World's Best AI Learning Platform with profoundly Demanding Certification Programs
Designed by IITians, only for AI Learners.
Designed by IITians, only for AI Learners.
New to InsideAIML? Create an account
Employer? Create an account
Download our e-book of Introduction To Python
4.5 (1,292 Ratings)
559 Learners
Neha Kumawat
a year ago
import urllib3
http = urllib3.PoolManager()
resp = http.request('GET', 'http://tutorialspoint.com/robots.txt')
print resp.data
# get the status of the response
print resp.status
User-agent: *
Disallow: /tmp
Disallow: /logs
Disallow: /rate/*
Disallow: /cgi-bin/*
Disallow: /videotutorials/video_course_view.php?*
Disallow: /videotutorials/course_view.php?*
Disallow: /videos/*
Disallow: /*/*_question_bank/*
Disallow: //*/*/*/*/src/*
200
import urllib3
http = urllib3.PoolManager()
resp = http.request('GET', 'http://tutorialspoint.com/robot.txt')
print resp.data
# get the status of the response
print resp.status
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>403 Forbidden</title>
</head><body>
<h1>Forbidden</h1>
<p>You don't have permission to access /robot.txt
on this server.</p>
</body></html>
403