World's Best AI Learning Platform with profoundly Demanding Certification Programs
Designed by IITian's, only for AI Learners.
Download our e-book of Introduction To Python
4 (4,001 Ratings)
218 Learners
Neha Kumawat
5 months ago
import urllib3
http = urllib3.PoolManager()
resp = http.request('GET', 'http://tutorialspoint.com/robots.txt')
print resp.data
# get the status of the response
print resp.status
User-agent: *
Disallow: /tmp
Disallow: /logs
Disallow: /rate/*
Disallow: /cgi-bin/*
Disallow: /videotutorials/video_course_view.php?*
Disallow: /videotutorials/course_view.php?*
Disallow: /videos/*
Disallow: /*/*_question_bank/*
Disallow: //*/*/*/*/src/*
200
import urllib3
http = urllib3.PoolManager()
resp = http.request('GET', 'http://tutorialspoint.com/robot.txt')
print resp.data
# get the status of the response
print resp.status
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>403 Forbidden</title>
</head><body>
<h1>Forbidden</h1>
<p>You don't have permission to access /robot.txt
on this server.</p>
</body></html>
403