World's Best AI Learning Platform with profoundly Demanding Certification Programs
Designed by IITian's, only for AI Learners.
Download our e-book of Introduction To Python
4 (4,001 Ratings)
218 Learners
Neha Kumawat
6 months ago
A Status-line
Zero or more header (General|Response|Entity) fields followed by CRLF
An empty line (i.e., a line with nothing preceding the CRLF)
indicating the end of the header fields
Optionally a message-body
Status-Line = HTTP-Version SP Status-Code SP Reason-Phrase CRLF
HTTP-Version = HTTP/1.1
import urllib3
http = urllib3.PoolManager()
resp = http.request('GET', 'http://tutorialspoint.com/robots.txt')
print resp.data
# get the status of the response
print resp.status
User-agent: *
Disallow: /tmp
Disallow: /logs
Disallow: /rate/*
Disallow: /cgi-bin/*
Disallow: /videotutorials/video_course_view.php?*
Disallow: /videotutorials/course_view.php?*
Disallow: /videos/*
Disallow: /*/*_question_bank/*
Disallow: //*/*/*/*/src/*
200