World's Best AI Learning Platform with profoundly Demanding Certification Programs
Designed by IITians, only for AI Learners.
Download our e-book of Introduction To Python
Related Blog
Top Discussion
Exception Type: JSONDecodeError at /update/ Exception Value: Expecting value: line 1 column 1 (char 0) How can I write Python code to change a date string from "mm/dd/yy hh: mm" format to "YYYY-MM-DD HH: mm" format? How to leave/exit/deactivate a Python virtualenvironment What is the process of executing a Python script within Power Automate? Change the hyperparameter means ? Why is my f1_scores different when i calculate them manually vs output by sklearn.metrics How can a sentence be matched to a cluster of keywords? How Python interpreter Works? Join Discussion
Top Courses
Master's In Artificial Intelligence Job Guarantee Program
4.5 (1,292 Ratings)
559 Learners
Webinars
Why You Should Learn Data Science in 2023?
Jun 8th (7:00 PM) 289 Registered
Python - Reading HTML Pages
Raj Mehta
a year ago
Table of Contents
- Introduction
- Install Beautifulsoup
- Reading the HTML file
- Extracting Tag Value
- Extracting All Tags
Introduction
Today in this article, we are going to learn about how we can handle HTML file using python very easily.
Python provides us with a library known as Beautifulsoup. Using this library, we can search for the values of HTML tags and get specific data like the title of the page and the list of headers in the page.
Install Beautifulsoup
Use the Anaconda package manager to install the required package and its dependent packages.
conda install BeaustifulsoapReading the HTML file
In the below example we make a request to an URL to be loaded into the python environment. Then use the HTML parser parameter to
read the entire HTML file. Next, we print the first few lines of the HTML page.
import urllib.request
from bs4 import BeautifulSoup
request_url = urllib.request.urlopen('https://insideaiml.com/')
html_doc = request_url.read()
# Parse the html file
soup = BeautifulSoup(html_doc, 'html.parser')
# Format the parsed html file
strhtm = soup.prettify()
# Print the first few characters
print (strhtm[:225])
When we execute the above code, it produces the following result.
<!DOCTYPE html>
<html lang="en">
<head>
<!-- Google Tag Manager -->
<!-- <script>(function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':
new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],
Extracting Tag Value
We can extract tag value from the first instance of the tag using the following code.
import urllib.request
from bs4 import BeautifulSoup
request_url = urllib.request.urlopen('https://insideaiml.com/')
html_doc = request_url.read()
# Parse the html file
soup = BeautifulSoup(html_doc, 'html.parser')
print (soup.title)
print(soup.title.string)
When we execute the above code, it produces the following result.
InsideAIML
InsideAIML
Extracting All Tags
We can extract tag value from all the instances of a tag using the following code.
import urllib.request
from bs4 import BeautifulSoup
request_url = urllib.request.urlopen('https://insideaiml.com/python/python_overview')
html_doc = request_url.read()
# Parse the html file
soup = BeautifulSoup(html_doc, 'html.parser')
for x in soup.find_all('b'):
print(x.string)
When we execute the above code, it produces the following result.
Python is Interpreted
Python is Interactive
Python is Object-Oriented
Python is a Beginner's Language
Easy-to-learn
Easy-to-read
Easy-to-maintain
A broad standard library
Interactive Mode
Portable
Extendable
Databases
GUI Programming
Scalable
I hope you enjoyed reading this article and finally, you came
to know about Python - Reading HTML Pages.
For more such blogs/courses on data science, machine
learning, artificial intelligence and emerging new technologies do visit us at InsideAIML.
Thanks for reading…
Happy Learning…