World's Best AI Learning Platform with profoundly Demanding Certification Programs
Designed by IITian's, only for AI Learners.
Download our e-book of Attention Mechanism
Related Blog
Top Discussion
How to use Enum in python? What is a Bag-of-Words Model ? What is use of rank() function? What are local and global scope? Backpropagation: In second-order methods, would ReLU derivative be 0? and what its effect on training? How to Choose a classification algorithm for particular problem? What is TF/IDF vectorization? What is file hashing in python? Join Discussion
Top Courses
Machine Learning with Python & Statistics
5 (4,001 Ratings)
220 Learners
Webinars
Join AI Webinar With Top AI Experts on 20th June 2021 at 06 PM
.jpg)
Jun 20th (6:00 PM) 690 Registered
Python - Text Munging
Neha Kumawat
a year ago
Munging in general means cleaning up anything messy by transforming them. In our case we will see how we can transform text to get some result which gives us some desirable changes to data. At a simple level it is only about transforming the text we are dealing with.
Example
In the below example we plan to shuffle and then rearrange all the letters of a sentence except the first and the last one to get the possible alternate words which may get generated as a mis-spelled word during writing by a human. This rearrangement helps us in
import random
import re
def replace(t):
inner_word = list(t.group(2))
random.shuffle(inner_word)
return t.group(1) + "".join(inner_word) + t.group(3)
text = "Hello, You should reach the finish line."
print re.sub(r"(\w)(\w+)(\w)", replace, text)
print re.sub(r"(\w)(\w+)(\w)", replace, text)
When we run the above program we get the following output −
Hlleo, You slouhd raech the fsiinh lnie.
Hlleo, You suolhd raceh the fniish line.
Here you can see how the words are jumbled except for the first and the last letters. By taking a statistical approach to wrong spelling we can decided what are the commonly mis0spelled words and supply the correct spelling for them.