Clever Geek Handbook

Removing non-ASCII from a corpus

I am using NLTK for my project. However, if there is a non-ascii word like "â€ĸ". NLTK can't fake it. I am using nltk.word_tokenize

as a tokenizer. How to remove such words from the whole body or make the tokenizer aware of such words?

+3


source to share


1 answer


Use the below code to remove nonascii from your corpus:



ip=open(nonascii.txt,'r')
#Edit should be in w mode
op=open(ascii.txt,'w')
for line in ip:
        line=line.strip().decode("ascii","ignore").encode("ascii")
        if line=="":continue
        op.write(line)
ip.close()
op.close()
+3


source






More articles:


All Articles