Shulou(Shulou.com)05/31 Report--

This article mainly introduces "how Python uses Spacy for word segmentation". In daily operation, I believe many people have doubts about how Python uses Spacy for word segmentation. The editor consulted all kinds of materials and sorted out simple and easy-to-use operation methods. I hope it will be helpful to answer the doubts about "how Python uses Spacy for word segmentation". Next, please follow the editor to study!

Description

1. Spacy language model contains some powerful text analysis functions, such as part of speech tagging and named entity recognition.

2. After importing spacy-related modules, you need to load the Chinese processing package. Then read the novel data, nlp deals with The Demi-Gods & Semi-Devils 's novels, including word segmentation, quantification, part of speech tagging, grammatical analysis, named entity recognition, using symbols / separated novels. Finally, the is_stop function is used to determine whether the word in the word is an invalid word, and after deleting the invalid word, the result is written to the txt file.

Example

Import spacyimport pandas as pdimport timefrom spacy.lang.zh.stop_words import STOP_WORDS nlp = spacy.load ('zh_core_web_sm') def fenci_stopwords (data,newdata1): fenci = [] qc_stopwords = [] article = pd.read_table (data,encoding= "utf-8") start1 = time.time () with open (newdata1,'w' Encoding='utf-8') as F1: for i in article ["The Demi-Gods & Semi-Devils"]: # participle doc = nlp (I) result1 ='/ '.join ([t.text for t in doc]) fenci.append (result1) for j in fenci:# remove the stop word words = nlp.vocab [j] if words.is_stop = = False: qc_stopwords.append (j) result2 ='/ '.join (qc_stopwords) f1.write (result2) end1 = time.time () return end1-start1 so far The study on "how to use Spacy for word segmentation in Python" is over. I hope I can solve your doubts. The collocation of theory and practice can better help you learn, go and try it! If you want to continue to learn more related knowledge, please continue to follow the website, the editor will continue to work hard to bring you more practical articles!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.