What is the preprocessing method of nlp Chinese data
This article mainly introduces "what is the nlp Chinese data preprocessing method". In the daily operation, I believe that many people have doubts about what the nlp Chinese data preprocessing method is. The editor consulted all kinds of materials and sorted out simple and easy-to-use operation methods. I hope it will be helpful for you to answer the doubts about "what is the nlp Chinese data preprocessing method?" Next, please follow the editor to study!
Data loading (default csv format)
Import pandas as pd
Datas = pd.read_csv (". / test.csv", header=0, index_col=0) # DataFrame
N_datas = data.to_numpy () # ndarray converted to numpy is easier to handle (personal preference)
Remove blank lines
Def delete_blank_lines (sentences):
Return [s for s in sentences if s.split ()]
No_line_datas = delete_blank_lines (n_datas)
Remove the number
DIGIT_RE = re.compile (r'\ daddy')
No_digit_datas = DIGIT_RE.sub ('', no_line_datas)
Def delete_digit (sentences):
Return [DIGIT_RE.sub (', s) for s in sentences]
Judge the form of a sentence (simple sentence or complex sentence)
STOPS = ['.' ,
Def is_sample_sentence (sentence):
Count = 0
For word in sentence:
If word in STOPS:
Count + = 1
If count > 1:
Return False
Return True
Remove Chinese and English punctuation
From string import punctuation
Import re
Punc = punctuation + u'
Def delete_punc (sentences):
Return [re.sub (r "[{}] +" .format (punc),', s) for s in a]
Remove English (only Chinese characters)
ENGLISH_RE = re.compile (r'[a-zA-Z] +')
Def delete_e_word (sentences):
Return [ENGLISH_RE.sub (', s) for s in sentences]
Remove garbled codes and special symbols
Use regular expressions to remove related useless symbols and garbled codes
# this operation can remove all symbols, punctuation and English, as punctuation may be needed to further determine whether the sentence is a simple sentence, so this operation can be used last. Price http://www.zzkdfk.com/ for gynecological examination in Zhengzhou
SPECIAL_SYMBOL_RE = re.compile (r'[^\ w\ s\ u4e00 -\ u9fa5] +')
Def delete_special_symbol (sentences):
Return [SPECIAL_SYMBOL_RE.sub (', s) for s in sentences]
Chinese word segmentation
# using jieba
Def seg_sentences (sentences):
Cut_words = map (lambda s: list (jieba.cut (s)), sentences)
Return list (cut_words)
# use pyltp participle
Def seg_sentences (sentences):
Segmentor = Segmentor ()
Segmentor.load ('. / cws.model') # loads word segmentation model parameters
Seg_sents = [list (segmentor.segment (sent)) for sent in sentences]
Segmentor.release ()
Return seg_sents
Remove stop words
# stop word list needs to be downloaded by yourself
Stopwords = []
Def delete_stop_word (sentences):
Return [[word for word in s if word not in stopwords] for s in sentences]
At this point, the study on "what is the preprocessing method of nlp Chinese data" is over. I hope to be able to solve your doubts. The collocation of theory and practice can better help you learn, go and try it! If you want to continue to learn more related knowledge, please continue to follow the website, the editor will continue to work hard to bring you more practical articles!
