Shulou(Shulou.com)06/01 Report--

This article introduces the knowledge of "how to use python pkuseg tools". In the operation of actual cases, many people will encounter such a dilemma, so let the editor lead you to learn how to deal with these situations. I hope you can read it carefully and be able to achieve something!

The Chinese word segmentation tool in the subdivision field is easy to use and improves the accuracy of word segmentation compared with the existing open source word segmentation tools.

As a test environment, Linux tests the accuracy of different toolkits on news data (MSRA), mixed text (CTB8) and network text (WEIBO) data.

Detailed field training and test results

The following is a comparison of different datasets:

Cross-domain test results

We choose the training set of mixed-domain CTB8 corpus for training, and test it in other fields to simulate the segmentation effect of the model on "black box data". The reason for choosing CTB8 corpus is that CTB8 belongs to mixed corpus and the ideal effect will be better; and in the test, we find that all toolkits can achieve higher average results in cross-domain testing of the model trained on CTB8. The following are the results of the cross-domain test:

Pkuseg has the following characteristics:

Multi-domain participle. Different from the previous general Chinese word segmentation tools, this toolkit is also committed to providing personalized pre-training models for data in different fields. According to the domain characteristics of the text to be segmented, users are free to choose different models. At present, we support word segmentation pre-training models in news, online text and mixed fields, and we also plan to launch more detailed domain pre-training models in the near future, such as medicine, tourism, patents, novels and so on.

Higher accuracy of word segmentation. Compared with other word segmentation toolkits, pkuseg can achieve higher word segmentation accuracy when using the same training data and test data.

Support user self-training model. Support users to use new label data for training.

Mode of use

Code example 1: use default model and default dictionary word segmentation

Import pkuseg

Seg = pkuseg.pkuseg () # load the model with the default configuration

Text = seg.cut ('I love Tiananmen Square in Beijing') # for participle

Print (text)

Result

Loading model

Finish

['I', 'Love', 'Beijing', 'Tiananmen Square']

This is the end of "how to use python pkuseg tools". Thank you for reading. If you want to know more about the industry, you can follow the website, the editor will output more high-quality practical articles for you!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.