Shulou(Shulou.com)06/03 Report--

Natural language processing (NLP) is a field of computer science, artificial intelligence and linguistics concerned with the interaction between computers and human (natural) language. Natural language processing is an important direction in the field of computer science and artificial intelligence. It studies various theories and methods that can realize effective communication between people and computers using natural language. Natural language processing is a science that integrates linguistics, computer science and mathematics. Therefore, the research in this field will involve natural language, that is, the language that people use every day, so it is closely related to the study of linguistics, but there are important differences. Natural language processing is not to study natural language in general, but to develop a computer system, especially the software system, which can effectively realize natural language communication. So it's part of computer science.

Natural language processing technology is a general term for all technologies related to computer processing of natural language. its purpose is to make computers understand and accept instructions inputted by human beings in natural language and complete the translation function from one language to another. the research of natural language processing technology can enrich the research content of computer knowledge processing and promote the development of artificial intelligence technology.

The big fast NLP module is a component of the big fast big data integrated platform. Users can refer to this component to effectively carry out natural language processing, such as summarizing articles, semantic discrimination and improving the accuracy and effectiveness of content retrieval.

Nowadays, natural language processing is studied not only as the core topic of artificial intelligence, but also as the core topic of the new generation of computers. From the point of view of knowledge industry, expert system, database, knowledge base, computer aided design system (CAD), computer aided instruction system (CAI), computer aided decision system, office automation management system, intelligent robot and so on all need to be processed by natural language. The natural language understanding system with text comprehension ability can be used in the fields of automatic machine translation, information retrieval, automatic indexing, automatic summarization, automatic story writing and so on, all of which can be processed by our tool class DKNLPBase.

Standard participle

Method signature: List StandardTokenizer.segment (String txt)

Return: list of participles.

Signature parameter description: txt: the statement to be participle.

Example: the following example verifies that the fifth participle in a paragraph is AlphaGo.

Public void testSegment () throws Exception

{

String text = "goods and services"

List termList = DKNLPBase.segment (text)

AssertEquals ("Commodity", termList.get (0) .word)

AssertEquals ("and", termList.get (1) .word)

AssertEquals ("Services", termList.get (2) .word)

Text = "Ke Jie explains that Lee se-dol VS AlphaGo's second inning ends like this."

TermList = DKNLPBase.segment (text)

AssertEquals ("AlphaGo", termList.get (5) .word); / / able to identify "AlphaGo"

}

Keyword extraction

Method signature: List extractKeyword (String txt,int keySum)

Return: keyword list.

Signature parameter description: txt: the statement to extract keywords, and the number of keywords to be extracted by keySum

Example: give a paragraph to extract a key word is "programmer".

Public void testExtractKeyword () throws Exception

{

String content = "programmers (English Programmer) are professionals engaged in program development and maintenance." +

Programmers are generally divided into programmers and programmers.

"but the line between the two is not very clear, especially in China." +

"Software practitioners are divided into junior programmers, senior programmers and systems" +

"there are four categories of analysts and project managers."

List keyword = DKNLPBase.extractKeyword (content, 1)

AssertEquals (1, keyword.size ())

AssertEquals (programmer, keyword.get (0))

}

Phrase extraction

Method signature: List extractPhrase (String txt, int phSum)

Return: phrase

Signature parameter description: txt: statement to extract phrase, number of phSum phrase

Example: give a passage that represents the five phrases of the article, the first of which is an algorithm engineer.

Entering the 21 century, we have entered the era of massive information with the Internet as the main symbol, most of which are expressed in natural language. On the one hand, the vast amount of information also provides more "material" for computer learning of human language, on the other hand, it also provides a broader application stage for natural language processing. For example, as an important application of natural language processing, search engine has gradually become an important tool for people to obtain information, and search engine giants represented by Baidu and Google have emerged. Machine translation has also entered the homes of ordinary people from the laboratory. Google, Baidu and other companies have provided machine translation and auxiliary translation tools based on massive network data. Chinese based on natural language processing (input methods such as Sogou, Microsoft, Google, etc.) has become a necessary tool for computer users; computers and mobile phones with speech recognition are also popular to help users work and study more effectively. In short, with the popularity of the Internet and the emergence of huge amounts of information, natural language processing is playing a more and more important role in people's daily life.

However, at the same time, we are faced with a grim fact, that is, how to make effective use of massive information has become an overall bottleneck restricting the development of information technology. Natural language processing has inevitably become a new strategic commanding point for the long-term development of information science and technology. At the same time, people gradually realize that it is impossible to learn language knowledge from massive data quickly and effectively by relying solely on statistical methods, and only by giving full play to the respective advantages of rule-based rationalism and statistics-based empiricism at the same time, the two complement each other in order to carry out natural language processing better and faster.

As a new discipline which is less than a century old, natural language processing is developing by leaps and bounds. Reviewing the development of natural language processing, it is not plain sailing, there have been troughs, but also have the best part. Now we are facing new challenges and opportunities. For example, at present, the network search engine basically stays in keyword matching, lack of in-depth natural language processing and understanding. At present, speech recognition, text recognition, question answering system and machine translation can only reach a very basic level. The journey is long. Natural language processing, as a highly intersecting new discipline, whether it is exploring the nature of nature or putting it into practical application, there will be unexpected surprises and unusually rapid development in the future.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.