Shulou(Shulou.com)06/01 Report--

This article shows you how to understand the introduction and indexing process of Lucene. The content is concise and easy to understand, which will definitely brighten your eyes. I hope you can get something through the detailed introduction of this article.

Introduction and use of Lucene

Introduction to Lucene

Lucene is a full-text search engine developed by Doug Cutting on weekends. Doug Cutting led the development of Lucene and Nutch. Based on the Hadoop separated from Nutch, a Hadoop project team was set up in Yahoo to continue to promote the research and development of Hadoop. Lucene is not an application software, but is similar to a full-text retrieval function library. It provides a basic function interface for the application software to realize the document to the retrieval API.

What is full-text search?

Consider the data type first.

1. Structured data can have a fixed format, length of data. Such as a database.

two。 Unstructured data has no fixed length, no strict format division, no schema information and so on. Such as logs, emails, documents, etc.

The database method to solve the problem of unstructured data retrieval is also possible. But the efficiency is low. Then the usage scenario for this kind of log is basically content-based retrieval.

Application scenario of full-text Retrieval

1. Log analysis: analysis of log data with no detailed structure defined. For example, query based on keywords. (at present, our log data is in this way, the log output is relatively random, and some keywords are used to retrieve abnormal problems in the log.)

two。 Search engine search: full-text search of search engine is a typical scene, which also uses keywords to retrieve web pages, documents and other related content captured by crawlers.

3. E-commerce search: search related goods based on category, title, content, etc.

Full-text Retrieval based on Lucene

Based on the above figure, we get the whole process of full-text search using Lucene if needed.

Decomposition based on the image above

Original document:

The original document can be a web page, email, word document and so on. Then Lucene does not provide related web page data crawling, but Doug Cutting developed Nutch to provide the function of web crawlers. Here are some common crawler tools

1.1:Nutch: a web crawler tool developed by Doug Cutting to achieve distributed web page data collection.

1.2:Scrapy: Python domain professional crawler development framework, has completed the commonly used crawler tools.

The web crawler library developed by 1.3:WebMagic:Java based on the idea of Scrapy has a very high popularity in the direction of Java. ...

Create a document object:

The document object is created so that the content of the document can be obtained by retrieval. For example, a search engine searches for "PHP is the best language in the world" to obtain a web page content, which can be defined as a document object. So here we define a web page as a document object (Document), and each document object contains a variety of Field (title, content, time, author, etc.). Of course, in the web page collection as far as possible to extract the Field, and similar to Google, Baidu and so on will have corresponding rules, so that the crawler program can identify where is the title, content and so on. But each document will have a unique address, such as the URL of the web page

Analyze the contents of the document:

Analyze the content of the document, that is, analyze the contents of various Field in the document, carry out word segmentation, case conversion, special symbol filtering, removal of stop words and so on to generate the final vocabulary unit, that is, a word.

For example:

PHP is the best language in the world.

The lexical units after word segmentation are:

PHP, the best in the world, language

Each word is called Term, and different Term is separated from different Document and Field. Term contains DOcumentID and word content.

Create an index:

The purpose of creating an index is to retrieve relevant documents, so the final Term of full-text retrieval is actually located to a Document. It's so simple to think that the Term contained in the index library basically has multiple DocumentId used to locate documents that can be retrieved by the Term.

The simple ones can be:

TemrsDocumentIdPHPdoc_1,doc_2,doc_3 World doc_1,doc_3 language doc_2,doc_3

At this point, the index of Lucene is built, and the index database can be retrieved through the retrieval Api provided by Lucene.

The above content is how to understand the introduction and indexing process of Lucene. Have you learned any knowledge or skills? If you want to learn more skills or enrich your knowledge reserve, you are welcome to follow the industry information channel.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.