Shulou(Shulou.com)06/01 Report--

This article mainly explains "how to use LuceneD's API to merge multiple index files", the content of the article is simple and clear, easy to learn and understand, the following please follow the editor's ideas slowly in depth, together to study and learn "how to use LuceneD's API to merge multiple index files"!

Lucene's index system is a write-exclusive, read-shared structure, which means that when we use multiple threads to add an index, the performance will not be significantly improved, so only one thread can write to the index at any time, and the security of this operation comes from Lucene's unique locking mechanism (when the write operation is in progress. We can see a lock file named write.lock in the index root directory of Lucene. If multiple different IndexWriter writes to the index at the same time, an exception of lock overlap will be thrown, so the special index structure of Lucene determines that it can only use one IndexWriter to add the index.

Even if the Lucen is limited to one thread for write operation, the write performance of Lucene is very efficient, especially after Lucene4.x. We can tune some parameters according to the hardware environment of our server. Using the characteristics of the previous batch processing, we can greatly improve the write performance.

As mentioned earlier, Lucene can only be written with one thread, so what if we want to use multithreaded writes to speed up?

The answer is yes, although the Lucene limit can only be written by one thread, but this restriction only refers to the restriction on one index file. We can take a compromise approach, using multiple threads to write to multiple index folder directories, and finally merging these index files to improve the index speed. Lucene's API also supports the merging of multiple index files, so we use this way to build the index. It can also greatly improve index performance, which is especially suitable for indexing the data of the database. We can use paged reading to build the index by a fixed number of threads.

Merge operation most of the time requires our data structure to be consistent, of course, Lucene is a document-type loose storage structure, a document can also store its own unique fields, and other documents, there is no, but since we need to merge, then most of the structure is required to be consistent, otherwise two completely different types of indexes, merged together is also illogical.

To demonstrate merging, two indexes are created, and then the two indexes are merged. The screenshot is as follows:

The core code for the merge is as follows:

/ * @ author Qin Dongliang * lucene Technology Exchange Group: 324714439 * Test the method of * merging between multiple indexes * * / public static void combineMoreIndex () {try {Directory d1=FSDirectory.open ("E:\\ 1\\ a")) / / Open the path Directory d2=FSDirectory.open where Index 1 is stored (new File ("E:\\ 2\\ a"); / / Open the path Directory d3=FSDirectory.open where Index 2 is stored (new File ("E:\\ 3\\ ab"); / / merge into Index 3 IndexWriter writer=new IndexWriter (D3, new IndexWriterConfig (Version.LUCENE_44, new IKAnalyzer () Writer.addIndexes (D1 and IndexReader); / input their respective Diretory or IndexReader to merge writer.commit (); / / submit index writer.close (); System.out.println ("merge index completed.");} catch (Exception e) {e.printStackTrace () }}

For the third index generated, the screenshot is as follows:

Let's take a look at the data changes before the merge, the 1 minute 2 index and the 3 index after the merge.

System.out.println ("= 1a ="); showAll ("E:\\ 1\\ a"); System.out.println ("= 2a ="); showAll ("E:\\ 2\\ a"); System.out.println ("= after merger ="); showAll ("E:\\ 3\\ ab")

The output is as follows. Notice that there is a field with a date of null, which indicates that the document has no date.

China date: > nullb=== > France date: > China date: = > nulld=== > UK date: = = > null=2a=q=== > China date: = > nullw=== > France date: = > China date: = > nulld=== > UK date: = > China date: = > nulld=== > UK date: = > China date: = > nullb=== > France date: = > China date: = = > nulld=== > UK date: = > nullq=== > China date: = = > nullw=== > French date: = > China date: = = > 1389783980586r date = > China date Date: nulld=== > UK date: null Thank you for your reading The above is the content of "how to use LuceneD's API to merge multiple index files". After the study of this article, I believe you have a deeper understanding of how to use LuceneD's API to merge multiple index files. Here is, the editor will push for you more related knowledge points of the article, welcome to follow!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.