Shulou(Shulou.com)06/02 Report--

In this issue, the editor will bring you about how to update the incremental index in sphinx. The article is rich in content and analyzes and narrates it from a professional point of view. I hope you can get something after reading this article.

The idea of realization is:

Need to prepare: 1 "create a table to record the maximum id value of the data." 2 "two indexes

1. First insert a count table and two index tables in MySQL

CREATE TABLE sph_counter (counter_id INTEGER PRIMARY KEY NOT NULL, max_doc_id INTEGER NOT NULL)

two。 Modify sphinx.conf

Source main_src {

Type = mysql

Sql_host = localhost

Sql_user = yourusername

Sql_pass = yourpassword

Sql_db = test / / the database you use

Sql_port = 3306 / / Port used. Default is 3306.

Sql_query_pre = SET NAMES utf8

Sql_query_pre = SET SESSION query_cache_type=OFF

The following statement updates the max_doc_id in the sph_counter table.

Sql_query_pre = REPLACE INTO sph_counter SELECT 1, MAX (id) FROM documents

Sql_query = SELECT id, group_id, UNIX_TIMESTAMP (date_added) AS date_added, title,content FROM documents WHERE id (SELECT max_doc_id FROM sph_counter WHERE counter_id=1)

}

# Primary Index

Index main {

Source = main_src

Path = / path/to/main

# example: / usr/local/sphinx/var/data/main

Charset_type = utf-8 # this must be set to support Chinese.

# Chinese search path

Chinese_dictionary = / usr/local/mmseg3/etc/

}

# delta can copy the primary index all, and then change the source and path as follows

# incremental indexing

Index delta: main {

Source = delta_src

Path = / path/to/delta

# example: / usr/local/sphinx/var/data/delta...

# Chinese search path

Chinese_dictionary = / usr/local/mmseg3/etc/

}

Other configurations can be used by default, if you set the index for distributed retrieval, then change the corresponding index name.

3. Re-index:

/ usr/local/coreseek/bin/indexer-c / usr/local/coreseek/etc/sphinx.conf-- all-- rotate

The next question is how to merge the incremental index with the primary index.

4. Index merging

Merging two existing indexes is sometimes more effective than re-indexing all data, although, when indexes are merged, the two indexes to be merged will be read into memory once, and the merged content needs to be written to disk once, that is, merging the two indexes of 100GB and 1GB will result in the IO operation of 202GB

Command prototype: indexer-- merge DSTINDEX SRCINDEX [--rotate] merges SRCINDEX into DSTINDEX, so only DSTINDEX will change, and the-- rotate parameter is required if both indexes are providing services. For example: merge delta into main.

Indexer-merge main delta

5. Automatic index update

You need to use scripts.

Create two scripts: build_main_index.sh and build_delta_index.sh.

Build_main_index.sh:

#! / bin/sh

# stop the running searchd

/ usr/local/sphinx/bin/searchd-c / usr/local/sphinx/etc/mersphinx.conf-- stop > > / usr/local/sphinx/var/log/sphinx/searchd.log

# create a primary index

/ usr/local/sphinx/bin/indexer-c / usr/local/sphinx/etc/mersphinx.conf main > > / usr/local/sphinx/var/log/sphinx/mainindex.log

# start the searchd daemon

/ usr/local/sphinx/bin/searchd > > / usr/local/sphinx/var/log/sphinx/searchd.log

Build_delta_index.sh

#! / bin/sh

# stop the sphinx service and redirect the output

/ usr/local/sphinx/bin/searchd-stop > > / usr/local/sphinx/var/log/sphinx/searchd.log

# re-index delta to redirect the output

/ usr/local/sphinx/bin/indexer delta-c / usr/local/sphinx/etc/sphinx.conf > > / usr/lcoal/sphinx/var/log/sphinx/deltaindex.log

# merge delta into main

/ usr/local/sphinx/bin/indexer-merge main delta-c / usr/local/sphinx/etc/sphinx.conf > > / usr/lcoal/sphinx/var/log/sphinx/deltaindex.log

# start the service

/ usr/local/sphinx/bin/searchd > > / usr/local/sphinx/var/log/sphinx/searchd.log

After the script is written, you need to compile chmod + x filename to run it. That is,

Chmod + x build_main_index.sh

Chmod + x build_delta_index.sh

Finally, we need the script to run automatically to achieve that the delta index is rebuilt every 5 minutes, and the main index is rebuilt only at 2: 30 midnight.

Using the crontab command, there are two places to refer to the crontab crontab file

Crontab-e to edit the crontab file, if not previously used, it will be an empty file. Write down the following two sentences

* / 30 * / bin/sh / usr/local/sphinx/etc/build_delta_index.sh > / dev/null 2 > & 1

30 2 * / bin/sh / usr/local/sphinx/etc/build_main_index.sh > / dev/null 2 > & 1

The first is to run the build_delta_index.sh script under / usr/local/sphinx/etc/ every 30 minutes to redirect the output.

The second is to run the build_main_inde.sh script under / usr/local/sphinx/etc at 2: 30 a. M. every day to redirect the output.

The settings for the previous five values are described in detail in the crontab file above. For an explanation of redirection, see the top Crontab note, which is also described by crontab.

After saving: restart the service

[root@test1 init.d] # service crond stop

[root@test1 init.d] # service crond start

Or

/ etc/init.d/crontab start

It is OK to carry out the planned task. The merged index is suitable for the situation where there are few modifications to the database. If you modify the database data, you need to read the database again to generate the primary index, and make a plan according to the business requirements.

The above is how to update the incremental index in the sphinx shared by the editor. If you happen to have similar doubts, you might as well refer to the above analysis to understand. If you want to know more about it, you are welcome to follow the industry information channel.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.