Shulou(Shulou.com)06/01 Report--

This article mainly explains "how to use Elasticsearch data synchronization tool". Friends who are interested may wish to have a look. The method introduced in this paper is simple, fast and practical. Let's let the editor take you to learn how to use Elasticsearch data synchronization tools.

Import maven coordinates com.bbossgroups.plugins bboss-elasticsearch-rest-kafka2x 5.9.9 compile build deployment preparation

You need to build the release version through gradle, and the gradle installation configuration reference documentation

Download the source project-based on gradle

Download the source project from the above address, then import idea or eclipse, and modify the importer logic according to your own requirements

Org.frameworkset.elasticsearch.imp.Kafka2ESdemo

If you need to test and debug the import function, you can run the main method of Kafka2ESdemo:

Public class Dbdemo {public static void main (String args []) {Kafka2ESdemo dbdemo = new Kafka2ESdemo (); boolean dropIndice = true;//CommonLauncher.getBooleanAttribute ("dropIndice", false); / / the default value dbdemo.scheduleTimestampImportData (dropIndice) is also specified;}.}

Modify the es configuration-kafka2x-elasticsearch\ src\ main\ resources\ application.properties

After you have modified the configuration, you can debug the function.

After the test and debugging pass, you can build a release runnable version: enter the command line mode and run the following gradle directive to package the release version under the root directory of the source code project kafka2x-elasticsearch

Release.bat

Run a job

After gradle is built successfully, a runnable zip package will be generated in the build/distributions directory, and the importer will be unzipped and run.

Linux:

Chmod + x restart.sh

. / restart.sh

Windows: restart.bat

Job jvm configuration

Modify jvm.options to set memory size and other jvm parameters

-Xms1g

-Xmx1g

Job parameter configuration

When using kafka2x-elasticsearch, in order to avoid constantly packaging and publishing data synchronization tools during debugging, you can configure some control parameters into the startup configuration file resources/application.properties, and then obtain the configuration parameters in the code by the following methods:

# tool main program mainclass=org.frameworkset.elasticsearch.imp.Kafka2ESdemo# parameter configuration # get the method in the code: CommonLauncher.getBooleanAttribute ("dropIndice", false); / / also specify the default value falsedropIndice=false

Get the parameter dropIndice method in the code:

Boolean dropIndice = CommonLauncher.getBooleanAttribute ("dropIndice", false); / / the default value false is also specified

In addition, you can configure some parameters that control job execution in resources/application.properties, such as the number of worker threads, the number of waiting queues, batch size, and so on:

QueueSize=50workThreads=10batchSize=20

Get and use the above parameters in the job execution method:

Int batchSize = CommonLauncher.getIntProperty ("batchSize", 10); / / also specify the default value int queueSize = CommonLauncher.getIntProperty ("queueSize", 50); / / also specify the default value int workThreads = CommonLauncher.getIntProperty ("workThreads", 10); / / also specify the default value importBuilder.setBatchSize (batchSize); importBuilder.setQueue (queueSize); / / set the bulk import thread pool waiting queue length importBuilder.setThreadCount (workThreads) / / set the number of worker threads in the batch import thread pool. I believe you have a better understanding of "how to use the Elasticsearch data synchronization tool". You might as well do it in practice. Here is the website, more related content can enter the relevant channels to inquire, follow us, continue to learn!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.