Elasticsearch batch storage bulk performance instance test

This article focuses on "elasticsearch batch storage bulk performance example testing", interested friends may wish to take a look. The method introduced in this paper is simple, fast and practical. Now let the editor to take you to learn the "elasticsearch batch storage bulk performance example test" it!

Background

My requirement is very simple, is to use httpCilent components, through the REST batch interface bulk to submit some data. It took 0.843 seconds to complete 1W pieces of data import. I could have happily handed over the job here, but I did another thing. I ran 1W pieces of data with the Java native client and found that the import speed was 0.359 seconds. Whatever. get the job done. But, but I struggled, as a purposeful, thoughtful programmer, I struggled. The time gap is a little big. I suspected there was something wrong with the code. I used the official JEST client to try the speed again. It's not fast to find out. Then, my psychology has been entangled with a question, which is better to import in bulk? Just did a series of tests. It would have been fine to test a Http, but there is another but. I can't stop at all.

Test result

The unit is seconds, the amount of data is 1W, and the total size is 672 kb. HttpClient v1 is written in the stupidest way. V2s is added to the thread pool. Initialize Client first. The speed is averaged three times. The client of Java has 0.2 seconds when it is fast, that is, more than 5W records per second. A bit of an accident occurred while testing 10W records. XContentBuilder ate too much memory, that is, JSON parsing. My scum machine with 2G memory can't run. Changed the server to 3G memory test. The test data is simple, as follows.

{"create": {"_ index": "nq_test2", "_ type": "base"}\ n {"name": "value1"}\ n

Group one

0.359 java

0.843 httpClient v1

0.593 httpClient v2

0.719 jest

0.581 javascript

0.739 python

0.389 nest (.net client)

Group 2

10W 6825 kb

5.343 python

2.746 httpClient v2

2.145 java

Test conclusion

The REST interface of the http protocol is always slower than the Java native client of the TCP protocol. Accidentally, it is a Http component of C#, which has a good performance.

Test code

Python

Def test (): actions= [] for i in range (1Power100001): actions.append ({"_ index": 'nq_test2', "_ type": "python", "_ source": {"name": "value" + str (I)}}) starttime = time.clock () helpers.bulk (es,actions,chunk_size=50000) endtime = time.clock () print "cost =" + str (endtime-starttime)

JS

Var client = new elasticsearch.Client ({host: 'http://myes:9200', sniffOnStart: false, sniffInterval: 60000,}); function buildData () {var actions= []; for (var iTunes 1)