Shulou(Shulou.com)06/01 Report--

I. background

In the past few days, we have done a benchmark stress test on all the basic components. At present, none of our open source basic components have been tested. Developers often ask, "how much can our RDS and MongoDB clusters resist?" at this time, I can't reply, because I don't know, although a database cluster can resist a lot, under the condition of fixed software and hardware configuration. It has a lot to do with business scenarios. If the amount of data is small and the query SQL is simple, then the throughput is naturally very high. If the amount of data is very large and it is all complex SQL, then the throughput will not go up naturally. But since people ask, they must hope to have an answer. If you say you don't know, it will give people a feeling of unreliability, so do a benchmark stress test. We know how much throughput our cluster can have in a particular scenario, and only when we have a clear idea can we give confidence to others. This week in the stress test MongoDB, Google, MongoDB stress test tools are very few, there are several articles are introduced through the YCSB stress test MongoDB, ask Sister Dan (logical thinking chief DBA) to recommend a MongoDB stress test tool, Dan Sister also recommends YCSB, all right, then it, open up.

2. Environment description 1. MongoDB cluster configuration (a fragmented shard cluster)

2. MongoDB version

4.0.4-62-g7e345a7

4. System and kernel version

CentOS Linux release 7.5.1804 (Core) 3.10.0-862.14.4.el7.x86_643, YCSB version

YCSB-0.16.0-RC1.

4. Test description

Installation 1, jdk and maven installation reference official

Https://github.com/brianfrankcooper/YCSB/tree/master/mongodb

2. Install YCSBwget https://github.com/brianfrankcooper/YCSB/archive/0.16.0-RC1.tar.gztar-zxvf YCSB-0.16.0-RC1.tar.gzcd YCSB-0.16.0-RC1/mvn clean package-Dmaven.test.skip=true

PS:

Maven download depends on × × during installation. If there is a package that failed to install, you need to download manual installation on a server that can. For example, mongodb-async-driver-2.0.1.jar requires × × ×. Here is the manual installation method.

Download the jar package manually

Wget http://www.allanbank.com/repo/com/allanbank/mongodb-async-driver/2.0.1/mongodb-async-driver-2.0.1.jar

B, pressure package, view groupId, artifactId, version from pom.xml file

C, manual installation

Mvn install:install-file-Dfile=/tmp/mongodb-async-driver-2.0.1.jar-DgroupId=com.allanbank-DartifactId=mongodb-async-driver-Dversion=2.0.1-Dpackaging=jarmvn-pl com.yahoo.ycsb:mongodb-binding-am clean package IV, pressure test 1, preparation of pressure test file

There are many files used in the stress test files in the workloads directory, and we copy one from one of them, and the editor adds our own definition.

Vim workloads/2000wongodb.url=mongodb://root:123456@172.21.244.101:27000mongodb.writeConcern=normaltable=chj_2000wrecordcount=20000000operationcount=50000000readallfields=truereadproportion=0updateproportion=0scanproportion=0insertproportion=1requestdistribution=zipfianworkload=com.yahoo.ycsb.workloads.CoreWorkload

Each parameter of the stress test file for YCSB is explained as follows:

Fieldcount: number of fields per record (default: 10) fieldlength: length of each field (default: 100) readallfields: whether to read all fields true or read one field false (default: true) readproportion: read job ratio (default: 0.95) updateproportion: update job ratio (default: 0.05) insertproportion: insert job ratio (default: 0) scanproportion: scan job ratio (default: 0) readmodifywriteproportion: read a record, modify it and write it back (default: 0) requestdistribution: requested distribution rule uniform Zipfian or latest (default: uniform) maxscanlength: maximum number of records in scan job (default: 1000) scanlengthdistribution: distribution rule between 1 and maximum number of scan records (default: uniform) insertorder: rule ordered or hashed (default: hashed) operationcount: operations performed. Maxexecutiontime: maximum time to perform an operation Of course, if it does not exceed this time, it is mainly run time. Table: the name of the test table (default: usertable) recordcount: the number of records loaded into the database (default: 0) 2, create data, also test write performance

. / bin/ycsb load mongodb-threads 100-P workloads/2000w

The output result shows that

[OVERALL], RunTime (ms), 37182 # time taken to load data (Ms) [OVERALL], Throughput (ops/sec), 13447.367005540314 # Throughput of load operation (ops/sec) [TOTAL_GCS_PS_Scavenge], Count, 37 [TOTAL_GC_TIME_PS_Scavenge], Time (ms), 146 [TOTAL_GC_TIME_%_PS_Scavenge], Time (%), 0.3926631165617772 [TOTAL_GCS_PS_MarkSweep], Count, 0 [TOTAL_GC_TIME_PS_MarkSweep] Time (ms), 0 [TOTAL_GC_TIME_%_PS_MarkSweep], Time (%), 0.0 [TOTAL_GCs], Count, 37 [TOTAL_GC_TIME], Time (ms), 146 [TOTAL_GC_TIME_%], Time (%), 0.3926631165617772 [CLEANUP], Operations, 64 [CLEANUP], AverageLatency (us), 422.09375 [CLEANUP], MinLatency (us), 0 [CLEANUP], MaxLatency (us), 26911 [CLEANUP], 95thPercentileLatency (us), 3 [CLEANUP], 99thPercentileLatency (us), 30 [INSERT] Total number of insert operations performed by Operations, 500000 # [INSERT], AverageLatency (us), 4658.931652 # average delay per insert operation (microsecond) [INSERT], MinLatency (us), 831 # minimum delay for all insert operations (microsecond) [INSERT], MaxLatency (us), 1784831 # maximum delay for all insert operations (microsecond) [INSERT], 95thPercentileLatency (us), 9711 # 95% insert operation delay less than 9 milliseconds [INSERT], 99thPercentileLatency (us) 17903 # 99% of insert operation latency is less than 17 milliseconds [INSERT], Return=OK, 5000003, pressure test

The mixed operation of read-only and read-write is simulated by adjusting the ratio of read and update in the pressure test file.

. / bin/ycsb run mongodb-threads 100-P workloads/2000w

[OVERALL], RunTime (ms), 1735408 [OVERALL], Throughput (ops/sec), 2881.1668495247227 [TOTAL_GCS_PS_Scavenge], Count, 3975 [TOTAL_GC_TIME_PS_Scavenge], Time (ms), 6180 [TOTAL_GC_TIME_%_PS_Scavenge], Time (%), 0.3561122226012557 [TOTAL_GCS_PS_MarkSweep], Count, 0 [TOTAL_GC_TIME_PS_MarkSweep], Time (ms), 0 [TOTAL_GC_TIME_%_PS_MarkSweep], Time (%) 3975 [TOTAL_GC_TIME], Time (ms), 6180 [TOTAL_GC_TIME_%], Time (%), 0.3561122226012557 [READ], Operations, 500346 [READ], AverageLatency (us), 2851.9638989819045 [READ], MinLatency (us), 696 [READ], MaxLatency (us), 646655 [READ], 95thPercentileLatency (us), 6991 [READ], 99thPercentileLatency (us), 23103 [READ], Return=OK, 500346 [CLEANUP], Operations, 10 [CLEANUP], AverageLatency (us) 3131.0 [CLEANUP], MinLatency (us), 1 [CLEANUP], MaxLatency (us), 31295 [CLEANUP], 95thPercentileLatency (us), 31295 [CLEANUP], 99thPercentileLatency (us), 31295 [UPDATE], Operations, 4499654 [UPDATE], AverageLatency (us), 3534.2083122391186 [UPDATE], MinLatency (us), 704 [UPDATE], MaxLatency (us), 1078271 [UPDATE], 95thPercentileLatency (us), 11647 [UPDATE], 99thPercentileLatency (us), 27343 [UPDATE], Return=OK, 44996554

1. Server metrics, which mainly observe the utilization and latency of CPU, memory and disk IO. You can check the real-time situation through top and iostat tools.

2. MongoDB can view the real-time situation through the mongostat tool

Output description of mongostat

Inserts: inserts per second query: queries per second update: updates per second delete: deletions per second getmore: execution of getmore per second command: commands per second, more than the above inserts, searches, updates and deletions combined. It also counts the percentage of dirty data in other commands dirty:WiredTiger storage engine cache percentage flushs: the number of times fsync is executed per second to write data to hard disk in used:WiredTiger storage engine. Vsize: virtual memory usage, unit MBres: physical memory usage, unit MBqrw: length of client waiting to read, length in queue arw: length of queue waiting for client to write netIn and netOut: network traffic, in bytes byteconn: current number of connections time: timestamp 6, test result

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.