Shulou(Shulou.com)06/01 Report--

What are the core Tips of Elasticsearch development, operation and maintenance? I believe many inexperienced people are at a loss about this. Therefore, this paper summarizes the causes and solutions of the problem. Through this article, I hope you can solve this problem.

1. Pay attention to the hard disk space of the evaluation node at the cluster planning level. Combined with esrally and other third-party tools to evaluate the cluster resource writing, retrieval throughput and other indicators. Reasonably configure the number of fragments for each index. 2. Data preprocessing layer data should be cleaned before entering Elasticsearch. Elasticsearch is good at retrieval and non-complex aggregation, other live to relational databases or third-party big data open source libraries such as: clickhouse and so on. 3, data modeling level rather than strict mode, I prefer dynamic mapping, through the field name prefix mapping type, since the use of this set of rules, field conflicts caused by kibana can not do the report problem swept away ah, really not too fragrant. Whether to score, whether to sort, aggregate, filter, if not, (doc_values (dvm, dvd) norm (nvd, nvm)) properties need to be turned off, and so on. Template template is more flexible than mapping. It is recommended to use dynamic templates in combination with aliases, especially in business scenarios where the daily increment of data volume is huge. Fields are clearly fixed and will not be added in the future. Consider setting "dynamic": "strict" when mapping is created to strictly control Mapping flooding. Choose a word splitter or even a custom word splitter according to the business. 4. If you need to consider the optimization of query speed at the retrieval level, and the sorting field is basically fixed, you can consider matching indexSort, which will be interrupted in advance.

IndexSort can effectively avoid global scanning, interrupt the query in advance and improve query performance through pre-sorting, which is very suitable for scenarios where the query is sorted by a certain column (note that it is not suitable for correlation sorting).

Query according to the actual business considerations, it is recommended to limit Wildcard fuzzy queries, *. * and other queries that will lead to a large amount of data. Limit limit + offset, limit the length of text queries such as query_string, limit the length of term, and pay attention to slow query logs at any time. Es is very powerful, but depending on how you use it, you never know how to adjust your interface. 5. Hardware resource level 5.1 is the disk size sufficient at the disk level, and the default speed Compression is used for the compression format? Or Best Compression?5.2 memory layer uses the default NIOFS or MMAP, using MMAP which need to be pre-cached out of the heap. 6. At the cluster management level, remember to configure delayed sharding index.unassigned.node_left.delayed_timeout. The time of refresh and flush is adjusted according to the actual business needs. The more comprehensive the performance monitoring of the cluster, the better, find the slow query in time, evaluate the usage according to the business as comprehensively as possible, and be able to find and upgrade the configuration during the bottleneck period. Multi-node cluster, reasonable division of node roles, especially separation: master node, data node, coordination node. 7. Disabling batch deletion of indexes at the security and disaster recovery level is more important than the default random deletion. Regular or incremental backups are more important than no backups (if conditions permit). Security is a must. We encrypt the core fields when the log is clear. The entire technology stack of elk only allows access through the internal network, and the external service interface is also soft token. To provide ES to business R & D for use, it is more necessary to consider control authority and lower the threshold. It is best to encapsulate a layer of network management for business R & D use, and then share more training to improve business side R & D awareness of ES. 8. Shut down the system swap at the performance optimization level. If the amount of data is large, use bulk bulk operations as much as possible.

(1) write-level bulk operations, including but not limited to: bulk API performs batch write, update, and delete multi-document operations.

(2) search level bulk operations, including but not limited to: Multi Get (mget), Scorll, MultiSearch.

It is recommended to enable the slow query log according to the earlier settings of business requirements. Do not exceed the heap memory size of 32GB. When using script scripts, consider possible negative impacts such as slowness, security risks (earlier versions), and so on. Under certain conditions, if you perform a forced merge segment, the query speed will be greatly improved. After reading the above, have you mastered the methods of Elasticsearch to develop Tips, the core of operation and maintenance? If you want to learn more skills or want to know more about it, you are welcome to follow the industry information channel, thank you for reading!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.