Shulou(Shulou.com)06/01 Report--

Generally speaking, there are two biggest performance problems affecting the database, one is the read and write operations of the database, and the other is the slow operation caused by the large data in the database. For the former, we can reduce part of the read operations with the help of cache, while for some complex report analysis and search, we can hand it over to hadoop and elasticsearch. For writing concurrent large and reading concurrent large, we can consider dividing the database into tables, separating master and slave reading and writing, or combining the two to improve concurrency and timeliness. For example, PG large concurrent write, large data view can be read synchronously with elasticsearch and PG data, which can achieve good results.

ElasticSearch as a search server, in terms of performance really outstanding, is the current popular enterprise-level search engine. It provides a full-text search engine with distributed multi-user capabilities, based on RESTful web interfaces. It is mainly used for real-time search and analysis engine, supporting structured data and unstructured data processing retrieval.

For example, the database we use is Postgres database, master-slave configuration, and the slave database is mainly used for data analysis and retrieval. If postgres is used to mine valuable data such as multi-table, multi-dimensional and full-volume retrieval and analysis of user behavior, the performance cannot meet the customer's timeliness requirements in time. Therefore, we can use Elasticsearch database instead of PG slave database as mining analysis database. During the use process, we found that the compatibility between tables and data of both is good. And the performance of presenting data is really fast,

As shown in the figure below, the table corresponding to the same table data in the postgres database corresponds to the Elasticsearch database, indicating that the data is compatible.

However, in the postgres database, the full query for the 3073920 data in the visit_plan_qz table cannot be displayed normally if it takes more than 5 minutes.

If a full query using Elasticsearch takes only 0.0005 seconds to present the data, Elasticsearch will automatically divide the table into 5 slices to present the data.

How it works:

PG Relational Database <$Database <$Tables <$Rows <$Columns

Elasticsearch <$Index <$type <$Docments <$Fields

It can be seen that Elasticsearch's performance advantage lies in indexing, which provides powerful indexing capabilities. Elasticsearch uses Lucene's inverted index technology to achieve faster filtering than relational databases.

Elasticsearch's indexing idea: move things from disk into memory as much as possible, reduce the number of random disk reads (and also use disk sequential read characteristics), combine various algorithms, and use memory with its harsh attitude.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.