Shulou(Shulou.com)06/02 Report--

About what the OLAP scenario of tens of millions of data is like, many novices are not very clear about it. In order to help you solve this problem, the following editor will explain it in detail. People with this need can come and learn. I hope you can get something.

I took over a tricky project some time ago.

60 million stock market data aggregation scene.

Request:

It takes less than 2 seconds for the user to initiate query, parse, aggregate, and finally present to the user.

Because there are many aggregation dimensions, it can be divided into time dimension and index dimension. There are more than 10 index dimensions.

Scenario 1:

With mongo, about 20 million data is thrown in first, and it is almost impossible to produce results without indexing.

If each dimension uses the index, then the cost is too high, and finally decisively give up mongo.

Scenario 2:

Use mysql, data slicing, very fine-grained, ready to trade space for time.

It mainly uses the time dimension to do data redundancy, and prepares the data in advance for several major time periods.

Other dimensions are calculated by hard calculation.

The idea is good, and the result of the test is quite cruel to me.

1. It takes too long to calculate.

two。 Non-major time dimensions span is too small, not all do data pre.

Scenario 3:

Using elasticsearch, five servers are prepared. Prepare to use 5 es clusters to deal with OLAP scenarios.

After importing 50 million data, start the test.

Time dimension + other single dimension is less than 1 second

Time dimension + other multi-dimensions (within 3) within 3 seconds

Is it helpful for you to read the above content? If you want to know more about the relevant knowledge or read more related articles, please follow the industry information channel, thank you for your support.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.