Shulou(Shulou.com)05/31 Report--

What this article shares with you is about what the real-time data statistics service based on akka and data-sketch technology is like. The editor thinks it is very practical, so I share it with you. I hope you can get something after reading this article. Let's take a look at it with the editor.

Real-time stream data analysis service: submit analysis data in real time and obtain analysis results in real time

Previously, I had been working in a company as a background data processing and analysis program. After encountering a lot of problems, I had an idea whether I could directly set up a data analysis service.

Users can call the interface of the analysis service to submit analysis data and obtain analysis results at any time.

Technical background:

Data-sketch makes a trade-off between accuracy, storage cost and calculation cost for common statistical requirements, such as count (distinct), quantiles, topN, etc., and obtains statistical results with high accuracy when each calculation element is only touch once.

Akka is an asynchronous programming library that implements actor pattern. Business functions are closed into actor, mutual calls or state changes are completed through message transmission, stateless sharing.

At present, three computing models are encapsulated:

Theta sketch: cardinality calculation, that is, calculating count (distinct)

Quantile sketch: quantile calculation, that is, calculating the distribution of a series of values

Frequency sketch: frequency calculation, or topn calculation

Data security:

Wss is used to submit data, and https is used to obtain data. Data will not be intercepted halfway.

The backend service does not store any original data and throws it away after calculation. The server only has the analyzed data.

Related calculation parameters, such as userid,business, can be hash or confused before passing values

Since only authenticate is done at present, developers should pay attention to the confidentiality of appid. It is recommended that the backend call

At present, the whole service backend has completed about 0.5. because it is experimental, only one Aliyun ES is used at present. There are two aspects of the work to be done next.

1. Add statistical model

two。 Add support for akka cluster to make the entire backend service scalable

The above is what the real-time data statistics service based on akka and data-sketch technology is like. The editor believes that there are some knowledge points that we may see or use in our daily work. I hope you can learn more from this article. For more details, please follow the industry information channel.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.