Shulou(Shulou.com)05/31 Report--

How to carry out excellent performance analysis based on spark, I believe that many inexperienced people are helpless about this, for this reason this article summarizes the causes and solutions of the problem, through this article I hope you can solve this problem.

Designed for exploratory and ad hoc analysis

YDB is a real-time, multidimensional, interactive query, statistics, and analysis engine based on Hadoop distributed architecture, with second-level performance at trillion data scales, and enterprise-level stable and reliable performance.

YDB is a fine-grained index: an index of precise granularity. Data is imported instantly, index is generated instantly, and relevant data is efficiently located through index. YDB is deeply integrated with Spark, and Spark directly analyzes and calculates the YDB retrieval result set. The same scenario speeds up Spark performance by a hundred times.

1. Audit deployment control scenario performance

2. Superior search and analysis performance vs. Spark txt performance (multiple improvement)

Compare to Parquet format (in seconds)

Performance comparison with ORACLE

3. Excellent sorting performance

Sorting in reverse chronological order is a hard metric for many logging systems. In Yanyun YDB system, we have changed the traditional brute force sorting method. Through index technology, we can sort data in a single column very quickly without the need for full table brute force scanning. This technology is called BlockSort, and currently supports tlong, tdouble, int, and tfloat.

Since BlockSort is implemented by means of search index, BlockSort sorting does not require brute force scanning, and performance is greatly improved.

BlockSort sorting is not a pre-computed method. It can be sorted by the whole table or filtered based on any filter criteria.

Detailed test address: http://blog.csdn.net/qq_33160722/article/details/54447022

30 billion pieces of data sorted demo video http://blog.csdn.net/qq_33160722/article/details/54834896

Test Results (Time in seconds)

After reading the above, do you know how to perform superior performance analysis based on spark? If you still want to learn more skills or want to know more related content, welcome to pay attention to the industry information channel, thank you for reading!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.