Shulou(Shulou.com)06/01 Report--

This article shows you how to change the Elastic open source agreement, the content is concise and easy to understand, it will definitely brighten your eyes. I hope you can get something through the detailed introduction of this article.

On January 15th, CEO Shay Banon of Elastic, a world-famous big data search and real-time processing company, suddenly announced that one of the open source license agreements between Elasticsearch and Kibana would be changed.

It is reported that the change in the license agreement has no impact on most community users who use the default distribution for free, and the main restrictions are cloud service providers.

Of course, this is not the first time Elastic has changed the open source license agreement. Experts say Elastic changed its open source license agreement once in 2018.

Similarly, in the open source field, database software MongDB, Redis Lab, graph database Neo4j and so on modified the relevant open source license agreements in 2018 to change the "bloodsucking" dilemma they faced in the commercialization of databases.

The revision of the open source software license agreement by Elastic is bound to have a significant impact on the production of a large number of users using Elasticsearch and Kibana, especially on the public cloud. How to avoid the impact and ensure the stable and sustainable operation of applications is a major problem faced by a large number of Chinese users.

In addition, in the case of a large number of applications of open source software, users have to face the huge risk brought by the modification of open source software protocols.

1. China also has its own big data comprehensive search engine Transwarp New Search.

As a very popular data search and real-time analysis engine in the world, Elasticsearch is famous for its downloads of more than 225 million people in 2018 alone, and many companies around the world use it in one form.

The first version of Elasticsearch was released in 2012. After more than 7 years of update iteration, the ecology of Elastic Stack has become increasingly mature, with more and more users in China, and the number of developers applying Elasticsearch is also expanding.

Developers use Elastic Stack to develop flexible software, which is widely used in search, log recording, security protection, operation and maintenance index monitoring, database acceleration and other application scenarios, as well as in the Internet and software industry, financial industry and other industries.

In recent years, cloud service providers have been using open source products, modifying their code, and developing hosted (paid) service solutions. However, the modified code will not be accessible as open source code.

At the same time, the business behavior of cloud service providers also hinders the commercialization of open source software companies. Under the open source license, how to make a profit and achieve healthier development has become the biggest challenge for open source software companies.

As a result, Elasticsearch and Kibana have made significant changes in licenses from an open source Apache 2.0 license to SSPL (server-side public license).

As early as 2018, MongoDB changed its license agreement and adopted SSPL (Server-side Public license) to protect open source code from being used by cloud service providers to develop their own SaaS/DBaaS products.

There is no doubt that Elastic's change of license agreement will have a huge impact on users, especially on cloud-hosted applications. Many countries regard the modification of license agreements for open source software as a major risk to the development of the software industry, causing panic among relevant users around the world.

Fortunately, with the strong support of the national policy, China's innovation industry continues to grow. In the localization and self-control of big data comprehensive search, China has launched its own products.

China big data and star ring technology, the leader of AI basic software, have launched the big data comprehensive search engine that can completely replace Elasticsearch-Transwarp New Search, an international leading large-scale statistics and search fusion engine, which can not only complete users' accurate query and analysis of full-text search and relationships, but also excel in semi-structured data retrieval, spatio-temporal data retrieval, semantic retrieval, fuzzy retrieval and so on.

Supported by the national independent and controllable policy, as well as the continuous breakthroughs in star ring science and technology, star ring science and technology has completed the completely independent research and development of big data basic software, and there will be no open source software Hadoop in the future, and its big data basic software products begin to replace Oracle, IBM and other foreign software in different fields.

Users from more than 2000 different industries have chosen to use the platform independently developed by Star Ring Technology to build the infrastructure at the bottom of big data, giving relevant business capabilities to various industries, including finance, government, energy, manufacturing, transportation, education and so on.

2.New Search emerges from the blue but surpasses the blue.

Star Ring Technology self-developed Transwarp New Search is used to build big data search engine within the enterprise. New Search supports the storage and retrieval of unstructured data formats such as Word/Excel/PDF/CSV/ Internet data / pictures / audio and video. When retrieving the amount of data at PB level, it can be returned in seconds.

In terms of development interface, New Search provides complete SQL syntax, supports and provides search syntax SQL extension, through effective combination with Star Ring Technology's analytical database Inceptor optimizer, developers do not need to understand the underlying architecture to develop an efficient search engine.

Compared with the open source big data search engine Elasticsearch (ES), Star Ring Technology self-developed New Search has more advantages:

New Search provides a distributed computing engine, which can meet the needs of scenarios such as multi-table association and complex aggregation analysis, changing the problem of inaccurate results of open source Elasticsearch aggregation and providing accurate aggregation.

New Search supports standard SQL and SQL extended search semantics, as well as Oracle and DB2L dialects, supporting Star Ring's own security control platform Guardian and big data management platform Manager to facilitate security management and operation and maintenance management.

In the aspect of full-text retrieval, New Search supports the storage and search of documents in common formats such as pdf/word/excel, provides Chinese, Uygur, Tibetan, English, French, Japanese, Korean, German, Spanish, Portuguese and other language word splitters, and supports natural language processing functions such as article similarity matching, keyword extraction, abstract extraction and so on.

The New Search spatio-temporal database module supports standard graphic types defined by OGC, including points, lines, polygons, set types, tile services based on WMTS protocol, and spatio-temporal database algorithms such as adjoint analysis and trajectory similarity matching.

In addition, New Search performs well in a large data volume and large cluster environment, which is significantly better than the open source Elasticsearch (ES):

When the hardware input cost is limited, users want to improve the resource utilization as much as possible, so the number of instances supported by a single node of the cluster is very important. New Search supports single-instance 50TB, which is far more than open source ES single-node single-instance 10TB.

When the total amount of user data reaches the order of 100 TB-PB, when using open source ES software, when the instance has more than 100 nodes, there will be stability problems such as serial disconnection. The New Search products of Star Ring Technology specialize in tuning parameters for large clusters, which can greatly alleviate the problem of node loss, while using the latest generation of New Search, it can still have better stability when supporting more than 200nodes or instances.

Big data search has high requirements for high availability, and should be able to guarantee more than 99% of SLA service level agreements, which can be automatically and quickly recovered when nodes are abnormal, and can provide tools to quickly diagnose and repair clusters when manual OPS is needed. It usually takes hours to restart an order of magnitude node of open source ES software TB, while it takes only a few minutes for Star Ring Technology's New Search,TB order of magnitude node to restart. The following figure shows the restart time of NS with different stock data sizes. The difference between cold and hot lies in whether to exclude the influence of operating system pageCache.

For businesses with high data dimensions and need to do multi-table association, New Search of Star Ring Technology has changed the practice that open source ES does not directly support multi-table association operations to meet the requirements of multi-table association queries where field data is extracted from two or more tables for a query result.

For short and fast queries, the maximum concurrency supported by open source ES is generally 700800, and the query performance is burr due to the impact of GC. On the other hand, the query process of New Search first-generation products of Star Ring Technology is optimized, reducing one rpc, reducing response time by 30%, and reducing heap footprint through offheap, automerge, cooling and other technologies, which is less affected by GC.

The latest generation of New Search, through thread pool optimization and lucene optimization, continues to reduce the memory footprint by 1 / 3, and is less affected by GC, which not only meets the requirements of short and fast queries in seconds, but also meets the query peak period with high concurrency. As shown in the following figure, when the stand-alone stock data is 4.5T, through the efficient use of out-of-heap memory, New Search greatly reduces the heap memory footprint and significantly reduces the GC pressure.

The user's query is mainly aimed at the data in the last N days, and the query frequency of the old data is relatively low. In the face of this demand, open source ES does not do special processing for hot and cold data, while New Search of Star Ring Technology optimizes hot and cold data to improve query performance.

When the size of the cluster is very large and there are many table fragments, due to the limitations of its own PP architecture and balancing strategy, open source ES will cause very high DDL operation latency, resulting in obvious stutter and high cluster load. The central architecture of New Search can significantly optimize this performance, and the test comparison is shown in the following figure.

When the daily incremental data increases, users will put forward special requirements for storage performance. How to ensure the performance of incoming data? The data storage performance of open source ES will gradually decrease as the amount of data increases. The New Search first-generation product of Star Ring Technology optimizes the storage format and improves the performance by 10% to 20%, while the second-generation products join the bloomfilter index to minimize the impact of the increase in the amount of data, and continue to improve the storage performance by 30% and 70%. In addition, support bulkload function, through the way of BulkLoad to quickly import massive data.

As shown in the following figure, the performance of New Search in batch write tests using TPC-DS standard data is compared with that of ES, in single node / MB/ seconds.

In addition to comprehensively searching for the technology and performance of products, Star Ring Technology has many noteworthy aspects in the basic software of big data. For example, Star Ring Technology is considered to be the enterprise with the richest product line in the field of big data. The products involve many application needs and scenarios in the field of big data, there are more successful cases in China, and they can also provide better localization services and support for Chinese users.

The above content is what to do if the Elastic open source protocol is changed. Have you learned the knowledge or skills? If you want to learn more skills or enrich your knowledge reserve, you are welcome to follow the industry information channel.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.