Shulou(Shulou.com)11/24 Report--

In the golden autumn of October 2023, there is not only a lively celebration of double festivals throughout the country, but also a time for even numbers and their industry customers to harvest the results of product iterations. On October 13, even Technology released the latest real-time Huancang data platform Skylab 5.4. Among them, cloud native distributed database OushuDB, data analysis and application platform Kepler, data asset management platform Orbit, data factory Wasp, data development and scheduling platform Flow, system management software Lava have all been iteratively updated in this new version upgrade.

Next, let's take a look at the updates to the various components of Skylab 5.4.

Update of cloud native distributed database OushuDB 5.3

OushuDB 5.3 supports online incremental backup recovery in Native ORC format. Compared with full backup recovery, incremental backup recovery reduces backup and recovery time and provides faster data recovery. At the same time, compared with traditional offline backup (database service needs to be stopped), online backup reduces downtime. This is critical for businesses that require high availability and run at 7x24. Industry customers pay more and more attention to the security of the data platform, and online incremental backup and recovery is becoming more and more necessary. It not only ensures data security, but also reduces platform maintenance costs, reduces business interference, and improves system availability.

Historical SQL monitoring function is supported. Historical SQL monitoring can track and analyze the performance of SQL queries and transactions executed in the database. By monitoring the execution plan, response time and resource consumption of historical queries, DBA can identify potential performance bottlenecks and take appropriate optimization measures to improve database performance.

The new version of the default permissions feature sets the default permissions for new objects (tables, views, functions, sequences) for specified users and schema. Therefore, permission information is automatically added when you specify a user or when you specify schema to create an object, without the need for manual authorization. Especially for large-scale, multi-object rights management, the function of default permissions can effectively reduce the time cost and operation cost of user rights management.

OushuDB 5.3 implements partition column DISTINCT performance optimization, which means that when executing a DISTINCT query, it is no longer necessary to scan all the data in the sub-table, which greatly reduces the number of data scans, improves query performance, and returns complex query results in milliseconds.

Hive External Catalog added support for Description. At the same time, on the basis of reading-only Hive tables in previous versions, multiple different Database in Hive can be accessed at the same time to realize multi-table association calculation and provide users with a more convenient query method.

OushuDB's client-side workbook Worksheet has further optimized the performance, functionality, and interactive experience in this upgrade. For example, query results columns can be dragged, exported query results with headers, hotkeys / aliases, resource queue switching, and so on.

Update of data analysis and application platform Kepler 4.1

From version 4.0 to version 4.1, the core modules such as Kepler report development, index development, visualization and platform services have been updated.

The report development module adds the functions of overall style, undo / redo, hidden rows / columns, frozen scrolling and parameter jump, which improves the flexibility and efficiency of report design and enables users to create more attractive reports.

A variety of chart types and map configuration properties are added to the data visualization module. By increasing the diversity of data presentation, it helps users analyze the data more clearly.

The platform service module adds real-time index service and data service detail page upgrade, which provides the ability of real-time data analysis to help users make business decisions in a more timely manner.

SQL index placeholder, field list editing, association relationship, null value filtering, custom measurement and other functions are added to the index development module to simplify the index development process, speed up the data analysis process and improve the data quality.

In addition, Kepler adds access control and consanguinity map node data download functions to provide better security and data traceability. The complex qualification operator in / not in is supported, which improves the flexibility of data query and makes it easier to filter target data.

Data factory Wasp 5.4 update

Wasp 5.4 has been updated and optimized from the point of view of error tolerance and ease of use.

Kafka and file data sources support ignoring case to ensure that field matching and alignment are not affected by case, thus improving data consistency and avoiding matching errors caused by case differences; support skipping dirty data, excluding format errors, invalid or incomplete data to prevent dirty data from interfering with subsequent processing and analysis, and maintaining data quality.

Support the remapping of data synchronization objects, in the case of changes in the data source or target table structure, allow to re-modify the field mapping of the source and target table, reducing the complexity of operation and maintenance.

In addition, Wasp 5.4 also supports the movement of tasks and folders, optimizes metric monitoring, and supports the display of primary key information of source tables. Overall, the robustness of the new version of Wasp is further improved, and its new features and ease of use will help users effectively reduce operation and maintenance costs.

Update of data Asset Management platform Orbit 2.3

First of all, the new version of Orbit focuses on improving the stability and fault tolerance of automatic metadata collection and consanguinity collection. When conducting metadata and consanguinity analysis, users will encounter fewer interruptions and failures, and it will be easier to track the flow and relationship of the data, which will contribute to the accuracy of data decisions.

Secondly, Orbit optimizes the configuration page of the data service interface in the master data module on the basis of the previous master data management module, in order to improve the user operation experience. A more smooth and intuitive interface provides users with a more convenient mode of operation. For example, when configuring multi-table association queries, users can more easily create and manage multi-table associations, improving the efficiency of data retrieval and analysis.

In addition, the new version of Orbit has security upgrades to ensure the confidentiality and integrity of data, as well as guard against potential data risks and threats.

Data development and scheduling platform Flow 5.4. update Flow 5.4.It further enriches the functions of the control panel, adding functions such as timed calendar, task proportion, job history statistical chart, etc., to help users understand the use of Flow more conveniently. In addition, new functions such as job movement and replication, task history and so on are added.

System management software Lava 5.4 update

The new version of Lava enhances the HA (High Availability) feature to ensure that the system keeps running in the face of hardware failures, software failures, or other unpredictable events.

In the monitoring operation and maintenance part of Lava, functions such as OushuDB backup and recovery, script migration tools, SQL monitoring, custom metrics monitoring and other functions have been added. Add Product specialist Wechat to understand and experience the product: oushukeji

Even Skylab is a real-time data platform with massive data storage, calculation, analysis, application and AI capabilities, providing a complete ecological closed loop for data management, helping users to achieve batch and flow integration, real-time data processing, automated machine learning, self-help visual analysis and data asset management. Relying on Skylab, users can easily implement data application scenarios such as intelligent marketing, intelligent risk control, intelligent audit, intelligent supervision, and so on.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.