Shulou(Shulou.com)05/31 Report--

How to analyze DataWorks data services + MaxCompute Lightning docking DataV best practices, I believe that many inexperienced people do not know what to do, so this paper summarizes the causes of the problem and solutions, through this article I hope you can solve this problem.

1. Overview

Data services (https://ds-cn-shanghai.data.aliyun.com) is a member of the DataWorks product family, which provides the ability to quickly generate data tables into API. Through visual wizards, API can be generated in a minute of "zero code", making API development so convenient! At the same time, it supports custom API query SQL function, which makes it easy to support your personalized complex query logic.

DataWorks data service provides HTTP API services and adopts Serverless architecture. You only need to pay attention to the query logic of API itself, no need to care about infrastructure such as running environment, and zero operation and maintenance costs.

DataWorks provides a full-link data research and development platform covering "data integration-data development-machine learning-data service". Data service serves as a bridge between data application and data warehouse. Through MaxCompute Lightning acceleration engine, data service can directly generate data API from MaxCompute tables that can only be queried offline and query them in real time, eliminating the trouble of manually synchronizing data. Data service is the best tool for you to export the results of data warehouse construction into service.

At present, DataWorks data service has been deeply connected with the artifact of data visualization-DataV (https://data.aliyun.com/visual/datav). The API generated by data service can be visually displayed in DataV directly. You do not need to synchronize data manually, write complex Java code, or build WebServer. Ordinary data development engineers, algorithm development engineers, data analysts and even product business personnel can use data services to "develop" data API, and then quickly call API in DataV and display data results from MaxCompute. The data service solves the last kilometer between the data warehouse development and the data display.

The following will focus on how to quickly generate real-time query API from MaxCompute table through data service and interface with DataV for large screen analysis and display.

two。 Use data services to generate API in one minute

Generating API through data service mainly includes three steps: create data source-> configure API- > publish API. This document briefly introduces how to use it.

2.1 New data source

Data services support a wide range of data source types, including basically all common data sources, including:

Relational database: RDS/DRDS/MySQL/PostgreSQL/Oracle/SQL Server

Analytical database: AnalyticDB (ADS)

NoSQL database: TableStore (OTS) / MongoDB

Big data Storage: Lightning (MaxCompute)

In the data service, click the New button in Service Development, and select New data Source from the drop-down menu.

Cdn.com/6bc368871d425d0377562039571aaae10ab39a37.png ">

This will then open the data sources page in the data integration, where you can create a new data source that you need to access.

Note: the "sslmode=require&prepareThreshold=0" in the JDBC extension parameter is required and cannot be deleted, otherwise it will not be able to connect.

2.2 New API

After creating the data source, go back to the data service product page, we can generate API configuration, this article takes the wizard mode to generate API as an example.

Click "Service Development"-"New"-"generate API"-"Wizard Mode" to generate API in wizard mode visual configuration. Enter the basic information of API in the pop-up dialog box. Here, take the query transaction volume growth trend API as an example, as shown below:

Then click "return parameters" on the right to set the parameter description information.

Note: if you do not set the request parameters, you need to turn on the "return result paging" switch to conduct a paging query to avoid excessive data returned by a single query affecting performance.

At this point, an API has been generated, is not super simple!

2.3 release API

Once the API test has passed, it can be released. Click "publish" to the right of the toolbar to publish the API. After publishing, you can click "Service Management" in the navigation bar of the item section to view the details of API. If you want to call API, please check the "Service Management"-"API invocation" page. The data service provides two authentication methods: simple authentication (AppCode) and encrypted signature authentication (AppKey&AppSecret). You are free to choose. The invocation of the data service API in DataV is described below.

Aren't you a little excited? "developing" an API Tathagata has never been so simple! This can hardly be called "development"!

3. Invoke the API of the data service in DataV

The next step is to configure the DataV data screen, which is mainly divided into two steps: "add data" and "New Visualization".

3.1 add a data service as a data source

Click "my data"-"add data", select "DataWorks data Service" in "Type", fill in the data source name, and then select your DataWorks project (workspace). DataV interfacing data service uses more secure encryption and signature authentication, so you need to enter AppKey and AppSecret here.

Once the data service data source is added, the data can be displayed on a large screen.

3.2 invoke the data service API in a large screen

Click New Visualization in my Visualization, and this article selects the Smart Factory template, as shown below:

The components in the template have their own static data. Here we will explain how to use the data service API in the component by changing the "basic line chart" in the middle of the template to calling the API created above to "query the transaction volume growth trend".

Select the basic line chart component, switch to the data panel, select "DataWorks data Service" in the "data Source Type", then select the data source "my data Service" that you just created, select "query transaction volume growth trend" this API, and then set the query parameters, here set pageSize to "31" to query one month's data.

Click "View data response results" to see the query results of API.

Then fill in the field mapping relationship, fill in "date" in "x" to take the date as the horizontal axis, and fill in "y" in "amount" to take the transaction amount as the vertical axis.

As you can see, currently x and y cannot match the field. This is because DataV has certain requirements for data format and cannot identify fields with a deeper structure, so here you need to add a data filter to filter out unnecessary fields and, in this case, directly return the "rows" array.

Check "use filter" and click "New filter". Here, you can write JS code to filter and process the data results. The data parameter of the filter is API, which returns the result JSON object. In this case, we only need to return the rows array in the API results, so we just need to enter the code "return data.data.rows;", and then we can view the filtered results below and click "done".

After adding the filter, you can see that the fields can be matched successfully at this time.

But the line chart is not displayed correctly at this time, because the date format returned by API is different from the default format of the component, so we also need to set the date format of the horizontal axis of the broken line. Switch to the "configuration" panel, in the "x axis"-"axis label", select the data type as "time type", select the data format returned by this API "2016-01-01", you can see that the line chart can be displayed normally.

Finally, take a preview to see the finished product.

At this point, we have completed all the steps of generating the MaxCompute table into API through the data service, and then displaying it on the DataV data screen. Taking a similar step, configuring the data sources of other components of the large screen to DataWorks data service API can complete the creation of the whole large screen, doesn't it feel very easy!

4. Summary

After the seamless connection between DataWorks data service and DataV, you do not need to use the "API" data source in DataV to fill in a URL to call API, directly create a new DataWorks data service as a data source, you can directly choose API in the data service, do not need to set AppKey and AppSecret authentication information for each API, and support filling in API parameters through the form, which is very fast, convenient, safe and reliable to use. Through data services, you can directly present the data results processed in MaxCompute in DataV, and "data development-data service-data analysis display" in one go!

Finally, some tips and notes are provided for your reference:

DataWorks data service wizard mode generates API only supports single table simple conditional query, script mode allows users to write query SQL statements, and supports multi-table associated queries, functions and complex conditions. Everyone can choose flexibly according to their own needs.

Lightning uses the syntax of PostgreSQL, so when writing SQL, you should pay attention to using the PostgreSQL function instead of MaxCompute's UDF. Currently, Lightning only supports max_pt as a MaxCompute UDF, which can be used to get the latest partition. Also, use "| |" when connecting strings.

Lightning only supports second-level query, and the MaxCompute of the query should not be too large (controlled at GB level). Try to use partition as the request parameter to avoid scanning too many partitions, otherwise it will be slow.

If you require millisecond API queries, it is recommended that you use a relational database, NoSQL database, or AnalyticDB as the data source.

The data format required by the DataV component is an array, and the API generated by the data service returns a complete JSON with an error code, so a filter is used to process the API results. You can choose to add a filter in DataV, or you can choose to add a filter directly when the data service configures API. Generally speaking, for an API that is not paged, just return the "data" array directly, and for the API of a paging query, it directly returns the "data.rows" array.

If you want to add multiple series to the line chart or bar chart of DataV, DataV generally requires that the data of each series is an object and distinguishes the series by the field "s". At this time, you should pay attention to the use of filters for format conversion.

After reading the above, have you mastered how to analyze the best practices of DataWorks data Services + MaxCompute Lightning docking DataV? If you want to learn more skills or want to know more about it, you are welcome to follow the industry information channel, thank you for reading!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.