Shulou(Shulou.com)06/01 Report--

How to understand the deployment and operation and maintenance of DLI under the micro-service architecture? to solve this problem, this article introduces the corresponding analysis and solution in detail, hoping to help more partners who want to solve this problem to find a more simple and feasible method.

Huawei Cloud data Lake Exploration DLI is a Serverless big data computing service that supports multi-mode engines, which well implements the characteristics of Serverless:

1. Weakens the link between storage and computing

two。 Code execution no longer requires manual allocation of resources

3. Charge by usage.

Then how to better achieve Serverlessed services, while avoiding becoming a traditional single distributed application, micro-service architecture is undoubtedly the best choice. The overall deployment architecture of DLI based on the microservice architecture model is as follows:

That is to say, the service is provided in the form of pure API, and the server-lessened big data computing service is realized by taking API Gateway as the entrance of the application and dividing the micro-services according to the sub-domain based on the domain model.

For such a Serverless service based on micro-service architecture, how do we deploy and operate in the production environment, so as to achieve fast iterative launch on the premise of ensuring the service SLA?

Key to DLI deployment

With the development of technology, the process and architecture of deployment have undergone fundamental changes, and now it has entered a technology era of lightweight and short life cycle.

From big data computing platform originally deployed on physical machines, to big data platform deployed by elastic computing servers based on public cloud, to Serverless services such as DLI, it well demonstrates the evolution of big data computing services. So how can we better achieve the deployment of Server-lessened big data computing services? the answer of DLI is to deploy micro-services based on Kubernetes+Docker.

Kubernetes deployment is a good way to deploy services without downtime, but how to deal with errors after receiving production traffic to make the new version of the service more reliable? This can be seen by dividing the problem in two:

1. Deployment, bringing the service online to run in a production environment

two。 Publish, even if the service can be used to handle production traffic.

Traditionally, separating the deployment process from the release process has been a challenge. But now we have a good choice, which is based on the service grid. In the deployment of DLI, we combine Kubernetes+Istio, and use the traffic management of Istio to achieve service discovery and traffic routing, so as to easily separate deployment from release, and make the new version of the service more reliable.

Monitoring and alarm to improve the overall ability of operation and maintenance

Free operation and maintenance is also an important feature of DLI when facing customers as a Serverless cloud service. How do we realize the operation and maintenance of the whole service? Today, let's talk about how DLI implements monitoring alarms to improve the overall operation and maintenance capability, so as to better provide customers with Serverless DLI.

The figure above shows the overall deployment architecture of DLI services. As a Serverless service, it fully embraces cloud native technology. Whether it is a micro-service that provides task management or a computing unit that ultimately executes tasks, it is deployed based on Kubernetes, which better realizes the fast and flexible scaling of Serverless.

Currently, we mainly consider the monitoring and alarm of DLI service from the following aspects:

1. Global dimension, mainly the QPS, success rate and response delay of the overall API

As a Serverless big data computing service, DLI provides services in the form of REST API, so the QPS and response delay of API directly reflect the external capability of the service, and the success rate is the direct embodiment of the service SLA.

2. OS dimension, mainly CPU utilization, memory utilization, disk utilization, upstream and downstream traffic of the container host.

No matter how the deployed architecture and technology evolves, the monitoring of basic resources is the most basic and necessary.

3. Container dimension, mainly CPU utilization, memory utilization, K8s space and user space utilization, POD health

Container is the evolution of virtual machine, so the resource monitoring of container is the most basic. Our microservices or computing units run as containers on the Kubernetes cluster, so it is also necessary to monitor the health of POD.

4. Micro-service dimension, mainly traffic, performance, health check and critical log, etc.

Monitoring is to better identify and solve problems, so the core is still business-level monitoring. DLI is a complex distributed Serverless application, which is divided into different micro-services according to different domain models, so monitoring the internal traffic and performance of micro-services is an important index to measure the reliability of micro-services. A good system often has a perfect log system, and monitoring key logs can help us find and locate problems quickly, so this is also the focus of our monitoring in the business dimension.

This is the answer to the question about the deployment of DLI and how to understand the operation and maintenance under the micro-service architecture. I hope the above content can be of some help to you. If you still have a lot of doubts to be solved, you can follow the industry information channel for more relevant knowledge.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.