Shulou(Shulou.com)06/01 Report--

This article mainly introduces what factors need to be considered in building a Prometheus platform, which is very detailed and has a certain reference value. Friends who are interested must finish reading it!

The growing container complicates the situation.

Relatively speaking, it is often easier to monitor a single environment because the number of static physical servers and virtual machines is determined, and the number of monitoring metrics is limited. However, there has been a surge in the number of instance programs to track and monitor because of containers and the need to migrate to a micro-service architecture.

If the server in the data center is a pet and needs our constant attention, the cloud instance is more like a cow (because there are so many, you don't have to care about a single instance), and the container is more like a little bee. They are numerous, sometimes there are hundreds of containers per machine, and new containers keep popping up all the time, and their life can be very short when used with container orchestration engines such as Kubernetes. This makes it more difficult to track and monitor them, and they can cause a lot of damage if you accidentally misoperate them.

As complexity and distributed environments increase, so does the number of entities you need to monitor. In addition, you may want to monitor more properties to ensure that you have an accurate understanding of what is happening, or that you can understand what is happening when troubleshooting or responding to events. The latter is especially problematic in a short-lived environment, because when you want to understand the root cause of the problem, the relevant resources are usually deactivated, which means that the monitoring solution must provide a way to store sufficient historical records for forensics.

Popular monitoring tool: Prometheus

More and more teams that need cloud monitoring are turning to Prometheus, an open source CNCF project. Prometheus has become the preferred monitoring tool for developers to collect and understand metrics in a native cloud environment. It is supported by a large community with 6300 contributors from more than 700 companies, 13500 code submissions and 7200 pull requests.

By default, typical cloud native application stacks (such as Kubernetes, Ngnix, MongoDB, Kafka, golang, etc.) expose Prometheus metrics. Prometheus is a vertically scalable Go program that is easy to deploy for a single container or a single host. In other words, using Prometheus is extremely easy at first, and you can easily monitor your first Kubernetes cluster, but this also means that monitoring will become more and more complex as the infrastructure grows.

Scaling problems caused by application growth

As the size of the environment grows, you need to track and monitor the rapid growth of time series data, and after the data volume reaches a certain point, a single Prometheus instance can no longer track and monitor. In this case, the most immediate option is to run a set of Prometheus servers throughout the enterprise, but this presents some challenges. For example, it is not easy to manage and merge data across dozens or even hundreds of Prometheus servers. Similarly, it is not easy to understand corporate workflows, single sign-on, role-based access control, and compliance with SLA or compliance. As applications grow, running a comprehensive monitoring solution without interrupting the work of developers will become an issue of manageability and reliability.

In order to solve this problem, enterprises have adopted many methods.

The simple way is to prepare a separate Prometheus server for each namespace or cluster. This method will be difficult to sustain at a certain scale, in addition, it also has the disadvantage that it will result in a large number of disconnected data islands. This can make troubleshooting troublesome because most problems span multiple services / teams / clusters. Not only is it hard to find the same metrics in every environment, but you also need to piece the data together to try to understand what's going on.

Another common approach is to use open source tools like Cortex or Thanos to aggregate multiple Prometheus servers. These efficient tools allow you to centrally query the server, collect data, and share it in a unified dashboard. However, like any data-intensive distributed system, they require a lot of skills and resources to run.

Six factors to consider

For companies that start with Prometheus and then seek commercial solutions for global monitoring, it is important not to lose all standardized development work done on Prometheus-dashboard, alarms, exporter, and so on. However, this is not the only thing to consider, and if you continue to use Prometheus, you need to adhere to the following criteria:

1. Compatibility to support all Prometheus features

Your vendor / tools / SaaS solution needs to be able to consume data in any physical program that generates Prometheus metrics, whether it's local Kubernetes or cloud services. Consuming Prometheus metrics is relatively trivial, but don't ignore small things, such as extracting metrics to storage or being able to re-label metrics as you add data, which makes more sense to your environment. Taken together, the data that can be collected will accumulate and vary greatly.

2. PromQL compatibility

The Prometheus query language, invented by the creator of Prometheus, is used to extract information stored in Prometheus. PromQL allows you to query metrics for specified services or users, and it also aggregates or subdivides data. For example, you can use it to display the CPU usage of each application in all containers. Or just display the data from the Cassandra container and display it as a single value for each cluster. It can be said that PromQL releases the true value of Prometheus, so if you integrate the metrics of Prometheus into a product that does not fully support PromQL, it completely violates the original intention of using Prometheus.

3. Support hot plug and unplug

To be truly compatible with Prometheus, the solution must be hot-swappable so that it can be used with your existing dashboard, alerts, and scripts. For example, many enterprises that use Prometheus use Grafana for dashboard. This open source tool integrates well with Prometheus, including at the query level, and can be used to generate a series of useful charts and dashboard. Therefore, commercial products that claim to be compatible with Prometheus should be compatible with tools such as Grafana. It is not enough to say that the solution allows you to view the numbers in Grafana, you need to be able to extract the existing Grafana dashboard as it is and reapply it to the installed data in the business solution.

4. Access control

Access control is another security issue you need to consider when evaluating tools. The ability to protect user authentication using industry standard protocols, including LDAP, Google Oauth, SAML, and OpenID, enables companies to isolate and protect resources through service-based access control.

5. Troubleshooting

Kubernetes simplifies deployment, self-scaling, and managing containerized applications and micro-services. This helps keep the service up and running, but to identify and resolve fundamental problems such as performance degradation, deployment failures, and connection errors, you need to be able to collect and visualize infrastructure, application, and performance data from across the environment. Since real-time information and contextual data cannot be accessed at the same time, it is almost impossible to correlate metrics in the environment, so you can solve the problem faster.

6. Compatible with existing alarms

Finally, if you are looking for a commercial solution to help solve Prometheus scalability issues, make sure it supports all levels of alerts. The key to achieving this goal is full support for Alert Manager functionality, while Alert Manager also requires 100% integration and PromQL compatibility.

If you find a commercial tool that meets the above criteria, you should be able to easily integrate it into existing Prometheus and avoid the scalability problems encountered by the company. Developers have every reason to love Prometheus, so conducting comprehensive and due diligence before adopting a commercial solution will ensure that they can still use the metrics they like.

These are all the contents of the article "what factors need to be considered in building a Prometheus platform". Thank you for reading! Hope to share the content to help you, more related knowledge, welcome to follow the industry information channel!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.