Shulou(Shulou.com)05/31 Report--

This article mainly explains "what are the log management tools in Kubernetes". Interested friends may wish to have a look. The method introduced in this paper is simple, fast and practical. Let's let the editor take you to learn what are the log management tools in Kubernetes.

Kubernetes dominates the container orchestration market and is often used to host micro services. However, each instance of the microservice generates a large number of log events, which quickly become difficult to manage. To make matters worse, when problems occur, it is difficult to find the root cause because of complex interactions between services and unpredictable failure patterns.

At present, we have so many logging tools? Is there a perfect tool to meet all requirements and make monitoring, logging, and fault cause analysis as efficient and fast as possible?

Most Kubernetes log management tools are variants of ELK with similar functionality and the same limitations. These tools can help you access logs and search for information. However, most of these tools also need to resolve log rules and alert rules to work properly.

1. Zebrium

I put Zebrium in the first place because I found that this tool has the potential to be the next important tool in Kubernetes log management.

The new startup was recently selected as one of the "25 start-ups worth watching by Gartner 2020" and "Forbes AI 50: America's most promising artificial Intelligence Company."

Zebrium also recently helped Sweetwater reduce event tracking time from 3 hours to just a few minutes. Zebrium can even find software problems that have not been discovered before. This is an excellent feature because it can help you find problems before they arise.

So what makes Zebrium stand out from the competition? Zebrium uses artificial intelligence (AI) for problem discovery and automatic discovery, while all other tools rely on users to add rules manually. Zebrium can also be used as a stand-alone log management platform or integrated with ELK Stack or other log managers.

It sounds like a dream come true, so I tested it on a very simple project. In this test, Zebrium can automatically detect a network timeout problem. I didn't establish any rules for this, nor did I monitor the system manually. Zebrium solved this problem with its ML-based algorithm and informed me immediately.

Advantages:

It is easy to start and requires only a simple helm or kubectl command.

Automatically detect problems and software failures without the need to manually configure rules.

It can be used as a stand-alone log management tool or as an ML attachment to existing log management tools, such as ELK Stack.

Disadvantages:

Free plan: the log is limited to daily 500MB and only 3 days of records are kept.

Supports Kubernetes,Docker and most common platforms, but does not yet support Windows.

Official website link: https://www.zebrium.com/

2. Sematext

Sematext is a solution for log management and application performance monitoring. Sematex provides visibility into the system state.

Sematext is not limited to K8s logging, but also provides monitoring and alarm capabilities. The collected logs are automatically parsed / structured according to different known log formats, and users can also provide custom log modes.

Sematext also exposes Elasticsearch API, so you can also use any tool that works with Elasticsearch, such as Filebeat and Logstash and Sematex.

Sematext can create specific rules to monitor specific situations and catch exceptions. With Sematex's comprehensive real-time dashboard, users can control and monitor all services.

Advantages:

Integrate with other Sematext Cloud tools such as monitoring.

Configuration is convenient and flexible.

The flexibility of ELK.

Disadvantages:

Sematex and Kibana cannot be mixed on the same dashboard.

Custom parsing needs to be done in the log transporter, and Sematext parses Syslog and JSON only on the server side.

Tracking is weak, although they plan to improve it.

Official website link: https://sematext.com/

Loki of 3.Grafana

The third place in the list of K8s log monitoring tools is not ELK, but Loki.

Loki is an Prometheus-inspired tool that supports multi-tenancy and highly available log aggregation. This tool helps to collect logs, but users need to manually establish rules for them.

Loki works with Grafana,Prometheus and Kubernetes. Loki can make your internal processes more efficient. For example, it saves Paytm Insider 75% of logging and monitoring costs.

Loki does not index your log content, but only a set of tags for each event stream, so it is very efficient.

Advantages:

Large ecosystems.

Rich visualization functions.

Increased efficiency due to unindexed log content

Disadvantages:

Kubernetes log management is not optimized.

A lot of manual work.

Lack of a content index may limit search performance.

Official website link: https://grafana.com/oss/loki/

4. ELK Stack (Elastic Stack)

ELK ranked fourth. Generally speaking, ELK is probably the most famous open source tool for log management. ELK is the initials of Elasticsearch,Logstash and Kibana. Each component is responsible for a different part of the logging process.

Elasticsearch is a powerful and scalable search system. Logstash aggregates and merges logs, while Kibana provides a data analysis and visualization interface to help users understand data. Together, they provide a comprehensive logging solution for K8s.

Note that there are many other variants of the ELK stack (such as EFK Stack-Elasticsearch,Fluentd and Kibana).

ELK is used by many big companies such as Adobe,T-Mobile and Walmart, so you can trust its robustness. Typically, this is a reliable and proven tool.

However, because of its complexity and the amount of resources required, I put it in fourth place.

Advantages:

The tool is well known and has a large community.

Very wide range of platform support.

Rich analysis and visualization capabilities in Kibana.

Manually defined alert rules are required, and log analysis is complex.

Disadvantages:

Maintenance difficulty

In a large environment, you need to adjust a lot of properties

A large number of resource requirements

Some functions need to be paid for.

Official website link: https://www.elastic.co/cn/what-is/elk-stack

5.Google Operations (Stackdriver)

Google Operations (also known as Stackdriver) is a tool for monitoring, troubleshooting, and improving application performance in a Google environment.

It collects metrics, log information, and software tracking information for the entire Google Cloud and your application. Google Operations is equivalent to CloudWatch on AWS and, like CloudWatch, is a solution with logging and monitoring.

Cloud Logging is tightly integrated with GKE and is added to every GKE cluster you create by default. Your logs are stored in Logging's data store and indexed for search and visualization.

Cloud Logging supports flexible queries and seamlessly integrates with other tools in the Google infrastructure.

Advantages:

Real-time log management and analysis.

Built-in large-scale metrics.

Other tools in the Google infrastructure can be integrated.

Disadvantages:

Because the request passes through all levels of Google Cloud Platform (GCP), it is difficult to track the actual delay.

Applies only to GCP environments.

Complex pricing system. It is difficult to estimate in advance how much it will cost.

Official website link: https://cloud.google.com/products/operations

6. CloudWatch

CloudWatch is a product of Amazon Web Services.

It collects data from AWS and visualizes it in a single automated dashboard. This allows you to view logs and metrics and relate them to each other to understand the root cause of the problem.

Users can analyze logs using CloudWatch's own proprietary query language, which supports aggregations, filters, and regular expressions. You can also send logs to Elasticsearch via Lambda.

Overall, if you are already using Amazon services, CloudWatch is a good choice. It can also be used for hybrid cloud architectures and uses proxies or API to monitor software resources. Many well-known brands such as AirWatch,Deliveroo,9GAG are using CloudWatch.

Advantages:

Designed to monitor AWS resources.

With the explosive property of the index

Detailed monitoring and auto-scaling groups.

Disadvantages:

It can only be used for AWS services.

There are not many customization options for the dashboard.

Transaction tracking is not supported.

Official website link: https://aws.amazon.com/cloudwatch/

7.Fluentd

Fluentd is a cross-platform open source data collector that provides a unified logging layer (but it is not a separate log manager).

Fluentd is a popular tool with more than 5000 users such as Atlassian,Microsoft and Amazon. Fluentd has high reliability and performance.

In addition, Fluentd has created a unified logging layer to help you use your data more efficiently. This tool can help you process 120000 records per second like LINE.

Advantages:

Large community and plug-in ecosystem.

Unified logging layer.

Proven reliability and performance.

Easy to install

Disadvantages:

Complex configuration

Support for data conversion is limited.

Not a complete logging solution

Official website link: https://www.fluentd.org/

At this point, I believe you have a deeper understanding of "what are the log management tools in Kubernetes?" you might as well do it in practice. Here is the website, more related content can enter the relevant channels to inquire, follow us, continue to learn!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.