Shulou(Shulou.com)05/31 Report--

In this issue, the editor will bring you what is the solution for Kubernetes log collection. The article is rich in content and analyzes and describes for you from a professional point of view. I hope you can get something after reading this article.

Selection and Scheme of Log Collection component in kubernetes Cluster Environment

1. Source of demand

In the scenario of large-scale cluster deployment, the container instance will be deployed to multiple nodes, and the logs generated by the nodes and the applications on the nodes will be scattered on the hosts of each container, and the traditional cluster applications are mostly persisted locally, which brings great challenges to the log monitoring and troubleshooting of the whole application system, but in the Kubernetes large-scale cluster environment. We need to consider the unified collection, management and display of the logs scattered on each node.

2. Log source

Host kernel log

Error logs generated by the host kernel can usually help developers diagnose service exceptions caused by host or OS exceptions, such as network exceptions, file system exceptions, and so on.

Core component log

The logs of docker help users view the running status of pod internal containers and the logs of APIServer. The logs generated by Scheduler can help users view the logs generated by the operation of Kubernetes.

Apply your own log

Usually, when a business is upgraded or an exception occurs in a certain scenario, you can troubleshoot it through the log.

3. Log collection method

Host file

The data of the Pod application is stored in the host file system. For example, if we declare that the business log is stored in a certain directory through the hostpath declaration, we usually deploy fluentd or filebeat in the form of DaemonSet on each node, and mount the host file system to fluentd or filebeat Pod for collection. Of course, we can also collect other logs (logs generated by the operating system, logs generated by Kubernetes components, etc.) as shown below:

Files in the container

A log collection mode of sidecar that deploys the log collection container and the application container in the same pod, collects container logs in the form of a shared volume, and then outputs them to the node. This collection is generally aimed at applications requiring high log accuracy. In this way, we can customize the file name in the current container, the ip of pod, and so on. As shown in the following figure:

Direct output in container

Pod applications store data directly in a shared file system (NFS, hdfs, ceph, GlusterFS, etc.). Introduction to the use of nfs log storage in this case, we can view the log directly in the current file system, or deploy log collection pod in the node where the log is stored and transfer the log to the log system. As shown in the following figure:

4. Log collection and storage instance

In terms of log storage and query, it is recommended to use ELK (logstash consumes a lot of resources, it is recommended to switch to filebeat or fluentd for log collection and transmission) mature solution, because ES natively supports multi-tenant usage scenarios and supports different indexing methods to distinguish data from different users and different business types. After startup, fluentd generates an index with the specified prefix in ES according to the logstash_prefix in the configuration file. When creating a display index in the Kibana interface, you can match and monitor the log data based on the previous log prefix settings, as shown below:

In view of the large-scale and continuous growth of application business logs, fixed data for a few days will be stored under the traditional stand-alone business mode. in today's Internet of everything, we not only need to quickly and real-time monitor the log data in the cluster. what's more, we need to store these data persistently, so that it is convenient for us to carry out data mining, statistics, analysis and modeling or make predictions based on users' behavior logs. Of course, we can use big data Analysis solution (hadoop+spark) to analyze and manage the data.

This paper mainly introduces the three log collection modes under the Kubernetes cluster mode, and adopts different log collection schemes to meet the specific needs in combination with the actual use scenarios.

The above is the solution of the Kubernetes log collection shared by the editor. If you happen to have similar doubts, please refer to the above analysis to understand. If you want to know more about it, you are welcome to follow the industry information channel.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.