Shulou(Shulou.com)06/01 Report--

This article introduces how to access the general application log in the ELK log system, the content is very detailed, interested friends can refer to, I hope it can be helpful to you.

Log specification

The standard log storage path and output format will bring great convenience for our subsequent collection and analysis. There is no need to consider the compatibility of different paths and formats, but only need to adapt to fixed types of logs. The specific specifications are as follows:

Specification of log storage path

Project logs can only be output to a fixed location, for example, under the / data/logs/ directory

Log file names of the same type (for example, java web) remain the same, for example, they are all called application.log

A type of project can record multiple different log files, such as exception.log and business.log

Specification of log output format

It is important that the log output must be in JSON format

The same type of project should adopt a unified log output standard, modularize the log output as far as possible, and all projects reference the same module.

The output log must contain standard time (timestamp), application name (appname) and level (level) fields, and the log content is clear and easy to understand.

Log information level specification

Log level description numeric debug debug log, log information the most 7info general information log, the most commonly used level 6notice the most important general condition information 5warning warning level 4error error level, a function does not work properly 3critical severity level, the whole system does not work properly 2alert needs to be modified immediately log 1emerg kernel crash and other serious information 0

From top to bottom, the number of logs is from low to high, and the number of logs is from more to less. The correct selection of log level helps to quickly troubleshoot problems in the later stage.

Why do we have to make such a specification?

All our projects are in Docker, and the Docker image is composed of basic image and project code.

The basic image packages the underlying environment in which the project runs, such as the spring cloud micro-service project, and packages the jre service.

After standardizing the log storage and output, we can package the filebeat as the log collection agent into the basic image, because the log path and format of the same type of project are the same, and the filebeat configuration file can be common.

In this way, we do not need to care about the log-related content in the subsequent deployment process. As long as the project image references this basic image, we can automatically connect to our log service and realize the collection, processing, storage and display of logs.

Log collection

Our general log collection scheme is shown below:

The program runs in the container, and the container has its own Filebeat program to collect logs.

After the collection is completed, it is passed to the kafka cluster, and the logstash reads the kafka cluster data and writes it to the elasticsearch cluster.

Kibana reads elasticsearch cluster data and is displayed on web. Developers, operators and other users who need to view logs log in to kibana to view.

Client-side Filebeat configuration

Filebeat.prospectors:- input_type: log paths:-/ home/logs/app/business.log-/ home/logs/app/exception.log json.message_key: log json.keys_under_root: trueoutput.kafka: hosts: ["10.82.9.202 home/logs/app/business.log 9092", "10.82.9.203 home/logs/app/business.log 9092", "10.82.9.204 home/logs/app/business.log 9092"] topic: filebeat_docker_java

Format of data received by Kafka

{"@ timestamp": "2018-09-05T13:17:46.051Z", "appname": "app01", "beat": {"hostname": "52fc9bef4575", "name": "52fc9bef4575", "version": "5.4.0"}, "classname": "com.domain.pay.service.ApiService", "date": "2018-09-05 2117953mm 0800", "filename": "ApiService.java", "hostname": "172.17.0.2", "level": "INFO" "linenumber": 285, "message": "param [{\" email\ ":\" TEST@163.COM\ ",\" claimeeIP\ ":\" 123.191.2.75\ ",\" AccountName\ ":\"\ "}]", "source": "/ home/logs/business.log", "thread": "Thread-11", "timestamp": 1536153465953, "type": "log"}

Server-side Logstash configuration

Input {kafka {bootstrap_servers = > "10.82.9.202 date 9092, 10.82.9.203 topics 9092, 10.82.9.204 topics > [" filebeat_docker_java "]}} filter {json {match = >" message "} date {match = > [" timestamp " "UNIX_MS"] target = > "@ timestamp"} output {elasticsearch {hosts = > ["10.82.9.205", "10.82.9.206", "10.82.9.207"] index = > "filebeat-docker-java-% {+ YYYY.MM.dd}"}}

The basic configuration is very simple, without too much explanation, the log collection of any application can be realized through the above simple configuration.

Log display

After collecting logs to elasticsearch, you can display the application logs through kibana configuration, which is convenient for developers to find problems in time and locate problems online.

Log printing Json format is not convenient to view locally? This can be written in the configuration file as the log output format, and different environments load different configurations, just as the development environment loads the development database.

Log system has been running steadily for nearly 2 years since it was launched. Except that it is not suitable at the beginning, it is more and more easy to use. Now they can not do without ELK log system, which greatly improves the efficiency of work.

On how to access the general application log in the ELK log system is shared here, I hope the above content can be of some help to you, can learn more knowledge. If you think the article is good, you can share it for more people to see.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.