Shulou(Shulou.com)06/01 Report--

How to solve the problem that filebeat occupies unreleased Linux space? in view of this problem, this article introduces the corresponding analysis and solution in detail, hoping to help more partners who want to solve this problem to find a more simple and feasible method.

One of our application servers, the operating system is Red Hat Linux, monitoring alarm, / opt/applog file system utilization exceeds the threshold, the overall capacity is 50g, but found that the actual file capacity is 20g, what is the remaining 30g space?

We know that in the Linux environment, everything exists in the form of files. In the background, the system assigns a file descriptor to each application, which provides a general interface for the interaction between the application and the operating system. Since it is a file, it will take up space. At this time, you can use the lsof instruction, which can list the files that are being opened by the current system.

> lsof

COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME

...

Filebeat 111442 app 1r REG 253,3 209715229 1040407 / opt/applog/E.20171016.info.012.log

Filebeat 111442 app 2R REG 253 Magi 3 209715254 385080 / opt/applog/E.20171015.info.001.log (deleted)

...

The header fields have the following meanings:

COMMAND: the name of the process

PID: process identifier

USER: process owner

FD: a file descriptor that the application recognizes by the file descriptor. Such as cwd, txt, etc.

TYPE: file type, such as DIR, REG, etc.

DEVICE: specifies the name of the disk

SIZE: the size of the file

NODE: Inode (identification of files on disk)

NAME: the exact name of the open file

As you can see, in some lines, NAME identifies (deleted)

/ opt/applog/E.20171015.info.001.log (deleted)

What he means is that the file has been deleted, but the handle to the file has not been closed. The name of COMMAND is filebeat,USER process owner is app. This is our log collection process, and app users have opened the filebeat process.

Insert and broadcast the log collection platform

Elasticsearch is an open source distributed search engine, distributed, zero configuration, automatic discovery, index automatic slicing, index copy mechanism, restful style interface, multiple data sources, automatic search load and so on.

Logstash is an open source collection tool that collects, filters, and stores logs for later use.

Kibana is an open source graphical Web tool that provides a log analysis friendly Web interface for Logstash and ElasticSearch to summarize, analyze, and search important data logs.

What is the filebeat mentioned above? What's the connection with ELK?

Zhihu has an introduction by Niu Rao Chenlin (author of the authoritative Guide to ELKstack), which is very incisive.

Quoted from https://www.zhihu.com/question/54058964/answer/137882919

Because logstash is run by jvm and consumes a lot of resources, the author later wrote a lightweight logstash-forwarder with less function but less resource consumption in golang. But the author is only a person, after joining http://elastic.co, because es itself also acquired another open source project, packetbeat, and this project is dedicated to golang, there is a whole team, so es simply merges the development work of logstash-forwarder into the same golang team, so the new project is called filebeat. To put it simply, filebeat is the process agent of log collection, which is responsible for collecting application log files.

For my above problem, there are a large number of (deleted), unreleased file handles, and a background, that is, because the disk space is very limited, temporary tasks are added to delete logs 12 hours ago every hour. In other words, scheduled tasks automatically delete some files that are open by filebeat at this time, so these files become unreleased files, so the actual files are deleted. But the space has not been released.

Solution 1:

The configuration file filebeat.yml of filebeat actually has two parameters

Close_older: 1h

Description: Close older closes the file handler for which were not modified for longer then close_older. Time strings like 2h (2 hours), 5m (5 minutes) can be used.

That is, if a file has not been updated within a certain period of time, the monitored file handle will be closed. The default is 1 hour.

Force_close_files: false

Description: This option closes a file, as soon as the file name changes. This config option is recommended on windows only. Filebeat keeps the files it's reading open. This can cause issues when the file is removed, as the file will not be fully removed until also Filebeat closes the reading. Filebeat closes the file handler after ignore_older. During this time no new file with the same name can be created. Turning this feature on the other hand can lead to loss of data on rotate files. It can happen that after file rotation the beginning of the new file is skipped, as the reading starts at the end. We recommend to leave this option on false but lower the ignore_older value to release files faster.

That is, when the name of a file changes, including renaming and deletion, a file will be automatically closed.

Combined with these two parameters, according to the application requirements, if a file is not updated within 30 minutes, the handle needs to be closed, the file is renamed or deleted, and the handle needs to be closed.

Close_older: 30m

Force_close_files: true

It can meet the basic requirements of collecting logs by filebeat and deleting historical files regularly.

This is the answer to the question about how to solve the problem that filebeat occupies unreleased Linux space. I hope the above content can be of some help to you. If you still have a lot of doubts to be solved, you can follow the industry information channel for more related knowledge.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.