Shulou(Shulou.com)06/01 Report--

This article mainly introduces "why the space is not released after the Linux deletes the file". In the daily operation, I believe that many people have doubts about why the space is not released after deleting the file by Linux. The editor consulted all kinds of information and sorted out a simple and easy-to-use method of operation. I hope it will be helpful to answer the question of "why the space is not released after the Linux deletes the file". Next, please follow the editor to study!

Have you ever encountered a situation in Linux where the file has been deleted but the space has not been released? This article will introduce a scenario of this problem and the corresponding solution.

One of our application servers, the operating system is Red Hat Linux, monitoring alarm, / opt/applog file system utilization exceeds the threshold, the overall capacity is 50g, but found that the actual file capacity is 20g, what is the remaining 30g space?

We know that in the Linux environment, everything exists in the form of files. In the background, the system assigns a file descriptor to each application, which provides a general interface for the interaction between the application and the operating system. Since it is a file, it will take up space. At this time, you can use the lsof instruction, which can list the files that are being opened by the current system.

> lsof COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME... Filebeat 111442 app 1r REG 253 REG 253 209715229 1040407 / opt/applog/E.20171016.info.012.log filebeat 111442 app 2R REG 253 REG 253 209715254 385080 / opt/applog/E.20171015.info.001.log (deleted)

The header fields have the following meanings:

COMMAND: the name of the process

PID: process identifier

USER: process owner

FD: a file descriptor that the application recognizes by the file descriptor. Such as cwd, txt, etc.

TYPE: file type, such as DIR, REG, etc.

DEVICE: specifies the name of the disk

SIZE: the size of the file

NODE: Inode (identification of files on disk)

NAME: the exact name of the open file

As you can see, in some lines, NAME identifies (deleted)

/ opt/applog/E.20171015.info.001.log (deleted)

What he means is that the file has been deleted, but the handle to the file has not been closed. The name of COMMAND is filebeat,USER process owner is app. This is our log collection process, and app users have opened the filebeat process.

Insert and broadcast the log collection platform

The traditional open source logging platform, ELK, consists of three open source tools, ElasticSearch, Logstash and Kiabana, among which:

Elasticsearch is an open source distributed search engine, distributed, zero configuration, automatic discovery, index automatic slicing, index copy mechanism, restful style interface, multiple data sources, automatic search load and so on.

Logstash is an open source collection tool that collects, filters, and stores logs for later use.

Kibana is an open source graphical Web tool that provides a log analysis friendly Web interface for Logstash and ElasticSearch to summarize, analyze, and search important data logs.

Common deployment diagrams, as shown below

What is the filebeat mentioned above? What's the connection with ELK?

Because logstash is run by jvm and consumes a lot of resources, the author later wrote a lightweight logstash-forwarder with less function but less resource consumption in golang. But the author is only a person, after joining http://elastic.co, because es itself also acquired another open source project, packetbeat, and this project is dedicated to golang, there is a whole team, so es simply merges the development work of logstash-forwarder into the same golang team, so the new project is called filebeat.

To put it simply, filebeat is the process agent of log collection, which is responsible for collecting application log files.

For my above problem, there are a large number of (deleted), unreleased file handles, and a background, that is, because the disk space is very limited, temporary tasks are added to delete logs 12 hours ago every hour. In other words, scheduled tasks automatically delete some files that are open by filebeat at this time, so these files become unreleased files, so the actual files are deleted. But the space has not been released.

Solution 1:

The most direct way to quickly free up space is the kill-9 filebeat process, where space is freed. But it is not fundamentally solved, the scheduled task will also delete these files opened by filebeat, resulting in full space.

Solution 2:

The configuration file filebeat.yml of filebeat actually has two parameters:

Close_older: 1h

Description: Close older closes the file handler for which were not modified for longer then close_older. Time strings like 2h (2 hours), 5m (5 minutes) can be used.

That is, if a file has not been updated within a certain period of time, the monitored file handle will be closed. The default is 1 hour.

Force_close_files: false

Description: This option closes a file, as soon as the file name changes. This config option is recommended on windows only. Filebeat keeps the files it's reading open. This can cause issues when the file is removed, as the file will not be fully removed until also Filebeat closes the reading. Filebeat closes the file handler after ignore_older. During this time no new file with the same name can be created. Turning this feature on the other hand can lead to loss of data on rotate files. It can happen that after file rotation the beginning of the new file is skipped, as the reading starts at the end. We recommend to leave this option on false but lower the ignore_older value to release files faster.

That is, when the name of a file changes, including renaming and deletion, a file will be automatically closed.

Combined with these two parameters, according to the application requirements, if a file is not updated within 30 minutes, the handle needs to be closed, the file is renamed or deleted, and the handle needs to be closed.

Close_older: 30m force_close_files: true

It can meet the basic requirements of collecting logs by filebeat and deleting historical files regularly.

At this point, the study on "why the space is not released after Linux deletes the file" is over. I hope to be able to solve everyone's doubts. The collocation of theory and practice can better help you learn, go and try it! If you want to continue to learn more related knowledge, please continue to follow the website, the editor will continue to work hard to bring you more practical articles!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.