Shulou(Shulou.com)06/02 Report--

Preface

Recently, I have encountered a very thorny problem in my work. I need to read all the files under a certain directory in the ubuntu system. Because there are so many files stored in the server, this process is very inefficient. It is easy to wait for as long as an hour, and it is only a directory. So how to quickly get the list of files is the top priority of these two days, after a long time to find a faster way, record as follows, say no more, let's take a look at the detailed introduction.

Multiple implementation methods

A variety of methods have been tried, both programmatic and non-programmable.

1 、 walk

Python's walk library can recursively read all the files in the directory, which is the most common method, but the efficiency is a little slow. The implementation is simple and will not be dwelt on.

2 、 os.scandir

The os.scandir method in python is officially interpreted as reading directories quickly, and tests that the speed is improved compared with walk, but it still does not meet the requirements, and you need to write your own recursion. The code is as follows:

Def scan_path (file_path, level = 3): files = [] if level > = 0: path = os.scandir (file_path) for p in path: if p.is_dir (): files.extend (scan_path (p.path, level-1)) else: files.append (p.path) return files

When neither of these methods worked, I began to think about using non-programming methods. Theoretically, the execution efficiency of python is quite high, although it may not reach the speed of c or C++, but it is fast enough compared with java and C#, so it no longer considers the way of programming, but turns to the native way of linux system.

3 、 ls

The first thing that comes to mind is the ls command, using the following command

Ls-l-R (or-lR) src > list.txt

This command can list all the files in the src directory, but it is still not efficient enough, and the result contains directory information and file information, which is not neat and needs to be processed later.

4 、 tree

The tree command itself is used to list the structure tree of the file system, and the function of listing all directories and files can be achieved by setting some parameters.

Tree-afi-L 3-o 2.txt-- noreport src

-a lists all files,-f lists the full path (the result is absolute path or relative path is the same as find usage),-I does not draw the structure line of tree,-L lists how many layers of directory,-o outputs to files,-- noreport does not want the final summary.

5 、 find

The find command itself is a command to find files, but if used properly, you can quickly list the files in the directory, as follows:

Find src > 1.txt

This command is fast enough to basically meet the needs. The result of find is relative to the path of the current src, that is, each result begins with src. If src is an absolute path, the result is an absolute path, and if src is a relative path, the result starts with a relative path.

6 、 locate

Google for a while, found that the function of locate and find is similar, locate can also find files, so guess locate can also achieve this function, try it, sure enough, the writing is the same.

Locate src > 1.txt

The difference is that whether src is a relative path or an absolute path, the result is an absolute path.

Using the time command to test the command execution time, it is found that the find and locate times are basically the same, sometimes the locate is slightly faster, while the tree command is a little slow.

Conclusion

All the files under the folder can be accessed above, and it is most convenient to integrate with the program using walk and scandir, but the speed is a little slow. Find and locate commands are faster, and tree commands are powerful, but the speed is a little slower than that of find and locate. If you want to integrate these three with python, you need to use pipeline mechanisms such as os.popen to execute spliced bash commands in the program. Therefore, each of the above orders has his own strengths and chooses according to his own needs.

Summary

The above is the whole content of this article, I hope that the content of this article has a certain reference and learning value for your study or work, if you have any questions, you can leave a message and exchange, thank you for your support.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.