Shulou(Shulou.com)06/01 Report--

Today, I will talk to you about how APP keeps logs efficiently, which may not be well understood by many people. In order to make you understand better, the editor has summarized the following contents for you. I hope you can get something according to this article.

Anyone who has experience in embedded development knows that log storage is a troublesome problem. Previously, ARM opened up a project: the cmbacktrace,github address will not be posted, and the stack information of crash can be saved. But for the mobile phone app, it is far from enough to save the log information when the program crashes, and it is impossible to locate the specific reason. Before I explain the log system used by Taos, let's take a look at the most general log system.

General plan one

The mainstream log module, such as logback, belongs to the real-time logging system, that is, every log generated is encrypted and stored in the disk file. A disadvantage of this is that Iramp O is too intensive and takes up a lot of CPU resources, which affects the performance of the program and is easy to stutter. Generally used with the app development and debugging phase, the official release version is sure to turn off the log function, and when users have reaction problems such as program crashes, they will reproduce a debug package. But many times the scene is not exactly the same, it is difficult to reproduce the problem.

Earlier, I talked about the relationship between cache data and disk data: how do processes use memory?

When the program writes files, it does not directly manipulate the files on the disk, but first writes the data to the system cache, that is, the dirty pages, and then the operating system daemon, the update process, updates the dirty pages to the disk regularly (usually 30s). There are two copies of data written to disk: from user space memory to kernel space memory, and then from kernel space flush to disk. In addition, the specific operation of writing to disk cannot be controlled by the program, so the high peak value of CPU will lead to performance degradation.

General plan 2

Of course, it is convenient to use Android's logsdk directly, but it can cause performance problems, so we can first save the log to the in-memory cache and then encrypt the write file after reaching a certain size. At the same time, in order to reduce the data flow, we first compress the data and then write it (learn from the practice of HTTP network transmission). The overall plan is shown below:

This scheme does not need to save logs in real time and will not generate CPU spikes. But the problem of losing the log is still unsolved. For example, when the program crash exits abnormally, there will be no notification of system events in many scenarios, so the stack of crash cannot be saved. Tencenttiny OS, the embedded operating system of Goose Factory, claims to solve this problem on the stm32 series of veneers, but it is still not satisfactory to use. More importantly, the Android system is much more complex than the embedded system, and many schemes can not be applied directly.

Optimization scheme of hand Amoy

Handy app mainly improves the efficiency of log use from two aspects.

Mmap

By calling the mmap storage mapping Ibank O, the specific solution is to map the disk files to the user space buffer, and when performing data operations on the memory buffer, it is equivalent to manipulating the files in the disk. The advantage is that you don't need to use read and write. The mapping scheme is shown in the following figure.

The advantage of using mmap is to avoid a real-time data copy, and you can write dirty data pages back to disk files by calling msync and munmap, which is controllable for the application layer (as long as a background thread executes asynchronously). The specific effect after optimization can be seen in figure 14-28 of APUE.

Note here that intmadvise (caddr_t addr, size_t len, int advice); this interface explains in Unix that you can preload disk files into memory by using the madv_willneed parameter, but the experiment on mac has no effect, so the proper solution is to read one byte per page to ensure that the files are loaded into memory.

Data compression

Should the data be encrypted or compressed first? The standard practice is to compress and then encrypt. Plaintext generally has redundancy, and the content will become less after compression. On the contrary, if you encrypt it first, it will destroy the redundancy of the file. The general compression methods mainly include gzip, huffman and so on. Handao draws lessons from HTTP2's hpack header compression, and optimizes the network transmission on the basis of facebook's open source zstd compression algorithm. Because there are usually a large number of phrases with the same characteristics in the log, a dictionary is resident on the server, and timely updates are pulled by the client to improve the compression ratio through dictionary compression. At the same time, in order to prevent data corruption from affecting the whole compression package, streaming compression is adopted (that is, compression is carried out when the log reaches 32k). Although the overall efficiency of block compression is slightly lower, it will not cause CPU peak value because it will not be packed and compressed in a short time.

In the actual use, the external SD card is also adapted. When the SD card is deleted, the log is stored in the temporary cache, and the cache is cleaned every time the process is killed, so as to prevent public opinion caused by the occupation of user space for a long time.

After reading the above, do you have any further understanding of how APP keeps logs efficiently? If you want to know more knowledge or related content, please follow the industry information channel, thank you for your support.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.