Shulou(Shulou.com)06/01 Report--

Introduction: in terms of the performance of the whole software, resource performance is like the ice below the tip of the iceberg. This part is made up of traditional parts of the disk, CPU, memory and network, as well as batteries (power consumption) that are particularly important because of the mobile network. In this article, we will focus on the disk section.

This article is selected from "Android Mobile performance practice".

1 principle

Before there is no SSD hard drive, everyone will think that our HDD hard drive is very easy to use, what 5400 rpm, 7200 rpm, advertising is very good. Until one day, SSD appeared, found that when starting Windows, unexpectedly can open in a second, this just woke up. Therefore, for the layman, the disk Iripple O performance is always the most easily ignored, and the effort will be more focused on CPU. But for insiders, we all know that performance is nothing more than CPU-intensive and Imax-O-intensive. Disk IPUBO is one of them. So what is the performance of our memory chips in the mobile era? Before we discuss this issue, let's take a look at a test data.

As shown in the figure above, our sequential read / write performance has improved very rapidly, and many new models have greatly improved sequential read / write performance compared to the previous ones, and the gap with SSD has been narrowed a lot. But here's the bad news: random read / write performance is still poor, see MOTO X, S7, iPhone 6S Plus. At this point, we must introduce you to the first concept: random read / write.

Random read / write

Random writing is everywhere. Give me two simple examples. The first example is the simplest. The journal file of the database will result in random writes. Random writes are triggered when writes occur back and forth between the db file and the journal file in the database. In the following table, simply insert a piece of data into test.db and monitor the interface of pwrite64. You can see that the shading in the table is written randomly. In the second example, if you insert multiple pieces of data into a database table with AUTOINCREMENT (the value of the automatically created primary key field), you need to manipulate two database tables for each piece of data inserted, which means there are random writes.

As can be seen from the above example, random read / write is relative to sequential read / write, and offset is randomly generated during read or write. But why is random reading / writing so slow?

1. Random reading loses the optimization effect of read-ahead.

two。 Compared with sequential writing, random writing not only produces a large number of invalid pages, but also increases the probability of triggering "write amplification" effect.

So what is "write amplification"? Let's introduce the second concept: the write amplification effect.

"write amplification" effect

When data is written for the first time, because all the particles are erased, the data can be written directly in the smallest unit of the page. When new data writes need to replace the old data, the master controller writes the new data to another blank flash space (erased state) and then updates the logical LBA address to point to the new physical FTL address. At this point, the old address content becomes invalid data, but the main controller does not perform an erase operation but marks the corresponding "page" as invalid. When the disk needs to be written again in the above invalid area, in order to get free space, flash memory must first copy all valid "pages" in the "block" to the new "block" and erase the old "block" before writing. (for further study, see: http://bbs.pceva.com.cn/forum.php?mod=viewthread&action=print able&tid=8277. )

For example, if you write a 4KB data now, the worst-case scenario is that there is no clean space in a block, but there happens to be a "page" of invalid data to be erased, so the master reads all the data out, erases the block, and writes back the new 4KB data. Looking back at the whole process, I only wanted to write the data of 4KB, which resulted in the write operation of the whole block (512KB). At the same time, the operation that only needs to simply write 4KB has become "flash read (512KB)-> cache change (4KB)-> flash erase (512KB)-> flash write (512KB)", which results in a great increase in latency, and slow speed is natural. This is the so-called write amplification (Write Amplification) problem.

Let's verify the existence of the write amplification effect by constructing the scene.

Scenario 1: write the 1MB file to the SD card normally, and count the time it takes to write the file.

Scenario 2: first fill the SD card with a small file of 6KB, and then delete the written file. This ensures that the SD card does not have clean data blocks. At this time, write the 1MB file to the SD card, and count the time it takes to write the file.

The following figure is the test data carried out on Samsung 9100, Samsung 9006 and Samsung 9300 respectively. From the test data, when there is no clean data block on the SD card, the writing time of the file is 1.9 to 6.5 times that of normal writing, so the test results can well illustrate the existence of the "write magnification" effect.

So when is the write amplification effect most likely to occur? External cause: the mobile phone has been used for a long time and the disk space is insufficient. Internal cause: the application triggers a large number of random writes. At this time, the time consumption of disk I _ peg O will fluctuate violently, and there is only one thing App can do, that is, to reduce the amount of operation of disk I _ hand O, especially the amount of operation of the main thread. So how to find, locate, and solve the performance problems of these disks? Of course we have to use our tools.

2 tool set

The toolset is shown in the following table.

STRICTMODE should be an entry-level must-have tool to find and locate the most affected main thread in the disk Imax O problem. As you can see from the following code, the enabling method is very simple.

Public void onCreate () {if (DEVELOPER_MODE) {StrictMode.setThreadPolicy (new StrictMode.ThreadPolicy.Builder () .sprintDiskReads () .returtDiskWrites () .returtNetwork () .penaltyLog () .build ()); super.onCreate ();}}

The principle is also very simple, mainly file operation (BlockGuardOs.java), database operation (SQLiteConnection. The check code is inserted into the interface between java) and SharePreferences operation (SharedPreferencesImpl.java). We have intercepted a piece of code to monitor the file operation in the Android source code, as follows. Finally, we actually call the onWriteToDisk method in StrictMode, and print the stack of the call to BlockGuardPolicyException to help locate the problem.

Detailed code:

Http://androidxref.com/4.4.4_r1/xref/libcore/luni/src/main/java/libcore/io/BlockGuardOs.java#91

Perfbox:I/OMonitor

Principle: the function of I/OMonitor can be summed up by collecting the information of Imax O that distinguishes process and scene through the method of Hook Java layer system Imax O.

1. Hook java method

The method of xposed O Monitor Hook java draws lessons from the open source project xposed. There are many articles about it on the Internet. Here we use a flow chart to briefly explain how to obtain the information of this operation.

two。 Istroke O information collection that distinguishes processes from scenarios

The following four steps are involved in the collection of Iamp O information that distinguishes processes from scenarios.

(1) app_process replacement

App_process is the entrance to the Java program in Android. By replacing app_process, you can control the entry and run our code in any application. The workflow of the replaced app_process is as follows.

(2) add libfork.so to the environment variable LD_PRELOAD

In UNIX, LD_PRELOAD is an environment variable that affects the runtime links of a program, allowing you to define dynamic link libraries that load first before the program runs. This feature can be used to selectively load the same functions in different dynamic link libraries. Set the LD_PRELOAD environment variable before the zygote process starts, so that all child processes of zygote inherit this environment variable. Libfork.so implements a fork function. When app_process uses the fork function to start the zygote process, it gives priority to the fork function implemented in libfork.so. The flow of the fork function is as follows.

(3) add XPlatform.jar to the environment variable CLASSPATH

XPlatform.jar is added to CLASSPATH so that plug-in jar like common.jar can use classes in XPlatform.jar. There is a similar thing in Mobile QQ, the development colleagues compiled the entire project into two dex files, after Mobile QQ started, put the second dex file into CLASSPATH (different from the XPlatform implementation method, but the same effect), so that the main dex can directly import and use the classes in the second dex. If you don't join CLASSPATH, you need to use the DexClassLoader class to use a class in another jar package, which is cumbersome and restrictive.

During system startup, the app_process process is actually the predecessor of the zygote process, so XPlatform.jar runs in the zygote process.

There are mainly two java methods Hook in XPlatform to monitor the startup of system_server process and application process, and to do some initialization operations in these processes. A feature of fork is used in this. The parent process uses fork to create the child process, and the child process inherits all the variables of the parent process. Because zygote uses fork to create the child process, Hook is performed in the zygote process, and any application process and system_server process created by it is also valid.

The XPlatform workflow flow chart is as follows.

In this way, when the application process starts, the function of controlling the running of I _ big O Monitor in the specified process is realized.

(4) Istroke O information collection based on distinguishing scenarios.

In order to realize the IWeiO information collection of different scenarios, we add a switch to the iUnip O Monitor, which corresponds to the Python control script, so that we can collect the iUniver O information of the specified scenario and make the test results more accurate.

In this way, we have implemented the Imap O information collection that distinguishes between processes and scenarios.

After introducing the principles of our tool, let's take a look at the collected Icano log information, including file paths, processes, threads, the number of times to read / write files, size and time, and the stack of calls.

The data in the XPlatform workflow chart shows that a file at a time corresponds to a line in the CSV file. Each time the system's API (read or write method) is called, the number of reads / writes (readcount, writecount) is increased by 1. Read / write time (readtime, writetime) is the time from open to close is calculated.

SQLite performance Analysis / Monitoring tool SQL I Compact O Monitor

We know that the final operation of the database operation is the DB file on the disk, and there is no difference between the DB file and the ordinary file in nature, and the performance of the DB system has always been the bottleneck of the computer, so the ultimate goal of optimizing the database often lies in how to reduce the disk Imax O.

Whether it is to optimize the table structure, use the index, increase the cache, adjust the page size, etc., the ultimate goal is to reduce the disk I mano, and these are our conventional means to optimize the database. Habits range from analyzing business characteristics to trying to optimize strategies to validating test results, so why can't we do it backwards? Since the purpose of database optimization is to reduce disk Icano, can we start directly from disk Icano data to see if there will be any unexpected gains.

1. Collect data of database Icano

In order to realize our idea, the first step is, of course, to collect the corresponding disk Imax O data during the database operation. Due to the previous Java Hook technology, we have obtained the Native Hook layer's IUnip O operation data. Although the SQLite's Icano operation is carried out in libsqlite.so and belongs to the Native layer, we will naturally think of collecting SQLite's Icano data through Java.

Native Hook is mainly implemented in the following ways.

(1) modify the environment variable LD_PRELOAD.

(2) modify sys_call_table.

(3) modify the register.

(4) modify GOT table.

(5) Inline Hook.

The following mainly introduces three implementation methods: (1), (4) and (5).

(1) modify the environment variable LD_PRELOAD

This method is the easiest to implement, rewriting the system functions open, read, write and close, and putting the so library into the environment variable LD_PRELOAD, so that when the program calls the system function, it will first look in the environment variable, so that the rewritten system function will be called. You can refer to the Snow Forum article "Android uses LD_PRELOAD for Hook" (http://bbs.pediy.com/showthread.php?t=185693).

But this kind of Hook is effective for the whole system, that is, all the Imax O operations of the system are Hook, resulting in a huge amount of data in Hook, and the system is easily stuck.

(4) modify GOT table

When referencing an external function, the address of the external function is stored in the .got table in the form of Stub at compile time, and the linker is relocated when loaded to write the real external function to this stub. The idea of Hook is to replace the address of the external function in the. Got table. However, the libsqlite.so operation is performed by calling the system function in libc.so, so it is feasible to modify the Hook scheme of GOT table.

However, the reality is not always plain sailing, when our scheme is implemented, we find that we can only record the open and close function calls in libsqlite.so, but we can not record the read/write calls caused by the internal mechanism of sqlite.

(5) Inline Hook

After the first two schemes are unsuccessful, you can only try Inline Hook. Inline Hook can be used to Hook so the internal functions of the library, and the first thing that comes to mind is the Hook libsqlite.so internal I seekandread O interface posixOpen, seekandread, seekandwrite, and robust_close. However, the road to success is always full of twists and turns, and most of the key functions in sqlite are defined as static functions, such as posixOpen. In C language, static functions do not export symbols, while Inline Hook is to find the corresponding function position in the symbol table. As a result, the path through Hook sqlite internal functions doesn't work again.

Static int posixOpen (const char * zFile, int flags, int mode) {return open (zFile, flags, mode);}

Since this doesn't work, we can only use the open, read, write, and close methods in Hook libc.so more violently. Because no matter what changes are made in sqlite, the system function will eventually be called, and the only bad thing is that all the IO data of the process is recorded in this way. This method is proved to be feasible in the self-compiled libsqlite.so.

When I was happy to call the libsqlite.so library that came with my phone, the read / write data was once again unrecorded, and my heart was almost broken. Why can I compile the libsqlite.so library myself, but not with the one on my phone? No way, can only look at the following source code, and finally found in seekAndRead, sqlite defines a lot of macro switches, you can decide to call the system functions pread, pread64 and read to read the file. Is it possible that so compiled by myself is not compiled in the same way as the so in my phone?

Static int seekAndRead (unixFile * id, sqlite3_int64 offset, void * pBuf, int cnt) {int got; int prior = 0 positron if (! defined (USE_PREAD) & &! defined (USE_PREAD64)) i64 newOffset;#endif TIMER_START; do {# if defined (USE_PREAD) got = osPread (id- > h, pBuf, cnt, offset); SimulateIOError (got =-1); # elif defined (USE_PREAD64) got = osPread64 (id- > h, pBuf, cnt, offset); SimulateIOError (got =-1) # else newOffset = lseek (id- > h, offset, SEEK_SET); SimulateIOError (newOffset--)

The author also Hook pread and pread64, this time finally recorded the complete Icano data, it turns out that the libsqlite.so in the mobile phone calls the pread64 and pwrite64 functions of the system to carry out the page O operation, and obtains the type of page when the database reads / writes the disk through Inline Hook. The page types of sqlite include table leaf page, table internal page, index leaf page, index internal page and overflow page.

After a lot of hard work, I finally got the information about the database read / write disk, but what's the use of this information? We can think of the following uses.

The optimization effect of the database is directly verified by the amount of Icano data.

Through the offset to find out the random read / write to optimize.

However, we are faced with another problem, because the disk information obtained is based on DB files, while the operation of the database in the application layer is based on tables and lack of stack, so it is difficult to locate the problem. Based on this, we think of another solution, that is, the database operation of the Hook application code, which corresponds to the two through the stack, so that the application code can be linked together and it is more convenient to analyze the problem.

2. Hook application layer SQL operation

The Hook application code is actually the database addition, deletion, modification and query operation in Hook SQLiteDatabase. The application code SQL statement is as follows, and the Java layer Hook is implemented based on Xposed.

Eventually, you can correspond to the disk information through the stack.

After obtaining so much data, we will introduce some database-related cases to you in the following push to see how they are applied.

This article is selected from "Android Mobile performance Battle". Click this link to view this book on the blog Viewpoint website.

For more wonderful articles in time, search for "blog viewpoints" on Wechat or scan the QR code below and follow.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.