Shulou(Shulou.com)06/02 Report--

This article will explain in detail about the working principle of SurfaceFlinger in Android, the content of the article is of high quality, so the editor will share it with you for reference. I hope you will have a certain understanding of the relevant knowledge after reading this article.

Concept

SurfaceFlinger is a system service, such as: audioflinger,audiopolicyservice, etc., the main services of the system through this article to understand, Android system services at a glance. In other words, in the implementation of Android, it is a service that provides system-wide surface composer functions, which can combine 2D and 3D surface of various applications.

Principle analysis

Let's first take a look at the following screen sketch:

Each application may correspond to one or more graphical interfaces, and each interface is called a surface, or window. In the above picture, we can see four surface, one is the home interface, and there are three surface represented by red, green and blue, and the two button are actually the content of the home surface. Here we can see the problems that we need to solve for graphic display:

First of all, each surface has its location and size on the screen, and then there is content, content, size and location to be displayed in each surface. These elements may change when we change the application. What should we do when we change the application?

B, and then there may be overlap between the various surface. For example, in the thumbnail above, green covers blue, while red covers green and blue, as well as the home below, and has some transparency. How should the relationship between these layers be described?

Let's first look at the second problem. We can imagine that there is a Z axis in the vertical direction of the screen plane. All the surface determine the front and back according to the coordinates on the Z axis, so that we can describe the upper and lower coverage relationship between each surface. In this order on the Z axis, there is a technical term called Z-order on the graph.

For the first question, we need a structure to record the location and size of the application interface, and a buffer to record what needs to be displayed, so this is our concept of surface, surface actually we can understand it as a container, this container records the control information of the application interface, such as size and location, and it also has buffer to store the content that needs to be displayed.

There is also a problem here, that is, when there is graphic overlap, how to deal with it, and some surface may also have transparent information, here is our SurfaceFlinger need to solve the problem, it needs to combine each surface (compose/merge) into a main Surface, and finally send the contents of Main Surface to FB/V4l2 Output, so that we can see the effect we want on the screen. In practice, there are two ways to merge these Surface, one is to use the form of software to merge, and the other is to use hardware, the way of software is our SurfaceFlinger, and the way of hardware is Overlay.

OverLay

Because the hardware merge content is relatively simple, let's first take a look at overlay. There are many ways to implement Overlay, but all of them need hardware support. Take IMX51 as an example. When IPU applies for FB from the kernel, it will apply for three FB, one for the main screen, one for the secondary screen, and one for Overlay. To put it simply, Overlay means that we send the format data and control information acceptable to the hardware to this Overlay FrameBuffer, and the hardware driver is responsible for the content of the merge Overlay buffer and the home screen buffer.

Generally speaking, current hardware supports only one Overlay, which is mainly used for video playback and camera preview, because it is much more efficient to use hardware Merge than software Merge for changing video content. Here is the process of using Overlay and not using Overlay:

Overlay hal has been added to SurfaceFlinger. As long as the Overlay hal is implemented, you can use the function of overlay. The header file is in:

/ hardware/libhardware/include/harware/Overlay.h

It can be implemented using FB or V4L2 output, which may be what we will work on in the future. After you implement Overlay hal, the sequence that uses the Overlay interface is as follows:

/ frameworks/base/libs/surfaceflinger/tests/overlays/Overlays.cpp

This sequnce is very important, which we will talk about later.

However, in practice, we do not necessarily need to implement Overlay hal, if we know the hardware, we can send this information directly to Overlay Buffer in the driver, without the need to go to the upper Android. Fsl's current Camera preview is in this way, and I took a cursory look at the contents of the R3 patch, which should also implement Overlay in the video playback of opencore.

SurfaceFlinger

Now let's take a look at the most complex SurfaceFlinger. First of all, we should make it clear that SurfaceFlinger is only responsible for the control of merge Surface, such as calculating the overlapping areas of two Surface, and the content that Surface needs to display is calculated through skia,opengl and pixflinger. So before we introduce SurfaceFlinger, we first ignore what is stored in it, figure out a series of processes of its control over merge, and then look at its processing with 2D and 3D engines.

3.1.The process of creating Surface

As mentioned earlier, each application may have one or more Surface, we need some data structures to store our window information, we also need buffer to store our window contents, and most importantly, we should determine a scheme to exchange this information with SurfaceFlinger. Let's first take a look at the following class diagram of the Surface creation process:

On the left side of the IBinder is the client side, which is the application that needs to be displayed in the window, and on the right is our Surface Flinger service. Creating a surface is divided into two processes, one is to create a management structure for each application (Client) on the SurfaceFlinger side, and the other is to create a buffer to store the content, and a series of operations such as drawing on this buffer.

Because SurfaceFlinger manages multiple window interfaces of multiple applications, it provides a Client class for management, and each application requesting a service corresponds to a Client. Because surface is created in SurfaceFlinger, a structure must be returned to let the application know the surface information it applied for, so SurfaceFlinger returns the control structure per_client_cblk_t created by Client to SurfaceComposerClient after being encapsulated by BClient, and provides the application with a set of operations to create and destroy surface:

After creating a Client for the application, all you need to do is assign the Surface to the Client. Flinger provides 8m of space for each Client, including the buffer that controls the information and stores the content. Before we talk about creating surface, we first need to understand the concept of layer. Let's go back to the screen sketch we saw earlier. In fact, each window is a layer on the z-axis. Layer provides the operation of window control information and the processing of content (call opengl or skia). That is to say, SurfaceFlinger only controls when the information should be processed and the process of processing. All the actual processing is carried out in layer. It can be understood to create a Surface is to create a Layer. I have to say that the messy names of Android have kept me around for a long time.

In the process of creating a Layer, the Client of the application first generates a unique layer ID based on the application's pid, and then creates the Layer based on information such as size, location, format, and so on. There is a nested Surface class in Layer, which mainly contains an ISurfaceFlingerClient::Surface_data_t, which contains the uniform identifier of the Surace and buffer information, etc., which are provided to the application. Finally, the application creates its own Surface based on the returned ISurface information, and so on.

Android provides four types of layer to choose from, each layer corresponds to a type of window, and corresponds to the corresponding operation of this window:

Layer, LayerBlur, LayerBuffer, LayerDim .

LayerBuffer is easily understood as the Buffer of Layer, which is actually a type of Layer. For the effects of each Layer, please refer to the description in Surface.java: / frameworks/base/core/java/android/view/surface.java. Here we will focus on two kinds of Layer, one is Layer (norm layer), and the other is LayerBuffer.

Norm Layer is one of the most widely used Layer in Android. General applications use this kind of layer when creating surface. Knowing Normal Layer can let us know some basic principles of Android in the process of display. Normal Layer assigns two buffer to each Surface: front buffer and back buffer, which are relative concepts, and they can Flip. Front buffer is used for SurfaceFlinger display, while Back buffer is used for application drawing. When Back buffer is filled with data (dirty), it will be flip, back buffer will become front buffer for display, and front buffer will become back buffer for drawing. The size of these two buffer is dynamically changed according to the size format of surface. I didn't take a closer look at the implementation of this dynamic change. I can refer to setbuffers () in / frameworks/base/lib/surfaceflinger/layer.cpp.

The two-buffer flip approach is an important implementation in Android display, not only for each Surface, but also for the main surface that is finally written into FB.

LayerBuffer is also a Layer that will be used in the future, personally, it is also the most complex layer, it does not have render buffer, it is mainly used in camera preview / video playback. It provides two ways of implementation, one is post buffer, and the other is overlay, which we mentioned earlier. The interface of Overlay is actually implemented on this layer. Both overlay and post buffer refer to the data of the layer coming from other sources, but whether the post buffer finally uses the software or the main FB of the layer merge, while the overlay is implemented through the hardware merge. Closely linked to this layer is the ISurface interface, which registers data sources. Let me give an example to illustrate the use of these two methods:

The first few steps are generic:

/ / to use Surfaceflinger services, you must first create a client.

Sp client = new SurfaceComposerClient ()

/ / then apply for a Surface from Surfaceflinger. The surface type is PushBuffers.

Sp surface = client- > createSurface (getpid (), 0320,240, PIXEL_FORMAT_UNKNOWN, ISurfaceComposer::ePushBuffers)

/ / then get the interface ISurface. If the function getISurface () is called with permission restrictions, it must be opened in Surface.h: / framewoks/base/include/ui/Surface.h.

Sp isurface = Test::getISurface (surface)

/ / create overlay in overlay mode, and then you can use the interface of overlay

Sp ref = isurface- > createOverlay (320,240, PIXEL_FORMAT_RGB_565)

Sp verlay = new Overlay (ref)

/ / in post buffer mode, first create a buffer, and then register the buffer on the ISurface

ISurface::BufferHeap buffers (w, h, w, h

PIXEL_FORMAT_YCbCr_420_SP

Transform

0

MHardware- > getPreviewHeap ()

MSurface- > registerBuffers (buffers)

3.2. Application control and drawing of windows

After the Surface is created, the application can draw pictures in buffer, so here are the following problems:

A. How do you know which buffer to draw on?

B, that is, how to inform SurfaceFlinger to flip after drawing?

C, in addition to drawing, if we move the window and change the size of the window, how to tell SurfaceFlinger to deal with it?

Before we understand these issues, we need to understand how the SurfaceFlinger service works:

You can see from the class diagram that SurfaceFlinger is a threaded class that inherits the Thread class. When the SurfaceFlinger service is created, a SurfaceFlinger listener thread will be started. The thread will wait for the occurrence of events, such as the need for sruface flip, or the size of the window has changed, and so on. Once these events are generated, SurfaceComposerClient will send a signal through IBinder, and the thread will end waiting for these events to be processed, and will continue to wait after the processing is completed, and so on.

SurfaceComposerClient and SurfaceFlinger synchronize signals through the class SurfaceFlingerSynchro, which, to put it bluntly, is a conditional variable. The value of the waiting condition of the listening thread becomes OPEN. Once it becomes OPEN, it ends the waiting and sets the condition to CLOSE and then handles the event. After the processing is completed, the value of the waiting condition becomes OPEN. Once the Surface of Client is changed, it notifies SurfaceFlinger through IBinder to change the value of the condition variable into OPEN, and wakes up the waiting thread. Thus, a dynamic processing mechanism is implemented through thread classes and condition variables.

Now that we understand the event mechanism of SurfaceFlinger, let's go back to the problems mentioned earlier. First of all, you must lock the layer of the Surface before drawing the Surface, which is actually the variable swapstate in the Layer_cblk_t. SurfaceComposerClient uses the value of swapsate to determine which buffer to use for drawing. If swapstate is the following value, Client will be blocked, so copy will not be translated directly:

/ / We block the client if:

/ / eNextFlipPending: we've used both buffers already, so we need to

/ / wait for one to become availlable.

/ / eResizeRequested: the buffer we're going to acquire is being

/ / resized. Block until it is done.

/ / eFlipRequested & & eBusy: the buffer we're going to acquire is

/ / currently in use by the server.

/ / eInvalidSurface: this is a special case, we don't block in this

/ / case, we just return an error.

So the application first calls lockSurface () to lock the swapstate of the layer, gets the buffer of the drawing, and then can draw on it, and then calls unlockSurfaceAndPost () to tell the SurfaceFlinger to Flip. Or just call unlockSurface () without notifying SurfaceFlinger.

Generally speaking, the process of drawing needs to redraw all the pixels on the Surface, because in general, the pixels displayed are not saved, but you can also save some pixels by setting and draw only part of the pixels. Here, it involves copying the pixels, and you need to copy the contents of the Front buffer to the Back buffer. Copying pixels is a frequent operation in SurfaceFlinger service implementation, and it may also involve the conversion of the copying process, such as screen rotation, flipping and so on. Therefore, Android provides hal for copying pixels, which may be what we need to achieve in the future, because using hardware to copy pixels and possible matrix transformation in the copying process is more efficient and resource-saving than using memcpy. The HAL header file is located in:

/ hardware/libhardware/hardware/include/copybit.h

PS: Windows Manager is a service of the java layer, which provides the management function of all windows. I haven't read this part in detail. I think it's something I need to know in the future.

Window state changes include position movement, window size, transparency, z-order, and so on. First let's take a look at how SurfaceComposerClient and SurfaceFlinger interact with this information. When the application needs to change the state of the window, it packages all the state change information and sends it to SurfaceFlinger together. After SurfaceFlinger changes these state information, it will wake up the waiting listening thread and set a flag bit to tell the listening thread that the state of the window has changed and must be processed. In the implementation of Android, the packaging process is a Transaction. All changes to window state (layer_state_t) must be in a Transaction.

At this point, the processing process of the application client is finished, which is basically divided into two parts, one is to draw a picture in the window, and the other is to deal with the change of the window state.

4. The process of SurfaceFlinger

After understanding the interaction between Flinger and the client, let's take a closer look at the processing of SurfaceFlinger. As mentioned earlier, the SurfaceFlinger service starts a listening thread when it is created, and this thread is responsible for handling each window update. Let's take a closer look at the event handling of this thread, which is roughly shown in the following figure:

First talk about the principle of Android combination of each window: Android is actually by calculating the visible region of each window, that is, the window region that we can see on the screen (visibleRegionScreen in Android's vocabulary), then draw the visible area of each window to the corresponding part of a main layer, and finally splice it into a complete screen, and then send the main layer to the FB display. The process of drawing the visible area of each window to the main layer involves a hardware implementation and a software implementation. If it is a software implementation, it will be redrawn through Opengl, including alpha calculation with transparency; if copybit hal is implemented, this part of the data of the window can be copied directly, and possible rotation, flipping, and alhpa calculation can be completed.

Let's take a look at how Android assembles each layer and sends it to FB for display:

4.1 、 handleConsoleEvent

After receiving the signal or singalEvent event, the thread stops waiting to start processing the Client request. The first step is handleConsoleEvent. This step is related to the / dev/console device. It will get the screen or release the screen. Only when it gets the screen can it draw on the screen.

4.2 、 handleTransaction

As mentioned earlier, window state changes can only be made in one Transaction. Because the change in the state of the window may cause changes in the visible area of this window and other windows, it is necessary to recalculate the visible area of the window. During this process, Android traverses all layer based on flag bits, and once it finds that the state of which window has changed, it sets flag bits to recalculate the visible area of the window in the future. After traversing all the child layer, the Android will also process the main layer according to the flag bit. For example, if the sensor senses that the phone is sideways, it will display the window horizontally, so it is necessary to reorient the main layer.

4.3 、 handlePageFlip

The flipping between the surface buffer of each window is handled here, and it is decided whether to flip according to the swapsate of the layer_state_t, which will be flipped when the value of swapsate is eNextFlipPending. After dealing with the flip, it recalculates the visible area of each layer. I don't quite understand the process of recalculation, but it's something like this:

Start with the layer with the highest Z value, that is, from the top layer, remove the transparent area and the opaque area covered by it, and get the visible region of the layer. Then the opaque area of the layer is accumulated to the opaque coverage area, and the visible area of the layer is placed into the visible area of the main layer, and then the next layer is calculated until the visible area of all the layer is calculated. This intermediate calculation is achieved through a graphical logic operation defined in skia, similar to the and or illogical graph in our mathematics.

4.4 、 handleRepaint

After calculating the visible region of each layer, this step is to draw the contents of all visible regions to the corresponding part of the main layer, that is to say, copy the corresponding contents of each surface buffer to the corresponding buffer of the main layer, which may also involve alpha operation, pixel flipping, rotation and other operations. As I said earlier, it can be implemented in hardware or software. In the use of software opengl to do the calculation process will also use PixFlinger to do pixel synthesis, this part of the content I do not have time to take a closer look.

4.5 、 postFrameBuffer

The final task is to flip the two buffer of the main layer and display what you have just written into the FB.

About what the working principle of SurfaceFlinger in Android is shared here, I hope the above content can be of some help to you, can learn more knowledge. If you think the article is good, you can share it for more people to see.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.