Shulou(Shulou.com)06/02 Report--

This article mainly introduces the relevant knowledge of "what is the design idea and internal implementation method of BufferQueue". The editor shows you the operation process through an actual case. The operation method is simple, fast and practical. I hope this article "what is the design idea and internal implementation method of BufferQueue" can help you solve the problem.

1. Background

For business developers, there is no access to BufferQueue or even what BufferQueue is. For the system, BufferQueue is a very important component for data transmission. The Android display system depends on BufferQueue. As long as the content is displayed to the "screen" (in this case, an abstract screen and sometimes an encoder can be included), BufferQueue must be used. It can be said that BufferQueue is everywhere in the display / player-related field. Even if you directly call Opengl ES to draw, the underlying layer still needs BufferQueue to display on the screen.

Understanding BufferQueue can not only enhance your understanding of the Android system, but also understand / troubleshoot related problems, such as why Mediacodec calls dequeueBuffer always return-1? Why does the draw method of ordinary View draw content directly, and why does SurfaceView still need unlockCanvasAndPost after draw?

Note: the code analyzed in this article comes from Android6.0.1.

2. Internal operation of BufferQueue

BufferQueue is the core of Android display system, and its design philosophy is producer-consumer model. As long as data is filled in BufferQueue, it is regarded as producer, and as long as data is obtained from BufferQueue, it is considered as consumer. Sometimes the same class may be either a producer or a consumer in different scenarios. For example, SurfaceFlinger, when compositing and displaying UI content, the UI element produces content as a producer, and SurfaceFlinger consumes it as a consumer. When taking a screenshot, SurfaceFlinger, as a producer, fills the current composite displayed UI content into another BufferQueue, and the screenshot application, as a consumer, obtains data from the BufferQueue and produces screenshots.

The following are common steps for using BufferQueue:

Initialize a BufferQueue

The producer of graphic data applies for a GraphicBuffer through BufferQueue, corresponding to the dequeueBuffer method in the diagram.

After applying for the GraphicBuffer, get the GraphicBuffer and obtain it through the function requestBuffer

After obtaining the GraphicBuffer, fill the GraphicBuffer with graphic data in various forms, and then queue the GraphicBuffer into the BufferQueue, corresponding to the queueBuffer method in the figure above.

When a new GraphicBuffer joins the BufferQueue, BufferQueue will notify consumers of graphics data through callback that new graphics data has been produced.

Then the consumer sends out a GraphicBuffer from the BufferQueue to correspond to the acquireBuffer method in the figure

After the consumer consumes the graphic data, the empty GraphicBuffer is returned to BufferQueue for reuse, which corresponds to the releaseBuffer method in the figure above.

At this time, the BufferQueue notifies the producer of the graphic data that the GraphicBuffer is available through callback, and the producer of the graphic data can obtain an empty GraphicBuffer from the BufferQueue to fill the data.

Cycle 2-8 steps all the time, so that the production-consumption of graphic data is completed methodically.

Of course, graphic data producers can not wait for BufferQueue callback to reproduce data, but keep producing data and then queue up to BufferQueue until BufferQueue is full. Consumers of graphic data can also try to obtain data from BufferQueue every time without waiting for the callback notification from BufferQueue, but it is inefficient and requires constant rotation training of BufferQueue (because BufferQueue has synchronous blocking and asynchronous blocking, failure to obtain data under the asynchronous blocking mechanism will not block the thread until there is data to wake up the thread, but directly return-1).

Producers and consumers who use BufferQueue at the same time are often in different processes. BufferQueue uses shared memory and Binder to transfer data in different processes to reduce data copies and improve efficiency.

Several classes related to BufferQueue are:

The actual implementation of BufferBufferCore:BufferQueue

BufferSlot: used to store GraphicBuffer

BufferState: indicates the status of the GraphicBuffer

The producer interface of IGraphicBufferProducer:BufferQueue, and the implementation class is BufferQueueProducer

The consumer interface of IGraphicBufferConsumer:BufferQueue, and the implementation class is BufferQueueConsumer

GraphicBuffer: represents a Buffer that can be filled with image data

Parent class of ANativeWindow_Buffer:GraphicBuffer

ConsumerBase: implements the ConsumerListener interface, which is called when data is queued to notify consumers

In BufferQueue, BufferSlot is used to store GraphicBuffer and an array is used to store a series of BufferSlot. The default size of the array is 64.

GraphicBuffer uses BufferState to represent its status, which has the following states:

FREE: indicates that the Buffer is not used by the producer-consumer, and the ownership of the Buffer belongs to BufferQueue

DEQUEUED: indicates that the Buffer has been acquired by the producer, and the ownership of the Buffer belongs to the producer

QUEUED: indicates that the Buffer is populated with data by the producer and has joined the queue to BufferQueue. The ownership of the Buffer belongs to BufferQueue.

ACQUIRED: indicates that the Buffer has been acquired by the consumer, and the ownership of the Buffer belongs to the consumer

Why do you need these states? Assuming that these states are not needed, implement a simple BufferQueue, assuming the following implementation:

BufferQueue {

Vector slots

Void push (GraphicBuffer slot) {

Slots.push (slot)

}

GraphicBuffer pull () {

Return slots.pull ()

}

}

After the producer has finished producing the data, the data is inserted into the vector by calling the push function of BufferQueue. The consumer calls BufferQueue's pull function to send out an Buffer data.

The problem with the above implementation is that producers need to create their own GraphicBuffer each time, while consumers release the GraphicBuffer each time they consume the data, and the GraphicBuffer is not recycled. In Android, because the producer-consumer of BufferQueue is often in different processes, GraphicBuffer needs to connect the producer-consumer process through shared memory. Every time a GraphicBuffer is created, it means that shared memory needs to be created, which is inefficient.

Using BufferState to represent the state of GraphicBuffer in BufferQueue solves this problem. Each GraphicBuffer has its current state, and the reuse of GraphicBuffer is completed by maintaining the state of GraphicBuffer.

Since the internal implementation of BufferQueue is BufferQueueCore, BufferQueueCore is used instead of BufferQueue below. This paper first introduces the corresponding data structure within BufferQueueCore, and then introduces the state twisting process and production-consumption process of BufferQueue.

The following is the process of queue / dequeue operation of Buffer and state reversal of BufferState. Only asynchronous blocking mode is introduced here.

2.1 BufferQueueCore internal data structure

The core data structure is as follows:

BufferQueueDefs::SlotsType mSlots: Slot stored in an array. The default size of the array is BufferQueueDefs::NUM_BUFFER_SLOTS, specifically 64, which represents all Slot.

Std::set mFreeSlots: all Slot with the status of FREE at present. These Slot are not associated with a specific GraphicBuffer, and need to be associated with GraphicBuffer for subsequent use.

Std::list mFreeBuffers: all Slot with the status of FREE. These Slot are associated with specific GraphicBuffer and can be used directly.

Fifo mQueue: a first-in, first-out queue that holds producer production data

When BufferQueueCore is initialized, since there is no data in the queue at this time, according to the above description, mFreeSlots should contain all Slot. The element size is the same as mSlots. The initialization code is as follows:

For (int slot = 0; slot)

< BufferQueueDefs::NUM_BUFFER_SLOTS; ++slot) { mFreeSlots.insert(slot); }2.2 生产者dequeueBuffer 当生产者可以生产图形数据时，首先向BufferQueue中申请一块GraphicBuffer。调用函数BufferQueueProducer.dequeueBuffer，如果当前BufferQueue中有可用的GraphicBuffer，则返回其对用的索引；如果不存在，则返回-1，代码在BufferQueueProducer，流程如下： status_t BufferQueueProducer::dequeueBuffer(int *outSlot, sp *outFence, bool async, uint32_t width, uint32_t height, PixelFormat format, uint32_t usage) { //1. 寻找可用的Slot，可用指Buffer状态为FREE status_t status = waitForFreeSlotThenRelock("dequeueBuffer", async, &found, &returnFlags); if (status != NO_ERROR) { return status; } //2.找到可用的Slot，将Buffer状态设置为DEQUEUED，由于步骤1找到的Slot状态为FREE,因此这一步完成了FREE到DEQUEUED的状态切换 *outSlot = found; ATRACE_BUFFER_INDEX(found); attachedByConsumer = mSlots[found].mAttachedByConsumer; mSlots[found].mBufferState = BufferSlot::DEQUEUED; //3. 找到的Slot如果需要申请GraphicBuffer，则申请GraphicBuffer，这里采用了懒加载机制，如果内存没有申请，申请内存放在生产者来处理 if (returnFlags & BUFFER_NEEDS_REALLOCATION) { status_t error; sp graphicBuffer(mCore->

MAllocator- > createGraphicBuffer (width, height, format, usage, & error)

GraphicBuffer- > setGenerationNumber (mCore- > mGenerationNumber)

MSlots [* outSlot] .mGraphicBuffer = graphicBuffer

}

}

The key is that the process for finding available Slot,waitForFreeSlotThenRelock is as follows:

Status_t BufferQueueProducer::waitForFreeSlotThenRelock (const char* caller

Bool async, int* found, status_t* returnFlags) const {

/ / 1. Whether there is too much mQueue

Bool tooManyBuffers = mCore- > mQueue.size () > static_cast (maxBufferCount)

If (tooManyBuffers) {

} else {

/ / 2. First find out whether there is anything available in mFreeBuffers. As can be seen from the introduction of 2.1, the elements in mFreeBuffers are associated with GraphicBuffer and are directly available.

If (! mCore- > mFreeBuffers.empty ()) {

Auto slot = mCore- > mFreeBuffers.begin ()

* found = * slot

MCore- > mFreeBuffers.erase (slot)

} else if (mCore- > mAllowAllocation & &! mCore- > mFreeSlots.empty ()) {

/ / 3. Then find out if there is anything available in mFreeSlots, as shown by 2.1. the list will be filled during initialization, so the first call must not be empty. At the same time, the elements in this list need to be associated with GraphicBuffer before they can be used directly, and the associated process is implemented by outer functions.

Auto slot = mCore- > mFreeSlots.begin ()

/ / Only return free slots up to the max buffer count

If (* slot

< maxBufferCount) { *found = *slot; mCore->

MFreeSlots.erase (slot)

}

}

}

TryAgain = (* found = = BufferQueueCore::INVALID_BUFFER_SLOT) | |

TooManyBuffers

/ / 4. If you cannot find the available Slot or if there are too many buffers (in synchronous blocking mode), you may need to wait

If (tryAgain) {

If (mCore- > mDequeueBufferCannotBlock & &

(acquiredCount mMaxAcquiredBufferCount)) {

Return WOULD_BLOCK

}

MCore- > mDequeueCondition.wait (mCore- > mMutex)

}

}

The waitForFreeSlotThenRelock function tries to find an available Slot, the available Slot state must be FREE (because it is obtained from the list of two FREE states), and then dequeueBuffer changes the state to DEQUEUED, which completes the state reversal.

There are two types of Slot that can be returned by waitForFreeSlotThenRelock:

Obtained from mFreeBuffers, the element in mFreeBuffers is associated with GraphicBuffer and is directly available.

The one obtained from mFreeSlots is not associated with GraphicBuffer, so you need to apply for GraphicBuffer and associate it with Slot. Apply for a GraphicBuffer through createGraphicBuffer, and then assign a value to the mGraphicBuffer of Slot to complete the association.

Summary dequeueBuffer: try to find a Slot, and complete the association between Slot and GraphicBuffer (if necessary), then reverse the state of Slot from FREE to DEQUEUED, and return the index of Slot corresponding to mSlots in BufferQueueCore.

2.3 producer requestBuffer

After the index of the available Slot is obtained by the dequeueBuffer function, the corresponding GraphicBuffer is obtained through requestBuffer. The process is as follows:

Status_t BufferQueueProducer::requestBuffer (int slot, sp* buf) {

/ / 1. Determine whether the slot parameter is legal or not

If (slot

< 0 || slot >

= BufferQueueDefs::NUM_BUFFER_SLOTS) {

BQ_LOGE ("requestBuffer: slot index% d out of range [0,% d)"

Slot, BufferQueueDefs::NUM_BUFFER_SLOTS)

Return BAD_VALUE

} else if (mSlots[ slot] .mBufferState! = BufferSlot::DEQUEUED) {

BQ_LOGE ("requestBuffer: slot% d is not owned by the producer"

"(state =% d)", slot, mslots [slot] .mBufferState)

Return BAD_VALUE

}

/ / 2. Set mRequestBufferCalled to true

M slope [slot] .mRequestBufferCalled = true

* buf = mSlts[ slot] .mGraphicBuffer

Return NO_ERROR

}

This step is not necessary. The business layer can obtain the corresponding GraphicBuffer directly through the index of Slot.

2.4 producer queueBuffer

After the above dequeueBuffer obtains a Slot, you can complete the production of image data on the GraphicBuffer corresponding to Slot, which can be the main thread Draw process of View, the child thread drawing process of SurfaceView, or even the decoding process of MediaCodec.

After filling in the image data, you need to enter the Slot into the BufferQueueCore (after the data is written, you can pass it to the producer-consumer queue for consumers to consume), and call the queueBuffer function in the queue. The process of queueBuffer is as follows:

Status_t BufferQueueProducer::queueBuffer (int slot

Const QueueBufferInput & input, QueueBufferOutput * output) {

/ / 1. First determine whether the incoming Slot is legal

If (slot

< 0 || slot >

= maxBufferCount) {

BQ_LOGE ("queueBuffer: slot index% d out of range [0,% d)"

Slot, maxBufferCount)

Return BAD_VALUE

}

/ / 2. Reverse the Buffer state to QUEUED, which completes the process of Buffer state from DEQUEUED to QUEUED.

MSlots.mFence = fence

M slope [slot] .mBufferState = BufferSlot::QUEUED

+ + mCore- > mFrameCounter

M slots. MFrameNumber = mCore- > mFrameCounter

/ / 3. Join the team mQueue

If (mCore- > mQueue.empty ()) {

MCore- > mQueue.push_back (item)

FrameAvailableListener = mCore- > mConsumerListener

}

/ / 4. Call back frameAvailableListener to inform consumers that they have data to join the queue.

If (frameAvailableListener! = NULL) {

FrameAvailableListener- > onFrameAvailable (item)

} else if (frameReplacedListener! = NULL) {

FrameReplacedListener- > onFrameReplaced (item)

}

}

As you can see from the comments above, the main steps for queueBuffer are as follows:

Reverse the Buffer state to QUEUED, which completes the process of Buffer state from DEQUEUED to QUEUED.

Queue Buffer into the mQueue queue of BufferQueueCore

Callback frameAvailableListener, informing consumers that they have data to join the queue and can consume data. FrameAvailableListener is a callback for consumer registration.

Summary queueBuffer: change the status of Slot to QUEUED, add it to mQueue, and finally notify consumers that there is data to join the queue.

2.5 Consumer acquireBuffer

When the consumer receives the onFrameAvailable callback or the consumer actively wants to consume data, call acquireBuffer to try to get a data from BufferQueueCore for consumption. The consumer code is in BufferQueueConsumer, and the acquireBuffer process is as follows:

Status_t BufferQueueConsumer::acquireBuffer (BufferItem* outBuffer

Nsecs_t expectedPresent, uint64_t maxFrameNumber) {

/ / 1. If the queue is empty, return directly

If (mCore- > mQueue.empty ()) {

Return NO_BUFFER_AVAILABLE

}

/ / 2. Take the first element of the mQueue queue and remove it from the queue

BufferQueueCore::Fifo::iterator front (mCore- > mQueue.begin ())

Int slot = front- > mSlot

* outBuffer = * front

MCore- > mQueue.erase (front)

/ / 3. In the case of dealing with expectedPresent, the "display" time of several consecutive Slot may be less than that of expectedPresent. In this case, these Slot are already "outdated". Go directly to the releaseBuffer consumption process below. The code is relatively long and ignored.

/ / 4. Update the status of Slot to ACQUIRED

If (mCore- > stillTracking (front)) {

M slope [slot] .mAcquireCalled = true

M slots [slot] .mNeedsCleanupOnRelease = false

M slope [slot] .mBufferState = BufferSlot::ACQUIRED

MSlots.mFence = Fence::NO_FENCE

}

/ / 5. If there is a direct releaseBuffer process in step 3, call back to the producer, and some data is consumed.

If (listener! = NULL) {

For (int I = 0; I

< numDroppedBuffers; ++i) { listener->

OnBufferReleased ()

}

}

}

As you can see from the comments above, the main steps for acquireBuffer are as follows:

Take and remove an element from the mQueue queue

Change the corresponding state of Slot to ACQUIRED

If there is frame drop logic, the callback informs the producer that some data is being consumed and the producer can prepare the production data.

Summary acquireBuffer: reverse the status of the Slot to ACQUIRED, remove it from the mQueue, and finally notify the producer that there is data out of the team.

2.6Consumer releaseBuffer

Consumers start to consume data after obtaining Slot (typical consumption, such as UI synthesis of SurfaceFlinger). After consumption, you need to inform BufferQueueCore that the Slot has been consumed by consumers, and you can re-produce data to producers. The releaseBuffer process is as follows:

Status_t BufferQueueConsumer::releaseBuffer (int slot, uint64_t frameNumber

Const sp& releaseFence, EGLDisplay eglDisplay,EGLSyncKHR eglFence) {

/ / 1. Check whether Slot is legal

If (slot

< 0 || slot >

= BufferQueueDefs::NUM_BUFFER_SLOTS | |

Return BAD_VALUE

}

/ / 2. Fault-tolerant processing: if the Slot to be processed exists in mQueue, then the source of the Slot is illegal, and it is not a Slot obtained from acquireBuffer 2.5. Processing is refused.

BufferQueueCore::Fifo::iterator current (mCore- > mQueue.begin ())

While (current! = mCore- > mQueue.end ()) {

If (current- > mSlot = = slot) {

Return BAD_VALUE

}

+ + current

}

/ / 3. Reverse the state of Slot to FREE, which used to be ACQUIRED, and add the Slot to BufferQueueCore's mFreeBuffers list (introduction to mFreeBuffers definition Ref. 2.1)

If (mSlots [slot] .mBufferState = = BufferSlot::ACQUIRED) {

M slope [slot] .mEglDisplay = eglDisplay

M slope [slot] .mEglFence = eglFence

MSlots.mFence = releaseFence

M slope [slot] .mBufferState = BufferSlot::FREE

MCore- > mFreeBuffers.push_back (slot)

Listener = mCore- > mConnectedProducerListener

BQ_LOGV ("releaseBuffer: releasing slot% d", slot)

}

/ / 4. Callback producer, some data has been consumed

If (listener! = NULL) {

Listener- > onBufferReleased ()

}

}

As you can see from the comments above, the main steps for releaseBuffer are as follows:

Reverse the state of Slot to FREE

Add the consumed Slot to the mFreeBuffers for use by subsequent producer dequeueBuffer

The callback informs the producer that some data is being consumed and that the producer can prepare the production data.

Summary releaseBuffer: change the status of Slot to FREE, add it to the BufferQueueCore mFreeBuffers queue, and finally notify the producer that data is out of the queue.

Summarize the process of state change:

This is the end of the content about "what are the design ideas and internal implementation methods of BufferQueue". Thank you for your reading. If you want to know more about the industry, you can follow the industry information channel. The editor will update different knowledge points for you every day.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.