Shulou(Shulou.com)06/01 Report--

Today, the editor will share with you the relevant knowledge points of KeyDB. The content is detailed and the logic is clear. I believe most people still know too much about this knowledge, so share this article for your reference. I hope you can get something after reading this article. Let's learn about it.

Thread model

KeyDB splits the original redis main thread into the main thread and the worker thread. Each worker thread is an io thread that listens for ports, accept requests, reads data, and parses protocols. As shown in the figure:

KeyDB uses the SO_REUSEPORT feature, which allows multiple threads to bind and listen on the same port.

Each worker thread does cpu binding, and reads data using the SO_INCOMING_CPU feature, specifying cpu to receive data.

After parsing the protocol, each thread manipulates the data in memory, and a global lock controls multi-thread access to memory data.

In fact, the main thread is also a worker thread, including the work of the worker thread, as well as work that can only be done by the main thread. The subscript 0 in the worker thread array is the main thread.

The main work of the main thread is to implement serverCron, including:

Processing statistics

Client link management

Resize and reshard of db data

Dealing with aof

Replication active / standby synchronization

Tasks in cluster mode

Link management

All link management in redis is done in a single thread. In KeyDB's design, each worker thread is responsible for a set of links, all of which are inserted into the thread's link list for maintenance. The generation, work and destruction of links must be in the same thread. Add a field for each link

Int iel; / * the event loop index we're registered with * /

Used to indicate which thread the link belongs to.

KeyDB maintains three key data structures for link management:

Clients_pending_write: a thread-specific linked list that maintains queues that synchronize data sent to customer links

Clients_pending_asyncwrite: a thread-specific linked list that maintains queues that send data asynchronously to customer links

Clients_to_close: global linked list to maintain customer links that need to be closed asynchronously

It is divided into synchronous and asynchronous queues because redis has some linkage api. For example, after pub/sub,pub, you need to send messages to the client of sub. The thread executed by pub is not the same thread as the client of sub. In order to deal with this situation, KeyDB will need to send data to clients other than this thread and maintain it in an asynchronous queue.

The logic of synchronous sending is relatively simple, and it is done in this thread. The following figure shows how to send data synchronously to the client:

As mentioned above, a link must be created, received, sent, and released on the same thread. Asynchronous sending involves the interaction between two threads. KeyDB passes messages between two threads through pipes.

When the local thread needs to send data asynchronously, it first checks whether the client belongs to the local thread, and the non-local thread obtains the client-specific thread ID, and then manages the operation of sending AE_ASYNC_OP::CreateFileEvent to the exclusive thread, requiring that a write socket event be added. The dedicated thread adds the corresponding request to the write event when processing the pipeline message, as shown in the figure:

Some redis requests to close the client are not performed entirely on the thread in which the link resides, so a global asynchronous closed list is maintained here.

Locking mechanism

KeyDB implements a lock mechanism similar to spinlock, called fastlock.

The main data structures of fastlock are:

Int fdCmdWrite; / / write pipeline

Int fdCmdRead; / / read pipeline

Use the atomic operation _ _ atomic_load_2,__atomic_fetch_add,__atomic_compare_exchange to determine whether the lock can be acquired by comparing the m_active=m_avail.

Fastlock provides two ways to acquire locks:

Try_lock: once failed to get, return directly

Lock: busy wait. After every 1024 * 1024 busy waits, use sched_yield to actively hand over the cpu, and move to the end of the cpu task to wait for execution.

Combine try_lock with events in KeyDB to avoid busy waits. Each client has a dedicated lock, which attempts to lock the client data before it is read, and exits if it fails, because the data has not yet been read, so it can be processed again in the next epoll_wait processing event loop.

Active-Replica

KeyDB implements the mechanism of multi-activity, each replica can be set to be writable and not read-only, and the replica synchronizes data with each other. The main features are:

Each replica has a uuid flag that removes circular replication.

Add rreplay API, package incremental commands into rreplay commands, with local uuid

Key,value plus the timestamp version number, as a conflict check, if the local key is the same and the timestamp version number is larger than the synchronized data, the new write fails. The timestamp version number of the key is obtained by moving the current timestamp 20 bits to the left and adding the last 44 bits to the left.

These are all the contents of this article entitled "what are the relevant knowledge points of KeyDB?" Thank you for reading! I believe you will gain a lot after reading this article. The editor will update different knowledge for you every day. If you want to learn more knowledge, please pay attention to the industry information channel.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.