Shulou(Shulou.com)06/03 Report--

This article mainly explains the "pika cluster horizontal expansion of how to make the performance capacity is no longer limited", the content of the article is simple and clear, easy to learn and understand, now please follow the editor's ideas slowly in depth, together to study and learn "pika cluster horizontal expansion of how to make the performance capacity is no longer limited" bar!

Background

Pika is a persistent high-capacity redis storage service, which is compatible with most of the interfaces of string, hash, list, zset and set (compatibility details). It solves the bottleneck of insufficient memory caused by the huge amount of data stored in redis. Users can migrate from redis to pika services without modifying any code. With good compatibility and stability, more than 3000 instances are used within the company, and more than 3.8K star is used in the github community. As the capacity of a single pika is limited by the capacity of a single hard disk, there is a growing demand for distributed pika clusters in the business and community of 360. therefore, we have launched a native distributed pika cluster and released pika version v3.4. Compared with the pika+codis cluster solution, codis's support for pika creation and management of slot operations is not friendly and requires a large number of operations and maintenance personnel. Pika native clusters do not require additional deployment of codis-proxy modules.

Cluster deployment structure

Take a cluster with 3 pika nodes as an example. The cluster deployment structure is shown in the figure above:

Deploy the Etcd cluster as the meta-information store for pika manager.

Deploy pika manager on three physical machines, and configure the service port of Etcd. Pika manager will register with etcd and compete to become leader. One and only one pika manager in the cluster can become a leader and write cluster data to the etcd.

Deploy the pika node on the three physical machines, and then add the information of the pika node to the pika manager.

For load balancing, register the service port of pika with LVS.

Data distribution

In order to isolate data according to business, Pika cluster introduces the concept of table, and different business data are stored in different table. Business data is stored on the corresponding slot according to the hash value of key. There are multiple copies of each slot, forming a replication group. All slot replicas in replication group have the same slot ID, one of which is leader and the other is follower. In order to ensure the consistency of data, only leader provides read and write services. You can use pika manager to schedule and migrate slot, so that the data and read and write pressure are evenly distributed to the entire pika cluster, thus ensuring the full utilization of the resources of the entire cluster and horizontal capacity expansion and reduction according to business pressure and storage capacity.

Pika uses rocksdb as the storage engine, and each slot creates a corresponding rocksdb. Each slot in pika supports reading and writing five redis data structures. Therefore, it is very convenient to migrate data, as long as you migrate the slot in pika. But at the same time, there is also the problem of taking up too much resources. The current pika creates five rocksdb by default when creating a slot, each of which stores five data structures. When there is a large amount of slot in table or a large number of table is created, a single pika node will contain multiple slot, thus creating too many rocksdb instances and consuming too much system resources. In subsequent versions, on the one hand, it will support the creation of one or more data structures according to business needs when creating slot, on the other hand, it will continue to optimize the blackwidow interface layer in pika to reduce the use of rocksdb.

Data processing.

When the pika node receives the user request, the parsing layer parses the redis protocol and gives the parsed result to the router layer for judgment.

According to the hash result of key, router finds the corresponding slot of key and determines whether the slot is on the local node.

If the slot where the key is located is on another node, a task is created and placed in the queue according to the request, and the request is forwarded to the peer node for processing. When the task receives the processing result of the request, it returns the request to the client.

If the slot where the key resides belongs to the local node, the request is processed locally and returned to the client.

For write requests that need to be processed locally, first write binlog through the replication manager module and copy it asynchronously to other slot replicas. Process layer writes to leader slot according to the requirement of consistency. Where blackwidow is the interface encapsulation of rocksdb.

We embed proxy into pika, which does not need to be deployed separately. Compared with redis cluster, the client does not need to be aware of the existence of proxy, just use the cluster as if it were a stand-alone machine. The service port of the pika node can be mounted to the LVS to achieve load balancing on the whole cluster.

Log replication

The replication manager module in pika is responsible for the master-slave synchronization of logs. For compatibility with redis,pika to support inconsistent log replication, leader slot writes data directly to the db without waiting for an ack reply from the follower slot. Raft consistency protocol log replication is also supported, and the ack that receives most copies is required to write to the db.

Inconsistent log replication

In an inconsistent scenario, the processing flow is as follows:

The processing thread receives the request from the client, writes to binlog and manipulates db after being locked directly.

The processing thread returns the client response.

The worker thread sends a BinlogSync synchronization request to the follower slot to synchronize the log.

Follower slot returns BinlogSyncAck to report synchronization.

Consistency log replication

In a consistency log replication scenario:

The processing thread writes the client request to the binlog file

Synchronize to the slave library by sending a BinlogSync request

Return BinlogSyncAck from the library to report synchronization status

Check that the response from the library satisfies most of the requests and writes the corresponding request to db

Return response to the client

Cluster metadata processing

We have developed pika manager (PM for short) on the basis of codis-dashboard, which is used as the global control node of the whole cluster to deploy and dispatch and manage the cluster. The metadata and routing information of the entire cluster are stored in PM.

The function of creating multiple tables in the cluster is added to facilitate business data isolation according to different tables.

You can specify the number of slot and the number of copies when creating tables, making it easy for OPS to create table based on business scale and fault tolerance.

Logically change the concept of group to replication group, so that the original process-level data and log replication is transformed into slot-level replication.

You can create passwords when creating table to isolate the use of business. The client only needs to execute auth and select statements to authenticate and operate on the specified table.

Slot migration is supported to facilitate capacity expansion and reduction according to business needs.

With the integration of the Sentinel module, PM will continuously send heartbeats to the pika nodes in the cluster to monitor the survival status. When PM finds leader slot down, it automatically promotes the slave slot with the largest binlog offset to leader.

The storage backend supports writing metadata to etcd to ensure the high availability of metadata.

Pika manager becomes leader by constantly scrambling for locks from etcd to achieve the high availability of pika manager.

Thank you for your reading. the above is the content of "how the horizontal expansion of pika clusters makes the performance capacity no longer limited". After the study of this article, I believe you have a deeper understanding of how to make the performance capacity of pika clusters no longer limited, and the specific use needs to be verified in practice. Here is, the editor will push for you more related knowledge points of the article, welcome to follow!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.