In addition to Weibo, there is also WeChat
Please pay attention
WeChat public account
Shulou
2025-01-19 Update From: SLTechnology News&Howtos shulou NAV: SLTechnology News&Howtos > Database >
Share
Shulou(Shulou.com)05/31 Report--
This article will explain in detail the example analysis of Redis in-memory database fragmentation. The content of the article is of high quality, so the editor will share it with you for reference. I hope you will have a certain understanding of the relevant knowledge after reading this article.
Software download address
Allows larger datasets to be stored in the memory of multiple machines. If there is no sharding, then you can only store the dataset with the maximum value of a single machine's memory.
Allows for increased computing power and network bandwidth with multiple cores and multiple machines.
Common slicing methods:
1. Slice according to the range
2. Hash fragmentation, such as consistent hash
Common implementation of slicing:
① client sharding
② does shredding in the agent, such as twemproxy
③ query routing: sends a query to a random instance, which ensures that your query is forwarded to the correct node. With the help of the client, the redis cluster implements a mixed form of query routing. Instead of forwarding the request directly from the redis instance to another instance, the client receives the redirection to the correct node.
④ is shredded on the server side, and Redis is sliced on the server side using hash slot:
The Redis cluster has 16384 hash slots. Use Jian CrC16 pair 16384 to calculate the hash slot to which a key belongs.
Disadvantages of slicing
Some of the features of Redis don't work very well with sharding:
1. Operations involving multiple keys are usually not supported. For example, you can't perform an intersection of keys mapped on two different Redis instances (there is actually a way to do that, but not directly).
2. Transactions involving multiple keys cannot be used.
3. The granularity of sharding is a key, so you can't use a large key to fragment a dataset, such as a large ordered set.
4. When sharding is used, data processing becomes more complex, for example, you need to deal with multiple RDB/AOF files, and when you back up data, you need to aggregate persistent files from multiple instances and hosts.
5. Adding and removing capacity is also complex. For example, Redis clusters have the ability to dynamically add and remove nodes at run time to support transparent rebalancing of data, but other ways, such as client sharding and agents, do not support this feature. However, there is a technique called pre-slicing (Presharding) that can help.
Disadvantages of Redis fragmentation
1. Operations involving multiple builds, such as mget, are not supported. If all the keys are on the same node, they will be executed normally, otherwise an error will be prompted.
2. The granularity of the shard is key, so the corresponding value of each key should not be too large.
3. Data backup can be troublesome. When backing up data, you need to aggregate persistent files of multiple instances and hosts.
4. It is troublesome to deal with capacity expansion.
5. The recovery of the failure will be troublesome, and it may be necessary to recomb the relationship between Master and Slave and adjust the data in each replication set.
Pre-slicing Technology of Redis
The pre-sharding technology of Redis can perform instance migration by following the steps below:
(1) start a new redis instance on the new machine
(2) use the new redis instance as slave and the original redis instance as master, and migrate the data from the original redis instance to the new redis instance.
(3) stop the client (when the sharding operation is on the client) or the proxy server (the sharding operation is on the proxy)
(4) update the configuration information in the client or proxy server, remove the ip and port information of the migrated original redis instance, and add the IP address and port of the newly launched redis instance
(5) send a SLAVEOF NOONE command to the newly launched redis to terminate the dependency of the new redis instance to the original redis instance
(6) restart the client program or agent program, and they will use the new redis instance
(7) shut down the original redis instance where the data was migrated
Preslicing
We already know that a problem with sharding is that unless we use Redis as a cache, adding and deleting nodes is tricky, and using fixed keys and instance mappings is much easier.
However, the requirements for data storage may be changing all the time. I can accept 10 Redis nodes (instances) today, but I may need 50 nodes tomorrow.
Because Redis has very little footprint and is lightweight (an idle instance only uses 1MB memory), a simple solution is to open a lot of instances in the first place. Even if you start with only one server, you can decide to live in a distributed world on the first day, using sharding to run multiple Redis instances on one server.
You can choose a large number of instances from the start. For example, 32 or 64 instances can satisfy most users and provide enough space for future growth.
So, when your data storage needs to grow and you need more Redis servers, all you have to do is simply move the instance from one server to another. When you add the first server, you need to move half of the Redis instances from the first server to the second server, and so on.
With Redis replication, you can move data in little or no downtime:
Start an empty instance on your new server.
Move the data and configure the new instance as the slave service of the source instance.
Stop your client.
Update the server IP address configuration of the moved instance.
Send a SLAVEOF NO ONE command to the slave node on the new server.
Start your client with a new updated configuration.
Finally, close the instances that are no longer in use on the old server.
Redis cluster
Because the amount of data is too large to support a single replication set, it is necessary to cluster multiple replication sets to form horizontal expansion. Each replication set is only responsible for storing part of the data set, which is the cluster of Redis.
1. In previous versions, the cluster of Redis relied on client fragmentation, but it would have many disadvantages, such as high maintenance cost, client coding, adding and removing nodes, etc.
2. One of the new features of Redis3.0 is to support clusters, which also provides the accessibility of network partitions and supports the recovery of master database failures without compromising performance.
3. After using the cluster, you can only use the default database No. 0.
4. Each redis cluster node needs two TCP connections to open. Normal TCP ports are used to serve clients, such as 6379 and home 10000, which are used as data ports. You must ensure that the firewall opens these two ports.
5. The Redis cluster does not guarantee strong consistency, which means that under certain conditions, the Redis cluster may lose some write request commands received by the system (the Master propagation command will hang up after it returns to OK, and slave has not received the broadcast yet)
Architecture of the cluster
1. All Redis nodes are interconnected with each other, and binary protocols are used internally to optimize transmission speed and broadband.
2. The fail of a node takes effect only when it is detected by more than half of the nodes in the cluster.
3. The client is directly connected to the redis node and does not need an intermediate proxy layer. The client does not need to connect to all the nodes in the cluster, just connect to any available node in the cluster
4. The cluster maps all physical nodes to [0-16383] slots, and the cluster is responsible for maintaining the node-slot-value relationship.
Basic commands for cluster operation
CLUSTER INFO
CLUSTER NODES
CLUSTER MEET
CLUSTER FORGET
CLUSTER REPLICATE
CLUSTER SAVECONFIG
CLUSTER ADDSLOTS
CLUSTER DELSLOTS
CLUSTER FLUSHSLOTS
CLUSTER SETSLOT NODE
CLUSTER SETSLOT MIGRATING
CLUSTER SETSLOT IMPORTING
CLUSTER SETSLOT STABLE
CLUSTER KEYSLOT
CLUSTER COUNTKEYSINSLOT
CLUSTER GETKEYSINSLOT
MIGRATE destination node key name database number timeout [copy] [replace]
This is the end of the example analysis of Redis in-memory database fragmentation. I hope the above content can be of some help to you and learn more knowledge. If you think the article is good, you can share it for more people to see.
Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.
Views: 0
*The comments in the above article only represent the author's personal views and do not represent the views and positions of this website. If you have more insights, please feel free to contribute and share.
Continue with the installation of the previous hadoop.First, install zookooper1. Decompress zookoope
"Every 5-10 years, there's a rare product, a really special, very unusual product that's the most un
© 2024 shulou.com SLNews company. All rights reserved.