Shulou(Shulou.com)06/01 Report--

[TOC]

Redis Java API use (1): stand-alone version of Redis API use

Redis's Java API operates through Jedis, so the third-party library of Jedis is needed first. Because the Maven project is used, the dependency of Jedis is given first:

Redis.clients jedis basic code example

Jedis also provides all the commands that Redis can provide, and it is very similar to use, so only part of the code for the operation is given below.

The java API * jedis of package com.uplooking.bigdata;import org.junit.After;import org.junit.Before;import org.junit.Test;import redis.clients.jedis.Jedis;import java.util.List;import java.util.Map;import java.util.Set;/** * Redis operation is the entrance to the java api of our operation Redis * A Jedis object represents a connection of Redis * CRUD * / public class RedisTest {private Jedis jedis; private String host; private int port @ Before public void setUp () {host = "uplooking01"; port = 6379; jedis = new Jedis (host, port);} @ Test public void testCRUD () {/ / the collection of all key Set keys = jedis.keys ("*"); / / jedis.select (index) Specify the database to perform the operation. The default operation is 0 data System.out.println (keys); / / string System.out.println ("* String*"); / / delete key nam1 Long del = jedis.del ("nam1") in redis; System.out.println (del = = 1L? "deletion succeeded ~": "deletion failed ~"); List mget = jedis.mget ("name", "age"); System.out.println (mget); / / hash System.out.println ("* Hash*"); Map person = jedis.hgetAll ("person") / / keyset / / entryset for (Map.Entry me: person.entrySet ()) {String field = me.getKey (); String value = me.getValue (); System.out.println (field + "- -" + value) } / / list System.out.println ("* List*"); List seasons = jedis.lrange ("season", 0,-1); for (String season: seasons) {System.out.println (season) } / / set System.out.println ("* Set*"); Set nosql = jedis.smembers ("nosql"); for (String db: nosql) {System.out.println (db) } / / zset System.out.println ("* Zset*"); Set website = jedis.zrange ("website", 0,-1); for (String ws: website) {System.out.println (ws) } @ After public void cleanUp () {jedis.close ();}} JedisPool tool class development

The previous code is to establish a Jedis connection at a time, which is more resource-consuming, so you can use JedisPool to solve this problem, and in order to improve the efficiency of later development, you can develop a tool class based on JedisPool.

JedisUtil.javapackage com.uplooking.bigdata.common.util.redis;import com.uplooking.bigdata.constants.redis.JedisConstants;import redis.clients.jedis.Jedis;import redis.clients.jedis.JedisPool;import redis.clients.jedis.JedisPoolConfig;import java.io.IOException;import java.util.Properties / * * tools for Redis Java API operations * mainly provide us with Java operation Redis objects Jedis mimics a similar database connection pool * * JedisPool * / public class JedisUtil {private JedisUtil () {} private static JedisPool jedisPool; static {Properties prop = new Properties (); try {prop.load (JedisUtil.class.getClassLoader (). GetResourceAsStream ("redis/redis.properties")) JedisPoolConfig poolConfig = new JedisPoolConfig (); / / maximum number of connections in the jedis connection pool poolConfig.setMaxTotal (Integer.valueOf (prop.getProperty (JedisConstants.JEDIS_MAX_TOTAL); / / maximum number of idle connections in the jedis connection pool poolConfig.setMaxIdle (Integer.valueOf (prop.getProperty (JedisConstants.JEDIS_MAX_IDLE) / / minimum number of idle connections in jedis connection pool poolConfig.setMinIdle (Integer.valueOf (prop.getProperty (JedisConstants.JEDIS_MIN_IDLE); / / maximum waiting time for jedis connection pool ms value poolConfig.setMaxWaitMillis (Long.valueOf (prop.getProperty (JedisConstants.JEDIS_MAX_WAIT_MILLIS) / / Server hostname String host = prop.getProperty (JedisConstants.JEDIS_HOST) for jedis; String JEDIS_PORT = "jedis.port"; int port = Integer.valueOf (prop.getProperty (JedisConstants.JEDIS_PORT)); / / Service password for jedis String password = prop.getProperty (JedisConstants.JEDIS_PASSWORD) JedisPool = new JedisPool (poolConfig, host, port, 10000);} catch (IOException e) {e.printStackTrace ();}} / * provides the object of Jedis * @ return * / public static Jedis getJedis () {return jedisPool.getResource () } / * * Resource release * @ param jedis * / public static void returnJedis (Jedis jedis) {jedis.close ();}} JedisConstants.javapackage com.uplooking.bigdata.constants.redis;/** * is specially used to store the constant class of Jedis * / public interface JedisConstants {/ / indicates the server hostname of jedis String JEDIS_HOST = "jedis.host" / / Port String JEDIS_PORT for jedis service = "jedis.port"; / / Service password for jedis String JEDIS_PASSWORD = "jedis.password"; / / maximum number of connections in jedis connection pool String JEDIS_MAX_TOTAL = "jedis.max.total"; / / maximum number of idle connections in jedis connection pool String JEDIS_MAX_IDLE = "jedis.max.idle" / / minimum number of idle connections in jedis connection pool String JEDIS_MIN_IDLE = "jedis.min.idle"; / / maximum waiting time for jedis connection pool ms value String JEDIS_MAX_WAIT_MILLIS = "jedis.max.wait.millis" } redis.properties### redis configuration file # indicates the server hostname jedis.host=uplooking01# of jedis # # Port jedis.port=6379### of jedis indicates service password of jedis maximum number of connections in jedis.password=uplooking###jedis connection pool maximum number of idle connections in jedis.max.total=60###jedis connection pool minimum number of idle connections in jedis.max.idle=30###jedis connection pool maximum ms value jedis.max.wait.millis=30000 in waiting connection pool

Later, you can easily use this utility class to operate on Redis:

/ obtain Jedis connection object Jedis jedis = JedisUtil.getJedis (); / release Jedis object resource JedisUtil.returnJedis (jedis); build Redis cluster theory in Redis distributed cluster environment. Redis cluster is a distributed Redis storage architecture that can share data among multiple nodes to solve the problems of high availability and scalability of Redis. Redis clusters provide the following two benefits: 1. Automatically split the data to multiple nodes 2. Redis can also continue to process client requests when one of the nodes in the cluster fails. A Redis cluster contains 16384 hash slot, and each data in the database belongs to one of these 16384 hash slots. The cluster uses the formula CRC16 (key) 384 to calculate which slot the key belongs to. Each node in the cluster is responsible for handling part of the hash slot. Each node in the master-slave replication cluster in the cluster has 1 to N replicas, one of which is a master node and the rest is a slave node. If the master node goes offline, the cluster will set a slave node of this master node as the new master node and continue to work. The cluster will not fail to work properly because of the offline of a primary node. If a master node and all its slave nodes go offline, the redis cluster stops working. Redis clusters do not guarantee strong consistency of data, and under certain circumstances, redis clusters will lose commands that have already been executed. Using asynchronous replication (asynchronous replication) is one of the reasons why Redis clusters may lose write commands. Sometimes due to network reasons, if the network is disconnected for too long, the redis cluster will enable a new master node, and the data gathering previously sent to the master node will be lost.

The above theoretical knowledge, after the completion of the following Reids master-slave replication environment and distributed environment, I believe there will be a very intuitive understanding.

Redis master-slave replication cluster installation

Three devices are used here, and the environment is described as follows:

Uplooking01 masteruplooking02 slaveuplooking03 slave is uplooking01 master node, 02 and 03 are slave nodes, master node is mainly responsible for writing, slave node is mainly responsible for reading, can not write. In addition, the password will be configured on the two slave servers to test the connection method when using the password (note that the master server does not set the password). These requirements are reflected in the configuration below.

The uplooking01 redis.conf configuration is as follows:

Bind uplooking01daemonize yes (background running) logfile / opt/redis-3.2.0/logs/redis.log (log file, directory must exist)

The uplooking02 redis.conf configuration is as follows:

Bind uplooking02daemonize yes (background running) logfile / opt/redis-3.2.0/logs/redis.log (log file, directory must exist) slave-read-only yesrequirepass uplookingslaveof uplooking01 6379

The uplooking03 redis.conf configuration is as follows:

Bind uplooking03daemonize yes (background running) logfile / opt/redis-3.2.0/logs/redis.log (log file, directory must exist) slave-read-only yesrequirepass uplookingslaveof uplooking01 6379

After the above configuration is completed, start redis on the two slave servers to complete the configuration of the master-slave replication cluster. You need to pay attention to the following problems:

1. Authentication problem if you need to connect to uplooking02 or uplooking03, you need to add a password, otherwise you can't complete the authentication. There are two ways: you can specify a password when you connect: redis-cli-h uplooking03-a uplooking can also connect first, and then authenticate to the terminal: auth uplooking2. Problem of data reading and writing: data reading and writing can be completed on all three servers: data can only be written on the master node. In the code used earlier, if you are connecting to a slave server, you also need to configure a password. Take the JedisPool tool class we developed as an example, you need to specify a password when creating a JedisPool: jedisPool = new JedisPool (poolConfig, host, port, 10000, password). Redis distributed cluster installation and deployment

Cluster description:

1. The cluster built above is only a master-slave replication cluster, which means that the data is the same on all three machines, and its purpose is only to separate read from write, improve the efficiency of reading and play the role of redundancy at the same time. Once the master node fails, the slave node can be replaced, but obviously, this is just a cluster, not a distributed one. two。 However, there may be a problem, that is, when the amount of data is too large, all the data are stored on the same node (although the two servers have made backups, but because the saved data is the same, it is regarded as a node), the data storage pressure on a single server will be very great, so you can consider using a distributed environment to save it, which is the distributed cluster of Redis. Distributed: data is divided into several pieces and stored in clusters on different devices: for the same data, there is at least one copy to save. This can be compared to hdfs in hadoop or partition in kafka (topic can set the number of partition and copy factor) 3. In Redis, at least 6 nodes are needed to build a distributed cluster environment, so for the sake of devices, it will operate on the same device. That is to say, the pseudo-distributed environment is built, with 3 master nodes and the other 3 slave nodes respectively, which are used to store their replica data. According to the previous theoretical knowledge, in a distributed environment, the key value is calculated as follows: CRC16 (16)% 16384 to calculate which slot the key value belongs to, while for our environment, the number of slots per master node is about 16384 / 3 = 54611. Extract the installation package [uplooking@uplooking01 ~] $mkdir-p app/redis-cluster [uplooking@uplooking01 ~] $tar-zxvf soft/redis-3.2.0.tar.gz-C app/redis-cluster/2. Compile and install [uplooking@uplooking01 ~] $cd app/redis-cluster/redis-3.2.0/ [uplooking@uplooking01 redis-3.2.0] $pwd/home/uplooking/app/redis-cluster/redis-3.2.0 [uplooking@uplooking01 redis-3.2.0] $make [uplooking@uplooking01 redis-3.2.0] $make install PREFIX=/home/uplooking/app/redis-cluster/redis-3.2.03. Create a Redis node [uplooking@uplooking01 redis-cluster] $mv redis-3.2.0/ 7000 [uplooking@uplooking01 redis-cluster] $cp-r 7000 7001 [uplooking@uplooking01 redis-cluster] $cp-r 7000 7002 [uplooking@uplooking01 redis-cluster] $cp-r 7000 7003 [uplooking@uplooking01 redis-cluster] $cp-r 7000 7004 [uplooking@uplooking01 redis-cluster] $cp-r 7000 70054. Modify the configuration of each node

Take 7000 as an example:

Daemonize yes / / configure redis background to run bind uplooking01 / / bind host uplooking01logfile "/ home/uplooking/app/redis-cluster/7000/redis-7000.log" / / Note that the directory must exist pidfile / var/run/redis-7000.pid / / pidfile file corresponding to 7002 7003 port 7000 / / Port cluster-enabled yes / / enable the cluster to comment # remove the configuration file of the cluster-config-file nodes-7000.conf / / cluster for the first time start automatic generation of cluster-node-timeout 15000 / / request timeout setting 15 seconds enough for appendonly yes / / aof log to be enabled if necessary It records a log for each write operation.

On the other nodes, you just need to change it to 7001pm 7002. That's it.

Tip: after the configuration of 7000 is complete, you can directly copy to other nodes, cp redis.conf.. / 7001, and then make full use of the vim's 1 # vim to replace 7000 with other numbers, such as 7001, etc.

5. Start each node

First create a script that starts in batches:

[uplooking@uplooking01 redis-cluster] $cat startmurall.shrunken cd 7004bin/redis-server. / redis.confcd.. cd 7001bin/redis-server. / redis.confcd. Cd 7002bin/redis-server. / redis.confcd. Cd 7003bin/redis-server. / redis.confcd.. cd 7004bin/redis-server. / redis.confcd.. cd 7005bin/redis-server. / redis.confcd.

Then execute the script to start.

6. View the service [uplooking@uplooking01 redis-cluster] $ps-ef | grep redis500 1460 1 001: 17? 00:00:01 bin/redis-server uplooking01:7000 [cluster] 1464 1 001: 17? 00:00:01 bin/redis-server uplooking01:7001 [cluster] 500 1468 1 001: 17? 00:00:01 bin/redis-server uplooking01:7002 [cluster] 500 1472 10 01:17? 00:00:01 bin/redis-server uplooking01:7003 [cluster] 500 1474 10 01:17? 00:00:01 bin/redis-server uplooking01:7004 [cluster] 500 1480 10 01:17? 00:00:01 bin/redis-server uplooking01:7005 [cluster] 500 3233 1018 0 01:53 pts/0 00:00:00 grep redis7. Create a cluster (core)

Now it's time to use the previously prepared redis nodes and concatenate them to build a cluster. An official tool has been provided: redis-trib.rb ($REDIS_HOME/src uses a program written by ruby, so you need to install ruby):

$sudo yum-y install ruby ruby-devel rubygems rpm-build

Then use the command gem to install the redis interface (gem is a toolkit for ruby):

Gem install redis [- v 3.2.0] # [] specify a specific software version for the option # when I install, I prompt that the ruby version needs to be > = 2.2.2, but there will be no problem after connecting the redis version above.

Next, run redis-trib.rb:

[uplooking@uplooking01 7000] $src/redis-trib.rb create-- replicas 1192.168.56.101: 7000192.168.56.101replicas 7002 192.168.56.101 Performing hash slots allocation on 6 nodes...Using 3 masters:192.168.56.101:7000192.168.56.101:7001192.168.56.101:7002Adding replica 192.168.56.101 .101to 7003 to 192.168.56.101:7000Adding replica 192.168.56.101 to 192.168.56.101:7001Adding replica 192.168.56.101 to 192.168.56.101 to 7002M: 497bce5118057198afb0511cc7b88479bb0c3938 192.168.56.101 7000slots:0-5460 (5461 slots) masterM: f0568474acad5c707f25843add2d68455d2cbbb2 192.168.56.101Rd 7001 slots:5461-10922 (5462slots) masterM: ebe86ea74af5612e6393c8e5c5b3363928a4b7b2 192.168.56.101 masterM 7002 slots:10923-16383 (5461 slots) masterS : c99c55ab3fcea2d65ca3be5b4786390a6e463ea2 192.168.56.101:7003 replicates 497bce5118057198afb0511cc7b88479bb0c3938S: 0a847801493a45d32487d701cd0fe37790d4b2f9 192.168.56.101:7004 replicates f0568474acad5c707f25843add2d68455d2cbbb2S: 7f9e4bec579fda23a574a62d362a04463140bbc2 192.168.56.101:7005 replicates ebe86ea74af5612e6393c8e5c5b3363928a4b7b2Can I set the above configuration? (type 'yes' to accept): yes > > Nodes configuration updated > Assign a different config epoch to each node > > Sending CLUSTER MEET messages to join the clusterWaiting for the cluster to join. > > Performing Cluster Check (using node 192.168.56.101 masterM 7000) M: 497bce5118057198afb0511cc7b88479bb0c3938 192.168.56.101 masterM 7000 slots:0-5460 (5461 slots) masterM: f0568474acad5c707f25843add2d68455d2cbbb2 192.168.56.101 Vane 7001 slots:5461-10922 (5462 slots) masterM: ebe86ea74af5612e6393c8e5c5b3363928a4b7b2 192.168.56.101 purge 7002 slots:10923-16383 ( 5461 slots) masterM: c99c55ab3fcea2d65ca3be5b4786390a6e463ea2 192.168.56.101 slots 7003 slots: (0 slots) master replicates 497bce5118057198afb0511cc7b88479bb0c3938M: 0a847801493a45d32487d701cd0fe37790d4b2f9 192.168.56.101 slots 7004 slots: (0 slots) master replicates f0568474acad5c707f25843add2d68455d2cbbb2M: 7f9e4bec579fda23a574a62d362a04463140bbc2 192.168.56.101 slots: (0 slots) master replicates ebe86ea74af5612e6393c8e5c5b3363928a4b7b2 [OK] All nodes agree about slots configuration. > > Check for open slots... > Check slots coverage... [OK] All 16384slots covered.

If you take a closer look at the hints, you will have a clearer understanding of Redis distributed clusters. It is also important to note that because redis-trib.rb does not support domain names or hostnames well, it is necessary to use ip:port when creating clusters.

8. Test [uplooking@uplooking01 7000] $bin/redis-cli-h uplooking01-p 7000-cuplooking01:7000 > set name xpleaf- > Redirected to slot [5798] located at 192.168.56.101:7001OK192.168.56.101:7001 > get name "xpleaf" 192.168.56.101 uplooking01 7001 > [uplooking@uplooking01 7000] $bin/redis-cli-h uplooking01-p 7004-cuplooking01:7004 > get name- > Redirected to slot [5798] located at 192.168.56.101 uplooking01 7001 "xpleaf" 192.168.56. 101uplooking01 7001 > [uplooking@uplooking01 7000] $bin/redis-cli-h uplooking01-p 7004-cuplooking01:7004 > set name yyh- > Redirected to slot [5798] located at 192.168.56.101:7001OK192.168.56.101:7001 > get name "yyh" [uplooking@uplooking01 7000] $bin/redis-cli-h uplooking01-p 7002-cuplooking01:7002 > keys * (empty list or set) uplooking01:7002 > get name- > Redirected to slot [5798] located at 192.168.56.101 Velcro 7001 "yyh"

The above tests fully illustrate the following problems:

1. Distributed data is distributed storage and will be saved to different master nodes according to different key. two。 Data backup slave node acts as a backup node, just like the previous master-slave replication cluster, it is only used to read data, and when you need to write or modify data, you need to switch to the master node. Redis Java API use (2): Cluster API use

The previous code is only suitable for operating the stand-alone version of Redis. If you are using a distributed Redis cluster, you need to modify the code. Here, we directly develop a tool class JedisClusterUtil, as follows:

The utility class for package com.uplooking.bigdata.common.util.redis;import redis.clients.jedis.*;import java.io.IOException;import java.util.HashSet;import java.util.Properties;import java.util.Set;/** * Redis Java API operations * is specifically responsible for the cluster mode of redis * / public class JedisClusterUtil {private JedisClusterUtil () {} private static JedisCluster jedisCluster; static {Set nodes = new HashSet () Nodes.add (new HostAndPort ("uplooking01", 7000); nodes.add (new HostAndPort ("uplooking01", 7001)); nodes.add (new HostAndPort ("uplooking01", 7002)); nodes.add (new HostAndPort ("uplooking01", 7003)); nodes.add (new HostAndPort ("uplooking01", 7004)); nodes.add (new HostAndPort ("uplooking01", 7005)); jedisCluster = new JedisCluster (nodes) / / the redis cluster mode} / * provides the Jedis object * @ return * / public static JedisCluster getJedis () {return jedisCluster;} / * Resource release * @ param jedis * / public static void returnJedis (JedisCluster jedis) {try {jedis.close () } catch (IOException e) {e.printStackTrace ();}

It should be noted that when using API operations such as mget, JedisCluster is not allowed to obtain data on multiple nodes at the same time, for example: List mget = jedis.mget ("name", "age"); if name and age are on different nodes, an exception will be reported, so it is not recommended to use this method to obtain data.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.