Shulou(Shulou.com)06/01 Report--

CAP theorem:

C:Consistency consistency, data backup nodes in distributed systems need to maintain data consistency in real time.

A:Availability availability. The failure of a node in the cluster does not affect the services provided by the whole cluster.

P:Partition tolerance partition fault tolerance, after part of the system data is lost, it can still provide services.

In a distributed system, CAP cannot have both; high availability and consistent data are the goals of many system designs, but the phenomenon of partitioning is inevitable:

CA: the phenomenon of partitioning always exists. Generally speaking, the CA system refers to the maintenance of CA in the subpartitions after each partition.

CP: the system maintains strong consistency, and it is impossible to provide multiple services in synchronizing data between nodes. In relational database transactions, it can reflect the characteristics of CP.

AP: to ensure high availability, you must give up strong consistency. Once partitioning occurs in the cluster, each node can use its own local data to provide services, but this will lead to inconsistency of global data, which is the basic data feature of NoSQL.

Due to network hardware and other problems, the phenomenon of delayed packet loss is bound to occur, so partition tolerance is something we must achieve. In designing a distributed system, there can only be a tradeoff between consistency and availability.

BASE scenario:

BA (Basically Available) is basically available. When a distributed system fails, partial availability is allowed to be lost to ensure core availability.

S (Soft state) soft state, which allows the system to have an intermediate state without affecting the overall availability of the system.

E (Eventually consistent) is finally consistent, and all the copies of data in the system can finally reach a consistent state after a period of time.

NoSQL's four main technology genres:

Dynamo V storage: key-value, such as Dynamo, Redis

Column Family: column database, such as HBase

Document: document database, such as mongoDB

GraphDB: schema database.

Redis:Remote DICtionary Server: data structure server.

Characteristics of Redis:

(1) completely open source and free of charge, abiding by BSD agreement

(2) persistent data storage is supported.

(3) support data backup.

Advantages of Redis:

(1) High performance-110000 reads per second and 81000 writes per second

(2) many data types are supported-there are string (String), hash (Hash), list (List), set (Sets), ordered set (Sorted sets) and other types of values.

(3) atomicity-all operations are atomic

(4) rich features-support publish/subscribe, notification, key expiration and other features.

Redis installation: version 3.0 or above, you need to download the rpm package from the official website.

Installing the rpm package for Redis depends on the jemalloc installation package, so you need to configure the yum source for epel:

Yum install. / redis-3.2.3-1.el7.x86_64.rpm

Redis configuration:

The configuration file of Redis is / etc/redis.conf, and the default listening port is 6379 of TCP. You can log in to the interactive command line interface through rediscli:

[root@node7 ~] # systemctl start redis.service

[root@node7 ~] # redis-cli

127.0.0.1purl 6379 >

Redis data type:

Redis supports five data types: string (String), hash (Hash), list (List), set (Sets), ordered set (Sorted sets).

(1) string (String)

127.0.0.1 key 6379 > SET name 'hisen' # set the key of a string to name,value and hisen

OK # set the result status to OK

127.0.0.1 name 6379 > GET name # get the value whose key is name

Result obtained by "hisen" #

Note: a key can store 512MB at most.

(2) Hash (Hash)

Hash type is one-key multi-value type, one key corresponds to the value of a collection, commands HMSET, HGETALL.

127.0.0.1 purl 6379 > HMSET hisen id name age gender

OK

127.0.0.1 purl 6379 > HGETALL hisen

1) "id"

2) "name"

3) "age"

4) "gender"

Each hash can store 2 ^ 32-1 key-value pairs.

(3) list (List)

List type values, which can be added to the string list from the left or right, with the commands LPUSH, RPUSH, LRANGE:

127.0.0.1 purl 6379 > LPUSH friends he

(integer) 1

127.0.0.1 purl 6379 > LPUSH friends tao

(integer) 2

127.0.0.1 purl 6379 > LPUSH friends ying

(integer) 3

127.0.0.1 purl 6379 > RPUSH friends yu

(integer) 4

127.0.0.1 purl 6379 > RPUSH friends zhen

(integer) 5

127.0.0.1 LRANGE friends 6379 > 0 4

1) "ying"

2) "tao"

3) "he"

4) "yu"

5) "zhen"

Each list can store up to 2 ^ 32-1 elements.

(4) set (Sets)

Set is an unordered collection of type string, with commands SADD, SMEMBERS.

Adding a String element returns 1 successfully, or 0 if the inserted element is the same as an element in the collection.

127.0.0.1 purl 6379 > SADD animals cat

(integer) 1

127.0.0.1 purl 6379 > SADD animals pig

(integer) 1

127.0.0.1 purl 6379 > SADD animals dog

(integer) 1

127.0.0.1 purl 6379 > SADD animals panda

(integer) 1

127.0.0.1 purl 6379 > SADD animals dog

(integer) 0

127.0.0.1 purl 6379 > SMEMBERS animals

1) "pig"

2) "panda"

3) "cat"

4) "dog"

(5) ZSET ordered set (Sorted sets)

ZSET is also a collection of string type elements and does not repeat

Each element is associated with a score of type double when it is added to the collection. Redis sorts the collection elements from the largest to the smallest by scores, and the scores can be the same.

127.0.0.1 6379 > ZADD animals 0 pig

(integer) 1

127.0.0.1 6379 > ZADD animals 2 cat

(integer) 1

127.0.0.1 6379 > ZADD animals 4 dog

(integer) 1

127.0.0.1 6379 > ZADD animals 6 panda

(integer) 1

127.0.0.1 ZRANGEBYSCORE animals 6379 > 010

1) "pig"

2) "cat"

3) "dog"

4) "panda"

Redis configuration:

The configuration file for Redis is the redis.conf file in the installation directory, or you can view it from the command line:

127.0.0.1 6379 > CONFIG GET *

1) "dbfilename"

2) "dump.rdb"

3) "requirepass"

4) "hisen"

5) "masterauth"

6) "

7) "unixsocket"

8) "

9) "logfile"

10) "/ var/log/redis/redis.log"

11) "pidfile"

12) "/ var/run/redis/redis.pid"

13) "slave-announce-ip"

14) "

15) "maxmemory"

16) "0"

17) "maxmemory-samples"

18) "5"

19) "timeout"

20) "0"

21) "auto-aof-rewrite-percentage"

22) "100"

23) "auto-aof-rewrite-min-size"

24) "67108864"

25) "hash-max-ziplist-entries"

26) "512"

27) "hash-max-ziplist-value"

28) "64"

29) "list-max-ziplist-size"

30) "- 2"

31) "list-compress-depth"

32) "0"

...

Redis.conf profile entry:

Bind 0.0.0.0 # listening port

Protected-mode yes

Port 6379 # default listening port

Tcp-backlog 511

Timeout 0 # timeout, 0: no timeout

Tcp-keepalive 300

Does daemonize no # run in daemon mode

Supervised no

Pidfile / var/run/redis/redis.pid # files stored in pid

Loglevel notice # logging levels: debug, verbose, notice, warning

Files stored in logfile / var/log/redis/redis.log # logs

Number of databases 16 # databases

Save 9001 # synchronizes the data to the disk file with one update within 900s

Save 30010 # synchronizes data to disk files after 10 updates within 300s

Save 60 10000 # synchronizes data to disk files with 10000 updates within 60 seconds

Stop-writes-on-bgsave-error yes

Rdbcompression yes # whether local data is compressed

Rdbchecksum yes

Dbfilename dump.rdb # Local database name, default is dump.rdb

Dir / var/lib/redis # Local database storage path

Slaveof # the specified master address and port number when the native machine is slave

Masterauth # password for slave connection to master

Slave-serve-stale-data yes

Slave-read-only yes

Repl-diskless-sync no

Repl-diskless-sync-delay 5

Repl-disable-tcp-nodelay no

Slave-priority 100

Requirepass foobared # set the redis connection password

Maxclients 10000 # maximum number of client connections at a time

Maxmemory # specifies the maximum available memory space for redis

Appendonly no # specifies whether to log after each update operation

Appendfilename "appendonly.aof" # Update log file name

Appendfsync everysec # specifies the update log condition: no indicates that the operating system synchronizes the data cache to disk; always manually writes the data to disk with fsync () after each update operation; everysec synchronizes once a second

No-appendfsync-on-rewrite no

Auto-aof-rewrite-percentage 100

Auto-aof-rewrite-min-size 64mb

Aof-load-truncated yes

Lua-time-limit 5000

Slowlog-log-slower-than 10000

Slowlog-max-len 128

Latency-monitor-threshold 0

Notify-keyspace-events ""

Hash-max-ziplist-entries 512

Hash-max-ziplist-value 64

List-max-ziplist-size-2

List-compress-depth 0

Set-max-intset-entries 512

Zset-max-ziplist-entries 128

Zset-max-ziplist-value 64

Hll-sparse-max-bytes 3000

Activerehashing yes

Client-output-buffer-limit normal 0 0 0

Client-output-buffer-limit slave 256mb 64mb 60

Client-output-buffer-limit pubsub 32mb 8mb 60

Hz 10

Aof-rewrite-incremental-fsync yes

Redis publish and subscribe

Redis publish subscription is divided into sending message side (pub) and receiving information side (sub).

Example:

(1) create a subscription channel WeChat:

127.0.0.1 purl 6379 > SUBSCRIBE WeChat

Reading messages... (press Ctrl-C to quit)

1) "subscribe"

2) "WeChat"

3) (integer) 1

(2) Open another client and publish information to WeChat channel:

127.0.1 6379 > PUBLISH WeChat "Today is a funny day!"

(integer) 1

127.0.1 6379 > PUBLISH WeChat "I Love Linux!"

(integer) 1

(3) return to the subscription client to view:

127.0.0.1 purl 6379 > SUBSCRIBE WeChat

Reading messages... (press Ctrl-C to quit)

1) "subscribe"

2) "WeChat"

3) (integer) 1

1) "message"

2) "WeChat"

3) "Today is a funny day!"

1) "message"

2) "WeChat"

3) "I Love Linux!"

Command

Description

PSUBSCRIBE pattern [pattern...]

Subscribe to one or more channels that match the pattern

PUBSUB subcommand [argument [argument...]]

View subscription and publishing system status

PUBLISH channel message

Send information to a specified channel

PUNSUBSCRIBE [pattern [pattern...]]

Unsubscribe from all channels of a given mode

SUBSCRIBE channel [channel...]

Subscribe to one or more given channel information

UNSUBSCRIBE [channel [channel...]]

Unsubscribe from a given channel

Redis transactions:

Transaction isolation: all commands in a transaction are executed sequentially and are not interrupted by commands from other clients.

Transaction atomicity: commands in a transaction are either executed or not executed at all.

Example:

MULTI is the flag that opens a transaction, after which multiple commands are queued in the transaction, and when EXEC is executed, all commands in the queue are executed in turn.

127.0.0.1 purl 6379 > MULTI

OK

127.0.0.1 purl 6379 > SET name hisen

QUEUED

127.0.0.1 purl 6379 > GET name

QUEUED

127.0.0.1 purl 6379 > SADD xingshi zhao qiao sun li

QUEUED

127.0.0.1 purl 6379 > SMEMBERS xingshi

QUEUED

127.0.0.1 purl 6379 > EXEC

1) OK

2) "hisen"

3) (integer) 4

4) 1) "qiao"

2) "li"

3) "sun"

4) "zhao"

Command

Description

DISCARD

Cancel a transaction

EXEC

Execute a transaction

MULTI

Start a transaction

UNWATCH

Cancel monitoring of all key

WATCH key [key...]

Monitor key, if the key is changed before the transaction is executed, the transaction will be interrupted

Redis Security:

Set the service connection password, either through the configuration file or from the command line:

(1) check whether Redis enables password authentication:

127.0.0.1 purl 6379 > CONFIG GET requirepass

1) "requirepass"

2) "" # empty string means no authentication password has been set

(2) set Redis password authentication:

127.0.0.1 hisen 6379 > CONFIG SET requirepass "hisen" # set the password to hisen

OK

127.0.0.1 6379 > CONFIG GET requirepass # View password

1) "requirepass"

2) the "hisen" # password has been set to hisen

(3) verify the validity of the password:

127.0.0.1 GET name 6379 > try to get data before entering a password

(error) NOAUTH Authentication required. # display an error without authentication

127.0.0.1 6379 > AUTH hisen # enter the authentication password

OK # status is OK

127.0.0.1 6379 > GET name # get the data again

"hisen" # returns data correctly

Redis-benchmark of Redis performance testing tool

Redis-benchmark [option] [option value]

Redis-benchmark options:

Option

Description

Default value

-h

Specify hostname

127.0.0.1

-p

Designated port

6379

S

Server socket

-c

Number of concurrent connections

fifty

-n

Number of requests

10000

-d

Specify the data size of the SET/ get value in bytes

two

-k

1=keep alive 0=reconnect

one

-r

SET/GET/INCR uses random key,SADD to use random values

-P

Transmit the request through the pipeline

one

-Q

Forcibly exit redis. Show only query/ secvalue

-- csv

Export in CSV format

-l

Generate a loop to permanently execute the test

-t

Run only a comma-separated list of test commands

-l

Ldle mode, open only N idle connections and wait

-a

Specify authentication password

-dbnum

Select the database number

0

-e

Error message of standard output server reply

Example:

[root@node7] # redis-benchmark-h 127.0.0.1-p 6379-t set,lpush-c 100-n 10000

= SET =

10000 requests completed in 0.07 seconds

100 parallel clients

3 bytes payload

Keep alive: 1

98.23%

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.