Shulou(Shulou.com)06/01 Report--

This article was compiled and first published by the technical team of Grape City.

Please indicate the source of the reprint: grape City official website, Grape City provides developers with professional development tools, solutions and services, empowering developers.

Introduction

At the beginning of the development of Web application, relational database received more and more attention and application, because the access and concurrency of Web sites were not high and the interaction was less at that time. Later, with the increase of traffic, Web sites using relational databases more or less began to have some performance bottlenecks, and the source of the bottleneck is generally on the disk I hand O. With the further development of Internet technology, various types of applications emerge in endlessly, which leads to more demand for performance in the era of cloud computing and big data, which is mainly reflected in the following four aspects:

Low latency reading and writing speed: the rapid response of applications can greatly improve user satisfaction

Support huge amounts of data and traffic: for large-scale applications like search, you need to take advantage of PB-level data and be able to handle millions of levels of traffic

Management of large-scale clusters: system administrators want distributed applications to be easier to deploy and manage

Consideration of huge operating costs: IT hopes to significantly reduce hardware, software and labor costs

In order to overcome this problem, NoSQL arises at the historic moment. It has the advantages of high performance, strong scalability and high availability, and is favored by a wide range of developers and warehouse managers.

What is Redis?

Redis is one of the most popular NoSQL databases today. Redis is an open source, multi-data structure, network-based, memory-based, optional persistence key-pair storage database written in ANSI C with the following features:

Run based on memory with high performance

It supports distribution and can be expanded infinitely in theory.

Key-value storage system

Open source log database written in ANSI C, complying with BSD protocol, supporting network, memory-based and persistent, Key-Value database, and providing API in multiple languages

Compared with other database types, Redis has the following characteristics:

Cpact S communication model

Single process and single thread model

Rich data types

The operation is atomic.

Persistence

Highly concurrent read and write

Support for lua scripts

Which big factories are using Redis?

Github

Twitter

micro-blog

Stack Overflow

Alibaba

Baidu

Meituan

Sohu

What are the application scenarios of Redis?

Redis's application scenarios include: caching systems ("hot" data: high-frequency reads, low-frequency writes), counters, message queuing systems, rankings, social networks and real-time systems.

Data types and main characteristics of Redis

The data types provided by Redis are mainly divided into five own types and one custom type, which include: String type, hash type, list type, collection type and sequential collection type.

String type:

It is a binary secure string, which means that it can store not only strings, but also pictures, videos and other types, with a maximum length of 512m.

For each data type, Redis provides a wealth of operation commands, such as:

GET/MGET

SET/SETEX/MSET/MSETNX

INCR/DECR

GETSET

DEL

Hash type:

This type is a map consisting of a field and an associated value. Where field and value are both string types.

The operation commands for Hash are as follows:

HGET/HMGET/HGETALL

HSET/HMSET/HSETNX

HEXISTS/HLEN

HKEYS/HDEL

HVALS

List type:

This type is a collection of string elements sorted in insertion order, based on a double-linked list implementation.

The operation commands for List are as follows:

LPUSH/LPUSHX/LPOP/RPUSH/RPUSHX/RPOP/LINSERT/LSET

LINDEX/LRANGE

LLEN/LTRIM

Collection type:

The Set type is an unordered collection, and the biggest difference between it and the List type is that the elements in the collection are unordered and unique.

The bottom layer of the Set type is implemented through a hash table, and its operation command is:

SADD/SPOP/SMOVE/SCARD

SINTER/SDIFF/SDIFFSTORE/SUNION

Set type is mainly used in some scenarios, such as social scenarios, through intersection, union and subtraction operations, through the Set type can easily find common friends, common concerns and common preferences and other social relationships.

Sequential collection type:

ZSet is an ordered collection type, and each element is associated with a fractional weight of type double, which is used to sort the members of the collection from small to large. Like the Set type, the underlying layer is implemented through a hash table.

ZSet command:

ZADD/ZPOP/ZMOVE/ZCARD/ZCOUNT

ZINTER/ZDIFF/ZDIFFSTORE/ZUNION

Data structure of Redis

The data structure of Redis is shown in the following figure:

With regard to some of the definitions in the table above:

A compressed list is one of the underlying implementations of list keys and hash keys. When a list key contains only a small number of list items, and each list item is either a small integer or a short string, Redis uses a compressed list as the underlying implementation of the list key

The set of integers is one of the underlying implementations of the set key. When a set contains only integer-valued elements and the number of elements in the set is small, Redis will use the set of integers as the underlying implementation of the set key.

The following is an example of defining an Struct data structure:

Simple dynamic string SDS (Simple Dynamic String)

Based on the shortcomings of traditional strings in C language, Redis has constructed an abstract type called simple dynamic string, called SDS for short, whose structure is as follows:

SDS runs through almost all the data structures of Redis and is widely used.

Characteristics of SDS

Compared with the C string, SDS has the following characteristics:

1. Constant complexity gets string length

In Redis, the length of the saved string can be directly obtained by using the len property of the SDS string.

Degree, which directly reduces the complexity required to obtain the length of a string from O (N) of a C string to O (1).

two。 Reduce the number of memory reallocations caused by string modification

Through the nature of the C string, we know that for a C string containing N characters, the underlying implementation is always an array of 1 characters long (ending with an extra null character)

So if the string needs to be modified at this time, the program needs to make a memory reallocation (possibly an extension or release) to the C string array in advance.

Memory reallocation means that it is a time-consuming operation.

Redis cleverly uses SDS to avoid the shortcomings of C strings. In SDS, the length of the buf array is not necessarily the number of characters in the string plus one. The buf array can contain unused bytes, which are recorded by the free attribute.

At the same time, SDS adopts the strategy of space pre-allocation, which avoids the time-consuming operation of memory reallocation when C string is modified, and reduces the memory reallocation from N times to N times at most.

Here is a simple definition of SDS by Redis:

Redis feature 1: transaction

Command serialization, executed sequentially

Atomicity

Three phases: start the transaction-order to join the queue-execute the transaction

Command: MULTI/EXEC/DISCARD

Redis feature 2: publish subscription (Pub/Sub)

Pub/sub is a mode of message communication

Pub sends messages, Sub accepts messages

Redis clients can subscribe to any number of channels

"fire and forgot", send and forget

Command: Publish/Subscribe/Psubscribe/UnSub

Redis feature 3:Stream

Redis 5.0Add

Waiting for consumption

Consumer group (intra-group competition)

Consumption historical data

FIFO

These are the basic concepts of Redis, and below we will introduce the "holes" that may be stepped into in the development process.

Analysis of common problems in Redis: breakdown

Concept: when Redis acquires a key, the act that a request must be made to DB because the key does not exist is called "Redis breakdown".

The cause of the breakdown:

First visit

Malicious access to key that does not exist

Key expires

Reasonable circumvention plan:

When the server starts, write in advance

Standardize the naming of key and intercept through middleware

Set a reasonable TTL or never expire for some high-frequency access Key

Analysis of common problems in Redis: avalanche

Concept: after the Redis cache layer goes down for some reason, all requests will flock to the storage layer, and high concurrent requests in a short period of time may cause the storage layer to hang up, which is called "Redis avalanche".

Reasonable circumvention plan:

Use Redis clustering

Current limit

Application and practice of Redis in Product Development

For this reason, I am very pleased to introduce to you that Jim, the architect of Grape City, will bring you an open class at 14:00 on 2019-11-27, in which Jim will not only explain the basis of Redis, but also actually demonstrate the problems and solutions encountered by his project team when using Redis, which is of more reference significance and learning value for students who have just come into contact with Redis. Welcome to join us at that time. Open course address: https://live.vhall.com/661463644.

NodeJS is used in the back end.

Redis services using Azure

Usage scenarios of Redis

-token cache for token verification

-IP whitelist

The problems encountered

"Network jitter" or Redis service exception causes Redis access timeout

Stability of Redis client driver

-connection pool "Broken connection" problem

-Redis reset problem caused by JS's Promise

Let's take a brief look at the advanced knowledge of Redis.

Brief introduction of Advanced Redis Protocol

Redis client communication protocol: RESP (Redis Serialization Protocol), which is characterized by:

simple

Fast parsing speed

Good readability

Redis cluster internal communication protocol: RECP (Redis Cluster Protocol), which is characterized by:

Two tcp connections per node

One responsible for client-server communications (P: 6379)

One responsible for communication between node (P: 10000 + 6379)

Data types supported by Redis protocol:

Simple character (first byte: "+")

"+ OK\ r\ n"

Error (first byte: "-")

"- error msg\ r\ n"

Number (first byte: ":")

": 123\ r\ n"

Batch characters (first byte: "$")

"& hello\ r\ nWhoa re you\ r\ n"

Array (first byte: "*")

"* 0\ r\ n"

"*-1\ r\ n"

Besides Redis, is there any NoSQL database?

There are many NoSQL databases similar to Redis on the market, as shown in the following figure, in addition to Redis, there are also MemCache, Cassadra and Mongo. Next, let's give a brief introduction to these databases:

Memcache: this is a database very similar to Redis, but its data types are not as rich as Redis. Developed by Brad Fitzpatrick of LiveJournal, Memcache, as a distributed cache system, is used by many websites to improve the access speed of websites, and has a significant effect on the access speed of some large websites that need to access the database frequently.

Apache Cassandra: (commonly referred to as C* in the community) this is an open source distributed NoSQL database system. Originally developed by Facebook, it is used to store data in simple formats such as inboxes, integrating the data model of Google BigTable and the fully distributed architecture of Amazon Dynamo. Facebook opened Cassandra in 2008. Because of its good scalability and performance, it has been adopted by Apple, Comcast, Instagram, Spotify, eBay, Rackspace, Netflix and other well-known websites, and has become a popular distributed structured data storage scheme.

MongoDB: a document-oriented NoSQL database based on distributed file storage, written by C++, designed to provide scalable, high-performance data storage solutions for WEB applications. MongoDB is a product between relational database and non-relational database, which is the most functional and most like relational database in non-relational database. It supports a very loose data structure and is a BSON format similar to json.

Summary

The above is an introduction to Redis. If you want to know more, please feel free to let me know through comments and private messages.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.