Shulou(Shulou.com)06/03 Report--

Recently, we have been following up the problem of storm, from the stability of storm cluster to monitoring to the problem of upgrading to bolt to write redis, because the company does not have a professional operation and maintenance redis, so we have to do it on our own. Here are some of the problems encountered:

Summarize the current problems of storm writing redis:

1.redis peak write exception, increased redis monitoring, found cpu performance bottleneck (redis single thread, the highest 10w/s processing capacity)

two。 Previously, the concurrency of redis bolt was more than 200. too much concurrency had a great impact on the performance of redis, and now it has been reduced to 5.

3. The monitor monitoring of redis is turned off, and the performance loss of resident monitor monitoring to redis is about 30%.

4. The rdb persistence mode of redis is turned off, the aof mode is enabled, and the low peak aofrewrite is enabled.

5. Expand the capacity to 8 instances. Using jedissharding, the capacity of a single machine exceeds the capacity of 5W/s during peak hours.

6. Remove the select operation and use the default db0

7. According to the analysis of the peak data, the ping operation accounts for more than 50% of the processing capacity of 40w/s. Adjusting the setting of jedispool basically shields the operation of ping.

8. Batch processing on the Bolt side to reduce the amount of writing

9.40% of the expire operations test the performance of ttl+expire vs expire, and the performance loss of the ttl+expire-based approach in one operation is about 35%.

There will be a performance improvement if the same key operates sequentially in one thread (we don't have this scenario at present)

1) Direct expire

HardedJedis.set (key,value)

HardedJedis.expire (key,1000)

2) ttl+expire

HardedJedis.set (key,value)

Long re = shardedJedis.ttl (key)

If ((re = =-1) | | (re = =-2)) {hardedJedis.expire (key,1000)}

10. From the point 8 test, 40% of the expire operation can not be saved, and can only be optimized by increasing the single processing capacity (pipline).

11. The scheme of lvs- > twemproxy- > redis is tested, which is not stable. Considering that there are many referenced components, twemproxy is also a black box for us.

The 12.jedissharding solution will have some delays during peak hours, and the stand-alone solution is relatively stable. If the amount of access data becomes larger, you still have to follow the sharding mode. The reason for the delay needs to be followed up.

Finally, several monitoring charts are attached:

1.redis cpu

2.redis conns

3.redis command/s

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.