Shulou(Shulou.com)06/03 Report--

This case happened to others. I think it is meaningful to learn from it. It is specially reproduced to record it.

PM said that there was a small demand similar to panic buying, and our first thought was to prevent inventory from being oversold, so the firm chose Redis locally. As long as it is guaranteed to be an atomic operation, you can prevent inventory from being oversold, so you naturally think of using atomic operations such as Incr/Decr.

Check the description of the Incr method in PHP's Redis extension:

/ * Increment the number stored at key by one. * * @ param string $key * @ return int the new value * @ link http://redis.io/commands/incr * * / public function incr ($key) {}

It can be seen that the Incr method returns the new value after the key operation, that is, the value after + 1, so we write the following code:

$num = $redis- > incr ($key); if ($num

< $max) { //入抢购成功队列，异步去执行抢购成功逻辑} else { //不好意思呢，已经被抢完了} 不知道你有没有闻到这段代码的坏味道，在大部分情况下会如你所想地运行，但是特殊场景下会 出现判断失效 的逻辑问题，例如： 1、key 由于某些原因失效了； 2、Incr 操作失败了，不会抛异常并返回 false； 上述两种情况，都会导致$num < $max条件成立，进而导致更严重的逻辑问题，最终超卖。 问题描述与分析 我们就抢购开始后就遇到了上述的第二种情况，下面描述整个过程。先通过 Cat 监控平台观察到访问量急剧上升，开始担心应用服务坑不住，随后日志平台报警 Incr 操作存在异常几率，再然后就出现超卖情况，紧急情况只能关闭业务开关。是什么原因导致判断条件成立？ 通过日志定位到 Incr 操作问题，便 Telnet 连接到线上 Redis 服务，发现了异常情况： \# 查看值GET key100# 尝试修改INCR keyREADONLY You can't write against a read only slaveINFO# Replicationrole:slave 可以看出来，该连接的机器目前处于从机状态，不可写操作，所以 Incr 操作返回 false，同时 PHP 不同类型比较会存在隐式转化，所以false < $num恒成立，导致计数器失效。而这一切又是由于 Redis 高可用不完善，当主从切换后，VIP 未能成功漂移，这部分是运维的锅，研发代码不够健壮，这锅同样要背 >

Rush incr ($key); if ($num > 0 & & $num < $max) {/ / join the panic purchase success queue and asynchronously execute the panic purchase success logic} else {/ / Sorry, it has been robbed}

Then, switch the Redis source to the high availability cluster (Codis), test and go online again. The panic purchase on the second day has been normal. Watching the traffic on Cat gradually stabilize, you feel at ease.

View sourc

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.