Shulou(Shulou.com)05/31 Report--

This article mainly introduces "how to understand RabbitMQ under HAProxy load balancer". In daily operation, I believe many people have doubts about how to understand RabbitMQ under HAProxy load balancer. The editor consulted all kinds of materials and sorted out simple and easy-to-use operation methods. I hope it will be helpful for you to answer the doubts of "how to understand RabbitMQ under HAProxy load balancer"! Next, please follow the editor to study!

Specific error message:

2018-05-04 11 Channel shutdown 21 ERROR 48.116 ERROR 60848-[.168.0.202: 8001] o.s.a.r.c.CachingConnectionFactory: Channel shutdown: connection error 2018-05-04 11 Switzerland 21 Swiss 48.116 ERROR 60848-[.168.0.202: 8001] o.s.a.r.c.CachingConnectionFactory: Channel shutdown: connection error 2018-05-04 11 21 Switzerland 48.125 INFO 60848-[nge.consumer1-8 ] o.s.a.r.l.SimpleMessageListenerContainer: Restarting Consumer@5196e177: tags= [{amq.ctag-d_wIlZIGxM3f0fsxkmYQfA=my_test_exchange.consumer1}] Channel=Cached Rabbit Channel: AMQChannel (amqp://admin@192.168.0.202:8001/,1), conn: Proxy@25a73de1 Shared Rabbit Connection: SimpleConnection@25fca927 [delegate=amqp://admin@192.168.0.202:8001/, localPort= 56258] AcknowledgeMode=AUTO local queue size=0 2018-05-04 11 nge.consumer1 21 INFO 48.126 60848-[nge.consumer1-9] o.s.a.r.c.CachingConnectionFactory: Attempting to connect to: [manager1:8001] 2018-05-04 11 Fringe 21 Swiss 48.393 INFO 60848-[nge.consumer1-9] o.s.a.r.c.CachingConnectionFactory: Created new connection: rabbitConnectionFactory#2b8bd14b:12/SimpleConnection@3fb9795a [delegate=amqp://admin@192.168.0.202:8001/ LocalPort= 56260] 2018-05-04 1115 nge.consumer1-8] o.s.a.r.l.SimpleMessageListenerContainer: Restarting Consumer@58b42519: tags= [{amq.ctag-T1HyrOd5Ykr_VQZDwxRslA=stream_exchange.consumer1}], channel=Cached Rabbit Channel: AMQChannel (amqp://admin@192.168.0.202:8001/,2), conn: Proxy@25a73de1 Shared Rabbit Connection: SimpleConnection@3fb9795a [delegate=amqp://admin@192.168.0.202:8001/, localPort= 56260], acknowledgeMode=AUTO local queue size=0

Spring Boot configure RabbitMQ (using HAProxy load balancer):

Spring: application: name: stream-rabbitmq-producer rabbitmq: host: manager1 port: 8001 username: admin password: admin123456

Recently, when using RabbitMQ cluster (HAProxy load balancer), the above error messages appear frequently, but the messages can be consumed normally. If you only use stand-alone RabbitMQ (no HAProxy), there is no error.

Perplexed by this problem for a long time, Google found a lot of information, but also did not find a solution, stumbled upon an article: RabbitMQ and HAProxy: a timeout issue

The article says that if you use HAProxy to configure a RabbitMQ high-availability cluster, you will encounter client connection timeouts.

Why is there such a problem? Because HAProxy configures the client connection timeout parameter (timeout client ms), if the client connection exceeds the configured parameter, HAProxy will delete the client connection.

The RabbitMQ client uses * * to connect to the agent and never times out, so why is there a problem? Because if RabbitMQ is inactive for a period of time, HAProxy will automatically close the connection.

So how to solve this problem? We see that HAProxy provides a clitcpka parameter configuration that can send TCP keepalive packets from the client.

We used it, but found that after the configuration, there was still the above problem.

Why?

[.] The exact behaviour of tcp keep-alive is determined by the underlying OS/Kernel configuration […]

What does it mean? It means that the sending of TCP keepalive packets depends on the operating system / kernel configuration.

We can use the command to view (the tcp_keepalive_time configuration in the server where HAProxy is located):

[root@manager1 ~] # cat / proc/sys/net/ipv4/tcp_keepalive_time 7200

The default configuration time of tcp_keepalive_time is 2 hours, which means that the interval between sending TCP keepalive packets is 2 hours, or sending TCP keepalive packets every 2 hours.

To be clear, although we configure the clitcpka parameter in HAProxy, because the interval between sending TCP keepalive packets is too long, far exceeding the timeout client timeout in HAProxy (the default seems to be 2 seconds), so the client connection is mercilessly deleted by HAProxy every 2 seconds, and then constantly rebuilt.

After all that has been said, how can we solve this problem?

Two options:

Modify the tcp_keepalive_time configuration of the system with an interval lower than the timeout client timeout configured by HAProxy (not recommended because it may affect other system services).

Modify the timeout client timeout in HAProxy to configure a greater tcp_keepalive_time interval than the system (recommended)

Because the interval between system tcp_keepalive_time sending TCP keepalive packets is 2 hours, we set the timeout client timeout in HAProxy to 3 hours:

Timeout client 3h timeout server 3h

Complete sample configuration:

[root@manager1] # cat / etc/haproxy/haproxy.cfg global log 127.0.0.1 local0 info global log 127.0.0.1 local1 notice daemon global maxconn 4096 defaults log global mode tcp option tcplog option dontlognull retries 3 option abortonclose maxconn 4096 timeout connect 5000ms timeout client 3000ms global timeout server 3000ms balance roundrobin listen private_monitoring bind 0.0.0. 0:8000 mode http option httplog stats refresh 5s stats uri / stats stats realm Haproxy stats auth admin:admin listen rabbitmq_admin bind 0.0.0.0:8002 server manager1 manager1:15672 server manager2 manager2:15672 server manager3 manager3:15672 listen rabbitmq_cluster bind 0.0.0.0:8001 mode tcp option tcplog balance roundrobin timeout client 3h timeout server 3h Server manager1 manager1:5672 check inter 5000 rise 2 fall 3 server manager2 manager2:5672 check inter 5000 rise 2 fall 3 server manager3 manager3:5672 check inter 5000 rise 2 fall 3

Rerun HAProxy, and the RabbitMQ test succeeds:

At this point, the study on "how to understand RabbitMQ under HAProxy load balancing" is over. I hope to be able to solve your doubts. The collocation of theory and practice can better help you learn, go and try it! If you want to continue to learn more related knowledge, please continue to follow the website, the editor will continue to work hard to bring you more practical articles!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.