Shulou(Shulou.com)06/01 Report--

Environment description:

The structure of one center and three platforms in the callcenter system of a state-owned enterprise needs to add security equipment such as check point firewall in front of the central server cluster.

Problem description:

After the check point firewall is cut online, some traffic seats are randomly selected for testing, signaling transmission, three-party call, call transfer and so on are all normal, but after most of the agents go to work, all kinds of call failures occur randomly, and the signaling transmission is normal.

Troubleshooting process:

If you find the above problems, conduct investigation immediately.

1. Log in to smartDashboard first to view the security policy.

From the policy status, it is shown that during the period between the cutoff test and the failure, there is no artificial change to the relevant policy, and everything is fine.

2. Secondly, check the smartlog system and randomly enter the IP address of the faulty phone in the log system

No drop packet information was found.

3. Then use smartview tracker to further check the packet situation, and add the faulty phone IP address to the filter condition:

View filter results

No abnormal information was found.

4. Grab the packet at the device interface by using the command line.

; [cpu_4]; [fw4_1]; fw_log_drop_ex: Packetproto=1 219.141.216.254 dropped byfwha_select_ip_packet Reason 0-> 219.141.216.12 dropped byfwha_select_ip_packet Reason: icmp probe reply to our request

; [cpu_2]; [fw4_3]; fw_log_drop_ex: Packetproto=17 0.0.0.0 Packetproto=17 8116-> 10.96.165.28 dropped by fw_handle_first_packetReason: Rulebase drop-rule 629

; [cpu_2]; [fw4_3]; fw_log_drop_ex: Packetproto=17 0.0.0.0 Packetproto=17 8116-> 219.141.216.12 fw_log_drop_ex 8116 dropped byfw_handle_first_packet Reason: Rulebase drop-rule 629

; [cpu_2]; [fw4_3]; fw_log_drop_ex: Packetproto=17 0.0.0.0 Packetproto=17 8116-> 10.96.165.28 dropped by fw_handle_first_packetReason: Rulebase drop-rule 629

; [cpu_5]; [fw4_0]; fw_log_drop_ex: Packetproto=17 192.168.254.4 dropped byfw_handle_first_packet Reason 137-> 10.96.21.136 dropped byfw_handle_first_packet Reason: Rulebase drop-rule 629

; [cpu_2]; [fw4_3]; fw_log_drop_ex: Packetproto=17 0.0.0.0 Packetproto=17 8116-> 10.96.165.28 dropped by fw_handle_first_packetReason: Rulebase drop-rule 629

; [cpu_2]; [fw4_3]; fw_log_drop_ex: Packetproto=17 0.0.0.0 Packetproto=17 8116-> 219.141.216.12 fw_log_drop_ex 8116 dropped byfw_handle_first_packet Reason: Rulebase drop-rule 629

; [cpu_2]; [fw4_3]; fw_log_drop_ex: Packetproto=17 0.0.0.0 Packetproto=17 8116-> 10.96.165.28 dropped by fw_handle_first_packetReason: Rulebase drop-rule 629

; [cpu_4]; [fw4_1]; fw_log_drop_ex: Packetproto=17 172.23.140.36 Packetproto=17 51221-> 10.96.4.249 dropped by fw_handle_first_packetReason 2055 dropped by fw_handle_first_packetReason: Rulebase drop-rule 629

; [cpu_4]; [fw4_1]; fw_log_drop_ex: Packetproto=1 10.96.165.20 dropped by fwha_select_ip_packetReason 0-> 10.96.165.28 dropped by fwha_select_ip_packetReason: icmp probe reply to our request

; [cpu_2]; [fw4_3]; fw_log_drop_ex: Packet proto=170.0.0.0:8116-> 219.141.216.12 dropped by fw_handle_first_packetReason 8116 dropped by fw_handle_first_packetReason: Rulebase drop-rule 629

; [cpu_2]; [fw4_3]; fw_log_drop_ex: Packetproto=17 0.0.0.0 Packetproto=17 8116-> 10.96.165.28 dropped by fw_handle_first_packetReason: Rulebase drop-rule 629

; [cpu_2]; [fw4_3]; fw_log_drop_ex: Packetproto=17 0.0.0.0 Packetproto=17 8116-> 10.96.165.28 dropped by fw_handle_first_packetReason: Rulebase drop-rule 629

; [cpu_2]; [fw4_3]; fw_log_drop_ex: Packetproto=17 0.0.0.0 Packetproto=17 8116-> 10.96.165.28 dropped by fw_handle_first_packetReason: Rulebase drop-rule 629

; [cpu_2]; [fw4_3]; fw_log_drop_ex: Packetproto=17 0.0.0.0 Packetproto=17 8116-> 219.141.216.12 fw_log_drop_ex 8116 dropped byfw_handle_first_packet Reason: Rulebase drop-rule 629

; [cpu_2]; [fw4_3]; fw_log_drop_ex: Packetproto=17 0.0.0.0 Packetproto=17 8116-> 10.96.165.28 dropped by fw_handle_first_packetReason: Rulebase drop-rule 629

There is no drop information related to the callcenter IP address.

5. After the inspection of the above steps, initially eliminate the check point firewall problem, and cooperate with the network team to check the other devices launched this time one by one, and finally found an abnormal state on an IPS device between the core exchanges. The log of the IPS device shows that during the period of call center traffic failure, the device was discarded by DDOS***, and a large number of suspected UDP packets.

6. After confirming with the voice team that call center uses UDP random port to transmit voice, after turning off the anti-DDOS function of IPS, the traffic agent fault disappears and the call returns to normal.

Cause analysis:

It turns out that when the call center system is working, it first transmits voice signaling between the phone and the server, and then communicates directly between the traffic seats, using the UDP1023-65535 random port. During the splicing test, there are few phones working at the same time, and the transmission is normal at this time. When all the agents in the three places are in place, because all the agents are in the same network segment, the UDP packet exceeds the preset DDOS threshold of IPS per unit time, so IPS determines that the UDP packet from the cc network segment is discarded by DDOS***, resulting in incomplete voice packets and random voice failures.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.