Shulou(Shulou.com)06/01 Report--

Removal of relevant sensitive information

1. Fault situation

The feedback from the staff on duty was received on July 11, 2019. From 23:45, the three sets of network management terminals showed that all the network elements of XXXXX, XXXXX and XXXXX were out of control, and resumed at 00:00 for 15 minutes. At the same time, the relevant technicians also reported that during this period, the servers of the three sets of network management were also cut off from their respective network elements, which also lasted for 15 minutes. The fault was discovered on July 11 and resolved on July 14, which lasted a total of 4 days.

two。 Topology description

The firewall is deployed between the server and the network management network in a transparent mode, and the corresponding security policy is configured.

The whole network uses layer 2 data exchange without routing forwarding.

3. Take measures

After line detection, device detection, packet capture, firewall log analysis, device log analysis and host log analysis all failed to locate the cause of the fault. After the firewall after-sales engineer to help locate, locate the cause of the fault, and at the same time change the firewall configuration to solve the fault.

4. Fault resolution

Packet capture Analysis of 4.1.XX Network Management Network

In the packet capture analysis of the XX network management system, a large number of OSPF HELLO packets in the fault event segment are forwarded from the XX network element aggregation switch, but can not be forwarded to the server through the firewall, indicating that the relevant configuration of the firewall prevents OSPF messages from being captured and sent.

Packet capture Analysis of 4.2.XX Network Management Network

According to the packet capture analysis of the XXXXX,XXXXX transmission network management network, in the fault event segment, a large number of layer 2 Ethernet frames are forwarded from the XXXXX to the aggregation switch, but they cannot be forwarded to the server through the firewall, indicating that the firewall blocks the relevant layer 2 packets.

4.3. Change firewall configuration

In view of the above phenomena, and through the test, adjust the relevant configuration of the firewall:

1. Configure global network-non-IP message forwarding

In view of the problem that OSPF can not be forwarded in XX network management network, a firewall is set to forward non-IP messages.

2. Configure the virtual switch-forward tagged packets

Aiming at the problem that some layer 2 data packets can not be captured and sent in the XX network management network, the firewall is set to forward marked data packets.

After the above configuration and full testing at the same time, the fault disappeared.

5. Analysis of network outage events

After analysis and inference, why the network management network of the 23:45-00:00 fixed event segment was interrupted, and why the failure did not occur in the firewall for more than a month, the following possible reasons are summarized:

1. The equipment in the XX network management network needs to send OSPF messages to establish a communication mechanism with the server to carry out some business operation from 23:45 to 00:00, but this service operation is not taken before the failure.

2. From 23:00 to 00:00, the equipment in the XX network management network needs to establish a layer 2 data packet with TAG to establish a communication mechanism with the server to carry out some kind of service operation, which is not taken before the fault.

3. The service operation mechanism of the three sets of XX,XX network management network at the same time in the fault event section is unknown, if necessary, it should be thoroughly investigated.

6. Relevant suggestions

The firewall filters and captures packets in strict accordance with the specified policy rules. This business failure is indeed caused by improper configuration of the firewall, but it also shows that there are network environment irregularities and abnormal behavior throughout the network. It is recommended to avoid risks in accordance with the following aspects:

1. Verify whether the equipment in the XX and XX network management network has timing service operations to be performed from 23:45 to 00:00.

2. Conduct vulnerability scanning and security reinforcement on the network management server.

3. Promote the evaluation of protection, promote the use of fortress machines in business.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.