Shulou(Shulou.com)06/01 Report--

Sina Weibo: @ wandering

Blog address: dayong.info

I have been working on the Internet full-time for almost five years, which is exactly the same age as my daughter, and both of them are the most important experiences in my life. When the daughter was born, she was completely unaware of the change in her role. As the daughter grew up, she became more and more aware of the changes in her body and mind as a father. Similarly, after experiencing a painful career transformation almost from scratch, through several years of online experience, I finally found a kind of down-to-earth confidence and joy.

To get to the point, as mentioned in the previous article "three important services of network management: AAA, NTP and SYSLOG", there are a large number of difficulties and challenges in the initial stage of network work, of which the most urgent and important is fault handling.

Why is fault handling a top priority?

In the event of a failure, especially a major failure, no matter whether it is learning or communication, whether it is learning or communication, or optimizing the network or architectural design, sufficient resources must be mobilized to give priority to service impact as much as possible. shorten fault handling time. Fault handling, in particular, tests the comprehensive ability of a network engineer, understanding of network architecture, familiarity with various devices, mastery of related technologies, analysis and control of problems, understanding of product applications, psychological tolerance in the face of emergencies, and so on.

Therefore, to improve their own fault handling level has become the primary development goal.

To achieve this goal, a number of measures have been taken, including:

1) log in all the IDC management servers, core devices, border devices, × × devices and important aggregation switches in the network for several times to ensure that you can log in to the failure-related devices in the event of problems, and have a certain impression and understanding of the devices.

2) be familiar with the basic operation commands of equipment from different manufacturers, and formulate some tasks or scenarios, such as checking CPU to determine common factors and solutions that affect CPU, such as information collection, state detection and operation processing related to various failures that may occur, such as state detection of key layer 2 and layer 3 protocols and common operation methods (STP, OSPF, etc.), such as how to quickly go online to replace equipment For example, how to quickly locate the fault location and so on.

3) be familiar with the physical and logical connections of key devices in different IDC, IP/ network segments, functions, key configurations, special features, etc.

4) read a large number of online equipment-related documents, work-related classic technical books, communicate with old colleagues, and consult each other as soon as possible to understand each other's practices and whether their own handling methods are appropriate.

5) re-learn and understand some network principles, such as what is Ethernet, such as how much bandwidth can the GE interface run, and why? For example, why does an CRC error occur on an optical interface?

6) began to use text to record some key network information, and later developed into a personal CMDB.

* Note 1: why use text? Cross-platform, efficient retrieval, can be called by external programs.

* Note 2: why not do online CMDB? Energy capacity is limited, the company already has CMDB, and if you already have CMDB to use.

The above measures allow me to quickly get into work and, more importantly, to be on a good technical team.

Next, more and more transactions and failures are found, and you need to be familiar with and understand the connectivity between IDC and its role functions.

The north-south traffic of IDC relies on the Internet exit bandwidth provided by ISP, while the east-west traffic depends on a large number of direct connect and × ×. When some individual and local problems can be solved, some structural and interconnection-related problems will naturally be concerned.

Therefore, I set another goal for myself, drawing the dedicated line and routing connection topology of the whole network of the company.

Another reason that prompted me to do so was that there was no physical or logical topology of the whole network in the network team at that time. This can lead to some problems, such as the fact that new members of the team spend a lot of time familiarizing themselves with the network, although this is a necessary process. For example, not everyone is very clear about the overall network architecture, and everyone has a different network structure, which brings inconvenience or even obstacles to communication and collaboration.

The following six topologies are network topologies drawn at different times over the years, and these versions are basically a continuation of development. From the change of the topology diagram, it can reflect the change of my understanding, understanding and attitude towards the network, as well as the change of the company network.

Stage one

Topology covers only a few important IDC that you are concerned about, as well as the important dedicated lines and other connections between them, which is relatively simple. Many colleagues have painted similar topologies, which seems to be the only way for network engineers to grow up. But this topology helps me a lot. Its existence enables me to accumulate the harvest of every thought, and constantly improve and improve it.

Stage two

The content is rich, and the important change is the addition of interconnection interface and address information. At that time, OSPF and static routes were mixed in the network, and the route adjustment was very complex. Often because of the congestion of a link, it is necessary to find a large number of traffic and split the traffic through static routes, which will lead to routing errors. When individual link failures are interrupted, traffic analysis and rerouting are all manual operations, which is very scary.

When you are called up in the middle of the night and need to determine link failures, develop temporary solutions, and make a large number of static road adjustments as soon as possible, the determination to change the status quo is really a torture.

Stage three

So there is stage 3, at this time, the main links of the whole network have been covered by OSPF, although there are static routes in some narrow-band links, but basically most of the cases of manual routing are gone. The scope of attention is extended to all routing nodes in the network, and richer information is added to the topology. It is expected that the most critical information of fault handling and daily changes can be found from the topology. For example, route-id, such as cost values, such as interconnect interfaces and interconnect addresses.

In addition, when drawing the topology, try to do as much as possible: the structure is clear and the relationship is intuitive when the picture is scaled down to see the whole, when the graph is enlarged and viewed locally, it is rich in information and easy to be associated with the outside, and all routing nodes are shown clearly in the limited space as far as possible. Soon, the network topology became a reference document for team members and was regularly updated and maintained by themselves.

When the whole team faces the same topology, everyone is freed from the event of building a network structure in mind, analyzing and discussing various problems on the basis of the same information, which nodes are not redundant, which link traffic is unreasonable, where expansion is needed, and so on, things are getting better and better.

Stage four

At this point, the focus of the network team is how to extricate themselves from passive expansion and link redundancy.

First of all, remove the unimportant nodes from the topology, change the triangular IDC relationship, try to make use of the existing links, refer to the core nodes located in different ISP, select a group of ring nodes from all IDC, create a logical backbone network, and all other nodes are downgraded to access nodes connected to the backbone network.

In order to meet the transmission needs of the majority of the network traffic between the two main ISP, two backbone nodes are selected for each ISP, and two logical links are established between two pairs of different ISP nodes, so as to achieve the effect of four logical links between the two ISP. In order to achieve this goal, all the cost values of the OSPF links in the whole network are specified manually and set strictly according to the designed transmission level.

In addition, in order to achieve the effect of backbone transmission, several large-capacity dedicated lines have even been abandoned because once these lines are used, they will lead to structural damage. In order to make the whole team accept a similar point of view, a lot of time has been spent on communication, technical exchange, fault simulation based on topological model, and has been improved and perfected many times.

Finally, it develops from the design and planning stage of stage 4 to stage 5.

Stage five

At this point, the concept of backbone transmission network has finally been established, and a series of work, such as IDC selection, dedicated line expansion, network planning and so on, have become reasonable and evidence-based. And the network work, also slowly away from the role of the fire brigade, although the network has become more complex and larger, but everything is in order, you can clearly know what problems exist and the priority of solving the problems. The highly reliable and stable network structure ensures that the occurrence of service interruption is reduced to the maximum extent when a failure occurs, which frees the network team from the huge business impact pressure, and can solve the fault with peace of mind when a fault occurs.

Next, new challenges are encountered unexpectedly, a large number of inter-IDC data calls, resulting in a rapid shortage of transmission network capacity. It is precisely because the fundamental deficiency of the structure has been solved in the early stage, so the problem of capacity expansion brings to the network team only which way to expand capacity, and the main energy can be devoted to studying things that need to be paid attention to at present and in the future.

Stage 6

According to the recent situation of the company's network, we finally choose to expand the existing dedicated line through wavelength division technology, and its expansion capacity can at least meet the needs of development in the next 3 years. In addition to the improvement of the capacity of the transmission backbone network, more energy has been invested in reducing single point failures of equipment and links, reducing mutual interference between transmission network and switching network, traffic analysis, network monitoring and so on.

Not long ago, the maintenance of the network topology was officially handed over to a new colleague who entered the department for more than a year, which felt like handing over his well-cared for and grown-up child to someone else. At present, the company has been on the right track in network interconnection, as long as it is not to change for the sake of change or performance, but to objectively analyze problems and seek solutions based on the actual needs of network development and business development. It will develop better and better.

The above is the development process of the network of my company, and I hope it will be helpful to my colleagues on the network. Still does not involve too many details, measures to local conditions, vary from person to person, the important thing is the ideas and ideas.

* Note: the topology in this article has tried its best to narrow down and deal with the key information, so as not to disclose the company's secrets and to show the elements of the topology. please forgive me if there is any unknown.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.