Shulou(Shulou.com)06/01 Report--

Recently, I received a book from my friend-MySQL. In my spare time, I read some chapters, and I was impressed by the section on troubleshooting in the book. Some of the methodologies extracted from the book are exactly the ones I promoted in the team before. Its purpose is to standardize the fault emergency operation, and then improve the processing efficiency. By extension, this is actually a reflection of structured thinking in specific work. This kind of thinking can play an important role in a specific work, as well as individual and team development. Write down this article.

Fault handling flow 1.1 example: database fault handling

Here are some experiences from NetEase, sorted out from the book MySQL.

1.1.1 beforehand: fault handling principles

1) Communication first

In the event of database failure, be sure to maintain efficient communication with other teams such as operation and maintenance, development, products, etc. When DBA encounters a fault, don't forget the importance of communication. Even if time is tight, a brief communication can often get twice the result with half the effort. In the long run, it also helps to cultivate cooperation and trust with other people and other teams.

2) pay attention to artificial

Man-made faults account for a large proportion. By communicating in a timely manner and checking the history, confirm whether the operation is wrong and communicate with other teams for special actions. Of course, the best way to solve man-made failures is to automate, standardize and standardize the operation and maintenance of the database.

3) Fast recovery

When dealing with failures, a clear idea is to give priority to restoring services to ensure maximum availability of services, while others do not have to be given priority.

4) think twice before you act.

Some fault handling methods may have an impact on the database that is difficult to recover, so be careful and make a good backup as far as possible. Additional problems caused by unfamiliarity with the operation itself should be avoided as far as possible. Carefully consider the possible consequences of the command to avoid secondary damage to the system.

5) Service classification

Usually, services, applications and databases should be graded, and in the event of a large area failure, the core business can be restored according to the priority of the service.

1.1.2 in-process: fault handling process

1) Fault discovery

OS index

Load CPU usage disk space IO usage SWAP usage

DB index

Slow number of database surviving connections SQL master-slave delay

2) Fault location

Check operation

The program is released on the line table to change the online data modification background tasks, data statistics database parameter adjustment and other misoperations.

Check OS

System process CPU memory, SWAPIO system log

Check DB

After the connection slow query lock waits for QPS error log 1.1.3: troubleshooting

1) slow SQL

There is no index in the selection condition or the index is inefficient. There is an index, but no index is used, or the wrong index is selected. The filter condition is not strong, and the result set is too large.

2) High frequency of SQL execution

Malicious attacks on cache invalidation applications to achieve a sudden increase in logical unreasonable traffic

3) Lock conflict

Hot issues of major affairs

4) hardware issu

Raid card cache problem hardware corruption

5) the parameters are unreasonable

1.2 example: exception handling in GP database (my experience)

The following is the exception handling flow for GP that I summarized in the previous unit. The [] section of the figure corresponds to the specific processing steps (corresponding to the script or action document).

As can be seen from the above two examples, this is a "co-ordinated" way of working, not an "emergency" one. It emphasizes that after the failure, analyze and investigate according to the planned principles and steps to find out the core problems, and then deal with the existing problems in accordance with the existing plans. At the same time, in the process of processing, pay attention to risk avoidance and communication and coordination, in order to achieve a rapid solution to the fault. Obviously, this way represents a kind of foresight to the work, taking precautions against trouble, avoiding the kind of busy fighting fire, making the work passive forever. The above is actually the embodiment of a kind of "structured thinking".

Structured thinking 2.1 what is "structured thinking"? There is no logic when thinking, and most of the time I don't know where to start. There is no organization in the speech, but it is difficult to make things clear with a lot of words. When dealing with problems, it is inefficient and leaks from east to west, but the effect of being busy is not good. When you are faced with the above dilemma, you can consider training your own structured thinking to solve the problem.

Structured thinking: it means that a person can think from many aspects when facing a task or problem, deeply analyze the causes of the problem, systematically formulate an action plan, and take appropriate measures to make the work work efficiently and achieve high performance. When you do things in this way, you will have structured thinking, which will be of great help to your career promotion. Thinking determines development, and different levels of thinking lead to different results. In short, structured thinking refers to a hierarchical mode of thinking from the whole to the part. Is to borrow some thinking framework to assist thinking, the fragmented information for systematic thinking and processing, so as to expand the level of thinking, more comprehensive thinking.

2.2 structured way of thinking

How to carry out structured thinking, there is also a methodology, generally speaking, there are two steps, the first is to "establish a center", and then to "decompose".

1) Establishment of a centre

The establishment of the center is to clearly define the problems to be solved, to make clear the goal, and it is also a way of thinking with the end as the beginning. In other words, you need to figure out why first, and then how. There are two ways to establish a center: top-down and bottom-up. We will explain in more detail later.

The establishment of the center is not usually formed at once, and the abstraction of the center will be adjusted accordingly as the understanding of the problem changes. Different levels of abstraction face different breadth of problems. Which level of abstraction should be used as the "center" depends on the specific situation. The higher the level of abstraction, the wider the problem domain to be solved and the greater the extension. For example, in the face of the problem of "too much bug in the system", upward abstraction is "improving code quality" and downward abstraction is "enhanced testing". Which one you choose as the center depends on what you are currently trying to solve.

2) structured decomposition

Use structured thinking to decompose the problem. Decomposition strategies are four common logical orders, namely, deductive order, time order, spatial order and degree order.

3) logical order

The following figure shows the corresponding legend of the XMind tool.

Deductive (causal) sequence

The deductive reasoning mode of "major premise, minor premise and conclusion" is the deductive sequence. For example, classical syllogism: everyone is going to die, Socrates is human, Socrates is going to die.

Time (step) sequence

"first, second, third", "first, then, then" and so on, many of the time order is also the order of cause and effect.

Spatial (structural) order

"Front end, back end, data", "Boston, New York, Washington", breaking up into parts (breaking the whole into parts) are all spatial order. When doing spatial decomposition, we should pay attention to meet the MECE (Mutually Exclusive Collectively Exhaustive, mutually independent, completely exhaustive) principle.

Degree (importance) order

For example, "the most important, secondary, unimportant" and so on.

2.3 thinking of "top-down"

Top-down thinking is applicable to situations where the problem is relatively clear. We just need to find the core elements of the problem and expand it. This is a very typical structural way of thinking with a total score. Summarize first, then diverge. Thinking in this way helps to form, organize and construct a mind map, which promotes the brain to think in a natural and orderly manner, allowing you to analyze a problem in a more comprehensive way. Here are several common top-down thinking models:

1) STAR rule

Situation background Target Target Action Action Result result

2) SWOT analysis method

Strengths advantage Weaknesses inferiority Opportunities opportunity Threats threat

3) problem solving

Analyze the problem > find the cause > set the goal > propose the solution > implement

2.4 "bottom-up" thinking

For the situation where the problem is not clear enough, it is necessary to classify, prune and summarize a variety of messy contents into one center. According to the Pyramid principle, "anything can be summed up into a central argument, which can be supported by three to seven arguments, and each first-level argument can derive other sub-arguments." In this way, the following pyramid structure of thinking can be formed.

However, it is difficult to use this way of thinking directly when we have not yet mastered this structured way of thinking. At this point, we can use a bottom-up way of thinking to find the structure.

List all the main points of thinking as far as possible, find out the relationship, classify and summarize the main points, refine the viewpoints, and perfect the ideas.

The summary is: diverge first, then summarize. Thinking in this way not only makes it easier to find logical structures, but also makes it easier to cultivate your structured thinking. For example, when we are faced with career development choices, how to summarize and refine our own decisions.

2.5 enhanced-Extensible thinking

The core goal of expansive thinking is to improve the breadth of thinking, which can be expanded in three directions:

Give an example: solve N problems of the same type

This way of thinking is characterized by inference and analogy, which is equivalent to the effect of batch processing, which can greatly improve the efficiency of solving problems and avoid repeated processing.

Looking for possibilities: expanding different ways to solve problems

The common means of expanding thinking is whether it is possible to change more ways of understanding, or more solutions.

Dig deep into the root: dig into the deep-seated causes of the problem

This way of thinking is to break through the superficial solution of the existing problems, but to dig deep into the causes and explore the fundamental problems. Only in this way can the problem be solved fundamentally.

2.6 example: my experience of a failure (dig deep into the cause)

Write at the end

There are many ways of thinking, and you can try to use the above methods in practical work. After sticking to it for a period of time, you will find that you are more logical when thinking, and your speech is more organized and persuasive. Not only that, you can also use this structured thinking to build and construct your own thinking system.

Author: Han Feng

The first release is on the author's personal official name "Han Feng Channel".

Source: Yixin Institute of Technology

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.