Shulou(Shulou.com)06/01 Report--

Why does the container stop automatically?

Why does the server operate stutter?

Where does the mysterious connection of the process point to?

Discovery-a container that stops automatically

One day I found that a container deployed on the server had been stopped, and I began to think that it had been stopped or deleted by a colleague.

However, when the login server restarts the container, it finds a strange phenomenon: the container stops automatically a few seconds after it starts.

Generally speaking, there may be something wrong with the container itself.

However, looking at the container log does not get any error messages, and the container image has been steadily deployed and running on other servers, so there should be no bug.

So the guess is that the system is short of resources, such as disk, memory, CPU.

There is still a lot of disk remaining to check, but an exception was found when looking at CPU and memory with the top command: the CPU utilization of a process reached 99%.

Of course, this situation is not particularly surprising for our company's servers, because we usually perform some computing tasks on the server, and it is normal to take up a lot of CPU.

But because this server is used by almost no other colleagues except me, and the process command line can't see it, it arouses my suspicion.

Verification-confirmation of the identity of the mining process in more than one anomaly

Such a high rate of CPU usage reminds me of the recent epidemic of mining viruses.

Use the netstat-anp command to see if the process has established an external network connection.

Sure enough, a connection was established to point to the IP address of 5.196.26.96. Look up the IP address at https://www.ipip.net/ip.html and point to a foreign place.

Further validates my guess. Because domestic servers have a strict filing management mechanism, many people will deploy their servers abroad.

For further confirmation, query the https://x.threatbook.cn/ip/5.196.26.96 on the threat intelligence platform again.

The platform also gives a threat warning, which can be boldly presumed that this is a mining process.

Of course, if you want to further confirm, you can extract the MD5 value of the execution file to the relevant website for identification.

Where does the mining procedure come from?

Mining programs are usually downloaded and executed by *, so check the download behavior with the history command.

No suspicious downloads were found, and it is likely that the operation records have been cleared or downloaded through other means.

To further rule out the possibility that other virus programs may start regularly as daemons or boot to start the mining process, check the crontab configuration information.

No new suspicious files were found, so * should not have set a scheduled task.

At the same time, no suspicious startup item configuration was found.

Suspicious images and containers

At this point, the clue is interrupted. We can only think from a different point of view.

According to the administrator, this server is rarely used and uses a strong password, so the possibility of password disclosure is very small.

Combined with the analysis of the stop time of the container I deployed, the server should have been * within a few hours after my deployment.

So the suspicion may have something to do with my operation.

A new situation is found when you use the docker command to search.

Some containers use unknown images (heybb/theimg2) or unofficial images (zoolu/ubuntu).

Search for these images on docker hub, and you can't find Dockerfile or instructions such as readme. And the upload time is very new, but the number of downloads is growing rapidly.

It is strange that this kind of unspecified and strangely named image can be downloaded many times, so it can be inferred that it is an uploaded image carrying *.

Then use the docker inspect command to look at these containers and find that the container does not write to the system file by mounting the directory, but executes a script file for mac.sh.

Use the cat command to view the file, with only one line of command

This is obviously digging for Monroe coins.

Summary

Now it is found that there is more than one server, some have deployed mining containers, and some have deployed the mining process and deleted records.

Processing-clearing processes, closing vulnerabilities

The first task, of course, is to clear the mining process and containers, as well as the corresponding execution files and images.

Of course, this is only a temporary cure rather than a permanent cure.

In order to solve the problem fundamentally, we need to carry out traceability analysis to prevent the server from being attacked again.

Combined with the above clues and personal experience analysis, it is likely to take advantage of the vulnerabilities of Docker.

I started the Docker remote API service when I deployed the container, which is likely to be exposed on the public network. Enter the server IP address and corresponding port in the browser immediately, and sure enough, you can access it!

It turns out that the server operator does not provide default firewall services, and the ports on the machine are directly exposed on the public network.

* the way can basically be guessed, through the Docker remote API server to operate the container, with the mining process of the container deployed to the server.

Or the mining program can be copied to the server by directory mount and triggered and executed in some way.

It is also easy to fix this vulnerability by stopping the exposure service.

Suggestion

Network security is actually a very important issue, but developers often do not pay enough attention to it.

In view of this incident, several experiences have been summed up:

Do not use the default port except for some web services (http/https).

* operations are generally automated and batch.

The operation is to use the port scan tool to scan a specific default port.

For example, in this example, it must be scanned to port 2375 of this server (2375 is the default port of Docker remote API).

This principle is actually a bit like phone fraud, using some very low-level deception to screen out vulnerable people.

So we usually try to avoid using the default port when writing programs.

Do not expose services that do not need to be accessed by binding 0.0.0.0.

Previously, when I started the Docker remote API service, I was listening to 0.0.0.0 IP because I saw that many online tutorials were configured in this way, but there were actually great security risks. There is a big difference between getting things done and getting things done. )

In fact, this service does not need to be provided to the public network in use. In fact, it only needs to listen to the subnet IP. For example, 127.0.0.1, 172.17.0.1.

Consider abnormal situations as much as possible

When developing, we not only consider the normal input and output of the program, but also need to assume some special circumstances for testing.

The following is the difference between the way developers and * * think:

Developers: a-> programs-> programs.

What the developer thinks about is to make sure that if you enter A, you will get B. * most of the time, you will enter S that the developer has not considered, so as to find bug or vulnerabilities.

Use a firewall to restrict port access.

Network services, firewalls are very important.

This time, it has something to do with the mindset that cloud server manufacturers have their own firewalls.

Verify the identity of the visitor through the certificate.

For services that need to provide external access, the use of authentication is also an effective way to avoid *.

For example, Docker supports TLS certificates to verify the identity of servers and clients.

Summary

The process of investigating * is very much like playing a detective, finding the killer and the modus operandi through clues at the crime scene.

Fortunately, when we found the problem, we did not immediately take the simple and rude way of reinstalling the system to solve the problem, otherwise the loophole would still exist and the server would still be *.

For more information about more authoritative network security, please refer to OWASP TOP10 2017, which contains the 10 most common types of vulnerabilities and preventive measures.

Vulnerabilities such as the Docker remote unauthorized vulnerability in this article and similar redis unauthorized vulnerabilities are among the vulnerabilities in OWASP TOP 10.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.