Shulou(Shulou.com)06/01 Report--

This article "Docker container port mapping suddenly unable to connect how to troubleshoot" article, most people do not understand, so the editor summed up the following content, detailed, clear steps, with a certain reference value, I hope you can get something after reading this article, let's take a look at this "Docker container port mapping suddenly can not connect how to troubleshoot" article.

I. background

Generally speaking, docker containers that provide services are required. After startup, we use the-p command to expose the external access port to the outside. For example, when we start docker registry, we map port 5000 for external access:

Docker run-d-p 5000 registry

Recently, however, we encountered a very strange situation: docker registry was deployed in a centos 7 test environment in the R & D team, and the port was exposed. After starting the container, it will work normally for a period of time, but after an indefinite time interval, the external host will be unable to pull the image from the repository, prompting timeout:

However, the repository can be accessed normally on the docker host machine:

As for this problem, external access will not be available until the defective docker daemon service is manually restarted, but this problem will occur again after a while.

II. Troubleshooting

My first reaction to this question was to ask anyone in the group if anyone had restarted centos 7's own firewalld.

Because I configured this server, although the firewall is on, but I have opened the port access, so it must not be because the firewall blocks the connection. However, since this article is a trampling investigation document, I still write about this situation.

Case 1: firewall is on but no port is open

Centos 7 comes with firewall firewalld enabled, and we can check the status of firewalld with the following command:

Firewall-cmd-state

If the output is "not running", the firewalld is not running, and all the protection policies are not enabled, so you can rule out the situation where the firewall blocks the connection.

If the output is "running", which indicates that firewalld is running, you need to enter the following command to see which ports and services are now open:

Firewall-cmd-list-portsfirewall-cmd-list-services

You can see that the current firewall only opens 80/tcp ports, ssh services (22/tcp) and dhcpv6-client services, and does not open 5000/tcp ports mapped by docker containers.

There are two solutions:

1. Shut down the firewalld service:

If you don't need a firewall, just turn off the firewalld service.

Systemctl stop firewalld.service

two。 Add a policy to open the specified port:

For example, if we want to open the external 5000/tcp port, we can use the following command:

Firewall-cmd-add-port=5000/tcp-permanentfirewall-cmd-reload

If you only temporarily open the port and remove the "--permanent" parameter from the first command, this policy will not work when the firewalld service is restarted again.

Case 2: artificially restart the firewalld service of centos 7

Firewalld is a new component introduced by the centos system in version 7, which is simply the wrapper of iptables to simplify firewall-related settings.

However, firewalld and docker do not get along very well. When firewalld starts (or restarts), the docker chain is removed from iptables, causing docker not to work properly:

Firewalld

Centos-7 introduced firewalld, which is a wrapper around iptables and can conflict with docker.

When firewalld is started or restarted it will remove the docker chain from iptables, preventing docker from working properly.

When using systemd, firewalld is started before docker, but if you start or restart firewalld after docker, you will have to restart the docker daemon.

Excerpt from docker official document "centos-docker documentation"

In centos 7, there will be no problem if the docker service is set to boot with systemd, because docker clearly indicates "after= firewalld.service" in the systemd configuration file to ensure that docker daemon starts after firewalld starts.

(docker: can't you afford to hide if you can't piss me off?)

However, every time a user manually restarts the firewalld service, the firewalld service deletes the docker chain in which the docker daemon is written to iptables, so you need to restart the docker daemon service manually and let the docker daemon service rebuild the docker chain.

However, I asked the other two R & D in the group, and both said they had not moved. Checked shell's history and couldn't find the corresponding record.

That's weird. But after a period of squatting, I finally found a new reason:

Case 3: ip_forward is not enabled

Since we have been unable to locate the problem, our research and development team manually logs in to the host to restart the docker daemon service when we find that the warehouse cannot be accessed normally.

Before logging in to the host server to restart the docker daemon service, I suddenly remembered that I had encountered another problem when using docker: if the host does not enable the ip_forward feature, the docker container will output a warning message when it starts:

Warning: ipv4 forwarding is disabled. Networking will not work.

And the external network cannot be accessed in the launched container, and the external ports exposed by the container cannot be accessed normally:

Could this failure be caused by the fact that the ip_forward function of the host is not enabled?

Sysctl net.ipv4.ip_forward

Sure enough, the output indicates that the ip_forward function of the current system is disabled!

But the problem is, when you started the container, it was all good, and there was no output, so how could the ip_forward function be disabled when you use it?

Wait, wait,

With this assumption, I manually restart the docker daemon service:

Sure enough, the docker daemon service will check the ip_forward configuration item of the system during startup. If the ip_forward function of the current system is disabled, it will help us to temporarily enable the ip_forward function, but the temporarily enabled ip_forward function will fail for a variety of other reasons.

Although there is no conclusive evidence for the cause of this failure, I seriously suspect that it is caused by the restart of network services. Because the web project being developed by our R & D team is running on the server host in question, one of the functions is to modify the ip address of the Nic. After modifying the Nic ip, this function will automatically call the following command to restart the network service:

Systemctl restart network.service

Restarting the network service will invalidate the temporarily enabled ip_forward configuration automatically set by the docker daemon service:

In addition, because the program invokes the command directly, it does not leave a trace in the history command.

As for the fix, it is very simple, just one line of command is fine:

Echo 'net.ipv4.ip_forward = 1' > > / usr/lib/sysctl.d/50-default.conf

When the execution is complete, restart the server or load the configuration from the file using the following command:

Sysctl-p / usr/lib/sysctl.d/50-default.conf

The above is about the article "how to troubleshoot suddenly unable to connect after Docker container port mapping". I believe we all have some understanding. I hope the content shared by the editor will be helpful to you. If you want to know more about the relevant knowledge, please follow the industry information channel.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.