Shulou(Shulou.com)06/01 Report--

Problem description:

The production environment found that CPU is running at full capacity, and the number of connections to MongoDB has never exceeded 1000.

Solution:

1. Check the log of mongodb and report the following error:

Wed Nov 21 15:26:09 [initandlisten] pthread_create failed: errno:11 Resource temporarily unavailable Wed Nov 21 15:26:09 [initandlisten] can't create new thread, closing connection

2. Tested on the same centos5 machine, it was found that there was no problem connecting 2000 connections at all.

3. Find the problem on google with the keyword "mongod.conf can't create new thread, closing connection"

4. Find the problem. The original centos6 is different from the previous centos5. There is a default configuration file that limits the user's nproc: / etc/security/limits.d/90-nproc.conf. By default, the nproc of ordinary users is set to 1024, and mongodb happens to be run by non-root users using mongod, so the number of connections can not go up all the time.

5. Change / etc/security/limits.d/90-nproc.conf, change 1024 to 20480, and the problem is solved.

[root@test ~] # cat / etc/security/limits.d/90-nproc.conf # Default limit for number of user's processes to prevent # accidental fork bombs. # See rhbz # 432903 for reasoning.* soft nproc 20480

Limit the number of open file handles and the maximum number of user processes:

When deploying applications under Linux, you will sometimes encounter the problem of Socket/File: Can't open so many files. This value will also affect the maximum number of concurrency of the server. In fact, Linux is limited by file handle, and the Linux is not very high by default, which is usually 1024. It is easy to reach this number for production servers. The following is how to correct this system default value by correcting the configuration.

View method

We can use ulimit-a to see all the limit values

[root@test] # ulimit-a core file size (blocks,-c) 0 data seg size (kbytes,-d) unlimited scheduling priority (- e) 0 file size (blocks,-f) unlimited pending signals (- I) 256469 max locked memory (kbytes,-l) 64 max memory size (kbytes) -m) unlimited open files (- n) 64000 pipe size (512 bytes,-p) 8 POSIX message queues (bytes,-Q) 819200 real-time priority (- r) 0 stack size (kbytes,-s) 10240 cpu time (seconds) -t) unlimited max user processes (- u) 65536 virtual memory (kbytes,-v) unlimited file locks (- x) unlimited

Where "open files (- n)" is the Linux operating system limit on the number of file handles opened by a process, and the default is 1024.

(it also includes the number of SOCKET opened, which can affect the number of concurrent connections to the database.)

The right thing to do is to modify / etc/security/limits.conf

There are very detailed notes, such as

Hadoop soft nofile 32768

Hadoop hard nofile 65536

Hadoop soft nproc 32768

Hadoop hard nproc 65536

You can change the file handle limit to soft 32768 and hard 65536. The first part of the configuration file refers to domain, which is set to an asterisk to represent the global situation. In addition, you can also make different restrictions for different users.

Note: the hard limit in this is the actual limit, while the soft limit, which is the warnning limit, will only make warning;. In fact, the ulimit command itself has a soft and hard setting, adding-H means hard, and adding-S means soft.

The soft limit is displayed by default, and if you don't add it when you run the ulimit command to modify it, the two parameters change together.

Changes to RHE6 and later nproc are in / etc/security/limits.d/90-nproc.conf

How to modify the connection limit:

Temporary modification (number of processes that can be opened by users to change under the current shell):

# ulimit-u xxx

Permanent modification, the insurance practice is to modify / etc/security/limits.d/90-nproc.conf and / etc/security/limits.conf at the same time as follows:

Limits_conf = / etc/security/limits.conf:

* soft nproc S1

* hard nproc H2

Nproc_conf = / etc/security/limits.d/90-nproc.conf:

* soft nproc S2

* hard nproc H3

S 1 # 2 # 2 # 2 # 3 must be a specific and meaningful number. At this point, the value shown by ulimit-u is = min (h _ 2 ~ H _ 3).

Therefore, it is common to set up s1=s2=h2=h3, such as adding both limits_conf and nproc_conf:

* soft nproc 65536

* hard nproc 65536

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.