Shulou(Shulou.com)06/03 Report--

Demand:

Back to the top.

In the work, there is a server program running in Linux (debian8) environment, which is implemented in Python language. There are different priority logs to be recorded in the code. When developing, the logging module of python is used to output to a file. The sample code is as follows:

1 import logging, os 2 3 logger = None 4 def get_logger (): 5 global logger 6 if not logger: 7 logger = logging.getLogger ('ServerLog') 8 logger.setLevel (logging.INFO) 9 filehandler = logging.FileHandler (os.environ [' HOME'] +'/ Server.log' Encoding='utf8') 10 filehandler.setFormatter (logging.Formatter ("% (asctime) s -% (levelname) s -% (message) s") 11 logger.addHandler (filehandler) 12 return logger13 14 def some_func (): 15 get_logger (). Info ("call some_func") 16 17 if _ name__ ='_ main__':18 some_func ()

Running the above code will generate a server.log file under the home directory.

Later, the data analysis department said that they hoped to get some of the logs in real time, and they had a server dedicated to processing logs, so how to send the logs to them? The author has no relevant experience before, and colleagues in the data analysis department say that they all turn to operation and maintenance personnel for help. The solution given by the OPS colleague is simple: the product writes the log to syslog, and then they are responsible for forwarding the log with certain keywords to the data analysis department. Under the guidance of the OPS colleague, the code is changed like this:

1 import logging 2 import logging.handlers 3 4 logger = None 5 def get_logger (): 6 global logger 7 if not logger: 8 logger = logging.getLogger ('ServerLog') 9 logger.setLevel (logging.INFO) 10 11 sys_handler = logging.handlers.SysLogHandler (' / dev/log') Facility=logging.handlers.SysLogHandler.LOG_LOCAL0) 12 syslog_tag = 'ServerLog'13 sys_handler.setFormatter (logging.Formatter (syslog_tag + ":% (asctime) s -% (name) s -% (levelname) s -% (message) s")) 14 15 logger.addHandler (sys_handler) 16 return logger17 18 def some_func (): 19 get_logger (). Info ("call some_func") 20 21 if _ name__ = ='_ _ main__':22 some_func ()

The above code modifies the output form of the log, and the intuitive feeling is from the file server.log to / dev/log, but / dev/log corresponds to SysLogHandler, not FileHandler, so it is definitely not an ordinary file. At this point, I have two questions: first, I did not output the log to the Server.log file in the home directory, but the program generated such a file when the program was running; second, how to send the log to the server of the data analysis department.

If you don't understand, ask:

Q: how to generate the Server.log file under the new code, and how to forward the log content to the server of the data analysis department?

A: this is / etc/init.d/rsyslog, the daemon outputs logs to different files, including network files, that is, other servers according to the configuration file of / etc/rsyslog.conf. Just look at the / etc/rsyslog.conf configuration.

Q:OK, what does the python code have to do with rsyslog to output the file to / dev/log?

A:python 's sysloghandler sends logs to rsyslog, and they communicate with each other using unix domain socket, which can be seen in the source code of the logging module.

Unix domain socket:

Back to the top.

According to the meaning of the above dialogue, the python program first sends the log to the rsyslog program, and then rsyslog processes the received log data, so first look at the logging code:

The class SysLogHandler is in logging.handlers.py, and the core code is as follows:

1 def _ init__ (self, address= ('localhost', SYSLOG_UDP_PORT), 2 facility=LOG_USER, socktype=socket.SOCK_DGRAM): 3 "" 4 Initialize a handler. 5 6 If address is specified as a string, a UNIX socket is used. To log to a 7 local syslogd, "SysLogHandler (address=" / dev/log ")" can be used. 8 If facility is not specified, LOG_USER is used. 9 "" 10 logging.Handler.__init__ (self) 11 12 self.address = address13 self.facility = facility14 self.socktype = socktype15 16 if isinstance (address Basestring): 17 self.unixsocket = 118 self._connect_unixsocket (address) 19 else:20 self.unixsocket = 021 self.socket = socket.socket (socket.AF_INET, socktype) 22 if socktype = = socket.SOCK_STREAM:23 self.socket.connect (address) 24 self.formatter = None25 26 def _ connect_unixsocket (self Address): 27 self.socket = socket.socket (socket.AF_UNIX, socket.SOCK_DGRAM) 28 # syslog may require either DGRAM or STREAM sockets29 try:30 self.socket.connect (address) 31 except socket.error:32 self.socket.close () 33 self.socket = socket.socket (socket.AF_UNIX, socket.SOCK_STREAM) 34 self.socket.connect (address)

It is clear in _ _ init__.doc that if address is a string (the default is a tuple), then a unix socket (unix domain socket) will be created. If address is "/ dev/log" (as in our previous python code), then output to the native syslogd program. In addition, in line 27, the first parameter family of self.socket = socket.socket (socket.AF_UNIX, socket.SOCK_DGRAM) socket.socket is AF_UNIX, rather than the AF_INET (IPV4) or AF_INET6 (IPV6) that we often use. So what is unix domain socket?

Unix domain socket is a form of interprocess communication (IPC:inter-process communication), and others include pipes, named pipes, message queues, shared memory, socket, and so on. What is the difference between unix domain socket and the commonly used socket (narrow internet socket), that is, unix domain socket can only communicate between processes on the same host, and ordinary socket can also communicate on the same host through 'localhost', so what are the advantages of unix domain socket?

First: no need to go through the network protocol stack

Second, there is no need to pack and unpack, calculate checksum, maintain serial number and reply, etc.

Therefore, the advantage is good performance, one word, fast.

Let's take a simple server-client example to see how and how to use unix domain socket:

Server: uds_server.py

1 ADDR ='/ tmp/uds_tmp' 2 3 import socket, os 4 5 def main (): 6 try: 7 sock = socket.socket (socket.AF_UNIX, socket.SOCK_STREAM) 8 if os.path.exists (ADDR): 9 os.unlink (ADDR) 10 sock.bind (ADDR) 11 sock.listen (5) 12 while True:13 connection Address = sock.accept () 14 print "Data:% s"% connection.recv (1024) 15 connection.send ("hello uds client") 16 connection.close () 17 finally:18 sock.close () 19 20 if _ name__ ='_ main__':21 main ()

Client: uds_client.py

1 ADDR ='/ tmp/uds_tmp' 2 3 import socket 4 5 def main (): 6 sock = socket.socket (socket.AF_UNIX, socket.SOCK_STREAM) 7 sock.connect (ADDR) 8 sock.send ('hello unix domain socket server') 9 print' client recieve', sock.recv (1024) 10 sock.close () 11 12 if _ _ name__ ='_ main__':13 main ()

First, run the server python uds_server.py, and the file is generated in the / tmp directory. Use ls to view the details as follows:

As you can see, the file type (the first field) is s, which represents the socket file. (PS: if the process communicates through a command pipeline, also using intermediate files, the file type displayed by ls is p)

Running the client-side python uds_client.py, there is a corresponding output on both the client side and the server side, and the method of using it is not much different from that of the ordinary socket.

Log forwarding process:

Back to the top.

After understanding the concept of unix domain socket, the following is relatively simple, the first is the / dev/log file, we use ls to view the information of this file

You can see that this file is a symbolic link file, and the real file is / run/systemd/journal/dev-log, so let's look at this file again.

Ok, is a socket file, compound expectation. According to the previous example of unix domain socket, rsyslog should also listen on this file. Let's take a look.

Lsof fd can list all the processes that use this file (the concept of files under linux is relatively broad). In fact, we see only two unknown processes, systemd and systemd-j. So look directly at the unix domain socket used by rsyslog.

Well, you can see that the socket domain socket used by rsyslogd is / run/systemd/journal/syslog, not / run/systemd/journal/dev-log, and the two files are in the same directory, so let's see which other processes use / run/systemd/journal/syslog.

Both so,systemd and rsyslogd use this file, which feels like an application process (e.g. The above python program) sends the log to systemd via / run/systemd/journal/dev-log (the real file behind / dev/log), and then systemd sends the log to rsyslogd via / run/systemd/journal/syslog. Isn't that right? google found the article understand-logging-in-linux, which is indeed such a process:

Systemd has a single monolithic log management program, systemd-journald. This runs as a service managed by systemd.

It reads / dev/kmsg for kernel log data.

It reads / dev/log (a symbolic link to / run/systemd/journal/dev-log) for application log data from the GNU C library's syslog () function.

It listens on the AF_LOCAL stream socket at / run/systemd/journal/stdout for log data coming from systemd-managed services.

It listens on the AF_LOCAL datagram socket at / run/systemd/journal/socket for log data coming from programs that speak the systemd-specific journal protocol (i.e. Sd_journal_sendv () et al.).

It mixes these all together.

It writes to a set of system-wide and per-user journal files, in / run/log/journal/ or / var/log/journal/.

If it can connect (as a client) to an AF_LOCAL datagram socket at / run/systemd/journal/syslogit writes journal data there, if forwarding to syslog is configured.

Ok, so far, we know how the log of the application is forwarded to rsyslog, so the secret of how rsyslog handles the received log is / etc/rsyslog.conf. Before opening this configuration file, let's take a look at the brief description of the official website of rsyslog:

RSYSLOG is the rocket-fast system for log processing.

So R means rocket-fast! Fast as a rocket! The official website claims that it can handle millions of logs per second. Rsyslogd is the default syslogd program in some linux environments (at least on the author's machine), d means daemon, background process. When the system starts, the process is started to process the logs (including the logs of the operating system itself and user processes). Open the modified / etc/rsyslog.conf, and then it's time to witness a miracle.

It turns out that every move is under surveillance. This file is provided by the system, and it is obviously not wise to make changes directly on this file. For example, in the red part of the image above, you can add your own configuration file under the rysyslog.d folder to customize the log filtering rules. Then take a look at the new tmp.conf under the rsyslog.d folder.

1$ FileOwner USERNAME 2$ FileGroup USERNAME 3$ FileCreateMode 0644 4$ DirCreateMode 0755 5$ Umask 0022 6$ template serverLog, "/ home/USERNAME/Server.log" 7$ template LogFormat, "% msg%\ n" 8 if $syslogfacility-text = = 'local0' and $syslogtag contains' ServerLog' then -? serverLog;LogFormat 9 10 # if $syslogfacility-text = = 'local0' and $syslogtag contains' ServerLog' then @ someip:port11 & stop

Let's review the corresponding application code:

1 import logging 2 import logging.handlers 3 4 logger = None 5 def get_logger (): 6 global logger 7 if not logger: 8 logger = logging.getLogger ('ServerLog') 9 logger.setLevel (logging.INFO) 10 11 sys_handler = logging.handlers.SysLogHandler (' / dev/log') Facility=logging.handlers.SysLogHandler.LOG_LOCAL0) 12 syslog_tag = 'ServerLog'13 sys_handler.setFormatter (logging.Formatter (syslog_tag + ":% (asctime) s -% (name) s -% (levelname) s -% (message) s")) 14 15 logger.addHandler (sys_handler) 16 return logger17 18 def some_func (): 19 get_logger (). Info ("call some_func") 20 21 if _ name__ = ='_ _ main__':22 some_func ()

Note: the configuration file needs to match the application code, for example, line 11 facility=logging.handlers.SysLogHandler.LOG_LOCAL0 corresponds to $syslogfacility-text = = 'local0' in the configuration; line 12 syslog_tag =' ServerLog' corresponds to the configuration file $syslogtag contains' ServerLog'. For the setting of syslogtag in the python code, refer to this question and answer on stackoverflow.

When we modify the configuration, we need to restart rsyslogd with the command / etc/init.d/rsyslog restart, and then run the previous python file after restart.

Send to the remote server:

The above tmp.conf file comments out line 10, which is used to send qualified logs to other specified machines, and IP:Port is used to specify the remote rsyslogd program that accepts the logs. By default, rsyslogd listens on port 514. Suppose I need to send syslog to 10.240.10.10 on the local area network, and line 10 will be changed to this:

If $syslogfacility-text = = 'local0' and $syslogtag contains' ServerLog' then @ 10.240.10.10

Then 10.240.10.10 mainly enables remote listening of rsyslogd and specifies the output rules of remote logs, for example:

This configuration allows rsyslogd to listen on port 514 using both UDP and TCP protocols and output non-native logs to a file corresponding to the remote hostname. Note that all the above changes need to restart rsyslogd to take effect.

Summary:

Back to the top.

The flow of the log from the application to the final log file (or remote server) is as follows:

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.