Shulou(Shulou.com)06/02 Report--

As the most attractive advantage, systemd has powerful processing and system logging capabilities. When using other tools, logs are often scattered across the entire system, handled by different daemons and processes, which means that it is difficult to interpret their contents across multiple applications.

In contrast, systemd attempts to provide a centralized management solution to take care of the log information of all kernel and user-level processes. This system can collect and manage log content, which is also known as journal.

The implementation of Journal is attributed to the journald daemon, which handles the information generated by the kernel, initrd, services, and so on. In today's tutorial, we will explore how to use the journalctl tool to access and manipulate data within journal.

General thinking

The underlying driving force of Systemd journal is to centrally manage log information from any source. Since most of the boot processes are handled by the systemd process, there is a reason for log collection and access in a standardized manner. The jornald daemon collects data from all sources and stores it in binary format, making it easy to operate dynamically.

This approach can achieve a variety of benefits. By interacting with data with a single tool, administrators can display log data dynamically. In addition, we can easily view the historical boot data, or combine log entries with other related services to complete the debugging of communication problems.

Storing log data in binary form also means that the data can be displayed in binary output format at any time as needed. For example, you can view logs in standard syslog format for day-to-day management, and hand over entries as JSON objects to graphical services when you need to use graphics services. Since the data is not written to disk in plain text, we do not need to do any format conversion.

You can use systemd journal with existing syslog solutions, or you can use it to replace existing syslog capabilities, depending on your actual needs. Although systemd journal is sufficient to cover most of the management requirements, it also complements existing logging mechanisms. For example, you can set up a centralized syslog server to compile data from multiple servers, or we can use systemd journal to aggregate logs from multiple services into a single system.

Set system time

One of the great benefits of using binary journal is that it can display log records at UTC or local time. By default, systemd displays the results at local time.

For this reason, before we start using journal, we need to make sure that the time zone is set correctly. A timedatectl tool is also available in the Systemd suite to solve such problems.

First, use the list-timezones option to view the available time zone:

Timedatectl list-timezones

The result lists all the time zones available on the system. Then select the project that matches the location of the server and use the set-timezone option to set it:

Sudo timedatectl set-timezone zone

To ensure that our device uses the correct time, you can use the timedatectl command alone or add the status option. The display results are as follows:

Timedatectl statusLocal time: Thu 2015-02-05 14:08:06 ESTUniversal time: Thu 2015-02-05 19:08:06 UTC RTC time: Thu 2015-02-05 19:08:06 Time zone: America/New_York (EST,-0500) NTP enabled: noNTP synchronized: noRTC in local TZ: no DST active: Numbera

The first line should show the correct time.

Basic log view

To view the logs collected by the journald daemon, use the journalctl command.

When used alone, each journal entry in the system is displayed in a single pager for us to browse. The earlier the entry, the higher the arrangement:

Journalctl-- Logs begin at Tue 2015-02-03 21:48:52 UTC, end at Tue 2015-02-03 22:29:38 UTC. -- Feb 03 21:48:52 localhost.localdomain systemd-journal [2443]: Runtime journal is using 6.2m (max allowed 49.Feb 03 21:48:52 localhost.localdomain systemd-journal]: Runtime journal is using 6.2m (max allowed 49.Feb 03 21:48:52 localhost.localdomain systemd-journald [139]: Received SIGTERM from PID 1 (systemd). Feb 03 21:48:52 localhost.localdomain kernel: audit: type=1404 audit: enforcing=1 old_enFeb 03 21:48:52 localhost.localdomain kernel: SELinux: 2048 avtab hash slots 104131 rules.Feb 03 21:48:52 localhost.localdomain kernel: SELinux: 2048 avtab hash slots, 104131 rules.Feb 03 21:48:52 localhost.localdomain kernel: input: ImExPS/2 Generic Explorer Mouse as / devices/platform/Feb 03 21:48:52 localhost.localdomain kernel: SELinux: 8 users, 102 roles, 4976 types, 294 bools, 1 sens,Feb 03 21:48:52 localhost.localdomain kernel: SELinux: 83 classes, 104131 rules. . .

You can flip through the pages, but if the system takes a long time to run, there will be thousands of logs in systemd, which proves the considerable amount of data in the journal database.

Its format is very similar to the standard syslog log. However, it collects data from much richer sources than syslog. It contains logs from previous boot processes, kernels, initrd, and application standard errors and output. All this can be found in journal.

You may also notice that all timestamps are based on local time. Because the local time has been set correctly for the system, the timestamps displayed are also accurate.

If you want to display the timestamp in UTC, you can use the-utc flag:

Journalctl-utc performs journal filtering by time

Browsing through a lot of data certainly helps, but too much information makes it difficult or impossible for us to find what really matters. As a result, journalctl provides critical filtering options.

Displays the logs under the current boot process

The most common of these is the-b flag, which displays all journal entries collected after the most recent reboot.

Journalctl-b

In this way, we can identify and manage information from the current environment.

If you do not use this feature and the number of bootstraps is displayed for more than one day, journalctl inserts a description at the system shutdown:

. . .-- Reboot. . .

This approach can help us effectively distinguish information from different boot sessions.

Past guide record

People usually only need to look at the information in the current boot environment, but sometimes it is necessary to check the past boot records. Journal can save a lot of past guide information, thus allowing journalctl to easily display the relevant content.

Some versions save past boot information by default, while others disable this feature by default. To enable this feature, you can use the following features to create a directory for storing journal information:

Sudo mkdir-p / var/log/journal

Or edit the journal configuration file directly:

Sudo nano / etc/systemd/journald.conf

Set the Storage= option to "persistent" under the [Journal] section to enable persistent logging:

/ etc/systemd/journald.conf. . . [Journal] Storage=persistent

When the ability to save past boot information is enabled, journalctl will provide additional commands to help you operate each boot record as a separate unit. To view the boot information that has been recorded in Journald, use the-list-boots option:

Journalctl-- list-boots-2 caf0524a1d394ce0bdbcff75b94444fe Tue 2015-02-03 21:48:52 UTC-Tue 2015-02-03 22:17:00 UTC-1 13883d180dc0420db0abcb5fa26d6198 Tue 2015-02-03 22:17:03 UTC-Tue 2015-02-03 22:19:08 UTC 0 bed718b17a73415fade0e4e7f4bea609 Tue 2015-02-03 22:19:12 UTC-Tue 2015-02-03 23:01:01 UTC

Each boot here will be displayed as a line. The first column can be used to reference the secondary boot in journalctl. If you need a more accurate reference, you can find the boot ID in the second column. The two times recorded at the end are the start and end times of the current boot.

To display the specific information in these guides, you can use the information provided in the first or second column.

For example, to view the last booted journal record, use the-1 relative pointer with the-b flag:

Journalctl-b-1

Alternatively, you can use the boot ID:

Journalctl-b caf0524a1d394ce0bdbcff75b94444fe time window

It is of course important to view log entries according to the boot environment, but we often need to use a time window that has nothing to do with system boot as a browsing benchmark. This is more common in long-running servers.

You can use the-since and-until options to set the time period, which is responsible for explaining the records before and after a given time, respectively.

Time values can be output in multiple formats. For absolute time values, you can use the following format:

YYYY-MM-DD HH:MM:SS

For example, we can view all entries after 5:15 in the afternoon of January 10, 2015 with the following command:

Journalctl-since "2015-01-10 17:15:00"

If some of the components in the above format are not filled in, the system will populate by default directly. For example, if the date section is not filled in, the current date is displayed directly. If the time section is left empty, 00:00:00 (midnight) is used by default. The second field can also be left blank, and the default value is "00":

Journalctl-since "2015-01-10"-- until "2015-01-11 03:00"

In addition, journal can also understand some relative values and naming abbreviations. For example, you can use expressions such as "yesterday", "today", "tomorrow" or "now". In addition, we can also use "-" or "+" to set the relative value, or use the expression before "ago".

The command to get yesterday's data is as follows:

Journalctl-since yesterday

To obtain reports for the period from 9:00 to one hour ago, use the following command:

Journalctl-since 09:00-until "1 hour ago"

As you can see, the filtering mechanism of the time window is very flexible and easy to use.

Filter by information type

Now we will explore how to implement filtering using the service or component type of interest. Systemd journal also offers a variety of ways for you to choose from.

By unit

The most commonly used method of this kind of filtering is to filter the number by unit. We can do this using the-u option.

For example, to view all logs on the system from the Nginx unit, use the following command:

Journalctl-u nginx.service

Generally speaking, we may need to filter information by unit and time at the same time. For example, check the running status of a service today:

Journalctl-u nginx.service-- since today

We can also take full advantage of journal's ability to view multiple unit information. For example, if our Nginx process accesses a PHP-FPM unit to handle dynamic content, we can merge the two units and get query results sorted by time:

Journalctl-u nginx.service-u php-fpm.service-- since today

This ability is obviously very important for the interaction between different programs and system debugging.

ID by process, user, or group

Because some services contain multiple child processes, if we want to implement the query through the process ID, we can also use the relevant filtering mechanism.

The _ PID field needs to be specified here. For example, if PID is 8088, you can enter:

Journalctl _ PID=8088

Sometimes we may want to display all log entries from a specific user or group, which requires the use of _ UID or _ GID. For example, if your Web server is running under the www-data user, you can find the user ID like this:

Id-u www-data33

Next, we can use the ID to return the filtered journal result:

Journalctl _ UID=33-- since today

Systemd journal has a variety of fields that implement filtering functions. Some of these are from logged processes, while others are used by journald to collect logs from the system for a specific period of time.

The _ PID mentioned earlier belongs to the latter. Journal automatically records and retrieves the process PID for later filtering. You can check all the currently available journal fields:

Man systemd.journal-fields

Let's take a look at the filtering mechanism for these fields. The-F option can be used to display all available values in a specific journal field.

For example, to view the group ID for which systemd journal has entries, use the following command:

Journalctl-F _ GID3299102133818410012487

It will display all the values that the journal has stored in the group ID field and can be used for future filtering requirements.

By component path

We can also provide path locations for filtering.

If the path points to an executable, journalctl displays all entries related to that executable. For example, to find entries related to the bash executable:

Journalctl / usr/bin/bash

In general, if a unit is available for the executable, this approach is more explicit and provides better relevant information (entries related to child processes, etc.). But sometimes it doesn't work.

Display kernel information

Kernel information is usually present in the dmesg output, which can also be retrieved by journal. To display only this type of information, add the-k or-dmesg tag:

Journalctl-k

By default, it displays all kernel information in the current boot environment. You can also use the regular boot selection flag to query the previous boot record. For example, to query information about the previous boot environment five times:

Journalctl-k-b-5 by priority

Another filtering mechanism that administrators may be interested in is information priority. Although it is necessary to view logs in a more detailed way, low-priority logs tend to distract us and lead to confusion when understanding existing information.

You can use the journalctl with-p option to display specific priority information to filter out lower priority information.

For example, only log entries with error level or higher are displayed:

Journalctl-p err-b

Only messages marked as error, critical, warning, or emergency level will be displayed. This implementation of Journal is consistent with standard syslog information at a level. You can use the priority name or its related quantized value. The following numbers are the highest to lowest priority:

0: emerg1: alert2: crit3: err4: warning5: notice6: info7: debug above are the numbers or names that can be used in the-p option. Selecting a priority displays the same level and higher information. Modify journal display content

At this point, the filtering part has been introduced. We can also modify the output in a variety of ways to adjust the display of the journalctl.

Truncate or expand the output result

We can zoom out or expand the output to adjust the way journalctl is displayed. By default, journalctl displays each entry within the pager and accesses its information through the right arrow key. If you want to truncate the output and insert an ellipsis to represent the removed information, you can use the-no-full option:

Journalctl-- no-full. . .Feb 04 20:54:13 journalme sshd [937]: Failed password for root from 83.234.207.60...h3Feb 04 20:54:13 journalme sshd [937]: Connection closed by 83.234.207.60 [preauth] Feb 04 20:54:13 journalme sshd [937]: PAM 2 more authentication failures; logname...ot

You can also ask it to display all the information, regardless of whether it contains non-exportable characters or not. The specific way is to add the-a tag:

Journalctl-a standard output result

By default, journalctl displays the output in pager for easy reference. If you want to use text manipulation tools to process the data, you may need to use a standard format. In this case, we need to use the-no-pager option:

Journalclt-no-pager

In this way, the relevant results can be redirected to files on disk or processing tools as needed.

Output format

If you need to work with journal entries, you may need to use an easier-to-use format to simplify data parsing. Fortunately, journal can be displayed in a variety of formats by adding the-o option plus a format description.

For example, we can output journal entries to JSON format:

Journalctl-b-u nginx-o json {"_ _ CURSOR": "swords 13a21661cf4948289c63075db6c25c00 | 116f1mitb1b58db8fd9046ab9f847ddb82a2fa2d X=e307daadb4858635 "," _ _ REALTIME_TIMESTAMP ":" 1422990364739502 "," _ _ MONOTONIC_TIMESTAMP ":" 27200938 "," _ BOOT_ID ":" 81b58db8fd9046ab9f847ddb82a2fa2d "," PRIORITY ":" 6 "," _ UID ":" 0 "," _ GID ":" 0 "," _ CAP_EFFECTIVE ":" 3fffffffff "," _ MACHINE_ID ":" 752737531a9d1a9c1e3cb52a4ab967ee "," _ HOSTNAME ":" desktop " "SYSLOG_FACILITY": "3", "CODE_FILE": "src/core/unit.c", "CODE_LINE": "1402", "CODE_FUNCTION": "unit_status_log_starting_stopping_reloading", "SYSLOG_IDENTIFIER": "systemd", "MESSAGE_ID": "7d4958e842da4a758f6c1cdc7b36dcc5", "_ TRANSPORT": "journal", "_ PID": "1", "_ COMM": "systemd" "_ EXE": "/ usr/lib/systemd/systemd", "_ CMDLINE": "/ usr/lib/systemd/systemd", "_ SYSTEMD_CGROUP": "/", "UNIT": "nginx.service", "MESSAGE": "Starting A high performance web server and a reverse proxy server...", "_ SOURCE_REALTIME_TIMESTAMP": "1422990364737973"}. . .

This approach is very important for tool parsing. You can also use the json-pretty format to better handle data structures:

Journalctl-b-u nginx-o json-pretty {"_ _ CURSOR": "swords 13a21661cf4948289c63075db6c25c00 | 116f1mitb1b58db8fd9046ab9f847ddb82a2fa2d X=e307daadb4858635 "," _ _ REALTIME_TIMESTAMP ":" 1422990364739502 "," _ _ MONOTONIC_TIMESTAMP ":" 27200938 "," _ BOOT_ID ":" 81b58db8fd9046ab9f847ddb82a2fa2d "," PRIORITY ":" 6 "," _ UID ":" 0 "," _ GID ":" 0 "," _ CAP_EFFECTIVE ":" 3fffffffff "," _ MACHINE_ID ":" 752737531a9d1a9c1e3cb52a4ab967ee "," _ HOSTNAME ":" desktop "," SYSLOG_FACILITY ":" 3 " "CODE_FILE": "src/core/unit.c", "CODE_LINE": "1402", "CODE_FUNCTION": "unit_status_log_starting_stopping_reloading", "SYSLOG_IDENTIFIER": "systemd", "MESSAGE_ID": "7d4958e842da4a758f6c1cdc7b36dcc5", "_ TRANSPORT": "journal", "_ PID": "1", "_ COMM": "systemd", "_ EXE": "/ usr/lib/systemd/systemd" "_ CMDLINE": "/ usr/lib/systemd/systemd", "_ SYSTEMD_CGROUP": "/", "UNIT": "nginx.service", "MESSAGE": "Starting A high performance web server and a reverse proxy server...", "_ SOURCE_REALTIME_TIMESTAMP": "1422990364737973"}. . .

The following are the various formats available for display:

Cat: only the information field itself is displayed.

Export: a binary format suitable for transfer or backup.

Json: standard JSON with one entry per line.

Json-pretty: JSON format, suitable for human reading habits.

Json-sse: JSON format, packaged to be compatible with server-sent events.

Short: the default syslog class output format.

Short-iso: default format, emphasizing the ISO 8601 wall clock timestamp.

Short-monotonic: default format, which provides a normal timestamp.

Short-precise: default format, which provides microsecond precision.

Verbose: displays all available journal fields for this entry, including fields that are usually hidden internally.

These options allow you to display journal entries in the format that best suits your needs.

Activity process monitoring

The Journalctl command can also help administrators monitor activities or recent processes in a manner similar to tail. This feature is built into journalctl and allows access without the need for other tools.

Show recent logs

To display a specific number of records, you can use the-n option as tail-n.

By default, it displays the last ten records:

Journalctl-n

You can specify the number of entries to view after-n:

Journalctl-n 20 tracking log

To actively track the logs currently being written, you can use the-f flag. The method is also tail-f:

Journalctl-fJournal maintenance

Storing so much data is certainly stressful, so we also need to know how to clean up some of the stale logs to free up storage space.

Understand existing disk usage

You can use the-disk-usage flag to view the current disk usage of journal:

Journalctl-- disk-usageJournals take up 8.0m on disk. Delete old logs

If you plan to clean up the journal records, you can use two different ways (for systemd 218 and later).

"if you use the-vacuum-size option, you can rigidly specify the overall volume of the log, which means that it continues to delete old records until the capacity meets the requirements:"

Sudo journalctl-vacuum-size=1G

Another way is to use the-vacuum-time option. Any entries earlier than this point in time will be deleted.

For example, entries after last year can only be retained:

Sudo journalctl-vacuum-time=1years qualifies the Journal extension

You can configure your own servers to limit the maximum capacity that journal can occupy. To do this, we need to edit the / etc/systemd/journald.conf file.

The following entries can be used to limit the rate at which the journal volume expands:

SystemMaxUse=: specifies the maximum persistent storage capacity that journal can use.

SystemKeepFree=: specifies the remaining space that journal needs to reserve when adding new entries.

SystemMaxFileSize=: controls the size of a single journal file, and those who meet the requirements can be converted to persistent storage.

RuntimeMaxUse=: specifies the maximum available disk capacity in volatile storage (within the / run file system).

RuntimeKeepFree=: specifies the amount of space reserved for other applications (within the / run file system) when writing data to volatile storage.

RuntimeMaxFileSize=: specifies the maximum volatile storage capacity that a single journal file can occupy (within the / run file system).

By setting the above values, you can control how journald consumes and retains server space.

Summary

At this point, systemd journal has introduced the collection and management mechanism of system and application data. Its excellent flexibility comes from automatically recording a wide range of metadata into centralized logs. In addition, the journalctl command significantly simplifies the use of journal, allowing more administrators to use it to complete analysis and related debugging work for different application components.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.