Shulou(Shulou.com)06/01 Report--

This article will explain in detail how to use BPF to enhance the security audit of SSH sessions in linux. The editor finds it very practical, so I share it with you as a reference. I hope you can get something after reading this article.

Teleport is a ssh server developed in Golang language to replace sshd. It supports remote access through SSH or HTTPS, supports cluster, Web login, SSH history replay for operation sharing and security audit, builds and develops based on Golang ssh packages, and is fully compatible with OpenSSH.

Recently, ssh enhanced callback logging has been added in Teleport 4.2 to receive unstructured ssh sessions and output them through structured event streams. This feature uses a new technology integrated into the Linux kernel, eBPF (BPF for short). It is used to improve the deficiency of Teleport audit function. In this article, Bugs will show you how to do it in Teleport.

Background

One of the advantages of Teleport is that it can record the entire ssh session. This feature captures all the content displayed on the user's screen. You can then use the record playback to restore all operations for audit purposes. These records are easy to understand and provide information about what the user does and displays during the session

However, like other similar tools (fortress machines), Teleport has its drawbacks, and users can use some ways to bypass session records. These include:

Obfuscation command:

For example, confuse commands by encryption or other means. For example, the following statement confuses the command to be executed with base64 coding.

Echo Y3VybCBodHRwOi8vd3LmV4YW1wbGUuY29tCgwings = | base64-d | sh

In this way, the actual execution command (curl example.com) is not recorded directly.

Shell script:

If the user cannot capture the commands running in the script by uploading and executing the script, only the output of the script can be captured.

Terminal control:

Ssh terminals can support a variety of controls, such as controls that users can disable terminal echo, which are usually used in some applications. For example, the sudo command when they prompt the user for a password. This is also not possible to capture them in the ssh record.

In addition, session records can be difficult to capture and monitor to some extent due to the unstructured nature of TTY streams.

Technical realization

To address these shortcomings of security auditing, Teleport needs a way to convert unstructured ssh sessions into structured event streams. This is the enhanced call recording feature that this article is going to talk about. The specific features are shown as follows:

Technology selection

Through the comparison of a variety of methods, including content from temporary methods such as regular expression pattern matching to methods such as parsing the original SSH session, and exploring various API and systems provided by Linux, such as Audit,fanotify and BPF, the two criteria to consider when comparing technologies are accuracy and performance issues:

Reduce false positives, preferably to zero. Monitoring system (security audit), the first consideration is the accuracy of the problem, if there are many alarms, the attention to the alarm will be reduced. May cause serious problems to be ignored.

Reduce any performance impact caused by monitoring, preferably zero. The monitoring system should avoid affecting the performance of the system as far as possible.

In many methods, the problem of false positives is very serious. The byte streams that make up the SSH session cannot be accurately parsed and interpreted, and alarm fatigue cannot be prevented.

There are also some methods that will have performance problems. Like Linux Audit.

By looking at file system access, there are some alternatives. The two most prominent are inotify and fanotify. Inotify is good as an initial alternative, but it lacks recursive directory monitoring. Fanotify is more promising, and there are two problems.

Brendan. Greg mentioned in the BPF performance toolbook that under heavy loads, opensnoop accounts for 67 per cent of CPU compared to fanotify,fanotify, while opensnoop uses only 1 per cent.

The Linux kernel was recently merged into a patch to improve the performance of fanotify and make it more suitable for monitoring the entire file system, so that its use in Teleport fits more closely. However, patches need to be merged in Linux 5.1, and there is currently no out-of-the-box mainstream distribution support.

With these considerations in mind, Teleport finally chose a solution based on the BPF program chain, which reduced false positives and had minimal impact on performance.

BPF introduction

BPF, originally a packet filter of Unix, Berkeley bag filter (Berkeley Packet Filter). It was later transplanted to Linux. In 2013, Alexei Starovoitov improved and improved BPF. The new version was named eBPF (extended BPF), abbreviated as BPF, and the old BPF was called cBPF (classic BPF). EBPF adds new features such as mapping and tail calls, and rewrites the JIT compiler. The new language is closer to the native machine language than cBPF. BPF allows user-space programs to hook up and issue events from certain locations in the kernel in a secure and efficient manner.

Security means that BPF programs do not fall into an infinite loop and cause the system to paralyze. BPF programs are unlikely to crash the entire operating system like kernel modules.

BPF programs are also effective, and if they are not used quickly enough, events are discarded rather than dragging down the performance of the entire system.

At present, there are a large number of system tools in Linux refactoring based on BPF (such as iptables), Brendan. Greg's book "BPF High performance tools" (blog) introduces a large number of tools for you to learn.

Teleport docking BPF

Three BPF programs are used in Teleport: the execution of the execsnoop capture program, the files opened by the opensnoop capture program, and the TCP connection established by the tcpconnect capture program.

To better understand the functions of these BPF programs, you can look at the output man ls of execsnoop at run time.

It seems that simply running the man binaries is the execution of many other programs behind the scenes.

Teleport embeds these programs in its binaries, and when enhanced session logging is enabled, it generates and runs them.

These programs themselves are excellent tools for debugging and tracking, giving you an idea of what is going on on the system, not just one user.

Cgroups (especially cgroupv2) is also used to associate program execution with a specific SSH session. When Teleport starts a SSH session, it first restarts itself and places it in cgroup. Not only does this allow this process, but you can also use a unique ID to track all future processes started by Teleport. The BPF programs that Teleport runs have been updated, and you can also issue the cgroup ID of the program that executes them. This allows you to associate events with specific SSH sessions and identities.

Limitation

Currently, Teleport has not completed the enhanced session recording. There are still some gaps that will be addressed in future versions.

It is worth pointing out that through session logging, Teleport can capture the byte stream that makes up the session because of its privileged location (the byte stream must flow through the Teleport). Crucially, the integrity of the session record does not depend on any information reported by the host itself. However, the enhanced auditing functionality depends on the host accurately reporting information to the Teleport. If the integrity of the host is compromised, the integrity of the enhanced audit is also affected. In addition, Teleport monitors only some of the most critical system calls in the system, not all.

Currently, enhanced session logging is most suitable for non-root users, and enhanced session logging can be disabled in a number of ways for users who have access to root.

Practice

In the absence of background, you can use the following script for enhanced session recording.

Start Ubuntu 19.04 or RHEL/CentOS 8 VM first, and then run the script above. The script installs the kernel header file and bcc-tools, which is a prerequisite for running enhanced session logging. In addition, the script installs jq, a command json parsing tool (Bug has been introduced in previous articles, which can be referenced), which helps to visualize the structured event flow.

After entering the terminal according to the instructions, the following similar information is seen on the screen:

Thus it can be seen that curl programs are executed by users in two ways. The first is the execution of the program itself. The second is the behavior of the program, which can also be seen when the curl makes a network request. You can try to perform other operations, such as confusing commands, or script execution, and you should check the results in the log.

Request

The minimum requirement for Teleport enhanced session logging requires starting the Linux kernel 4.18 supported by BPF. It is currently available in several distributions, including Ubuntu 19.04 Debian 10 and RHEL/CentOS 8.

You also need to install kernel header files and bcc-tools. For the operating systems listed above, install them in the package manager of the distribution, for example, just run Centos 8

Yum install-y kernel-headers bcc-tools

Or execute under Ubuntu

Apt install-y linux-headers-$ (uname-r) bpfcc-tools

If you have not already packaged bcc-tools, you must build them from the source code.

To enable enhanced session logging in Teleport, simply enable it in the file configuration, as follows:

Ssh_service: enhanced_recording: enabled: yes article on "how to use BPF in linux to enhance the security audit of SSH sessions" ends here. I hope the above content can be of some help to you, so that you can learn more knowledge. If you think the article is good, please share it for more people to see.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.