Shulou(Shulou.com)06/01 Report--

What this article shares with you is about the function and usage of Awk in Linux. The editor thinks it is very practical, so I share it with you to learn. I hope you can get something after reading this article.

We know the three swordsmen of Linux, they are grep, sed, awk, and today we are going to share the more powerful awk.

Sed can achieve non-interactive string substitution, and grep can achieve effective filtering functions. Compared with the two, awk is a powerful text analysis tool, which is particularly powerful when analyzing data and generating reports.

The powerful function of awk is incomparable to the general Linux command. I won't tell you that awk is also a programming language, lest it scare you. We just need to think of it as the next powerful text analysis tool for Linux.

Scene

Before learning how to use it, let's take a look at what awk can do:

1. The given text content can be output and displayed according to our desired format and printed into a report.

two。 Analyze and process system logs, quickly analyze and mine the data we care about, and generate statistical information

3. Conveniently used for statistical data, such as the number of visits to the website, the number of IP visits, etc.

4. Through the combination of various tools, quickly summarize and analyze the operation information of the system, so that you know the operation of the system like the back of your hand.

5. Powerful scripting language expression ability, supporting loops, conditions, arrays and other syntax to help you analyze more complex data

.

Of course, awk can not only do these things, when you integrate its usage, you can do efficient data analysis and statistics as you like.

However, we need to know that awk is not omnipotent, it is good at dealing with formatted text, such as logs, csv format data, etc.

Principle

Let's start with a brief understanding of the basic working principles of awk. Through the following pictures and texts, I hope you can understand how awk works.

Awk basic command format

Use the following figure to illustrate in detail how awk works.

First, execute the command in {} identified by the keyword BEGIN

After completing the command in BEGIN curly braces, start executing the body command

Read the data row by row, and read by default that\ nthe segmented content is a record, which is actually the concept of a row.

Dividing a record into fields according to the specified delimiter is actually the concept of a column

Loop the commands in the body block, execute the body for each row read, and finally complete the body execution

Finally, execute the END command, which usually outputs the final result in END

Awk is input-driven, and the body command will be executed as many lines as there are input lines.

In the following example study, we should always keep in mind that the Record is the row, the Field is the column, the BEGIN is the preprocessing phase, the body is the real work stage of the awk, and the END is the final processing phase.

Actual combat-getting started

Starting with the following, let's go straight to the actual combat. To facilitate the example, I will first save the following information to file.txt

All right, let's start with the simplest and most commonly used example of awk, outputting columns 1, 4, and 8:

Inside the curly braces is the awk statement, which can only be contained in single quotation marks, where $1. N represents the column, and $0 represents the entire line.

Let's take a look at the more practical function formatting output of awk. Like the printf output in C language, I personally like this format rather than the streaming in C++.

% s represents a string placeholder,-4 indicates a column width of 4 and is left-aligned, and we can list more complex formats as needed. I won't give a detailed example here.

Actual combat-advanced

(I) filtering records

Some data may not be what you want and can be filtered as needed.

The above filter condition is that the rows in column 3 root and 6 column 10 will be output.

Awk supports a variety of comparison symbols!, >, =, and redirect each line directly to the month-named file. Of course, you can also output the specified columns to the file.

(3) if statement

For complex condition judgment, you can use awk's if statement. Awk is powerful because it is a script interpreter and has the programming ability of a general scripting language. The following example splits files with slightly more complex conditions

Note that the if statement is in curly braces.

(4) Statistics

Count the sum of the space occupied by all *. C, *. H files in the current directory

Column 5 represents the file size, which is calculated in the sum variable for each row read, and the sum, which is the sum of all the file sizes, is printed out in the final END phase.

Let's take a look at another example. Count the memory consumed by each user's process. Note that the column that takes the value is the RSS column.

Arrays and for loops are used here, and it is worth mentioning that the array of awk can be understood as a dictionary or Map,key can be numeric and string, which is a common data type.

(5) string

Demonstrate awk's support for string manipulation through the following simple example

Awk has built-in support for a series of string functions, length calculates the string length, and the toupper function converts strings to uppercase.

Actual combat-skill

To understand how awk works as a whole, let's look at a comprehensive example, assuming that there is a student transcript:

Because this sample program is a little complicated, it is not easy to read on the command line. In addition, we would like to introduce another way to execute awk through this case. Our awk script is as follows:

The result of executing awk is as follows

We can write complex awk statements to the script file cal.awk and then specify execution from the script file with the-f option.

In the BEGIN phase, we initialize the relevant variables and print the format of the header

In the body phase, we read each row of data and calculate the total scores of the subject and the student.

In the END phase, we first printed the format of the end of the table, printed the total score, and calculated the average

This simple example fully reflects the working mechanism and principle of awk. I hope this example can help you really understand how awk works.

Summary and induction

Through the above examples, we have learned how awk works. Let's summarize the following concepts and common knowledge points.

(1) built-in variables

1. Each line of content record, called record, English name Record

two。 Each column in each line separated by a delimiter is called a field, with the English name Field

With these concepts clear, let's summarize several important built-in variables:

NR: indicates the current number of rows

NF: indicates the current number of columns

RS: line delimiter. Newline is the default.

FS: column delimiter, default is space and tab

OFS: output column delimiter, which is used to split fields when printing. Default is space.

ORS: output line delimiter, which is used to split records when printing. Default is newline character.

(2) output format

Awk provides the printf function to format the output function, which is basically consistent with the C syntax.

Basic usage

Commonly used formatting:

% d decimal signed integer

% u unsigned decimal integer

% f floating point number

% s string

% c single character

Floating point numbers in the form of% e exponent

% x% X unsigned integer in hexadecimal

0 unsigned integer in octal

% g automatically selects the appropriate representation

\ nLine feed character

\ t Tab character

(3) programming statements

Awk is not only a Linux command line tool, it is actually a scripting language that supports all the control structures of the programming language. It supports:

Conditional statement

Loop statement

Array

Function

(4) commonly used functions

Awk has a lot of useful functions built in, and it also supports custom functions, allowing you to write your own functions to extend the built-in functions.

Here we simply list some of the more commonly used string functions:

Index (s, t) returns the position of the substring t in s

Length (s) returns the length of the string s

Split (s, a, sep) splits the string and stores the segmented fields in the array a

Substr (s, p, n) returns the substring according to the parameter

Tolower (s) converts a string to lowercase

Toupper (s) converts a string to uppercase

Here is only a simple summary of some commonly used string functions, specific usage, but also need you to refer to the previous example program, examples, applied to practical problems.

These are the functions and usage of Awk in Linux. The editor believes that there are some knowledge points that we may see or use in our daily work. I hope you can learn more from this article. For more details, please follow the industry information channel.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.