Shulou(Shulou.com)06/03 Report--

This article shows you what the Awk, Sed and Grep commands are in Linux. The content is concise and easy to understand. It will definitely brighten your eyes. I hope you can get something through the detailed introduction of this article.

Detailed explanation of Linux Awk command

Command name

Awk pattern scanning and processing language

Command function

Processing text and data

detailed description

Awk is a programming language used to process text and data under linux/unix. Data can come from standard output (stdin), one or more files, or the output of other commands. It is used on the command line, but more often as a script. Awk has many built-in functions, such as arrays, functions, etc., which is what it has in common with the C language. Flexibility is the biggest advantage of awk.

Grammatical format

Awk [options] 'scripts' var=value filename

Common parameters

-F specifies the delimiter (which can be a string or regular expression)-f reads the awk command-v var=value assignment variable from the script file, passing the external variable to awk

Basic structure of script

Awk 'BEGIN {print "start"} pattern {commands} END {print "end"}' filename

An awk script usually consists of three parts: BEGIN statement + pattern matching + END statement, all of which are optional

How it works:

The first step is to execute the BEGIN statement, the second step is to read a line from the file or standard input, and then the pattern statement is executed. The file is scanned line by line until all the files are read. The third step is to execute the END statement.

Example shows:

Echo "hello" | awk 'BEGIN {print "welcome"} END {print "2017-08-08"}'

Welcome

2017-08-08

Echo-e "hello" | awk 'BEGIN {print "welcome"} {print} END {print "2017-08-08"}'

Welcome

Hello

2017-08-08

# only the current line is printed by default without the print parameter

Echo | awk'{a = "hello"; b = "nihao"; c = "mingongge"; print

Hello nihao mingongge

# when using print separated by commas, printing is delimited by spaces

Echo | awk'{a = "mgg"; b = "mingg"; c = "mingongge"; print a "is" b "or" c;}'

Mgg is mingg or mingongge

The double quotation marks in the print statement of # awk are actually a concatenation.

Variables of Awk

Built-in variable

$0 # current record

# Nth field of the current record

FS # input field delimiter (- F equivalent) default space

RS # enter record separator, default newline character

The number of NF # fields is a column

The number of NR # records, which is the line number, starts from 1 by default

OFS # output field delimiter, default space

ORS # output record separator, default newline character

External variable

[mingongge@ ~] # axi100

[mingongge@ ~] # baked 100

[mingongge@ ~] # echo | awk'{print v1*v2}'v1, v2, roomb

10000

Awk operation and judgment

Arithmetic operator

+-add and subtract

* / & multiply and divide for remainder

^ * exponentiation

+-increase or decrease as a prefix or suffix

[mingongge@ ~] # awk 'BEGIN {a = "b"; print a maxim, a merry maestro, a muri mai, a mai Yu;}'

B 0 1 1

[mingongge@ ~] # awk 'BEGIN {a = "0"; print a maxim, a mai mai, a muri mai, a mai Yu;}'

0 0 1 1

[mingongge@ ~] # awk 'BEGIN {a = "0"; print a-mai-na;}'

0 0 0 1

# like other programming languages, all operands are used as arithmetic operators, operands are automatically converted to numeric values, and all non-numeric values become 0

Assignment operator

= + =-= / =% = ^ = *

Regular operator

~! ~ match regular expression / mismatch regular expression

Logical operator

| | & & Logic or logic is related to

Relational operator

< >

=!

Other operators

$field reference

Space string linker

?: ternary operator

Whether a key value exists in the ln array

Awk regularity

^ line beginning locator

$end of line locator

. Match any single character

* match 0 or more leading characters (including carriage return)

+ match one or more leading characters

? Match 0 or 1 leading character

[] matches any character in the specified character group / ^ [ab]

[^] matches any character that is not in the specified character group

() subexpression

| or |

\ escape character

~,! ~ matching or mismatched conditional statements

X {m} x characters are repeated m times

X {m,} x characters are repeated at least m times

X {m posix n} x characters are repeated at least m times but not more than n times (you need to specify the parameter-m or-- re-interval)

Detailed explanation of Linux Sed command

Command name

Sed, a powerful streaming text editor

detailed description

Sed is a stream editor and a very good tool for text processing. When using more powerful processing with regularities, the currently processed lines are stored in a temporary buffer, called "pattern space", and then use the sed command to process the contents of the buffer, output to the screen after completion, and then process the next line.

Command format

Sed [options] 'command' file (s)

Sed [options]-f scriptfile file (s)

Common parameters

-e # processes the input text file with the specified instruction

-n # cancel the default output (if you use the p command at the same time, only the lines that will change will be printed)

-h # help

-V # displays version information

Common command

A # insert text below the current line

I # insert text above the current line

C # change the selected line to new text

D # Delete, delete selected row

D # Delete the first line of the template block

S # replaces the specified character

H # copy the contents of the module plate to the buffer in memory

H # append the contents of the template block to the buffer in memory

G # gets the contents of the memory buffer and replaces the text in the current mold plate

G # gets the contents of the memory buffer and appends it to the text of the current module block

L # list A list of characters that cannot be printed

N # read the next input line and process the new line with the next command instead of the first command

N # append the next input line to the template plate and embed a new line between the two to change the current line number

P # print matching lines

P # (uppercase) print the first line of the template

Q # quit Sed

B # lable branch to the marked place in the script, or to the end of the script if the branch does not exist

R # file reads lines from file

T # label if branch, starting from the last line, will cause the branch to branch to the labeled command, or to the end of the script, once the condition is met or the TMaget command is met.

T # label error branch, starting from the last line, will cause the branch to branch to the labeled command or to the end of the script in the event of an error or the command t

W # file writes and appends the mold plate to the end of file * *

W # file writes and appends the first line of the template block to the end of file * *

! # indicates that subsequent commands have an effect on all rows that have not been selected * *

= # print the current line number * *

# # extend comments to the next newline character * *

Sed replace command

G # indicates full in-line replacement (global replacement is used with the s command)

P # represents a print line

W # means to write a line to a file

X # represents the text in the interchange mode plate and the text in the buffer zone

Y # means to translate one character into another (but not for regular expressions)

1 # substring matching tag

& # matched string token

Sed regularity

^ # match line start

$# end of matching line

. # matches any character that is not a newline character

* # matches 0 or more characters

[] # matches a character in a specified range

[^] # matches a character that is not within the specified range

(.) # matching substring

& # Save search characters to replace other characters

< #匹配单词的开始 >

# the end of the matching word

X {m} # repeating characters x, m times

X {m,} # repeat character x, at least m times

X {mdirection n} # repeat character x, at least m times, no more than n times

Common examples of Sed

1. Replace operation

Echo "hello world" | sed's / /-/ 1g'

Hello-world

# replace globally with-starting with the first space, except that there is only one space in the text

2. Delete operation

Sed'/ ^ $/ d'filename # Delete blank lines

Sed '2d' filename # Delete the second line

Sed'2 filename # delete the second until all lines have not been tailed

Sed'$d' filename # Delete the last line

Sed'/ ^ test/'d filename # Delete the line starting with test

3. Match and replace

Echo "hello world" | sed 's/w+/ [&] / g'

[hello] [world]

Echo "hello world" | sed 's/w+/ "&" / g'

"hello"world"

# w + matches every word, and & represents the string to which it matches

Echo AAA bbb | sed's / ([Amurz] +) ([Amurz] +) / [2] [1] /'

[bbb] [AAA]

# substring matching replacement

4. Selected range

Sed-n'/ = 0GHAGUAGUBG

# min-encryption = 0

# max-encryption = 256

# all lines in the range from = 0 to max will be printed

5. Sed multipoint editing function (- e)

[root@centos001] # cat-n test

1 this is a test file

2 welcome

3 to

4 here

5 hello WORLD

six

7 linux centos6.8

8 redhat

Sed-e '2pr 6d'-e 's/linux centos6.8/Linux Centos6.8/' test

This is a test file

Linux Centos6.8

Redhat

# if the two commands have the same function, then you need to use the following parameters

Sed-expression='s/linux centos6.8/Linux Centos6.8/'-expression='s/to/TO/' test**

This is a test file

Welcome

TO

Here

Hello WORLD

Linux CenTOs6.8

Redhat

6. Read and write

[root@centos001 ~] # cat test1

Welcom

To

Here

[root@centos001 ~] # sed'/ here/r test1' test

This is a test file

Welcome

To

Here

# welcom

To

Here#

Hello WORLD

Linux centos6.8

Redhat

# read the contents of the test1 file and show the back of all matching here lines

Sed-n'/ centos6.8/w test2' test

[root@centos001 ~] # cat test2

Linux centos6.8

# all lines that match the test file to centos6.8 are written to the test2 file, and the file may not exist.

# if the file exists, it will be redirected rather than appended

7. Append and insert

[root@centos001 ~] # sed'/ ^ l/a2017-08-08' test2

Linux centos6.8

2017-08-08

# append 2017-08-08 to the line that matches the line starting with l

[root@centos001] # sed '1a2017-08-08' test2

Linux centos6.8

2017-08-08

# append 2017-08-08 after the first line

[root@centos001 ~] # sed'/ ^ l/i2017-08-08' test2

2017-08-08

Linux centos6.8

# insert 2017-08-08 before matching lines starting with l

# the above operations will not change the content of the file #

[root@centos001] # sed-I'/ ^ l/i2017-08-08' test2

[root@centos001 ~] # cat test2

2017-08-08

Linux centos6.8

8. Other command examples

[root@centos001] # cat-n test2

1 2017-08-08

2 linux centos6.8

3 08

four

5 test

[root@centos001] # * * sed'/ 08 / {n; s _ root@centos001 _ L _ hand;} 'test2

2017-08-08

Linux centos6.8

08

Test

# if 08 matches, skip to the next line, replace lowercase l with uppercase, and notice that the third line is also matched to

# but the latter conditions are not met, all have not been replaced

[root@centos001 ~] # sed'1 test2 4yqqqqqxxxxxxxxxxxxxxxxxxxxxxxxx

2017-09-09

Linux centos6.9

09

Test

# replace all the numbers 8 in lines 1 to 4 with 9

[root@centos001] # * * sed '1q' test2**

2017-08-08

# exit after printing the first line

9. Print odd or even lines

[root@centos001] # sed-n'ptern 'test2

20170808

08

[root@centos001] # sed-n'nterterp 'test2

Linux centos6.8

Test

[root@centos001 ~] # sed-n '1mm 2p' test2

20170808

08

[root@centos001 ~] # sed-n '2cm 2p' test2

Linux centos6.8

Test

10. Print the next line that matches the character sequence

[root@centos001 ~] # sed-n'/ linux/ {nittp} 'test2

08

[root@centos001 ~] # awk'/ linux/ {getline; print} 'test2

08

Detailed explanation of Linux Grep command

Command name

Grep

Command function

Text search or search tool

detailed description

It can also be used with regular expressions to search for text and print out matching lines. It can also be used to filter and search specific strings, which is very flexible.

Common parameters

-a # Don't ignore binary data

-A # displays not only the line that matches the template style, but also the content after that line

-b # in addition to the line that matches the template style, and displays the content before the line

-B # displays not only the line that matches the style, but also the content before the line

-c # calculates the number of columns that match the template style

-C # displays the contents of the column before and after the column in addition to the column that matches the template style

-d # this parameter must be used when you specify that you are looking for a directory instead of a file, otherwise the grep command will return information and stop the action

-e # specifies the string as the template style for finding the contents of the file

-E # uses the template style as an extended normal representation, which means that you can use extended regular expressions

-f # specifies a template file with one or more template styles and lets grep find the content of the file that meets the criteria of the template in the format of the template style for each column

-F # treats the template style as a list of fixed strings

-G # treats the template style as a common representation

-h # do not indicate the name of the file to which the column belongs until the column that matches the template style is displayed

-H # Mark the name of the file for that column before displaying the column that matches the template style

-I # ignores the difference in character case

-l # lists the file names whose contents match the specified template style

-L # lists the file names whose contents do not match the specified template style

-n # Mark the number of the column before displaying the column that matches the template style

-Q # does not display any information

The effect of this parameter is the same as that of specifying "- d recurse" parameter.

-s # does not display error message

-v # reverse lookup

-V # displays version information

-w # displays only columns that match the whole word

-x # displays only the columns that match the entire column

-y # this parameter has the same effect as "- I"

-o # output only those parts of the file that match

Regular expression

^ # match lines that start with XX

$# matches lines ending in XX

Common examples

1. Look in multiple files:

Grep "file" file_1 file_2 file_3

2. Output all lines except-v option:

Grep-v "file" file_name

3. Tag matching color-- color=auto option:

Grep "file" file_name-color=auto

4. Use the regular expression-E option:

Grep-E "[1-9] +"

Egrep "[1-9] +"

5. Output only the-o option that matches in the file:

Echo this is a test line. | grep-o-E "[amurz] +."

Line.

Echo this is a test line. | | egrep-o "[amurz] +."

Line.

6. Count the number of lines in the file or text that contain matching strings-c option:

Grep-c "text" file_name

two

7. Output the number of lines containing matching strings-n option:

Grep "text"-n file_name

Or

Cat file_name | grep "text"-n

8. Multiple files

Grep "text"-n file_1 file_2

9. Search multiple files and find out which files match the text:

Grep-l "text" file1 file2 file3...

10. Grep recursively searches for files

Recursively search for text in a multi-level directory:

Grep "text". -r-n

11. Ignore the character case in the matching style:

Echo "hello world" | grep-I "HELLO"

Hello

12. Option-e specifies multiple matching styles:

Echo this is a text line | grep-e "is"-e "line"-o

Is

Line

13. You can also use the-f option to match multiple styles and write out the characters you need to match line by line in the style file.

Cat patfile

Aaa

Bbb

Echo aaa bbb ccc ddd eee | grep-f patfile-o

14. Include or exclude specified files from grep search results:

Only recursively search for the character "main ()" in all .php and .html files in the directory

Grep "main ()". -r-- include *. {php,html}

15. Exclude all README files from the search results

Grep "main ()". -r-- exclude "README"

16. Exclude files from the list of filelist files in the search results

Grep "main ()". -r-- exclude-from filelist

Linux command actual combat operation

Introduction of Grep example

Grep "San" testfile

# filter rows with San

Grep'^ J' testfile

# display lines that begin with J

Grep'70 $'testfile

# display lines ending in 70

Grep-v "834" testfile

# display all lines that do not include 834

Grep': 12max 'testfile

# shows the line of 12 /

Grep': 498 Merry 'testfile

# display: line of 498-

Grep'[Amurz] [Amurz] {4}: [[: space:]] [Amurz] 'testfile

# shows a line of one uppercase letter + four lowercase letters + spaces + one uppercase letter

Grep'[a murz] {1,} [[: space:]] [Kk] 'testfile

# display lines including K k

Grep-n'[0-9] {6,} $'testfile

# display 6-digit lines and print line numbers

Grep-I "lincoln" testfile

# display lines with lincoln, case-insensitive

Introduction of Sed example

Sed's _

# Global replacement (Jon replaced with Jonathan)

Sed'1m 3d' testfile

# Delete file lines 1-3

Sed-n'5 10p 'testfile

# print 5-10 lines of the file

Sed'/ Lane/d' testfile

# Delete the line containing lane in the package

Sed-ne'/ [1-9] {5}: 1 [12] / p 'testfile

# print a line with 5 digits +: 11lap12

Sed's / ^ Fred / * * & / 'testfile

# add * before the line that begins with Fred

Sed-e 's/.*Jose.*/JOSE HAS RETIRE/g' testfile

# replace the line containing Jose with JOSE HAS RETIRE

Sed-n'/ ^ Popeye/p' testfile | sed's / [0-9] {1,} / [0-9] {1,} / [0-9] {1,} / 11 *

# print lines that start with Popeye, and then replace the string of numbers in the format "number / number / number" with 11-14-46

# # pattern {n} matching pattern occurs n times.

# # pattern {n,} matching pattern occurs at least n times.

# # the matching pattern of pattern {n ~ m} occurs between n and m times, and n ~ m is any integer in 0-255.

Sed'/ ^ $/ d'testfile

# Delete all blank lines

Sed's Universe G'

# delete to. End line

Sed's / ^ [] [] * / / g'

# remove the beginning of the line space

Sed's Universe. [] [] * / [] / g'

# Delete a full stop followed by two or more spaces and replace it with a space

Sed's / ^ / g'

# Delete the first character

Sed 's/COL/ (... /) / / g'

# Delete the last three characters immediately following COL

Sed's / ^ / g'

# Delete the first / in the path

Sed-n'3 Universe 245700 Universe testfile

# query from the third line to 245700 and print

Sed-n'2 for 26 years testfile

# Lines of printed files (except 2-26)

Introduction of Awk example

Awk-F:'{print $2} 'datafile

# to: separate print the second column

Awk-F:'/ ^ Dan/ {print $2} 'datafile

# to: separate the second column of the line starting with Dan

Awk-F:'/ ^ [CE] / {print $1} 'datafile

# print the first column starting with C or E

Awk-F:'{if (length ($1) = = 4) print $1} 'datafile

# print the first column separated by: 4 characters in length

Awk-F:'/ [916] / {print $1} 'datafile

# match 916 rows to: separate print the first column

Awk-F:'/ ^ Vinh/ {print "a" $5} '2.txt

# display the line starting with Dan and add a before the fifth column

Awk-F:'{print $2 "," $1} 'datafile

# print the second column and the first column and separate them with

Awk-F:'($5 = = 68900) {print $1} '2.txt

# to: separate the rows where the fifth column is 68900 and the first column

Awk-F:'{if (length ($1) = = 11) print $1} '2.txt

# print the first column separated by: 4 characters in length

Awk-F:'$1~/Tommy Savage/ {print $5} '2.txt

Awk-F:'($1 = = "Tommy Savage") {print $5} '2.txt

# print as: delimited the fifth column with the first column Tommy Savage

Ll | awk 'BEGIN {size=0;} {size=size+$5;} END {print "[end] size is", size}'

# count all bytes of files in the directory

Awk 'BEGIN {size=0;} {size=size+$5;} END {print "[end] size is", size/1024/1024, "M"}'

# display all bytes in the directory in M

Awk 'BEGIN {axiathe10 domestic print a}'

twenty

# axi10 is equivalent to a=a+10

Echo | awk 'BEGIN {a = "100testaaa"} a~/test/ {print "ok"}'

# regular match whether a has test characters, set up print ok

Awk 'BEGIN {a = "b"; print asides = "b"? "ok": "err"}'

Ok

Awk 'BEGIN {a = "b"; print asides = "c"? "ok": "err"}'

Err

# ternary operator?:

Awk'/ root/ {print $0} 'passwd

# match all lines containing root

Awk-F:'$5~/root/ {print $0} 'passwd

# use a semicolon as the delimiter to match the line where the fifth field is root

Ifconfig eth0 | awk 'BEGIN {FS= "[[: space:]:] +"} NR==2 {print $4}'

# print IP address

Awk'{print toupper ($0)} 'test.txt

# toupper is an awk built-in function that converts lowercase letters to uppercase

Awk, Sed, Grep commands in Linux are explained in detail, and this is the end of the introduction.

The above is what the Awk and Sed and Grep commands are in Linux. Have you learned any knowledge or skills? If you want to learn more skills or enrich your knowledge reserve, you are welcome to follow the industry information channel.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.