Shulou(Shulou.com)05/31 Report--

This article mainly introduces the use of skills in Linux Shell, has a certain reference value, interested friends can refer to, I hope you can learn a lot after reading this article, the following let the editor take you to understand it.

1. Split a string with a variable extension

People often use cut or even awk just to subtract part of a string through a pattern or using a delimiter.

In addition, many people use ${VARIABLE:start_position:length} for substring bash operations, which is very fast.

But bash provides a powerful way to handle text strings using #, #,%, and%-it's called bash variable extension.

Using this syntax, you can reduce the need for a pattern without executing external commands, so it will work very quickly.

The following example shows how to use cut or variable extension to get the third column (shell) from a string, with values separated by a colon «username:homedir:shell »(we use the *: mask and # # commands, which means: cut all characters to the left until the last colon is found):

$STRING= "username:homedir:shell" $echo "$STRING" | cut-d ":"-f 3 shell $echo "${STRING##*:}" shell

The second option does not start the child process (cut) and does not use pipes at all, which makes it work faster. Also, if you use the bash subsystem on a Windows where the pipe barely moves, the speed difference can be significant.

Let's take a look at the example on Ubuntu: loop through our commands 1000 times

$cat test.sh #! / usr/bin/env bash STRING= "Name:Date:Shell" echo "using cut" time for An in {1.. 1000} do cut-d ":"-f 3 > / dev/null / dev/null done secret.key; chmod 600 secret.key

Let's encrypt the string using aes-256-cbc:

$echo "string_to_encrypt" | openssl enc-pass file:secret.key-e-aes-256-cbc-a U2FsdGVkX194R0GmFKCL/krYCugS655yLhf8aQyKNcUnBs30AE5lHN5MXPjjSFML

You can put this encrypted string into any configuration file stored in git or anywhere else-without secret.key, it's almost impossible to decrypt it.

To decrypt and execute the same command, simply replace-e with-d:

$echo 'U2FsdGVkX194R0GmFKCLash krYCugS655yLhf8aQyKNcUnBs30AE5lHN5MXPjSFML' | openssl enc-pass file:secret.key-d-aes-256-cbc-a string_to_encrypt5. Grep command

Everyone should know the grep command. And be friendly to regular expressions. In general, you can write something like this:

Tail-f application.log | grep-I error

Even like this:

Tail-f application.log | grep-I-P "(error | warning | failure)"

But don't forget that grep has a lot of great choices. For example,-v, which restores your search and displays all messages except the "info" message:

Tail-f application.log | grep-v-I "info"

Other contents:

The option-P is very useful because by default, grep uses the rather outdated «basic regular expression: », and-P enables PCRE without even knowing the grouping.

-I ignore case.

-- line-buffered parses lines immediately instead of waiting for the standard 4k buffer to be reached (useful for tail-f | grep).

If you know a lot about regular expressions, you can use-- only-matching /-o to really achieve the excellent function of cutting text. You only need to compare the following two commands to extract the shell of myuser:

$grep myuser / etc/passwd | cut-d ":"-f 7$ grep-Po "^ myuser (:. *) {5}:\ K.*" / etc/passwd

The second command looks more compiled, but it only runs grep instead of grep and cut, so it takes less time to execute.

6. How to reduce the size of log files

In * nix, if you delete log files that the application is currently using, you can not only delete all logs, but also prevent the application from writing new logs before restarting.

Because the file descriptor opens the iNode structure instead of the file name, the application will continue to write the file descriptor to a file without a directory entry, and the file will be automatically deleted by the file system after the application stops. (your application can open and close log files every time you want to write something to avoid this problem, but this can affect performance.)

Therefore, how to clear the log file without deleting it:

Echo "" > application.log

Or we can use the truncate command:

Truncate-size=1M application.log

Mentioned that the truncate command will delete the rest of the file, so you will lose the latest log events. Another example of how to store the last 1000 rows:

Echo "$(tail-n 1000 application.log)" > application.log

PS in Linux, we have a standard service rotatelog. You can add logs to automatic truncation / rotation, or you can do so using an existing log library (such as log4j in Java).

7. Watch

In some cases, you are waiting for the event to end. For example, you are waiting to finish (repeating ls dozens of times) when another user logs in to shell (you execute who commands continuously), or when someone should use scp or ftp to copy files to your computer.

In this case, you can use the

Watch

By default, it is executed every 2 seconds, and the screen is cleared in advance until you press Ctrl + C. You can configure the frequency of execution.

This feature is useful when you want to view real-time logs.

8.bash sequence

Creating a scope is very useful. For example, instead of like this:

For srv in 1 2 3 4 5; do echo "server$ {srv}"; done server1 server2 server3 server4 server5

You can write the following:

For srv in server {1... 5}; do echo "$srv"; done server1 server2 server3 server4 server5

You can also use the seq command to generate formatting ranges. For example, we can use seq to create values that automatically adjust the twitch based on the width (00, 01 instead of 0, 1):

For srv in $(seq-w 5 10); do echo "server$ {srv}"; done server05 server06 server07 server08 server09 server10

Another example of replacing with a command-renaming a file. To get a file name without an extension, we use the "basename" command:

For file in * .txt; do name=$ (basename "$file" .txt); mv $name {.txt, .lst}; done

Even shorter than'%':

For file in * .txt; do mv ${file%.txt} {.txt, .lst}; done

PS actually, for renaming files, you can try to use the rename tool with many options.

Another example-let's create a structure for the new Java project:

Mkdir-p project/src/ {main,test} / {java,resources}

Result

9.tail, multiple files, multiple users...

I have mentioned multitail to read files and view multiple real-time logs. However, this feature is not available by default, and permissions to install some content are not always available.

But a standard tail can also do this:

Tail-f / var/logs/*.log

It also lets you remember information about users who view the application log using the 'tail-f' alias.

Multiple users can view log files at the same time using "tail-f". The conversation of some of them is not very accurate. For some reason, they may leave 'tail-f' in the background and forget it.

If you restart the application, some running "tail-f" processes are monitoring log files that do not exist, and the process may be suspended for days or even months.

Usually this is not a big problem, but it is not neat.

If you use an alias to view the log, you can use the-- pid option to modify the alias:

Alias TFapplog='tail-f-pid=$ (cat / opt/app/tmp/app.pid) / opt/app/logs/app.log'

In this case, all tails are automatically terminated when the target application is restarted.

10. Create a file of the specified size

Dd is one of the most popular tools for using block and binary data. For example, creating a 1 MB file and filling in zeros would be:

Dd if=/dev/zero of=out.txt bs=1M count=10

However, I recommend using fallocate:

Fallocate-l 10m file.txt

On file systems that support the allocation function (xfs,ext4,Btrfs...), fallocate executes immediately, unlike the dd tool. In addition, allocation refers to the actual allocation of blocks, not the creation of alternate files.

11. Xargs

Many people know the popular xargs command. But not everyone uses the following two options, so scripts can be greatly improved.

First-you can get a very long list of arguments that may exceed the length of the command line (~ 4 kb by default).

However, you can restrict execution with the-n option, so xargs will run the command multiple times, sending a specified number of parameters at a time:

$# lets print 5 arguments and send them to echo with xargs: $echo 1 2 3 4 5 | xargs echo 1 2 3 4 5 $# now let's repeat, but limit argument processing by 3 per execution $echo 1 2 3 4 5 | xargs-n 3 echo 1 2 3 4 5

Come on, it may take a lot of time to deal with a long list because it runs in a single thread. However, if we have several cores, we can tell xargs to run in parallel:

Echo 1 2 3 4 5 6 7 8 9 10 | xargs-n 2-P 3 echo

In the above example, we told xargs that list; of the three threads would accept and process two parameters for each execution. If you don't know how many cores you have, use "nproc" to optimize:

Echo12 3 4 5 6 7 8 9 10 | xargs-n 2-P $(nproc) echo12.sleep? While? Read!

Sometimes you have to wait a few seconds. Or wait for the user to enter the following:

Read-p "Press any key to continue"-n 1

But you only need to add a timeout option to read the command, and the script pauses the specified number of seconds, but in the case of interactive execution, the user can easily skip waiting.

Read-p "Press any key to continue (auto continue in 30 seconds)"-t 30-n 1

Therefore, you just need to forget the sleep command.

I suspect that not all the tricks look interesting, but in my opinion, there are twelve numbers that can be filled in.

Thank you for reading this article carefully. I hope the article "what skills to use in Linux Shell" shared by the editor will be helpful to you. At the same time, I also hope that you will support us and pay attention to the industry information channel. More related knowledge is waiting for you to learn!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.