Linux queries large log table information by time period and divides it by date
# check the nohup.out log table 2019-11-18 20:00:00-2019-11-18 22:00:00 (provided there is such a format in the log), and generate exchange_update.log
Cat nohup.out | sed-n'/ 2019-11-18 20 exchange_update.log 0012-11-18 20 exchange_update.log
2020-02-31 18:21
Log query
Scenario: the access.log log is a large file that contains information about all dates. However, when you need to query the log at a certain time on a certain day, both grep and sed are time-consuming, so you need to save the log of this date before querying.
Acess.log log format:
[12/Feb/2020:15:40:46 + 0800]-- 101.20.18.75-- 57197 "POST / api?apiCode=meu_mksm:info HTTP/1.1" 0.011 "-" 0.011 "- -" gdn.cn 154.121.10.206 9443 localhost "-" https://gdn.cn:9443/view/xtgl/mksm_view.html?apiCode=meu_mksm:info&code=M9502"
Access.log format settings for nginx.conf:
'[$time_local]-- $remote_addr-- $remote_port "$request_method $request_uri $server_protocol" $status "$upstream_response_time"-- "$request_time"-- "$host $server_addr $server_port $server_name"-"$http_referer"'
First query by date, and save the results
Sed-n'/ 12\ / Feb\ / 2020 Feb 150.Plus / 2020Feb / 2020 Feb 160.hand acess.log > 202021215.log # plus\ is for decompilation to prevent parsers from using / symbols as compiled symbols.
Sed:
-n,-- quiet,-- silent
Suppress automatic printing of pattern space # suppresses automatic printing mode space
Query the 502 response code in 202021215.log
Cat 202021215.log | grep "^ 502 $" # find a segment with a separate word of 502
Result: NULL # because there is no such thing in the log
Grep:
^ means that the matching string is at the beginning of the line
$indicates that the matching string is at the end of the line
Precise query:
Sed-n'/ 2020-02-12 15-15-00-12-12-20-02-12 16-12-00-02-12 16-12-12-02-12 16-12-02-12 16-12-20-02-12 16-12-20-02-12
Running result: NULL
Pit points:
If it is not in the start time log, then the query result is empty, for example, the start time is not in the log at 15:00:00 on 2020-02-12.
If it is not in the end time log, the result of the query is the full log from the start time to the last
Fuzzy query:
Sed-n'/ 2020-02-12 15 purl 0.gamma ris.log # start with the first words 2020-02-12 15: 0 in the log until the end of the first 2020-02-12 16:0
Log segmentation
This section is reproduced to: https://blog.csdn.net/feinifi/article/details/78455563
Scenario: access.log logs for all dates are concentrated here, so it is not convenient to find problems, and the log needs to be separated by date.
Let the log be saved as a single file by date every day, script
#! / bin/bash
YESTERDAY=$ (date-d "yesterday" + "% Y-%m-%d")
LOGPATH=/usr/local/openresty/nginx/logs/
PID=$ {LOGPATH} nginx.pid
Mv ${LOGPATH} access.log ${LOGPATH} access-$ {YESTERDAY} .log
Mv ${LOGPATH} error.log ${LOGPATH} error-$ {YESTERDAY} .log
Kill-USR1 `cat ${PID} `
Change the script to an executable file, and put the script into crontab as a scheduled task startup script.
Chmod + x nginx_logback.sh
Crontab-e
#! / bin/bash
0 * / bin/bash / root/nginx_logback.sh
