Shulou(Shulou.com)06/03 Report--

In the "hello World" example, we have seen and introduced the basic syntax of the running process and configuration of Logstash. Remember one rule: the Logstash configuration must have an input and an output. During the demonstration, if input is not specified, the input/stdin that we have demonstrated in "hello world" will be used by default. Similarly, the output that is not specified is output/stdout.

If you have any questions, please check the document: http://udn.yyuap.com/doc/logstash-best-practice-cn/input/index.html. The following is a specific explanation of the input plug-in:

(1) Standard input. Type and tags are special fields in the logstash event. Type is used to mark the event type-we must know in advance what type the event belongs to. Tags is added or deleted by specific plug-ins in the process of data processing.

[root@localhost test] # vim stdin.conf input {stdin {add_field = > {"key" = > "value"} codec = > "plain" tags = > ["add"] type = > "std-lqb"}} output {stdout {codec = > rubydebug}} [root@localhost logstash] # / usr/local/logstash/bin/logstash-f Test/stdin.conf Settings: Default pipeline workers: 1Logstash startup completedhello world {"message" = > "hello world" @ version "= >" 1 "," @ timestamp "= >" 2017-05-24T08:11:45.852Z "," type "= >" std-lqb "," key "= >" value "," tags "= > [[0]" add "]," host "= >" localhost.localdomain "} abclqb {" message "= >" abclqb "," @ version "= >" 1 " "@ timestamp" = > "2017-05-24T08:13:21.192Z", "type" = > "std-lqb", "key" = > "value", "tags" = > [[0] "add"], "host" = > "localhost.localdomain"} # modify stdin Add tags column [root@localhost test] # vim stdin.conf input {stdin {add_field = > {"key" = > "value22222222222222222222222222222222222222222222"} codec = > "plain" tags = > ["add", "xxyy" "abc"] type = > "std-lqb"} output {stdout {codec = > rubydebug}} [root@localhost logstash] # / usr/local/logstash/bin/logstash-f test/stdin.conf Settings: Default pipeline workers: 1Logstash startup completedhello world {"message" = > "hello world", "@ version" = > "1", "@ timestamp" = > "2017-05-24T09:07:43.228Z" "type" = > "std-lqb", "key" = > "value22222222222222222222222222222222222222222222", "tags" = > [[0] "add", [1] "xxyy", [2] "abc"] "host" = > "localhost.localdomain"} # judge by tags: [root@localhost test] # vim stdin_2.conf input {stdin {add_field = > {"key11" = > "value22"} codec= > "plain" tags= > ["add" "xxyy"] type= > "std"} output {if "tttt" in [tags] {stdout {codec= > rubydebug {}} else if "add" in [tags] {stdout {codec= > json} [root@localhost logstash] # / usr/local/logstash/bin/logstash-f test/stdin_2.conf Settings: Default pipeline workers: 1Logstash startup completedyyxxx {"message": "yyxxx", "@ version": "1", "@ timestamp": "2017-05-24T09:32:25.840Z", "type": "std" "key11": "value22", "tags": ["add", "xxyy"], "host": "localhost.localdomain"} {"message": "," @ version ":" 1 "," @ timestamp ":" 2017-05-24T09:32:32.480Z "," type ":" std "," key11 ":" value22 "," tags ": [" add "," xxyy "]," host ":" localhost.localdomain "} xxyy {" message ":" xxyy "," @ version ":" 1 " "@ timestamp": "2017-05-24T09:32:42.249Z", "type": "std", "key11": "value22", "tags": ["add", "xxyy"], "host": "localhost.localdomain"}

(2)。 Read the file. Logstash uses a Ruby Gem library called FileWatch to listen for file changes. The library supports the glob expansion file path and records a database file called .uploedb to track the current read location of the listening log file. So don't worry that logstash will miss your data.

[root@localhost test] # cat log.confinput {file {path = > "/ usr/local/nginx/logs/access.log" type= > "system" start_position = > "beginning"}} output {stdout {codec = > rubydebug}} [root@localhost logstash] # / usr/local/logstash/bin/logstash-f test/log.conf Settings: Default Pipeline workers: 1Logstash startup completed {"message" = > "192.168.181.231-[24/May/2017:15:04:29 + 0800]\" GET / HTTP/1.1\ "502 537\" -\ "\" Mozilla/5.0 (Windows NT 6.1) Win64 X64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.36\ "\" -\ "," @ version "= >" 1 "," @ timestamp "= >" 2017-05-24T09:39:16.600Z "," path "= >" / usr/local/nginx/logs/access.log "," host "= >" localhost.localdomain " "type" = > "system"} {"message" = > "192.168.181.231-[24/May/2017:15:04:32 + 0800]\" GET / HTTP/1.1\ "502 537\" -\ "\" Mozilla/5.0 (Windows NT Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.36\ "\" -\ "," @ version "= >" 1 "," @ timestamp "= >" 2017-05-24T09:39:16.614Z "," path "= >" / usr/local/nginx/logs/access.log "," host "= >" localhost.localdomain "," type "= >" system "}

Explanation:

There are some useful configuration items that can be used to specify the behavior of the FileWatch library:

Discover_interval

How often does logstash check to see if there are new files under the monitored path. The default value is 15 seconds.

Exclude

Files that do not want to be monitored can be excluded. Glob deployment is supported here as well as path.

Sincedb_path

If you do not want to use the default $HOME/.sincedb (on the Windows platform at C:\ Windows\ System32\ config\ systemprofile\ .uploedb), you can define the sincedb file to another location through this configuration.

Sincedb_write_interval

How often does logstash write sincedb files? the default is 15 seconds.

Stat_interval

How often does logstash check the status of monitored files (for updates). The default is 1 second.

Start_position

Where logstash starts reading file data, the default is the end location, which means that the logstash process runs in a form similar to tail-F. If you are importing the original data, change this setting to "beginning", the logstash process will read from scratch, a bit like cat, but the last line will not stop, but will continue to become tail-F.

Be careful

Usually if you want to import the original data into Elasticsearch, you also need the filter/date plug-in to change the default "@ timestamp" field value. I will learn about this later.

FileWatch only supports absolute paths to files and does not automatically recurse directories. So if necessary, please write down the specific files in an array.

LogStash::Inputs::File simply initializes a FileWatch object during the registration phase of the process. So it cannot support path = > "/ path/to/% {+ yyyy/MM/dd/hh} .log" like fluentd. To achieve the same goal, you can only write path = > "/ path/to/*.log".

Start_position works only if the file has never been monitored. If there is already an inode record for this file in the sincedb file, logstash will still read the data from the recorded pos. So you need to delete the sincedb file each time you repeat the test.

Because there is no inode concept on the windows platform, it is not very reliable for some versions of Logstash to listen to files on the windows platform. On the windows platform, it is recommended to consider using nxlog as the collection end

(3) .TCP input. In the future, you may use a Redis server or other message queuing system as the role of logstash broker. However, Logstash actually has its own TCP/UDP plug-in, which can be used for temporary tasks, especially in the test environment.

[root@localhost test] # cat tcp.conf input {tcp {port = > 8888 mode= > "server" ssl_enable = > false}} output {stdout {codec = > rubydebug}} [root@localhost logstash] # / usr/local/logstash/bin/logstash-f test/tcp.conf Settings: Default pipeline workers: 1Logstash startup completed {"message" = > "GET / jenkins/ HTTP/1.1\ r" "@ version" = > "1", "@ timestamp" = > "2017-05-24T10:09:53.980Z", "host" = > "192.168.181.231", "port" = > 59426} {"message" = > "Host: 192.168.180.9 port 8888\ r", "@ version" = > "1", "@ timestamp" = > "2017-05-24T10:09:54.175Z" "host" = > "192.168.181.231", "port" > 59426} {"message" = > "Connection: keep-alive\ r", "@ version" = > "1", "@ timestamp" = > "2017-05-24T10:09:54.180Z", "host" = > "192.168.181.231", "port" = > 59426}

Note: first close the application on port 8888, and then turn it on, the following log will be output.

(4) coding plug-in Codec:

Codec is a new concept introduced by logstash since version 1.3.0 (Codec comes from the acronym of the word Coder/decoder). Until now, logstash only supported plain text input and then processed it as a filter. But now we can handle different types of data during the input period, all because of the codec setting. We have already used Codec coding in the first "Hello world" column, and rubydebug is a Codec, although it is generally only used in stdout plug-ins as a tool for configuration testing or debugging.

(4.1) using JSON coding, directly enter the predefined JSON data, so that you can omit the filter/grok configuration!

Take nginx as an example to configure the instance. The specific steps are as follows:

A, edit the configuration nginx configuration file nginx.conf. Comment out the original configuration file, change it to json format, and then restart your nginx

[root@localhost test] # vim / usr/local/nginx/conf/nginx.confuser ftp;worker_processes 2 / workerboards rushing events Nofile 65535 events {use epoll; worker_connections 1024;} http {include mime.types; default_type application/octet-stream; include proxy.conf # log_format main'$remote_addr-$remote_user [$time_local] "$request" #'$status $body_bytes_sent "$http_referer" #'"$http_user_agent"$http_x_forwarded_for"' # log_format json'{"@ timestamp": "$time_iso8601",''"@ version": "1",''"host": "$server_addr",''"client": "$remote_addr",''"size": $body_bytes_sent,''"responsetime": $request_time '' "domain": "$host", "url": "$uri",''"status": "$status"}' Access_log logs/nginx_access.log json; # access_log logs/access.log main; # Note: there are no double quotes at the ends of the $request_time and $body_bytes_sent variables, which should be numeric types in JSON.

B, edit your logstash configuration file json.conf

[root@localhost test] # vim json.conf input {file {path = > "/ usr/local/nginx/logs/nginx_access.log" type = > "nginx" start_position = > "beginning" add_field = > {"key" = > "value"} codec = > "json"} output {stdout {codec = > rubydebug {}

Cperfine logstash load startup test:

[root@localhost logstash] # / usr/local/logstash/bin/logstash-f test/json.conf Settings: Default pipeline workers: 1Logstash startup completed {"@ timestamp" = > "2017-05-25T03:26:19.000Z", "@ version" = > "1", "host" = > "192.168.180.9", "client" = > "192.168.181.231", "size" = > 8250, "responsetime" = > 0.157 "domain" = > "192.168.180.9", "url" = > "/", "status" = > "200"," path "= >" / usr/local/nginx/logs/nginx_access.log "," type "= >" nginx "," key "= >" value "} {" @ timestamp "= >" 2017-05-25T03:26:19.000Z " "@ version" = > "1", "host" = > "192.168.180.9", "client" = > "192.168.181.231", "size" = > 450, "responsetime" = > 0.017, "domain" = > "192.168.180.9", "url" = > "/ sc.do", "status" = > "200" "path" = > "/ usr/local/nginx/logs/nginx_access.log", "type" = > "nginx", "key" = > "value"} {"@ timestamp" = > "2017-05-25T03:26:19.000Z", "@ version" = > "1", "host" = > "192.168.180.9", "client" = > "192.168.181.231" "size" = > 16, "responsetime" = > 0.083, "domain" = > "192.168.180.9", "url" = > "/ logger/catch.do", "status" = > "200"," path "= >" / usr/local/nginx/logs/nginx_access.log "," type "= >" nginx " "key" = > "value"} {"@ timestamp" = > "2017-05-25T03:26:19.000Z", "@ version" = > "1", "host" = > "192.168.180.9", "client" = > "192.168.181.231", "size" = > 41153, "responsetime" = > 0.362, "domain" = > "192.168.180.9" "url" = > "/ getPageData.do", "status" = > "200"," path "= >" / usr/local/nginx/logs/nginx_access.log "," type "= >" nginx "," key "= >" value "} {" @ timestamp "= >" 2017-05-25T03:26:20.000Z "," @ version "= >" 1 " "host" = > "192.168.180.9", "client" = > "192.168.181.231", "size" = > 51042, "responsetime" = > 0.565, "domain" = > "192.168.180.9", "url" = > "/ getPageData.do", "status" = > "200" "path" = > "/ usr/local/nginx/logs/nginx_access.log", "type" = > "nginx", "key" = > "value"

Merge multiple rows of data (Multiline): sometimes, the application debug log contains a lot of content, printing out many lines for an event. This kind of log is usually difficult to analyze through command line parsing. Logstash is preparing the codec/multiline plug-in for this. The multiline plug-in can also be used for other similar stackable information, such as linux's kernel log.

When you start the logstash and configuration file, you will be asked to enter a series of characters, and the current input will not be terminated until you enter [, as follows:

[

Root@localhost test] # vim multiline.confinput {stdin {codec = > multiline {pattern = > "^\ [" negate = > true what = > "previous"}} output {stdout {codec = > rubydebug {} [root@localhost logstash] # / usr/local/logstash/ Bin/logstash-f test/multiline.conf Settings: Default pipeline workers: 1Logstash startup completedhellohello worldhow are you abc2345 [{"@ timestamp" = > "2017-05-25T03:44:35.604Z" "message" = > "[\ nhello\ nhello world\ nhow are you\ nabc2345", "@ version" = > "1", "tags" = > [[0] "multiline"], "host" = > "localhost.localdomain"}

In short, the principle of this plug-in is simple: add the data of the current row to the end of the previous row until the new current row matches "[" regular. You can also use grok expressions for this rule.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.