Analysis of Common cases of Logstash Grammar (1)
Abstract
A brief description of the common plug-ins of logstash and simple use cases
One: basic operation
It is recommended to use supervisor to manage the components in ELK to facilitate the same management.
There are instructions on installing https://www.aolens.cn/?p=809
Provide a common configuration:
[program:logstash] command=/opt/logstash/bin/logstash-f / opt/logstash/conf/index.confnumprocs=1; Open several processes dirrectory=/opt/logstashuser=root; user stdout_logfile=/opt/logstash/logs/logstash.logstdout_logfile_maxbytes=1MB; each log size stdout_logfile_backups=10; keep 10 log files stderr_logfile=/opt/logstash/logs/logstash_err.logstderr_logfile_maxbytes=1MBstderr_logfile_backups=10
Operation parameters:
Start the logstash service (common supervisor daemons)
. / bin/logstash-f / etc/logstash/conf.d/*-t # check whether the configuration file is ok
. / bin/logstash-f conf.d/nginx.conf-w 5-l / var/log/logstash/
Second: configuration syntax
1, area: (section)
Logstash uses {} to define areas. Multiple plug-in regions can be defined in the area, and key-value pairs can be defined in the plug-in area.
Eg:
Input {# input data file {path= ["/ var/log/messages", "/ var/log/*.log"] type= "system" start_position= "beginning"} filter {# data filtering processing if [type] = "system" {grok {match= ["message" % {COMBINEDAPACHELOG}]}} output {# data processing output stdout {codec=rubydebug}}
2. Data type:
String-- normal string
Name = > "Hello world"
Name = >'It\'s a beautiful day'
Array-- arrays can be single or multiple string values.
Path = > ["/ var/log/messages", "/ var/log/*.log"]
Path = > "/ data/mysql/mysql.log"
Hash-- key-value pairs, note that multiple key-value pairs are separated by spaces instead of commas.
Match = > {
"field1" = > "value1"
"field2" = > "value2"
...}
Codec-- is used to represent data encoding. Used for input and output segments. It is convenient for data processing.
Codec = > "json"
Number-- must be a valid numeric, floating point, or integer.
Port = > 33
The boolean-- Boolean value must be TRUE or false.
Ssl_enable = > true
Bytes-- specifies the byte unit. The default is byte.
My_bytes = > "1113" # 1113 bytes
My_bytes = > "10MiB" # 10485760 bytes
My_bytes = > "100kib" # 102400 bytes Binary (Ki,Mi,Gi,Ti,Pi,Ei,Zi,Yi) Unit 1024
My_bytes = > "180 mb" # 180000000 bytes SI based on 1000
Password-- a separate string.
My_password = > "password"
Path-- represents a valid operating system path.
My_path = > "/ tmp/logstash"
3: field reference
To use the value of the field in the Logstash configuration, simply write the name of the field in square brackets []. As long as the value entered by input can be referenced
Eg:
[geoip] [location] [- 1]
4, condition judgment
Operators supported by expressions
= = (equal to),! = (unequal), (greater than), = (large, etc.)
= ~ (match regular),! ~ (mismatch regular)
In (included), not in (not included)
And (and), or (or), nand (and not), xor (non-OR)
() (compound expression),! () (take the result of the inverse compound expression)
Three: common plug-ins
1, plug-in management
. / bin/plugin-h
Install
Uninstall
Update
List
Eg:
Bin/plugin install logstash-output-webhdfs
Bin/plugin update logstash-input-tcp
2, common plug-ins input,output,filter,codec
2.1 Input plug-in
Stdin: standard input, commonly used for testing
Input {stdin {type = "string" tags = ["add"] codec= "plain"}}
File: reads files from the file system, similar to tail-f under linux. Most commonly used
Input {file {path = ["/ var/log/*.log", "/ var/log/message"] # logstash only supports the absolute path to the file type = "system" # type record file type, and the defined variables are global variables. Other plug-ins can call start_position = "beginning"}}.
Redis: read from the redis server, using both redis channel and redis list.
Input {redis {data_type= "list" key= "logstash-nginx" host= "192.168.1.250" port=6379 db=1 threads=5}} writes source data to redisoutput {redis {host= "192.168.1.250" port=6379 db=1 data_type= " List "key=" logstash-nginx "}}
TCP/UDP: input
# nc127.0.0.18888 / var/log/nginx/access.json # can be json file direct value # echo' {"name": "liuziping", "age": "18"}'| nc127.0.0.18888 input {tcp {port = 8888 # definition tcp listening port codec= "json" # specifies that the incoming data is in json format KBH structure is convenient to analyze mode = "server"}}.
Syslog: listens to Syslog information on port 514 and parses it into RFC3164 format.
Input {syslog {port = "514"}}
Beats: sends events through Filebeat.
2.2:Output plug-in
Stdout: standard output
Output {stdout {codec = rubydebug workers = 2}}
File: saving to a file
Output {file {path = "/ path/to/% {+ yyyy/MM/dd/HH} /% {host} .log.gz" message_format = "% {message}" gzip = true}}
Elasticsearch: save it into elasticsearch, which is also the most important
Output {elasticsearch {hosts = > ["192.168.0.2 hosts 9200"] # or cluster = > "ClusterName" index = > "logstash-% {type} -% {+ YYYY.MM.dd}" # index name, unified format, convenient for kibana import, can talk about the same type of log Import all the type values in the type=input here document_type = > "nginx" workers = > 1 # start a process flush_size = > 20000 # save 20000 pieces of data and send it to ES at once Default idle_flush_time = > 10 # if there are not enough 20000 messages within 10 seconds, send them to ES once. Default is 1s template_overwrite = > true}}.
Redis: saved to redis is explained in the input plug-in
TCP: output TCP
Output {tcp {host = "192.168.0.2" port = 8888 codec = json_lines}}
Email: sending mail
Exec: call command execution
