Shulou(Shulou.com)06/03 Report--

overview

logstash

Logstash is a pipeline with real-time data transmission capability, responsible for transmitting data information from the input end of the pipeline to the output end of the pipeline; at the same time, this pipeline also allows you to add filters in the middle according to your own needs. Logstash provides many powerful filters to meet your various application scenarios.

Nginx

Nginx(engine x) is a high-performance HTTP and reverse proxy server, and an IMAP/POP3/SMTP server. It distributes its source code under a BSD-like license and is known for its stability, rich feature set, sample configuration files, and low system resource consumption. Nginx is a lightweight Web server/reverse proxy server and email (IMAP/POP3) proxy server distributed under a BSD-like protocol. It is characterized by less memory and strong concurrency capability. In fact, nginx's concurrency capability does perform well in web servers of the same type.

1.logstash installation (jdk early installation 1.8)

rpm -ivh logstash-6.6.2.rpm

2. Upload database driver jar package

wget https://dev.mysql.com/get/Downloads/Connector-J/mysql-connector-java-5.1.48.tar.gz

mkdir -p /usr/share/logstash/vendor/jar/jdbccd /usr/share/logstash/vendor/jar/jdbc[root@localhost soft]# cd /usr/share/logstash/vendor/jar/jdbc[root@localhost jdbc]# lltotal 984-rw-r--r--. 1 logstash logstash 1006959 Jul 11 19:43 mysql-connector-java-5.1.48-bin.jar[root@localhost jdbc]#

3. Database Create Library and Authorize Users

create database nginxlog;use nginxlog;CREATE TABLE `consumerlog` ( `client_ip` varchar(128) DEFAULT NULL, `log_time` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP, `status` int(6) DEFAULT NULL, `http_referer` text, `AgentVersion` varchar(512) DEFAULT NULL,) grant all on nginxlog.* to nginxlog@'%' identified by '123456';

4. Install logstash-output-jdbc plugin

vim /usr/share/logstash/Gemfile# source "https://rubygems.org" Change foreign source comments to domestic source "https://gems.ruby-china.com/"/usr/share/logstash/bin/logstash-plugin install logstash-output-jdbcValidating logstash-output-jdbcInstalling logstash-output-jdbcInstallation successful /usr/share/logstash/bin/logstash-plugin list| grep jdbclogstash-input-jdbclogstash-output-jdbc

5.nginx log formatting

log_format access_log_json '{"client_ip":"$remote_addr","log_time":"$time_local","request":"$request","status":"$status","body_bytes_sent":"$body_bytes_sent","http_referer":"$http_referer","AgentVersion":"$http_user_agent","upstream_addr":"$upstream_addr","request_time":"$request_time","upstream_response_time":"$upstream_response_time"}';

6.

[root@localhost conf.d]# cat consumer_log.conf

/etc/logstash/conf.d

input{ file{ path => "/usr/local/tengine-2.1.2/logs/sxt-consumer.log" start_position => "beginning" stat_interval => "2" codec => "json" } } filter { if [status] != "200" { drop{} }}output{ jdbc{ connection_string => "jdbc:mysql://192.168.14.61/nginxlog? user=nginxlog&password=123456&useUnicode=true&characterEncoding=UTF8" statement => ["insert into consumerlog(client_ip,status,http_referer,AgentVersion) VALUES(?,?,?,?) ", "client_ip","status","http_referer","AgentVersion"] }}

7. Start logstash

systemctl start logstash

8. validation data

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.