Shulou(Shulou.com)06/02 Report--

This article mainly explains "how to use Nginx to do page collection", the content of the article is simple and clear, easy to learn and understand, the following please follow the editor's ideas slowly in depth, together to study and learn "how to use Nginx to do page collection" bar!

0. Introduction to Architectur

Simulate the online real-time stream, such as the user's operation log, collect the data, and then process it. For the time being, only consider the data collection, and use Html+Jquery+Nginx+Ngx_kafka_module+Kafka to implement it. Ngx_kafka_module is an open source component dedicated to docking Nginx and Kafka.

1. Requirements description 1.1 simulates the user request log with html and jquery

These include the following:

User id:user_id, access time: act_time, Action: (action, including click,job_collect,cv_send,cv_upload)

Enterprise code job_code

1.2After accepting the request in 1.1with Nginx, send the data to the subject tp_individual of Kafka using ngx_kafka_module. 1. 4 use a consumer consumption theme in kafka and observe 2. 4. Set up step 2.1 Kafka installation

Since you are using a ready-made installed docker-kafka image, you can start it directly.

Install Nginx and start $cd / usr/local/src$ git clone git@github.com:edenhill/librdkafka.git# to enter librdkafka, then compile $cd librdkafka$ yum install-y gcc gcc-c++ pcre-devel zlib-devel$. / configure$ make & & make install$ yum-y install make zlib-devel gcc-c++ libtool openssl openssl-devel$ cd / opt/hoult/software# 1. Download $wget http://nginx.org/download/nginx-1.18.0.tar.gz# 2. Extract $tar-zxf nginx-1.18.0.tar.gz-C / opt/hoult/servers# 3. Download the module source code $cd / opt/hoult/software$ git clone git@github.com:brg-liuwei/ngx_kafka_module.git# 4. Compile $cd / opt/hoult/servers/nginx-1.18.0 $. / configure-- add-module=/opt/hoult/software/ngx_kafka_module/$ make & & make install # 5. Delete the Nginx installation package $rm / opt/hoult/software/nginx-1.18.0.tar.gz# 6. Start nginx$ cd / opt/hoult/servers/nginx-1.18.0 $nginx3. Related configuration 3.1 nginx configuration nginx.conf#pid logs/nginx.pid;events {worker_connections 1024;} http {include mime.types; default_type application/octet-stream # log_format main'$remote_addr-$remote_user [$time_local] "$request" #'$status $body_bytes_sent "$http_referer" #'"$http_user_agent"$http_x_forwarded_for"; # access_log logs/access.log main; sendfile on; # tcp_nopush on; # keepalive_timeout 0; keepalive_timeout 65 # gzip on; kafka; kafka_broker_list linux121:9092; server {listen 9090; server_name localhost; # charset koi8-r; # access_log logs/host.access.log main #-start of kafka related configuration-location = / kafka/log {# Cross domain related configuration add_header 'Access-Control-Allow-Origin' $http_origin; add_header' Access-Control-Allow-Credentials' 'true' Add_header 'Access-Control-Allow-Methods'' GET, POST, OPTIONS'; kafka_topic tp_individual;} # error_page 404 / 404.html Start kafka producer and Consumer # create topickafka-topics.sh-- zookeeper linux121:2181/myKafka-- create-- topic tp_individual-- partitions 1-- replication-factor "create Consumer kafka-console-consumer.sh-- bootstrap-server linux121:9092-- topic tp_individual-- from-beginning# create producer Test kafka-console-producer.sh-- broker-list linux121:9092-- topic tp_individual3.3 Writing Html + Jquery code index function operate (action) {var json = {'user_id':' upright donated' 'act_time': current (). ToString (), 'action': action,' job_code': 'donald'} $.ajax ({url: "http://linux121:8437/kafka/log", type:" POST ", crossDomain: true, data: JSON.stringify (json) / / the following sentence allows cross-domain cookie to access xhrFields: {withCredentials: true}, success:function (data, status, xhr) {/ / console.log ("Operation successful:'" + action)} Error:function (err) {/ / console.log (err.responseText) );}; function current () {var d = new Date (), str =''; str + = d.getFullYear () +'-'; str + = d.getMonth () + 1 +'-'; str + = d.getDate () +'' Str + = d.getHours () +':'; str + = d.getMinutes () +':'; str + = d.getSeconds (); return str;}

Put the a.html in the directory of nginx, and the browser accesses 192.168.18.128 9090

4. Demo

4.1Firstly, start zk cluster, kafka cluster

4.2Then create topic, create consumers, create producers, test topic

4.3 launch the nginx access page, click, and observe the status of consumers

The whole process is shown below:

Thank you for your reading, the above is the content of "how to use Nginx to do page collection". After the study of this article, I believe you have a deeper understanding of how to use Nginx to do page collection, and the specific use needs to be verified in practice. Here is, the editor will push for you more related knowledge points of the article, welcome to follow!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.