Shulou(Shulou.com)06/03 Report--

Background

Nginx-kafka-module is a plug-in of nginx, which can integrate kafka into nginx to facilitate the collection of embedded data on front-end pages in web projects. for example, if the front-end page is set up, some access and request data of users can be sent directly to the message middleware kafka through http requests, and the back-end can consume messages in kafka for real-time calculation. For example, through SparkStream to real-time consumption of Kafka data to analyze user PV,UV, some user behavior and page funnel model conversion rate, to better optimize the system or real-time dynamic analysis of visiting users.

Specific integration steps 1. Install git yum install-y git2. Change to the / usr/local/src directory and clone the c client source code of kafka to the local cd / usr/local/src git clone https://github.com/edenhill/librdkafka3. Go to librdkafka and compile cd librdkafka yum install-y gcc gcc-c++ pcre-devel zlib-devel. / configure make & & make install4. Install the plug-in for nginx integration kafka and enter the / usr/local/src,clone nginx integration kafka source code cd / usr/local/src git clone https://github.com/brg-liuwei/ngx_kafka_module5. Go to the nginx source package directory (compile nginx, and then compile the plug-in at the same time) cd / usr/local/src/nginx-1.12.2. / configure-- add-module=/usr/local/src/ngx_kafka_module/ make & & make install6. Modify the configuration file of nginx: set a topic for location and kafaka. For more information, please see the nginx.conf# add configuration in the current directory (2) kafka;kafka_broker_list f1VO9092 f2VONING 9092 f3RAPHING 9092 kafka_topic access888; location = / kafka/access {kafka_topic access888;}

As shown below:

7. Start zk and kafka clusters (create topic) zkServer.sh start kafka-server-start.sh-daemon config/server.properties8. Start nginx, report an error, cannot find the file error while loading shared libraries: librdkafka.so.1: cannot open shared object file: No such file or directory9 of kafka.so.1. Load so library # boot load / usr/local/lib the library echo "/ usr/local/lib" > > / etc/ld.so.conf # load ldconfig10 manually. Test, write data to nginx, and then see if kafka consumers can consume data curl http://localhost/kafka/access-d "message send to kafka topic" curl http://localhost/kafka/access-d "Xiao Wei 666" test.

You can also simulate the page burying request API to send information:

The consumption information of backend Kafka is shown in the figure:

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.