Shulou(Shulou.com)06/02 Report--

Background

Nginx-kafka-module is a plug-in of nginx, which can integrate kafka into nginx to facilitate the collection of embedded data on front-end pages in web projects. for example, if the front-end page is set up, some access and request data of users can be sent directly to the message middleware kafka through http requests, and the back-end can consume messages in kafka for real-time calculation. For example, through SparkStream to real-time consumption of Kafka data to analyze user PV,UV, some user behavior and page funnel model conversion rate, to better optimize the system or real-time dynamic analysis of visiting users.

Specific integration steps

1. Install git

Yum install-y git

two。 Change to the / usr/local/src directory and clone the c client source code of kafka locally

Cd / usr/local/srcgit clone https://github.com/edenhill/librdkafka

3. Go to librdkafka and compile

Cd librdkafkayum install-y gcc gcc-c++ pcre-devel zlib-devel./configuremake & & make install

4. Install the plug-in of nginx integration kafka and enter the source code of / usr/local/src,clone nginx integration kafka

Cd / usr/local/srcgit clone https://github.com/brg-liuwei/ngx_kafka_module

5. Go to the source package directory of nginx (compile nginx, then the plug-in will be compiled at the same time)

Cd / usr/local/src/nginx-1.12.2./configure-- add-module=/usr/local/src/ngx_kafka_module/make & & make install

6. Modify the configuration file of nginx: set a topic for location and kafaka. For more information, please see the nginx.conf of the current directory.

# add configuration (2 places) kafka;kafka_broker_list f1 kafka_topic access888; 9092 f2 kafka_topic access888; 9092 f3 kafka_topic access888; 9092

As shown below:

7. Start zk and kafka clusters (create topic)

ZkServer.sh startkafka-server-start.sh-daemon config/server.properties

8. Start nginx, error reported, cannot find the file of kafka.so.1

Error while loading shared libraries: librdkafka.so.1: cannot open shared object file: No such file or directory

9. Load the so library

# load the library echo "/ usr/local/lib" > > / etc/ld.so.conf# under / usr/local/lib manually load ldconfig

10. Test, write data to nginx, and then see if consumers of kafka can consume data

Curl http://localhost/kafka/access-d "message send to kafka topic" curl http://localhost/kafka/access-d "Xiao Wei 666" test

You can also simulate the page burying request API to send information:

The consumption information of backend Kafka is shown in the figure:

The above is the whole content of this article, I hope it will be helpful to your study, and I also hope that you will support it.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.