Shulou(Shulou.com)05/31 Report--

Most people do not understand the knowledge points of this "Nginx how to integrate Kafka" article, so the editor summarizes the following content, detailed content, clear steps, and has a certain reference value. I hope you can get something after reading this article. Let's take a look at this "Nginx how to integrate Kafka" article.

Background

Nginx-kafka-module is a plug-in of nginx, which can integrate kafka into nginx to facilitate the collection of embedded data on front-end pages in web projects. for example, if the front-end page is set up, some access and request data of users can be sent directly to the message middleware kafka through http requests, and the back-end can consume messages in kafka for real-time calculation. For example, through sparkstream to real-time consumption of kafka data to analyze user pv,uv, some user behavior and page funnel model conversion rate, to better optimize the system or real-time dynamic analysis of visiting users.

Specific integration steps

1. Install git

Yum install-y git

two。 Change to the / usr/local/src directory and clone the c client source code of kafka locally

Cd / usr/local/srcgit clone https://github.com/edenhill/librdkafka

3. Go to librdkafka and compile

Cd librdkafkayum install-y gcc gcc-c++ pcre-devel zlib-devel./configuremake & & make install

4. Install the plug-in of nginx integration kafka and enter the source code of / usr/local/src,clone nginx integration kafka

Cd / usr/local/srcgit clone https://github.com/brg-liuwei/ngx_kafka_module

5. Go to the source package directory of nginx (compile nginx, then the plug-in will be compiled at the same time)

Cd / usr/local/src/nginx-1.12.2./configure-- add-module=/usr/local/src/ngx_kafka_module/make & & make install

6. Modify the configuration file for nginx: set up a topic for location and kafaka

# add configuration (2 places) kafka;kafka_broker_list f1 kafka_topic access888; 9092 f2 kafka_topic access888; 9092 f3 kafka_topic access888; 9092

As shown below:

7. Start zk and kafka clusters (create topic)

Zkserver.sh startkafka-server-start.sh-daemon config/server.properties

8. Start nginx, error reported, cannot find the file of kafka.so.1

Error while loading shared libraries: librdkafka.so.1: cannot open shared object file: no such file or directory

9. Load the so library

# load the library echo "/ usr/local/lib" > > / etc/ld.so.conf# under / usr/local/lib manually load ldconfig

10. Test, write data to nginx, and then see if consumers of kafka can consume data

Curl http://localhost/kafka/access-d "message send to kafka topic" curl http://localhost/kafka/access-d "Xiao Wei 666" test

You can also simulate the page burying request API to send information:

The consumption information of backend kafka is shown in the figure:

The above is about the content of this article on "how Nginx integrates Kafka". I believe we all have a certain understanding. I hope the content shared by the editor will be helpful to you. If you want to know more about the relevant knowledge, please follow the industry information channel.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.