Shulou(Shulou.com)06/03 Report--

Thursday, 2019-3-14

Linux initialization script (centos6 centos7 generic) Linux initialization script (centos6 centos7 generic)

Zookeeper production environment building zookeeper production environment building

Be sure to install zookeeper before installation and check the above 2 link addresses!

Kafka Optimization:

Sometimes the system needs to deal with sudden peak data, which may slow down the disk. (for example, batch operations at the beginning of each hour, etc.)

At this time, we need to allow more dirty data to be stored in memory, so that the background process can slowly write the data to disk asynchronously.

Vm.dirty_background_ratio = 5

Vm.dirty_ratio = 80

At this time, the background starts asynchronous cleaning when the dirty data reaches 5%, but the system will not force synchronous disk writes before 80%. This makes the IO smoother.

Actual operation:

Echo "vm.dirty_background_ratio=5" > > / etc/sysctl.conf

Echo "vm.dirty_ratio=80" > > / etc/sysctl.conf

Sysctl-p

Construction of kafka production environment

# cd / opt/ins/ [root@emm-kafka01-10 Mar 174Mar] # lltotal 233044-rwxr-xr-x 1 root root 166044032 Mar 13 15:58 jdk-8u102-linux-x64.rpm-rw-r--r-- 1 root root 50326212 Mar 13 16:14 kafka_2.12-1.1.0.tgzMui Mar r Mar-1 root root 22261552 Mar 13 16:14 zookeeper-3.4.8.tar.gz [root@emm-kafka01-10murmur174 ins] # tar-zxvf kafka_2.12-1.1.0.tgz-C / usr/local/ [root@emm-kafka01-10Murray 174ins] # cd / usr/local/ [root@emm-kafka01-10Murray 174local] # ln-s kafka_2.12-1.1.0 / kafka [root@emm-kafka01-10Murray 174local] # lltotal 4drwxr-xr-x. 2 root root 6 Apr 11 2018 bindrwxr-xr-x. 2 root root 6 Apr 11 2018 etcdrwxr-xr-x. 2 root root 6 Apr 11 2018 gamesdrwxr-xr-x. 2 root root 6 Apr 11 2018 includelrwxrwxrwx 1 root root 17 Mar 14 09:51 kafka-> kafka_2.12-1.1.0/drwxr-xr-x 6 root root 83 Mar 24 2018 kafka_2.12-1.1.0drwxr-xr-x. 2 root root 6 Apr 11 2018 libdrwxr-xr-x. 2 root root 6 Apr 11 2018 lib64drwxr-xr-x. 2 root root 6 Apr 11 2018 libexecdrwxr-xr-x. 2 root root 6 Apr 11 2018 sbindrwxr-xr-x. 5 root root 46 Apr 11 2018 sharedrwxr-xr-x. 2 root root 6 Nov 12 13:03 srclrwxrwxrwx 1 root root 15 Mar 13 18:20 zookeeper- > zookeeper-3.4.8drwxr-xr-x 11 root root 4096 Mar 13 18:22 zookeeper-3.4.8

Modify the configuration file

We will not make any special configuration for zookeeper.properties producer.properties consumer.properties.

We only modify the configuration of server.properties.

[root@emm-kafka01-10 Mar 174 config] # pwd/usr/local/kafka/config [root@emm-kafka01-10 Mar 174 config] # lltotal 64 connect-console-source.properties-rw-r--r-- RW Mar-1 root root 906 Mar 24 2018 connect-console-sink.properties-rw-r--r-- 1 root root 909 Mar 24 2018 connect-console-source.properties-rw-r--r-- 1 root root 5807 Mar 24 2018 connect-distributed.properties-rw-r--r-- 1 root root 883 Mar 24 2018 connect-file-sink.properties-rw-r--r-- 1 root root 881 Mar 24 2018 connect-file-source.properties-rw-r--r-- 1 root root 1111 Mar 24 2018 connect-log4j.properties-rw-r--r-- 1 root root 2730 Mar 24 2018 connect-standalone.properties-rw-r--r-- 1 root root 1221 Mar 24 2018 consumer.properties-rw-r--r-- 1 root root 4727 Mar 24 2018 log4j.properties-rw-r--r- -1 root root 1919 Mar 24 2018 producer.properties-rw-r--r-- 1 root root 6851 Mar 24 2018 server.properties-rw-r--r-- 1 root root 1032 Mar 24 2018 tools-log4j.properties-rw-r--r-- 1 root root 1023 Mar 24 2018 zookeeper.properties # ls-l server.properties-rw-r--r-- 1 root root 6851 Mar 24 2018 server.properties Config] # vim server.properties.

As follows

Server.properties under config

Broker.id=10-if the id is deployed as a kafka cluster, the id cannot be the same

Port=9092-default kafka port. If multiple kafka instances are deployed on a machine, two instance ports are required to be different.

Log.dirs=/var/log/kafka/kafka-logs this needs to be modified. This is to store topic related information.

Zookeeper.connect=10.2.10.174:2181,10.2.10.175:2181,10.2.10.176:2181/kafkagroup

[root@emm-kafka01-10murmur174 config] # grep'^ [aMel Z] 'server.properties broker.id=174 / / each instance is different num.network.threads=3num.io.threads=8socket.send.buffer.bytes=102400socket.receive.buffer.bytes=102400socket.request.max.bytes=104857600log.dirs=/var/log/kafka/kafka-logsnum.partitions=1 / / the default partition for each topic is 1num.recovery.threads.per.data.dirroom1offsets.topic.replication.factorkeeper 1transaction.state.log.replication.factorroom1transaction.state.log.min.isrroom1log.retention.hours168log.segment.byteseggs 1073741824log.retention.check.interval.mseggs 300000zookeeper.connectmakers 10.2.10.174u2181ther 10.2.175fre2181ther 10.2.176pur2181hand kafkagprouzookeeper.connection.timeout.mswatches 6000group.initial.rebalance.delay.mseggs

Note:

Kafka implementation requires zookeeper scheduling, so this is the connection to the zk stand-alone or cluster. The above is the way to connect to the cluster, and / kafkagroup can also be removed, but the znode structure of zk will be confusing, so it is recommended to add a path.

In this way, kafkagroup will appear under zk's znode.

The broker.id= in the server.properties configuration file of the other two is changed to 175176

Everything else remains the same.

[root@emm-kafka01-10 Murmai 174 config] # scp server.properties root@10.2.10.175:/usr/local/kafka/config/

Server.properties 100% 6911 2.5MB/s 00:00

[root@emm-kafka01-10 Murmai 174 config] # scp server.properties root@10.2.10.176:/usr/local/kafka/config/

Server.properties 100% 6911 2.8MB/s 00:0

Modify kafka startup script (adjust startup memory footprint)

[root@emm-kafka01-10 Murmai 174 bin] # vim / usr/local/kafka/bin/kafka-server-start.sh

If ["x$KAFKA_HEAP_OPTS" = "x"]; then

Export KAFKA_HEAP_OPTS= "- Xmx1G-Xms1G"

Fi

Change to

Our memory is 8 gigabytes. Here's how we change it.

Export KAFKA_HEAP_OPTS= "- Xmx4G-Xms1G"

Maximum 4G initial 1G

When the memory is 4G, it is changed to

Export KAFKA_HEAP_OPTS= "- Xmx2G-Xms1G"

[root@emm-kafka01-10 bin 174 bin] # scp kafka-server-start.sh root@10.2.10.175:/usr/local/kafka/bin/kafka-server-start.sh 100% 1376 1.5MB/s 00:00 [root@emm-kafka01-10 Murray 174 bin] # scp kafka-server-start.sh root@10.2.10 .176: / usr/local/kafka/bin/kafka-server-start.sh 100% 1376 1.2MB/s 00:00

4. Before starting kafka / / background startup and then starting, let's set the environment variable first.

[root@emm-kafka01-10murmur174 bin] # vim / etc/profileexport PATH=/usr/local/kafka/bin:/usr/local/zookeeper/bin:$PATH [root@emm-kafka01-10murmur174 bin] # source / etc/profile [root@emm-kafka01-10murmur174 bin] # which kafka-server-start.sh / usr/local/kafka/bin/kafka-server-start.sh [root@emm-kafka01-10murmur174 bin] # which zkServer.sh/usr/local/zookeeper/bin/zkServer.sh

The other 2 stations are the same as above

Start kafka in the background

[root@emm-kafka01-10murmur174 bin] # cd ~ # nohup kafka-server-start.sh / usr/local/kafka/config/server.properties 1 > / dev/null 2 > & 1 & [1] 26314 [root@emm-kafka01-10murmur174] # jps14290 QuorumPeerMain26643 Jps26314 Kafka [root@emm-kafka01-10murmur174] # ps-ef | grep kafkaroot 26314 24915 35 10:12 pts/0 00:00:11 java-Xmx2G Xms1G-server [root@emm-kafka01-10 murmur174] # sh zkCli.sh [zk: localhost:2181 (CONNECTED) 0] ls / [kafkagroup Zookeeper] [zk: localhost:2181 (CONNECTED) 1] ls / kafkagroup [cluster, controller, controller_epoch, brokers, admin, isr_change_notification, consumers, log_dir_event_notification, latest_producer_id_block, config]

The above installation is successful

Our next step is to test kafka.

# kafka-topics.sh-- zookeeper 10.2.10.174/kafkagroup-- create-- topic majihui-- partitions 2-- replication-factor 2Created topic "majihui". [root@emm-kafka01-10 zookeeper 10.2.10.174/kafkagroup] # kafka-topics.sh-- zookeeper 10.2.10.174/kafkagroup-- describe-- topic majihui Topic:majihui PartitionCount:2 ReplicationFactor:2 Configs: Topic:majihui Partition: 0 Leader: 174Replicas: 174175 Isr: 174175 Topic: majihui Partition: 1 Leader: 175Replicas: 175176 Isr: 175176 # kafka-topics.sh-- zookeeper 10.2.10.174-- list [root@emm-kafka02-10Merle] # kafka-topics.sh-- zookeeper 10.2.10.174/kafkagroup-- list__consumer_offsetsmajihui launch producer [root@emm- Kafka02-10Meltel 175 ~] # kafka-console-producer.sh-- zookeeper 10.2.10.174Rose 2181-- topic majihui zookeeper is not a recognized option this command is incorrect Command changed # kafka-console-producer.sh-- broker-list 10.2.10.174broker-list 9092-- topic majihui > hello start the consumer [root@emm-kafka01-10mury174] # kafka-console-consumer.sh-- bootstrap-server 10.2.10.174Vist9092-- topic majihui hello delete topic majihui [root@emm-kafka02-10Muth175] # kafka-topics.sh-- zookeeper 10.2.10 .174 / kafkagroup-- delete-- topic majihuiTopic majihui is marked for deletion.Note: This will have no impact if delete.topic.enable is not set to true. # kafka-topics.sh-- zookeeper 10.2.10.174/kafkagroup-- list__consumer_ offsets [root @ emm-kafka01-10muri 174] # kafka-topics.sh-- zookeeper 10.2.10.174/kafkagroup-- delete-- topic _ _ consumer_offsetsError while executing topic Command: Topic _ _ consumer_offsets is a kafka internal topic and is not allowed to be marked for deletion.// error executing theme command: topic _ _ consumer_offsets is an internal theme of kafka Marking for deletion is not allowed. [2019-03-14 11 consumer_offsets is a kafka internal topic and is not allowed to be marked for deletion 2415] ERROR kafka.admin.AdminOperationException: Topic _ consumer_offsets is a kafka internal topic and is not allowed to be marked for deletion. At kafka.admin.TopicCommand$.$anonfun$deleteTopic$1 (TopicCommand.scala:188) at kafka.admin.TopicCommand$.$anonfun$deleteTopic$1$ adapted (TopicCommand.scala:185) at scala.collection.mutable.ResizableArray.foreach (ResizableArray.scala:59) at scala.collection.mutable.ResizableArray.foreach$ (ResizableArray.scala:52) at scala.collection.mutable.ArrayBuffer.foreach (ArrayBuffer.scala:48) at kafka.admin.TopicCommand$.deleteTopic (TopicCommand.scala:185) At kafka.admin.TopicCommand$.main (TopicCommand.scala:71) at kafka.admin.TopicCommand.main (TopicCommand.scala) (kafka.admin.TopicCommand$) [root@emm-kafka01-10 Murmuri 174] # kafka-topics.sh-- zookeeper 10.2.10.174/kafkagroup-- list__consumer_offsets

Due to the upgrade of the version, the previous command is no longer available

Pay attention here.

Kafka Optimization:

[root@emm-kafka01-10murmur174] # echo "vm.swappiness=1" > > / etc/sysctl.conf [root@emm-kafka01-10murmur174] # echo "vm.dirty_background_ratio=5" > > / etc/sysctl.conf [root@emm-kafka01-10murmur174] # echo "vm.dirty_ratio=80" > > / etc/sysctl.conf [root@emm-kafka01-10murmur174i] # sysctl-pvm.swappiness = 1vm.dirty_background_ratio = 5vm.dirty_ratio = 80 [root@emm-kafka01-10 529.6KB/s 174] # sysctl-pvm.swappiness = 1vm.dirty_background_ratio = 5vm.dirty_ratio = 80 [root@emm-kafka01-10muri 174] # scp / etc/sysctl.conf root@10.2.10.175:/etc/sysctl.conf sysctl.conf 100% 511 529.6KB/s 00:00 [root@emm-kafka01-10 root@emm-kafka01] # scp / etc/sysctl.conf root@10.2.10.176:/etc/sysctl.conf sysctl.conf 100% 511 497.6KB/s 00:00

Kafka startup script

[root@emm-kafka01-10 May 174 scripts] # pwd/opt/scripts [root@emm-kafka01-10 May 174 scripts] # lltotal 16-rwxr-xr-x 1 root root 6704 Mar 13 15:10 initialization.sh-rw-r--r-- 1 root root 10 11:05 kafkastart.sh-rw-r--r-- 1 root root 88 May 10 10:26 zkstart.sh [root@emm-kafka01-10 May scripts] # cat kafkastart.sh #! / bin/bashsource / etc/profilenohup / usr/local/kafka/bin/kafka-server-start.sh / usr/local/kafka/config/server.properties 1 > / dev/null 2 > & 1 & [root@emm-kafka01-10 usr/local/kafka/config/server.properties 174 70.8KB/s] # scp kafkastart.sh root@10.2.10.175:/opt/scripts/kafkastart.sh 100% 122 70.8KB/s 00:00 [root@emm-kafka01-10 root@emm-kafka01 root@emm-kafka01 174 scripts] # scp kafkastart.sh root@10.2.10.176:/opt/scripts/kafkastart.sh 100% 122 61.6KB/s 00:00 [root@emm-kafka01-10 Muhami 174 scripts] # sh kafkastart.sh [root@emm-kafka01-10 Muhami 174 scripts] # Jps11185 Jps7974 QuorumPeerMain10856 Kafka [root@emm-kafka01-10 Murray 174 scripts] # netstat-lntupActive Internet connections (only servers) Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name tcp 0 0127.0.0.1 jps11185 Jps7974 QuorumPeerMain10856 Kafka 25 0.0.0.0 * LISTEN 1379/master tcp 00 0.0.0.0: 22 0.0.0.0 LISTEN 1110/sshd tcp6 0:: 1:25:: * LISTEN 1379/master tcp6 0 0: 39549:: * LISTEN 10856/java Tcp6 0 0: 9092: * LISTEN 10856/java tcp6 0 0: 2181: * LISTEN 7974/java tcp6 00 10.2.10.174 LISTEN 7974/java tcp6 3888: * LISTEN 7974/java tcp6 0 0: 33264: * LISTEN 7974/java tcp6 0 0: 22: * LISTEN 1110/sshd

Add to boot self-boot

[root@emm-kafka03-10 Murray 176 scripts] # echo "/ usr/bin/sh / opt/scripts/kafkastart.sh" > > / etc/rc.d/rc.local [root@emm-kafka03-10 Rafael 176 scripts] # cat / EtcUnip rc.localbat THIS FILE IS ADDED FOR COMPATIBILITY PURPOSES## It is highly advisable to create own systemd services or udev rules# to run scripts during boot instead of using this file.## In contrast to previous versions due to parallel execution during boot # this script will NOT be run after all other services.## Please note that you must run 'chmod + x / etc/rc.d/rc.local' to ensure# that this script will be executed during boot.touch / var/lock/subsys/localulimit-SHn 65535echo' never' > / sys/kernel/mm/transparent_hugepage/enabledecho 'never' > / sys/kernel/mm/transparent_hugepage/defrag/usr/bin/sh / opt/scripts/zkstart.sh/usr/bin/sh / opt/scripts/kafkastart.sh [root @ emm-kafka03-10 Murtel 176 scripts] # chmod + x / etc/rc.d/rc.local

Kafka Optimization reference Link: https://www.cnblogs.com/yinzhengjie/p/9994207.html

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.