Shulou(Shulou.com)06/02 Report--

Warm Tip: to see the high-definition no-code picture, please open it with your mobile phone and click the picture to enlarge.

1. Overview

In the default installation package of CDH, Kafka,Kudu and Spark2 are not included, and a specific Parcel package needs to be downloaded separately to install the corresponding service. This document mainly describes the process of installing Kudu, Spark2 and Kafka through Parcel packages using Cloudera Manager on an offline environment based on a CDH5.12.1 cluster on the CentOS6.5 operating system.

Content summary Kudu installation Spark2 installation service verification test environment operating system version: CentOS6.5CM and CDH version 5.12.1 using CM administrator admin user operating system using root user operation conditions CDH cluster is running normally

2.Kudu installation

CDH5.12.1 is packaged and integrated with Kudu1.4, and Cloudera provides official support. No longer need to install the csd file of Kudu, you can directly operate Kudu after installing Kudu,Impala.

The following installation steps describe how to use Cloudera Manager to install and deploy Kudu1.4

Parcel deployment of 2.1Kudu

1. Download Kudu's Parcel package from Cloudera's official website at the following address

Http://archive.cloudera.com/kudu/parcels/5.12.1/KUDU-1.4.0-1.cdh6.12.1.p0.10-el6.parcelhttp://archive.cloudera.com/kudu/parcels/5.12.1/KUDU-1.4.0-1.cdh6.12.1.p0.10-el6.parcel.sha1http://archive.cloudera.com/kudu/parcels/5.12.1/manifest.json

two。 Download the above three files to the / var/www/html/kudu1.4 directory of the server where the http service is located

[root@ip-172-31-6-148~] # cd / var/www/html/ [root @ ip-172-31-6-148html] # mkdir Kudu1.4 [root @ ip-172-31-6-148html] # cd kudu1.4/ [root @ ip-172-31-6-148 kudu1.4] # lltotal 474140 lltotal 474140 Mustang Rwmuri-1 rootroot 485506175 Aug 30 14:55 KUDU-1.4.0-1.cdh6.12.1.p0.10-el6.parcel -rw-r--r-- 1 rootroot 41 Aug 30 14:55KUDU-1.4.0-1.cdh6.12.1.p0.10Murtel 6.parcel.sha1koum RWKEL RWKE-1 rootroot 2646 Aug 30 14:55 fu.json [root @ ip-172-31-6-148RW] #

3. Verify that http can be accessed properly

2.2 install the Kudu service

1. Configure the Parcel address of Kudu through the CM interface, and download, distribute, and activate Kudu.

Assigned Activation

two。 Go back to the CM home page and add Kudu services

Select the Kudu service and click "continue"

Select Master and Tablet Server, and click continue

Configure the corresponding directories, note: according to the actual situation, whether it is Master or Tablet, there should be multiple data directories (fs_data_dir) to improve concurrent read and write, thereby improving Kudu performance.

Start the Kudu service

Installation completed

2.3 configure Impala

Starting from CDH5.10, after installing Kudu, the default Impala can directly operate Kudu for SQL operation, but in order to save the need to add kudu_master_addresses attribute to TBLPROPERTIES every time the table is built, it is recommended to set the address and port of KuduMaster in the advanced configuration item of Impala:-- kudu_master_hosts=ip-172-31-6-148.fayson.com:7051.

Multiple master can be segmented with "," such as:

-- kudu_master_hosts=ip-172-31-6-148.fayson.complex 7051

3.Spark2 installation

The jdk version of the cluster is jdk1.7.0_67, and previous versions of Java7, Python2.6, and Hadoop2.6.5 are no longer supported from the Spark2.2.0 version, so choose Spark 2.1.0 deployment here.

3.1 install csd files

1. Download the csd file at the following address:

Http://archive.cloudera.com/spark2/csd/SPARK2_ON_YARN-2.1.0.cloudera1.jar

two。 Move the csd file to the / opt/cloudera/csd directory

[root@ip-172-31-6-148csd] # pwd/opt/cloudera/ CSD [root @ ip-172-31-6-148csd] # lltotal 16 RWFUR-1 rootroot 16109 Mar 29 06:58 SPARK2_ON_YARN-2.1.0.cloudera1.jar [root-31-6-148csd] #

If the csd directory does not exist, create a

[root@ip-172-31-6-148cloudera] # mkdir CSD [root @ ip-172-31-6-148cloudera] # chown cloudera-scm:cloudera-scm csd/

3. Restart the Cloudera Manager service

[root@ip-172-31-6-148~] # service cloudera-scm-serverrestartStopping cloudera-scm-server: [OK] Starting cloudera-scm-server: [OK] [root@ip-172-31-6-148~] #

Parcel deployment of 3.2Spark2

1. Download the Parcel package for Spark2 at the following address

Http://archive.cloudera.com/spark2/parcels/2.1.0/SPARK2-2.1.0.cloudera1-1.cdh6.7.0.p0.120904-el6.parcelhttp://archive.cloudera.com/spark2/parcels/2.1.0/SPARK2-2.1.0.cloudera1-1.cdh6.7.0.p0.120904-el6.parcel.sha1http://archive.cloudera.com/spark2/parcels/2.1.0/manifest.json

two。 Download the above three files to the / var/www/html/spark2.1.0 directory

[root-31-6-148html] # cd / var/www/html/ [root @ ip-172-31-6-148html] # mkdir Spark2.1.0 [root @ ip-172-31-6-148html] # cd spark2.1.0/ [root @ ip-172-31-6-148spark2.1.0] # lltotal 173052While RWHUBE-1 rootroot 4677 Mar 29 06:58 manifest.json-rw-r--r-- 1 rootroot 177185276 Mar 29 06:58SPARK2-2.1.0.cloudera1-1.cdh6.7.0.p0.120904 Murtel 6.ParcelMurr RWQULY-1 rootroot 41 Mar 29 06:58SPARK2-2.1.0.cloudera1-1.cdh6.7.0.p0.120904-el6.parcel.sha1 [root@ip-172-31-6-148 spark2.1.0] #

3. Verify that the deployment is successful

3. 3 install Spark2

1. Configure the Parcel address of Spark2 through the CM management interface and save it

two。 Click to download, assign and activate

3. Go back to the CM home page and add Spark2

4. Select Spark2 and click "continue"

5. Select a set of dependencies for the new Spark2 and click "continue"

6. Select the History Server and Gateway nodes, and click continue

7. Start the Spark2 service. When the service startup is complete, click "continue"

8.Spark2 installation completed

4.Kafka installation

4.1Kafka version selection

Kafka version feature minimum support CM version support CDH version is integrated into CDH2.2.x ​ Cloudera Manager 5.9.xCDH 5.9.x and higher No 2.1.xSentry authorizationCloudera Manager 5.9.xCDH 5.9.x and higher No 2.0.xEnhanced securityCloudera Manager 5.5.3CDH 5.4.x and higher No 1.4.xDistributed both as package and parcelCloudera Manager 5.2.xCDH 5.4.x, 5.5.x 5.6.x No 1.3.xIncludes Kafka MonitoringCloudera Manager 5.2.xCDH 5.4.x, 5.5.x, 5.6.x No 1.2.x ​ Cloudera Manager 5.2.xCDH 5.4.x, 5.5.x, 5.6.x No

Reference official website: https://www.cloudera.com/documentation/enterprise/release-notes/topics/rn\_consolidated\_pcm.html#pcm\_kafka

Parcel deployment of 4.2Kafka

1. Download Kafka's Parcel package from Cloudera's official website at the following address

Http://archive.cloudera.com/kafka/parcels/2.1.1.18/KAFKA-2.1.1-1.2.1.1.p0.18-el6.parcelhttp://archive.cloudera.com/kafka/parcels/2.1.1.18/KAFKA-2.1.1-1.2.1.1.p0.18-el6.parcel.sha1http://archive.cloudera.com/kafka/parcels/2.1.1.18/manifest.json

two。 Download the above three files to the / var/www/html/kafka2.1.1.18 directory

[root@ip-172-31-6-148html] # cd / var/www/html/ [root @ ip-172-31-6-148html] # mkdir Kafka2.1.1.18 [root @ ip-172-31-6-148html] # cd kafka2.1.1.18/ [root @ ip-172-31-6148 kafka2.1.1.18] # lltotal 66536RWMurtel RWMurtel-1 rootroot 68116503 Mar 27 17:39 KAFKA-2.1.1-1.2 .1.1.p0.18-el6.parcel-rw-r--r-- 1 rootroot 41 Mar 27 17:39KAFKA-2.1.1-1.2.1.1.p0.18 rootroot 6.parcel.sha1When RWKE RHUBE-1 rootroot 5252 kafka2.1.1.18 27 17:40 Jesson [root @ ip-172-31-6-148Root] #

3. Verify that the deployment is successful

4.3 install the Kafka service

1. Configure the Parcel packet address of Kafka through CM and save

two。 Click to download, assign and activate

3. Go back to the CM home page and add Kafka services

4. Select the Kafka service and click "continue"

5. Select a set of dependencies for Kafka and click continue

6. Select Kafka Broker and Gateway, and click continue

7. To modify the Kafka configuration according to the cluster environment, click "continue"

8.Kafka installation completed

9. Modify the heap size of Kafka Broker. The default is 50m, which may cause Kafka startup failure.

Save the configuration, redeploy the client, and restart the service.

5. Service verification

5.1Kudu verification

The statement of the table is as follows:

CREATE TABLE my_first_table (id BIGINT, name STRING, PRIMARY KEY (id)) PARTITION BY HASH PARTITIONS 16 STORED AS KUDU

Create a Kudu table through Impala-shell

[impala@ip-172-31-6-148root] $impala-shell-iip-172-31-10-118.fayson.com. [IP-31-31-10-118.fayson.com:21000] > show tables Query: show tables+-+ | name | +-+ | test | | test_table | +-+ Fetched 2 row (s) in 0.06s [IP-172-31-10-118.fayson.com:21000] > CREATE TABLEmy_first_table (> id BIGINT, > name STRING, > PRIMARY KEY (id) >) > PARTITION BY HASH PARTITIONS 16 > STORED AS KUDU Query: create TABLE my_first_table (id BIGINT, name STRING, PRIMARY KEY (id)) PARTITION BY HASH PARTITIONS 16STORED AS KUDUFetched 0 row (s) in 2.43s [IP-172-31-10-118.fayson.com:21000] >

Insert data and query

[ip-172-31-10-118.fayson.com:21000] > insert into my_first_table values; Query: insert into my_first_table values. Modified 1 row (s), 0 row error (s) in 3.92s [IP-172-31-10-118.fayson.com:21000] > select * from my_first_table ... id | name | +-+-+ | 1 | fayson | +-+-+ Fetched 1 row (s) in 1.02s [IP-172-31-10-118.fayson.com:21000] >

View via Kudu Master Web UI

5.2Spark2 verification

[root@ip-172-31-6-148C] # spark2-shell Setting default log level to "WARN" .to adjust logging level use sc.setLogLevel (newLevel). ForSparkR, use setLogLevel (newLevel). 09:46:22 on 17-09-11 WARN spark.SparkContext: Support for Java 7 is deprecated as of Spark 2.0.0Spark context Web UI available at http://172.31.6.148:4040Spark context available as' sc' (master = yarn) App id = application_1505121236974_0001). Spark session available as' spark'.Welcome to _ / / _ / / _\ / _ _ / `/ _ _ / / _ /. _ _ /. _ /\ _ version 2.1.0.cloudera1 / _ / Using Scala version 2.11.8 (Java HotSpot (TM) 64-Bit Server VM, Java 1.7.0x67) Type in expressions tohave them evaluated.Type: help for more information.scala > var textFile=sc.textFile ("/ fayson/test/a.txt") textFile: org.apache.spark.rdd.RDD [String] = / fayson/test/a.txt MapPartitionsRDD [1] at textFile at: 24scala > textFile.count () res0: Long = 3scala >

5.3Kafka verification

1. Create a topic for test

[root@ip-172-31-6-148hive] # kafka-topics-create--zookeeper ip-172-31-6-148.fayson.com:2181-- replication-factor 3-- partitions1-- topic test

two。 Send a message to topic

[root@ip-172-31-6-148hive] # kafka-console-producer--broker-list ip-172-31-10-118.fayson.com:9092-- topic test

3. Consume topic messages

[root@ip-172-31-6-148hive] # kafka-console-consumer-- zookeeperip-172-31-6-148.fayson.com:2181-- topic test-- from-beginning

4. View topic description information

[root@ip-172-31-6-148hive] # kafka-topics-- describe--zookeeper ip-172-31-6-148.fayson.com:2181-topic test

Drunken whips are famous horses, and teenagers are so pompous! Lingnan Huan Xisha, under the vomiting liquor store! The best friend refuses to let go, the flower of data play!

Warm Tip: to see the high-definition no-code picture, please open it with your mobile phone and click the picture to enlarge.

It is recommended to follow Hadoop practice, the first time, share more Hadoop practical information, welcome to forward and share.

Original article, welcome to reprint, reprint please indicate: reproduced from the official account of Wechat Hadoop

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.