Shulou(Shulou.com)06/01 Report--

Preparatory work

1. To install JDK, the path and version of JDK installed by each member host in the cluster must be unified. two。 Install a MySQL Server on the host in the cluster (MySQL is optional, the default database for Ambari is PostgreSQL) 1. MySQL is configured to create the database of ambari-server and connect to users, and authorize # mysql-u root-pCREATE DATABASE ambari DEFAULT CHARSET=utf8;CREATE USER'@% 'IDENTIFIED BY'; GRANT ALL PRIVILEGES ON ambari.* TO'@'%'; # Delete MySQL's default anonymous account, omitted here. FLUSH PRIVILEGES;2. Join test MySQL and create structure table # mysql-u ambari- pSHOW DATABASES;SHOW GRANTS;# import structure table SOURCE / var/lib/ambari-server/resources/Ambari-DDL-MySQL-CREATE.sql;3. Configure mysql-jdbc connector wget http://java.com/****/mysql-connector-java-5.1.40.tar.gztar xf mysql-connector-java-5.1.40.tar.gzcd mysql-connector-java-5.1.40mkdir / usr/share/javacp mysql-connector-java-5.1.40-bin.jar / usr/share/java/cd / usr/share/java ln-s mysql-connector-java-5.1.40-bin.jar mysql-connector-java.jar

Or install mysql-connector-java directly from yum.

1. Configure ambari repo local source # VERSION_NUMBER=2.4.2.0-136 [Updates-ambari-2.4.2.0] name=ambari-2.4.2.0-Updatesbaseurl= http://172.17.20.21/ambari/gpgcheck=1gpgkey=http://172.17.20.21/ambari/RPM-GPG-KEY/RPM-GPG-KEY-Jenkinsenabled=1priority=12. Configure HDP local source [HDP-2.5.3.0] name=HDP Version-HDP-2.5.3.0baseurl= http://172.17.20.21/HDP/centos6/gpgcheck=1gpgkey=http://172.17.20.21/HDP/centos6/RPM-GPG-KEY/RPM-GPG-KEY-Jenkinsenabled=1priority=1[HDP-UTILS-2.5.3.0]name=HDP-UTILS Version-HDP-UTILS-2.5.3.0baseurl= http://172.17.20.21/HDP-UTILS-1.1 .0.21 / repos/centos6/gpgcheck=1gpgkey= http://172.17.20.21/HDP-UTILS-1.1.0.21/repos/centos6/RPM-GPG-KEY/RPM-GPG-KEY-Jenkinsenabled=1priority=13. Install ambari-serveryum install ambari-server

Yum installation of ambari-server requires database support to install postgresql,ambari-server. Default is PostgreSQL, or you can choose MySQL (here, because of the dependency, install PostgreSQL first, but not later, just install MySQL instead).

Yum install ambari-agent

Manually install ambari-agent on all member hosts in the cluster

4. Initialize ambari

Use the command:

Ambari-server setupUsing python / usr/bin/pythonSetup ambari-serverChecking SELinux...SELinux status is' disabled'Customize user account for ambari-server daemon [yPao] (n)? # ambari-server runs Adjusting ambari-server permissions and ownership...Checking firewall status...Checking JDK... [1] oracle jdk 1.8 + java cryptography extension (jce) policy files 8 [2] oracle jdk 1.7 + java cryptography extension (jce) policy files 7 [3] custom jdk # if jdk is already installed locally Then select this option, otherwise ambari initialization will install jdk====Enter choice (1): 3WARNING: JDK must be installed on all hosts and JAVA_HOME must be valid on all hosts.WARNING: JCE Policy files are required for configuring Kerberos security through the yum source. If you plan to use Kerberos,please makesure JCE Unlimited Strength Jurisdiction Policy Files are valid on all hosts.Path to JAVA_HOME: / opt/java/jdk # specifies the locally installed jdk path, that is, the value of the environment variable JAVA_HOME. Validating JDK on Ambari Server...done.Completing setup...Configuring database...Enter advanced database configuration [YBO] (n)? Y # whether you need custom database configuration Configuring database...====Choose one of the following options: [1]-PostgreSQL (Embedded) # default uses PostgreSQL [2]-Oracle [3]-MySQL / MariaDB # We choose locally installed MySQL [4]-PostgreSQL [5]-Microsoft SQL Server (Tech Preview) [6]-SQL Anywhere [7]-BDB====Enter choice (1): 3Hostname (localhost) : Port (3306): # whether you need to specify a database port Default 3306Database name (ambari): # whether you need to specify the database of ambari, default ambari library Username (ambari): # whether you need to specify the database user of ambari-server, default ambariEnter Database Password (bigdata): # whether you need to use the default database connection password The default is bigdataRe-enter password:Configuring ambari database...Copying JDBC drivers to server resources...Configuring remote database connection properties...WARNING: Before starting Ambari Server, you must run the following DDL against the database to create the schema:/var/lib/ambari-server/resources/Ambari-DDL-MySQL-CREATE.sqlProceed with configuring remote database connection properties [yPao] (y)? Y # initialize database connection data Extracting system views.Adjusting ambari-server permissions and ownership...Ambari Server 'setup' completed successfully. # successfully configuration completed

Start the ambari-server service:

Service ambari-server startUsing python / usr/bin/pythonStarting ambari-serverAmbari Server running with administrator privileges.Organizing resource files at / var/lib/ambari-server/resources...Ambari database consistency check started...No errors were found.Ambari database consistency check finishedServer PID at: / var/run/ambari-server/ambari-server.pidServer out at: / var/log/ambari-server/ambari-server.outServer log at: / var/log/ambari-server/ambari-server.logWaiting for server start. Ambari Server 'start' completed successfully.

After the configuration is completed, the web page can open http://HOST_IP:8080. After installation, the default login password is admin.

5. Web configuration of ambari-server

Click Launch install Wizard to begin installing and configuring the cluster.

-1. Cluster naming

-2. Choose HDP version. Since the local source version is 2.5, I choose the default version 2.5.

Select Use Local Repository (all cluster hosts need to install the role package from the Yum source, which defaults to the official source).

Delete the remaining unrelated system categories

Enter the HDP source and the Base URL of HDP-UTILS

-3. Enter the FQDN names of all hosts in the cluster. Upload or paste the private key (the ssh private key trusted by all hosts in the cluster).

After clicking the Registration and confirmation button, ambari-server will use the local source address specified in the previous step to connect to all the hosts in the cluster to install necessary ambari-agent and other packages, and to create system users for each service.

-4. Submit cluster initialization configuration

This step is to check whether the member hosts in the cluster have ambari-agent installed. If not, the program will be automatically installed through yum, as long as the yum source is configured in advance.

You need to configure the ntpd service of each member host in the cluster and start the service.

It is recommended to disable the Transparent Huge Pages (THP) function of the kernel by appending the configuration transparent_hugepage=never restart in the kenel startup parameters of grub.conf.

-5. Select the services to be installed

-6. Assign master roles for each host in the cluster

-7. Assign Slave roles for each host in the cluster

-8. You can configure the storage directory, storage space and memory allocation size of DataNode DataNode by customizing the parameters of each service.

-9. Previous configuration summary preview

-10. Start deployment

If you encounter an error prompt, you can click Message to view the error details.

6. HDP configure lzoyum-y install lzo lzo-devel hadooplzo hadooplzo-native

Configuration on Web: find the Advanced core-site entry and add com.hadoop.compression.lzo.LzoCodec to the original value of io.compression.codecs

Find the Custom core-site item, add key to io.compression.codec.lzo.class and value to com.hadoop.compression.lzo.LzoCodec.

Save the configuration. Just restart the cluster service.

Test lzo: execute

Hbase org.apache.hadoop.hbase.util.CompressionTest hdfs://mycluster/test_lzo lzo

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.