Shulou(Shulou.com)06/03 Report--

In addition to Ambria, you can also use Cloudera Manager to quickly build big data platform in the enterprise. These two tools are indeed much more convenient than manual building, but they have both advantages and disadvantages. Compared with beginners, it is not easy to grasp internal principles, difficult to troubleshoot, etc., while pure manual building has more mistakes and is not easy to succeed.

one。 Planning:

192.168.3.201 server/agent mysql

192.168.3.202 agent namenode

192.168.3.203 agent namenode (standby) resourcemanager

192.168.3.204 agent resourcemanager (standby)

192.168.3.205 agent

two。 Environmental preparation

System: Centos 6.564 bit

Configure IP,hostname

IP configuration of each server: # vim / etc/sysconfig/network-script/ifcfg-eth0

Service network restart

Configuration of each server name: # vim / etc/sysconfig/network

Vim / etc/hosts 192.168.3.201 rac201.com 192.168.3.202 rac202.com 192.168.3.203 rac203.com 192.168.3.204 rac204.com 192.168.3.205 rac205.com 3. Check the DNS configuration to make sure you can access the Internet # vim / etc/resolve.conf # ping www.baidu.com 4. Turn off the firewall and disable selinux getenforce to view the status of selinux and modify it to disable status # vim / etc/sysconfig/selinux SELINUX=disable restart takes effect 5. Disable and disable firewall temporarily disable # iptables-F # setenforce 0 or set boot not to start # service iptables stop # chkconfig iptables off 6. All servers uninstall their own JDK and install the new version For example, # rpm-qa in version 1.7 or 1.8 | grep java # rpm-e-- nodeps tzdata-java-2012j-1.el6.noarch java-1.6.0-openjdk-1.6.0.0-1.50.1.11.5.el6_3.x86_64 java-1.7.0-openjdk-1.7.0.9-2.3.4.1.el6_3.x86_64 # mkdir-p / opt/ Modules # tar zxf jdk-8u141-linux-x64.gz-C / opt/modules 7. Configure the JAVA environment variable # vi / etc/profile export JAVA_HOME=/opt/modules/jdk1.8.0_141 export PATH=$PATH:$JAVA_HOME/bin # source / etc/profile 8. Copy the configuration to another server # scp-r / opt/modules/jdk1.8.0_141 root@rac202.com:/opt/modules/ # scp-r / opt/modules/jdk1.8.0_141 root@rac203.com:/opt/modules/ # scp-r / opt/modules/jdk1.8.0_141 root@rac204.com:/opt/modules/ # scp-r / opt/modules/jdk1.8. 00141 root@rac205.com:/opt/modules/ check: java-version 9. Configure key-free login between servers 9.1 # ssh-keygen # ssh-copy-id-I ~ .ssh/id_rsa.pub rac201.com # ssh-copy-id-I ~ .ssh/id_rsa.pub rac202.com # ssh-copy-id-I ~ / .ssh/id_rsa.pub rac203.com # ssh-copy-id-I ~ / .ssh / id_rsa.pub rac204.com # ssh-copy-id-I ~ / .ssh/id_rsa.pub rac205.com 9.2 # ssh rac01.com # exit # Don't forget to exit # ssh rac02.com # exit # ssh rac03.com # exit # ssh rac04.com after connecting # exit # ssh rac05.com # exit 10. Server time synchronization starts ntpd process service # service ntpd start # chkconfig ntpd on # chkconfig-- list | grep ntpd 10.2 modify server configuration # vim / etc/ntp.conf Open comment Modify the network segment of the cost server # restrict 192.168.3.0 mask 255.255.255.0 nomodify notrap comment out the following lines It may be three lines or five lines # server 0.centos.pool.ntp.org # server 1.centos.pool.ntp.org # server 2.centos.pool.ntp.org manually add the following two lines: server 127.127.1.0 # local clock fudge 127.127.1.0 s start and configure to start # service at boot time Ntpd start & & chkconfig ntpd on & & chkconfig-- list | grep ntpd11. Set the maximum number of files and processes that can be opened for all servers Available memory, such as # vi / etc/security/limits.conf * soft nofile 32728 * hard nofile 1024567 * soft nproc 65535 * hard nproc unlimited * Soft memlock unlimited * hard memlock unlimited copy limits.conf to another server # scp / etc/security/limits.conf root@rac202.com:/etc/security/limits.conf # scp / etc/security/limits.conf root@rac203.com:/etc/security/limits.conf # scp / etc/security/limits.conf root@rac204.com:/etc/security/limits. Conf # scp / etc/security/limits.conf root@rac205.com:/etc/security/limits.conf restart takes effect: ulimit-a

three。 Mysql database installation

Check and uninstall the automatically installed version of Mysql on the rac201 server

Rpm-qa | grep mysql # rpm-e-- nodeps mysql-libs-5.1.71-1.el6.x86_64

To install Mysql5.6 using yum source, download the yum source file first

Wget http://repo.mysql.com/mysql57-community-release-el6-8.noarch.rpm # rpm-Uvh mysql57-community-release-el6-8.noarch.rpm modify yum source file: vim / etc/yum.repos.d/mysql-community.repo [mysql56-community] enabled=1 gpgcheck=1 [mysql57-community] enabled=0 gpgcheck=1 vim / etc/yum.repos.d/mysql-community-source.repo [mysql56-community-source] enabled=1 gpgcheck=1 [mysql-tools-preview-source] name=MySQL Tools Preview-Source baseurl= http://repo.mysql.com/yum/mysql-tools-preview/el/6/SRPMS enabled=1 gpgcheck=1 use the command yum repolist mysql to see if there is already a mysql installable file # yum repolist all | grep mysql yum installable mysql All the way down to yes # yum-y install mysql-community-server after successful installation Launch and initialize # service mysqld start # service mysqld status mysql has just been installed, the password of the root user of mysql is empty by default. You need to set the password for the root user. Execute the following command for security settings # mysql_secure_installation enter the current user: enter the root password: 123456 production environment. Remove anonymous users? [Y/n] y... Success! Normally, root should only be allowed to connect from 'localhost'. This ensures that someone cannot guess at the root password from the network. Disallow root login remotely? [Y/n] n... Skipping. By default, MySQL comes with a database named 'test' that anyone can access. This is also intended only for testing, and should be removed before moving into a production environment. Remove test database and access to it? [Y/n] n... Skipping. Reloading the privilege tables will ensure that all changes made so far will take effect immediately. Reload privilege tables now? [Y/n] y... Success!

four。 Cloudera Manager installation

Install the third-party dependency package on each server, or install the development package when the Centos system is installed

Yum-y install chkconfig python bind-utils psmisc libxslt zlib sqlite cyrus-sasl-plain cyrus-sasl-gssapi fuse portmap fuse-libs redhat-lsb

Download cloudera-manager-el6-cm5.3.6_x86_64.tar.gz in advance.

Http://archive.cloudera.com/cm5/cm/5/cloudera-manager-el6-cm5.3.6_x86_64.tar.gz

Create an unzipped directory of CM on the rac201 server

Mkdir-p / opt/cloudera-managertar-zxf cloudera-manager-el6-cm5.3.6_x86_64.tar.gz-C / opt/cloudera-manager

Modify the agent configuration on the rac201 server

Vi / opt/cloudera-manager/cm-5.3.6/etc/cloudera-scm-agent/config.iniHostname of the CM server.server_host=rac201.com server_port=7182

Copy the extracted CM folder to another server

Scp-r / opt/cloudera-manager/ root@rac202.com:/opt/# scp-r / opt/cloudera-manager/ root@rac203.com:/opt/# scp-r / opt/cloudera-manager/ root@rac204.com:/opt/# scp-r / opt/cloudera-manager/ root@rac205.com:/opt/

Users who create CM on all servers

Useradd-- system-- home=/opt/cloudera-manager/cm-5.3.6/run/cloudera-scm-server-- no-create-home-- shell=/bin/false cloudera-scm# cat / etc/passwd | grep cloudera-scm

Set permissions for the CM database

Mysql-uroot-p123456mysql > grant all privileges on *. * to 'temp'@'%' identified by' temp' with grant option;mysql > grant all privileges on *. * to 'temp'@'rac01.com' identified by' temp' with grant option;mysql > flush privileges;mysql > use mysql;mysql > select host, user from user Go to the folder below the rac201 server and execute the script command. Put the mysql-connector-java.jar in the / usr/share/java folder # cd / opt/cloudera-manager/cm-5.3.6/share/cmf/schema#. / scm_prepare_database.sh mysql- h rac01.com-utemp-ptemp-- scm-host rac01.com scm scm scm

Download the CDH source file in advance

CDH-5.3.6-1.cdh6.3.6.p0.11-el6.parcel

CDH-5.3.6-1.cdh6.3.6.p0.11-el6.parcel.sha

Create the following directory on the rac201 server

Mkdir-p / opt/cloudera/parcel-repo# mv CDH-5.3.6-1.cdh6.3.6.p0.11Murel6.parcel* / opt/cloudera/parcel-repo/ create # mkdir-p / opt/cloudera/parcels on all servers change the owner of the parcel directory on the rac201 server # chown-R cloudera-scm:cloudera-scm / opt/cloudera/parcel-repo/ on all servers # chown-R cloudera-scm:cloudera-scm / opt/cloudera/parcels

Start the process of CM

On the rac201 server

Cd / opt/cloudera-manager/cm-5.3.6/etc/init.d #. / cloudera-scm-server start

Start the agent process on all servers

Cd / opt/cloudera-manager/cm-5.3.6/etc/init.d/ #. / cloudera-scm-agent start

Server login interface of CM

Http://rac01.com:7180

Login account and password admin admin

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.