Shulou(Shulou.com)06/03 Report--

Thank you. Prepare to do 1. Offline deployment outline MySQL offline deployment CM offline deployment Parcel offline source deployment 2. Planning

Linux version: CentOS 7.2

Node MySQL component Parcel "offline source CM service process" data component hadoop001MySQLParcelAlert Publisher Event ServerNN RM DN NM ZKhadoop002Alert Publisher Event ServerDN NM ZKhadoop003Host Monitor Service MonitorDN NM ZK3. Download source

CM

Cloudera-manager-centos7-cm5.16.1x8664.tar.gz

Parcel

CDH-5.16.1-1.cdh6.16.1.p0.3-el7.parcel

CDH-5.16.1-1.cdh6.16.1.p0.3-el7.parcel.sha1

Manifest.json

JDK

JDK8

Download jdk-8u202-linux-x64.tar.gz

MySQL

MYSQL5.7

Download mysql-5.7.26-el7-x86_64.tar.gz

MySQL JDBC jar

Mysql-connector-java-5.1.47.jar

Rename and remove the version number after the download

Mv mysql-connector-java-5.1.47.jar mysql-connector-java.jar boys. Cluster node initialization 1. Aliyun buys 3 virtual machines (minimum configuration 2core 8G) and chooses pay-per-view CentOS7.22. Current Notebook (win) hosts profile

Path: C:\ Windows\ System32\ drivers\ etc\ hosts

39.97.188.249 hadoop001 hadoop00139.97.225.112 hadoop002 hadoop00239.97.224.68 hadoop003 hadoop003

Note: IP is your virtual machine public network IP

3. Set the hosts file echo '172.17.144.104 hadoop001' > > / etc/hostsecho' 172.17.144.103 hadoop002' > > / etc/hostsecho '172.17.144.105 hadoop003' > > / etc/hosts# check cat / etc/hosts127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4::1 localhost localhost.localdomain localhost6 localhost6.localdomain6172.17.144.104 hadoop001172.17.144.103 hadoop002172.17.144.105 hadoop003 for all nodes

Note: IP is a private network IP

4. Disable all node firewalls and empty rule CVMs

For the CVM we use, the firewalls of both Ali Cloud and Tencent Cloud are disabled, so we do not need to turn off the firewall of the server. However, we need to check whether the web access port is automatically opened, and if not, add it ourselves.

(1) Open the security group configuration

Click configure rules after entering

(2) add security group rules

Note:

1. Click on the blue exclamation point and there will be a rule description 2. If the authorized object needs to set a network segment within the company, specify the ip network segment according to the figure above. If there is no limit, you can directly use 0.0.0.0swap 0 intranet server.

It is best to turn off the firewall when the internal server is deployed, turn it off temporarily if not, and turn it on again when the deployment is successful.

Systemctl stop firewalldsystemctl disable firewalldiptables-F 5. Close all node selinux

Ali CVM has disabled selinux, so there is no need to configure

Your own server will probably turn on selinux, so you need to shut it down.

Set the SELINUX=disabled in before the restart takes effect.

Vim / etc/selinux/config # This file controls the state of SELinux on the system.# SELINUX= can take one of these three values:# enforcing-SELinux security policy is enforced.# permissive-SELinux prints warnings instead of enforcing.# disabled-No SELinux policy is loaded.SELINUX=disabled# SELINUXTYPE= can take one of three two values:# targeted-Targeted processes are protected,# minimum-Modification of targeted policy. Only selected processes are protected. # mls-Multi Level Security protection.SELINUXTYPE=targeted6. Set the time zone consistency and clock synchronization of all nodes

Aliyun has synchronized the node time zone and time.

Let's actually operate the time zone time synchronization of the company environment.

6.1 time zone [root@hadoop001 ~] # timedatectl Local time: Tue 2019-05-28 15:37:53 CST Universal time: Tue 2019-05-28 07:37:53 UTC RTC time: Tue 2019-05-28 15:37:53 Time zone: Asia/Shanghai (CST, + 0800) NTP enabled: yes RTC in local TZ: yes DST active: n/a# View Command help, Learning is important, you need Baidu Too low [root@hadoop001 ~] # timedatectl-- helptimedatectl [OPTIONS...] COMMAND... Query or change system time and date settings.-h-help Show this help message--version Show package version--no-pager Do not pipe output into a pager--no-ask-password Do not prompt for password-H-host= [user @] HOST Operate on remote host-M-machine=CONTAINER Operate on local container--adjust-system-clock Adjust system clock when changing local RTC modeCommands:status Show current time settingsset-time TIME Set system timeset-timezone ZONE Set system time zonelist-timezones Show known time zonesset-local-rtc BOOL Control whether RTC is in local timeset-ntp BOOL Control whether NTP is enabled# to check which time zone [root@hadoop001 ~] # timedatectl list-timezonesAfrica/AbidjanAfrica/AccraAfrica/Addis_AbabaAfrica/AlgiersAfrica/AsmaraAfrica/Bamako# all nodes set the Asian Shanghai time zone [root@hadoop001 ~] # timedatectl set-timezone Asia/Shanghai [root@hadoop002 ~] # timedatectl set-timezone Asia/Shanghai [root@hadoop003 ~] # timedatectl set-timezone Asia/Shanghai6.2. Time # install NTP [root @ hadoop001 ~] # yum install-y ntp# choose hadoop001 as the primary node of ntp [root@hadoop001 ~] # vi / etc/ntp.conf#timeserver 0.asia.pool.ntp.orgserver 1.asia.pool.ntp.orgserver 2.asia.pool.ntp.orgserver 3.asia.pool.ntp.org# when external time is not available Can use local hardware time server 127.127.1.0 iburst local clock# which network segment machines are allowed to synchronize the time to modify the synchronization time to their own private network segment restrict 172.17.144.0 mask 255.255.255.0 nomodify notrap# enable ntpd and check the status [root@hadoop001 ~] # systemctl start ntpd [root@hadoop001 ~] # systemctl status ntpd ● ntpd.service-Network Time ServiceLoaded: loaded (/ usr/lib/systemd/system/ntpd.service) Enabled; vendor preset: disabled) Active: active (running) since Sat 2019-05-11 10:15:00 CST 11min agoMain PID: 18518 (ntpd) CGroup: / system.slice/ntpd.service! "18518 / usr/sbin/ntpd-u ntp:ntp-gMay 11 10:15:00 hadoop001 systemd [1]: Starting Network Time Service...May 11 10:15:00 hadoop001 ntpd [18518]: proto: precision = 0.088 usecMay 11 10:15:00 hadoop001 ntpd [18518]: 0.0.0.0 c01d 0d kern kernel time sync enabledMay 11 10:15:00 hadoop001 systemd [1]: Started Network Time Service.# verification Certificate [root@hadoop001 ~] # ntpq-premote refid st t when poll reach delay offset jitter====LOCAL (0) .LOCL. 10 l 726 64 0 0.000 0.000 0.005 other slave nodes disable ntpd services [root@hadoop002 ~] # systemctl stop ntpd [root@hadoop002 ~] # systemctl disable ntpdRemoved symlink / etc/systemd/system/multi-user.target.wants/ntpd.service. [root@hadoop002 ~] # / usr/sbin/ntpdate hadoop00111 May 10:29:22 ntpdate [9370]: Adjust time server 172.19.7.96 offset 0.000867 sec# synchronize hadoop001 node time [root@hadoop002 ~] # crontab-e00 * / usr/sbin/ntpdate hadoop001 [root@hadoop003 ~] # systemctl stop ntpd [root@hadoop004 ~] # systemctl disable ntpdRemoved symlink / etc/systemd/system/multi-user.target.wants/ntpd.service. [root@hadoop005 ~] # / usr/sbin/ntpdate hadoop00111 May 10:29:22 ntpdate [9370]: adjust time server 172.19.7.96 offset 0.000867 sec# synchronize hadoop001 node time in the early hours of every morning [root@hadoop003 ~] # crontab-e00 * / usr/sbin/ntpdate hadoop0017.JDK deployment mkdir / usr/javatar-xzvf jdk-8u45-linux-x64.tar.gz-C / usr/java/# keep in mind that the chown-R root:root / usr/java/jdk1.8.0_45 of the subscriber and user group must be corrected [root@hadoop001 cdh6.16.1] # vim / etc/profileexport JAVA_HOME=/usr/java/jdk1.8.0_45export PATH=$ {JAVA_HOME} / bin:$ {PATH} source / etc/profilewhich java

If there are too many nodes, then only make a mirror template, and after the basic work is completed, distribute the clones. no, no, no. (you'd better ask the operation and maintenance brother to do it.)

8.hadoop001 node offline deployment of MySQL5.7

(according to production standards)

8.1 extract and create folder # extract [root@hadoop001 cdh6.16.1] # tar xzvf mysql-5.7.11-linux-glibc2.5-x86_64.tar.gz-C / usr/local/# switch directory [root@hadoop001 cdh6.16.1] # cd / usr/local/# modify mysql name [root@hadoop001 local] # mv mysql-5.7.11-linux-glibc2.5-x86_64/ mysql# create folder [root@hadoop001] Local] # mkdir mysql/arch mysql/data mysql/tmp8.2 create my.cnfrm / etc/my.cnfvim / etc/ my.cnf [client] port = 3306socket = / usr/local/mysql/data/mysql.sockdefault-character-set= utf8mb4 [mysqld] port = 3306socket = / usr/local/mysql/data/mysql.sockskip-slave-startskip-external-lockingkey_buffer_size = 256Msort_buffer_size = 2Mread_buffer_size = 2Mread_rnd _ buffer_size= 4Mqueryroomcachesisize= 32Mmax_allowed_packet = 16Mmyisam_sort_buffer_size=128Mtmp_table_size=32Mtable_open_cache = 512thread_cache_size = 8wait_timeout = 86400max_connections = 60 percent Try number of CPU's*2 for thread_concurrency#thread_concurrency = 32 # isolation level and default engine default-storage-engine = INNODBtransaction-isolation = READ-COMMITTEDserver-id = 1739basedir = / usr/local/mysqldatadir = / usr/local/mysql/datapid-file = / usr / local/mysql/data/hostname.pid#open performance schemalog-warningssysdate-is-nowbinlog_format = ROWlog_bin_trust_function_creators=1log-error = / usr/local/mysql/data/hostname.errlog-bin = / usr/local/mysql/arch/mysql-binexpire_logs_days = 7innodb_write_io_threads=16relay-log = / usr/local/mysql/relay_log/relay-logrelay-log-index = / usr/local/mysql/relay_log/relay-log.indexrelay_log_info_file= / usr / local/mysql/relay_log/relay-log.infolog_slave_updates=1gtid_mode=OFFenforce_gtid_consistency=OFF# slaveslave-parallel-type=LOGICAL_CLOCKslave-parallel-workers=4master_info_repository=TABLErelay_log_info_repository=TABLErelay_log_recovery=ON#other logs#general_log = 1#general_log_file = / usr/local/mysql/data/general_log.err#slow_query_log=1#slow_query_log_file=/usr/local/mysql/data/slow_log.err#for replication slavesync_binlog = 500#for innodb options innodb _ data_home_dir = / usr/local/mysql/data/innodb_data_file_path = ibdata1:1G Ibdata2:1G:autoextendinnodb_log_group_home_dir = / usr/local/mysql/archinnodb_log_files_in_group = 4innodb_log_file_size = 1Ginnodb_log_buffer_size = 200M# according to production needs Adjust pool size innodb_buffer_pool_size = 2G#innodb_additional_mem_pool_size = 50m # deprecated in 5.6tmpdir = / usr/local/mysql/tmpinnodb_lock_wait_timeout = 1000#innodb_thread_concurrency = 0innodb_flush_log_at_trx_commit = 2innodb_locks_unsafe_for_binlog=1#innodb io features: add for mysql5.5.8performance_schemainnodb_read_io_threads=4innodb-write-io-threads=4innodb-io-capacity=200#purge threads change default (0) to 1 for purgeinnodb_purge _ threads=1innodb_use_native_aio=on#case-sensitive file names and separate tablespaceinnodb_file_per_table = 1 [mysqldump] quickmax_allowed_packet = 128m [MySQL] no-auto-rehashdefault-character-set=utf8mb4 [mysqlhotcopy] interactive- timeout [myisamchk] key_buffer_size = 256Msort_buffer_size = 256Mread_buffer = 2Mwrite_buffer = 2M8.3 create user groups and users [root@hadoop001 local] # groupadd-g 101 dba [root@hadoop001 local] # useradd-u 514-g dba- G root-d / usr/local/mysql mysqladmin [root@hadoop001 local] # id mysqladminuid=514 (mysqladmin) gid=101 (dba) groups=101 (dba) 0 (root) # # generally, you do not need to set the password for mysqladmin Switch the 8.4 copy environment variable profile directly from root or LDAP user sudo

Copy environment variable configuration file (hidden file) to the home directory of the mysqladmin user, configure personal environment variables for the following steps

Cp / etc/skel/.* / usr/local/mysql 8.5 configure the environment variable [root@hadoop001 local] # vi mysql/.bash_profile#. Bash _ profile# Get the aliases and functionsif [- f ~ /. Bashrc]; then. ~ / .bashrcfi # User specific environment and startup programsexport MYSQL_BASE=/usr/local/mysqlexport PATH=$ {MYSQL_BASE} / bin:$PATHunset USERNAME#stty erase ^ Hset umask to 022umask 022PS1 = `uname-n` ":"'$USER' ":"'$PWD' ": >" Export PS18.6 authorization and user group switching user mysqladmin installation [root@hadoop001 local] # chown mysqladmin:dba / etc/my.cnf [root@hadoop001 local] # chmod 640 / etc/my.cnf [root@hadoop001 local] # chown-R mysqladmin:dba / usr/local/mysql [root@hadoop001 local] # chmod-R 755 / usr/local/mysql 8.7 configure service and boot self-boot [root@hadoop001 local] # cd / usr/local/mysql# copy service files to init.d And renamed to mysql [root@hadoop001 mysql] # cp support-files/mysql.server / etc/rc.d/init.d/mysql# to give executable permission [root@hadoop001 mysql] # chmod + x / etc/rc.d/init.d/mysql# delete service [root@hadoop001 mysql] # chkconfig-- del mysql# add service [root@hadoop001 mysql] # chkconfig-- add mysql [root@hadoop001 mysql] # chkconfig-- level 345 mysql on8.8 install libaio and install mysql Initial db [root@hadoop001 mysql] # yum-y install libaio [root@hadoop001 mysql] # su-mysqladminLast login: Tue May 28 17:04:49 CST 2019 on pts/0hadoop001:mysqladmin:/usr/local/mysql: > bin/mysqld\ >-- defaults-file=/etc/my.cnf\ >-- user=mysqladmin\ >-- basedir=/usr/local/mysql/\ >-- datadir=/usr/local/mysql/data/\ >-- initialize

If-initial-insecure is added during initialization, a root@localhost account with an empty password will be created, otherwise a root@localhost account with a password will be created, and the password will be written directly in the log-error log file (in version 5.6, it is placed in the ~ / .mysql_secret file, which is more hidden. If you are not familiar with it, you may not know what to do.)

8.9 View temporary password # View password hadoop001:mysqladmin:/usr/local/mysql/data: > cat hostname.err | grep password2019-05-28T09:28:40.447701Z 1 [Note] A temporary password is generated for root@localhost: J=/usr/local/mysql/bin/mysqld_safe-- defaults-file=/etc/my.cnf & [1] 21740hadoop001:mysqladmin:/usr/local/mysql: > 2019-05-28T09:38:16.127060Z mysqld_safe Logging to'/ usr/local/mysql/ Data/hostname.err'.2019-05-28T09:38:16.196799Z mysqld_safe Starting mysqld daemon with databases from / usr/local/mysql/data# Press enter twice # # exit mysqladmin user # # View mysql process number [root@hadoop001 mysql] # ps-ef | grep mysqlmysqlad+ 21740 10 17:38 pts/0 00:00:00 / bin/sh / usr/local/mysql/bin/mysqld_safe-- defaults-file=/etc/my.cnfmysqlad+ 22557 21740 0 17:38 pts/0 00:00:00 / usr/local/mysql/bin/mysqld-- defaults-file=/etc/my.cnf-- basedir=/usr/local/mysql-- datadir=/usr/local/mysql/data-- plugin-dir=/usr/local/mysql/lib/plugin-- log-error=/usr/local/mysql/data/hostname.err-- pid-file=/usr/local/mysql/data/hostname.pid-- socket=/usr/local/mysql/data/mysql.sock-- port=3306root 22609 9194 : 39 pts/0 00:00:00 grep-- color=auto mysql## checks the MySQL port number [root@hadoop001 mysql] # netstat-nlp through the mysql process number | grep 2255 switch to mysqladmin [root@hadoop001 mysql] # su-mysqladminLast login: Tue May 28 17:24:45 CST 2019 on pts/0hadoop001:mysqladmin:/usr/local/mysql: > # # check whether mysql is running hadoop001:mysqladmin:/usr/local/mysql: > service mysql statusMySQL running (22557) [OK] 8.11 login And modify the user password # initial password hadoop001:mysqladmin:/usr/local/mysql: > mysql-uroot-paired J =

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.