Shulou(Shulou.com)06/03 Report--

Alluxio is a memory speed virtual distributed storage system.Alluxio, an open source memory-based distributed storage system, is now one of the fastest growing big data open source projects in the open source community.

Alluxio download address https://www.alluxio.org/download

To consider the availability of the system, HA Alluxio Master. The specific steps are as follows:

-1. Add the zookeeper information to the alluxio-env.sh and attach the path to the hdfs. If it is a stand-alone alluxio master, you do not need hdfs as the storage of the alluxio. But for Master HA mode, the shared file path is needed to store files, so hdfs is selected as the information storage for master.

Cat alluxio-env.sh JAVA_HOME=/usr/local/jvm/jdk1.7.0_75ALLUXIO_HOME=/data1/alluxio-1.4.0ALLUXIO_RAM_FOLDER=/data1/alluxio-1.4.0/ramdiskALLUXIO_UNDERFS_ADDRESS=/data1/alluxio-dataALLUXIO_WORKER_MEMORY_SIZE=1000MBALLUXIO_JAVA_OPTS= "- Dalluxio.zookeeper.enabled=true-Dalluxio.zookeeper.address=192.168.101.72:2181192.168.101.73:2181192.168.101.74:2181-Dalluxio.master.journal.folder=hdfs://liangbeta/user/hadoop/alluxio/journal"

-2. Also add the information of zookeeper to alluxio-site.properties and attach the path of hdfs

# cat alluxio-site.properties alluxio.worker.tieredstore.levels=2alluxio.worker.tieredstore.level0.alias=MEMalluxio.worker.tieredstore.level0.dirs.path=/mnt/ramdiskalluxio.worker.tieredstore.level0.dirs.quota=1GBalluxio.worker.tieredstore.level0.watermark.high.ratio=0.9alluxio.worker.tieredstore.level0.watermark.low.ratio=0.7alluxio.worker.tieredstore.level1.alias=HDDalluxio.worker.tieredstore.level1.dirs.path=/data1/alluxio-dataalluxio.worker.tieredstore.level1.dirs.quota=500GBalluxio.worker.tieredstore.level1.watermark.high.ratio=0 .9alluxio.worker.tieredstore.level1.watermark.low.ratio = 0.7~HA related settings ~ alluxio.zookeeper.enabled=truealluxio.zookeeper.address=192.168.101.72:2181192.168.101.73:2181192.168.101.74:2181alluxio.master.journal.folder=hdfs://liangbeta/user/hadoop/alluxio/journalalluxio.worker.block.heartbeat.timeout.ms=300000~~~

-3. In the configuration file, when it comes to the hdfs directory, you need to create

Su-hdfshdfs dfs-mkdir-p / user/hadoop/alluxio/journalhdfs dfs-chown alluxio:hadoop / user/hadoop/alluxiohdfs://liangbeta/user/hadoop/alluxio/journal

-4. Set up / etc/profile

Export JAVA_HOME=/usr/local/jvm/jdk1.7.0_75export HADOOP_HOME=/usr/local/hadoop/hadoop-releaseexport HADOOP_CONF_DIR=/usr/local/hadoop/hadoop-release/etc/hadoopexport ALLUXIO_HOME=/data1/alluxio-1.4.0export PATH=$HADOOP_HOME/sbin:$HADOOP_HOME/bin:$JAVA_HOME/bin:$ALLUXIO_HOME/bin:$PATH

-5. Need to copy the core-site.xml and hdfs-site.xml of hadoop to alluxio conf

-# ls / data1/alluxio-1.4.0/confalluxio-env.sh alluxio-site.properties core-site.xml hdfs-site.xml log4j.properties workers

-6. For newly built clusters, you need to format master and worker to start the service.

Format master

Format Masteralluxio format

Format worker

Format Workeralluxio formatWorker

-7. Add the secret-free sudo permission of the user of the Worker service to the Worker machine, otherwise you need to enter the alluxio password when starting worker

Add the following to / etc/sudoers

ALL= (ALL) NOPASSWD: ALL is actually set as follows: alluxio ALL= (ALL) NOPASSWD: ALL

-8. It is particularly emphasized here that due to security requirements, big data's environment is authenticated through kerberos. It is necessary to create alluxio users on KDC, generate keytab, copy them to alluxio servers, create scheduled tasks, and perform kinit.

$iduid=1100 (alluxio) gid=1091 (hadoop) groups=1091 (hadoop) $crontab-10 * kinit-kt / etc/security/keytab/alluxio.keytab alluxio$ klistTicket cache: FILE:/tmp/krb5cc_1100Default principal: alluxio@LIANG.COMValid starting Expires Service principal12/26/17 18:00:01 12-27-17 18:00:01 krbtgt/LIANG.COM@LIANG.COM renew until 12-26-17 18:00:01

-9. Start Master

Sh alluxio-start.sh master

-10. Start Worker

Sh alluxio-start.sh worker

-11. Start and close Proxy

Alluxio proxy processes can provide REST interfaces to the outside and can be started on master nodes and worker nodes.

Currently we have started the proxy process on both the master node and the worker node.

Sh alluxio-start.sh proxy

-12. After startup, you can query it through the web page

-13. Worker Information Page

-14. The directory you created, on the Browse page, you can see

-15. Create a / user/test directory

$alluxio fs mkdir / user/testSuccessfully created directory / user/test

-16. Summary of Alluxio commands

Start and close Master

Sh alluxio-start.sh mastersh alluxio-stop.sh master

Start and close Worker

Sh alluxio-start.sh workersh alluxio-stop.sh worker

Start and close Proxy

Sh alluxio-start.sh proxysh alluxio-stop.sh proxy

-17. Create a new directory and authorization method

Check the user who started alluxio

Ps-ef | grep alluxio

Switch to user

Su-alluxio

Create a user

Alluxio fs mkdir

Empower users

Alluxio fs chown

Updated on 2018-4-3

It should be noted that in the configuration of worker, you need to configure the alluxio-site.properties file and set the access level of alluxio. Otherwise, you will only use memory by default, not SSD or disk.

Cat alluxio-site.propertiesalluxio.worker.tieredstore.levels=2alluxio.worker.tieredstore.level0.alias=MEMalluxio.worker.tieredstore.level0.dirs.path=/data1/alluxio-ramdiskalluxio.worker.tieredstore.level0.dirs.quota=20GBalluxio.worker.tieredstore.level0.watermark.high.ratio=0.9alluxio.worker.tieredstore.level0.watermark.low.ratio=0.7alluxio.worker.tieredstore.level1.alias=SSDalluxio.worker.tieredstore.level1.dirs.path=/data1/alluxio-dataalluxio.worker.tieredstore.level1.dirs.quota=450GBalluxio.worker.tieredstore.level1.watermark.high.ratio=0 .9alluxio.worker.tieredstore.level1.watermark.low.ratio = 0.7 User properties-# alluxio.user.file.readtype.default=CACHE_PROMOTE-# alluxio.user.file.writetype.default=MUST_CACHE-# HAalluxio.zookeeper.enabled=truealluxio.zookeeper.address=zookeeper1:2181 Zookeeper2:2181,zookeeper3:2181alluxio.master.journal.folder=hdfs://lialng/user/alluxio/journalalluxio.worker.block.heartbeat.timeout.ms=300000 storage hierarchy problem Reference link: https://blog.csdn.net/lipeng_bigdata/article/details/50948184

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.