In addition to Weibo, there is also WeChat
Please pay attention
WeChat public account
Shulou
2025-01-17 Update From: SLTechnology News&Howtos shulou NAV: SLTechnology News&Howtos > Servers >
Share
Shulou(Shulou.com)06/01 Report--
This article is about how Tachyon 0.7.1 implements pseudo-distributed cluster installation and testing. The editor thinks it is very practical, so share it with you as a reference and follow the editor to have a look.
Tachyon is a highly fault-tolerant distributed file system that allows files to be reliably shared in a cluster framework at memory speed, just like Spark and MapReduce. Tachyon achieves high performance through the use of information inheritance and memory intrusion. The Tachyon working set file is cached in memory and allows different Jobs/Queries and frameworks to access the cache file at the speed of memory. Therefore, Tachyon can reduce the number of times that datasets that need to be used frequently are obtained by accessing disk.
Source code download
Source code address: https://github.com/amplab/tachyon
Git clone https://github.com/amplab/tachyon.git
Compile
Enter TACHYON_HOME
Git tagv0.1.0v0.2.0v0.2.1v0.3.0v0.4.0v0.4.1v0.4.1-thriftv0.5.0v0.6.0v0.6.1v0.6.2v0.6.3v0.6.4v0.7.0v0.7.0-rc1v0.7.112345678910111213141516171819
Switch to v0.7.1 version
Git checkout v0.7.1 Note: checking out 'v0.7.1'.You are in' detached HEAD' state. You can look around, make experimentalchanges and commit them, and you can discard any commits you make in thisstate without impacting any branches by performing another checkout.If you want to create a new branch to retain commits you create, you maydo so (now or later) by using-b with the checkout command again. Example: git checkout-b new_branch_nameHEAD is currently located at a2b69c9. [maven-release-plugin] prepare release v0.7.11234567891011121314
Execute the following command:
Export MAVEN_OPTS= "- Xmx2g-XX:MaxPermSize=512M-XX:ReservedCodeCacheSize=512m" mvn clean package-Dhadoop.version=2.3.0-cdh6.1.0-Dspark.version=1.4.1-DskipTests12
Configuration
Create a journal directory
Mkdir-p / home/cluster/apps/tachyon/journal
Create ramdisk
Mkdir-p / home/cluster/apps/tachyon/ramdisk
Change tachyon_home/conf/tachyon-env.sh.template to tachyon_home/conf/tachyon-env.sh
Mv conf/tachyon-env.sh.template conf/tachyon-env.sh
Edit tachyon_home/conf/tachyon-env.sh by adding the following
Export JAVA_HOME=/home/cluster/share/java1.7export HADOOP_HOME=/home/cluster/apps/hadoopexport TACHYON_HOME=/home/cluster/apps/tachyonexport TACHYON_RAM_FOLDER=$TACHYON_HOME/ramdiskexport TACHYON_MASTER_ADDRESS=master12345
Modify the storage path of tachyon logs:
Create a logs directory under tachyon_home/
Mkdir logs
Vim conf/log4j.properties adds the following:
Tachyon.logs.dir=/home/cluster/apps/tachyon/logs
Change tachyon_home/conf/core-site.xml.template to tachyon_home/conf/core-site.xml
Mv conf/core-site.xml.template conf/core-site.xml
Edit the conf/core-site.xml and add the following:
Fs.tachyon.impl
Tachyon.hadoop.TFS
Format Tachyon
Bin/tachyon formatConnecting to localhost as wangyue...Formatting TachyonWorker @ wangyue-umConnection to localhost closed.Formatting TachyonMaster @ master12345 starts bin/tachyon-start.sh localKilled 0 processes on wangyue-umKilled 0 processes on wangyue-umConnecting to localhost as wangyue...Killed 0 processes on wangyue-umConnection to localhost closed. [sudo] password for wangyue: Formatting RamFS: / home/cluster/apps/tachyon/ramdisk (1gb) Starting master @ masterStarting worker @ wangyue-umjps: 21025 ZeppelinServer22214 Worker21987 Master21360 TachyonWorker32212 RemoteInterpreterServer23361 NameNode29603 RemoteInterpreterServer21324 TachyonMaster23493 DataNode21412 Jps27005 RemoteInterpreterServer has started TachyonMaster & TachyonWorker12345678910111213141516171819202122232425
The tachyon-start.sh local command starts both the Master and Worker processes locally. It is important to note that to run the tachyon-start.sh local command, you must have the password to switch root, otherwise it will fail to start. This is because root permission is required for RamFS formatting.
test
Webui: http://master:19999
/ home/cluster/apps/tachyon$ bin/tachyon runTest Basic CACHE_THROUGH/default_tests_files/BasicFile_CACHE_THROUGH has been removed2015-09-09 20 MasterClient.java:connect 0000 MasterClient.java:connect 17911 INFO (MasterClient.java:connect)-Tachyon client (version 0.7.1) is trying to connect with master @ master/127.0.0.1:199982015-09-09 20 00 Ride 00001 17938 INFO (MasterClient.java:connect)-User registered with the master @ master/127.0.0.1:19998 Got UserId 52015-09-09 20 CommonUtils.java:printTimeTakenMs CommonUtils.java:printTimeTakenMs-createFile with fileId 3 took 57 ms.2015-09-09 20 00 took 17999 INFO (WorkerClient.java:connect)-Trying to get local worker host: wangyue-um.local2015-09-09 20 00 CommonUtils.java:printTimeTakenMs 1813 INFO (WorkerClient.java:connect)-Connecting local worker @ wangyue-um.local/10.32.24.78:299982015-09-09 2000 createFile with fileId 1898 INFO (BlockOutStream.java:get)-Writing with local stream. TachyonFile: / default_tests_files/BasicFile_CACHE_THROUGH, blockIndex: 0, opType: CACHE_THROUGH2015-09-09 20 blockIndex 0000 tachyonFile 18136 INFO (CommonUtils.java:createBlockPath)-Folder / home/cluster/apps/tachyon/ramdisk/tachyonworker/5 was createdwatches 2015-09-09 20 00 Vega 18140 INFO (LocalBlockOutStream.java:)-/ home/cluster/apps/tachyon/ramdisk/tachyonworker/5/3221225472 was created! TachyonFile: / default_tests_files/BasicFile_CACHE_THROUGH, blockIndex: 0, blockId: 3221225472, blockCapacityByte: 536870912 2015-09-09 20 blockId 0000 INFO 18203 INFO (CommonUtils.java:printTimeTakenMs)-writeFile to file / default_tests_files/BasicFile_CACHE_THROUGH took 238 ms.2015-09-09 20 INFO (CommonUtils.java:printTimeTakenMs)-readFile file / default_tests_files/BasicFile_CACHE_THROUGH took 43 ms.Passed the test! Perform the bin/tachyon runTests cycle test 123456789101112131415 to stop wangyue@wangyue-um:/home/cluster/apps/tachyon$ bin/tachyon-stop.sh Killed 1 processes on wangyue-umKilled 1 processes on wangyue-umConnecting to localhost as wangyue...Killed 0 processes on wangyue-umConnection to localhost closed. There is no tachyon related process to execute jps:21025 ZeppelinServer22214 Worker21987 Master32212 RemoteInterpreterServer23361 NameNode29603 RemoteInterpreterServer27469 Jps23493 DataNode27005 RemoteInterpreterServer 123456789101112131415161718 Thank you for reading! On "Tachyon 0.7.1 how to achieve pseudo-distributed cluster installation and testing" this article is shared here, I hope the above content can be of some help to you, so that you can learn more knowledge, if you think the article is good, you can share it out for more people to see it!
Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.
Views: 0
*The comments in the above article only represent the author's personal views and do not represent the views and positions of this website. If you have more insights, please feel free to contribute and share.
Continue with the installation of the previous hadoop.First, install zookooper1. Decompress zookoope
"Every 5-10 years, there's a rare product, a really special, very unusual product that's the most un
© 2024 shulou.com SLNews company. All rights reserved.