Shulou(Shulou.com)06/03 Report--

I went to an offline party a few days ago, and the people who attended the party were basically engaged in Internet work. At the meeting, someone mentioned the blockchain, which triggered a discussion about big data. It was only last year that I officially got in touch with big data and have been studying hadoop. I believe anyone who has come into contact with hadoop knows that. Each build in a separate hadoop requires a running environment, configuration file modification, testing, etc., and almost every component has a hole, especially for beginners. Previously, I have learned about their configuration process with some bloggers on Weibo, and some of them were stupefied for several days but did not succeed.

The configuration of the Hadoop runtime environment can only be understood by those who walk by, but in March, I found a more interesting domestic hadoop distribution-the fast open source release DKHadoop. DKHadoop integrates a number of common components in the open source community, including HDFS,Hbase,Storm,Flume,Kafka,Mahout,ES, into this platform, so the configuration and installation of the entire runtime environment is greatly simplified.

I sorted out the previous installation process to share with interested friends, mainly divided into: preparation phase, server operating system configuration, DKH system installation, common problem solving four departments. As the article is too long, this article focuses on sharing the first part-the preparatory stage.

Part I: preparatory work

1. Configuration required for big data platform:

(1) system: CentOS 6.564 bit (Desktop is required to be installed by default)

(2) CPU: intel E3 and above.

(3) memory: a minimum of 8 gigabytes is recommended, 32 gigabytes are recommended and 128 gigabytes are recommended.

(4) hard disk: 256g or above, solid state disk is recommended.

(5) system partition requirements: if there are no special needs, except for swap partition, all remaining space can be allocated to / (root) partition

(6) Network requirements: if there are no special circumstances, you can access the external network as far as possible.

(7) at least three servers (the names of three servers are arbitrary, and the password must be the same).

(8) install using root user as default user

2. Remote upload and remote connection tools:

(1) if you directly use the virtual machine enabled by the personal PC as the basis of the server, you can directly copy it to the virtual machine.

(2) if you use the computer room server as the platform to build the server, if you can not directly contact the server, you need to use remote tools to connect and send the installation package.

The first part is mainly about the preparation before installation. In addition to doing the above, you also need to download the installation version. I searched directly and found a three-node distribution downloaded from the official website of Daxuai. com. Friends who need it can also search for the main point, and if you can't find it, you can leave a message for me.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.