Shulou(Shulou.com)06/03 Report--

In the past few weeks, I have been building, writing and sharing around the running environment of DKhadoop. Some friends have left messages and asked for the dkhadoop installation package. I don't know if I have gone to download and install it. The download and installation of DKHadoop has been basically made clear. I have sorted out some of the contents of the operation and maintenance management platform of Express DKM big data these days. As a matching management platform for DKHadoop, it is necessary to have some understanding of DKM.

DKM is the DKHadoop management platform. As an end-to-end Apache Hadoop management application of big data platform, DKM provides fine-grained visualization and control of every part of DKH. Through DKM, operation and maintenance personnel can improve the performance of the cluster, improve the quality of service, improve compliance and reduce management costs.

DKM is designed to make the management of enterprise data centers simple and intuitive. Through DKM, it is easy to deploy and centrally operate the complete big data software stack. The application automates the installation process, thus reducing the time it takes to deploy the cluster. Through DKM, you can provide a cluster-wide view of the real-time running status of nodes. At the same time, a central console is provided that can be used to configure the cluster. To sum up, the main functions that DKM can provide are as follows:

1. Automate the Hadoop installation process, significantly reducing deployment time

two。 Provide a real-time overview of the cluster, such as the health of nodes and services

3. Provides a centralized central console to change the configuration of the cluster

4. Includes comprehensive reporting and diagnostic tools to help optimize performance and utilization

Basic functions: the basic functions of DKM can be divided into four modules: management function, monitoring function, diagnosis function and integration function. In this article, let's first look at the following management functions:

1. Batch deployment

We all know that Hadoop itself is a distributed system, so during installation, you need to install components for each node, and because it is open source software, the installation process is relatively complex, and each component of Hadoop needs to do a lot of configuration work. I believe you have a good understanding of this. DKH provides DKM to automate installation and deployment of Hadoop. It greatly shortens the installation time of Hadoop and simplifies the process of installing Hadoop. (for DKHADOOP installation steps, please refer to the article shared earlier.)

The process of automated installation is as follows:

1. Prepare the installation environment, download the installation files of DKM and DKH, and install JDK,yum and other basic software.

two。 Pick a node and install DKM. Users only need to start the installation script, which usually takes a few minutes.

3.DKM is a web application that provides a browser-based interface through which users can visually install and deploy DKH.

4. Through the DKM interface, add other required installation nodes, select the Hadoop components to install, and the role of each node, select installation, DKM will automatically distribute the software that needs to be installed to the corresponding nodes, and complete the installation.

5. When the software for all nodes is installed, DKM starts all services. From the above installation process, we can see that the installation of DKH mainly embodies two characteristics: batch and automation. It only needs to be done in one of the nodes, and the other nodes can be automatically installed in batches.

2. Cluster configuration

(1) Visualized parameter configuration interface

Hadoop contains many components, and different components contain a variety of configurations and are distributed on different hosts. DKM provides interface parameter configuration for this situation, and can be automatically deployed to each node.

(2) highly reliable configuration

DKM uses HA deployment scheme for key components to avoid single point of failure. At the same time, DKH provides automatic recovery processing for abnormal errors of components to maximize the reliability of the service.

(3) HDFS is highly reliable.

In a standard configuration, NameNode is a single point of failure (SPOF) in a HDFS cluster. Each cluster has a NameNode, and if the machine or process becomes unavailable, the cluster as a whole becomes unavailable until NameNode restarts or goes online on the new host. Secondary NameNode does not provide failover capabilities. To keep the status of the "standby" NameNode synchronized with the "active" NameNode in this implementation, both nodes communicate with a set of independent daemons called JournalNode. When any Namespace changes are performed by the "active" NameNode, it continues to record changes to most of the JournalNode. The "standby" NameNode can read editing operations from JournalNode and constantly monitor them for changes made to the editing log. When the standby node discovers editing operations, it applies those edits to its own Namespace. In the event of a failover, the standby node ensures that all edits are read from the JournalNode before upgrading itself to "active". This ensures that the Namespace state is fully synchronized before a further failover occurs.

To provide fast failover, the standby NameNode also needs to have up-to-date information about the location of the blocks in the cluster. To achieve this, DataNode configures the location of the two NameNode, and they send the location information and the detection signal to the two NameNode.

Only one of the NameNode can be active at a time, which is critical to the proper operation of the HA cluster. Otherwise, the Namespace status can quickly diverge between the two, leading to the risk of data loss or other incorrect results. To ensure this attribute and prevent the so-called "brain splitting condition", JournalNode allows only one NameNode to be a writer at a time. During the failover process, the NameNode that wants to enter the "active" state will take over the write role of the JournalNode, which effectively prevents other NameNode from remaining in the "active" state, so that the new "active" NameNode can safely continue with the failover.

DKH turns on HA by default. Users do not have to worry about this problem.

(4) YARN is highly reliable.

YARN ResourceManager (RM) is responsible for tracking resources in the cluster and scheduling applications (for example, MapReduce jobs). The RM High availability (HA) feature adds redundancy in the form of active / standby RM pairs to remove this single point of failure. In addition, when failing over from a standby RM to an active RM, the application can recover from its last checkpoint state; for example, map tasks completed in a MapReduce job are not rerun in subsequent attempts. This allows you to handle the following events without any significant performance impact on the running application:

Unplanned events, such as a computer crash.

Planned maintenance events, such as software or hardware upgrades on computers running ResourceManager.

RM HA requires Zookeeper and HDFS services to be running. RM HA is implemented as an active-standby RM pair. At startup, each RM is on standby; the startup process, but not loaded. When you transition to the active state, RM loads the internal state from the specified state store and starts all internal services. The administrator (through CLI) or through an integrated failover controller (when automatic failover is enabled) can facilitate the transition to an active state.

DKH enables Resource Manager HA by default. Users don't need to worry.

3. Rights management

System administrators, database administrators, and other administrators must be granted different levels of administrative rights.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.