Shulou(Shulou.com)06/01 Report--

This article shows you what the use of the Cloudera big data platform is, the content is concise and easy to understand, it can definitely brighten your eyes. I hope you can get something through the detailed introduction of this article.

CDH: Cloudera released a self-packaged commercial version of Hadoop software distribution package, which contains not only the commercial version of Cloudera Hadoop, but also a variety of commonly used open source data processing and storage frameworks, such as Spark, Hive, Hbase and so on.

Cloudera Manager: CM for short is to facilitate the installation, monitoring and management of related services such as Hadoop in the cluster by big data, which greatly simplifies the installation and configuration management of host, Hadoop, Hive, Spark and other services in the cluster. It is the software distribution and management monitoring platform of Hadoop cluster, through which a Hadoop cluster can be quickly deployed and the nodes and services of the cluster can be monitored in real time.

The core function of CM is divided into four modules

I. Management function

1. Batch automated deployment of nodes: CM provides powerful Hadoop cluster deployment capabilities, which can automate the deployment of nodes in batches. To install a Hadoop cluster, you only need to add the nodes to be installed, install the required components and assign roles, which greatly shortens the installation time of Hadoop and simplifies the installation process of Hadoop.

2, visual parameter configuration function: Hadoop contains many components, different components contain a variety of XML configuration files, CM provides interface GUI visual parameter configuration function.

3. Intelligent parameter verification and optimization: when there is a problem with some parameter values in the user configuration, CM will give an intelligent error prompt to help the user modify the configuration parameters more reasonably.

4. High availability configuration: CM uses HA deployment for key components, such as NameNode High availability, you can start HDFS HA through the CMweb management interface according to the wizard.

5. Permission management: different levels of administrative permissions are provided. For example, when a read-only user accesses the interface of CM, the operation options such as start and stop for all services are not available.

II. Monitoring function

1. Service monitoring: check the results of health checks at the service and instance levels, comprehensively monitor the setting of various indicators and the operation of the system, and the system will make recommendations on the actions that administrators should take.

2. Host monitoring: monitor the relevant information of all hosts in the cluster, including the memory currently consumed on the host, the role allocation running on the host, etc., which can not only display a summary view of all cluster hosts, but also further display a detailed view of the key metrics of a single host.

3. Behavior monitoring: CM provides lists and charts to view the activities going on on the cluster, not only to show the activities currently under way, but also to view historical activities through the dashboard.

4. Event activity: the monitoring interface can view events, and system administrators can filter events through time range, service, host, keyword and other field information.

5. Alarm: through the CM interface, you can generate an alarm for a specified event and notify it by mail or SNMP.

6. Logs and reports: you can easily click a link to view log entries for specific services, and CM can generate reports on collected historical monitoring data.

III. Diagnostic function

1. Periodic service diagnosis: CM will periodically diagnose the services running in the cluster, check whether the status of these services is normal, and notify them in time if there are any abnormalities.

2. Log collection and retrieval: for a large-scale cluster, CM provides log collection, which can view the logs of each machine and various services in the cluster through a unified interface, and can be retrieved according to the log level.

3. System performance usage report: CM can generate system performance reports, including CPU utilization of the cluster, CPU utilization of a single node, CPU utilization of a single process and other performance data, which is very important for hadoop cluster tuning.

IV. Integrated function

1. Security configuration: in order to facilitate the integration of Hadoop big data platform with original identity authentication systems such as AD and LDAP, CM only needs to be configured on the interface.

2. CM API: through CM API, CM can be easily integrated into the original enterprise management system.

3. SNMP Integration [simple Network Management Protocol (SNMP)]: CM provides SNMP integration capability, which can integrate SNMP with simple configuration and forward alarm information in the cluster.

Advanced features of CM (paid)

1. Software rolling upgrade: Hadoop version upgrade and bug repair, supporting the continued provision of services and applications during the upgrade process.

2. Parameter version control: any time the configuration is modified and guaranteed, CM will generate a version of the configuration to view the historical configuration and roll back to different versions, thus providing a reliable basis for cluster recovery and problem diagnosis.

3. Backup and disaster recovery system BDR: realize interface data backup and disaster recovery.

4. Data audit: support audit and access to data

5. Security Integration Wizard: start Kerberos integration and external security authentication integration, such as supporting user authentication through internal database and external services.

The above content is what is the use of Cloudera big data platform? have you learned the knowledge or skills? If you want to learn more skills or enrich your knowledge reserve, you are welcome to follow the industry information channel.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.