Shulou(Shulou.com)05/31 Report--

This article introduces what is the internal structure and function of Cloudera Manager. The content is very detailed. Interested friends can use it for reference. I hope it will be helpful to you.

1. Related catalogue

/ var/log/cloudera-scm-installer: installation log directory.

/ var/log/*: related log files (related services and CM).

/ usr/share/cmf/: the program installation directory.

/ usr/lib64/cmf/: Agent program code.

/ var/lib/cloudera-scm-server-db/data: embedded database directory.

/ usr/bin/postgres: embedded database program.

/ etc/cloudera-scm-agent/: the configuration directory for agent.

/ etc/cloudera-scm-server/: the configuration directory for server.

/ opt/cloudera/parcels/: installation directory for Hadoop related services.

/ opt/cloudera/parcel-repo/: downloaded service package data in parcels format.

/ opt/cloudera/parcel-cache/: the downloaded service package caches data.

/ etc/hadoop/*: client profile directory.

two。 Configuration

Hadoop profile

The configuration file is placed in the / var/run/cloudera-scm-agent/process/ directory. For example: / var/run/cloudera-scm-agent/process/193-hdfs-NAMENODE/core-site.xml. These configuration files are generated when the corresponding service (such as HDFS) is started through Cloudera Manager, and the contents are obtained from the database (that is, parameters configured through the interface).

Changing the configuration on the CM interface is not immediately reflected in the configuration file, and the information is stored in the database until the next time the service is restarted. And a new configuration file is generated each time it starts.

The main database of CM Server is configs, which is configured for placement in the scm base. It contains the configuration information of the service, and each configuration change will add all the configuration contents of the current page to the database to save the configuration modification history.

The scm database is configured to be accessible only from localhost. If you need to connect to the database externally, modify the vim / var/lib/cloudera-scm-server-db/data/pg_hba.conf file, and then restart the database. The user running the database is cloudera-scm.

View configuration content

Directly query the contents of the configs data table in the scm database.

Visit REST API: http://hostname:7180/api/v5/cm/deployment and return the deployment configuration information in JSON format.

Configure the generation mode

CM generates a separate configuration directory (file) for each service process. All configurations are generated by querying the database on the server (because the scm database can only be accessed under localhost) to generate configuration files, and then agent downloads the zip package containing the configuration files through the network and decompresses them locally to the specified directory.

Configuration modification

CM is predefined for configurations that need to be modified, and for configurations that are not predefined, it is configured by using xml configuration fragments in advanced configuration items. The configuration file under / etc/hadoop/ is the client configuration, which can be generated by deploying the client in CM.

3. Database

The main database of Cloudera manager is scm, which stores the information needed for Cloudera manager operation: configuration, host, user, etc.

4. CM structure

CM is divided into two parts: Server and Agent and database (embedded Postgresql with changes). It mainly does three events:

Manage and monitor cluster hosts.

Unify the management configuration.

Manage and maintain Hadoop platform system.

Agent is responsible for executing the commands sent by the server for the client, which is generally implemented by invoking the corresponding service shell script using python. The Server side is a Java REST service, which provides the REST API,Web management side to call the server side functions through REST API, and the Web interface uses rich client technology (Knockout).

The body on the Server side is implemented using Java.

The Agent body uses Python, and the service is started by calling the corresponding shell script. If the startup fails, the startup script will be called 4 times.

Agent and Server keep their heartbeats, using the Thrift RPC framework.

5. Upgrade

In CM, you can upgrade related services through the interface wizard. The upgrade process is three steps:

Download the service package.

Distribute the downloaded service packages to the managed machines in the cluster.

Install the service package and use soft links to link the service program directory to the newly installed package directory.

6. Unloading

Sudo / usr/share/cmf/uninstall-scm-express.sh, and then delete the / var/lib/cloudera-scm-server-db/ directory, otherwise the next installation may not be successful.

7. Enable postgresql remote access

The CM embedded database is configured to be accessible only from localhost. If you need to view the data from outside, the data modifies the vim / var/lib/cloudera-scm-server-db/data/pg_hba.conf file, and then restarts the database. The user running the database is cloudera-scm.

On the internal structure and function of Cloudera Manager what is shared here, I hope that the above content can be of some help to you, can learn more knowledge. If you think the article is good, you can share it for more people to see.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.