Shulou(Shulou.com)06/01 Report--

This article mainly shows you "how to deploy Ozone1.0", the content is easy to understand, clear, hope to help you solve your doubts, the following let the editor lead you to study and learn "how to deploy Ozone1.0" this article.

1. Background introduction

HDFS provides low-latency metadata access to clients by loading all file system metadata into Namenode memory. Because the metadata needs to be fully loaded into memory, the maximum number of files that a HDFS cluster can support is limited by JAVA heap memory, which is about 400 million files. Therefore, HDFS is suitable for clusters with a large number of large files (hundreds of megabytes). If there are a lot of small files in the cluster, the metadata access performance of HDFS will be affected. Ozone is a distributed, multi-copy object storage system, which is specially optimized for big data scenario. Ozone is mainly designed for extensibility, targeting object storage of more than a billion magnitude.

2. Component architecture

2.1 basic functions of Ozone Manager2.1.1

Ozone Manager (OM) manages the namespace of Ozone, providing all new, update and delete operations for Volume, Bucket, and Key.

Ozone Manager communicates only with Ozone Client and Storage Container Manager, not directly with Datanode.

When writing data to Ozone, you need to request a block from OM, and OM will return a block and record the relevant information. When you want to read that file, you also need to get the address of that block through OM.

OM allows users to manage keys under volumes and buckets, both of which are part of the namespace and are also managed by OM.

Each volume is the root of a separate namespace under OM, unlike HDFS, which provides a file system with a single root directory.

Compared to the single-root tree structure in HDFS, the namespace of Ozone is a collection of volumes, or can be thought of as a forest, so it is very easy to deploy multiple OM for extension.

2.1.2. Metadata management

OM maintains a list of volumes, buckets, and keys. It maintains a list of volumes for each user, a list of buckets for each volume, and a list of keys for each bucket.

2.1.3. Single point and high availability

A single point of OM persists metadata through RocksDB

The underlying layer of HA schema implements the HA of metadata by extending the number of copies of metadata through Ratis (which implements the Raft protocol).

2.1.4. Key write

2.1.5. Key read

2.2. Storage Container Manager

SCM is the leader node of Ozone's block space management, and its main responsibility is to create and manage container,container, which is the main replication unit of Ozone.

2.2.1. Cluster management

SCM is responsible for creating the Ozone cluster, and when the cluster is initialized through init, SCM creates the cluster identity and root certificate required for certificate authorization. SCM manages the life cycle of the data node datanode in the cluster.

2.2.2. Service security management

SCM's certificate authority issues an identity certificate for each service in the cluster, which makes it easier for the service to start mTLS (two-way authentication) at the network layer, and the underlying token of the block also depends on this base certificate.

2.2.3. Block management

SCM is a block manager that allocates blocks and assigns them to data nodes, and clients can read and write to these blocks directly.

2.2.4. Copy management

SCM keeps track of all block copies, and if the data node or disk is damaged, SCM detects and instructs the data node to replicate the lost block to ensure high availability.

2.3 Containers

Block is a block object that actually stores the user's data and is not managed by SCM.

Container is a logical concept, which is a collection of Block that are not related to each other.

In Ozone, data is replicated at the granularity of Container, with a default size of 5G (ozone.scm.container.size), managed by the SCM service.

Pipeline to ensure the number of copies desired by the Container implementation. At present, SCM supports two Pipeline modes to implement multi-copy, single-copy Standalone mode and three-copy Ratis mode (ozone.replication).

When Ozone Manager requests a new block allocation from SCM, SCM will identify the appropriate container and generate a block id containing ContainerId+LocalId. The Datanode,Datanode that the client connects to the storage container can manage separate blocks according to LocalId.

Container has two states, OPEN and CLOSED:

When container is created, it starts the Open state, and when it is full (the default is 5G), it closes and becomes the state of CLOSED, and then it cannot be modified. | | OPEN | CLOSED | |:-|:-| | variable | immutable | | copy via ratis | copy using asynchronous container copy | | Raft leader node is used for data reading and writing | all nodes are used for reading |

2.4 Datanodes

All the data is stored, the client writes the data by block, and the data node aggregates these block into the storage container, and the container contains the data blocks written by the user and the metadata of these blocks.

The Ozone container storage container is a self-contained super block that contains a series of Block and disk files that store the actual data, which is the default storage container format.

When the client wants to read data from Ozone, the client sends the name of the data file to OM, OM will return a list of blocks that make up the file, an Ozone block contains a container ID and a local ID,container ID to let the client find the location of the container, the relevant information about the container is provided by SCM, and in most cases, the container location is cached by Ozone Manager and returned together with the Ozone block.

When the client finds the container, that is, which data nodes contain the container, the client connects to the data node and reads the data stream specified by the container ID:Local ID. In other words, the Local ID is used as the index of the container to read the desired data stream from the container.

Location of the container found

How does SCM get the location of the container? This is very similar to existing HDFS. Data nodes regularly send container reports similar to block reports, which are much more concise than block reports. for example, for a cluster with a storage capacity of 196 TB, Ozone will have about 40, 000 containers, compared with HDFS's 1.5 million blocks, the number of block reports has been reduced to 1/40.

This indirect management method greatly improves the scalability of Ozone, because the block data that SCM needs to deal with is greatly reduced, and naming service (OM), as a unique service principal, is of great significance for extending Ozone.

2.5. Recon

Monitoring page of Ozone

2.6 S3G

S3 compatible REST gateway of Ozone

3. Basic introduction 3.1. Component architecture

The namespace of Ozone consists of volume, while volume is also used for storage account management.

Ozone Manager (OM) manages namespaces, Storage Container Manager (SCM) manages the underlying data, and Recon is the management interface for Ozone.

3.2. Logical architecture

Logical storage architecture of Ozone data

Volumes (volume): similar to user accounts, only administrators can create volumes and do quota management to store buckets. A volume can contain multiple buckets.

Buckets (bucket): similar to directories, users can create any number of buckets under their own volumes. A bucket can contain countless keys,buckets but not other buckets, and they do not interfere with each other.

Keys (key): similar to a file, located under bucket, used to read and write data.

3.3. Port description service port parameters remarks Datanode9858dfs.container.ratis.ipcDatanode node Datanode9859dfs.container.ipcDatanode node SCM9860ozone.scm.client.address

SCM9861ozone.scm.datanode.addressscm and datanode communication port OM9862ozone.om.address

SCM9863ozone.scm.block.client.port

OM9874ozone.om.http-addressOM web page SCM9876ozone.scm.http-addressSCM web page S3G9878ozone.s3g.http-addressS3GatewayDatanode9882hdds.datanode.http-addressDatanode node Recon9888ozone.recon.http-addressRecon web page Recon9891ozone.recon.address is none by default, Datanode node must be equipped with parameters, so that Recon can monitor Datanode service 4. 5. Installation and deployment

OZONE and HDFS should not be deployed on the same node, because the environment variables of OZONE will be shared with HADOOP, and the service startup will conflict. Scripts related to environment variables can be modified to make changes.

4.1. Basic environment 4.1.1. Configure / etc/hosts# cat / etc/hosts127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4::1 localhost localhost.localdomain localhost6 localhost6.localdomain6172.16.40.21 ozom-1172.16.40.22 ozom-2172.16.40.23 ozom-3172.16.40.11 ozdn-1172.16.40.12 ozdn-2172.16.40.13 ozdn-34.1.2. Turn off the firewall

# systemctl stop firewalld & & systemctl disable firewalld

4.1.3. Close selinux

# setenforce 0 & & sed-I "s / ^ Selinus = enforcing/SELINUX=disabled/g" / etc/selinux/config

4.1.4. Time synchronization

Ntpd

4.1.5. Create a user

# useradd hadoop

4.1.6. Configure mutual trust

Slightly

4.2. Single point deployment of HostnameIpServiceozom-1172.16.40.21OM, SCM, DataNodeozom-2172.16.40.22DataNodeozom-3172.16.40.23DataNodeozdn-1172.16.40.11DataNodeozdn-2172.16.40.12DataNodeozdn-3172.16.40.13DataNode4.2.1. Create a directory

Create a data directory for Datanode and mount it to disk

# mkdir-p / data/disk {1.. 2} # chown-R hadoop:hadoop / data/disk {1.. 2}

Create ozone related directories

# mkdir-p / opt/dlw/ {conf,soft,logs,metadata,tmp} # chown-R hadoop:hadoop / opt/dlw$ mkdir-p / opt/dlw/logs/ozone$ mkdir-p / opt/dlw/tmp/run$ mkdir-p / opt/dlw/metadata/ozone/ {om,node,recon,ratis,scm} 4.2.2. Environment variable 1, configure java

All nodes need to be configured

Configure ozone

Export OZONE_HOME=/opt/dlw/soft/ozoneexport PATH=$OZONE_HOME/sbin:$OZONE_HOME/bin:$PATHexport HADOOP_OPTS= "- XX:ParallelGCThreads=8-XX:+UseConcMarkSweepGC-XX:CMSInitiatingOccupancyFraction=75-XX:+CMSParallelRemarkEnabled" export HADOOP_PID_DIR=/opt/dlw/tmp/runexport HADOOP_LOG_DIR=/opt/dlw/logs/hadoop4.2.3. Copy softwar

Binary package downloaded by Ozone, decompressed directly and copied to all nodes

$tar-zxvf hadoop-ozone-1.0.0.tar.gz-C / opt/dlw/soft/$ cd / opt/dlw/soft/$ mv ozone-1.0.0 ozone4.2.4. Modify configuration

Generate configuration

$ozone genconf $OZONE_HOME/etc/hadoop

Modify configuration

$vim $OZONE_HOME/etc/hadoop/ozone-site.xml ozone.om.address ozom-1 ozone.om.http-address ozom-1:9874 ozone.om.db.dirs / opt/dlw/metadata/ozone/om ozone.metadata.dirs / opt/dlw/metadata/ozone ozone.om.ratis.enable False ozone.om.service.ids omdlw001 ozone.om.nodes.omdlw001 om1 Om2 Om3 ozone.om.address.omdlw001.om1 ozom-1 ozone.om.address.omdlw001.om2 ozom-2 ozone.om.address.omdlw1.om3 ozom-3 ozone.scm.client.address ozom-1 ozone.scm.names ozom-1 ozone.scm.container.size 5GB ozone.scm.db.dirs / opt/dlw/metadata/ozone/scm ozone.scm.pipeline.owner.container.count 3 ozone.scm.datanode.id.dir / opt/dlw/metadata/ozone/node ozone.scm.block.client.address ozom-1 ozone.scm.datanode.address Ozom-1 hdds.datanode.dir / data/disk1 / data/disk2 ozone.recon.db.dir / opt/dlw/metadata/ozone/recon ozone.recon.address ozom-1:9891 recon.om.snapshot.task.interval.delay 1m dfs.container.ratis.datanode.storage.dir / opt/dlw/metadata/ozone/ratis $vim $OZONE_HOME/etc/hadoop/workersozom-1ozom-2ozom -3ozdn-1ozdn-2ozdn-34.2.5. Start the service

Scm

$ozone scm-init$ ozone-daemon start scm

Om

$ozone om-init$ ozone-daemon start om

Datanode-all datanode nodes

$ozone-daemon start datanode

Recon

$ozone-daemon start recon

S3g

$ozone-daemon start s3g4.2.6. Verification of school inspection $jps6721 ReconServer7589 Gateway4934 StorageContainerManagerStarter5479 OzoneManagerStarter5705 HddsDatanodeService7662 Jps

Scm

Om

S3g

Recon

4.3. Highly available deployment of HostnameIpServiceozom-1172.16.40.21OM, SCM, DataNodeozom-2172.16.40.22OMozom-3172.16.40.23OMozdn-1172.16.40.11DataNodeozdn-2172.16.40.12DataNodeozdn-3172.16.40.13DataNode

High availability currently only OM services support HA,SCM but not: a single point of OM persists metadata through RocksDB; the underlying layer of high availability mode implements metadata HA by extending the number of copies of metadata through Ratis (implementing Raft protocol).

Modify the configuration to enable the HA mode of OM

Ozone.om.ratis.enable true

An ozone-site.xml supports multiple Ozone HA configurations, and in order to choose between available HA clusters, each cluster needs a logical name that can be resolved to the IP address (or domain name) of OM.

Ozone.om.service.ids cluster1,cluster2

There is only one cluster here. Customize a cluster name omdlw001.

Ozone.om.service.ids omdlw001

For this cluster, all its om can be defined and each om node can be configured.

Ozone.om.nodes.omdlw001 om1,om2,om3 ozone.om.address.omdlw001.om1 ozom-1 ozone.om.address.omdlw001.om2 ozom-2 ozone.om.address.omdlw001.om3 ozom-3

Start the om service on all three om nodes, but it must be noted that the highly available om service cannot be on the same node as the datanode service, because both use ratis will conflict.

$ozone om-- init$ ozone-- daemon start om above is all the content of this article "how to deploy Ozone1.0". Thank you for reading! I believe we all have a certain understanding, hope to share the content to help you, if you want to learn more knowledge, welcome to follow the industry information channel!

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.