Shulou(Shulou.com)06/02 Report--

HDFS Enable SSD Storage 1. Configure HDFS Data Nodes

[DISK]/hadoop/hdfs/data,[SSD]/hadoop/hdfs/ssd

Mount ssd disks on all data nodes to the path/hadoop/hdfs/ssd

The requested URL/hadoop/hdfs/ssd/was not found on this server.

drwxr-x--- 3 hdfs hadoop 4096 Oct 17 19:10 /hadoop/hdfs/ssd

Restart data node

2. Create hdfs path using ssd hdfs dfs -mkdir /ssd3. Set storage policy for/ssd: ALL_SSDHdfs storagepolicies -setStoragePolicy -path /ssd-policy ALL_SSDHAWQ Create Tablespace 1. Create a file space configuration file and execute $hawq filespace -o tpc_h_config filespace:fs_tpc_hfsreplica:3dfs_url::mycluster/ssd/fs_tpc_h2. Create HDFS directory $hdfs dfs -mkdir /ssd$hdfs dfs -chown gpadmin:gpadmin /ssd$hdfs dfs -ls /3. Create file space $hawq filespace -c tpc_h_config

4. create tablespace

psql

create tablespace ts_tpc_h filespace fs_tpc_h;

5. View all current tablespaces SELECT spcname AS tblspc, fsname AS filespc, fsedbid AS seg_dbid, fselocation AS datadir FROM pg_tablespace pgts, pg_filespace pgfs, pg_filespace_entry pgfse WHERE pgts.spcfsoid=pgfse.fsefsoid AND pgfse.fsefsoid=pgfs.oid ORDER BY tblspc, seg_dbid;

HAWQ Creation Table 1. Create table region (r_regionkey integer,r_name char(25),r_comment varchar(152),r_extra char(1))with (appendonly=true,orientation=parquet,compresstype=snappy) tablespace ts_tpc_hdistributed by (r_regionkey) ; 2. View the table space used by the table select c.relname, d.dat2tablespace_id, d.oid database_id, c.relfilenode table_id from pg_database d, pg_class c, pg_namespace n where c.relnamespace = n.oid and d.datname = current_database() and n.nspname = 'qbyps' and c.relname = 'p';SELECT pgfs.oid fs_id,pgts.oid ts_id, spcname AS tblspc, fsname AS filespc, fsedbid AS seg_dbid, fselocation AS datadir FROM pg_tablespace pgts, pg_filespace pgfs, pg_filespace_entry pgfse WHERE pgts.spcfsoid=pgfse.fsefsoid AND pgfse.fsefsoid=pgfs.oid ORDER BY tblspc, seg_dbid; maintenance

HAWQ uses the API of libhdfs3.so to access HDFS and currently does not support storage policies. Therefore, maintenance of the written data is required.

hdfs mover -p /ssd/fs_tpc_h Appendix:

storage policy command

List all storage policies

hdfs storagepolicies -listPolicies

Set Storage Policy

hdfs storagepolicies -setStoragePolicy -path -policy

for example

hdfs storagepolicies -setStoragePolicy -path /tmp -policy ALL_SSD

Cancel Storage Policy

hdfs storagepolicies -unsetStoragePolicy -path

After that, the directory or file is subject to its superior directory. If it is the root directory, it is HOT.

Get access policy

hdfs storagepolicies -getStoragePolicy -path

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.