Shulou(Shulou.com)05/31 Report--

This article will explain in detail how to build a BeeGFS development environment for you. The editor thinks it is very practical, so I share it with you as a reference. I hope you can get something after reading this article.

BeeGFS is often used for distributed file storage in high-performance computing. Its support for large and small files is much better than most other file systems. The detailed process of building is described here. A total of three servers are used, one disk is divided into two zones (for metadata and data storage respectively), and two network cards (one management network and one storage network) are used.

Network configuration Network assignment

The management network is used for SSH login and other functions, all are gigabit ports, the storage network is used for data transmission, and double 10 Gigabit ports are bound by Mode 6.

Hostname Management Network Storage Network Service deployment instructions sacd01172.29.201.125172.29.39.125mgmt,meta (2), storage (401402), clientsacd02172.29.201.126172.29.39.126meta (3), storage (501502), clientsacd03172.29.201.133172.29.39.113client

The following configuration takes sacd01 as an example.

DNS is bound to hostname sacd01 $cat / etc/hosts172.29.201.125 sacd01172.29.201.126 sacd02172.29.201.133 sacd03sacd01 $hostnamesacd01sacd01 $cat / etc/hostname sacd01 Nic

Check the binding of the Nic:

Sacd01 $eths=$ (lspci | grep Ethernet); for bond in `ls-d / sys/class/net/bond [0-9] `; do nics=$ (ls-d $bond/slave* | awk-Fv bond=$bond'$1~bond {print $2}'| tr-d ":"); for nic in $nics; do addr=$ (ethtool-I $nic | awk'$1 ~ "bus-info" {print $2}'| awk-F:'{printf $2 ":" $3}')) Echo-e $(echo $bond | awk-F /'{print $5}') "\ t" $nic "\ t" $addr "\ t" $(echo-e "$eths" | grep $addr | awk-F:'{printf $3}'); done Donebond0 enp26s0f2 1a:00.2 Intel Corporation Ethernet Connection X722 for 1GbE (rev 09) bond1 enp95s0f0 5f:00.0 Intel Corporation Ethernet Controller X710 for 10GbE SFP+ (rev 01) bond1 enp95s0f1 5f:00.1 Intel Corporation Ethernet Controller X710 for 10GbE SFP+ (rev 01) sacd01 $ip a | grep bond16: enp95s0f0: mtu 1600 qdisc mq master bond1 state UP group default qlen 10007: enp95s0f1: mtu 1600 qdisc mq master bond1 state UP group default qlen 10008: bond1: mtu 1600 qdisc noqueue state UP group default qlen 10009: bond1.1039@bond1: mtu 1600 qdisc Noqueue state UP group default qlen 1000 inet 172.29.39.125 Ethernet 24 brd 172.29.39.255 scope global noprefixroute bond1.1039sacd01 $ifconfig bond1.1039bond1.1039: flags=4163 mtu 1600 inet 172.29.39.125 netmask 255.255.255.0 broadcast 172.29.39.255 inet6 fe80::6e92:bfff:fe67:9bdc prefixlen 64 scopeid 0x20 ether 6c:92:bf:67:9b:dc txqueuelen 1000 (Ethernet) RX packets 10 bytes .0B) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 16 bytes 992 (992.0 B) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0sacd01 $cat / sys/class/net/bond1/bonding/mode balance-alb 6

View the Nic binding configuration:

Sacd01 $cat / etc/sysconfig/network-scripts/ifcfg-enp95s0f0TYPE=EthernetNAME= "enp95s0f0" DEVICE=enp95s0f0ONBOOT=yesBOOTPROTO=noneMASTER=bond1SLAVE=yessacd01 $cat / etc/sysconfig/network-scripts/ifcfg-enp95s0f1TYPE=EthernetNAME= "enp95s0f1" DEVICE=enp95s0f1ONBOOT=yesBOOTPROTO=noneMASTER=bond1SLAVE=yessacd01 $cat / etc/sysconfig/network-scripts/ifcfg-bond1DEVICE=bond1BOOTPROTO=noneONBOOT=yesTYPE=BondNM_CONTROLED=noIPV6INIT=noUSERCTL=noIPV6INIT=noBONDING_MASTER=yesBONDING_OPTS= "miimon=100 mode=6" # BONDING_OPTS= "xmit_hash_policy=layer2+3 mode=4 miimon=100" MTU=1600sacd01 $cat / etc/sysconfig/network-scripts/ifcfg-bond1.1039 DEVICE=bond1.1039ONBOOT=yesBOOTPROTO=staticIPADDR=172.29.39.125NETMASK=255.255.255.0VLAN=yes Storage configuration disk Information sacd01 $lsblkNAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTsda 8:0 0 222.6G 0 disk ├─ sda1 8:1 0200M 0 part / boot └─ sda2 8:2 0222.4G 0 part part centos-root 253 centos-root 0 0 122.4G 0 lvm / └─ centos-home 253 Fringe 1 0 100G 0 lvm / homesdb 8:16 0 7.3T 0 disksacd01 $ Lsblk-d | awk'$1 echo ~ "sda" & & ($1 ~ "^ sd" | | $1 ~ "^ nv") {print $1}'| xargs-I% sh-c 'echo-e "/ dev/%\ t" Smartctl-I / dev/% | grep-e "Product"-e "Device Model"-e "Model Number"-e "Rotation Rate"-e "Capacity" | awk-F\: "{print $2}"'| awk-F: 'NF==1 {printf "\ n" $1} NF==2 {sub (/ ^ [] + /, ", $2) Printf $2 "\ t"} END {printf "\ n"}'/ dev/sdb Logical Volume 8000999784448 bytes [8.00 TB] sacd01 $smartctl-I / dev/sdbsmartctl 6.5 2016-05-07 r4318 [x86_64-linux-3.10.0-693.el7.x86_64] (local build) Copyright (C) 2002-16, Bruce Allen, Christian Franke Www.smartmontools.org=== START OF INFORMATION SECTION = Vendor: LSIProduct: Logical VolumeRevision: 3000Compliance: SPC-4User Capacity: 8000999784448 bytes [8.00 TB] Logical block size: 512 bytesPhysical block size: 4096 bytesLogical Unit id: 0x600508e000000000fadfaab948205506Device type: diskLocal Time is: Wed May 8 09:56:05 2019 CSTSMART support is: Unavailable-device lacks SMART capability. Disk partition

There are two 2T disks:

$sgdisk / dev/sdb-- zap-all-- clear-- mbrtogpt$ sgdisk / dev/sdb-n 1V 0V 2T-c 1: "beegfs meta"-t 1V 8300-n 2V 0V V 2T-c 2: "beegfs data"-t 2:8300Setting nameplate partnum is 0REALLY setting nameplate setting ting nameplate partNum is 1REALLY setting operation has completed successfully.$ sgdisk / dev/sdb-pDisk / dev/sdb: 15626952704 sectors 7.3 TiBLogical sector size: 512 bytesDisk identifier (GUID): A3A34A32-C0FE-4133-BDDB-05A28E2F5455Partition table holds up to 128 entriesFirst usable sector is 34 Last usable sector is 15626952670Partitions will be aligned on 2048-sector boundariesTotal free space is 7037018045 sectors (3.3TiB) Number Start (sector) End (sector) Size Code Name 1 2048 4294969343 2.0 TiB 8300 beegfs meta 2 4294969344 8589936639 2.0 TiB 8300 beegfs data$ lsblkNAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTsda 8:0 0222.6G 0 disk ├─ sda1 8:1 0 200M 0 part / boot └─ sda2 8:2 0222.4G 0 part centos-root 253 lvm / └─ centos-home 253 └─ lvm 0 100G 0 lvm / homesdb 8:16 0 7.3T 0 disk ├─ sdb1 8:17 02T 0 part └─ sdb2 8:18 02T 0 part disk format

Metadata partitions use the EXT4 file system:

$mkfs.ext4-I 2048-I 512-J size=400-Odir_index,filetype / dev/sdb1mke2fs 1.42.9 (28-Dec-2013) Filesystem label=OS type: LinuxBlock size=4096 (log=2) Fragment size=4096 (log=2) Stride=0 blocks, Stripe width=0 blocks1073741824 inodes, 536870912 blocks26843545 blocks (5.00%) reserved for the super userFirst data block=0Maximum filesystem blocks=161061273632768 block groups16384 blocks per group, 16384 fragments per group32768 inodes per groupSuperblock backups stored on blocks: 16384,49152,81920,114688, 147456,409600,442368802816,1327104,2048000, 39813125619712102400,11943931808 39337984, 51200000, 107495424, 256000000, 275365888, 322486272Allocating group tables: doneWriting inode tables: done Creating journal (102400 blocks): doneWriting superblocks and filesystem accounting information: done

Storage partitions use the XFS file system:

$mkfs.xfs / dev/sdb2meta-data=/dev/sdb2 isize=512 agcount=4, agsize=134217728 blks = sectsz=4096 attr=2, projid32bit=1 = crc=1 finobt=0, sparse=0data = bsize=4096 blocks=536870912 Imaxpct=5 = sunit=0 swidth=0 blksnaming = version 2 bsize=4096 ascii-ci=0 ftype=1log = internal log bsize=4096 blocks=262144, version=2 = sectsz=4096 sunit=1 blks, lazy-count=1realtime = none extsz=4096 blocks=0 Rtextents=0 disk mount $mkdir-pv / beegfs_meta / beegfs_data$ mount / dev/sdb1 / beegfs_meta$ mount / dev/sdb2 / beegfs_data$ echo "/ dev/sdb1 / beegfs_meta xfs defaults 00" | tee-a / etc/fstab$ echo "/ dev/sdb2 / beegfs_data xfs defaults 00" | tee-a / etc/fstab$ df-h | grep beegfs/dev/sdb1 2.0T 33M 2.0T 1% / beegfs_meta/dev/sdb2 2.0T 33M 2.0T 1% / beegfs_data$ mount | grep beegfs/dev/sdb1 on / beegfs_meta type xfs (rw Relatime,attr2,inode64,noquota) / dev/sdb2 on / beegfs_data type xfs (rw,relatime,attr2,inode64,noquota) CPU configuration $lscpuArchitecture: x86_64CPU op-mode (s): 32-bit 64-bitByte Order: Little EndianCPU (s): 72On-line CPU (s) list: 0-71Thread (s) per core: 2Core (s) per socket: 18Socket (s): 2NUMA node (s): 2Vendor ID: GenuineIntelCPU family: 6Model: 85Model name: Intel (R) Xeon ( R) Gold 6140 CPU @ 2.30GHzStepping: 4CPU MHz: 1499.941CPU max MHz: 3700.0000CPU min MHz: 1000.0000BogoMIPS: 4600.00Virtualization: VT-xL1d cache: 32KL1i cache: 32KL2 cache: 1024KL3 cache: 25344KNUMA node0 CPU (s): 0-17meme 36 -53NUMA node1 CPU (s): 18-35 rdrand lahf_lm abm 3dnowprefetch epb cat_ 54-71Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 ds_cpl vmx smx est tm2 ssse3 fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch epb cat_ L3 cdp_l3 intel_pt tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm cqm mpx rdt_a avx512f avx512dq rdseed adx smap clflushopt clwb avx512cd avx512bw avx512vl xsaveopt xsavec xgetbv1 cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local dtherm ida arat pln pts hwp hwp_act_window hwp_epp hwp_pkg_req memory configuration $dmidecode-t memory | grep "Size: [0-9] *"-B7-A12 | grep-e "^ Handle"-e Size: "- e" Type: "- e" Speed: "- e" Manufacturer: "- e" Part Number "Handle 0x0038 DMI type 17, 40 bytes Size: 32 GB Type: DDR4 Speed: 2666 MT/s Manufacturer: Hynix Part Number: HMA84GR7AFR4N-VK Handle 0x003B, DMI type 17, 40 bytes Size: 32 GB Type: DDR4 Speed: 2666 MT/s Manufacturer: Hynix Part Number: HMA84GR7AFR4N-VK Handle 0x003E, DMI type 17 40 bytes Size: 32 GB Type: DDR4 Speed: 2666 MT/s Manufacturer: Hynix Part Number: HMA84GR7AFR4N-VK Handle 0x0043, DMI type 17, 40 bytes Size: 32 GB Type: DDR4 Speed: 2666 MT/s Manufacturer: Hynix Part Number: HMA84GR7AFR4N-VK Handle 0x0046, DMI type 17 40 bytes Size: 32 GB Type: DDR4 Speed: 2666 MT/s Manufacturer: Hynix Part Number: HMA84GR7AFR4N-VK Handle 0x0049, DMI type 17, 40 bytes Size: 32 GB Type: DDR4 Speed: 2666 MT/s Manufacturer: Hynix Part Number: HMA84GR7AFR4N-VK Handle 0x004E, DMI type 17 40 bytes Size: 32 GB Type: DDR4 Speed: 2666 MT/s Manufacturer: Hynix Part Number: HMA84GR7AFR4N-VK Handle 0x0051, DMI type 17, 40 bytes Size: 32 GB Type: DDR4 Speed: 2666 MT/s Manufacturer: Hynix Part Number: HMA84GR7AFR4N-VK Handle 0x0054, DMI type 17 40 bytes Size: 32 GB Type: DDR4 Speed: 2666 MT/s Manufacturer: Hynix Part Number: HMA84GR7AFR4N-VK Handle 0x0059, DMI type 17, 40 bytes Size: 32 GB Type: DDR4 Speed: 2666 MT/s Manufacturer: Hynix Part Number: HMA84GR7AFR4N-VK Handle 0x005C, DMI type 17 40 bytes Size: 32 GB Type: DDR4 Speed: 2666 MT/s Manufacturer: Hynix Part Number: HMA84GR7AFR4N-VK Handle 0x005F, DMI type 17 40 bytes Size: 32 GB Type: DDR4 Speed: 2666 MT/s Manufacturer: Hynix Part Number: HMA84GR7AFR4N-VK$ dmidecode-t memory | grep "Size: [0-9] *" | wc-L12 $free-h total used free shared buff/cache availableMem: 376G 6.8G 368G 10M 983M 368GSwap: 0B 0B 0B this is the end of the article on how to build a BeeGFS development environment. Hope that the above content can be helpful to you, so that you can learn more knowledge, if you think the article is good, please share it for more people to see.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.