Shulou(Shulou.com)06/01 Report--

This article introduces you how to build BeeGFS development environment, the content is very detailed, interested friends can refer to, hope to be helpful to you.

Install the package

My test environment uses CentOS 7 and installs the V6 version of BeeGFS.

Install using the downloaded RPM package:

$rpm-ivh beegfs_rpm/*.rpm$ rpm-qa | grep beebeegfs-meta-6.18-el7.x86_64beegfs-admon-6.18-el7.x86_64beegfs-utils-6.18-el7.x86_64beegfs-helperd-6.18-el7.x86_64beegfs-storage-6.18-el7.x86_64beegfs-opentk-lib-6.18-el7.x86_64beegfs-client-6.18-el7.noarchbeegfs-common-6.18-el7.noarchbeegfs-mgmtd-6.18-el7.x86_64

Install using Yum:

$wget https://www.beegfs.io/release/beegfs_6/dists/beegfs-rhel7.repo-O / etc/yum.repos.d/beegfs-rhel7.repo$ yum install-y beegfs-mgmtd beegfs-meta libbeegfs-ib beegfs-storage libbeegfs-ib beegfs-client beegfs-helperd beegfs-utils beegfs-admon$ yum install-y mdadm initialization configuration initialization management service sacd01 $rm-rfv / beegfs_mgmtdsacd01 $/ opt/beegfs/sbin/beegfs-setup-mgmtd-p / beegfs_mgmtdPreparing storage directory: / beegfs_mgmtd * Creating format.conf file...Updating config file: / etc/beegfs/beegfs-mgmtd.conf * Setting storage directory in config file... * Disabling usage of uninitialized storage directory in config file...All done. Initialize metadata service

Select 2 and 3 for the ID of the two metadata nodes:

Sacd01 $rm / beegfs_meta/meta*-rfsacd01 $/ opt/beegfs/sbin/beegfs-setup-meta-p / beegfs_meta/meta1-s 2-m 172.29.39.125Preparing storage directory: / beegfs_meta/meta1 * Creating format.conf file... * Creating server numeric ID file: / beegfs_meta/meta1/nodeNumIDUpdating config file: / etc/beegfs/beegfs-meta.conf * Setting management host: 172.29.39.125 * Setting storage directory in config file... * Disabling usage of uninitialized storage directory in config file... * Setting usage of extended attributes to: trueAll done.sacd02 $rm / beegfs_meta/meta*-rfsacd02 $/ opt/beegfs/sbin/beegfs-setup-meta-p / beegfs_meta/meta2-s 3-m 172.29.39.125Preparing storage directory: / beegfs_meta/meta2 * Creating format.conf file... * Creating server numeric ID file: / beegfs_meta/meta2/nodeNumIDUpdating config file: / etc/beegfs/beegfs-meta.conf * Setting management host: 172.29.39.125 * Setting storage directory in config file... * Disabling usage of uninitialized storage directory in config file... * Setting usage of extended attributes to: trueAll done. Initialize the storage service sacd01 $rm / beegfs_data/data*-rfsacd01 $/ opt/beegfs/sbin/beegfs-setup-storage-p / beegfs_data/data1-s 4-I 401-m 172.29.39.125Preparing storage target directory: / beegfs_data/data1 * Creating format.conf file... * Creating chunks directory... * Creating buddymir directory... * Creating target numeric ID file: / beegfs_data/data1/targetNumID * Creating server numeric ID file: / beegfs_data/data1/nodeNumIDUpdating config file: / etc/beegfs/beegfs-storage.conf * Setting management host: 172.29.39.125 * Appending to target directory list in config file... * WARNING: Skipping append. Target directory seems to be included in current targets list already. ('- f 'disables this check.) * Disabling usage of uninitialized storage targets in config file...All done.sacd01 $/ opt/beegfs/sbin/beegfs-setup-storage-p / beegfs_data/data2-s 4-I 402-m 172.29.39.125Preparing storage target directory: / beegfs_data/data2 * Creating format.conf file... * Creating chunks directory... * Creating buddymir directory... * Creating target numeric ID file: / beegfs_data/data2/targetNumID * Creating server numeric ID file: / beegfs_data/data2/nodeNumIDUpdating config file: / etc/beegfs/beegfs-storage.conf * Setting management host: 172.29.39.125 * Appending to target directory list in config file... * WARNING: Skipping append. Target directory seems to be included in current targets list already. ('- f 'disables this check.) * Disabling usage of uninitialized storage targets in config file...All done.sacd02 $rm / beegfs_data/data*-rfsacd02 $/ opt/beegfs/sbin/beegfs-setup-storage-p / beegfs_data/data3-s 5-I 501-m 172.29.39.125Preparing storage target directory: / beegfs_data/data3 * Creating format.conf file... * Creating chunks directory... * Creating buddymir directory... * Creating target numeric ID file: / beegfs_data/data3/targetNumID * Creating server numeric ID file: / beegfs_data/data3/nodeNumIDUpdating config file: / etc/beegfs/beegfs-storage.conf * Setting management host: 172.29.39.125 * Appending to target directory list in config file... * WARNING: Skipping append. Target directory seems to be included in current targets list already. ('- f 'disables this check.) * Disabling usage of uninitialized storage targets in config file...All done.sacd02 $/ opt/beegfs/sbin/beegfs-setup-storage-p / beegfs_data/data4-s 5-I 502-m 172.29.39.125Preparing storage target directory: / beegfs_data/data4 * Creating format.conf file... * Creating chunks directory... * Creating buddymir directory... * Creating target numeric ID file: / beegfs_data/data4/targetNumID * Creating server numeric ID file: / beegfs_data/data4/nodeNumIDUpdating config file: / etc/beegfs/beegfs-storage.conf * Setting management host: 172.29.39.125 * Appending to target directory list in config file... * WARNING: Skipping append. Target directory seems to be included in current targets list already. ('- f 'disables this check.) * Disabling usage of uninitialized storage targets in config file...All done. Initialize the client service sacd03 $/ opt/beegfs/sbin/beegfs-setup-client-m 172.29.39.125Updating config file: / etc/beegfs/beegfs-client.conf * Setting management host: 172.29.39.125All done. Configure the service network interface

Let the entire storage cluster communicate using the specified network port:

Sacd01 $echo bond1.1039 > / etc/beegfs/network/beegfs-network-interface.confsacd02 $echo bond1.1039 > / etc/beegfs/network/beegfs-network-interface.confsacd03 $echo bond1.1039 > / etc/beegfs/network/beegfs-network-interface.conf# Managementsacd01 $vi / etc/beegfs/beegfs-mgmtd.confconnInterfacesFile = / etc/beegfs/network/beegfs-network-interface.conf# Metadatasacd01 $vi / etc/beegfs/beegfs-meta.confconnInterfacesFile = / etc/beegfs/network/beegfs-network-interface.confsacd02 $vi / etc/beegfs/beegfs-meta.confconnInterfacesFile = / etc/beegfs/network/beegfs-network-interface.conf# Storagesacd01 $vi / etc/beegfs/beegfs-storage.confconnInterfacesFile = / etc/beegfs/network/beegfs-network-interface.confsacd02 $vi vi / etc/beegfs/beegfs-storage.confconnInterfacesFile = / Etc/beegfs/network/beegfs-network-interface.conf# Clientsacd03 $vi / etc/beegfs/beegfs-client.confconnInterfacesFile = / etc/beegfs/network/beegfs-network-interface.conf startup service (except clients) sacd01 $systemctl start beegfs-mgmtdsacd01 $systemctl start beegfs-metasacd02 $systemctl start beegfs-metasacd01 $systemctl start beegfs-storagesacd02 $systemctl start beegfs-storagesacd03 $beegfs-ctl-- listtargets-- nodetype=metadata-- stateTargetID Reachability Consistency NodeID= = = 2 Online Good 2 3 Online Good 3sacd03 $beegfs-ctl-- listtargets-- nodetype=storage-- stateTargetID Reachability Consistency NodeID= 401 Online Good 4402 Online Good 4501 Online Good 5502 Online Good 5 configure metadata service mirror sacd03 $beegfs-ctl-- addmirrorgroup-- automatic-- nodetype=metaNew mirror groups:BuddyGroupID Node type Node= 1 primary 2 @ beegfs-meta sacd01 [ID: 2] secondary 3 @ beegfs-meta sacd02 [ID: 3] Mirror buddy group successfully set: groupID 1-> target IDs 2 3sacd03 $beegfs-ctl-- listmirrorgroups-- nodetype=meta BuddyGroupID PrimaryNodeID SecondaryNodeID = 12 3sacd03 $beegfs-ctl-- mirrormdsacd01 $systemctl restart beegfs-metasacd02 $systemctl restart beegfs-meta configure and start the client service sacd03 $vi / etc/beegfs/beegfs-client.conflogClientID = truelogLevel = 5#sysMountSanityCheckMS = 11000sysMountSanityCheckMS = 0sacd03 $systemctl start beegfs-helperd & & systemctl start beegfs-clientsacd03 $systemctl status beegfs-helperd & & systemctl status beegfs-clientsacd01 $systemctl enable beegfs-mgmtd beegfs-meta beegfs-storagesacd02 $systemctl enable beegfs-meta beegfs-storagesacd03 $systemctl enable beegfs-client View the network port used by the service $beegfs-ctl-- listnodes-- nodetype=management-- nicdetailssacd01 [ID: 1] Ports: UDP: 8008 TCP: 8008 Interfaces: + bond1.1039 [ip addr: 172.29.39.125; type: TCP] Number of nodes: 1$ beegfs-ctl-- listnodes-- nodetype=metadata-- nicdetailssacd01 [ID: 2] Ports: UDP: 8005; TCP: 8005 Interfaces: + bond1.1039 [ip addr: 172.29.39.125; type: TCP] sacd02 [ID: 3] Ports: UDP: 8005; TCP: 8005 Interfaces: + bond1.1039 [ip addr: 172.29.39.126 Type: TCP] Number of nodes: 2Root: 2$ beegfs-ctl-- listnodes-- nodetype=storage-- nicdetailssacd01 [ID: 4] Ports: UDP: 8003; TCP: 8003 Interfaces: + bond1.1039 [ip addr: 172.29.39.125; type: TCP] sacd02 [ID: 5] Ports: UDP: 8003; TCP: 8003 Interfaces: + bond1.1039 [ip addr: 172.29.39.126 Type: TCP] Number of nodes: 2$ beegfs-ctl-- listnodes-- nodetype=client-- nicdetailsC26A-5CD18762-sacd03 [ID: 5] Ports: UDP: 8004; TCP: 0 Interfaces: + bond1.1039 [ip addr: 172.29.39.133; type: TCP] Number of nodes: 1 solve the kernel version problem

When starting the client service, the following error may occur:

$systemctl start beegfs-clientJob for beegfs-client.service failed because the control process exited with error code. See "systemctl status beegfs-client.service" and "journalctl-xe" for details.$ systemctl status beegfs-client.service-l ● beegfs-client.service-Start BeeGFS Client Loaded: loaded (/ usr/lib/systemd/system/beegfs-client.service; enabled; vendor preset: disabled) Active: failed (Result: exit-code) since Tue 2019-05-07 16:37:01 CST 17s ago Process: 134148 ExecStart=/etc/init.d/beegfs-client start (code=exited, status=1/FAILURE) Main PID: 134148 (code=exited Status=1/FAILURE) May 07 16:37:00 sacd03 systemd [1]: Starting Start BeeGFS Client...May 07 16:37:00 sacd03 beegfs-client [134148]: Starting BeeGFS Client:May 07 16:37:00 sacd03 beegfs-client [134148]:-Loading BeeGFS modulesMay 07 16:37:00 sacd03 beegfs-client [134148]:-BeeGFS module autobuildMay 07 16:37:01 sacd03 beegfs-client [134148]: Makefile:117: * Linux kernel build directory not found. Please check if the kernel module development packages are installed for the current kernel version. (RHEL: kernel-devel; SLES: linux-kernel-headers, kernel-source; Debian: linux-headers) Stop.May 07 16:37:01 sacd03 beegfs-client [134148]: make: * [auto_rebuild] Error 2May 07 16:37:01 sacd03 systemd [1]: beegfs-client.service: main process exited, code=exited, status=1/FAILUREMay 07 16:37:01 sacd03 systemd [1]: Failed to start Start BeeGFS Client.May 07 16:37:01 sacd03 systemd [1]: Unit beegfs-client.service entered failed state.May 07 16:37:01 sacd03 systemd [1]: beegfs-client.service failed.

This is caused by no kernel-devel installed and no kernel header files.

Or the following error:

$vi / var/log/beegfs-client.logMessage from syslogd@sacd03 at May 8 09:44:12. Dracut:dracut: creation of / boot/initramfs-3.10.0-957.12.1.el7.x86_64.tmp failed (5) May08 09:44:33 * df (63614) [C206-5CD22EA8-sacd03: FhgfsOps_getattr] > called. Path: /; EntryID: root (0) May08 09:44:33 * df (63614) [C206-5CD22EA8-sacd03: Messaging (RPC node)] > > Invalid mirror buddy group ID: 0 (5) May08 09:44:33 * df (63614) [C206-5CD22EA8-sacd03: FhgfsOps_statfs] > > called. Path: n 5CD22EA8-sacd03 a (no dentry); EntryID: n 5CD22EA8-sacd03 a (no inode) (1) May08 09:44:33 * df (63614) [C206-5CD22EA8-sacd03: Remoting (stat storage targets)] > No storage targets known. (5) May08 09:45:04 * df (65354) [C206-5CD22EA8-sacd03: FhgfsOps_getattr] > > called. Path: /; EntryID: root (0) May08 09:45:04 * df (65354) [C206-5CD22EA8-sacd03: Messaging (RPC node)] > > Invalid mirror buddy group ID: 0 (5) May08 09:45:04 * df (65354) [C206-5CD22EA8-sacd03: FhgfsOps_statfs] > > called. Path: n 5CD22EA8-sacd03 a (no dentry); EntryID: n No storage targets known.$ ls a (no inode) (1) May08 09:45:04 * df (65354) [C206-5CD22EA8-sacd03: Remoting (stat storage targets)] > > No storage targets known.$ ls / mnt/beegfs/ls: reading directory / mnt/beegfs/: Not ad irectory

This is caused by a Bug of BeeGFS 6.18 on the RHEL 3.10.0-862 kernel:

Installing kernel-3.10.0-693kernels and development kits on all nodes solves the problem:

$yum list kernelLoaded plugins: fastestmirror LangpacksLoading mirror speeds from cached hostfile * base: ap.stykers.moe * epel: mirrors.tuna.tsinghua.edu.cn * extras: ap.stykers.moe * updates: mirrors.tuna.tsinghua.edu.cnInstalled Packageskernel.x86_64 3.10.0-957.5.1.el7 @ updates Available Packageskernel.x86_64 3.10.0-957.12.1.el7 updates $ Wget ftp://ftp.pbone.net/mirror/ftp.scientificlinux.org/linux/scientific/7.0/x86_64/updates/security/kernel-3.10.0-693.el7.x86_64.rpm$ wget ftp://ftp.pbone.net/mirror/ftp.scientificlinux.org/linux/scientific/7.0/x86_64/updates/security/kernel-devel-3.10.0-693.el7.x86_64.rpm# wget ftp://ftp.pbone.net/mirror/ Ftp.redhat.com/pub/redhat/linux/enterprise/7Server/en/RHEV/SRPMS/kernel-3.10.0-693.el7.src.rpm$ rpm-ivh kernel*-- force$ grep "menuentry" / boot/grub2/grub.cfg | awk-F "'{print $2} 'CentOS Linux (3.10.0-693.el7.x86_64) 7 (Core) CentOS Linux (3.10.0-957.12.1.el7.x86_64) 7 (Core) CentOS Linux (0-rescue-e3a40c9c3add41e3b2eb72e341b52ec1) 7 (Core) $vi / etc/default/grubGRUB_DEFAULT= "CentOS Linux (3.10.0-693.el7.x86_64) 7 (Core)" $grub2-mkconfig-o / boot/grub2/grub.cfg$ reboot$ rpm-evh kernel-3.10.0-957.12.1.el7.x86_64 $grep "menuentry" / boot/grub2/grub.cfg | awk-F "'{print $2} 'CentOS Linux (3.10.0-693. El7.x86_64) 7 (Core) CentOS Linux (0-rescue-e3a40c9c3add41e3b2eb72e341b52ec1) 7 (Core) on how to build a BeeGFS development environment is here. I hope the above content can be of some help to you and learn more knowledge. If you think the article is good, you can share it for more people to see.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.