Shulou(Shulou.com)06/01 Report--

This article will explain in detail how to build linux gfs system iscsi+GFS to achieve network storage. The editor thinks it is very practical, so I share it with you as a reference. I hope you can get something after reading this article.

Build gfs system by virtual machine

System environment: CentOS release 5.5-2.6.18-194.el5

Gfs Node 1 192.168.1.231 gfs1

Gfs Node 2 192.168.1.232 gfs2

Gfs node 3VR 192.168.1.233 gfs3

Iscsi-target storage device: 192.168.1.240 iscsi-storage (IP corresponding hostname)

GFS briefly states that there are two types:

1. Google file system: GFS is an extensible distributed file system implemented by GOOGLE, which is used for large, distributed applications that access large amounts of data. It runs on cheap ordinary hardware, but it can provide fault tolerance. It can provide high-performance services to a large number of users. To learn more, you can visit http://www.codechina.org/doc/google/gfs-paper/introduction.html

2. GFS (Global File System) of Redhat

GFS (Global File System) itself appears as a local file system. Multiple Linux machines share storage devices over the network, and each machine can treat the network shared disk as a local disk. If a machine writes to a file, the machine that later accesses the file will read the result of the write. Different scenarios can be deployed according to performance or scalability, or based on multiple principles of economy.

GFS main components, cluster volume management, lock management, cluster management, fencing and recovery, cluster configuration management.

This paper mainly introduces the GFS system of Redhat.

REDHAT CLUSTER SUITEWITH GFS:

RHCS (REDHAT CLUSTER SUITE) is a cluster toolset that can provide high performance, high reliability, load balancing and high availability. A cluster usually has two or more computers (called "nodes" or "members") to perform a task together.

Main components of RHCS:

Cluster architecture:

Provides a basic function that enables nodes to work together as clusters: profile management, membership management, lock management, and gate devices.

High availability * * Management:

Provide node failure transfer service, and transfer the service to another node when one node fails.

Cluster management tools:

Configure and manage Red Hat clusters through configuration and management tools.

Linux Virtual Server (LVS)

LVS provides a load balancing function based on IP, through which customer requests can be evenly distributed to cluster nodes through LVS.

Other Red Hat cluster components:

Cluster Logical Volume Manager (CLVM)

Provides logical volume management cluster storage.

Cluster Manager:

CMAN is a distributed cluster manager (DLM), which runs on each cluster node. CMAN provides a legal number of nodes (quorum) by monitoring cluster nodes. When more than half of the nodes in the cluster are active, the legal number of nodes is met, and the cluster continues to be available. When only half or less of the nodes are active, the legal number of nodes has not been reached. The entire cluster becomes unavailable at this point. CMAN determines the membership of each node by monitoring the nodes in the cluster. When the membership in the cluster changes, CMAN communicates with other components in the architecture to adjust accordingly.

DLM Lock Management:

Distributed lock manager runs on all cluster nodes. Lock management is a common infrastructure that provides a management mechanism for clusters to share cluster resources. GFS uses lock managers to synchronously access file system metadata, and CLVM uses lock managers to synchronously update data to LVM volumes and volume groups.

Assurance of data integrity:

In order to ensure the integrity of the data, RHCS uses the Fence device to cut off the I / O of the failed node from the shared storage. When CMAN determines that a node fails, it advertises the failed node (multicast) in the cluster structure, and the fenced process isolates the failed node to ensure that the failed node does not destroy the shared data.

REDHAT cluster configuration system:

Cluster configuration file: (/ etc/cluster/cluster.conf) is a XML file that describes the following cluster characteristics:

Cluster name: lists the cluster name, cluster profile version, and an isolation time when a new node joins or is isolated from the cluster.

Cluster: list each node in the cluster, specify the node name, node ID, legal number of votes, and grid mode.

Fence devices: define fence devices.

Manage resources: define the resources needed to create a cluster service. Management resources include failed transfer domains, resources, and services.

Brief description of initiator and target of iscsi:

ISCSI (Internet SCSI) is a standard developed by IETF (InternetEngineering Task Force, Internet Engineering Task Force) in 2003. This set of instructions can run SCSI protocol on IP network and enable it to route over high-speed Gigabit Ethernet. SCSI (Small Computer System Interface) is a block data transfer protocol, which is widely used in the storage industry, and is the most basic standard protocol for storage devices. ISCSI protocol is a method of using IP network to transmit SCSI data blocks with short latency. ISCSI uses Ethernet protocol to transmit SCSI commands, responses and data. ISCSI can build IP storage area networks using Ethernet that we are already familiar with and use every day. In this way, iSCSI overcomes the limitation of directly connecting to storage, enables us to share storage resources across different servers, and expands storage capacity without downtime.

The working process of iSCSI: when the iSCSI host application sends a data read and write request, the operating system generates a corresponding SCSI command. The SCSI command is encapsulated into an iSCSI message packet in the iSCSI Initiator layer and transmitted to the device side through TCP/IP. The iSCSI Target layer of the device side will unlock the iSCSI message packet, get the contents of the SCSI command, and then transmit it to the SCSI device for execution. The response of the device after executing the SCSI command is encapsulated into the iSCSI response PDU when it passes through the iSCSI Target layer on the device side, and is transmitted to the host through the iSCSI Initiator layer of the TCP/IP network. ISCS Initiator will parse the SCSI response from the iSCSI response PDU and transmit it to the operating system, and then the operating system responds to the application. In order to read and write iSCSI, in addition to using specific hardware, the server can also be simulated as the initiator (Initiator) or target end (target) of iSCSI by software, and the iSCSI connection can be realized by using the existing processor and ordinary Ethernet card resources.

This article has realized the gfs function on the Centos5.5 virtual machine by software!

Install the target side of iscsi on 192.168.1.240:

[root@iscsi-storage ~] # yum install scsi-target-utils (Note: there is no letter I before scsi)

(using centos's default yum source, the software scsi-target-utils-0.0-6.20091205snap.el5_5.3 will be installed.)

[root@iscsi-storage ~] # yum install libibverbs-devel libibverbs librdmacm librdmacm-devel

If the above software is not installed, the following error will be reported in / var/log/messages:

Iscsi-storage tgtd: libibverbs.so: cannot open shared object file: No such file or

Directory-iser transport not used

Iscsi-storage tgtd:librdmacm.so: cannot open shared object file: No such file or directory-iser

Transport not used

Disk sda holds the system, and disk sdb is used to store and partition it.

[root@iscsi-storage ~] # fdisk / dev/sdb

I am divided into / dev/sdb1 3G, / dev/sdb2 7G.

Instead of formatting the partition after the partition is completed, execute the command partprobe to let the system read the partition table again without rebooting the system.

[root@iscsi-storage ~] # partprobe

After scsi-targe-utils is installed, the service is called tgtd, start it, and set it to boot.

[root@iscsi-storage ~] # service tgtd start

Starting SCSI target daemon: Starting target framework daemon

[root@iscsi-storage ~] # chkconfig tgtd on

Next, establish target device. Use the tgtadm instruction in centos5 to set vi / etc/ietd.conf in 4.

The parameters of the tgtadm command are a bit difficult to remember, but the scsi-target-utils suite contains steps to build target device using the tgtadm instruction, which can be set by referring to the file: / usr/share/doc/scsi-target-utils-0.0/README.iscsi

I would like to add a new iqn named iqn.2011-01.com.chinaitv:rhcs-storage target device.

=

The iqn (iSCSI Qualified Name) format is usually as follows:

Format

Meaning

Example

Yyyy-mm

Year-month

2011-01

Reversed domain name

Write the domain name in reverse, usually the company's domain name in reverse

Com.chinaitv

Identifier

Identify words, usually indicating the purpose of this storage space

Rhcs-storage

=

The new target device command is as follows:

# tgtadm-lld iscsi-op new-mode target-tid 1-T iqn.2011-01.com.chinaitv:rhcs-storage

After execution, view the target device with the following command:

# tgtadm-lld iscsi-op show-mode target

Add the partition you originally created to the target device.

# tgtadm-lld iscsi-op new-mode logicalunit-tid 1-lun 1-b / dev/sdb1

View the output after joining target device:

# tgtadm-lld iscsi-op show-mode target

Sets the initiator node that can access this target device. This machine allows gfs1 gfs2 gfs3 access, and the settings are as follows:

# tgtadm-lld iscsi-op bind-mode target-tid 1-I 192.168.1.231

# tgtadm-lld iscsi-op bind-mode target-tid 1-I 192.168.1.232

# tgtadm-lld iscsi-op bind-mode target-tid 1-I 192.168.1.233 (I is uppercase I)

(if all initiators access is allowed, you can replace ip with ALL, as follows:

# tgtadm-lld iscsi-op bind-mode target-tid 1-I ALL

If you need to remove it, you need to replace bind with unbind, for example

# tgtadm-lld iscsi-op unbind-mode target-tid 1-I 192.168.1.233)

Only sdb1 is used to test storage in this operation, so only sdb1 is added. If you need to use multiple partitions for storage, you can do the following:

Tgtadm-- lld iscsi-- op new-- mode target-- tid 1-T iqn.2011-01.com.chinaitv:rhcs-storage

Tgtadm-- lld iscsi-- op new-- mode target-- tid 2-T iqn.2011-01.com.chinaitv:rhcs-storage2

Tgtadm-lld iscsi-op new-mode logicalunit-tid 1-lun 1-b / dev/sdb1

Tgtadm-lld iscsi-op new-mode logicalunit-tid 2-lun 1-b / dev/sdb2

Tgtadm-lld iscsi-op bind-mode target-tid 1-I ip

Tgtadm-- lld iscsi-- op bind-- mode target-- tid 2-I ip

Note: if you restart the tgtd service, all the above configurations will disappear, so do not restart casually, otherwise you need to reconfigure according to the above steps. In order to run automatically at the next boot, you can add the above command to the / etc/rc.local file.

First, add the following at the end of the hosts file of the three node machines:

# vi / etc/hosts

Install iscsi-initiator-utils on the node machine

# yum install iscsi-initiator-utils

Install the required software packages

[root@gfs1 ~] # yum install-y cman gfs-utils kmod-gfs kmod-dlm cluster-snmp lvm2-cluster rgmanager

Specify initiator alias name

# echo "InitiatorAlias=gfs1" > > / etc/iscsi/initiatorname.iscsi

Create a cluster.conf configuration file, which is not available after installation. You need to create it yourself:

# vi / etc/cluster/cluster.conf

# # this line defines the name of cluster, where config_version is the number of times the profile has been configured

# # (in RedHat Cluster, each node is the same configuration version value, that is, the configuration file with the highest number of configuration times for learning synchronization report)

# # this line defines the name, id and voting rights of the cluster node. The node name is usually the hostname of the node.

Fence is also a product of RedHat Cluster, and GFS cannot work without it. The function of fence is that when a node goes down (disconnects with cluster), other normal nodes will use fence to set up the device fence and kick the device out of cluster to ensure the normal operation of the whole cluster. Fence devices are generally the server's own hardware devices, such as hp ilo port, IBM and dell ipmi, etc., due to the different server hardware. As a result, different fence devices are configured differently. There is no fence device for the virtual machine here, so we use the method of fence_manual to set it manually. The function of manual fence is: when a node goes down, we need to enter the command "# fence_ack_manual-n dropped node name" on the normal node to isolate the host of the cluster, so that the normal node can work normally. Now I am not very familiar with cluster.conf, so I can not give a more detailed answer, but you can enter the command: man 5 cluster.conf to view the help file. This cluster.conf file needs to be created on each node machine. )

(note: the configuration file is not very standard. You can add more content and modifications as needed. Here is just a simple example.)

Start iscsi daemon

# service iscsi start

# chkconfig iscsi on

Use the iscsiadm command to probe iscsi device:

# iscsiadm-m discovery-t sendtargets-p 192.168.1.240pur3260

192.168.1.240pur3260 iqn.2011-01.com.chinaitv:rhcs-storage

Log in to iscsi target

# iscsiadm-m node-T iqn.2011-01.com.chinaitv:rhcs-storage-p 192.168.1.240 purl 3260-l

Logging in to [iface: default, target: iqn.2011-01.com.chinaitv:rhcs-storage, portal: 192.168.1.240 Magi 3260]

Login to [iface: default, target: iqn.2011-01.com.chinaitv:rhcs-storage, portal: 192.168.1.240 Magne3260]: successful

After logging in successfully, you can use fdisk-l to find more than two hard disk partitions.

Create a LVM volume group named gfsvg on this node

# pvcreate-ff / dev/sdb

# vgcreate gfsvg / dev/sdb

List the VG size:

# vgdisplay gfsvg | grep "Total PE"

Total PE 717

Create lv

# lvcreate-l 717-n gfs gfsvg

Logical volume "gfs" created

# cman_tool status | grep "Cluster Name"

Cman_tool: Cannot open connection to cman, is it running?

The reason is that the service is not started, the service is started:

# service cman start

Create a GFS volume on this node and format the GFS file system, using the command:

Gfs_mkfs-p lock_dlm-t ClusterName:FSName-j Number BlockDevice

Format the gfs file format:

The parameter-p is followed by the gfs lock mechanism. Generally speaking, lock_dlm is used.

-t is followed by ClusterName:FSName

Where ClusterName is the cluster name, which is the cluster name specified in the cluster.conf configuration file, and FSName is the name of the newly formatted gfs partition.

-j represents the number of journal. Generally speaking, 2 are not required to be adjusted separately unless there are some special cases.

BlockDevice is the name of the device to be formatted.

Here is an example of formatting the GFS command:

# gfs_mkfs-p lock_dlm-t GFSCluster:gfs-j 3 / dev/gfsvg/gfs

Load the appropriate gfs module and see if lv is successful

# modprobe gfs

# modprobe gfs2

# chkconfig gfs on

# chkconfig gfs2 on

# chkconfig clvmd on

# / etc/init.d/gfs restart

# / etc/init.d/gfs2 restart

# / etc/init.d/clvmd restart

# lvscan

ACTIVE'/ dev/gfsvg/gfs' [2.80 GB] inherit # # indicates success

Mount the newly created logical disk locally:

# mount-t gfs / dev/gfsvg/gfs / opt

# df-h

Do the following on the gfs2 node and the gfs3 node, respectively

# modprobe gfs

# modprobe gfs2

# / etc/init.d/gfs restart

# / etc/init.d/gfs2 restart

# / etc/init.d/clvmd restart

# chkconfig-add cman

# chkconfig-add clvmd

# chkconfig-add gfs

# chkconfig-level 35 cman on

# chkconfig-level 35 clvmd on

# chkconfig-level 35 gfs on

# lvscan

# mount / dev/gfsvg/gfs / opt

Go to the / opt directory to see if there is the file file1 that was previously established on gfs1

As shown in the figure, any operation can be done in the / opt directory, and the three machines will be consistent, thus achieving the goal of operating on one machine and keeping the other machines synchronized to improve usability. When something goes wrong with gfs1, gfs2 and gfs3 can also be used to provide services! You can execute the command clustat on each node machine to view the activity status of each node in the gfs cluster:

You can also execute the following command to check whether the connection is successful

# ccs_test connect

Connect successful.

Connection descriptor = 9600

Check to see if it is normal

# ccs_tool lsnode

# ccs_tool lsfence

# cman_tool services

Note: the clvmd service must be started on the gfs2 and gfs3 nodes, or the / dev/gfsvg/gfs device cannot be mounted.

If you want to stop the tgtd (service tgtd stop) service on the storage server, that is, on the iscsi-storage in this lab, and if there are external nodes connected to this storage, you have the following prompts:

Stopping SCSI target daemon: Stopping target framework daemon

Some initiators are still connected-could not stop tgtd

Indicates that it cannot be stopped, but when it is stopped by killing the pid of tgtd, when you start the tgtd service with service tgtd start, you will have the following prompt:

In fact, tgtd did not start successfully, and you cannot view the tgtd service process with ps. To solve this problem, because a / var/lock/subsys/tgtd file exists, you just need to delete it.

# rm / var/lock/subsys/tgtd

On the node machine, modify the / etc/fstab file if you want to power on the automatic mount, and add the following:

/ dev/mapper/gfsvg-gfs / opt gfs defaults 0 0

When I deployed this gfs cluster storage on the virtual machine, I encountered a lot of problems. It took a long time to implement gfs on the virtual machine after consulting all kinds of materials, among which there were many errors, for example:

Jan 19 04:04:00 gfs1 ccsd [19610]: Cluster is not quorate. Refusing connection.

Jan 19 04:04:00 gfs1 ccsd [19610]: Error while processing connect: Connection refused

Jan 19 04:04:01 gfs1 dlm_controld [18685]: connect to ccs error-111l, check ccsd or cluster status

This error is related to the startup of fence. This is because fencing cannot be started when I start cman on the first node. To enable the fence function, multiple machine nodes must be enabled. When a single node is enabled, it will not take effect. In other words, half of the fence must be enabled to take effect, so you need to execute service cman start on multiple nodes to quickly solve the fenc startup problem.

Final question:

1 check the status of target is always ready, but cannot get running status

2 there is always the following error when entering the tgtd command:

[root@iscsi-storage ~] # tgtd

Librdmacm: couldn't read ABI version.

Librdmacm: assuming: 4

CMA: unable to get RDMA device list # # Note, because there is no rdma device on the virtual machine, the following socket does not understand

(null): iscsi_rdma_init (1217) cannot initialize RDMA; load kernel modules?

(null): iscsi_tcp_init unable to bind server socket, Address already in use

(null): iscsi_tcp_init unable to bind server socket, Address already in use

No available low level driver!

This is the end of the article on "how to build linux gfs system iscsi+GFS to achieve network storage". I hope the above content can be of some help to you, so that you can learn more knowledge. if you think the article is good, please share it out for more people to see.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.