Shulou(Shulou.com)05/31 Report--

This article introduces how to analyze iscsi, nfs and ceph. The content is very detailed. Interested friends can use it for reference. I hope it will be helpful to you.

I. iscsi setting

1. Server side

[1] Install administration tools.

[root@dlp ~] # yum-y install scsi-target-utils

[2] Configure iSCSI Target.

For example, create a disk image under the [/ iscsi_disks] directory and set it as a shared disk.

# create a disk image [root@dlp ~] # mkdir / iscsi_ disks [root @ dlp ~] # dd if=/dev/zero of=/iscsi_disks/disk01.img count=0 bs=1 seek=80G [root@dlp ~] # vi / etc/tgt/targets.conf# add follows to the end# if you set some devices Add-and set the same way with follows# naming rule: [iqn.yaer-month.domain:any name] # provided devicce as an iSCSI target backing-store / iscsi_disks/disk01.img # iSCSI Initiator's IP address you allow to connect initiator-address 10.0.0.31 # authentication info (set anyone you like for "username" "password") incominguser username password [root@dlp ~] # / etc/rc.d/init.d/tgtd startStarting SCSI target daemon: [OK] [root@dlp ~] # chkconfig tgtd on# confirm status [root@dlp ~] # tgtadm-- mode target-- op showTarget 1: iqn.2014-08.world.server:target00 System information: Driver: iscsi State: ready import nexus information: LUN Information: LUN: 0 Type: controller SCSI ID: IET 00010000 SCSI SN: beaf10 Size: 0 MB Block size: 1 Online: Yes Removable media: No Prevent removal: No Readonly: No Backing store type: null Backing store path: None Backing store flags: LUN: 1 Type: disk SCSI ID: IET 00010001 SCSI SN: beaf11 Size: 85899 MB Block size: 512 Online: Yes Removable media: No Prevent removal: No Readonly: No Backing store type: rdwr Backing store path: / iscsi_disks/disk01.img Backing store flags: Account information: username ACL information: 10.0.0.31

two。 Client configuration

[1] Configure iSCSI Initiator.

[root@www] # yum-y install iscsi-initiator-utils [root@www] # vi / etc/iscsi/iscsid.conf# line 53: uncommentnode.session.auth.authmethod = CHAP# line 57 root@www 58: uncomment and set username and password which set on iSCSI Targetnode.session.auth.username = usernamenode.session.auth.password = password# discover target [root@www] # iscsiadm-m discovery-ts endtargets-p 10.0.0.30Starting iscsid: [OK] 10.0.0.30 08.world.server:target00# confirm status after discovery 3260 iqn.2014-08.world.server:target00# confirm status after discovery [root@www ~] # iscsiadm-m node-o show# BEGIN RECORD 6.2.0-873.10.el6node.name = iqn.2014-08.world.server:target00node.tpgt = 1node.startup = automaticnode.leading_login = No.node.conn [0] .iscsi.IFMarker = Nonode.conn [0] .iscsi. OFMarker = No# END RECORD# login to target [root@www ~] # iscsiadm-m node-- loginLogging in to [iface: default Target: iqn.2014-08.world.server:target00, portal: 10.0.0.30 iqn.2014 3260] (multiple) Login to [iface: default, target: iqn.2014-08.world.server:target00 Portal: 10.0.0.30 session 3260] successful.# confirm established session [root@www ~] # iscsiadm-m session-o showtcp: [1] 10.0.0.30 iscsiadm 3260 blocks name 1 iqn.2014-08.world.server:target00# confirm partitions [root@www ~] # cat / proc/partitionsmajor minor # blocks name 8 0 209715200 sda 8 1 512000 sda1 82 209202176 sda2 253 0200966144 dm-0 253 18232960 dm-1 8 16 83886080 sdb# added new device provided from target as [sdb]

[2] It's possible to use iSCSI device like follows.

[root@www ~] # yum-y install parted# create a label [root@www ~] # parted-- script / dev/sdb "mklabel msdos" # create a partition [root@www ~] # parted-script / dev/sdb "mkpart primary 100%" # format with EXT4 [root@www ~] # mkfs.ext4 / dev/sdb1# mount [root@www ~] # mount / dev/sdb1 / mnt [root@www ~] # df-hTFilesystem Type Size Used Avail Use% Mounted on / dev/mapper/vg_dlp-lv_root ext4 189G 1.1G 179G 1% / tmpfs tmpfs 1.9G 0 1.9G 0% / dev/shm/dev/sda1 ext4 485M 75M 385M 17% / boot/dev/sdb1 ext4 79G 184M 75G 1% / mnt

Add:

After the client is installed, you need to start the daemon and then discover:

/ etc/init.d/iscsid startiscsiadm-m discovery-t st-p 12.123.0.51iscsiadm-m discovery-T iqn.2015-06.com.oracle:zjxl-p 12.123.0.51 12.123.0.51iscsiadm 3260-l# modify configuration file vim / var/lib/iscsi/send_targets/12.123.0.51\, 3260/iqn.2015-06.com.oracle\: oracle\: zjxl\, 12.123.0.51\, 3260\, 1\, default/default

Configure multipath:

A machine must hang at least 2 storage IP, one of which is broken, and the other can be connected. Edit / etc/multipath.conf

Blacklist {devnode "^ sda"} defaults {user_friendly_names yesudev_dir / devpath_grouping_policy multibusfailback immediateno_path_retry fail}

Blacklist means that the system disk does not do multipath.

Then service multipathd restart; executes cat / proc/partitions to see devices that start with dm-*. These are multipath devices.

II. Ceph setting

In fact, for ceph, directly through rbd create a block, and then rbd map this block, and then rbd showmapped can see the path of this block map, such as / dev/rbd0. Then configure it in / etc/tgt/targets.conf on the iscsi server side:

Direct-store / dev/rbd0

In this way, the block device can be used on the iscsi client.

But it is expensive to use because it mounts rbd in the form of ceph rbd kernel module. In this way, there will be a large number of switching between kernel state and user mode, which is bound to affect the performance.

This problem has been solved:

Http://ceph.com/dev-notes/adding-support-for-rbd-to-stgt/

Http://ceph.com/dev-notes/updates-to-ceph-tgt-iscsi-support/

Http://www.sebastien-han.fr/blog/2014/07/07/start-with-the-rbd-support-for-tgt/

Iscsi on ubuntu already supports rbd;fedora 20 and there is an iscsi rpm package that supports rbd. However, on centos 6.5, iscsi with rbd support cannot be configured.

If you have also configured this feature, you are welcome to communicate.

Third, use scsi-target-utils that supports rbd

There is no problem for ceph to derive iscsi devices by using the above method, but because it uses the kernel module of rbd, it leads to frequent switching between kernel mode and user mode, which is bound to affect the performance. Is it possible to export rbd block devices directly to iscsi devices? The answer is yes.

First download the scsi-target-utilsrpm package that supports rbd from the following address:

# wget http://ceph.com/packages/ceph-extras/rpm/centos6/x86_64/scsi-target-utils-1.0.38-48.bf6981.ceph.el6.x86_64.rpm # rpm-ivh scsi-target-utils-1.0.38-48.bf6981.ceph.el6.x86_64.rpm

After the installation is complete, check whether the current tgt supports rbd driver:

# tgtadm-lld iscsi-mode system-op show System: State: ready debug: off LLDs: iser: error iscsi: ready Backing stores: rbd (bsoflags sync:direct) rdwr (bsoflags sync:direct) ssc null bsg sg sheepdog Device types: passthrough tape changer controller osd cd/dvd disk iSNS: iSNS=Off iSNSServerIP= iSNSServerPort=3205 iSNSAccessControl=Off

Create a rbd device:

# rbd create iscsi/tgt1-s 10240

Modify / etc/tgt/targets.conf to export the rbd device you just created:

Include / etc/tgt/conf.d/*.conf driver iscsi bs-type rbd backing-store iscsi/tgt1

Restart tgt:

# / etc/init.d/tgtd restart Stopping target framework daemon Starting target framework daemon

Connect the iscsi target on the iscsi initiator side:

[root@ceph-osd-1] # iscsiadm-m discovery-t sendtargets-p 10.10.200.165 Starting iscsid: [OK] 10.10.200.165 sendtargets 3260 iqn.2014-11.rbdstore.com:iscsi [root@ceph-osd-1] # iscsiadm-m node-T iqn.2014-11.rbdstore.com:iscsi-l Logging in to [iface: default, target: iqn.2014-11.rbdstore.com:iscsi Portal: 10.10.200.165jue 3260] (multiple) Login to [iface: default, target: iqn.2014-11.rbdstore.com:iscsi, portal: 10.10.200.165jue 3260] successful. [root@ceph-osd-1] # fdisk-l Disk / dev/sdb: 5788.2 GB, 5788206759936 bytes 255heads, 63 sectors/track, 703709 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Sector size (logical/physical): 512 bytes / 512 bytes I bytes Sector size O size (minimum/optimal): 512 bytes / 512 bytes Disk identifier: 0x00000000 Disk / dev/sda: 209.7 GB, 209715068928 bytes 255heads, 63 sectors/track 25496 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Sector size (logical/physical): 512 bytes / 512 bytes I size O size (minimum/optimal): 512 bytes / 512 bytes Disk identifier: 0x0009a9dd Device Boot Start End Blocks Id System / dev/sda1 * 1131 1048576 83 Linux Partition 1 does not end on cylinder boundary. / dev/sda2 131 1176 8392704 82 Linux swap / Solaris / dev/sda3 1176 25497 195356672 8e Linux LVM Disk / dev/mapper/vg_swift-LogVol00: 200.0 GB, 200043134976 bytes 255 heads, 63 sectors/track 24320 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Sector size (logical/physical): 512 bytes / 512 bytes I size O size (minimum/optimal): 512 bytes / 512 bytes Disk identifier: 0x00000000 Disk / dev/mapper/docker-253:0-3539142-pool: 107.4 GB, 107374182400 bytes heads, 63 sectors/track 13054 cylinders Units = cylinders of 16065 * 8225280 bytes Sector size (logical/physical): 512 bytes / 512 bytes I size O size (minimum/optimal): 512 bytes / 65536 bytes Disk identifier: 0x00000000 Disk / dev/sdc: 10.7 GB, 10737418240 bytes 64 heads, 32 sectors/track 10240 cylinders Units = cylinders of 2048 * 512 = 1048576 bytes Sector size (logical/physical): 512 bytes / 4194304 bytes I use O size (minimum/optimal): 4194304 bytes / 4194304 bytes Disk identifier: 0x00000000

As you can see from the above, the export is successful.

IV. Nfs settings

A node in ceph takes advantage of rbd map a block device, which is then formatted and mounted in a directory, such as / mnt. Install the rpm package for nfs on this node:

Yum-y install nfs-utils

Set the mount directory:

[root@mon0 mnt] # cat / etc/exports / mnt 192.168.101.157 (rw,async,no_subtree_check,no_root_squash) / mnt 192.168.108.4 (rw,async,no_subtree_check,no_root_squash)

Start and export:

Service rpcbind startchkconfig rpcbind onservice nfs startchkconfig nfs onexportfs-r

Check on the client:

[root@osd2 /] # showmount-e mon0Export list for mon0:/mnt 192.168.108.4192.168.101.157

Then mount:

Mount-t nfs mon0:/mnt / mnt

It should be noted that UDP protocol is used by default for NFS. If the network is unstable, it can be replaced with TCP protocol:

Mount-t nfs mon0:/mnt / mnt-o proto=tcp-o nolock

5. Rbd-fuse settings

After configuring ceph.repo on the client, install the rpm package for rbd-fuse, and then you can mount pool:

Rbd-fuse-p test / mnt

The above example is to mount test pool to the client's / mnt without cephx. You can then see the blocks in test pool in / mnt. At this point, you can mount the img using losetup.

Uninstall uses fusermount-u / mnt directly.

On how to analyze iscsi, nfs and ceph to share here, I hope the above content can be of some help to you, can learn more knowledge. If you think the article is good, you can share it for more people to see.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.