Shulou(Shulou.com)06/03 Report--

1. ISCSI concept

I take iscsi target as the first article of TARGET, mainly because many people know about the export protocol of iscsi O, and they are the first to think of it.

What is iscsi. First of all, let's talk about its concept.

SCSI, the minicomputer system interface, is a storage interface designed based on the client-server model. The client (initiator) of SCSI sends commands to server (target), and server processes commands and responds to result to client. Because the corresponding transport protocol must be used to transmit commands, iSCSI is one of the transport protocols, in addition to FC and SAS.

ISCSI is the abbreviation of SCSI over IP. It uses TCP/IP protocol to transmit SCSI commands and responses. It borrows the existing mature and cheap Ethernet to achieve SCSI communication. It has low cost and is very convenient to manage and use.

ISCSI has two parts: initiator and target. Initiator and target corresponding to initiator of SCSI and target,iSCSI respectively use TCP transmission protocol to realize the data transmission between initiator and target of SCSI.

2. Basic concepts of iSCSI protocol.

SCSI target contains many LU (logical units), each SCSI command has its own target LU, and these LU are responsible for processing and responding to the SCSI commands against it. The command unit of iSCSI is PDU. On the iSCSI initiator side, PDU contains the LUN (logical unit number) and SCSI command (CDB) sent to it by SCSI initiator. PDU is passed to iSCSI target as the playload of TCP, and then SCSI target core throws the SCSI command into the target LU according to LUN. LU delivers the response result to SCSI target,SCSI target after processing the command, then delivers it to iSCSI target,iSCSI target to encapsulate result into iSCSI PDU, and then sends it back to iSCSI initiator using TCP. ISCSI initiator then submits the results to the upper level.

3. The concept of iSCSI subsystem member

ISCSI Device

The SCSI device that uses the iSCSI protocol to deliver the service subsystem is typically IP-SAN.

ISCSI Node

The iSCSI node represents an iSCSI initiator or iSCSI target. In a network entity, there may be one or more iSCSI nodes. ISCSI nodes can be accessed through one or more network ports. The identity of the iSCSI node is the iSCSI name. When iSCSI names and addresses are separated, multiple iSCSI nodes are allowed to use the same address, or an iSCSI node uses multiple addresses, which is the basic unit that makes up IP-SAN fabric.

Network Entity

Network entity. It represents a device or gateway that can be accessed through an IP network. A network entity must have one or more network ports, which can be used by iSCSI nodes contained in the network entity to access the IP network.

Network Portal

Network port. A component of a network entity that has an TCP/IP address. Network ports are identified by IP addresses in initiator and by IP addresses + listening TCP ports in target.

Session

A set of TCP connections that connect initiator and target form a session (which can be simply referred to as an iTunes nexus). You can add a TCP connection to session, or you can remove a TCP connection from session. Through all connections to a session, initiator sees only the same target.

Connection

A pair of TCP connections. One or more TCP connections can be used to communicate between Initiator and target.

CID (Connection ID)

Each connection in a session is identified by a CID, which is unique within the session range. CID is generated by initiator and passed to target when login requests and closes the connection using logout.

Portal Groups

Network port group. ISCSI session supports multiple connections, and some implementations can bind multiple connections established through multiple ports to a single session. Multiple network ports of an iSCSI network entity are defined as a network port group. If the group is associated with a session, the session can bind multiple connections established through multiple ports in the group, and then make them work together to achieve the purpose of bundling. Each session of the group does not need to include all the network ports of the group. An iSCSI node may have one or more network port groups, but the network ports used by each iSCSI can only belong to one group of the iSCSI node.

Portal Group Tag

Network port group identification. Use a 16-bit number to identify a network port group. In an iSCSI node, all ports with the same group flag form a network port group.

I_T nexus

Idle nexus refers to the relationship between a SCSI initiator port and a SCSI target port. For iSCSI, this relationship corresponds to a session, which refers to the relationship between the initiator side of the session and the iSCSI target network port group. The identity of the iTunt nexus is a pair of port names (iSCSI initiator name + iSuppli ISID target name + t + network port group ID).

PDU (Protocol Data Unit): splits information into messages when communicating between initiator and target. These messages are called iSCSI PDU.

SSID (Session ID)

The session between iSCSI initiator and iSCSI target is identified by SSID, which is composed of the ISID of the initiator part and the TPGT of the target part. ISID is explicitly given by initiator when session is established, and the implied TPGT is determined by the network port selected when establishing the connection. When the TargetName is given, in the process of establishing a connection, the target must send the TPGT keyword as a confirmation message.

TSIH (Target Session Identifying Handle): the identity assigned by the Target to the session established with a specific name initiator. But 0 is reserved for initiator to tell target that this is a new session. When adding a connect to a session, the TSIH has implicitly indicated.

4. Iscsi network mapping relationship

In SCSI arch, a Device (SCSI initiator or SCSI target) can contain multiple ports, the connection between a SCSI initiator port and a SCSI target port is an iTunes nexus, and only one iTunes nexus is allowed between a SCSI initiator port and a SCSI target port.

ITunes nexus corresponds to a session (normal) in iSCSI, SCSI target port corresponds to a portal group in iSCSI target (a group of ports,iSCSI target with the same tag listens on these ports to wait for the connection of iSCSI initiator to log in login, all these ports support an iTunes nexus), and SCSI initiator port corresponds to the port of session (normal) in iSCSI initiator, as shown in the following figure.

An iSCSI node can have only one and only one SCSI device; the SCSI device can only be accessed through a functioning iSCSI session; the SCSI name is the name of the iSCSI node.

The SCSI initiator port, which is mapped to one end of a certain functioning iSCSI session. In the login process, an iSCSI initiator node and an iSCSI target node negotiate to establish a normal session;. When the session is successfully established, a SCSI initiator port belonging to the iSCSI initiator device is created (a virtual port). The name and identity of the SCSI initiator port are defined as: iSCSI initiator name + I + ISID identity.

The definitions of iSCSI,SCSI initiator port and SCSI target port are not the same. ISCSI target port, which is mapped to a network port group of iSCSI target.

ISCSI target port name and identity: both are defined as: iSCSI target name + t + iSCSI target PGT (port group tag).

5. Naming and identification and login process in iSCSI protocol

Each iSCSI initiator or iSCSI target has a globally unique name, initiator name for iSCSI initiator and target name for iSCSI target.

Port group in iSCSI target uses port group tag to identify

Session uses ISID (initiator session identifier) identification on the iSCSI initiator side and TSIH (target session identifier handle) identification on the iSCSI target side.

Fourth, the state division and login of session and connection. For connection, there are two states, namely login phase (login state) and full function phase (full function state), that is, the login state is in the process of login negotiation between iSCSI initiator and iSCSI target; the full function state means that the login process has been completed, and the connection has been able to provide services for the upper SCSI. Like connection, Session also has login phase and full function phase. When only one connection in session is in login state, session is in login state, and session is in full function state when at least one connection in session is in full function state.

The process of logging in to iSCSI target by iSCSI initiator consists of three processes:

1) first, iSCSI initiator sends an initial login request to iSCSI target, which includes the protocol version supported by iSCSI initiator, iSCSI initiator name and iSCSI target name, ISID and TSIH, and CID

2) the second is security negotiation, that is, chap authentication, which is used for authorization authentication between iSCSI initiator and iSCSI target.

3) the last part is the operation parameter negotiation, which is used to negotiate some communication parameters in the full function period.

There are two ways to log out, one is that iSCSI initiator initiates a logout request and completes the logout after receiving the logout response from iSCSI target; the other is that iSCSI target sends an Async protocol packet containing "request logout", and the initiator sends a logout request.

5 iSCSI PDU format

1. One Basic Header Segment (BHS,48 bytes)

two。 Optional n (arbitrary) Additional Header Segment (AHS, variable length, integer multiple of 4 bytes (not enough must be filled with 0), PDU sent by iSCSI target to iSCSI initiator does not contain AHS)

3. Optionally a Header Digest (header check information, 4 bytes)

4. Optionally a Data Segment (data segment, variable length, integer multiple of 4 bytes (0 must be filled if not enough))

5. An optional Data Digest (data parity information, 4 bytes).

5.1 BHS (Basic Header Segment) format

1. I bit:1bit, indicating that this PDU needs to be sent immediately

2. Opcode opcode: 6bit, which identifies the type of this PDU

(1) request PDU opcode sent from iSCSI initiator to iSCSI target

0x00 NOP-Out 0x01 SCSI Command (encapsulates a SCSI Command Descriptor Block) 0x02 SCSI Task Management function request 0x03 Login Request 0x04 Text Request 0x05 SCSI Data-Out (for WRITE operations) 0x06 Logout Request 0x10 SNACK Request 0x1c-0x1e Vendor specific codes

(2) the corresponding PDU opcode sent from iSCSI target to iSCSI initiator

0x20 NOP-In 0x21 SCSI Response-contains SCSI status and possibly sense information or other response information. 0x22 SCSI Task Management function response 0x23 Login Response 0x24 Text Response 0x25 SCSI Data-In-for READ operations. 0x26 Logout Response 0x31 Ready To Transfer (R2T)-sent by target when it is ready to receive data. 0x32 Asynchronous Message-sent by target to indicate certain special conditions. 0x3c-0x3e Vendor specific codes 0x3f Reject

3. F bit:1bit, identifying this PDU as the last one in a PDU sequence

4. Total AHS Length:1byte, the length of all AHS in this PDU in 4 bytes, including 0 (padding) that may need to be filled

5. Data Segment Length:3byte, the length of the data segment in this PDU, in bytes, excluding 0 (padding) that may need to be filled

6. LUN: if this (command / response / data) PDU is associated with a specific LU, use LUN (Logical Unit Number) to identify the LU

7. Initial Task Tag (ITT): use ITT to identify any iSCSI task initiated by iSCSI initiator in a session. Any PDU associated with an iSCSI task, whether issued by iSCSI initiator or iSCSI target, carries the corresponding ITT of the iSCSI task. This ITT is the command identifier of scsi cmd and represents a scsi cmd.

Transmission of 6 pdu

6.1 SCSI read command

① iSINI (iscsi initiator) sends SCSI cmd pdu (OP=R,F=1) to iSTGT, and ① pdu seq ends.

② iSTGT sends data pdu and respone pdu to ISINI.

Data pdu contains three times of datain pdu,dataSN (seq number) in increment, and ends with the last tag, Flem1Magol PDU seq.

The response pdu will carry the ExpDataSN, which represents the number of DATA PDU sent by this iSCSI taget.

6.2 SCSI write command

The flow of iscsi writing commands, including both immediate data and non-immediate data.

① iSINI sends scsi cmd pdu (op=w,F=0,I=1 (immediate)), which contains immediate data.

ISINI dataout pdu (DataSN=0,F=1) deserves to be the end of the current dataout seq.

The next flow of ② is the interaction between R2T (ready to transfer) and dataout.

7 iSCSI error handling

Error repair scope (class) within command

Target: the target loses dataout pdu (digest error or sequence error) during the Write command. Solution: use recovery R2T.

Initiator: the initiator loses datain pdu or R2t pdu (digest error or sequence error) during the Read command. Solution: use snack.

Within connection

Initiator: the command issued did not get a response (ExpCmdSN), lost response pdu (digest error or sequence error). Solution: the former uses retry and the latter uses snack.

Target: issuing the response pdu has not been confirmed by the initiator (ExpStatSN). Solution: use nop-in to notify the initiator to cause the initiator to send snack requests for retransmission.

Connection recovery

Initiator: detects a tcp connection failure or receives an asynchronous message (Asynchronous Message) from the target notifying it that tcp connection failed. Solution: connection recovery (task reassign).

Target: failed to detect tcp connetion. Solution: close connection and notify initiator of tcp connection failure via asynchronous message if connection is still available.

Session recovery

Consider using session repair only after the other three repair attempts.

Session repair for initiators: close all connection connection, abort all pending commands to return the upper SCSI with appropriate results, and re-establish a new session connection to the target.

Error repair hierarchy / level (hierarchy/level)

The error repair hierarchy / level is used to define the ability of iSCSI initiator/iSCSI target to fix errors, as follows:

Level 0, no matter what error you encounter, you can only do session recovery, and the repair ability is very poor.

Level 1, can be repaired within the scope of command and connection, and has the ability of session recovery at the same time. If you encounter an error, consider within command repair first, then consider within connection repair, and then consider session recovery.

Level 2, it can be repaired within the scope of command and connection, and has the capabilities of connection recovery and session recovery at the same time. If you encounter an error, consider within command repair first, then consider within connection repair, and then consider connection recovery, and then consider session recovery.

Task reassign

In iSCSI, each iSCSI command (task) is associated with a unique connection, and all pdu related to this command need to be sent / received on this connection. After connection recovery, iSCSI initiator can consider task reassign, that is, re-associate the task (iSCSI command) on the original connection to another or new connection, which is accomplished by sending the iSCSI task management command-task reassign on the new connection.

Retry

When iSCSI initiator sends iSCSI cmd pdu to iSCSI target but does not receive the confirmation of iSCSI target (via ExpCmdSN or datain/r2t pdu or response pdu), iSCSI initiator needs to resend the iSCSI cmd pdu (using the same ITT and CmdSN as the original).

Reject pdu

If the reject is for cmd pdu, it means that the iSCSI target will discard the cmd;. If it is for a non-cmd pdu, the iSCSI target must also send a response pdu to complete the command, and the iSCSI initiator can choose whether to retransmit the pdu that has been reject.

Digest Error

For header digest error, the data in the pdu is received and discarded or the current connection is shut down directly.

Payload (data) digest error

ISCSI initiator discards the pdu directly, and if the pdu is datain pdu, send a data SNACK request for retransmission or abort the task (task management command) to complete the command. If the pdu is a response pdu, send an status SNACK request for retransmission, either do a connection repair (complete the command in the new connection), or simply close the connection. If the pdu is an unsolicited pdu (such as Async,Reject, etc.) no further processing is required. ISCSI target responds that the reason in a reject pdu,pdu is set to Data-Digest-Error, and then discards the pdu. If the pdu is a dataout pdu (R2T requested or unrequested), send R2T to request retransmission, or abort the task in response to a cmd response pdu (complete the command) (the Status field is "CHECK CONDITION" and the sense data is "protocol service CRC error"). If the pdu is not a dataout pdu, no further processing is required.

Sequense Error

When iSCSI initiator receives a R2TSN/DataSN out-of-order r2t/datain pdu or a response pdu with ExpDataSN indicating that r2t/datain pdu is missing, it means that iSCSI initiator has previously received one or more Digest Error r2t/datain pdu.

When iSCSI target receives an out-of-order dataout pdu of DataSN, it means that iSCSI target has received at least one dataout pdu of Digest Error before, using the method described in 1.7.3.

When iSCSI initiator receives a StatSN out-of-order response pdu, it means that you have received digest error's response pdu before, so use the method described in 1.7.3 to deal with it.

Format error

If iSCSI initiator or iSCSI target detects a received pdu format error (illegal or inconsistent content), the current session will be closed and a session (session recovery) will be rebuilt.

8 pdu classification and overview

8.1 PDU with only SCSI information

8.1.1 SCSI Command

ISCSI initiator à iSCSI target. PDU contains SCSI CDB (Command Description Block) and all the parameters needed for command execution (such as task attributes, length of data to be transferred, LUN,ITT, etc.), as well as information needed for iSCSI protocol communication:

CmdSN (the sequence number of the iSCSI command within the session range, used with ExpCmdSN to detect whether the iSCSI command is lost during transmission).

ExpStatSN (expected connection status serial number sent by iSCSI target, which is used to confirm to iSCSI target that all status responses in the connection whose StatSN is less than ExpStatSN have been received by iSCSI initiator, and together with StatSN to detect whether the iSCSI pdu containing Status information is lost during transmission)

8.1.2 SCSI Response

ISCSI target à iSCSI initiator, which is used by iSCSI target to feedback the implementation results of SCSI Command. PDU contains the result of the execution of the SCSI command, whether it has not been executed or has been executed, but there are some problems in the execution; if there is a data transfer in the command, it also includes the deviation value Residual count of the data transmission (for example, iSCSI initiator initially indicates to write 5k bytes, the target receives 5k+2 bytes, etc., Residual count is 2). The SCSI command execution result fields: response and status, if response is 0 (command completed) and status is 2 (check condition), sense data (with details of check condition) will be included in the data field of pdu.

Fields to be used for iSCSI protocol communication:

ExpCmdSN

The command sequence number that iSCSI target expects to be sent by iSCSI initiator within the session range, which is used to confirm to iSCSI initiator that all commands with CmdSN less than ExpCmdSN have been received by iSCSI target.

MaxCmdSN

The maximum command sequence number that can be sent by iSCSI initiator in session, used to build a window buffer that matches the command sending rate and command processing rate of iSCSI initiator and iSCSI target.

StatSN

The status sequence number of the iSCSI response pdu within the connection range, which is used for the status response number)

ExpDataSN (the number of Data PDU sent by iSCSI target in this command.

Task Management Function Request

ISCSI initiator à iSCSI target, for iSCSI initiator control management (such as abort) prior to multiple tasks sent to iSCSI target.

Task Management Function Response

ISCSI target à iSCSI initiator, which is used by iSCSI target to feedback the execution results of task management requests.

SCSI Data Out and SCSI Data In

ISCSI initiator contacts à iSCSI target, which is used to transmit SCSI data.

In addition to SCSI data, PDU also contains information for iSCSI protocol communication: Buffer Offset and Data Segment Length of the data carried in all the data to be transmitted by the command.

DataSN

The sequence number of data in a Data PDU sequence that is used to detect whether any Data PDU is lost in transmission.

Ready To Transfer (R2T)

ISCSI initiator é iSCSI target is used when iSCSI initiator transmits data to iSCSI target, iSCSI target notifies iSCSI initiator that it is ready to receive data, and after receiving this PDU, iSCSI initiator can send data to iSCSI target.

Target Transfer Tag (TTT)

The Data Out PDU that iSCSI initiator sends for this PDU contains the same TTT to identify the relevance of this packet to this R2T.

R2TSN

Is the sequence number of R2T within the scope of an iSCSI command, from which iSCSI initiator can detect the loss of R2T pdu.

8.2 PDU with SCSI information and iSCSI information

Asynchronous Message

An asynchronous message is a pdu that iSCSI target can actively send to iSCSI initiator to inform iSCSI initiator of some events. Asynchronous messages pdu can be either SCSI messages or iSCSI messages.

There is a field AsyncEvent in Pdu to indicate a specific event, such as 0 indicates an SCSI asynchronous event, and the specific content is placed in the data part of the pdu (called sense data); for example, 1 indicates that iSCSI target requires iSCSI initiator logout; and so on.

8.3 PDU with only iSCSI information

Text request and Text response

Text request and text response are applied to the negotiation of implementation parameters (initiators and targets are used in communication, see Chapter 12 of RFC3720). The data part of the Pdu appears in the form "key=value".

Multiple text request / response pdu contains the same ITT to identify that these requests / responses belong to the same negotiation operation. In addition, TTT in Pdu uses different mechanisms to identify whether to continue a negotiation operation or initiate a new negotiation operation.

Login request and login response

Login requests and responses are used to implement iSCSI initiator login to iSCSI target. Pdu header contains:

(1) Landing phase information T bit, C bit, CSG, NSG (login is divided into two stages, security negotiation phase and operation parameter negotiation phase)

(2) ISID and TSIH (uniquely identify session in iSCSI initiator and iSCSI target, respectively) to indicate the ID of this login to session

CID (the unique ID connection in session), indicating the ID of this login to connection

(3) CmdSN, ExpStatSN (login request)

(4) StatSN, ExpCmdSN, MaxCmdSN (login response).

The data part of Pdu appears in the form "key=value" like the text request / response pdu and contains login parameters, including security negotiation parameters (see Chapter 11 of RFC3720) and operation parameters (see Chapter 12 of RFC3720).

The login process of the first connection of a session will determine / negotiate some parameters within the entire session range, such as the iSCSI protocol version number, session ID, and the number of connection allowed to be created within the session.

Logout request and logout response

(1) Logout request and response are used to implement iSCSI logout iSCSI target. Logout request pdu includes the reason for logout:

a. Close the entire session

b. Close a connecton in the session (the connection that sends the logout request is determined by the CID in the pdu header)

c. Delete a connecton in the session for repair, and the pending iSCSI commands in the original connection are transferred to another connection for transfer processing, which is called connection recovery.

(2) the logout response pdu header contains a response field to report the response result of the logout request:

a. Successfully shut down session or connection

b. Connection not found

C.iSCSI target does not support connection recovery

d. Failed to close session/connection or repair connection.

If it is a connection repair response, the pdu,pdu header also contains two fields:

Time2Wait: if iSCSI initiator wants to reassign tasks (iSCSI command) to another connection (task reasign), at least wait time (in seconds)

Time2Retain:iSCSI target waits for iSCSI initiator to task reasign after Time2Wait, after which the task is discarded (iSCSI command).

SNACK request

SNACK is the ack of SN, that is, the pdu of all kinds of serial numbers, including pdu, datain pdu, and R2t pdu containing Status/response information, is confirmed (for correct cases) or iSCSI target retransmission is required (for cases of error).

Specifically, include the field Type in the SNACK request pdu header:

a. Request retransmission of datain pdu or R2t pdu

1. Request for retransmission of pdu containing Status/Response information

two。 Confirm receipt of datain pdu (for A bit set datain pdu)

3. Request datain pdu retransmission after the MaxRecvDataSegmentLength has been changed

The other two fields BegRun and RunLength indicate the number of starting SN and pdu of the pdus that needs to be retransmitted (in the case of retransmission) or the SN of the expected next datain pdu (when confirming reception).

Reject

Reject pdu is used to prompt iSCSI initiator for an iSCSI error to reject a pdu sent by the previous iSCSI initiator. The Pdu header contains a field Reason that indicates the reason for the rejection:

The data field verification error of a.pdu requires retransmission of iSCSI initiator.

b. Reject a previous SNACK pdu, which requires retransmission

c. Protocol error

d. The iSCSI command is not supported

In addition, reject pdu also copies the header part of the rejected pdu to the data part of the reject pdu and sends it to the iSCSI initiator.

Nop in and nop out

Nop in and nop out are used to periodically check whether the opposite end of iSCSI initiator and iSCSI target is still operable, which is equivalent to "ping" in TCP/IP protocol.

A nop out pdu initiated by iSCSI initiator that requires an iSCSI target response must contain a valid ITT (incomplete 1), and other nop out pdu must contain an ITT of all 1.

A nop in pdu initiated by iSCSI target that requires an iSCSI initiator response must contain a valid TTT (incomplete 1), and other nop in pdu must contain a TTT of all 1.

The nop out pdu sent by the iSCSI initiator responding to the nop in pdu of the iSCSI target (containing the TTT that is not complete 1) must contain the same TTT as the nop in pdu it responds to, except that the TTT contained in all other nop out pdu is all 1.

The nop in pdu sent by the iSCSI target responding to the nop out pdu of the iSCSI initiator (containing the ITT of the incomplete 1) must contain the same ITT as the nop out pdu of its response, and the ITT contained in other nop in pdu must be all 1.

[this article is only posted by 51cto blogger "underlying Storage Technology" https://blog.51cto.com/12580077, official account release: storage Valley], if you need to reprint, please contact me, thank you.

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.