Fibre Channel - SAN Protocols Explained
Scroll Down
What is Fibre Channel?
FC (Fibre Channel) is a network technology, predominantly used within storage area networks, to provide high-speed, loss-less delivery of raw block data between computer data storage and server devices.
The Fibre Channel network (aka fabric) is a dedicated high-speed, low latency storage network, supporting bandwidth speeds of 2, 4, 6, 8 and 16Gbps, connecting:
- HBAs (Host Bus Adapter) - Dedicated server storage adapters (aka the Initiator)
- Storage Systems - Contains storage controllers, and disks (aka the target).
- FC Switches - High-speed Fibre Channel switches.
The FCP (Fibre Channel Protocol) operates over the FC fabric. Predominantly FCP is the implementation of SCSI over an FC network, or in other words,
SCSI data is encapsulated and transported within FC frames.[1]
FC Layers
When it comes to learning the various layers of Fibre Channel it is important to understand that FC does not follow the traditional OSI 7 layer model. Instead, it is broken down into 5 layers. As shown below:
FC-0 - Defines the physical media used to link two Fibre Channel ports, including cabling types, optical and electrical parameters for a variety of data rates, maximum transfer distances, and noise limits. Fibre Channel supports two types of cables: Copper and optical.[2]
FC-1 - Defines the transmission protocol including serial encoding and decoding rules, special characters and error control.[3]
FC-2 - Defines the transport mechanism of Fibre Channel and the framing rules of the data to be transferred between ports, the different mechanisms for controlling the service of classes and the means of managing the sequence of a data transfer.[4]
FC-3 - Defines common services required for advanced features such as striping, hunt group, and multicast.
FC-4 - Defines the application interfaces that can execute over Fibre Channel. It specifies the mapping rules of upper layer protocols using the FC levels below.[5]
Figure 1: FC Layers
FC Frames
Fibre Channel defines a variable length frame consisting of 36 bytes of overhead and up to 2112 bytes of payload for a total maximum size of 2148 bytes. A Start of Frame (SOF) delimiter and End of Frame (EOF) delimiter mark the beginning and end of each Fibre Channel frame.
The CRC is used to detect transmission errors.[6]
Figure 2: FC Layers.[7]
Addressing/Naming
World Wide Names
WWN
FC addressing is based upon WWN (World Wide Names). These addresses are unique to each of the FC devices (such as an FC HBA or SAN's) and are 64-bit addresses, consisting of 16 x Hexadecimal values, like so: 15:00:00:f0:8c:08:95:de.
Each device in the SAN is identified by a unique WWN. The WWN contains vendor identifier field, which is defined and maintained by the IEEE.[8]
The WWN can be compared to a MAC address in the Ethernet world.
WWPN
Whereas the WWN is assigned to the device. The World Wide Port Name's (WWPN) is a unique address assigned to every port of a node (be it HBA or storage system).
WWNN
The World Wide Node Name (WWNN) is assigned to a node in the storage network. For example, the same WWNN can represent multiple network interfaces of a single network node.[9]
Aliases
Aliases are used to aid in the configuration and troubleshooting, by mapping the World Wide name (from the FC switch or storage system) to a string. For example: the WWPN 15:00:00:f0:8c:08:95:de could be aliased to MAIL-SERVER.
Switch Domain IDs
Within the FC network, each switch is assigned a unique Domain ID. A principle switch is elected within the network which assigns the Domain IDs to the other switches. This Domain ID is then used to form part of the FCID, as per the FLOGI process (described later).
FCNS
The Fibre Channel Name Service (FCNS) is used to exchange the FLOGI database, between switches within the fabric. Allowing each switch to learn where each WWPN is, and how to route traffic there.[10]
Security
Zoning
Zoning is a security feature that allows the administrator to control which hosts can communicate with each other. This allows the administrator to create zones that prevent servers from communicating with each other over the fabric, ensuring that each server can only communicate to the storage system.
Figure 3: Zoning.[11]
LUN Masking
LUN masking is used to restrict which LUN is presented to which host or set of hosts. This prevents unauthorized hosts from accessing LUNs and also prevents situations of data corruption that could occur should a host access the wrong LUN.
Login Processes
Fabric Login (FLOGI)
When a device is connected to the fabric, it sends a Fabric Login (FLOGI) request. The switch assigns a 24-bit FCID address, which is equivalent to an IP address.
The FCID is then used by the FC switches to route traffic across the fabric.[12]
In addition, as part of this process buffer credits are also exchanged with the switch.
Port Login (PLOGI)
Once the FLOGI is complete, the initiator will send the PLOGI request to the switch, in order to inform the Fabric Name Server (FCNS) of its personality and capabilities. Such as:[13]
- WWNN, WWPN
- Buffer credits for flow control
- clock frequency ('speed capability')
- Upper layer protocol support (eg. SCSI-3, IP)
Process Login (PLRI)
The initiator host will send a PLRI request to the storage target. The storage system will then granted access to the host based on its configured LUN masking.[14]
Ports/Links
Port Types
There are a number of FC port types. Below outlines the main ones:
| Port Type | Port Name | Port Description |
|---|---|---|
| G_Port | Generic | This is the port type that all ports first start with, prior to transitioning to another port type. |
| F_Port | Fabric | Connects to an N_Port (aka Initiator or Target) |
| N_Port | Node | The Initiator or Target. Connects to an F_Port (aka Fabric switch). |
| E_Port | Extension | Connects to another fabric switch. |
Figure 4: FC Ports.
Link Types
- Inter-Switch Link (ISL) - A connection between two switches.
- Trunk - Allows for multiple ISL's to be bundled together as a single link. Traffic is then distributed across each of the members. This is synonymous to port-channels in the world of ethernet.
Figure 5: FC Links.
Classes of Service (CoS)
Applications may require different levels of service and guarantees regarding delivery, connectivity, and bandwidth. Fibre Channel provided different classes of service to accommodate different application needs.[15]
| Class | Summary | Details | Use Case |
|---|---|---|---|
| Class 1 | Acknowledged, connection-oriented, full bandwidth service. | Reserves 100% full bandwidth. Therefore no B2B flow control is required, only E2E. | Used when data needs to be continuous and time critical, such as voice or video. |
| Class 2 | Acknowledged, connectionless service (similar to TCP). | Due to being connectionless a port can transmit frames to and receive frames from more than one N_Port. As a result, the N_Ports share the bandwidth of the links with other network traffic. Frames are not guaranteed to arrive in the order in which they were transmitted. Uses both B2B and E2E flow control. | Used when the order and timeliness of delivery are not so important. |
| Class 3 | Unacknowledged, connectionless service (similar to UDP). | Similar to Class 2, but only useds E2E flow control. Therefore optimizing the fabrics resources, leaving the upper protocols to handle frame sequence ordering. | Works well with SCSI and is the most commonly used service in FC networks. |
| Class 4 | Acknowledged, connection-oriented, partial bandwidth service. | Similar to class 1, but Virtual Circuits are used between N_ports to provide QoS. Uses E2E flow control. | Used by multimedia applications that require allocate bandwidth and latency. |
| Class F | Acknowledged, connectionless service. | Used for switch-to-switch, aka ISL links. | Used by E_Ports for control and management of the fabric. |
Flow Control
The goal of flow control is to prevent a situation where a device receives frames faster then it can process. FC provides 2 flow control mechanisms and the mechanism used is dependent upon the service class.
Buffer-to-Buffer (BB_Credits)
Buffer-to-buffer (B2B) is used for Class 3 traffic to provide N_Port to F_Port flow control. Each N_Port begins with a credit pool, that is established during the FLOGI login. Each frame that is sent results in the credit pool is reduced by 1. Once the receiver is ready a R_RDY frame is sent and the sender's credits are replenished.
End-to-End (EE_Credits)
Provides N_Port to N_Port flow control and is used by Class 2 traffic. An initial credit pool is established as part of the PLOGI login. The credit pool is replenished when ACK frames are received from the target.
Useful Links
https://www.iol.unh.edu/testing/storage/fc/tutorial
https://hsi.web.cern.ch/hsi/fcs/spec/overview.htm
https://www.storageinfra.com/fibre-channel-classes-of-service/
References
"4.3 Overview of Fibre Channel (FC) SAN Protocol ... - TSM Tutorials." 8 Aug. 2016, http://www.tsmtutorials.com/2016/08/fc-san-protocols.html. Accessed 13 Oct. 2018. ↩︎
"UNH-IOL Fibre Channel Tutorial | InterOperability Laboratory." https://www.iol.unh.edu/testing/storage/fc/tutorial. Accessed 15 Oct. 2018. ↩︎
"UNH-IOL Fibre Channel Tutorial | InterOperability Laboratory." https://www.iol.unh.edu/testing/storage/fc/tutorial. Accessed 15 Oct. 2018. ↩︎
"Fibre Channel Overview - High Speed Interconnect - CERN." https://hsi.web.cern.ch/hsi/fcs/spec/overview.htm. Accessed 15 Oct. 2018. ↩︎
"Fibre Channel Overview - High Speed Interconnect - CERN." https://hsi.web.cern.ch/hsi/fcs/spec/overview.htm. Accessed 15 Oct. 2018. ↩︎
"Fibre Channel Overview - High Speed Interconnect - CERN." https://hsi.web.cern.ch/hsi/fcs/spec/overview.htm. Accessed 15 Oct. 2018. ↩︎
"Fiber Channel Frame - Frame Design & Reviews - myframe.co Frame ...." https://myframe.co/fiber-channel-frame/. Accessed 15 Oct. 2018. ↩︎
"Fibre Channel addressing – Storage Freak." 6 Oct. 2014, https://www.storagefreak.net/2014/10/fibre-channel-addressing. Accessed 12 Oct. 2018. ↩︎
"Fibre Channel addressing – Storage Freak." 6 Oct. 2014, https://www.storagefreak.net/2014/10/fibre-channel-addressing. Accessed 12 Oct. 2018. ↩︎
"Fibre Channel SAN Storage Overview Tutorial Video - YouTube." 3 Jul. 2016, https://www.youtube.com/watch?v=zb2kEtjMmPg. Accessed 12 Oct. 2018. ↩︎
"Fibre Channel SAN Storage Overview Tutorial Video - YouTube." 3 Jul. 2016, https://www.youtube.com/watch?v=zb2kEtjMmPg. Accessed 15 Oct. 2018. ↩︎
"Fibre Channel SAN Storage Overview Tutorial Video - YouTube." 3 Jul. 2016, https://www.youtube.com/watch?v=zb2kEtjMmPg. Accessed 12 Oct. 2018. ↩︎
"3766 - VMware Communities : Discussion List - All Communities." http://vmware1254.rssing.com/chan-14022298/all_p3766.html. Accessed 12 Oct. 2018. ↩︎
"Fibre Channel SAN Storage Overview Tutorial Video - YouTube." 3 Jul. 2016, https://www.youtube.com/watch?v=zb2kEtjMmPg. Accessed 12 Oct. 2018. ↩︎
"Fibre Channel Classes of service | StorageInfra." 31 Mar. 2017, https://www.storageinfra.com/fibre-channel-classes-of-service/. Accessed 12 Oct. 2018. ↩︎