Introduction
Server and storage technology is undergoing a fundamental change that promises significant increases in processing power, availability, and distributed access to massive amounts of data. Multi-processor servers, high performance backplanes, server clustering applications, the physical separation of storage from servers, and multi-terabyte capacity of storage arrays are rapidly exceeding the capabilities of the traditional SCSI architecture that previously bound servers and storage together.
Fortunately, technical innovations in server and storage products have not occurred in isolation. Interconnect technology has undergone its own metamorphosis, with SCSI protocol over parallel bus re-emerging as a serial transport over Fibre Channel. This convergence of new server designs, server clustering, autonomous gigabyte storage arrays, and Fibre Channel architecture is raising the server/storage paradigm to a higher plateau: storage networking. From the vantage point of storage networking, managers of Enterprise networks can now envision and implement innovative designs to enhance data availability and accommodate the growing demands of their users.
Fibre Channel Topologies
Fibre Channel architecture offers three topologies for network design: point-to-point, Arbitrated Loop, and switched fabric. All are based on gigabit speeds, with effective 100 megabyte per second throughput (200 megabyte full duplex). All allow for both copper and fiber optic cable plant, with maximum distances appropriate to the media (30m for copper, 500m for short-wave laser over multimode fiber, 10k for long-wave laser over singlemode fiber).
Point-to-point is a simple dedicated connection between two devices, and is used for minimal server/storage configurations. Point-to-point cabling typically runs directly from one device to another without an intervening hub using a subset of Fibre Channel protocol between the two devices. For additional devices, storage managers can extend the point-to-point cabling scheme, but since the media is no longer under the exclusive control of two nodes, Arbitrated Loop protocol must be used to negotiate access.
Figure 1 Point-to-Point Fibre Channel Connection
Fibre Channel-Arbitrated Loop is a shared gigabit media for up to 127 nodes (one of which may be attached to a switched fabric). Arbitrated Loop is analogous to Token Ring or FDDI, in that two communicating nodes possess the shared media only for the duration of a transaction, and then yield control to other nodes. Arbitrated Loop uses an additional subset of Fibre Channel commands to handle negotiating access to the loop, and specific sequences for assigning loop addresses (Arbitrated Loop Port Address, or AL_PA) to the nodes.
Arbitrated loops originally connected transmit and receive interfaces between multiple nodes creating an extended ring topology, similar to early Token Ring implementations. This configuration, however, was vulnerable to the same issues that surfaced in point-to-point LAN topologies. A break anywhere along the point-to-point chain would bring down the entire segment and would be very difficult to troubleshoot.
Arbitrated Loop hubs facilitate loop implementation by aggregating loop ports via a physical star configuration. Loop hubs typically provide 7 to 12 ports, and can be used to build larger loops via cascading. As with hubs in Ethernet and Token Ring LAN environments, Arbitrated Loop hubs provide greater control and reliability. Arbitrated Loop hubs employ bypass circuitry at each port to keep dysfunctional nodes from disrupting loop traffic. Vixel's hubs provide status and diagnostic LED's at each port, and the Vixel Rapport 2000 hub offers sophisticated loop integrity features and SNMP management.
Since one of the Arbitrated Loop node addresses is reserved for attachment to a Fibre Channel switch, a loop can participate in a broader network or fabric built with multiple switches and loops. The combination of Arbitrated Loop hubs and switches provides flexibility in allocating bandwidth and designing storage network segmentation.
Figure 2 A small Arbitrated Loop. The Vixel Rapport 1000 AL hub provides a physical star configuration for a logical loop data path over shared gigabit speeds.
A Fibre Channel switch typically provides 8 to 16 ports, with full gigabit speeds available at each port. Following the model previously established by Ethernet switches, a Fibre Channel switch port may be configured to support a single node or a shared segment of multiple nodes (e.g. a loop). Because a switch requires more processing power, memory and microcode at each port to properly route frames, switch per-port costs are usually >6 times Arbitrated Loop hub per-port costs.
Figure 3 Fabric Switch. The Vixel Rapport 4000 switch provides 100 MBps (200 MBps full duplex) per port for high end storage applications and Arbitrated Loop support
A Fibre Channel switching hub is a hybrid technology that offers the advantages of both Arbitrated Loop and fabric switching. A switching hub manages the address space of two or more Arbitrated Loop segments to create a larger, logical loop. This allows nodes on physically separate loops to transparently communicate with one another, while maintaining higher available bandwidth on each physical loop. Switching hub products optimize extended Arbitrated Loop performance. They also give some of the benefits of fabric switching at a favorable price point. Vixel's Rapport 3000 switching hub (4Q98) provides concurrent 100 MBps access between multiple physical Arbitrated Loops.
Figure 4 Switching Hub. The Vixel Rapport 3000 switching hub will provide a switched path between multiple physically separate Arbitrated Loops. The loops share a common address space, and so appear to end nodes as a single logical loop. Each loop enjoys its own 100 MBps transport between its own nodes and 100 MBps bandwidth to the other loops.
Customer Selection of Fibre Channel Products
Anyone with experience with storage interconnects will instantly see applications appropriate for Fibre Channel switches, switching hubs and hubs. A single Arbitrated Loop hub is a viable and cost-effective solution for two-to-12 node configurations of one or more servers and associated storage arrays. Multiple Arbitrated Loop hubs can be cascaded to build 10 to >60 node networks with reasonable bandwidth. Storage managers can design a combination of Arbitrated Loop hubs and switching hubs to balance individual loop performance with high availability between loops. Managers can also position a fabric switch to provide a combination of dedicated gigabit access with shared gigabit access via hubs and/or switching hubs.
Unfortunately, vendor positioning of Fibre Channel products is sometimes misleading. A vendor that only manufactures Arbitrated Loop hubs may be tempted to complain that Fibre Channel switches are far too expensive and complex to deploy. A vendor who only supplies switches may attempt to counter objections to the higher per port cost with denigrating remarks against Arbitrated Loop hubs. While healthy competition between vendors is essential for driving technology to higher levels, misinformation will not aid the adoption of Fibre Channel solutions for storage networking.
Just as today's enterprise networks continue to deploy a diversity of products and topologies for LAN and WAN solutions, scaling each component according to performance, application and cost, storage networking will extend on the basis of diverse products, including coexistence with legacy SCSI devices. Fibre Channel switches, switching hubs, hubs and Fibre Channel-to-SCSI bridges are not competing solutions to the same problem, but integral components of a single, more comprehensive solution.
Designing Storage Networks
There are a number of design considerations for a storage network that effect the initial selection of products and their deployment:
• Application requirements
• Protocol support
• Cable plant
• Distance between devices
• Number of devices
• Accommodation of legacy devices
• Anticipated growth
• Traffic volumes
• Departmental segmentation
• Redundancy
• Disaster recovery
• SNMP management
• Cost/performance ratio
All of these factors are interrelated and should be balanced for optimum use and cost savings.
Application Requirements
The first question that you should ask when designing a storage network is: "What is the application?" Full motion video, prepress graphics processing, relational database queries, data mining, server clustering, tape backup, disaster recovery, etc. each have significantly different bandwidth, port population, segmentation and distance requirements.
A single storage network design, moreover, may have to accommodate multiple applications concurrently, e.g. data mining + tape backup + disaster recovery. Designing for present and future storage applications requires an analysis of ongoing transaction needs and an understanding of what combination of Fibre Channel products can fulfill them. Fabric switches, switching hubs, and hubs, plus cabling options for short and long wave laser and copper, offer a rich toolset for building to both simple and complex storage application specifications.
Protocol Support
The most common protocol for storage networks is SCSI-3 over Fibre Channel. Vendors of host bus adapters (HBA's) and storage arrays rely on SCSI-3 to seamlessly replace parallel SCSI with Fibre Channel and universally provide device drivers for upper layer SCSI application support. It may also be desirable to run other storage protocols (e.g. HIPPI) or IP over Fibre Channel. Support of IP and other protocol stacks vary among HBA vendors, so you should select a particular HBA with both current and future protocol requirements in mind. In Arbitrated Loop, the upper level protocol is transparent to the hub or switching hub and so is not a consideration in hub selection.
Cable Plant
Fibre Channel supports both copper and optical fiber cabling, although fiber's immunity to EMI and support for longer distances makes it the clear choice for reliable connectivity. Fibre Channel vendors early on agreed to rename "Fiber" to "Fibre", so as not to exclude the use of copper.
For cost considerations, some vendors of Fibre Channel storage arrays, HBA's, and even Arbitrated Loop hubs have elected to provide only copper interfaces on some products. You needs to understand the distance limitations and EMI susceptibility of copper to properly provision these products, and you should take care to stay well within specified guidelines.
A passive copper interface provides no signal balancing, and should not exceed 15m in length. An active copper interface does provide signal balancing, and can be run as long as 30m. Vixel's hubs provide support for mixing the use of copper and optical fiber on a port by port basis (via GBIC's). For more information on this issue, read Guidelines for Implementing Fibre Channel with Copper, pub. 3/98, available at www.vixel.com.
Fibre Channel hub and switch products that offer copper-only ports may not only have EMI issues to overcome, but require external media interface adapters (MIA's) to support optical fiber. You need to be aware of this additional expense and reliability issue in the selection process.
Optical fiber cabling provides superior flexibility in storage network design. Multi-mode fiber can be run up to 500m; single-mode fiber can extend to 10k. Since optical fiber is immune to external EMI, storage managers can deploy it within or between buildings and extend storage networking across an entire campus.
Distance Between Devices
The physical organization of servers and storage devices may impact both the selection of cable plant on specific ports and the number of devices on any one Fibre Channel segment. In the majority of server/storage configurations, managers will locate nodes within the same room or building. For Arbitrated Loop, the total loop length can be quite large (10's of kilometers), although propagation delay through any media should be factored into network design.
The Fibre Channel specification for Arbitrated Loop, for example, allows up to 10k for long wave, singlemode fiber between nodes without retiming the signal. That does not mean, however, that you could design a network with 126 nodes separated by 10k each (a total of 1260 kilometers) and expect a robust, high bandwidth network. In practice, a 10k run might be extended on a few ports (e.g. for disaster recovery), with the majority of ports well under 500m.
Fabric switches and switching hubs (e.g. the Vixel Rapport 4000 and Rapport 3000 respectively) provide additional flexibility for extended distances on a per port basis. Since bandwidth is allocated per port, it would be possible to design multiple 10k runs.
Distance between nodes is a very real consideration when selecting copper or fiber interfaces. If device distance is at the threshold of copper's maximum length (e.g. 30m on active copper), it would be advisable to install fiber instead. Storage networking is the most business-critical network space, and nothing is gained by pushing recommended specifications to the maximum limit in order to cut costs.
Number of Devices
The address space for Fibre Channel switch fabric allows for millions of devices, more than adequate for large, enterprise networks. Switch products are typically 8 to 16 port configurations, but can integrate many more devices via Fabric Loop ports attached to Arbitrated Loop hubs, and switching hubs or extensions to additional switches.
Arbitrated Loop provides 127 addresses per loop, with one address reserved for switch attachment. Theoretically, it is possible to cascade multiple hubs to create a 126 node loop. Practically, such a configuration would not support normal server/storage transactions, since each node represents a latency factor against loop performance. It would be more efficient use of bandwidth to configure multiple loops and employ a switching hub or switch to connect them.
Typical Arbitrated Loops contain 3 to 12 nodes. You should not exceed a maximum of 50-60 nodes to insure high bandwidth and availability. If you require additional nodes within the same logical address space, a switching hub can provide transparent activity between loop segments without violating bandwidth requirements for each loop segment.
Your device count should also include the internal configuration of disk arrays. If a disk array (e.g. JBOD) uses Arbitrated Loop as an internal architecture for linking disks, and the internal loop is connected to the external loop port, then each disk within an enclosure should be counted as a node.
Accommodation of Legacy Devices
A well-conceived migration path from parallel SCSI to Fibre Channel may leverage a network's current investment in SCSI disks and tape subsystems by employing Fibre Channel-to-SCSI bridges (sometimes referred to as FC-SCSI 'routers'). FC-SCSI bridges offer a means to integrate parallel SCSI into a storage network design with minimal religious warfare between old and new technology proponents.
FC-SCSI bridges provide both Fibre Channel and parallel SCSI interfaces, and handle the conversion between serial SCSI-3 and previous versions of SCSI protocol. A FC-provisioned server can therefore transparently access storage or tape regardless of the ultimate downstream interface.
Computer networks continue to grow exponentially. Your selection of switch and hub products for current requirements should therefore accommodate projected growth and thus maximize the initial investment. Your initial storage network may require only a single 7 port Arbitrated Loop hub (3 servers + 3-4 storage arrays). Depending on anticipated bandwidth and port requirements, however, a switching hub might be a more suitable investment. Additional devices could then be accommodated with hubs without subsequently dividing the bandwidth of a single Arbitrated Loop.
Traffic volumes By calculating average frame size, buffer capacity on both servers and storage arrays, number of active nodes, etc. it is possible to determine a reasonable configuration given traffic volumes. As in LAN internetworking, however, you continually adjust Fibre Channel bandwidth provisioning in response to user application's changes. Full motion audio/video applications, for example, are better served by switching hubs or switches since each port can deliver 100 MBps throughput. You can easily support typical SQL database applications, on the other hand, with medium-to-large Arbitrated Loop configurations that would be difficult to cost justify using dedicated switch ports.
Departmental segmentation
Fibre Channel offers a number of solutions for segmentation of storage on a departmental or workgroup basis. A fabric switch can provide a backbone for tying multiple Arbitrated Loops together. A manufacturing department and engineering workgroup may have separate servers and storage arrays, but occasionally require access to each other's data. Each department could install Arbitrated Loops, with a port on each loop connected to the fabric switch. Each would preserve a shared 100 megabyte bandwidth for their own transactions, with a switched 100 megabyte pipe between them.
In addition, Arbitrated Loop addressing provides both public and private address schemes, which allows selected devices on the same loop to operate in conjunction with or in isolation from a switched fabric. A department that requires access to the fabric (via public addresses) can share the same loop topology as a department that has only local requirements (via private addresses).
The Vixel Rapport 4000 fabric switch provides additional means to functionally segment attached nodes. The Rapport 4000's Zoning implementation allows individual ports to be associated for mutual transactions in isolation from other ports. This configuration in effect creates a virtual private storage network between the assigned devices. The Rapport 4000 also offers a Stealth mode for attached Arbitrated Loops. You can define multiple physically separate loops as a single logical loop, thus supplying the functionality of a switching hub within the fabric switch.
Redundancy Traditional concepts of redundancy focus on backup (or load sharing) power supplies and multiple fans. These components, however, are rarely the cause of network disruption. Loss of network availability is more often caused by the erratic behavior of an attached node or breaks in the cable plant that either downs the segment or sends it into suspended animation.
You can achieve high availability for storage networking by configuring dual Arbitrated Loops. If one loop fails (e.g. a marginal HBA takes the loop down), the redundant loop provides an alternate path between devices. You can also implement this dual-path configuration with Fibre Channel switches.
Figure 5 A dual loop configuration using Vixel Rapport 1000 hubs. Each storage array and server is provisioned with dual Fibre Channel interfaces. If the primary loop fails, the redundant loop insures availability.
Disaster recovery
Disaster recovery (sometimes known under less alarming euphemisms) was at best difficult to achieve with parallel SCSI storage configurations. Typically, storage managers use servers connected to high speed routers to periodically backup data to a remote disaster recovery site.
Fibre Channel's support for 10k lengths at 100 megabytes per second offers a superior solution for disaster recovery. You can back-up or mirror an entire storage configuration via a switch, switching hub or Arbitrated Loop hub connection to a remote, off-campus location.
By taking backups and disaster recovery transfers off the LAN, Fibre Channel offers a significant performance benefit: Not only is the transfer rate much higher (to 100 MBps), but data can be transferred in native SCSI protocol without the overhead of TCP/IP or LAN protocol conversion.
SNMP management
SNMP management support is by now a given for all enterprise-level LAN/WAN devices. No vendor would offer an Ethernet switch or Frame Relay product into a business-critical environment without full MIB-II support and extensive vendor-specific MIB extensions.
Although servers may provide SNMP statistics and storage arrays may provide SCSI Enclosure Services data via SNMP platforms, storage networking is a relative newcomer to SNMP requirements. Vixel's Rapport 2000 is the only Arbitrated Loop hub to provide a full featured and robust SNMP implementation on par with mature and sophisticated enterprise router or LAN switch products.
Designing manageability into a storage network adds costs to the individual components, but significantly reduces overall operational costs. Simplifying device configuration, integrating diagnostic and rapid recovery features, and reducing support requirements all contribute to cost savings. The greatest savings that manageability offers, however, is in maximizing system uptime. When networks go down, companies lose money. For most companies, an hour or two of downtime during business hours would have paid for manageability many times over.
If SNMP management is a mandatory requirement for an enterprise network's design criteria, you should carefully evaluate the level of SNMP support of a Fibre Channel hub or switch. If a vendor fulfills only the minimal SNMP device and enclosure status features in order to qualify as "SNMP managed," its products will be insufficiently manageable to maintain stable operation.
Figure 6 Vixel's Rapport 2000 SNMP managed hub provides sophisticated diagnostics for trouble-shooting Arbitrated Loop problems. A combination of hardware and software features facilitates automatic problem detection, isolation and recovery to maximize loop availability.
Vixel has produced an additional white paper, "A Management Strategy for Fibre Channel Loop", which examines SNMP requirements and diagnostic capability essential for high availability storage networks.
Cost/performance ratio The decision to implement a new technology or upgrade is always a business-based decision. A company calculates the return on investment (ROI) on cost of implementation versus the increased performance (number of business transactions, i.e. dollars, per second) that the implementation provides.
Fibre Channel as an infrastructure has a favorable return on investment, since it is the only viable technology refresh for storage that allows faster servers and massive databases. The more transactions per second, the higher the ROI.
Within the Fibre Channel suite of products, application requirements drive the return on investment for specific configurations, cost per port, manageability, reliability, vendor support, and other factors. This is not easily determined by a simple formula, but as healthy competition between Fibre Channel vendors raises the standards for product features and concurrently lowers the overall implementation costs, designers of storage networks will have even more enviable choices to make.
Summary
Fibre Channel provides a robust, high availability, flexible and cost effective solution to the contradiction between server/storage innovations and legacy parallel SCSI transport. Fibre Channel switches, switching hubs, Arbitrated Loop hubs, and FC-SCSI bridges all play an essential part in storage network design. Despite occasional excesses in vendor rhetoric, both fabric switch and loop technology will contribute to the success of this breakthrough technology.
Server and storage technology is undergoing a fundamental change that promises significant increases in processing power, availability, and distributed access to massive amounts of data. Multi-processor servers, high performance backplanes, server clustering applications, the physical separation of storage from servers, and multi-terabyte capacity of storage arrays are rapidly exceeding the capabilities of the traditional SCSI architecture that previously bound servers and storage together.
Fortunately, technical innovations in server and storage products have not occurred in isolation. Interconnect technology has undergone its own metamorphosis, with SCSI protocol over parallel bus re-emerging as a serial transport over Fibre Channel. This convergence of new server designs, server clustering, autonomous gigabyte storage arrays, and Fibre Channel architecture is raising the server/storage paradigm to a higher plateau: storage networking. From the vantage point of storage networking, managers of Enterprise networks can now envision and implement innovative designs to enhance data availability and accommodate the growing demands of their users.
Fibre Channel Topologies
Fibre Channel architecture offers three topologies for network design: point-to-point, Arbitrated Loop, and switched fabric. All are based on gigabit speeds, with effective 100 megabyte per second throughput (200 megabyte full duplex). All allow for both copper and fiber optic cable plant, with maximum distances appropriate to the media (30m for copper, 500m for short-wave laser over multimode fiber, 10k for long-wave laser over singlemode fiber).
Point-to-point is a simple dedicated connection between two devices, and is used for minimal server/storage configurations. Point-to-point cabling typically runs directly from one device to another without an intervening hub using a subset of Fibre Channel protocol between the two devices. For additional devices, storage managers can extend the point-to-point cabling scheme, but since the media is no longer under the exclusive control of two nodes, Arbitrated Loop protocol must be used to negotiate access.
Figure 1 Point-to-Point Fibre Channel Connection
Fibre Channel-Arbitrated Loop is a shared gigabit media for up to 127 nodes (one of which may be attached to a switched fabric). Arbitrated Loop is analogous to Token Ring or FDDI, in that two communicating nodes possess the shared media only for the duration of a transaction, and then yield control to other nodes. Arbitrated Loop uses an additional subset of Fibre Channel commands to handle negotiating access to the loop, and specific sequences for assigning loop addresses (Arbitrated Loop Port Address, or AL_PA) to the nodes.
Arbitrated loops originally connected transmit and receive interfaces between multiple nodes creating an extended ring topology, similar to early Token Ring implementations. This configuration, however, was vulnerable to the same issues that surfaced in point-to-point LAN topologies. A break anywhere along the point-to-point chain would bring down the entire segment and would be very difficult to troubleshoot.
Arbitrated Loop hubs facilitate loop implementation by aggregating loop ports via a physical star configuration. Loop hubs typically provide 7 to 12 ports, and can be used to build larger loops via cascading. As with hubs in Ethernet and Token Ring LAN environments, Arbitrated Loop hubs provide greater control and reliability. Arbitrated Loop hubs employ bypass circuitry at each port to keep dysfunctional nodes from disrupting loop traffic. Vixel's hubs provide status and diagnostic LED's at each port, and the Vixel Rapport 2000 hub offers sophisticated loop integrity features and SNMP management.
Since one of the Arbitrated Loop node addresses is reserved for attachment to a Fibre Channel switch, a loop can participate in a broader network or fabric built with multiple switches and loops. The combination of Arbitrated Loop hubs and switches provides flexibility in allocating bandwidth and designing storage network segmentation.
Figure 2 A small Arbitrated Loop. The Vixel Rapport 1000 AL hub provides a physical star configuration for a logical loop data path over shared gigabit speeds.
A Fibre Channel switch typically provides 8 to 16 ports, with full gigabit speeds available at each port. Following the model previously established by Ethernet switches, a Fibre Channel switch port may be configured to support a single node or a shared segment of multiple nodes (e.g. a loop). Because a switch requires more processing power, memory and microcode at each port to properly route frames, switch per-port costs are usually >6 times Arbitrated Loop hub per-port costs.
Figure 3 Fabric Switch. The Vixel Rapport 4000 switch provides 100 MBps (200 MBps full duplex) per port for high end storage applications and Arbitrated Loop support
A Fibre Channel switching hub is a hybrid technology that offers the advantages of both Arbitrated Loop and fabric switching. A switching hub manages the address space of two or more Arbitrated Loop segments to create a larger, logical loop. This allows nodes on physically separate loops to transparently communicate with one another, while maintaining higher available bandwidth on each physical loop. Switching hub products optimize extended Arbitrated Loop performance. They also give some of the benefits of fabric switching at a favorable price point. Vixel's Rapport 3000 switching hub (4Q98) provides concurrent 100 MBps access between multiple physical Arbitrated Loops.
Figure 4 Switching Hub. The Vixel Rapport 3000 switching hub will provide a switched path between multiple physically separate Arbitrated Loops. The loops share a common address space, and so appear to end nodes as a single logical loop. Each loop enjoys its own 100 MBps transport between its own nodes and 100 MBps bandwidth to the other loops.
Customer Selection of Fibre Channel Products
Anyone with experience with storage interconnects will instantly see applications appropriate for Fibre Channel switches, switching hubs and hubs. A single Arbitrated Loop hub is a viable and cost-effective solution for two-to-12 node configurations of one or more servers and associated storage arrays. Multiple Arbitrated Loop hubs can be cascaded to build 10 to >60 node networks with reasonable bandwidth. Storage managers can design a combination of Arbitrated Loop hubs and switching hubs to balance individual loop performance with high availability between loops. Managers can also position a fabric switch to provide a combination of dedicated gigabit access with shared gigabit access via hubs and/or switching hubs.
Unfortunately, vendor positioning of Fibre Channel products is sometimes misleading. A vendor that only manufactures Arbitrated Loop hubs may be tempted to complain that Fibre Channel switches are far too expensive and complex to deploy. A vendor who only supplies switches may attempt to counter objections to the higher per port cost with denigrating remarks against Arbitrated Loop hubs. While healthy competition between vendors is essential for driving technology to higher levels, misinformation will not aid the adoption of Fibre Channel solutions for storage networking.
Just as today's enterprise networks continue to deploy a diversity of products and topologies for LAN and WAN solutions, scaling each component according to performance, application and cost, storage networking will extend on the basis of diverse products, including coexistence with legacy SCSI devices. Fibre Channel switches, switching hubs, hubs and Fibre Channel-to-SCSI bridges are not competing solutions to the same problem, but integral components of a single, more comprehensive solution.
Designing Storage Networks
There are a number of design considerations for a storage network that effect the initial selection of products and their deployment:
• Application requirements
• Protocol support
• Cable plant
• Distance between devices
• Number of devices
• Accommodation of legacy devices
• Anticipated growth
• Traffic volumes
• Departmental segmentation
• Redundancy
• Disaster recovery
• SNMP management
• Cost/performance ratio
All of these factors are interrelated and should be balanced for optimum use and cost savings.
Application Requirements
The first question that you should ask when designing a storage network is: "What is the application?" Full motion video, prepress graphics processing, relational database queries, data mining, server clustering, tape backup, disaster recovery, etc. each have significantly different bandwidth, port population, segmentation and distance requirements.
A single storage network design, moreover, may have to accommodate multiple applications concurrently, e.g. data mining + tape backup + disaster recovery. Designing for present and future storage applications requires an analysis of ongoing transaction needs and an understanding of what combination of Fibre Channel products can fulfill them. Fabric switches, switching hubs, and hubs, plus cabling options for short and long wave laser and copper, offer a rich toolset for building to both simple and complex storage application specifications.
Protocol Support
The most common protocol for storage networks is SCSI-3 over Fibre Channel. Vendors of host bus adapters (HBA's) and storage arrays rely on SCSI-3 to seamlessly replace parallel SCSI with Fibre Channel and universally provide device drivers for upper layer SCSI application support. It may also be desirable to run other storage protocols (e.g. HIPPI) or IP over Fibre Channel. Support of IP and other protocol stacks vary among HBA vendors, so you should select a particular HBA with both current and future protocol requirements in mind. In Arbitrated Loop, the upper level protocol is transparent to the hub or switching hub and so is not a consideration in hub selection.
Cable Plant
Fibre Channel supports both copper and optical fiber cabling, although fiber's immunity to EMI and support for longer distances makes it the clear choice for reliable connectivity. Fibre Channel vendors early on agreed to rename "Fiber" to "Fibre", so as not to exclude the use of copper.
For cost considerations, some vendors of Fibre Channel storage arrays, HBA's, and even Arbitrated Loop hubs have elected to provide only copper interfaces on some products. You needs to understand the distance limitations and EMI susceptibility of copper to properly provision these products, and you should take care to stay well within specified guidelines.
A passive copper interface provides no signal balancing, and should not exceed 15m in length. An active copper interface does provide signal balancing, and can be run as long as 30m. Vixel's hubs provide support for mixing the use of copper and optical fiber on a port by port basis (via GBIC's). For more information on this issue, read Guidelines for Implementing Fibre Channel with Copper, pub. 3/98, available at www.vixel.com.
Fibre Channel hub and switch products that offer copper-only ports may not only have EMI issues to overcome, but require external media interface adapters (MIA's) to support optical fiber. You need to be aware of this additional expense and reliability issue in the selection process.
Optical fiber cabling provides superior flexibility in storage network design. Multi-mode fiber can be run up to 500m; single-mode fiber can extend to 10k. Since optical fiber is immune to external EMI, storage managers can deploy it within or between buildings and extend storage networking across an entire campus.
Distance Between Devices
The physical organization of servers and storage devices may impact both the selection of cable plant on specific ports and the number of devices on any one Fibre Channel segment. In the majority of server/storage configurations, managers will locate nodes within the same room or building. For Arbitrated Loop, the total loop length can be quite large (10's of kilometers), although propagation delay through any media should be factored into network design.
The Fibre Channel specification for Arbitrated Loop, for example, allows up to 10k for long wave, singlemode fiber between nodes without retiming the signal. That does not mean, however, that you could design a network with 126 nodes separated by 10k each (a total of 1260 kilometers) and expect a robust, high bandwidth network. In practice, a 10k run might be extended on a few ports (e.g. for disaster recovery), with the majority of ports well under 500m.
Fabric switches and switching hubs (e.g. the Vixel Rapport 4000 and Rapport 3000 respectively) provide additional flexibility for extended distances on a per port basis. Since bandwidth is allocated per port, it would be possible to design multiple 10k runs.
Distance between nodes is a very real consideration when selecting copper or fiber interfaces. If device distance is at the threshold of copper's maximum length (e.g. 30m on active copper), it would be advisable to install fiber instead. Storage networking is the most business-critical network space, and nothing is gained by pushing recommended specifications to the maximum limit in order to cut costs.
Number of Devices
The address space for Fibre Channel switch fabric allows for millions of devices, more than adequate for large, enterprise networks. Switch products are typically 8 to 16 port configurations, but can integrate many more devices via Fabric Loop ports attached to Arbitrated Loop hubs, and switching hubs or extensions to additional switches.
Arbitrated Loop provides 127 addresses per loop, with one address reserved for switch attachment. Theoretically, it is possible to cascade multiple hubs to create a 126 node loop. Practically, such a configuration would not support normal server/storage transactions, since each node represents a latency factor against loop performance. It would be more efficient use of bandwidth to configure multiple loops and employ a switching hub or switch to connect them.
Typical Arbitrated Loops contain 3 to 12 nodes. You should not exceed a maximum of 50-60 nodes to insure high bandwidth and availability. If you require additional nodes within the same logical address space, a switching hub can provide transparent activity between loop segments without violating bandwidth requirements for each loop segment.
Your device count should also include the internal configuration of disk arrays. If a disk array (e.g. JBOD) uses Arbitrated Loop as an internal architecture for linking disks, and the internal loop is connected to the external loop port, then each disk within an enclosure should be counted as a node.
Accommodation of Legacy Devices
A well-conceived migration path from parallel SCSI to Fibre Channel may leverage a network's current investment in SCSI disks and tape subsystems by employing Fibre Channel-to-SCSI bridges (sometimes referred to as FC-SCSI 'routers'). FC-SCSI bridges offer a means to integrate parallel SCSI into a storage network design with minimal religious warfare between old and new technology proponents.
FC-SCSI bridges provide both Fibre Channel and parallel SCSI interfaces, and handle the conversion between serial SCSI-3 and previous versions of SCSI protocol. A FC-provisioned server can therefore transparently access storage or tape regardless of the ultimate downstream interface.
Computer networks continue to grow exponentially. Your selection of switch and hub products for current requirements should therefore accommodate projected growth and thus maximize the initial investment. Your initial storage network may require only a single 7 port Arbitrated Loop hub (3 servers + 3-4 storage arrays). Depending on anticipated bandwidth and port requirements, however, a switching hub might be a more suitable investment. Additional devices could then be accommodated with hubs without subsequently dividing the bandwidth of a single Arbitrated Loop.
Traffic volumes By calculating average frame size, buffer capacity on both servers and storage arrays, number of active nodes, etc. it is possible to determine a reasonable configuration given traffic volumes. As in LAN internetworking, however, you continually adjust Fibre Channel bandwidth provisioning in response to user application's changes. Full motion audio/video applications, for example, are better served by switching hubs or switches since each port can deliver 100 MBps throughput. You can easily support typical SQL database applications, on the other hand, with medium-to-large Arbitrated Loop configurations that would be difficult to cost justify using dedicated switch ports.
Departmental segmentation
Fibre Channel offers a number of solutions for segmentation of storage on a departmental or workgroup basis. A fabric switch can provide a backbone for tying multiple Arbitrated Loops together. A manufacturing department and engineering workgroup may have separate servers and storage arrays, but occasionally require access to each other's data. Each department could install Arbitrated Loops, with a port on each loop connected to the fabric switch. Each would preserve a shared 100 megabyte bandwidth for their own transactions, with a switched 100 megabyte pipe between them.
In addition, Arbitrated Loop addressing provides both public and private address schemes, which allows selected devices on the same loop to operate in conjunction with or in isolation from a switched fabric. A department that requires access to the fabric (via public addresses) can share the same loop topology as a department that has only local requirements (via private addresses).
The Vixel Rapport 4000 fabric switch provides additional means to functionally segment attached nodes. The Rapport 4000's Zoning implementation allows individual ports to be associated for mutual transactions in isolation from other ports. This configuration in effect creates a virtual private storage network between the assigned devices. The Rapport 4000 also offers a Stealth mode for attached Arbitrated Loops. You can define multiple physically separate loops as a single logical loop, thus supplying the functionality of a switching hub within the fabric switch.
Redundancy Traditional concepts of redundancy focus on backup (or load sharing) power supplies and multiple fans. These components, however, are rarely the cause of network disruption. Loss of network availability is more often caused by the erratic behavior of an attached node or breaks in the cable plant that either downs the segment or sends it into suspended animation.
You can achieve high availability for storage networking by configuring dual Arbitrated Loops. If one loop fails (e.g. a marginal HBA takes the loop down), the redundant loop provides an alternate path between devices. You can also implement this dual-path configuration with Fibre Channel switches.
Figure 5 A dual loop configuration using Vixel Rapport 1000 hubs. Each storage array and server is provisioned with dual Fibre Channel interfaces. If the primary loop fails, the redundant loop insures availability.
Disaster recovery
Disaster recovery (sometimes known under less alarming euphemisms) was at best difficult to achieve with parallel SCSI storage configurations. Typically, storage managers use servers connected to high speed routers to periodically backup data to a remote disaster recovery site.
Fibre Channel's support for 10k lengths at 100 megabytes per second offers a superior solution for disaster recovery. You can back-up or mirror an entire storage configuration via a switch, switching hub or Arbitrated Loop hub connection to a remote, off-campus location.
By taking backups and disaster recovery transfers off the LAN, Fibre Channel offers a significant performance benefit: Not only is the transfer rate much higher (to 100 MBps), but data can be transferred in native SCSI protocol without the overhead of TCP/IP or LAN protocol conversion.
SNMP management
SNMP management support is by now a given for all enterprise-level LAN/WAN devices. No vendor would offer an Ethernet switch or Frame Relay product into a business-critical environment without full MIB-II support and extensive vendor-specific MIB extensions.
Although servers may provide SNMP statistics and storage arrays may provide SCSI Enclosure Services data via SNMP platforms, storage networking is a relative newcomer to SNMP requirements. Vixel's Rapport 2000 is the only Arbitrated Loop hub to provide a full featured and robust SNMP implementation on par with mature and sophisticated enterprise router or LAN switch products.
Designing manageability into a storage network adds costs to the individual components, but significantly reduces overall operational costs. Simplifying device configuration, integrating diagnostic and rapid recovery features, and reducing support requirements all contribute to cost savings. The greatest savings that manageability offers, however, is in maximizing system uptime. When networks go down, companies lose money. For most companies, an hour or two of downtime during business hours would have paid for manageability many times over.
If SNMP management is a mandatory requirement for an enterprise network's design criteria, you should carefully evaluate the level of SNMP support of a Fibre Channel hub or switch. If a vendor fulfills only the minimal SNMP device and enclosure status features in order to qualify as "SNMP managed," its products will be insufficiently manageable to maintain stable operation.
Figure 6 Vixel's Rapport 2000 SNMP managed hub provides sophisticated diagnostics for trouble-shooting Arbitrated Loop problems. A combination of hardware and software features facilitates automatic problem detection, isolation and recovery to maximize loop availability.
Vixel has produced an additional white paper, "A Management Strategy for Fibre Channel Loop", which examines SNMP requirements and diagnostic capability essential for high availability storage networks.
Cost/performance ratio The decision to implement a new technology or upgrade is always a business-based decision. A company calculates the return on investment (ROI) on cost of implementation versus the increased performance (number of business transactions, i.e. dollars, per second) that the implementation provides.
Fibre Channel as an infrastructure has a favorable return on investment, since it is the only viable technology refresh for storage that allows faster servers and massive databases. The more transactions per second, the higher the ROI.
Within the Fibre Channel suite of products, application requirements drive the return on investment for specific configurations, cost per port, manageability, reliability, vendor support, and other factors. This is not easily determined by a simple formula, but as healthy competition between Fibre Channel vendors raises the standards for product features and concurrently lowers the overall implementation costs, designers of storage networks will have even more enviable choices to make.
Summary
Fibre Channel provides a robust, high availability, flexible and cost effective solution to the contradiction between server/storage innovations and legacy parallel SCSI transport. Fibre Channel switches, switching hubs, Arbitrated Loop hubs, and FC-SCSI bridges all play an essential part in storage network design. Despite occasional excesses in vendor rhetoric, both fabric switch and loop technology will contribute to the success of this breakthrough technology.
