The purpose of this article is to:
- Provide an introduction to the InfiniBand (IB) networking specification and architecture
- Highlight important vendors (i.e. Mellanox) and industry contributors (such as Broadcom, Microsoft, etc.)
What is InfiniBand?
- InfiniBand is an industry standard, channel-based, switched fabric interconnect architecture for server and storage connectivity
- InfiniBand devices connect over either copper or optical fiber bidirectional serial cables
- InfiniBand allows for reliable messaging and memory manipulation (RDMA) without in-transit software intervention
- InfiniBand comprises an ecosystem of devices as well as cross-platform, open-source management software that has been developed in collaboration with industry partners
What is RDMA?
- RDMA stands for Remote Direct Memory Access
- This allows for storing and retrieving of data directly from the memory of the nodes in the network
- InfiniBand adapter cards have on-board processing power that can aggregate and send data between two or more nodes while minimizing CPU utilization
What are the key advantages of InfiniBand over traditional Ethernet?
- High Bandwidth: Up to 12 combined lanes of throughput, with each serial link (individual lane) ranging from 2.5 Gb/s to 25 Gb/s speeds (planned)
- Low Latency: All layers up to Application layer are optimized to reduce latency
- Flexibility and Scalability: Multiple applications in data centers and HPC, with scalability up to 48,000 nodes in a single subnet
- Quality of Service (QoS): I/O channel definitions at the adapter level, virtual lane definitions at the link level
- CPU Offloads: RDMA support, hardware-based transport protocol, kernel bypass, reliable transport protocol (reduced packet loss)
- Simplified Management: Centralized route management through Subnet Manager, enables Software-Defined Networking (SDN) and plug-and-play environment for nodes in the network
InfiniBand Bandwidth Roadmap: Where are we headed next?
What does an IB Network look like?
Here is an example from Mellanox Academy's free, introductory course, "InfiniBand Essentials".
Source: http://academy.mellanox.com ; register for free to view courses and learn more about InfiniBand and RDMA
InfiniBand Trade Association FAQ Highlights:
InfiniBand enables high-performance message passing and I/O to accelerate computation and storage of large datasets. This is well-suited to applications such as:
- Bioscience and drug research
- Data mining
- Digital rendering
- Electronic design automation
- Fluid dynamics and weather analysis
Additional HPC applications can be found in the areas of:
- Fraud detection
- Virtualization and web services
- Financial services
- Insurance services
How does InfiniBand fit into a virtualized data center?
- Trends in virtualization are driving the need for more I/O connectivity per physical server
- Typical VMware ESXi server environments require multiple Gigabit Ethernet NICs and Fibre Channel HBAs, which is costly and increasingly difficult to manage over time
- InfiniBand I/O virtualization can ease this burden through unification of I/O in the compute server farm
- Unifying and virtualizing I/O enables significantly higher LAN and SAN performance from virtual machines
- IB allows for easy segregation of the compute, LAN and SAN domains for independent resource management and scaling
What is RDMA over Converged Ethernet (RoCE)?
- RoCE (pronounced "rocky") is an industry standard transport protocol that enables Remote Direct Memory Access (RDMA) to operate on ordinary Ethernet layer 2 and 3 networks.
- RDMA enables server-to-server data movement directly between application memory without any CPU involvement, resulting in performance and efficiency gains, while also significantly reducing latency.
- RoCE allows RDMA to run over traditional ethernet as well as InfiniBand architectures
What is the relationship between InfiniBand and Fibre Channel or Gigabit Ethernet?
- InfiniBand is complementary to Fibre Channel and Gigabit Ethernet but offers higher performance and better I/O efficiency
- InfiniBand is uniquely positioned to become the I/O interconnect of choice and is replacing Fibre Channel in many data centers
- Ethernet connects seamlessly into the edge of the InfiniBand fabric and benefits from better access to InfiniBand architecture-enabled compute resources
What type of cabling does InfiniBand support?
IB supports both active and passive copper (up to 30 meters long) and fiber-optic cabling (up to 10km long).
Additional Highlights and Advantages of InfiniBand Architectures:
- Currently more than 47% of the TOP500 supercomputers leverage the InfiniBand interconnect
- Latencies can be as low as 1µs end-to-end
- Networking, clustering, and storage can be consolidated and administered over a single fabric
- Fully redundant and lossless I/O fabrics with automatic path failover and link layer multi-pathing abilities for high availability
- Data integrity through Cyclic Redundancy Checks (CRCs) at each fabric hop and end-to-end across the fabric
- Open-source management software developed collaboratively through organizations such as the OpenFabrics Alliance (see OFA section below)
- IBTA compliance and interoperability testing conducted by the OpenFabrics Alliance, promotes product choice and vendor independence
Infiniband Trade Association (IBTA) Steering Committee Roster:
OpenFabrics Alliance (OFA):
The OpenFabrics Alliance (OFA) is an open source-based organization that develops, tests, licenses, supports and distributes OpenFabrics Software (OFS).
The Alliance’s mission is to develop and promote software that enables maximum application efficiency by delivering wire-speed messaging, ultra-low latencies and maximum bandwidth directly to applications with minimal CPU overhead.
The OFA has promoted and facilitated the development of:
- Vendor-Neutral Linux-based InfiniBand software stack
- Windows support for IB software stack
- iWARP support
- RoCE (RDMA over Converged Ethernet) support
OFA Promoters include:
- Emulex (Avago)
- Lawrence Livermore National Laboratory (LLNL)
- Los Alamos National Laboratory (LANL)
- Oak Ridge National Laboratory (ORNL)
- Sandia National Laboratories (SNL)
OFA Adopters include:
- Argonne National Laboratory (ANL)
- Mellanox Academy: http://academy.mellanox.com/
- OpenFabrics Alliance Training: https://www.openfabrics.org/index.php/training-file-downloads.html