Menu

Subscribe

A Primer on InfiniBand

on IB, networking, protocols, Mellanox, InfiniBand

The purpose of this article is to:

  1. Provide an introduction to the InfiniBand (IB) networking specification and architecture
  2. Highlight important vendors (i.e. Mellanox) and industry contributors (such as Broadcom, Microsoft, etc.)

What is InfiniBand?

  • InfiniBand is an industry standard, channel-based, switched fabric interconnect architecture for server and storage connectivity
  • InfiniBand devices connect over either copper or optical fiber bidirectional serial cables
  • InfiniBand allows for reliable messaging and memory manipulation (RDMA) without in-transit software intervention
  • InfiniBand comprises an ecosystem of devices as well as cross-platform, open-source management software that has been developed in collaboration with industry partners

What is RDMA?

  • RDMA stands for Remote Direct Memory Access
  • This allows for storing and retrieving of data directly from the memory of the nodes in the network
  • InfiniBand adapter cards have on-board processing power that can aggregate and send data between two or more nodes while minimizing CPU utilization

What are the key advantages of InfiniBand over traditional Ethernet?

  • High Bandwidth: Up to 12 combined lanes of throughput, with each serial link (individual lane) ranging from 2.5 Gb/s to 25 Gb/s speeds (planned)
  • Low Latency: All layers up to Application layer are optimized to reduce latency
  • Flexibility and Scalability: Multiple applications in data centers and HPC, with scalability up to 48,000 nodes in a single subnet
  • Quality of Service (QoS): I/O channel definitions at the adapter level, virtual lane definitions at the link level
  • CPU Offloads: RDMA support, hardware-based transport protocol, kernel bypass, reliable transport protocol (reduced packet loss)
  • Simplified Management: Centralized route management through Subnet Manager, enables Software-Defined Networking (SDN) and plug-and-play environment for nodes in the network

InfiniBand Bandwidth Roadmap: Where are we headed next?

InfiniBand Bandwidth Roadmap

Source: https://www.nextplatform.com/2017/01/30/relentless-yet-predictable-pace-infiniband-speed-bumps/

What does an IB Network look like?

Here is an example from Mellanox Academy's free, introductory course, "InfiniBand Essentials".

Mellanox Academy IB Fabric Components

Source: http://academy.mellanox.com ; register for free to view courses and learn more about InfiniBand and RDMA

InfiniBand Trade Association FAQ Highlights:

Source: http://www.infinibandta.org/content/pages.php?pg=technology_faq

InfiniBand enables high-performance message passing and I/O to accelerate computation and storage of large datasets. This is well-suited to applications such as:

  • Bioscience and drug research
  • Data mining
  • Digital rendering
  • Electronic design automation
  • Fluid dynamics and weather analysis

Additional HPC applications can be found in the areas of:

  • Fraud detection
  • Database
  • Virtualization and web services
  • Financial services
  • Insurance services

How does InfiniBand fit into a virtualized data center?

  • Trends in virtualization are driving the need for more I/O connectivity per physical server
  • Typical VMware ESXi server environments require multiple Gigabit Ethernet NICs and Fibre Channel HBAs, which is costly and increasingly difficult to manage over time
  • InfiniBand I/O virtualization can ease this burden through unification of I/O in the compute server farm
  • Unifying and virtualizing I/O enables significantly higher LAN and SAN performance from virtual machines
  • IB allows for easy segregation of the compute, LAN and SAN domains for independent resource management and scaling

What is RDMA over Converged Ethernet (RoCE)?

  • RoCE (pronounced "rocky") is an industry standard transport protocol that enables Remote Direct Memory Access (RDMA) to operate on ordinary Ethernet layer 2 and 3 networks.
  • RDMA enables server-to-server data movement directly between application memory without any CPU involvement, resulting in performance and efficiency gains, while also significantly reducing latency.
  • RoCE allows RDMA to run over traditional ethernet as well as InfiniBand architectures

What is the relationship between InfiniBand and Fibre Channel or Gigabit Ethernet?

  • InfiniBand is complementary to Fibre Channel and Gigabit Ethernet but offers higher performance and better I/O efficiency
  • InfiniBand is uniquely positioned to become the I/O interconnect of choice and is replacing Fibre Channel in many data centers
  • Ethernet connects seamlessly into the edge of the InfiniBand fabric and benefits from better access to InfiniBand architecture-enabled compute resources

What type of cabling does InfiniBand support?

IB supports both active and passive copper (up to 30 meters long) and fiber-optic cabling (up to 10km long).

Additional Highlights and Advantages of InfiniBand Architectures:

  • Currently more than 47% of the TOP500 supercomputers leverage the InfiniBand interconnect
  • Latencies can be as low as 1µs end-to-end
  • Networking, clustering, and storage can be consolidated and administered over a single fabric
  • Fully redundant and lossless I/O fabrics with automatic path failover and link layer multi-pathing abilities for high availability
  • Data integrity through Cyclic Redundancy Checks (CRCs) at each fabric hop and end-to-end across the fabric
  • Open-source management software developed collaboratively through organizations such as the OpenFabrics Alliance (see OFA section below)
  • IBTA compliance and interoperability testing conducted by the OpenFabrics Alliance, promotes product choice and vendor independence

Infiniband Trade Association (IBTA) Steering Committee Roster:

  • Cray
  • Broadcom
  • HP
  • IBM
  • Intel
  • Mellanox
  • Microsoft
  • Oracle
  • QLogic

Source: http://infinibandta.org/content/pages.php?pg=about_us_infiniband

OpenFabrics Alliance (OFA):

The OpenFabrics Alliance (OFA) is an open source-based organization that develops, tests, licenses, supports and distributes OpenFabrics Software (OFS).

The Alliance’s mission is to develop and promote software that enables maximum application efficiency by delivering wire-speed messaging, ultra-low latencies and maximum bandwidth directly to applications with minimal CPU overhead.

The OFA has promoted and facilitated the development of:

  • Vendor-Neutral Linux-based InfiniBand software stack
  • Windows support for IB software stack
  • iWARP support
  • RoCE (RDMA over Converged Ethernet) support

OFA Promoters include:

  • Broadcom
  • Cray
  • Emulex (Avago)
  • HP
  • Huawei
  • IBM
  • Intel
  • Lawrence Livermore National Laboratory (LLNL)
  • Los Alamos National Laboratory (LANL)
  • Mellanox
  • NetApp
  • Oak Ridge National Laboratory (ORNL)
  • Sandia National Laboratories (SNL)
  • Oracle
  • Unisys

OFA Adopters include:

  • Argonne National Laboratory (ANL)
  • DataDirect
  • NASA
  • Qlogic

Source: https://www.openfabrics.org/index.php/organization.html

Learn More...