Before Oracle switched to RoCE (Ethernet-based networking) with the release of the Exadata X8M, the "gold standard" for the Exadata internal fabric was InfiniBand.

If you are looking at an older Exadata model (X8 or earlier), you are looking at an InfiniBand-powered machine. Here is why InfiniBand was the backbone of Exadata for over a decade.

1. The Internal "Backplane"

In older Exadata models, InfiniBand was the private high-speed interconnect that linked all the Database Servers and Storage Cells together.

It wasn't used for your users to connect to the database (that was standard Ethernet); instead, it was used for the "heavy lifting" inside the rack:

• Data Transfer: Moving data blocks from storage to the database.

• Cluster Communication: Handling "heartbeats" and cache synchronization between nodes in an Oracle RAC (Real Application Clusters) environment.

• Backups: Moving data to an appliance like the Zero Data Loss Recovery Appliance (ZDLRA).

2. Why InfiniBand? (The Pre-RoCE Era)

Back in the day, standard Ethernet was "slow" (1Gbps or 10Gbps) and had high overhead. InfiniBand was chosen because it offered three things Ethernet couldn't:

Remote Direct Memory Access (RDMA)

This is the big one. InfiniBand was designed from the ground up to support RDMA. It allowed the Database Server to pull data directly from the Storage Cell's memory without waking up the Storage Cell's CPU. This is the feature that made Exadata famous for its speed.

Massive Bandwidth (40Gbps)

While most data centers were still struggling with 10Gbps Ethernet, InfiniBand provided 40Gbps (QDR - Quad Data Rate) throughput. This ensured that the "pipe" between the disks and the CPU was never the bottleneck.

Extremely Low Latency

InfiniBand is a "lossless" fabric. Unlike Ethernet, which might drop a packet if it gets too busy and ask for it to be resent, InfiniBand manages flow at the hardware level. This resulted in consistent, sub-millisecond response times.

3. The Physical Difference

You can easily spot an older InfiniBand Exadata by looking at the back of the rack:

• The Cables: Instead of the typical RJ45 (plastic clip) or SFP+ Ethernet cables, InfiniBand used thick, sturdy QSFP (Quad Small Form-factor Pluggable) cables.

• The Switches: Every rack contained two internal InfiniBand switches (Leaf Switches) for redundancy.

4. Why did it go away?

If InfiniBand was so good, why did Oracle replace it with RoCE in 2019?

1. Specialized Knowledge: InfiniBand required specialized networking skills that many traditional IT admins didn't have.

2. Hardware Silos: You couldn't just plug an InfiniBand cable into a standard corporate switch; you needed specific (and expensive) gateways.

3. The Rise of 100Gb Ethernet: Ethernet eventually "caught up." Once Ethernet reached 100Gbps and learned how to do RDMA (becoming RoCE), the complexity of maintaining a separate InfiniBand architecture no longer made sense.

InfiniBand vs. RoCE (The Transition)

Summary

In older models, InfiniBand was the exclusive VIP lane that allowed Oracle servers and storage to talk to each other at speeds that seemed impossible at the time. It paved the way for the RDMA technology that still makes Exadata the fastest database platform in the world today.