How does Exadata rack expansion affect fabric latency topology?
In the context of a 1M TPS (Transactions Per Second) workload, expanding an Oracle Exadata rack isn't just about adding more "boxes." It is a surgical reconfiguration of the RoCE (RDMA over Converged Ethernet) fabric that shifts the network from a local cluster to a distributed multi-tier topology.
Here is how rack expansion fundamentally changes your fabric latency and topology.
In a Single-Rack Exadata (X8M through X11M), the topology is a simple, collapsed Leaf structure.
Connectivity: Database and Storage servers connect directly to two redundant Leaf Switches (at U20/U22).
The "Zero-Spine" State: In a single rack, the leaf switches are directly interconnected with 8–13 high-speed links. Traffic never leaves the rack, resulting in the lowest possible latency (<14$\mu$s for RDMA reads).
When you Expand to Multi-Rack, the topology evolves into a formal Leaf-Spine (Clos) Architecture.
The Spine Layer: A third switch, the Spine Switch (usually at U1), is activated or added.
The Re-Cabling: The direct links between the Leaf switches in Rack A are removed.
The Hop Count: This ensures that any database server can reach any storage server in any rack in exactly 3 hops (Leaf $\rightarrow$ Spine $\rightarrow$ Leaf).
At 1M TPS, every nanosecond counts. Expanding your rack introduces a measurable change in your tail latency:
| Metric | Single-Rack (Intra-Rack) | Multi-Rack (Inter-Rack) | Impact |
| Switch Hops | 1 (Direct Leaf) | 3 (Leaf-Spine-Leaf) | Increase: Slight processing delay. |
| Physical Cable Length | 1–3 meters | 10–30+ meters | Increase: Light speed in copper/fiber ($\sim$5ns per meter). |
| XRMEM Latency | $\sim$14 $\mu$s | $\sim$16–19 $\mu$s | Noticeable: $\sim$15-20% increase for cross-rack RDMA. |
Pro Tip for 1M TPS: Oracle’s Auto-Placement logic tries to keep "Hot" data on storage cells within the same physical rack as the database instance to maintain that sub-15$\mu$s "local" performance.
While latency increases slightly, aggregate bandwidth scales linearly.
X11M Fabric: Uses 100 Gbps RoCE.
Scaling: Expanding to a 4-rack cluster doesn't just quadruple your storage; it quadruples your Total Fabric Capacity.
Throughput: A multi-rack X11M configuration can achieve over 100 GB/s per storage server in Smart Scan throughput.
A critical advantage of the Exadata RoCE fabric during expansion is Instant Failure Detection.
The Problem: In a massive multi-rack cluster, identifying a "sick" switch port (one that is up but dropping packets) is like finding a needle in a haystack.
The Solution: Exadata uses RDMA-based heartbeats. If a cross-rack RDMA read fails across all paths, the system evicts the "sick" component in milliseconds, preventing a single faulty cable in Rack 4 from dragging down the 1M TPS performance of Rack 1.
| Feature | Single Rack | Multi-Rack Expansion (Up to 14 Racks) |
| Topology | Collapsed Leaf | Leaf-Spine (Full Mesh) |
| Redundancy | Dual-Path | Multi-Path (ECMP) |
| RDMA Speed | Pure 100Gbps Local | 100Gbps with $\sim$2-5$\mu$s "Spine Tax" |
| 1M TPS Stability | Maximum Determinism | Massive Throughput / High Availability |
Expanding your Exadata rack is a trade-off: you exchange a few microseconds of Latency for a massive increase in Throughput and Density. To maintain 1M TPS after an expansion, you must ensure your ASM (Automatic Storage Management) is configured to be "Rack Aware," keeping primary data mirrors within the same fabric hop whenever possible.