What is impact of spine-leaf oversubscription on RAC global cache?
In a 1M TPS Oracle Real Application Clusters (RAC) environment, the "Global Cache" is the nervous system of the database. It allows multiple nodes to share data blocks via the Interconnect (RoCE/InfiniBand) without hitting the disk.
When you move from a single-rack to a multi-rack Spine-Leaf topology, the Oversubscription Ratio of your Spine switches becomes the single most important variable in your "Log File Sync" and "GC CR Block Received" wait times.
To maintain ACID compliance, if Node A needs a data block that Node B currently has in its RAM, the block must travel across the fabric.
The Payload: Usually an 8KB or 16KB database block.
The Protocol: RDMA (Remote Direct Memory Access).
The Latency Goal: $<20 \mu s$ for a 1M TPS system.
If your Spine-Leaf fabric is oversubscribed (e.g., you have 3:1 oversubscription where 480Gbps of Leaf capacity shares only 160Gbps of Spine bandwidth), the "Global Cache" is no longer a dedicated highway; it becomes a congested city street.
A 1M TPS workload doesn't flow like water; it moves in Micro-Bursts.
The Storm: Thousands of transactions commit at the exact same millisecond.
The Incast: Multiple Leaf switches attempt to send Global Cache blocks to a single "Master" node simultaneously.
The Oversubscription Trap: If the Spine-to-Leaf links are oversubscribed, the Spine switch buffers fill up instantly.
The Result: PFC (Priority Flow Control) kicks in, "pausing" the traffic to prevent packet loss. This "Pause" adds $50\mu s$ to $200\mu s$ of jitter to your RAC Interconnect.
Oracle's LMS (Global Cache Service) processes are designed to be ultra-fast. They expect the network to be a "Zero-Loss" fabric.
Without Oversubscription (1:1): The LMS process sends a block, and it arrives at the requester in near-wire speed.
With Oversubscription (3:1): The LMS process experiences "Network Wait." While the CPU core is ready to process the next transaction, it is stalled waiting for the fabric to clear. In a 1M TPS system, a $100\mu s$ network stall can lead to a queueing cascade that slows down every node in the cluster.
How do you know if your spine-leaf oversubscription is killing your RAC performance? Look for these three specific Oracle wait events:
| Wait Event | Normal Range | Oversubscription Symptom |
| gc cr block receive | $<1 ms$ | Spikes to $5ms+$ during peak load. |
| gc current block receive | $<1.5 ms$ | High variance (Jitter) in response times. |
| log file sync | $<2 ms$ | Increases because the "Commit" must wait for the Global Cache to sync across the oversubscribed fabric. |
To maintain 1M TPS in a multi-rack Spine-Leaf environment:
Maintain 1:1 Non-Blocking Fabric: For the RAC Interconnect, ensure the number of uplinks from the Leaf to the Spine equals the total bandwidth of the downlinks to the servers.
RDMA-Only Links: Dedicate specific physical ports on the Spine for RAC Interconnect traffic, isolating it from general "Client" or "Backup" traffic.
RoCE Congestion Control (DCQCN): In 2026, we use hardware-level congestion notification. The Spine switch tells the Leaf to "slow down" before the buffer overflows, preventing the dreaded "Pause Frame" that halts the entire cluster.
At 1M TPS, Oversubscription is the enemy of Consistency. If your Spine-Leaf ratio is anything other than 1:1, your RAC Global Cache will suffer from "Micro-Stalls" that prevent the database from scaling linearly. In a multi-rack Exadata or OCI Dedicated Region, always verify that your "Spine-to-Leaf" bandwidth can handle the peak aggregate "East-West" traffic of every node combined.