What is effect of interconnect jitter on global cache fusion?
In an Oracle Real Application Clusters (RAC) environment, Global Cache Fusion is the mechanism that allows data blocks to move between the RAM of different nodes.
However, the efficiency of this "fusion" depends entirely on the stability of the private interconnect. While we often talk about Latency, the real performance killer is Jitter (latency variance).
Here is how interconnect jitter destabilizes your cluster and why your database "feels" slower than the hardware specs suggest.
Latency is the time it takes for a block to travel from Node A to Node B (e.g., $100\mu\text{s}$). Jitter is the deviation in that timing.
If one block takes $100\mu\text{s}$ but the next takes $2,000\mu\text{s}$ due to a micro-burst or a switch buffer conflict, you have high jitter. To a database engine, an unpredictable $100\mu\text{s}$ connection is often more damaging than a steady, predictable $500\mu\text{s}$ connection.
The LMS (Global Cache Service) processes are responsible for shipping blocks. When jitter occurs:
The Request: Node 1 requests a block from Node 2.
The Wait: Node 1 enters a gc current block receive wait.
The Jitter: Because the network is "jittery," the acknowledgment or the data packet arrives late.
The Result: The CPU on Node 1 cannot move forward. Even a small $1\text{ms}$ jitter spike causes the CPU to "spin" or context-switch, wasting thousands of clock cycles that could have been used for SQL execution.
Cache Fusion isn't just about moving data; it's about moving Locks.
If a "Lock Grant" message is delayed by jitter, the session on the requesting node sits idle.
Other sessions on that same node, waiting for the same row, start to pile up.
The Result: A tiny network jitter spike creates a software bottleneck. By the time the network stabilizes, you have a "convoy" of hundreds of sessions fighting for the same latch, leading to a massive spike in CPU utilization as the database tries to clear the backlog.
Oracle RAC tracks interconnect performance in "buckets" (e.g., <1ms, 1-2ms, 2-10ms).
The Symptom: If your AWR report shows a significant number of gc waits falling into the 2-10ms or 10ms+ buckets on a high-speed (100Gbps) network, you are a victim of jitter.
The Cause: This is usually caused by Micro-bursting—where a non-database traffic source (like a backup or VM migration) momentarily saturates a switch port, causing a "micro-stall" for the RAC heartbeat and cache packets.
| Metric | Stable Interconnect | Jittery Interconnect |
Typical gc Wait | $150\mu\text{s} - 250\mu\text{s}$ | $200\mu\text{s} - 5,000\mu\text{s}$ |
| Throughput | Linear Scaling | Erratic / Non-linear |
| CPU Impact | Low Overhead | High "System" CPU (Spinning) |
| Node Eviction Risk | Negligible | High (Missed Heartbeats) |
To stabilize Cache Fusion, you must treat the interconnect as a "sanctuary" for database traffic:
Physical Isolation: Never share the RAC private interconnect switches with public traffic or storage traffic (unless using properly configured VLANs and QoS).
Disable Interrupt Coalescing: On the NIC level, disable or "fix" interrupt coalescing. While coalescing saves CPU, it introduces jitter by waiting to "batch" packets. For RAC, we want the packet now, even if it costs more CPU.
Jumbo Frames (MTU 9000): Ensure MTU is consistent across all nodes and switches. "Path MTU Discovery" failures are a leading cause of erratic jitter as the OS tries to fragment large database blocks (8KB+) into small Ethernet frames.
Flow Control (PFC): Use Priority Flow Control to ensure that critical RAC traffic gets "Lossless" treatment, preventing the packet drops that lead to retransmission jitter.
Interconnect jitter is the "friction" that prevents a RAC cluster from scaling. When the network becomes unpredictable, the Global Cache Fusion mechanism loses its efficiency, leading to CPU waste and application "hiccups." For a RAC DBA, predictability is the highest form of performance.