How does network bandwidth saturation affect RAC?
In an Oracle Real Application Clusters (RAC) environment, the network is the nervous system. While a standalone database mostly cares about the connection between the CPU and the disk, a RAC cluster relies on the network to keep multiple servers synchronized.
When network bandwidth saturates, the cluster doesn't just "slow down"—it begins to destabilize, leading to performance "brownouts" or even the forced eviction of nodes from the cluster.
To understand saturation, we have to look at the two distinct roles the network plays:
The Public Network: Handles client connections and SQL queries.
The Private Interconnect: The "heartbeat" and data exchange path between nodes. This is where saturation is most lethal.
The primary job of the private interconnect is Cache Fusion. If Node A needs a block of data that is currently sitting in the memory (SGA) of Node B, Oracle transfers that block over the network rather than reading it from the slow disk.
When Bandwidth Saturates:
Increased Latency: Data blocks that should arrive in microseconds take milliseconds.
Wait Events: You will see a massive spike in gc current block 2-way and gc cr block busy wait events in your AWR reports.
CPU Spikes: The CPU spends more cycles managing network retransmissions and "waiting" for the network buffer to clear, leading to a secondary CPU bottleneck.
The most dangerous consequence of network saturation is a loss of cluster heartbeat. RAC nodes constantly "ping" each other to confirm they are alive.
If the network is 100% saturated by a massive data move or a backup, the Clusterware (GI) heartbeats might not get through. If a node fails to respond within the misscount threshold:
The cluster assumes a network failure has occurred.
To prevent data corruption (a "split-brain" scenario), the cluster will evict one or more nodes.
The evicted node will perform an immediate, hard reboot.
Network saturation in RAC is rarely caused by standard SQL traffic. It is usually triggered by:
Parallel Queries: A large "Select" statement that spans multiple nodes, forcing massive amounts of data to be shuffled across the interconnect.
LMON/LMS Process Backlog: When the Global Enqueue Service can't keep up with lock requests due to network lag.
Non-Database Traffic: Running RMAN backups or OS-level file copies over the private interconnect (a major configuration "no-no").
If you suspect the network is choking your RAC performance, look for these "smoking guns":
| Metric | Threshold for Concern |
gc (Global Cache) Waits | If these are your "Top 5" wait events, the interconnect is struggling. |
| Network Retransmissions | Use netstat -s. High retransmission rates mean the pipe is full or dropping packets. |
| Interconnect Latency | Use the oraping utility. In a healthy RAC, latency should be < 1ms. |
| AWR "Interconnect" Section | Check the "Global Cache Efficiency" section; if "Throughput" hits your NIC's theoretical max, you are saturated. |
Dedicated Hardware: Never share the private interconnect with public traffic or storage traffic (unless using a highly managed converged network like InfiniBand or RoCE on Exadata).
Jumbo Frames: Set the MTU to 9000. This allows larger packets, reducing the CPU overhead of breaking down large data blocks.
Link Aggregation (LACP/Bonding): Use multiple physical NICs bonded together to increase the total available "pipe."
Workload Isolation: Use Services to keep related data/users on the same node, reducing the need for Cache Fusion to move blocks across the wire.
In Oracle RAC, the network is a physical extension of the memory bus. When the network saturates, you aren't just losing bandwidth; you are losing the ability to maintain a consistent state across your database. Keeping your interconnect "wide and fast" is the single most important factor in RAC scalability.