How does cross-socket memory access latency influence RAC interconnect efficiency?
In an Oracle Real Application Clusters (RAC) environment, we often talk about the "Interconnect" as the physical network cables (InfiniBand, RoCE, or Ethernet) connecting the nodes. However, in modern high-core-count servers, there is a "hidden interconnect" inside the server itself: the cross-socket link (Intel UPI or AMD Infinity Fabric).
The relationship between cross-socket memory latency and RAC interconnect efficiency is a story of compounding delays. If your internal memory access is slow, your external RAC scaling will suffer.
When a RAC node receives a request for a data block via the network (Cache Fusion), the Global Cache Service (LMS) process must retrieve that block from local RAM.
The Local Hit: If the data is in the memory attached to the CPU running the LMS process, latency is minimal (approx. 80–100ns).
The Remote Hit (Cross-Socket): If the data resides in RAM attached to the other socket, the CPU must traverse the cross-socket link. This adds 40–60ns of hardware latency.
The Compound Effect: While 60ns sounds small, it delays the LMS process's ability to "ship" the block. This increases the total "Global Cache Service Time," which is the primary metric for RAC health.
Oracle RAC is a NUMA (Non-Uniform Memory Access) aware application. If the OS scheduler moves an LMS process from Socket 0 to Socket 1, but the "hot" buffers remain in Socket 0's memory:
The Impact: Every Cache Fusion request now requires a cross-socket "hop."
The Efficiency Loss: This consumes bandwidth on the CPU's internal fabric, competing with other database processes and slowing down the overall packet processing for the external Interconnect.
The NIC (Network Interface Card) typically delivers packets to a specific CPU core via an interrupt.
The Conflict: If the NIC is physically wired to PCIe lanes on Socket 0, but the Oracle LMS process is running on Socket 1, every incoming Cache Fusion packet must cross the internal socket link before the database even sees it.
The Result: This increases "Interrupt Latency." In high-traffic RAC environments, this mismatch can lead to dropped packets at the network layer, triggering retransmits and spiking gc current block busy wait events.
Modern CPU microarchitectures have improved the efficiency of these cross-socket "jumps":
Intel UPI (Ultra Path Interconnect): Higher bandwidth and lower latency compared to the older QPI. It allows for "Direct Transfer" modes where a cache line can be moved from one socket's L3 to another socket's L3 without hitting RAM.
AMD Infinity Fabric: Uses a "Mesh" approach that treats all memory as a unified pool, though the "NUMA distance" penalty still exists.
| Access Path | Typical Latency | Impact on RAC |
| Local L3 Cache | ~10-15ns | Instant Cache Fusion response. |
| Local RAM | ~80-100ns | Standard "Fast" RAC response. |
| Remote Socket RAM | ~150-180ns | Increased gc service time; CPU "System" time spikes. |
| External Interconnect (RoCE) | ~1,000ns - 5,000ns | The "visible" network delay. |
To maximize RAC interconnect efficiency, you must optimize the "internal" interconnect first:
Strict NUMA Alignment: Ensure Oracle's _enable_numa_support is handled correctly and the OS numad or lstopo shows processes pinned near their memory.
NIC Placement: Ensure your RAC Interconnect NIC is physically plugged into a PCIe slot managed by the CPU socket where the LMS processes reside.
Single-Socket Strategy: In the era of 128-core CPUs (like AMD EPYC), it is often more efficient to run a "one-socket-per-node" RAC configuration. By keeping all memory local to a single socket, you eliminate cross-socket latency entirely, making your Cache Fusion traffic significantly more predictable.