What is NUMA-aware virtualization?
In high-end database servers like Oracle Exadata or SPARC T-Series, a single machine often has multiple CPU sockets and hundreds of gigabytes of RAM. However, not all memory is "equal" in terms of distance from the CPU.
NUMA-aware virtualization is a configuration strategy that ensures a virtual machine (VM) is aware of the physical "neighborhoods" of processors and memory on the host, preventing the massive performance drops caused by "remote" memory access.
NUMA (Non-Uniform Memory Access) is a hardware design where a server is divided into "Nodes." Each node consists of a group of CPU cores and a local bank of RAM.
Local Access: A CPU core accessing RAM in its own node is lightning fast (low latency).
Remote Access: A CPU core accessing RAM in a different node must travel across the "interconnect" (like Intel’s QPI or AMD’s Infinity Fabric). This is significantly slower and can create a bottleneck.
Without NUMA awareness, a hypervisor might accidentally place a VM's virtual CPUs (vCPUs) on Node 0 while its memory is allocated on Node 1.
Every time that VM does work, it is forced to perform "remote" memory accesses. For a database like Oracle, which moves massive amounts of data in and out of the Buffer Cache, this can lead to a 30% to 50% performance penalty.
There are two main layers to a NUMA-aware setup:
The hypervisor ensures that a VM’s resources are "contained" within a physical NUMA boundary. If a VM needs 8 cores and 64GB of RAM, the hypervisor will "pin" those 8 vCPUs to the 8 physical cores on Node 0 and allocate the 64GB from the RAM directly attached to Node 0.
If a VM is so large that it must span across two physical nodes (e.g., a "Monster VM" with 128 cores), the hypervisor presents a virtual NUMA (vNUMA) topology to the guest OS.
The Guest OS (Solaris or Linux) "sees" two virtual nodes inside the VM.
The Database (Oracle/SQL Server) sees this topology and intelligently places its background processes (like DBWR) close to the data they are writing.
On modern Exadata RoCE-based systems, NUMA awareness is critical for the RDMA features:
RoCE Network Cards are physically attached to a specific NUMA node.
For maximum speed, the VM cluster should be aligned with the same NUMA node as the network card. This allows data to flow from the network directly into the VM's memory without crossing the CPU interconnect.
| Feature | Non-Aware Virtualization | NUMA-Aware Virtualization |
| Memory Latency | High (Frequent remote hops). | Minimal (Local access). |
| Interconnect Traffic | Congested (Data moving between sockets). | Clear (Data stays within the node). |
| Predictability | Poor (Performance varies by placement). | Consistent and Deterministic. |
| Best For | Generic web servers. | Oracle RAC, SQL Server, SAP HANA. |
If you are virtualizing a mission-critical database, NUMA is not optional. You must ensure that your hypervisor (KVM or LDOM) is configured to keep CPUs and memory "married" within the same physical node.