When a database is suffering from "silent" performance degradation—where CPU usage is high but throughput is low—the culprit is often NUMA imbalance. This happens when a process on Socket A is constantly reaching across the interconnect to fetch data from memory attached to Socket B.

While standard OS tools like top won't show this, the CPU’s Hardware Performance Counters provide the raw truth. Here are the specific counters and ratios you need to monitor to identify a NUMA bottleneck.

1. The "Golden Ratio": RMA vs. LMA

The most direct indicator is the relationship between local and remote memory traffic.

• LMA (Local Memory Access): Data fetched from the RAM directly attached to the current CPU socket.

• RMA (Remote Memory Access): Data fetched from RAM attached to a different socket via the interconnect (UPI/Infinity Fabric).

• The Imbalance Indicator: Use numatop or pcm-numa. A healthy database should have an RMA/LMA ratio of less than 0.05 (5%). If your RMA/LMA ratio climbs above 0.20 (20%), your CPU is spending more time "waiting on the bus" than executing SQL.

2. Remote Node Memory Hits (numa_miss & numa_foreign)

Standard Linux kernel counters (accessible via numastat) track page allocation failures.

• numa_miss: A process wanted to allocate memory on its local node but was forced to go remote because the local node was full.

• numa_foreign: The inverse—this node is hosting memory for a process that actually lives on another node.

• The Imbalance Indicator: A high numa_miss count combined with a high numa_foreign count on another node indicates a "Memory Straddle." This often happens when Oracle's SGA is allocated before the OS has properly balanced its hugepages across nodes.

3. UPI/QPI/Infinity Fabric Utilization

If the internal "highway" between sockets is congested, everything slows down.

• Intel Counter: UNC_M_CAS_COUNT.RD (Remote) vs. UNC_M_CAS_COUNT.WR.

• The Imbalance Indicator: Using pcm.x, look at the QPI/UPI Bandwidth. If you see sustained traffic above 50% of the link’s max GB/s, you are likely suffering from a "NUMA Thrash," where the CPU is constantly shuttling cache lines between sockets to maintain coherency.

4. Instruction Stall Cycles (RESOURCE_STALLS.ANY)

When a NUMA imbalance occurs, the CPU isn't "busy"—it's stalled.

• The Counter: CYCLE_ACTIVITY.STALLS_L3_MISS (Intel) or DATA_CACHE_STALLS (AMD).

• The Imbalance Indicator: If your Instructions Per Cycle (IPC) drops below 1.0 while RMA is high, the hardware is explicitly telling you that the CPU pipeline is empty because it is waiting for data to arrive from the remote socket’s memory controller.

How to Collect These Counters (The Toolbox)

Summary for Performance Engineers

NUMA imbalance is essentially "Latency Leakage." By monitoring the ratio of Remote vs. Local memory hits, you can catch configuration errors—like an incorrectly set vm.swappiness or a misaligned NIC—before they manifest as "unexplainable" database slowness.