Memory affinity—how closely a CPU (or partition) is tied to the memory it accesses—has a major impact on performance, especially on NUMA systems like IBM POWER10 servers.

At a high level:

The closer your memory is to your CPU, the faster your partition runs.

🧠 What memory affinity means in partitions

In virtualized environments (e.g., using PowerVM):

• Each LPAR (partition) is assigned:
  • Virtual CPUs (vCPUs)
  • Memory pages
• Good affinity: vCPUs access memory on the same NUMA node (local memory)
• Poor affinity: vCPUs frequently access remote memory (other sockets/nodes)

⚡ 1. Latency impact (most critical)

• Local memory access → lowest latency
• Remote memory access → higher latency (extra hops via interconnect like PowerAXON)

👉 Poor affinity leads to:

• Slower load/store operations
• Increased CPU stall cycles
• Lower instruction throughput

📉 2. Cache efficiency degradation

When memory is remote:

• Cache lines take longer to fill
• Higher chance of:
  • Cache misses
  • Reduced prefetch effectiveness

👉 Result:

• Lower L1/L2 cache hit efficiency
• More pressure on L3 and memory fabric

🔄 3. Increased interconnect traffic

• Remote memory access uses system interconnect (e.g., PowerAXON)
• Multiple partitions competing → congestion

👉 Effects:

• Bandwidth contention
• Higher effective latency for all partitions

📊 4. Impact on SMT efficiency

On POWER10 (with SMT up to 8 threads per core):

• SMT relies on hiding latency
• With poor memory affinity:
  • All threads may stall on remote memory
  • Latency hiding becomes less effective

👉 Result:

• Diminishing returns from SMT
• Throughput drops under memory-heavy workloads

🧩 5. Workload-specific sensitivity

Highly sensitive:

• Databases (e.g., OLTP)
• In-memory analytics
• JVM-based apps

Less sensitive:

• Compute-bound workloads
• Streaming or sequential access workloads

📈 6. Effects in dynamic environments

With features like:

• Live Partition Mobility
• Memory Inception (shared memory clustering)

Affinity can degrade if:

• Memory stays on source node but CPUs move
• Memory is spread across nodes

👉 Requires rebalancing to avoid performance loss

⚖️ Good vs Poor Affinity (Summary)

🛠️ How systems mitigate affinity issues

In POWER systems:

• Hypervisor (PowerVM) tries to:
  • Place memory close to assigned CPUs
  • Maintain NUMA locality during allocation
• Techniques:
  • First-touch memory allocation
  • NUMA-aware scheduling
  • Dynamic memory migration (in some cases)

🧠 Big insight

Memory affinity is often more important than raw CPU allocation in modern systems.

A partition with fewer CPUs but good locality can outperform one with more CPUs but poor memory affinity.