Hardware-assisted memory page migration across NUMA nodes is about moving data closer to the CPU that actually uses it—without heavy software overhead. On systems like IBM POWER10, this is a coordinated effort between hardware, firmware, and the OS/hypervisor.

🧠 Why page migration is needed

In NUMA systems:

• Each node has local memory (fast) and remote memory (slower)
• If a thread frequently accesses remote pages → performance drops

👉 Migration moves those pages to the local NUMA node to restore locality.

⚙️ 1. Hardware monitoring of memory access patterns

POWER10 includes performance monitoring units (PMUs) that:

• Track:
  • Memory access frequency
  • Local vs remote accesses
  • Cache miss patterns
• Identify “hot pages” (frequently accessed memory)

👉 Hardware provides fine-grained telemetry to guide migration decisions.

🧩 2. OS / Hypervisor decision layer

• The OS (AIX/Linux) or PowerVM:
  • Analyzes hardware counters
  • Decides:
    • Which pages to migrate
    • When migration is worth the cost

👉 Important: Hardware does not blindly migrate—it assists intelligent decisions.

🚚 3. Hardware-assisted page copy (fast data movement)

Once a page is selected:

• Hardware uses:
  • DMA engines / memory controllers
  • High-speed interconnect (e.g., PowerAXON)
• Page is copied:
  • Source NUMA node → Target NUMA node

👉 This avoids CPU-heavy copying and minimizes overhead.

🔄 4. Atomic page remapping

After copying:

• Page tables are updated to point to the new location
• Translation Lookaside Buffers (TLBs) are:
  • Flushed or updated

Hardware support ensures:

• Atomic switch-over
• No inconsistent reads/writes

🔐 5. Coherency and consistency handling

During migration:

• Cache coherency protocol ensures:
  • No stale data is used
  • Writes are properly synchronized

Typical steps:

• Quiesce access briefly (or use copy-on-write techniques)
• Ensure all cores see consistent memory

⚡ 6. Minimizing migration overhead

Hardware helps reduce impact by:

• Migrating only hot pages (not entire memory regions)
• Performing migration in background
• Overlapping migration with execution (using SMT and OoO execution)

👉 Result: Migration cost is amortized over future locality gains

📊 7. Adaptive and dynamic behavior

• Continuous monitoring → continuous tuning
• If access patterns change:
  • Pages can be migrated again

👉 System adapts to workload behavior in real time

⚖️ Trade-off: Migration cost vs benefit

👉 Migration is only done when:

Expected performance gain > migration cost

🧠 Big insight

Hardware doesn’t “own” page migration—it accelerates and enables it:

The OS decides what to move,
Hardware makes moving it fast, safe, and low-overhead.