On multi-socket systems built with the IBM POWER architecture (e.g., servers using the IBM POWER10 processor or IBM POWER9 processor), cache coherency ensures that all cores—across all sockets—see a consistent view of memory, even though each core has its own private caches.

Here’s how it works at a system level.

🔹 1. The Problem: Multiple Copies of Data

Each core has:

• L1 and L2 (private)
• L3 (shared within a chip/module)

Across sockets:

• The same memory line may exist in multiple caches

👉 Without coherency:

• One core updates data
• Others keep stale copies → incorrect results

🔹 2. Coherency Protocol (MESI/MOESI-like)

POWER systems use an advanced directory-based coherency protocol (similar to MESI/MOESI but extended).

Each cache line has states like:

• Modified (M) – only this cache has the latest copy
• Exclusive (E) – clean and private
• Shared (S) – multiple caches share it
• Invalid (I) – not valid

POWER adds enhancements for:

• Scalability across sockets
• Reduced broadcast traffic

🔹 3. Directory-Based Coherency (Key Difference)

Unlike simple snooping systems:

👉 POWER uses a directory-based model:

• A directory tracks which socket/core holds each cache line
• Typically stored alongside L3 or memory controller

Why this matters:

• Avoids broadcasting to all sockets
• Sends messages only to relevant nodes

👉 Result:

• Scales efficiently to large multi-socket systems

🔹 4. Inter-Socket Fabric (X-Bus / PowerAXON)

Sockets are connected via high-speed interconnects:

• POWER9: X-Bus / NVLink-based fabric
• POWER10: PowerAXON fabric

These links:

• Carry coherency messages
• Transfer cache lines between sockets

🔹 5. Example: Read Across Sockets

Scenario:

Core A (Socket 1) needs data owned by Socket 2

Steps:

1. Core A issues a load miss
2. Directory identifies owner (Socket 2)
3. Request sent over interconnect
4. Socket 2 supplies data:
   • From its L3 cache (if present)
   • Or from memory

👉 Result:

• Data fetched without full system broadcast

🔹 6. Example: Write (Invalidation Flow)

Scenario:

Core A modifies a shared cache line

Steps:

1. Core A requests ownership (Modified state)
2. Directory finds other sharers
3. Sends invalidate messages to those sockets
4. Other caches mark line Invalid
5. Core A updates data

👉 Result:

• Only one “owner” → ensures correctness

🔹 7. Cache-to-Cache Transfers

Instead of going to memory:

• If another socket already has the latest copy:
  • Data is transferred directly cache-to-cache

👉 Benefits:

• Lower latency than DRAM access
• Reduced memory bandwidth usage

🔹 8. NUMA Awareness

Multi-socket POWER systems are NUMA (Non-Uniform Memory Access):

• Local memory → faster
• Remote memory → slower

Coherency system:

• Minimizes remote accesses via caching
• Works with OS (like AIX/Linux) for:
  • Thread placement
  • Memory locality

🔹 9. L3 Cache Role in Coherency

Large shared L3 cache (per chip/module):

• Acts as a coherency hub
• Stores directory information
• Reduces cross-socket traffic

👉 Many requests are resolved:

• Within the chip
• Without going to other sockets

🔹 10. Scalability Features in POWER10

Compared to earlier generations:

• More efficient directory structures
• Higher bandwidth interconnect (PowerAXON)
• Better coherency filtering
• Reduced false sharing impact

👉 Result:

• Scales to large systems with:
  • Many sockets
  • Thousands of threads

🔹 11. Impact on Database Workloads

For workloads like Oracle:

Positive effects:

• Consistent shared data across nodes
• Faster cache-to-cache transfers
• Reduced locking overhead

Challenges:

• Cross-socket sharing can add latency
• False sharing can degrade performance

👉 Best practice:

• Keep frequently shared data NUMA-local

🔑 Summary

🧠 Key Insight

In IBM POWER architecture systems:

Cache coherency is selective, directory-driven, and fabric-accelerated
—not broadcast-heavy like older designs.

This is what allows POWER systems to scale efficiently for large databases, SAP, and enterprise workloads across multiple sockets.