How does persistent memory flush instruction latency compare to DRAM write latency?
In the evolution of database storage, Persistent Memory (PMEM) promised the speed of RAM with the permanence of a disk. However, for a database developer, the "speed" of PMEM is a double-edged sword.
While reading from PMEM is nearly as fast as DRAM, writing to it involves a specialized "flush" operation that changes the latency profile entirely.
To understand the latency gap, we have to look at the "flight time" of a single store operation.
| Operation Type | Typical Latency (nanoseconds) | Comparison |
| DRAM Write (Store to L1/L2) | ~1ns - 2ns | Near-instant (volatile). |
| DRAM Write (Retire to RAM) | ~80ns - 100ns | The "RAM speed" benchmark. |
| PMEM Write (Direct Load/Store) | ~100ns - 300ns | Slightly slower than DRAM. |
| PMEM Flush (Persistence) | ~500ns - 2,000ns | The Bottleneck. |
The Crucial Distinction: A standard "write" to PMEM only reaches the CPU's internal buffers or the memory controller's write queue. It is not "persistent" until it is flushed to the actual non-volatile media.
CLWB and CLFLUSHOPTTo ensure a database log entry survives a power loss, the CPU must execute specific instructions to push data out of the volatile cache and into the "Persistence Domain."
CLWB (Cache Line Write Back): This is the modern standard. It pushes the data to the PMEM media but keeps a copy in the CPU cache for future reads.
Latency Impact: While CLWB is non-destructive, the CPU must wait for an ACK (Acknowledgment) from the memory controller that the data has reached the ADR (Asynchronous DRAM Refresh) safe zone. This "round-trip" typically takes 500ns to 1μs.
In a database environment, a flush is rarely a solo act. To guarantee the order of operations (e.g., "Write Data" then "Write Commit Flag"), the flush must be followed by a Memory Fence (SFENCE).
Write to Cache: $1\text{ns}$
CLWB (Flush to PMEM): $500\text{ns} - 1,000\text{ns}$
SFENCE (Wait for Confirmation): Adds another $100\text{ns} - 200\text{ns}$ of stall time.
Total Persistence Latency: You are looking at roughly $1.2\mu\text{s}$ per persistent write. Compared to a standard DRAM write ($100\text{ns}$), persisting data to PMEM is roughly $12\times$ slower than simply writing to RAM.
For an Oracle LGWR or a PostgreSQL WAL writer, this $12\times$ latency penalty is a massive hurdle. If the database code treats PMEM exactly like DRAM, the commit performance will plummet.
The Optimization Strategies:
Log Buffering: Databases avoid frequent flushes by "batching" changes in a DRAM buffer and issuing one massive CLWB sequence for the entire block.
Write-Combining: Modern CPUs can combine multiple stores into a single flush operation if they are contiguous, reducing the number of "Ack" wait cycles.
Bypassing the Cache: Some engines use "Non-Temporal" stores (MOVNTI), which write directly to PMEM and bypass the L1/L2 caches entirely, reducing the overhead of cache-coherency "chatter" during the flush.
Persistent Memory is revolutionary because it is $100\times$ faster than an NVMe SSD ($1\mu\text{s}$ vs $100\mu\text{s}$), but it is still $10\times$ slower than DRAM when you factor in the mandatory flush instructions.
For the database architect, the goal is to use DRAM for the "work" and PMEM only for the "proof" (the logs), carefully managing those microsecond-long flush stalls to maintain high transaction throughput.