How does PCIe topology mapping influence storage placement?
PCIe Topology Mapping is the practice of aligning your high-speed storage devices (NVMe) with the physical wiring of the motherboard. If your storage is on "the wrong side" of the CPU, your 1M TPS engine will be throttled by the very bus meant to accelerate it.
Modern servers are multi-socket (NUMA) systems. Each CPU socket has its own PCIe Root Complex—the "traffic controller" for the PCIe lanes physically wired to that socket.
The Wrong Way: You place your Redo Log NVMe drive in Slot 1 (wired to CPU 0), but your Database Instance is running on CPU 1.
The Penalty: Every I/O request must travel across the CPU Interconnect (UPI or Infinity Fabric). This adds $100ns$ to $300ns$ of latency and consumes "East-West" bandwidth that should be reserved for cache coherency.
The Right Way: Socket-Direct Placement. You map the PCIe topology to ensure the NVMe drive containing the most active data is physically wired to the CPU socket running that data's processes.
In high-density 1M TPS storage arrays, there aren't always enough direct CPU lanes for 24+ NVMe drives. This leads to the use of PCIe Switches.
The Bottleneck: A PCIe switch allows multiple NVMe drives to share a single "Upstream" x16 link to the CPU.
The Impact: If four NVMe Gen5 drives (each capable of 14GB/s) are behind a single x16 switch port, they will contend for bandwidth during a "Checkpointer" burst or a massive "Smart Scan."
The Strategy: Map your topology to ensure that Redo Logs (latency-sensitive) are on Direct-to-CPU lanes, while Data Files (throughput-sensitive) can sit behind a PCIe switch.
The "Holy Grail" of 1M TPS performance is P2P DMA. This allows a NIC to move data directly to an NVMe drive without touching the CPU or System RAM.
The Topology Requirement: For P2P DMA to work, both the NIC and the NVMe drive must be sitting behind the same PCIe Switch or the same Root Complex.
The Benefit: Data moving from the network to the disk skips the "Memory Wall" entirely. This frees up the CPU and the RAM bandwidth to focus exclusively on transaction logic, effectively doubling the efficiency of the server.
| Placement Strategy | IOPS Latency | CPU Overhead | Scalability |
| Random Placement | High Jitter (~250$\mu$s) | High (Interconnect stalls) | Poor (Bottlenecks at 500k TPS) |
| NUMA-Aligned | Low (~100$\mu$s) | Low (Local Memory) | Good |
| P2P / Socket-Direct | Ultra-Low (<50$\mu$s) | Near Zero | Optimal (1M+ TPS) |
To ensure your storage placement isn't killing your throughput:
Map the Bus: Use lspci -tv to see the physical tree of your PCIe devices. Look for which devices share a bridge.
Verify Affinity: Check /sys/class/pci_bus/.../cpulist_allowed to see which CPU cores are "native" to your NVMe slots.
Align the Workload: Use Interrupt Affinity (smp_affinity) to ensure the CPU core handling the "Storage Completion" interrupt is the same core that requested the data.
In the world of 1M TPS, where you plug in the drive matters as much as the drive itself. By mapping your PCIe topology, you eliminate the "Hidden Hops" inside the server, ensuring that your data takes the shortest, fastest path from the silicon of the disk to the silicon of the CPU.