What is memory hierarchy in Exadata compute nodes?
In a database server, the closer data is to the CPU, the faster the query runs. Oracle Exadata compute nodes (the Database Servers) utilize a sophisticated, multi-tiered memory hierarchy designed to ensure that the CPU never has to "wait" for data.
Think of it as a series of high-speed staging areas, each one larger but slightly slower than the one before it.
At the very top of the pyramid are the buffers built directly into the Intel Xeon processors.
Registers: The fastest memory on earth, used for immediate calculations.
L1/L2 Cache: Dedicated to individual cores.
L3 Cache: Shared across all cores in a processor socket.
Exadata relies heavily on these caches for Vector Processing (using SIMD instructions). During a Smart Scan, the database pulls data into these caches to perform lightning-fast comparisons on multiple rows at once.
This is the "main" memory of the compute node. Modern Exadata nodes can be configured with massive amounts of RAM (up to 1.5TB or more).
System Global Area (SGA): This is where the Oracle Buffer Cache lives. When you read a block from storage, it is placed here so that the next time a user needs it, it can be served at nanosecond speeds without any network I/O.
PGA (Program Global Area): This memory is used for individual session tasks like sorting, hashing, and joining.
In newer RoCE-based Exadata models (X8M, X9M, X10M), the memory hierarchy actually extends outside the compute node.
Through Remote Direct Memory Access (RDMA), a compute node treats the Persistent Memory (PMEM) or DDR Cache on the storage cells as if it were a local tier of memory.
If a block isn't in local RAM, the compute node "reaches out" via the 100Gbps network and pulls it from the storage cell's memory in under 19 microseconds.
This effectively creates a "Global Memory Pool" across the entire rack.
| Tier | Location | Speed (Latency) | Role |
| L1/L2/L3 Cache | Inside CPU | < 1 - 20 ns | Immediate processing/Vectorization. |
| Local DRAM | Compute Node RAM | ~100 ns | Primary Buffer Cache and SQL sorting. |
| Remote PMEM/RAM | Storage Cell (via RDMA) | < 19 $\mu$s | Ultra-fast remote cache hit. |
| Smart Flash Cache | Storage Cell (NVMe) | ~100 - 200 $\mu$s | High-speed data aging/staging. |
Exadata software intelligently moves data between these tiers to maximize "hot" data availability:
Columnarization: In the flash and memory tiers, Exadata can transform traditional row-based data into In-Memory Columnar formats. This allows the CPU to scan the memory tiers even faster by only reading the specific columns needed for a query.
Parallel Query: Exadata spreads the memory load across all compute nodes. If one node's memory is full, the cluster can use Cache Fusion to ship blocks from another node's memory via the high-speed interconnect.
The goal of the Exadata memory hierarchy is to make the storage feel invisible. By layering local CPU caches, massive DRAM, and RDMA-accessible remote memory, Exadata ensures that the vast majority of your "hot" database operations never have to touch a physical disk platter.