In an Exadata environment, performance isn't just about raw speed; it’s about parallelism. If one storage cell is working harder than the others, it becomes a bottleneck that slows down the entire database.

Exadata balances I/O through a sophisticated "striping and mirroring" architecture managed by Oracle Automatic Storage Management (ASM) and the Exadata Storage Server software. Here is how the grid ensures no single disk or cell is left carrying the heavy lifting.

1. Fine-Grained Striping (The Foundation)

Exadata uses ASM to distribute data across all available storage cells. When you create an ASM Diskgroup, the data is broken down into Allocation Units (AU)—typically 4MB or larger in Exadata.

These AUs are spread evenly across every Grid Disk in the storage grid. This ensures that a single large table isn't sitting on one cell; instead, pieces of that table are distributed across Cell 1, Cell 2, Cell 3, and so on.

2. Intelligent Mirroring

Exadata doesn't use traditional RAID 5 or 6 (which can have performance overhead). Instead, it uses ASM Mirroring (Normal or High redundancy):

• Normal Redundancy: Two copies of every AU.

• High Redundancy: Three copies of every AU.

The architecture is "Partner Aware." ASM ensures that a primary AU and its mirrored copy are never placed on the same storage cell. This way, if a cell fails, the I/O load is immediately and evenly picked up by the surviving cells that hold the mirrors.

3. I/O Resource Management (IORM)

While striping handles the physical distribution, IORM handles the "traffic control." In a multi-tenant environment, one database might try to hog all the I/O bandwidth.

IORM balances I/O at the software level by:

• Prioritizing Critical Tasks: Ensuring Log Writer (LGWR) and critical OLTP transactions get prioritized over background backups or reports.

• Fair Share Distribution: Giving each database or consumer group a guaranteed percentage of the total I/O bandwidth of the grid.

• Latency Monitoring: If one cell is responding slower than others (due to a disk reconstruction or heavy load), the database can intelligently redirect "read-only" requests to a mirrored copy on a faster-responding cell.

4. Rebalancing on the Fly

One of the most powerful features of the Exadata storage grid is its ability to rebalance dynamically.

• Adding a Cell: If you add a new storage cell to the rack, ASM automatically starts moving AUs from the old cells to the new one in the background.

• Removing/Failure: If a disk or cell fails, the grid redistributes the data to maintain redundancy levels.

Because this happens across all cells simultaneously, the "rebalance" operation itself is parallelized, minimizing the impact on production performance.

5. The Role of RoCE and RDMA

The 100Gbps RoCE (RDMA over Converged Ethernet) interconnect allows the Database Servers to "see" the entire storage grid as a single pool of low-latency memory.

Because the DB server uses Direct-to-Wire protocols to talk to the storage cells, there is virtually zero CPU overhead for managing the network traffic. This allows the database to pull data from 10 different cells as easily as if it were pulling from one local drive.

Summary: The "Many Hands" Approach