Adding a new storage cell to an Exadata rack is like adding a new lane to a congested highway—except the highway automatically moves the cars around to ensure every lane is used perfectly.

In Exadata, this process is known as ASM Rebalancing. It is a background operation that redistributes data across the entire storage grid to ensure that no single cell is a bottleneck and that redundancy is maintained.

1. The Trigger: Increasing the "Storage Pool"

When you add a new storage server, you first configure its physical disks into Cell Disks and Grid Disks. Once these Grid Disks are added to an existing ASM Diskgroup, Oracle ASM (Automatic Storage Management) immediately recognizes the "imbalance."

The system sees that the new disks are empty while the old ones are nearly full. To restore the principle of balanced I/O, it triggers a rebalance.

2. The Mechanics: Moving Allocation Units (AUs)

Exadata data is stored in Allocation Units (AUs)—typically 4MB blocks. The rebalance process is a "many-to-many" operation:

1. Background Migration: Every existing storage cell in the grid begins reading AUs from its local disks.

2. Network Transfer: These AUs are sent over the 100Gbps RoCE (or InfiniBand) fabric to the new storage cell.

3. Writing: The new storage cell writes these AUs to its fresh disks.

3. Parallelism is the Secret Sauce

Unlike traditional RAID rebuilds that might happen one drive at a time, Exadata rebalancing is massively parallel.

• Every disk in the existing grid reads data.

• Every disk in the new cell receives data.

• Because the work is shared across the entire grid, the rebalance finishes much faster than a standard storage expansion.

4. Performance Tuning: The Rebalance Power

A common concern is whether adding storage will slow down the production database. Exadata gives you a "volume knob" called the ASM Rebalance Power (ASM_POWER_LIMIT):

• Low Power (1-10): The rebalance moves slowly, consuming very little I/O and CPU. This is ideal for peak business hours.

• High Power (Up to 1024 in newer versions): The rebalance moves as fast as the hardware allows. This is used during maintenance windows to integrate new hardware as quickly as possible.

5. Intelligent Data Placement

Exadata doesn't just move data randomly. It ensures that:

• Primary and Mirror copies are never placed on the same storage cell (Partner Awareness).

• Smart Scan metadata is updated so the database knows exactly where the new pieces of the table are located.

• Storage Indexes are rebuilt or updated for the data arriving at the new cell so that query performance remains optimized.

6. Zero Downtime Architecture

The most important part of this process? The database remains 100% online.

• Applications can continue to read and write to the table while the AUs are moving.

• If a query asks for a block that is currently "in flight" (moving from an old cell to the new one), ASM knows exactly where to find it and handles the request transparently.

Summary: The Rebalance Lifecycle