How does storage firmware queue arbitration influence RAC nodes differently?
In an Oracle Real Application Clusters (RAC) environment, we often focus on the "Interconnect" or the "Global Cache Service" as the primary drivers of performance. However, there is a lower-level gatekeeper that can cause asymmetric performance across your cluster: Storage Firmware Queue Arbitration.
When multiple RAC nodes share the same physical storage (SAN or All-Flash Array), the firmware inside the storage controllers must decide which node’s I/O request gets processed first. If this arbitration isn't perfectly balanced, you end up with the "Limping Node" syndrome.
Storage controllers use an Arbiter (a hardware-level scheduler) to manage incoming commands from multiple Host Bus Adapters (HBAs). Since a controller has finite resources (CPU cycles and cache bandwidth), it uses algorithms like Round Robin, Weighted Fair Queuing, or First-In-First-Out (FIFO) to drain its entry queues.
In a RAC environment, every node is fighting for the same "slots" in that firmware queue.
If Node 1 is performing a massive Parallel Query (Smart Scan) and Node 2 is processing high-frequency OLTP commits, the firmware arbiter may prioritize the "depth" of Node 1’s queue.
The Impact: Node 2’s small Redo Log writes get stuck behind Node 1’s large block reads.
The RAC Result: Node 2 experiences a spike in log file sync waits. Because Node 2 is slow to commit, it holds onto row locks longer, eventually causing "Enqueue" waits on Node 1. The whole cluster slows down because one node was "favored" by the storage firmware.
RAC nodes often have different physical paths to the storage (different switches or HBAs).
The Conflict: If Node 1 has a slightly "cleaner" path (lower CRC error rate) than Node 2, the storage firmware may complete Node 1’s requests faster to keep its internal pipeline full.
The Result: This creates LMS (Global Cache) Asymmetry. If Node 2 is consistently slower at fetching blocks from disk than Node 1, the Global Cache Fusion becomes unbalanced. You’ll see gc cr block lost or gc current block busy events spiking on the "fast" node because it’s constantly waiting for the "slow" node to catch up.
In Fibre Channel SANs, the firmware uses "credits" to allow a node to send data.
The Influence: If Node 1 is "hogging" buffer credits at the switch or controller level, Node 2's HBA will literally sit idle, waiting for a "Ready" signal from the storage firmware.
The RAC Symptom: You may see one node with 10% CPU I/O Wait and another with 40% I/O Wait, even though their database workloads are identical.
This is why Oracle Exadata and high-end arrays use I/O Resource Management (IORM).
Standard Firmware: Arbitrates based on "Packets" or "Bytes." It doesn't know Node 1 is a Production DB and Node 2 is a Test DB.
Smart Arbitration (Exadata): Arbitrates based on Database Identity. It ensures that "Critical" Redo writes from any RAC node jump to the front of the firmware queue, regardless of how many "Batch" reads are coming from other nodes.
| Strategy | Impact on RAC | Risk |
| Strict FIFO | High Jitter | A single "heavy" node can block the entire cluster. |
| Round Robin | Balanced | Can under-utilize the storage if one node is idle. |
| Priority-Based | Optimal | Requires the storage to be "Database Aware" (e.g., Exadata/ZFS). |
To prevent storage firmware from playing favorites with your RAC nodes:
Uniform HBA Settings: Ensure Execution Throttle and Queue Depth settings are identical across all nodes in the cluster (e.g., set all to 64 or 128).
Isolate Workloads: If possible, don't mix "Scan-Heavy" nodes and "Commit-Heavy" nodes on the same physical storage ports. Use Zoning to give them dedicated controller ports.
Monitor "Port Utilization": Use SAN analytics tools to look for "Port Over-subscription." If one port is at 90% and another is at 20%, the firmware arbiter is likely struggling to maintain fairness.
Storage firmware is the "invisible judge" of RAC performance. When queue arbitration becomes unbalanced, it creates a ripple effect that starts at the physical disk and ends with cluster-wide global cache congestion. In a RAC environment, Fairness is more important than raw speed.