In the world of high-performance computing (HPC) and Oracle Exadata clusters, InfiniBand is prized for its "lossless" nature. Unlike Ethernet, which might drop a packet if things get too crowded, InfiniBand uses a strict Credit-Based Flow Control mechanism.

However, this safety mechanism has a failure mode that can cause "silent" performance degradation: Buffer Credit Starvation. When this happens, your database nodes aren't "slow" because of CPU or disk—they are literally waiting for "permission" to speak.

1. The Mechanism: How Credits Work

In InfiniBand, a sender cannot transmit a packet until the receiver (the switch or the target node) confirms it has enough buffer space to hold it.

• The "Credit": The receiver sends "Credits" to the sender, representing available blocks of 64 bytes in its receive buffer.

• The Spend: Every time the sender transmits a packet, it "spends" credits.

• The Refresh: Once the receiver processes the data and clears its buffer, it sends a "Credit Update" back to the sender.

2. What is Credit Starvation?

Credit Starvation occurs when a sender has data ready to go, but its "Remote Credit" counter is zero. The sender must sit idle, even if the physical cable has $100\text{Gbps}$ of available bandwidth.

The Three Main Causes:

1. The "Slow Receiver": If Node A is sending data to Node B, but Node B’s CPU is too busy to pull data out of the HBA (Host Channel Adapter) buffer, Node B stops sending credit updates. Node A "starves."

2. Long-Distance Latency: If you are running InfiniBand over a long distance (e.g., between data centers), the "Round Trip Time" for credit updates increases. If the "Credit Limit" is too small, the sender spends all its credits before the first "Update" returns from the far end.

3. The "Fan-In" Bottleneck: If four database nodes are all trying to send results to one application server simultaneously, that single server's buffers fill up instantly. It stops issuing credits to all four nodes, causing a cluster-wide stall.

3. Impact on Database Performance

4. Detecting the "Invisible" Stall

You won't see "Packet Loss" or "Retransmissions" in InfiniBand logs because no packets were ever dropped. Instead, you must look at the Performance Counters on the HBA or the Switch:

• PortXmitWait: This is the smoking gun. It counts the number of ticks the port had data to send but couldn't because it lacked credits.

• SymbolErrors / LinkErrorRecovery: While usually physical, high credit starvation can sometimes lead to link instability if the "Keep-Alive" packets are also delayed.

5. Hardware Tuning: Solving Starvation

To fix credit starvation, you have to adjust the "Buffer-to-Buffer" (B2B) credit management:

1. Increase VL (Virtual Lane) Buffers: Most InfiniBand switches (like the Mellanox Quantum series) allow you to reallocate internal memory. By increasing the buffer size for the specific Virtual Lane used by your database traffic, you increase the "Credit Limit."

2. MTU Alignment: Ensure your MTU is set to 4096. Smaller MTUs require more "Credit Updates" for the same amount of data, increasing the overhead and the risk of starvation.

3. Adaptive Routing: Enable Adaptive Routing so that if one path is "Credit Constrained" (perhaps due to a slow switch-to-switch link), the traffic can take an alternative path where credits are more plentiful.

Summary

Buffer Credit Starvation is the "polite" version of a network crash. Because InfiniBand is designed never to lose a packet, it will choose to stop all traffic rather than drop a bit. For a DBA, this means your "Interconnect" waits might actually be a hardware-level buffer negotiation issue. If your PortXmitWait counters are climbing, your network isn't full—it's just out of "permission."