What is storage latency benchmarking method?
Benchmarking storage latency is often misunderstood. Many people look at "throughput" (how much data can move per second), but for databases and virtual machines, latency (how long a single request takes) is the metric that actually determines if the system feels fast or "laggy."
To get an accurate measurement, you can't just copy a file. You need a scientific method that isolates the storage from the operating system's cache.
Before testing, you must define the three types of latency. If you don't know which one you are measuring, your data is useless.
Average Latency: The "typical" speed. Good for general health checks.
Tail Latency (99th Percentile): The speed of the slowest 1% of requests. This is what causes "stuttering" in applications.
Interrupt/System Latency: The overhead added by the CPU and the hypervisor before the request even hits the disk.
Modern Operating Systems are "liars"—they store data in RAM and tell the benchmark it was written to disk instantly. To measure the real storage speed, you must use the O_DIRECT flag, which tells the OS to bypass the RAM cache and talk directly to the hardware.
Throughput tests use large files (1MB blocks). Latency tests must use small, random blocks (4KB). This forces the disk head (on HDDs) or the controller logic (on SSDs) to work hard for every single request, revealing the true hardware response time.
Latency increases as more requests pile up.
To measure minimum latency, test with QD=1 (one request at a time).
To measure stressed latency, test with QD=32 or higher.
In the enterprise world, FIO is the gold standard. It is more accurate than simple tools like dd or CrystalDiskMark because it provides a full latency distribution.
The "Magic" Latency Command:
fio --name=latency-test --filename=/dev/sdb --ioengine=libaio --direct=1 --rw=randread --bs=4k --size=1G --numjobs=1 --iodepth=1 --runtime=60 --time_based --group_reporting --lat_log=test
When the test finishes, don't just look at the msec average. Look at the clat (Completion Latency) percentiles:
| Percentile | Meaning | Performance Grade |
| 90th | 90% of requests were faster than this. | Standard expectation. |
| 99th | 1% of requests were slower than this. | The "Jitter" indicator. |
| 99.99th | The "Worst Case" scenario. | Critical for real-time financial apps. |
| Do... | Don't... |
| Test on a "Clean" (empty) disk when possible. | Test on a disk while a backup is running. |
| Run the test for at least 10 minutes to allow for heat-soaking. | Run a 5-second "burst" test. |
Use direct=1 to bypass the OS cache. | Forget that the OS cache exists. |
| Test both Random Read and Random Write. | Assume Read and Write latency are the same. |
Storage latency benchmarking is the art of removing the "noise" of the software to see the truth of the hardware. If your average latency is 1ms but your 99th percentile is 100ms, your users will complain that the system is slow, even if your "averages" look perfect.