Samsung PM863 SSD: surprisingly high Write IOPS measured using `fio`, over 4.6 times more than spec!?
Hello everyone,
I've arrived at a very surprising number measuring IOPS write performance on my SSDs' "bare metal" (ie, straight on the /dev/$DISK, no filesystem involved):
export COMMON_OPTIONS='--ioengine=libaio --direct=1 --runtime=120 --time_based --group_reporting'
ls -l /dev/disk/by-id | grep 'ata-.*sda'
lrwxrwxrwx 1 root root 9 Feb 13 17:19 ata-SAMSUNG_MZ7LM1T9HCJM-00003_XXXXXXXXXXXXXX -> ../../sda
TANGO=/dev/disk/by-id/ata-SAMSUNG_MZ7LM1T9HCJM-00003_XXXXXXXXXXXXXX
sudo fio --filename=${TANGO} --name=device_iops_write --rw=randwrite --bs=4k --iodepth=256 --numjobs=4 ${COMMON_OPTIONS}
[...]
write: *IOPS=83.1k*, BW=325MiB/s (341MB/s)(38.1GiB/120007msec)
[...](please find the complete output at the end of this message, in case I should have looked at some other lines and/or you are curious)
As per the official manufacturer specs (both in this whitepaper at their website[1]), and also in this datasheet I found somewhere else[2]), it's supposed to be only *18K IOPS*.
All the other base performance numbers I've measured (read IOPS, read and write MB/s, read and write latencies) are at or very near the manufacturer specs.
What's going on?
At first I thought that, despite --direct=1 being explicitly indicated, my machine's 64GB RAM (via the Linux buffer cache) could be caching the writes (even if the number, in that case, should have been much higher)... so, I tested it again with --runtime=120 to saturate the buffer cache in case it was really the 'culprit'... lo and behold, the result was:
[...] write: IOPS=83.1k, BW=325MiB/s (341MB/s)(190GiB/600019msec) [...]
So, the surprising over-4.6x-times-the-spec Write IOPS is mantained, even for 190GiB total data.
And with 190GiB data written (about 10% the total device capacity), I do not believe it's any kind of cache (RAM, MLC or whatever) inside the SSD either.
I even considered that I could have got some kind of 'unicorn' device, so I repeated all tests on my other SSD (same model and firmware, but a little older -- date of manufacture on the paper label about 3 months earlier), and got almost the exact same results (less than 1% variation). I do not believe I got *two* over-4.6x-times-faster-than-spec 'unicorns' out of a used eBay SSD sale...
So, what gives? With me being no fio expert, the obvious answer is that I made some kind of mistake in its command-line above, but if so, for the life of me I can't see it.
Thanks in advance for all tips and hints and cluebats from all you fio connoiseurs...
and please have no mercy, in case I messed up both the face and the palm here are ready to meet each other... ;-)
PS: in case it matters, this was running Ubuntu 18.04.6 with kernel 4.15.0-167-generic and fio-3.1 installed via apt-get install from the official distro repo.
[1] https://www.samsung.com/semiconductor/global.semi.static/PM863_White_Paper-0.pdf,
p.4: "4 KB Random R/*W* (IOPs) Up to 99,000 / *18,000 IOPS*";
[2] https://www.compuram.de/documents/datasheet/PM863_SAMSUNG.pdf,
p.6: "Random Write IOPS (4 KB) [then, in the column for the 1,920GB model] 18K IOPS"
Cheers,
Signature · 37 lines
Durval.
$ export COMMON_OPTIONS='--ioengine=libaio --direct=1 --runtime=120 --time_based --group_reporting'
$ ls -l /dev/disk/by-id | grep 'ata-.*sda'
lrwxrwxrwx 1 root root 9 Feb 13 17:19 ata-SAMSUNG_MZ7LM1T9HCJM-00003_XXXXXXXXXXXXXX -> ../../sda
$ TANGO=/dev/disk/by-id/ata-SAMSUNG_MZ7LM1T9HCJM-00003_XXXXXXXXXXXXXX
$ sudo fio --filename=${TANGO} --name=device_iops_write --rw=randwrite --bs=4k --iodepth=256 --numjobs=4 ${COMMON_OPTIONS}
device_iops_write: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=256
...
fio-3.1
Starting 4 processes
Jobs: 4 (f=4): [w(4)][100.0%][r=0KiB/s,w=326MiB/s][r=0,w=83.4k IOPS][eta 00m:00s]
device_iops_write: (groupid=0, jobs=4): err= 0: pid=27042: Sun Feb 13 16:19:10 2022
write: IOPS=83.1k, BW=325MiB/s (341MB/s)(38.1GiB/120007msec)
slat (nsec): min=1504, max=13545k, avg=46653.86, stdev=230086.16
clat (usec): min=769, max=39675, avg=12267.87, stdev=3172.09
lat (usec): min=772, max=39681, avg=12314.59, stdev=3187.43
clat percentiles (usec):
| 1.00th=[ 6521], 5.00th=[ 7963], 10.00th=[ 8717], 20.00th=[ 9765],
| 30.00th=[10421], 40.00th=[11207], 50.00th=[11863], 60.00th=[12518],
| 70.00th=[13435], 80.00th=[14484], 90.00th=[16319], 95.00th=[17957],
| 99.00th=[22414], 99.50th=[23987], 99.90th=[26870], 99.95th=[28181],
| 99.99th=[30802]
bw ( KiB/s): min=58880, max=101216, per=25.00%, avg=83130.06, stdev=8366.08, samples=959
iops : min=14720, max=25304, avg=20782.51, stdev=2091.52, samples=959
lat (usec) : 1000=0.01%
lat (msec) : 2=0.01%, 4=0.02%, 10=23.35%, 20=74.16%, 50=2.47%
cpu : usr=3.21%, sys=9.27%, ctx=425656, majf=0, minf=28
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1%
issued rwt: total=0,9977029,0, short=0,0,0, dropped=0,0,0
latency : target=0, window=0, percentile=100.00%, depth=256
Run status group 0 (all jobs):
WRITE: bw=325MiB/s (341MB/s), 325MiB/s-325MiB/s (341MB/s-341MB/s), io=38.1GiB (40.9GB), run=120007-120007msec
$ Hi,
Quoted 42 lines · Durval Menezes
Hello everyone,
I've arrived at a very surprising number measuring IOPS write performance on my SSDs' "bare metal" (ie, straight on the
/dev/$DISK, no filesystem involved):export COMMON_OPTIONS='--ioengine=libaio --direct=1 --runtime=120 --time_based --group_reporting' ls -l /dev/disk/by-id | grep 'ata-.*sda' lrwxrwxrwx 1 root root 9 Feb 13 17:19 ata-SAMSUNG_MZ7LM1T9HCJM-00003_XXXXXXXXXXXXXX -> ../../sda TANGO=/dev/disk/by-id/ata-SAMSUNG_MZ7LM1T9HCJM-00003_XXXXXXXXXXXXXX sudo fio --filename=${TANGO} --name=device_iops_write --rw=randwrite --bs=4k --iodepth=256 --numjobs=4 ${COMMON_OPTIONS} [...] write: *IOPS=83.1k*, BW=325MiB/s (341MB/s)(38.1GiB/120007msec) [...](please find the complete output at the end of this message, in case I should have looked at some other lines and/or you are curious)
As per the official manufacturer specs (both in this whitepaper at their website[1]), and also in this datasheet I found somewhere else[2]), it's supposed to be only *18K IOPS*.
All the other base performance numbers I've measured (read IOPS, read and write MB/s, read and write latencies) are at or very near the manufacturer specs.
What's going on?
At first I thought that, despite
--direct=1being explicitly indicated, my machine's 64GB RAM (via the Linux buffer cache) could be caching the writes (even if the number, in that case, should have been much higher)... so, I tested it again with--runtime=120to saturate the buffer cache in case it was really the 'culprit'... lo and behold, the result was:[...] write: IOPS=83.1k, BW=325MiB/s (341MB/s)(190GiB/600019msec) [...]
Quoted 6 lines
So, the surprising over-4.6x-times-the-spec Write IOPS is mantained, even for 190GiB total data.
And with 190GiB data written (about 10% the total device capacity), I do not believe it's any kind of cache (RAM, MLC or whatever) inside the SSD either.
You're running your workload for a comparatively short time and additionally we don't know how "fresh" your SSD is. The 18K IOPS value might be when the drive has been fully written and there are no pre-erased blocks available (via so-called preconditioning)... I'll also note the whitepaper [1] mentions this:
Quoted 1 line
SSD Precondition: Sustained state (or steady state)
[...]
Quoted 1 line
It's important to note that all performance items mentioned in this white paper have been measured at the sustained state, except the sequential read/write performance
I notice that your SSD appears to be SATA (sda) so I'd be surprised that a total queue depth greater than 32 makes a difference (your total queue depth is 1024). Do you get a similar result with just the one job with an iodepth=32?
It's unlikely but if the jobs were submitting I/O to the same areas as other jobs at the same time then some of the I/O could be elided but given what you've posted this should not be the case.
Signature · 1 line
Sitsofe