Samsung PM863 SSD: surprisingly high Write IOPS measured using `fio`, over 4.6 times more than spec!?
Hi Sitsofe,
First of all, thanks for your detailed, thoughtful response. More, below:
Quoted 52 lines · Sitsofe Wheeler
Quoted 8 lines · Durval Menezes
Hello everyone,
I've arrived at a very surprising number measuring IOPS write performance on my SSDs' "bare metal" (ie, straight on the
/dev/$DISK, no filesystem involved):export COMMON_OPTIONS='--ioengine=libaio --direct=1--runtime=120 --time_based --group_reporting'
Quoted 3 lines
ls -l
/dev/disk/by-id| grep 'ata-.*sda'
lrwxrwxrwx 1 root root 9 Feb 13 17:19ata-SAMSUNG_MZ7LM1T9HCJM-00003_XXXXXXXXXXXXXX -> ../../sda
Quoted 2 lines
TANGO=/dev/disk/by-id/ata-SAMSUNG_MZ7LM1T9HCJM-00003_XXXXXXXXXXXXXX
Quoted 1 line
sudo fio --filename=${TANGO} --name=device_iops_write--rw=randwrite --bs=4k --iodepth=256 --numjobs=4 ${COMMON_OPTIONS}
Quoted 2 lines
[...] write: *IOPS=83.1k*, BW=325MiB/s(341MB/s)(38.1GiB/120007msec)
Quoted 3 lines
[...](please find the complete output at the end of this message, in case I
should
Quoted 8 lines
have looked at some other lines and/or you are curious)
As per the official manufacturer specs (both in this whitepaper at their website[1]), and also in this datasheet I found somewhere else[2]), it's supposed to be only *18K IOPS*.
All the other base performance numbers I've measured (read IOPS, read and write MB/s, read and write latencies) are at or very near the
manufacturer
Quoted 7 lines
specs.
What's going on?
At first I thought that, despite
--direct=1being explicitly indicated, my machine's 64GB RAM (via the Linux buffer cache) could be caching the writes (even if the number, in that case, should have been muchhigher)...
Quoted 1 line
so, I tested it again with
--runtime=120to saturate the buffer cachein
Quoted 7 lines
case it was really the 'culprit'... lo and behold, the result was:
[...] write: IOPS=83.1k, BW=325MiB/s (341MB/s)(190GiB/600019msec) [...]
Quoted 8 lines
Quoted 7 lines
So, the surprising over-4.6x-times-the-spec Write IOPS is mantained, even for 190GiB total data.
And with 190GiB data written (about 10% the total device capacity), I do not believe it's any kind of cache (RAM, MLC or whatever) inside the SSD either.
You're running your workload for a comparatively short time
OK, I was able to find in the whitepaper (see below) the manufacturer stating that the rand writes should be run for twice the capacity of the disk. That will also imply a much longer test time...
Quoted 1 line
and additionally we don't know how "fresh" your SSD is.
Good point; here's its "freshness"-relevant data straight from smartctl -a:
9 Power_On_Hours 0x0032 091 091 000 Old_age Always - 43694 177 Wear_Leveling_Count 0x0013 094 094 005 Pre-fail Always - 394 241 Total_LBAs_Written 0x0032 099 099 000 Old_age Always - 535797643689 242 Total_LBAs_Read 0x0032 099 099 000 Old_age Always - 1848967801660 251 NAND_Writes 0x0032 100 100 000 Old_age Always - 1642499721864
So, I think it's a pretty 'mature' disk already (but hopefully with a lot of 'life' still in it).
In other words, I don't think it's "fresh" enough to explain a 4x I/O increase.
Perhaps "freshness" in this case refers to it being recently secure-erased (which I did prior to start testing)?
Quoted 11 lines
The 18K IOPS value
might be when the drive has been fully written and there are no pre-erased blocks available (via so-called preconditioning)... I'll also note the whitepaper [1] mentions this:SSD Precondition: Sustained state (or steady state) [...]
It's important to note that all performance items mentioned in this white paper have been measured at the sustained state, except the sequential read/write performance
Thanks for going through the whitepaper and picking this up. It passed right by me...
I went through the whitepaper again, and found this:
The sustained state in this document refers to the status that a 128 KB sequential write has been completed equal to the drive capacity and then 4 KB random write has completed twice as much as the drive capacity
OK, so at least there's a "recipe" for this preconditioning. I will try it and come back later to report.
Quoted 4 lines
I notice that your SSD appears to be SATA (sda) so I'd be surprised that a total queue depth greater than 32 makes a difference (your total queue depth is 1024). Do you get a similar result with just the one job with an iodepth=32?
I tested with iodepth=32 (instead of 256) and got the same result, so I guess you are not surprised ;-)
Just did it again, this time with --numjobs=1 (instead of 4) and here's
the result:
write: IOPS=83.0k, BW=324MiB/s (340MB/s)(38.0GiB/120001msec)
So that's not it either.
Quoted 3 lines
It's unlikely but if the jobs were submitting I/O to the same areas as other jobs at the same time then some of the I/O could be elided but given what you've posted this should not be the case.
Agreed.
Cheers,
Signature · 1 line
Durval.
Quoted 6 lines
--
SitsofeQuoted 2 lines