block: set reasonable default for discard max
PATCH 1/1 block: set reasonable default for discard max
Some drive manufacturers export a very large supported max discard size. However, when the operating system sends I/O of the max size to the device, extreme I/O latency can often be encountered. Since hardware does not provide an optimal discard value in addition to the max, and there is no way to foreshadow how well a drive handles the large size, take the method from max_sectors setting, and use BLK_DEF_MAX_SECTORS to set a more reasonable default discard max. This should avoid the extreme latency while still allowing the user to increase the value for specific needs.
--- Documentation/ABI/stable/sysfs-block | 4 +++- block/blk-settings.c | 3 ++- 2 files changed, 5 insertions(+), 2 deletions(-)
Documentation/ABI/stable/sysfs-block +3 -1
@@ -235,7 +235,9 @@ Description: large latencies when large discards are issued, setting this value lower will make Linux issue smaller discards and potentially help reduce latencies induced by large discard - operations. + operations. For this reason, the max is currently defaulted to + four times BLK_DEF_MAX_SECTORS, but can be increased via sysfs + as needed. What: /sys/block/<disk>/queue/discard_max_hw_bytes
block/blk-settings.c +2 -1
@@ -179,7 +179,8 @@ void blk_queue_max_discard_sectors(struct request_queue *q, unsigned int max_discard_sectors) { q->limits.max_hw_discard_sectors = max_discard_sectors; - q->limits.max_discard_sectors = max_discard_sectors; + q->limits.max_discard_sectors =min (max_discard_sectors, +BLK_DEF_MAX_SECTORS *4 ); }EXPORT_SYMBOL (blk_queue_max_discard_sectors);
Signature · 1 line
2.38.1
John,
Quoted 9 lines
Some drive manufacturers export a very large supported max discard size. However, when the operating system sends I/O of the max size to the device, extreme I/O latency can often be encountered. Since hardware does not provide an optimal discard value in addition to the max, and there is no way to foreshadow how well a drive handles the large size, take the method from
max_sectorssetting, and useBLK_DEF_MAX_SECTORSto set a more reasonable default discard max. This should avoid the extreme latency while still allowing the user to increase the value for specific needs.
What's reasonable for one device may be completely unreasonable for another. 4 * BLK_DEF_MAX_SECTORS is *tiny* and will penalize performance on many devices.
If there's a problem with a device returning something that doesn't make sense, let's quirk it.
Signature · 1 line
Martin K. Petersen Oracle Linux Engineering
Thanks alot for responding Martin. Forgive my ignorance; just trying to gain understanding. For example, if we find a device with a 2TiB max discard (way too high for any device to handle reasonably from what I've seen), and we make a quirk for it that brings the max discard down, how do we decide what value to bring that down to? Would we ask the hardware vendor for an optimal value? Is there some way we could decide the value? Thanks again for any help.
Quoted 24 lines · Martin K. Petersen
John,
Quoted 9 lines
Some drive manufacturers export a very large supported max discard size. However, when the operating system sends I/O of the max size to the device, extreme I/O latency can often be encountered. Since hardware does not provide an optimal discard value in addition to the max, and there is no way to foreshadow how well a drive handles the large size, take the method from
max_sectorssetting, and useBLK_DEF_MAX_SECTORSto set a more reasonable default discard max. This should avoid the extreme latency while still allowing the user to increase the value for specific needs.What's reasonable for one device may be completely unreasonable for another. 4 *
BLK_DEF_MAX_SECTORSis *tiny* and will penalize performance on many devices.If there's a problem with a device returning something that doesn't make sense, let's quirk it.
--
Martin K. Petersen Oracle Linux Engineering
Quoted 7 lines · John Pittman
Thanks alot for responding Martin. Forgive my ignorance; just trying to gain understanding. For example, if we find a device with a 2TiB max discard (way too high for any device to handle reasonably from what I've seen), and we make a quirk for it that brings the max discard down, how do we decide what value to bring that down to? Would we ask the hardware vendor for an optimal value? Is there some way we could decide the value? Thanks again for any help.
asking a H/W vendor is definitely good start, but you have to test that value and make sure it is working before adding a quirk is right way to go, because sometimes those values won't work from my experience ...
Also, in absence of information from H/W vendor I'd run few experiments with the range of this values to establish the correctness that will not be too low to significantly degrade the performance or to high to create potential problems.
-ck
John,
Quoted 4 lines
Thanks alot for responding Martin. Forgive my ignorance; just trying to gain understanding. For example, if we find a device with a 2TiB max discard (way too high for any device to handle reasonably from what I've seen),
That really depends. On some devices performing a 2TB discard may just involve updating a few entries in a block translation table. It could be close to constant time regardless of the number of terabytes unmapped.
Quoted 4 lines
and we make a quirk for it that brings the max discard down, how do we decide what value to bring that down to? Would we ask the hardware vendor for an optimal value? Is there some way we could decide the value?
The best thing would be to ask the vendor. Or maybe if you provide the output of
# sg_vpd -p bl /dev/sdN
and we can try to see if we can come up with a suitable heuristic for a quirk.
Signature · 1 line
Martin K. Petersen Oracle Linux Engineering
FWIW the most recent discard performance issue I looked at was an early SSD that only supported a small max discard size. Doing a discard of the whole device (mkfs) took several minutes with lots of little discards. So it isn't always just an issue of the max size.
-Ewan
Quoted 29 lines · Martin K. Petersen
John,
Quoted 4 lines
Thanks alot for responding Martin. Forgive my ignorance; just trying to gain understanding. For example, if we find a device with a 2TiB max discard (way too high for any device to handle reasonably from what I've seen),
That really depends. On some devices performing a 2TB discard may just involve updating a few entries in a block translation table. It could be close to constant time regardless of the number of terabytes unmapped.
Quoted 4 lines
and we make a quirk for it that brings the max discard down, how do we decide what value to bring that down to? Would we ask the hardware vendor for an optimal value? Is there some way we could decide the value?
The best thing would be to ask the vendor. Or maybe if you provide the output of
# sg_vpd -p bl /dev/sdNand we can try to see if we can come up with a suitable heuristic for a quirk.
--
Martin K. Petersen Oracle Linux Engineering
Quoted 1 line · John Pittman
Some drive manufacturers export a very large supported max discard size.
Most of devices exports very large max discard size, and it shouldn't be just some.
Quoted 2 lines
However, when the operating system sends I/O of the max size to the device, extreme I/O latency can often be encountered. Since hardware
It often depends on drive internal state.
Quoted 6 lines
does not provide an optimal discard value in addition to the max, and there is no way to foreshadow how well a drive handles the large size, take the method from
max_sectorssetting, and useBLK_DEF_MAX_SECTORSto set a more reasonable default discard max. This should avoid the extreme latency while still allowing the user to increase the value for specific needs.
As Martin and Chaitanya mentioned, reducing max discard size to level of BLK_DEF_MAX_SECTORS won't be one ideal approach, and shouldn't be accepted, since discarding command is supposed to be cheap from device viewpoint, such as, SSD FW usually just marks the LBA ranges as invalid for handling discard.
However, how well discard performs may depend on SSD internal, such as if GC is in-progress. And it might be one more generic issue for SSD, and it was discussed several times, such as "Issues around discard" in lsfmm2019:
https://lwn.net/Articles/787272/
BTW, blk-wbt actually throttles discard, but it is just in queue depth level. If max discard size is too big, single big discard request still may cause device irresponsive.
Thanks,
Ming
Hi Jens, I saw in the v2 of your original patchset that made max discard rw, you were going to set a 64M default, but Mike mentioned some issues it might present in dm, so you said you would come back to it later and decide on a default max [1]. In the link that Ming provided [2], the general consensus was that the hw manufacturers should do what they say; they've sold this function as fast, so they should make it fast. But unfortunately, from what I've read, it still seems there are very few, if any, devices that can truly handle a 2TiB discard well. Of course, I don't have any insight into the hw manufacturers intentions, but in my opinion, we shouldn't be keeping problematic kernel limits based on a selling point by the hw manufacturers.
So, the thought (after bugging Ming offline) is we could take one of the below approaches:
1) Set a 64M limit as you've suggested in the past. It seems more prudent to tune the value upward for the few devices that can actually handle a 2TiB discard rather than tune downward for the large majority.
2) Create a udev rule to go into RHEL that would set 64M as we still steadily see support cases where max discard needs to be tuned downward.
In either case, I suppose we'd have to figure a way to not impact dm. Maybe we could do that with a driver setting or udev rule?
Jens and Mike, Do you guys have any thoughts on this? Asking here instead of RH bug (2160828) since Jens said he wanted to come back to this issue later in the thread mentioned.
Thanks for the continued assistance.
[1] https://lkml.org/lkml/2015/7/15/859
[2] https://lwn.net/Articles/787272/
Quoted 40 lines · Ming Lei
Quoted 1 line · John Pittman
Some drive manufacturers export a very large supported max discard size.
Most of devices exports very large max discard size, and it shouldn't be just some.
Quoted 2 lines
However, when the operating system sends I/O of the max size to the device, extreme I/O latency can often be encountered. Since hardware
It often depends on drive internal state.
Quoted 6 lines
does not provide an optimal discard value in addition to the max, and there is no way to foreshadow how well a drive handles the large size, take the method from
max_sectorssetting, and useBLK_DEF_MAX_SECTORSto set a more reasonable default discard max. This should avoid the extreme latency while still allowing the user to increase the value for specific needs.As Martin and Chaitanya mentioned, reducing max discard size to level of
BLK_DEF_MAX_SECTORSwon't be one ideal approach, and shouldn't be accepted, since discarding command is supposed to be cheap from device viewpoint, such as, SSD FW usually just marks the LBA ranges as invalid for handling discard.However, how well discard performs may depend on SSD internal, such as if GC is in-progress. And it might be one more generic issue for SSD, and it was discussed several times, such as "Issues around discard" in lsfmm2019:
https://lwn.net/Articles/787272/BTW, blk-wbt actually throttles discard, but it is just in queue depth level. If max discard size is too big, single big discard request still may cause device irresponsive.
Thanks,
Ming
John,
Quoted 4 lines
1) Set a 64M limit as you've suggested in the past. It seems more prudent to tune the value upward for the few devices that can actually handle a 2TiB discard rather than tune downward for the large majority.
This makes the assumption that smaller always equals faster. However, some devices track allocations in very large units. And as a result, sending smaller discards will result in *slower* performance for those devices. In addition, for doings things like the full block device sweep we do during mkfs, it is often imperative that we issue large sequential ranges.
Please provide the VPD pages from the device in question.
Signature · 1 line
Martin K. Petersen Oracle Linux Engineering
Thanks Martin. Sorry for the wait; it took a while for the user to respond. They don't have scsi devs but rather nvme devs. The sg_vpd command failed b/c there was no bl page available. I asked them to provide the supported vpd pages that are there. I do have some nvme data though from a sosreport. I'll send you that in a separate email. Thanks again.
Quoted 21 lines · Martin K. Petersen
John,
Quoted 4 lines
1) Set a 64M limit as you've suggested in the past. It seems more prudent to tune the value upward for the few devices that can actually handle a 2TiB discard rather than tune downward for the large majority.
This makes the assumption that smaller always equals faster. However, some devices track allocations in very large units. And as a result, sending smaller discards will result in *slower* performance for those devices. In addition, for doings things like the full block device sweep we do during mkfs, it is often imperative that we issue large sequential ranges.
Please provide the VPD pages from the device in question.
--
Martin K. Petersen Oracle Linux Engineering
John,
Quoted 5 lines
Sorry for the wait; it took a while for the user to respond. They don't have scsi devs but rather nvme devs. The
sg_vpdcommand failed b/c there was no bl page available. I asked them to provide the supported vpd pages that are there. I do have some nvme data though from a sosreport. I'll send you that in a separate email.
The device vendor should be fixing their firmware to report a suitable set of DMRL/DMRSL/DMSL values which align with the expected performance of the device.
Signature · 1 line
Martin K. Petersen Oracle Linux Engineering
Thanks again Martin. We're going to try to engage the hardware folks now.
Quoted 17 lines · Martin K. Petersen
John,
Quoted 5 lines
Sorry for the wait; it took a while for the user to respond. They don't have scsi devs but rather nvme devs. The
sg_vpdcommand failed b/c there was no bl page available. I asked them to provide the supported vpd pages that are there. I do have some nvme data though from a sosreport. I'll send you that in a separate email.The device vendor should be fixing their firmware to report a suitable set of DMRL/DMRSL/DMSL values which align with the expected performance of the device.
--
Martin K. Petersen Oracle Linux Engineering