scsi_lib: Align max_sectors to kb
Subsystems (2)
PATCH 1/1 scsi_lib: Align max_sectors to kb
max_sectors can be modified via sysfs, but only in kb units. Which leads to a misalignment on stacked devices if the original max_sector size is an odd number. So align the max_sectors setting to kb to avoid this issue.
--- drivers/scsi/scsi_lib.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)
drivers/scsi/scsi_lib.c +2 -1
@@ -1983,7 +1983,8 @@ void __scsi_init_queue(struct Scsi_Host *shost, struct request_queue *q)blk_queue_max_integrity_segments (q, shost->sg_prot_tablesize); } -blk_queue_max_hw_sectors (q, shost->max_sectors); +/* Align to kb to avoid conflicts with Sysfs settings */ +blk_queue_max_hw_sectors (q, shost->max_sectors & ~0x1 );blk_queue_segment_boundary (q, shost->dma_boundary);dma_set_seg_boundary (dev, shost->dma_boundary);
Signature · 1 line
2.35.3
Quoted 1 line · Hannes Reinecke
max_sectorscan be modified via sysfs, but only in kb units.
Yes.
Quoted 2 lines
Which leads to a misalignment on stacked devices if the original
max_sectorsize is an odd number.
How?
Note that we really should not stack max_sectors anyway, as it's only used for splitting in the lower device to start with.
Quoted 10 lines · Christoph Hellwig
Quoted 1 line · Hannes Reinecke
max_sectorscan be modified via sysfs, but only in kb units.Yes.
Quoted 2 lines
Which leads to a misalignment on stacked devices if the original
max_sectorsize is an odd number.How?
That's an issue we've been seeing during testing:
https://lore.kernel.org/dm-devel/[email protected]/
While this can be fixed in userspace (Martin Wilck provided a patchset
to multipath-tools), what I find irritating is that we will always
display the max_sectors setting in kb, even if the actual value is not
kb aligned.
_And_ we allow to modify that value (again in kb units). Which means that we _never_ are able to reset it to its original value.
I would vastly prefer to have the actual values displayed in sysfs,
ie either have a max_sectors_kb value for the block limits or have anmax_sectors setting in sysfs, but we really should avoid this 'shift by 1' when displaying the value.
Quoted 2 lines
Note that we really should not stack
max_sectorsanyway, as it's only used for splitting in the lower device to start with.
If that's the case, why don't we inhibit the modification for max_sectors on the lower devices?
Cheers,
Hannes
Quoted 15 lines · Hannes Reinecke
Quoted 10 lines · Christoph Hellwig
Quoted 1 line · Hannes Reinecke
max_sectorscan be modified via sysfs, but only in kb units.Yes.
Quoted 2 lines
Which leads to a misalignment on stacked devices if the original
max_sectorsize is an odd number.How?
That's an issue we've been seeing during testing:
https://lore.kernel.org/dm-devel/[email protected]/
Quoted 8 lines
While this can be fixed in userspace (Martin Wilck provided a patchset
to multipath-tools), what I find irritating is that we will always display themax_sectorssetting in kb, even if the actual value is not
kb aligned.
_And_ we allow to modify that value (again in kb units). Which means that we _never_ are able to reset it to its original value.
User space has no way to determine whether the actual max_sectors value in the kernel is even or odd. By reading max_sectors_kb and writing it back, one may actually change the kernel-internal value by rounding it down to the next even number. This can cause breakage if the device being changed is a multipath path device.
Wrt Hannes' patch: It would fix the issue on the kernel side, but user space would still have no means to determine whether this patch applied or not, except for checking the kernel version, which is unreliable. For user space, it would be more helpful to add a "max_sectors" sysfs attribute that exposes the actual value in blocks.
Quoted 6 lines
Quoted 3 lines
Note that we really should not stack
max_sectorsanyway, as it's only
used for splitting in the lower device to start with.If that's the case, why don't we inhibit the modification for
max_sectorson the lower devices?
I vote for allowing changes to max_sectors_kb only for devices that don't have anything stacked on top, even though my late testing indicates that it's only a real problem with request-based dm aka multipath. After all, max_sectors only needs to be manipulated in rare situations, and would be generally recommended to do this in an udev rule early during boot.
Regards,
Martin
Quoted 4 lines · Hannes Reinecke
While this can be fixed in userspace (Martin Wilck provided a patchset to multipath-tools), what I find irritating is that we will always display the
max_sectorssetting in kb, even if the actual value is not kb aligned.
The problem you are running into it exactly a problem of pointlessly inheriting the value when we should not.
Other secondary issues are that we:
a) should allow the modification in the granularity that we actually internally use, that is sectors
b) apparently some drivers still use the silly BLK_SAFE_MAX_SECTORS limits that is too low for just about any modern hardwre.
Quoted 5 lines
Quoted 2 lines
Note that we really should not stack
max_sectorsanyway, as it's only used for splitting in the lower device to start with.If that's the case, why don't we inhibit the modification for
max_sectorson the lower devices?
Why would we? It makes absolutly no sense to inherit these limits, the lower device will split anyway which is very much the point of the immutable bio_vec work.
Quoted 5 lines · Christoph Hellwig
Quoted 5 lines
Quoted 4 lines
Note that we really should not stack
max_sectorsanyway, as it's only
used for splitting in the lower device to start with.
Quoted 8 lines
Quoted 3 lines
If that's the case, why don't we inhibit the modification for
max_sectors
on the lower devices?Why would we? It makes absolutly no sense to inherit these limits, the lower device will split anyway which is very much the point of the immutable
bio_vecwork.
Sorry, I don't follow. With (request-based) dm-multipath on top of SCSI, we hit the "over max size limit" condition in blk_insert_cloned_request() [1], which will cause IO to fail at the dm level. So at least in this configuration, it's crucial that the upper device inherit the lower device's limits.
Regards,
Martin
[1] https://elixir.bootlin.com/linux/latest/source/block/blk-mq.c#L3048
Quoted 10 lines · Martin Wilck
Quoted 4 lines
Why would we? It makes absolutly no sense to inherit these limits, the lower device will split anyway which is very much the point of the immutable
bio_vecwork.Sorry, I don't follow. With (request-based) dm-multipath on top of SCSI, we hit the "over max size limit" condition in
blk_insert_cloned_request()[1], which will cause IO to fail at the dm level. So at least in this configuration, it's crucial that the upper device inherit the lower device's limits.
Oh, indeed. Request based multipath is different from everyone else. I really wish we could spend the effort to convert it to bio based and remove this special case..
(added Mike, Ben, and dm-devel)
Quoted 16 lines · Christoph Hellwig
Quoted 7 lines · Martin Wilck
Quoted 6 lines
Why would we? It makes absolutly no sense to inherit these limits,
the lower device will split anyway which is very much the point of
the immutablebio_vecwork.Quoted 7 lines
Sorry, I don't follow. With (request-based) dm-multipath on top of SCSI, we hit the "over max size limit" condition in
blk_insert_cloned_request()[1], which will cause IO to fail at the dm
level. So at least in this configuration, it's crucial that the upper
device inherit the lower device's limits.
Quoted 4 lines
Oh, indeed. Request based multipath is different from everyone else. I really wish we could spend the effort to convert it to bio based and remove this special case..
Well, it's just a matter of setting "queue_mode" in the multipath configuration. But request-based has been the default since kernel v3.0 (f40c67f0f7e2 ("dm mpath: change to be request based")). While we got bio-based dm-multipath back in v4.8 (76e33fe4e2c4 ("dm mpath:
reinstate bio-based support")), making it the default or even removing request-based dm (in order to free the block layer from limit- stacking requirements) will obviously require a broad discussion.
Mike has pointed out in 76e33fe4e2c4 that many of the original reasons for introducing request-based dm-multipath have been obsoleted by faster hardware and improvements in the generic block layer.
I am open for discussion on this subject, but with my distribution maintainer hat on, I don't expect the default being changed for end customers soon. Actually, with NVMe rising, I wonder if a major technology switch for dm-multipath (which is effectively SCSI multipath these days) makes sense at this point in time.
Thanks,
Martin
Quoted 2 lines · Martin Wilck
Well, it's just a matter of setting "
queue_mode" in the multipath configuration.
Yes, and no. We'd probably at very last want a blk_plug callback to allow making decisions for entire large I/Os instead of on a per-bio basis.
Quoted 3 lines
Mike has pointed out in
76e33fe4e2c4that many of the original reasons for introducing request-based dm-multipath have been obsoleted by faster hardware and improvements in the generic block layer.
Yes.
Quoted 5 lines
I am open for discussion on this subject, but with my distribution maintainer hat on, I don't expect the default being changed for end customers soon. Actually, with NVMe rising, I wonder if a major technology switch for dm-multipath (which is effectively SCSI multipath these days) makes sense at this point in time.
Well, it'll take ages until anything upstream gets into the enterprise distros, which array users tend to use anyway. But I'd really love to get the ball rolling upstream rather sooner than later even if there is no need for immediate action. The reason is that request based dm multipath causes a lot of special casing in the block core and dm, and that has become a significant maintenance burden.
So if you have a chance to test the current bio based path, or help with testing a plug callback (I can try to write a patch for that, but I don't really have a good test setup) we can start to see where we are in practice in terms of still needing the request based stacking, and if there is a feasible path to move away from it.