Skip to contentSkip to conversation

block: make queue limits workable in case of 64K PAGE_SIZE

PATCH In review Ming Lei 4 participants 14 messages Jan 2, 2025 – Jan 8, 2025 Block layer

lore
Share & tools
lore

Ming Lei

author
#0 lore raw

PATCH 1/1 block: make queue limits workable in case of 64K PAGE_SIZE

PAGE_SIZE is applied in some block device queue limits, this way is very fragile and is wrong:

- queue limits are read from hardware, which is often one readonly hardware property

- PAGE_SIZE is one config option which can be changed during build time.

In RH lab, it has been found that max segment size of some mmc card is less than 64K, then this kind of card can't work in case of 64K PAGE_SIZE.

Fix this issue by using SZ_4K in related code for dealing with queue limits.

Cc Yi Zhang <[email protected]>
Cc Luis Chamberlain <[email protected]>
Cc John Garry <[email protected]>
---
 block/blk-settings.c | 6 +++---
 block/blk.h          | 2 +-
 2 files changed, 4 insertions(+), 4 deletions(-)
block/blk-settings.c +3 -3
index 8f09e33f41f6..3b6918c134b1 100644
@@ -300,7 +300,7 @@ int blk_validate_limits(struct queue_limits *lim)
 	max_hw_sectors = min_not_zero(lim->max_hw_sectors,
 				lim->max_dev_sectors);
 	if (lim->max_user_sectors) {
-		if (lim->max_user_sectors < PAGE_SIZE / SECTOR_SIZE)
+		if (lim->max_user_sectors < SZ_4K / SECTOR_SIZE)
 			return -EINVAL;
 		lim->max_sectors = min(max_hw_sectors, lim->max_user_sectors);
 	} else if (lim->io_opt > (BLK_DEF_MAX_SECTORS_CAP << SECTOR_SHIFT)) {
@@ -338,7 +338,7 @@ int blk_validate_limits(struct queue_limits *lim)
 	 */
 	if (!lim->seg_boundary_mask)
 		lim->seg_boundary_mask = BLK_SEG_BOUNDARY_MASK;
-	if (WARN_ON_ONCE(lim->seg_boundary_mask < PAGE_SIZE - 1))
+	if (WARN_ON_ONCE(lim->seg_boundary_mask < SZ_4K - 1))
 		return -EINVAL;
 
 	/*
@@ -359,7 +359,7 @@ int blk_validate_limits(struct queue_limits *lim)
 		 */
 		if (!lim->max_segment_size)
 			lim->max_segment_size = BLK_MAX_SEGMENT_SIZE;
-		if (WARN_ON_ONCE(lim->max_segment_size < PAGE_SIZE))
+		if (WARN_ON_ONCE(lim->max_segment_size < SZ_4K))
 			return -EINVAL;
 	}
 
block/blk.h +1 -1
index 2c26abf505b8..570d66f1e338 100644
@@ -357,7 +357,7 @@ static inline bool bio_may_need_split(struct bio *bio,
 		const struct queue_limits *lim)
 {
 	return lim->chunk_sectors || bio->bi_vcnt != 1 ||
-		bio->bi_io_vec->bv_len + bio->bi_io_vec->bv_offset > PAGE_SIZE;
+		bio->bi_io_vec->bv_len + bio->bi_io_vec->bv_offset > SZ_4K;
 }
 
 /**
Signature · 1 line
2.44.0
#1 lore raw
Quoted 2 lines · Ming Lei

In RH lab, it has been found that max segment size of some mmc card is less than 64K, then this kind of card can't work in case of 64K PAGE_SIZE.

That means that that MMC card is incompatible with the 64K page size.

Additionally, this patch looks wrong to me. There are good reasons why the block layer requires that the DMA segment size is at least as large as the page size.

You may want to take a look at this rejected patch series:
Bart Van Assche, "PATCH v6 0/8] Support limits below the page size",
June 2023 (https://lore.kernel.org/linux-block/[email protected]/).

Bart.

author
#2 lore raw
Quoted 5 lines · Bart Van Assche
Quoted 2 lines · Ming Lei

In RH lab, it has been found that max segment size of some mmc card is less than 64K, then this kind of card can't work in case of 64K PAGE_SIZE.

That means that that MMC card is incompatible with the 64K page size.

Is there any such linux kernel compatibility standard? If yes, please share it here.

Quoted 4 lines

Additionally, this patch looks wrong to me. There are good reasons why the block layer requires that the DMA segment size is at least as large as the page size.

Do you think 512byte sector can't be DMAed?

Quoted 4 lines

You may want to take a look at this rejected patch series:
Bart Van Assche, "PATCH v6 0/8] Support limits below the page size",
June 2023 (https://lore.kernel.org/linux-block/[email protected]/).

'502 Bad Gateway' is returned for the above link.

Thanks,
Ming

#3 lore raw
Quoted 6 lines · Ming Lei
Quoted 3 lines · Bart Van Assche

Additionally, this patch looks wrong to me. There are good reasons why the block layer requires that the DMA segment size is at least as large as the page size.

Do you think 512byte sector can't be DMAed?

A clarification: I was referring to the maximum DMA segment size. The 512 byte number in your email refers to the actual DMA transfer size. My statement does not apply to the actual DMA transfer size.

Quoted 5 lines
Quoted 3 lines

You may want to take a look at this rejected patch series:
Bart Van Assche, "PATCH v6 0/8] Support limits below the page size",
June 2023 (https://lore.kernel.org/linux-block/[email protected]/).

'502 Bad Gateway' is returned for the above link.

That link works fine here on multiple devices (laptop, workstation, smartphone) so please check your setup.

Thanks,

Bart.

author
#4 lore raw
Quoted 11 lines · Bart Van Assche
Quoted 6 lines · Ming Lei
Quoted 3 lines · Bart Van Assche

Additionally, this patch looks wrong to me. There are good reasons why the block layer requires that the DMA segment size is at least as large as the page size.

Do you think 512byte sector can't be DMAed?

A clarification: I was referring to the maximum DMA segment size. The 512 byte number in your email refers to the actual DMA transfer size. My statement does not apply to the actual DMA transfer size.

But why does DMA segment size have to be >= PAGE_SIZE(4KB, 64KB)?

I don't think that the constraint is from hardware because DMA controller is capable of doing DMA on buffer with smaller alignment & size.

IMO, the limit is actually from block layer, that is why I posted out this patch.

Quoted 9 lines
Quoted 6 lines
Quoted 4 lines

You may want to take a look at this rejected patch series:
Bart Van Assche, "PATCH v6 0/8] Support limits below the page size",
June 2023 (https://lore.kernel.org/linux-block/[email protected]/).

'502 Bad Gateway' is returned for the above link.

That link works fine here on multiple devices (laptop, workstation, smartphone) so please check your setup.

It is reachable now.

From the link, you have storage controllers with DMA segment size which is less than 4K, which may never get supported by linux kernel.

I am looking at hardware which works fine on kernel with 4K page size, but it becomes not workable when kernel is built as 64K page size, which confuses final kernel users.

Thanks,
Ming

#5 lore raw
Quoted 1 line · Ming Lei

But why does DMA segment size have to be >= PAGE_SIZE(4KB, 64KB)?

From the description of patch 5/8 of my patch series: "If the segment size is smaller than the page size there may be multiple segments per bvec even if a bvec only contains a single page." The current block layer implementation is based on the assumption that a single page fits in a single DMA segment. Please take a look at patch 5/8 of my patch series.

Quoted 2 lines

From the link, you have storage controllers with DMA segment size which is less than 4K, which may never get supported by linux kernel.

As mentioned in the cover letter of that patch series, I came up with that patch series to support a DMA controller with max_segment_size of 4 KiB on a system with a PAGE_SIZE of 16 KiB.

Thanks,

Bart.

#6 lore raw
Quoted 9 lines · Bart Van Assche
Quoted 1 line · Ming Lei

But why does DMA segment size have to be >= PAGE_SIZE(4KB, 64KB)?

From the description of patch 5/8 of my patch series: "If the segment size is smaller than the page size there may be multiple segments per bvec even if a bvec only contains a single page." The current block layer implementation is based on the assumption that a single page fits in a single DMA segment. Please take a look at patch 5/8 of my patch series.

While that addresses a smaller segment then page size this still leaves open the question of if a dma segment can be larger than page size, given our adoption of folios I don't see why we would constrain ourselves to a page size. I already have PoC code which enables this.

Luis

#7 lore raw
Quoted 2 lines · Luis Chamberlain

While that addresses a smaller segment then page size this still leaves open the question of if a dma segment can be larger than page size,

Hmm ... aren't max_segment_size values that are larger than the page size supported since day one of the Linux kernel? Or are you perhaps referring to Ming's multi-page bvec patch series that was merged about six years ago?

Thanks,

Bart.

#8 lore raw
Quoted 7 lines · Bart Van Assche
Quoted 2 lines · Luis Chamberlain

While that addresses a smaller segment then page size this still leaves open the question of if a dma segment can be larger than page size,

Hmm ... aren't max_segment_size values that are larger than the page size supported since day one of the Linux kernel? Or are you perhaps referring to Ming's multi-page bvec patch series that was merged about six years ago?

Try aiming high for a single 2 MiB for a single IO on x86_64 on NVMe, that is currently not possible. At the max 128 NVMe number of DMA segments, and we have 4 KiB per DMA segment, for a 512 KiB IO limit. Should multi-page bvec enable to lift this?

Luis

#9 lore raw
Quoted 4 lines · Luis Chamberlain

Try aiming high for a single 2 MiB for a single IO on x86_64 on NVMe, that is currently not possible. At the max 128 NVMe number of DMA segments, and we have 4 KiB per DMA segment, for a 512 KiB IO limit. Should multi-page bvec enable to lift this?

4 KiB per DMA segment for NVMe? I think that the DMA segment size limit for PRP and SGL modes is much larger than 4 KiB. See also the description of the CC.MPS parameter and PRP Lists in the NVMe base
specification. From a system with an NVMe controller:

$ cat /sys/block/nvme0n1/queue/max_segment_size
4294967295

Thanks,

Bart.

#10 lore raw
Quoted 8 lines · Bart Van Assche
Quoted 4 lines · Luis Chamberlain

Try aiming high for a single 2 MiB for a single IO on x86_64 on NVMe, that is currently not possible. At the max 128 NVMe number of DMA segments, and we have 4 KiB per DMA segment, for a 512 KiB IO limit. Should multi-page bvec enable to lift this?

4 KiB per DMA segment for NVMe? I think that the DMA segment size limit for PRP and SGL modes is much larger than 4 KiB.

Of course in terms of device capability, I was referring to what we support in software.

Quoted 6 lines

See also the
description of the CC.MPS parameter and PRP Lists in the NVMe base specification. From a system with an NVMe controller:

$ cat /sys/block/nvme0n1/queue/max_segment_size
4294967295

To enable such things you need either huge pages or large folios, and for a deterministic large dma segment you can use min-order, my experimentation shows we have a bit of work to lever really large dma segments above 64k.

Luis

#11 lore raw
In reply to Luis Chamberlain
Quoted 9 lines · Luis Chamberlain
Quoted 7 lines · Bart Van Assche
Quoted 2 lines · Luis Chamberlain

While that addresses a smaller segment then page size this still leaves open the question of if a dma segment can be larger than page size,

Hmm ... aren't max_segment_size values that are larger than the page size supported since day one of the Linux kernel? Or are you perhaps referring to Ming's multi-page bvec patch series that was merged about six years ago?

Quoted 4 lines

Try aiming high for a single 2 MiB for a single IO on x86_64 on NVMe, that is currently not possible. At the max 128 NVMe number of DMA segments, and we have 4 KiB per DMA segment, for a 512 KiB IO limit. Should multi-page bvec enable to lift this?

You need huge pages to guarantee you can reach those transfer sizes in a single command. Otherwise you need to get lucky with larger contiguous folios, which can certainly happen but it's just not as desterministic.

There are many nvme devices that absoulutely require 4KiB as the largest segment, but the driver says it can handle much larger segments anyway. It's trivial for the driver to split a large bio vec into a bunch of smaller device specific descriptors.

#12 lore raw
Quoted 10 lines · Keith Busch
Quoted 9 lines · Luis Chamberlain
Quoted 7 lines · Bart Van Assche
Quoted 2 lines · Luis Chamberlain

While that addresses a smaller segment then page size this still leaves open the question of if a dma segment can be larger than page size,

Hmm ... aren't max_segment_size values that are larger than the page size supported since day one of the Linux kernel? Or are you perhaps referring to Ming's multi-page bvec patch series that was merged about six years ago?

Quoted 8 lines
Quoted 4 lines

Try aiming high for a single 2 MiB for a single IO on x86_64 on NVMe, that is currently not possible. At the max 128 NVMe number of DMA segments, and we have 4 KiB per DMA segment, for a 512 KiB IO limit. Should multi-page bvec enable to lift this?

You need huge pages to guarantee you can reach those transfer sizes in a single command. Otherwise you need to get lucky with larger contiguous folios, which can certainly happen but it's just not as desterministic.

Now with min-order support, it is deterministic.

Luis

author
#13 lore raw
In reply to Bart Van Assche
Quoted 9 lines · Bart Van Assche
Quoted 1 line · Ming Lei

But why does DMA segment size have to be >= PAGE_SIZE(4KB, 64KB)?

From the description of patch 5/8 of my patch series: "If the segment size is smaller than the page size there may be multiple segments per bvec even if a bvec only contains a single page." The current block layer implementation is based on the assumption that a single page fits in a single DMA segment. Please take a look at patch 5/8 of my patch series.

OK, I guess you agree it is one block layer constraint now, which need to be relaxed for both big logical block size and >4k PAGE_SIZE.

Yes, your patch 5/8 is still needed.

Quoted 7 lines
Quoted 3 lines

From the link, you have storage controllers with DMA segment size which is less than 4K, which may never get supported by linux kernel.

As mentioned in the cover letter of that patch series, I came up with that patch series to support a DMA controller with max_segment_size of 4 KiB on a system with a PAGE_SIZE of 16 KiB.

Probably the conception of subpage need to avoid, because PAGE_SIZE is config option, and not see strong reason to couple with fixed & readable hardware properties with configurable kernel page size.

Thanks,
Ming