Skip to contentSkip to conversation

RDMA/core: Fix best page size finding when it can cross SG entries

PATCH Applied Michael Margolin 3 participants 16 messages Feb 9, 2025 – Feb 17, 2025 InfiniBand

v1 of 2 Applied Latest is v2 →

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 RDMA/core: Fix best page size finding when it can cross SG entries

A single scatter-gather entry is limited by a 32 bits "length" field that is practically 4GB - PAGE_SIZE. This means that even when the memory is physically contiguous, we might need more than one entry to represent it. Additionally when using dmabuf, the sg_table might be originated outside the subsystem and optimized for other needs.

For instance an SGT of 16GB GPU continuous memory might look like this:
(a real life example)

dma_address 34401400000, length fffff000
dma_address 345013ff000, length fffff000
dma_address 346013fe000, length fffff000
dma_address 347013fd000, length fffff000
dma_address 348013fc000, length 4000

Since ib_umem_find_best_pgsz works within SG entries, in the above case we will result with the worst possible 4KB page size.

Fix this by taking into consideration only the alignment of addresses of real discontinuity points rather than treating SG entries as such, and adjust the page iterator to correctly handle cross SG entry pages.

There is currently an assumption that drivers do not ask for pages bigger than maximal DMA size supported by their devices.

---
 drivers/infiniband/core/umem.c  | 34 +++++++++++++++++++++++----------
 drivers/infiniband/core/verbs.c | 11 ++++++-----
 2 files changed, 30 insertions(+), 15 deletions(-)
drivers/infiniband/core/umem.c +24 -10
index 07c571c7b699..e7e428369159 100644
@@ -80,8 +80,10 @@ unsigned long ib_umem_find_best_pgsz(struct ib_umem *umem,
 				     unsigned long pgsz_bitmap,
 				     unsigned long virt)
 {
-	struct scatterlist *sg;
+	unsigned long curr_len = 0;
+	dma_addr_t curr_base = ~0;
 	unsigned long va, pgoff;
+	struct scatterlist *sg;
 	dma_addr_t mask;
 	int i;
 
@@ -107,17 +109,29 @@ unsigned long ib_umem_find_best_pgsz(struct ib_umem *umem,
 	pgoff = umem->address & ~PAGE_MASK;
 
 	for_each_sgtable_dma_sg(&umem->sgt_append.sgt, sg, i) {
-		/* Walk SGL and reduce max page size if VA/PA bits differ
-		 * for any address.
+		/* If the current entry is physically contiguous with the previous
+		 * one, no need to take its start addresses into consideration.
 		 */
-		mask |= (sg_dma_address(sg) + pgoff) ^ va;
+		if (curr_base + curr_len != sg_dma_address(sg)) {
+
+			curr_base = sg_dma_address(sg);
+			curr_len = 0;
+
+			/* Reduce max page size if VA/PA bits differ */
+			mask |= (curr_base + pgoff) ^ va;
+
+			/* The alignment of any VA matching a discontinuity point
+			* in the physical memory sets the maximum possible page
+			* size as this must be a starting point of a new page that
+			* needs to be aligned.
+			*/
+			if (i != 0)
+				mask |= va;
+		}
+
+		curr_len += sg_dma_len(sg);
 		va += sg_dma_len(sg) - pgoff;
-		/* Except for the last entry, the ending iova alignment sets
-		 * the maximum possible page size as the low bits of the iova
-		 * must be zero when starting the next chunk.
-		 */
-		if (i != (umem->sgt_append.sgt.nents - 1))
-			mask |= va;
+
 		pgoff = 0;
 	}
 
drivers/infiniband/core/verbs.c +6 -5
index 473ee0831307..dc40001072a5 100644
@@ -3109,22 +3109,23 @@ EXPORT_SYMBOL(__rdma_block_iter_start);
 bool __rdma_block_iter_next(struct ib_block_iter *biter)
 {
 	unsigned int block_offset;
-	unsigned int sg_delta;
+	unsigned int delta;
 
 	if (!biter->__sg_nents || !biter->__sg)
 		return false;
 
 	biter->__dma_addr = sg_dma_address(biter->__sg) + biter->__sg_advance;
 	block_offset = biter->__dma_addr & (BIT_ULL(biter->__pg_bit) - 1);
-	sg_delta = BIT_ULL(biter->__pg_bit) - block_offset;
+	delta = BIT_ULL(biter->__pg_bit) - block_offset;
 
-	if (sg_dma_len(biter->__sg) - biter->__sg_advance > sg_delta) {
-		biter->__sg_advance += sg_delta;
-	} else {
+	while (biter->__sg_nents && biter->__sg &&
+	       sg_dma_len(biter->__sg) - biter->__sg_advance <= delta) {
+		delta -= sg_dma_len(biter->__sg) - biter->__sg_advance;
 		biter->__sg_advance = 0;
 		biter->__sg = sg_next(biter->__sg);
 		biter->__sg_nents--;
 	}
+	biter->__sg_advance += delta;
 
 	return true;
 }
Signature · 1 line
2.47.1
#1 lore raw
Quoted 25 lines · Michael Margolin

A single scatter-gather entry is limited by a 32 bits "length" field that is practically 4GB - PAGE_SIZE. This means that even when the memory is physically contiguous, we might need more than one entry to represent it. Additionally when using dmabuf, the sg_table might be originated outside the subsystem and optimized for other needs.

For instance an SGT of 16GB GPU continuous memory might look like this:
(a real life example)

dma_address 34401400000, length fffff000
dma_address 345013ff000, length fffff000
dma_address 346013fe000, length fffff000
dma_address 347013fd000, length fffff000
dma_address 348013fc000, length 4000

Since ib_umem_find_best_pgsz works within SG entries, in the above case we will result with the worst possible 4KB page size.

Fix this by taking into consideration only the alignment of addresses of real discontinuity points rather than treating SG entries as such, and adjust the page iterator to correctly handle cross SG entry pages.

There is currently an assumption that drivers do not ask for pages bigger than maximal DMA size supported by their devices.

Quoted 7 lines
---
 drivers/infiniband/core/umem.c  | 34 +++++++++++++++++++++++----------
 drivers/infiniband/core/verbs.c | 11 ++++++-----
 2 files changed, 30 insertions(+), 15 deletions(-)

Applied with the following change to prevent arithmetic overflow.

drivers/infiniband/core/umem.c +1 -2
index e7e428369159..63a92d6cfbc2 100644
@@ -112,8 +112,7 @@ unsigned long ib_umem_find_best_pgsz(struct ib_umem *umem,
                /* If the current entry is physically contiguous with the previous
                 * one, no need to take its start addresses into consideration.
                 */
-               if (curr_base + curr_len != sg_dma_address(sg)) {
-
+               if (curr_base != sg_dma_address(sg) - curr_len) {
                        curr_base = sg_dma_address(sg);
                        curr_len = 0;
 
maintainer
#2 lore raw

drivers/infiniband/core/umem.c unsigned long ib_umem_find_best_pgsz(struct ib_umem *umem, · line 117

Quoted 13 lines · Leon Romanovsky
drivers/infiniband/core/umem.c +1 -2
index e7e428369159..63a92d6cfbc2 100644
@@ -112,8 +112,7 @@ unsigned long ib_umem_find_best_pgsz(struct ib_umem *umem,
                /* If the current entry is physically contiguous with the previous
                 * one, no need to take its start addresses into consideration.
                 */
-               if (curr_base + curr_len != sg_dma_address(sg)) {
-
+               if (curr_base != sg_dma_address(sg) - curr_len) {
                        curr_base = sg_dma_address(sg);
                        curr_len = 0;

I'm not sure about this, what ensures sg_dma_address() > curr_len?

curr_base + curr_len could also overflow, we've seen that AMD IOMMU sometimes uses the very high addresess already

Jason

#3 lore raw
Quoted 20 lines · Jason Gunthorpe

CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe.

drivers/infiniband/core/umem.c unsigned long ib_umem_find_best_pgsz(struct ib_umem *umem, · line 117

Quoted 13 lines · Leon Romanovsky
drivers/infiniband/core/umem.c +1 -2
index e7e428369159..63a92d6cfbc2 100644
@@ -112,8 +112,7 @@ unsigned long ib_umem_find_best_pgsz(struct ib_umem *umem,
                 /* If the current entry is physically contiguous with the previous
                  * one, no need to take its start addresses into consideration.
                  */
-               if (curr_base + curr_len != sg_dma_address(sg)) {
-
+               if (curr_base != sg_dma_address(sg) - curr_len) {
                         curr_base = sg_dma_address(sg);
                         curr_len = 0;

I'm not sure about this, what ensures sg_dma_address() > curr_len?

Quoted 2 lines

curr_base + curr_len could also overflow, we've seen that AMD IOMMU sometimes uses the very high addresess already

I think the only case we care about where curr_base + curr_len can overflow is when next sg_dma_address() == 0.

But maybe we should just add an explicit check:

- if (curr_base + curr_len != sg_dma_address(sg)) {
+ if (curr_base + curr_len < curr_base ||

+                   curr_base + curr_len != sg_dma_address(sg)) {
                        curr_base = sg_dma_address(sg);
                        curr_len = 0;

Michael

maintainer
#4 lore raw
Quoted 30 lines · Michael
Quoted 20 lines · Jason Gunthorpe

CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe.

drivers/infiniband/core/umem.c unsigned long ib_umem_find_best_pgsz(struct ib_umem *umem, · line 117

Quoted 13 lines · Leon Romanovsky
drivers/infiniband/core/umem.c +1 -2
index e7e428369159..63a92d6cfbc2 100644
@@ -112,8 +112,7 @@ unsigned long ib_umem_find_best_pgsz(struct ib_umem *umem,
                 /* If the current entry is physically contiguous with the previous
                  * one, no need to take its start addresses into consideration.
                  */
-               if (curr_base + curr_len != sg_dma_address(sg)) {
-
+               if (curr_base != sg_dma_address(sg) - curr_len) {
                         curr_base = sg_dma_address(sg);
                         curr_len = 0;

I'm not sure about this, what ensures sg_dma_address() > curr_len?

Quoted 2 lines

curr_base + curr_len could also overflow, we've seen that AMD IOMMU sometimes uses the very high addresess already

I think the only case we care about where curr_base + curr_len can overflow is when next sg_dma_address() == 0.

But maybe we should just add an explicit check:

drivers/infiniband/core/umem.c unsigned long ib_umem_find_best_pgsz(struct ib_umem *umem, · line 117

Quoted 5 lines
drivers/infiniband/core/umem.c +2 -1
-               if (curr_base + curr_len != sg_dma_address(sg)) {
+               if (curr_base + curr_len < curr_base ||
+                   curr_base + curr_len != sg_dma_address(sg)) {
                        curr_base = sg_dma_address(sg);
                        curr_len = 0;

Ugh

I wonder if we should try to make a overflow.h helper for these kinds of problems.

/* Check if a + n == b, failing if a+n overflows */ check_consecutive(a, n, b)

?

It is a fairly common problem

I suggest to take the patch as it originally was and try to propose the above helper?

Jason

#5 lore raw
Quoted 6 lines · Jason Gunthorpe

I wonder if we should try to make a overflow.h helper for these kinds of problems.

/* Check if a + n == b, failing if a+n overflows */ check_consecutive(a, n, b)

Quoted 6 lines

?

It is a fairly common problem

I suggest to take the patch as it originally was and try to propose the above helper?

Sounds reasonable, I can propose such check helpers.

Michael

#6 lore raw
In reply to Jason Gunthorpe
Quoted 45 lines · Jason Gunthorpe
Quoted 30 lines · Michael
Quoted 20 lines · Jason Gunthorpe

CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe.

drivers/infiniband/core/umem.c unsigned long ib_umem_find_best_pgsz(struct ib_umem *umem, · line 117

Quoted 13 lines · Leon Romanovsky
drivers/infiniband/core/umem.c +1 -2
index e7e428369159..63a92d6cfbc2 100644
@@ -112,8 +112,7 @@ unsigned long ib_umem_find_best_pgsz(struct ib_umem *umem,
                 /* If the current entry is physically contiguous with the previous
                  * one, no need to take its start addresses into consideration.
                  */
-               if (curr_base + curr_len != sg_dma_address(sg)) {
-
+               if (curr_base != sg_dma_address(sg) - curr_len) {
                         curr_base = sg_dma_address(sg);
                         curr_len = 0;

I'm not sure about this, what ensures sg_dma_address() > curr_len?

Quoted 2 lines

curr_base + curr_len could also overflow, we've seen that AMD IOMMU sometimes uses the very high addresess already

I think the only case we care about where curr_base + curr_len can overflow is when next sg_dma_address() == 0.

But maybe we should just add an explicit check:

drivers/infiniband/core/umem.c unsigned long ib_umem_find_best_pgsz(struct ib_umem *umem, · line 117

Quoted 5 lines
drivers/infiniband/core/umem.c +2 -1
-               if (curr_base + curr_len != sg_dma_address(sg)) {
+               if (curr_base + curr_len < curr_base ||
+                   curr_base + curr_len != sg_dma_address(sg)) {
                        curr_base = sg_dma_address(sg);
                        curr_len = 0;

Ugh

I wonder if we should try to make a overflow.h helper for these kinds of problems.

/* Check if a + n == b, failing if a+n overflows */ check_consecutive(a, n, b)

Quoted 6 lines

?

It is a fairly common problem

I suggest to take the patch as it originally was and try to propose the above helper?

My concern was with this line:
if (curr_base + curr_len != sg_dma_address(sg)) {

Initially curr_base is 0xFF.....FF and curr_len is 0.
So if this "if ..." is skipped (not possible but static checkers don't know), we will advance curr_len and curr_base + curr_len will overflow.

I don't want to take original patch.

Thanks

Quoted 2 lines

Jason

maintainer
#7 lore raw
Quoted 1 line · Leon Romanovsky

Initially curr_base is 0xFF.....FF and curr_len is 0.

curr base can't be so unaligned can it?

Quoted 4 lines

So if this "if ..." is skipped (not possible but static checkers don't know), we will advance curr_len and curr_base + curr_len will overflow.

I don't want to take original patch.

Subtracting is no better, it will just randomly fail for low dma addrs instead of high.

You need to call check_add_overflow()

Jason

#8 lore raw
Quoted 5 lines · Jason Gunthorpe
Quoted 1 line · Leon Romanovsky

Initially curr_base is 0xFF.....FF and curr_len is 0.

curr base can't be so unaligned can it?

It is only for first iteration where it is compared with sg_dma_address(), immediately after that it is overwritten.

Quoted 8 lines
Quoted 5 lines

So if this "if ..." is skipped (not possible but static checkers don't know), we will advance curr_len and curr_base + curr_len will overflow.

I don't want to take original patch.

Subtracting is no better, it will just randomly fail for low dma addrs instead of high.

Aren't sg_dma_address placed in increasing order? If not, whole if loop is not correct.
If yes, we won't see any failures.

Quoted 4 lines

You need to call check_add_overflow()

Jason

maintainer
#9 lore raw
Quoted 9 lines · Leon Romanovsky
Quoted 5 lines · Jason Gunthorpe
Quoted 1 line · Leon Romanovsky

Initially curr_base is 0xFF.....FF and curr_len is 0.

curr base can't be so unaligned can it?

It is only for first iteration where it is compared with sg_dma_address(), immediately after that it is overwritten.

But this is all working with inherently page aligned stuff, cur_base + len1 + len2 + len3 + len_n should be page aligned for interior segments..

Quoted 9 lines
Quoted 7 lines
Quoted 4 lines

So if this "if ..." is skipped (not possible but static checkers don't know), we will advance curr_len and curr_base + curr_len will overflow.

I don't want to take original patch.

Subtracting is no better, it will just randomly fail for low dma addrs instead of high.

Aren't sg_dma_address placed in increasing order?

Not necessarily, dma direct produces something random

Quoted 1 line

If not, whole if loop is not correct.

The point is to detect cases where they happen to be in order because they were split up due to the 32 bit limit in the sgl.

Jason

#10 lore raw
Quoted 22 lines · Jason Gunthorpe
Quoted 9 lines · Leon Romanovsky
Quoted 5 lines · Jason Gunthorpe
Quoted 1 line · Leon Romanovsky

Initially curr_base is 0xFF.....FF and curr_len is 0.

curr base can't be so unaligned can it?

It is only for first iteration where it is compared with sg_dma_address(), immediately after that it is overwritten.

But this is all working with inherently page aligned stuff, cur_base + len1 + len2 + len3 + len_n should be page aligned for interior segments..

Quoted 8 lines
Quoted 7 lines
Quoted 4 lines

So if this "if ..." is skipped (not possible but static checkers don't know), we will advance curr_len and curr_base + curr_len will overflow.

I don't want to take original patch.

Subtracting is no better, it will just randomly fail for low dma addrs instead of high.

Quoted 8 lines
Quoted 1 line

Aren't sg_dma_address placed in increasing order?

Not necessarily, dma direct produces something random

Quoted 1 line

If not, whole if loop is not correct.

The point is to detect cases where they happen to be in order because they were split up due to the 32 bit limit in the sgl.

There is no promise that this split will create consecutive SG entries and iterator will fetch them in that order too.

This discussion leads me to the understanding that we need to drop the patch.

Thanks

Quoted 2 lines

Jason

maintainer
#11 lore raw
Quoted 5 lines · Leon Romanovsky
Quoted 2 lines

The point is to detect cases where they happen to be in order because they were split up due to the 32 bit limit in the sgl.

There is no promise that this split will create consecutive SG entries and iterator will fetch them in that order too.

Yes, that is guarenteed, if the memory is contiguous and split then the SG entries have to be consecutive and the sg iterator has to go in order.

Otherwise the DMA transfer would be scrambled.

In normal cases we except the SGL to already be fully aggregated so all SGL boundaries also have a dma_addr discontinuity. This special case of exceeding the 32 bit limit is the only time we should have consecutive sgls that are dma_addr contiguous.

Jason

#12 lore raw
In reply to Jason Gunthorpe
Quoted 6 lines · Jason Gunthorpe
Quoted 5 lines · Leon Romanovsky
Quoted 4 lines · Jason Gunthorpe
Quoted 1 line · Leon Romanovsky

Initially curr_base is 0xFF.....FF and curr_len is 0.

Quoted 7 lines
Quoted 4 lines
Quoted 1 line

curr base can't be so unaligned can it?

It is only for first iteration where it is compared with sg_dma_address(), immediately after that it is overwritten.

But this is all working with inherently page aligned stuff, cur_base + len1 + len2 + len3 + len_n should be page aligned for interior segments..

This is unknown to static code analyze tools. I'm not concerned about logical change, but about possible static code analyze failures.

Thanks

#13 lore raw
Quoted 11 lines · Leon Romanovsky

CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe.

Quoted 6 lines · Jason Gunthorpe
Quoted 5 lines · Leon Romanovsky
Quoted 4 lines · Jason Gunthorpe
Quoted 1 line · Leon Romanovsky

Initially curr_base is 0xFF.....FF and curr_len is 0.

curr base can't be so unaligned can it?

Quoted 8 lines
Quoted 4 lines
Quoted 2 lines

It is only for first iteration where it is compared with sg_dma_address(), immediately after that it is overwritten.

But this is all working with inherently page aligned stuff, cur_base + len1 + len2 + len3 + len_n should be page aligned for interior segments..

This is unknown to static code analyze tools. I'm not concerned about logical change, but about possible static code analyze failures.

Thanks

OK, I'll edit the patch to make sure it doesn't produce any SA failures.

Michael

#14 lore raw
In reply to Michael Margolin
Quoted 10 lines · Michael Margolin

A single scatter-gather entry is limited by a 32 bits "length" field that is practically 4GB - PAGE_SIZE. This means that even when the memory is physically contiguous, we might need more than one entry to represent it. Additionally when using dmabuf, the sg_table might be originated outside the subsystem and optimized for other needs.

For instance an SGT of 16GB GPU continuous memory might look like this:
(a real life example)

[...]

Applied, thanks!

[1/1] RDMA/core: Fix best page size finding when it can cross SG entries

      https://git.kernel.org/rdma/rdma/c/a4b57de5dfef29

Best regards,

Signature · 1 line
Leon Romanovsky <[email protected]>
#15 lore raw
Quoted 11 lines · Leon Romanovsky
Quoted 9 lines · Michael Margolin

A single scatter-gather entry is limited by a 32 bits "length" field that is practically 4GB - PAGE_SIZE. This means that even when the memory is physically contiguous, we might need more than one entry to represent it. Additionally when using dmabuf, the sg_table might be originated outside the subsystem and optimized for other needs.

For instance an SGT of 16GB GPU continuous memory might look like this:
(a real life example)

Quoted 6 lines
Quoted 1 line

[...]

Applied, thanks!

[1/1] RDMA/core: Fix best page size finding when it can cross SG entries

      https://git.kernel.org/rdma/rdma/c/a4b57de5dfef29

I dropped this patch for now.

Thanks

Quoted 6 lines

Best regards,

Signature · 1 line
Leon Romanovsky <[email protected]>