RDMA/core: Fix best page size finding when it can cross SG entries
PATCH 1/1 RDMA/core: Fix best page size finding when it can cross SG entries
A single scatter-gather entry is limited by a 32 bits "length" field that is practically 4GB - PAGE_SIZE. This means that even when the memory is physically contiguous, we might need more than one entry to represent it. Additionally when using dmabuf, the sg_table might be originated outside the subsystem and optimized for other needs.
For instance an SGT of 16GB GPU continuous memory might look like this:
(a real life example)
dma_address 34401400000, length fffff000dma_address 345013ff000, length fffff000dma_address 346013fe000, length fffff000dma_address 347013fd000, length fffff000dma_address 348013fc000, length 4000
Since ib_umem_find_best_pgsz works within SG entries, in the above case we will result with the worst possible 4KB page size.
Fix this by taking into consideration only the alignment of addresses of real discontinuity points rather than treating SG entries as such, and adjust the page iterator to correctly handle cross SG entry pages.
There is currently an assumption that drivers do not ask for pages bigger than maximal DMA size supported by their devices.
--- drivers/infiniband/core/umem.c | 34 +++++++++++++++++++++++---------- drivers/infiniband/core/verbs.c | 11 ++++++----- 2 files changed, 30 insertions(+), 15 deletions(-)
drivers/infiniband/core/umem.c +24 -10
@@ -80,8 +80,10 @@ unsigned long ib_umem_find_best_pgsz(struct ib_umem *umem, unsigned long pgsz_bitmap, unsigned long virt) { -struct scatterlist *sg; +unsigned long curr_len =0 ; +dma_addr_t curr_base = ~0 ;unsigned long va, pgoff; +struct scatterlist *sg;dma_addr_t mask;int i;
@@ -107,17 +109,29 @@ unsigned long ib_umem_find_best_pgsz(struct ib_umem *umem, pgoff = umem->address & ~PAGE_MASK ;for_each_sgtable_dma_sg (&umem->sgt_append.sgt, sg, i) { -/* Walk SGL and reduce max page size if VA/PA bits differ -* for any address. +/* If the current entry is physically contiguous with the previous +* one, no need to take its start addresses into consideration. */ - mask |= (sg_dma_address (sg) + pgoff) ^ va; +if (curr_base + curr_len !=sg_dma_address (sg)) { + + curr_base =sg_dma_address (sg); + curr_len =0 ; + +/* Reduce max page size if VA/PA bits differ */ + mask |= (curr_base + pgoff) ^ va; + +/* The alignment of any VA matching a discontinuity point +* in the physical memory sets the maximum possible page +* size as this must be a starting point of a new page that +* needs to be aligned. +*/ +if (i !=0 ) + mask |= va; + } + + curr_len +=sg_dma_len (sg); va +=sg_dma_len (sg) - pgoff; -/* Except for the last entry, the ending iova alignment sets -* the maximum possible page size as the low bits of the iova -* must be zero when starting the next chunk. -*/ -if (i != (umem->sgt_append.sgt.nents -1 )) - mask |= va; + pgoff =0 ; }
drivers/infiniband/core/verbs.c +6 -5
@@ -3109,22 +3109,23 @@ EXPORT_SYMBOL(__rdma_block_iter_start);bool __rdma_block_iter_next (struct ib_block_iter *biter) {unsigned int block_offset; -unsigned int sg_delta; +unsigned int delta;if (!biter->__sg_nents || !biter->__sg)return false; biter->__dma_addr =sg_dma_address (biter->__sg) + biter->__sg_advance; block_offset = biter->__dma_addr & (BIT_ULL (biter->__pg_bit) -1 ); - sg_delta =BIT_ULL (biter->__pg_bit) - block_offset; + delta =BIT_ULL (biter->__pg_bit) - block_offset; -if (sg_dma_len (biter->__sg) - biter->__sg_advance > sg_delta) { - biter->__sg_advance += sg_delta; - }else { +while (biter->__sg_nents && biter->__sg && +sg_dma_len (biter->__sg) - biter->__sg_advance <= delta) { + delta -=sg_dma_len (biter->__sg) - biter->__sg_advance; biter->__sg_advance =0 ; biter->__sg =sg_next (biter->__sg); biter->__sg_nents--; } + biter->__sg_advance += delta;return true; }
Signature · 1 line
2.47.1
Quoted 25 lines · Michael Margolin
A single scatter-gather entry is limited by a 32 bits "length" field that is practically 4GB -
PAGE_SIZE. This means that even when the memory is physically contiguous, we might need more than one entry to represent it. Additionally when using dmabuf, thesg_tablemight be originated outside the subsystem and optimized for other needs.For instance an SGT of 16GB GPU continuous memory might look like this:
(a real life example)
dma_address34401400000, lengthfffff000dma_address345013ff000, lengthfffff000dma_address346013fe000, lengthfffff000dma_address347013fd000, lengthfffff000dma_address348013fc000, length 4000Since
ib_umem_find_best_pgszworks within SG entries, in the above case we will result with the worst possible 4KB page size.Fix this by taking into consideration only the alignment of addresses of real discontinuity points rather than treating SG entries as such, and adjust the page iterator to correctly handle cross SG entry pages.
There is currently an assumption that drivers do not ask for pages bigger than maximal DMA size supported by their devices.
Quoted 7 lines
Reviewed-by Firas Jahjah <[email protected]>Reviewed-by Yonatan Nachum <[email protected]>Signed-off-by Michael Margolin <[email protected]>--- drivers/infiniband/core/umem.c | 34 +++++++++++++++++++++++---------- drivers/infiniband/core/verbs.c | 11 ++++++----- 2 files changed, 30 insertions(+), 15 deletions(-)
Applied with the following change to prevent arithmetic overflow.
drivers/infiniband/core/umem.c +1 -2
@@ -112,8 +112,7 @@ unsigned long ib_umem_find_best_pgsz(struct ib_umem *umem,/* If the current entry is physically contiguous with the previous * one, no need to take its start addresses into consideration. */ -if (curr_base + curr_len !=sg_dma_address (sg)) { - +if (curr_base !=sg_dma_address (sg) - curr_len) { curr_base =sg_dma_address (sg); curr_len =0 ;
Jason Gunthorpe
maintainerdrivers/infiniband/core/umem.c unsigned long ib_umem_find_best_pgsz(struct ib_umem *umem, · line 117
Quoted 13 lines · Leon Romanovsky
drivers/infiniband/core/umem.c +1
-2@@ -112,8 +112,7 @@ unsigned long ib_umem_find_best_pgsz(struct ib_umem *umem,/* If the current entry is physically contiguous with the previous * one, no need to take its start addresses into consideration. */ -if (curr_base + curr_len !=sg_dma_address (sg)) { - +if (curr_base !=sg_dma_address (sg) - curr_len) { curr_base =sg_dma_address (sg); curr_len =0 ;
I'm not sure about this, what ensures sg_dma_address() > curr_len?
curr_base + curr_len could also overflow, we've seen that AMD IOMMU sometimes uses the very high addresess already
Jason
Quoted 20 lines · Jason Gunthorpe
CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe.
drivers/infiniband/core/umem.c unsigned long ib_umem_find_best_pgsz(struct ib_umem *umem, · line 117
Quoted 13 lines · Leon Romanovsky
drivers/infiniband/core/umem.c +1
-2@@ -112,8 +112,7 @@ unsigned long ib_umem_find_best_pgsz(struct ib_umem *umem,/* If the current entry is physically contiguous with the previous * one, no need to take its start addresses into consideration. */ -if (curr_base + curr_len !=sg_dma_address (sg)) { - +if (curr_base !=sg_dma_address (sg) - curr_len) { curr_base =sg_dma_address (sg); curr_len =0 ;I'm not sure about this, what ensures
sg_dma_address()>curr_len?
Quoted 2 lines
curr_base+curr_lencould also overflow, we've seen that AMD IOMMU sometimes uses the very high addresess already
I think the only case we care about where curr_base + curr_len can overflow is when next sg_dma_address() == 0.
But maybe we should just add an explicit check:
- if (curr_base + curr_len != sg_dma_address(sg)) {
+ if (curr_base + curr_len < curr_base ||
+ curr_base + curr_len != sg_dma_address(sg)) {
curr_base = sg_dma_address(sg);
curr_len = 0;Michael
Jason Gunthorpe
maintainerQuoted 30 lines · Michael
Quoted 20 lines · Jason Gunthorpe
CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe.
drivers/infiniband/core/umem.c unsigned long ib_umem_find_best_pgsz(struct ib_umem *umem, · line 117
Quoted 13 lines · Leon Romanovsky
drivers/infiniband/core/umem.c +1
-2@@ -112,8 +112,7 @@ unsigned long ib_umem_find_best_pgsz(struct ib_umem *umem,/* If the current entry is physically contiguous with the previous * one, no need to take its start addresses into consideration. */ -if (curr_base + curr_len !=sg_dma_address (sg)) { - +if (curr_base !=sg_dma_address (sg) - curr_len) { curr_base =sg_dma_address (sg); curr_len =0 ;I'm not sure about this, what ensures
sg_dma_address()>curr_len?Quoted 2 lines
curr_base+curr_lencould also overflow, we've seen that AMD IOMMU sometimes uses the very high addresess alreadyI think the only case we care about where
curr_base+curr_lencan overflow is when nextsg_dma_address()== 0.But maybe we should just add an explicit check:
drivers/infiniband/core/umem.c unsigned long ib_umem_find_best_pgsz(struct ib_umem *umem, · line 117
Quoted 5 lines
drivers/infiniband/core/umem.c +2
-1-if (curr_base + curr_len !=sg_dma_address (sg)) { +if (curr_base + curr_len < curr_base || + curr_base + curr_len !=sg_dma_address (sg)) { curr_base =sg_dma_address (sg); curr_len =0 ;
Ugh
I wonder if we should try to make a overflow.h helper for these kinds of problems.
/* Check if a + n == b, failing if a+n overflows */ check_consecutive(a, n, b)
?
It is a fairly common problem
I suggest to take the patch as it originally was and try to propose the above helper?
Jason
Quoted 6 lines · Jason Gunthorpe
I wonder if we should try to make a
overflow.hhelper for these kinds of problems./* Check if a + n == b, failing if a+n overflows */
check_consecutive(a, n, b)
Quoted 6 lines
?
It is a fairly common problem
I suggest to take the patch as it originally was and try to propose the above helper?
Sounds reasonable, I can propose such check helpers.
Michael
Quoted 45 lines · Jason Gunthorpe
Quoted 30 lines · Michael
Quoted 20 lines · Jason Gunthorpe
CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe.
drivers/infiniband/core/umem.c unsigned long ib_umem_find_best_pgsz(struct ib_umem *umem, · line 117
Quoted 13 lines · Leon Romanovsky
drivers/infiniband/core/umem.c +1
-2@@ -112,8 +112,7 @@ unsigned long ib_umem_find_best_pgsz(struct ib_umem *umem,/* If the current entry is physically contiguous with the previous * one, no need to take its start addresses into consideration. */ -if (curr_base + curr_len !=sg_dma_address (sg)) { - +if (curr_base !=sg_dma_address (sg) - curr_len) { curr_base =sg_dma_address (sg); curr_len =0 ;I'm not sure about this, what ensures
sg_dma_address()>curr_len?Quoted 2 lines
curr_base+curr_lencould also overflow, we've seen that AMD IOMMU sometimes uses the very high addresess alreadyI think the only case we care about where
curr_base+curr_lencan overflow is when nextsg_dma_address()== 0.But maybe we should just add an explicit check:
drivers/infiniband/core/umem.c unsigned long ib_umem_find_best_pgsz(struct ib_umem *umem, · line 117
Quoted 5 lines
drivers/infiniband/core/umem.c +2
-1-if (curr_base + curr_len !=sg_dma_address (sg)) { +if (curr_base + curr_len < curr_base || + curr_base + curr_len !=sg_dma_address (sg)) { curr_base =sg_dma_address (sg); curr_len =0 ;Ugh
I wonder if we should try to make a
overflow.hhelper for these kinds of problems./* Check if a + n == b, failing if a+n overflows */
check_consecutive(a, n, b)
Quoted 6 lines
?
It is a fairly common problem
I suggest to take the patch as it originally was and try to propose the above helper?
My concern was with this line:
if (curr_base + curr_len != sg_dma_address(sg)) {
Initially curr_base is 0xFF.....FF and curr_len is 0.
So if this "if ..." is skipped (not possible but static checkers don't know), we will advance curr_len and curr_base + curr_len will overflow.
I don't want to take original patch.
Thanks
Quoted 2 lines
Jason
Jason Gunthorpe
maintainerQuoted 1 line · Leon Romanovsky
Initially
curr_baseis 0xFF.....FF andcurr_lenis 0.
curr base can't be so unaligned can it?
Quoted 4 lines
So if this "if ..." is skipped (not possible but static checkers don't know), we will advance
curr_lenandcurr_base+curr_lenwill overflow.I don't want to take original patch.
Subtracting is no better, it will just randomly fail for low dma addrs instead of high.
You need to call check_add_overflow()
Jason
Quoted 5 lines · Jason Gunthorpe
Quoted 1 line · Leon Romanovsky
Initially
curr_baseis 0xFF.....FF andcurr_lenis 0.curr base can't be so unaligned can it?
It is only for first iteration where it is compared with sg_dma_address(), immediately after that it is overwritten.
Quoted 8 lines
Quoted 5 lines
So if this "if ..." is skipped (not possible but static checkers don't know), we will advance
curr_lenandcurr_base+curr_lenwill overflow.I don't want to take original patch.
Subtracting is no better, it will just randomly fail for low dma addrs instead of high.
Aren't sg_dma_address placed in increasing order? If not, whole if loop is not correct.
If yes, we won't see any failures.
Quoted 4 lines
You need to call
check_add_overflow()Jason
Jason Gunthorpe
maintainerQuoted 9 lines · Leon Romanovsky
Quoted 5 lines · Jason Gunthorpe
Quoted 1 line · Leon Romanovsky
Initially
curr_baseis 0xFF.....FF andcurr_lenis 0.curr base can't be so unaligned can it?
It is only for first iteration where it is compared with
sg_dma_address(), immediately after that it is overwritten.
But this is all working with inherently page aligned stuff, cur_base + len1 + len2 + len3 + len_n should be page aligned for interior segments..
Quoted 9 lines
Quoted 7 lines
Quoted 4 lines
So if this "if ..." is skipped (not possible but static checkers don't know), we will advance
curr_lenandcurr_base+curr_lenwill overflow.I don't want to take original patch.
Subtracting is no better, it will just randomly fail for low dma addrs instead of high.
Aren't
sg_dma_addressplaced in increasing order?
Not necessarily, dma direct produces something random
Quoted 1 line
If not, whole if loop is not correct.
The point is to detect cases where they happen to be in order because they were split up due to the 32 bit limit in the sgl.
Jason
Quoted 22 lines · Jason Gunthorpe
Quoted 9 lines · Leon Romanovsky
Quoted 5 lines · Jason Gunthorpe
Quoted 1 line · Leon Romanovsky
Initially
curr_baseis 0xFF.....FF andcurr_lenis 0.curr base can't be so unaligned can it?
It is only for first iteration where it is compared with
sg_dma_address(), immediately after that it is overwritten.But this is all working with inherently page aligned stuff,
cur_base+ len1 + len2 + len3 +len_nshould be page aligned for interior segments..Quoted 8 lines
Quoted 7 lines
Quoted 4 lines
So if this "if ..." is skipped (not possible but static checkers don't know), we will advance
curr_lenandcurr_base+curr_lenwill overflow.I don't want to take original patch.
Subtracting is no better, it will just randomly fail for low dma addrs instead of high.
Quoted 8 lines
Quoted 1 line
Aren't
sg_dma_addressplaced in increasing order?Not necessarily, dma direct produces something random
Quoted 1 line
If not, whole if loop is not correct.
The point is to detect cases where they happen to be in order because they were split up due to the 32 bit limit in the sgl.
There is no promise that this split will create consecutive SG entries and iterator will fetch them in that order too.
This discussion leads me to the understanding that we need to drop the patch.
Thanks
Quoted 2 lines
Jason
Jason Gunthorpe
maintainerQuoted 5 lines · Leon Romanovsky
Quoted 2 lines
The point is to detect cases where they happen to be in order because they were split up due to the 32 bit limit in the sgl.
There is no promise that this split will create consecutive SG entries and iterator will fetch them in that order too.
Yes, that is guarenteed, if the memory is contiguous and split then the SG entries have to be consecutive and the sg iterator has to go in order.
Otherwise the DMA transfer would be scrambled.
In normal cases we except the SGL to already be fully aggregated so all SGL boundaries also have a dma_addr discontinuity. This special case of exceeding the 32 bit limit is the only time we should have consecutive sgls that are dma_addr contiguous.
Jason
Quoted 6 lines · Jason Gunthorpe
Quoted 5 lines · Leon Romanovsky
Quoted 4 lines · Jason Gunthorpe
Quoted 1 line · Leon Romanovsky
Initially
curr_baseis 0xFF.....FF andcurr_lenis 0.
Quoted 7 lines
Quoted 4 lines
Quoted 1 line
curr base can't be so unaligned can it?
It is only for first iteration where it is compared with
sg_dma_address(), immediately after that it is overwritten.But this is all working with inherently page aligned stuff,
cur_base+ len1 + len2 + len3 +len_nshould be page aligned for interior segments..
This is unknown to static code analyze tools. I'm not concerned about logical change, but about possible static code analyze failures.
Thanks
Quoted 11 lines · Leon Romanovsky
CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe.
Quoted 6 lines · Jason Gunthorpe
Quoted 5 lines · Leon Romanovsky
Quoted 4 lines · Jason Gunthorpe
Quoted 1 line · Leon Romanovsky
Initially
curr_baseis 0xFF.....FF andcurr_lenis 0.curr base can't be so unaligned can it?
Quoted 8 lines
Quoted 4 lines
Quoted 2 lines
It is only for first iteration where it is compared with
sg_dma_address(), immediately after that it is overwritten.But this is all working with inherently page aligned stuff,
cur_base+ len1 + len2 + len3 +len_nshould be page aligned for interior segments..This is unknown to static code analyze tools. I'm not concerned about logical change, but about possible static code analyze failures.
Thanks
OK, I'll edit the patch to make sure it doesn't produce any SA failures.
Michael
Leon Romanovsky
AppliedQuoted 10 lines · Michael Margolin
A single scatter-gather entry is limited by a 32 bits "length" field that is practically 4GB -
PAGE_SIZE. This means that even when the memory is physically contiguous, we might need more than one entry to represent it. Additionally when using dmabuf, thesg_tablemight be originated outside the subsystem and optimized for other needs.For instance an SGT of 16GB GPU continuous memory might look like this:
(a real life example)[...]
Applied, thanks!
[1/1] RDMA/core: Fix best page size finding when it can cross SG entries
https://git.kernel.org/rdma/rdma/c/a4b57de5dfef29
Best regards,
Signature · 1 line
Leon Romanovsky <[email protected]>
Quoted 11 lines · Leon Romanovsky
Quoted 9 lines · Michael Margolin
A single scatter-gather entry is limited by a 32 bits "length" field that is practically 4GB -
PAGE_SIZE. This means that even when the memory is physically contiguous, we might need more than one entry to represent it. Additionally when using dmabuf, thesg_tablemight be originated outside the subsystem and optimized for other needs.For instance an SGT of 16GB GPU continuous memory might look like this:
(a real life example)
Quoted 6 lines
Quoted 1 line
[...]
Applied, thanks!
[1/1] RDMA/core: Fix best page size finding when it can cross SG entries
https://git.kernel.org/rdma/rdma/c/a4b57de5dfef29
I dropped this patch for now.
Thanks
Quoted 6 lines
Best regards,
Signature · 1 line
Leon Romanovsky <[email protected]>