Skip to contentSkip to conversation

RDMA/irdma: Fix FRWR IRDMAQPSQ_PBLADDR field population

PATCH Applied Jacob Moroni 4 participants 5 messages Oct 15 – Oct 16

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

PATCH 1/1 RDMA/irdma: Fix FRWR IRDMAQPSQ_PBLADDR field population

Previously, the IRDMAQPSQ_PBLADDR FRWR WQE field was being populated using a mask that was right-shifted, whereas it should have been the address being right-shifted.

This ended up working fine in many cases because the lowermost 12 bits are always zero (the addresses are page aligned) and the uppermost 12 bits are usually zero too, but it could break for DMA addresses that are wider than 52 bits.

Fix this by right-shifting the address instead of the mask, similar to how it was done in the original i40iw driver.

Tested with NFS over RDMA with some added debug logs to ensure it was hitting both the direct (single page) path as well as multi page (< 512).

Fixes 3f49d6842569 ("RDMA/irdma: Implement HW Admin Queue OPs")
---
 drivers/infiniband/hw/irdma/ctrl.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
drivers/infiniband/hw/irdma/ctrl.c +1 -1
index 0db73fd7d01b..b62bf4d11829 100644
@@ -1727,7 +1727,7 @@ int irdma_sc_mr_fast_register(struct irdma_sc_qp *qp,
 			 info->first_pm_pbl_index >> 16);
 	set_64bit_val(wqe, 8,
 		      FIELD_PREP(IRDMAQPSQ_FIRSTPMPBLIDXHI, temp) |
-		      FIELD_PREP(IRDMAQPSQ_PBLADDR >> IRDMA_HW_PAGE_SHIFT, info->reg_addr_pa));
+		      FIELD_PREP(IRDMAQPSQ_PBLADDR, info->reg_addr_pa >> IRDMA_HW_PAGE_SHIFT));
 	set_64bit_val(wqe, 16,
 		      info->total_len |
 		      FIELD_PREP(IRDMAQPSQ_FIRSTPMPBLIDXLO, info->first_pm_pbl_index));
Signature · 1 line
2.56.0.385.gd3acb90ef8-goog
#1 lore raw
Sashiko has reviewed this patch and found no issues. It looks great!
Quoted 12 lines

Previously, the IRDMAQPSQ_PBLADDR FRWR WQE field was being populated using a mask that was right-shifted, whereas it should have been the address being right-shifted.

This ended up working fine in many cases because the lowermost 12 bits are always zero (the addresses are page aligned) and the uppermost 12 bits are usually zero too, but it could break for DMA addresses that are wider than 52 bits.

Fix this by right-shifting the address instead of the mask, similar to how it was done in the original i40iw driver.

Quoted 6 lines

Tested with NFS over RDMA with some added debug logs to ensure it was hitting both the direct (single page) path as well as multi page (< 512).

Fixes 3f49d6842569 ("RDMA/irdma: Implement HW Admin Queue OPs")

Sashiko has reviewed this patch and found no issues. It looks great!

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=1
#2 lore raw
In reply to Jacob Moroni
Quoted 10 lines · Jacob Moroni

Previously, the IRDMAQPSQ_PBLADDR FRWR WQE field was being populated using a mask that was right-shifted, whereas it should have been the address being right-shifted.

This ended up working fine in many cases because the lowermost 12 bits are always zero (the addresses are page aligned) and the uppermost 12 bits are usually zero too, but it could break for DMA addresses that are wider than 52 bits.

[...]

Applied, thanks!

[1/1] RDMA/irdma: Fix FRWR IRDMAQPSQ_PBLADDR field population

      https://git.kernel.org/rdma/rdma/c/59df57262e6fa9

Best regards,

Signature · 1 line
Leon Romanovsky <[email protected]>
#3 lore raw

Am 15.10.26 um 20:44 schrieb Leon Romanovsky:

Quoted 11 lines · Jacob Moroni
Quoted 9 lines

Previously, the IRDMAQPSQ_PBLADDR FRWR WQE field was being populated using a mask that was right-shifted, whereas it should have been the address being right-shifted.

This ended up working fine in many cases because the lowermost 12 bits are always zero (the addresses are page aligned) and the uppermost 12 bits are usually zero too, but it could break for DMA addresses that are wider than 52 bits.

Quoted 6 lines
Quoted 1 line

[...]

Applied, thanks!

[1/1] RDMA/irdma: Fix FRWR IRDMAQPSQ_PBLADDR field population

       https://git.kernel.org/rdma/rdma/c/59df57262e6fa9

My initial tests with this resolved the smbdirect related data corruption.

So it replaces this with success:
https://lore.kernel.org/linux-rdma/[email protected]/

Thanks!
metze

#4 lore raw

Thanks for verifying. There's still another fix for PBLE_LEVEL_2 on the way as well.

I should have it out shortly.

Thanks,
Jake