IB/hfi1: fix the PIO_CRED credit-return mmap

The PIO_CRED case of hfi1_file_mmap() hands user space the credit-return page for its send context. On a two-socket host with a translating IOMMU it instead returns a page mapped from a frame above MAXPHYADDR, and the first user read takes "Corrupted page table" / "Oops: Bad pagetable". Correcting only the offset arithmetic replaces the Oops with a silent wrong-page mapping, so every transfer that uses send PIO hangs instead.

The failure is intermittent: it depends on which of the credit-return pages the context's entry lands on, which follows the hardware send context index and so varies from boot to boot.

Patch 1 fixes the node used to resolve the credit-return buffer, which has been wrong since the driver was merged. Patch 2 fixes the byte offset applied to a typed pointer and the use of dma_mmap_coherent(), both introduced when this case was converted to the DMA API. They are separate patches because they have different Fixes: tags, but only the two together make the mapping correct: patch 2 without patch 1 turns the hang into an -ENXIO from iommu_dma_mmap()'s bounds check, because the cross-node offset exceeds the buffer.

Found while debugging psm2_ep_open() Oopsing a Dell T7610 (Xeon E5-2650 v2, Intel IOMMU in DMA-FQ mode) talking to a Threadripper PRO 3995WX, both Omni-Path 100. With the series applied to both hosts, psm2_ep_open() succeeds, send PIO, send DMA and the default mixed mode all work, and MPI over the PSM2 MTL sustains 95 Gb/s. Forcing send PIO only (PSM2_SDMA=0), which hung indefinitely before, now completes.

Based on rdma/for-rc at 31b7c700670830a0e8a4cdcd451c88a13cc5dc48.

Shuhei Takeshita (2):
IB/hfi1: Resolve the credit-return buffer through the send context's
node
IB/hfi1: Fix the PIO_CRED credit-return mmap

 drivers/infiniband/hw/hfi1/file_ops.c | 23 +++++++++++++++++------
 1 file changed, 17 insertions(+), 6 deletions(-)

2.43.0