Allow relaxed ordering read in VFs and VMs

From: Avihai Horon <[email protected]>

pcie_relaxed_ordering_enabled() check was added to avoid a syndrome when creating a MKey with relaxed ordering (RO) enabled when the driver's relaxed_ordering_{read,write} HCA capabilities are out of sync with FW.

While this can happen with relaxed_ordering_read, it can't happen with relaxed_ordering_write as it's set if the device supports RO write, regardless of RO in PCI config space, and thus can't change during runtime.

Therefore, drop the pcie_relaxed_ordering_enabled() check for relaxed_ordering_write while keeping it for relaxed_ordering_read. Doing so will also allow the usage of RO write in VFs and VMs (where RO in PCI config space is not reported/emulated properly).

Signed-off-by Avihai Horon <[email protected]>

Reviewed-by Shay Drory <[email protected]>

Signed-off-by Leon Romanovsky <[email protected]>

---

From: Avihai Horon <[email protected]>

relaxed_ordering_read HCA capability is set if both the device supports relaxed ordering (RO) read and RO is set in PCI config space.

RO in PCI config space can change during runtime. This will change the value of relaxed_ordering_read HCA capability in FW, but the driver will not see it since it queries the capabilities only once.

This can lead to the following scenario:
1. RO in PCI config space is enabled.
2. User creates MKey without RO.
3. RO in PCI config space is disabled.
As a result, relaxed_ordering_read HCA capability is turned off in FW but remains on in driver copy of the capabilities.
4. User requests to reconfig the MKey with RO via UMR.
5. Driver will try to reconfig the MKey with RO read although it shouldn't (as relaxed_ordering_read HCA capability is really off).

To fix this, check pcie_relaxed_ordering_enabled() before setting RO read in UMR.

Fixes 896ec9735336 ("RDMA/mlx5: Set mkey relaxed ordering by UMR with ConnectX-7")

Signed-off-by Avihai Horon <[email protected]>

Reviewed-by Shay Drory <[email protected]>

Signed-off-by Leon Romanovsky <[email protected]>

---

From: Avihai Horon <[email protected]>

Rename existing HCA capability relaxed_ordering_read to relaxed_ordering_read_pci_enabled. This is in accordance with recent PRM change to better describe the capability, as it's set only if both the device supports relaxed ordering (RO) read and RO is enabled in PCI config space.

In addition, add new HCA capability relaxed_ordering_read which is set if the device supports RO read, regardless of RO in PCI config space. This will be used in the following patch to allow RO in VFs and VMs.

Signed-off-by Avihai Horon <[email protected]>

Reviewed-by Shay Drory <[email protected]>

Signed-off-by Leon Romanovsky <[email protected]>

---

From: Avihai Horon <[email protected]>

According to PCIe spec, Enable Relaxed Ordering value in the VF's PCI config space is wired to 0 and PF relaxed ordering (RO) setting should be applied to the VF. In QEMU (and maybe others), when assigning VFs, the RO bit in PCI config space is not emulated properly and is always set to 0.

Therefore, pcie_relaxed_ordering_enabled() always returns 0 for VFs and VMs and thus MKeys can't be created with RO read even if the PF supports it.

pcie_relaxed_ordering_enabled() check was added to avoid a syndrome when creating a MKey with relaxed ordering (RO) enabled when the driver's relaxed_ordering_read_pci_enabled HCA capability is out of sync with FW. With the new relaxed_ordering_read capability this can't happen, as it's set regardless of RO value in PCI config space and thus can't change during runtime.

Hence, to allow RO read in VFs and VMs, use the new HCA capability relaxed_ordering_read without checking pcie_relaxed_ordering_enabled(). The old capability checks are kept for backward compatibility with older FWs.

Allowing RO in VFs and VMs is valuable since it can greatly improve performance on some setups. For example, testing throughput of a VF on an AMD EPYC 7763 and ConnectX-6 Dx setup showed roughly 60% performance improvement.

Signed-off-by Avihai Horon <[email protected]>

Reviewed-by Shay Drory <[email protected]>

Reviewed-by Aya Levin <[email protected]>

Signed-off-by Leon Romanovsky <[email protected]>

---

From: Leon Romanovsky <[email protected]>

From Avihai,

Currently, Relaxed Ordering (RO) can't be used in VFs directly and in VFs assigned to QEMU, even if the PF supports RO. This is due to issues in reporting/emulation of PCI config space RO bit and due to current HCA capability behavior.

This series fixes it by using a new HCA capability and by relying on FW to do the "right thing" according to the PF's PCI config space RO value.

Allowing RO in VFs and VMs is valuable since it can greatly improve performance on some setups. For example, testing throughput of a VF on an AMD EPYC 7763 and ConnectX-6 Dx setup showed roughly 60% performance improvement.

Thanks

Avihai Horon (4):
RDMA/mlx5: Remove pcie_relaxed_ordering_enabled() check for RO write RDMA/mlx5: Check pcie_relaxed_ordering_enabled() in UMR net/mlx5: Update relaxed ordering read HCA capabilities RDMA/mlx5: Allow relaxed ordering read in VFs and VMs

Quoted 17 lines · Leon Romanovsky

From: Leon Romanovsky <[email protected]>

From Avihai,

Currently, Relaxed Ordering (RO) can't be used in VFs directly and in VFs assigned to QEMU, even if the PF supports RO. This is due to issues in reporting/emulation of PCI config space RO bit and due to current HCA capability behavior.

This series fixes it by using a new HCA capability and by relying on FW to do the "right thing" according to the PF's PCI config space RO value.

Allowing RO in VFs and VMs is valuable since it can greatly improve performance on some setups. For example, testing throughput of a VF on an AMD EPYC 7763 and ConnectX-6 Dx setup showed roughly 60% performance improvement.

Quoted 7 lines

Thanks

Avihai Horon (4):
RDMA/mlx5: Remove pcie_relaxed_ordering_enabled() check for RO write RDMA/mlx5: Check pcie_relaxed_ordering_enabled() in UMR net/mlx5: Update relaxed ordering read HCA capabilities RDMA/mlx5: Allow relaxed ordering read in VFs and VMs

This looks OK, but the patch structure is pretty confusing.

It seems to me there are really only two patches here, the first is to add some static inline

'mlx5 supports read ro'

which supports both the cap bits described in
the PRM, with a little comment to explain that old devices only set the old cap.

And a second patch to call it in all the places we need to check before setting the mkc ro read bit.

Maybe a final third patch to sort out that mistake in the write side.

But this really doesn't have anything to do with VFs and VMs, this is adjusting the code to follow the current PRM because the old one was mis-desgined.

Jason

Quoted 17 lines · Leon Romanovsky

From: Leon Romanovsky <[email protected]>

From Avihai,

Currently, Relaxed Ordering (RO) can't be used in VFs directly and in VFs assigned to QEMU, even if the PF supports RO. This is due to issues in reporting/emulation of PCI config space RO bit and due to current HCA capability behavior.

This series fixes it by using a new HCA capability and by relying on FW to do the "right thing" according to the PF's PCI config space RO value.

Allowing RO in VFs and VMs is valuable since it can greatly improve performance on some setups. For example, testing throughput of a VF on an AMD EPYC 7763 and ConnectX-6 Dx setup showed roughly 60% performance improvement.

Quoted 7 lines

Thanks

Avihai Horon (4):
RDMA/mlx5: Remove pcie_relaxed_ordering_enabled() check for RO write RDMA/mlx5: Check pcie_relaxed_ordering_enabled() in UMR net/mlx5: Update relaxed ordering read HCA capabilities RDMA/mlx5: Allow relaxed ordering read in VFs and VMs

This looks OK, but the patch structure is pretty confusing.

It seems to me there are really only two patches here, the first is to add some static inline

'mlx5 supports read ro'

which supports both the cap bits described in
the PRM, with a little comment to explain that old devices only set the old cap.

And a second patch to call it in all the places we need to check before setting the mkc ro read bit.

Maybe a final third patch to sort out that mistake in the write side.

But this really doesn't have anything to do with VFs and VMs, this is adjusting the code to follow the current PRM because the old one was mis-desgined.

Jason

Quoted 40 lines · Jason Gunthorpe

Quoted 17 lines · Leon Romanovsky

From: Leon Romanovsky <[email protected]>

From Avihai,

Currently, Relaxed Ordering (RO) can't be used in VFs directly and in VFs assigned to QEMU, even if the PF supports RO. This is due to issues in reporting/emulation of PCI config space RO bit and due to current HCA capability behavior.

This series fixes it by using a new HCA capability and by relying on FW to do the "right thing" according to the PF's PCI config space RO value.

Allowing RO in VFs and VMs is valuable since it can greatly improve performance on some setups. For example, testing throughput of a VF on an AMD EPYC 7763 and ConnectX-6 Dx setup showed roughly 60% performance improvement.

Quoted 7 lines

Thanks

Avihai Horon (4):
RDMA/mlx5: Remove pcie_relaxed_ordering_enabled() check for RO write RDMA/mlx5: Check pcie_relaxed_ordering_enabled() in UMR net/mlx5: Update relaxed ordering read HCA capabilities RDMA/mlx5: Allow relaxed ordering read in VFs and VMs

This looks OK, but the patch structure is pretty confusing.

It seems to me there are really only two patches here, the first is to add some static inline

'mlx5 supports read ro'

which supports both the cap bits described in
the PRM, with a little comment to explain that old devices only set the old cap.

And a second patch to call it in all the places we need to check before setting the mkc ro read bit.

Quoted 7 lines

Maybe a final third patch to sort out that mistake in the write side.

But this really doesn't have anything to do with VFs and VMs, this is adjusting the code to follow the current PRM because the old one was mis-desgined.

Jason

FWIW I think Jason's outline here makes sense too and might be slightly better. However, reading through the series I was reasonably able to understand things enough that I think its fine as-is.

In some sense its not about VF or VM, but fixing this has the result that it fixes a setup with VF and VM, so I think thats an ok thing to call out as the goal.

Quoted 2 lines

that it fixes a setup with VF and VM, so I think thats an ok thing to call out as the goal.

VF or VM came from user perspective of where this behavior is not correct. Avihai saw this in QEMU, so he described it in terms which are more clear to the end user.

Quoted 6 lines · Leon Romanovsky

Quoted 2 lines

that it fixes a setup with VF and VM, so I think thats an ok thing to call out as the goal.

VF or VM came from user perspective of where this behavior is not correct. Avihai saw this in QEMU, so he described it in terms which are more clear to the end user.

Except it is not clear, the VF/VM issue is more properly solved by showing the real relaxed order cap to the VM.

From: Leon Romanovsky <[email protected]>

From Avihai,

Currently, Relaxed Ordering (RO) can't be used in VFs directly and in VFs assigned to QEMU, even if the PF supports RO. This is due to issues in reporting/emulation of PCI config space RO bit and due to current HCA capability behavior.

[...]