Skip to contentSkip to conversation

Please pull mlx5 VFIO PCI DMA conversion

GIT PULL Leon Romanovsky 2 participants 4 messages May 13, 2025 – May 15, 2025 KVM

lore
Share & tools
lore
#0 lore raw

Hi Alex,

Please accept this pull request, which presents subset of new DMA-API patchset [1] specific for VFIO subsystem.

It is based on Marek's dma-mapping-for-6.16-two-step-api branch, so merging now will allow us to reduce possible rebase errors in mlx5 vfio code and give enough time to start to work on second driver conversion. Such conversion will allow us to generalize the API for VFIO kernel drivers, in similiar way that was done for RDMA, HMM and block layers.

Thanks

[1] [PATCH v10 00/24] Provide a new two step DMA mapping API
https://lore.kernel.org/all/[email protected]/

----------------------------------------------------------------
The following changes since commit 3ee7d9496342246f4353716f6bbf64c945ff6e2d:

docs: core-api: document the IOVA-based API (2025-05-06 08:36:54 +0200)

are available in the Git repository at:

https://git.kernel.org/pub/scm/linux/kernel/git/leon/linux-rdma.git tags/vfio-dma-two-step

for you to fetch changes up to 855c89a25e1756b7699b863afd4d6afcbd0de0d6:

vfio/mlx5: Enable the DMA link API (2025-05-13 03:58:27 -0400)

----------------------------------------------------------------
Convert mlx5 VFIO PCI driver to new two step DMA API

This PR is based on newly accepted DMA API, which allows us to avoid building scatter-gather lists just to batch mapping and unmapping of pages.

VFIO PCI live migration code is building a very large "page list" for the device. Instead of allocating a scatter list entry per allocated page it can just allocate an array of 'struct page *', saving a large amount of memory.

----------------------------------------------------------------
Leon Romanovsky (3):
vfio/mlx5: Explicitly use number of pages instead of allocated length vfio/mlx5: Rewrite create mkey flow to allow better code reuse vfio/mlx5: Enable the DMA link API

 drivers/vfio/pci/mlx5/cmd.c  | 375 +++++++++++++++++++++----------------------
 drivers/vfio/pci/mlx5/cmd.h  |  35 ++--
 drivers/vfio/pci/mlx5/main.c |  87 +++++-----
 3 files changed, 239 insertions(+), 258 deletions(-)
#1 lore raw
Quoted 10 lines · On Tue, 13 May 2025 13:48:10 +0300 Leon Romanovsky <[email protected]> wrote:

Hi Alex,

Please accept this pull request, which presents subset of new DMA-API patchset [1] specific for VFIO subsystem.

It is based on Marek's dma-mapping-for-6.16-two-step-api branch, so merging now will allow us to reduce possible rebase errors in mlx5 vfio code and give enough time to start to work on second driver conversion. Such conversion will allow us to generalize the API for VFIO kernel drivers, in similiar way that was done for RDMA, HMM and block layers.

Hi Leon,

Pull requests are not my typical workflow. Are these mlx5-vfio-pci changes intended to enter mainline through the vfio tree or your rdma tree? Why do the commits not include a review/ack from Yishai?

Typically I'd expect a patch series for the mlx5-vfio-pci changes that I would apply, with Yishai's approval, to a shared branch containing the commits Marek has already accepted. I'm not sure why we're preempting that process here. Thanks,

Alex

Quoted 41 lines

[1] [PATCH v10 00/24] Provide a new two step DMA mapping API
https://lore.kernel.org/all/[email protected]/

----------------------------------------------------------------
The following changes since commit 3ee7d9496342246f4353716f6bbf64c945ff6e2d:

docs: core-api: document the IOVA-based API (2025-05-06 08:36:54 +0200)

are available in the Git repository at:

https://git.kernel.org/pub/scm/linux/kernel/git/leon/linux-rdma.git tags/vfio-dma-two-step

for you to fetch changes up to 855c89a25e1756b7699b863afd4d6afcbd0de0d6:

vfio/mlx5: Enable the DMA link API (2025-05-13 03:58:27 -0400)

----------------------------------------------------------------
Convert mlx5 VFIO PCI driver to new two step DMA API

This PR is based on newly accepted DMA API, which allows us to avoid building scatter-gather lists just to batch mapping and unmapping of pages.

VFIO PCI live migration code is building a very large "page list" for the device. Instead of allocating a scatter list entry per allocated page it can just allocate an array of 'struct page *', saving a large amount of memory.

----------------------------------------------------------------
Leon Romanovsky (3):
vfio/mlx5: Explicitly use number of pages instead of allocated length vfio/mlx5: Rewrite create mkey flow to allow better code reuse vfio/mlx5: Enable the DMA link API

 drivers/vfio/pci/mlx5/cmd.c  | 375 +++++++++++++++++++++----------------------
 drivers/vfio/pci/mlx5/cmd.h  |  35 ++--
 drivers/vfio/pci/mlx5/main.c |  87 +++++-----
 3 files changed, 239 insertions(+), 258 deletions(-)
#2 lore raw
Quoted 14 lines · Alex Williamson
Quoted 10 lines · On Tue, 13 May 2025 13:48:10 +0300 Leon Romanovsky <[email protected]> wrote:

Hi Alex,

Please accept this pull request, which presents subset of new DMA-API patchset [1] specific for VFIO subsystem.

It is based on Marek's dma-mapping-for-6.16-two-step-api branch, so merging now will allow us to reduce possible rebase errors in mlx5 vfio code and give enough time to start to work on second driver conversion. Such conversion will allow us to generalize the API for VFIO kernel drivers, in similiar way that was done for RDMA, HMM and block layers.

Quoted 5 lines

Hi Leon,

Pull requests are not my typical workflow. Are these mlx5-vfio-pci changes intended to enter mainline through the vfio tree or your rdma tree?

VFIO changes will come through your tree. DMA patches are the same as in Marek's DMA tree and in our RDMA tree.

I prepared PR to save from your hassle of merging dma/dma-mapping-for-6.16-two-step-api topic from Marek and collecting VFIO patches from ML.

Quoted 1 line

Why do the commits not include a review/ack from Yishai?

They have Jason's review tags and as far as I know Yishai, he trusts Jason's judgement.

Quoted 5 lines

Typically I'd expect a patch series for the mlx5-vfio-pci changes that I would apply, with Yishai's approval, to a shared branch containing the commits Marek has already accepted. I'm not sure why we're preempting that process here. Thanks,

This is exactly what is in this PR: reviewed VFIO patches which were posted to the ML on top of Marek's shared branch.

If you prefer, I can repost the VFIO patches.

Thanks

Quoted 45 lines

Alex

Quoted 41 lines

[1] [PATCH v10 00/24] Provide a new two step DMA mapping API
https://lore.kernel.org/all/[email protected]/

----------------------------------------------------------------
The following changes since commit 3ee7d9496342246f4353716f6bbf64c945ff6e2d:

docs: core-api: document the IOVA-based API (2025-05-06 08:36:54 +0200)

are available in the Git repository at:

https://git.kernel.org/pub/scm/linux/kernel/git/leon/linux-rdma.git tags/vfio-dma-two-step

for you to fetch changes up to 855c89a25e1756b7699b863afd4d6afcbd0de0d6:

vfio/mlx5: Enable the DMA link API (2025-05-13 03:58:27 -0400)

----------------------------------------------------------------
Convert mlx5 VFIO PCI driver to new two step DMA API

This PR is based on newly accepted DMA API, which allows us to avoid building scatter-gather lists just to batch mapping and unmapping of pages.

VFIO PCI live migration code is building a very large "page list" for the device. Instead of allocating a scatter list entry per allocated page it can just allocate an array of 'struct page *', saving a large amount of memory.

----------------------------------------------------------------
Leon Romanovsky (3):
vfio/mlx5: Explicitly use number of pages instead of allocated length vfio/mlx5: Rewrite create mkey flow to allow better code reuse vfio/mlx5: Enable the DMA link API

 drivers/vfio/pci/mlx5/cmd.c  | 375 +++++++++++++++++++++----------------------
 drivers/vfio/pci/mlx5/cmd.h  |  35 ++--
 drivers/vfio/pci/mlx5/main.c |  87 +++++-----
 3 files changed, 239 insertions(+), 258 deletions(-)
#3 lore raw
Quoted 15 lines · On Thu, 15 May 2025 22:51:16 +0300 Leon Romanovsky <[email protected]> wrote:
Quoted 14 lines · Alex Williamson
Quoted 10 lines · On Tue, 13 May 2025 13:48:10 +0300 Leon Romanovsky <[email protected]> wrote:

Hi Alex,

Please accept this pull request, which presents subset of new DMA-API patchset [1] specific for VFIO subsystem.

It is based on Marek's dma-mapping-for-6.16-two-step-api branch, so merging now will allow us to reduce possible rebase errors in mlx5 vfio code and give enough time to start to work on second driver conversion. Such conversion will allow us to generalize the API for VFIO kernel drivers, in similiar way that was done for RDMA, HMM and block layers.

Quoted 8 lines
Quoted 5 lines

Hi Leon,

Pull requests are not my typical workflow. Are these mlx5-vfio-pci changes intended to enter mainline through the vfio tree or your rdma tree?

VFIO changes will come through your tree. DMA patches are the same as in Marek's DMA tree and in our RDMA tree.

Ok

Quoted 7 lines

I prepared PR to save from your hassle of merging dma/dma-mapping-for-6.16-two-step-api topic from Marek and collecting VFIO patches from ML.

Quoted 1 line

Why do the commits not include a review/ack from Yishai?

They have Jason's review tags and as far as I know Yishai, he trusts Jason's judgement.

I don't doubt that, but Yishai is listed as the maintainer, so barring some special circumstances we should at least have an ack.

Quoted 9 lines
Quoted 4 lines

Typically I'd expect a patch series for the mlx5-vfio-pci changes that I would apply, with Yishai's approval, to a shared branch containing the commits Marek has already accepted. I'm not sure why we're preempting that process here. Thanks,

This is exactly what is in this PR: reviewed VFIO patches which were posted to the ML on top of Marek's shared branch.

If you prefer, I can repost the VFIO patches.

I appreciate the attempt to streamline things and expect this largely only affects the chain of sign-off approvals, but at the same time that's an important aspect. If the code here is identical to v10, we could use that with Yishai's ack, but it's probably most direct (irony vs PR noted) to repost just just the mlx5-vfio-pci changes with a reference to Marek's shared topic branch. Thanks,

Alex