Skip to content

RFC Removing struct page from P2PDMA

Logan Gunthorpe linux-block linux-nvme linux-pci linux-rdma lkml 1 version In review

Get series with b4
b4 am [email protected]

Versions newest first

  1. v1 RFC Latest28 patches 60 repliesIn review

Patches in v1 No applies recorded yet

#Subject and filesSizeRepliesApplied
1/28block: Introduce DMA direct request typeinclude/linux/blk_types.h include/linux/blkdev.h+18 -10 replies
2/28block: Add dma_vec structureinclude/linux/bio.h include/linux/blk_types.h include/linux/bvec.h+59 -10 replies
3/28block: Warn on mis-use of dma-direct biosblock/bio.c block/blk-core.c+36 -00 replies
4/28block: Never bounce dma-direct biosblock/bounce.c+8 -02 replies
5/28block: Skip dma-direct bios in bio_integrity_prep()block/bio-integrity.c+4 -00 replies
6/28block: Support dma-direct bios in bio_advance_iter()include/linux/bio.h+2 -00 replies
7/28block: Use dma_vec length in bio_cur_bytes() for dma-direct biosinclude/linux/bio.h+5 -30 replies
8/28block: Introduce dmavec_phys_mergeable()block/blk.h+22 -60 replies
9/28block: Introduce vec_gap_to_prev()block/blk.h+19 -20 replies
10/28block: Create generic vec_split_segs() from bvec_split_segs()block/blk-merge.c+14 -70 replies
11/28block: Create blk_segment_split_ctxblock/blk-merge.c+35 -200 replies
12/28block: Create helper for bvec_should_split()block/blk-merge.c+46 -400 replies
13/28block: Generalize bvec_should_split()block/blk-merge.c+17 -140 replies
14/28block: Support splitting dma-direct biosblock/blk-merge.c+37 -80 replies
15/28block: Support counting dma-direct bio segmentsblock/blk-merge.c+30 -110 replies
16/28block: Implement mapping dma-direct requests to SGs in blk_rq_map_sg()block/blk-merge.c+65 -00 replies
17/28block: Introduce queue flag to indicate support for dma-direct biosinclude/linux/blkdev.h+3 -00 replies
18/28block: Introduce bio_add_dma_addr()block/bio.c include/linux/bio.h+48 -00 replies
19/28nvme-pci: Support dma-direct biosdrivers/nvme/host/core.c drivers/nvme/host/nvme.h drivers/nvme/host/pci.c+10 -30 replies
20/28IB/core: Introduce API for initializing a RW ctx from a DMA addressdrivers/infiniband/core/rw.c include/rdma/rw.h+69 -114 replies
21/28nvmet: Split nvmet_bdev_execute_rw() into a helper functiondrivers/nvme/target/io-cmd-bdev.c+31 -210 replies
22/28nvmet: Use DMA addresses instead of struct pages for P2Pdrivers/nvme/target/core.c drivers/nvme/target/io-cmd-bdev.c drivers/nvme/target/nvmet.h +1 more+71 -210 replies
23/28nvme-pci: Remove support for PCI_P2PDMA requestsdrivers/nvme/host/core.c drivers/nvme/host/nvme.h drivers/nvme/host/pci.c+11 -210 replies
24/28block: Remove PCI_P2PDMA queue flaginclude/linux/blkdev.h+1 -40 replies
25/28IB/core: Remove P2PDMA mapping support in rdma_rw_ctxdrivers/infiniband/core/rw.c+2 -90 replies
26/28PCI/P2PDMA: Remove SGL helpersDocumentation/driver-api/pci/p2pdma.rst drivers/pci/p2pdma.c include/linux/pci-p2pdma.h+3 -1200 replies
27/28PCI/P2PDMA: Remove struct pages that back P2PDMA memorydrivers/pci/p2pdma.c+33 -740 replies
28/28memremap: Remove PCI P2PDMA page memory typeinclude/linux/memremap.h include/linux/mm.h+0 -180 replies

Reviews trailers per patch

No Reviewed-by, Acked-by or Tested-by trailers for v1 yet.

Read the v1 thread

Cover letter [RFC PATCH 00/28] Removing struct page from P2PDMA

For eons there has been a debate over whether or not to use struct pages for peer-to-peer DMA transactions. Pro-pagers have argued that struct pages are necessary for interacting with existing code like scatterlists or the bio_vecs. Anti-pagers assert that the tracking of the memory is unecessary and allocating the pages is a waste of memory. Both viewpoints are valid, however developers working on GPUs and RDMA tend to be able to do away with struct pages relatively easily compared to those wanting to work with NVMe devices through the block layer. So it would be of great value to be able to universally do P2PDMA transactions without the use of struct pages.

Previously, there have been multiple attempts[1][2] to replace struct page usage with pfn_t but this has been unpopular seeing it creates dangerous edge cases where unsuspecting code might run accross pfn_t's they are not ready for.

Currently, we have P2PDMA using struct pages through the block layer and the dangerous cases are avoided by using a queue flag that indicates support for the special pages.

This RFC proposes a new solution: allow the block layer to take DMA addresses directly for queues that indicate support. This will provide a more general path for doing P2PDMA-like requests and will allow us to remove the struct pages that back P2PDMA memory thus paving the way to build a more uniform P2PDMA ecosystem.

This is a fairly long patch set but most of the patches are quite small. Patches 1 through 18 introduce the concept of a dma_vec that is similar to a bio_vec (except it takes dma_addr_t's instead of pages and offsets) as well as a special dma-direct bio/request. Most of these patches just prevent the new type of bio from being mis-used and also support splitting and mapping them in the same way that struct page bios can be operated on. Patches 19 through 22 modify the existing P2PDMA support in nvme-pci, ib-core and nvmet to use DMA addresses directly. Patches 23 through 25 remove the P2PDMA specific code from the block layer and ib-core. Finally, patches 26 through 28 remove the struct pages from the PCI P2PDMA code.

This RFC is based on v5.2-rc5 and a git branch is available here:

https://github.com/sbates130272/linux-p2pmem.git dma_direct_rfc1

[1] https://lwn.net/Articles/647404/
[2] https://lore.kernel.org/lkml/1495662147-18277-1-git-send-email-logang at deltatee.com/

--

Logan Gunthorpe (28):
block: Introduce DMA direct request type
block: Add dma_vec structure
block: Warn on mis-use of dma-direct bios
block: Never bounce dma-direct bios
block: Skip dma-direct bios in bio_integrity_prep()
block: Support dma-direct bios in bio_advance_iter()
block: Use dma_vec length in bio_cur_bytes() for dma-direct bios
block: Introduce dmavec_phys_mergeable()
block: Introduce vec_gap_to_prev()
block: Create generic vec_split_segs() from bvec_split_segs()
block: Create blk_segment_split_ctx
block: Create helper for bvec_should_split()
block: Generalize bvec_should_split()
block: Support splitting dma-direct bios
block: Support counting dma-direct bio segments
block: Implement mapping dma-direct requests to SGs in blk_rq_map_sg()
block: Introduce queue flag to indicate support for dma-direct bios
block: Introduce bio_add_dma_addr()
nvme-pci: Support dma-direct bios
IB/core: Introduce API for initializing a RW ctx from a DMA address
nvmet: Split nvmet_bdev_execute_rw() into a helper function
nvmet: Use DMA addresses instead of struct pages for P2P
nvme-pci: Remove support for PCI_P2PDMA requests
block: Remove PCI_P2PDMA queue flag
IB/core: Remove P2PDMA mapping support in rdma_rw_ctx PCI/P2PDMA: Remove SGL helpers
PCI/P2PDMA: Remove struct pages that back P2PDMA memory
memremap: Remove PCI P2PDMA page memory type

 Documentation/driver-api/pci/p2pdma.rst |   9 +-
 block/bio-integrity.c                   |   4 +
 block/bio.c                             |  71 +++++++
 block/blk-core.c                        |   3 +
 block/blk-merge.c                       | 256 ++++++++++++++++++------
 block/blk.h                             |  49 ++++-
 block/bounce.c                          |   8 +
 drivers/infiniband/core/rw.c            |  85 ++++++--
 drivers/nvme/host/core.c                |   4 +-
 drivers/nvme/host/nvme.h                |   2 +-
 drivers/nvme/host/pci.c                 |  29 ++-
 drivers/nvme/target/core.c              |  12 +-
 drivers/nvme/target/io-cmd-bdev.c       |  82 +++++---
 drivers/nvme/target/nvmet.h             |   5 +-
 drivers/nvme/target/rdma.c              |  43 +++-
 drivers/pci/p2pdma.c                    | 202 +++----------------
 include/linux/bio.h                     |  32 ++-
 include/linux/blk_types.h               |  14 +-
 include/linux/blkdev.h                  |  16 +-
 include/linux/bvec.h                    |  43 ++++
 include/linux/memremap.h                |   5 -
 include/linux/mm.h                      |  13 --
 include/linux/pci-p2pdma.h              |  19 --
 include/rdma/rw.h                       |   6 +
 24 files changed, 648 insertions(+), 364 deletions(-)

--
2.20.1