Support VFIO sparse mmap in PCI bus
Subsystems (8)
cover Support VFIO sparse mmap in PCI bus
This series introduces a VFIO standard capability, called sparse mmap to PCI bus. In linux kernel, it's defined as VFIO_REGION_INFO_CAP_SPARSE_MMAP. Sparse mmap means instead of mmap whole BAR region into DPDK process, only mmap part of the BAR region after getting sparse mmap information from kernel. For the rest of BAR region that is not mmap-ed, DPDK process can use pread/pwrite system calls to access. Sparse mmap is useful when kernel does not want userspace to mmap whole BAR region, or kernel wants to control over access to specific BAR region. Vendors can choose to enable this feature or not for their devices in their specific kernel modules.
In this patchset:
Patch 1-3 is mainly for introducing BAR access APIs so that driver could use them to access specific BAR using pread/pwrite system calls when part of the BAR is not mmap-able.
Patch 4 adds the VFIO sparse mmap support finally. A question is for all sparse mmap regions, should they be mapped to a continuous virtual address region that follows device-specific BAR layout or not. In theory, there could be three options to support this feature.
Option 1: Map sparse mmap regions independently
======================================================
In this approach, we mmap each sparse mmap region one by one and each region could be located anywhere in process address space. But accessing the mmaped BAR will not be as easy as 'bar_base_address + bar_offset', driver needs to check the sparse mmap information to access specific BAR register.
Patch 4 in this patchset adopts this option. Driver API change is introduced in bus_pci_driver.h. Corresponding changes in all drivers are also done and currently I am assuming drivers do not support this feature so they will not check the 'is_sparse' flag but assumes it to be false. Note that it will not break any driver and each vendor can add related logic when they start to support this feature. This is only because I don't want to introduce complexity to drivers that do not want to support this feature.
Option 2: Map sparse mmap regions based on device-specific BAR layout
======================================================================
In this approach, the sparse mmap regions are mapped to continuous virtual address region that follows device-specific BAR layout. For example, the BAR size is 0x4000 and only 0-0x1000 (sparse mmap region #1) and 0x3000-0x4000 (sparse mmap region #2) could be mmaped. Region #1 will be mapped at 'base_addr' and region #2 will be mapped at 'base_addr + 0x3000'. The good thing is if we implement like this, driver can still access all BAR registers using 'bar_base_address + bar_offset' way and we don't need to introduce any driver API change. But the address space range 'base_addr + 0x1000' to 'base_addr + 0x3000' may need to be reserved so it could result in waste of address space or memory (when we use MAP_ANONYMOUS and MAP_PRIVATE flag to reserve this range). Meanwhile, driver needs to know which part of BAR is mmaped (this is possible since the range is defined by vendor's specific kernel module).
Option 3: Support both option 1 & 2
===================================
We could define a driver flag to let driver choose which way it perfers since either option has its own Pros & Cons.
Please share your comments, Thanks!
Chenbo Xia (4):
bus/pci: introduce an internal representation of PCI device bus/pci: avoid depending on private value in kernel source bus/pci: introduce helper for MMIO read and write bus/pci: add VFIO sparse mmap support
drivers/baseband/acc/rte_acc100_pmd.c | 6 +- drivers/baseband/acc/rte_vrb_pmd.c | 6 +- .../fpga_5gnr_fec/rte_fpga_5gnr_fec.c | 6 +- drivers/baseband/fpga_lte_fec/fpga_lte_fec.c | 6 +- drivers/bus/pci/bsd/pci.c | 43 +- drivers/bus/pci/bus_pci_driver.h | 24 +- drivers/bus/pci/linux/pci.c | 91 +++- drivers/bus/pci/linux/pci_init.h | 14 +- drivers/bus/pci/linux/pci_uio.c | 34 +- drivers/bus/pci/linux/pci_vfio.c | 445 ++++++++++++++---- drivers/bus/pci/pci_common.c | 57 ++- drivers/bus/pci/pci_common_uio.c | 12 +- drivers/bus/pci/private.h | 25 +- drivers/bus/pci/rte_bus_pci.h | 48 ++ drivers/bus/pci/version.map | 3 + drivers/common/cnxk/roc_dev.c | 4 +- drivers/common/cnxk/roc_dpi.c | 2 +- drivers/common/cnxk/roc_ml.c | 22 +- drivers/common/qat/dev/qat_dev_gen1.c | 2 +- drivers/common/qat/dev/qat_dev_gen4.c | 4 +- drivers/common/sfc_efx/sfc_efx.c | 2 +- drivers/compress/octeontx/otx_zip.c | 4 +- drivers/crypto/ccp/ccp_dev.c | 4 +- drivers/crypto/cnxk/cnxk_cryptodev_ops.c | 2 +- drivers/crypto/nitrox/nitrox_device.c | 4 +- drivers/crypto/octeontx/otx_cryptodev_ops.c | 6 +- drivers/crypto/virtio/virtio_pci.c | 6 +- drivers/dma/cnxk/cnxk_dmadev.c | 2 +- drivers/dma/hisilicon/hisi_dmadev.c | 6 +- drivers/dma/idxd/idxd_pci.c | 4 +- drivers/dma/ioat/ioat_dmadev.c | 2 +- drivers/event/dlb2/pf/dlb2_main.c | 16 +- drivers/event/octeontx/ssovf_probe.c | 38 +- drivers/event/octeontx/timvf_probe.c | 18 +- drivers/event/skeleton/skeleton_eventdev.c | 2 +- drivers/mempool/octeontx/octeontx_fpavf.c | 6 +- drivers/net/ark/ark_ethdev.c | 4 +- drivers/net/atlantic/atl_ethdev.c | 2 +- drivers/net/avp/avp_ethdev.c | 20 +- drivers/net/axgbe/axgbe_ethdev.c | 4 +- drivers/net/bnx2x/bnx2x_ethdev.c | 6 +- drivers/net/bnxt/bnxt_ethdev.c | 8 +- drivers/net/cpfl/cpfl_ethdev.c | 4 +- drivers/net/cxgbe/cxgbe_ethdev.c | 2 +- drivers/net/cxgbe/cxgbe_main.c | 2 +- drivers/net/cxgbe/cxgbevf_ethdev.c | 2 +- drivers/net/cxgbe/cxgbevf_main.c | 2 +- drivers/net/e1000/em_ethdev.c | 4 +- drivers/net/e1000/igb_ethdev.c | 4 +- drivers/net/ena/ena_ethdev.c | 4 +- drivers/net/enetc/enetc_ethdev.c | 2 +- drivers/net/enic/enic_main.c | 4 +- drivers/net/fm10k/fm10k_ethdev.c | 2 +- drivers/net/gve/gve_ethdev.c | 4 +- drivers/net/hinic/base/hinic_pmd_hwif.c | 14 +- drivers/net/hns3/hns3_ethdev.c | 2 +- drivers/net/hns3/hns3_ethdev_vf.c | 2 +- drivers/net/hns3/hns3_rxtx.c | 4 +- drivers/net/i40e/i40e_ethdev.c | 2 +- drivers/net/iavf/iavf_ethdev.c | 2 +- drivers/net/ice/ice_dcf.c | 2 +- drivers/net/ice/ice_ethdev.c | 2 +- drivers/net/idpf/idpf_ethdev.c | 4 +- drivers/net/igc/igc_ethdev.c | 2 +- drivers/net/ionic/ionic_dev_pci.c | 2 +- drivers/net/ixgbe/ixgbe_ethdev.c | 4 +- drivers/net/liquidio/lio_ethdev.c | 4 +- drivers/net/nfp/nfp_ethdev.c | 2 +- drivers/net/nfp/nfp_ethdev_vf.c | 6 +- drivers/net/nfp/nfpcore/nfp_cpp_pcie_ops.c | 4 +- drivers/net/ngbe/ngbe_ethdev.c | 2 +- drivers/net/octeon_ep/otx_ep_ethdev.c | 2 +- drivers/net/octeontx/base/octeontx_pkivf.c | 6 +- drivers/net/octeontx/base/octeontx_pkovf.c | 12 +- drivers/net/qede/qede_main.c | 6 +- drivers/net/sfc/sfc.c | 2 +- drivers/net/thunderx/nicvf_ethdev.c | 2 +- drivers/net/txgbe/txgbe_ethdev.c | 2 +- drivers/net/txgbe/txgbe_ethdev_vf.c | 2 +- drivers/net/virtio/virtio_pci.c | 6 +- drivers/net/vmxnet3/vmxnet3_ethdev.c | 4 +- drivers/raw/cnxk_bphy/cnxk_bphy.c | 10 +- drivers/raw/cnxk_bphy/cnxk_bphy_cgx.c | 6 +- drivers/raw/ifpga/afu_pmd_n3000.c | 4 +- drivers/raw/ifpga/ifpga_rawdev.c | 6 +- drivers/raw/ntb/ntb_hw_intel.c | 8 +- drivers/vdpa/ifc/ifcvf_vdpa.c | 6 +- drivers/vdpa/sfc/sfc_vdpa_hw.c | 2 +- drivers/vdpa/sfc/sfc_vdpa_ops.c | 2 +- lib/eal/include/rte_vfio.h | 1 - 90 files changed, 853 insertions(+), 352 deletions(-)
Signature · 1 line
2.17.1
RFC 1/4 bus/pci: introduce an internal representation of PCI device
This patch introduces an internal representation of the PCI device which will be used to store the internal information that don't have to be exposed to drivers, e.g., the VFIO region sizes/offsets.
In this patch, the internal structure is simply a wrapper of the rte_pci_device structure. More fields will be added.
--- drivers/bus/pci/bsd/pci.c | 13 ++++++++----- drivers/bus/pci/linux/pci.c | 28 ++++++++++++++++------------ drivers/bus/pci/pci_common.c | 12 ++++++------ drivers/bus/pci/private.h | 14 +++++++++++++- 4 files changed, 43 insertions(+), 24 deletions(-)
drivers/bus/pci/bsd/pci.c +8 -5
@@ -208,16 +208,19 @@ pci_uio_map_resource_by_index(struct rte_pci_device *dev, int res_idx,static int pci_scan_one (int dev_pci_fd,struct pci_conf *conf) { +struct rte_pci_device_internal *pdev;struct rte_pci_device *dev;struct pci_bar_io bar;unsigned i, max; - dev =malloc (sizeof (*dev)); -if (dev ==NULL ) { + pdev =malloc (sizeof (*pdev)); +if (pdev ==NULL ) { +RTE_LOG (ERR ,EAL ,"Cannot allocate memory for internal pci device\n" );return -1 ; } -memset (dev,0 ,sizeof (*dev)); +memset (pdev,0 ,sizeof (*pdev)); + dev = &pdev->device; dev->device.bus = &rte_pci_bus.bus; dev->addr.domain = conf->pc_sel.pc_domain;
@@ -303,7 +306,7 @@ pci_scan_one(int dev_pci_fd, struct pci_conf *conf)memmove (dev2->mem_resource, dev->mem_resource,sizeof (dev->mem_resource)); -pci_free (dev); +pci_free (pdev); }return 0 ; }
@@ -313,7 +316,7 @@ pci_scan_one(int dev_pci_fd, struct pci_conf *conf)return 0 ; skipdev: -pci_free (dev); +pci_free (pdev);return 0 ; }
drivers/bus/pci/linux/pci.c +16 -12
@@ -211,22 +211,26 @@ pci_scan_one(const char *dirname, const struct rte_pci_addr *addr) {char filename[PATH_MAX ];unsigned long tmp; +struct rte_pci_device_internal *pdev;struct rte_pci_device *dev;char driver[PATH_MAX ];int ret; - dev =malloc (sizeof (*dev)); -if (dev ==NULL ) + pdev =malloc (sizeof (*pdev)); +if (pdev ==NULL ) { +RTE_LOG (ERR ,EAL ,"Cannot allocate memory for internal pci device\n" );return -1 ; + } -memset (dev,0 ,sizeof (*dev)); +memset (pdev,0 ,sizeof (*pdev)); + dev = &pdev->device; dev->device.bus = &rte_pci_bus.bus; dev->addr = *addr;/* get vendor id */ snprintf (filename,sizeof (filename),"%s/vendor" , dirname);if (eal_parse_sysfs_value (filename, &tmp) <0 ) { -pci_free (dev); +pci_free (pdev);return -1 ; } dev->id.vendor_id = (uint16_t )tmp;
@@ -234,7 +238,7 @@ pci_scan_one(const char *dirname, const struct rte_pci_addr *addr)/* get device id */ snprintf (filename,sizeof (filename),"%s/device" , dirname);if (eal_parse_sysfs_value (filename, &tmp) <0 ) { -pci_free (dev); +pci_free (pdev);return -1 ; } dev->id.device_id = (uint16_t )tmp;
@@ -243,7 +247,7 @@ pci_scan_one(const char *dirname, const struct rte_pci_addr *addr)snprintf (filename,sizeof (filename),"%s/subsystem_vendor" , dirname);if (eal_parse_sysfs_value (filename, &tmp) <0 ) { -pci_free (dev); +pci_free (pdev);return -1 ; } dev->id.subsystem_vendor_id = (uint16_t )tmp;
@@ -252,7 +256,7 @@ pci_scan_one(const char *dirname, const struct rte_pci_addr *addr)snprintf (filename,sizeof (filename),"%s/subsystem_device" , dirname);if (eal_parse_sysfs_value (filename, &tmp) <0 ) { -pci_free (dev); +pci_free (pdev);return -1 ; } dev->id.subsystem_device_id = (uint16_t )tmp;
@@ -261,7 +265,7 @@ pci_scan_one(const char *dirname, const struct rte_pci_addr *addr)snprintf (filename,sizeof (filename),"%s/class" , dirname);if (eal_parse_sysfs_value (filename, &tmp) <0 ) { -pci_free (dev); +pci_free (pdev);return -1 ; }/* the least 24 bits are valid: class, subclass, program interface */
@@ -297,7 +301,7 @@ pci_scan_one(const char *dirname, const struct rte_pci_addr *addr)snprintf (filename,sizeof (filename),"%s/resource" , dirname);if (pci_parse_sysfs_resource (filename, dev) <0 ) {RTE_LOG (ERR ,EAL ,"%s(): cannot parse resource\n" , __func__); -pci_free (dev); +pci_free (pdev);return -1 ; }
@@ -306,7 +310,7 @@ pci_scan_one(const char *dirname, const struct rte_pci_addr *addr) ret =pci_get_kernel_driver_by_path (filename, driver,sizeof (driver));if (ret <0 ) {RTE_LOG (ERR ,EAL ,"Fail to get kernel driver\n" ); -pci_free (dev); +pci_free (pdev);return -1 ; }
@@ -320,7 +324,7 @@ pci_scan_one(const char *dirname, const struct rte_pci_addr *addr)else dev->kdrv =RTE_PCI_KDRV_UNKNOWN ; }else { -pci_free (dev); +pci_free (pdev);return 0 ; }/* device is valid, add in list (sorted) */
@@ -375,7 +379,7 @@ pci_scan_one(const char *dirname, const struct rte_pci_addr *addr)pci_common_set (dev2); } } -pci_free (dev); +pci_free (pdev); }return 0 ; }
drivers/bus/pci/pci_common.c +6 -6
@@ -121,12 +121,12 @@ pci_common_set(struct rte_pci_device *dev) }void -pci_free (struct rte_pci_device *dev) +pci_free (struct rte_pci_device_internal *pdev) { -if (dev ==NULL ) +if (pdev ==NULL )return ; -free (dev->bus_info); -free (dev); +free (pdev->device.bus_info); +free (pdev); }/* map a particular resource from a file */
@@ -465,7 +465,7 @@ pci_cleanup(void)rte_intr_instance_free (dev->vfio_req_intr_handle); dev->vfio_req_intr_handle =NULL ; -pci_free (dev); +pci_free (RTE_PCI_DEVICE_INTERNAL (dev)); }return error;
@@ -681,7 +681,7 @@ pci_unplug(struct rte_device *dev)if (ret ==0 ) {rte_pci_remove_device (pdev);rte_devargs_remove (dev->devargs); -pci_free (pdev); +pci_free (RTE_PCI_DEVICE_INTERNAL (pdev)); }return ret; }
drivers/bus/pci/private.h +13 -1
@@ -13,6 +13,14 @@#include <rte_os_shim.h> #include <rte_pci.h> +/* +* Convert struct rte_pci_device to struct rte_pci_device_internal +*/ +#define RTE_PCI_DEVICE_INTERNAL (ptr) \ + container_of(ptr, struct rte_pci_device_internal, device) +#define RTE_PCI_DEVICE_INTERNAL_CONST (ptr) \ + container_of(ptr, const struct rte_pci_device_internal, device) +/** * Structure describing the PCI bus */
@@ -34,6 +42,10 @@ extern struct rte_pci_bus rte_pci_bus;struct rte_pci_driver ;struct rte_pci_device ; +struct rte_pci_device_internal { +struct rte_pci_device device; +}; +/** * Scan the content of the PCI bus, and the devices in the devices * list
@@ -53,7 +65,7 @@ pci_common_set(struct rte_pci_device *dev);* Free a PCI device. */ void -pci_free (struct rte_pci_device *dev); +pci_free (struct rte_pci_device_internal *pdev);/** * Validate whether a device with given PCI address should be ignored or not.
Signature · 1 line
2.17.1
RFC 2/4 bus/pci: avoid depending on private value in kernel source
In reply to Chenbo XiaThe value 40 used in VFIO_GET_REGION_ADDR() is a private value (VFIO_PCI_OFFSET_SHIFT) defined in Linux kernel source [1]. It is not part of VFIO API, and we should not depend on it.
[1] https://github.com/torvalds/linux/blob/v6.2/include/linux/vfio_pci_core.h
--- drivers/bus/pci/linux/pci.c | 4 +- drivers/bus/pci/linux/pci_init.h | 4 +- drivers/bus/pci/linux/pci_vfio.c | 195 +++++++++++++++++++++++-------- drivers/bus/pci/private.h | 9 ++ lib/eal/include/rte_vfio.h | 1 - 5 files changed, 158 insertions(+), 55 deletions(-)
drivers/bus/pci/linux/pci.c +2 -2
@@ -645,7 +645,7 @@ int rte_pci_read_config(const struct rte_pci_device *device,return pci_uio_read_config (intr_handle, buf, len, offset);#ifdef VFIO_PRESENT case RTE_PCI_KDRV_VFIO : -return pci_vfio_read_config (intr_handle, buf, len, offset); +return pci_vfio_read_config (device, buf, len, offset);#endif default : rte_pci_device_name(&device->addr, devname,
@@ -669,7 +669,7 @@ int rte_pci_write_config(const struct rte_pci_device *device,return pci_uio_write_config (intr_handle, buf, len, offset);#ifdef VFIO_PRESENT case RTE_PCI_KDRV_VFIO : -return pci_vfio_write_config (intr_handle, buf, len, offset); +return pci_vfio_write_config (device, buf, len, offset);#endif default : rte_pci_device_name(&device->addr, devname,
drivers/bus/pci/linux/pci_init.h +2 -2
@@ -66,9 +66,9 @@ int pci_uio_ioport_unmap(struct rte_pci_ioport *p);#endif /* access config space */ -int pci_vfio_read_config (const struct rte_intr_handle *intr_handle, +int pci_vfio_read_config (const struct rte_pci_device *dev,void *buf,size_t len,off_t offs); -int pci_vfio_write_config (const struct rte_intr_handle *intr_handle, +int pci_vfio_write_config (const struct rte_pci_device *dev,const void *buf,size_t len,off_t offs); int pci_vfio_ioport_map(struct rte_pci_device *dev, int bar,
drivers/bus/pci/linux/pci_vfio.c +145 -50
363 lines not on this page · raw message
drivers/bus/pci/private.h +9 -0
@@ -13,6 +13,8 @@#include <rte_os_shim.h> #include <rte_pci.h> +#define RTE_MAX_PCI_REGIONS 9 +/* * Convert struct rte_pci_device to struct rte_pci_device_internal */
@@ -42,8 +44,15 @@ extern struct rte_pci_bus rte_pci_bus;struct rte_pci_driver ;struct rte_pci_device ; +struct rte_pci_region { +uint64_t size; +uint64_t offset; +}; +struct rte_pci_device_internal {struct rte_pci_device device; +/* PCI regions provided by e.g. VFIO. */ +struct rte_pci_region region[RTE_MAX_PCI_REGIONS ]; };/**
lib/eal/include/rte_vfio.h +0 -1
@@ -38,7 +38,6 @@ extern "C" {#define VFIO_CONTAINER_PATH "/dev/vfio/vfio"#define VFIO_GROUP_FMT "/dev/vfio/%u"#define VFIO_NOIOMMU_GROUP_FMT "/dev/vfio/noiommu-%u" -#define VFIO_GET_REGION_ADDR (x) ((uint64_t) x << 40ULL)#define VFIO_GET_REGION_IDX (x) (x >> 40)#define VFIO_NOIOMMU_MODE \ "/sys/module/vfio/parameters/enable_unsafe_noiommu_mode"
Signature · 1 line
2.17.1
RFC 3/4 bus/pci: introduce helper for MMIO read and write
In reply to Chenbo XiaThe MMIO regions may not be mmap-able for VFIO-PCI devices. In this case, the driver should explicitly do read and write to access these regions.
--- drivers/bus/pci/bsd/pci.c | 22 +++++++++++++++ drivers/bus/pci/linux/pci.c | 46 ++++++++++++++++++++++++++++++ drivers/bus/pci/linux/pci_init.h | 10 +++++++ drivers/bus/pci/linux/pci_uio.c | 22 +++++++++++++++ drivers/bus/pci/linux/pci_vfio.c | 36 ++++++++++++++++++++++++ drivers/bus/pci/rte_bus_pci.h | 48 ++++++++++++++++++++++++++++++++ drivers/bus/pci/version.map | 3 ++ 7 files changed, 187 insertions(+)
drivers/bus/pci/bsd/pci.c +22 -0
@@ -489,6 +489,28 @@ int rte_pci_write_config(const struct rte_pci_device *dev,return -1 ; } +/* Read PCI MMIO space. */ +int rte_pci_mmio_read (const struct rte_pci_device *dev,int bar, +void *buf,size_t len,off_t offset) +{ +if (bar >=PCI_MAX_RESOURCE || dev->mem_resource[bar].addr ==NULL || + (uint64_t )offset + len > dev->mem_resource[bar].len) +return -1 ; +memcpy (buf, (uint8_t *)dev->mem_resource[bar].addr + offset, len); +return len; +} + +/* Write PCI MMIO space. */ +int rte_pci_mmio_write (const struct rte_pci_device *dev,int bar, +const void *buf,size_t len,off_t offset) +{ +if (bar >=PCI_MAX_RESOURCE || dev->mem_resource[bar].addr ==NULL || + (uint64_t )offset + len > dev->mem_resource[bar].len) +return -1 ; +memcpy ((uint8_t *)dev->mem_resource[bar].addr + offset, buf, len); +return len; +} +int rte_pci_ioport_map (struct rte_pci_device *dev,int bar,struct rte_pci_ioport *p)
drivers/bus/pci/linux/pci.c +46 -0
52 lines not on this page · raw message
drivers/bus/pci/linux/pci_init.h +10 -0
@@ -37,6 +37,11 @@ int pci_uio_read_config(const struct rte_intr_handle *intr_handle,int pci_uio_write_config (const struct rte_intr_handle *intr_handle,const void *buf,size_t len,off_t offs); +int pci_uio_mmio_read (const struct rte_pci_device *dev,int bar, +void *buf,size_t len,off_t offset); +int pci_uio_mmio_write (const struct rte_pci_device *dev,int bar, +const void *buf,size_t len,off_t offset); +int pci_uio_ioport_map (struct rte_pci_device *dev,int bar,struct rte_pci_ioport *p); void pci_uio_ioport_read(struct rte_pci_ioport *p,
@@ -71,6 +76,11 @@ int pci_vfio_read_config(const struct rte_pci_device *dev,int pci_vfio_write_config (const struct rte_pci_device *dev,const void *buf,size_t len,off_t offs); +int pci_vfio_mmio_read (const struct rte_pci_device *dev,int bar, +void *buf,size_t len,off_t offset); +int pci_vfio_mmio_write (const struct rte_pci_device *dev,int bar, +const void *buf,size_t len,off_t offset); +int pci_vfio_ioport_map (struct rte_pci_device *dev,int bar,struct rte_pci_ioport *p); void pci_vfio_ioport_read(struct rte_pci_ioport *p,
drivers/bus/pci/linux/pci_uio.c +22 -0
@@ -55,6 +55,28 @@ pci_uio_write_config(const struct rte_intr_handle *intr_handle,return pwrite (uio_cfg_fd, buf, len, offset); } +int +pci_uio_mmio_read (const struct rte_pci_device *dev,int bar, +void *buf,size_t len,off_t offset) +{ +if (bar >=PCI_MAX_RESOURCE || dev->mem_resource[bar].addr ==NULL || + (uint64_t )offset + len > dev->mem_resource[bar].len) +return -1 ; +memcpy (buf, (uint8_t *)dev->mem_resource[bar].addr + offset, len); +return len; +} + +int +pci_uio_mmio_write (const struct rte_pci_device *dev,int bar, +const void *buf,size_t len,off_t offset) +{ +if (bar >=PCI_MAX_RESOURCE || dev->mem_resource[bar].addr ==NULL || + (uint64_t )offset + len > dev->mem_resource[bar].len) +return -1 ; +memcpy ((uint8_t *)dev->mem_resource[bar].addr + offset, buf, len); +return len; +} +static int pci_uio_set_bus_master (int dev_fd) {
drivers/bus/pci/linux/pci_vfio.c +36 -0
@@ -1258,6 +1258,42 @@ pci_vfio_ioport_unmap(struct rte_pci_ioport *p)return -1 ; } +int +pci_vfio_mmio_read (const struct rte_pci_device *dev,int bar, +void *buf,size_t len,off_t offs) +{ +uint64_t size, offset; +int fd; + + fd =rte_intr_dev_fd_get (dev->intr_handle); + +if (pci_vfio_get_region (dev, bar, &size, &offset) !=0 ) +return -1 ; + +if ((uint64_t )len + offs > size) +return -1 ; + +return pread64 (fd, buf, len, offset + offs); +} + +int +pci_vfio_mmio_write (const struct rte_pci_device *dev,int bar, +const void *buf,size_t len,off_t offs) +{ +uint64_t size, offset; +int fd; + + fd =rte_intr_dev_fd_get (dev->intr_handle); + +if (pci_vfio_get_region (dev, bar, &size, &offset) !=0 ) +return -1 ; + +if ((uint64_t )len + offs > size) +return -1 ; + +return pwrite64 (fd, buf, len, offset + offs); +} +int pci_vfio_is_enabled (void ) {
drivers/bus/pci/rte_bus_pci.h +48 -0
54 lines not on this page · raw message
drivers/bus/pci/version.map +3 -0
@@ -21,6 +21,9 @@ EXPERIMENTAL { # added in 21.08 rte_pci_set_bus_master; + # added in 23.07 + rte_pci_mmio_read; + rte_pci_mmio_write; }; INTERNAL {
Signature · 1 line
2.17.1
RFC 4/4 bus/pci: add VFIO sparse mmap support
In reply to Chenbo XiaThis patch adds sparse mmap support in PCI bus. Sparse mmap is a capability defined in VFIO which allows multiple mmap areas in one VFIO region.
--- drivers/baseband/acc/rte_acc100_pmd.c | 6 +- drivers/baseband/acc/rte_vrb_pmd.c | 6 +- .../fpga_5gnr_fec/rte_fpga_5gnr_fec.c | 6 +- drivers/baseband/fpga_lte_fec/fpga_lte_fec.c | 6 +- drivers/bus/pci/bsd/pci.c | 20 +- drivers/bus/pci/bus_pci_driver.h | 24 +- drivers/bus/pci/linux/pci.c | 13 +- drivers/bus/pci/linux/pci_uio.c | 24 +- drivers/bus/pci/linux/pci_vfio.c | 214 +++++++++++++++--- drivers/bus/pci/pci_common.c | 45 ++-- drivers/bus/pci/pci_common_uio.c | 12 +- drivers/bus/pci/private.h | 2 + drivers/common/cnxk/roc_dev.c | 4 +- drivers/common/cnxk/roc_dpi.c | 2 +- drivers/common/cnxk/roc_ml.c | 22 +- drivers/common/qat/dev/qat_dev_gen1.c | 2 +- drivers/common/qat/dev/qat_dev_gen4.c | 4 +- drivers/common/sfc_efx/sfc_efx.c | 2 +- drivers/compress/octeontx/otx_zip.c | 4 +- drivers/crypto/ccp/ccp_dev.c | 4 +- drivers/crypto/cnxk/cnxk_cryptodev_ops.c | 2 +- drivers/crypto/nitrox/nitrox_device.c | 4 +- drivers/crypto/octeontx/otx_cryptodev_ops.c | 6 +- drivers/crypto/virtio/virtio_pci.c | 6 +- drivers/dma/cnxk/cnxk_dmadev.c | 2 +- drivers/dma/hisilicon/hisi_dmadev.c | 6 +- drivers/dma/idxd/idxd_pci.c | 4 +- drivers/dma/ioat/ioat_dmadev.c | 2 +- drivers/event/dlb2/pf/dlb2_main.c | 16 +- drivers/event/octeontx/ssovf_probe.c | 38 ++-- drivers/event/octeontx/timvf_probe.c | 18 +- drivers/event/skeleton/skeleton_eventdev.c | 2 +- drivers/mempool/octeontx/octeontx_fpavf.c | 6 +- drivers/net/ark/ark_ethdev.c | 4 +- drivers/net/atlantic/atl_ethdev.c | 2 +- drivers/net/avp/avp_ethdev.c | 20 +- drivers/net/axgbe/axgbe_ethdev.c | 4 +- drivers/net/bnx2x/bnx2x_ethdev.c | 6 +- drivers/net/bnxt/bnxt_ethdev.c | 8 +- drivers/net/cpfl/cpfl_ethdev.c | 4 +- drivers/net/cxgbe/cxgbe_ethdev.c | 2 +- drivers/net/cxgbe/cxgbe_main.c | 2 +- drivers/net/cxgbe/cxgbevf_ethdev.c | 2 +- drivers/net/cxgbe/cxgbevf_main.c | 2 +- drivers/net/e1000/em_ethdev.c | 4 +- drivers/net/e1000/igb_ethdev.c | 4 +- drivers/net/ena/ena_ethdev.c | 4 +- drivers/net/enetc/enetc_ethdev.c | 2 +- drivers/net/enic/enic_main.c | 4 +- drivers/net/fm10k/fm10k_ethdev.c | 2 +- drivers/net/gve/gve_ethdev.c | 4 +- drivers/net/hinic/base/hinic_pmd_hwif.c | 14 +- drivers/net/hns3/hns3_ethdev.c | 2 +- drivers/net/hns3/hns3_ethdev_vf.c | 2 +- drivers/net/hns3/hns3_rxtx.c | 4 +- drivers/net/i40e/i40e_ethdev.c | 2 +- drivers/net/iavf/iavf_ethdev.c | 2 +- drivers/net/ice/ice_dcf.c | 2 +- drivers/net/ice/ice_ethdev.c | 2 +- drivers/net/idpf/idpf_ethdev.c | 4 +- drivers/net/igc/igc_ethdev.c | 2 +- drivers/net/ionic/ionic_dev_pci.c | 2 +- drivers/net/ixgbe/ixgbe_ethdev.c | 4 +- drivers/net/liquidio/lio_ethdev.c | 4 +- drivers/net/nfp/nfp_ethdev.c | 2 +- drivers/net/nfp/nfp_ethdev_vf.c | 6 +- drivers/net/nfp/nfpcore/nfp_cpp_pcie_ops.c | 4 +- drivers/net/ngbe/ngbe_ethdev.c | 2 +- drivers/net/octeon_ep/otx_ep_ethdev.c | 2 +- drivers/net/octeontx/base/octeontx_pkivf.c | 6 +- drivers/net/octeontx/base/octeontx_pkovf.c | 12 +- drivers/net/qede/qede_main.c | 6 +- drivers/net/sfc/sfc.c | 2 +- drivers/net/thunderx/nicvf_ethdev.c | 2 +- drivers/net/txgbe/txgbe_ethdev.c | 2 +- drivers/net/txgbe/txgbe_ethdev_vf.c | 2 +- drivers/net/virtio/virtio_pci.c | 6 +- drivers/net/vmxnet3/vmxnet3_ethdev.c | 4 +- drivers/raw/cnxk_bphy/cnxk_bphy.c | 10 +- drivers/raw/cnxk_bphy/cnxk_bphy_cgx.c | 6 +- drivers/raw/ifpga/afu_pmd_n3000.c | 4 +- drivers/raw/ifpga/ifpga_rawdev.c | 6 +- drivers/raw/ntb/ntb_hw_intel.c | 8 +- drivers/vdpa/ifc/ifcvf_vdpa.c | 6 +- drivers/vdpa/sfc/sfc_vdpa_hw.c | 2 +- drivers/vdpa/sfc/sfc_vdpa_ops.c | 2 +- 86 files changed, 477 insertions(+), 285 deletions(-)
drivers/baseband/acc/rte_acc100_pmd.c +3 -3
@@ -4260,12 +4260,12 @@ acc100_bbdev_init(struct rte_bbdev *dev, struct rte_pci_driver *drv) !strcmp (drv->driver.name,RTE_STR (ACC100PF_DRIVER_NAME )); ((struct acc_device *) dev->data->dev_private)->mmio_base = - pci_dev->mem_resource[0 ].addr; + pci_dev->pci_mem[0 ].mem_res.addr;rte_bbdev_log_debug ("Init device %s [%s] @ vaddr %p paddr %#" PRIx64"" , drv->driver.name, dev->data->name, - (void *)pci_dev->mem_resource[0 ].addr, - pci_dev->mem_resource[0 ].phys_addr); + (void *)pci_dev->pci_mem[0 ].mem_res.addr, + pci_dev->pci_mem[0 ].mem_res.phys_addr); } static int acc100_pci_probe(struct rte_pci_driver *pci_drv,
drivers/baseband/acc/rte_vrb_pmd.c +3 -3
@@ -3349,7 +3349,7 @@ vrb_bbdev_init(struct rte_bbdev *dev, struct rte_pci_driver *drv) dev->dequeue_fft_ops = vrb_dequeue_fft; d->pf_device = !strcmp (drv->driver.name,RTE_STR (VRB_PF_DRIVER_NAME )); - d->mmio_base = pci_dev->mem_resource[0 ].addr; + d->mmio_base = pci_dev->pci_mem[0 ].mem_res.addr;/* Device variant specific handling. */ if ((pci_dev->id.device_id == RTE_VRB1_PF_DEVICE_ID) ||
@@ -3367,8 +3367,8 @@ vrb_bbdev_init(struct rte_bbdev *dev, struct rte_pci_driver *drv)rte_bbdev_log_debug ("Init device %s [%s] @ vaddr %p paddr %#" PRIx64"" , drv->driver.name, dev->data->name, - (void *)pci_dev->mem_resource[0 ].addr, - pci_dev->mem_resource[0 ].phys_addr); + (void *)pci_dev->pci_mem[0 ].mem_res.addr, + pci_dev->pci_mem[0 ].mem_res.phys_addr); } static int vrb_pci_probe(struct rte_pci_driver *pci_drv,
drivers/baseband/fpga_5gnr_fec/rte_fpga_5gnr_fec.c +3 -3
@@ -2149,13 +2149,13 @@ fpga_5gnr_fec_init(struct rte_bbdev *dev, struct rte_pci_driver *drv) !strcmp (drv->driver.name,RTE_STR (FPGA_5GNR_FEC_PF_DRIVER_NAME )); ((struct fpga_5gnr_fec_device *) dev->data->dev_private)->mmio_base = - pci_dev->mem_resource[0 ].addr; + pci_dev->pci_mem[0 ].mem_res.addr;rte_bbdev_log_debug ("Init device %s [%s] @ virtaddr %p phyaddr %#" PRIx64, drv->driver.name, dev->data->name, - (void *)pci_dev->mem_resource[0 ].addr, - pci_dev->mem_resource[0 ].phys_addr); + (void *)pci_dev->pci_mem[0 ].mem_res.addr, + pci_dev->pci_mem[0 ].mem_res.phys_addr); } static int
drivers/baseband/fpga_lte_fec/fpga_lte_fec.c +3 -3
@@ -2326,13 +2326,13 @@ fpga_lte_fec_init(struct rte_bbdev *dev, struct rte_pci_driver *drv) !strcmp (drv->driver.name,RTE_STR (FPGA_LTE_FEC_PF_DRIVER_NAME )); ((struct fpga_lte_fec_device *) dev->data->dev_private)->mmio_base = - pci_dev->mem_resource[0 ].addr; + pci_dev->pci_mem[0 ].mem_res.addr;rte_bbdev_log_debug ("Init device %s [%s] @ virtaddr %p phyaddr %#" PRIx64, drv->driver.name, dev->data->name, - (void *)pci_dev->mem_resource[0 ].addr, - pci_dev->mem_resource[0 ].phys_addr); + (void *)pci_dev->pci_mem[0 ].mem_res.addr, + pci_dev->pci_mem[0 ].mem_res.phys_addr); } static int
drivers/bus/pci/bsd/pci.c +10 -10
@@ -186,17 +186,17 @@ pci_uio_map_resource_by_index(struct rte_pci_device *dev, int res_idx,/* if matching map is found, then use it */ offset = res_idx * pagesz; mapaddr =pci_map_resource (NULL , fd, (off_t )offset, - (size_t )dev->mem_resource[res_idx].len,0 ); + (size_t )dev->pci_mem[res_idx].mem_res.len,0 );close (fd);if (mapaddr ==NULL ) goto error; - maps[map_idx].phaddr = dev->mem_resource[res_idx].phys_addr; - maps[map_idx].size = dev->mem_resource[res_idx].len; + maps[map_idx].phaddr = dev->pci_mem[res_idx].mem_res.phys_addr; + maps[map_idx].size = dev->pci_mem[res_idx].mem_res.len; maps[map_idx].addr = mapaddr; maps[map_idx].offset = offset;strcpy (maps[map_idx].path, devname); - dev->mem_resource[res_idx].addr = mapaddr; + dev->pci_mem[res_idx].mem_res.addr = mapaddr;return 0 ;
@@ -493,10 +493,10 @@ int rte_pci_write_config(const struct rte_pci_device *dev,int rte_pci_mmio_read (const struct rte_pci_device *dev,int bar,void *buf,size_t len,off_t offset) { -if (bar >=PCI_MAX_RESOURCE || dev->mem_resource[bar].addr ==NULL || - (uint64_t )offset + len > dev->mem_resource[bar].len) +if (bar >=PCI_MAX_RESOURCE || dev->pci_mem[bar].mem_res.addr ==NULL || + (uint64_t )offset + len > dev->pci_mem[bar].mem_res.len)return -1 ; -memcpy (buf, (uint8_t *)dev->mem_resource[bar].addr + offset, len); +memcpy (buf, (uint8_t *)dev->pci_mem[bar].mem_res.addr + offset, len);return len; }
@@ -504,10 +504,10 @@ int rte_pci_mmio_read(const struct rte_pci_device *dev, int bar,int rte_pci_mmio_write (const struct rte_pci_device *dev,int bar,const void *buf,size_t len,off_t offset) { -if (bar >=PCI_MAX_RESOURCE || dev->mem_resource[bar].addr ==NULL || - (uint64_t )offset + len > dev->mem_resource[bar].len) +if (bar >=PCI_MAX_RESOURCE || dev->pci_mem[bar].mem_res.addr ==NULL || + (uint64_t )offset + len > dev->pci_mem[bar].mem_res.len)return -1 ; -memcpy ((uint8_t *)dev->mem_resource[bar].addr + offset, buf, len); +memcpy ((uint8_t *)dev->pci_mem[bar].mem_res.addr + offset, buf, len);return len; }
drivers/bus/pci/bus_pci_driver.h +22 -2
@@ -28,6 +28,27 @@ enum rte_pci_kernel_driver { RTE_PCI_KDRV_NET_UIO, /* NetUIO for Windows */ }; +struct rte_mem_map_area { +void *addr; +uint64_t offset; +uint64_t size; +}; + +struct rte_sparse_mem_map { +uint64_t phys_addr; +uint64_t len; +uint32_t nr_maps; +struct rte_mem_map_area *areas; +}; + +struct rte_pci_mem_resource { +bool is_sparse; +union { +struct rte_mem_resource mem_res; +struct rte_sparse_mem_map sparse_mem; + }; +}; +/** * A structure describing a PCI device. */
@@ -36,8 +57,7 @@ struct rte_pci_device {struct rte_device device;/**< Inherit core device */ struct rte_pci_addr addr;/**< PCI location. */ struct rte_pci_id id;/**< PCI ID. */ -struct rte_mem_resource mem_resource[PCI_MAX_RESOURCE ]; -/**< PCI Memory Resource */ +struct rte_pci_mem_resource pci_mem[PCI_MAX_RESOURCE ];/**< PCI Memory Resource */ struct rte_intr_handle *intr_handle;/**< Interrupt handle */ struct rte_pci_driver *driver;/**< PCI driver used in probing */ uint16_t max_vfs;/**< sriov enable if not zero */
drivers/bus/pci/linux/pci.c +6 -7
@@ -179,7 +179,7 @@ pci_parse_sysfs_resource(const char *filename, struct rte_pci_device *dev)return -1 ; } -for (i =0 ; i<PCI_MAX_RESOURCE ; i++) { +for (i =0 ; i <PCI_MAX_RESOURCE ; i++) {if (fgets (buf,sizeof (buf), f) ==NULL ) { RTE_LOG(ERR, EAL,
@@ -191,10 +191,10 @@ pci_parse_sysfs_resource(const char *filename, struct rte_pci_device *dev) goto error;if (flags &IORESOURCE_MEM ) { - dev->mem_resource[i].phys_addr = phys_addr; - dev->mem_resource[i].len = end_addr - phys_addr +1 ; + dev->pci_mem[i].mem_res.phys_addr = phys_addr; + dev->pci_mem[i].mem_res.len = end_addr - phys_addr +1 ;/* not mapped for now */ - dev->mem_resource[i].addr =NULL ; + dev->pci_mem[i].mem_res.addr =NULL ; } }fclose (f);
@@ -347,9 +347,8 @@ pci_scan_one(const char *dirname, const struct rte_pci_addr *addr) dev2->max_vfs = dev->max_vfs; dev2->id = dev->id;pci_common_set (dev2); -memmove (dev2->mem_resource, - dev->mem_resource, -sizeof (dev->mem_resource)); +memmove (dev2->pci_mem, dev->pci_mem, +sizeof (dev->pci_mem)); }else {/** * If device is plugged and driver is
drivers/bus/pci/linux/pci_uio.c +12 -12
@@ -59,10 +59,10 @@ int pci_uio_mmio_read(const struct rte_pci_device *dev, int bar, void *buf, size_t len, off_t offset) { - if (bar >= PCI_MAX_RESOURCE || dev->mem_resource[bar].addr == NULL || - (uint64_t)offset + len > dev->mem_resource[bar].len) + if (bar >= PCI_MAX_RESOURCE || dev->pci_mem[bar].mem_res.addr == NULL || + (uint64_t)offset + len > dev->pci_mem[bar].mem_res.len) return -1; - memcpy(buf, (uint8_t *)dev->mem_resource[bar].addr + offset, len); + memcpy(buf, (uint8_t *)dev->pci_mem[bar].mem_res.addr + offset, len);return len; }
@@ -70,10 +70,10 @@ int pci_uio_mmio_write(const struct rte_pci_device *dev, int bar, const void *buf, size_t len, off_t offset) { - if (bar >= PCI_MAX_RESOURCE || dev->mem_resource[bar].addr == NULL || - (uint64_t)offset + len > dev->mem_resource[bar].len) + if (bar >= PCI_MAX_RESOURCE || dev->pci_mem[bar].mem_res.addr == NULL || + (uint64_t)offset + len > dev->pci_mem[bar].mem_res.len) return -1; - memcpy((uint8_t *)dev->mem_resource[bar].addr + offset, buf, len); + memcpy((uint8_t *)dev->pci_mem[bar].mem_res.addr + offset, buf, len);return len; }
@@ -388,22 +388,22 @@ pci_uio_map_resource_by_index(struct rte_pci_device *dev, int res_idx, pci_map_addr =pci_find_max_end_va (); mapaddr =pci_map_resource (pci_map_addr, fd,0 , - (size_t )dev->mem_resource[res_idx].len,0 ); + (size_t )dev->pci_mem[res_idx].mem_res.len,0 );close (fd);if (mapaddr ==NULL ) goto error; pci_map_addr =RTE_PTR_ADD (mapaddr, - (size_t )dev->mem_resource[res_idx].len); + (size_t )dev->pci_mem[res_idx].mem_res.len); pci_map_addr =RTE_PTR_ALIGN (pci_map_addr,sysconf (_SC_PAGE_SIZE)); - maps[map_idx].phaddr = dev->mem_resource[res_idx].phys_addr; - maps[map_idx].size = dev->mem_resource[res_idx].len; + maps[map_idx].phaddr = dev->pci_mem[res_idx].mem_res.phys_addr; + maps[map_idx].size = dev->pci_mem[res_idx].mem_res.len; maps[map_idx].addr = mapaddr; maps[map_idx].offset =0 ;strcpy (maps[map_idx].path, devname); - dev->mem_resource[res_idx].addr = mapaddr; + dev->pci_mem[res_idx].mem_res.addr = mapaddr;return 0 ;
@@ -463,7 +463,7 @@ pci_uio_ioport_map(struct rte_pci_device *dev, int bar,RTE_LOG (DEBUG ,EAL ,"%s(): PIO BAR %08lx detected\n" , __func__, base); }else if (flags &IORESOURCE_MEM ) { - base = (unsigned long )dev->mem_resource[bar].addr; + base = (unsigned long )dev->pci_mem[bar].mem_res.addr;RTE_LOG (DEBUG ,EAL ,"%s(): MMIO BAR %08lx detected\n" , __func__, base); }else {RTE_LOG (ERR ,EAL ,"%s(): unknown BAR type\n" , __func__);
drivers/bus/pci/linux/pci_vfio.c +182 -32
275 lines not on this page · raw message
drivers/bus/pci/pci_common.c +31 -14
62 lines not on this page · raw message
drivers/bus/pci/pci_common_uio.c +7 -5
@@ -71,7 +71,8 @@ pci_uio_map_secondary(struct rte_pci_device *dev) }return -1 ; } - dev->mem_resource[i].addr = mapaddr; + dev->pci_mem[i].is_sparse = false; + dev->pci_mem[i].mem_res.addr = mapaddr; }return 0 ; }
@@ -108,7 +109,8 @@ pci_uio_map_resource(struct rte_pci_device *dev)/* Map all BARs */ for (i =0 ; i !=PCI_MAX_RESOURCE ; i++) {/* skip empty BAR */ - phaddr = dev->mem_resource[i].phys_addr; + dev->pci_mem[i].is_sparse = false; + phaddr = dev->pci_mem[i].mem_res.phys_addr;if (phaddr ==0 )continue ;
@@ -164,10 +166,10 @@ pci_uio_remap_resource(struct rte_pci_device *dev)/* Remap all BARs */ for (i =0 ; i !=PCI_MAX_RESOURCE ; i++) {/* skip empty BAR */ -if (dev->mem_resource[i].phys_addr ==0 ) +if (dev->pci_mem[i].mem_res.phys_addr ==0 )continue ; - map_address =mmap (dev->mem_resource[i].addr, - (size_t )dev->mem_resource[i].len, + map_address =mmap (dev->pci_mem[i].mem_res.addr, + (size_t )dev->pci_mem[i].mem_res.len,PROT_READ |PROT_WRITE ,MAP_PRIVATE |MAP_ANONYMOUS |MAP_FIXED ,-1 ,0 );if (map_address ==MAP_FAILED ) {
drivers/bus/pci/private.h +2 -0
@@ -121,6 +121,8 @@ struct pci_map {uint64_t offset;uint64_t size;uint64_t phaddr; +uint32_t nr_areas; +struct rte_mem_map_area *areas; };struct pci_msix_table {
drivers/common/cnxk/roc_dev.c +2 -2
@@ -1151,8 +1151,8 @@ dev_init(struct dev *dev, struct plt_pci_device *pci_dev)if (!dev_cache_line_size_valid ())return -EFAULT ; - bar2 = (uintptr_t )pci_dev->mem_resource[2 ].addr; - bar4 = (uintptr_t )pci_dev->mem_resource[4 ].addr; + bar2 = (uintptr_t )pci_dev->pci_mem[2 ].mem_res.addr; + bar4 = (uintptr_t )pci_dev->pci_mem[4 ].mem_res.addr;if (bar2 ==0 || bar4 ==0 ) {plt_err ("Failed to get PCI bars" ); rc = -ENODEV ;
drivers/common/cnxk/roc_dpi.c +1 -1
@@ -152,7 +152,7 @@ roc_dpi_dev_init(struct roc_dpi *roc_dpi)struct plt_pci_device *pci_dev = roc_dpi->pci_dev;uint16_t vfid; - roc_dpi->rbase = pci_dev->mem_resource[0 ].addr; + roc_dpi->rbase = pci_dev->pci_mem[0 ].mem_res.addr; vfid = ((pci_dev->addr.devid &0x1F ) <<3 ) | (pci_dev->addr.function &0x7 ); vfid -=1 ;
drivers/common/cnxk/roc_ml.c +12 -10
@@ -100,9 +100,9 @@ roc_ml_addr_pa_to_offset(struct roc_ml *roc_ml, uint64_t phys_addr)struct ml *ml =roc_ml_to_ml_priv (roc_ml);if (roc_model_is_cn10ka ()) -return phys_addr - ml->pci_dev->mem_resource[0 ].phys_addr; +return phys_addr - ml->pci_dev->pci_mem[0 ].mem_res.phys_addr;else -return phys_addr - ml->pci_dev->mem_resource[0 ].phys_addr -ML_MLAB_BLK_OFFSET ; +return phys_addr - ml->pci_dev->pci_mem[0 ].mem_res.phys_addr -ML_MLAB_BLK_OFFSET ; }uint64_t
@@ -111,9 +111,9 @@ roc_ml_addr_offset_to_pa(struct roc_ml *roc_ml, uint64_t offset)struct ml *ml =roc_ml_to_ml_priv (roc_ml);if (roc_model_is_cn10ka ()) -return ml->pci_dev->mem_resource[0 ].phys_addr + offset; +return ml->pci_dev->pci_mem[0 ].mem_res.phys_addr + offset;else -return ml->pci_dev->mem_resource[0 ].phys_addr +ML_MLAB_BLK_OFFSET + offset; +return ml->pci_dev->pci_mem[0 ].mem_res.phys_addr +ML_MLAB_BLK_OFFSET + offset; }void
@@ -543,13 +543,14 @@ roc_ml_dev_init(struct roc_ml *roc_ml) ml->pci_dev = pci_dev; dev->roc_ml = roc_ml; - ml->ml_reg_addr = ml->pci_dev->mem_resource[0 ].addr; + ml->ml_reg_addr = ml->pci_dev->pci_mem[0 ].mem_res.addr; ml->ml_mlr_base =0 ; ml->ml_mlr_base_saved = false; -plt_ml_dbg ("ML: PCI Physical Address : 0x%016lx" , ml->pci_dev->mem_resource[0 ].phys_addr); +plt_ml_dbg ("ML: PCI Physical Address : 0x%016lx" , + ml->pci_dev->pci_mem[0 ].mem_res.phys_addr);plt_ml_dbg ("ML: PCI Virtual Address : 0x%016lx" , -PLT_U64_CAST (ml->pci_dev->mem_resource[0 ].addr)); +PLT_U64_CAST (ml->pci_dev->pci_mem[0 ].mem_res.addr));plt_spinlock_init (&roc_ml->sp_spinlock);plt_spinlock_init (&roc_ml->fp_spinlock);
@@ -589,11 +590,12 @@ roc_ml_blk_init(struct roc_bphy *roc_bphy, struct roc_ml *roc_ml)plt_ml_dbg ("MLAB: Physical Address : 0x%016lx" , -PLT_PTR_ADD_U64_CAST (ml->pci_dev->mem_resource[0 ].phys_addr,ML_MLAB_BLK_OFFSET )); +PLT_PTR_ADD_U64_CAST (ml->pci_dev->pci_mem[0 ].mem_res.phys_addr, +ML_MLAB_BLK_OFFSET ));plt_ml_dbg ("MLAB: Virtual Address : 0x%016lx" , -PLT_PTR_ADD_U64_CAST (ml->pci_dev->mem_resource[0 ].addr,ML_MLAB_BLK_OFFSET )); +PLT_PTR_ADD_U64_CAST (ml->pci_dev->pci_mem[0 ].mem_res.addr,ML_MLAB_BLK_OFFSET )); - ml->ml_reg_addr =PLT_PTR_ADD (ml->pci_dev->mem_resource[0 ].addr,ML_MLAB_BLK_OFFSET ); + ml->ml_reg_addr =PLT_PTR_ADD (ml->pci_dev->pci_mem[0 ].mem_res.addr,ML_MLAB_BLK_OFFSET ); ml->ml_mlr_base =0 ; ml->ml_mlr_base_saved = false;
drivers/common/qat/dev/qat_dev_gen1.c +1 -1
@@ -214,7 +214,7 @@ qat_reset_ring_pairs_gen1(struct qat_pci_device *qat_pci_dev __rte_unused)const struct rte_mem_resource *qat_dev_get_transport_bar_gen1 (struct rte_pci_device *pci_dev) { -return &pci_dev->mem_resource[0 ]; +return &pci_dev->pci_mem[0 ].mem_res; }int
drivers/common/qat/dev/qat_dev_gen4.c +2 -2
@@ -271,14 +271,14 @@ qat_reset_ring_pairs_gen4(struct qat_pci_device *qat_pci_dev)static const struct rte_mem_resource *qat_dev_get_transport_bar_gen4 (struct rte_pci_device *pci_dev) { -return &pci_dev->mem_resource[0 ]; +return &pci_dev->pci_mem[0 ].mem_res; }static int qat_dev_get_misc_bar_gen4 (struct rte_mem_resource **mem_resource,struct rte_pci_device *pci_dev) { - *mem_resource = &pci_dev->mem_resource[2 ]; + *mem_resource = &pci_dev->pci_mem[2 ].mem_res;return 0 ; }
drivers/common/sfc_efx/sfc_efx.c +1 -1
@@ -77,7 +77,7 @@ sfc_efx_find_mem_bar(efsys_pci_config_t *configp, int bar_index, result.esb_rid = bar_index; result.esb_dev = dev; - result.esb_base = dev->mem_resource[bar_index].addr; + result.esb_base = dev->pci_mem[bar_index].mem_res.addr; *barp = result;
drivers/compress/octeontx/otx_zip.c +2 -2
@@ -149,10 +149,10 @@ zipvf_create(struct rte_compressdev *compressdev)void *vbar0;uint64_t reg; -if (pdev->mem_resource[0 ].phys_addr ==0ULL ) +if (pdev->pci_mem[0 ].mem_res.phys_addr ==0ULL )return -EIO ; - vbar0 = pdev->mem_resource[0 ].addr; + vbar0 = pdev->pci_mem[0 ].mem_res.addr;if (!vbar0) {ZIP_PMD_ERR ("Failed to map BAR0 of %s" , dev_name);return -ENODEV ;
drivers/crypto/ccp/ccp_dev.c +2 -2
@@ -67,7 +67,7 @@ ccp_read_hwrng(uint32_t *value)struct ccp_device *dev; TAILQ_FOREACH(dev, &ccp_list, next) { -void *vaddr = (void *)(dev->pci->mem_resource[2 ].addr); +void *vaddr = (void *)(dev->pci->pci_mem[2 ].mem_res.addr);while (dev->hwrng_retries++ <CCP_MAX_TRNG_RETRIES ) { *value =CCP_READ_REG (vaddr,TRNG_OUT_REG );
@@ -493,7 +493,7 @@ ccp_add_device(struct ccp_device *dev) dev->id = ccp_dev_id++; dev->qidx =0 ; - vaddr = (void *)(dev->pci->mem_resource[2 ].addr); + vaddr = (void *)(dev->pci->pci_mem[2 ].mem_res.addr);if (dev->pci->id.device_id ==AMD_PCI_CCP_5B ) {CCP_WRITE_REG (vaddr,CMD_TRNG_CTL_OFFSET ,0x00012D57 );
drivers/crypto/cnxk/cnxk_cryptodev_ops.c +1 -1
@@ -400,7 +400,7 @@ cnxk_cpt_queue_pair_setup(struct rte_cryptodev *dev, uint16_t qp_id, pci_dev =RTE_DEV_TO_PCI (dev->device); -if (pci_dev->mem_resource[2 ].addr ==NULL ) { +if (pci_dev->pci_mem[2 ].mem_res.addr ==NULL ) {plt_err ("Invalid PCI mem address" );return -EIO ; }
drivers/crypto/nitrox/nitrox_device.c +2 -2
@@ -35,7 +35,7 @@ ndev_init(struct nitrox_device *ndev, struct rte_pci_device *pdev)enum nitrox_vf_mode vf_mode; ndev->pdev = pdev; - ndev->bar_addr = pdev->mem_resource[0 ].addr; + ndev->bar_addr = pdev->pci_mem[0 ].mem_res.addr; vf_mode =vf_get_vf_config_mode (ndev->bar_addr); ndev->nr_queues =vf_config_mode_to_nr_queues (vf_mode); }
@@ -70,7 +70,7 @@ nitrox_pci_probe(struct rte_pci_driver *pci_drv __rte_unused,int err;/* Nitrox CSR space */ -if (!pdev->mem_resource[0 ].addr) +if (!pdev->pci_mem[0 ].mem_res.addr)return -EINVAL ; ndev =ndev_allocate (pdev);
drivers/crypto/octeontx/otx_cryptodev_ops.c +3 -3
@@ -157,7 +157,7 @@ otx_cpt_que_pair_setup(struct rte_cryptodev *dev, pci_dev =RTE_DEV_TO_PCI (dev->device); -if (pci_dev->mem_resource[0 ].addr ==NULL ) { +if (pci_dev->pci_mem[0 ].mem_res.addr ==NULL ) {CPT_LOG_ERR ("PCI mem address null" );return -EIO ; }
@@ -1004,7 +1004,7 @@ otx_cpt_dev_create(struct rte_cryptodev *c_dev)char dev_name[32 ];int ret; -if (pdev->mem_resource[0 ].phys_addr ==0ULL ) +if (pdev->pci_mem[0 ].mem_res.phys_addr ==0ULL )return -EIO ;/* for secondary processes, we don't initialise any further as primary
@@ -1025,7 +1025,7 @@ otx_cpt_dev_create(struct rte_cryptodev *c_dev)snprintf (dev_name,32 ,"%02x:%02x.%x" , pdev->addr.bus, pdev->addr.devid, pdev->addr.function); - reg_base = pdev->mem_resource[0 ].addr; + reg_base = pdev->pci_mem[0 ].mem_res.addr;if (!reg_base) {CPT_LOG_ERR ("Failed to map BAR0 of %s" , dev_name); ret = -ENODEV ;
drivers/crypto/virtio/virtio_pci.c +3 -3
@@ -322,14 +322,14 @@ get_cfg_addr(struct rte_pci_device *dev, struct virtio_pci_cap *cap)return NULL ; } -if (offset + length > dev->mem_resource[bar].len) { +if (offset + length > dev->pci_mem[bar].mem_res.len) {VIRTIO_CRYPTO_INIT_LOG_ERR ("invalid cap: overflows bar space: %u > %" PRIu64, - offset + length, dev->mem_resource[bar].len); + offset + length, dev->pci_mem[bar].mem_res.len);return NULL ; } - base = dev->mem_resource[bar].addr; + base = dev->pci_mem[bar].mem_res.addr;if (base ==NULL ) {VIRTIO_CRYPTO_INIT_LOG_ERR ("bar %u base addr is NULL" , bar);return NULL ;
drivers/dma/cnxk/cnxk_dmadev.c +1 -1
@@ -637,7 +637,7 @@ cnxk_dmadev_probe(struct rte_pci_driver *pci_drv __rte_unused,struct roc_dpi *rdpi =NULL ;int rc; -if (!pci_dev->mem_resource[0 ].addr) +if (!pci_dev->pci_mem[0 ].mem_res.addr)return -ENODEV ; rc =roc_plt_init ();
drivers/dma/hisilicon/hisi_dmadev.c +3 -3
@@ -894,7 +894,7 @@ hisi_dma_create(struct rte_pci_device *pci_dev, uint8_t queue_id, hw->data = dev->data; hw->revision = revision; hw->reg_layout =hisi_dma_reg_layout (revision); - hw->io_base = pci_dev->mem_resource[REG_PCI_BAR_INDEX ].addr; + hw->io_base = pci_dev->pci_mem[REG_PCI_BAR_INDEX ].mem_res.addr; hw->queue_id = queue_id; hw->sq_tail_reg =hisi_dma_queue_regaddr (hw,HISI_DMA_QUEUE_SQ_TAIL_REG );
@@ -950,7 +950,7 @@ hisi_dma_probe(struct rte_pci_driver *pci_drv __rte_unused,rte_pci_device_name (&pci_dev->addr, name,sizeof (name)); -if (pci_dev->mem_resource[2 ].addr ==NULL ) { +if (pci_dev->pci_mem[2 ].mem_res.addr ==NULL ) {HISI_DMA_LOG (ERR ,"%s BAR2 is NULL!\n" , name);return -ENODEV ; }
@@ -961,7 +961,7 @@ hisi_dma_probe(struct rte_pci_driver *pci_drv __rte_unused,HISI_DMA_LOG (DEBUG ,"%s read PCI revision: 0x%x" , name, revision);if (rte_eal_process_type () ==RTE_PROC_PRIMARY ) -hisi_dma_init_gbl (pci_dev->mem_resource[2 ].addr, revision); +hisi_dma_init_gbl (pci_dev->pci_mem[2 ].mem_res.addr, revision);for (i =0 ; i <HISI_DMA_MAX_HW_QUEUES ; i++) { ret =hisi_dma_create (pci_dev, i, revision);
drivers/dma/idxd/idxd_pci.c +2 -2
@@ -188,12 +188,12 @@ init_pci_device(struct rte_pci_device *dev, struct idxd_dmadev *idxd,rte_spinlock_init (&pci->lk);/* assign the bar registers, and then configure device */ - pci->regs = dev->mem_resource[0 ].addr; + pci->regs = dev->pci_mem[0 ].mem_res.addr; grp_offset = (uint16_t )pci->regs->offsets[0 ]; pci->grp_regs =RTE_PTR_ADD (pci->regs, grp_offset *0x100 ); wq_offset = (uint16_t )(pci->regs->offsets[0 ] >>16 ); pci->wq_regs_base =RTE_PTR_ADD (pci->regs, wq_offset *0x100 ); - pci->portals = dev->mem_resource[2 ].addr; + pci->portals = dev->pci_mem[2 ].mem_res.addr; pci->wq_cfg_sz = (pci->regs->wqcap >>24 ) &0x0F ;/* sanity check device status */
drivers/dma/ioat/ioat_dmadev.c +1 -1
@@ -644,7 +644,7 @@ ioat_dmadev_create(const char *name, struct rte_pci_device *dev) ioat = dmadev->data->dev_private; ioat->dmadev = dmadev; - ioat->regs = dev->mem_resource[0 ].addr; + ioat->regs = dev->pci_mem[0 ].mem_res.addr; ioat->doorbell = &ioat->regs->dmacount; ioat->qcfg.nb_desc =0 ; ioat->desc_ring =NULL ;
drivers/event/dlb2/pf/dlb2_main.c +8 -8
@@ -170,32 +170,32 @@ dlb2_probe(struct rte_pci_device *pdev, const void *probe_args)*/ /* BAR 0 */ -if (pdev->mem_resource[0 ].addr ==NULL ) { +if (pdev->pci_mem[0 ].mem_res.addr ==NULL ) {DLB2_ERR (dlb2_dev,"probe: BAR 0 addr (csr_kva) is NULL\n" ); ret = -EINVAL ; goto pci_mmap_bad_addr; } - dlb2_dev->hw.func_kva = (void *)(uintptr_t )pdev->mem_resource[0 ].addr; - dlb2_dev->hw.func_phys_addr = pdev->mem_resource[0 ].phys_addr; + dlb2_dev->hw.func_kva = (void *)(uintptr_t )pdev->pci_mem[0 ].mem_res.addr; + dlb2_dev->hw.func_phys_addr = pdev->pci_mem[0 ].mem_res.phys_addr;DLB2_INFO (dlb2_dev,"DLB2 FUNC VA=%p, PA=%p, len=%p\n" , (void *)dlb2_dev->hw.func_kva, (void *)dlb2_dev->hw.func_phys_addr, - (void *)(pdev->mem_resource[0 ].len)); + (void *)(pdev->pci_mem[0 ].mem_res.len));/* BAR 2 */ -if (pdev->mem_resource[2 ].addr ==NULL ) { +if (pdev->pci_mem[2 ].mem_res.addr ==NULL ) {DLB2_ERR (dlb2_dev,"probe: BAR 2 addr (func_kva) is NULL\n" ); ret = -EINVAL ; goto pci_mmap_bad_addr; } - dlb2_dev->hw.csr_kva = (void *)(uintptr_t )pdev->mem_resource[2 ].addr; - dlb2_dev->hw.csr_phys_addr = pdev->mem_resource[2 ].phys_addr; + dlb2_dev->hw.csr_kva = (void *)(uintptr_t )pdev->pci_mem[2 ].mem_res.addr; + dlb2_dev->hw.csr_phys_addr = pdev->pci_mem[2 ].mem_res.phys_addr;DLB2_INFO (dlb2_dev,"DLB2 CSR VA=%p, PA=%p, len=%p\n" , (void *)dlb2_dev->hw.csr_kva, (void *)dlb2_dev->hw.csr_phys_addr, - (void *)(pdev->mem_resource[2 ].len)); + (void *)(pdev->pci_mem[2 ].mem_res.len)); dlb2_dev->pdev = pdev;
drivers/event/octeontx/ssovf_probe.c +19 -19
@@ -148,23 +148,23 @@ ssowvf_probe(struct rte_pci_driver *pci_drv, struct rte_pci_device *pci_dev)if (rte_eal_process_type () !=RTE_PROC_PRIMARY )return 0 ; -if (pci_dev->mem_resource[0 ].addr ==NULL || - pci_dev->mem_resource[2 ].addr ==NULL || - pci_dev->mem_resource[4 ].addr ==NULL ) { +if (pci_dev->pci_mem[0 ].mem_res.addr ==NULL || + pci_dev->pci_mem[2 ].mem_res.addr ==NULL || + pci_dev->pci_mem[4 ].mem_res.addr ==NULL ) {mbox_log_err ("Empty bars %p %p %p" , - pci_dev->mem_resource[0 ].addr, - pci_dev->mem_resource[2 ].addr, - pci_dev->mem_resource[4 ].addr); + pci_dev->pci_mem[0 ].mem_res.addr, + pci_dev->pci_mem[2 ].mem_res.addr, + pci_dev->pci_mem[4 ].mem_res.addr);return -ENODEV ; } -if (pci_dev->mem_resource[4 ].len !=SSOW_BAR4_LEN ) { +if (pci_dev->pci_mem[4 ].mem_res.len !=SSOW_BAR4_LEN ) {mbox_log_err ("Bar4 len mismatch %d != %d" , -SSOW_BAR4_LEN , (int )pci_dev->mem_resource[4 ].len); +SSOW_BAR4_LEN , (int )pci_dev->pci_mem[4 ].mem_res.len);return -EINVAL ; } - id = pci_dev->mem_resource[4 ].addr; + id = pci_dev->pci_mem[4 ].mem_res.addr; vfid = id->vfid;if (vfid >=SSO_MAX_VHWS ) {mbox_log_err ("Invalid vfid(%d/%d)" , vfid,SSO_MAX_VHWS );
@@ -173,9 +173,9 @@ ssowvf_probe(struct rte_pci_driver *pci_drv, struct rte_pci_device *pci_dev) res = &sdev.hws[vfid]; res->vfid = vfid; - res->bar0 = pci_dev->mem_resource[0 ].addr; - res->bar2 = pci_dev->mem_resource[2 ].addr; - res->bar4 = pci_dev->mem_resource[4 ].addr; + res->bar0 = pci_dev->pci_mem[0 ].mem_res.addr; + res->bar2 = pci_dev->pci_mem[2 ].mem_res.addr; + res->bar4 = pci_dev->pci_mem[4 ].mem_res.addr; res->domain = id->domain; sdev.total_ssowvfs++;
@@ -229,14 +229,14 @@ ssovf_probe(struct rte_pci_driver *pci_drv, struct rte_pci_device *pci_dev)if (rte_eal_process_type () !=RTE_PROC_PRIMARY )return 0 ; -if (pci_dev->mem_resource[0 ].addr ==NULL || - pci_dev->mem_resource[2 ].addr ==NULL ) { +if (pci_dev->pci_mem[0 ].mem_res.addr ==NULL || + pci_dev->pci_mem[2 ].mem_res.addr ==NULL ) {mbox_log_err ("Empty bars %p %p" , - pci_dev->mem_resource[0 ].addr, - pci_dev->mem_resource[2 ].addr); + pci_dev->pci_mem[0 ].mem_res.addr, + pci_dev->pci_mem[2 ].mem_res.addr);return -ENODEV ; } - idreg = pci_dev->mem_resource[0 ].addr; + idreg = pci_dev->pci_mem[0 ].mem_res.addr; idreg +=SSO_VHGRP_AQ_THR ; val =rte_read64 (idreg);
@@ -250,8 +250,8 @@ ssovf_probe(struct rte_pci_driver *pci_drv, struct rte_pci_device *pci_dev) res = &sdev.grp[vfid]; res->vfid = vfid; - res->bar0 = pci_dev->mem_resource[0 ].addr; - res->bar2 = pci_dev->mem_resource[2 ].addr; + res->bar0 = pci_dev->pci_mem[0 ].mem_res.addr; + res->bar2 = pci_dev->pci_mem[2 ].mem_res.addr; res->domain = val &0xffff ; sdev.total_ssovfs++;
drivers/event/octeontx/timvf_probe.c +9 -9
@@ -112,15 +112,15 @@ timvf_probe(struct rte_pci_driver *pci_drv, struct rte_pci_device *pci_dev)if (rte_eal_process_type () !=RTE_PROC_PRIMARY )return 0 ; -if (pci_dev->mem_resource[0 ].addr ==NULL || - pci_dev->mem_resource[4 ].addr ==NULL ) { +if (pci_dev->pci_mem[0 ].mem_res.addr ==NULL || + pci_dev->pci_mem[4 ].mem_res.addr ==NULL ) {timvf_log_err ("Empty bars %p %p" , - pci_dev->mem_resource[0 ].addr, - pci_dev->mem_resource[4 ].addr); + pci_dev->pci_mem[0 ].mem_res.addr, + pci_dev->pci_mem[4 ].mem_res.addr);return -ENODEV ; } - val =rte_read64 ((uint8_t *)pci_dev->mem_resource[0 ].addr + + val =rte_read64 ((uint8_t *)pci_dev->pci_mem[0 ].mem_res.addr +0x100 /* TIM_VRINGX_BASE */ ); vfid = (val >>23 ) &0xff ;if (vfid >=TIM_MAX_RINGS ) {
@@ -130,16 +130,16 @@ timvf_probe(struct rte_pci_driver *pci_drv, struct rte_pci_device *pci_dev) res = &tdev.rings[tdev.total_timvfs]; res->vfid = vfid; - res->bar0 = pci_dev->mem_resource[0 ].addr; - res->bar2 = pci_dev->mem_resource[2 ].addr; - res->bar4 = pci_dev->mem_resource[4 ].addr; + res->bar0 = pci_dev->pci_mem[0 ].mem_res.addr; + res->bar2 = pci_dev->pci_mem[2 ].mem_res.addr; + res->bar4 = pci_dev->pci_mem[4 ].mem_res.addr; res->domain = (val >>7 ) &0xffff ; res->in_use = false; tdev.total_timvfs++;rte_wmb ();timvf_log_dbg ("Domain=%d VFid=%d bar0 %p total_timvfs=%d" , res->domain, - res->vfid, pci_dev->mem_resource[0 ].addr, + res->vfid, pci_dev->pci_mem[0 ].mem_res.addr, tdev.total_timvfs);return 0 ; }
drivers/event/skeleton/skeleton_eventdev.c +1 -1
@@ -360,7 +360,7 @@ skeleton_eventdev_init(struct rte_eventdev *eventdev) pci_dev =RTE_DEV_TO_PCI (eventdev->dev); - skel->reg_base = (uintptr_t )pci_dev->mem_resource[0 ].addr; + skel->reg_base = (uintptr_t )pci_dev->pci_mem[0 ].mem_res.addr;if (!skel->reg_base) {PMD_DRV_ERR ("Failed to map BAR0" ); ret = -ENODEV ;
drivers/mempool/octeontx/octeontx_fpavf.c +3 -3
@@ -785,11 +785,11 @@ fpavf_probe(struct rte_pci_driver *pci_drv, struct rte_pci_device *pci_dev)if (rte_eal_process_type () !=RTE_PROC_PRIMARY )return 0 ; -if (pci_dev->mem_resource[0 ].addr ==NULL ) { -fpavf_log_err ("Empty bars %p " , pci_dev->mem_resource[0 ].addr); +if (pci_dev->pci_mem[0 ].mem_res.addr ==NULL ) { +fpavf_log_err ("Empty bars %p " , pci_dev->pci_mem[0 ].mem_res.addr);return -ENODEV ; } - idreg = pci_dev->mem_resource[0 ].addr; + idreg = pci_dev->pci_mem[0 ].mem_res.addr;octeontx_fpavf_setup ();
drivers/net/ark/ark_ethdev.c +2 -2
@@ -329,8 +329,8 @@ eth_ark_dev_init(struct rte_eth_dev *dev) dev->rx_pkt_burst = rte_eth_pkt_burst_dummy; dev->tx_pkt_burst = rte_eth_pkt_burst_dummy; - ark->bar0 = (uint8_t *)pci_dev->mem_resource[0 ].addr; - ark->a_bar = (uint8_t *)pci_dev->mem_resource[2 ].addr; + ark->bar0 = (uint8_t *)pci_dev->pci_mem[0 ].mem_res.addr; + ark->a_bar = (uint8_t *)pci_dev->pci_mem[2 ].mem_res.addr; ark->sysctrl.v = (void *)&ark->bar0[ARK_SYSCTRL_BASE ]; ark->mpurx.v = (void *)&ark->bar0[ARK_MPU_RX_BASE ];
drivers/net/atlantic/atl_ethdev.c +1 -1
@@ -384,7 +384,7 @@ eth_atl_dev_init(struct rte_eth_dev *eth_dev)/* Vendor and Device ID need to be set before init of shared code */ hw->device_id = pci_dev->id.device_id; hw->vendor_id = pci_dev->id.vendor_id; - hw->mmio = (void *)pci_dev->mem_resource[0 ].addr; + hw->mmio = (void *)pci_dev->pci_mem[0 ].mem_res.addr;/* Hardware configuration - hardcode */ adapter->hw_cfg.is_lro = false;
drivers/net/avp/avp_ethdev.c +10 -10
@@ -368,8 +368,8 @@ avp_dev_translate_address(struct rte_eth_dev *eth_dev,void *addr;unsigned int i; - addr = pci_dev->mem_resource[RTE_AVP_PCI_MEMORY_BAR ].addr; - resource = &pci_dev->mem_resource[RTE_AVP_PCI_MEMMAP_BAR ]; + addr = pci_dev->pci_mem[RTE_AVP_PCI_MEMORY_BAR ].mem_res.addr; + resource = &pci_dev->pci_mem[RTE_AVP_PCI_MEMMAP_BAR ].mem_res; info = (struct rte_avp_memmap_info *)resource->addr; offset =0 ;
@@ -421,7 +421,7 @@ avp_dev_check_regions(struct rte_eth_dev *eth_dev)/* Dump resource info for debug */ for (i =0 ; i <PCI_MAX_RESOURCE ; i++) { - resource = &pci_dev->mem_resource[i]; + resource = &pci_dev->pci_mem[i].mem_res;if ((resource->phys_addr ==0 ) || (resource->len ==0 ))continue ;
@@ -554,7 +554,7 @@ _avp_set_queue_counts(struct rte_eth_dev *eth_dev)struct rte_avp_device_info *host_info;void *addr; - addr = pci_dev->mem_resource[RTE_AVP_PCI_DEVICE_BAR ].addr; + addr = pci_dev->pci_mem[RTE_AVP_PCI_DEVICE_BAR ].mem_res.addr; host_info = (struct rte_avp_device_info *)addr;/*
@@ -664,7 +664,7 @@ avp_dev_interrupt_handler(void *data) {struct rte_eth_dev *eth_dev = data;struct rte_pci_device *pci_dev =RTE_ETH_DEV_TO_PCI (eth_dev); -void *registers = pci_dev->mem_resource[RTE_AVP_PCI_MMIO_BAR ].addr; +void *registers = pci_dev->pci_mem[RTE_AVP_PCI_MMIO_BAR ].mem_res.addr;uint32_t status, value;int ret;
@@ -723,7 +723,7 @@ static int avp_dev_enable_interrupts(struct rte_eth_dev *eth_dev) {struct rte_pci_device *pci_dev =RTE_ETH_DEV_TO_PCI (eth_dev); -void *registers = pci_dev->mem_resource[RTE_AVP_PCI_MMIO_BAR ].addr; +void *registers = pci_dev->pci_mem[RTE_AVP_PCI_MMIO_BAR ].mem_res.addr;int ret;if (registers ==NULL )
@@ -748,7 +748,7 @@ static int avp_dev_disable_interrupts(struct rte_eth_dev *eth_dev) {struct rte_pci_device *pci_dev =RTE_ETH_DEV_TO_PCI (eth_dev); -void *registers = pci_dev->mem_resource[RTE_AVP_PCI_MMIO_BAR ].addr; +void *registers = pci_dev->pci_mem[RTE_AVP_PCI_MMIO_BAR ].mem_res.addr;int ret;if (registers ==NULL )
@@ -793,7 +793,7 @@ static int avp_dev_migration_pending(struct rte_eth_dev *eth_dev) {struct rte_pci_device *pci_dev =RTE_ETH_DEV_TO_PCI (eth_dev); -void *registers = pci_dev->mem_resource[RTE_AVP_PCI_MMIO_BAR ].addr; +void *registers = pci_dev->pci_mem[RTE_AVP_PCI_MMIO_BAR ].mem_res.addr;uint32_t value;if (registers ==NULL )
@@ -824,7 +824,7 @@ avp_dev_create(struct rte_pci_device *pci_dev,struct rte_mem_resource *resource;unsigned int i; - resource = &pci_dev->mem_resource[RTE_AVP_PCI_DEVICE_BAR ]; + resource = &pci_dev->pci_mem[RTE_AVP_PCI_DEVICE_BAR ].mem_res;if (resource->addr ==NULL ) {PMD_DRV_LOG (ERR ,"BAR%u is not mapped\n" ,RTE_AVP_PCI_DEVICE_BAR );
@@ -1992,7 +1992,7 @@ avp_dev_configure(struct rte_eth_dev *eth_dev) goto unlock; } - addr = pci_dev->mem_resource[RTE_AVP_PCI_DEVICE_BAR ].addr; + addr = pci_dev->pci_mem[RTE_AVP_PCI_DEVICE_BAR ].mem_res.addr; host_info = (struct rte_avp_device_info *)addr;/* Setup required number of queues */
drivers/net/axgbe/axgbe_ethdev.c +2 -2
@@ -2216,12 +2216,12 @@ eth_axgbe_dev_init(struct rte_eth_dev *eth_dev) pdata->pci_dev = pci_dev; pdata->xgmac_regs = - (void *)pci_dev->mem_resource[AXGBE_AXGMAC_BAR ].addr; + (void *)pci_dev->pci_mem[AXGBE_AXGMAC_BAR ].mem_res.addr; pdata->xprop_regs = (void *)((uint8_t *)pdata->xgmac_regs +AXGBE_MAC_PROP_OFFSET ); pdata->xi2c_regs = (void *)((uint8_t *)pdata->xgmac_regs +AXGBE_I2C_CTRL_OFFSET ); - pdata->xpcs_regs = (void *)pci_dev->mem_resource[AXGBE_XPCS_BAR ].addr; + pdata->xpcs_regs = (void *)pci_dev->pci_mem[AXGBE_XPCS_BAR ].mem_res.addr;/* version specific driver data*/ if (pci_dev->id.device_id ==AMD_PCI_AXGBE_DEVICE_V2A )
drivers/net/bnx2x/bnx2x_ethdev.c +3 -3
@@ -656,12 +656,12 @@ bnx2x_common_dev_init(struct rte_eth_dev *eth_dev, int is_vf) sc->flags =BNX2X_IS_VF_FLAG ; sc->pcie_func = pci_dev->addr.function; - sc->bar[BAR0 ].base_addr = (void *)pci_dev->mem_resource[0 ].addr; + sc->bar[BAR0 ].base_addr = (void *)pci_dev->pci_mem[0 ].mem_res.addr;if (is_vf) sc->bar[BAR1 ].base_addr = (void *) - ((uintptr_t )pci_dev->mem_resource[0 ].addr +PXP_VF_ADDR_DB_START ); + ((uintptr_t )pci_dev->pci_mem[0 ].mem_res.addr +PXP_VF_ADDR_DB_START );else - sc->bar[BAR1 ].base_addr = pci_dev->mem_resource[2 ].addr; + sc->bar[BAR1 ].base_addr = pci_dev->pci_mem[2 ].mem_res.addr;assert (sc->bar[BAR0 ].base_addr);assert (sc->bar[BAR1 ].base_addr);
drivers/net/bnxt/bnxt_ethdev.c +4 -4
@@ -4682,8 +4682,8 @@ static int bnxt_map_pci_bars(struct rte_eth_dev *eth_dev)struct bnxt *bp = eth_dev->data->dev_private;/* enable device (incl. PCI PM wakeup), and bus-mastering */ - bp->bar0 = (void *)pci_dev->mem_resource[0 ].addr; - bp->doorbell_base = (void *)pci_dev->mem_resource[2 ].addr; + bp->bar0 = (void *)pci_dev->pci_mem[0 ].mem_res.addr; + bp->doorbell_base = (void *)pci_dev->pci_mem[2 ].mem_res.addr;if (!bp->bar0 || !bp->doorbell_base) {PMD_DRV_LOG (ERR ,"Unable to access Hardware\n" );return -ENODEV ;
@@ -5932,8 +5932,8 @@ bnxt_dev_init(struct rte_eth_dev *eth_dev, void *params __rte_unused)PMD_DRV_LOG (INFO ,"Found %s device at mem %" PRIX64 ", node addr %pM\n" ,DRV_MODULE_NAME , - pci_dev->mem_resource[0 ].phys_addr, - pci_dev->mem_resource[0 ].addr); + pci_dev->pci_mem[0 ].mem_res.phys_addr, + pci_dev->pci_mem[0 ].mem_res.addr);return 0 ;
drivers/net/cpfl/cpfl_ethdev.c +2 -2
@@ -1172,8 +1172,8 @@ cpfl_adapter_ext_init(struct rte_pci_device *pci_dev, struct cpfl_adapter_ext *astruct idpf_hw *hw = &base->hw;int ret =0 ; - hw->hw_addr = (void *)pci_dev->mem_resource[0 ].addr; - hw->hw_addr_len = pci_dev->mem_resource[0 ].len; + hw->hw_addr = (void *)pci_dev->pci_mem[0 ].mem_res.addr; + hw->hw_addr_len = pci_dev->pci_mem[0 ].mem_res.len; hw->back = base; hw->vendor_id = pci_dev->id.vendor_id; hw->device_id = pci_dev->id.device_id;
drivers/net/cxgbe/cxgbe_ethdev.c +1 -1
@@ -1729,7 +1729,7 @@ static int eth_cxgbe_dev_init(struct rte_eth_dev *eth_dev)return -1 ; adapter->use_unpacked_mode =1 ; - adapter->regs = (void *)pci_dev->mem_resource[0 ].addr; + adapter->regs = (void *)pci_dev->pci_mem[0 ].mem_res.addr;if (!adapter->regs) {dev_err (adapter,"%s: cannot map device registers\n" , __func__); err = -ENOMEM ;
drivers/net/cxgbe/cxgbe_main.c +1 -1
@@ -2228,7 +2228,7 @@ int cxgbe_probe(struct adapter *adapter)if (qpp > num_seg)dev_warn (adapter,"Incorrect SGE EGRESS QUEUES_PER_PAGE configuration, continuing in debug mode\n" ); - adapter->bar2 = (void *)adapter->pdev->mem_resource[2 ].addr; + adapter->bar2 = (void *)adapter->pdev->pci_mem[2 ].mem_res.addr;if (!adapter->bar2) {dev_err (adapter,"cannot map device bar2 region\n" ); err = -ENOMEM ;
drivers/net/cxgbe/cxgbevf_ethdev.c +1 -1
@@ -148,7 +148,7 @@ static int eth_cxgbevf_dev_init(struct rte_eth_dev *eth_dev)return -1 ; adapter->use_unpacked_mode =1 ; - adapter->regs = (void *)pci_dev->mem_resource[0 ].addr; + adapter->regs = (void *)pci_dev->pci_mem[0 ].mem_res.addr;if (!adapter->regs) {dev_err (adapter,"%s: cannot map device registers\n" , __func__); err = -ENOMEM ;
drivers/net/cxgbe/cxgbevf_main.c +1 -1
@@ -184,7 +184,7 @@ int cxgbevf_probe(struct adapter *adapter)return err;if (!is_t4 (adapter->params.chip)) { - adapter->bar2 = (void *)adapter->pdev->mem_resource[2 ].addr; + adapter->bar2 = (void *)adapter->pdev->pci_mem[2 ].mem_res.addr;if (!adapter->bar2) {dev_err (adapter,"cannot map device bar2 region\n" ); err = -ENOMEM ;
drivers/net/e1000/em_ethdev.c +2 -2
@@ -265,13 +265,13 @@ eth_em_dev_init(struct rte_eth_dev *eth_dev)rte_eth_copy_pci_info (eth_dev, pci_dev); - hw->hw_addr = (void *)pci_dev->mem_resource[0 ].addr; + hw->hw_addr = (void *)pci_dev->pci_mem[0 ].mem_res.addr; hw->device_id = pci_dev->id.device_id; adapter->stopped =0 ;/* For ICH8 support we'll need to map the flash memory BAR */ if (eth_em_dev_is_ich8 (hw)) - hw->flash_address = (void *)pci_dev->mem_resource[1 ].addr; + hw->flash_address = (void *)pci_dev->pci_mem[1 ].mem_res.addr;if (e1000_setup_init_funcs (hw, TRUE) !=E1000_SUCCESS ||em_hw_init (hw) !=0 ) {
drivers/net/e1000/igb_ethdev.c +2 -2
@@ -743,7 +743,7 @@ eth_igb_dev_init(struct rte_eth_dev *eth_dev)rte_eth_copy_pci_info (eth_dev, pci_dev); - hw->hw_addr= (void *)pci_dev->mem_resource[0 ].addr; + hw->hw_addr = (void *)pci_dev->pci_mem[0 ].mem_res.addr;igb_identify_hardware (eth_dev, pci_dev);if (e1000_setup_init_funcs (hw, FALSE) !=E1000_SUCCESS ) {
@@ -938,7 +938,7 @@ eth_igbvf_dev_init(struct rte_eth_dev *eth_dev) hw->device_id = pci_dev->id.device_id; hw->vendor_id = pci_dev->id.vendor_id; - hw->hw_addr = (void *)pci_dev->mem_resource[0 ].addr; + hw->hw_addr = (void *)pci_dev->pci_mem[0 ].mem_res.addr; adapter->stopped =0 ;/* Initialize the shared code (base driver) */
drivers/net/ena/ena_ethdev.c +2 -2
@@ -2121,8 +2121,8 @@ static int eth_ena_dev_init(struct rte_eth_dev *eth_dev) intr_handle = pci_dev->intr_handle; - adapter->regs = pci_dev->mem_resource[ENA_REGS_BAR ].addr; - adapter->dev_mem_base = pci_dev->mem_resource[ENA_MEM_BAR ].addr; + adapter->regs = pci_dev->pci_mem[ENA_REGS_BAR ].mem_res.addr; + adapter->dev_mem_base = pci_dev->pci_mem[ENA_MEM_BAR ].mem_res.addr;if (!adapter->regs) { PMD_INIT_LOG(CRIT, "Failed to access registers BAR(%d)\n",
drivers/net/enetc/enetc_ethdev.c +1 -1
@@ -883,7 +883,7 @@ enetc_dev_init(struct rte_eth_dev *eth_dev) eth_dev->tx_pkt_burst = &enetc_xmit_pkts;/* Retrieving and storing the HW base address of device */ - hw->hw.reg = (void *)pci_dev->mem_resource[0 ].addr; + hw->hw.reg = (void *)pci_dev->pci_mem[0 ].mem_res.addr; hw->device_id = pci_dev->id.device_id; error =enetc_hardware_init (hw);
drivers/net/enic/enic_main.c +2 -2
@@ -1914,8 +1914,8 @@ int enic_probe(struct enic *enic)if (rte_eal_process_type () !=RTE_PROC_PRIMARY )return 0 ; - enic->bar0.vaddr = (void *)pdev->mem_resource[0 ].addr; - enic->bar0.len = pdev->mem_resource[0 ].len; + enic->bar0.vaddr = (void *)pdev->pci_mem[0 ].mem_res.addr; + enic->bar0.len = pdev->pci_mem[0 ].mem_res.len;/* Register vNIC device */ enic->vdev =vnic_dev_register (NULL , enic, enic->pdev, &enic->bar0,1 );
drivers/net/fm10k/fm10k_ethdev.c +1 -1
@@ -3090,7 +3090,7 @@ eth_fm10k_dev_init(struct rte_eth_dev *dev) hw->subsystem_device_id = pdev->id.subsystem_device_id; hw->subsystem_vendor_id = pdev->id.subsystem_vendor_id; hw->revision_id =0 ; - hw->hw_addr = (void *)pdev->mem_resource[0 ].addr; + hw->hw_addr = (void *)pdev->pci_mem[0 ].mem_res.addr;if (hw->hw_addr ==NULL ) {PMD_INIT_LOG (ERR ,"Bad mem resource." " Try to refuse unused devices." );
drivers/net/gve/gve_ethdev.c +2 -2
@@ -777,13 +777,13 @@ gve_dev_init(struct rte_eth_dev *eth_dev) pci_dev =RTE_DEV_TO_PCI (eth_dev->device); - reg_bar = pci_dev->mem_resource[GVE_REG_BAR ].addr; + reg_bar = pci_dev->pci_mem[GVE_REG_BAR ].mem_res.addr;if (!reg_bar) {PMD_DRV_LOG (ERR ,"Failed to map pci bar!" );return -ENOMEM ; } - db_bar = pci_dev->mem_resource[GVE_DB_BAR ].addr; + db_bar = pci_dev->pci_mem[GVE_DB_BAR ].mem_res.addr;if (!db_bar) {PMD_DRV_LOG (ERR ,"Failed to map doorbell bar!" );return -ENOMEM ;
drivers/net/hinic/base/hinic_pmd_hwif.c +7 -7
@@ -398,7 +398,7 @@ static int hinic_init_hwif(struct hinic_hwdev *hwdev, void *cfg_reg_base,int err; pci_dev = (struct rte_pci_device *)(hwdev->pcidev_hdl); - db_bar_len = pci_dev->mem_resource[HINIC_DB_MEM_BAR ].len; + db_bar_len = pci_dev->pci_mem[HINIC_DB_MEM_BAR ].mem_res.len; hwif = hwdev->hwif;
@@ -470,16 +470,16 @@ static void hinic_get_mmio(struct hinic_hwdev *hwdev, void **cfg_regs_base,uint64_t bar0_phy_addr;uint64_t pagesize =sysconf (_SC_PAGESIZE); - *cfg_regs_base = pci_dev->mem_resource[HINIC_CFG_REGS_BAR ].addr; - *intr_base = pci_dev->mem_resource[HINIC_INTR_MSI_BAR ].addr; - *db_base = pci_dev->mem_resource[HINIC_DB_MEM_BAR ].addr; + *cfg_regs_base = pci_dev->pci_mem[HINIC_CFG_REGS_BAR ].mem_res.addr; + *intr_base = pci_dev->pci_mem[HINIC_INTR_MSI_BAR ].mem_res.addr; + *db_base = pci_dev->pci_mem[HINIC_DB_MEM_BAR ].mem_res.addr; - bar0_size = pci_dev->mem_resource[HINIC_CFG_REGS_BAR ].len; - bar2_size = pci_dev->mem_resource[HINIC_INTR_MSI_BAR ].len; + bar0_size = pci_dev->pci_mem[HINIC_CFG_REGS_BAR ].mem_res.len; + bar2_size = pci_dev->pci_mem[HINIC_INTR_MSI_BAR ].mem_res.len;if (pagesize ==PAGE_SIZE_64K && (bar0_size % pagesize !=0 )) { bar0_phy_addr = - pci_dev->mem_resource[HINIC_CFG_REGS_BAR ].phys_addr; + pci_dev->pci_mem[HINIC_CFG_REGS_BAR ].mem_res.phys_addr;if (bar0_phy_addr % pagesize !=0 && (bar0_size + bar2_size <= pagesize) && bar2_size >= bar0_size) {
drivers/net/hns3/hns3_ethdev.c +1 -1
@@ -4525,7 +4525,7 @@ hns3_init_pf(struct rte_eth_dev *eth_dev)PMD_INIT_FUNC_TRACE ();/* Get hardware io base address from pcie BAR2 IO space */ - hw->io_base = pci_dev->mem_resource[2 ].addr; + hw->io_base = pci_dev->pci_mem[2 ].mem_res.addr;/* Firmware command queue initialize */ ret =hns3_cmd_init_queue (hw);
drivers/net/hns3/hns3_ethdev_vf.c +1 -1
@@ -1414,7 +1414,7 @@ hns3vf_init_vf(struct rte_eth_dev *eth_dev)PMD_INIT_FUNC_TRACE ();/* Get hardware io base address from pcie BAR2 IO space */ - hw->io_base = pci_dev->mem_resource[2 ].addr; + hw->io_base = pci_dev->pci_mem[2 ].mem_res.addr;/* Firmware command queue initialize */ ret =hns3_cmd_init_queue (hw);
drivers/net/hns3/hns3_rxtx.c +2 -2
@@ -2923,8 +2923,8 @@ hns3_tx_push_get_queue_tail_reg(struct rte_eth_dev *dev, uint16_t queue_id)* * The quick doorbell located at 64B offset in the TQP region. */ -return (char *)pci_dev->mem_resource[bar_id].addr + - (pci_dev->mem_resource[bar_id].len >>1 ) + +return (char *)pci_dev->pci_mem[bar_id].mem_res.addr + + (pci_dev->pci_mem[bar_id].mem_res.len >>1 ) +HNS3_TX_PUSH_TQP_REGION_SIZE * queue_id +HNS3_TX_PUSH_QUICK_DOORBELL_OFFSET ; }
drivers/net/i40e/i40e_ethdev.c +1 -1
@@ -1449,7 +1449,7 @@ eth_i40e_dev_init(struct rte_eth_dev *dev, void *init_params __rte_unused) pf->dev_data = dev->data; hw->back =I40E_PF_TO_ADAPTER (pf); - hw->hw_addr = (uint8_t *)(pci_dev->mem_resource[0 ].addr); + hw->hw_addr = (uint8_t *)(pci_dev->pci_mem[0 ].mem_res.addr);if (!hw->hw_addr) {PMD_INIT_LOG (ERR ,"Hardware is not available, as address is NULL" );
drivers/net/iavf/iavf_ethdev.c +1 -1
@@ -2605,7 +2605,7 @@ iavf_dev_init(struct rte_eth_dev *eth_dev) hw->bus.bus_id = pci_dev->addr.bus; hw->bus.device = pci_dev->addr.devid; hw->bus.func = pci_dev->addr.function; - hw->hw_addr = (void *)pci_dev->mem_resource[0 ].addr; + hw->hw_addr = (void *)pci_dev->pci_mem[0 ].mem_res.addr; hw->back =IAVF_DEV_PRIVATE_TO_ADAPTER (eth_dev->data->dev_private); adapter->dev_data = eth_dev->data; adapter->stopped =1 ;
drivers/net/ice/ice_dcf.c +1 -1
@@ -618,7 +618,7 @@ ice_dcf_init_hw(struct rte_eth_dev *eth_dev, struct ice_dcf_hw *hw) hw->resetting = false; - hw->avf.hw_addr = pci_dev->mem_resource[0 ].addr; + hw->avf.hw_addr = pci_dev->pci_mem[0 ].mem_res.addr; hw->avf.back = hw; hw->avf.bus.bus_id = pci_dev->addr.bus;
drivers/net/ice/ice_ethdev.c +1 -1
@@ -2293,7 +2293,7 @@ ice_dev_init(struct rte_eth_dev *dev) pf->adapter =ICE_DEV_PRIVATE_TO_ADAPTER (dev->data->dev_private); pf->dev_data = dev->data; hw->back = pf->adapter; - hw->hw_addr = (uint8_t *)pci_dev->mem_resource[0 ].addr; + hw->hw_addr = (uint8_t *)pci_dev->pci_mem[0 ].mem_res.addr; hw->vendor_id = pci_dev->id.vendor_id; hw->device_id = pci_dev->id.device_id; hw->subsystem_vendor_id = pci_dev->id.subsystem_vendor_id;
drivers/net/idpf/idpf_ethdev.c +2 -2
@@ -1135,8 +1135,8 @@ idpf_adapter_ext_init(struct rte_pci_device *pci_dev, struct idpf_adapter_ext *astruct idpf_hw *hw = &base->hw;int ret =0 ; - hw->hw_addr = (void *)pci_dev->mem_resource[0 ].addr; - hw->hw_addr_len = pci_dev->mem_resource[0 ].len; + hw->hw_addr = (void *)pci_dev->pci_mem[0 ].mem_res.addr; + hw->hw_addr_len = pci_dev->pci_mem[0 ].mem_res.len; hw->back = base; hw->vendor_id = pci_dev->id.vendor_id; hw->device_id = pci_dev->id.device_id;
drivers/net/igc/igc_ethdev.c +1 -1
@@ -1343,7 +1343,7 @@ eth_igc_dev_init(struct rte_eth_dev *dev) dev->data->dev_flags |=RTE_ETH_DEV_AUTOFILL_QUEUE_XSTATS ; hw->back = pci_dev; - hw->hw_addr = (void *)pci_dev->mem_resource[0 ].addr; + hw->hw_addr = (void *)pci_dev->pci_mem[0 ].mem_res.addr;igc_identify_hardware (dev, pci_dev);if (igc_setup_init_funcs (hw, false) !=IGC_SUCCESS ) {
drivers/net/ionic/ionic_dev_pci.c +1 -1
@@ -234,7 +234,7 @@ eth_ionic_pci_probe(struct rte_pci_driver *pci_drv __rte_unused, bars.num_bars =0 ;for (i =0 ; i <PCI_MAX_RESOURCE && i <IONIC_BARS_MAX ; i++) { - resource = &pci_dev->mem_resource[i]; + resource = &pci_dev->pci_mem[i].mem_res;if (resource->phys_addr ==0 || resource->len ==0 )continue ;
drivers/net/ixgbe/ixgbe_ethdev.c +2 -2
@@ -1136,7 +1136,7 @@ eth_ixgbe_dev_init(struct rte_eth_dev *eth_dev, void *init_params __rte_unused)/* Vendor and Device ID need to be set before init of shared code */ hw->device_id = pci_dev->id.device_id; hw->vendor_id = pci_dev->id.vendor_id; - hw->hw_addr = (void *)pci_dev->mem_resource[0 ].addr; + hw->hw_addr = (void *)pci_dev->pci_mem[0 ].mem_res.addr; hw->allow_unsupported_sfp =1 ;/* Initialize the shared code (base driver) */
@@ -1634,7 +1634,7 @@ eth_ixgbevf_dev_init(struct rte_eth_dev *eth_dev) hw->device_id = pci_dev->id.device_id; hw->vendor_id = pci_dev->id.vendor_id; - hw->hw_addr = (void *)pci_dev->mem_resource[0 ].addr; + hw->hw_addr = (void *)pci_dev->pci_mem[0 ].mem_res.addr;/* initialize the vfta */ memset (shadow_vfta,0 ,sizeof (*shadow_vfta));
drivers/net/liquidio/lio_ethdev.c +2 -2
@@ -2071,8 +2071,8 @@ lio_eth_dev_init(struct rte_eth_dev *eth_dev)rte_eth_copy_pci_info (eth_dev, pdev); -if (pdev->mem_resource[0 ].addr) { - lio_dev->hw_addr = pdev->mem_resource[0 ].addr; +if (pdev->pci_mem[0 ].mem_res.addr) { + lio_dev->hw_addr = pdev->pci_mem[0 ].mem_res.addr; }else {PMD_INIT_LOG (ERR ,"ERROR: Failed to map BAR0\n" );return -ENODEV ;
drivers/net/nfp/nfp_ethdev.c +1 -1
@@ -543,7 +543,7 @@ nfp_net_init(struct rte_eth_dev *eth_dev) pci_dev->addr.domain, pci_dev->addr.bus, pci_dev->addr.devid, pci_dev->addr.function); - hw->ctrl_bar = (uint8_t *)pci_dev->mem_resource[0 ].addr; + hw->ctrl_bar = (uint8_t *)pci_dev->pci_mem[0 ].mem_res.addr;if (hw->ctrl_bar ==NULL ) {PMD_DRV_LOG (ERR ,"hw->ctrl_bar is NULL. BAR0 not configured" );
drivers/net/nfp/nfp_ethdev_vf.c +3 -3
@@ -290,7 +290,7 @@ nfp_netvf_init(struct rte_eth_dev *eth_dev) hw =NFP_NET_DEV_PRIVATE_TO_HW (eth_dev->data->dev_private); - hw->ctrl_bar = (uint8_t *)pci_dev->mem_resource[0 ].addr; + hw->ctrl_bar = (uint8_t *)pci_dev->pci_mem[0 ].mem_res.addr;if (hw->ctrl_bar ==NULL ) {PMD_DRV_LOG (ERR ,"hw->ctrl_bar is NULL. BAR0 not configured" );
@@ -351,9 +351,9 @@ nfp_netvf_init(struct rte_eth_dev *eth_dev)PMD_INIT_LOG (DEBUG ,"tx_bar_off: 0x%" PRIx64"" , tx_bar_off);PMD_INIT_LOG (DEBUG ,"rx_bar_off: 0x%" PRIx64"" , rx_bar_off); - hw->tx_bar = (uint8_t *)pci_dev->mem_resource[2 ].addr + + hw->tx_bar = (uint8_t *)pci_dev->pci_mem[2 ].mem_res.addr + tx_bar_off; - hw->rx_bar = (uint8_t *)pci_dev->mem_resource[2 ].addr + + hw->rx_bar = (uint8_t *)pci_dev->pci_mem[2 ].mem_res.addr + rx_bar_off; PMD_INIT_LOG(DEBUG, "ctrl_bar: %p, tx_bar: %p, rx_bar: %p",
drivers/net/nfp/nfpcore/nfp_cpp_pcie_ops.c +2 -2
@@ -790,7 +790,7 @@ nfp6000_set_barsz(struct rte_pci_device *dev, struct nfp_pcie_user *desc)unsigned long tmp;int i =0 ; - tmp = dev->mem_resource[0 ].len; + tmp = dev->pci_mem[0 ].mem_res.len;while (tmp >>=1 ) i++;
@@ -836,7 +836,7 @@ nfp6000_init(struct nfp_cpp *cpp, struct rte_pci_device *dev)if (nfp6000_set_barsz (dev, desc) <0 ) goto error; - desc->cfg = (char *)dev->mem_resource[0 ].addr; + desc->cfg = (char *)dev->pci_mem[0 ].mem_res.addr;nfp_enable_bars (desc);
drivers/net/ngbe/ngbe_ethdev.c +1 -1
@@ -364,7 +364,7 @@ eth_ngbe_dev_init(struct rte_eth_dev *eth_dev, void *init_params __rte_unused)rte_eth_copy_pci_info (eth_dev, pci_dev); eth_dev->data->dev_flags |=RTE_ETH_DEV_AUTOFILL_QUEUE_XSTATS ; - hw->hw_addr = (void *)pci_dev->mem_resource[0 ].addr; + hw->hw_addr = (void *)pci_dev->pci_mem[0 ].mem_res.addr;/* Vendor and Device ID need to be set before init of shared code */ hw->back = pci_dev;
drivers/net/octeon_ep/otx_ep_ethdev.c +1 -1
@@ -490,7 +490,7 @@ otx_ep_eth_dev_init(struct rte_eth_dev *eth_dev) }rte_eth_random_addr (vf_mac_addr.addr_bytes);rte_ether_addr_copy (&vf_mac_addr, eth_dev->data->mac_addrs); - otx_epvf->hw_addr = pdev->mem_resource[0 ].addr; + otx_epvf->hw_addr = pdev->pci_mem[0 ].mem_res.addr; otx_epvf->pdev = pdev;otx_epdev_init (otx_epvf);
drivers/net/octeontx/base/octeontx_pkivf.c +3 -3
@@ -195,13 +195,13 @@ pkivf_probe(struct rte_pci_driver *pci_drv, struct rte_pci_device *pci_dev)if (rte_eal_process_type () !=RTE_PROC_PRIMARY )return 0 ; -if (pci_dev->mem_resource[0 ].addr ==NULL ) { +if (pci_dev->pci_mem[0 ].mem_res.addr ==NULL ) {octeontx_log_err ("PKI Empty bar[0] %p" , - pci_dev->mem_resource[0 ].addr); + pci_dev->pci_mem[0 ].mem_res.addr);return -ENODEV ; } - bar0 = pci_dev->mem_resource[0 ].addr; + bar0 = pci_dev->pci_mem[0 ].mem_res.addr; val =octeontx_read64 (bar0); domain = val &0xffff ; vfid = (val >>16 ) &0xffff ;
drivers/net/octeontx/base/octeontx_pkovf.c +6 -6
@@ -586,15 +586,15 @@ pkovf_probe(struct rte_pci_driver *pci_drv, struct rte_pci_device *pci_dev)if (rte_eal_process_type () !=RTE_PROC_PRIMARY )return 0 ; -if (pci_dev->mem_resource[0 ].addr ==NULL || - pci_dev->mem_resource[2 ].addr ==NULL ) { +if (pci_dev->pci_mem[0 ].mem_res.addr ==NULL || + pci_dev->pci_mem[2 ].mem_res.addr ==NULL ) {octeontx_log_err ("Empty bars %p %p" , - pci_dev->mem_resource[0 ].addr, - pci_dev->mem_resource[2 ].addr); + pci_dev->pci_mem[0 ].mem_res.addr, + pci_dev->pci_mem[2 ].mem_res.addr);return -ENODEV ; } - bar0 = pci_dev->mem_resource[0 ].addr; - bar2 = pci_dev->mem_resource[2 ].addr; + bar0 = pci_dev->pci_mem[0 ].mem_res.addr; + bar2 = pci_dev->pci_mem[2 ].mem_res.addr;octeontx_pkovf_setup ();
drivers/net/qede/qede_main.c +3 -3
@@ -37,9 +37,9 @@ qed_update_pf_params(struct ecore_dev *edev, struct ecore_pf_params *params)static void qed_init_pci (struct ecore_dev *edev,struct rte_pci_device *pci_dev) { - edev->regview = pci_dev->mem_resource[0 ].addr; - edev->doorbells = pci_dev->mem_resource[2 ].addr; - edev->db_size = pci_dev->mem_resource[2 ].len; + edev->regview = pci_dev->pci_mem[0 ].mem_res.addr; + edev->doorbells = pci_dev->pci_mem[2 ].mem_res.addr; + edev->db_size = pci_dev->pci_mem[2 ].mem_res.len; edev->pci_dev = pci_dev; }
drivers/net/sfc/sfc.c +1 -1
@@ -773,7 +773,7 @@ sfc_mem_bar_init(struct sfc_adapter *sa, const efx_bar_region_t *mem_ebrp)struct rte_pci_device *pci_dev =RTE_ETH_DEV_TO_PCI (eth_dev);efsys_bar_t *ebp = &sa->mem_bar;struct rte_mem_resource *res = - &pci_dev->mem_resource[mem_ebrp->ebr_index]; + &pci_dev->pci_mem[mem_ebrp->ebr_index].mem_res;SFC_BAR_LOCK_INIT (ebp, eth_dev->data->name); ebp->esb_rid = mem_ebrp->ebr_index;
drivers/net/thunderx/nicvf_ethdev.c +1 -1
@@ -2223,7 +2223,7 @@ nicvf_eth_dev_init(struct rte_eth_dev *eth_dev) pci_dev->addr.domain, pci_dev->addr.bus, pci_dev->addr.devid, pci_dev->addr.function); - nic->reg_base = (uintptr_t )pci_dev->mem_resource[0 ].addr; + nic->reg_base = (uintptr_t )pci_dev->pci_mem[0 ].mem_res.addr;if (!nic->reg_base) {PMD_INIT_LOG (ERR ,"Failed to map BAR0" ); ret = -ENODEV ;
drivers/net/txgbe/txgbe_ethdev.c +1 -1
@@ -594,7 +594,7 @@ eth_txgbe_dev_init(struct rte_eth_dev *eth_dev, void *init_params __rte_unused)rte_eth_copy_pci_info (eth_dev, pci_dev); - hw->hw_addr = (void *)pci_dev->mem_resource[0 ].addr; + hw->hw_addr = (void *)pci_dev->pci_mem[0 ].mem_res.addr;/* Vendor and Device ID need to be set before init of shared code */ hw->device_id = pci_dev->id.device_id;
drivers/net/txgbe/txgbe_ethdev_vf.c +1 -1
@@ -211,7 +211,7 @@ eth_txgbevf_dev_init(struct rte_eth_dev *eth_dev) hw->vendor_id = pci_dev->id.vendor_id; hw->subsystem_device_id = pci_dev->id.subsystem_device_id; hw->subsystem_vendor_id = pci_dev->id.subsystem_vendor_id; - hw->hw_addr = (void *)pci_dev->mem_resource[0 ].addr; + hw->hw_addr = (void *)pci_dev->pci_mem[0 ].mem_res.addr;/* initialize the vfta */ memset (shadow_vfta,0 ,sizeof (*shadow_vfta));
drivers/net/virtio/virtio_pci.c +3 -3
@@ -603,14 +603,14 @@ get_cfg_addr(struct rte_pci_device *dev, struct virtio_pci_cap *cap)return NULL ; } -if (offset + length > dev->mem_resource[bar].len) { +if (offset + length > dev->pci_mem[bar].mem_res.len) {PMD_INIT_LOG (ERR ,"invalid cap: overflows bar space: %u > %" PRIu64, - offset + length, dev->mem_resource[bar].len); + offset + length, dev->pci_mem[bar].mem_res.len);return NULL ; } - base = dev->mem_resource[bar].addr; + base = dev->pci_mem[bar].mem_res.addr;if (base ==NULL ) {PMD_INIT_LOG (ERR ,"bar %u base addr is NULL" , bar);return NULL ;
drivers/net/vmxnet3/vmxnet3_ethdev.c +2 -2
@@ -345,8 +345,8 @@ eth_vmxnet3_dev_init(struct rte_eth_dev *eth_dev)/* Vendor and Device ID need to be set before init of shared code */ hw->device_id = pci_dev->id.device_id; hw->vendor_id = pci_dev->id.vendor_id; - hw->hw_addr0 = (void *)pci_dev->mem_resource[0 ].addr; - hw->hw_addr1 = (void *)pci_dev->mem_resource[1 ].addr; + hw->hw_addr0 = (void *)pci_dev->pci_mem[0 ].mem_res.addr; + hw->hw_addr1 = (void *)pci_dev->pci_mem[1 ].mem_res.addr; hw->num_rx_queues =1 ; hw->num_tx_queues =1 ;
drivers/raw/cnxk_bphy/cnxk_bphy.c +5 -5
@@ -331,10 +331,10 @@ bphy_rawdev_probe(struct rte_pci_driver *pci_drv,if (rte_eal_process_type () !=RTE_PROC_PRIMARY )return 0 ; -if (!pci_dev->mem_resource[0 ].addr) { +if (!pci_dev->pci_mem[0 ].mem_res.addr) {plt_err ("BARs have invalid values: BAR0 %p\n BAR2 %p" , - pci_dev->mem_resource[0 ].addr, - pci_dev->mem_resource[2 ].addr); + pci_dev->pci_mem[0 ].mem_res.addr, + pci_dev->pci_mem[2 ].mem_res.addr);return -ENODEV ; }
@@ -355,8 +355,8 @@ bphy_rawdev_probe(struct rte_pci_driver *pci_drv, bphy_rawdev->driver_name = pci_dev->driver->driver.name; bphy_dev = (struct bphy_device *)bphy_rawdev->dev_private; - bphy_dev->mem.res0 = pci_dev->mem_resource[0 ]; - bphy_dev->mem.res2 = pci_dev->mem_resource[2 ]; + bphy_dev->mem.res0 = pci_dev->pci_mem[0 ].mem_res; + bphy_dev->mem.res2 = pci_dev->pci_mem[2 ].mem_res; bphy_dev->bphy.pci_dev = pci_dev; ret =roc_bphy_dev_init (&bphy_dev->bphy);
drivers/raw/cnxk_bphy/cnxk_bphy_cgx.c +3 -3
@@ -302,7 +302,7 @@ cnxk_bphy_cgx_rawdev_probe(struct rte_pci_driver *pci_drv,if (rte_eal_process_type () !=RTE_PROC_PRIMARY )return 0 ; -if (!pci_dev->mem_resource[0 ].addr) +if (!pci_dev->pci_mem[0 ].mem_res.addr)return -ENODEV ; ret =roc_plt_init ();
@@ -326,8 +326,8 @@ cnxk_bphy_cgx_rawdev_probe(struct rte_pci_driver *pci_drv, } rcgx = cgx->rcgx; - rcgx->bar0_pa = pci_dev->mem_resource[0 ].phys_addr; - rcgx->bar0_va = pci_dev->mem_resource[0 ].addr; + rcgx->bar0_pa = pci_dev->pci_mem[0 ].mem_res.phys_addr; + rcgx->bar0_va = pci_dev->pci_mem[0 ].mem_res.addr; ret =roc_bphy_cgx_dev_init (rcgx);if (ret) goto out_free;
drivers/raw/ifpga/afu_pmd_n3000.c +2 -2
@@ -1524,7 +1524,7 @@ static void *n3000_afu_get_port_addr(struct afu_rawdev *dev)if (!pci_dev)return NULL ; - addr = (uint8_t *)pci_dev->mem_resource[0 ].addr; + addr = (uint8_t *)pci_dev->pci_mem[0 ].mem_res.addr; val =rte_read64 (addr +PORT_ATTR_REG (dev->port));if (!PORT_IMPLEMENTED (val)) {IFPGA_RAWDEV_PMD_INFO ("FIU port %d is not implemented" , dev->port);
@@ -1537,7 +1537,7 @@ static void *n3000_afu_get_port_addr(struct afu_rawdev *dev)return NULL ; } - addr = (uint8_t *)pci_dev->mem_resource[bar].addr +PORT_OFFSET (val); + addr = (uint8_t *)pci_dev->pci_mem[bar].mem_res.addr +PORT_OFFSET (val);return addr; }
drivers/raw/ifpga/ifpga_rawdev.c +3 -3
@@ -1580,9 +1580,9 @@ ifpga_rawdev_create(struct rte_pci_device *pci_dev,/* init opae_adapter_data_pci for device specific information */ for (i =0 ; i <PCI_MAX_RESOURCE ; i++) { - data->region[i].phys_addr = pci_dev->mem_resource[i].phys_addr; - data->region[i].len = pci_dev->mem_resource[i].len; - data->region[i].addr = pci_dev->mem_resource[i].addr; + data->region[i].phys_addr = pci_dev->pci_mem[i].mem_res.phys_addr; + data->region[i].len = pci_dev->pci_mem[i].mem_res.len; + data->region[i].addr = pci_dev->pci_mem[i].mem_res.addr; } data->device_id = pci_dev->id.device_id; data->vendor_id = pci_dev->id.vendor_id;
drivers/raw/ntb/ntb_hw_intel.c +4 -4
@@ -179,7 +179,7 @@ intel_ntb_dev_init(const struct rte_rawdev *dev)return -EINVAL ; } - hw->hw_addr = (char *)hw->pci_dev->mem_resource[0 ].addr; + hw->hw_addr = (char *)hw->pci_dev->pci_mem[0 ].mem_res.addr;if (is_gen3_ntb (hw)) ret =intel_ntb3_check_ppd (hw);
@@ -207,7 +207,7 @@ intel_ntb_dev_init(const struct rte_rawdev *dev)for (i =0 ; i < hw->mw_cnt; i++) { bar = intel_ntb_bar[i]; - hw->mw_size[i] = hw->pci_dev->mem_resource[bar].len; + hw->mw_size[i] = hw->pci_dev->pci_mem[bar].mem_res.len; }/* Reserve the last 2 spad registers for users. */
@@ -238,7 +238,7 @@ intel_ntb_get_peer_mw_addr(const struct rte_rawdev *dev, int mw_idx) bar = intel_ntb_bar[mw_idx]; -return hw->pci_dev->mem_resource[bar].addr; +return hw->pci_dev->pci_mem[bar].mem_res.addr; } static int
@@ -271,7 +271,7 @@ intel_ntb_mw_set_trans(const struct rte_rawdev *dev, int mw_idx,/* Limit reg val should be EMBAR base address plus MW size. */ base = addr; - limit = hw->pci_dev->mem_resource[bar].phys_addr + size; + limit = hw->pci_dev->pci_mem[bar].mem_res.phys_addr + size;rte_write64 (base, xlat_addr);rte_write64 (limit, limit_addr);
drivers/vdpa/ifc/ifcvf_vdpa.c +3 -3
@@ -204,11 +204,11 @@ ifcvf_vfio_setup(struct ifcvf_internal *internal)for (i =0 ; i <RTE_MIN (PCI_MAX_RESOURCE ,IFCVF_PCI_MAX_RESOURCE ); i++) { internal->hw.mem_resource[i].addr = - internal->pdev->mem_resource[i].addr; + internal->pdev->pci_mem[i].mem_res.addr; internal->hw.mem_resource[i].phys_addr = - internal->pdev->mem_resource[i].phys_addr; + internal->pdev->pci_mem[i].mem_res.phys_addr; internal->hw.mem_resource[i].len = - internal->pdev->mem_resource[i].len; + internal->pdev->pci_mem[i].mem_res.len; }return 0 ;
drivers/vdpa/sfc/sfc_vdpa_hw.c +1 -1
@@ -192,7 +192,7 @@ sfc_vdpa_mem_bar_init(struct sfc_vdpa_adapter *sva,struct rte_pci_device *pci_dev = sva->pdev;efsys_bar_t *ebp = &sva->mem_bar;struct rte_mem_resource *res = - &pci_dev->mem_resource[mem_ebrp->ebr_index]; + &pci_dev->pci_mem[mem_ebrp->ebr_index].mem_res;SFC_BAR_LOCK_INIT (ebp, pci_dev->name); ebp->esb_rid = mem_ebrp->ebr_index;
drivers/vdpa/sfc/sfc_vdpa_ops.c +1 -1
@@ -861,7 +861,7 @@ sfc_vdpa_get_notify_area(int vid, int qid, uint64_t *offset, uint64_t *size) *offset); pci_dev =sfc_vdpa_adapter_by_dev_handle (dev)->pdev; - doorbell = (uint8_t *)pci_dev->mem_resource[reg.index].addr + *offset; + doorbell = (uint8_t *)pci_dev->pci_mem[reg.index].mem_res.addr + *offset;/* * virtio-net driver in VM sends queue notifications before
Signature · 1 line
2.17.1
Hello Chenbo,
Quoted 12 lines · Chenbo Xia
This series introduces a VFIO standard capability, called sparse mmap to PCI bus. In linux kernel, it's defined as
VFIO_REGION_INFO_CAP_SPARSE_MMAP. Sparse mmap means instead of mmap whole BAR region into DPDK process, only mmap part of the BAR region after getting sparse mmap information from kernel. For the rest of BAR region that is not mmap-ed, DPDK process can use pread/pwrite system calls to access. Sparse mmap is useful when kernel does not want userspace to mmap whole BAR region, or kernel wants to control over access to specific BAR region. Vendors can choose to enable this feature or not for their devices in their specific kernel modules.
Sorry, I did not take the time to look into the details. Could you summarize what would be the benefit of this series?
Quoted 54 lines
In this patchset:
Patch 1-3 is mainly for introducing BAR access APIs so that driver could use them to access specific BAR using pread/pwrite system calls when part of the BAR is not mmap-able.
Patch 4 adds the VFIO sparse mmap support finally. A question is for all sparse mmap regions, should they be mapped to a continuous virtual address region that follows device-specific BAR layout or not. In theory, there could be three options to support this feature.
Option 1: Map sparse mmap regions independently
======================================================
In this approach, we mmap each sparse mmap region one by one and each region could be located anywhere in process address space. But accessing the mmaped BAR will not be as easy as 'bar_base_address+bar_offset', driver needs to check the sparse mmap information to access specific BAR register.Patch 4 in this patchset adopts this option. Driver API change is introduced in
bus_pci_driver.h. Corresponding changes in all drivers are also done and currently I am assuming drivers do not support this feature so they will not check the 'is_sparse' flag but assumes it to be false. Note that it will not break any driver and each vendor can add related logic when they start to support this feature. This is only because I don't want to introduce complexity to drivers that do not want to support this feature.Option 2: Map sparse mmap regions based on device-specific BAR layout
======================================================================
In this approach, the sparse mmap regions are mapped to continuous virtual address region that follows device-specific BAR layout. For example, the BAR size is0x4000and only 0-0x1000 (sparse mmap region #1) and 0x3000-0x4000 (sparse mmap region #2) could be mmaped. Region #1 will be mapped at 'base_addr' and region #2 will be mapped at 'base_addr+0x3000'. The good thing is if we implement like this, driver can still access all BAR registers using 'bar_base_address+bar_offset' way and we don't need to introduce any driver API change. But the address space range 'base_addr+0x1000' to 'base_addr+0x3000' may need to be reserved so it could result in waste of address space or memory (when we useMAP_ANONYMOUSandMAP_PRIVATEflag to reserve this range). Meanwhile, driver needs to know which part of BAR is mmaped (this is possible since the range is defined by vendor's specific kernel module).Option 3: Support both option 1 & 2
===================================
We could define a driver flag to let driver choose which way it perfers since either option has its own Pros & Cons.
Quoted 5 lines
Please share your comments, Thanks!
Chenbo Xia (4):
bus/pci: introduce an internal representation of PCI device
I think this first patch main motivation was to avoid ABI issues. Since v22.11, the rte_pci_device object is opaque to applications.
So, do we still need this patch?
Quoted 95 lines
bus/pci: avoid depending on private value in kernel source bus/pci: introduce helper for MMIO read and write bus/pci: add VFIO sparse mmap support
drivers/baseband/acc/rte_acc100_pmd.c | 6 +- drivers/baseband/acc/rte_vrb_pmd.c | 6 +- .../fpga_5gnr_fec/rte_fpga_5gnr_fec.c | 6 +- drivers/baseband/fpga_lte_fec/fpga_lte_fec.c | 6 +- drivers/bus/pci/bsd/pci.c | 43 +- drivers/bus/pci/bus_pci_driver.h | 24 +- drivers/bus/pci/linux/pci.c | 91 +++- drivers/bus/pci/linux/pci_init.h | 14 +- drivers/bus/pci/linux/pci_uio.c | 34 +- drivers/bus/pci/linux/pci_vfio.c | 445 ++++++++++++++---- drivers/bus/pci/pci_common.c | 57 ++- drivers/bus/pci/pci_common_uio.c | 12 +- drivers/bus/pci/private.h | 25 +- drivers/bus/pci/rte_bus_pci.h | 48 ++ drivers/bus/pci/version.map | 3 + drivers/common/cnxk/roc_dev.c | 4 +- drivers/common/cnxk/roc_dpi.c | 2 +- drivers/common/cnxk/roc_ml.c | 22 +- drivers/common/qat/dev/qat_dev_gen1.c | 2 +- drivers/common/qat/dev/qat_dev_gen4.c | 4 +- drivers/common/sfc_efx/sfc_efx.c | 2 +- drivers/compress/octeontx/otx_zip.c | 4 +- drivers/crypto/ccp/ccp_dev.c | 4 +- drivers/crypto/cnxk/cnxk_cryptodev_ops.c | 2 +- drivers/crypto/nitrox/nitrox_device.c | 4 +- drivers/crypto/octeontx/otx_cryptodev_ops.c | 6 +- drivers/crypto/virtio/virtio_pci.c | 6 +- drivers/dma/cnxk/cnxk_dmadev.c | 2 +- drivers/dma/hisilicon/hisi_dmadev.c | 6 +- drivers/dma/idxd/idxd_pci.c | 4 +- drivers/dma/ioat/ioat_dmadev.c | 2 +- drivers/event/dlb2/pf/dlb2_main.c | 16 +- drivers/event/octeontx/ssovf_probe.c | 38 +- drivers/event/octeontx/timvf_probe.c | 18 +- drivers/event/skeleton/skeleton_eventdev.c | 2 +- drivers/mempool/octeontx/octeontx_fpavf.c | 6 +- drivers/net/ark/ark_ethdev.c | 4 +- drivers/net/atlantic/atl_ethdev.c | 2 +- drivers/net/avp/avp_ethdev.c | 20 +- drivers/net/axgbe/axgbe_ethdev.c | 4 +- drivers/net/bnx2x/bnx2x_ethdev.c | 6 +- drivers/net/bnxt/bnxt_ethdev.c | 8 +- drivers/net/cpfl/cpfl_ethdev.c | 4 +- drivers/net/cxgbe/cxgbe_ethdev.c | 2 +- drivers/net/cxgbe/cxgbe_main.c | 2 +- drivers/net/cxgbe/cxgbevf_ethdev.c | 2 +- drivers/net/cxgbe/cxgbevf_main.c | 2 +- drivers/net/e1000/em_ethdev.c | 4 +- drivers/net/e1000/igb_ethdev.c | 4 +- drivers/net/ena/ena_ethdev.c | 4 +- drivers/net/enetc/enetc_ethdev.c | 2 +- drivers/net/enic/enic_main.c | 4 +- drivers/net/fm10k/fm10k_ethdev.c | 2 +- drivers/net/gve/gve_ethdev.c | 4 +- drivers/net/hinic/base/hinic_pmd_hwif.c | 14 +- drivers/net/hns3/hns3_ethdev.c | 2 +- drivers/net/hns3/hns3_ethdev_vf.c | 2 +- drivers/net/hns3/hns3_rxtx.c | 4 +- drivers/net/i40e/i40e_ethdev.c | 2 +- drivers/net/iavf/iavf_ethdev.c | 2 +- drivers/net/ice/ice_dcf.c | 2 +- drivers/net/ice/ice_ethdev.c | 2 +- drivers/net/idpf/idpf_ethdev.c | 4 +- drivers/net/igc/igc_ethdev.c | 2 +- drivers/net/ionic/ionic_dev_pci.c | 2 +- drivers/net/ixgbe/ixgbe_ethdev.c | 4 +- drivers/net/liquidio/lio_ethdev.c | 4 +- drivers/net/nfp/nfp_ethdev.c | 2 +- drivers/net/nfp/nfp_ethdev_vf.c | 6 +- drivers/net/nfp/nfpcore/nfp_cpp_pcie_ops.c | 4 +- drivers/net/ngbe/ngbe_ethdev.c | 2 +- drivers/net/octeon_ep/otx_ep_ethdev.c | 2 +- drivers/net/octeontx/base/octeontx_pkivf.c | 6 +- drivers/net/octeontx/base/octeontx_pkovf.c | 12 +- drivers/net/qede/qede_main.c | 6 +- drivers/net/sfc/sfc.c | 2 +- drivers/net/thunderx/nicvf_ethdev.c | 2 +- drivers/net/txgbe/txgbe_ethdev.c | 2 +- drivers/net/txgbe/txgbe_ethdev_vf.c | 2 +- drivers/net/virtio/virtio_pci.c | 6 +- drivers/net/vmxnet3/vmxnet3_ethdev.c | 4 +- drivers/raw/cnxk_bphy/cnxk_bphy.c | 10 +- drivers/raw/cnxk_bphy/cnxk_bphy_cgx.c | 6 +- drivers/raw/ifpga/afu_pmd_n3000.c | 4 +- drivers/raw/ifpga/ifpga_rawdev.c | 6 +- drivers/raw/ntb/ntb_hw_intel.c | 8 +- drivers/vdpa/ifc/ifcvf_vdpa.c | 6 +- drivers/vdpa/sfc/sfc_vdpa_hw.c | 2 +- drivers/vdpa/sfc/sfc_vdpa_ops.c | 2 +- lib/eal/include/rte_vfio.h | 1 - 90 files changed, 853 insertions(+), 352 deletions(-)
Signature · 1 line
David Marchand
Hi David,
Quoted 23 lines
-----Original Message-----
From: David Marchand <[email protected]>
Sent: Tuesday, April 18, 2023 3:47 PM
To: Xia, Chenbo <[email protected]>
Cc: [email protected]; [email protected]
Subject: Re: [RFC 0/4] Support VFIO sparse mmap in PCI busHello Chenbo,
Quoted 12 lines · Chenbo Xia
This series introduces a VFIO standard capability, called sparse mmap to PCI bus. In linux kernel, it's defined as
VFIO_REGION_INFO_CAP_SPARSE_MMAP. Sparse mmap means instead of mmap whole BAR region into DPDK process, only mmap part of the BAR region after getting sparse mmap information from kernel. For the rest of BAR region that is not mmap-ed, DPDK process can use pread/pwrite system calls to access. Sparse mmap is useful when kernel does not want userspace to mmap whole BAR region, or kernel wants to control over access to specific BAR region. Vendors can choose to enable this feature or not for their devices in their specific kernel modules.
Quoted 2 lines
Sorry, I did not take the time to look into the details. Could you summarize what would be the benefit of this series?
It could be different benefit for different vendor. There was one discussion:
http://inbox.dpdk.org/dev/CO6PR18MB386016A2634AF375F5B4BA8CB4899@CO6PR18MB3860.namprd18.prod.outlook.com/
Above problem is some device has very large BAR, and we don't want DPDK to map the whole BAR.
For Intel devices, one benefit is that we want our kernel module to control over access to specific BAR region so we will let DPDK process unable to mmap that region. (Because after mmap, kernel will not know if userspace is accessing device BAR).
So that's why I summarize as 'Sparse mmap is useful when kernel does not want userspace to mmap whole BAR region, or kernel wants to control over access to specific BAR region'. It could be more usage for other vendors that I have not realized
Thanks,
Chenbo
Quoted 167 lines
Quoted 56 lines
In this patchset:
Patch 1-3 is mainly for introducing BAR access APIs so that driver could use them to access specific BAR using pread/pwrite system calls when part of the BAR is not mmap-able.
Patch 4 adds the VFIO sparse mmap support finally. A question is for all sparse mmap regions, should they be mapped to a continuous virtual address region that follows device-specific BAR layout or not. In theory, there could be three options to support this feature.
Option 1: Map sparse mmap regions independently
======================================================
In this approach, we mmap each sparse mmap region one by one and each region could be located anywhere in process address space. But accessing the mmaped BAR will not be as easy as 'bar_base_address+bar_offset', driver needs to check the sparse mmap information to access specific BAR register.Patch 4 in this patchset adopts this option. Driver API change is introduced in
bus_pci_driver.h. Corresponding changes in all drivers are also done and currently I am assuming drivers do not support this feature so they will not check the 'is_sparse' flag but assumes it to be false. Note that it will not break any driver and each vendor can add related logic when they start to support this feature. This is only because I don't want to introduce complexity to drivers that do not want to support this feature.Option 2: Map sparse mmap regions based on device-specific BAR layout
======================================================================
In this approach, the sparse mmap regions are mapped to continuous virtual address region that follows device-specific BAR layout. For example, the BAR size is0x4000and only 0-0x1000 (sparse mmap region #1) and 0x3000-0x4000 (sparse mmap region #2) could be mmaped. Region #1 will be mapped at 'base_addr' and region #2 will be mapped at 'base_addr+0x3000'. The good thing is if we implement like this, driver can still access all BAR registers using 'bar_base_address+bar_offset' way and we don't need to introduce any driver API change. But the address space range 'base_addr+0x1000' to 'base_addr+0x3000' may need to be reserved so it could result in waste of address space or memory (when we useMAP_ANONYMOUSandMAP_PRIVATEflag to reserve this range). Meanwhile, driver needs to know which part of BAR is mmaped (this is possible since the range is defined by vendor's specific kernel module).Option 3: Support both option 1 & 2
===================================
We could define a driver flag to let driver choose which way it perfers since either option has its own Pros & Cons.Quoted 5 lines
Please share your comments, Thanks!
Chenbo Xia (4):
bus/pci: introduce an internal representation of PCI deviceI think this first patch main motivation was to avoid ABI issues. Since v22.11, the
rte_pci_deviceobject is opaque to applications.So, do we still need this patch?
Quoted 87 lines
bus/pci: avoid depending on private value in kernel source bus/pci: introduce helper for MMIO read and write bus/pci: add VFIO sparse mmap support
drivers/baseband/acc/rte_acc100_pmd.c | 6 +- drivers/baseband/acc/rte_vrb_pmd.c | 6 +- .../fpga_5gnr_fec/rte_fpga_5gnr_fec.c | 6 +- drivers/baseband/fpga_lte_fec/fpga_lte_fec.c | 6 +- drivers/bus/pci/bsd/pci.c | 43 +- drivers/bus/pci/bus_pci_driver.h | 24 +- drivers/bus/pci/linux/pci.c | 91 +++- drivers/bus/pci/linux/pci_init.h | 14 +- drivers/bus/pci/linux/pci_uio.c | 34 +- drivers/bus/pci/linux/pci_vfio.c | 445 ++++++++++++++---- drivers/bus/pci/pci_common.c | 57 ++- drivers/bus/pci/pci_common_uio.c | 12 +- drivers/bus/pci/private.h | 25 +- drivers/bus/pci/rte_bus_pci.h | 48 ++ drivers/bus/pci/version.map | 3 + drivers/common/cnxk/roc_dev.c | 4 +- drivers/common/cnxk/roc_dpi.c | 2 +- drivers/common/cnxk/roc_ml.c | 22 +- drivers/common/qat/dev/qat_dev_gen1.c | 2 +- drivers/common/qat/dev/qat_dev_gen4.c | 4 +- drivers/common/sfc_efx/sfc_efx.c | 2 +- drivers/compress/octeontx/otx_zip.c | 4 +- drivers/crypto/ccp/ccp_dev.c | 4 +- drivers/crypto/cnxk/cnxk_cryptodev_ops.c | 2 +- drivers/crypto/nitrox/nitrox_device.c | 4 +- drivers/crypto/octeontx/otx_cryptodev_ops.c | 6 +- drivers/crypto/virtio/virtio_pci.c | 6 +- drivers/dma/cnxk/cnxk_dmadev.c | 2 +- drivers/dma/hisilicon/hisi_dmadev.c | 6 +- drivers/dma/idxd/idxd_pci.c | 4 +- drivers/dma/ioat/ioat_dmadev.c | 2 +- drivers/event/dlb2/pf/dlb2_main.c | 16 +- drivers/event/octeontx/ssovf_probe.c | 38 +- drivers/event/octeontx/timvf_probe.c | 18 +- drivers/event/skeleton/skeleton_eventdev.c | 2 +- drivers/mempool/octeontx/octeontx_fpavf.c | 6 +- drivers/net/ark/ark_ethdev.c | 4 +- drivers/net/atlantic/atl_ethdev.c | 2 +- drivers/net/avp/avp_ethdev.c | 20 +- drivers/net/axgbe/axgbe_ethdev.c | 4 +- drivers/net/bnx2x/bnx2x_ethdev.c | 6 +- drivers/net/bnxt/bnxt_ethdev.c | 8 +- drivers/net/cpfl/cpfl_ethdev.c | 4 +- drivers/net/cxgbe/cxgbe_ethdev.c | 2 +- drivers/net/cxgbe/cxgbe_main.c | 2 +- drivers/net/cxgbe/cxgbevf_ethdev.c | 2 +- drivers/net/cxgbe/cxgbevf_main.c | 2 +- drivers/net/e1000/em_ethdev.c | 4 +- drivers/net/e1000/igb_ethdev.c | 4 +- drivers/net/ena/ena_ethdev.c | 4 +- drivers/net/enetc/enetc_ethdev.c | 2 +- drivers/net/enic/enic_main.c | 4 +- drivers/net/fm10k/fm10k_ethdev.c | 2 +- drivers/net/gve/gve_ethdev.c | 4 +- drivers/net/hinic/base/hinic_pmd_hwif.c | 14 +- drivers/net/hns3/hns3_ethdev.c | 2 +- drivers/net/hns3/hns3_ethdev_vf.c | 2 +- drivers/net/hns3/hns3_rxtx.c | 4 +- drivers/net/i40e/i40e_ethdev.c | 2 +- drivers/net/iavf/iavf_ethdev.c | 2 +- drivers/net/ice/ice_dcf.c | 2 +- drivers/net/ice/ice_ethdev.c | 2 +- drivers/net/idpf/idpf_ethdev.c | 4 +- drivers/net/igc/igc_ethdev.c | 2 +- drivers/net/ionic/ionic_dev_pci.c | 2 +- drivers/net/ixgbe/ixgbe_ethdev.c | 4 +- drivers/net/liquidio/lio_ethdev.c | 4 +- drivers/net/nfp/nfp_ethdev.c | 2 +- drivers/net/nfp/nfp_ethdev_vf.c | 6 +- drivers/net/nfp/nfpcore/nfp_cpp_pcie_ops.c | 4 +- drivers/net/ngbe/ngbe_ethdev.c | 2 +- drivers/net/octeon_ep/otx_ep_ethdev.c | 2 +- drivers/net/octeontx/base/octeontx_pkivf.c | 6 +- drivers/net/octeontx/base/octeontx_pkovf.c | 12 +- drivers/net/qede/qede_main.c | 6 +- drivers/net/sfc/sfc.c | 2 +- drivers/net/thunderx/nicvf_ethdev.c | 2 +- drivers/net/txgbe/txgbe_ethdev.c | 2 +- drivers/net/txgbe/txgbe_ethdev_vf.c | 2 +- drivers/net/virtio/virtio_pci.c | 6 +- drivers/net/vmxnet3/vmxnet3_ethdev.c | 4 +- drivers/raw/cnxk_bphy/cnxk_bphy.c | 10 +- drivers/raw/cnxk_bphy/cnxk_bphy_cgx.c | 6 +-Quoted 8 lines
drivers/raw/ifpga/afu_pmd_n3000.c | 4 +- drivers/raw/ifpga/ifpga_rawdev.c | 6 +- drivers/raw/ntb/ntb_hw_intel.c | 8 +- drivers/vdpa/ifc/ifcvf_vdpa.c | 6 +- drivers/vdpa/sfc/sfc_vdpa_hw.c | 2 +- drivers/vdpa/sfc/sfc_vdpa_ops.c | 2 +- lib/eal/include/rte_vfio.h | 1 - 90 files changed, 853 insertions(+), 352 deletions(-)--
David Marchand
David,
Sorry that I missed one comment...
Quoted 84 lines
-----Original Message-----
From: David Marchand <[email protected]>
Sent: Tuesday, April 18, 2023 3:47 PM
To: Xia, Chenbo <[email protected]>
Cc: [email protected]; [email protected]
Subject: Re: [RFC 0/4] Support VFIO sparse mmap in PCI busHello Chenbo,
Quoted 12 lines · Chenbo Xia
This series introduces a VFIO standard capability, called sparse mmap to PCI bus. In linux kernel, it's defined as
VFIO_REGION_INFO_CAP_SPARSE_MMAP. Sparse mmap means instead of mmap whole BAR region into DPDK process, only mmap part of the BAR region after getting sparse mmap information from kernel. For the rest of BAR region that is not mmap-ed, DPDK process can use pread/pwrite system calls to access. Sparse mmap is useful when kernel does not want userspace to mmap whole BAR region, or kernel wants to control over access to specific BAR region. Vendors can choose to enable this feature or not for their devices in their specific kernel modules.Sorry, I did not take the time to look into the details. Could you summarize what would be the benefit of this series?
Quoted 57 lines
In this patchset:
Patch 1-3 is mainly for introducing BAR access APIs so that driver could use them to access specific BAR using pread/pwrite system calls when part of the BAR is not mmap-able.
Patch 4 adds the VFIO sparse mmap support finally. A question is for all sparse mmap regions, should they be mapped to a continuous virtual address region that follows device-specific BAR layout or not. In theory, there could be three options to support this feature.
Option 1: Map sparse mmap regions independently
======================================================
In this approach, we mmap each sparse mmap region one by one and each region could be located anywhere in process address space. But accessing the mmaped BAR will not be as easy as 'bar_base_address+bar_offset', driver needs to check the sparse mmap information to access specific BAR register.Patch 4 in this patchset adopts this option. Driver API change is introduced in
bus_pci_driver.h. Corresponding changes in all drivers are also done and currently I am assuming drivers do not support this feature so they will not check the 'is_sparse' flag but assumes it to be false. Note that it will not break any driver and each vendor can add related logic when they start to support this feature. This is only because I don't want to introduce complexity to drivers that do not want to support this feature.Option 2: Map sparse mmap regions based on device-specific BAR layout
======================================================================
In this approach, the sparse mmap regions are mapped to continuous virtual address region that follows device-specific BAR layout. For example, the BAR size is0x4000and only 0-0x1000 (sparse mmap region #1) and 0x3000-0x4000 (sparse mmap region #2) could be mmaped. Region #1 will be mapped at 'base_addr' and region #2 will be mapped at 'base_addr+0x3000'. The good thing is if we implement like this, driver can still access all BAR registers using 'bar_base_address+bar_offset' way and we don't need to introduce any driver API change. But the address space range 'base_addr+0x1000' to 'base_addr+0x3000' may need to be reserved so it could result in waste of address space or memory (when we useMAP_ANONYMOUSandMAP_PRIVATEflag to reserve this range). Meanwhile, driver needs to know which part of BAR is mmaped (this is possible since the range is defined by vendor's specific kernel module).Option 3: Support both option 1 & 2
===================================
We could define a driver flag to let driver choose which way it perfers since either option has its own Pros & Cons.Please share your comments, Thanks!
Quoted 7 lines
Quoted 2 lines
Chenbo Xia (4):
bus/pci: introduce an internal representation of PCI deviceI think this first patch main motivation was to avoid ABI issues. Since v22.11, the
rte_pci_deviceobject is opaque to applications.So, do we still need this patch?
I think it could be good to reduce unnecessary driver APIs.. Hiding these region information could be friendly to driver developer?
Thanks,
Chenbo
Quoted 101 lines
Quoted 89 lines
bus/pci: avoid depending on private value in kernel source bus/pci: introduce helper for MMIO read and write bus/pci: add VFIO sparse mmap support
drivers/baseband/acc/rte_acc100_pmd.c | 6 +- drivers/baseband/acc/rte_vrb_pmd.c | 6 +- .../fpga_5gnr_fec/rte_fpga_5gnr_fec.c | 6 +- drivers/baseband/fpga_lte_fec/fpga_lte_fec.c | 6 +- drivers/bus/pci/bsd/pci.c | 43 +- drivers/bus/pci/bus_pci_driver.h | 24 +- drivers/bus/pci/linux/pci.c | 91 +++- drivers/bus/pci/linux/pci_init.h | 14 +- drivers/bus/pci/linux/pci_uio.c | 34 +- drivers/bus/pci/linux/pci_vfio.c | 445 ++++++++++++++---- drivers/bus/pci/pci_common.c | 57 ++- drivers/bus/pci/pci_common_uio.c | 12 +- drivers/bus/pci/private.h | 25 +- drivers/bus/pci/rte_bus_pci.h | 48 ++ drivers/bus/pci/version.map | 3 + drivers/common/cnxk/roc_dev.c | 4 +- drivers/common/cnxk/roc_dpi.c | 2 +- drivers/common/cnxk/roc_ml.c | 22 +- drivers/common/qat/dev/qat_dev_gen1.c | 2 +- drivers/common/qat/dev/qat_dev_gen4.c | 4 +- drivers/common/sfc_efx/sfc_efx.c | 2 +- drivers/compress/octeontx/otx_zip.c | 4 +- drivers/crypto/ccp/ccp_dev.c | 4 +- drivers/crypto/cnxk/cnxk_cryptodev_ops.c | 2 +- drivers/crypto/nitrox/nitrox_device.c | 4 +- drivers/crypto/octeontx/otx_cryptodev_ops.c | 6 +- drivers/crypto/virtio/virtio_pci.c | 6 +- drivers/dma/cnxk/cnxk_dmadev.c | 2 +- drivers/dma/hisilicon/hisi_dmadev.c | 6 +- drivers/dma/idxd/idxd_pci.c | 4 +- drivers/dma/ioat/ioat_dmadev.c | 2 +- drivers/event/dlb2/pf/dlb2_main.c | 16 +- drivers/event/octeontx/ssovf_probe.c | 38 +- drivers/event/octeontx/timvf_probe.c | 18 +- drivers/event/skeleton/skeleton_eventdev.c | 2 +- drivers/mempool/octeontx/octeontx_fpavf.c | 6 +- drivers/net/ark/ark_ethdev.c | 4 +- drivers/net/atlantic/atl_ethdev.c | 2 +- drivers/net/avp/avp_ethdev.c | 20 +- drivers/net/axgbe/axgbe_ethdev.c | 4 +- drivers/net/bnx2x/bnx2x_ethdev.c | 6 +- drivers/net/bnxt/bnxt_ethdev.c | 8 +- drivers/net/cpfl/cpfl_ethdev.c | 4 +- drivers/net/cxgbe/cxgbe_ethdev.c | 2 +- drivers/net/cxgbe/cxgbe_main.c | 2 +- drivers/net/cxgbe/cxgbevf_ethdev.c | 2 +- drivers/net/cxgbe/cxgbevf_main.c | 2 +- drivers/net/e1000/em_ethdev.c | 4 +- drivers/net/e1000/igb_ethdev.c | 4 +- drivers/net/ena/ena_ethdev.c | 4 +- drivers/net/enetc/enetc_ethdev.c | 2 +- drivers/net/enic/enic_main.c | 4 +- drivers/net/fm10k/fm10k_ethdev.c | 2 +- drivers/net/gve/gve_ethdev.c | 4 +- drivers/net/hinic/base/hinic_pmd_hwif.c | 14 +- drivers/net/hns3/hns3_ethdev.c | 2 +- drivers/net/hns3/hns3_ethdev_vf.c | 2 +- drivers/net/hns3/hns3_rxtx.c | 4 +- drivers/net/i40e/i40e_ethdev.c | 2 +- drivers/net/iavf/iavf_ethdev.c | 2 +- drivers/net/ice/ice_dcf.c | 2 +- drivers/net/ice/ice_ethdev.c | 2 +- drivers/net/idpf/idpf_ethdev.c | 4 +- drivers/net/igc/igc_ethdev.c | 2 +- drivers/net/ionic/ionic_dev_pci.c | 2 +- drivers/net/ixgbe/ixgbe_ethdev.c | 4 +- drivers/net/liquidio/lio_ethdev.c | 4 +- drivers/net/nfp/nfp_ethdev.c | 2 +- drivers/net/nfp/nfp_ethdev_vf.c | 6 +- drivers/net/nfp/nfpcore/nfp_cpp_pcie_ops.c | 4 +- drivers/net/ngbe/ngbe_ethdev.c | 2 +- drivers/net/octeon_ep/otx_ep_ethdev.c | 2 +- drivers/net/octeontx/base/octeontx_pkivf.c | 6 +- drivers/net/octeontx/base/octeontx_pkovf.c | 12 +- drivers/net/qede/qede_main.c | 6 +- drivers/net/sfc/sfc.c | 2 +- drivers/net/thunderx/nicvf_ethdev.c | 2 +- drivers/net/txgbe/txgbe_ethdev.c | 2 +- drivers/net/txgbe/txgbe_ethdev_vf.c | 2 +- drivers/net/virtio/virtio_pci.c | 6 +- drivers/net/vmxnet3/vmxnet3_ethdev.c | 4 +- drivers/raw/cnxk_bphy/cnxk_bphy.c | 10 +- drivers/raw/cnxk_bphy/cnxk_bphy_cgx.c | 6 +-Quoted 8 lines
drivers/raw/ifpga/afu_pmd_n3000.c | 4 +- drivers/raw/ifpga/ifpga_rawdev.c | 6 +- drivers/raw/ntb/ntb_hw_intel.c | 8 +- drivers/vdpa/ifc/ifcvf_vdpa.c | 6 +- drivers/vdpa/sfc/sfc_vdpa_hw.c | 2 +- drivers/vdpa/sfc/sfc_vdpa_ops.c | 2 +- lib/eal/include/rte_vfio.h | 1 - 90 files changed, 853 insertions(+), 352 deletions(-)--
David Marchand
Gentle ping for some comments..
After rethink, personally I may choose option 2 as it will have no driver API change for PMDs and as I look at current code, that's how we do when MSI-X table can't be mmap-ed.
Thanks,
Chenbo
Quoted 175 lines
-----Original Message-----
From: Chenbo Xia <[email protected]>
Sent: Tuesday, April 18, 2023 1:30 PM
To: [email protected]
Cc: [email protected]
Subject: [RFC 0/4] Support VFIO sparse mmap in PCI busThis series introduces a VFIO standard capability, called sparse mmap to PCI bus. In linux kernel, it's defined as
VFIO_REGION_INFO_CAP_SPARSE_MMAP. Sparse mmap means instead of mmap whole BAR region into DPDK process, only mmap part of the BAR region after getting sparse mmap information from kernel. For the rest of BAR region that is not mmap-ed, DPDK process can use pread/pwrite system calls to access. Sparse mmap is useful when kernel does not want userspace to mmap whole BAR region, or kernel wants to control over access to specific BAR region. Vendors can choose to enable this feature or not for their devices in their specific kernel modules.In this patchset:
Patch 1-3 is mainly for introducing BAR access APIs so that driver could use them to access specific BAR using pread/pwrite system calls when part of the BAR is not mmap-able.
Patch 4 adds the VFIO sparse mmap support finally. A question is for all sparse mmap regions, should they be mapped to a continuous virtual address region that follows device-specific BAR layout or not. In theory, there could be three options to support this feature.
Option 1: Map sparse mmap regions independently
======================================================
In this approach, we mmap each sparse mmap region one by one and each region could be located anywhere in process address space. But accessing the mmaped BAR will not be as easy as 'bar_base_address+bar_offset', driver needs to check the sparse mmap information to access specific BAR register.Patch 4 in this patchset adopts this option. Driver API change is introduced in
bus_pci_driver.h. Corresponding changes in all drivers are also done and currently I am assuming drivers do not support this feature so they will not check the 'is_sparse' flag but assumes it to be false. Note that it will not break any driver and each vendor can add related logic when they start to support this feature. This is only because I don't want to introduce complexity to drivers that do not want to support this feature.Option 2: Map sparse mmap regions based on device-specific BAR layout
======================================================================
In this approach, the sparse mmap regions are mapped to continuous virtual address region that follows device-specific BAR layout. For example, the BAR size is0x4000and only 0-0x1000 (sparse mmap region #1) and 0x3000-0x4000 (sparse mmap region #2) could be mmaped. Region #1 will be mapped at 'base_addr' and region #2 will be mapped at 'base_addr+0x3000'. The good thing is if we implement like this, driver can still access all BAR registers using 'bar_base_address+bar_offset' way and we don't need to introduce any driver API change. But the address space range 'base_addr+0x1000' to 'base_addr+0x3000' may need to be reserved so it could result in waste of address space or memory (when we useMAP_ANONYMOUSandMAP_PRIVATEflag to reserve this range). Meanwhile, driver needs to know which part of BAR is mmaped (this is possible since the range is defined by vendor's specific kernel module).Option 3: Support both option 1 & 2
===================================
We could define a driver flag to let driver choose which way it perfers since either option has its own Pros & Cons.Please share your comments, Thanks!
Chenbo Xia (4):
bus/pci: introduce an internal representation of PCI device bus/pci: avoid depending on private value in kernel source bus/pci: introduce helper for MMIO read and write bus/pci: add VFIO sparse mmap supportdrivers/baseband/acc/rte_acc100_pmd.c | 6 +- drivers/baseband/acc/rte_vrb_pmd.c | 6 +- .../fpga_5gnr_fec/rte_fpga_5gnr_fec.c | 6 +- drivers/baseband/fpga_lte_fec/fpga_lte_fec.c | 6 +- drivers/bus/pci/bsd/pci.c | 43 +- drivers/bus/pci/bus_pci_driver.h | 24 +- drivers/bus/pci/linux/pci.c | 91 +++- drivers/bus/pci/linux/pci_init.h | 14 +- drivers/bus/pci/linux/pci_uio.c | 34 +- drivers/bus/pci/linux/pci_vfio.c | 445 ++++++++++++++---- drivers/bus/pci/pci_common.c | 57 ++- drivers/bus/pci/pci_common_uio.c | 12 +- drivers/bus/pci/private.h | 25 +- drivers/bus/pci/rte_bus_pci.h | 48 ++ drivers/bus/pci/version.map | 3 + drivers/common/cnxk/roc_dev.c | 4 +- drivers/common/cnxk/roc_dpi.c | 2 +- drivers/common/cnxk/roc_ml.c | 22 +- drivers/common/qat/dev/qat_dev_gen1.c | 2 +- drivers/common/qat/dev/qat_dev_gen4.c | 4 +- drivers/common/sfc_efx/sfc_efx.c | 2 +- drivers/compress/octeontx/otx_zip.c | 4 +- drivers/crypto/ccp/ccp_dev.c | 4 +- drivers/crypto/cnxk/cnxk_cryptodev_ops.c | 2 +- drivers/crypto/nitrox/nitrox_device.c | 4 +- drivers/crypto/octeontx/otx_cryptodev_ops.c | 6 +- drivers/crypto/virtio/virtio_pci.c | 6 +- drivers/dma/cnxk/cnxk_dmadev.c | 2 +- drivers/dma/hisilicon/hisi_dmadev.c | 6 +- drivers/dma/idxd/idxd_pci.c | 4 +- drivers/dma/ioat/ioat_dmadev.c | 2 +- drivers/event/dlb2/pf/dlb2_main.c | 16 +- drivers/event/octeontx/ssovf_probe.c | 38 +- drivers/event/octeontx/timvf_probe.c | 18 +- drivers/event/skeleton/skeleton_eventdev.c | 2 +- drivers/mempool/octeontx/octeontx_fpavf.c | 6 +- drivers/net/ark/ark_ethdev.c | 4 +- drivers/net/atlantic/atl_ethdev.c | 2 +- drivers/net/avp/avp_ethdev.c | 20 +- drivers/net/axgbe/axgbe_ethdev.c | 4 +- drivers/net/bnx2x/bnx2x_ethdev.c | 6 +- drivers/net/bnxt/bnxt_ethdev.c | 8 +- drivers/net/cpfl/cpfl_ethdev.c | 4 +- drivers/net/cxgbe/cxgbe_ethdev.c | 2 +- drivers/net/cxgbe/cxgbe_main.c | 2 +- drivers/net/cxgbe/cxgbevf_ethdev.c | 2 +- drivers/net/cxgbe/cxgbevf_main.c | 2 +- drivers/net/e1000/em_ethdev.c | 4 +- drivers/net/e1000/igb_ethdev.c | 4 +- drivers/net/ena/ena_ethdev.c | 4 +- drivers/net/enetc/enetc_ethdev.c | 2 +- drivers/net/enic/enic_main.c | 4 +- drivers/net/fm10k/fm10k_ethdev.c | 2 +- drivers/net/gve/gve_ethdev.c | 4 +- drivers/net/hinic/base/hinic_pmd_hwif.c | 14 +- drivers/net/hns3/hns3_ethdev.c | 2 +- drivers/net/hns3/hns3_ethdev_vf.c | 2 +- drivers/net/hns3/hns3_rxtx.c | 4 +- drivers/net/i40e/i40e_ethdev.c | 2 +- drivers/net/iavf/iavf_ethdev.c | 2 +- drivers/net/ice/ice_dcf.c | 2 +- drivers/net/ice/ice_ethdev.c | 2 +- drivers/net/idpf/idpf_ethdev.c | 4 +- drivers/net/igc/igc_ethdev.c | 2 +- drivers/net/ionic/ionic_dev_pci.c | 2 +- drivers/net/ixgbe/ixgbe_ethdev.c | 4 +- drivers/net/liquidio/lio_ethdev.c | 4 +- drivers/net/nfp/nfp_ethdev.c | 2 +- drivers/net/nfp/nfp_ethdev_vf.c | 6 +- drivers/net/nfp/nfpcore/nfp_cpp_pcie_ops.c | 4 +- drivers/net/ngbe/ngbe_ethdev.c | 2 +- drivers/net/octeon_ep/otx_ep_ethdev.c | 2 +- drivers/net/octeontx/base/octeontx_pkivf.c | 6 +- drivers/net/octeontx/base/octeontx_pkovf.c | 12 +- drivers/net/qede/qede_main.c | 6 +- drivers/net/sfc/sfc.c | 2 +- drivers/net/thunderx/nicvf_ethdev.c | 2 +- drivers/net/txgbe/txgbe_ethdev.c | 2 +- drivers/net/txgbe/txgbe_ethdev_vf.c | 2 +- drivers/net/virtio/virtio_pci.c | 6 +- drivers/net/vmxnet3/vmxnet3_ethdev.c | 4 +- drivers/raw/cnxk_bphy/cnxk_bphy.c | 10 +- drivers/raw/cnxk_bphy/cnxk_bphy_cgx.c | 6 +- drivers/raw/ifpga/afu_pmd_n3000.c | 4 +- drivers/raw/ifpga/ifpga_rawdev.c | 6 +- drivers/raw/ntb/ntb_hw_intel.c | 8 +- drivers/vdpa/ifc/ifcvf_vdpa.c | 6 +- drivers/vdpa/sfc/sfc_vdpa_hw.c | 2 +- drivers/vdpa/sfc/sfc_vdpa_ops.c | 2 +- lib/eal/include/rte_vfio.h | 1 - 90 files changed, 853 insertions(+), 352 deletions(-)--
2.17.1
+1 for option 2.
Thanks & Regards
Sunil Kumar Kori
Quoted 198 lines
-----Original Message-----
From: Xia, Chenbo <[email protected]>
Sent: Monday, May 8, 2023 7:43 AM
To: [email protected]
Cc: Sunil Kumar Kori <[email protected]>; [email protected]; [email protected]; Richardson, Bruce <[email protected]>; [email protected]; [email protected]; Cao, Yahui <[email protected]>; Li, Miao <[email protected]>
Subject: [EXT] RE: [RFC 0/4] Support VFIO sparse mmap in PCI busExternal Email
----------------------------------------------------------------------
Gentle ping for some comments..After rethink, personally I may choose option 2 as it will have no driver API change for PMDs and as I look at current code, that's how we do when MSI-X table can't be mmap-ed.
Thanks,
ChenboQuoted 10 lines
-----Original Message-----
From: Chenbo Xia <[email protected]>
Sent: Tuesday, April 18, 2023 1:30 PM
To: [email protected]
Cc: [email protected]
Subject: [RFC 0/4] Support VFIO sparse mmap in PCI busThis series introduces a VFIO standard capability, called sparse mmap to PCI bus. In linux kernel, it's defined as
VFIO_REGION_INFO_CAP_SPARSE_MMAP. Sparse mmap means instead ofmmap
Quoted 37 lines
whole BAR region into DPDK process, only mmap part of the BAR region after getting sparse mmap information from kernel.
For the rest of BAR region that is not mmap-ed, DPDK process can use pread/pwrite system calls to access. Sparse mmap is useful when kernel does not want userspace to mmap whole BAR region, or kernel wants to control over access to specific BAR region. Vendors can choose to enable this feature or not for their devices in their specific kernel modules.In this patchset:
Patch 1-3 is mainly for introducing BAR access APIs so that driver could use them to access specific BAR using pread/pwrite system calls when part of the BAR is not mmap-able.
Patch 4 adds the VFIO sparse mmap support finally. A question is for all sparse mmap regions, should they be mapped to a continuous virtual address region that follows device-specific BAR layout or not. In theory, there could be three options to support this feature.
Option 1: Map sparse mmap regions independently
======================================================
In this approach, we mmap each sparse mmap region one by one and each region could be located anywhere in process address space. But accessing the mmaped BAR will not be as easy as 'bar_base_address+bar_offset', driver needs to check the sparse mmap information to access specific BAR register.Patch 4 in this patchset adopts this option. Driver API change is introduced in
bus_pci_driver.h. Corresponding changes in all drivers are also done and currently I am assuming drivers do not support this feature so they will not check the 'is_sparse' flag but assumes it to be false. Note that it will not break any driver and each vendor can add related logic when they start to support this feature. This is only because I don't want to introduce complexity to drivers that do not want to support this feature.Quoted 2 lines
Option 2: Map sparse mmap regions based on device-specific BAR layout
================================================================
======Quoted 11 lines
In this approach, the sparse mmap regions are mapped to continuous virtual address region that follows device-specific BAR layout. For example, the BAR size is
0x4000and only 0-0x1000 (sparse mmap region #1) and 0x3000-0x4000 (sparse mmap region #2) could be mmaped. Region #1 will be mapped at 'base_addr' and region #2 will be mapped at 'base_addr+0x3000'. The good thing is if we implement like this, driver can still access all BAR registers using 'bar_base_address+bar_offset' way and we don't need to introduce any driver API change. But the address space range 'base_addr+0x1000' to 'base_addr+0x3000' may need to be reserved so it could result in waste of address space or memory (when we useMAP_ANONYMOUSandMAP_PRIVATEflagto
Quoted 112 lines
reserve this range). Meanwhile, driver needs to know which part of BAR is mmaped (this is possible since the range is defined by vendor's specific kernel module).
Option 3: Support both option 1 & 2
===================================
We could define a driver flag to let driver choose which way it perfers since either option has its own Pros & Cons.Please share your comments, Thanks!
Chenbo Xia (4):
bus/pci: introduce an internal representation of PCI device bus/pci: avoid depending on private value in kernel source bus/pci: introduce helper for MMIO read and write bus/pci: add VFIO sparse mmap supportdrivers/baseband/acc/rte_acc100_pmd.c | 6 +- drivers/baseband/acc/rte_vrb_pmd.c | 6 +- .../fpga_5gnr_fec/rte_fpga_5gnr_fec.c | 6 +- drivers/baseband/fpga_lte_fec/fpga_lte_fec.c | 6 +- drivers/bus/pci/bsd/pci.c | 43 +- drivers/bus/pci/bus_pci_driver.h | 24 +- drivers/bus/pci/linux/pci.c | 91 +++- drivers/bus/pci/linux/pci_init.h | 14 +- drivers/bus/pci/linux/pci_uio.c | 34 +- drivers/bus/pci/linux/pci_vfio.c | 445 ++++++++++++++---- drivers/bus/pci/pci_common.c | 57 ++- drivers/bus/pci/pci_common_uio.c | 12 +- drivers/bus/pci/private.h | 25 +- drivers/bus/pci/rte_bus_pci.h | 48 ++ drivers/bus/pci/version.map | 3 + drivers/common/cnxk/roc_dev.c | 4 +- drivers/common/cnxk/roc_dpi.c | 2 +- drivers/common/cnxk/roc_ml.c | 22 +- drivers/common/qat/dev/qat_dev_gen1.c | 2 +- drivers/common/qat/dev/qat_dev_gen4.c | 4 +- drivers/common/sfc_efx/sfc_efx.c | 2 +- drivers/compress/octeontx/otx_zip.c | 4 +- drivers/crypto/ccp/ccp_dev.c | 4 +- drivers/crypto/cnxk/cnxk_cryptodev_ops.c | 2 +- drivers/crypto/nitrox/nitrox_device.c | 4 +- drivers/crypto/octeontx/otx_cryptodev_ops.c | 6 +- drivers/crypto/virtio/virtio_pci.c | 6 +- drivers/dma/cnxk/cnxk_dmadev.c | 2 +- drivers/dma/hisilicon/hisi_dmadev.c | 6 +- drivers/dma/idxd/idxd_pci.c | 4 +- drivers/dma/ioat/ioat_dmadev.c | 2 +- drivers/event/dlb2/pf/dlb2_main.c | 16 +- drivers/event/octeontx/ssovf_probe.c | 38 +- drivers/event/octeontx/timvf_probe.c | 18 +- drivers/event/skeleton/skeleton_eventdev.c | 2 +- drivers/mempool/octeontx/octeontx_fpavf.c | 6 +- drivers/net/ark/ark_ethdev.c | 4 +- drivers/net/atlantic/atl_ethdev.c | 2 +- drivers/net/avp/avp_ethdev.c | 20 +- drivers/net/axgbe/axgbe_ethdev.c | 4 +- drivers/net/bnx2x/bnx2x_ethdev.c | 6 +- drivers/net/bnxt/bnxt_ethdev.c | 8 +- drivers/net/cpfl/cpfl_ethdev.c | 4 +- drivers/net/cxgbe/cxgbe_ethdev.c | 2 +- drivers/net/cxgbe/cxgbe_main.c | 2 +- drivers/net/cxgbe/cxgbevf_ethdev.c | 2 +- drivers/net/cxgbe/cxgbevf_main.c | 2 +- drivers/net/e1000/em_ethdev.c | 4 +- drivers/net/e1000/igb_ethdev.c | 4 +- drivers/net/ena/ena_ethdev.c | 4 +- drivers/net/enetc/enetc_ethdev.c | 2 +- drivers/net/enic/enic_main.c | 4 +- drivers/net/fm10k/fm10k_ethdev.c | 2 +- drivers/net/gve/gve_ethdev.c | 4 +- drivers/net/hinic/base/hinic_pmd_hwif.c | 14 +- drivers/net/hns3/hns3_ethdev.c | 2 +- drivers/net/hns3/hns3_ethdev_vf.c | 2 +- drivers/net/hns3/hns3_rxtx.c | 4 +- drivers/net/i40e/i40e_ethdev.c | 2 +- drivers/net/iavf/iavf_ethdev.c | 2 +- drivers/net/ice/ice_dcf.c | 2 +- drivers/net/ice/ice_ethdev.c | 2 +- drivers/net/idpf/idpf_ethdev.c | 4 +- drivers/net/igc/igc_ethdev.c | 2 +- drivers/net/ionic/ionic_dev_pci.c | 2 +- drivers/net/ixgbe/ixgbe_ethdev.c | 4 +- drivers/net/liquidio/lio_ethdev.c | 4 +- drivers/net/nfp/nfp_ethdev.c | 2 +- drivers/net/nfp/nfp_ethdev_vf.c | 6 +- drivers/net/nfp/nfpcore/nfp_cpp_pcie_ops.c | 4 +- drivers/net/ngbe/ngbe_ethdev.c | 2 +- drivers/net/octeon_ep/otx_ep_ethdev.c | 2 +- drivers/net/octeontx/base/octeontx_pkivf.c | 6 +- drivers/net/octeontx/base/octeontx_pkovf.c | 12 +- drivers/net/qede/qede_main.c | 6 +- drivers/net/sfc/sfc.c | 2 +- drivers/net/thunderx/nicvf_ethdev.c | 2 +- drivers/net/txgbe/txgbe_ethdev.c | 2 +- drivers/net/txgbe/txgbe_ethdev_vf.c | 2 +- drivers/net/virtio/virtio_pci.c | 6 +- drivers/net/vmxnet3/vmxnet3_ethdev.c | 4 +- drivers/raw/cnxk_bphy/cnxk_bphy.c | 10 +- drivers/raw/cnxk_bphy/cnxk_bphy_cgx.c | 6 +- drivers/raw/ifpga/afu_pmd_n3000.c | 4 +- drivers/raw/ifpga/ifpga_rawdev.c | 6 +- drivers/raw/ntb/ntb_hw_intel.c | 8 +- drivers/vdpa/ifc/ifcvf_vdpa.c | 6 +- drivers/vdpa/sfc/sfc_vdpa_hw.c | 2 +- drivers/vdpa/sfc/sfc_vdpa_ops.c | 2 +- lib/eal/include/rte_vfio.h | 1 - 90 files changed, 853 insertions(+), 352 deletions(-)--
2.17.1
cover Support VFIO sparse mmap in PCI bus
In reply to Chenbo XiaThis series introduces a VFIO standard capability, called sparse mmap to PCI bus. In linux kernel, it's defined as VFIO_REGION_INFO_CAP_SPARSE_MMAP. Sparse mmap means instead of mmap whole BAR region into DPDK process, only mmap part of the BAR region after getting sparse mmap information from kernel. For the rest of BAR region that is not mmap-ed, DPDK process can use pread/pwrite system calls to access. Sparse mmap is useful when kernel does not want userspace to mmap whole BAR region, or kernel wants to control over access to specific BAR region. Vendors can choose to enable this feature or not for their devices in their specific kernel modules.
In this patchset:
Patch 1-3 is mainly for introducing BAR access APIs so that driver could use them to access specific BAR using pread/pwrite system calls when part of the BAR is not mmap-able. Patch 4 adds the VFIO sparse mmap support finally.
Chenbo Xia (3):
bus/pci: introduce an internal representation of PCI device bus/pci: avoid depending on private value in kernel source bus/pci: introduce helper for MMIO read and write
Miao Li (1):
bus/pci: add VFIO sparse mmap support
drivers/bus/pci/bsd/pci.c | 35 +++- drivers/bus/pci/linux/pci.c | 78 +++++-- drivers/bus/pci/linux/pci_init.h | 14 +- drivers/bus/pci/linux/pci_uio.c | 22 ++ drivers/bus/pci/linux/pci_vfio.c | 335 +++++++++++++++++++++++++------ drivers/bus/pci/pci_common.c | 12 +- drivers/bus/pci/private.h | 25 ++- drivers/bus/pci/rte_bus_pci.h | 48 +++++ drivers/bus/pci/version.map | 3 + lib/eal/include/rte_vfio.h | 1 - 10 files changed, 482 insertions(+), 91 deletions(-)
Signature · 1 line
2.25.1
PATCH 1/4 bus/pci: introduce an internal representation of PCI device
From: Chenbo Xia <[email protected]>
This patch introduces an internal representation of the PCI device which will be used to store the internal information that don't have to be exposed to drivers, e.g., the VFIO region sizes/offsets.
In this patch, the internal structure is simply a wrapper of the rte_pci_device structure. More fields will be added.
--- drivers/bus/pci/bsd/pci.c | 13 ++++++++----- drivers/bus/pci/linux/pci.c | 28 ++++++++++++++++------------ drivers/bus/pci/pci_common.c | 12 ++++++------ drivers/bus/pci/private.h | 14 +++++++++++++- 4 files changed, 43 insertions(+), 24 deletions(-)
drivers/bus/pci/bsd/pci.c +8 -5
@@ -208,16 +208,19 @@ pci_uio_map_resource_by_index(struct rte_pci_device *dev, int res_idx,static int pci_scan_one (int dev_pci_fd,struct pci_conf *conf) { +struct rte_pci_device_internal *pdev;struct rte_pci_device *dev;struct pci_bar_io bar;unsigned i, max; - dev =malloc (sizeof (*dev)); -if (dev ==NULL ) { + pdev =malloc (sizeof (*pdev)); +if (pdev ==NULL ) { +RTE_LOG (ERR ,EAL ,"Cannot allocate memory for internal pci device\n" );return -1 ; } -memset (dev,0 ,sizeof (*dev)); +memset (pdev,0 ,sizeof (*pdev)); + dev = &pdev->device; dev->device.bus = &rte_pci_bus.bus; dev->addr.domain = conf->pc_sel.pc_domain;
@@ -303,7 +306,7 @@ pci_scan_one(int dev_pci_fd, struct pci_conf *conf)memmove (dev2->mem_resource, dev->mem_resource,sizeof (dev->mem_resource)); -pci_free (dev); +pci_free (pdev); }return 0 ; }
@@ -313,7 +316,7 @@ pci_scan_one(int dev_pci_fd, struct pci_conf *conf)return 0 ; skipdev: -pci_free (dev); +pci_free (pdev);return 0 ; }
drivers/bus/pci/linux/pci.c +16 -12
@@ -211,22 +211,26 @@ pci_scan_one(const char *dirname, const struct rte_pci_addr *addr) {char filename[PATH_MAX ];unsigned long tmp; +struct rte_pci_device_internal *pdev;struct rte_pci_device *dev;char driver[PATH_MAX ];int ret; - dev =malloc (sizeof (*dev)); -if (dev ==NULL ) + pdev =malloc (sizeof (*pdev)); +if (pdev ==NULL ) { +RTE_LOG (ERR ,EAL ,"Cannot allocate memory for internal pci device\n" );return -1 ; + } -memset (dev,0 ,sizeof (*dev)); +memset (pdev,0 ,sizeof (*pdev)); + dev = &pdev->device; dev->device.bus = &rte_pci_bus.bus; dev->addr = *addr;/* get vendor id */ snprintf (filename,sizeof (filename),"%s/vendor" , dirname);if (eal_parse_sysfs_value (filename, &tmp) <0 ) { -pci_free (dev); +pci_free (pdev);return -1 ; } dev->id.vendor_id = (uint16_t )tmp;
@@ -234,7 +238,7 @@ pci_scan_one(const char *dirname, const struct rte_pci_addr *addr)/* get device id */ snprintf (filename,sizeof (filename),"%s/device" , dirname);if (eal_parse_sysfs_value (filename, &tmp) <0 ) { -pci_free (dev); +pci_free (pdev);return -1 ; } dev->id.device_id = (uint16_t )tmp;
@@ -243,7 +247,7 @@ pci_scan_one(const char *dirname, const struct rte_pci_addr *addr)snprintf (filename,sizeof (filename),"%s/subsystem_vendor" , dirname);if (eal_parse_sysfs_value (filename, &tmp) <0 ) { -pci_free (dev); +pci_free (pdev);return -1 ; } dev->id.subsystem_vendor_id = (uint16_t )tmp;
@@ -252,7 +256,7 @@ pci_scan_one(const char *dirname, const struct rte_pci_addr *addr)snprintf (filename,sizeof (filename),"%s/subsystem_device" , dirname);if (eal_parse_sysfs_value (filename, &tmp) <0 ) { -pci_free (dev); +pci_free (pdev);return -1 ; } dev->id.subsystem_device_id = (uint16_t )tmp;
@@ -261,7 +265,7 @@ pci_scan_one(const char *dirname, const struct rte_pci_addr *addr)snprintf (filename,sizeof (filename),"%s/class" , dirname);if (eal_parse_sysfs_value (filename, &tmp) <0 ) { -pci_free (dev); +pci_free (pdev);return -1 ; }/* the least 24 bits are valid: class, subclass, program interface */
@@ -297,7 +301,7 @@ pci_scan_one(const char *dirname, const struct rte_pci_addr *addr)snprintf (filename,sizeof (filename),"%s/resource" , dirname);if (pci_parse_sysfs_resource (filename, dev) <0 ) {RTE_LOG (ERR ,EAL ,"%s(): cannot parse resource\n" , __func__); -pci_free (dev); +pci_free (pdev);return -1 ; }
@@ -306,7 +310,7 @@ pci_scan_one(const char *dirname, const struct rte_pci_addr *addr) ret =pci_get_kernel_driver_by_path (filename, driver,sizeof (driver));if (ret <0 ) {RTE_LOG (ERR ,EAL ,"Fail to get kernel driver\n" ); -pci_free (dev); +pci_free (pdev);return -1 ; }
@@ -320,7 +324,7 @@ pci_scan_one(const char *dirname, const struct rte_pci_addr *addr)else dev->kdrv =RTE_PCI_KDRV_UNKNOWN ; }else { -pci_free (dev); +pci_free (pdev);return 0 ; }/* device is valid, add in list (sorted) */
@@ -375,7 +379,7 @@ pci_scan_one(const char *dirname, const struct rte_pci_addr *addr)pci_common_set (dev2); } } -pci_free (dev); +pci_free (pdev); }return 0 ; }
drivers/bus/pci/pci_common.c +6 -6
@@ -121,12 +121,12 @@ pci_common_set(struct rte_pci_device *dev) }void -pci_free (struct rte_pci_device *dev) +pci_free (struct rte_pci_device_internal *pdev) { -if (dev ==NULL ) +if (pdev ==NULL )return ; -free (dev->bus_info); -free (dev); +free (pdev->device.bus_info); +free (pdev); }/* map a particular resource from a file */
@@ -465,7 +465,7 @@ pci_cleanup(void)rte_intr_instance_free (dev->vfio_req_intr_handle); dev->vfio_req_intr_handle =NULL ; -pci_free (dev); +pci_free (RTE_PCI_DEVICE_INTERNAL (dev)); }return error;
@@ -681,7 +681,7 @@ pci_unplug(struct rte_device *dev)if (ret ==0 ) {rte_pci_remove_device (pdev);rte_devargs_remove (dev->devargs); -pci_free (pdev); +pci_free (RTE_PCI_DEVICE_INTERNAL (pdev)); }return ret; }
drivers/bus/pci/private.h +13 -1
@@ -13,6 +13,14 @@#include <rte_os_shim.h> #include <rte_pci.h> +/* +* Convert struct rte_pci_device to struct rte_pci_device_internal +*/ +#define RTE_PCI_DEVICE_INTERNAL (ptr) \ + container_of(ptr, struct rte_pci_device_internal, device) +#define RTE_PCI_DEVICE_INTERNAL_CONST (ptr) \ + container_of(ptr, const struct rte_pci_device_internal, device) +/** * Structure describing the PCI bus */
@@ -34,6 +42,10 @@ extern struct rte_pci_bus rte_pci_bus;struct rte_pci_driver ;struct rte_pci_device ; +struct rte_pci_device_internal { +struct rte_pci_device device; +}; +/** * Scan the content of the PCI bus, and the devices in the devices * list
@@ -53,7 +65,7 @@ pci_common_set(struct rte_pci_device *dev);* Free a PCI device. */ void -pci_free (struct rte_pci_device *dev); +pci_free (struct rte_pci_device_internal *pdev);/** * Validate whether a device with given PCI address should be ignored or not.
Signature · 1 line
2.25.1
PATCH 2/4 bus/pci: avoid depending on private value in kernel source
In reply to Miao LiFrom: Chenbo Xia <[email protected]>
The value 40 used in VFIO_GET_REGION_ADDR() is a private value (VFIO_PCI_OFFSET_SHIFT) defined in Linux kernel source [1]. It is not part of VFIO API, and we should not depend on it.
[1] https://github.com/torvalds/linux/blob/v6.2/include/linux/vfio_pci_core.h
--- drivers/bus/pci/linux/pci.c | 4 +- drivers/bus/pci/linux/pci_init.h | 4 +- drivers/bus/pci/linux/pci_vfio.c | 195 +++++++++++++++++++++++-------- drivers/bus/pci/private.h | 9 ++ lib/eal/include/rte_vfio.h | 1 - 5 files changed, 158 insertions(+), 55 deletions(-)
drivers/bus/pci/linux/pci.c +2 -2
@@ -645,7 +645,7 @@ int rte_pci_read_config(const struct rte_pci_device *device,return pci_uio_read_config (intr_handle, buf, len, offset);#ifdef VFIO_PRESENT case RTE_PCI_KDRV_VFIO : -return pci_vfio_read_config (intr_handle, buf, len, offset); +return pci_vfio_read_config (device, buf, len, offset);#endif default : rte_pci_device_name(&device->addr, devname,
@@ -669,7 +669,7 @@ int rte_pci_write_config(const struct rte_pci_device *device,return pci_uio_write_config (intr_handle, buf, len, offset);#ifdef VFIO_PRESENT case RTE_PCI_KDRV_VFIO : -return pci_vfio_write_config (intr_handle, buf, len, offset); +return pci_vfio_write_config (device, buf, len, offset);#endif default : rte_pci_device_name(&device->addr, devname,
drivers/bus/pci/linux/pci_init.h +2 -2
@@ -66,9 +66,9 @@ int pci_uio_ioport_unmap(struct rte_pci_ioport *p);#endif /* access config space */ -int pci_vfio_read_config (const struct rte_intr_handle *intr_handle, +int pci_vfio_read_config (const struct rte_pci_device *dev,void *buf,size_t len,off_t offs); -int pci_vfio_write_config (const struct rte_intr_handle *intr_handle, +int pci_vfio_write_config (const struct rte_pci_device *dev,const void *buf,size_t len,off_t offs); int pci_vfio_ioport_map(struct rte_pci_device *dev, int bar,
drivers/bus/pci/linux/pci_vfio.c +145 -50
363 lines not on this page · raw message
drivers/bus/pci/private.h +9 -0
@@ -13,6 +13,8 @@#include <rte_os_shim.h> #include <rte_pci.h> +#define RTE_MAX_PCI_REGIONS 9 +/* * Convert struct rte_pci_device to struct rte_pci_device_internal */
@@ -42,8 +44,15 @@ extern struct rte_pci_bus rte_pci_bus;struct rte_pci_driver ;struct rte_pci_device ; +struct rte_pci_region { +uint64_t size; +uint64_t offset; +}; +struct rte_pci_device_internal {struct rte_pci_device device; +/* PCI regions provided by e.g. VFIO. */ +struct rte_pci_region region[RTE_MAX_PCI_REGIONS ]; };/**
lib/eal/include/rte_vfio.h +0 -1
@@ -38,7 +38,6 @@ extern "C" {#define VFIO_CONTAINER_PATH "/dev/vfio/vfio"#define VFIO_GROUP_FMT "/dev/vfio/%u"#define VFIO_NOIOMMU_GROUP_FMT "/dev/vfio/noiommu-%u" -#define VFIO_GET_REGION_ADDR (x) ((uint64_t) x << 40ULL)#define VFIO_GET_REGION_IDX (x) (x >> 40)#define VFIO_NOIOMMU_MODE \ "/sys/module/vfio/parameters/enable_unsafe_noiommu_mode"
Signature · 1 line
2.25.1
PATCH 3/4 bus/pci: introduce helper for MMIO read and write
In reply to Miao LiFrom: Chenbo Xia <[email protected]>
The MMIO regions may not be mmap-able for VFIO-PCI devices. In this case, the driver should explicitly do read and write to access these regions.
--- drivers/bus/pci/bsd/pci.c | 22 +++++++++++++++ drivers/bus/pci/linux/pci.c | 46 ++++++++++++++++++++++++++++++ drivers/bus/pci/linux/pci_init.h | 10 +++++++ drivers/bus/pci/linux/pci_uio.c | 22 +++++++++++++++ drivers/bus/pci/linux/pci_vfio.c | 36 ++++++++++++++++++++++++ drivers/bus/pci/rte_bus_pci.h | 48 ++++++++++++++++++++++++++++++++ drivers/bus/pci/version.map | 3 ++ 7 files changed, 187 insertions(+)
drivers/bus/pci/bsd/pci.c +22 -0
@@ -489,6 +489,28 @@ int rte_pci_write_config(const struct rte_pci_device *dev,return -1 ; } +/* Read PCI MMIO space. */ +int rte_pci_mmio_read (const struct rte_pci_device *dev,int bar, +void *buf,size_t len,off_t offset) +{ +if (bar >=PCI_MAX_RESOURCE || dev->mem_resource[bar].addr ==NULL || + (uint64_t )offset + len > dev->mem_resource[bar].len) +return -1 ; +memcpy (buf, (uint8_t *)dev->mem_resource[bar].addr + offset, len); +return len; +} + +/* Write PCI MMIO space. */ +int rte_pci_mmio_write (const struct rte_pci_device *dev,int bar, +const void *buf,size_t len,off_t offset) +{ +if (bar >=PCI_MAX_RESOURCE || dev->mem_resource[bar].addr ==NULL || + (uint64_t )offset + len > dev->mem_resource[bar].len) +return -1 ; +memcpy ((uint8_t *)dev->mem_resource[bar].addr + offset, buf, len); +return len; +} +int rte_pci_ioport_map (struct rte_pci_device *dev,int bar,struct rte_pci_ioport *p)
drivers/bus/pci/linux/pci.c +46 -0
52 lines not on this page · raw message
drivers/bus/pci/linux/pci_init.h +10 -0
@@ -37,6 +37,11 @@ int pci_uio_read_config(const struct rte_intr_handle *intr_handle,int pci_uio_write_config (const struct rte_intr_handle *intr_handle,const void *buf,size_t len,off_t offs); +int pci_uio_mmio_read (const struct rte_pci_device *dev,int bar, +void *buf,size_t len,off_t offset); +int pci_uio_mmio_write (const struct rte_pci_device *dev,int bar, +const void *buf,size_t len,off_t offset); +int pci_uio_ioport_map (struct rte_pci_device *dev,int bar,struct rte_pci_ioport *p); void pci_uio_ioport_read(struct rte_pci_ioport *p,
@@ -71,6 +76,11 @@ int pci_vfio_read_config(const struct rte_pci_device *dev,int pci_vfio_write_config (const struct rte_pci_device *dev,const void *buf,size_t len,off_t offs); +int pci_vfio_mmio_read (const struct rte_pci_device *dev,int bar, +void *buf,size_t len,off_t offset); +int pci_vfio_mmio_write (const struct rte_pci_device *dev,int bar, +const void *buf,size_t len,off_t offset); +int pci_vfio_ioport_map (struct rte_pci_device *dev,int bar,struct rte_pci_ioport *p); void pci_vfio_ioport_read(struct rte_pci_ioport *p,
drivers/bus/pci/linux/pci_uio.c +22 -0
@@ -55,6 +55,28 @@ pci_uio_write_config(const struct rte_intr_handle *intr_handle,return pwrite (uio_cfg_fd, buf, len, offset); } +int +pci_uio_mmio_read (const struct rte_pci_device *dev,int bar, +void *buf,size_t len,off_t offset) +{ +if (bar >=PCI_MAX_RESOURCE || dev->mem_resource[bar].addr ==NULL || + (uint64_t )offset + len > dev->mem_resource[bar].len) +return -1 ; +memcpy (buf, (uint8_t *)dev->mem_resource[bar].addr + offset, len); +return len; +} + +int +pci_uio_mmio_write (const struct rte_pci_device *dev,int bar, +const void *buf,size_t len,off_t offset) +{ +if (bar >=PCI_MAX_RESOURCE || dev->mem_resource[bar].addr ==NULL || + (uint64_t )offset + len > dev->mem_resource[bar].len) +return -1 ; +memcpy ((uint8_t *)dev->mem_resource[bar].addr + offset, buf, len); +return len; +} +static int pci_uio_set_bus_master (int dev_fd) {
drivers/bus/pci/linux/pci_vfio.c +36 -0
@@ -1258,6 +1258,42 @@ pci_vfio_ioport_unmap(struct rte_pci_ioport *p)return -1 ; } +int +pci_vfio_mmio_read (const struct rte_pci_device *dev,int bar, +void *buf,size_t len,off_t offs) +{ +uint64_t size, offset; +int fd; + + fd =rte_intr_dev_fd_get (dev->intr_handle); + +if (pci_vfio_get_region (dev, bar, &size, &offset) !=0 ) +return -1 ; + +if ((uint64_t )len + offs > size) +return -1 ; + +return pread64 (fd, buf, len, offset + offs); +} + +int +pci_vfio_mmio_write (const struct rte_pci_device *dev,int bar, +const void *buf,size_t len,off_t offs) +{ +uint64_t size, offset; +int fd; + + fd =rte_intr_dev_fd_get (dev->intr_handle); + +if (pci_vfio_get_region (dev, bar, &size, &offset) !=0 ) +return -1 ; + +if ((uint64_t )len + offs > size) +return -1 ; + +return pwrite64 (fd, buf, len, offset + offs); +} +int pci_vfio_is_enabled (void ) {
drivers/bus/pci/rte_bus_pci.h +48 -0
54 lines not on this page · raw message
drivers/bus/pci/version.map +3 -0
@@ -21,6 +21,9 @@ EXPERIMENTAL { # added in 21.08 rte_pci_set_bus_master; + # added in 23.07 + rte_pci_mmio_read; + rte_pci_mmio_write; }; INTERNAL {
Signature · 1 line
2.25.1
PATCH 4/4 bus/pci: add VFIO sparse mmap support
In reply to Miao LiThis patch adds sparse mmap support in PCI bus. Sparse mmap is a capability defined in VFIO which allows multiple mmap areas in one VFIO region.
In this patch, the sparse mmap regions are mapped to one continuous virtual address region that follows device-specific BAR layout. So, driver can still access all mapped sparse mmap regions by using 'bar_base_address + bar_offset'.
--- drivers/bus/pci/linux/pci_vfio.c | 104 +++++++++++++++++++++++++++---- drivers/bus/pci/private.h | 2 + 2 files changed, 94 insertions(+), 12 deletions(-)
drivers/bus/pci/linux/pci_vfio.c +92 -12
@@ -673,6 +673,54 @@ pci_vfio_mmap_bar(int vfio_dev_fd, struct mapped_pci_resource *vfio_res,return 0 ; } +static int +pci_vfio_sparse_mmap_bar (int vfio_dev_fd,struct mapped_pci_resource *vfio_res, +int bar_index,int additional_flags) +{ +struct pci_map *bar = &vfio_res->maps[bar_index]; +struct vfio_region_sparse_mmap_area *sparse; +void *bar_addr; +uint32_t i; + +if (bar->size ==0 ) { +RTE_LOG (DEBUG ,EAL ,"Bar size is 0, skip BAR%d\n" , bar_index); +return 0 ; + } + +/* reserve the address using an inaccessible mapping */ + bar_addr =mmap (bar->addr, bar->size,0 ,MAP_PRIVATE | +MAP_ANONYMOUS | additional_flags,-1 ,0 ); +if (bar_addr !=MAP_FAILED ) { +void *map_addr =NULL ; +for (i =0 ; i < bar->nr_areas; i++) { + sparse = &bar->areas[i]; +if (sparse->size) { +void *addr =RTE_PTR_ADD (bar_addr, sparse->offset); + map_addr =pci_map_resource (addr, vfio_dev_fd, + bar->offset + sparse->offset, sparse->size, +RTE_MAP_FORCE_ADDRESS ); +if (map_addr ==NULL ) { +munmap (bar_addr, bar->size); +RTE_LOG (ERR ,EAL ,"Failed to map pci BAR%d\n" , + bar_index); + goto err_map; + } + } + } + }else { +RTE_LOG (ERR ,EAL ,"Failed to create inaccessible mapping for BAR%d\n" , + bar_index); + goto err_map; + } + + bar->addr = bar_addr; +return 0 ; + +err_map: + bar->nr_areas =0 ; +return -1 ; +} +/* * region info may contain capability headers, so we need to keep reallocating * the memory until we match allocated memory size with argsz.
@@ -875,6 +923,8 @@ pci_vfio_map_resource_primary(struct rte_pci_device *dev)for (i =0 ; i < vfio_res->nb_maps; i++) {void *bar_addr; +struct vfio_info_cap_header *hdr; +struct vfio_region_info_cap_sparse_mmap *sparse; ret =pci_vfio_get_region_info (vfio_dev_fd, ®, i);if (ret <0 ) {
@@ -920,12 +970,33 @@ pci_vfio_map_resource_primary(struct rte_pci_device *dev) maps[i].size = reg->size; maps[i].path =NULL ;/* vfio doesn't have per-resource paths */ - ret =pci_vfio_mmap_bar (vfio_dev_fd, vfio_res, i,0 ); -if (ret <0 ) { -RTE_LOG (ERR ,EAL ,"%s mapping BAR%i failed: %s\n" , - pci_addr, i,strerror (errno)); -free (reg); - goto err_vfio_res; + hdr =pci_vfio_info_cap (reg,VFIO_REGION_INFO_CAP_SPARSE_MMAP ); + +if (hdr !=NULL ) { + sparse =container_of (hdr, + struct vfio_region_info_cap_sparse_mmap, header); +if (sparse->nr_areas >0 ) { + maps[i].nr_areas = sparse->nr_areas; + maps[i].areas = sparse->areas; + } + } + +if (maps[i].nr_areas >0 ) { + ret =pci_vfio_sparse_mmap_bar (vfio_dev_fd, vfio_res, i,0 ); +if (ret <0 ) { +RTE_LOG (ERR ,EAL ,"%s sparse mapping BAR%i failed: %s\n" , + pci_addr, i,strerror (errno)); +free (reg); + goto err_vfio_res; + } + }else { + ret =pci_vfio_mmap_bar (vfio_dev_fd, vfio_res, i,0 ); +if (ret <0 ) { +RTE_LOG (ERR ,EAL ,"%s mapping BAR%i failed: %s\n" , + pci_addr, i,strerror (errno)); +free (reg); + goto err_vfio_res; + } } dev->mem_resource[i].addr = maps[i].addr;
@@ -1008,11 +1079,20 @@ pci_vfio_map_resource_secondary(struct rte_pci_device *dev) maps = vfio_res->maps;for (i =0 ; i < vfio_res->nb_maps; i++) { - ret =pci_vfio_mmap_bar (vfio_dev_fd, vfio_res, i,MAP_FIXED ); -if (ret <0 ) { -RTE_LOG (ERR ,EAL ,"%s mapping BAR%i failed: %s\n" , - pci_addr, i,strerror (errno)); - goto err_vfio_dev_fd; +if (maps[i].nr_areas >0 ) { + ret =pci_vfio_sparse_mmap_bar (vfio_dev_fd, vfio_res, i,0 ); +if (ret <0 ) { +RTE_LOG (ERR ,EAL ,"%s sparse mapping BAR%i failed: %s\n" , + pci_addr, i,strerror (errno)); + goto err_vfio_dev_fd; + } + }else { + ret =pci_vfio_mmap_bar (vfio_dev_fd, vfio_res, i,0 ); +if (ret <0 ) { +RTE_LOG (ERR ,EAL ,"%s mapping BAR%i failed: %s\n" , + pci_addr, i,strerror (errno)); + goto err_vfio_dev_fd; + } } dev->mem_resource[i].addr = maps[i].addr;
@@ -1062,7 +1142,7 @@ find_and_unmap_vfio_resource(struct mapped_pci_res_list *vfio_res_list,break ; } -if (vfio_res ==NULL ) +if (vfio_res ==NULL )return vfio_res; RTE_LOG(INFO, EAL, "Releasing PCI mapped resource for %s\n",
drivers/bus/pci/private.h +2 -0
@@ -121,6 +121,8 @@ struct pci_map {uint64_t offset;uint64_t size;uint64_t phaddr; +uint32_t nr_areas; +struct vfio_region_sparse_mmap_area *areas; };struct pci_msix_table {
Signature · 1 line
2.25.1
v2 cover Support VFIO sparse mmap in PCI bus
This series introduces a VFIO standard capability, called sparse mmap to PCI bus. In linux kernel, it's defined as VFIO_REGION_INFO_CAP_SPARSE_MMAP. Sparse mmap means instead of mmap whole BAR region into DPDK process, only mmap part of the BAR region after getting sparse mmap information from kernel. For the rest of BAR region that is not mmap-ed, DPDK process can use pread/pwrite system calls to access. Sparse mmap is useful when kernel does not want userspace to mmap whole BAR region, or kernel wants to control over access to specific BAR region. Vendors can choose to enable this feature or not for their devices in their specific kernel modules.
In this patchset:
Patch 1-3 is mainly for introducing BAR access APIs so that driver could use them to access specific BAR using pread/pwrite system calls when part of the BAR is not mmap-able. Patch 4 adds the VFIO sparse mmap support finally.
v2:
1. add PCI device internal structure in bus/pci/windows/pci.c
2. fix parameter type error
Chenbo Xia (3):
bus/pci: introduce an internal representation of PCI device bus/pci: avoid depending on private value in kernel source bus/pci: introduce helper for MMIO read and write
Miao Li (1):
bus/pci: add VFIO sparse mmap support
drivers/bus/pci/bsd/pci.c | 35 +++- drivers/bus/pci/linux/pci.c | 78 +++++-- drivers/bus/pci/linux/pci_init.h | 14 +- drivers/bus/pci/linux/pci_uio.c | 22 ++ drivers/bus/pci/linux/pci_vfio.c | 335 +++++++++++++++++++++++++------ drivers/bus/pci/pci_common.c | 12 +- drivers/bus/pci/private.h | 25 ++- drivers/bus/pci/rte_bus_pci.h | 48 +++++ drivers/bus/pci/version.map | 3 + drivers/bus/pci/windows/pci.c | 14 +- lib/eal/include/rte_vfio.h | 1 - 11 files changed, 491 insertions(+), 96 deletions(-)
Signature · 1 line
2.25.1
PATCH v2 1/4 bus/pci: introduce an internal representation of PCI device
From: Chenbo Xia <[email protected]>
This patch introduces an internal representation of the PCI device which will be used to store the internal information that don't have to be exposed to drivers, e.g., the VFIO region sizes/offsets.
In this patch, the internal structure is simply a wrapper of the rte_pci_device structure. More fields will be added.
--- drivers/bus/pci/bsd/pci.c | 13 ++++++++----- drivers/bus/pci/linux/pci.c | 28 ++++++++++++++++------------ drivers/bus/pci/pci_common.c | 12 ++++++------ drivers/bus/pci/private.h | 14 +++++++++++++- drivers/bus/pci/windows/pci.c | 14 +++++++++----- 5 files changed, 52 insertions(+), 29 deletions(-)
drivers/bus/pci/bsd/pci.c +8 -5
@@ -208,16 +208,19 @@ pci_uio_map_resource_by_index(struct rte_pci_device *dev, int res_idx,static int pci_scan_one (int dev_pci_fd,struct pci_conf *conf) { +struct rte_pci_device_internal *pdev;struct rte_pci_device *dev;struct pci_bar_io bar;unsigned i, max; - dev =malloc (sizeof (*dev)); -if (dev ==NULL ) { + pdev =malloc (sizeof (*pdev)); +if (pdev ==NULL ) { +RTE_LOG (ERR ,EAL ,"Cannot allocate memory for internal pci device\n" );return -1 ; } -memset (dev,0 ,sizeof (*dev)); +memset (pdev,0 ,sizeof (*pdev)); + dev = &pdev->device; dev->device.bus = &rte_pci_bus.bus; dev->addr.domain = conf->pc_sel.pc_domain;
@@ -303,7 +306,7 @@ pci_scan_one(int dev_pci_fd, struct pci_conf *conf)memmove (dev2->mem_resource, dev->mem_resource,sizeof (dev->mem_resource)); -pci_free (dev); +pci_free (pdev); }return 0 ; }
@@ -313,7 +316,7 @@ pci_scan_one(int dev_pci_fd, struct pci_conf *conf)return 0 ; skipdev: -pci_free (dev); +pci_free (pdev);return 0 ; }
drivers/bus/pci/linux/pci.c +16 -12
@@ -211,22 +211,26 @@ pci_scan_one(const char *dirname, const struct rte_pci_addr *addr) {char filename[PATH_MAX ];unsigned long tmp; +struct rte_pci_device_internal *pdev;struct rte_pci_device *dev;char driver[PATH_MAX ];int ret; - dev =malloc (sizeof (*dev)); -if (dev ==NULL ) + pdev =malloc (sizeof (*pdev)); +if (pdev ==NULL ) { +RTE_LOG (ERR ,EAL ,"Cannot allocate memory for internal pci device\n" );return -1 ; + } -memset (dev,0 ,sizeof (*dev)); +memset (pdev,0 ,sizeof (*pdev)); + dev = &pdev->device; dev->device.bus = &rte_pci_bus.bus; dev->addr = *addr;/* get vendor id */ snprintf (filename,sizeof (filename),"%s/vendor" , dirname);if (eal_parse_sysfs_value (filename, &tmp) <0 ) { -pci_free (dev); +pci_free (pdev);return -1 ; } dev->id.vendor_id = (uint16_t )tmp;
@@ -234,7 +238,7 @@ pci_scan_one(const char *dirname, const struct rte_pci_addr *addr)/* get device id */ snprintf (filename,sizeof (filename),"%s/device" , dirname);if (eal_parse_sysfs_value (filename, &tmp) <0 ) { -pci_free (dev); +pci_free (pdev);return -1 ; } dev->id.device_id = (uint16_t )tmp;
@@ -243,7 +247,7 @@ pci_scan_one(const char *dirname, const struct rte_pci_addr *addr)snprintf (filename,sizeof (filename),"%s/subsystem_vendor" , dirname);if (eal_parse_sysfs_value (filename, &tmp) <0 ) { -pci_free (dev); +pci_free (pdev);return -1 ; } dev->id.subsystem_vendor_id = (uint16_t )tmp;
@@ -252,7 +256,7 @@ pci_scan_one(const char *dirname, const struct rte_pci_addr *addr)snprintf (filename,sizeof (filename),"%s/subsystem_device" , dirname);if (eal_parse_sysfs_value (filename, &tmp) <0 ) { -pci_free (dev); +pci_free (pdev);return -1 ; } dev->id.subsystem_device_id = (uint16_t )tmp;
@@ -261,7 +265,7 @@ pci_scan_one(const char *dirname, const struct rte_pci_addr *addr)snprintf (filename,sizeof (filename),"%s/class" , dirname);if (eal_parse_sysfs_value (filename, &tmp) <0 ) { -pci_free (dev); +pci_free (pdev);return -1 ; }/* the least 24 bits are valid: class, subclass, program interface */
@@ -297,7 +301,7 @@ pci_scan_one(const char *dirname, const struct rte_pci_addr *addr)snprintf (filename,sizeof (filename),"%s/resource" , dirname);if (pci_parse_sysfs_resource (filename, dev) <0 ) {RTE_LOG (ERR ,EAL ,"%s(): cannot parse resource\n" , __func__); -pci_free (dev); +pci_free (pdev);return -1 ; }
@@ -306,7 +310,7 @@ pci_scan_one(const char *dirname, const struct rte_pci_addr *addr) ret =pci_get_kernel_driver_by_path (filename, driver,sizeof (driver));if (ret <0 ) {RTE_LOG (ERR ,EAL ,"Fail to get kernel driver\n" ); -pci_free (dev); +pci_free (pdev);return -1 ; }
@@ -320,7 +324,7 @@ pci_scan_one(const char *dirname, const struct rte_pci_addr *addr)else dev->kdrv =RTE_PCI_KDRV_UNKNOWN ; }else { -pci_free (dev); +pci_free (pdev);return 0 ; }/* device is valid, add in list (sorted) */
@@ -375,7 +379,7 @@ pci_scan_one(const char *dirname, const struct rte_pci_addr *addr)pci_common_set (dev2); } } -pci_free (dev); +pci_free (pdev); }return 0 ; }
drivers/bus/pci/pci_common.c +6 -6
@@ -121,12 +121,12 @@ pci_common_set(struct rte_pci_device *dev) }void -pci_free (struct rte_pci_device *dev) +pci_free (struct rte_pci_device_internal *pdev) { -if (dev ==NULL ) +if (pdev ==NULL )return ; -free (dev->bus_info); -free (dev); +free (pdev->device.bus_info); +free (pdev); }/* map a particular resource from a file */
@@ -465,7 +465,7 @@ pci_cleanup(void)rte_intr_instance_free (dev->vfio_req_intr_handle); dev->vfio_req_intr_handle =NULL ; -pci_free (dev); +pci_free (RTE_PCI_DEVICE_INTERNAL (dev)); }return error;
@@ -681,7 +681,7 @@ pci_unplug(struct rte_device *dev)if (ret ==0 ) {rte_pci_remove_device (pdev);rte_devargs_remove (dev->devargs); -pci_free (pdev); +pci_free (RTE_PCI_DEVICE_INTERNAL (pdev)); }return ret; }
drivers/bus/pci/private.h +13 -1
@@ -13,6 +13,14 @@#include <rte_os_shim.h> #include <rte_pci.h> +/* +* Convert struct rte_pci_device to struct rte_pci_device_internal +*/ +#define RTE_PCI_DEVICE_INTERNAL (ptr) \ + container_of(ptr, struct rte_pci_device_internal, device) +#define RTE_PCI_DEVICE_INTERNAL_CONST (ptr) \ + container_of(ptr, const struct rte_pci_device_internal, device) +/** * Structure describing the PCI bus */
@@ -34,6 +42,10 @@ extern struct rte_pci_bus rte_pci_bus;struct rte_pci_driver ;struct rte_pci_device ; +struct rte_pci_device_internal { +struct rte_pci_device device; +}; +/** * Scan the content of the PCI bus, and the devices in the devices * list
@@ -53,7 +65,7 @@ pci_common_set(struct rte_pci_device *dev);* Free a PCI device. */ void -pci_free (struct rte_pci_device *dev); +pci_free (struct rte_pci_device_internal *pdev);/** * Validate whether a device with given PCI address should be ignored or not.
drivers/bus/pci/windows/pci.c +9 -5
@@ -336,6 +336,7 @@ set_kernel_driver_type(PSP_DEVINFO_DATA device_info_data,static int pci_scan_one (HDEVINFO dev_info,PSP_DEVINFO_DATA device_info_data) { +struct rte_pci_device_internal *pdev;struct rte_pci_device *dev =NULL ;int ret =-1 ;char pci_device_info[REGSTR_VAL_MAX_HCID_LEN ];
@@ -370,11 +371,14 @@ pci_scan_one(HDEVINFO dev_info, PSP_DEVINFO_DATA device_info_data) goto end; } - dev =malloc (sizeof (*dev)); -if (dev ==NULL ) + pdev =malloc (sizeof (*pdev)); +if (pdev ==NULL ) { +RTE_LOG (ERR ,EAL ,"Cannot allocate memory for internal pci device\n" ); goto end; + } -memset (dev,0 ,sizeof (*dev)); +memset (pdev,0 ,sizeof (*pdev)); + dev = &pdev->device; dev->device.bus = &rte_pci_bus.bus; dev->addr = addr;
@@ -409,7 +413,7 @@ pci_scan_one(HDEVINFO dev_info, PSP_DEVINFO_DATA device_info_data) dev2->max_vfs = dev->max_vfs;memmove (dev2->mem_resource, dev->mem_resource,sizeof (dev->mem_resource)); -pci_free (dev); +pci_free (pdev); }return 0 ; }
@@ -418,7 +422,7 @@ pci_scan_one(HDEVINFO dev_info, PSP_DEVINFO_DATA device_info_data)return 0 ; end: -pci_free (dev); +pci_free (pdev);return ret; }
Signature · 1 line
2.25.1
PATCH v2 2/4 bus/pci: avoid depending on private value in kernel source
In reply to Miao LiFrom: Chenbo Xia <[email protected]>
The value 40 used in VFIO_GET_REGION_ADDR() is a private value (VFIO_PCI_OFFSET_SHIFT) defined in Linux kernel source [1]. It is not part of VFIO API, and we should not depend on it.
[1] https://github.com/torvalds/linux/blob/v6.2/include/linux/vfio_pci_core.h
--- drivers/bus/pci/linux/pci.c | 4 +- drivers/bus/pci/linux/pci_init.h | 4 +- drivers/bus/pci/linux/pci_vfio.c | 195 +++++++++++++++++++++++-------- drivers/bus/pci/private.h | 9 ++ lib/eal/include/rte_vfio.h | 1 - 5 files changed, 158 insertions(+), 55 deletions(-)
drivers/bus/pci/linux/pci.c +2 -2
@@ -645,7 +645,7 @@ int rte_pci_read_config(const struct rte_pci_device *device,return pci_uio_read_config (intr_handle, buf, len, offset);#ifdef VFIO_PRESENT case RTE_PCI_KDRV_VFIO : -return pci_vfio_read_config (intr_handle, buf, len, offset); +return pci_vfio_read_config (device, buf, len, offset);#endif default : rte_pci_device_name(&device->addr, devname,
@@ -669,7 +669,7 @@ int rte_pci_write_config(const struct rte_pci_device *device,return pci_uio_write_config (intr_handle, buf, len, offset);#ifdef VFIO_PRESENT case RTE_PCI_KDRV_VFIO : -return pci_vfio_write_config (intr_handle, buf, len, offset); +return pci_vfio_write_config (device, buf, len, offset);#endif default : rte_pci_device_name(&device->addr, devname,
drivers/bus/pci/linux/pci_init.h +2 -2
@@ -66,9 +66,9 @@ int pci_uio_ioport_unmap(struct rte_pci_ioport *p);#endif /* access config space */ -int pci_vfio_read_config (const struct rte_intr_handle *intr_handle, +int pci_vfio_read_config (const struct rte_pci_device *dev,void *buf,size_t len,off_t offs); -int pci_vfio_write_config (const struct rte_intr_handle *intr_handle, +int pci_vfio_write_config (const struct rte_pci_device *dev,const void *buf,size_t len,off_t offs); int pci_vfio_ioport_map(struct rte_pci_device *dev, int bar,
drivers/bus/pci/linux/pci_vfio.c +145 -50
363 lines not on this page · raw message
drivers/bus/pci/private.h +9 -0
@@ -13,6 +13,8 @@#include <rte_os_shim.h> #include <rte_pci.h> +#define RTE_MAX_PCI_REGIONS 9 +/* * Convert struct rte_pci_device to struct rte_pci_device_internal */
@@ -42,8 +44,15 @@ extern struct rte_pci_bus rte_pci_bus;struct rte_pci_driver ;struct rte_pci_device ; +struct rte_pci_region { +uint64_t size; +uint64_t offset; +}; +struct rte_pci_device_internal {struct rte_pci_device device; +/* PCI regions provided by e.g. VFIO. */ +struct rte_pci_region region[RTE_MAX_PCI_REGIONS ]; };/**
lib/eal/include/rte_vfio.h +0 -1
@@ -38,7 +38,6 @@ extern "C" {#define VFIO_CONTAINER_PATH "/dev/vfio/vfio"#define VFIO_GROUP_FMT "/dev/vfio/%u"#define VFIO_NOIOMMU_GROUP_FMT "/dev/vfio/noiommu-%u" -#define VFIO_GET_REGION_ADDR (x) ((uint64_t) x << 40ULL)#define VFIO_GET_REGION_IDX (x) (x >> 40)#define VFIO_NOIOMMU_MODE \ "/sys/module/vfio/parameters/enable_unsafe_noiommu_mode"
Signature · 1 line
2.25.1
PATCH v2 3/4 bus/pci: introduce helper for MMIO read and write
In reply to Miao LiFrom: Chenbo Xia <[email protected]>
The MMIO regions may not be mmap-able for VFIO-PCI devices. In this case, the driver should explicitly do read and write to access these regions.
--- drivers/bus/pci/bsd/pci.c | 22 +++++++++++++++ drivers/bus/pci/linux/pci.c | 46 ++++++++++++++++++++++++++++++ drivers/bus/pci/linux/pci_init.h | 10 +++++++ drivers/bus/pci/linux/pci_uio.c | 22 +++++++++++++++ drivers/bus/pci/linux/pci_vfio.c | 36 ++++++++++++++++++++++++ drivers/bus/pci/rte_bus_pci.h | 48 ++++++++++++++++++++++++++++++++ drivers/bus/pci/version.map | 3 ++ 7 files changed, 187 insertions(+)
drivers/bus/pci/bsd/pci.c +22 -0
@@ -489,6 +489,28 @@ int rte_pci_write_config(const struct rte_pci_device *dev,return -1 ; } +/* Read PCI MMIO space. */ +int rte_pci_mmio_read (const struct rte_pci_device *dev,int bar, +void *buf,size_t len,off_t offset) +{ +if (bar >=PCI_MAX_RESOURCE || dev->mem_resource[bar].addr ==NULL || + (uint64_t )offset + len > dev->mem_resource[bar].len) +return -1 ; +memcpy (buf, (uint8_t *)dev->mem_resource[bar].addr + offset, len); +return len; +} + +/* Write PCI MMIO space. */ +int rte_pci_mmio_write (const struct rte_pci_device *dev,int bar, +const void *buf,size_t len,off_t offset) +{ +if (bar >=PCI_MAX_RESOURCE || dev->mem_resource[bar].addr ==NULL || + (uint64_t )offset + len > dev->mem_resource[bar].len) +return -1 ; +memcpy ((uint8_t *)dev->mem_resource[bar].addr + offset, buf, len); +return len; +} +int rte_pci_ioport_map (struct rte_pci_device *dev,int bar,struct rte_pci_ioport *p)
drivers/bus/pci/linux/pci.c +46 -0
52 lines not on this page · raw message
drivers/bus/pci/linux/pci_init.h +10 -0
@@ -37,6 +37,11 @@ int pci_uio_read_config(const struct rte_intr_handle *intr_handle,int pci_uio_write_config (const struct rte_intr_handle *intr_handle,const void *buf,size_t len,off_t offs); +int pci_uio_mmio_read (const struct rte_pci_device *dev,int bar, +void *buf,size_t len,off_t offset); +int pci_uio_mmio_write (const struct rte_pci_device *dev,int bar, +const void *buf,size_t len,off_t offset); +int pci_uio_ioport_map (struct rte_pci_device *dev,int bar,struct rte_pci_ioport *p); void pci_uio_ioport_read(struct rte_pci_ioport *p,
@@ -71,6 +76,11 @@ int pci_vfio_read_config(const struct rte_pci_device *dev,int pci_vfio_write_config (const struct rte_pci_device *dev,const void *buf,size_t len,off_t offs); +int pci_vfio_mmio_read (const struct rte_pci_device *dev,int bar, +void *buf,size_t len,off_t offset); +int pci_vfio_mmio_write (const struct rte_pci_device *dev,int bar, +const void *buf,size_t len,off_t offset); +int pci_vfio_ioport_map (struct rte_pci_device *dev,int bar,struct rte_pci_ioport *p); void pci_vfio_ioport_read(struct rte_pci_ioport *p,
drivers/bus/pci/linux/pci_uio.c +22 -0
@@ -55,6 +55,28 @@ pci_uio_write_config(const struct rte_intr_handle *intr_handle,return pwrite (uio_cfg_fd, buf, len, offset); } +int +pci_uio_mmio_read (const struct rte_pci_device *dev,int bar, +void *buf,size_t len,off_t offset) +{ +if (bar >=PCI_MAX_RESOURCE || dev->mem_resource[bar].addr ==NULL || + (uint64_t )offset + len > dev->mem_resource[bar].len) +return -1 ; +memcpy (buf, (uint8_t *)dev->mem_resource[bar].addr + offset, len); +return len; +} + +int +pci_uio_mmio_write (const struct rte_pci_device *dev,int bar, +const void *buf,size_t len,off_t offset) +{ +if (bar >=PCI_MAX_RESOURCE || dev->mem_resource[bar].addr ==NULL || + (uint64_t )offset + len > dev->mem_resource[bar].len) +return -1 ; +memcpy ((uint8_t *)dev->mem_resource[bar].addr + offset, buf, len); +return len; +} +static int pci_uio_set_bus_master (int dev_fd) {
drivers/bus/pci/linux/pci_vfio.c +36 -0
@@ -1258,6 +1258,42 @@ pci_vfio_ioport_unmap(struct rte_pci_ioport *p)return -1 ; } +int +pci_vfio_mmio_read (const struct rte_pci_device *dev,int bar, +void *buf,size_t len,off_t offs) +{ +uint64_t size, offset; +int fd; + + fd =rte_intr_dev_fd_get (dev->intr_handle); + +if (pci_vfio_get_region (dev, bar, &size, &offset) !=0 ) +return -1 ; + +if ((uint64_t )len + offs > size) +return -1 ; + +return pread64 (fd, buf, len, offset + offs); +} + +int +pci_vfio_mmio_write (const struct rte_pci_device *dev,int bar, +const void *buf,size_t len,off_t offs) +{ +uint64_t size, offset; +int fd; + + fd =rte_intr_dev_fd_get (dev->intr_handle); + +if (pci_vfio_get_region (dev, bar, &size, &offset) !=0 ) +return -1 ; + +if ((uint64_t )len + offs > size) +return -1 ; + +return pwrite64 (fd, buf, len, offset + offs); +} +int pci_vfio_is_enabled (void ) {
drivers/bus/pci/rte_bus_pci.h +48 -0
54 lines not on this page · raw message
drivers/bus/pci/version.map +3 -0
@@ -21,6 +21,9 @@ EXPERIMENTAL { # added in 21.08 rte_pci_set_bus_master; + # added in 23.07 + rte_pci_mmio_read; + rte_pci_mmio_write; }; INTERNAL {
Signature · 1 line
2.25.1
PATCH v2 4/4 bus/pci: add VFIO sparse mmap support
In reply to Miao LiThis patch adds sparse mmap support in PCI bus. Sparse mmap is a capability defined in VFIO which allows multiple mmap areas in one VFIO region.
In this patch, the sparse mmap regions are mapped to one continuous virtual address region that follows device-specific BAR layout. So, driver can still access all mapped sparse mmap regions by using 'bar_base_address + bar_offset'.
--- drivers/bus/pci/linux/pci_vfio.c | 104 +++++++++++++++++++++++++++---- drivers/bus/pci/private.h | 2 + 2 files changed, 94 insertions(+), 12 deletions(-)
drivers/bus/pci/linux/pci_vfio.c +92 -12
@@ -673,6 +673,54 @@ pci_vfio_mmap_bar(int vfio_dev_fd, struct mapped_pci_resource *vfio_res,return 0 ; } +static int +pci_vfio_sparse_mmap_bar (int vfio_dev_fd,struct mapped_pci_resource *vfio_res, +int bar_index,int additional_flags) +{ +struct pci_map *bar = &vfio_res->maps[bar_index]; +struct vfio_region_sparse_mmap_area *sparse; +void *bar_addr; +uint32_t i; + +if (bar->size ==0 ) { +RTE_LOG (DEBUG ,EAL ,"Bar size is 0, skip BAR%d\n" , bar_index); +return 0 ; + } + +/* reserve the address using an inaccessible mapping */ + bar_addr =mmap (bar->addr, bar->size,0 ,MAP_PRIVATE | +MAP_ANONYMOUS | additional_flags,-1 ,0 ); +if (bar_addr !=MAP_FAILED ) { +void *map_addr =NULL ; +for (i =0 ; i < bar->nr_areas; i++) { + sparse = &bar->areas[i]; +if (sparse->size) { +void *addr =RTE_PTR_ADD (bar_addr, (uintptr_t )sparse->offset); + map_addr =pci_map_resource (addr, vfio_dev_fd, + bar->offset + sparse->offset, sparse->size, +RTE_MAP_FORCE_ADDRESS ); +if (map_addr ==NULL ) { +munmap (bar_addr, bar->size); +RTE_LOG (ERR ,EAL ,"Failed to map pci BAR%d\n" , + bar_index); + goto err_map; + } + } + } + }else { +RTE_LOG (ERR ,EAL ,"Failed to create inaccessible mapping for BAR%d\n" , + bar_index); + goto err_map; + } + + bar->addr = bar_addr; +return 0 ; + +err_map: + bar->nr_areas =0 ; +return -1 ; +} +/* * region info may contain capability headers, so we need to keep reallocating * the memory until we match allocated memory size with argsz.
@@ -875,6 +923,8 @@ pci_vfio_map_resource_primary(struct rte_pci_device *dev)for (i =0 ; i < vfio_res->nb_maps; i++) {void *bar_addr; +struct vfio_info_cap_header *hdr; +struct vfio_region_info_cap_sparse_mmap *sparse; ret =pci_vfio_get_region_info (vfio_dev_fd, ®, i);if (ret <0 ) {
@@ -920,12 +970,33 @@ pci_vfio_map_resource_primary(struct rte_pci_device *dev) maps[i].size = reg->size; maps[i].path =NULL ;/* vfio doesn't have per-resource paths */ - ret =pci_vfio_mmap_bar (vfio_dev_fd, vfio_res, i,0 ); -if (ret <0 ) { -RTE_LOG (ERR ,EAL ,"%s mapping BAR%i failed: %s\n" , - pci_addr, i,strerror (errno)); -free (reg); - goto err_vfio_res; + hdr =pci_vfio_info_cap (reg,VFIO_REGION_INFO_CAP_SPARSE_MMAP ); + +if (hdr !=NULL ) { + sparse =container_of (hdr, + struct vfio_region_info_cap_sparse_mmap, header); +if (sparse->nr_areas >0 ) { + maps[i].nr_areas = sparse->nr_areas; + maps[i].areas = sparse->areas; + } + } + +if (maps[i].nr_areas >0 ) { + ret =pci_vfio_sparse_mmap_bar (vfio_dev_fd, vfio_res, i,0 ); +if (ret <0 ) { +RTE_LOG (ERR ,EAL ,"%s sparse mapping BAR%i failed: %s\n" , + pci_addr, i,strerror (errno)); +free (reg); + goto err_vfio_res; + } + }else { + ret =pci_vfio_mmap_bar (vfio_dev_fd, vfio_res, i,0 ); +if (ret <0 ) { +RTE_LOG (ERR ,EAL ,"%s mapping BAR%i failed: %s\n" , + pci_addr, i,strerror (errno)); +free (reg); + goto err_vfio_res; + } } dev->mem_resource[i].addr = maps[i].addr;
@@ -1008,11 +1079,20 @@ pci_vfio_map_resource_secondary(struct rte_pci_device *dev) maps = vfio_res->maps;for (i =0 ; i < vfio_res->nb_maps; i++) { - ret =pci_vfio_mmap_bar (vfio_dev_fd, vfio_res, i,MAP_FIXED ); -if (ret <0 ) { -RTE_LOG (ERR ,EAL ,"%s mapping BAR%i failed: %s\n" , - pci_addr, i,strerror (errno)); - goto err_vfio_dev_fd; +if (maps[i].nr_areas >0 ) { + ret =pci_vfio_sparse_mmap_bar (vfio_dev_fd, vfio_res, i,0 ); +if (ret <0 ) { +RTE_LOG (ERR ,EAL ,"%s sparse mapping BAR%i failed: %s\n" , + pci_addr, i,strerror (errno)); + goto err_vfio_dev_fd; + } + }else { + ret =pci_vfio_mmap_bar (vfio_dev_fd, vfio_res, i,0 ); +if (ret <0 ) { +RTE_LOG (ERR ,EAL ,"%s mapping BAR%i failed: %s\n" , + pci_addr, i,strerror (errno)); + goto err_vfio_dev_fd; + } } dev->mem_resource[i].addr = maps[i].addr;
@@ -1062,7 +1142,7 @@ find_and_unmap_vfio_resource(struct mapped_pci_res_list *vfio_res_list,break ; } -if (vfio_res ==NULL ) +if (vfio_res ==NULL )return vfio_res; RTE_LOG(INFO, EAL, "Releasing PCI mapped resource for %s\n",
drivers/bus/pci/private.h +2 -0
@@ -121,6 +121,8 @@ struct pci_map {uint64_t offset;uint64_t size;uint64_t phaddr; +uint32_t nr_areas; +struct vfio_region_sparse_mmap_area *areas; };struct pci_msix_table {
Signature · 1 line
2.25.1
Re: [PATCH v1 4/4] bus/pci: add VFIO sparse mmap support
In reply to Miao Lidrivers/bus/pci/linux/pci_vfio.c pci_vfio_mmap_bar(int vfio_dev_fd, struct mapped_pci_resource *vfio_res, · line 705
Quoted 7 lines · On Mon, 15 May 2023 06:47:00 +0000 Miao Li <[email protected]> wrote:
drivers/bus/pci/linux/pci_vfio.c +7
-0+ map_addr =pci_map_resource (addr, vfio_dev_fd, + bar->offset + sparse->offset, sparse->size, +RTE_MAP_FORCE_ADDRESS ); +if (map_addr ==NULL ) { +munmap (bar_addr, bar->size); +RTE_LOG (ERR ,EAL ,"Failed to map pci BAR%d\n" , + bar_index);
If mmap() fails then printing errno would help diagnose why.
RE: [PATCH v1 4/4] bus/pci: add VFIO sparse mmap support
Hi,
I will add errno print in patch v3.
Quoted 25 lines
-----Original Message-----
From: Stephen Hemminger <[email protected]>
Sent: Monday, May 15, 2023 11:53 PM
To: Li, Miao <[email protected]>
Cc: [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; Xia, Chenbo <[email protected]>; Cao, Yahui <[email protected]>; Burakov, Anatoly <[email protected]>
Subject: Re: [PATCH v1 4/4] bus/pci: add VFIO sparse mmap supportReview
Quoted 1 line · On Mon, 15 May 2023 06:47:00 +0000 Miao Li <[email protected]> wrote:
(quoted diff) +1
-0+ map_addr = pci_map_resource(addr,
vfio_dev_fd,drivers/bus/pci/linux/pci_vfio.c pci_vfio_mmap_bar(int vfio_dev_fd, struct mapped_pci_resource *vfio_res,
Quoted 6 lines
(quoted diff) +1
-0+ bar->offset + sparse->offset, sparse-size,
drivers/bus/pci/linux/pci_vfio.c +4
-0+ RTE_MAP_FORCE_ADDRESS); +if (map_addr ==NULL ) { +munmap (bar_addr, bar->size); + RTE_LOG(ERR, EAL, "Failed to map pciBAR%d\n",
Review
Quoted 1 line
(quoted diff) +1
-0+ bar_index);If
mmap()fails then printing errno would help diagnose why.
RE: [PATCH v1 4/4] bus/pci: add VFIO sparse mmap support
In reply to Stephen HemmingerHi Stephen,
Quoted 20 lines
-----Original Message-----
From: Stephen Hemminger <[email protected]>
Sent: Monday, May 15, 2023 11:53 PM
To: Li, Miao <[email protected]>
Cc: [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; Xia, Chenbo <[email protected]>; Cao, Yahui <[email protected]>; Burakov, Anatoly <[email protected]>
Subject: Re: [PATCH v1 4/4] bus/pci: add VFIO sparse mmap supportdrivers/bus/pci/linux/pci_vfio.c pci_vfio_mmap_bar(int vfio_dev_fd, struct mapped_pci_resource *vfio_res,
Quoted 6 lines · On Mon, 15 May 2023 06:47:00 +0000 Miao Li <[email protected]> wrote:
drivers/bus/pci/linux/pci_vfio.c +6
-0+ map_addr =pci_map_resource (addr, vfio_dev_fd, + bar->offset + sparse->offset, sparse->size, +RTE_MAP_FORCE_ADDRESS ); +if (map_addr ==NULL ) { +munmap (bar_addr, bar->size); + RTE_LOG(ERR, EAL, "Failed to map pciBAR%d\n",
Quoted 3 lines
Review
Quoted 1 line
(quoted diff) +1
-0+ bar_index);If
mmap()fails then printing errno would help diagnose why.
Thanks for your review! It seems errno will be printed in function pci_map_resource() when mmap() fails. So I guess we don't need it here?
Thanks,
Chenbo
v3 cover Support VFIO sparse mmap in PCI bus
In reply to Miao LiThis series introduces a VFIO standard capability, called sparse mmap to PCI bus. In linux kernel, it's defined as VFIO_REGION_INFO_CAP_SPARSE_MMAP. Sparse mmap means instead of mmap whole BAR region into DPDK process, only mmap part of the BAR region after getting sparse mmap information from kernel. For the rest of BAR region that is not mmap-ed, DPDK process can use pread/pwrite system calls to access. Sparse mmap is useful when kernel does not want userspace to mmap whole BAR region, or kernel wants to control over access to specific BAR region. Vendors can choose to enable this feature or not for their devices in their specific kernel modules.
In this patchset:
Patch 1-3 is mainly for introducing BAR access APIs so that driver could use them to access specific BAR using pread/pwrite system calls when part of the BAR is not mmap-able. Patch 4 adds the VFIO sparse mmap support finally.
v3:
fix variable 'pdev' and 'info' uninitialized error
v2:
1. add PCI device internal structure in bus/pci/windows/pci.c
2. fix parameter type error
Chenbo Xia (3):
bus/pci: introduce an internal representation of PCI device bus/pci: avoid depending on private value in kernel source bus/pci: introduce helper for MMIO read and write
Miao Li (1):
bus/pci: add VFIO sparse mmap support
drivers/bus/pci/bsd/pci.c | 35 +++- drivers/bus/pci/linux/pci.c | 78 +++++-- drivers/bus/pci/linux/pci_init.h | 14 +- drivers/bus/pci/linux/pci_uio.c | 22 ++ drivers/bus/pci/linux/pci_vfio.c | 337 +++++++++++++++++++++++++------ drivers/bus/pci/pci_common.c | 12 +- drivers/bus/pci/private.h | 25 ++- drivers/bus/pci/rte_bus_pci.h | 48 +++++ drivers/bus/pci/version.map | 3 + drivers/bus/pci/windows/pci.c | 14 +- lib/eal/include/rte_vfio.h | 1 - 11 files changed, 492 insertions(+), 97 deletions(-)
Signature · 1 line
2.25.1
PATCH v3 1/4 bus/pci: introduce an internal representation of PCI device
From: Chenbo Xia <[email protected]>
This patch introduces an internal representation of the PCI device which will be used to store the internal information that don't have to be exposed to drivers, e.g., the VFIO region sizes/offsets.
In this patch, the internal structure is simply a wrapper of the rte_pci_device structure. More fields will be added.
--- drivers/bus/pci/bsd/pci.c | 13 ++++++++----- drivers/bus/pci/linux/pci.c | 28 ++++++++++++++++------------ drivers/bus/pci/pci_common.c | 12 ++++++------ drivers/bus/pci/private.h | 14 +++++++++++++- drivers/bus/pci/windows/pci.c | 14 +++++++++----- 5 files changed, 52 insertions(+), 29 deletions(-)
drivers/bus/pci/bsd/pci.c +8 -5
@@ -208,16 +208,19 @@ pci_uio_map_resource_by_index(struct rte_pci_device *dev, int res_idx,static int pci_scan_one (int dev_pci_fd,struct pci_conf *conf) { +struct rte_pci_device_internal *pdev;struct rte_pci_device *dev;struct pci_bar_io bar;unsigned i, max; - dev =malloc (sizeof (*dev)); -if (dev ==NULL ) { + pdev =malloc (sizeof (*pdev)); +if (pdev ==NULL ) { +RTE_LOG (ERR ,EAL ,"Cannot allocate memory for internal pci device\n" );return -1 ; } -memset (dev,0 ,sizeof (*dev)); +memset (pdev,0 ,sizeof (*pdev)); + dev = &pdev->device; dev->device.bus = &rte_pci_bus.bus; dev->addr.domain = conf->pc_sel.pc_domain;
@@ -303,7 +306,7 @@ pci_scan_one(int dev_pci_fd, struct pci_conf *conf)memmove (dev2->mem_resource, dev->mem_resource,sizeof (dev->mem_resource)); -pci_free (dev); +pci_free (pdev); }return 0 ; }
@@ -313,7 +316,7 @@ pci_scan_one(int dev_pci_fd, struct pci_conf *conf)return 0 ; skipdev: -pci_free (dev); +pci_free (pdev);return 0 ; }
drivers/bus/pci/linux/pci.c +16 -12
@@ -211,22 +211,26 @@ pci_scan_one(const char *dirname, const struct rte_pci_addr *addr) {char filename[PATH_MAX ];unsigned long tmp; +struct rte_pci_device_internal *pdev;struct rte_pci_device *dev;char driver[PATH_MAX ];int ret; - dev =malloc (sizeof (*dev)); -if (dev ==NULL ) + pdev =malloc (sizeof (*pdev)); +if (pdev ==NULL ) { +RTE_LOG (ERR ,EAL ,"Cannot allocate memory for internal pci device\n" );return -1 ; + } -memset (dev,0 ,sizeof (*dev)); +memset (pdev,0 ,sizeof (*pdev)); + dev = &pdev->device; dev->device.bus = &rte_pci_bus.bus; dev->addr = *addr;/* get vendor id */ snprintf (filename,sizeof (filename),"%s/vendor" , dirname);if (eal_parse_sysfs_value (filename, &tmp) <0 ) { -pci_free (dev); +pci_free (pdev);return -1 ; } dev->id.vendor_id = (uint16_t )tmp;
@@ -234,7 +238,7 @@ pci_scan_one(const char *dirname, const struct rte_pci_addr *addr)/* get device id */ snprintf (filename,sizeof (filename),"%s/device" , dirname);if (eal_parse_sysfs_value (filename, &tmp) <0 ) { -pci_free (dev); +pci_free (pdev);return -1 ; } dev->id.device_id = (uint16_t )tmp;
@@ -243,7 +247,7 @@ pci_scan_one(const char *dirname, const struct rte_pci_addr *addr)snprintf (filename,sizeof (filename),"%s/subsystem_vendor" , dirname);if (eal_parse_sysfs_value (filename, &tmp) <0 ) { -pci_free (dev); +pci_free (pdev);return -1 ; } dev->id.subsystem_vendor_id = (uint16_t )tmp;
@@ -252,7 +256,7 @@ pci_scan_one(const char *dirname, const struct rte_pci_addr *addr)snprintf (filename,sizeof (filename),"%s/subsystem_device" , dirname);if (eal_parse_sysfs_value (filename, &tmp) <0 ) { -pci_free (dev); +pci_free (pdev);return -1 ; } dev->id.subsystem_device_id = (uint16_t )tmp;
@@ -261,7 +265,7 @@ pci_scan_one(const char *dirname, const struct rte_pci_addr *addr)snprintf (filename,sizeof (filename),"%s/class" , dirname);if (eal_parse_sysfs_value (filename, &tmp) <0 ) { -pci_free (dev); +pci_free (pdev);return -1 ; }/* the least 24 bits are valid: class, subclass, program interface */
@@ -297,7 +301,7 @@ pci_scan_one(const char *dirname, const struct rte_pci_addr *addr)snprintf (filename,sizeof (filename),"%s/resource" , dirname);if (pci_parse_sysfs_resource (filename, dev) <0 ) {RTE_LOG (ERR ,EAL ,"%s(): cannot parse resource\n" , __func__); -pci_free (dev); +pci_free (pdev);return -1 ; }
@@ -306,7 +310,7 @@ pci_scan_one(const char *dirname, const struct rte_pci_addr *addr) ret =pci_get_kernel_driver_by_path (filename, driver,sizeof (driver));if (ret <0 ) {RTE_LOG (ERR ,EAL ,"Fail to get kernel driver\n" ); -pci_free (dev); +pci_free (pdev);return -1 ; }
@@ -320,7 +324,7 @@ pci_scan_one(const char *dirname, const struct rte_pci_addr *addr)else dev->kdrv =RTE_PCI_KDRV_UNKNOWN ; }else { -pci_free (dev); +pci_free (pdev);return 0 ; }/* device is valid, add in list (sorted) */
@@ -375,7 +379,7 @@ pci_scan_one(const char *dirname, const struct rte_pci_addr *addr)pci_common_set (dev2); } } -pci_free (dev); +pci_free (pdev); }return 0 ; }
drivers/bus/pci/pci_common.c +6 -6
@@ -121,12 +121,12 @@ pci_common_set(struct rte_pci_device *dev) }void -pci_free (struct rte_pci_device *dev) +pci_free (struct rte_pci_device_internal *pdev) { -if (dev ==NULL ) +if (pdev ==NULL )return ; -free (dev->bus_info); -free (dev); +free (pdev->device.bus_info); +free (pdev); }/* map a particular resource from a file */
@@ -465,7 +465,7 @@ pci_cleanup(void)rte_intr_instance_free (dev->vfio_req_intr_handle); dev->vfio_req_intr_handle =NULL ; -pci_free (dev); +pci_free (RTE_PCI_DEVICE_INTERNAL (dev)); }return error;
@@ -681,7 +681,7 @@ pci_unplug(struct rte_device *dev)if (ret ==0 ) {rte_pci_remove_device (pdev);rte_devargs_remove (dev->devargs); -pci_free (pdev); +pci_free (RTE_PCI_DEVICE_INTERNAL (pdev)); }return ret; }
drivers/bus/pci/private.h +13 -1
@@ -13,6 +13,14 @@#include <rte_os_shim.h> #include <rte_pci.h> +/* +* Convert struct rte_pci_device to struct rte_pci_device_internal +*/ +#define RTE_PCI_DEVICE_INTERNAL (ptr) \ + container_of(ptr, struct rte_pci_device_internal, device) +#define RTE_PCI_DEVICE_INTERNAL_CONST (ptr) \ + container_of(ptr, const struct rte_pci_device_internal, device) +/** * Structure describing the PCI bus */
@@ -34,6 +42,10 @@ extern struct rte_pci_bus rte_pci_bus;struct rte_pci_driver ;struct rte_pci_device ; +struct rte_pci_device_internal { +struct rte_pci_device device; +}; +/** * Scan the content of the PCI bus, and the devices in the devices * list
@@ -53,7 +65,7 @@ pci_common_set(struct rte_pci_device *dev);* Free a PCI device. */ void -pci_free (struct rte_pci_device *dev); +pci_free (struct rte_pci_device_internal *pdev);/** * Validate whether a device with given PCI address should be ignored or not.
drivers/bus/pci/windows/pci.c +9 -5
@@ -336,6 +336,7 @@ set_kernel_driver_type(PSP_DEVINFO_DATA device_info_data,static int pci_scan_one (HDEVINFO dev_info,PSP_DEVINFO_DATA device_info_data) { +struct rte_pci_device_internal *pdev =NULL ;struct rte_pci_device *dev =NULL ;int ret =-1 ;char pci_device_info[REGSTR_VAL_MAX_HCID_LEN ];
@@ -370,11 +371,14 @@ pci_scan_one(HDEVINFO dev_info, PSP_DEVINFO_DATA device_info_data) goto end; } - dev =malloc (sizeof (*dev)); -if (dev ==NULL ) + pdev =malloc (sizeof (*pdev)); +if (pdev ==NULL ) { +RTE_LOG (ERR ,EAL ,"Cannot allocate memory for internal pci device\n" ); goto end; + } -memset (dev,0 ,sizeof (*dev)); +memset (pdev,0 ,sizeof (*pdev)); + dev = &pdev->device; dev->device.bus = &rte_pci_bus.bus; dev->addr = addr;
@@ -409,7 +413,7 @@ pci_scan_one(HDEVINFO dev_info, PSP_DEVINFO_DATA device_info_data) dev2->max_vfs = dev->max_vfs;memmove (dev2->mem_resource, dev->mem_resource,sizeof (dev->mem_resource)); -pci_free (dev); +pci_free (pdev); }return 0 ; }
@@ -418,7 +422,7 @@ pci_scan_one(HDEVINFO dev_info, PSP_DEVINFO_DATA device_info_data)return 0 ; end: -pci_free (dev); +pci_free (pdev);return ret; }
Signature · 1 line
2.25.1
PATCH v3 2/4 bus/pci: avoid depending on private value in kernel source
In reply to Miao LiFrom: Chenbo Xia <[email protected]>
The value 40 used in VFIO_GET_REGION_ADDR() is a private value (VFIO_PCI_OFFSET_SHIFT) defined in Linux kernel source [1]. It is not part of VFIO API, and we should not depend on it.
[1] https://github.com/torvalds/linux/blob/v6.2/include/linux/vfio_pci_core.h
--- drivers/bus/pci/linux/pci.c | 4 +- drivers/bus/pci/linux/pci_init.h | 4 +- drivers/bus/pci/linux/pci_vfio.c | 197 +++++++++++++++++++++++-------- drivers/bus/pci/private.h | 9 ++ lib/eal/include/rte_vfio.h | 1 - 5 files changed, 159 insertions(+), 56 deletions(-)
drivers/bus/pci/linux/pci.c +2 -2
@@ -645,7 +645,7 @@ int rte_pci_read_config(const struct rte_pci_device *device,return pci_uio_read_config (intr_handle, buf, len, offset);#ifdef VFIO_PRESENT case RTE_PCI_KDRV_VFIO : -return pci_vfio_read_config (intr_handle, buf, len, offset); +return pci_vfio_read_config (device, buf, len, offset);#endif default : rte_pci_device_name(&device->addr, devname,
@@ -669,7 +669,7 @@ int rte_pci_write_config(const struct rte_pci_device *device,return pci_uio_write_config (intr_handle, buf, len, offset);#ifdef VFIO_PRESENT case RTE_PCI_KDRV_VFIO : -return pci_vfio_write_config (intr_handle, buf, len, offset); +return pci_vfio_write_config (device, buf, len, offset);#endif default : rte_pci_device_name(&device->addr, devname,
drivers/bus/pci/linux/pci_init.h +2 -2
@@ -66,9 +66,9 @@ int pci_uio_ioport_unmap(struct rte_pci_ioport *p);#endif /* access config space */ -int pci_vfio_read_config (const struct rte_intr_handle *intr_handle, +int pci_vfio_read_config (const struct rte_pci_device *dev,void *buf,size_t len,off_t offs); -int pci_vfio_write_config (const struct rte_intr_handle *intr_handle, +int pci_vfio_write_config (const struct rte_pci_device *dev,const void *buf,size_t len,off_t offs); int pci_vfio_ioport_map(struct rte_pci_device *dev, int bar,
drivers/bus/pci/linux/pci_vfio.c +146 -51
371 lines not on this page · raw message
drivers/bus/pci/private.h +9 -0
@@ -13,6 +13,8 @@#include <rte_os_shim.h> #include <rte_pci.h> +#define RTE_MAX_PCI_REGIONS 9 +/* * Convert struct rte_pci_device to struct rte_pci_device_internal */
@@ -42,8 +44,15 @@ extern struct rte_pci_bus rte_pci_bus;struct rte_pci_driver ;struct rte_pci_device ; +struct rte_pci_region { +uint64_t size; +uint64_t offset; +}; +struct rte_pci_device_internal {struct rte_pci_device device; +/* PCI regions provided by e.g. VFIO. */ +struct rte_pci_region region[RTE_MAX_PCI_REGIONS ]; };/**
lib/eal/include/rte_vfio.h +0 -1
@@ -38,7 +38,6 @@ extern "C" {#define VFIO_CONTAINER_PATH "/dev/vfio/vfio"#define VFIO_GROUP_FMT "/dev/vfio/%u"#define VFIO_NOIOMMU_GROUP_FMT "/dev/vfio/noiommu-%u" -#define VFIO_GET_REGION_ADDR (x) ((uint64_t) x << 40ULL)#define VFIO_GET_REGION_IDX (x) (x >> 40)#define VFIO_NOIOMMU_MODE \ "/sys/module/vfio/parameters/enable_unsafe_noiommu_mode"
Signature · 1 line
2.25.1
PATCH v3 3/4 bus/pci: introduce helper for MMIO read and write
In reply to Miao LiFrom: Chenbo Xia <[email protected]>
The MMIO regions may not be mmap-able for VFIO-PCI devices. In this case, the driver should explicitly do read and write to access these regions.
--- drivers/bus/pci/bsd/pci.c | 22 +++++++++++++++ drivers/bus/pci/linux/pci.c | 46 ++++++++++++++++++++++++++++++ drivers/bus/pci/linux/pci_init.h | 10 +++++++ drivers/bus/pci/linux/pci_uio.c | 22 +++++++++++++++ drivers/bus/pci/linux/pci_vfio.c | 36 ++++++++++++++++++++++++ drivers/bus/pci/rte_bus_pci.h | 48 ++++++++++++++++++++++++++++++++ drivers/bus/pci/version.map | 3 ++ 7 files changed, 187 insertions(+)
drivers/bus/pci/bsd/pci.c +22 -0
@@ -489,6 +489,28 @@ int rte_pci_write_config(const struct rte_pci_device *dev,return -1 ; } +/* Read PCI MMIO space. */ +int rte_pci_mmio_read (const struct rte_pci_device *dev,int bar, +void *buf,size_t len,off_t offset) +{ +if (bar >=PCI_MAX_RESOURCE || dev->mem_resource[bar].addr ==NULL || + (uint64_t )offset + len > dev->mem_resource[bar].len) +return -1 ; +memcpy (buf, (uint8_t *)dev->mem_resource[bar].addr + offset, len); +return len; +} + +/* Write PCI MMIO space. */ +int rte_pci_mmio_write (const struct rte_pci_device *dev,int bar, +const void *buf,size_t len,off_t offset) +{ +if (bar >=PCI_MAX_RESOURCE || dev->mem_resource[bar].addr ==NULL || + (uint64_t )offset + len > dev->mem_resource[bar].len) +return -1 ; +memcpy ((uint8_t *)dev->mem_resource[bar].addr + offset, buf, len); +return len; +} +int rte_pci_ioport_map (struct rte_pci_device *dev,int bar,struct rte_pci_ioport *p)
drivers/bus/pci/linux/pci.c +46 -0
52 lines not on this page · raw message
drivers/bus/pci/linux/pci_init.h +10 -0
@@ -37,6 +37,11 @@ int pci_uio_read_config(const struct rte_intr_handle *intr_handle,int pci_uio_write_config (const struct rte_intr_handle *intr_handle,const void *buf,size_t len,off_t offs); +int pci_uio_mmio_read (const struct rte_pci_device *dev,int bar, +void *buf,size_t len,off_t offset); +int pci_uio_mmio_write (const struct rte_pci_device *dev,int bar, +const void *buf,size_t len,off_t offset); +int pci_uio_ioport_map (struct rte_pci_device *dev,int bar,struct rte_pci_ioport *p); void pci_uio_ioport_read(struct rte_pci_ioport *p,
@@ -71,6 +76,11 @@ int pci_vfio_read_config(const struct rte_pci_device *dev,int pci_vfio_write_config (const struct rte_pci_device *dev,const void *buf,size_t len,off_t offs); +int pci_vfio_mmio_read (const struct rte_pci_device *dev,int bar, +void *buf,size_t len,off_t offset); +int pci_vfio_mmio_write (const struct rte_pci_device *dev,int bar, +const void *buf,size_t len,off_t offset); +int pci_vfio_ioport_map (struct rte_pci_device *dev,int bar,struct rte_pci_ioport *p); void pci_vfio_ioport_read(struct rte_pci_ioport *p,
drivers/bus/pci/linux/pci_uio.c +22 -0
@@ -55,6 +55,28 @@ pci_uio_write_config(const struct rte_intr_handle *intr_handle,return pwrite (uio_cfg_fd, buf, len, offset); } +int +pci_uio_mmio_read (const struct rte_pci_device *dev,int bar, +void *buf,size_t len,off_t offset) +{ +if (bar >=PCI_MAX_RESOURCE || dev->mem_resource[bar].addr ==NULL || + (uint64_t )offset + len > dev->mem_resource[bar].len) +return -1 ; +memcpy (buf, (uint8_t *)dev->mem_resource[bar].addr + offset, len); +return len; +} + +int +pci_uio_mmio_write (const struct rte_pci_device *dev,int bar, +const void *buf,size_t len,off_t offset) +{ +if (bar >=PCI_MAX_RESOURCE || dev->mem_resource[bar].addr ==NULL || + (uint64_t )offset + len > dev->mem_resource[bar].len) +return -1 ; +memcpy ((uint8_t *)dev->mem_resource[bar].addr + offset, buf, len); +return len; +} +static int pci_uio_set_bus_master (int dev_fd) {
drivers/bus/pci/linux/pci_vfio.c +36 -0
@@ -1258,6 +1258,42 @@ pci_vfio_ioport_unmap(struct rte_pci_ioport *p)return -1 ; } +int +pci_vfio_mmio_read (const struct rte_pci_device *dev,int bar, +void *buf,size_t len,off_t offs) +{ +uint64_t size, offset; +int fd; + + fd =rte_intr_dev_fd_get (dev->intr_handle); + +if (pci_vfio_get_region (dev, bar, &size, &offset) !=0 ) +return -1 ; + +if ((uint64_t )len + offs > size) +return -1 ; + +return pread64 (fd, buf, len, offset + offs); +} + +int +pci_vfio_mmio_write (const struct rte_pci_device *dev,int bar, +const void *buf,size_t len,off_t offs) +{ +uint64_t size, offset; +int fd; + + fd =rte_intr_dev_fd_get (dev->intr_handle); + +if (pci_vfio_get_region (dev, bar, &size, &offset) !=0 ) +return -1 ; + +if ((uint64_t )len + offs > size) +return -1 ; + +return pwrite64 (fd, buf, len, offset + offs); +} +int pci_vfio_is_enabled (void ) {
drivers/bus/pci/rte_bus_pci.h +48 -0
54 lines not on this page · raw message
drivers/bus/pci/version.map +3 -0
@@ -21,6 +21,9 @@ EXPERIMENTAL { # added in 21.08 rte_pci_set_bus_master; + # added in 23.07 + rte_pci_mmio_read; + rte_pci_mmio_write; }; INTERNAL {
Signature · 1 line
2.25.1
PATCH v3 4/4 bus/pci: add VFIO sparse mmap support
In reply to Miao LiThis patch adds sparse mmap support in PCI bus. Sparse mmap is a capability defined in VFIO which allows multiple mmap areas in one VFIO region.
In this patch, the sparse mmap regions are mapped to one continuous virtual address region that follows device-specific BAR layout. So, driver can still access all mapped sparse mmap regions by using 'bar_base_address + bar_offset'.
--- drivers/bus/pci/linux/pci_vfio.c | 104 +++++++++++++++++++++++++++---- drivers/bus/pci/private.h | 2 + 2 files changed, 94 insertions(+), 12 deletions(-)
drivers/bus/pci/linux/pci_vfio.c +92 -12
@@ -673,6 +673,54 @@ pci_vfio_mmap_bar(int vfio_dev_fd, struct mapped_pci_resource *vfio_res,return 0 ; } +static int +pci_vfio_sparse_mmap_bar (int vfio_dev_fd,struct mapped_pci_resource *vfio_res, +int bar_index,int additional_flags) +{ +struct pci_map *bar = &vfio_res->maps[bar_index]; +struct vfio_region_sparse_mmap_area *sparse; +void *bar_addr; +uint32_t i; + +if (bar->size ==0 ) { +RTE_LOG (DEBUG ,EAL ,"Bar size is 0, skip BAR%d\n" , bar_index); +return 0 ; + } + +/* reserve the address using an inaccessible mapping */ + bar_addr =mmap (bar->addr, bar->size,0 ,MAP_PRIVATE | +MAP_ANONYMOUS | additional_flags,-1 ,0 ); +if (bar_addr !=MAP_FAILED ) { +void *map_addr =NULL ; +for (i =0 ; i < bar->nr_areas; i++) { + sparse = &bar->areas[i]; +if (sparse->size) { +void *addr =RTE_PTR_ADD (bar_addr, (uintptr_t )sparse->offset); + map_addr =pci_map_resource (addr, vfio_dev_fd, + bar->offset + sparse->offset, sparse->size, +RTE_MAP_FORCE_ADDRESS ); +if (map_addr ==NULL ) { +munmap (bar_addr, bar->size); +RTE_LOG (ERR ,EAL ,"Failed to map pci BAR%d\n" , + bar_index); + goto err_map; + } + } + } + }else { +RTE_LOG (ERR ,EAL ,"Failed to create inaccessible mapping for BAR%d\n" , + bar_index); + goto err_map; + } + + bar->addr = bar_addr; +return 0 ; + +err_map: + bar->nr_areas =0 ; +return -1 ; +} +/* * region info may contain capability headers, so we need to keep reallocating * the memory until we match allocated memory size with argsz.
@@ -875,6 +923,8 @@ pci_vfio_map_resource_primary(struct rte_pci_device *dev)for (i =0 ; i < vfio_res->nb_maps; i++) {void *bar_addr; +struct vfio_info_cap_header *hdr; +struct vfio_region_info_cap_sparse_mmap *sparse; ret =pci_vfio_get_region_info (vfio_dev_fd, ®, i);if (ret <0 ) {
@@ -920,12 +970,33 @@ pci_vfio_map_resource_primary(struct rte_pci_device *dev) maps[i].size = reg->size; maps[i].path =NULL ;/* vfio doesn't have per-resource paths */ - ret =pci_vfio_mmap_bar (vfio_dev_fd, vfio_res, i,0 ); -if (ret <0 ) { -RTE_LOG (ERR ,EAL ,"%s mapping BAR%i failed: %s\n" , - pci_addr, i,strerror (errno)); -free (reg); - goto err_vfio_res; + hdr =pci_vfio_info_cap (reg,VFIO_REGION_INFO_CAP_SPARSE_MMAP ); + +if (hdr !=NULL ) { + sparse =container_of (hdr, + struct vfio_region_info_cap_sparse_mmap, header); +if (sparse->nr_areas >0 ) { + maps[i].nr_areas = sparse->nr_areas; + maps[i].areas = sparse->areas; + } + } + +if (maps[i].nr_areas >0 ) { + ret =pci_vfio_sparse_mmap_bar (vfio_dev_fd, vfio_res, i,0 ); +if (ret <0 ) { +RTE_LOG (ERR ,EAL ,"%s sparse mapping BAR%i failed: %s\n" , + pci_addr, i,strerror (errno)); +free (reg); + goto err_vfio_res; + } + }else { + ret =pci_vfio_mmap_bar (vfio_dev_fd, vfio_res, i,0 ); +if (ret <0 ) { +RTE_LOG (ERR ,EAL ,"%s mapping BAR%i failed: %s\n" , + pci_addr, i,strerror (errno)); +free (reg); + goto err_vfio_res; + } } dev->mem_resource[i].addr = maps[i].addr;
@@ -1008,11 +1079,20 @@ pci_vfio_map_resource_secondary(struct rte_pci_device *dev) maps = vfio_res->maps;for (i =0 ; i < vfio_res->nb_maps; i++) { - ret =pci_vfio_mmap_bar (vfio_dev_fd, vfio_res, i,MAP_FIXED ); -if (ret <0 ) { -RTE_LOG (ERR ,EAL ,"%s mapping BAR%i failed: %s\n" , - pci_addr, i,strerror (errno)); - goto err_vfio_dev_fd; +if (maps[i].nr_areas >0 ) { + ret =pci_vfio_sparse_mmap_bar (vfio_dev_fd, vfio_res, i,0 ); +if (ret <0 ) { +RTE_LOG (ERR ,EAL ,"%s sparse mapping BAR%i failed: %s\n" , + pci_addr, i,strerror (errno)); + goto err_vfio_dev_fd; + } + }else { + ret =pci_vfio_mmap_bar (vfio_dev_fd, vfio_res, i,0 ); +if (ret <0 ) { +RTE_LOG (ERR ,EAL ,"%s mapping BAR%i failed: %s\n" , + pci_addr, i,strerror (errno)); + goto err_vfio_dev_fd; + } } dev->mem_resource[i].addr = maps[i].addr;
@@ -1062,7 +1142,7 @@ find_and_unmap_vfio_resource(struct mapped_pci_res_list *vfio_res_list,break ; } -if (vfio_res ==NULL ) +if (vfio_res ==NULL )return vfio_res; RTE_LOG(INFO, EAL, "Releasing PCI mapped resource for %s\n",
drivers/bus/pci/private.h +2 -0
@@ -121,6 +121,8 @@ struct pci_map {uint64_t offset;uint64_t size;uint64_t phaddr; +uint32_t nr_areas; +struct vfio_region_sparse_mmap_area *areas; };struct pci_msix_table {
Signature · 1 line
2.25.1
RE: [EXT] [PATCH v3 1/4] bus/pci: introduce an internal representation of PCI device
In reply to Miao LiQuoted 22 lines
-----Original Message-----
From: Miao Li <[email protected]>
Sent: Thursday, May 25, 2023 10:01 PM
To: [email protected]
Cc: Sunil Kumar Kori <[email protected]>; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]
Subject: [EXT] [PATCH v3 1/4] bus/pci: introduce an internal representation of PCI deviceExternal Email
----------------------------------------------------------------------
From: Chenbo Xia <[email protected]>This patch introduces an internal representation of the PCI device which will be used to store the internal information that don't have to be exposed to drivers, e.g., the VFIO region sizes/offsets.
In this patch, the internal structure is simply a wrapper of the
rte_pci_devicestructure. More fields will be added.
Quoted 9 lines
Signed-off-by Chenbo Xia <[email protected]>--- drivers/bus/pci/bsd/pci.c | 13 ++++++++----- drivers/bus/pci/linux/pci.c | 28 ++++++++++++++++------------ drivers/bus/pci/pci_common.c | 12 ++++++------ drivers/bus/pci/private.h | 14 +++++++++++++- drivers/bus/pci/windows/pci.c | 14 +++++++++----- 5 files changed, 52 insertions(+), 29 deletions(-)
...
[snipped]
...
Quoted 1 line
2.25.1
RE: [EXT] [PATCH v3 2/4] bus/pci: avoid depending on private value in kernel source
In reply to Miao LiQuoted 28 lines
-----Original Message-----
From: Miao Li <[email protected]>
Sent: Thursday, May 25, 2023 10:01 PM
To: [email protected]
Cc: Sunil Kumar Kori <[email protected]>; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; Anatoly Burakov <[email protected]>
Subject: [EXT] [PATCH v3 2/4] bus/pci: avoid depending on private value in kernel sourceExternal Email
----------------------------------------------------------------------
From: Chenbo Xia <[email protected]>The value 40 used in
VFIO_GET_REGION_ADDR()is a private value (VFIO_PCI_OFFSET_SHIFT) defined in Linux kernel source [1]. It is not part of VFIO API, and we should not depend on it.[1] https://urldefense.proofpoint.com/v2/url?u=https-
3A__github.com_torvalds_linux_blob_v6.2_include_linux_vfio-5Fpci- 5Fcore.h&d=DwIDAg&c=nKjWec2b6R0mOyPaz7xtfQ&r=dXeXaAMkP5COgn1z xHMyaF1_d9IIuq6vHQO6NrIPjaE&m=JmiGBKn8A8yznTdPB1knOyEeYM4moYy ws5F5wylRMz9_Jp8-FRr- _FDWaUpA6a7U&s=DBgE0M81mcB0EWqXuq8apKbHmhKQIQ52RFcPWdHXat s&e=
Quoted 9 lines
Signed-off-by Chenbo Xia <[email protected]>--- drivers/bus/pci/linux/pci.c | 4 +- drivers/bus/pci/linux/pci_init.h | 4 +- drivers/bus/pci/linux/pci_vfio.c | 197 +++++++++++++++++++++++-------- drivers/bus/pci/private.h | 9 ++ lib/eal/include/rte_vfio.h | 1 - 5 files changed, 159 insertions(+), 56 deletions(-)
...
[snipped]
...
Quoted 1 line
2.25.1
RE: [EXT] [PATCH v3 3/4] bus/pci: introduce helper for MMIO read and write
In reply to Miao LiQuoted 21 lines
-----Original Message-----
From: Miao Li <[email protected]>
Sent: Thursday, May 25, 2023 10:01 PM
To: [email protected]
Cc: Sunil Kumar Kori <[email protected]>; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; Anatoly Burakov <[email protected]>
Subject: [EXT] [PATCH v3 3/4] bus/pci: introduce helper for MMIO read and writeExternal Email
----------------------------------------------------------------------
From: Chenbo Xia <[email protected]>The MMIO regions may not be mmap-able for VFIO-PCI devices.
In this case, the driver should explicitly do read and write to access these regions.Signed-off-by Chenbo Xia <[email protected]>
Quoted 10 lines
--- drivers/bus/pci/bsd/pci.c | 22 +++++++++++++++ drivers/bus/pci/linux/pci.c | 46 ++++++++++++++++++++++++++++++ drivers/bus/pci/linux/pci_init.h | 10 +++++++ drivers/bus/pci/linux/pci_uio.c | 22 +++++++++++++++ drivers/bus/pci/linux/pci_vfio.c | 36 ++++++++++++++++++++++++ drivers/bus/pci/rte_bus_pci.h | 48 ++++++++++++++++++++++++++++++++ drivers/bus/pci/version.map | 3 ++ 7 files changed, 187 insertions(+)
...
[snipped]
...
Quoted 1 line
2.25.1
RE: [EXT] [PATCH v3 4/4] bus/pci: add VFIO sparse mmap support
In reply to Miao LiQuoted 21 lines
-----Original Message-----
From: Miao Li <[email protected]>
Sent: Thursday, May 25, 2023 10:01 PM
To: [email protected]
Cc: Sunil Kumar Kori <[email protected]>; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; Anatoly Burakov <[email protected]>
Subject: [EXT] [PATCH v3 4/4] bus/pci: add VFIO sparse mmap supportExternal Email
----------------------------------------------------------------------
This patch adds sparse mmap support in PCI bus. Sparse mmap is a capability defined in VFIO which allows multiple mmap areas in one VFIO region.In this patch, the sparse mmap regions are mapped to one continuous virtual address region that follows device-specific BAR layout. So, driver can still access all mapped sparse mmap regions by using '
bar_base_address+bar_offset'.
Quoted 7 lines
Signed-off-by Miao Li <[email protected]>Signed-off-by Chenbo Xia <[email protected]>--- drivers/bus/pci/linux/pci_vfio.c | 104 +++++++++++++++++++++++++++---- drivers/bus/pci/private.h | 2 + 2 files changed, 94 insertions(+), 12 deletions(-)
...
[snipped]
...
Quoted 1 line
2.25.1
Re: [PATCH v3 1/4] bus/pci: introduce an internal representation of PCI device
In reply to Miao LiQuoted 9 lines · Miao Li
From: Chenbo Xia <[email protected]>
This patch introduces an internal representation of the PCI device which will be used to store the internal information that don't have to be exposed to drivers, e.g., the VFIO region sizes/offsets.
In this patch, the internal structure is simply a wrapper of the
rte_pci_devicestructure. More fields will be added.
Quoted 9 lines
Signed-off-by Chenbo Xia <[email protected]>--- drivers/bus/pci/bsd/pci.c | 13 ++++++++----- drivers/bus/pci/linux/pci.c | 28 ++++++++++++++++------------ drivers/bus/pci/pci_common.c | 12 ++++++------ drivers/bus/pci/private.h | 14 +++++++++++++- drivers/bus/pci/windows/pci.c | 14 +++++++++----- 5 files changed, 52 insertions(+), 29 deletions(-)
Re: [PATCH v3 2/4] bus/pci: avoid depending on private value in kernel source
In reply to Miao LiQuoted 8 lines · Miao Li
From: Chenbo Xia <[email protected]>
The value 40 used in
VFIO_GET_REGION_ADDR()is a private value (VFIO_PCI_OFFSET_SHIFT) defined in Linux kernel source [1]. It is not part of VFIO API, and we should not depend on it.[1] https://github.com/torvalds/linux/blob/v6.2/include/linux/vfio_pci_core.h
Quoted 9 lines
Signed-off-by Chenbo Xia <[email protected]>--- drivers/bus/pci/linux/pci.c | 4 +- drivers/bus/pci/linux/pci_init.h | 4 +- drivers/bus/pci/linux/pci_vfio.c | 197 +++++++++++++++++++++++-------- drivers/bus/pci/private.h | 9 ++ lib/eal/include/rte_vfio.h | 1 - 5 files changed, 159 insertions(+), 56 deletions(-)
Re: [PATCH v3 3/4] bus/pci: introduce helper for MMIO read and write
In reply to Miao LiQuoted 7 lines · Miao Li
From: Chenbo Xia <[email protected]>
The MMIO regions may not be mmap-able for VFIO-PCI devices. In this case, the driver should explicitly do read and write to access these regions.
Signed-off-by Chenbo Xia <[email protected]>
Quoted 10 lines
--- drivers/bus/pci/bsd/pci.c | 22 +++++++++++++++ drivers/bus/pci/linux/pci.c | 46 ++++++++++++++++++++++++++++++ drivers/bus/pci/linux/pci_init.h | 10 +++++++ drivers/bus/pci/linux/pci_uio.c | 22 +++++++++++++++ drivers/bus/pci/linux/pci_vfio.c | 36 ++++++++++++++++++++++++ drivers/bus/pci/rte_bus_pci.h | 48 ++++++++++++++++++++++++++++++++ drivers/bus/pci/version.map | 3 ++ 7 files changed, 187 insertions(+)
Re: [PATCH v3 4/4] bus/pci: add VFIO sparse mmap support
In reply to Miao LiQuoted 9 lines · Miao Li
This patch adds sparse mmap support in PCI bus. Sparse mmap is a capability defined in VFIO which allows multiple mmap areas in one VFIO region.
In this patch, the sparse mmap regions are mapped to one continuous virtual address region that follows device-specific BAR layout. So, driver can still access all mapped sparse mmap regions by using '
bar_base_address+bar_offset'.
Quoted 7 lines
Signed-off-by Miao Li <[email protected]>Signed-off-by Chenbo Xia <[email protected]>--- drivers/bus/pci/linux/pci_vfio.c | 104 +++++++++++++++++++++++++++---- drivers/bus/pci/private.h | 2 + 2 files changed, 94 insertions(+), 12 deletions(-)
RE: [PATCH v3 4/4] bus/pci: add VFIO sparse mmap support
In reply to Miao LiQuoted 111 lines
-----Original Message-----
From: Li, Miao <[email protected]>
Sent: Friday, May 26, 2023 12:31 AM
To: [email protected]
Cc: [email protected]; [email protected]; [email protected]; [email protected]; Xia, Chenbo <[email protected]>; Cao, Yahui <[email protected]>; Burakov, Anatoly <[email protected]>
Subject: [PATCH v3 4/4] bus/pci: add VFIO sparse mmap supportThis patch adds sparse mmap support in PCI bus. Sparse mmap is a capability defined in VFIO which allows multiple mmap areas in one VFIO region.
In this patch, the sparse mmap regions are mapped to one continuous virtual address region that follows device-specific BAR layout. So, driver can still access all mapped sparse mmap regions by using '
bar_base_address+bar_offset'.Signed-off-by Miao Li <[email protected]>Signed-off-by Chenbo Xia <[email protected]>--- drivers/bus/pci/linux/pci_vfio.c | 104 +++++++++++++++++++++++++++---- drivers/bus/pci/private.h | 2 + 2 files changed, 94 insertions(+), 12 deletions(-)drivers/bus/pci/linux/pci_vfio.c +29
-0@@ -673,6 +673,54 @@ pci_vfio_mmap_bar(int vfio_dev_fd, structmapped_pci_resource *vfio_res, return 0; } +static int +pci_vfio_sparse_mmap_bar (int vfio_dev_fd,struct mapped_pci_resource *vfio_res, +int bar_index,int additional_flags) +{ +struct pci_map *bar = &vfio_res->maps[bar_index]; +struct vfio_region_sparse_mmap_area *sparse; +void *bar_addr; +uint32_t i; + +if (bar->size ==0 ) { +RTE_LOG (DEBUG ,EAL ,"Bar size is 0, skip BAR%d\n" , bar_index); +return 0 ; + } + +/* reserve the address using an inaccessible mapping */ + bar_addr =mmap (bar->addr, bar->size,0 ,MAP_PRIVATE | +MAP_ANONYMOUS | additional_flags,-1 ,0 ); +if (bar_addr !=MAP_FAILED ) { +void *map_addr =NULL ; +for (i =0 ; i < bar->nr_areas; i++) { + sparse = &bar->areas[i]; +if (sparse->size) { +void *addr =RTE_PTR_ADD (bar_addr, (uintptr_t )sparse->offset); + map_addr =pci_map_resource (addr, vfio_dev_fd, + bar->offset + sparse->offset, sparse->size, +RTE_MAP_FORCE_ADDRESS ); +if (map_addr ==NULL ) { +munmap (bar_addr, bar->size); + RTE_LOG(ERR, EAL, "Failed to map pciBAR%d\n",
drivers/bus/pci/linux/pci_vfio.c +7
-0+ bar_index); + goto err_map; + } + } + } + }else { + RTE_LOG(ERR, EAL, "Failed to create inaccessible mapping forBAR%d\n",
drivers/bus/pci/linux/pci_vfio.c +12
-0+ bar_index); + goto err_map; + } + + bar->addr = bar_addr; +return 0 ; + +err_map: + bar->nr_areas =0 ; +return -1 ; +} +/* * region info may contain capability headers, so we need to keep reallocating
* the memory until we match allocated memory size with argsz.drivers/bus/pci/linux/pci_vfio.c +2
-0@@ -875,6 +923,8 @@ pci_vfio_map_resource_primary(struct rte_pci_device *dev)for (i =0 ; i < vfio_res->nb_maps; i++) {void *bar_addr; +struct vfio_info_cap_header *hdr; +struct vfio_region_info_cap_sparse_mmap *sparse; ret =pci_vfio_get_region_info (vfio_dev_fd, ®, i);drivers/bus/pci/linux/pci_vfio.c +0
-4if (ret <0 ) {@@ -920,12 +970,33 @@ pci_vfio_map_resource_primary(struct rte_pci_device *dev) maps[i].size = reg->size; maps[i].path =NULL ;/* vfio doesn't have per-resource paths */ - ret =pci_vfio_mmap_bar (vfio_dev_fd, vfio_res, i,0 ); -if (ret <0 ) { -RTE_LOG (ERR ,EAL ,"%s mapping BAR%i failed: %s\n" , - pci_addr, i,strerror (errno));
drivers/bus/pci/linux/pci_vfio.c pci_vfio_map_resource_primary(struct rte_pci_device · line 982
Quoted 10 lines
drivers/bus/pci/linux/pci_vfio.c +8
-2-free (reg); - goto err_vfio_res; + hdr =pci_vfio_info_cap (reg,VFIO_REGION_INFO_CAP_SPARSE_MMAP ); + +if (hdr !=NULL ) { + sparse =container_of (hdr, + struct vfio_region_info_cap_sparse_mmap, header); +if (sparse->nr_areas >0 ) { + maps[i].nr_areas = sparse->nr_areas; + maps[i].areas = sparse->areas;
I just notice that this is wrong as the memory that pointer 'sparse' points to will be freed at the end. map[i].areas needs to be allocated by rte_zmalloc and freed correctly. Otherwise it could leads to secondary process segfault when it tries to access maps[i].areas.
Will fix this in v4.
Thanks,
Chenbo
Quoted 78 lines
drivers/bus/pci/linux/pci_vfio.c +7
-0+ } + } + +if (maps[i].nr_areas >0 ) { + ret =pci_vfio_sparse_mmap_bar (vfio_dev_fd, vfio_res, i,0 ); +if (ret <0 ) { + RTE_LOG(ERR, EAL, "%s sparse mapping BAR%ifailed: %s\n",
drivers/bus/pci/linux/pci_vfio.c +26
-5+ pci_addr, i, strerror(errno)); +free (reg); + goto err_vfio_res; + } + }else { + ret =pci_vfio_mmap_bar (vfio_dev_fd, vfio_res, i,0 ); +if (ret <0 ) { +RTE_LOG (ERR ,EAL ,"%s mapping BAR%i failed: %s\n" , + pci_addr, i,strerror (errno)); +free (reg); + goto err_vfio_res; + } } dev->mem_resource[i].addr = maps[i].addr;@@ -1008,11 +1079,20 @@ pci_vfio_map_resource_secondary(struct rte_pci_device *dev) maps = vfio_res->maps;for (i =0 ; i < vfio_res->nb_maps; i++) { - ret =pci_vfio_mmap_bar (vfio_dev_fd, vfio_res, i,MAP_FIXED ); -if (ret <0 ) { -RTE_LOG (ERR ,EAL ,"%s mapping BAR%i failed: %s\n" , - pci_addr, i,strerror (errno)); - goto err_vfio_dev_fd; +if (maps[i].nr_areas >0 ) { + ret =pci_vfio_sparse_mmap_bar (vfio_dev_fd, vfio_res, i,0 ); +if (ret <0 ) { +RTE_LOG (ERR ,EAL ,"%s sparse mapping BAR%i failed: %s\n", + pci_addr, i,strerror (errno)); + goto err_vfio_dev_fd; + } + }else { + ret =pci_vfio_mmap_bar (vfio_dev_fd, vfio_res, i,0 ); +if (ret <0 ) { +RTE_LOG (ERR ,EAL ,"%s mapping BAR%i failed: %s\n" , + pci_addr, i,strerror (errno)); + goto err_vfio_dev_fd; + }drivers/bus/pci/linux/pci_vfio.c +1
-1} dev->mem_resource[i].addr = maps[i].addr;@@ -1062,7 +1142,7 @@ find_and_unmap_vfio_resource(structmapped_pci_res_list *vfio_res_list, break; } -if (vfio_res ==NULL ) +if (vfio_res ==NULL )return vfio_res;drivers/bus/pci/linux/pci_vfio.c +0
-0RTE_LOG(INFO, EAL, "Releasing PCI mapped resource for %s\n",drivers/bus/pci/private.h +2
-1@@ -121,6 +121,8 @@ struct pci_map {uint64_t offset;uint64_t size;uint64_t phaddr; +uint32_t nr_areas; +struct vfio_region_sparse_mmap_area *areas; };struct pci_msix_table { --2.25.1
v4 cover Support VFIO sparse mmap in PCI bus
In reply to Miao LiThis series introduces a VFIO standard capability, called sparse mmap to PCI bus. In linux kernel, it's defined as VFIO_REGION_INFO_CAP_SPARSE_MMAP. Sparse mmap means instead of mmap whole BAR region into DPDK process, only mmap part of the BAR region after getting sparse mmap information from kernel. For the rest of BAR region that is not mmap-ed, DPDK process can use pread/pwrite system calls to access. Sparse mmap is useful when kernel does not want userspace to mmap whole BAR region, or kernel wants to control over access to specific BAR region. Vendors can choose to enable this feature or not for their devices in their specific kernel modules.
In this patchset:
Patch 1-3 is mainly for introducing BAR access APIs so that driver could use them to access specific BAR using pread/pwrite system calls when part of the BAR is not mmap-able. Patch 4 adds the VFIO sparse mmap support finally.
v4:
1. add sparse mmap information allocation and release
2. add release note for BAR access APIs
v3:
fix variable 'pdev' and 'info' uninitialized error
v2:
1. add PCI device internal structure in bus/pci/windows/pci.c
2. fix parameter type error
Chenbo Xia (3):
bus/pci: introduce an internal representation of PCI device bus/pci: avoid depending on private value in kernel source bus/pci: introduce helper for MMIO read and write
Miao Li (1):
bus/pci: add VFIO sparse mmap support
doc/guides/rel_notes/release_23_07.rst | 5 + drivers/bus/pci/bsd/pci.c | 35 ++- drivers/bus/pci/linux/pci.c | 78 +++++- drivers/bus/pci/linux/pci_init.h | 14 +- drivers/bus/pci/linux/pci_uio.c | 22 ++ drivers/bus/pci/linux/pci_vfio.c | 371 ++++++++++++++++++++----- drivers/bus/pci/pci_common.c | 12 +- drivers/bus/pci/private.h | 25 +- drivers/bus/pci/rte_bus_pci.h | 48 ++++ drivers/bus/pci/version.map | 3 + drivers/bus/pci/windows/pci.c | 14 +- lib/eal/include/rte_vfio.h | 1 - 12 files changed, 525 insertions(+), 103 deletions(-)
Signature · 1 line
2.25.1
PATCH v4 1/4 bus/pci: introduce an internal representation of PCI device
From: Chenbo Xia <[email protected]>
This patch introduces an internal representation of the PCI device which will be used to store the internal information that don't have to be exposed to drivers, e.g., the VFIO region sizes/offsets.
In this patch, the internal structure is simply a wrapper of the rte_pci_device structure. More fields will be added.
--- drivers/bus/pci/bsd/pci.c | 13 ++++++++----- drivers/bus/pci/linux/pci.c | 28 ++++++++++++++++------------ drivers/bus/pci/pci_common.c | 12 ++++++------ drivers/bus/pci/private.h | 14 +++++++++++++- drivers/bus/pci/windows/pci.c | 14 +++++++++----- 5 files changed, 52 insertions(+), 29 deletions(-)
drivers/bus/pci/bsd/pci.c +8 -5
@@ -208,16 +208,19 @@ pci_uio_map_resource_by_index(struct rte_pci_device *dev, int res_idx,static int pci_scan_one (int dev_pci_fd,struct pci_conf *conf) { +struct rte_pci_device_internal *pdev;struct rte_pci_device *dev;struct pci_bar_io bar;unsigned i, max; - dev =malloc (sizeof (*dev)); -if (dev ==NULL ) { + pdev =malloc (sizeof (*pdev)); +if (pdev ==NULL ) { +RTE_LOG (ERR ,EAL ,"Cannot allocate memory for internal pci device\n" );return -1 ; } -memset (dev,0 ,sizeof (*dev)); +memset (pdev,0 ,sizeof (*pdev)); + dev = &pdev->device; dev->device.bus = &rte_pci_bus.bus; dev->addr.domain = conf->pc_sel.pc_domain;
@@ -303,7 +306,7 @@ pci_scan_one(int dev_pci_fd, struct pci_conf *conf)memmove (dev2->mem_resource, dev->mem_resource,sizeof (dev->mem_resource)); -pci_free (dev); +pci_free (pdev); }return 0 ; }
@@ -313,7 +316,7 @@ pci_scan_one(int dev_pci_fd, struct pci_conf *conf)return 0 ; skipdev: -pci_free (dev); +pci_free (pdev);return 0 ; }
drivers/bus/pci/linux/pci.c +16 -12
@@ -211,22 +211,26 @@ pci_scan_one(const char *dirname, const struct rte_pci_addr *addr) {char filename[PATH_MAX ];unsigned long tmp; +struct rte_pci_device_internal *pdev;struct rte_pci_device *dev;char driver[PATH_MAX ];int ret; - dev =malloc (sizeof (*dev)); -if (dev ==NULL ) + pdev =malloc (sizeof (*pdev)); +if (pdev ==NULL ) { +RTE_LOG (ERR ,EAL ,"Cannot allocate memory for internal pci device\n" );return -1 ; + } -memset (dev,0 ,sizeof (*dev)); +memset (pdev,0 ,sizeof (*pdev)); + dev = &pdev->device; dev->device.bus = &rte_pci_bus.bus; dev->addr = *addr;/* get vendor id */ snprintf (filename,sizeof (filename),"%s/vendor" , dirname);if (eal_parse_sysfs_value (filename, &tmp) <0 ) { -pci_free (dev); +pci_free (pdev);return -1 ; } dev->id.vendor_id = (uint16_t )tmp;
@@ -234,7 +238,7 @@ pci_scan_one(const char *dirname, const struct rte_pci_addr *addr)/* get device id */ snprintf (filename,sizeof (filename),"%s/device" , dirname);if (eal_parse_sysfs_value (filename, &tmp) <0 ) { -pci_free (dev); +pci_free (pdev);return -1 ; } dev->id.device_id = (uint16_t )tmp;
@@ -243,7 +247,7 @@ pci_scan_one(const char *dirname, const struct rte_pci_addr *addr)snprintf (filename,sizeof (filename),"%s/subsystem_vendor" , dirname);if (eal_parse_sysfs_value (filename, &tmp) <0 ) { -pci_free (dev); +pci_free (pdev);return -1 ; } dev->id.subsystem_vendor_id = (uint16_t )tmp;
@@ -252,7 +256,7 @@ pci_scan_one(const char *dirname, const struct rte_pci_addr *addr)snprintf (filename,sizeof (filename),"%s/subsystem_device" , dirname);if (eal_parse_sysfs_value (filename, &tmp) <0 ) { -pci_free (dev); +pci_free (pdev);return -1 ; } dev->id.subsystem_device_id = (uint16_t )tmp;
@@ -261,7 +265,7 @@ pci_scan_one(const char *dirname, const struct rte_pci_addr *addr)snprintf (filename,sizeof (filename),"%s/class" , dirname);if (eal_parse_sysfs_value (filename, &tmp) <0 ) { -pci_free (dev); +pci_free (pdev);return -1 ; }/* the least 24 bits are valid: class, subclass, program interface */
@@ -297,7 +301,7 @@ pci_scan_one(const char *dirname, const struct rte_pci_addr *addr)snprintf (filename,sizeof (filename),"%s/resource" , dirname);if (pci_parse_sysfs_resource (filename, dev) <0 ) {RTE_LOG (ERR ,EAL ,"%s(): cannot parse resource\n" , __func__); -pci_free (dev); +pci_free (pdev);return -1 ; }
@@ -306,7 +310,7 @@ pci_scan_one(const char *dirname, const struct rte_pci_addr *addr) ret =pci_get_kernel_driver_by_path (filename, driver,sizeof (driver));if (ret <0 ) {RTE_LOG (ERR ,EAL ,"Fail to get kernel driver\n" ); -pci_free (dev); +pci_free (pdev);return -1 ; }
@@ -320,7 +324,7 @@ pci_scan_one(const char *dirname, const struct rte_pci_addr *addr)else dev->kdrv =RTE_PCI_KDRV_UNKNOWN ; }else { -pci_free (dev); +pci_free (pdev);return 0 ; }/* device is valid, add in list (sorted) */
@@ -375,7 +379,7 @@ pci_scan_one(const char *dirname, const struct rte_pci_addr *addr)pci_common_set (dev2); } } -pci_free (dev); +pci_free (pdev); }return 0 ; }
drivers/bus/pci/pci_common.c +6 -6
@@ -121,12 +121,12 @@ pci_common_set(struct rte_pci_device *dev) }void -pci_free (struct rte_pci_device *dev) +pci_free (struct rte_pci_device_internal *pdev) { -if (dev ==NULL ) +if (pdev ==NULL )return ; -free (dev->bus_info); -free (dev); +free (pdev->device.bus_info); +free (pdev); }/* map a particular resource from a file */
@@ -465,7 +465,7 @@ pci_cleanup(void)rte_intr_instance_free (dev->vfio_req_intr_handle); dev->vfio_req_intr_handle =NULL ; -pci_free (dev); +pci_free (RTE_PCI_DEVICE_INTERNAL (dev)); }return error;
@@ -681,7 +681,7 @@ pci_unplug(struct rte_device *dev)if (ret ==0 ) {rte_pci_remove_device (pdev);rte_devargs_remove (dev->devargs); -pci_free (pdev); +pci_free (RTE_PCI_DEVICE_INTERNAL (pdev)); }return ret; }
drivers/bus/pci/private.h +13 -1
@@ -13,6 +13,14 @@#include <rte_os_shim.h> #include <rte_pci.h> +/* +* Convert struct rte_pci_device to struct rte_pci_device_internal +*/ +#define RTE_PCI_DEVICE_INTERNAL (ptr) \ + container_of(ptr, struct rte_pci_device_internal, device) +#define RTE_PCI_DEVICE_INTERNAL_CONST (ptr) \ + container_of(ptr, const struct rte_pci_device_internal, device) +/** * Structure describing the PCI bus */
@@ -34,6 +42,10 @@ extern struct rte_pci_bus rte_pci_bus;struct rte_pci_driver ;struct rte_pci_device ; +struct rte_pci_device_internal { +struct rte_pci_device device; +}; +/** * Scan the content of the PCI bus, and the devices in the devices * list
@@ -53,7 +65,7 @@ pci_common_set(struct rte_pci_device *dev);* Free a PCI device. */ void -pci_free (struct rte_pci_device *dev); +pci_free (struct rte_pci_device_internal *pdev);/** * Validate whether a device with given PCI address should be ignored or not.
drivers/bus/pci/windows/pci.c +9 -5
@@ -336,6 +336,7 @@ set_kernel_driver_type(PSP_DEVINFO_DATA device_info_data,static int pci_scan_one (HDEVINFO dev_info,PSP_DEVINFO_DATA device_info_data) { +struct rte_pci_device_internal *pdev =NULL ;struct rte_pci_device *dev =NULL ;int ret =-1 ;char pci_device_info[REGSTR_VAL_MAX_HCID_LEN ];
@@ -370,11 +371,14 @@ pci_scan_one(HDEVINFO dev_info, PSP_DEVINFO_DATA device_info_data) goto end; } - dev =malloc (sizeof (*dev)); -if (dev ==NULL ) + pdev =malloc (sizeof (*pdev)); +if (pdev ==NULL ) { +RTE_LOG (ERR ,EAL ,"Cannot allocate memory for internal pci device\n" ); goto end; + } -memset (dev,0 ,sizeof (*dev)); +memset (pdev,0 ,sizeof (*pdev)); + dev = &pdev->device; dev->device.bus = &rte_pci_bus.bus; dev->addr = addr;
@@ -409,7 +413,7 @@ pci_scan_one(HDEVINFO dev_info, PSP_DEVINFO_DATA device_info_data) dev2->max_vfs = dev->max_vfs;memmove (dev2->mem_resource, dev->mem_resource,sizeof (dev->mem_resource)); -pci_free (dev); +pci_free (pdev); }return 0 ; }
@@ -418,7 +422,7 @@ pci_scan_one(HDEVINFO dev_info, PSP_DEVINFO_DATA device_info_data)return 0 ; end: -pci_free (dev); +pci_free (pdev);return ret; }
Signature · 1 line
2.25.1
PATCH v4 2/4 bus/pci: avoid depending on private value in kernel source
In reply to Miao LiFrom: Chenbo Xia <[email protected]>
The value 40 used in VFIO_GET_REGION_ADDR() is a private value (VFIO_PCI_OFFSET_SHIFT) defined in Linux kernel source [1]. It is not part of VFIO API, and we should not depend on it.
[1] https://github.com/torvalds/linux/blob/v6.2/include/linux/vfio_pci_core.h
--- drivers/bus/pci/linux/pci.c | 4 +- drivers/bus/pci/linux/pci_init.h | 4 +- drivers/bus/pci/linux/pci_vfio.c | 197 +++++++++++++++++++++++-------- drivers/bus/pci/private.h | 9 ++ lib/eal/include/rte_vfio.h | 1 - 5 files changed, 159 insertions(+), 56 deletions(-)
drivers/bus/pci/linux/pci.c +2 -2
@@ -645,7 +645,7 @@ int rte_pci_read_config(const struct rte_pci_device *device,return pci_uio_read_config (intr_handle, buf, len, offset);#ifdef VFIO_PRESENT case RTE_PCI_KDRV_VFIO : -return pci_vfio_read_config (intr_handle, buf, len, offset); +return pci_vfio_read_config (device, buf, len, offset);#endif default : rte_pci_device_name(&device->addr, devname,
@@ -669,7 +669,7 @@ int rte_pci_write_config(const struct rte_pci_device *device,return pci_uio_write_config (intr_handle, buf, len, offset);#ifdef VFIO_PRESENT case RTE_PCI_KDRV_VFIO : -return pci_vfio_write_config (intr_handle, buf, len, offset); +return pci_vfio_write_config (device, buf, len, offset);#endif default : rte_pci_device_name(&device->addr, devname,
drivers/bus/pci/linux/pci_init.h +2 -2
@@ -66,9 +66,9 @@ int pci_uio_ioport_unmap(struct rte_pci_ioport *p);#endif /* access config space */ -int pci_vfio_read_config (const struct rte_intr_handle *intr_handle, +int pci_vfio_read_config (const struct rte_pci_device *dev,void *buf,size_t len,off_t offs); -int pci_vfio_write_config (const struct rte_intr_handle *intr_handle, +int pci_vfio_write_config (const struct rte_pci_device *dev,const void *buf,size_t len,off_t offs); int pci_vfio_ioport_map(struct rte_pci_device *dev, int bar,
drivers/bus/pci/linux/pci_vfio.c +146 -51
371 lines not on this page · raw message
drivers/bus/pci/private.h +9 -0
@@ -13,6 +13,8 @@#include <rte_os_shim.h> #include <rte_pci.h> +#define RTE_MAX_PCI_REGIONS 9 +/* * Convert struct rte_pci_device to struct rte_pci_device_internal */
@@ -42,8 +44,15 @@ extern struct rte_pci_bus rte_pci_bus;struct rte_pci_driver ;struct rte_pci_device ; +struct rte_pci_region { +uint64_t size; +uint64_t offset; +}; +struct rte_pci_device_internal {struct rte_pci_device device; +/* PCI regions provided by e.g. VFIO. */ +struct rte_pci_region region[RTE_MAX_PCI_REGIONS ]; };/**
lib/eal/include/rte_vfio.h +0 -1
@@ -38,7 +38,6 @@ extern "C" {#define VFIO_CONTAINER_PATH "/dev/vfio/vfio"#define VFIO_GROUP_FMT "/dev/vfio/%u"#define VFIO_NOIOMMU_GROUP_FMT "/dev/vfio/noiommu-%u" -#define VFIO_GET_REGION_ADDR (x) ((uint64_t) x << 40ULL)#define VFIO_GET_REGION_IDX (x) (x >> 40)#define VFIO_NOIOMMU_MODE \ "/sys/module/vfio/parameters/enable_unsafe_noiommu_mode"
Signature · 1 line
2.25.1
PATCH v4 3/4 bus/pci: introduce helper for MMIO read and write
In reply to Miao LiFrom: Chenbo Xia <[email protected]>
The MMIO regions may not be mmap-able for VFIO-PCI devices. In this case, the driver should explicitly do read and write to access these regions.
--- doc/guides/rel_notes/release_23_07.rst | 5 +++ drivers/bus/pci/bsd/pci.c | 22 ++++++++++++ drivers/bus/pci/linux/pci.c | 46 ++++++++++++++++++++++++ drivers/bus/pci/linux/pci_init.h | 10 ++++++ drivers/bus/pci/linux/pci_uio.c | 22 ++++++++++++ drivers/bus/pci/linux/pci_vfio.c | 36 +++++++++++++++++++ drivers/bus/pci/rte_bus_pci.h | 48 ++++++++++++++++++++++++++ drivers/bus/pci/version.map | 3 ++ 8 files changed, 192 insertions(+)
doc/guides/rel_notes/release_23_07.rst +5 -0
@@ -55,6 +55,11 @@ New Features Also, make sure to start the actual text at the margin. ======================================================= +* **Added MMIO read and write APIs to PCI bus.** + + Introduced ``rte_pci_mmio_read()`` and ``rte_pci_mmio_write()`` APIs to PCI + bus so that PCI drivers can access PCI memory resources when they are not + mapped to process address space. Removed Items -------------
drivers/bus/pci/bsd/pci.c +22 -0
@@ -489,6 +489,28 @@ int rte_pci_write_config(const struct rte_pci_device *dev,return -1 ; } +/* Read PCI MMIO space. */ +int rte_pci_mmio_read (const struct rte_pci_device *dev,int bar, +void *buf,size_t len,off_t offset) +{ +if (bar >=PCI_MAX_RESOURCE || dev->mem_resource[bar].addr ==NULL || + (uint64_t )offset + len > dev->mem_resource[bar].len) +return -1 ; +memcpy (buf, (uint8_t *)dev->mem_resource[bar].addr + offset, len); +return len; +} + +/* Write PCI MMIO space. */ +int rte_pci_mmio_write (const struct rte_pci_device *dev,int bar, +const void *buf,size_t len,off_t offset) +{ +if (bar >=PCI_MAX_RESOURCE || dev->mem_resource[bar].addr ==NULL || + (uint64_t )offset + len > dev->mem_resource[bar].len) +return -1 ; +memcpy ((uint8_t *)dev->mem_resource[bar].addr + offset, buf, len); +return len; +} +int rte_pci_ioport_map (struct rte_pci_device *dev,int bar,struct rte_pci_ioport *p)
drivers/bus/pci/linux/pci.c +46 -0
52 lines not on this page · raw message
drivers/bus/pci/linux/pci_init.h +10 -0
@@ -37,6 +37,11 @@ int pci_uio_read_config(const struct rte_intr_handle *intr_handle,int pci_uio_write_config (const struct rte_intr_handle *intr_handle,const void *buf,size_t len,off_t offs); +int pci_uio_mmio_read (const struct rte_pci_device *dev,int bar, +void *buf,size_t len,off_t offset); +int pci_uio_mmio_write (const struct rte_pci_device *dev,int bar, +const void *buf,size_t len,off_t offset); +int pci_uio_ioport_map (struct rte_pci_device *dev,int bar,struct rte_pci_ioport *p); void pci_uio_ioport_read(struct rte_pci_ioport *p,
@@ -71,6 +76,11 @@ int pci_vfio_read_config(const struct rte_pci_device *dev,int pci_vfio_write_config (const struct rte_pci_device *dev,const void *buf,size_t len,off_t offs); +int pci_vfio_mmio_read (const struct rte_pci_device *dev,int bar, +void *buf,size_t len,off_t offset); +int pci_vfio_mmio_write (const struct rte_pci_device *dev,int bar, +const void *buf,size_t len,off_t offset); +int pci_vfio_ioport_map (struct rte_pci_device *dev,int bar,struct rte_pci_ioport *p); void pci_vfio_ioport_read(struct rte_pci_ioport *p,
drivers/bus/pci/linux/pci_uio.c +22 -0
@@ -55,6 +55,28 @@ pci_uio_write_config(const struct rte_intr_handle *intr_handle,return pwrite (uio_cfg_fd, buf, len, offset); } +int +pci_uio_mmio_read (const struct rte_pci_device *dev,int bar, +void *buf,size_t len,off_t offset) +{ +if (bar >=PCI_MAX_RESOURCE || dev->mem_resource[bar].addr ==NULL || + (uint64_t )offset + len > dev->mem_resource[bar].len) +return -1 ; +memcpy (buf, (uint8_t *)dev->mem_resource[bar].addr + offset, len); +return len; +} + +int +pci_uio_mmio_write (const struct rte_pci_device *dev,int bar, +const void *buf,size_t len,off_t offset) +{ +if (bar >=PCI_MAX_RESOURCE || dev->mem_resource[bar].addr ==NULL || + (uint64_t )offset + len > dev->mem_resource[bar].len) +return -1 ; +memcpy ((uint8_t *)dev->mem_resource[bar].addr + offset, buf, len); +return len; +} +static int pci_uio_set_bus_master (int dev_fd) {
drivers/bus/pci/linux/pci_vfio.c +36 -0
@@ -1258,6 +1258,42 @@ pci_vfio_ioport_unmap(struct rte_pci_ioport *p)return -1 ; } +int +pci_vfio_mmio_read (const struct rte_pci_device *dev,int bar, +void *buf,size_t len,off_t offs) +{ +uint64_t size, offset; +int fd; + + fd =rte_intr_dev_fd_get (dev->intr_handle); + +if (pci_vfio_get_region (dev, bar, &size, &offset) !=0 ) +return -1 ; + +if ((uint64_t )len + offs > size) +return -1 ; + +return pread64 (fd, buf, len, offset + offs); +} + +int +pci_vfio_mmio_write (const struct rte_pci_device *dev,int bar, +const void *buf,size_t len,off_t offs) +{ +uint64_t size, offset; +int fd; + + fd =rte_intr_dev_fd_get (dev->intr_handle); + +if (pci_vfio_get_region (dev, bar, &size, &offset) !=0 ) +return -1 ; + +if ((uint64_t )len + offs > size) +return -1 ; + +return pwrite64 (fd, buf, len, offset + offs); +} +int pci_vfio_is_enabled (void ) {
drivers/bus/pci/rte_bus_pci.h +48 -0
54 lines not on this page · raw message
drivers/bus/pci/version.map +3 -0
@@ -21,6 +21,9 @@ EXPERIMENTAL { # added in 21.08 rte_pci_set_bus_master; + # added in 23.07 + rte_pci_mmio_read; + rte_pci_mmio_write; }; INTERNAL {
Signature · 1 line
2.25.1
PATCH v4 4/4 bus/pci: add VFIO sparse mmap support
In reply to Miao LiThis patch adds sparse mmap support in PCI bus. Sparse mmap is a capability defined in VFIO which allows multiple mmap areas in one VFIO region.
In this patch, the sparse mmap regions are mapped to one continuous virtual address region that follows device-specific BAR layout. So, driver can still access all mapped sparse mmap regions by using 'bar_base_address + bar_offset'.
--- drivers/bus/pci/linux/pci_vfio.c | 138 +++++++++++++++++++++++++++---- drivers/bus/pci/private.h | 2 + 2 files changed, 122 insertions(+), 18 deletions(-)
drivers/bus/pci/linux/pci_vfio.c +120 -18
@@ -673,6 +673,54 @@ pci_vfio_mmap_bar(int vfio_dev_fd, struct mapped_pci_resource *vfio_res,return 0 ; } +static int +pci_vfio_sparse_mmap_bar (int vfio_dev_fd,struct mapped_pci_resource *vfio_res, +int bar_index,int additional_flags) +{ +struct pci_map *bar = &vfio_res->maps[bar_index]; +struct vfio_region_sparse_mmap_area *sparse; +void *bar_addr; +uint32_t i; + +if (bar->size ==0 ) { +RTE_LOG (DEBUG ,EAL ,"Bar size is 0, skip BAR%d\n" , bar_index); +return 0 ; + } + +/* reserve the address using an inaccessible mapping */ + bar_addr =mmap (bar->addr, bar->size,0 ,MAP_PRIVATE | +MAP_ANONYMOUS | additional_flags,-1 ,0 ); +if (bar_addr !=MAP_FAILED ) { +void *map_addr =NULL ; +for (i =0 ; i < bar->nr_areas; i++) { + sparse = &bar->areas[i]; +if (sparse->size) { +void *addr =RTE_PTR_ADD (bar_addr, (uintptr_t )sparse->offset); + map_addr =pci_map_resource (addr, vfio_dev_fd, + bar->offset + sparse->offset, sparse->size, +RTE_MAP_FORCE_ADDRESS ); +if (map_addr ==NULL ) { +munmap (bar_addr, bar->size); +RTE_LOG (ERR ,EAL ,"Failed to map pci BAR%d\n" , + bar_index); + goto err_map; + } + } + } + }else { +RTE_LOG (ERR ,EAL ,"Failed to create inaccessible mapping for BAR%d\n" , + bar_index); + goto err_map; + } + + bar->addr = bar_addr; +return 0 ; + +err_map: + bar->nr_areas =0 ; +return -1 ; +} +/* * region info may contain capability headers, so we need to keep reallocating * the memory until we match allocated memory size with argsz.
@@ -798,7 +846,7 @@ pci_vfio_map_resource_primary(struct rte_pci_device *dev)char pci_addr[PATH_MAX ] = {0 };int vfio_dev_fd;struct rte_pci_addr *loc = &dev->addr; -int i, ret; +int i, j, ret;struct mapped_pci_resource *vfio_res =NULL ;struct mapped_pci_res_list *vfio_res_list =RTE_TAILQ_CAST (rte_vfio_tailq.head, mapped_pci_res_list);
@@ -875,13 +923,15 @@ pci_vfio_map_resource_primary(struct rte_pci_device *dev)for (i =0 ; i < vfio_res->nb_maps; i++) {void *bar_addr; +struct vfio_info_cap_header *hdr; +struct vfio_region_info_cap_sparse_mmap *sparse; ret =pci_vfio_get_region_info (vfio_dev_fd, ®, i);if (ret <0 ) {RTE_LOG (ERR ,EAL ,"%s cannot get device region info error " "%i (%s)\n" , pci_addr, errno,strerror (errno)); - goto err_vfio_res; + goto err_map; } pdev->region[i].size = reg->size;
@@ -891,7 +941,7 @@ pci_vfio_map_resource_primary(struct rte_pci_device *dev) ret =pci_vfio_is_ioport_bar (dev, vfio_dev_fd, i);if (ret <0 ) {free (reg); - goto err_vfio_res; + goto err_map; }else if (ret) {RTE_LOG (INFO ,EAL ,"Ignore mapping IO port bar(%d)\n" , i);
@@ -920,12 +970,41 @@ pci_vfio_map_resource_primary(struct rte_pci_device *dev) maps[i].size = reg->size; maps[i].path =NULL ;/* vfio doesn't have per-resource paths */ - ret =pci_vfio_mmap_bar (vfio_dev_fd, vfio_res, i,0 ); -if (ret <0 ) { -RTE_LOG (ERR ,EAL ,"%s mapping BAR%i failed: %s\n" , - pci_addr, i,strerror (errno)); -free (reg); - goto err_vfio_res; + hdr =pci_vfio_info_cap (reg,VFIO_REGION_INFO_CAP_SPARSE_MMAP ); + +if (hdr !=NULL ) { + sparse =container_of (hdr, + struct vfio_region_info_cap_sparse_mmap, header); +if (sparse->nr_areas >0 ) { + maps[i].nr_areas = sparse->nr_areas; + maps[i].areas =rte_zmalloc (NULL , +sizeof (*maps[i].areas) * maps[i].nr_areas,0 ); +if (maps[i].areas ==NULL ) { +RTE_LOG (ERR ,EAL , +"Cannot alloc memory for sparse map areas\n" ); + goto err_map; + } +memcpy (maps[i].areas, sparse->areas, +sizeof (*maps[i].areas) * maps[i].nr_areas); + } + } + +if (maps[i].nr_areas >0 ) { + ret =pci_vfio_sparse_mmap_bar (vfio_dev_fd, vfio_res, i,0 ); +if (ret <0 ) { +RTE_LOG (ERR ,EAL ,"%s sparse mapping BAR%i failed: %s\n" , + pci_addr, i,strerror (errno)); +free (reg); + goto err_map; + } + }else { + ret =pci_vfio_mmap_bar (vfio_dev_fd, vfio_res, i,0 ); +if (ret <0 ) { +RTE_LOG (ERR ,EAL ,"%s mapping BAR%i failed: %s\n" , + pci_addr, i,strerror (errno)); +free (reg); + goto err_map; + } } dev->mem_resource[i].addr = maps[i].addr;
@@ -935,19 +1014,26 @@ pci_vfio_map_resource_primary(struct rte_pci_device *dev)if (pci_rte_vfio_setup_device (dev, vfio_dev_fd) <0 ) {RTE_LOG (ERR ,EAL ,"%s setup device failed\n" , pci_addr); - goto err_vfio_res; + goto err_map; }#ifdef HAVE_VFIO_DEV_REQ_INTERFACE if (pci_vfio_enable_notifier (dev, vfio_dev_fd) !=0 ) {RTE_LOG (ERR ,EAL ,"Error setting up notifier!\n" ); - goto err_vfio_res; + goto err_map; }#endif TAILQ_INSERT_TAIL (vfio_res_list, vfio_res, next);return 0 ; +err_map: +for (j =0 ; j < i; j++) { +if (maps[j].addr) +pci_unmap_resource (maps[j].addr, maps[j].size); +if (maps[j].nr_areas >0 ) +rte_free (maps[j].areas); + } err_vfio_res:rte_free (vfio_res); err_vfio_dev_fd:
@@ -963,7 +1049,7 @@ pci_vfio_map_resource_secondary(struct rte_pci_device *dev)char pci_addr[PATH_MAX ] = {0 };int vfio_dev_fd;struct rte_pci_addr *loc = &dev->addr; -int i, ret; +int i, j, ret;struct mapped_pci_resource *vfio_res =NULL ;struct mapped_pci_res_list *vfio_res_list =RTE_TAILQ_CAST (rte_vfio_tailq.head, mapped_pci_res_list);
@@ -1008,11 +1094,20 @@ pci_vfio_map_resource_secondary(struct rte_pci_device *dev) maps = vfio_res->maps;for (i =0 ; i < vfio_res->nb_maps; i++) { - ret =pci_vfio_mmap_bar (vfio_dev_fd, vfio_res, i,MAP_FIXED ); -if (ret <0 ) { -RTE_LOG (ERR ,EAL ,"%s mapping BAR%i failed: %s\n" , - pci_addr, i,strerror (errno)); - goto err_vfio_dev_fd; +if (maps[i].nr_areas >0 ) { + ret =pci_vfio_sparse_mmap_bar (vfio_dev_fd, vfio_res, i,MAP_FIXED ); +if (ret <0 ) { +RTE_LOG (ERR ,EAL ,"%s sparse mapping BAR%i failed: %s\n" , + pci_addr, i,strerror (errno)); + goto err_vfio_dev_fd; + } + }else { + ret =pci_vfio_mmap_bar (vfio_dev_fd, vfio_res, i,MAP_FIXED ); +if (ret <0 ) { +RTE_LOG (ERR ,EAL ,"%s mapping BAR%i failed: %s\n" , + pci_addr, i,strerror (errno)); + goto err_vfio_dev_fd; + } } dev->mem_resource[i].addr = maps[i].addr;
@@ -1028,6 +1123,10 @@ pci_vfio_map_resource_secondary(struct rte_pci_device *dev)return 0 ; err_vfio_dev_fd: +for (j =0 ; j < i; j++) { +if (maps[j].addr) +pci_unmap_resource (maps[j].addr, maps[j].size); + }rte_vfio_release_device (rte_pci_get_sysfs_path (), pci_addr, vfio_dev_fd);return -1 ;
@@ -1062,7 +1161,7 @@ find_and_unmap_vfio_resource(struct mapped_pci_res_list *vfio_res_list,break ; } -if (vfio_res ==NULL ) +if (vfio_res ==NULL )return vfio_res; RTE_LOG(INFO, EAL, "Releasing PCI mapped resource for %s\n",
@@ -1080,6 +1179,9 @@ find_and_unmap_vfio_resource(struct mapped_pci_res_list *vfio_res_list, pci_addr, maps[i].addr);pci_unmap_resource (maps[i].addr, maps[i].size); } + +if (maps[i].nr_areas >0 ) +rte_free (maps[i].areas); }return vfio_res;
drivers/bus/pci/private.h +2 -0
@@ -121,6 +121,8 @@ struct pci_map {uint64_t offset;uint64_t size;uint64_t phaddr; +uint32_t nr_areas; +struct vfio_region_sparse_mmap_area *areas; };struct pci_msix_table {
Signature · 1 line
2.25.1
Thomas Monjalon
Applied31/05/2023 07:37, Miao Li:
Quoted 30 lines
This series introduces a VFIO standard capability, called sparse mmap to PCI bus. In linux kernel, it's defined as
VFIO_REGION_INFO_CAP_SPARSE_MMAP. Sparse mmap means instead of mmap whole BAR region into DPDK process, only mmap part of the BAR region after getting sparse mmap information from kernel. For the rest of BAR region that is not mmap-ed, DPDK process can use pread/pwrite system calls to access. Sparse mmap is useful when kernel does not want userspace to mmap whole BAR region, or kernel wants to control over access to specific BAR region. Vendors can choose to enable this feature or not for their devices in their specific kernel modules.In this patchset:
Patch 1-3 is mainly for introducing BAR access APIs so that driver could use them to access specific BAR using pread/pwrite system calls when part of the BAR is not mmap-able. Patch 4 adds the VFIO sparse mmap support finally.
v4:
1. add sparse mmap information allocation and release
2. add release note for BAR access APIsv3:
fix variable 'pdev' and 'info' uninitialized errorv2:
1. add PCI device internal structure inbus/pci/windows/pci.c
2. fix parameter type error
Quoted 7 lines
Chenbo Xia (3):
bus/pci: introduce an internal representation of PCI device bus/pci: avoid depending on private value in kernel source bus/pci: introduce helper for MMIO read and writeMiao Li (1):
bus/pci: add VFIO sparse mmap support
Applied, thanks.
Is there some drivers which may reuse the new MMIO helpers?
Hello,
This patchseries might have introduced a bug for building DPDK on windows. It failed on windows build when it went through our CI last week, and I am seeing other patch series fail on the windows build now that it is merged into main.
Tomorrow morning, I will check our windows system used for CI to verify it is still valid for building DPDK in terms of clang version, linker etc. But it seems fine from a quick glance right now. I will update if/when I learn more.
Thanks,
Patrick
Signature · 9 lines
Patrick Robb Technical Service Manager UNH InterOperability Laboratory 21 Madbury Rd, Suite 100, Durham, NH 03824 www.iol.unh.edu
Hi Thomas,
Quoted 45 lines
-----Original Message-----
From: Thomas Monjalon <[email protected]>
Sent: Thursday, June 8, 2023 12:31 AM
To: Li, Miao <[email protected]>
Cc: [email protected]; [email protected]; [email protected]; [email protected]; Xia, Chenbo <[email protected]>; Cao, Yahui <[email protected]>
Subject: Re: [PATCH v4 0/4] Support VFIO sparse mmap in PCI bus31/05/2023 07:37, Miao Li:
Quoted 35 lines
This series introduces a VFIO standard capability, called sparse mmap to PCI bus. In linux kernel, it's defined as
VFIO_REGION_INFO_CAP_SPARSE_MMAP. Sparse mmap means instead of mmap whole BAR region into DPDK process, only mmap part of the BAR region after getting sparse mmap information from kernel. For the rest of BAR region that is not mmap-ed, DPDK process can use pread/pwrite system calls to access. Sparse mmap is useful when kernel does not want userspace to mmap whole BAR region, or kernel wants to control over access to specific BAR region. Vendors can choose to enable this feature or not for their devices in their specific kernel modules.In this patchset:
Patch 1-3 is mainly for introducing BAR access APIs so that driver could use them to access specific BAR using pread/pwrite system calls when part of the BAR is not mmap-able. Patch 4 adds the VFIO sparse mmap support finally.
v4:
1. add sparse mmap information allocation and release
2. add release note for BAR access APIsv3:
fix variable 'pdev' and 'info' uninitialized errorv2:
1. add PCI device internal structure inbus/pci/windows/pci.c
2. fix parameter type errorChenbo Xia (3):
bus/pci: introduce an internal representation of PCI device bus/pci: avoid depending on private value in kernel source bus/pci: introduce helper for MMIO read and write
Quoted 6 lines
Quoted 2 lines
Miao Li (1):
bus/pci: add VFIO sparse mmap supportApplied, thanks.
Is there some drivers which may reuse the new MMIO helpers?
Yes, we will send patches to let Intel drivers use them soon. Other drivers could start to use when they want to support sparse mmap.
Thanks,
Chenbo
Quoted 1 line
Hi Patrick,
Oops.. Seems weird as patchwork does not report anything. Please reach out to me when you get some information about why it’s failing. I could fix it ASAP.
Thanks,
Chenbo
From: Patrick Robb <[email protected]>
Sent: Thursday, June 8, 2023 8:29 AM
To: Thomas Monjalon <[email protected]>
Cc: Li, Miao <[email protected]>; [email protected]; [email protected]; [email protected]; [email protected]; Xia, Chenbo <[email protected]>; Cao, Yahui <[email protected]>
Subject: Re: [PATCH v4 0/4] Support VFIO sparse mmap in PCI bus
Hello,
This patchseries might have introduced a bug for building DPDK on windows. It failed on windows build when it went through our CI last week, and I am seeing other patch series fail on the windows build now that it is merged into main.
Tomorrow morning, I will check our windows system used for CI to verify it is still valid for building DPDK in terms of clang version, linker etc. But it seems fine from a quick glance right now. I will update if/when I learn more.
Thanks,
Patrick
--
Patrick Robb
Technical Service Manager
UNH InterOperability Laboratory
21 Madbury Rd, Suite 100, Durham, NH 03824
www.iol.unh.edu<http://www.iol.unh.edu/>
[Image removed by sender.]
Quoted 9 lines
-----Original Message-----
From: Miao Li <[email protected]>
Sent: Wednesday, May 31, 2023 8:38 AM
To: [email protected]
Cc: [email protected]; NBU-Contact-Thomas Monjalon (EXTERNAL) <[email protected]>; [email protected]; [email protected]; [email protected]; [email protected]
Subject: [PATCH v4 0/4] Support VFIO sparse mmap in PCI bus
Quoted 12 lines
This series introduces a VFIO standard capability, called sparse mmap to PCI bus. In linux kernel, it's defined as
VFIO_REGION_INFO_CAP_SPARSE_MMAP. Sparse mmap means instead of mmap whole BAR region into DPDK process, only mmap part of the BAR region after getting sparse mmap information from kernel. For the rest of BAR region that is not mmap-ed, DPDK process can use pread/pwrite system calls to access. Sparse mmap is useful when kernel does not want userspace to mmap whole BAR region, or kernel wants to control over access to specific BAR region. Vendors can choose to enable this feature or not for their devices in their specific kernel modules.
Hello,
I see the build failure Patrick reported as well and can confirm it's caused by 095cf6e68b28 ("bus/pci: introduce MMIO read/write"). Bugzilla ticket: https://bugs.dpdk.org/show_bug.cgi?id=1245
Regards,
Ali
Quoted 32 lines
-----Original Message-----
From: Ali Alnubani <[email protected]>
Sent: Thursday, June 8, 2023 2:43 PM
To: Li, Miao <[email protected]>; [email protected]; NBU-Contact-Thomas Monjalon (EXTERNAL) <[email protected]>
Cc: [email protected]; [email protected]; [email protected]; Xia, Chenbo <[email protected]>; Cao, Yahui <[email protected]>; Patrick Robb <[email protected]>
Subject: RE: [PATCH v4 0/4] Support VFIO sparse mmap in PCI busQuoted 9 lines
-----Original Message-----
From: Miao Li <[email protected]>
Sent: Wednesday, May 31, 2023 8:38 AM
To: [email protected]
Cc: [email protected]; NBU-Contact-Thomas Monjalon (EXTERNAL) <[email protected]>; [email protected]; [email protected]; [email protected]; [email protected]
Subject: [PATCH v4 0/4] Support VFIO sparse mmap in PCI busQuoted 12 lines
This series introduces a VFIO standard capability, called sparse mmap to PCI bus. In linux kernel, it's defined as
VFIO_REGION_INFO_CAP_SPARSE_MMAP. Sparse mmap means instead of mmap whole BAR region into DPDK process, only mmap part of the BAR region after getting sparse mmap information from kernel. For the rest of BAR region that is not mmap-ed, DPDK process can use pread/pwrite system calls to access. Sparse mmap is useful when kernel does not want userspace to mmap whole BAR region, or kernel wants to control over access to specific BAR region. Vendors can choose to enable this feature or not for their devices in their specific kernel modules.
Quoted 5 lines
Hello,
I see the build failure Patrick reported as well and can confirm it's caused by
095cf6e68b28("bus/pci: introduce MMIO read/write"). Bugzilla ticket: https://bugs.dpdk.org/show_bug.cgi?id=1245
Thanks Ali. I just read the bz and understand what's missing. I will send a patch today.
But since last time CI is not reporting the error, this time how could I make sure the fix will perfectly work?
Regards,
Chenbo
Quoted 3 lines
Regards,
Ali
Hello Chenbo, Patrick,
Quoted 19 lines · Chenbo
Quoted 13 lines
Quoted 12 lines
This series introduces a VFIO standard capability, called sparse mmap to PCI bus. In linux kernel, it's defined as
VFIO_REGION_INFO_CAP_SPARSE_MMAP. Sparse mmap means instead of mmap whole BAR region into DPDK process, only mmap part of the BAR region after getting sparse mmap information from kernel. For the rest of BAR region that is not mmap-ed, DPDK process can use pread/pwrite system calls to access. Sparse mmap is useful when kernel does not want userspace to mmap whole BAR region, or kernel wants to control over access to specific BAR region. Vendors can choose to enable this feature or not for their devices in their specific kernel modules.Quoted 5 lines
Hello,
I see the build failure Patrick reported as well and can confirm it's caused by
095cf6e68b28("bus/pci: introduce MMIO read/write"). Bugzilla ticket: https://bugs.dpdk.org/show_bug.cgi?id=1245
Quoted 5 lines
Thanks Ali. I just read the bz and understand what's missing. I will send a patch today.
But since last time CI is not reporting the error, this time how could I make sure the fix will perfectly work?
Chenbo,
In theory, this error should have been reported so go ahead and post your fix.
Patrick,
This missing report could be a mail delivery issue (I can see the test ran at UNH).
I see no trace in test-report ml.
Can you look at the reason?
Thanks.
Signature · 1 line
David Marchand
Quoted 42 lines · David Marchand
Hello Chenbo, Patrick,
Quoted 15 lines · Chenbo
Quoted 15 lines
Quoted 12 lines
This series introduces a VFIO standard capability, called sparse mmap to PCI bus. In linux kernel, it's defined as
VFIO_REGION_INFO_CAP_SPARSE_MMAP. Sparse mmap means instead of mmap whole BAR region into DPDK process, only mmap part of the BAR region after getting sparse mmap information from kernel. For the rest of BAR region that is not mmap-ed, DPDK process can use pread/pwrite system calls to access. Sparse mmap is useful when kernel does not want userspace to mmap whole BAR region, or kernel wants to control over access to specific BAR region. Vendors can choose to enable this feature or not for their devices in their specific kernel modules.Hello,
Quoted 8 lines
Quoted 3 lines
I see the build failure Patrick reported as well and can confirm it's caused by
095cf6e68b28("bus/pci: introduce MMIO read/write"). Bugzilla ticket: https://bugs.dpdk.org/show_bug.cgi?id=1245Thanks Ali. I just read the bz and understand what's missing. I will send a patch today.
But since last time CI is not reporting the error, this time how could I
make
Quoted 1 line
sure the fix will perfectly work?
Chenbo,
In theory, this error should have been reported so go ahead and post your fix.
Patrick,
This missing report could be a mail delivery issue (I can see the test ran at UNH).
I see no trace in test-report ml.
Can you look at the reason?
Quoted 7 lines
Thanks.
--
David Marchand
I see where the issue is in our reporting process for the windows results and I will resolve it after the CI meeting this morning. Sorry everyone for the oversight.
Signature · 9 lines
Patrick Robb Technical Service Manager UNH InterOperability Laboratory 21 Madbury Rd, Suite 100, Durham, NH 03824 www.iol.unh.edu