vfio: selftest: Add NVIDIA GPU Falcon DMA test driver
Subsystems (4)
v13 cover vfio: selftest: Add NVIDIA GPU Falcon DMA test driver
Patch 1:
Add a chunking loop to vfio_pci_driver_memcpy() so that it handles arbitrarily sized memcpy requests by breaking them into max_memcpy_size-sized chunks. This allows tests to request any memcpy size. Update the test to use a size of 4x max_memcpy_size to exercise the chunking logic.
Patch 2:
Add vfio_pci_cmd_set()/vfio_pci_cmd_clear() macros for PCI_COMMAND operations.
Patch 3:
Allow drivers that cannot trigger MSI interrupts to leave the send_msi callback NULL. Tests check ops->send_msi directly and gracefully skip MSI-related operations when it is absent.
Patch 4:
Introduce the nv_falcon plugin driver, which extracts and adapts relevant functionality from NVIDIA's gpu-admin-tools project [1] and integrates it into the VFIO selftest framework. As a result, any system equipped with a PCIe slot and a supported NVIDIA GPU can now run VFIO DMA selftests using commonly available hardware.
Falcons are general-purpose microcontrollers present on NVIDIA GPUs that can perform DMA operations between system memory and device memory.
[1] https://github.com/NVIDIA/gpu-admin-tools
Note on version numbering: v1 through v9 were internal review iterations that were mistakenly carried over to the upstream submission. Apologies for the confusion, the internal changelog has been dropped.
Changes for lucky v13:
- Respin from Alex. No fundamental changes here, so keeping Rubin's authorships, sign-offs, and cover letter.
- Throughout: updated subject for consistency (David)
- Patch 1 & 3: No change, R-b collected
- Patch 2: Macros converted to static inlines (David)
- Patch 4:
- Formats %ll -> %l (David)
- Converted size_to_dma_encoding() size testing/manipulation to asserts (David)
- Re-introduced iterative transfer for non-power-of-2 memcpy (David) Testing this revealed we need some gymnastics to also avoid transfers crossing block boundaries, so we use the largest power- of-2 taking into account the transfer size as well as source and destination IOVA addresses.
- Function renaming for consistency (David).
Rubin Du (4):
vfio: selftests: Add memcpy chunking to vfio_pci_driver_memcpy()
vfio: selftests: Add generic PCI command register helpers
vfio: selftests: Allow drivers without send_msi() support
vfio: selftests: Add NVIDIA Falcon driver for DMA testing
.../selftests/vfio/lib/drivers/nv_falcon/hw.h | 350 ++++++++ .../vfio/lib/drivers/nv_falcon/nv_falcon.c | 789 ++++++++++++++++++ .../lib/include/libvfio/vfio_pci_device.h | 14 + tools/testing/selftests/vfio/lib/libvfio.mk | 2 + .../selftests/vfio/lib/vfio_pci_driver.c | 21 +- .../selftests/vfio/vfio_pci_driver_test.c | 57 +- 6 files changed, 1210 insertions(+), 23 deletions(-) create mode 100644 tools/testing/selftests/vfio/lib/drivers/nv_falcon/hw.h create mode 100644 tools/testing/selftests/vfio/lib/drivers/nv_falcon/nv_falcon.c
Signature · 1 line
2.51.0
PATCH v13 1/4 vfio: selftests: Add memcpy chunking to vfio_pci_driver_memcpy()
From: Rubin Du <[email protected]>
Add a chunking loop to vfio_pci_driver_memcpy() so that it breaks up large memcpy requests into max_memcpy_size-sized chunks. This allows callers to request any size without worrying about per-driver limits. The memcpy_start()/memcpy_wait() semantics are unchanged.
Update the test to use 4x max_memcpy_size so it exercises the new chunking path (4 iterations) while keeping execution fast for drivers with small DMA transfer sizes.
--- .../selftests/vfio/lib/vfio_pci_driver.c | 18 ++++++++++++++++-- .../selftests/vfio/vfio_pci_driver_test.c | 18 ++++++++++-------- 2 files changed, 26 insertions(+), 10 deletions(-)
tools/testing/selftests/vfio/lib/vfio_pci_driver.c +16 -2
@@ -106,7 +106,21 @@ int vfio_pci_driver_memcpy_wait(struct vfio_pci_device *device)int vfio_pci_driver_memcpy (struct vfio_pci_device *device,iova_t src,iova_t dst,u64 size) { -vfio_pci_driver_memcpy_start (device, src, dst, size,1 ); +struct vfio_pci_driver *driver = &device->driver; +u64 offset =0 ; + +while (offset < size) { +u64 chunk =min (size - offset, driver->max_memcpy_size); +int ret; + +vfio_pci_driver_memcpy_start (device, src + offset, + dst + offset, chunk,1 ); + ret =vfio_pci_driver_memcpy_wait (device); +if (ret) +return ret; + + offset += chunk; + } -return vfio_pci_driver_memcpy_wait (device); +return 0 ; }
tools/testing/selftests/vfio/vfio_pci_driver_test.c +10 -8
@@ -89,12 +89,12 @@ FIXTURE_SETUP(vfio_pci_driver_test) self->msi_fd = self->device->msi_eventfds[driver->msi];/* -* Use the maximum size supported by the device for memcpy operations, -* slimmed down to fit into the memcpy region (divided by 2 so src and -* dst regions do not overlap). +* Use 4x the driver's max_memcpy_size to exercise the chunking +* logic in vfio_pci_driver_memcpy(). Cap to half the memcpy +* region so src and dst do not overlap. */ - self->size = self->device->driver.max_memcpy_size; - self->size =min (self->size, self->memcpy_region.size /2 ); + self->size =min_t (u64, driver->max_memcpy_size *4 , + self->memcpy_region.size /2 ); self->src = self->memcpy_region.vaddr; self->dst = self->src + self->size;
@@ -211,6 +211,7 @@ TEST_F_TIMEOUT(vfio_pci_driver_test, memcpy_storm, 60) {struct vfio_pci_driver *driver = &self->device->driver;u64 total_size; +u64 size;u64 count;fcntl_set_nonblock (self->msi_fd);
@@ -221,13 +222,14 @@ TEST_F_TIMEOUT(vfio_pci_driver_test, memcpy_storm, 60)* will take too long. */ total_size =250UL *SZ_1G ; - count =min (total_size / self->size, driver->max_memcpy_count); + size =min (driver->max_memcpy_size, self->memcpy_region.size /2 ); + count =min (total_size / size, driver->max_memcpy_count); -printf ("Kicking off %lu memcpys of size 0x%lx\n" , count, self->size); +printf ("Kicking off %lu memcpys of size 0x%lx\n" , count, size);vfio_pci_driver_memcpy_start (self->device, self->src_iova, self->dst_iova, - self->size, count); + size, count);ASSERT_EQ (0 ,vfio_pci_driver_memcpy_wait (self->device));ASSERT_NO_MSI (self->msi_fd);
Signature · 1 line
2.51.0
PATCH v13 2/4 vfio: selftests: Add generic PCI command register helpers
In reply to Alex WilliamsonFrom: Rubin Du <[email protected]>
Add vfio_pci_cmd_set()/vfio_pci_cmd_clear() read-modify-write macros for PCI_COMMAND in vfio_pci_device.h.
--- .../vfio/lib/include/libvfio/vfio_pci_device.h | 14 ++++++++++++++ 1 file changed, 14 insertions(+)
tools/testing/selftests/vfio/lib/include/libvfio/vfio_pci_device.h +14 -0
@@ -65,6 +65,20 @@ void vfio_pci_config_access(struct vfio_pci_device *device, bool write,#define vfio_pci_config_writew (_d, _o, _v) vfio_pci_config_write(_d, _o, _v, u16)#define vfio_pci_config_writel (_d, _o, _v) vfio_pci_config_write(_d, _o, _v, u32) +static inline void vfio_pci_cmd_set (struct vfio_pci_device *device,u16 bits) +{ +u16 cmd =vfio_pci_config_readw (device,PCI_COMMAND ); + +vfio_pci_config_writew (device,PCI_COMMAND , cmd | bits); +} + +static inline void vfio_pci_cmd_clear (struct vfio_pci_device *device,u16 bits) +{ +u16 cmd =vfio_pci_config_readw (device,PCI_COMMAND ); + +vfio_pci_config_writew (device,PCI_COMMAND , cmd & ~bits); +} +void vfio_pci_irq_enable (struct vfio_pci_device *device,u32 index,u32 vector,int count);void vfio_pci_irq_disable (struct vfio_pci_device *device,u32 index);
Signature · 1 line
2.51.0
Re: [PATCH v13 2/4] vfio: selftests: Add generic PCI command register helpers
Quoted 7 lines · Alex Williamson
From: Rubin Du <[email protected]>
Add vfio_pci_cmd_set()/vfio_pci_cmd_clear() read-modify-write macros for
PCI_COMMANDinvfio_pci_device.h.Signed-off-by Rubin Du <[email protected]>Signed-off-by Alex Williamson <[email protected]>
PATCH v13 3/4 vfio: selftests: Allow drivers without send_msi() support
In reply to Alex WilliamsonFrom: Rubin Du <[email protected]>
Allow drivers that cannot trigger MSI interrupts to leave the send_msi callback NULL. Add an fcntl_set_msi_nonblock() wrapper that only sets nonblocking mode when send_msi is available, and update ASSERT_NO_MSI() to skip when the driver lacks MSI support. The send_msi test SKIPs and mix_and_match skips the MSI portion per iteration.
--- .../selftests/vfio/vfio_pci_driver_test.c | 39 ++++++++++++------- 1 file changed, 26 insertions(+), 13 deletions(-)
tools/testing/selftests/vfio/vfio_pci_driver_test.c +26 -13
@@ -11,11 +11,18 @@static const char *device_bdf; -#define ASSERT_NO_MSI (_eventfd) do { \ - u64 __value; \ - \ - ASSERT_EQ(-1, read(_eventfd, &__value, 8)); \ - ASSERT_EQ(EAGAIN, errno); \ +#define fcntl_set_msi_nonblock (_self) do { \ + if (_self->device->driver.ops->send_msi) \ + fcntl_set_nonblock(_self->msi_fd); \ +} while (0) + +#define ASSERT_NO_MSI (_self) do { \ + u64 __value; \ + \ + if (!_self->device->driver.ops->send_msi) \ + break; \ + ASSERT_EQ(-1, read(_self->msi_fd, &__value, 8)); \ + ASSERT_EQ(EAGAIN, errno); \ } while (0) static void region_setup(struct iommu *iommu,
@@ -129,7 +136,7 @@ TEST_F(vfio_pci_driver_test, init_remove) TEST_F(vfio_pci_driver_test, memcpy_success) { -fcntl_set_nonblock (self->msi_fd); +fcntl_set_msi_nonblock (self);memset (self->src,'x' , self->size);memset (self->dst,'y' , self->size);
@@ -140,12 +147,12 @@ TEST_F(vfio_pci_driver_test, memcpy_success) self->size));ASSERT_EQ (0 ,memcmp (self->src, self->dst, self->size)); -ASSERT_NO_MSI (self->msi_fd); +ASSERT_NO_MSI (self); } TEST_F(vfio_pci_driver_test, memcpy_from_unmapped_iova) { -fcntl_set_nonblock (self->msi_fd); +fcntl_set_msi_nonblock (self);/* * Ignore the return value since not all devices will detect and report
@@ -154,12 +161,12 @@ TEST_F(vfio_pci_driver_test, memcpy_from_unmapped_iova)vfio_pci_driver_memcpy (self->device, self->unmapped_iova, self->dst_iova, self->size); -ASSERT_NO_MSI (self->msi_fd); +ASSERT_NO_MSI (self); } TEST_F(vfio_pci_driver_test, memcpy_to_unmapped_iova) { -fcntl_set_nonblock (self->msi_fd); +fcntl_set_msi_nonblock (self);/* * Ignore the return value since not all devices will detect and report
@@ -168,13 +175,16 @@ TEST_F(vfio_pci_driver_test, memcpy_to_unmapped_iova)vfio_pci_driver_memcpy (self->device, self->src_iova, self->unmapped_iova, self->size); -ASSERT_NO_MSI (self->msi_fd); +ASSERT_NO_MSI (self); } TEST_F(vfio_pci_driver_test, send_msi) {u64 value; +if (!self->device->driver.ops->send_msi) +SKIP (return,"Driver does not support send_msi()\n" ); +vfio_pci_driver_send_msi (self->device);ASSERT_EQ (8 ,read (self->msi_fd, &value,8 ));ASSERT_EQ (1 , value);
@@ -201,6 +211,9 @@ TEST_F(vfio_pci_driver_test, mix_and_match) self->dst_iova, self->size); +if (!self->device->driver.ops->send_msi) +continue ; +vfio_pci_driver_send_msi (self->device);ASSERT_EQ (8 ,read (self->msi_fd, &value,8 ));ASSERT_EQ (1 , value);
@@ -214,7 +227,7 @@ TEST_F_TIMEOUT(vfio_pci_driver_test, memcpy_storm, 60)u64 size;u64 count; -fcntl_set_nonblock (self->msi_fd); +fcntl_set_msi_nonblock (self);/* * Perform up to 250GiB worth of DMA reads and writes across several
@@ -232,7 +245,7 @@ TEST_F_TIMEOUT(vfio_pci_driver_test, memcpy_storm, 60) size, count);ASSERT_EQ (0 ,vfio_pci_driver_memcpy_wait (self->device)); -ASSERT_NO_MSI (self->msi_fd); +ASSERT_NO_MSI (self); }static bool device_has_selftests_driver (const char *bdf)
Signature · 1 line
2.51.0
PATCH v13 4/4 vfio: selftests: Add NVIDIA Falcon driver for DMA testing
In reply to Alex WilliamsonFrom: Rubin Du <[email protected]>
Add a new VFIO PCI driver for NVIDIA GPUs that enables DMA testing via the Falcon (Fast Logic Controller) microcontrollers. This driver extracts and adapts the DMA test functionality from NVIDIA's gpu-admin-tools project and integrates it into the existing VFIO selftest framework.
Falcons are general-purpose microcontrollers present on NVIDIA GPUs that can perform DMA operations between system memory and device memory. By leveraging Falcon DMA, this driver allows NVIDIA GPUs to be tested alongside Intel IOAT and DSA devices using the same selftest infrastructure.
The driver is named 'nv_falcon' to reflect that it specifically controls the Falcon microcontrollers for DMA operations, rather than exposing general GPU functionality.
Reference implementation:
https://github.com/NVIDIA/gpu-admin-tools
--- .../selftests/vfio/lib/drivers/nv_falcon/hw.h | 350 ++++++++ .../vfio/lib/drivers/nv_falcon/nv_falcon.c | 789 ++++++++++++++++++ tools/testing/selftests/vfio/lib/libvfio.mk | 2 + .../selftests/vfio/lib/vfio_pci_driver.c | 3 + 4 files changed, 1144 insertions(+) create mode 100644 tools/testing/selftests/vfio/lib/drivers/nv_falcon/hw.h create mode 100644 tools/testing/selftests/vfio/lib/drivers/nv_falcon/nv_falcon.c
tools/testing/selftests/vfio/lib/drivers/nv_falcon/hw.h new file +350 -0
@@ -0,0 +1,350 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ +/* +* Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. All rights reserved. +*/ +#ifndef _NV_FALCON_HW_H_ +#define _NV_FALCON_HW_H_ + +/* PMC (Power Management Controller) Registers */ +#define NV_PMC_BOOT_0 0x00000000 +#define NV_PMC_ENABLE 0x00000200 +#define NV_PMC_ENABLE_PWR 0x00002000 +#define NV_PMC_ENABLE_HUB 0x20000000 + +/* Falcon Base Pages for Different Engines */ +#define NV_PPWR_FALCON_BASE 0x10a000 +#define NV_PGSP_FALCON_BASE 0x110000 + +/* Falcon Common Register Offsets (relative to base_page) */ +#define NV_FALCON_DMACTL_OFFSET 0x010c +#define NV_FALCON_ENGINE_RESET_OFFSET 0x03c0 + +/* DMEM Control Register Flags */ +#define NV_PPWR_FALCON_DMEMC_AINCR_TRUE 0x01000000 +#define NV_PPWR_FALCON_DMEMC_AINCW_TRUE 0x02000000 + +/* Falcon DMEM port offsets (for port 0) */ +#define NV_FALCON_DMEMC_OFFSET 0x1c0 +#define NV_FALCON_DMEMD_OFFSET 0x1c4 + +/* DMA Register Offsets (relative to base_page) */ +#define NV_FALCON_DMA_ADDR_LOW_OFFSET 0x110 +#define NV_FALCON_DMA_MEM_OFFSET 0x114 +#define NV_FALCON_DMA_CMD_OFFSET 0x118 +#define NV_FALCON_DMA_BLOCK_OFFSET 0x11c +#define NV_FALCON_DMA_ADDR_HIGH_OFFSET 0x128 + +/* DMA Global Address Top Bits Register */ +#define NV_GPU_DMA_ADDR_TOP_BITS_REG 0x100f04 + +/* DMA Command Register Bit Definitions */ +#define NV_FALCON_DMA_CMD_WRITE_BIT 0x20 +#define NV_FALCON_DMA_CMD_SIZE_SHIFT 8 +#define NV_FALCON_DMA_CMD_DONE_BIT 0x2 + +/* +* Falcon DMA is synchronous, so a transfer size and count larger than +* its per-operation maximum adds no value. +*/ + +/* DMA block size and alignment */ +#define NV_FALCON_DMA_MIN_TRANSFER_SIZE 4 +#define NV_FALCON_DMA_MAX_TRANSFER_SIZE 256 +#define NV_FALCON_DMA_BLOCK_SIZE 256 +#define NV_FALCON_DMA_MAX_TRANSFER_COUNT 1 + +/* DMACTL register bits */ +#define NV_FALCON_DMACTL_DMEM_SCRUBBING 0x1 +#define NV_FALCON_DMACTL_READY_MASK 0x6 + +/* Falcon Core Selection Register */ +#define NV_FALCON_CORE_SELECT_OFFSET 0x1668 +#define NV_FALCON_CORE_SELECT_MASK 0x30 + +/* Falcon mailbox register (for Ada+ reset check) */ +#define NV_FALCON_MAILBOX_TEST_OFFSET 0x40c +#define NV_FALCON_MAILBOX_RESET_MAGIC 0xbadf5620 + +/* Falcon Message Queue Register Offsets (relative to base_page) */ +#define NV_FALCON_QUEUE_HEAD_BASE_OFFSET 0x2c00 +#define NV_FALCON_QUEUE_TAIL_BASE_OFFSET 0x2c04 +#define NV_FALCON_QUEUE_STRIDE 0x8 +#define NV_FALCON_MSG_QUEUE_HEAD_BASE_OFFSET 0x2c80 +#define NV_FALCON_MSG_QUEUE_TAIL_BASE_OFFSET 0x2c84 + +/* FSP Falcon Base Pages */ +#define NV_FSP_FALCON_BASE 0x8f0100 +/* base_page = cpuctl & ~0xfff */ +#define NV_FSP_FALCON_BASE_PAGE 0x8f0000 +#define NV_FSP_EMEM_BASE 0x8f2000 + +/* FSP EMEM Port Offsets (relative to FSP EMEM base) */ +#define NV_FSP_EMEMC_OFFSET 0xac0 +#define NV_FSP_EMEMD_OFFSET 0xac4 +#define NV_FSP_EMEM_PORT_STRIDE 0x8 + +/* EMEM Control Register Flags (same as DMEM) */ +#define NV_FALCON_EMEMC_AINCR 0x01000000 +#define NV_FALCON_EMEMC_AINCW 0x02000000 + +/* FSP RPC channel configuration */ +#define NV_FSP_RPC_CHANNEL_SIZE 1024 +#define NV_FSP_RPC_MAX_PACKET_SIZE 1024 +#define NV_FSP_RPC_CHANNEL_HOPPER 2 +#define NV_FSP_RPC_EMEM_BASE \ + (NV_FSP_RPC_CHANNEL_HOPPER * NV_FSP_RPC_CHANNEL_SIZE) + +/* FSP EMEM port 2 registers (pre-computed for Hopper channel 2) */ +#define NV_FSP_EMEM_PORT2_CTRL (NV_FSP_EMEM_BASE + NV_FSP_EMEMC_OFFSET + \ + NV_FSP_RPC_CHANNEL_HOPPER * NV_FSP_EMEM_PORT_STRIDE) +#define NV_FSP_EMEM_PORT2_DATA (NV_FSP_EMEM_BASE + NV_FSP_EMEMD_OFFSET + \ + NV_FSP_RPC_CHANNEL_HOPPER * NV_FSP_EMEM_PORT_STRIDE) + +/* FSP queue register offsets (pre-computed for Hopper channel 2) */ +#define NV_FSP_QUEUE_HEAD \ + (NV_FSP_FALCON_BASE_PAGE + NV_FALCON_QUEUE_HEAD_BASE_OFFSET + \ + NV_FSP_RPC_CHANNEL_HOPPER * NV_FALCON_QUEUE_STRIDE) +#define NV_FSP_QUEUE_TAIL \ + (NV_FSP_FALCON_BASE_PAGE + NV_FALCON_QUEUE_TAIL_BASE_OFFSET + \ + NV_FSP_RPC_CHANNEL_HOPPER * NV_FALCON_QUEUE_STRIDE) +#define NV_FSP_MSG_QUEUE_HEAD \ + (NV_FSP_FALCON_BASE_PAGE + NV_FALCON_MSG_QUEUE_HEAD_BASE_OFFSET + \ + NV_FSP_RPC_CHANNEL_HOPPER * NV_FALCON_QUEUE_STRIDE) +#define NV_FSP_MSG_QUEUE_TAIL \ + (NV_FSP_FALCON_BASE_PAGE + NV_FALCON_MSG_QUEUE_TAIL_BASE_OFFSET + \ + NV_FSP_RPC_CHANNEL_HOPPER * NV_FALCON_QUEUE_STRIDE) + +/* MCTP Header */ +#define NV_MCTP_HDR_SEID_SHIFT 16 +#define NV_MCTP_HDR_SEID_MASK 0xff +#define NV_MCTP_HDR_SEQ_SHIFT 28 +#define NV_MCTP_HDR_SEQ_MASK 0x3 +#define NV_MCTP_HDR_EOM_BIT 0x40000000 +#define NV_MCTP_HDR_SOM_BIT 0x80000000 + +/* MCTP Message Header */ +#define NV_MCTP_MSG_TYPE_SHIFT 0 +#define NV_MCTP_MSG_TYPE_MASK 0x7f +#define NV_MCTP_MSG_TYPE_VENDOR_DEFINED 0x7e +#define NV_MCTP_MSG_VENDOR_ID_SHIFT 8 +#define NV_MCTP_MSG_VENDOR_ID_MASK 0xffff +#define NV_MCTP_MSG_VENDOR_ID_NVIDIA 0x10de +#define NV_MCTP_MSG_NVDM_TYPE_SHIFT 24 +#define NV_MCTP_MSG_NVDM_TYPE_MASK 0xff + +/* NVDM response type */ +#define NV_NVDM_TYPE_RESPONSE 0x15 + +/* Minimum response size: mctp_hdr + msg_hdr + status_hdr + type + status */ +#define NV_FSP_RPC_MIN_RESPONSE_WORDS 5 + +/* FBIF (Frame Buffer Interface) Registers */ +/* Legacy PMU FBIF offsets (Kepler, Maxwell Gen1) */ +#define NV_PMU_LEGACY_FBIF_CTL_OFFSET 0x624 +#define NV_PMU_LEGACY_FBIF_TRANSCFG_OFFSET 0x600 + +/* PMU FBIF offsets */ +#define NV_PMU_FBIF_CTL_OFFSET 0xe24 +#define NV_PMU_FBIF_TRANSCFG_OFFSET 0xe00 + +/* GSP FBIF offsets */ +#define NV_GSP_FBIF_CTL_OFFSET 0x624 +#define NV_GSP_FBIF_TRANSCFG_OFFSET 0x600 + +/* OFA Falcon Base Page and FBIF offsets (used for Hopper+ DMA) */ +#define NV_OFA_FALCON_BASE 0x844000 +#define NV_OFA_FBIF_CTL_OFFSET 0x424 +#define NV_OFA_FBIF_TRANSCFG_OFFSET 0x400 + +/* OFA DMA support check register (Hopper+) */ +#define NV_OFA_DMA_SUPPORT_CHECK_REG 0x8443c0 + +/* FSP NVDM command types */ +#define NV_NVDM_TYPE_FBDMA 0x22 +#define NV_FBDMA_SUBCMD_ENABLE 0x1 + +/* FBIF CTL2 offset (relative to fbif_ctl) */ +#define NV_FBIF_CTL2_OFFSET 0x60 + +/* FBIF TRANSCFG register bits */ +#define NV_FBIF_TRANSCFG_TARGET_MASK 0x3 +#define NV_FBIF_TRANSCFG_SYSMEM_DEFAULT 0x5 + +/* FBIF CTL register bits */ +#define NV_FBIF_CTL_ALLOW_PHYS_MODE 0x10 +#define NV_FBIF_CTL_ALLOW_FULL_PHYS_MODE 0x80 + +/* Memory clear register offsets */ +#define NV_MEM_CLEAR_OFFSET 0x100b20 +#define NV_BOOT_COMPLETE_OFFSET 0x118234 +#define NV_BOOT_COMPLETE_MASK 0x3ff + +/* FSP boot complete register (Hopper+) */ +#define NV_FSP_BOOT_COMPLETE_OFFSET 0x200bc +#define NV_FSP_BOOT_COMPLETE_MASK 0xff + +enum gpu_arch { +GPU_ARCH_UNKNOWN =-1 , +GPU_ARCH_KEPLER =0 , +GPU_ARCH_MAXWELL_GEN1 , +GPU_ARCH_MAXWELL_GEN2 , +GPU_ARCH_PASCAL , +GPU_ARCH_PASCAL_10X , +GPU_ARCH_VOLTA , +GPU_ARCH_TURING , +GPU_ARCH_AMPERE , +GPU_ARCH_ADA , +GPU_ARCH_HOPPER , +}; + +enum falcon_type { +FALCON_TYPE_PMU_LEGACY =0 , +FALCON_TYPE_PMU , +FALCON_TYPE_GSP , +FALCON_TYPE_OFA , +}; + +struct falcon { +u32 base_page; +u32 dmactl; +u32 engine_reset; +u32 fbif_ctl; +u32 fbif_ctl2; +u32 fbif_transcfg; +u32 dmem_control_reg; +u32 dmem_data_reg; +bool no_outside_reset; +}; + +struct gpu_properties { +u32 pmc_enable_mask; +bool memory_clear_supported; +enum falcon_type falcon_type; +}; + +static const u32 verified_gpu_map[] = { +0x0e40a0a2 ,/* K520 */ +0x0e6000a1 ,/* GTX660 */ +0x0e63a0a1 ,/* K4000 */ +0x0f22d0a1 ,/* K80 */ +0x108000a1 ,/* GT635 */ +0x117010a2 ,/* GTX750 */ +0x117020a2 ,/* GTX745 */ +0x124320a1 ,/* M60 */ +0x130000a1 ,/* P100 */ +0x134000a1 ,/* P4 */ +0x132000a1 ,/* P40 */ +0x140000a1 ,/* V100 */ +0x164000a1 ,/* T4 */ +0xb77000a1 ,/* A16 */ +0x170000a1 ,/* A100 */ +0xb72000a1 ,/* A10 */ +0x180000a1 ,/* H100 */ +0x194000a1 ,/* L4 */ +0x192000a1 ,/* L40S */ +}; + +#define VERIFIED_GPU_MAP_SIZE ARRAY_SIZE(verified_gpu_map) + +static const struct gpu_properties gpu_properties_map[] = { + [GPU_ARCH_KEPLER ] = { + .pmc_enable_mask =NV_PMC_ENABLE_PWR |NV_PMC_ENABLE_HUB , + .memory_clear_supported = false, + .falcon_type =FALCON_TYPE_PMU_LEGACY , + }, + [GPU_ARCH_MAXWELL_GEN1 ] = { + .pmc_enable_mask =NV_PMC_ENABLE_PWR |NV_PMC_ENABLE_HUB , + .memory_clear_supported = false, + .falcon_type =FALCON_TYPE_PMU_LEGACY , + }, + [GPU_ARCH_MAXWELL_GEN2 ] = { + .pmc_enable_mask =NV_PMC_ENABLE_PWR , + .memory_clear_supported = false, + .falcon_type =FALCON_TYPE_PMU , + }, + [GPU_ARCH_PASCAL ] = { + .pmc_enable_mask =NV_PMC_ENABLE_PWR , + .memory_clear_supported = false, + .falcon_type =FALCON_TYPE_PMU , + }, + [GPU_ARCH_PASCAL_10X ] = { + .pmc_enable_mask =0 , + .memory_clear_supported = false, + .falcon_type =FALCON_TYPE_PMU , + }, + [GPU_ARCH_VOLTA ] = { + .pmc_enable_mask =0 , + .memory_clear_supported = false, + .falcon_type =FALCON_TYPE_GSP , + }, + [GPU_ARCH_TURING ] = { + .pmc_enable_mask =0 , + .memory_clear_supported = true, + .falcon_type =FALCON_TYPE_GSP , + }, + [GPU_ARCH_AMPERE ] = { + .pmc_enable_mask =0 , + .memory_clear_supported = true, + .falcon_type =FALCON_TYPE_GSP , + }, + [GPU_ARCH_ADA ] = { + .pmc_enable_mask =0 , + .memory_clear_supported = true, + .falcon_type =FALCON_TYPE_PMU , + }, + [GPU_ARCH_HOPPER ] = { + .pmc_enable_mask =0 , + .memory_clear_supported = true, + .falcon_type =FALCON_TYPE_OFA , + }, +}; + +static const struct falcon falcon_map[] = { + [FALCON_TYPE_PMU_LEGACY ] = { + .base_page =NV_PPWR_FALCON_BASE , + .dmactl =NV_PPWR_FALCON_BASE +NV_FALCON_DMACTL_OFFSET , + .engine_reset =NV_PPWR_FALCON_BASE +NV_FALCON_ENGINE_RESET_OFFSET , + .fbif_ctl =NV_PPWR_FALCON_BASE +NV_PMU_LEGACY_FBIF_CTL_OFFSET , + .fbif_ctl2 =NV_PPWR_FALCON_BASE + +NV_PMU_LEGACY_FBIF_CTL_OFFSET +NV_FBIF_CTL2_OFFSET , + .fbif_transcfg =NV_PPWR_FALCON_BASE +NV_PMU_LEGACY_FBIF_TRANSCFG_OFFSET , + .dmem_control_reg =NV_PPWR_FALCON_BASE +NV_FALCON_DMEMC_OFFSET , + .dmem_data_reg =NV_PPWR_FALCON_BASE +NV_FALCON_DMEMD_OFFSET , + .no_outside_reset = false, + }, + [FALCON_TYPE_PMU ] = { + .base_page =NV_PPWR_FALCON_BASE , + .dmactl =NV_PPWR_FALCON_BASE +NV_FALCON_DMACTL_OFFSET , + .engine_reset =NV_PPWR_FALCON_BASE +NV_FALCON_ENGINE_RESET_OFFSET , + .fbif_ctl =NV_PPWR_FALCON_BASE +NV_PMU_FBIF_CTL_OFFSET , + .fbif_ctl2 =NV_PPWR_FALCON_BASE +NV_PMU_FBIF_CTL_OFFSET +NV_FBIF_CTL2_OFFSET , + .fbif_transcfg =NV_PPWR_FALCON_BASE +NV_PMU_FBIF_TRANSCFG_OFFSET , + .dmem_control_reg =NV_PPWR_FALCON_BASE +NV_FALCON_DMEMC_OFFSET , + .dmem_data_reg =NV_PPWR_FALCON_BASE +NV_FALCON_DMEMD_OFFSET , + .no_outside_reset = false, + }, + [FALCON_TYPE_GSP ] = { + .base_page =NV_PGSP_FALCON_BASE , + .dmactl =NV_PGSP_FALCON_BASE +NV_FALCON_DMACTL_OFFSET , + .engine_reset =NV_PGSP_FALCON_BASE +NV_FALCON_ENGINE_RESET_OFFSET , + .fbif_ctl =NV_PGSP_FALCON_BASE +NV_GSP_FBIF_CTL_OFFSET , + .fbif_ctl2 =NV_PGSP_FALCON_BASE +NV_GSP_FBIF_CTL_OFFSET +NV_FBIF_CTL2_OFFSET , + .fbif_transcfg =NV_PGSP_FALCON_BASE +NV_GSP_FBIF_TRANSCFG_OFFSET , + .dmem_control_reg =NV_PGSP_FALCON_BASE +NV_FALCON_DMEMC_OFFSET , + .dmem_data_reg =NV_PGSP_FALCON_BASE +NV_FALCON_DMEMD_OFFSET , + .no_outside_reset = false, + }, + [FALCON_TYPE_OFA ] = { + .base_page =NV_OFA_FALCON_BASE , + .dmactl =NV_OFA_FALCON_BASE +NV_FALCON_DMACTL_OFFSET , + .engine_reset =NV_OFA_FALCON_BASE +NV_FALCON_ENGINE_RESET_OFFSET , + .fbif_ctl =NV_OFA_FALCON_BASE +NV_OFA_FBIF_CTL_OFFSET , + .fbif_ctl2 =NV_OFA_FALCON_BASE +NV_OFA_FBIF_CTL_OFFSET +NV_FBIF_CTL2_OFFSET , + .fbif_transcfg =NV_OFA_FALCON_BASE +NV_OFA_FBIF_TRANSCFG_OFFSET , + .dmem_control_reg =NV_OFA_FALCON_BASE +NV_FALCON_DMEMC_OFFSET , + .dmem_data_reg =NV_OFA_FALCON_BASE +NV_FALCON_DMEMD_OFFSET , + .no_outside_reset = true, + }, +}; + +#endif /* _NV_FALCON_HW_H_ */
tools/testing/selftests/vfio/lib/drivers/nv_falcon/nv_falcon.c new file +789 -0
789 lines not on this page · raw message
tools/testing/selftests/vfio/lib/libvfio.mk +2 -0
@@ -14,6 +14,8 @@ LIBVFIO_C += drivers/ioat/ioat.c LIBVFIO_C += drivers/dsa/dsa.c endif +LIBVFIO_C += drivers/nv_falcon/nv_falcon.c + LIBVFIO_OUTPUT := $(OUTPUT)/libvfio LIBVFIO_O := $(patsubst %.c, $(LIBVFIO_OUTPUT)/%.o, $(LIBVFIO_C))
tools/testing/selftests/vfio/lib/vfio_pci_driver.c +3 -0
@@ -7,11 +7,14 @@ extern struct vfio_pci_driver_ops dsa_ops;extern struct vfio_pci_driver_ops ioat_ops;#endif +extern struct vfio_pci_driver_ops nv_falcon_ops; +static struct vfio_pci_driver_ops *driver_ops[] = { #ifdef __x86_64__ &dsa_ops, &ioat_ops, #endif + &nv_falcon_ops, };void vfio_pci_driver_probe (struct vfio_pci_device *device)
Signature · 1 line
2.51.0
David Matlack
maintainerRe: [PATCH v13 4/4] vfio: selftests: Add NVIDIA Falcon driver for DMA testing
Quoted 18 lines · Alex Williamson
From: Rubin Du <[email protected]>
Add a new VFIO PCI driver for NVIDIA GPUs that enables DMA testing via the Falcon (Fast Logic Controller) microcontrollers. This driver extracts and adapts the DMA test functionality from NVIDIA's gpu-admin-tools project and integrates it into the existing VFIO selftest framework.
Falcons are general-purpose microcontrollers present on NVIDIA GPUs that can perform DMA operations between system memory and device memory. By leveraging Falcon DMA, this driver allows NVIDIA GPUs to be tested alongside Intel IOAT and DSA devices using the same selftest infrastructure.
The driver is named '
nv_falcon' to reflect that it specifically controls the Falcon microcontrollers for DMA operations, rather than exposing general GPU functionality.
Quoted 5 lines
Reference implementation:
https://github.com/NVIDIA/gpu-admin-toolsSigned-off-by Rubin Du <[email protected]>Signed-off-by Alex Williamson <[email protected]>
Overall looks close, my comments are mostly nits.
I'm hoping to find a machine with one of the supported GPUs next week as well so I can test out the new driver.
tools/testing/selftests/vfio/lib/drivers/nv_falcon/hw.h line 6
Quoted 8 lines
+++
b/tools/testing/selftests/vfio/lib/drivers/nv_falcon/hw.htools/testing/selftests/vfio/lib/drivers/nv_falcon/hw.h new file +6
-0@@ -0,0 +1,350 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ +/* +* Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. All rights reserved. +*/ +#ifndef _NV_FALCON_HW_H_ +#define _NV_FALCON_HW_H_
Please add #include <linux/types.h> here to avoid depending on header include ordering.
Quoted 9 lines
tools/testing/selftests/vfio/lib/drivers/nv_falcon/nv_falcon.c new file +9
-0+static int gpu_poll_register (struct vfio_pci_device *device, +const char *name,u32 offset, +u32 expected,u32 mask,u32 timeout_ms) +{ +struct gpu_device *gpu =to_gpu_device (device); +struct timespec start, now; +u64 elapsed_ms; +u32 value; +
tools/testing/selftests/vfio/lib/drivers/nv_falcon/nv_falcon.c line 101
Quoted 10 lines
tools/testing/selftests/vfio/lib/drivers/nv_falcon/nv_falcon.c new file +10
-0+clock_gettime (CLOCK_MONOTONIC , &start); + +for (;;) { + value =gpu_read32 (gpu, offset); +if ((value & mask) == expected) +return 0 ; + +clock_gettime (CLOCK_MONOTONIC , &now); + elapsed_ms = (now.tv_sec - start.tv_sec) *1000 + + (now.tv_nsec - start.tv_nsec) /1000000 ;
Create a helper to avoid duplicating elapsed_ms below in fsp_poll_queue().
tools/testing/selftests/vfio/lib/drivers/nv_falcon/nv_falcon.c line 335
Quoted 7 lines
tools/testing/selftests/vfio/lib/drivers/nv_falcon/nv_falcon.c new file +7
-0+static int fsp_init (struct vfio_pci_device *device) +{ +int ret; + + ret =gpu_poll_register (device,"fsp_boot_complete" , +NV_FSP_BOOT_COMPLETE_OFFSET , +NV_FSP_BOOT_COMPLETE_MASK ,0xffffffff ,5000 );
Sashiko is wondering if the mask should be something other than 0xffffffff:
https://sashiko.dev/#/patchset/20260408225459.3088623-1-alex.williamson%40nvidia.com?part=4
tools/testing/selftests/vfio/lib/drivers/nv_falcon/nv_falcon.c line 401
Quoted 8 lines
tools/testing/selftests/vfio/lib/drivers/nv_falcon/nv_falcon.c new file +8
-0+static void falcon_select_core_falcon (struct vfio_pci_device *device) +{ +struct gpu_device *gpu =to_gpu_device (device); +const struct falcon *falcon = gpu->falcon; +u32 core_select_reg = falcon->base_page +NV_FALCON_CORE_SELECT_OFFSET ; +u32 core_select; + +/* Read current value */
nit: Unnecessary comment.
Quoted 2 lines
tools/testing/selftests/vfio/lib/drivers/nv_falcon/nv_falcon.c new file +2
-0+static int nv_falcon_dma_init (struct vfio_pci_device *device) +{
[...]
tools/testing/selftests/vfio/lib/drivers/nv_falcon/nv_falcon.c line 540
Quoted 5 lines
tools/testing/selftests/vfio/lib/drivers/nv_falcon/nv_falcon.c new file +5
-0+if (!falcon->no_outside_reset) { + ret =falcon_reset (device); +if (ret) +return ret; + }
falcon_enable() is called right before nv_falcon_dma_init(). Calling falcon_reset() here will do falcon_disable() and then falcon_enable() again. So it seems unnecessary?
tools/testing/selftests/vfio/lib/drivers/nv_falcon/nv_falcon.c line 565
Quoted 4 lines
tools/testing/selftests/vfio/lib/drivers/nv_falcon/nv_falcon.c new file +4
-0+static int nv_falcon_dma (struct vfio_pci_device *device, +u64 address, +u32 size_encoding, +bool write)
nit: Align with open parenthesis.
tools/testing/selftests/vfio/lib/drivers/nv_falcon/nv_falcon.c line 598
Quoted 4 lines
tools/testing/selftests/vfio/lib/drivers/nv_falcon/nv_falcon.c new file +4
-0+static int nv_falcon_memcpy_chunk (struct vfio_pci_device *device, +iova_t src, +iova_t dst, +u32 size_encoding)
nit: Align with open parenthesis.
tools/testing/selftests/vfio/lib/drivers/nv_falcon/nv_falcon.c line 605
Quoted 7 lines
tools/testing/selftests/vfio/lib/drivers/nv_falcon/nv_falcon.c new file +7
-0+{ +int ret; + + ret =nv_falcon_dma (device, src, size_encoding, false); +if (ret) { +dev_err (device,"Failed DMA read (src=0x%lx, encoding=%u)\n" , + src, size_encoding);
nit: Move this and the next dev_err() into nv_falcon_dma().
Quoted 41 lines
tools/testing/selftests/vfio/lib/drivers/nv_falcon/nv_falcon.c new file +41
-0+return ret; + } + + ret =nv_falcon_dma (device, dst, size_encoding, true); +if (ret) { +dev_err (device,"Failed DMA write (dst=0x%lx, encoding=%u)\n" , + dst, size_encoding); +return ret; + } + +return ret; +} + +static int nv_falcon_probe (struct vfio_pci_device *device) +{ +enum gpu_arch gpu_arch; +u32 pmc_boot_0; +void *bar0; +int i; + +if (vfio_pci_config_readw (device,PCI_VENDOR_ID ) !=PCI_VENDOR_ID_NVIDIA ) +return -ENODEV ; + +if (vfio_pci_config_readw (device,PCI_CLASS_DEVICE ) >>8 != +PCI_BASE_CLASS_DISPLAY ) +return -ENODEV ; + +/* Get BAR0 pointer for reading GPU registers */ + bar0 = device->bars[0 ].vaddr; +if (!bar0) +return -ENODEV ; + +/* Read PMC_BOOT_0 register from BAR0 to identify GPU */ + pmc_boot_0 =readl (bar0 +NV_PMC_BOOT_0 ); + +/* Look up GPU architecture to verify this is a supported GPU */ + gpu_arch =nv_gpu_arch_lookup (pmc_boot_0); +if (gpu_arch ==GPU_ARCH_UNKNOWN ) { +dev_err (device,"Unsupported GPU architecture for PMC_BOOT_0: 0x%x\n" , + pmc_boot_0); +return -ENODEV ;
tools/testing/selftests/vfio/lib/drivers/nv_falcon/nv_falcon.c line 656
Quoted 10 lines
tools/testing/selftests/vfio/lib/drivers/nv_falcon/nv_falcon.c new file +10
-0+ } + +/* Check verified GPU map */ +for (i =0 ; i <VERIFIED_GPU_MAP_SIZE ; i++) { +if (verified_gpu_map[i] == pmc_boot_0) +return 0 ; + } + +dev_info (device,"Unvalidated GPU: PMC_BOOT_0: 0x%x, possibly not supported\n" , + pmc_boot_0);
nit: Align with open parenthesis.
Quoted 9 lines
tools/testing/selftests/vfio/lib/drivers/nv_falcon/nv_falcon.c new file +9
-0+ +return 0 ; +} + +static void nv_falcon_init (struct vfio_pci_device *device) +{ +struct gpu_device *gpu =to_gpu_device (device); +const struct gpu_properties *props; +enum gpu_arch gpu_arch;
tools/testing/selftests/vfio/lib/drivers/nv_falcon/nv_falcon.c line 675
Quoted 10 lines
tools/testing/selftests/vfio/lib/drivers/nv_falcon/nv_falcon.c new file +10
-0+u32 pmc_boot_0; +int ret; + +VFIO_ASSERT_GE (device->driver.region.size,sizeof (*gpu)); + +/* Read PMC_BOOT_0 register from BAR0 to identify GPU */ + pmc_boot_0 =readl (device->bars[0 ].vaddr +NV_PMC_BOOT_0 ); + +/* Look up GPU architecture */ + gpu_arch =nv_gpu_arch_lookup (pmc_boot_0);
nit: gpu->arch can just be used to avoid needing extra local variable.
tools/testing/selftests/vfio/lib/drivers/nv_falcon/nv_falcon.c line 677
Quoted 2 lines
tools/testing/selftests/vfio/lib/drivers/nv_falcon/nv_falcon.c new file +2
-0+VFIO_ASSERT_NE (gpu_arch,GPU_ARCH_UNKNOWN , +"Unsupported GPU architecture (PMC_BOOT_0=0x%x)\n" , pmc_boot_0);
This assert should already be guaranteed by probe().
Quoted 6 lines
tools/testing/selftests/vfio/lib/drivers/nv_falcon/nv_falcon.c new file +6
-0+ + props = &gpu_properties_map[gpu_arch]; + +/* Populate GPU structure */ + gpu->arch = gpu_arch; + gpu->bar0 = device->bars[0 ].vaddr;
tools/testing/selftests/vfio/lib/drivers/nv_falcon/nv_falcon.c line 693
Quoted 10 lines
tools/testing/selftests/vfio/lib/drivers/nv_falcon/nv_falcon.c new file +10
-0+ gpu->is_memory_clear_supported = props->memory_clear_supported; + gpu->falcon = &falcon_map[props->falcon_type]; + gpu->pmc_enable_mask = props->pmc_enable_mask; + + ret =falcon_enable (device); +VFIO_ASSERT_EQ (ret,0 ,"Failed to enable falcon: %d\n" , ret); + +/* Initialize falcon for DMA */ + ret =nv_falcon_dma_init (device); +VFIO_ASSERT_EQ (ret,0 ,"Failed to initialize falcon DMA: %d\n" , ret);
Since the init() and remove() callbacks return void, propagating errors from functions that are only reachable from there is unnecessary. Consider using VFIO_ASSERT*() and making more functions return void.
Consider this an optional suggestion since will be a lot of churn and I'm not sure it will be a net improvement.
Quoted 8 lines
tools/testing/selftests/vfio/lib/drivers/nv_falcon/nv_falcon.c new file +8
-0+ + device->driver.max_memcpy_size =NV_FALCON_DMA_MAX_TRANSFER_SIZE ; + device->driver.max_memcpy_count =NV_FALCON_DMA_MAX_TRANSFER_COUNT ; +} + +static void nv_falcon_remove (struct vfio_pci_device *device) +{ +falcon_disable (device);
tools/testing/selftests/vfio/lib/drivers/nv_falcon/nv_falcon.c line 711
Quoted 10 lines
tools/testing/selftests/vfio/lib/drivers/nv_falcon/nv_falcon.c new file +10
-0+vfio_pci_cmd_clear (device,PCI_COMMAND_MASTER ); +} + +/* +* Falcon DMA can only process one transfer at a time, +* so the actual work is deferred to memcpy_wait() to conform to the +* memcpy_start()/memcpy_wait() contract. +*/ +static void nv_falcon_memcpy_start (struct vfio_pci_device *device, +iova_t src,iova_t dst,u64 size,u64 count)
nit: Align with open parenthesis.
tools/testing/selftests/vfio/lib/drivers/nv_falcon/nv_falcon.c line 715
Quoted 4 lines
tools/testing/selftests/vfio/lib/drivers/nv_falcon/nv_falcon.c new file +4
-0+{ +struct gpu_device *gpu =to_gpu_device (device); + +VFIO_ASSERT_EQ (gpu->memcpy_count,0 );
gpu->memcpy_count can be removed. The caller vfio_pci_drvier_memcpy_start() guarantees no memcpys are in-progress. I guess this was copied from the DSA driver, but that needs to track the count because that determines what type of completion it needs to wait on in memcpy_wait().
nv_falcon only supports a single copy, so no need to track the count across start/wait.
Quoted 13 lines
tools/testing/selftests/vfio/lib/drivers/nv_falcon/nv_falcon.c new file +13
-0+static int nv_falcon_memcpy_wait (struct vfio_pci_device *device) +{ +struct gpu_device *gpu =to_gpu_device (device); +iova_t src = gpu->memcpy_src; +iova_t dst = gpu->memcpy_dst; +u64 remaining = gpu->memcpy_size; +int ret =0 ; + +VFIO_ASSERT_EQ (gpu->memcpy_count,1 ); + +/* +* Falcon DMA supports power-of-2 transfer sizes in [4, 256] and +* cannot cross 256-byte block boundaries. Decompose the request
tools/testing/selftests/vfio/lib/drivers/nv_falcon/nv_falcon.c line 769
Quoted 10 lines
tools/testing/selftests/vfio/lib/drivers/nv_falcon/nv_falcon.c new file +10
-0+* into the largest valid chunk at each step. +*/ +while (remaining) { +u64 chunk =rounddown_pow_of_two (remaining); + + chunk =min (chunk,dma_block_remain (src)); + chunk =min (chunk,dma_block_remain (dst)); + + ret =nv_falcon_memcpy_chunk (device, src, dst, +size_to_dma_encoding (chunk));
nit: Suggest moving size_to_dma_encoding() all the way down into nv_falcon_dma() keep this function simple, avoid the line wrap, and it's an implementation detail of how the copy operation gets posted to the device.
Re: [PATCH v13 4/4] vfio: selftests: Add NVIDIA Falcon driver for DMA testing
Quoted 13 lines · On Fri, 10 Apr 2026 22:19:59 +0000 David Matlack <[email protected]> wrote:
tools/testing/selftests/vfio/lib/drivers/nv_falcon/nv_falcon.c line 335
Quoted 8 lines · Alex Williamson
From: Rubin Du <[email protected]>
tools/testing/selftests/vfio/lib/drivers/nv_falcon/nv_falcon.c new file +7
-0+static int fsp_init (struct vfio_pci_device *device) +{ +int ret; + + ret =gpu_poll_register (device,"fsp_boot_complete" , +NV_FSP_BOOT_COMPLETE_OFFSET , +NV_FSP_BOOT_COMPLETE_MASK ,0xffffffff ,5000 );Sashiko is wondering if the mask should be something other than
0xffffffff:
Quoted 1 line
https://sashiko.dev/#/patchset/20260408225459.3088623-1-alex.williamson%40nvidia.com?part=4
I consulted the gpu-admin-tools and it performs the same test. I think the hangup here is that we're labeling the expected value field with a macro named _MASK, when the mask is actually provided in the next field. I'll rename this to _SUCCESS for consistency with the reference implementation.
Quoted 10 lines
[...]
tools/testing/selftests/vfio/lib/drivers/nv_falcon/nv_falcon.c line 540
Quoted 5 lines
tools/testing/selftests/vfio/lib/drivers/nv_falcon/nv_falcon.c new file +5
-0+if (!falcon->no_outside_reset) { + ret =falcon_reset (device); +if (ret) +return ret; + }
falcon_enable()is called right beforenv_falcon_dma_init(). Callingfalcon_reset()here will dofalcon_disable()and thenfalcon_enable()again. So it seems unnecessary?
Yes, I think that initial falcon_enable() is unnecessary, and it's a bit subtle, but a no_outside_reset device doesn't do anything for enable or disable. I think the solution is to pull that test to the top of the enable function, at which point we can more clearly call falcon_reset() unconditionally here and remove the earlier falcon_enable() call.
I've incorporated all your other comments, aside from the optional refactoring, thanks! Testing to make sure the above change doesn't regress on any devices will take some time, so I'll expect this to miss the merge window and hopefully get queued for 7.2. Thanks,
Alex