vfio, iommufd: Enabling user space drivers to vend more granular access to client processes
Hello,
We've been running user space drivers (USD) in production built on top of VFIO, and have come to value the operational and development benefits of being able to deploy updates to device policy by simply shipping user space binaries.
In our architecture, a long-running USD process bootstraps and manages background device operations related to supporting client process workloads. Client processes communicate with the USD over Unix domain sockets to acquire the device resources necessary to dispatch work to the device.
We anticipate a growing need to provide more granular access to device resources beyond what the kernel currently affords to user space drivers similar to our model.
The purpose of this email is to:
- Gauge the extent to which ongoing work in VFIO and IOMMUFD can meet those needs.
- Seek guidance from VFIO and IOMMUFD maintainers about whether there is a path to supporting the remaining pieces across the VFIO and IOMMUFD UAPIs.
- Describe our current approach and get feedback on whether there are existing solutions that we've missed.
- Figure out the most useful places we can help contribute.
Inter-process communication latency (between client processes and USD) is prohibitively slow to support hot-path communication between the client and device, which targets round-trip times on the order of microseconds. To address this, we need to allow client processes and the device to access each other's memory directly, bypassing IPC with the USD and kernel syscalls.
a) For host-initiated access into device memory, this means mmap-ing BAR sub-regions into the client process.
b) For device-initiated access into host memory, it means establishing IOMMU mappings to memory underlying the client process address space.
Such things are more straightforward for in-kernel device drivers to accomplish:
they are free to define customized semantics for their associated fds and syscall handlers. Today, user space driver processes have fewer tools at their disposal for controlling these types of access.
----------
BAR Access
----------
To achieve (a), the USD sends the VFIO device fd to the client over Unix domain sockets using SCM_RIGHTS, along with descriptions of which device regions are for what. While this allows the client to mmap BARs into its address space, it comes at the cost of exposing more access to device BAR regions than is necessary or appropriate. In our use case, we don't need to contend with adversarial client processes, so the current situation is tenable, but not ideal.
Ongoing efforts to add dma-buf exporting to VFIO [1] seem relevant here. Though its current intent is around enabling peer-to-peer access, the fact that only a subset of device regions are bound to this fd could be useful for controlling access granularity to device regions.
Instead of vending the VFIO device fd to the client process, the USD could bind the necessary BAR regions to a dma-buf fd and share that with the client. If VFIO supported dma_buf_ops.mmap, the client could mmap those into its address space.
Adding such capability would mean that there would be two paths for mmap-ing device regions: VFIO device fd and dma-buf fd. I imagine this could be contentious; people may not like that there are two ways to achieve the same thing. It also seems complicated by the fact that there are ongoing discussions about how to extend the VFIO device fd UAPI to support features like write combining [2]. It would feel incomplete for such features to be available through one mmap-ing modality but not the other. This would have implications for how the “special regions” should be communicated across the UAPI.
The VFIO dma-buf UAPI currently being proposed [3] takes a region_index and an array of (offset, length) intervals within the region to assign to the dma-buf. From what I can tell, that seems coherent with the latest direction from [2], which will enable the creation of new region indices with special properties, which are aliases to the default BAR regions. The USD could theoretically create a dma-buf backed by "the region index corresponding to write-combined BAR 4" to share with the client.
Given some of the considerations above, would there be line of sight for adding support for dma_buf_ops.mmap to VFIO?
-------------
IOMMU Mapping
-------------
To achieve (b), we have been using the (now legacy) VFIO container interface to manage access to the IOMMU. We understand that new feature development has moved to IOMMUFD, and intend to migrate to using it when it's ready (we have some use cases that require P2P). We are not expecting to add features to VFIO containers. I will describe what we are doing today first.
In order to enable a device to access memory in multiple processes, we also share the VFIO container fd using SCM_RIGHTS between the USD and client processes. In this scheme, we partition the I/O address space (IOAS) for a given device's container to be used cooperatively amongst each process. The only enforcement of the partitioning convention is that each process only VFIO_IOMMU_{MAP,UNMAP}_DMA's to the IOVA ranges which have been assigned to it.
When the USD detects that the client process has exited, it is able to unmap any leftover dirty mappings with VFIO_IOMMU_UNMAP_DMA. This became possible after [4], which allowed one process to free the mappings created by another process. That patch's intent was to enable QEMU live update use cases, but benefited our use case as well.
Again, we don't have to contend with adversarial client processes, so this has been OK, but not ideal for now.
We are interested in the following incremental capabilities:
- We want the USD to be able to create and vend fds which provide restricted mapping access to the device's IOAS to the client, while preserving the ability of the USD to revoke device access to client memory via VFIO_IOMMU_UNMAP_DMA (or IOMMUFD_CMD_IOAS_UNMAP for IOMMUFD). Alternatively, to forcefully invalidate the entire restricted IOMMU fd, including mappings.
- It would be nice if mappings created with the restricted IOMMU fd were automatically freed when the underlying kernel object was freed (if the client process were to exit ungracefully without explicitly performing unmap cleanup after itself).
Some of those things sound very similar to the direction of vIOMMU, but it is difficult to tell if that could meet our needs exactly. The kinds of features I think we want should be achievable purely in software without any dedicated hardware support.
This is an area we are less familiar with, since we haven't been living on the IOMMUFD UAPI or following its development as closely yet. Perhaps we have missed something more obvious?
Overall, I'm curious to hear feedback on this. Allowing user space drivers to vend more granular device access would certainly benefit our use case, and perhaps others as well.
[1] https://lore.kernel.org/all/[email protected]/
[2] https://lore.kernel.org/all/[email protected]/
[3] https://lore.kernel.org/all/5e043d8b95627441db6156e7f15e6e1658e9d537.1754311439.git.leon@kernel.org/
[4] https://lore.kernel.org/all/[email protected]/
Thanks,
Alex
Quoted 3 lines · Alex Mastro
We anticipate a growing need to provide more granular access to device resources beyond what the kernel currently affords to user space drivers similar to our model.
I'm having a somewhat hard time wrapping my head around the security model that says your trust your related processes not use DMA in a way that is hostile their peers, but you don't trust them not to issue hostile ioctls..
Quoted 5 lines
To achieve (a), the USD sends the VFIO device fd to the client over Unix domain sockets using
SCM_RIGHTS, along with descriptions of which device regions are for what. While this allows the client to mmap BARs into its address space, it comes at the cost of exposing more access to device BAR regions than is necessary or appropriate.
IIRC VFIO should allow partial BAR mappings, so the client process can robustly have a subset mapped if you trust it to perform the unix SCM_RIGHTS/mapping ioctl/close() sequence.
Quoted 4 lines
Instead of vending the VFIO device fd to the client process, the USD could bind the necessary BAR regions to a dma-buf fd and share that with the client. If VFIO supported dma_buf_ops.mmap, the client could mmap those into its address space.
I wouldn't object to this, I think it is not too complicated at all.
And the idea to add some 'use writecombining' to the create dmabuf ioctl is certainly a novel and simple way to solve that problem.
Quoted 6 lines
We are interested in the following incremental capabilities:
- We want the USD to be able to create and vend fds which provide restricted mapping access to the device's IOAS to the client, while preserving the ability of the USD to revoke device access to client memory viaVFIO_IOMMU_UNMAP_DMA(orIOMMUFD_CMD_IOAS_UNMAPfor IOMMUFD). Alternatively, to forcefully invalidate the entire restricted IOMMU fd, including mappings.
I've had similarish requests for fwctl..
What I've been thinking is if the vending process could "dup" the FD and permanently attach a BPF program to the new FD that sits right after ioctl. The BPF program would inspect each ioctl when it is issued and enforce whatever policy the vending process wants.
Sort of like seccomp.
iommufd and fwctl have a similar ioctl design, so I would have no issue with something that could be easily reused for both.
What would give me alot of pause is your proposal where we effectively have the kernel enforce some arbitary policy, and I know from experience there will be endless asks for more and more policy options.
Quoted 4 lines
- It would be nice if mappings created with the restricted IOMMU fd were automatically freed when the underlying kernel object was freed (if the client process were to exit ungracefully without explicitly performing unmap cleanup after itself).
Maybe the BPF could trigger an eventfd or something when the FD closes?
Quoted 4 lines
Some of those things sound very similar to the direction of vIOMMU, but it is difficult to tell if that could meet our needs exactly. The kinds of features I think we want should be achievable purely in software without any dedicated hardware support.
I don't think viommu is really related to this, viommu is more about multiple physical devices.
Jason
Quoted 10 lines · Jason Gunthorpe
Quoted 3 lines · Alex Mastro
We anticipate a growing need to provide more granular access to device resources beyond what the kernel currently affords to user space drivers similar to our model.
I'm having a somewhat hard time wrapping my head around the security model that says your trust your related processes not use DMA in a way that is hostile their peers, but you don't trust them not to issue hostile ioctls..
I read this as more about having the granularity to automatically release resources associated with a client process when it dies (as mentioned below) rather than relying on the bootstrapping process to manage it all. Not really about hostile ioctls, but that an ungraceful ending of some client workload doesn't even send them.
Quoted 6 lines
Quoted 4 lines
- It would be nice if mappings created with the restricted IOMMU fd were automatically freed when the underlying kernel object was freed (if the client process were to exit ungracefully without explicitly performing unmap cleanup after itself).
Maybe the BPF could trigger an eventfd or something when the FD closes?
I wouldn't have considered a BPF dependency for this. I'll need to think about that one for a moment.
Quoted 17 lines
From: Keith Busch <[email protected]>
Sent: Friday, September 19, 2025 7:25 AMQuoted 3 lines · Jason Gunthorpe
Quoted 1 line · Alex Mastro
We anticipate a growing need to provide more granular access to device
resources
Quoted 1 line
Quoted 1 line
beyond what the kernel currently affords to user space drivers similar to
our
Quoted 6 lines
Quoted 1 line
model.
I'm having a somewhat hard time wrapping my head around the security model that says your trust your related processes not use DMA in a way that is hostile their peers, but you don't trust them not to issue hostile ioctls..
Quoted 6 lines
I read this as more about having the granularity to automatically release resources associated with a client process when it dies (as mentioned below) rather than relying on the bootstrapping process to manage it all. Not really about hostile ioctls, but that an ungraceful ending of some client workload doesn't even send them.
the proposal includes two parts: BAR access and IOMMU mapping. For the latter looks the intention is more around releasing resource. But the former sounds more like a security enhancement - instead of granting the client full access to the entire device it aims to expose only a region of BAR resource necessary into guest. Then as Jason questioned what is the value of doing so when one client can program arbitrary DMA address into the exposed BAR region to attack mapped memory of other clients and the USD... there is no hw isolation within a partitioned IOAS unless the device supports PASID then each client can be associated to its own IOAS space.
Quoted 3 lines · Kevin
memory of other clients and the USD... there is no hw isolation within a partitioned IOAS unless the device supports PASID then each client can be associated to its own IOAS space.
If the device does support pasid then both of the suggestions make a lot more security sense..
Jsaon
Quoted 24 lines · Kevin
Quoted 17 lines
From: Keith Busch <[email protected]>
Sent: Friday, September 19, 2025 7:25 AMQuoted 3 lines · Jason Gunthorpe
Quoted 1 line · Alex Mastro
We anticipate a growing need to provide more granular access to device
resources
Quoted 1 line
Quoted 1 line
beyond what the kernel currently affords to user space drivers similar to
our
Quoted 6 lines
Quoted 1 line
model.
I'm having a somewhat hard time wrapping my head around the security model that says your trust your related processes not use DMA in a way that is hostile their peers, but you don't trust them not to issue hostile ioctls..
Quoted 6 lines
I read this as more about having the granularity to automatically release resources associated with a client process when it dies (as mentioned below) rather than relying on the bootstrapping process to manage it all. Not really about hostile ioctls, but that an ungraceful ending of some client workload doesn't even send them.
Quoted 10 lines
the proposal includes two parts: BAR access and IOMMU mapping. For the latter looks the intention is more around releasing resource. But the former sounds more like a security enhancement - instead of granting the client full access to the entire device it aims to expose only a region of BAR resource necessary into guest. Then as Jason questioned what is the value of doing so when one client can program arbitrary DMA address into the exposed BAR region to attack mapped memory of other clients and the USD... there is no hw isolation within a partitioned IOAS unless the device supports PASID then each client can be associated to its own IOAS space.
That’s also my opinion, it seems that PASIDs are not supported in that case, that’s why the clients share the same IOVA address space, instead of each one having their own.
In that case I think as all of this is cooperative and can’t be enforced, one process can corrupt another process memory that is mapped the IOMMU.
It seems to me that any memory mapped in the IOMMU is that situation has to be explicitly shared between processes first through the kernel, so such memory can be accessed both by CPU and DMA by both processes.
Thanks,
Mostafa
Quoted 12 lines · Mostafa Saleh
Quoted 10 lines · Kevin
the proposal includes two parts: BAR access and IOMMU mapping. For the latter looks the intention is more around releasing resource. But the former sounds more like a security enhancement - instead of granting the client full access to the entire device it aims to expose only a region of BAR resource necessary into guest. Then as Jason questioned what is the value of doing so when one client can program arbitrary DMA address into the exposed BAR region to attack mapped memory of other clients and the USD... there is no hw isolation within a partitioned IOAS unless the device supports PASID then each client can be associated to its own IOAS space.
Quoted 3 lines
That’s also my opinion, it seems that PASIDs are not supported in that case, that’s why the clients share the same IOVA address space, instead of each one having their own.
Yes, we do have cases where PASID is not supported by our hardware.
Quoted 2 lines
In that case I think as all of this is cooperative and can’t be enforced, one process can corrupt another process memory that is mapped the IOMMU.
In systems lacking PASID, some degree of enforcement would still be possible via USD and device policies. In a ~similar way to how an in-kernel driver wanting to accomplish our same goals (enabling a client and device able to access each other's memory directly) would presumably need to enforce this.
I have been thinking along the following lines:
Imagine that we want the client and device to communicate with each other directly via queues in each other's memory, bypassing interaction with the driver.
As part of granting access to a client process:
- The USD allocates some IOAS slice for the client.
- The USD prepares a restricted IOMMU fd to be shared with the client which only has mapping permissions to the IOAS slice.
- The USD configures the device: "DMA initiated across this region of client-accessible BAR is only allowed to target the client's IOAS slice."
- The USD vends the client a dma-buf exposing a view of only that client's queue space, along with the restricted IOMMU fd.
Given the above setup, barring bugs in the USD, or the device hardware/firmware, it should be impossible for one client to corrupt another client's address space, since the side-effects it is able to create by accessing its BAR slice have been constrained by a combination of USD + device policy.
Next, we need to address revocation. The USD needs to be able revoke:
1) client access to BAR memory
2) device access to client memory
Issue (2) was touched on in the original tech topic email, but we haven't covered (1) yet.
For (1) to be possible, I think we need to grant the VFIO user (USD in this specific case) the ability to revoke a dma-buf in a way that prevents "peer" access to the device -- whether the peer is some other device, or a user space client process.
Following a dma_buf_ops.mmap, I suppose that revocation would mean:
- Poisoning the dma-buf fd to disallow the creation of additional mmaps.
- Zapping the PTEs backing existing mmaps. Subsequent access to the unmapped client address space should trigger page faults.
Thanks,
Alex
Quoted 1 line · Alex Mastro
Following a dma_buf_ops.mmap, I suppose that revocation would mean:
I'd investigate adding some ioctl to the dmabuf fd to permanently revoke it. The zapping/etc already has to be done just to get mmap in the first place. The vending process would retain a FD on the dmabuf and when it is time to revoke it then it can call the ioctl directly on the fd to revoke.
Jason
Quoted 5 lines · Keith Busch
I read this as more about having the granularity to automatically release resources associated with a client process when it dies (as mentioned below) rather than relying on the bootstrapping process to manage it all. Not really about hostile ioctls, but that an ungraceful ending of some client workload doesn't even send them.
You could achieve this between co-operating processes by monitoring the child with a pidfd, or handing it a pipe and watching for the pipe to close..
Quoted 9 lines
Quoted 6 lines
Quoted 4 lines
- It would be nice if mappings created with the restricted IOMMU fd were automatically freed when the underlying kernel object was freed (if the client process were to exit ungracefully without explicitly performing unmap cleanup after itself).
Maybe the BPF could trigger an eventfd or something when the FD closes?
I wouldn't have considered a BPF dependency for this. I'll need to think about that one for a moment.
Well, if you are going to be using BPF for policy then may as well use it for all policy. It would not be hard to also invoke the BPF duing the file descriptor close and presumably it can somehow to signal the vendor process in some easy BPF way?
Jason
Quoted 5 lines · On Thu, 18 Sep 2025 19:57:39 -0300 Jason Gunthorpe <[email protected]> wrote:
What I've been thinking is if the vending process could "dup" the FD and permanently attach a BPF program to the new FD that sits right after ioctl. The BPF program would inspect each ioctl when it is issued and enforce whatever policy the vending process wants.
Promising idea.
Quoted 4 lines
What would give me alot of pause is your proposal where we effectively have the kernel enforce some arbitary policy, and I know from experience there will be endless asks for more and more policy options.
Definitely. Also, is this at all considering the work that's gone into vfio-user? The long running USD sounds a lot like a vfio-user server, where if we're using vfio-user's socket interface we'd have a lot of opportunity to implement policy there and dma-bufs might be a means to expose direct, restricted access. Thanks,
Alex
Quoted 5 lines · Alex Williamson
Definitely. Also, is this at all considering the work that's gone into vfio-user? The long running USD sounds a lot like a vfio-user server, where if we're using vfio-user's socket interface we'd have a lot of opportunity to implement policy there and dma-bufs might be a means to expose direct, restricted access. Thanks,
Possibly. Though I think the USD's responsibilities and the semantics for how clients would negotiate various forms of device access would be very application-dependent. In addition to just vending vfio and iommu-related fds, our USD needs to do things like bootstrap the device by loading firmwares, collect metrics, and other background functionality.
I'm not sure if I'm addressing your point though.
We actually do use libvfio-user [1] for user space simulation of PCI devices, but it's not a part of our USD today.
[1] https://github.com/nutanix/libvfio-user
Alex
Quoted 4 lines · Jason Gunthorpe
I'm having a somewhat hard time wrapping my head around the security model that says your trust your related processes not use DMA in a way that is hostile their peers, but you don't trust them not to issue hostile ioctls..
Ah, yea. In my original message, I should have emphasized that vending the entire vfio device fd confers access to inappropriate ioctls *in addition to* inappropriate BAR regions that the client should be restricted from accessing.
Assuming we make headway on dma_buf_ops.mmap, granting a client process access to a dma-buf's worth of BAR space does not feel spiritually different than granting it to a peer device. The onus is on the combination of driver + device policy to constrain the side-effects of foreign access to the exposed BAR sub-regions.
Please let me know if I misunderstood your meaning.
Quoted 3 lines
IIRC VFIO should allow partial BAR mappings, so the client process can robustly have a subset mapped if you trust it to perform the unix SCM_RIGHTS/mapping ioctl/close() sequence.
Yes -- we already do this today actually. The USD just tells the client "these are the specific set of (offset, length) within the vfio device fd you should mmap". Those intervals are slices within BARs.
Quoted 6 lines
Quoted 4 lines
Quoted 1 line
Instead of vending the VFIO device fd to the client process, the USD could bind
the necessary BAR regions to a dma-buf fd and share that with the client. If VFIO supported dma_buf_ops.mmap, the client could mmap those into its address space.
I wouldn't object to this, I think it is not too complicated at all.
That's encouraging to hear! Thank you.
Quoted 4 lines
What I've been thinking is if the vending process could "dup" the FD and permanently attach a BPF program to the new FD that sits right after ioctl. The BPF program would inspect each ioctl when it is issued and enforce whatever policy the vending process wants.
This seems totally reasonable to me.
Quoted 4 lines
What would give me alot of pause is your proposal where we effectively have the kernel enforce some arbitary policy, and I know from experience there will be endless asks for more and more policy options.
Agreed. If we can engineer BPF to be able to interact with those ioctls to hoist these kinds of policy decisions up into user space, I can't argue with that.
Quoted 2 lines
I don't think viommu is really related to this, viommu is more about multiple physical devices.
Ack. I wasn't sure how much to read into the "representing a slice of the physical IOMMU instance" comment [1].
[1] https://docs.kernel.org/userspace-api/iommufd.html
Thanks,
Alex