IOASID extensions for guest SVA
Subsystems (7)
Re: [PATCH V4 05/18] iommu/ioasid: Redefine IOASID set and allocation APIs
In reply to Tian, KevinQuoted 20 lines · Kevin
Quoted 3 lines
From: Jason Gunthorpe <[email protected]>
Sent: Wednesday, April 28, 2021 1:12 AM[...]
Quoted 8 lines
Quoted 2 lines
As Alex says, if this line fails because of the group restrictions, that's not great because it's not very obvious what's gone wrong.
Okay, that is fair, but let's solve that problem directly. For instance netlink has been going in the direction of adding a "extack" from the kernel which is a descriptive error string. If the failing ioctl returned the string:
Quoted 7 lines
"cannot join this device to the IOASID because device XXX in the
same group #10 is in use"Would you agree it is now obvious what has gone wrong? In fact would you agree this is a lot better user experience than what applications do today even though they have the group FD?
Quoted 5 lines
Currently all the discussions are around implicit vs. explicit uAPI semantics on the group restriction. However if we look beyond group the implicit semantics might be inevitable when dealing with incompatible iommu domains. An existing example of iommu incompatibility is
IOMMU_CACHE.
I still think we need to get rid of these incompatibilities somehow. Having multiple HW incompatible IOASID in the same platform is just bad all around.
When modeling in userspace IOMMU_CACHE sounds like it is a property of each individual IOASID, not an attribute that requires a new domain.
People that want to create cache bypass IOASID's should just ask for that that directly.
Jason
RE: [PATCH V4 05/18] iommu/ioasid: Redefine IOASID set and allocation APIs
Quoted 30 lines
From: Jason Gunthorpe <[email protected]>
Sent: Wednesday, May 5, 2021 1:13 AMQuoted 20 lines · Kevin
Quoted 3 lines
From: Jason Gunthorpe <[email protected]>
Sent: Wednesday, April 28, 2021 1:12 AM[...]
Quoted 8 lines
Quoted 2 lines
As Alex says, if this line fails because of the group restrictions, that's not great because it's not very obvious what's gone wrong.
Okay, that is fair, but let's solve that problem directly. For instance netlink has been going in the direction of adding a "extack" from the kernel which is a descriptive error string. If the failing ioctl returned the string:
Quoted 7 lines
"cannot join this device to the IOASID because device XXX in the
same group #10 is in use"Would you agree it is now obvious what has gone wrong? In fact would you agree this is a lot better user experience than what applications do today even though they have the group FD?
Quoted 5 lines
Currently all the discussions are around implicit vs. explicit uAPI semantics on the group restriction. However if we look beyond group the implicit semantics might be inevitable when dealing with incompatible iommu domains. An existing example of iommu incompatibility is
IOMMU_CACHE.
Quoted 6 lines
I still think we need to get rid of these incompatibilities somehow. Having multiple HW incompatible IOASID in the same platform is just bad all around.
When modeling in userspace
IOMMU_CACHEsounds like it is a property of each individual IOASID, not an attribute that requires a new domain.
sure. the iommu domain is an kernel-internal concept. The userspace should focus everything on IOASID.
Quoted 4 lines
People that want to create cache bypass IOASID's should just ask for that that directly.
Yes, in earlier discussion we agreed on a scheme that ioasid module will return an error to userspace indicating incompatibility detected when binding a device to ioasid then the userspace should create a new IOASID for this device. This has to be done 'explicitly'.
When I used it as the example for 'implicit semantics" is that the kernel won't create another group-like object to contain devices with compatible attributes and 'explicitly' manage it in uAPI like group_fd. If we anyway rely on the userspace to have more intelligence on those hardware restrictions, it's little sense to only explicitly handle group_fd in uAPI.
Thanks
Kevin
RE: [PATCH V4 05/18] iommu/ioasid: Redefine IOASID set and allocation APIs
In reply to Jason GunthorpeQuoted 3 lines
From: Jason Gunthorpe <[email protected]>
Sent: Wednesday, April 28, 2021 1:12 AM
[...]
Quoted 5 lines
One option is VFIO can keep its group FD but nothing else will have anthing like it. However I don't much like the idea that VFIO will have a special and unique programming model to do that same things other subsystem will do. That will make it harder for userspace to implement.
Hi, Jason,
I have a question here. Based on discussions so far, it's clearly that the new ioasid uAPI will differ from existing VFIO uAPI a lot, e.g. ioasid- centric operations, no group fd, no incompatible domains, etc. Then I wonder how we plan to support legacy VFIO applications in this transition phase. Earlier you ever mentioned the desire of directly replacing /dev/vfio/vfio with /dev/ioasid and having ioasid to present both VFIO and new uAPI. Doesn't it imply that we have to copy the VFIO container/group semantics into /dev/ioasid although it's a special programming model only for VFIO?
Alternatively we could keep all the container/group legacy within VFIO and having /dev/ioasid support only the new uAPI semantics. In this case VFIO will include a shim iommu backend to connect its legacy uAPI into drivers/ioasid backend functions for backward compatibility. Then VFIO will also support a new model which only uses its device uAPI to bind to new ioasid fd w/o using any legacy container/group/iommu uAPI. Does this sound a plan?
Thanks
Kevin
Re: [PATCH V4 05/18] iommu/ioasid: Redefine IOASID set and allocation APIs
Quoted 10 lines · Kevin
Quoted 3 lines
From: Jason Gunthorpe <[email protected]>
Sent: Wednesday, April 28, 2021 1:12 AM[...]
Quoted 5 lines
One option is VFIO can keep its group FD but nothing else will have anthing like it. However I don't much like the idea that VFIO will have a special and unique programming model to do that same things other subsystem will do. That will make it harder for userspace to implement.
Quoted 7 lines
Hi, Jason,
I have a question here. Based on discussions so far, it's clearly that the new ioasid uAPI will differ from existing VFIO uAPI a lot, e.g. ioasid- centric operations, no group fd, no incompatible domains, etc. Then I wonder how we plan to support legacy VFIO applications in this transition phase.
I suspect the VFIO group fd will have to be registered with /dev/ioasid in addition to each device if we are to retain the same model.
Quoted 5 lines
Earlier you ever mentioned the desire of directly replacing
/dev/vfio/vfiowith/dev/ioasidand having ioasid to present both VFIO and new uAPI. Doesn't it imply that we have to copy the VFIO container/group semantics into/dev/ioasidalthough it's a special programming model only for VFIO?
I gave that as a something to think about, if it doesn't work out then it is just a bad idea to discard.
Quoted 7 lines
Alternatively we could keep all the container/group legacy within VFIO and having
/dev/ioasidsupport only the new uAPI semantics. In this case VFIO will include a shim iommu backend to connect its legacy uAPI intodrivers/ioasidbackend functions for backward compatibility. Then VFIO will also support a new model which only uses its device uAPI to bind to new ioasid fd w/o using any legacy container/group/iommu uAPI. Does this sound a plan?
It may be where we end up.. Though I fear it will make it overly complex inside VFIO to access the new stuff. It would be very nice if we could see a path where VFIO insides could only deal with the in-kernel ioasid handles, whatever they are.
Jason
Re: [PATCH V4 05/18] iommu/ioasid: Redefine IOASID set and allocation APIs
In reply to Alex WilliamsonQuoted 55 lines · Alex Williamson
Quoted 27 lines · On Wed, 21 Apr 2021 20:03:01 -0300 Jason Gunthorpe…
Quoted 7 lines · Alex Williamson
Quoted 3 lines
I still expect that
VFIO_GROUP_SET_CONTAINERwill be used to connect/dev/{ioasid,vfio}to the VFIO group and all the group and device logic stays inside VFIO.But that group and device logic is also tied to the container, where the IOMMU backend is the interchangeable thing that provides the IOMMU manipulation for that container.
I think that is an area where the discussion would need to be focused.
I don't feel very prepared to have it in details, as I haven't dug into all the group and iommu micro-operation very much.
But, it does seem like the security concept that VFIO is creating with the group also has to be present in the lower iommu layer too.
With different subsystems joining devices to the same ioasid's we still have to enforce the security propery the vfio group is creating.
Quoted 3 lines
If you're using
VFIO_GROUP_SET_CONTAINERto associate a group to a/dev/ioasid, then you're really either taking that group outside of vfio or you're re-implementing group management in/dev/ioasid.This sounds right.
Quoted 8 lines
Quoted 6 lines
Quoted 2 lines
Everything can be switched to
ioasid_containerall down the line. If it wasn't for PPC this looks fairly simple.At what point is it no longer vfio? I'd venture to say that replacing the container rather than invoking a different IOMMU backend is that point.
sorry, which is no longer vfio?
I'm suggesting that if we're replacing the container/group model with an ioasid then we're effectively creating a new thing that really only retains the vfio device uapi.
Quoted 12 lines
Quoted 9 lines
Quoted 5 lines
Since getting rid of PPC looks a bit hard, we'd be stuck with accepting a
/dev/ioasidand then immediately wrappering it in avfio_containeran shimming it through avfio_iommu_ops. It is not ideal at all, but in my look around I don't see a major problem if type1 implementation is moved to live under/dev/ioasid.But type1 is \just\ an IOMMU backend, not "
/dev/vfio". Given that nobody flinched at removing NVLink support, maybe just deprecate SPAPR now and see if anyone objects ;)Would simplify this project, but I wonder :)
Quoted 7 lines
Quoted 3 lines
In any event, it does look like today we'd expect the SPAPR stuff would be done through the normal iommu APIs, perhaps enhanced a bit, which makes me suspect an enhanced type1 can implement SPAPR.
David Gibson has argued for some time that SPAPR could be handled via a converged type1 model. We has mapped that out at one point, essentially a "type2", but neither of us had any bandwidth to pursue it.
Right. The sPAPR TCE backend is kind of an unfortunate accident of history. We absolutely could do a common interface, but no-one's had time to work on it.
Quoted 4 lines
Quoted 4 lines
I say this because the SPAPR looks quite a lot like PASID when it has APIs for allocating multiple tables and other things. I would be interested to hear someone from IBM talk about what it is doing and how it doesn't fit into today's IOMMU API.
Hm. I don't think it's really like PASID. Just like Type1, the TCE backend represents a single DMA address space which all devices in the container will see at all times. The difference is that there can be multiple (well, 2) "windows" of valid IOVAs within that address space. Each window can have a different TCE (page table) layout. For kernel drivers, a smallish translated window at IOVA 0 is used for 32-bit devices, and a large direct mapped (no page table) window is created at a high IOVA for better performance with 64-bit DMA capable devices.
With the VFIO backend we create (but don't populate) a similar smallish 32-bit window, userspace can create its own secondary window if it likes, though obvious for userspace use there will always be a page table. Userspace can choose the total size (but not address), page size and to an extent the page table format of the created window. Note that the TCE page table format is *not* the same as the POWER CPU core's page table format. Userspace can also remove the default small window and create its own.
The second wrinkle is pre-registration. That lets userspace register certain userspace VA ranges (*not* IOVA ranges) as being the only ones allowed to be mapped into the IOMMU. This is a performance optimization, because on pre-registration we also pre-account memory that will be effectively locked by DMA mappings, rather than doing it at DMA map and unmap time.
This came about because POWER guests always contain a vIOMMU. That (combined with the smallish default IOVA window) means that DMA maps and unmaps can become an important bottleneck, rather than being basically a small once-off cost when qemu maps all of guest memory into the IOMMU. That's optimized with a special interlink between KVM and VFIO that accelerates the guest-initiated maps/unmap operations. However, it's not feasible to do the accounting in that fast path, hence the need for the pre-registration.
Quoted 22 lines
[Cc David, Alexey]
Quoted 19 lines
It is very old and the iommu world has advanced tremendously lately, maybe I'm too optimisitic?
Quoted 11 lines
Quoted 7 lines
We end up with a
ioasid.hthat basically has thevfio_iommu_type1code lightly recast into some 'structiommu_container' and a set of ioasid_* function entry points that followvfio_iommu_driver_ops_type1:
ioasid_attach_group
ioasid_detatch_group
ioasid_<something about user pages> ioasid_read/ioasid_writeAgain, this looks like a vfio IOMMU backend. What are we accomplishing by replacing
/dev/vfiowith/dev/ioasidversus some manipulation ofVFIO_SET_IOMMUaccepting a/dev/ioasidfd?The point of all of this is to make the user api for the IOMMU cross-subsystem. It is not a vfio IOMMU backend, it is moving the IOMMU abstraction from VFIO into the iommu framework and giving the iommu framework a re-usable user API.
I like the idea of a common DMA/IOMMU handling system across platforms. However in order to be efficiently usable for POWER it will need to include multiple windows, allowing the user to change those windows and something like pre-registration to amortize accounting costs for heavy vIOMMU load.
Well... possibly we can do without the pre-reg now that 32-bit DMA limited devics are less common, as are POWER8 systems. With modern devices and modern kernels a guest is likely to use a single large 64-bit secondary window mapping all guest RAM, so the vIOMMU bottleneck shouldn't be such an issue.
Quoted 168 lines
Right, but I don't see that implies it cannot work within the vfio IOMMU model. Currently when an IOMMU is set, the
/dev/vfio/vfiocontainer becomes a conduit for file ops from the container to be forwarded to the IOMMU. But that's in part because the user doesn't have another object to interact with the IOMMU. It's entirely possible that with an ioasid shim, the user would continue to interact directly with the/dev/ioasidfd for IOMMU manipulation and only useVFIO_SET_IOMMUto associate a vfio container to that ioasid.Quoted 4 lines
My ideal outcome would be for VFIO to use only the new iommu/ioasid API and have no iommu pluggability at all. The iommu subsystem provides everything needed to VFIO, and provides it equally to VDPA and everything else.
As above, we don't necessarily need to have the vfio container be the access mechanism for the IOMMU, it can become just an means to association the container with an IOMMU. This has quite a few transitional benefits.
Quoted 7 lines
drivers/vfio/becomes primarily about 'structvfio_device' and everything related to its IOCTL interface.
drivers/iommuandioasid.cbecome all about a pluggable IOMMU interface, including a uAPI for it.IMHO it makes a high level sense, though it may be a pipe dream.
This is where we've dissolved all but the vfio device uapi, which suggests the group and container model were never necessary and I'm not sure exactly what that uapi looks like. We currently make use of an IOMMU api that is group aware, but that awareness extends out to the vfio uapi.
Quoted 5 lines
Quoted 5 lines
Quoted 4 lines
If we have this, and
/dev/ioasidimplements the legacy IOCTLs, then/dev/vfio==/dev/ioasidand we can compile outvfio_fopsand related fromvfio.cand tellioasid.cto create/dev/vfioinstead using the ops it owns.Quoted 7 lines
Quoted 3 lines
Why would we want
/dev/ioasidto implement legacy ioctls instead of simply implementing an interface to allow/dev/ioasidto be used as a vfio IOMMU backend?Only to make our own migration easier. I'd imagine everyone would want to sit down and design this new clear ioasid API that can co-exist on
/dev/ioasidwith the legacy once.vfio really just wants to be able to attach groups to an address space to consider them isolated, everything else about the IOMMU API could happen via a new ioasid file descriptor representing that context, ie. vfio handles the group ownership and device access, ioasid handles the actual mappings.
Quoted 15 lines
Quoted 3 lines
The pseudo code above really suggests you do want to remove
/dev/vfio/vfio, but this is only one of the IOMMU backends for vfio, so I can't quite figure out if we're talking past each other.I'm not quite sure what you mean by "one of the IOMMU backends?" You mean type1, right?
Quoted 7 lines
As I expressed in another thread, type1 has a lot of shortcomings. The mapping interface leaves userspace trying desperately to use statically mapped buffers because the map/unmap latency is too high. We have horrible issues with duplicate locked page accounting across containers. It suffers pretty hard from feature creep in various areas. A new IOMMU backend is an opportunity to redesign some of these things.
Quoted 5 lines
Sure, but also those kinds of transformational things go alot better if you can smoothly go from the old to the new and have technical co-existance in side the kernel. Having a shim that maps the old APIs to new APIs internally to Linux helps keep the implementation from becoming too bogged down with compatibility.
I'm afraid
/dev/ioasidproviding type1 compatibility would be just that.Quoted 5 lines
Quoted 4 lines
The IOMMU group also abstracts isolation and visibility relative to DMA. For example, in a PCIe topology a multi-function device may not have isolation between functions, but each requester ID is visible to the IOMMU.
Quoted 8 lines
Okay, I'm glad I have this all right in my head, as I was pretty sure this was what the group was about.
My next question is why do we have three things as a FD: group, device and container (aka IOMMU interface)?
Do we have container because the
/dev/vfio/vfiocan hold only a single page table so we need to swap containers sometimes?The container represents an IOMMU address space, which can be shared by multiple groups, where each group may contain one or more devices. Swapping a container would require releasing all the devices (the user cannot have access to a non-isolated device), then a group could be moved from one container to another.
Quoted 6 lines
If we start from a clean sheet and make a sketch..
/dev/ioasidis the IOMMU control interface. It can create multiple IOASIDs that have page tables and it can manipulate those page tables. Each IOASID is identified by some number.Quoted 8 lines
struct vfio_device/vdpa_device/etc are consumers of
/dev/ioasidWhen a device attaches to an ioasid userspace gives VFIO/VDPA the ioasid FD and the ioasid # in the FD.
The security rule for isolation is that once a device is attached to a
/dev/ioasidfd then all other devices in that security group must be attached to the same ioasid FD or left unused.Sounds like a group... Note also that if those other devices are not isolated from the user's device, the user could manipulate "unused" devices via DMA. So even unused devices should be within the same IOMMU context... thus attaching groups to IOMMU domains.
Quoted 3 lines
Thus
/dev/ioasidalso becomes the unit of security and the IOMMU subsystem level becomes aware of and enforces the group security rules. Userspace does not need to "see" the groupWhat tools does userspace have to understand isolation of individual devices without groups?
Quoted 32 lines
In sketch it would be like
ioasid_fd=open("/dev/ioasid");
vfio_device_fd=open("/dev/vfio/device0")
vdpa_device_fd=open("/dev/vdpa/device0")
ioctl(vifo_device_fd, JOIN_IOASID_FD, ioasifd)
ioctl(vdpa_device_fd, JOIN_IOASID_FD, ioasifd)
gpa_ioasid_id=ioctl(ioasid_fd, CREATE_IOASID, ..)ioctl(ioasid_fd, SET_IOASID_PAGE_TABLES, ..)
ioctl(vfio_device, ATTACH_IOASID, gpa_ioasid_id)ioctl(vpda_device, ATTACH_IOASID, gpa_ioasid_id).. both VDPA and VFIO see the guest physical map and the kernel has
enough info that both could use the same IOMMU page table structure ..// Guest viommu turns off bypass mode for the vfio device
ioctl(vfio_device, DETATCH_IOASID)// Guest viommu creates a new page table
rid_ioasid_id=ioctl(ioasid_fd, CREATE_IOASID, ..)ioctl(ioasid_fd, SET_IOASID_PAGE_TABLES, ..)// Guest viommu links the new page table to the RID
ioctl(vfio_device, ATTACH_IOASID, rid_ioasid_id)The group security concept becomes implicit and hidden from the uAPI.
JOIN_IOASID_FDimplicitly finds the device's group inside the kernel and requires that all members of the group be joined only to thisioasid_fd.Quoted 5 lines
Essentially we discover the group from the device instead of the device from the group.
Where does it fall down compared to the three FD version we have today?
The group concept is explicit today because how does userspace learn about implicit dependencies between devices? For example, if the user has a conventional PCI bus with a couple devices on it, how do they understand that those devices cannot be assigned to separate userspace drivers? The group fd fills that gap. Thanks,
Alex
Signature · 4 lines
David Gibson | I'll have my music baroque, and my code david AT gibson.dropbear.id.au | minimalist, thank you. NOT _the_ _other_ | _way_ _around_! http://www.ozlabs.org/~dgibson
Re: [PATCH V4 05/18] iommu/ioasid: Redefine IOASID set and allocation APIs
Quoted 14 lines · David Gibson
Quoted 4 lines
Quoted 4 lines
I say this because the SPAPR looks quite a lot like PASID when it has APIs for allocating multiple tables and other things. I would be interested to hear someone from IBM talk about what it is doing and how it doesn't fit into today's IOMMU API.
Hm. I don't think it's really like PASID. Just like Type1, the TCE backend represents a single DMA address space which all devices in the container will see at all times. The difference is that there can be multiple (well, 2) "windows" of valid IOVAs within that address space. Each window can have a different TCE (page table) layout. For kernel drivers, a smallish translated window at IOVA 0 is used for 32-bit devices, and a large direct mapped (no page table) window is created at a high IOVA for better performance with 64-bit DMA capable devices.
Quoted 8 lines
With the VFIO backend we create (but don't populate) a similar smallish 32-bit window, userspace can create its own secondary window if it likes, though obvious for userspace use there will always be a page table. Userspace can choose the total size (but not address), page size and to an extent the page table format of the created window. Note that the TCE page table format is *not* the same as the POWER CPU core's page table format. Userspace can also remove the default small window and create its own.
So what do you need from the generic API? I'd suggest if userspace passes in the required IOVA range it would benefit all the IOMMU drivers to setup properly sized page tables and PPC could use that to drive a single window. I notice this is all DPDK did to support TCE.
Quoted 6 lines
The second wrinkle is pre-registration. That lets userspace register certain userspace VA ranges (*not* IOVA ranges) as being the only ones allowed to be mapped into the IOMMU. This is a performance optimization, because on pre-registration we also pre-account memory that will be effectively locked by DMA mappings, rather than doing it at DMA map and unmap time.
This feels like nesting IOASIDs to me, much like a vPASID.
The pre-registered VA range would be the root of the tree and the vIOMMU created ones would be children of the tree. This could allow the map operations of the child to refer to already prepped physical memory held in the root IOASID avoiding the GUP/etc cost.
Seems fairly genericish, though I'm not sure about the kvm linkage..
Quoted 5 lines
I like the idea of a common DMA/IOMMU handling system across platforms. However in order to be efficiently usable for POWER it will need to include multiple windows, allowing the user to change those windows and something like pre-registration to amortize accounting costs for heavy vIOMMU load.
I have a feeling /dev/ioasid is going to end up with some HW specific escape hatch to create some HW specific IOASID types and operate on them in a HW specific way.
However, what I would like to see is that something simple like DPDK can have a single implementation - POWER should implement the standard operations and map them to something that will work for it.
As an ideal, only things like the HW specific qemu vIOMMU driver should be reaching for all the special stuff.
In this way the kernel IOMMU driver and the qemu user vIOMMU driver would form something of a classical split user/kernel driver pattern.
Jason
Re: [PATCH V4 05/18] iommu/ioasid: Redefine IOASID set and allocation APIs
Quoted 25 lines · Jason Gunthorpe
Quoted 14 lines · David Gibson
Quoted 4 lines
Quoted 4 lines
I say this because the SPAPR looks quite a lot like PASID when it has APIs for allocating multiple tables and other things. I would be interested to hear someone from IBM talk about what it is doing and how it doesn't fit into today's IOMMU API.
Hm. I don't think it's really like PASID. Just like Type1, the TCE backend represents a single DMA address space which all devices in the container will see at all times. The difference is that there can be multiple (well, 2) "windows" of valid IOVAs within that address space. Each window can have a different TCE (page table) layout. For kernel drivers, a smallish translated window at IOVA 0 is used for 32-bit devices, and a large direct mapped (no page table) window is created at a high IOVA for better performance with 64-bit DMA capable devices.
Quoted 8 lines
With the VFIO backend we create (but don't populate) a similar smallish 32-bit window, userspace can create its own secondary window if it likes, though obvious for userspace use there will always be a page table. Userspace can choose the total size (but not address), page size and to an extent the page table format of the created window. Note that the TCE page table format is *not* the same as the POWER CPU core's page table format. Userspace can also remove the default small window and create its own.
Quoted 4 lines
So what do you need from the generic API? I'd suggest if userspace passes in the required IOVA range it would benefit all the IOMMU drivers to setup properly sized page tables and PPC could use that to drive a single window. I notice this is all DPDK did to support TCE.
Yes. My proposed model for a unified interface would be that when you create a new container/IOASID, *no* IOVAs are valid. Before you can map anything you would have to create a window with specified base, size, pagesize (probably some flags for extension, too). That could fail if the backend IOMMU can't handle that IOVA range, it could be a backend no-op if the requested window lies within a fixed IOVA range the backend supports, or it could actually reprogram the back end for the new window (such as for POWER TCEs). Regardless of the hardware, attempts to map outside the created window(s) would be rejected by software.
I expect we'd need some kind of query operation to expose limitations on the number of windows, addresses for them, available pagesizes etc.
Quoted 7 lines
Quoted 6 lines
The second wrinkle is pre-registration. That lets userspace register certain userspace VA ranges (*not* IOVA ranges) as being the only ones allowed to be mapped into the IOMMU. This is a performance optimization, because on pre-registration we also pre-account memory that will be effectively locked by DMA mappings, rather than doing it at DMA map and unmap time.
Quoted 6 lines
This feels like nesting IOASIDs to me, much like a vPASID.
The pre-registered VA range would be the root of the tree and the vIOMMU created ones would be children of the tree. This could allow the map operations of the child to refer to already prepped physical memory held in the root IOASID avoiding the GUP/etc cost.
Huh... I never thought of it that way, but yeah, that sounds like it could work. More elegantly than the current system in fact.
Quoted 1 line
Seems fairly genericish, though I'm not sure about the kvm linkage..
I think it should be doable. We'd basically need to give KVM a handle on the parent AS, and the child AS, and the guest side handle (what PAPR calls a "Logical IO Bus Number" - liobn). KVM would then translate H_PUT_TCE etc. hypercalls on that liobn into calls into the IOMMU subsystem to map bits of the parent AS into the child. We'd probably have to have some requirements that either parent AS is identity-mapped to a subset of the userspace AS (effectively what we have now) or that parent AS is the same as guest physical address. Not sure which would work better.
Quoted 10 lines
Quoted 5 lines
I like the idea of a common DMA/IOMMU handling system across platforms. However in order to be efficiently usable for POWER it will need to include multiple windows, allowing the user to change those windows and something like pre-registration to amortize accounting costs for heavy vIOMMU load.
I have a feeling
/dev/ioasidis going to end up with some HW specific escape hatch to create some HW specific IOASID types and operate on them in a HW specific way.
Quoted 6 lines
However, what I would like to see is that something simple like DPDK can have a single implementation - POWER should implement the standard operations and map them to something that will work for it.
As an ideal, only things like the HW specific qemu vIOMMU driver should be reaching for all the special stuff.
I'm hoping we can even avoid that, usually. With the explicitly created windows model I propose above, it should be able to: qemu will create the windows according to the IOVA windows the guest platform expects to see and they either will or won't work on the host platform IOMMU. If they do, generic maps/unmaps should be sufficient. If they don't well, the host IOMMU simply cannot emulate the vIOMMU so you're out of luck anyway.
Quoted 5 lines
In this way the kernel IOMMU driver and the qemu user vIOMMU driver would form something of a classical split user/kernel driver pattern.
Jason
Signature · 4 lines
David Gibson | I'll have my music baroque, and my code david AT gibson.dropbear.id.au | minimalist, thank you. NOT _the_ _other_ | _way_ _around_! http://www.ozlabs.org/~dgibson
Re: [PATCH V4 05/18] iommu/ioasid: Redefine IOASID set and allocation APIs
Quoted 2 lines · David Gibson
Yes. My proposed model for a unified interface would be that when you create a new container/IOASID, *no* IOVAs are valid.
Hurm, it is quite tricky. All IOMMUs seem to have a dead zone around the MSI window, so negotiating this all in a general way is not going to be a very simple API.
To be general it would be nicer to say something like 'I need XXGB of IOVA space' 'I need 32 bit IOVA space' etc and have the kernel return ranges that sum up to at least that big. Then the kernel can do its all its optimizations.
I guess you are going to say that the qemu PPC vIOMMU driver needs more exact control..
Quoted 2 lines
I expect we'd need some kind of query operation to expose limitations on the number of windows, addresses for them, available pagesizes etc.
Is page size an assumption that hugetlbfs will always be used for backing memory or something?
Quoted 10 lines
Quoted 2 lines
As an ideal, only things like the HW specific qemu vIOMMU driver should be reaching for all the special stuff.
I'm hoping we can even avoid that, usually. With the explicitly created windows model I propose above, it should be able to: qemu will create the windows according to the IOVA windows the guest platform expects to see and they either will or won't work on the host platform IOMMU. If they do, generic maps/unmaps should be sufficient. If they don't well, the host IOMMU simply cannot emulate the vIOMMU so you're out of luck anyway.
It is not just P9 that has special stuff, and this whole area of PASID seems to be quite different on every platform
If things fit very naturally and generally then maybe, but I've been down this road before of trying to make a general description of a group of very special HW. It ended in tears after 10 years when nobody could understand the "general" API after it was Frankenstein'd up with special cases for everything. Cautionary tale
There is a certain appeal to having some 'PPC_TCE_CREATE_SPECIAL_IOASID' entry point that has a wack of extra information like windows that can be optionally called by the viommu driver and it remains well defined and described.
Jason
Re: [PATCH V4 05/18] iommu/ioasid: Redefine IOASID set and allocation APIs
Quoted 5 lines · Jason Gunthorpe
Quoted 2 lines · David Gibson
Yes. My proposed model for a unified interface would be that when you create a new container/IOASID, *no* IOVAs are valid.
Quoted 8 lines
Hurm, it is quite tricky. All IOMMUs seem to have a dead zone around the MSI window, so negotiating this all in a general way is not going to be a very simple API.
To be general it would be nicer to say something like 'I need XXGB of IOVA space' 'I need 32 bit IOVA space' etc and have the kernel return ranges that sum up to at least that big. Then the kernel can do its all its optimizations.
Ah, yes, sorry. We do need an API that lets the kernel make more of the decisions too. For userspace drivers it would generally be sufficient to just ask for XXX size of IOVA space wherever you can get it. Handling guests requires more precision. So, maybe a request interface with a bunch of hint variables and a matching set of MAP_FIXED-like flags to assert which ones aren't negotiable.
Quoted 2 lines
I guess you are going to say that the qemu PPC vIOMMU driver needs more exact control..
*Every* vIOMMU driver needs more exact control. The guest drivers will expect to program the guest devices with IOVAs matching the guest platform's IOMMU model. Therefore the backing host IOMMU has to be programmed to respond to those IOVAs. If it can't be, there's no way around it, and you want to fail out early. With this model that will happen when qemu (say) requests the host IOMMU window(s) to match the guest's expected IOVA ranges.
Actually, come to that even guests without a vIOMMU need more exact
control: they'll expect IOVA to match GPA, so if your host IOMMU can't be set up translate the full range of GPAs, again, you're out of luck.
The only reason x86 has been able to ignore this is that the assumption has been that all IOMMUs can translate IOVAs from 0..<a big enough number for any reasonable RAM size>. Once you really start to look at what the limits are, you need the exact window control I'm describing.
Quoted 5 lines
Quoted 2 lines
I expect we'd need some kind of query operation to expose limitations on the number of windows, addresses for them, available pagesizes etc.
Is page size an assumption that hugetlbfs will always be used for backing memory or something?
So for TCEs (and maybe other IOMMUs out there), the IO page tables are independent of the CPU page tables. They don't have the same format, and they don't necessarily have the same page size. In the case of a bare metal kernel working in physical addresses they can use that TCE page size however they like. For userspace you get another layer of complexity. Essentially to implement things correctly the backing IOMMU needs to have a page size granularity that's the minimum of whatever granularity the userspace or guest driver expects and the host page size backing the memory.
Quoted 20 lines
Quoted 10 lines
Quoted 2 lines
As an ideal, only things like the HW specific qemu vIOMMU driver should be reaching for all the special stuff.
I'm hoping we can even avoid that, usually. With the explicitly created windows model I propose above, it should be able to: qemu will create the windows according to the IOVA windows the guest platform expects to see and they either will or won't work on the host platform IOMMU. If they do, generic maps/unmaps should be sufficient. If they don't well, the host IOMMU simply cannot emulate the vIOMMU so you're out of luck anyway.
It is not just P9 that has special stuff, and this whole area of PASID seems to be quite different on every platform
If things fit very naturally and generally then maybe, but I've been down this road before of trying to make a general description of a group of very special HW. It ended in tears after 10 years when nobody could understand the "general" API after it was Frankenstein'd up with special cases for everything. Cautionary tale
Quoted 4 lines
There is a certain appeal to having some '
PPC_TCE_CREATE_SPECIAL_IOASID' entry point that has a wack of extra information like windows that can be optionally called by the viommu driver and it remains well defined and described.
Windows really aren't ppc specific. They're absolutely there on x86 and everything else as well - it's just that people are used to having a window at 0..<something largish> that you can often get away with treating it sloppily.
Signature · 4 lines
David Gibson | I'll have my music baroque, and my code david AT gibson.dropbear.id.au | minimalist, thank you. NOT _the_ _other_ | _way_ _around_! http://www.ozlabs.org/~dgibson
Re: [PATCH V4 05/18] iommu/ioasid: Redefine IOASID set and allocation APIs
Quoted 9 lines · David Gibson
Quoted 4 lines
There is a certain appeal to having some '
PPC_TCE_CREATE_SPECIAL_IOASID' entry point that has a wack of extra information like windows that can be optionally called by the viommu driver and it remains well defined and described.Windows really aren't ppc specific. They're absolutely there on x86 and everything else as well - it's just that people are used to having a window at 0..<something largish> that you can often get away with treating it sloppily.
My point is this detailed control seems to go on to more than just windows. As you say the vIOMMU is emulating specific HW that needs to have kernel interfaces to match it exactly.
I'm remarking that trying to unify every HW IOMMU implementation that ever has/will exist into a generic API complete enough to allow the vIOMMU to be created is likely to result in an API too complicated to understand..
Jason
Re: [PATCH V4 05/18] iommu/ioasid: Redefine IOASID set and allocation APIs
Quoted 6 lines · Jason Gunthorpe
Quoted 5 lines · David Gibson
Quoted 4 lines
There is a certain appeal to having some '
PPC_TCE_CREATE_SPECIAL_IOASID' entry point that has a wack of extra information like windows that can be optionally called by the viommu driver and it remains well defined and described.
Quoted 8 lines
Quoted 4 lines
Windows really aren't ppc specific. They're absolutely there on x86 and everything else as well - it's just that people are used to having a window at 0..<something largish> that you can often get away with treating it sloppily.
My point is this detailed control seems to go on to more than just windows. As you say the vIOMMU is emulating specific HW that needs to have kernel interfaces to match it exactly.
It's really not that bad. The case of emulating the PAPR vIOMMU on something else is relatively easy, because all updates to the IO page tables go through hypercalls. So, as long as the backend IOMMU can map all the IOVAs that the guest IOMMU can, then qemu's implementation of those hypercalls just needs to put an equivalent mapping in the backend, which it can do with a generic VFIO_DMA_MAP.
vIOMMUs with page tables in guest memory are harder, but only really in the usual ways that a vIOMMU of that type is harder (needs cache mode or whatever). At whatever point you need to shadow from the guest IO page tables to the host backend, you can again do that with generic maps, as long as the backend supports the necessary IOVAs, and has an IO page size that's equal to or a submultiple of the vIOMMU page size.
Quoted 4 lines
I'm remarking that trying to unify every HW IOMMU implementation that ever has/will exist into a generic API complete enough to allow the vIOMMU to be created is likely to result in an API too complicated to understand..
Maybe not every one, but I think we can get a pretty wide range with a reasonable interface. Explicitly handling IOVA windows does most of it. And we kind of need to handle that anyway to expose what ranges the IOMMU is capable of translating anyway. I think making handling valid IOVA windows explicit makes things simpler than having per-backend-family interfaces to expose the limits of their translation ranges, which is what's likely to happen without it.
Signature · 4 lines
David Gibson | I'll have my music baroque, and my code david AT gibson.dropbear.id.au | minimalist, thank you. NOT _the_ _other_ | _way_ _around_! http://www.ozlabs.org/~dgibson
Re: [PATCH V4 05/18] iommu/ioasid: Redefine IOASID set and allocation APIs
Quoted 16 lines · David Gibson
Quoted 6 lines · Jason Gunthorpe
Quoted 5 lines · David Gibson
Quoted 4 lines
There is a certain appeal to having some '
PPC_TCE_CREATE_SPECIAL_IOASID' entry point that has a wack of extra information like windows that can be optionally called by the viommu driver and it remains well defined and described.Quoted 8 lines
Quoted 4 lines
Windows really aren't ppc specific. They're absolutely there on x86 and everything else as well - it's just that people are used to having a window at 0..<something largish> that you can often get away with treating it sloppily.
My point is this detailed control seems to go on to more than just windows. As you say the vIOMMU is emulating specific HW that needs to have kernel interfaces to match it exactly.
Quoted 6 lines
It's really not that bad. The case of emulating the PAPR vIOMMU on something else is relatively easy, because all updates to the IO page tables go through hypercalls. So, as long as the backend IOMMU can map all the IOVAs that the guest IOMMU can, then qemu's implementation of those hypercalls just needs to put an equivalent mapping in the backend, which it can do with a generic
VFIO_DMA_MAP.
So you also want the PAPR vIOMMU driver to run on, say, an ARM IOMMU?
Quoted 7 lines
vIOMMUs with page tables in guest memory are harder, but only really in the usual ways that a vIOMMU of that type is harder (needs cache mode or whatever). At whatever point you need to shadow from the guest IO page tables to the host backend, you can again do that with generic maps, as long as the backend supports the necessary IOVAs, and has an IO page size that's equal to or a submultiple of the vIOMMU page size.
But this definitely all becomes HW specific.
For instance I want to have an ARM vIOMMU driver it needs to do some
ret = ioctl(ioasid_fd, CREATE_NESTED_IOASID, [page table format is ARMvXXX]) if (ret == -EOPNOTSUPP)
ret = ioctl(ioasid_fd, CREATE_NORMAL_IOASID, ..)
// and do completely different and more expensive emulation
I can get a little bit of generality, but at the end of the day the IOMMU must create a specific HW layout of the nested page table, if it can't, it can't.
Quoted 7 lines
Quoted 4 lines
I'm remarking that trying to unify every HW IOMMU implementation that ever has/will exist into a generic API complete enough to allow the vIOMMU to be created is likely to result in an API too complicated to understand..
Maybe not every one, but I think we can get a pretty wide range with a reasonable interface.
It sounds like a reasonable guideline is if the feature is actually general to all IOMMUs and can be used by qemu as part of a vIOMMU emulation when compatible vIOMMU HW is not available.
Having 'requested window' support that isn't actually implemented in every IOMMU is going to mean the PAPR vIOMMU emulation won't work, defeating the whole point of making things general?
Jason
Re: [PATCH V4 05/18] iommu/ioasid: Redefine IOASID set and allocation APIs
Quoted 17 lines · Jason Gunthorpe
Quoted 16 lines · David Gibson
Quoted 14 lines · Jason Gunthorpe
Quoted 9 lines · David Gibson
Quoted 4 lines
There is a certain appeal to having some '
PPC_TCE_CREATE_SPECIAL_IOASID' entry point that has a wack of extra information like windows that can be optionally called by the viommu driver and it remains well defined and described.Windows really aren't ppc specific. They're absolutely there on x86 and everything else as well - it's just that people are used to having a window at 0..<something largish> that you can often get away with treating it sloppily.
My point is this detailed control seems to go on to more than just windows. As you say the vIOMMU is emulating specific HW that needs to have kernel interfaces to match it exactly.
Quoted 8 lines
Quoted 6 lines
It's really not that bad. The case of emulating the PAPR vIOMMU on something else is relatively easy, because all updates to the IO page tables go through hypercalls. So, as long as the backend IOMMU can map all the IOVAs that the guest IOMMU can, then qemu's implementation of those hypercalls just needs to put an equivalent mapping in the backend, which it can do with a generic
VFIO_DMA_MAP.So you also want the PAPR vIOMMU driver to run on, say, an ARM IOMMU?
This is a good feature in general when let's say there is a linux supported device which has a proprietary device firmware update tool which only exists as an x86 binary and your hardware is not x86 - running qemu + vfio in full emulation would provide a way to run the tool to update a physical device.
Signature · 1 line
Alexey
Re: [PATCH V4 05/18] iommu/ioasid: Redefine IOASID set and allocation APIs
Quoted 4 lines · Alexey Kardashevskiy
This is a good feature in general when let's say there is a linux supported device which has a proprietary device firmware update tool which only exists as an x86 binary and your hardware is not x86 - running qemu + vfio in full emulation would provide a way to run the tool to update a physical device.
That specific use case doesn't really need a vIOMMU though, does it?
Jason
Re: [PATCH V4 05/18] iommu/ioasid: Redefine IOASID set and allocation APIs
Quoted 8 lines · Jason Gunthorpe
Quoted 4 lines · Alexey Kardashevskiy
This is a good feature in general when let's say there is a linux supported device which has a proprietary device firmware update tool which only exists as an x86 binary and your hardware is not x86 - running qemu + vfio in full emulation would provide a way to run the tool to update a physical device.
That specific use case doesn't really need a vIOMMU though, does it?
Possibly not, but the mechanics needed to do vIOMMU on different host IOMMU aren't really different from what you need for a no-vIOMMU guest. With a vIOMMU you need to map guest IOVA space into the host IOVA space. With no no-vIOMMU you need to map guest physical addresses into the host IOVA space. In either case the GPA/gIOVA to userspace and userspace to HPA mappings are basically arbitrary.
Signature · 4 lines
David Gibson | I'll have my music baroque, and my code david AT gibson.dropbear.id.au | minimalist, thank you. NOT _the_ _other_ | _way_ _around_! http://www.ozlabs.org/~dgibson
Re: [PATCH V4 05/18] iommu/ioasid: Redefine IOASID set and allocation APIs
Quoted 8 lines · David Gibson
Quoted 7 lines · Jason Gunthorpe
Quoted 4 lines · Alexey Kardashevskiy
This is a good feature in general when let's say there is a linux supported device which has a proprietary device firmware update tool which only exists as an x86 binary and your hardware is not x86 - running qemu + vfio in full emulation would provide a way to run the tool to update a physical device.
Quoted 5 lines
Quoted 1 line
That specific use case doesn't really need a vIOMMU though, does it?
Possibly not, but the mechanics needed to do vIOMMU on different host IOMMU aren't really different from what you need for a no-vIOMMU guest.
For very simple vIOMMUs this might be true, but this new features of nesting PASID, migration, etc, etc all make the vIOMMU complicated and emuluating it completely alot harder.
Stuffing a vfio-pci into a guest and creating a physical map using a single IOASID is comparably trivial.
Jason
Re: [PATCH V4 05/18] iommu/ioasid: Redefine IOASID set and allocation APIs
Quoted 11 lines · Jason Gunthorpe
Quoted 10 lines · David Gibson
Quoted 8 lines · Jason Gunthorpe
Quoted 4 lines · Alexey Kardashevskiy
This is a good feature in general when let's say there is a linux supported device which has a proprietary device firmware update tool which only exists as an x86 binary and your hardware is not x86 - running qemu + vfio in full emulation would provide a way to run the tool to update a physical device.
That specific use case doesn't really need a vIOMMU though, does it?
Quoted 7 lines
Quoted 3 lines
Possibly not, but the mechanics needed to do vIOMMU on different host IOMMU aren't really different from what you need for a no-vIOMMU guest.
For very simple vIOMMUs this might be true, but this new features of nesting PASID, migration, etc, etc all make the vIOMMU complicated and emuluating it completely alot harder.
Well, sure, emulating a complex vIOMMU is complex. But "very simple vIOMMUs" covers the vast majority of currently deployed hardware, and several are already emulated by qemu.
Quoted 2 lines
Stuffing a vfio-pci into a guest and creating a physical map using a single IOASID is comparably trivial.
Note that for PAPR (POWER guest) systems this is not an option: the PAPR platform *always* has a vIOMMU.
Signature · 4 lines
David Gibson | I'll have my music baroque, and my code david AT gibson.dropbear.id.au | minimalist, thank you. NOT _the_ _other_ | _way_ _around_! http://www.ozlabs.org/~dgibson
Re: [PATCH V4 05/18] iommu/ioasid: Redefine IOASID set and allocation APIs
In reply to Jason GunthorpeQuoted 17 lines · Jason Gunthorpe
Quoted 16 lines · David Gibson
Quoted 14 lines · Jason Gunthorpe
Quoted 9 lines · David Gibson
Quoted 4 lines
There is a certain appeal to having some '
PPC_TCE_CREATE_SPECIAL_IOASID' entry point that has a wack of extra information like windows that can be optionally called by the viommu driver and it remains well defined and described.Windows really aren't ppc specific. They're absolutely there on x86 and everything else as well - it's just that people are used to having a window at 0..<something largish> that you can often get away with treating it sloppily.
My point is this detailed control seems to go on to more than just windows. As you say the vIOMMU is emulating specific HW that needs to have kernel interfaces to match it exactly.
Quoted 8 lines
Quoted 6 lines
It's really not that bad. The case of emulating the PAPR vIOMMU on something else is relatively easy, because all updates to the IO page tables go through hypercalls. So, as long as the backend IOMMU can map all the IOVAs that the guest IOMMU can, then qemu's implementation of those hypercalls just needs to put an equivalent mapping in the backend, which it can do with a generic
VFIO_DMA_MAP.So you also want the PAPR vIOMMU driver to run on, say, an ARM IOMMU?
Well, I don't want to preclude it in the API. I'm not sure about that specific example, but in most cases it should be possible to run the PAPR vIOMMU on an x86 IOMMU backend. Obviously only something you'd want to do for testing and experimentation, but it could be quite useful for that.
Quoted 12 lines
Quoted 7 lines
vIOMMUs with page tables in guest memory are harder, but only really in the usual ways that a vIOMMU of that type is harder (needs cache mode or whatever). At whatever point you need to shadow from the guest IO page tables to the host backend, you can again do that with generic maps, as long as the backend supports the necessary IOVAs, and has an IO page size that's equal to or a submultiple of the vIOMMU page size.
But this definitely all becomes HW specific.
For instance I want to have an ARM vIOMMU driver it needs to do some
Quoted 8 lines
ret =
ioctl(ioasid_fd, CREATE_NESTED_IOASID, [page table format is ARMvXXX])if (ret ==-EOPNOTSUPP)
ret =ioctl(ioasid_fd, CREATE_NORMAL_IOASID, ..)
// and do completely different and more expensive emulationI can get a little bit of generality, but at the end of the day the IOMMU must create a specific HW layout of the nested page table, if it can't, it can't.
Erm.. I don't really know how your IOASID interface works here. I'm thinking about the VFIO interface where maps and unmaps are via explicit ioctl()s, which provides an obvious point to do translation between page table formats.
But.. even if you're exposing page tables to userspace.. with hardware that has explicit support for nesting you can probably expose the hw tables directly which is great for the cases that works for. But surely for older IOMMUs which don't do nesting you must have some way of shadowing guest IO page tables to host IO page tables to translate GPA to HPA at least? If you're doing that, I don't see that converting page table format is really any harder
Quoted 8 lines
Quoted 7 lines
Quoted 4 lines
I'm remarking that trying to unify every HW IOMMU implementation that ever has/will exist into a generic API complete enough to allow the vIOMMU to be created is likely to result in an API too complicated to understand..
Maybe not every one, but I think we can get a pretty wide range with a reasonable interface.
Quoted 7 lines
It sounds like a reasonable guideline is if the feature is actually general to all IOMMUs and can be used by qemu as part of a vIOMMU emulation when compatible vIOMMU HW is not available.
Having 'requested window' support that isn't actually implemented in every IOMMU is going to mean the PAPR vIOMMU emulation won't work, defeating the whole point of making things general?
The trick is that you don't necessarily need dynamic window support in the backend to emulate it in the vIOMMU. If your backend has fixed windows, then you emulate request window as:
if (requested window is within backend windows) no-op; else return ERROR;
It might not be a theoretically complete emulation of the vIOMMU, but it can support in-practice usage. In particular it works pretty well if your backend has a nice big IOVA range (like x86 IOMMUS) but your guest platform typically uses relatively small IOVA windows. PAPR on x86 is exactly that... well.. possibly not the 64-bit window, but because of old PAPR platforms that didn't support that, we can choose not to advertise that and guests will cope.
Signature · 4 lines
David Gibson | I'll have my music baroque, and my code david AT gibson.dropbear.id.au | minimalist, thank you. NOT _the_ _other_ | _way_ _around_! http://www.ozlabs.org/~dgibson
RE: [PATCH V4 05/18] iommu/ioasid: Redefine IOASID set and allocation APIs
Quoted 13 lines
From: David Gibson <[email protected]>
Sent: Thursday, May 13, 2021 2:01 PMQuoted 6 lines
But this definitely all becomes HW specific.
For instance I want to have an ARM vIOMMU driver it needs to do some
ret = ioctl(ioasid_fd,
CREATE_NESTED_IOASID, [page table format isARMvXXX])
Quoted 4 lines
if (ret ==
-EOPNOTSUPP)
ret =ioctl(ioasid_fd, CREATE_NORMAL_IOASID, ..)// and do completely different and more expensive emulation
Quoted 9 lines
Quoted 3 lines
I can get a little bit of generality, but at the end of the day the IOMMU must create a specific HW layout of the nested page table, if it can't, it can't.
Erm.. I don't really know how your IOASID interface works here. I'm thinking about the VFIO interface where maps and unmaps are via explicit ioctl()s, which provides an obvious point to do translation between page table formats.
We are working on a draft uAPI proposal based on the discussions in this thread. expect to send it out next week.
Thanks
Kevin
Re: [PATCH V4 05/18] iommu/ioasid: Redefine IOASID set and allocation APIs
In reply to David GibsonQuoted 6 lines · David Gibson
But.. even if you're exposing page tables to userspace.. with hardware that has explicit support for nesting you can probably expose the hw tables directly which is great for the cases that works for. But surely for older IOMMUs which don't do nesting you must have some way of shadowing guest IO page tables to host IO page tables to translate GPA to HPA at least?
I expect this would be in quemu and would be part of the expensive emulation I suggested. Converting the guest's page table structure into a sequence of map/unmaps to a non-nestable IOASID.
Quoted 2 lines
If you're doing that, I don't see that converting page table format is really any harder
It isn't, but it is a completely different flow and custom from the normal HW accelerated nesting.
Quoted 7 lines
It might not be a theoretically complete emulation of the vIOMMU, but it can support in-practice usage. In particular it works pretty well if your backend has a nice big IOVA range (like x86 IOMMUS) but your guest platform typically uses relatively small IOVA windows. PAPR on x86 is exactly that... well.. possibly not the 64-bit window, but because of old PAPR platforms that didn't support that, we can choose not to advertise that and guests will cope.
So maybe this multi-window thing is generic API somehow. You'll have to check what Kevin comes up with to ensure it fits in
Jason
Re: [PATCH V4 05/18] iommu/ioasid: Redefine IOASID set and allocation APIs
In reply to Alex WilliamsonQuoted 14 lines · Alex Williamson
Quoted 10 lines · On Wed, 21 Apr 2021 14:52:03 -0300 Jason Gunthorpe <[email protected]> wrote:
Quoted 1 line · Alex Williamson
That's essentially replacing vfio-core, where I think we're more
I am only talking about
/dev/vfiohere which is basically the IOMMU interface part.I still expect that
VFIO_GROUP_SET_CONTAINERwill be used to connect/dev/{ioasid,vfio}to the VFIO group and all the group and device logic stays inside VFIO.
Quoted 7 lines
But that group and device logic is also tied to the container, where the IOMMU backend is the interchangeable thing that provides the IOMMU manipulation for that container. If you're using
VFIO_GROUP_SET_CONTAINERto associate a group to a/dev/ioasid, then you're really either taking that group outside of vfio or you're re-implementing group management in/dev/ioasid. I'd expect the transition point atVFIO_SET_IOMMU.
per my understanding, transiting at the VFIO_SET_IOMMU point makes more sense as VFIO can still have the group and device logic, which is the key concept of group granularity isolation for userspace direct access.
Signature · 2 lines
Regards, Yi Liu
Re: [PATCH V4 05/18] iommu/ioasid: Redefine IOASID set and allocation APIs
In reply to Jason GunthorpeHi Jason,
Quoted 34 lines · Jason Gunthorpe
Quoted 33 lines · Auger Eric
Hi Jason,
Quoted 11 lines · Jason Gunthorpe
Quoted 8 lines · Yi L
DMA page faults are delivered to root-complex via page request message and it is per-device according to PCIe spec. Page request handling flow is:
1) iommu driver receives a page request from device
2) iommu driver parses the page request message. Get the RID,PASID, faulted page and requested permissions etc.
3) iommu driver triggers fault handler registered by device driver withiommu_report_device_fault()Quoted 3 lines
This seems confused.
The PASID should define how to handle the page fault, not the driver.
In my series I don't use PASID at all. I am just enabling nested stage and the guest uses a single context. I don't allocate any user PASID at any point.
When there is a fault at physical level (a stage 1 fault that concerns the guest), this latter needs to be reported and injected into the guest. The vfio pci driver registers a fault handler to the iommu layer and in that fault handler it fills a circ bugger and triggers an eventfd that is listened to by the VFIO-PCI QEMU device. this latter retrives the faault from the mmapped circ buffer, it knowns which vIOMMU it is attached to, and passes the fault to the vIOMMU.
Then the vIOMMU triggers and IRQ in the guest.We are reusing the existing concepts from VFIO, region, IRQ to do that.
Quoted 8 lines
Quoted 1 line
For that use case, would you also use
/dev/ioasid?
/dev/ioasidcould do all the things you described vfio-pci as doing, it can even do them the same way you just described.Stated another way, do you plan to duplicate all of this code someday for vfio-cxl? What about for vfio-platform? ARM SMMU can be hooked to platform devices, right?
vfio regions and IRQ related APIs are common user interfaces exposed by all vfio drivers, including platform. Then the actual circular buffer implementation details can be put in a common lib.
as for the thin vfio iommu wrappers, the ones you don't like, they are implemented in type1 code.
Maybe the need for /dev/ioasid is more crying for PASID management but for the nested use case, that's not obvious to me and in your different replies, it was not crystal clear where the use case belongs to.
The redesign requirement came pretty late in the development process. The iommu user API is upstream for a while, the VFIO interfaces have been submitted a long time ago and under review for a bunch of time. Redesigning everything with a different API, undefined at this point, is a major setback for our work and will have a large impact on the introduction of features companies are looking forward, hence our frustration.
Thanks
Eric
Quoted 9 lines
I feel what you guys are struggling with is some choice in the iommu kernel APIs that cause the events to be delivered to the
pci_deviceowner, not the PASID owner.That feels solvable.
Jason
Re: [PATCH V4 05/18] iommu/ioasid: Redefine IOASID set and allocation APIs
Quoted 7 lines · Auger Eric
The redesign requirement came pretty late in the development process. The iommu user API is upstream for a while, the VFIO interfaces have been submitted a long time ago and under review for a bunch of time. Redesigning everything with a different API, undefined at this point, is a major setback for our work and will have a large impact on the introduction of features companies are looking forward, hence our frustration.
I will answer both you and Jacob at once.
This is uAPI, once it is set it can never be changed.
The kernel process and philosophy is to invest heavily in uAPI development and review to converge on the best uAPI possible.
Many past submissions have take a long time to get this right, there are several high profile uAPI examples.
Do you think this case is so special, or the concerns so minor, that it should get to bypass all of the normal process?
Ask yourself, is anyone advocating for the current direction on technical merits alone?
Certainly the patches I last saw where completely disgusting from a uAPI design perspective.
It was against the development process to organize this work the way it was done. Merging a wack of dead code to the kernel to support a uAPI vision that was never clearly articulated was a big mistake.
Start from the beginning. Invest heavily in defining a high quality uAPI. Clearly describe the uAPI to all stake holders. Break up the implementation into patch series without dead code. Make the patches. Remove the dead code this group has already added.
None of this should be a surprise. The VDPA discussion and related "what is a mdev" over a year ago made it pretty clear VFIO is not the exclusive user of "IOMMU in userspace" and that places limits on what kind of uAPIs expansion it should experience going forward.
Jason
Re: [PATCH V4 05/18] iommu/ioasid: Redefine IOASID set and allocation APIs
Hi,
Quoted 14 lines · Jason Gunthorpe
Quoted 7 lines · Auger Eric
The redesign requirement came pretty late in the development process. The iommu user API is upstream for a while, the VFIO interfaces have been submitted a long time ago and under review for a bunch of time. Redesigning everything with a different API, undefined at this point, is a major setback for our work and will have a large impact on the introduction of features companies are looking forward, hence our frustration.
I will answer both you and Jacob at once.
This is uAPI, once it is set it can never be changed.
Quoted 8 lines
The kernel process and philosophy is to invest heavily in uAPI development and review to converge on the best uAPI possible.
Many past submissions have take a long time to get this right, there are several high profile uAPI examples.
Do you think this case is so special, or the concerns so minor, that it should get to bypass all of the normal process?
That's not my intent to bypass any process. I am just trying to understand what needs to be re-designed and for what use case.
Quoted 7 lines
Ask yourself, is anyone advocating for the current direction on technical merits alone?
Certainly the patches I last saw where completely disgusting from a uAPI design perspective.
Quoted 6 lines
It was against the development process to organize this work the way it was done. Merging a wack of dead code to the kernel to support a uAPI vision that was never clearly articulated was a big mistake.
Start from the beginning. Invest heavily in defining a high quality uAPI. Clearly describe the uAPI to all stake holders.
This was largely done during several confs including plumber, KVM forum, for several years. Also API docs were shared on the ML. I don't remember any voice was raised at those moments.
Break up the
Quoted 7 lines
implementation into patch series without dead code. Make the patches. Remove the dead code this group has already added.
None of this should be a surprise. The VDPA discussion and related "what is a mdev" over a year ago made it pretty clear VFIO is not the exclusive user of "IOMMU in userspace" and that places limits on what kind of uAPIs expansion it should experience going forward.
Maybe clear for you but most probably not for many other stakeholders.
Anyway I do not intend to further argue and I will be happy to learn from you and work with you, Jacob, Liu and all other stakeholders to define a better integration.
Thanks
Eric
Quoted 3 lines
Jason
Re: [PATCH V4 05/18] iommu/ioasid: Redefine IOASID set and allocation APIs
Quoted 3 lines · Auger Eric
This was largely done during several confs including plumber, KVM forum, for several years. Also API docs were shared on the ML. I don't remember any voice was raised at those moments.
I don't think anyone objects to the high level ideas, but implementation does matter. I don't think anyone presented "hey we will tunnel an uAPI through VFIO to the IOMMU subsystem" - did they?
Look at the fairly simple IMS situation, for example. This was presented at plumbers too, and the slides were great - but the implementation was too hacky. It required a major rework of the x86 interrupt handling before it was OK.
Jason
Re: [PATCH V4 05/18] iommu/ioasid: Redefine IOASID set and allocation APIs
Hi Jason,
Quoted 9 lines · Jason Gunthorpe
Quoted 3 lines · Auger Eric
This was largely done during several confs including plumber, KVM forum, for several years. Also API docs were shared on the ML. I don't remember any voice was raised at those moments.
I don't think anyone objects to the high level ideas, but implementation does matter. I don't think anyone presented "hey we will tunnel an uAPI through VFIO to the IOMMU subsystem" - did they?
But most obviously everything is documented in Documentation/userspace-api/iommu.rst where the VFIO tunneling is clearly stated ;-)
But well let's work together to design a better and more elegant solution then.
Thanks
Eric
Quoted 8 lines
Look at the fairly simple IMS situation, for example. This was presented at plumbers too, and the slides were great - but the implementation was too hacky. It required a major rework of the x86 interrupt handling before it was OK.
Jason
Re: [PATCH V4 05/18] iommu/ioasid: Redefine IOASID set and allocation APIs
In reply to Liu, Yi LQuoted 37 lines · Yi L
Quoted 16 lines
From: Jason Gunthorpe <[email protected]>
Sent: Wednesday, March 31, 2021 8:41 PMQuoted 4 lines · Yi L
The reason is
/dev/ioasidFD is per-VM since the ioasid allocated to the VM should be able to be shared by all assigned device for the VM. But the SVA operations (bind/unbind page table,cache_invalidate) should be per-device.It is not *per-device* it is *per-ioasid*
And as
/dev/ioasidis an interface for controlling multiple ioasid's there is no issue to also multiplex the page table manipulation for multiple ioasids as well.Quoted 8 lines
What you should do next is sketch out in some RFC the exactl ioctls each FD would have and show how the parts I outlined would work and point out any remaining gaps.
The device FD is something like the
vfio_deviceFD from VFIO, it has *nothing* to do with PASID beyond having a single ioctl to authorize the device to use the PASID. All control of the PASID is in/dev/ioasid.good to see this reply. Your idea is much clearer to me now. If I'm getting you correctly. I think the skeleton is something like below:
1) userspace opens a
/dev/ioasid, meanwhile there will be an ioasid allocated and a per-ioasid context which can be used to do bind page table and cache invalidate, an ioasid FD returned to userspace.
2) userspace passes the ioasid FD to VFIO, let it associated with a device FD (likevfio_deviceFD).
3) userspace binds page table on the ioasid FD with the page table info.
4) userspace unbinds the page table on the ioasid FD
5) userspace de-associates the ioasid FD and device FD
Quoted 1 line
Does above suit your outline?
Seems so
Quoted 3 lines
If yes, I still have below concern and wish to see your opinion.
- the ioasid FD and device association will happen at runtime instead of just happen in the setup phase.
Of course, this is required for security. The vIOMMU must perform the device association when the guest requires it. Otherwise a guest cannot isolate a PASID to a single process/device pair.
I'm worried Intel views the only use of PASID in a guest is with ENQCMD, but that is not consistent with the industry. We need to see normal nested PASID support with assigned PCI VFs.
Quoted 4 lines
- how about AMD and ARM's vSVA support? Their PASID allocation and page table happens within guest. They only need to bind the guest PASID table to host. Above model seems unable to fit them. (Jean, Eric, Jacob please feel free to correct me)
No, everything needs the device association step or it is not secure.
You can give a PASID to a guest and allow it to manipulate it's memory map directly, nested under the guest's CPU page tables.
However the guest cannot authorize a PCI BDF to utilize that PASID without going through some kind of step in the hypervisor. A Guest should not be able to authorize a PASID for a BDF it doesn't have access to - only the hypervisor can enforce this.
This all must also fit into the mdev model where only the device-specific mdev driver can do the device specific PASID authorization. A hypercall is essential, or we need to stop pretending mdev is a good idea.
I'm sure there will be some small differences, and you should clearly explain the entire uAPI surface so that soneone from AMD and ARM can review it.
Quoted 2 lines
- this per-ioasid SVA operations is not aligned with the native SVA usage model. Native SVA bind is per-device.
Seems like that is an error in native SVA.
SVA is a particular mode of the PASID's memory mapping table, it has nothing to do with a device.
Jason
RE: [PATCH V4 05/18] iommu/ioasid: Redefine IOASID set and allocation APIs
Quoted 2 lines
From: Jason Gunthorpe <[email protected]>
Sent: Thursday, April 1, 2021 7:47 PM
[...]
Quoted 3 lines
I'm worried Intel views the only use of PASID in a guest is with ENQCMD, but that is not consistent with the industry. We need to see normal nested PASID support with assigned PCI VFs.
I'm not quire flow here. Intel also allows PASID usage in guest without ENQCMD. e.g. Passthru a PF to guest, and use PASID on it without ENQCMD.
[...]
Quoted 3 lines
I'm sure there will be some small differences, and you should clearly explain the entire uAPI surface so that soneone from AMD and ARM can review it.
good suggestion, will do.
Quoted 7 lines
Quoted 2 lines
- this per-ioasid SVA operations is not aligned with the native SVA usage model. Native SVA bind is per-device.
Seems like that is an error in native SVA.
SVA is a particular mode of the PASID's memory mapping table, it has nothing to do with a device.
I think it still has relationship with device. This is determined by the DMA remapping hierarchy in hardware. e.g. Intel VT-d, the DMA isolation is enforced first in device granularity and then PASID granularity. SVA makes usage of both PASID and device granularity isolation.
Regards,
Yi Liu
Quoted 1 line
Jason
Re: [PATCH V4 05/18] iommu/ioasid: Redefine IOASID set and allocation APIs
Quoted 9 lines · Yi L
Quoted 2 lines
From: Jason Gunthorpe <[email protected]>
Sent: Thursday, April 1, 2021 7:47 PM[...]
Quoted 3 lines
I'm worried Intel views the only use of PASID in a guest is with ENQCMD, but that is not consistent with the industry. We need to see normal nested PASID support with assigned PCI VFs.
I'm not quire flow here. Intel also allows PASID usage in guest without ENQCMD. e.g. Passthru a PF to guest, and use PASID on it without ENQCMD.
Then you need all the parts, the hypervisor calls from the vIOMMU, and you can't really use a vPASID.
I'm not sure how Intel intends to resolve all of this.
Quoted 5 lines
Quoted 5 lines
Quoted 2 lines
- this per-ioasid SVA operations is not aligned with the native SVA usage model. Native SVA bind is per-device.
Seems like that is an error in native SVA.
Quoted 7 lines
Quoted 2 lines
SVA is a particular mode of the PASID's memory mapping table, it has nothing to do with a device.
I think it still has relationship with device. This is determined by the DMA remapping hierarchy in hardware. e.g. Intel VT-d, the DMA isolation is enforced first in device granularity and then PASID granularity. SVA makes usage of both PASID and device granularity isolation.
When the device driver authorizes a PASID the VT-d stuff should setup the isolation parameters for the give pci_device and PASID.
Do not leak implementation details like this as uAPI. Authorization and memory map are distinct ideas with distinct interfaces. Do not mix them.
Jason
RE: [PATCH V4 05/18] iommu/ioasid: Redefine IOASID set and allocation APIs
Quoted 11 lines
From: Jason Gunthorpe <[email protected]>
Sent: Thursday, April 1, 2021 9:16 PMQuoted 7 lines · Yi L
Quoted 2 lines
From: Jason Gunthorpe <[email protected]>
Sent: Thursday, April 1, 2021 7:47 PM[...]
Quoted 3 lines
I'm worried Intel views the only use of PASID in a guest is with ENQCMD, but that is not consistent with the industry. We need to see normal nested PASID support with assigned PCI VFs.
Quoted 6 lines
Quoted 2 lines
I'm not quire flow here. Intel also allows PASID usage in guest without ENQCMD. e.g. Passthru a PF to guest, and use PASID on it without
ENQCMD.
Then you need all the parts, the hypervisor calls from the vIOMMU, and you can't really use a vPASID.
This is a diagram shows the vSVA setup.
.-------------. .---------------------------.
| vIOMMU | | Guest process CR3, FL only|
| | '---------------------------' .----------------/
| PASID Entry |--- PASID cache flush -
'-------------' |
| | V
| | CR3 in GPA
'-------------'
Guest
------| Shadow |--------------------------|--------
v v v
Host
.-------------. .----------------------.
| pIOMMU | | Bind FL for GVA-GPA |
| | '----------------------'
.----------------/ |
| PASID Entry | V (Nested xlate)
'----------------\.------------------------------.
| | |SL for GPA-HPA, default domain|
| | '------------------------------'
'-------------'Where:
- FL = First level/stage one page tables
- SL = Second level/stage two page tables
Quoted 12 lines
I'm not sure how Intel intends to resolve all of this.
Quoted 1 line
Quoted 1 line
Quoted 1 line
- this per-ioasid SVA operations is not aligned with the native SVA
usage
Quoted 7 lines
Quoted 6 lines
Quoted 1 line
model. Native SVA bind is per-device.
Seems like that is an error in native SVA.
SVA is a particular mode of the PASID's memory mapping table, it has nothing to do with a device.
Quoted 8 lines
Quoted 2 lines
I think it still has relationship with device. This is determined by the DMA remapping hierarchy in hardware. e.g. Intel VT-d, the DMA isolation
is
Quoted 2 lines
enforced first in device granularity and then PASID granularity. SVA makes usage of both PASID and device granularity isolation.
When the device driver authorizes a PASID the VT-d stuff should setup the isolation parameters for the give
pci_deviceand PASID.
yes, both device and PASID is needed to setup VT-d stuff.
Quoted 3 lines
Do not leak implementation details like this as uAPI. Authorization and memory map are distinct ideas with distinct interfaces. Do not mix them.
got you. Let's focus on the uAPI things here and leave implementation details in RFC patches.
Thanks,
Yi Liu
Quoted 1 line
Jason
Re: [PATCH V4 05/18] iommu/ioasid: Redefine IOASID set and allocation APIs
Quoted 11 lines · Yi L
Quoted 11 lines
From: Jason Gunthorpe <[email protected]>
Sent: Thursday, April 1, 2021 9:16 PMQuoted 7 lines · Yi L
Quoted 2 lines
From: Jason Gunthorpe <[email protected]>
Sent: Thursday, April 1, 2021 7:47 PM[...]
Quoted 3 lines
I'm worried Intel views the only use of PASID in a guest is with ENQCMD, but that is not consistent with the industry. We need to see normal nested PASID support with assigned PCI VFs.
Quoted 8 lines
Quoted 6 lines
Quoted 2 lines
I'm not quire flow here. Intel also allows PASID usage in guest without ENQCMD. e.g. Passthru a PF to guest, and use PASID on it without
ENQCMD.
Then you need all the parts, the hypervisor calls from the vIOMMU, and you can't really use a vPASID.
This is a diagram shows the vSVA setup.
I'm not talking only about vSVA. Generic PASID support with arbitary mappings.
And how do you deal with the vPASID vs pPASID issue if the system has a mix of physical devices and mdevs?
Jason
RE: [PATCH V4 05/18] iommu/ioasid: Redefine IOASID set and allocation APIs
Quoted 22 lines
From: Jason Gunthorpe <[email protected]>
Sent: Thursday, April 1, 2021 9:47 PMQuoted 18 lines · Yi L
Quoted 17 lines
From: Jason Gunthorpe <[email protected]>
Sent: Thursday, April 1, 2021 9:16 PMQuoted 9 lines · Yi L
Quoted 2 lines
From: Jason Gunthorpe <[email protected]>
Sent: Thursday, April 1, 2021 7:47 PM[...]
Quoted 3 lines
I'm worried Intel views the only use of PASID in a guest is with ENQCMD, but that is not consistent with the industry. We need to see normal nested PASID support with assigned PCI VFs.
I'm not quire flow here. Intel also allows PASID usage in guest without ENQCMD. e.g. Passthru a PF to guest, and use PASID on it without
ENQCMD.
Then you need all the parts, the hypervisor calls from the vIOMMU, and you can't really use a vPASID.
Quoted 8 lines
Quoted 1 line
This is a diagram shows the vSVA setup.
I'm not talking only about vSVA. Generic PASID support with arbitary mappings.
And how do you deal with the vPASID vs pPASID issue if the system has a mix of physical devices and mdevs?
We plan to support two schemes. One is vPASID identity-mapped to pPASID then the mixed scenario just works, with the limitation of lacking of live migration support. The other is non-identity-mapped scheme, where live migration is supported but physical devices and mdevs should not be mixed in one VM if both expose SVA capability (requires some filtering check in Qemu). Although we have some idea relaxing this restriction in the non-identity scheme, it requires more thinking given how the vSVA uAPI is being refactored.
In both cases the virtual VT-d will report a virtual capability to the guest, indicating that the guest must request PASID through a vcmd register instead of creating its own namespace. The vIOMMU returns a vPASID to the guest upon request. The vPASID could be directly mapped to a pPASID or allocated from a new namespace based on user configuration.
We hope the /dev/ioasid can support both schemes, with the minimal requirement of allowing userspace to tag a vPASID to a pPASID and allowing mdev to translate vPASID into pPASID, i.e. not assuming that the guest will always use pPASID.
Thanks
Kevin
Re: [PATCH V4 05/18] iommu/ioasid: Redefine IOASID set and allocation APIs
Quoted 31 lines · Kevin
Quoted 22 lines
From: Jason Gunthorpe <[email protected]>
Sent: Thursday, April 1, 2021 9:47 PMQuoted 18 lines · Yi L
Quoted 17 lines
From: Jason Gunthorpe <[email protected]>
Sent: Thursday, April 1, 2021 9:16 PMQuoted 9 lines · Yi L
Quoted 2 lines
From: Jason Gunthorpe <[email protected]>
Sent: Thursday, April 1, 2021 7:47 PM[...]
Quoted 3 lines
I'm worried Intel views the only use of PASID in a guest is with ENQCMD, but that is not consistent with the industry. We need to see normal nested PASID support with assigned PCI VFs.
I'm not quire flow here. Intel also allows PASID usage in guest without ENQCMD. e.g. Passthru a PF to guest, and use PASID on it without
ENQCMD.
Then you need all the parts, the hypervisor calls from the vIOMMU, and you can't really use a vPASID.
Quoted 8 lines
Quoted 1 line
This is a diagram shows the vSVA setup.
I'm not talking only about vSVA. Generic PASID support with arbitary mappings.
And how do you deal with the vPASID vs pPASID issue if the system has a mix of physical devices and mdevs?
Quoted 6 lines
We plan to support two schemes. One is vPASID identity-mapped to pPASID then the mixed scenario just works, with the limitation of lacking of live migration support. The other is non-identity-mapped scheme, where live migration is supported but physical devices and mdevs should not be mixed in one VM if both expose SVA capability (requires some filtering check in Qemu).
That just becomes "block vPASID support if any device that doesn't use ENQCMD is plugged into the guest"
Which needs a special VFIO capability of some kind so qemu knows to block it. This really needs to all be layed out together so someone can understand it :(
Why doesn't the siov cookbook explaining this stuff??
Quoted 4 lines
We hope the
/dev/ioasidcan support both schemes, with the minimal requirement of allowing userspace to tag a vPASID to a pPASID and allowing mdev to translate vPASID into pPASID, i.e. not assuming that the guest will always use pPASID.
What I'm a unclear of is if /dev/ioasid even needs to care about vPASID or if vPASID is just a hidden artifact of the KVM connection to setup the translation table and the vIOMMU driver in qemu.
Since the physical HW never sees the vPASID I'm inclined to think the latter.
Jason
RE: [PATCH V4 05/18] iommu/ioasid: Redefine IOASID set and allocation APIs
Quoted 45 lines
From: Jason Gunthorpe <[email protected]>
Sent: Tuesday, April 6, 2021 7:40 AMQuoted 7 lines · Kevin
Quoted 7 lines
From: Jason Gunthorpe <[email protected]>
Sent: Thursday, April 1, 2021 9:47 PMQuoted 3 lines · Yi L
Quoted 3 lines
From: Jason Gunthorpe <[email protected]>
Sent: Thursday, April 1, 2021 9:16 PMQuoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines · Yi L
Quoted 5 lines
Quoted 2 lines
From: Jason Gunthorpe <[email protected]>
Sent: Thursday, April 1, 2021 7:47 PM[...]
Quoted 2 lines
I'm worried Intel views the only use of PASID in a guest is with ENQCMD, but that is not consistent with the industry. We need to
see
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 1 line
normal nested PASID support with assigned PCI VFs.
I'm not quire flow here. Intel also allows PASID usage in guest
without
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 1 line
ENQCMD. e.g. Passthru a PF to guest, and use PASID on it without
ENQCMD.
Then you need all the parts, the hypervisor calls from the vIOMMU,
and
Quoted 11 lines
Quoted 10 lines
Quoted 3 lines
Quoted 1 line
you can't really use a vPASID.
This is a diagram shows the vSVA setup.
I'm not talking only about vSVA. Generic PASID support with arbitary mappings.
And how do you deal with the vPASID vs pPASID issue if the system has a mix of physical devices and mdevs?
Quoted 6 lines
We plan to support two schemes. One is vPASID identity-mapped to pPASID then the mixed scenario just works, with the limitation of lacking of live migration support. The other is non-identity-mapped scheme, where live migration is supported but physical devices and mdevs should not be mixed in one VM if both expose SVA capability (requires some filtering check in Qemu).
Quoted 2 lines
That just becomes "block vPASID support if any device that doesn't use ENQCMD is plugged into the guest"
The limitation is only for physical device. and in reality it is not that bad. To support live migration with physical device we anyway need additional work to migrate the device state (e.g. based on Max's work), then it's not unreasonable to also mediate guest programming of device specific PASID register to enable vPASID (need to translate in the whole VM lifespan but likely is not a hot path).
Quoted 4 lines
Which needs a special VFIO capability of some kind so qemu knows to block it. This really needs to all be layed out together so someone can understand it :(
Or could simply based on whether the VFIO device supports live migration.
Quoted 11 lines
Why doesn't the siov cookbook explaining this stuff??
Quoted 4 lines
We hope the
/dev/ioasidcan support both schemes, with the minimal requirement of allowing userspace to tag a vPASID to a pPASID and allowing mdev to translate vPASID into pPASID, i.e. not assuming that the guest will always use pPASID.What I'm a unclear of is if
/dev/ioasideven needs to care about vPASID or if vPASID is just a hidden artifact of the KVM connection to setup the translation table and the vIOMMU driver in qemu.
Not just for KVM. Also required by mdev, which needs to translate vPASID into pPASID when ENQCMD is not used. As I replied in another mail, possibly we don't need /dev/ioasid to know this fact, which should only care about the operations related to pPASID. VFIO could carry vPASID information to mdev. KVM should have its own interface to know this information, as you suggested earlier.
Thanks
Kevin
Re: [PATCH V4 05/18] iommu/ioasid: Redefine IOASID set and allocation APIs
Quoted 48 lines · Kevin
Quoted 45 lines
From: Jason Gunthorpe <[email protected]>
Sent: Tuesday, April 6, 2021 7:40 AMQuoted 7 lines · Kevin
Quoted 7 lines
From: Jason Gunthorpe <[email protected]>
Sent: Thursday, April 1, 2021 9:47 PMQuoted 3 lines · Yi L
Quoted 3 lines
From: Jason Gunthorpe <[email protected]>
Sent: Thursday, April 1, 2021 9:16 PMQuoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines · Yi L
Quoted 5 lines
Quoted 2 lines
From: Jason Gunthorpe <[email protected]>
Sent: Thursday, April 1, 2021 7:47 PM[...]
Quoted 2 lines
I'm worried Intel views the only use of PASID in a guest is with ENQCMD, but that is not consistent with the industry. We need to
see
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 1 line
normal nested PASID support with assigned PCI VFs.
I'm not quire flow here. Intel also allows PASID usage in guest
without
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 1 line
ENQCMD. e.g. Passthru a PF to guest, and use PASID on it without
ENQCMD.
Then you need all the parts, the hypervisor calls from the vIOMMU,
and
Quoted 11 lines
Quoted 10 lines
Quoted 3 lines
Quoted 1 line
you can't really use a vPASID.
This is a diagram shows the vSVA setup.
I'm not talking only about vSVA. Generic PASID support with arbitary mappings.
And how do you deal with the vPASID vs pPASID issue if the system has a mix of physical devices and mdevs?
Quoted 6 lines
We plan to support two schemes. One is vPASID identity-mapped to pPASID then the mixed scenario just works, with the limitation of lacking of live migration support. The other is non-identity-mapped scheme, where live migration is supported but physical devices and mdevs should not be mixed in one VM if both expose SVA capability (requires some filtering check in Qemu).
Quoted 2 lines
That just becomes "block vPASID support if any device that doesn't use ENQCMD is plugged into the guest"
Quoted 6 lines
The limitation is only for physical device. and in reality it is not that bad. To support live migration with physical device we anyway need additional work to migrate the device state (e.g. based on Max's work), then it's not unreasonable to also mediate guest programming of device specific PASID register to enable vPASID (need to translate in the whole VM lifespan but likely is not a hot path).
IMHO that is pretty unreasonable.. More likely we end up with vPASID tables in each migratable device like KVM has.
Quoted 5 lines
Quoted 3 lines
Which needs a special VFIO capability of some kind so qemu knows to block it. This really needs to all be layed out together so someone can understand it :(
Or could simply based on whether the VFIO device supports live migration.
You need to define affirmative caps that indicate that vPASID will be supported by the VFIO device.
Quoted 7 lines
Quoted 7 lines
Why doesn't the siov cookbook explaining this stuff??
Quoted 4 lines
We hope the
/dev/ioasidcan support both schemes, with the minimal requirement of allowing userspace to tag a vPASID to a pPASID and allowing mdev to translate vPASID into pPASID, i.e. not assuming that the guest will always use pPASID.
Quoted 6 lines
Quoted 3 lines
What I'm a unclear of is if
/dev/ioasideven needs to care about vPASID or if vPASID is just a hidden artifact of the KVM connection to setup the translation table and the vIOMMU driver in qemu.Not just for KVM. Also required by mdev, which needs to translate vPASID into pPASID when ENQCMD is not used.
Do we have any mdev's that will do this?
Quoted 2 lines
should only care about the operations related to pPASID. VFIO could carry vPASID information to mdev.
It depends how common this is, I suppose
Jason
RE: [PATCH V4 05/18] iommu/ioasid: Redefine IOASID set and allocation APIs
Quoted 62 lines
From: Jason Gunthorpe <[email protected]>
Sent: Tuesday, April 6, 2021 8:21 PMQuoted 11 lines · Kevin
Quoted 11 lines
From: Jason Gunthorpe <[email protected]>
Sent: Tuesday, April 6, 2021 7:40 AMQuoted 7 lines · Kevin
Quoted 7 lines
From: Jason Gunthorpe <[email protected]>
Sent: Thursday, April 1, 2021 9:47 PMQuoted 3 lines · Yi L
Quoted 3 lines
From: Jason Gunthorpe <[email protected]>
Sent: Thursday, April 1, 2021 9:16 PMQuoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines · Yi L
Quoted 5 lines
Quoted 2 lines
From: Jason Gunthorpe <[email protected]>
Sent: Thursday, April 1, 2021 7:47 PM[...]
Quoted 2 lines
I'm worried Intel views the only use of PASID in a guest is with ENQCMD, but that is not consistent with the industry. We need
to
Quoted 6 lines
Quoted 6 lines
see
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 1 line
normal nested PASID support with assigned PCI VFs.
I'm not quire flow here. Intel also allows PASID usage in guest
without
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
ENQCMD. e.g. Passthru a PF to guest, and use PASID on it
without
Quoted 4 lines
Quoted 4 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
ENQCMD.
Then you need all the parts, the hypervisor calls from the vIOMMU,
and
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 3 lines
Quoted 1 line
you can't really use a vPASID.
This is a diagram shows the vSVA setup.
I'm not talking only about vSVA. Generic PASID support with arbitary mappings.
And how do you deal with the vPASID vs pPASID issue if the system
has
Quoted 13 lines
Quoted 10 lines
Quoted 9 lines
Quoted 2 lines
a mix of physical devices and mdevs?
We plan to support two schemes. One is vPASID identity-mapped to pPASID then the mixed scenario just works, with the limitation of lacking of live migration support. The other is non-identity-mapped scheme, where live migration is supported but physical devices and mdevs should not be mixed in one VM if both expose SVA capability (requires some filtering check in Qemu).
Quoted 2 lines
That just becomes "block vPASID support if any device that doesn't use ENQCMD is plugged into the guest"
Quoted 6 lines
The limitation is only for physical device. and in reality it is not that bad. To support live migration with physical device we anyway need additional work to migrate the device state (e.g. based on Max's work), then it's not unreasonable to also mediate guest programming of device specific PASID register to enable vPASID (need to translate in the whole VM lifespan but likely is not a hot path).
Quoted 2 lines
IMHO that is pretty unreasonable.. More likely we end up with vPASID tables in each migratable device like KVM has.
just like mdev needs to maintain allowed PASID list, this extends it to all migratable devices.
Quoted 9 lines
Quoted 6 lines
Quoted 4 lines
Which needs a special VFIO capability of some kind so qemu knows to block it. This really needs to all be layed out together so someone can understand it :(
Or could simply based on whether the VFIO device supports live migration.
You need to define affirmative caps that indicate that vPASID will be supported by the VFIO device.
Yes, this is required as I acked in another mail.
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Why doesn't the siov cookbook explaining this stuff??
Quoted 4 lines
We hope the
/dev/ioasidcan support both schemes, with the minimal requirement of allowing userspace to tag a vPASID to a pPASID and allowing mdev to translate vPASID into pPASID, i.e. not assuming that the guest will always use pPASID.
Quoted 8 lines
Quoted 6 lines
Quoted 3 lines
What I'm a unclear of is if
/dev/ioasideven needs to care about vPASID or if vPASID is just a hidden artifact of the KVM connection to setup the translation table and the vIOMMU driver in qemu.Not just for KVM. Also required by mdev, which needs to translate vPASID into pPASID when ENQCMD is not used.
Do we have any mdev's that will do this?
definitely. Actually any mdev which doesn't do ENQCMD needs to do this. In normal case, the PASID is programmed to a MMIO register (or in-memory context) associate with the backend resource of the mdev. The value programmed from the guest is vPASID, thus must be translated into pPASID before updating the physical register.
Quoted 6 lines
Quoted 3 lines
should only care about the operations related to pPASID. VFIO could carry vPASID information to mdev.
It depends how common this is, I suppose
based on above I think it's a common case.
Thanks
Kevin
A message that is not in the archive (sent off-list or not yet received): MWHPR11MB188628BDB37A4EE36F3D99338C769@MWHPR11MB1886.namprd11.prod.outlook.com
RE: [PATCH V4 05/18] iommu/ioasid: Redefine IOASID set and allocation APIs
Quoted 60 lines
From: Tian, Kevin
Sent: Tuesday, April 6, 2021 9:02 AMQuoted 17 lines
From: Jason Gunthorpe <[email protected]>
Sent: Tuesday, April 6, 2021 7:40 AMQuoted 13 lines · Kevin
Quoted 13 lines
From: Jason Gunthorpe <[email protected]>
Sent: Thursday, April 1, 2021 9:47 PMQuoted 9 lines · Yi L
Quoted 9 lines
From: Jason Gunthorpe <[email protected]>
Sent: Thursday, April 1, 2021 9:16 PMQuoted 5 lines · Yi L
Quoted 2 lines
From: Jason Gunthorpe <[email protected]>
Sent: Thursday, April 1, 2021 7:47 PM[...]
Quoted 2 lines
I'm worried Intel views the only use of PASID in a guest is with ENQCMD, but that is not consistent with the industry. We need
to
Quoted 28 lines
see
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 1 line
normal nested PASID support with assigned PCI VFs.
I'm not quire flow here. Intel also allows PASID usage in guest
without
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 1 line
ENQCMD. e.g. Passthru a PF to guest, and use PASID on it without
ENQCMD.
Then you need all the parts, the hypervisor calls from the vIOMMU,
and
Quoted 11 lines
Quoted 10 lines
Quoted 3 lines
Quoted 1 line
you can't really use a vPASID.
This is a diagram shows the vSVA setup.
I'm not talking only about vSVA. Generic PASID support with arbitary mappings.
And how do you deal with the vPASID vs pPASID issue if the system has a mix of physical devices and mdevs?
Quoted 6 lines
We plan to support two schemes. One is vPASID identity-mapped to pPASID then the mixed scenario just works, with the limitation of lacking of live migration support. The other is non-identity-mapped scheme, where live migration is supported but physical devices and mdevs should not be mixed in one VM if both expose SVA capability (requires some filtering check in Qemu).
Quoted 2 lines
That just becomes "block vPASID support if any device that doesn't use ENQCMD is plugged into the guest"
The limitation is only for physical device. and in reality it is not that bad. To support live migration with physical device we anyway need additional work to migrate the device state (e.g. based on Max's work), then it's not unreasonable to also mediate guest programming of device specific PASID register to enable vPASID (need to translate in the whole VM lifespan but likely is not a hot path).
Quoted 1 line
Quoted 5 lines
Quoted 3 lines
Which needs a special VFIO capability of some kind so qemu knows to block it. This really needs to all be layed out together so someone can understand it :(
Or could simply based on whether the VFIO device supports live migration.
Actually you are right on this point. VFIO should provide a per-device capability to indicate whether vPASID is allowed on this device. likely yes for mdev, by default no for pdev (unless explicitly opt in). Qemu should enable vPASID only if all assigned devices support it, and then provide vPASID information when using VFIO API to allow pPASIDs.
Quoted 22 lines
Quoted 4 lines
Why doesn't the siov cookbook explaining this stuff??
Quoted 8 lines
Quoted 4 lines
We hope the
/dev/ioasidcan support both schemes, with the minimal requirement of allowing userspace to tag a vPASID to a pPASID and allowing mdev to translate vPASID into pPASID, i.e. not assuming that the guest will always use pPASID.What I'm a unclear of is if
/dev/ioasideven needs to care about vPASID or if vPASID is just a hidden artifact of the KVM connection to setup the translation table and the vIOMMU driver in qemu.Not just for KVM. Also required by mdev, which needs to translate vPASID into pPASID when ENQCMD is not used. As I replied in another mail, possibly we don't need
/dev/ioasidto know this fact, which should only care about the operations related to pPASID. VFIO could carry vPASID information to mdev. KVM should have its own interface to know this information, as you suggested earlier.Thanks
Kevin
RE: [PATCH V4 05/18] iommu/ioasid: Redefine IOASID set and allocation APIs
In reply to Jason GunthorpeQuoted 50 lines
From: Tian, Kevin
Sent: Friday, April 2, 2021 3:58 PMQuoted 13 lines
From: Jason Gunthorpe <[email protected]>
Sent: Thursday, April 1, 2021 9:47 PMQuoted 9 lines · Yi L
Quoted 9 lines
From: Jason Gunthorpe <[email protected]>
Sent: Thursday, April 1, 2021 9:16 PMQuoted 5 lines · Yi L
Quoted 2 lines
From: Jason Gunthorpe <[email protected]>
Sent: Thursday, April 1, 2021 7:47 PM[...]
Quoted 2 lines
I'm worried Intel views the only use of PASID in a guest is with ENQCMD, but that is not consistent with the industry. We need to
see
Quoted 9 lines
Quoted 9 lines
Quoted 8 lines
Quoted 4 lines
Quoted 1 line
normal nested PASID support with assigned PCI VFs.
I'm not quire flow here. Intel also allows PASID usage in guest without ENQCMD. e.g. Passthru a PF to guest, and use PASID on it without
ENQCMD.
Then you need all the parts, the hypervisor calls from the vIOMMU, and you can't really use a vPASID.
Quoted 8 lines
Quoted 1 line
This is a diagram shows the vSVA setup.
I'm not talking only about vSVA. Generic PASID support with arbitary mappings.
And how do you deal with the vPASID vs pPASID issue if the system has a mix of physical devices and mdevs?
We plan to support two schemes. One is vPASID identity-mapped to pPASID then the mixed scenario just works, with the limitation of lacking of live migration support. The other is non-identity-mapped scheme, where live migration is supported but physical devices and mdevs should not be mixed in one VM if both expose SVA capability (requires some filtering check in Qemu). Although we have some idea relaxing this restriction in the non-identity scheme, it requires more thinking given how the vSVA uAPI is being refactored.
In both cases the virtual VT-d will report a virtual capability to the guest, indicating that the guest must request PASID through a vcmd register instead of creating its own namespace. The vIOMMU returns a vPASID to the guest upon request. The vPASID could be directly mapped to a pPASID or allocated from a new namespace based on user configuration.
Quoted 5 lines
We hope the
/dev/ioasidcan support both schemes, with the minimal requirement of allowing userspace to tag a vPASID to a pPASID and allowing mdev to translate vPASID into pPASID, i.e. not assuming that the guest will always use pPASID.
Per your comments in other threads I suppose this requirement should be implemented in VFIO_ALLOW_PASID command instead of going through /dev/ioasid which only needs to know pPASID and its pgtable management.
Thanks
Kevin
RE: [PATCH V4 05/18] iommu/ioasid: Redefine IOASID set and allocation APIs
In reply to Jason GunthorpeQuoted 22 lines
From: Jason Gunthorpe <[email protected]>
Sent: Tuesday, March 30, 2021 9:29 PMQuoted 7 lines
First, userspace may use ioasid in a non-SVA scenario where ioasid is bound to specific security context (e.g. a control vq in vDPA) instead of tying to mm. In this case there is no pgtable binding initiated from user space. Instead, ioasid is allocated from
/dev/ioasidand then programmed to the intended security context through specific passthrough framework which manages that context.This sounds like the exact opposite of what I'd like to see.
I do not want to see every subsystem gaining APIs to program a PASID. All of that should be consolidated in *one place*.
I do not want to see VDPA and VFIO have two nearly identical sets of APIs to control the PASID.
Drivers consuming a PASID, like VDPA, should consume the PASID and do nothing more than authorize the HW to use it.
Quoted 7 lines
quemu should have general code under the viommu driver that drives
/dev/ioasidto create PASID's and manage the IO mapping according to the guest's needs.Drivers like VDPA and VFIO should simply accept that PASID and configure/authorize their HW to do DMA's with its tag.
I agree with you on consolidating things in one place (especially for the general SVA support). But here I was referring to an usage without pgtable binding (Possibly Jason. W can say more here), where the userspace just wants to allocate PASIDs, program/accept PASIDs to various workqueues (device specific), and then use MAP/UNMAP interface to manage address spaces associated with each PASID. I just wanted to point out that the latter two steps are through VFIO/VDPA specific interfaces.
Thanks
Kevin
Re: [PATCH V4 05/18] iommu/ioasid: Redefine IOASID set and allocation APIs
Quoted 30 lines · Kevin
Quoted 22 lines
From: Jason Gunthorpe <[email protected]>
Sent: Tuesday, March 30, 2021 9:29 PMQuoted 7 lines
First, userspace may use ioasid in a non-SVA scenario where ioasid is bound to specific security context (e.g. a control vq in vDPA) instead of tying to mm. In this case there is no pgtable binding initiated from user space. Instead, ioasid is allocated from
/dev/ioasidand then programmed to the intended security context through specific passthrough framework which manages that context.This sounds like the exact opposite of what I'd like to see.
I do not want to see every subsystem gaining APIs to program a PASID. All of that should be consolidated in *one place*.
I do not want to see VDPA and VFIO have two nearly identical sets of APIs to control the PASID.
Drivers consuming a PASID, like VDPA, should consume the PASID and do nothing more than authorize the HW to use it.
Quoted 7 lines
quemu should have general code under the viommu driver that drives
/dev/ioasidto create PASID's and manage the IO mapping according to the guest's needs.Drivers like VDPA and VFIO should simply accept that PASID and configure/authorize their HW to do DMA's with its tag.
Quoted 8 lines
I agree with you on consolidating things in one place (especially for the general SVA support). But here I was referring to an usage without pgtable binding (Possibly Jason. W can say more here), where the userspace just wants to allocate PASIDs, program/accept PASIDs to various workqueues (device specific), and then use MAP/UNMAP interface to manage address spaces associated with each PASID. I just wanted to point out that the latter two steps are through VFIO/VDPA specific interfaces.
No, don't do that.
VFIO and VDPA has no buisness having map/unmap interfaces once we have /dev/ioasid. That all belongs in the iosaid side.
I know they have those interfaces today, but that doesn't mean we have to keep using them for PASID use cases, they should be replaced with a 'do dma from this pasid on /dev/ioasid' interface certainly not a 'here is a pasid from /dev/ioasid, go ahead and configure it youself' interface
This is because PASID is *complicated* in the general case! For instance all the two level stuff you are talking about must not leak into every user!
Jason
RE: [PATCH V4 05/18] iommu/ioasid: Redefine IOASID set and allocation APIs
Quoted 56 lines
From: Jason Gunthorpe
Sent: Tuesday, April 6, 2021 7:43 AMQuoted 8 lines · Kevin
Quoted 8 lines
From: Jason Gunthorpe <[email protected]>
Sent: Tuesday, March 30, 2021 9:29 PMQuoted 5 lines
First, userspace may use ioasid in a non-SVA scenario where ioasid is bound to specific security context (e.g. a control vq in vDPA) instead of tying to mm. In this case there is no pgtable binding initiated from user space. Instead, ioasid is allocated from
/dev/ioasidand thenprogrammed
Quoted 1 line
Quoted 1 line
Quoted 1 line
to the intended security context through specific passthrough
framework
Quoted 21 lines
Quoted 13 lines
Quoted 1 line
which manages that context.
This sounds like the exact opposite of what I'd like to see.
I do not want to see every subsystem gaining APIs to program a PASID. All of that should be consolidated in *one place*.
I do not want to see VDPA and VFIO have two nearly identical sets of APIs to control the PASID.
Drivers consuming a PASID, like VDPA, should consume the PASID and do nothing more than authorize the HW to use it.
Quoted 7 lines
quemu should have general code under the viommu driver that drives
/dev/ioasidto create PASID's and manage the IO mapping according to the guest's needs.Drivers like VDPA and VFIO should simply accept that PASID and configure/authorize their HW to do DMA's with its tag.
Quoted 8 lines
I agree with you on consolidating things in one place (especially for the general SVA support). But here I was referring to an usage without pgtable binding (Possibly Jason. W can say more here), where the userspace just wants to allocate PASIDs, program/accept PASIDs to various workqueues (device specific), and then use MAP/UNMAP interface to manage address spaces associated with each PASID. I just wanted to point out that the latter two steps are through VFIO/VDPA specific interfaces.
No, don't do that.
VFIO and VDPA has no buisness having map/unmap interfaces once we have
/dev/ioasid. That all belongs in the iosaid side.I know they have those interfaces today, but that doesn't mean we have to keep using them for PASID use cases, they should be replaced with a 'do dma from this pasid on
/dev/ioasid' interface certainly not a 'here is a pasid from/dev/ioasid, go ahead and configure it youself' interface
Quoted 4 lines
This is because PASID is *complicated* in the general case! For instance all the two level stuff you are talking about must not leak into every user!
Hi, Jason,
I didn't get your last comment how the two level stuff is leaked into every user. Could you elaborate it a bit?
and here is one example why using existing VFIO/VDPA interface makes sense. say dev1 (w/ sva) and dev2 (w/o sva) are placed in a single VFIO container. The container is associated to an iommu domain which contains a single 2nd-level page table, shared by both devices (when attached to the domain). The VFIO MAP operation is applied to the 2nd-level page table thus naturally applied to both devices. Then userspace could use /dev/ioasid to further allocate IOASIDs and bind multiple 1st-level page tables for dev1, nested on the shared 2nd-level page table.
If following your suggestion then VFIO must deny VFIO MAP operations on sva1 (assume userspace should not mix sva1 and sva2 in the same container and instead use /dev/ioasid to map for sva1)? and even for a sva-capable device there is a window before the guest actually enables sva on that device then VFIO should still accept MAP in that window and then deny it after sva is enabled by the guest? This all sounds unnecessary complex while there is already a clean way to achieve it...
Thanks
Kevin
Re: [PATCH V4 05/18] iommu/ioasid: Redefine IOASID set and allocation APIs
Quoted 4 lines · Kevin
and here is one example why using existing VFIO/VDPA interface makes sense. say dev1 (w/ sva) and dev2 (w/o sva) are placed in a single VFIO container.
Forget about SVA, it is an irrelevant detail of how a PASID is configured.
Quoted 3 lines
The container is associated to an iommu domain which contains a single 2nd-level page table, shared by both devices (when attached to the domain).
This level should be described by an ioasid.
Quoted 5 lines
The VFIO MAP operation is applied to the 2nd-level
page table thus naturally applied to both devices. Then userspace could use/dev/ioasidto further allocate IOASIDs and bind multiple 1st-level page tables for dev1, nested on the shared 2nd-level page table.
Because if you don't then we enter insane world where a PASID is being created under /dev/ioasid but its translation path flows through setup done by VFIO and the whole user API becomes an incomprehensible mess.
How will you even associate the PASID with the other translation??
The entire translation path for any ioasid or PASID should be defined only by /dev/ioasid. Everything else is a legacy API.
Quoted 3 lines
If following your suggestion then VFIO must deny VFIO MAP operations on sva1 (assume userspace should not mix sva1 and sva2 in the same container and instead use
/dev/ioasidto map for sva1)?
No, userspace creates an iosaid for the guest physical mapping and passes this ioasid to VFIO PCI which will assign it as the first layer mapping on the RID
When PASIDs are allocated the uAPI will be told to logically nested under the first ioasid. When VFIO authorizes a PASID for a RID it checks that all the HW rules are being followed.
If there are rules like groups of VFIO devices must always use the same IOASID then VFIO will check these too (and realistically qemu will have only one guest physical map ioasid anyhow)
There is no real difference between setting up an IOMMU table for a (RID,PASID) tuple or just a RID. We can do it universally with one interface for all consumers.
I wanted this when we were doing VDPA for the first time, now that we are doing pasid and more difficult stuff I view it as essential.
Jason
RE: [PATCH V4 05/18] iommu/ioasid: Redefine IOASID set and allocation APIs
Quoted 24 lines
From: Jason Gunthorpe <[email protected]>
Sent: Tuesday, April 6, 2021 8:35 PMQuoted 4 lines · Kevin
and here is one example why using existing VFIO/VDPA interface makes sense. say dev1 (w/ sva) and dev2 (w/o sva) are placed in a single VFIO container.
Forget about SVA, it is an irrelevant detail of how a PASID is configured.
Quoted 3 lines
The container is associated to an iommu domain which contains a single 2nd-level page table, shared by both devices (when attached to the domain).
This level should be described by an ioasid.
Quoted 5 lines
The VFIO MAP operation is applied to the 2nd-level
page table thus naturally applied to both devices. Then userspace could use/dev/ioasidto further allocate IOASIDs and bind multiple 1st-level page tables for dev1, nested on the shared 2nd-level page table.
Quoted 5 lines
Because if you don't then we enter insane world where a PASID is being created under
/dev/ioasidbut its translation path flows through setup done by VFIO and the whole user API becomes an incomprehensible mess.How will you even associate the PASID with the other translation??
PASID is attached to a specific iommu domain (created by VFIO/VDPA), which has GPA->HPA mappings already configured. If we view that mapping as an attribute of the iommu domain, it's reasonable to have the userspace-bound pgtable through /dev/ioasid to nest on it.
Quoted 11 lines
The entire translation path for any ioasid or PASID should be defined only by
/dev/ioasid. Everything else is a legacy API.Quoted 3 lines
If following your suggestion then VFIO must deny VFIO MAP operations on sva1 (assume userspace should not mix sva1 and sva2 in the same container and instead use
/dev/ioasidto map for sva1)?No, userspace creates an iosaid for the guest physical mapping and passes this ioasid to VFIO PCI which will assign it as the first layer mapping on the RID
Is it an dummy ioasid just for providing GPA mappings for nesting purpose of other IOASIDs? Then we waste one per VM?
Quoted 4 lines
When PASIDs are allocated the uAPI will be told to logically nested under the first ioasid. When VFIO authorizes a PASID for a RID it checks that all the HW rules are being followed.
As I explained above, why cannot we just use iommu domain to connect the dots? Every passthrough framework needs to create an iommu domain first. and It needs to support both devices w/ PASID and devices w/o PASID. For devices w/o PASID it needs to invent its own MAP interface anyway. Then why do we bother creating another MAP interface through /dev/ioasid which not only duplicates but also creating transition burden between two set of MAP interfaces when the guest turns on/off the pasid capability on the device?
Quoted 9 lines
If there are rules like groups of VFIO devices must always use the same IOASID then VFIO will check these too (and realistically qemu will have only one guest physical map ioasid anyhow)
There is no real difference between setting up an IOMMU table for a (RID,PASID) tuple or just a RID. We can do it universally with one interface for all consumers.
'universally' upon from which angle you look at this problem. From IOASID p.o.v possibly yes, but from device passthrough p.o.v. it's the opposite since the passthrough framework needs to handle devices w/o PASID anyway (or even for device w/ PASID it could send traffic w/o PASID) thus 'universally' makes more sense if the passthrough framework can use one interface of its own to manage GPA mappings for all consumers (apply to the case when a PASID is allowed/authorized).
Thanks
Kevin
Re: [PATCH V4 05/18] iommu/ioasid: Redefine IOASID set and allocation APIs
Quoted 10 lines · Kevin
Quoted 5 lines
Because if you don't then we enter insane world where a PASID is being created under
/dev/ioasidbut its translation path flows through setup done by VFIO and the whole user API becomes an incomprehensible mess.How will you even associate the PASID with the other translation??
PASID is attached to a specific iommu domain (created by VFIO/VDPA), which has
GPA->HPAmappings already configured. If we view that mapping as an attribute of the iommu domain, it's reasonable to have the userspace-bound pgtable through/dev/ioasidto nest on it.
A user controlled page table should absolutely not be an attribute of a hidden kernel object, nor should two parts of the kernel silently connect to each other via a hidden internal objects like this.
Security is important - the kind of connection must use some explicit FD authorization to access shared objects, not be made implicit!
IMHO this direction is a dead end for this reason.
Quoted 7 lines
Quoted 7 lines
The entire translation path for any ioasid or PASID should be defined only by
/dev/ioasid. Everything else is a legacy API.Quoted 3 lines
If following your suggestion then VFIO must deny VFIO MAP operations on sva1 (assume userspace should not mix sva1 and sva2 in the same container and instead use
/dev/ioasidto map for sva1)?
Quoted 6 lines
Quoted 3 lines
No, userspace creates an iosaid for the guest physical mapping and passes this ioasid to VFIO PCI which will assign it as the first layer mapping on the RID
Is it an dummy ioasid just for providing GPA mappings for nesting purpose of other IOASIDs? Then we waste one per VM?
Generic ioasid's are "free" they are just software constructs in the kernel.
Quoted 6 lines
Quoted 3 lines
When PASIDs are allocated the uAPI will be told to logically nested under the first ioasid. When VFIO authorizes a PASID for a RID it checks that all the HW rules are being followed.
As I explained above, why cannot we just use iommu domain to connect the dots?
Security.
Quoted 4 lines
Every passthrough framework needs to create an iommu domain first. and It needs to support both devices w/ PASID and devices w/o PASID. For devices w/o PASID it needs to invent its own MAP interface anyway.
No, it should consume a ioasid from /dev/ioasid, use a common ioasid map interface and assign that ioasid to a RID.
Don't get so fixated on PASID as a special case
Quoted 4 lines
Then why do we bother creating another MAP interface through
/dev/ioasidwhich not only duplicates but also creating transition burden between two set of MAP interfaces when the guest turns on/off the pasid capability on the device?
Don't transition. Always use the new interface. qemu detects the kernel supports /dev/ioasid and *all iommu page table configuration* goes through there. VFIO and VDPA APIs become unused for iommu configuration.
Quoted 7 lines
'universally' upon from which angle you look at this problem. From IOASID p.o.v possibly yes, but from device passthrough p.o.v. it's the opposite since the passthrough framework needs to handle devices w/o PASID anyway (or even for device w/ PASID it could send traffic w/o PASID) thus 'universally' makes more sense if the passthrough framework can use one interface of its own to manage GPA mappings for all consumers (apply to the case when a PASID is allowed/authorized).
You correctly named it /dev/ioasid, it is a generic way to allocate, manage and assign IOMMU page tables, which when generalized, only some of which may consume a limited PASID.
RID and RID,PASID are the same thing, just a small difference in how they match TLPs.
Jason
RE: [PATCH V4 05/18] iommu/ioasid: Redefine IOASID set and allocation APIs
Quoted 19 lines
From: Jason Gunthorpe <[email protected]>
Sent: Wednesday, April 7, 2021 8:21 PMQuoted 3 lines · Kevin
Quoted 3 lines
Because if you don't then we enter insane world where a PASID is being created under
/dev/ioasidbut its translation path flows through setup done by VFIO and the whole user API becomes an incomprehensiblemess.
Quoted 4 lines
Quoted 2 lines
How will you even associate the PASID with the other translation??
PASID is attached to a specific iommu domain (created by VFIO/VDPA),
which
Quoted 2 lines
has
GPA->HPAmappings already configured. If we view that mapping as an attribute of the iommu domain, it's reasonable to have the userspace-bound
Quoted 1 line
pgtable through
/dev/ioasidto nest on it.
Quoted 9 lines
A user controlled page table should absolutely not be an attribute of a hidden kernel object, nor should two parts of the kernel silently connect to each other via a hidden internal objects like this.
Security is important - the kind of connection must use some explicit FD authorization to access shared objects, not be made implicit!
IMHO this direction is a dead end for this reason.
Could you elaborate what exact security problem is brought with this approach? Isn't ALLOW_PASID the authorization interface for the connection?
Based on all your replies now I see what you actually want is generalizing all IOMMU related stuff through /dev/ioasid (sort of /dev/iommu), which requires factoring out the vfio_iommu_type1 into the general part. This is a huge work.
Is it really the only practice in Linux that any new feature has to be blocked as long as a refactoring work is identified? Don't people accept any balance between enabling new features and completing refactoring work through a staging approach, as long as we don't introduce an uAPI specifically for the staging purpose? ☹
Thanks
Kevin
Re: [PATCH V4 05/18] iommu/ioasid: Redefine IOASID set and allocation APIs
Quoted 26 lines · Kevin
Quoted 26 lines
From: Jason Gunthorpe <[email protected]>
Sent: Wednesday, April 7, 2021 8:21 PMQuoted 3 lines · Kevin
Quoted 3 lines
Because if you don't then we enter insane world where a PASID is being created under
/dev/ioasidbut its translation path flows through setup done by VFIO and the whole user API becomes an incomprehensiblemess.
Quoted 4 lines
Quoted 2 lines
How will you even associate the PASID with the other translation??
PASID is attached to a specific iommu domain (created by VFIO/VDPA),
which
Quoted 2 lines
has
GPA->HPAmappings already configured. If we view that mapping as an attribute of the iommu domain, it's reasonable to have the userspace-bound
Quoted 1 line
pgtable through
/dev/ioasidto nest on it.A user controlled page table should absolutely not be an attribute of a hidden kernel object, nor should two parts of the kernel silently connect to each other via a hidden internal objects like this.
Security is important - the kind of connection must use some explicit FD authorization to access shared objects, not be made implicit!
Quoted 6 lines
Quoted 2 lines
IMHO this direction is a dead end for this reason.
Could you elaborate what exact security problem is brought with this approach? Isn't
ALLOW_PASIDthe authorization interface for the connection?
If the kernel objects don't come out of FDs then no.
Quoted 2 lines
Is it really the only practice in Linux that any new feature has to be blocked as long as a refactoring work is identified?
The practice is to define uAPIs that make sense and have a good chance to be supported over a long time period, as the software evolves, not to hacky hacky a gaint uAPI mess just to get some feature out the door.
This proposal as it was oringial shown is exactly the kind of hacky hacky uapi nobody wants to see. Tunneling an IOMMU uapi through a whole bunch of other FDs is completely nutz.
Intel should basically be investing most of its time building a robust and well designed uAPI here, and don't complain that the community is not doing Intel's job for free.
Quoted 3 lines
Don't people accept any balance between enabling new features and completing refactoring work through a staging approach, as long as we don't introduce an uAPI specifically for the staging purpose? ☹
Since this is all uapi I don't see it as applicable here.
Jason
Re: [PATCH V4 05/18] iommu/ioasid: Redefine IOASID set and allocation APIs
In reply to Jason GunthorpeÔÚ 2021/4/6 ÉÏÎç7:42, Jason Gunthorpe дµÀ:
Quoted 38 lines · Kevin
Quoted 27 lines
Quoted 20 lines
From: Jason Gunthorpe <[email protected]>
Sent: Tuesday, March 30, 2021 9:29 PMQuoted 6 lines
First, userspace may use ioasid in a non-SVA scenario where ioasid is bound to specific security context (e.g. a control vq in vDPA) instead of tying to mm. In this case there is no pgtable binding initiated from user space. Instead, ioasid is allocated from
/dev/ioasidand then programmed to the intended security context through specific passthrough framework which manages that context.This sounds like the exact opposite of what I'd like to see.
I do not want to see every subsystem gaining APIs to program a PASID. All of that should be consolidated in *one place*.
I do not want to see VDPA and VFIO have two nearly identical sets of APIs to control the PASID.
Drivers consuming a PASID, like VDPA, should consume the PASID and do nothing more than authorize the HW to use it.
Quoted 7 lines
quemu should have general code under the viommu driver that drives
/dev/ioasidto create PASID's and manage the IO mapping according to the guest's needs.Drivers like VDPA and VFIO should simply accept that PASID and configure/authorize their HW to do DMA's with its tag.
Quoted 8 lines
I agree with you on consolidating things in one place (especially for the general SVA support). But here I was referring to an usage without pgtable binding (Possibly Jason. W can say more here), where the userspace just wants to allocate PASIDs, program/accept PASIDs to various workqueues (device specific), and then use MAP/UNMAP interface to manage address spaces associated with each PASID. I just wanted to point out that the latter two steps are through VFIO/VDPA specific interfaces.
No, don't do that.
Quoted 8 lines
VFIO and VDPA has no buisness having map/unmap interfaces once we have
/dev/ioasid. That all belongs in the iosaid side.I know they have those interfaces today, but that doesn't mean we have to keep using them for PASID use cases, they should be replaced with a 'do dma from this pasid on
/dev/ioasid' interface certainly not a 'here is a pasid from/dev/ioasid, go ahead and configure it youself' interface
So it looks like the PASID was bound to SVA in this design. I think it's not necessairly the case:
1) PASID can be implemented without SVA, in this case a map/unmap interface is still required
2) For the case that hypervisor want to do some mediation in the middle for a virtqueue. e.g in the case of control vq that is implemented in the VF/ADI/SF itself, the hardware virtqueue needs to be controlled by Qemu, Though binding qemu's page table to cvq can work but it looks like a overkill, a small dedicated buffers that is mapped for this PASID seems more suitalbe.
Quoted 6 lines
This is because PASID is *complicated* in the general case! For instance all the two level stuff you are talking about must not leak into every user!
Jason
So do you mean the device should not expose the PASID confiugration API to guest? I think it could happen if we assign the whole device and let guest to configure it for nested VMs.
Thanks
Quoted 1 line
Re: [PATCH V4 05/18] iommu/ioasid: Redefine IOASID set and allocation APIs
Quoted 11 lines · Jason Wang
Quoted 8 lines
VFIO and VDPA has no buisness having map/unmap interfaces once we have
/dev/ioasid. That all belongs in the iosaid side.I know they have those interfaces today, but that doesn't mean we have to keep using them for PASID use cases, they should be replaced with a 'do dma from this pasid on
/dev/ioasid' interface certainly not a 'here is a pasid from/dev/ioasid, go ahead and configure it youself' interfaceSo it looks like the PASID was bound to SVA in this design. I think it's not necessairly the case:
No, I wish people would stop talking about SVA.
SVA and vSVA are a very special narrow configuration of a PASID. There are lots of other PASID configurations! That is the whole point, a PASID is complicated, there are many configuration scenarios, they need to be in one place with a very clearly defined uAPI
Quoted 2 lines
1) PASID can be implemented without SVA, in this case a map/unmap interface is still required
Any interface to manipulate a PASID should be under /dev/ioasid. We do not want to duplicate this into every subsystem.
Quoted 6 lines
2) For the case that hypervisor want to do some mediation in the middle for a virtqueue. e.g in the case of control vq that is implemented in the VF/ADI/SF itself, the hardware virtqueue needs to be controlled by Qemu, Though binding qemu's page table to cvq can work but it looks like a overkill, a small dedicated buffers that is mapped for this PASID seems more suitalbe.
/dev/ioasid should allow userspace to setup any PASID configuration it wants. There are many choices. That is the whole point, instead of copying&pasting all the PASID configuration option into every subsystem we have on place to configure it.
If you want a PASID (or generic ioasid) that has the guest physical map, which is probably all that VDPA would ever want, then /dev/ioasid should be able to prepare that.
If you just want to map a few buffers into a PASID then it should be able to do that too.
Quoted 3 lines
So do you mean the device should not expose the PASID confiugration API to guest? I think it could happen if we assign the whole device and let guest to configure it for nested VMs.
This always needs co-operating with the vIOMMU driver. We can't have nested PASID use without both parts working together.
The vIOMMU driver configures the PASID and assigns the mappings (however complicated that turns out to be)
The VDPA/mdev driver authorizes the HW to use the ioasid mapping, eg by authorizing a queue to issue PCIe TLPs with a specific PASID.
The authorization is triggered by the guest telling the vIOMMU to allow a vRID to talk to a PASID, which qemu would have to translate to telling something like the VDPA driver under the vRID that it can use a PASID from /dev/ioasid
For security a VDPA/mdev device MUST NOT issue PASIDs that the vIOMMU has not authorized its vRID to use. Otherwise the security model of something like VFIO in the guest becomes completely broken.
Jason
Re: [PATCH V4 05/18] iommu/ioasid: Redefine IOASID set and allocation APIs
ÔÚ 2021/4/6 ÏÂÎç8:42, Jason Gunthorpe дµÀ:
Quoted 11 lines · Jason Wang
Quoted 10 lines
Quoted 9 lines
VFIO and VDPA has no buisness having map/unmap interfaces once we have
/dev/ioasid. That all belongs in the iosaid side.I know they have those interfaces today, but that doesn't mean we have to keep using them for PASID use cases, they should be replaced with a 'do dma from this pasid on
/dev/ioasid' interface certainly not a 'here is a pasid from/dev/ioasid, go ahead and configure it youself' interface
Quoted 8 lines
Quoted 2 lines
So it looks like the PASID was bound to SVA in this design. I think it's not necessairly the case:
No, I wish people would stop talking about SVA.
SVA and vSVA are a very special narrow configuration of a PASID. There are lots of other PASID configurations! That is the whole point, a PASID is complicated, there are many configuration scenarios, they need to be in one place with a very clearly defined uAPI
Right, that's my understanding as well.
Quoted 5 lines
Quoted 3 lines
1) PASID can be implemented without SVA, in this case a map/unmap interface is still required
Any interface to manipulate a PASID should be under
/dev/ioasid. We do not want to duplicate this into every subsystem.
Yes.
Quoted 31 lines
Quoted 7 lines
2) For the case that hypervisor want to do some mediation in the middle for a virtqueue. e.g in the case of control vq that is implemented in the VF/ADI/SF itself, the hardware virtqueue needs to be controlled by Qemu, Though binding qemu's page table to cvq can work but it looks like a overkill, a small dedicated buffers that is mapped for this PASID seems more suitalbe.
/dev/ioasidshould allow userspace to setup any PASID configuration it wants. There are many choices. That is the whole point, instead of copying&pasting all the PASID configuration option into every subsystem we have on place to configure it.If you want a PASID (or generic ioasid) that has the guest physical map, which is probably all that VDPA would ever want, then
/dev/ioasidshould be able to prepare that.If you just want to map a few buffers into a PASID then it should be able to do that too.
Quoted 3 lines
So do you mean the device should not expose the PASID confiugration API to guest? I think it could happen if we assign the whole device and let guest to configure it for nested VMs.
This always needs co-operating with the vIOMMU driver. We can't have nested PASID use without both parts working together.
The vIOMMU driver configures the PASID and assigns the mappings (however complicated that turns out to be)
The VDPA/mdev driver authorizes the HW to use the ioasid mapping, eg by authorizing a queue to issue PCIe TLPs with a specific PASID.
Quoted 8 lines
The authorization is triggered by the guest telling the vIOMMU to allow a vRID to talk to a PASID, which qemu would have to translate to telling something like the VDPA driver under the vRID that it can use a PASID from
/dev/ioasidFor security a VDPA/mdev device MUST NOT issue PASIDs that the vIOMMU has not authorized its vRID to use. Otherwise the security model of something like VFIO in the guest becomes completely broken.
Yes, that's how it should work.
Thanks
Quoted 3 lines
Jason
RE: [PATCH V4 05/18] iommu/ioasid: Redefine IOASID set and allocation APIs
In reply to Jason GunthorpeQuoted 18 lines
From: Jason Gunthorpe
Sent: Tuesday, April 6, 2021 8:43 PMQuoted 1 line · Jason Wang
Quoted 1 line
VFIO and VDPA has no buisness having map/unmap interfaces once we
have
Quoted 10 lines
Quoted 7 lines
/dev/ioasid. That all belongs in the iosaid side.I know they have those interfaces today, but that doesn't mean we have to keep using them for PASID use cases, they should be replaced with a 'do dma from this pasid on
/dev/ioasid' interface certainly not a 'here is a pasid from/dev/ioasid, go ahead and configure it youself' interfaceSo it looks like the PASID was bound to SVA in this design. I think it's not necessairly the case:
Quoted 7 lines
No, I wish people would stop talking about SVA.
SVA and vSVA are a very special narrow configuration of a PASID. There are lots of other PASID configurations! That is the whole point, a PASID is complicated, there are many configuration scenarios, they need to be in one place with a very clearly defined uAPI
I feel it also makes sense to allow a subsystem to specify which configurations are permitted when allowing a PASID on its device, e.g. excluding things like GPA mappings that existing subsystems (VFIO/VDPA) already handle well:
- Share GPA mappings between multiple devices (w/ or w/o PASID) for better IOTLB efficiency;
- Share GPA mappings between transactions w/ PASID and transactions w/o PASID from the same device (e.g. GPU) for better IOTLB efficiency;
- Use the same page table for GPA mappings before and after the guest turns on/off the PASID capability;
All above are given as long as we continue to let VFIO/VDPA manage the iommu domain and associated GPA mappings for PASID. The IOMMU driver already ensures a nested PASID entry linking to the established GPA paging structure of the domain when the 1st-level pgtable is bound through /dev/ioasid.
In contrast, above merits are lost if forcing a model where GPA mappings for PASID must be constructed through /dev/ioasid, as this will lead to multiple paging structures for the same GPA mappings implying worse IOTLB usage and unnecessary cost of invalidations.
Therefore, I envision a scheme where the subsystem could specify permitted PASID configurations when doing ALLOW_PASID, and then userspace queries per-PASID capability to learn which operations are allowed, e.g.:
1) To enable vSVA, VFIO/VDPA allows pgtable binding and related invalidation/ fault ops through /dev/ioasid;
2) for vDPA control vq usage, no configuration is allowed through /dev/ioasid;
3) for new subsystem which doesn't carry any legacy or similar usage as VFIO/VDPA, it could permit all configurations through /dev/ioasid including 1st-level binding and 2nd-level mapping ops;
This approach also allows us to grow the uAPI in a staging approach. Now focus on 1) and 2) as VFIO/VDPA are the only two users for now with good legacy to cover the GPA mappings. More ops can be introduced for 3) when there is a real example to show what exact ops are required for such a new subsystem.
Is this a good strategy to move forward?
btw this discussion was raised when discussing the I/O page fault handling process. Currently the IOMMU layer implements a per-device fault reporting mechanism, which requires VFIO to register a handler to receive all faults on its device and then forwards to ioasid if it's due to 1st-level. Possibly it makes more sense to convert it into a per-pgtable reporting scheme, and then the owner of each pgtable should register its own handler. It means for 1) VFIO will register a 2nd-level pgtable handler while /dev/ioasid will register a 1st-level pgtable handler, while for 3) /dev/ioasid will register handlers for both 1st-level and 2nd-level pgtable. Jean? also want to know your thoughts...
Thanks
Kevin
Re: [PATCH V4 05/18] iommu/ioasid: Redefine IOASID set and allocation APIs
Quoted 20 lines · Kevin
Quoted 20 lines
From: Jason Gunthorpe
Sent: Tuesday, April 6, 2021 8:43 PMQuoted 1 line · Jason Wang
Quoted 1 line
VFIO and VDPA has no buisness having map/unmap interfaces once we
have
Quoted 10 lines
Quoted 7 lines
/dev/ioasid. That all belongs in the iosaid side.I know they have those interfaces today, but that doesn't mean we have to keep using them for PASID use cases, they should be replaced with a 'do dma from this pasid on
/dev/ioasid' interface certainly not a 'here is a pasid from/dev/ioasid, go ahead and configure it youself' interfaceSo it looks like the PASID was bound to SVA in this design. I think it's not necessairly the case:
No, I wish people would stop talking about SVA.
Quoted 8 lines
Quoted 5 lines
SVA and vSVA are a very special narrow configuration of a PASID. There are lots of other PASID configurations! That is the whole point, a PASID is complicated, there are many configuration scenarios, they need to be in one place with a very clearly defined uAPI
I feel it also makes sense to allow a subsystem to specify which configurations are permitted when allowing a PASID on its device
huh? why?
Quoted 2 lines
e.g. excluding things like
GPA mappings that existing subsystems (VFIO/VDPA) already handle well:
They don't "handle well", they have some historical baggage that is no longer suitable for the complexity this area has in the modern world.
Forget about the existing APIs and replace them in /dev/ioasid.
Quoted 8 lines
- Share GPA mappings between multiple devices (w/ or w/o PASID) for better IOTLB efficiency;
- Share GPA mappings between transactions w/ PASID and transactions w/o PASID from the same device (e.g. GPU) for better IOTLB efficiency;
- Use the same page table for GPA mappings before and after the guest turns on/off the PASID capability;
All of these are cases you need to design the /dev/ioasid to handle.
It is pretty clear to me that you'll need non-PASID IOASID's as well.
Ideally a generic IOASID would just be a page table and it doesn't crystalize into a RID or RID,PASID routing until devices are attached to it.
Since IOASID can be nested the only thing that makes any sense is for each level of the nest to be visible under /dev/ioasid.
What a complete mess it would be if vfio-pci owns the GPA table, /dev/ioasid has a nested PASID, and vfio-mdev is running a mdev on top of that PASID.
Quoted 2 lines
All above are given as long as we continue to let VFIO/VDPA manage the iommu domain and associated GPA mappings for PASID.
So don't do that. Don't I keep saying this weird split is making a horrible mess?
You can't reasonably build the complex PASID scenarios you talk about above unless the entire translation path is owned by one entity:/dev/ioasid.
You need to focus on figuring out what that looks like then figure out how to move VDPA and VFIO to consume /dev/ioasid for all of their translation instead of open-coding half-baked internal versions.
Jason
Re: [PATCH V4 05/18] iommu/ioasid: Redefine IOASID set and allocation APIs
In reply to Tian, KevinQuoted 6 lines · Kevin
btw this discussion was raised when discussing the I/O page fault handling process. Currently the IOMMU layer implements a per-device fault reporting mechanism, which requires VFIO to register a handler to receive all faults on its device and then forwards to ioasid if it's due to 1st-level. Possibly it makes more sense to convert it into a per-pgtable reporting scheme, and then the owner of each pgtable should register its own handler.
Maybe, but you do need device information in there, since that's how the fault is reported to the guest and how the response is routed back to the faulting device (only PASID+PRGI would cause aliasing). And we need to report non-recoverable faults, as well as recoverable ones without PASID, once we hand control of level-1 page tables to guests.
Quoted 5 lines
It means
for 1) VFIO will register a 2nd-level pgtable handler while/dev/ioasidwill register a 1st-level pgtable handler, while for 3)/dev/ioasidwill register handlers for both 1st-level and 2nd-level pgtable. Jean? also want to know your thoughts...
Moving all IOMMU controls to /dev/ioasid rather that splitting them is probably better. Hopefully the implementation can reuse most of vfio_iommu_type1.
I'm trying to sketch what may work for Arm, if we have to reuse /dev/ioasid to avoid duplication of fault and inval queues:
* Get a container handle out of /dev/ioasid (or /dev/iommu, really.) No operation available since we don't know what the device and IOMMU capabilities are.
* Attach the handle to a VF. With VFIO that would be VFIO_GROUP_SET_CONTAINER. That causes the kernel to associate an IOMMU with the handle, and decide which operations are available.
* With a map/unmap vIOMMU (or shadow mappings), a single translation level is supported. With a nesting vIOMMU, we're populating the level-2 translation (some day maybe by binding the KVM page tables, but currently with map/unmap ioctl).
Single-level translation needs single VF per container. Two level would allow sharing stage-2 between multiple VFs, though it's a pain to define and implement.
* Without a vIOMMU or if the vIOMMU starts in bypass, populate the container page tables.
Start the guest.
* With a map/unmap vIOMMU, guest creates mappings, userspace populates the page tables with map/unmap ioctl.
It would be possible to add a PASID mode there: guest requests an address space with a specific PASID, userspace derives an IOASID handle from the container handle and populate that address space with map/unmap ioctl. That would enable PASID on sub-VF assignment, which requires the host to control which PASID is programmed into the VF (with DEVICE_ALLOW_IOASID, I guess). And either the host allocates the PASID in this case (which isn't supported by a vSMMU) or we have to do a vPASID -> pPASID. I don't know if it's worth the effort.
Or
* With a nesting vIOMMU, the guest attaches a PASID table to a VF, userspace issues a SET_PASID_TABLE ioctl on the container handle. If we support multiple VFs per container, we first need to derive a child container from the main one and the device, then attach the PASID table.
Guest programs the PASID table, sends invalidations when removing mappings which are relayed to the host on the child container. Page faults and response queue would be per container, so if multiple VF per container, we could have one queue for the parent (level-2 faults) and one for each child (level-1 faults).
Thanks,
Jean
Re: [PATCH V4 05/18] iommu/ioasid: Redefine IOASID set and allocation APIs
Quoted 7 lines · Jean-Philippe Brucker
* Get a container handle out of
/dev/ioasid(or/dev/iommu, really.) No operation available since we don't know what the device and IOMMU capabilities are.* Attach the handle to a VF. With VFIO that would be
VFIO_GROUP_SET_CONTAINER. That causes the kernel to associate an IOMMU with the handle, and decide which operations are available.
Right, this is basically the point, - the VFIO container (/dev/vfio) and the /dev/ioasid we are talking about have a core of similarity. ioasid is the generalized, modernized, and cross-subsystem version of the same idea. Instead of calling it "vfio container" we call it something that evokes the idea of controlling the iommu.
The issue is to seperate /dev/vfio generic functionality from vfio and share it with every subsystem.
It may be that /dev/vfio and /dev/ioasid end up sharing a lot of code, with a different IOCTL interface around it. The vfio_iommu_driver_ops is not particularly VFIOy.
Creating /dev/ioasid may primarily start as a code reorganization exercise.
Quoted 6 lines
* With a map/unmap vIOMMU (or shadow mappings), a single translation level is supported. With a nesting vIOMMU, we're populating the level-2 translation (some day maybe by binding the KVM page tables, but currently with map/unmap ioctl).
Single-level translation needs single VF per container.
Really? Why?
Jason
Re: [PATCH V4 05/18] iommu/ioasid: Redefine IOASID set and allocation APIs
Quoted 26 lines · Jason Gunthorpe
Quoted 7 lines · Jean-Philippe Brucker
* Get a container handle out of
/dev/ioasid(or/dev/iommu, really.) No operation available since we don't know what the device and IOMMU capabilities are.* Attach the handle to a VF. With VFIO that would be
VFIO_GROUP_SET_CONTAINER. That causes the kernel to associate an IOMMU with the handle, and decide which operations are available.Right, this is basically the point, - the VFIO container (
/dev/vfio) and the/dev/ioasidwe are talking about have a core of similarity. ioasid is the generalized, modernized, and cross-subsystem version of the same idea. Instead of calling it "vfio container" we call it something that evokes the idea of controlling the iommu.The issue is to seperate
/dev/vfiogeneric functionality from vfio and share it with every subsystem.It may be that
/dev/vfioand/dev/ioasidend up sharing a lot of code, with a different IOCTL interface around it. Thevfio_iommu_driver_opsis not particularly VFIOy.Creating
/dev/ioasidmay primarily start as a code reorganization exercise.
Quoted 8 lines
Quoted 6 lines
* With a map/unmap vIOMMU (or shadow mappings), a single translation level is supported. With a nesting vIOMMU, we're populating the level-2 translation (some day maybe by binding the KVM page tables, but currently with map/unmap ioctl).
Single-level translation needs single VF per container.
Really? Why?
The vIOMMU is started in bypass, so the device can do DMA to the GPA space until the guest configures the vIOMMU, at which point each VF is either kept in bypass or gets new DMA mappings, which requires the host to tear down the bypass mappings and set up the guest mappings on a per-VF basis (I'm not considering nesting translation in the host kernel for this, because it's not supported by all pIOMMUs and is expensive in terms of TLB and pinned memory). So keeping a single VF per container is simpler, but there are certainly other programming models possible.
Thanks,
Jean
RE: [PATCH V4 05/18] iommu/ioasid: Redefine IOASID set and allocation APIs
In reply to Jason GunthorpeQuoted 25 lines
From: Jason Gunthorpe <[email protected]>
Sent: Tuesday, March 30, 2021 12:32 AMQuoted 2 lines
In terms of usage for guest SVA, an
ioasid_setis mostly tied to a host mm, the use case is as the following:From that doc:
It is imperative to enforce
VM-IOASID ownership such that a malicious guest cannot target DMA traffic outside its own IOASIDs, or free an active IOASID that belongs to another VM.Huh?
Security in a PASID world comes from the IOMMU blocking access to the PASID except from approved PCI-ID's. If a VF/PF is assigned to a guest then that guest can cause the device to issue any PASID by having complete control and the vIOMMU is supposed to tell the real IOMMU what PASID's the device is alowed to access.
If a device is sharing a single PCI function with different security contexts (eg vfio mdev) then the device itself is responsible to ensure that only the secure interface can program a PASID and a less secure context can never self-enroll.
Quoted 9 lines
Here the mdev driver would have to consule with the vIOMMU to ensure the mdev device is allowed to access the PASID - is that what this set stuff is about?
If yes, it is backwards. The MDEV is the thing doing the security, the MDEV should have the list of allowed PASID's and a single PASID created under
/dev/ioasidshould be loaded into MDEV with some 'Ok you can use PASID xyz from FD abc' command.
The 'set' is per-VM. Once the mdev is assigned to a VM, all valid PASID's in the set of that VM are considered legitimate on this mdev. The mdev driver will mediate guest operations which program PASID to the backend context and load the PASID only if it is within the 'set' (i.e. already allocated through /dev/ioasid). This prevents a malicious VM from attacking others. Though it's not mdev which directly maintaining the list of allowed PASID's, the effect is the same in concept.
Thanks
Kevin
Re: [PATCH V4 05/18] iommu/ioasid: Redefine IOASID set and allocation APIs
Quoted 29 lines · Kevin
Quoted 29 lines
From: Jason Gunthorpe <[email protected]>
Sent: Tuesday, March 30, 2021 12:32 AMQuoted 2 lines
In terms of usage for guest SVA, an
ioasid_setis mostly tied to a host mm, the use case is as the following:From that doc:
It is imperative to enforce
VM-IOASID ownership such that a malicious guest cannot target DMA traffic outside its own IOASIDs, or free an active IOASID that belongs to another VM.Huh?
Security in a PASID world comes from the IOMMU blocking access to the PASID except from approved PCI-ID's. If a VF/PF is assigned to a guest then that guest can cause the device to issue any PASID by having complete control and the vIOMMU is supposed to tell the real IOMMU what PASID's the device is alowed to access.
If a device is sharing a single PCI function with different security contexts (eg vfio mdev) then the device itself is responsible to ensure that only the secure interface can program a PASID and a less secure context can never self-enroll.
Here the mdev driver would have to consule with the vIOMMU to ensure the mdev device is allowed to access the PASID - is that what this set stuff is about?
Quoted 8 lines
Quoted 5 lines
If yes, it is backwards. The MDEV is the thing doing the security, the MDEV should have the list of allowed PASID's and a single PASID created under
/dev/ioasidshould be loaded into MDEV with some 'Ok you can use PASID xyz from FD abc' command.The 'set' is per-VM. Once the mdev is assigned to a VM, all valid PASID's in the set of that VM are considered legitimate on this mdev.
No! That is a major security problem!
PASID authorization is *PER DEVICE*.
If I map a device into VFIO in userspace with full control over the HW that device MUST ONLY have access to PASID's that have been registered with vfio.
This means each time you register a PASID vfio must tell the IOMMU driver to authorize the pci_device to access the PASID, the vIOMMU driver must tell the hypervisor and the mdev under the PCI device MUST have a per-device list of allowed PASIDs.
Otherwise userspace in a VM with vfio could tell the mdev driver to talk to a PASID in the same VM but *that process doesn't own*. This is absolutely not allowed.
Most likely the entire ioasid set and related need to be deleted as a kernel concept.
Jason
RE: [PATCH V4 05/18] iommu/ioasid: Redefine IOASID set and allocation APIs
In reply to Jason GunthorpeQuoted 38 lines
From: Tian, Kevin
Sent: Tuesday, March 30, 2021 10:24 AMQuoted 25 lines
From: Jason Gunthorpe <[email protected]>
Sent: Tuesday, March 30, 2021 12:32 AMQuoted 2 lines
In terms of usage for guest SVA, an
ioasid_setis mostly tied to a host mm, the use case is as the following:From that doc:
It is imperative to enforce
VM-IOASID ownership such that a malicious guest cannot target DMA traffic outside its own IOASIDs, or free an active IOASID that belongs to another VM.Huh?
Security in a PASID world comes from the IOMMU blocking access to the PASID except from approved PCI-ID's. If a VF/PF is assigned to a guest then that guest can cause the device to issue any PASID by having complete control and the vIOMMU is supposed to tell the real IOMMU what PASID's the device is alowed to access.
If a device is sharing a single PCI function with different security contexts (eg vfio mdev) then the device itself is responsible to ensure that only the secure interface can program a PASID and a less secure context can never self-enroll.
Quoted 9 lines
Here the mdev driver would have to consule with the vIOMMU to ensure the mdev device is allowed to access the PASID - is that what this set stuff is about?
If yes, it is backwards. The MDEV is the thing doing the security, the MDEV should have the list of allowed PASID's and a single PASID created under
/dev/ioasidshould be loaded into MDEV with some 'Ok you can use PASID xyz from FD abc' command.
Quoted 8 lines
The 'set' is per-VM. Once the mdev is assigned to a VM, all valid PASID's in the set of that VM are considered legitimate on this mdev. The mdev driver will mediate guest operations which program PASID to the backend context and load the PASID only if it is within the 'set' (i.e. already allocated through
/dev/ioasid). This prevents a malicious VM from attacking others. Though it's not mdev which directly maintaining the list of allowed PASID's, the effect is the same in concept.
One correction. The mdev should still construct the list of allowed PASID's as you said (by listening to IOASID_BIND/UNBIND event), in addition to the ioasid set maintained per VM (updated when a PASID is allocated/freed). The per-VM set is required for inter-VM isolation (verified when a pgtable is bound to the mdev/PASID), while the mdev's own list is necessary for intra-VM isolation when multiple mdevs are assigned to the same VM (verified before loading a PASID to the mdev). This series just handles the general part i.e. per-VM ioasid set and leaves the mdev's own list to be managed by specific mdev driver which listens to various IOASID events).
Thanks
Kevin
Re: [PATCH V4 05/18] iommu/ioasid: Redefine IOASID set and allocation APIs
Quoted 9 lines · Kevin
One correction. The mdev should still construct the list of allowed PASID's as you said (by listening to IOASID_BIND/UNBIND event), in addition to the ioasid set maintained per VM (updated when a PASID is allocated/freed). The per-VM set is required for inter-VM isolation (verified when a pgtable is bound to the mdev/PASID), while the mdev's own list is necessary for intra-VM isolation when multiple mdevs are assigned to the same VM (verified before loading a PASID to the mdev). This series just handles the general part i.e. per-VM ioasid set and leaves the mdev's own list to be managed by specific mdev driver which listens to various IOASID events).
This is better, but I don't understand why we need such a convoluted design.
Get rid of the ioasid set.
Each driver has its own list of allowed ioasids.
Register a ioasid in the driver's list by passing the fd and ioasid #
No listening to events. A simple understandable security model.
Look - it took you three emails to even correctly explain the security model you are striving for here, it is *obviously* too complicated for anyone to understand or successfully implement. simplify smiplify simplify.
Jason
RE: [PATCH V4 05/18] iommu/ioasid: Redefine IOASID set and allocation APIs
Quoted 21 lines
From: Jason Gunthorpe <[email protected]>
Sent: Tuesday, March 30, 2021 9:28 PMQuoted 1 line · Kevin
One correction. The mdev should still construct the list of allowed PASID's
as
Quoted 1 line
you said (by listening to IOASID_BIND/UNBIND event), in addition to the
ioasid
Quoted 1 line
set maintained per VM (updated when a PASID is allocated/freed). The
per-VM
Quoted 1 line
set is required for inter-VM isolation (verified when a pgtable is bound to
the
Quoted 1 line
mdev/PASID), while the mdev's own list is necessary for intra-VM isolation
when
Quoted 1 line
multiple mdevs are assigned to the same VM (verified before loading a
PASID
Quoted 1 line
to the mdev). This series just handles the general part i.e. per-VM ioasid
set and
Quoted 1 line
leaves the mdev's own list to be managed by specific mdev driver which
listens
Quoted 8 lines
Quoted 1 line
to various IOASID events).
This is better, but I don't understand why we need such a convoluted design.
Get rid of the ioasid set.
Each driver has its own list of allowed ioasids.
First, I agree with you it's necessary to have a per-device allowed ioasid list. But besides it, I think we still need to ensure the ioasid used by a VM is really allocated to this VM. A VM should not use an ioasid allocated to another VM. right? Actually, this is the major intention for introducing ioasid_set.
Quoted 1 line
Register a ioasid in the driver's list by passing the fd and ioasid #
The fd here is a device fd. Am I right? If yes, your idea is ioasid is allocated via /dev/ioasid and associated with device fd via either VFIO or vDPA ioctl. right? sorry I may be asking silly questions but really need to ensure we are talking in the same page.
Quoted 1 line
No listening to events. A simple understandable security model.
For this suggestion, I have a little bit concern if we may have A-B/B-A lock sequence issue since it requires the /dev/ioasid (if it supports) to call back into VFIO/VDPA to check if the ioasid has been registered to device FD and record it in the per-device list. right? Let's have more discussion based on the skeleton sent by Kevin.
Regards,
Yi Liu
Re: [PATCH V4 05/18] iommu/ioasid: Redefine IOASID set and allocation APIs
Quoted 28 lines · Yi L
Quoted 28 lines
From: Jason Gunthorpe <[email protected]>
Sent: Tuesday, March 30, 2021 9:28 PMQuoted 1 line · Kevin
One correction. The mdev should still construct the list of allowed PASID's
as
Quoted 1 line
you said (by listening to IOASID_BIND/UNBIND event), in addition to the
ioasid
Quoted 1 line
set maintained per VM (updated when a PASID is allocated/freed). The
per-VM
Quoted 1 line
set is required for inter-VM isolation (verified when a pgtable is bound to
the
Quoted 1 line
mdev/PASID), while the mdev's own list is necessary for intra-VM isolation
when
Quoted 1 line
multiple mdevs are assigned to the same VM (verified before loading a
PASID
Quoted 1 line
to the mdev). This series just handles the general part i.e. per-VM ioasid
set and
Quoted 1 line
leaves the mdev's own list to be managed by specific mdev driver which
listens
Quoted 1 line
to various IOASID events).
This is better, but I don't understand why we need such a convoluted design.
Get rid of the ioasid set.
Quoted 7 lines
Quoted 1 line
Each driver has its own list of allowed ioasids.
First, I agree with you it's necessary to have a per-device allowed ioasid list. But besides it, I think we still need to ensure the ioasid used by a VM is really allocated to this VM. A VM should not use an ioasid allocated to another VM. right? Actually, this is the major intention for introducing
ioasid_set.
The /dev/ioasid FD replaces this security check. By becoming FD centric you don't need additional kernel security objects.
Any process with access to the /dev/ioasid FD is allowed to control those PASID. The seperation between VMs falls naturally from the seperation of FDs without creating additional, complicated, security infrastrucure in the kernel.
This is why all APIs must be FD focused, and you need to have a logical layering of responsibility.
Allocate a /dev/ioasid FD
Allocate PASIDs inside the FD
Assign memory to the PASIDS
Open a device FD, eg from VFIO or VDP
Instruct the device FD to authorize the device to access PASID A in an ioasid FD
* Prior to being authorized the device will have NO access to any PASID
* Presenting BOTH the device FD and the ioasid FD to the kernel is the security check. Any process with both FDs is allowed to make the connection. This is normal Unix FD centric thinking.
Quoted 3 lines
Quoted 1 line
Register a ioasid in the driver's list by passing the fd and ioasid #
The fd here is a device fd. Am I right?
It would be the vfio_device FD, for instance, and a VFIO IOCTL.
Quoted 4 lines
If yes, your idea is ioasid is allocated via
/dev/ioasidand associated with device fd via either VFIO or vDPA ioctl. right? sorry I may be asking silly questions but really need to ensure we are talking in the same page.
Yes, this is right
Quoted 7 lines
Quoted 1 line
No listening to events. A simple understandable security model.
For this suggestion, I have a little bit concern if we may have A-B/B-A lock sequence issue since it requires the
/dev/ioasid(if it supports) to call back into VFIO/VDPA to check if the ioasid has been registered to device FD and record it in the per-device list. right? Let's have more discussion based on the skeleton sent by Kevin.
Callbacks would be backwards.
User calls vfio with vfio_device fd and dev/ioasid fd
VFIO extracts some kernel representation of the ioasid from the ioasid fd using an API
VFIO does some kernel call to IOMMU/IOASID layer that says 'tell the IOMMU that this PCI device is allowed to use this PASID'
VFIO mdev drivers then record that the PASID is allowed in its own device specific struct for later checking during other system calls.
No lock inversions. No callbacks. Why do we need callbacks?? Simplify.
Jason
Re: [PATCH V4 05/18] iommu/ioasid: Redefine IOASID set and allocation APIs
Hi Jason,
Quoted 16 lines · Jason Gunthorpe
Quoted 3 lines
Quoted 3 lines
Get rid of the ioasid set.
Each driver has its own list of allowed ioasids.
[...]
The
/dev/ioasidFD replaces this security check. By becoming FD centric you don't need additional kernel security objects.Any process with access to the
/dev/ioasidFD is allowed to control those PASID. The seperation between VMs falls naturally from the seperation of FDs without creating additional, complicated, security infrastrucure in the kernel.This is why all APIs must be FD focused, and you need to have a logical layering of responsibility.
Quoted 7 lines
Allocate a
/dev/ioasidFD
Allocate PASIDs inside the FD
Assign memory to the PASIDSOpen a device FD, eg from VFIO or VDP
Instruct the device FD to authorize the device to access PASID A in an ioasid FD
How do we know user provided PASID A was allocated by the ioasid FD? Shouldn't we validate user input by tracking which PASIDs are allocated by which ioasid FD? This is one reason why we have ioasid_set and its xarray.
Quoted 28 lines
* Prior to being authorized the device will have NO access to any PASID
* Presenting BOTH the device FD and the ioasid FD to the kernel is the security check. Any process with both FDs is allowed to make the connection. This is normal Unix FD centric thinking.Quoted 4 lines
Quoted 2 lines
Register a ioasid in the driver's list by passing the fd and ioasid #
The fd here is a device fd. Am I right?
It would be the
vfio_deviceFD, for instance, and a VFIO IOCTL.Quoted 4 lines
If yes, your idea is ioasid is allocated via
/dev/ioasidand associated with device fd via either VFIO or vDPA ioctl. right? sorry I may be asking silly questions but really need to ensure we are talking in the same page.Yes, this is right
Quoted 7 lines
Quoted 1 line
No listening to events. A simple understandable security model.
For this suggestion, I have a little bit concern if we may have A-B/B-A lock sequence issue since it requires the
/dev/ioasid(if it supports) to call back into VFIO/VDPA to check if the ioasid has been registered to device FD and record it in the per-device list. right? Let's have more discussion based on the skeleton sent by Kevin.
Quoted 7 lines
Callbacks would be backwards.
User calls vfio with
vfio_devicefd and dev/ioasid fdVFIO extracts some kernel representation of the ioasid from the ioasid fd using an API
This lookup API seems to be asking for per ioasid FD storage array. Today, the ioasid_set is per mm and contains a Xarray. Since each VM, KVM can only open one ioasid FD, this per FD array would be equivalent to the per mm ioasid_set, right?
Quoted 3 lines
VFIO does some kernel call to IOMMU/IOASID layer that says 'tell the IOMMU that this PCI device is allowed to use this PASID'
Would it be redundant to what iommu_uapi_sva_bind_gpasid() does? I thought the idea is to use ioasid FD IOCTL to issue IOMMU uAPI calls. Or we can skip this step for now and wait for the user to do SVA bind.
Quoted 2 lines
VFIO mdev drivers then record that the PASID is allowed in its own device specific struct for later checking during other system calls.
Thanks,
Jacob
_______________________________________________
iommu mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/iommu
Re: [PATCH V4 05/18] iommu/ioasid: Redefine IOASID set and allocation APIs
Quoted 20 lines · Jacob Pan
Hi Jason,
Quoted 16 lines · Jason Gunthorpe
Quoted 3 lines
Quoted 3 lines
Get rid of the ioasid set.
Each driver has its own list of allowed ioasids.
[...]
The
/dev/ioasidFD replaces this security check. By becoming FD centric you don't need additional kernel security objects.Any process with access to the
/dev/ioasidFD is allowed to control those PASID. The seperation between VMs falls naturally from the seperation of FDs without creating additional, complicated, security infrastrucure in the kernel.This is why all APIs must be FD focused, and you need to have a logical layering of responsibility.
Quoted 8 lines
Quoted 7 lines
Allocate a
/dev/ioasidFD
Allocate PASIDs inside the FD
Assign memory to the PASIDSOpen a device FD, eg from VFIO or VDP
Instruct the device FD to authorize the device to access PASID A in an ioasid FDHow do we know user provided PASID A was allocated by the ioasid FD?
You pass in the ioasid FD and use a 'get pasid from fdno' API to extract the required kernel structure.
Quoted 2 lines
Shouldn't we validate user input by tracking which PASIDs are allocated by which ioasid FD?
Yes, but it is integral to the ioasid FD, not something separated.
Quoted 5 lines
Quoted 3 lines
VFIO extracts some kernel representation of the ioasid from the ioasid fd using an API
This lookup API seems to be asking for per ioasid FD storage array. Today, the
ioasid_setis per mm and contains a Xarray.
Right, put the xarray per FD. A set per mm is fairly nonsensical, we don't use the mm as that kind of security key.
Quoted 2 lines
Since each VM, KVM can only open one ioasid FD, this per FD array would be equivalent to the per mm
ioasid_set, right?
Why only one? Each interaction with the other FDs should include the PASID/FD pair. There is no restriction to just one.
Quoted 6 lines
Quoted 2 lines
VFIO does some kernel call to IOMMU/IOASID layer that says 'tell the IOMMU that this PCI device is allowed to use this PASID'
Would it be redundant to what
iommu_uapi_sva_bind_gpasid()does? I thought the idea is to use ioasid FD IOCTL to issue IOMMU uAPI calls. Or we can skip this step for now and wait for the user to do SVA bind.
I'm not sure what you are asking.
Possibly some of the IOMMU API will need a bit adjusting to make things split.
The act of programming the page tables and the act of authorizing a PCI BDF to use a PASID are distinct things with two different IOCTLs.
iommu_uapi_sva_bind_gpasid() is never called by anything, and it's uAPI is never implemented.
Joerg? Why did you merge dead uapi and dead code?
Jason
Re: [PATCH V4 05/18] iommu/ioasid: Redefine IOASID set and allocation APIs
Hi Jason,
Quoted 18 lines · Jason Gunthorpe
Quoted 17 lines · Jacob Pan
Hi Jason,
Quoted 13 lines · Jason Gunthorpe
Quoted 3 lines
Quoted 3 lines
Get rid of the ioasid set.
Each driver has its own list of allowed ioasids.
[...]
The
/dev/ioasidFD replaces this security check. By becoming FD centric you don't need additional kernel security objects.Any process with access to the
/dev/ioasidFD is allowed to control those PASID. The seperation between VMs falls naturally from the seperation of FDs without creating additional, complicated, security infrastrucure in the kernel.
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
This is why all APIs must be FD focused, and you need to have a logical layering of responsibility.
Allocate a
/dev/ioasidFD
Allocate PASIDs inside the FD
Just to be super clear. Do we allocate a FD for each PASID and return the FD to the user? Or return the plain PASID number back to the user space?
Quoted 10 lines
Quoted 6 lines
Quoted 5 lines
Assign memory to the PASIDS
Open a device FD, eg from VFIO or VDP
Instruct the device FD to authorize the device to access PASID A in an ioasid FDHow do we know user provided PASID A was allocated by the ioasid FD?
You pass in the ioasid FD and use a 'get pasid from fdno' API to extract the required kernel structure.
Seems you are talking about two FDs:
- /dev/ioasid FD
- per IOASID FD
This API ioasid = get_pasid_from_fd(dev_ioasid_fd, ioasid_fd); dev_ioasid_fd will find the xarray for all the PASIDs allocated under it, ioasid_fd wil be the index into the xarray to retrieve the actual ioasid. Correct?
Quoted 5 lines
Quoted 2 lines
Shouldn't we validate user input by tracking which PASIDs are allocated by which ioasid FD?
Yes, but it is integral to the ioasid FD, not something separated.
OK, if we have per IOASID FD in addition to the /dev/ioasid FD we can validate user input.
Quoted 9 lines
Quoted 5 lines
Quoted 3 lines
VFIO extracts some kernel representation of the ioasid from the ioasid fd using an API
This lookup API seems to be asking for per ioasid FD storage array. Today, the
ioasid_setis per mm and contains a Xarray.Right, put the xarray per FD. A set per mm is fairly nonsensical, we don't use the mm as that kind of security key.
Sounds good, one xarray per /dev/ioasid FD.
Quoted 6 lines
Quoted 2 lines
Since each VM, KVM can only open one ioasid FD, this per FD array would be equivalent to the per mm
ioasid_set, right?Why only one? Each interaction with the other FDs should include the PASID/FD pair. There is no restriction to just one.
OK, one per subsystem-VM. For example, if a VM has a VFIO and a VDPA device, it should only two /dev/ioasid FDs respectively. Correct?
Quoted 7 lines
Quoted 6 lines
Quoted 2 lines
VFIO does some kernel call to IOMMU/IOASID layer that says 'tell the IOMMU that this PCI device is allowed to use this PASID'
Would it be redundant to what
iommu_uapi_sva_bind_gpasid()does? I thought the idea is to use ioasid FD IOCTL to issue IOMMU uAPI calls. Or we can skip this step for now and wait for the user to do SVA bind.
Quoted 8 lines
I'm not sure what you are asking.
Possibly some of the IOMMU API will need a bit adjusting to make things split.
The act of programming the page tables and the act of authorizing a PCI BDF to use a PASID are distinct things with two different IOCTLs.
Why separate? I don't see a use case to just authorize a PASID but don't bind it with a page table. The very act of bind page table *is* the authorization.
Quoted 3 lines
iommu_uapi_sva_bind_gpasid()is never called by anything, and it's uAPI is never implemented.
Just a little background here. We have been working on the vSVA stack since 2017. At the time, VFIO was the de facto interface for IOMMU-aware driver framework. These uAPIs were always developed alone side with the accompanying VFIO patches served as consumers. By the time these IOMMU uAPIs were merged after reviews from most vendors, the VFIO patchset was approaching maturity in around v7. This is when we suddenly saw a new request to support VDPA, which attempted VFIO earlier but ultimately moved away.
For a complex stack like vSVA, I feel we have to reduce moving parts and do some divide and conquer.
Quoted 3 lines
Joerg? Why did you merge dead uapi and dead code?
Jason
Thanks,
Jacob
_______________________________________________
iommu mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/iommu
Re: [PATCH V4 05/18] iommu/ioasid: Redefine IOASID set and allocation APIs
Quoted 22 lines · Jacob Pan
Hi Jason,
Quoted 18 lines · Jason Gunthorpe
Quoted 17 lines · Jacob Pan
Hi Jason,
Quoted 13 lines · Jason Gunthorpe
Quoted 3 lines
Quoted 3 lines
Get rid of the ioasid set.
Each driver has its own list of allowed ioasids.
[...]
The
/dev/ioasidFD replaces this security check. By becoming FD centric you don't need additional kernel security objects.Any process with access to the
/dev/ioasidFD is allowed to control those PASID. The seperation between VMs falls naturally from the seperation of FDs without creating additional, complicated, security infrastrucure in the kernel.
Quoted 7 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
This is why all APIs must be FD focused, and you need to have a logical layering of responsibility.
Allocate a
/dev/ioasidFD
Allocate PASIDs inside the FDJust to be super clear. Do we allocate a FD for each PASID and return the FD to the user? Or return the plain PASID number back to the user space?
I would do multiple PASID's per /dev/ioasid FD because we expect alot of PASIDs to be in use and we'd run into FDno limits.
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Assign memory to the PASIDS
Open a device FD, eg from VFIO or VDP
Instruct the device FD to authorize the device to access PASID A in an ioasid FD
Quoted 7 lines
Quoted 5 lines
Quoted 1 line
How do we know user provided PASID A was allocated by the ioasid FD?
You pass in the ioasid FD and use a 'get pasid from fdno' API to extract the required kernel structure.
Seems you are talking about two FDs:
-/dev/ioasidFD
No, just this one.
Quoted 5 lines
- per IOASID FD
This API ioasid =get_pasid_from_fd(dev_ioasid_fd, ioasid_fd);dev_ioasid_fdwill find the xarray for all the PASIDs allocated under it,ioasid_fdwil be the index into the xarray to retrieve the actual ioasid. Correct?
'ioasid_fd' is just the ioasid number in whatever numberspace the /dev/ioasid FD's use.
Quoted 5 lines
Quoted 2 lines
Why only one? Each interaction with the other FDs should include the PASID/FD pair. There is no restriction to just one.
OK, one per subsystem-VM. For example, if a VM has a VFIO and a VDPA device, it should only two
/dev/ioasidFDs respectively. Correct?
No, only one.
For something like qemu's use case I mostly expect the vIOMMU driver will open /dev/ioasid for each vIOMMU instance it creates (basically only one)
Quoted 4 lines
Quoted 3 lines
The act of programming the page tables and the act of authorizing a PCI BDF to use a PASID are distinct things with two different IOCTLs.
Why separate?
Because they have different owners and different layers in the software.
It is not about use case, it is about putting the control points where they naturally belong.
Quoted 2 lines
For a complex stack like vSVA, I feel we have to reduce moving parts and do some divide and conquer.
uAPI should have all come together with a user and user application.
uAPI is hardest and most important part.
Jason
Re: [PATCH V4 05/18] iommu/ioasid: Redefine IOASID set and allocation APIs
In reply to Jean-Philippe BruckerHi Jean-Philippe,
Quoted 7 lines · Jean-Philippe Brucker
Quoted 3 lines
Slightly off the title. As we are moving to use cgroup to limit PASID allocations, it would be much simpler if we enforce on the current task.
Yes I think we should do that. Is there a problem with charging the process that does the PASID allocation even if the PASID indexes some other mm?
Besides complexity, my second concern is that we are sharing the misc cgroup controller with other resources that do not have such behavior.
Cgroup v2 also has unified hierarchy which also requires coherent behavior among controllers.
Thanks,
Jacob
_______________________________________________
iommu mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/iommu
PATCH v4 6/18 iommu/ioasid: Add free function and states
In reply to Jacob PanWhen an actively used IOASID is freed due to exceptions, users must be notified to perform the cleanup. The IOASID shall be put in a pending state until all users completed their cleanup work.
This patch adds ioasid_free() function to let the caller initiate the freeing process. Both ioasid_free() and ioasid_put() decrements reference counts. Unlike ioasid_put(), the ioasid_free() function also transition the IOASID to the free pending state where further ioasid_get() is prohibited. This paves the way for FREE event notifications that will be introduced next.
--- drivers/iommu/ioasid.c | 73 ++++++++++++++++++++++++++++++++++++++++++ include/linux/ioasid.h | 5 +++ 2 files changed, 78 insertions(+)
drivers/iommu/ioasid.c +73 -0
@@ -15,8 +15,26 @@static ioasid_t ioasid_capacity =PCI_PASID_MAX ;static ioasid_t ioasid_capacity_avail =PCI_PASID_MAX ;static DEFINE_XARRAY_ALLOC (ioasid_sets); + +enum ioasid_state { +IOASID_STATE_IDLE , +IOASID_STATE_ACTIVE , +IOASID_STATE_FREE_PENDING , +}; + +/** +* struct ioasid_data - Meta data about ioasid +* +* @id: Unique ID +* @refs: Number of active users +* @state: Track state of the IOASID +* @set: ioasid_set of the IOASID belongs to +* @private: Private data associated with the IOASID +* @rcu: For free after RCU grace period +*/ struct ioasid_data {ioasid_t id; +enum ioasid_state state;struct ioasid_set *set;void *private;struct rcu_head rcu;
@@ -597,6 +615,7 @@ ioasid_t ioasid_alloc(struct ioasid_set *set, ioasid_t min, ioasid_t max, goto exit_free; } data->id = id; + data->state =IOASID_STATE_IDLE ;/* Store IOASID in the per set data */ if (xa_err (xa_store (&set->xa, id, data,GFP_ATOMIC ))) {
@@ -631,6 +650,56 @@ static void ioasid_do_free_locked(struct ioasid_data *data)ioasid_set_free_locked (data->set); } +static void ioasid_free_locked (struct ioasid_set *set,ioasid_t ioasid) +{ +struct ioasid_data *data; + + data =xa_load (&active_allocator->xa, ioasid); +if (!data) { +pr_err_ratelimited ("Trying to free unknown IOASID %u\n" , ioasid); +return ; + } +if (data->set != set) { +pr_warn ("Cannot free IOASID %u due to set ownership\n" , ioasid); +return ; + } +/* Check if the set exists */ +if (WARN_ON (!xa_load (&ioasid_sets, data->set->id))) +return ; + +/* Free is already in progress */ +if (data->state ==IOASID_STATE_FREE_PENDING ) +return ; + + data->state =IOASID_STATE_FREE_PENDING ; +/* +* If the refcount is 1, it means there is no other users of the IOASID +* other than IOASID core itself. There is no need to notify anyone. +*/ +if (!refcount_dec_and_test (&data->refs)) +return ; + +ioasid_do_free_locked (data); +} + +/** +* ioasid_free - Drop reference on an IOASID. Free if refcount drops to 0, +* including free from its set and system-wide list. +* @set: The ioasid_set to check permission with. If not NULL, IOASID +* free will fail if the set does not match. +* @ioasid: The IOASID to remove +* +* TODO: return true if all references dropped, false if async work is in +* progress, IOASID is in FREE_PENDING state. wait queue to be used for blocking +* free task. +*/ +void ioasid_free (struct ioasid_set *set,ioasid_t ioasid) +{ +spin_lock (&ioasid_allocator_lock); +ioasid_free_locked (set, ioasid); +spin_unlock (&ioasid_allocator_lock); +} +EXPORT_SYMBOL_GPL (ioasid_free);int ioasid_get_locked (struct ioasid_set *set,ioasid_t ioasid) {struct ioasid_data *data;
@@ -640,6 +709,10 @@ int ioasid_get_locked(struct ioasid_set *set, ioasid_t ioasid)pr_err ("Trying to get unknown IOASID %u\n" , ioasid);return -EINVAL ; } +if (data->state ==IOASID_STATE_FREE_PENDING ) { +pr_err ("Trying to get IOASID being freed%u\n" , ioasid); +return -EBUSY ; + }/* Check set ownership if the set is non-null */ if (set && data->set != set) {
include/linux/ioasid.h +5 -0
@@ -72,6 +72,7 @@ int ioasid_get(struct ioasid_set *set, ioasid_t ioasid);int ioasid_get_locked (struct ioasid_set *set,ioasid_t ioasid);bool ioasid_put (struct ioasid_set *set,ioasid_t ioasid);bool ioasid_put_locked (struct ioasid_set *set,ioasid_t ioasid); +void ioasid_free (struct ioasid_set *set,ioasid_t ioasid);void *ioasid_find (struct ioasid_set *set,ioasid_t ioasid,bool (*getter)(void *));int ioasid_register_allocator (struct ioasid_allocator_ops *allocator);
@@ -105,6 +106,10 @@ static inline struct ioasid_set *ioasid_set_alloc(void *token, ioasid_t quota,return ERR_PTR (-ENOTSUPP ); } +static inline void ioasid_free (struct ioasid_set *set,ioasid_t ioasid) +{ +} +static inline struct ioasid_set *ioasid_find_mm_set (struct mm_struct *token) {return NULL ;
Signature · 1 line
2.25.1
PATCH v4 7/18 iommu/ioasid: Add ioasid_set iterator helper functions
In reply to Jacob PanUsers of an ioasid_set may not keep track of all the IOASIDs allocated under the set. When collective actions are needed for each IOASIDs, it is useful to iterate over all the IOASIDs within the set. For example, when the ioasid_set is freed, the user might perform the same cleanup operation on each IOASID.
This patch adds an API to iterate all the IOASIDs within the set.
--- drivers/iommu/ioasid.c | 84 ++++++++++++++++++++++++++++++++++++++++++ include/linux/ioasid.h | 20 ++++++++++ 2 files changed, 104 insertions(+)
drivers/iommu/ioasid.c +84 -0
@@ -700,6 +700,61 @@ void ioasid_free(struct ioasid_set *set, ioasid_t ioasid)spin_unlock (&ioasid_allocator_lock); }EXPORT_SYMBOL_GPL (ioasid_free); + +/** +* ioasid_free_all_in_set +* +* @brief +* Free all PASIDs from system-wide IOASID pool, all subscribers gets +* notified and do cleanup of their own. +* Note that some references of the IOASIDs within the set can still +* be held after the free call. This is OK in that the IOASIDs will be +* marked inactive, the only operations can be done is ioasid_put. +* No need to track IOASID set states since there is no reclaim phase. +* +* @param +* struct ioasid_set where all IOASIDs within the set will be freed. +*/ +void ioasid_free_all_in_set (struct ioasid_set *set) +{ +struct ioasid_data *entry; +unsigned long index; + +if (!ioasid_set_is_valid (set)) +return ; + +if (xa_empty (&set->xa)) +return ; + +if (!atomic_read (&set->nr_ioasids)) +return ; +spin_lock (&ioasid_allocator_lock); +xa_for_each (&set->xa, index, entry) { +ioasid_free_locked (set, index); +/* Free from per set private pool */ +xa_erase (&set->xa, index); + } +spin_unlock (&ioasid_allocator_lock); +} +EXPORT_SYMBOL_GPL (ioasid_free_all_in_set); + +/** +* ioasid_set_for_each_ioasid +* @brief +* Iterate over all the IOASIDs within the set +*/ +void ioasid_set_for_each_ioasid (struct ioasid_set *set, +void (*fn)(ioasid_t id,void *data), +void *data) +{ +struct ioasid_data *entry; +unsigned long index; + +xa_for_each (&set->xa, index, entry) +fn (index, data); +} +EXPORT_SYMBOL_GPL (ioasid_set_for_each_ioasid); +int ioasid_get_locked (struct ioasid_set *set,ioasid_t ioasid) {struct ioasid_data *data;
@@ -789,6 +844,35 @@ bool ioasid_put(struct ioasid_set *set, ioasid_t ioasid) }EXPORT_SYMBOL_GPL (ioasid_put); +/** +* @brief +* Find the ioasid_set of an IOASID. As long as the IOASID is valid, +* the set must be valid since the refcounting is based on the number of IOASID +* in the set. +* +* @param ioasid +* @return struct ioasid_set* +*/ +struct ioasid_set *ioasid_find_set (ioasid_t ioasid) +{ +struct ioasid_allocator_data *idata; +struct ioasid_data *ioasid_data; +struct ioasid_set *set =NULL ; + +rcu_read_lock (); + idata =rcu_dereference (active_allocator); + ioasid_data =xa_load (&idata->xa, ioasid); +if (!ioasid_data) { + set =ERR_PTR (-ENOENT ); + goto unlock; + } + set = ioasid_data->set; +unlock: +rcu_read_unlock (); +return set; +} +EXPORT_SYMBOL_GPL (ioasid_find_set); +/** * ioasid_find - Find IOASID data * @set: the IOASID set
include/linux/ioasid.h +20 -0
@@ -73,12 +73,17 @@ int ioasid_get_locked(struct ioasid_set *set, ioasid_t ioasid);bool ioasid_put (struct ioasid_set *set,ioasid_t ioasid);bool ioasid_put_locked (struct ioasid_set *set,ioasid_t ioasid);void ioasid_free (struct ioasid_set *set,ioasid_t ioasid); +void ioasid_free_all_in_set (struct ioasid_set *set);void *ioasid_find (struct ioasid_set *set,ioasid_t ioasid,bool (*getter)(void *)); +struct ioasid_set *ioasid_find_set (ioasid_t ioasid);int ioasid_register_allocator (struct ioasid_allocator_ops *allocator);void ioasid_unregister_allocator (struct ioasid_allocator_ops *allocator);int ioasid_attach_data (ioasid_t ioasid,void *data);void ioasid_detach_data (ioasid_t ioasid); +void ioasid_set_for_each_ioasid (struct ioasid_set *sdata, +void (*fn)(ioasid_t id,void *data), +void *data);#else /* !CONFIG_IOASID */ static inline void ioasid_install_capacity (ioasid_t total) {
@@ -158,5 +163,20 @@ static inline int ioasid_attach_data(ioasid_t ioasid, void *data)static inline void ioasid_detach_data (ioasid_t ioasid) { } + +static inline void ioasid_free_all_in_set (struct ioasid_set *set) +{ +} + +static inline struct ioasid_set *ioasid_find_set (ioasid_t ioasid) +{ +return ERR_PTR (-ENOTSUPP ); +} + +static inline void ioasid_set_for_each_ioasid (struct ioasid_set *sdata, +void (*fn)(ioasid_t id,void *data), +void *data) +{ +}#endif /* CONFIG_IOASID */ #endif /* __LINUX_IOASID_H */
Signature · 1 line
2.25.1
PATCH v4 8/18 iommu/ioasid: Introduce ioasid_set private ID
In reply to Jacob PanWhen an IOASID set is used for guest SVA, each VM will acquire its ioasid_set for IOASID allocations. IOASIDs within the VM must have a host/physical IOASID backing, mapping between guest and host IOASIDs can be non-identical. IOASID set private ID (SPID) is introduced in this patch to be used as guest IOASID. However, the concept of ioasid_set specific namespace is generic, thus named SPID.
As SPID namespace is within the IOASID set, the IOASID core can provide lookup services at both directions. SPIDs may not be available when its IOASID is allocated, the mapping between SPID and IOASID is usually established when a guest page table is bound to a host PASID.
--- drivers/iommu/ioasid.c | 104 +++++++++++++++++++++++++++++++++++++++++ include/linux/ioasid.h | 18 +++++++ 2 files changed, 122 insertions(+)
drivers/iommu/ioasid.c +104 -0
@@ -26,6 +26,7 @@ enum ioasid_state {* struct ioasid_data - Meta data about ioasid * * @id: Unique ID +* @spid: Private ID unique within a set * @refs: Number of active users * @state: Track state of the IOASID * @set: ioasid_set of the IOASID belongs to
@@ -34,6 +35,7 @@ enum ioasid_state {*/ struct ioasid_data {ioasid_t id; +ioasid_t spid;enum ioasid_state state;struct ioasid_set *set;void *private;
@@ -413,6 +415,107 @@ void ioasid_detach_data(ioasid_t ioasid) }EXPORT_SYMBOL_GPL (ioasid_detach_data); +static ioasid_t ioasid_find_by_spid_locked (struct ioasid_set *set,ioasid_t spid,bool get) +{ +ioasid_t ioasid =INVALID_IOASID ; +struct ioasid_data *entry; +unsigned long index; + +if (!xa_load (&ioasid_sets, set->id)) { +pr_warn ("Invalid set\n" ); + goto done; + } + +xa_for_each (&set->xa, index, entry) { +if (spid == entry->spid) { +if (get) +refcount_inc (&entry->refs); + ioasid = index; + } + } +done: +return ioasid; +} + +/** +* ioasid_attach_spid - Attach ioasid_set private ID to an IOASID +* +* @ioasid: the system-wide IOASID to attach +* @spid: the ioasid_set private ID of @ioasid +* +* After attching SPID, future lookup can be done via ioasid_find_by_spid(). +*/ +int ioasid_attach_spid (ioasid_t ioasid,ioasid_t spid) +{ +struct ioasid_data *data; +int ret =0 ; + +if (spid ==INVALID_IOASID ) +return -EINVAL ; + +spin_lock (&ioasid_allocator_lock); + data =xa_load (&active_allocator->xa, ioasid); + +if (!data) { +pr_err ("No IOASID entry %d to attach SPID %d\n" , + ioasid, spid); + ret = -ENOENT ; + goto done_unlock; + } +/* Check if SPID is unique within the set */ +if (ioasid_find_by_spid_locked (data->set, spid, false) !=INVALID_IOASID ) { + ret = -EINVAL ; + goto done_unlock; + } + data->spid = spid; + +done_unlock: +spin_unlock (&ioasid_allocator_lock); +return ret; +} +EXPORT_SYMBOL_GPL (ioasid_attach_spid); + +void ioasid_detach_spid (ioasid_t ioasid) +{ +struct ioasid_data *data; + +spin_lock (&ioasid_allocator_lock); + data =xa_load (&active_allocator->xa, ioasid); + +if (!data || data->spid ==INVALID_IOASID ) { +pr_err ("Invalid IOASID entry %d to detach\n" , ioasid); + goto done_unlock; + } + data->spid =INVALID_IOASID ; + +done_unlock: +spin_unlock (&ioasid_allocator_lock); +} +EXPORT_SYMBOL_GPL (ioasid_detach_spid); + +/** +* ioasid_find_by_spid - Find the system-wide IOASID by a set private ID and +* its set. +* +* @set: the ioasid_set to search within +* @spid: the set private ID +* @get: flag indicates whether to take a reference once found +* +* Given a set private ID and its IOASID set, find the system-wide IOASID. Take +* a reference upon finding the matching IOASID if @get is true. Return +* INVALID_IOASID if the IOASID is not found in the set or the set is not valid. +*/ +ioasid_t ioasid_find_by_spid (struct ioasid_set *set,ioasid_t spid,bool get) +{ +ioasid_t ioasid; + +spin_lock (&ioasid_allocator_lock); + ioasid =ioasid_find_by_spid_locked (set, spid, get); +spin_unlock (&ioasid_allocator_lock); +return ioasid; +} +EXPORT_SYMBOL_GPL (ioasid_find_by_spid); +static inline bool ioasid_set_is_valid (struct ioasid_set *set) {return xa_load (&ioasid_sets, set->id) == set;
@@ -616,6 +719,7 @@ ioasid_t ioasid_alloc(struct ioasid_set *set, ioasid_t min, ioasid_t max, } data->id = id; data->state =IOASID_STATE_IDLE ; + data->spid =INVALID_IOASID ;/* Store IOASID in the per set data */ if (xa_err (xa_store (&set->xa, id, data,GFP_ATOMIC ))) {
include/linux/ioasid.h +18 -0
@@ -81,6 +81,9 @@ int ioasid_register_allocator(struct ioasid_allocator_ops *allocator);void ioasid_unregister_allocator (struct ioasid_allocator_ops *allocator);int ioasid_attach_data (ioasid_t ioasid,void *data);void ioasid_detach_data (ioasid_t ioasid); +int ioasid_attach_spid (ioasid_t ioasid,ioasid_t spid); +void ioasid_detach_spid (ioasid_t ioasid); +ioasid_t ioasid_find_by_spid (struct ioasid_set *set,ioasid_t spid,bool get);void ioasid_set_for_each_ioasid (struct ioasid_set *sdata,void (*fn)(ioasid_t id,void *data),void *data);
@@ -173,6 +176,21 @@ static inline struct ioasid_set *ioasid_find_set(ioasid_t ioasid)return ERR_PTR (-ENOTSUPP ); } +static inline int ioasid_attach_spid (ioasid_t ioasid,ioasid_t spid) +{ +return -ENOTSUPP ; +} + +static inline void ioasid_detach_spid (ioasid_t ioasid) +{ +} + +static inline ioasid_t ioasid_find_by_spid (struct ioasid_set *set, +ioasid_t spid,bool get) +{ +return INVALID_IOASID ; +} +static inline void ioasid_set_for_each_ioasid (struct ioasid_set *sdata,void (*fn)(ioasid_t id,void *data),void *data)
Signature · 1 line
2.25.1
PATCH v4 9/18 iommu/ioasid: Introduce notification APIs
In reply to Jacob PanRelations among IOASID users largely follow a publisher-subscriber pattern. E.g. to support guest SVA on Intel Scalable I/O Virtualization (SIOV) enabled platforms, VFIO, IOMMU, device drivers, KVM are all users of IOASIDs. When a state change occurs, VFIO publishes the change event that needs to be processed by other users/subscribers.
This patch introduced two types of notifications: global and per ioasid_set. The latter is intended for users who only needs to handle events related to the IOASID of a given set.
For more information, refer to the kernel documentation at Documentation/ioasid.rst.
--- drivers/iommu/ioasid.c | 111 +++++++++++++++++++++++++++++++++++++++-- include/linux/ioasid.h | 54 ++++++++++++++++++++ 2 files changed, 161 insertions(+), 4 deletions(-)
drivers/iommu/ioasid.c +107 -4
@@ -10,12 +10,33 @@#include <linux/spinlock.h> #include <linux/xarray.h> +/* +* An IOASID can have multiple consumers where each consumer may have +* hardware contexts associated with the IOASID. +* When a status change occurs, like on IOASID deallocation, notifier chains +* are used to keep the consumers in sync. +* This is a publisher-subscriber pattern where publisher can change the +* state of each IOASID, e.g. alloc/free, bind IOASID to a device and mm. +* On the other hand, subscribers get notified for the state change and +* keep local states in sync. +*/ +static ATOMIC_NOTIFIER_HEAD (ioasid_notifier); +static DEFINE_SPINLOCK (ioasid_nb_lock); +/* Default to PCIe standard 20 bit PASID */ #define PCI_PASID_MAX 0x100000static ioasid_t ioasid_capacity =PCI_PASID_MAX ;static ioasid_t ioasid_capacity_avail =PCI_PASID_MAX ;static DEFINE_XARRAY_ALLOC (ioasid_sets); +struct ioasid_set_nb { +struct list_head list; +struct notifier_block *nb; +void *token; +struct ioasid_set *set; +bool active; +}; +enum ioasid_state {IOASID_STATE_IDLE ,IOASID_STATE_ACTIVE ,
@@ -415,6 +436,38 @@ void ioasid_detach_data(ioasid_t ioasid) }EXPORT_SYMBOL_GPL (ioasid_detach_data); +/** +* ioasid_notify - Send notification on a given IOASID for status change. +* +* @data: The IOASID data to which the notification will send +* @cmd: Notification event sent by IOASID external users, can be +* IOASID_BIND or IOASID_UNBIND. +* +* @flags: Special instructions, e.g. notify within a set or global by +* IOASID_NOTIFY_FLAG_SET or IOASID_NOTIFY_FLAG_ALL flags +* Caller must hold ioasid_allocator_lock and reference to the IOASID +*/ +static int ioasid_notify (struct ioasid_data *data, +enum ioasid_notify_val cmd,unsigned int flags) +{ +struct ioasid_nb_args args = {0 }; +int ret =0 ; + +if (flags & ~(IOASID_NOTIFY_FLAG_ALL |IOASID_NOTIFY_FLAG_SET )) +return -EINVAL ; + + args.id = data->id; + args.set = data->set; + args.pdata = data->private; + args.spid = data->spid; +if (flags &IOASID_NOTIFY_FLAG_ALL ) + ret =atomic_notifier_call_chain (&ioasid_notifier, cmd, &args); +if (flags &IOASID_NOTIFY_FLAG_SET ) + ret =atomic_notifier_call_chain (&data->set->nh, cmd, &args); + +return ret; +} +static ioasid_t ioasid_find_by_spid_locked (struct ioasid_set *set,ioasid_t spid,bool get) {ioasid_t ioasid =INVALID_IOASID ;
@@ -468,7 +521,7 @@ int ioasid_attach_spid(ioasid_t ioasid, ioasid_t spid) goto done_unlock; } data->spid = spid; - +ioasid_notify (data,IOASID_NOTIFY_BIND ,IOASID_NOTIFY_FLAG_SET ); done_unlock:spin_unlock (&ioasid_allocator_lock);return ret;
@@ -486,8 +539,8 @@ void ioasid_detach_spid(ioasid_t ioasid)pr_err ("Invalid IOASID entry %d to detach\n" , ioasid); goto done_unlock; } +ioasid_notify (data,IOASID_NOTIFY_UNBIND ,IOASID_NOTIFY_FLAG_SET ); data->spid =INVALID_IOASID ; - done_unlock:spin_unlock (&ioasid_allocator_lock); }
@@ -603,6 +656,8 @@ struct ioasid_set *ioasid_set_alloc(void *token, ioasid_t quota, int type) set->quota = quota; set->id = id;atomic_set (&set->nr_ioasids,0 ); +ATOMIC_INIT_NOTIFIER_HEAD (&set->nh); +/* * Per set XA is used to store private IDs within the set, get ready * for ioasid_set private ID and system-wide IOASID allocation
@@ -655,7 +710,9 @@ int ioasid_set_free(struct ioasid_set *set)int ret =0 ;spin_lock (&ioasid_allocator_lock); +spin_lock (&ioasid_nb_lock); ret =ioasid_set_free_locked (set); +spin_unlock (&ioasid_nb_lock);spin_unlock (&ioasid_allocator_lock);return ret; }
@@ -728,6 +785,7 @@ ioasid_t ioasid_alloc(struct ioasid_set *set, ioasid_t min, ioasid_t max, goto exit_free; }atomic_inc (&set->nr_ioasids); +ioasid_notify (data,IOASID_NOTIFY_ALLOC ,IOASID_NOTIFY_FLAG_SET ); goto done_unlock; exit_free:kfree (data);
@@ -780,9 +838,11 @@ static void ioasid_free_locked(struct ioasid_set *set, ioasid_t ioasid)* If the refcount is 1, it means there is no other users of the IOASID * other than IOASID core itself. There is no need to notify anyone. */ -if (!refcount_dec_and_test (&data->refs)) +if (!refcount_dec_and_test (&data->refs)) { +ioasid_notify (data,IOASID_NOTIFY_FREE , +IOASID_NOTIFY_FLAG_SET |IOASID_NOTIFY_FLAG_ALL );return ; - + }ioasid_do_free_locked (data); }
@@ -833,15 +893,39 @@ void ioasid_free_all_in_set(struct ioasid_set *set)if (!atomic_read (&set->nr_ioasids))return ;spin_lock (&ioasid_allocator_lock); +spin_lock (&ioasid_nb_lock);xa_for_each (&set->xa, index, entry) {ioasid_free_locked (set, index);/* Free from per set private pool */ xa_erase (&set->xa, index); } +spin_unlock (&ioasid_nb_lock);spin_unlock (&ioasid_allocator_lock); }EXPORT_SYMBOL_GPL (ioasid_free_all_in_set); +/* +* ioasid_find_mm_set - Retrieve IOASID set with mm token +* Take a reference of the set if found. +*/ +struct ioasid_set *ioasid_find_mm_set (struct mm_struct *token) +{ +struct ioasid_set *set; +unsigned long index; + +spin_lock (&ioasid_allocator_lock); + +xa_for_each (&ioasid_sets, index, set) { +if (set->type ==IOASID_SET_TYPE_MM && set->token == token) + goto exit_unlock; + } + set =NULL ; +exit_unlock: +spin_unlock (&ioasid_allocator_lock); +return set; +} +EXPORT_SYMBOL_GPL (ioasid_find_mm_set); +/** * ioasid_set_for_each_ioasid * @brief
@@ -1021,6 +1105,25 @@ void *ioasid_find(struct ioasid_set *set, ioasid_t ioasid, }EXPORT_SYMBOL_GPL (ioasid_find); +int ioasid_register_notifier (struct ioasid_set *set,struct notifier_block *nb) +{ +if (set) +return atomic_notifier_chain_register (&set->nh, nb); +else +return atomic_notifier_chain_register (&ioasid_notifier, nb); +} +EXPORT_SYMBOL_GPL (ioasid_register_notifier); + +void ioasid_unregister_notifier (struct ioasid_set *set, +struct notifier_block *nb) +{ +if (set) +atomic_notifier_chain_unregister (&set->nh, nb); +else +atomic_notifier_chain_unregister (&ioasid_notifier, nb); +} +EXPORT_SYMBOL_GPL (ioasid_unregister_notifier); +MODULE_AUTHOR ("Jean-Philippe Brucker <[email protected]>" );MODULE_AUTHOR ("Jacob Pan <jacob.jun.pan-VuQAYsv1563Yd54FQh9/[email protected]>" );MODULE_DESCRIPTION ("IO Address Space ID (IOASID) allocator" );
include/linux/ioasid.h +54 -0
72 lines not on this page · raw message
Signature · 1 line
2.25.1
PATCH v4 10/18 iommu/ioasid: Support mm token type ioasid_set notifications
In reply to Jacob PanAs a system-wide resource, IOASID is often shared by multiple kernel subsystems that are independent of each other. However, at the ioasid_set level, these kernel subsystems must communicate with each other for ownership checking, event notifications, etc. For example, on Intel Scalable IO Virtualization (SIOV) enabled platforms, KVM and VFIO instances under the same process/guest must be aware of a shared IOASID set.IOASID_SET_TYPE_MM token type was introduced to explicitly mark an IOASID set that belongs to a process, thus use the same mm_struct pointer as a token. Users of the same process can then identify with each other based on this token.
This patch introduces MM token specific event registration APIs. Event subscribers such as KVM instances can register IOASID event handler without the knowledge of its ioasid_set. Event handlers are registered based on its mm_struct pointer as a token. In case when subscribers register handler *prior* to the creation of the ioasid_set, the handler’s notification block is stored in a pending list within IOASID core. Once the ioasid_set of the MM token is created, the notification block will be registered by the IOASID core.
--- drivers/iommu/ioasid.c | 142 +++++++++++++++++++++++++++++++++++++++++ include/linux/ioasid.h | 18 ++++++ 2 files changed, 160 insertions(+)
drivers/iommu/ioasid.c +142 -0
@@ -21,6 +21,8 @@* keep local states in sync. */ static ATOMIC_NOTIFIER_HEAD (ioasid_notifier); +/* List to hold pending notification block registrations */ +static LIST_HEAD (ioasid_nb_pending_list);static DEFINE_SPINLOCK (ioasid_nb_lock);/* Default to PCIe standard 20 bit PASID */
@@ -574,6 +576,27 @@ static inline bool ioasid_set_is_valid(struct ioasid_set *set)return xa_load (&ioasid_sets, set->id) == set; } +static void ioasid_add_pending_nb (struct ioasid_set *set) +{ +struct ioasid_set_nb *curr; + +if (set->type !=IOASID_SET_TYPE_MM ) +return ; +/* +* Check if there are any pending nb requests for the given token, if so +* add them to the notifier chain. +*/ +spin_lock (&ioasid_nb_lock); +list_for_each_entry (curr, &ioasid_nb_pending_list, list) { +if (curr->token == set->token && !curr->active) { +atomic_notifier_chain_register (&set->nh, curr->nb); + curr->set = set; + curr->active = true; + } + } +spin_unlock (&ioasid_nb_lock); +} +/** * ioasid_set_alloc - Allocate a new IOASID set for a given token *
@@ -658,6 +681,11 @@ struct ioasid_set *ioasid_set_alloc(void *token, ioasid_t quota, int type)atomic_set (&set->nr_ioasids,0 );ATOMIC_INIT_NOTIFIER_HEAD (&set->nh); +/* +* Check if there are any pending nb requests for the given token, if so +* add them to the notifier chain. +*/ +ioasid_add_pending_nb (set);/* * Per set XA is used to store private IDs within the set, get ready * for ioasid_set private ID and system-wide IOASID allocation
@@ -675,6 +703,7 @@ EXPORT_SYMBOL_GPL(ioasid_set_alloc);static int ioasid_set_free_locked (struct ioasid_set *set) { +struct ioasid_set_nb *curr;int ret =0 ;if (!ioasid_set_is_valid (set)) {
@@ -688,6 +717,16 @@ static int ioasid_set_free_locked(struct ioasid_set *set) }WARN_ON (!xa_empty (&set->xa)); +/* Restore pending status of the set NBs */ +list_for_each_entry (curr, &ioasid_nb_pending_list, list) { +if (curr->token == set->token) { +if (curr->active) + curr->active = false; +else +pr_warn ("Set token exists but not active!\n" ); + } + } +/* * Token got released right away after the ioasid_set is freed. * If a new set is created immediately with the newly released token,
@@ -1117,6 +1156,22 @@ EXPORT_SYMBOL_GPL(ioasid_register_notifier);void ioasid_unregister_notifier (struct ioasid_set *set,struct notifier_block *nb) { +struct ioasid_set_nb *curr; + +spin_lock (&ioasid_nb_lock); +/* +* Pending list is registered with a token without an ioasid_set, +* therefore should not be unregistered directly. +*/ +list_for_each_entry (curr, &ioasid_nb_pending_list, list) { +if (curr->nb == nb) { +pr_warn ("Cannot unregister NB from pending list\n" ); +spin_unlock (&ioasid_nb_lock); +return ; + } + } +spin_unlock (&ioasid_nb_lock); +if (set)atomic_notifier_chain_unregister (&set->nh, nb);else
@@ -1124,6 +1179,93 @@ void ioasid_unregister_notifier(struct ioasid_set *set, }EXPORT_SYMBOL_GPL (ioasid_unregister_notifier); +/** +* ioasid_register_notifier_mm - Register a notifier block on the IOASID set +* created by the mm_struct pointer as the token +* +* @mm: the mm_struct token of the ioasid_set +* @nb: notfier block to be registered on the ioasid_set +* +* This a variant of ioasid_register_notifier() where the caller intends to +* listen to IOASID events belong the ioasid_set created under the same +* process. Caller is not aware of the ioasid_set, no need to hold reference +* of the ioasid_set. +*/ +int ioasid_register_notifier_mm (struct mm_struct *mm,struct notifier_block *nb) +{ +struct ioasid_set_nb *curr; +struct ioasid_set *set; +int ret =0 ; + +spin_lock (&ioasid_nb_lock); +/* Check for duplicates, nb is unique per set */ +list_for_each_entry (curr, &ioasid_nb_pending_list, list) { +if (curr->token == mm && curr->nb == nb) { + ret = -EBUSY ; + goto exit_unlock; + } + } + curr =kzalloc (sizeof (*curr),GFP_ATOMIC ); +if (!curr) { + ret = -ENOMEM ; + goto exit_unlock; + } +/* Check if the token has an existing set */ + set =ioasid_find_mm_set (mm); +if (!set) { +/* Add to the rsvd list as inactive */ + curr->active = false; + }else { +/* REVISIT: Only register empty set for now. Can add an option +* in the future to playback existing PASIDs. +*/ +if (atomic_read (&set->nr_ioasids)) { +pr_warn ("IOASID set %d not empty %d\n" , set->id, +atomic_read (&set->nr_ioasids)); + ret = -EBUSY ; + goto exit_free; + } + curr->token = mm; + curr->nb = nb; + curr->active = true; + curr->set = set; + +/* Set already created, add to the notifier chain */ +atomic_notifier_chain_register (&set->nh, nb); + } + +list_add (&curr->list, &ioasid_nb_pending_list); + goto exit_unlock; +exit_free: +kfree (curr); +exit_unlock: +spin_unlock (&ioasid_nb_lock); +return ret; +} +EXPORT_SYMBOL_GPL (ioasid_register_notifier_mm); + +void ioasid_unregister_notifier_mm (struct mm_struct *mm,struct notifier_block *nb) +{ +struct ioasid_set_nb *curr; + +spin_lock (&ioasid_nb_lock); +list_for_each_entry (curr, &ioasid_nb_pending_list, list) { +if (curr->token == mm && curr->nb == nb) { +list_del (&curr->list); +spin_unlock (&ioasid_nb_lock); +if (curr->active) { +atomic_notifier_chain_unregister (&curr->set->nh, + nb); + } +kfree (curr); +return ; + } + } +pr_warn ("No ioasid set found for mm token %llx\n" , (u64 )mm); +spin_unlock (&ioasid_nb_lock); +} +EXPORT_SYMBOL_GPL (ioasid_unregister_notifier_mm); +MODULE_AUTHOR ("Jean-Philippe Brucker <[email protected]>" );MODULE_AUTHOR ("Jacob Pan <[email protected]>" );MODULE_DESCRIPTION ("IO Address Space ID (IOASID) allocator" );
include/linux/ioasid.h +18 -0
@@ -132,6 +132,8 @@ void ioasid_unregister_notifier(struct ioasid_set *set,void ioasid_set_for_each_ioasid (struct ioasid_set *sdata,void (*fn)(ioasid_t id,void *data),void *data); +int ioasid_register_notifier_mm (struct mm_struct *mm,struct notifier_block *nb); +void ioasid_unregister_notifier_mm (struct mm_struct *mm,struct notifier_block *nb);#else /* !CONFIG_IOASID */ static inline void ioasid_install_capacity (ioasid_t total) {
@@ -250,5 +252,21 @@ static inline void ioasid_set_for_each_ioasid(struct ioasid_set *sdata, void *data) { } + +static inline int ioasid_register_notifier_mm (struct mm_struct *mm, +struct notifier_block *nb) +{ +return -ENOTSUPP ; +} + +static inline void ioasid_unregister_notifier_mm (struct mm_struct *mm, +struct notifier_block *nb) +{ +} + +static inline bool ioasid_queue_work (struct work_struct *work) +{ +return false; +}#endif /* CONFIG_IOASID */ #endif /* __LINUX_IOASID_H */
Signature · 6 lines
2.25.1 _______________________________________________ iommu mailing list [email protected] https://lists.linuxfoundation.org/mailman/listinfo/iommu
PATCH v4 11/18 iommu/ioasid: Add ownership check in guest bind
In reply to Jacob PanBind guest page table call comes with an IOASID provided by the userspace. To prevent attacks by malicious users, we must ensure the IOASID was allocated under the same process.
This patch adds a new API that will perform an ownership check that is based on whether the IOASID belongs to the ioasid_set allocated with the mm_struct pointer as a token.
--- drivers/iommu/ioasid.c | 37 +++++++++++++++++++++++++++++++++++++ drivers/iommu/iommu.c | 16 ++++++++++++++-- include/linux/ioasid.h | 6 ++++++ 3 files changed, 57 insertions(+), 2 deletions(-)
drivers/iommu/ioasid.c +37 -0
@@ -9,6 +9,7 @@#include <linux/slab.h> #include <linux/spinlock.h> #include <linux/xarray.h> +#include <linux/sched/mm.h> /* * An IOASID can have multiple consumers where each consumer may have
@@ -1028,6 +1029,42 @@ int ioasid_get(struct ioasid_set *set, ioasid_t ioasid) }EXPORT_SYMBOL_GPL (ioasid_get); +/** +* ioasid_get_if_owned - obtain a reference to the IOASID if the IOASID belongs +* to the ioasid_set with the current mm as token +* @ioasid: the IOASID to get reference +* +* +* Return: 0 on success, error if failed. +*/ +int ioasid_get_if_owned (ioasid_t ioasid) +{ +struct ioasid_set *set; +int ret; + +spin_lock (&ioasid_allocator_lock); + set =ioasid_find_set (ioasid); +if (IS_ERR_OR_NULL (set)) { + ret = -ENOENT ; + goto done_unlock; + } +if (set->type !=IOASID_SET_TYPE_MM ) { + ret = -EINVAL ; + goto done_unlock; + } +if (current->mm != set->token) { + ret = -EPERM ; + goto done_unlock; + } + + ret =ioasid_get_locked (set, ioasid); +done_unlock: +spin_unlock (&ioasid_allocator_lock); + +return ret; +} +EXPORT_SYMBOL_GPL (ioasid_get_if_owned); +bool ioasid_put_locked (struct ioasid_set *set,ioasid_t ioasid) {struct ioasid_data *data;
drivers/iommu/iommu.c +14 -2
@@ -2169,7 +2169,13 @@ int iommu_uapi_sva_bind_gpasid(struct iommu_domain *domain, struct device *dev,if (ret)return ret; -return domain->ops->sva_bind_gpasid (domain, dev, &data); + ret =ioasid_get_if_owned (data.hpasid); +if (ret) +return ret; + ret = domain->ops->sva_bind_gpasid (domain, dev, &data); +ioasid_put (NULL , data.hpasid); + +return ret; }EXPORT_SYMBOL_GPL (iommu_uapi_sva_bind_gpasid);
@@ -2196,7 +2202,13 @@ int iommu_uapi_sva_unbind_gpasid(struct iommu_domain *domain, struct device *devif (ret)return ret; -return iommu_sva_unbind_gpasid (domain, dev, data.hpasid); + ret =ioasid_get_if_owned (data.hpasid); +if (ret) +return ret; + ret =iommu_sva_unbind_gpasid (domain, dev, data.hpasid); +ioasid_put (NULL , data.hpasid); + +return ret; }EXPORT_SYMBOL_GPL (iommu_uapi_sva_unbind_gpasid);
include/linux/ioasid.h +6 -0
@@ -111,6 +111,7 @@ ioasid_t ioasid_alloc(struct ioasid_set *set, ioasid_t min, ioasid_t max, void *private);int ioasid_get (struct ioasid_set *set,ioasid_t ioasid);int ioasid_get_locked (struct ioasid_set *set,ioasid_t ioasid); +int ioasid_get_if_owned (ioasid_t ioasid);bool ioasid_put (struct ioasid_set *set,ioasid_t ioasid);bool ioasid_put_locked (struct ioasid_set *set,ioasid_t ioasid);void ioasid_free (struct ioasid_set *set,ioasid_t ioasid);
@@ -180,6 +181,11 @@ static inline int ioasid_get_locked(struct ioasid_set *set, ioasid_t ioasid)return -ENOTSUPP ; } +static inline int ioasid_get_if_owned (ioasid_t ioasid) +{ +return -ENOTSUPP ; +} +static inline bool ioasid_put (struct ioasid_set *set,ioasid_t ioasid) {return false;
Signature · 1 line
2.25.1
PATCH v4 12/18 iommu/vt-d: Remove mm reference for guest SVA
In reply to Jacob PanNow that IOASID core keeps track of the IOASID to mm_struct ownership in the forms of ioasid_set with IOASID_SET_TYPE_MM token type, there is no need to keep the same mapping in VT-d driver specific data. Native SVM usage is not affected by the change.
--- drivers/iommu/intel/svm.c | 7 ------- 1 file changed, 7 deletions(-)
drivers/iommu/intel/svm.c +0 -7
@@ -363,12 +363,6 @@ int intel_svm_bind_gpasid(struct iommu_domain *domain, struct device *dev, ret = -ENOMEM ; goto out; } -/* REVISIT: upper layer/VFIO can track host process that bind -* the PASID. ioasid_set = mm might be sufficient for vfio to -* check pasid VMM ownership. We can drop the following line -* once VFIO and IOASID set check is in place. -*/ - svm->mm =get_task_mm (current); svm->pasid = data->hpasid;if (data->flags &IOMMU_SVA_GPASID_VAL ) { svm->gpasid = data->gpasid;
@@ -376,7 +370,6 @@ int intel_svm_bind_gpasid(struct iommu_domain *domain, struct device *dev, }ioasid_attach_data (data->hpasid, svm);INIT_LIST_HEAD_RCU (&svm->devs); -mmput (svm->mm); } sdev =kzalloc (sizeof (*sdev),GFP_KERNEL );if (!sdev) {
Signature · 1 line
2.25.1
PATCH v4 13/18 iommu/ioasid: Add a workqueue for cleanup work
In reply to Jacob PanAn IOASID can have multiple users, such as IOMMU driver, KVM, and device
drivers. The atomic IOASID notifier is used to inform users of IOASID
state change. For example, the IOASID_NOTIFY_UNBIND event is issued when the IOASID is no longer bound to an address space. This requires ordered actions among users to tear down their contexts.
Not all work can be handled in the atomic notifier handler. This patch introduces a shared, ordered workqueue for all IOASID users who wish to perform work asynchronously upon notification.
--- drivers/iommu/ioasid.c | 25 +++++++++++++++++++++++++ include/linux/ioasid.h | 1 + 2 files changed, 26 insertions(+)
drivers/iommu/ioasid.c +25 -0
@@ -32,6 +32,9 @@ static ioasid_t ioasid_capacity = PCI_PASID_MAX;static ioasid_t ioasid_capacity_avail =PCI_PASID_MAX ;static DEFINE_XARRAY_ALLOC (ioasid_sets); +/* Workqueue for IOASID users to do cleanup upon notification */ +static struct workqueue_struct *ioasid_wq; +struct ioasid_set_nb {struct list_head list;struct notifier_block *nb;
@@ -1281,6 +1284,12 @@ int ioasid_register_notifier_mm(struct mm_struct *mm, struct notifier_block *nb) }EXPORT_SYMBOL_GPL (ioasid_register_notifier_mm); +bool ioasid_queue_work (struct work_struct *work) +{ +return queue_work (ioasid_wq, work); +} +EXPORT_SYMBOL_GPL (ioasid_queue_work); +void ioasid_unregister_notifier_mm (struct mm_struct *mm,struct notifier_block *nb) {struct ioasid_set_nb *curr;
@@ -1303,7 +1312,23 @@ void ioasid_unregister_notifier_mm(struct mm_struct *mm, struct notifier_block * }EXPORT_SYMBOL_GPL (ioasid_unregister_notifier_mm); +static int __init ioasid_init (void ) +{ + ioasid_wq =alloc_ordered_workqueue ("ioasid_wq" ,0 ); +if (!ioasid_wq) +return -ENOMEM ; + +return 0 ; +} + +static void __exit ioasid_cleanup(void) +{ +destroy_workqueue (ioasid_wq); +} +MODULE_AUTHOR ("Jean-Philippe Brucker <[email protected]>" );MODULE_AUTHOR ("Jacob Pan <jacob.jun.pan-VuQAYsv1563Yd54FQh9/[email protected]>" );MODULE_DESCRIPTION ("IO Address Space ID (IOASID) allocator" );MODULE_LICENSE ("GPL" ); +module_init (ioasid_init); +module_exit (ioasid_cleanup);
include/linux/ioasid.h +1 -0
@@ -135,6 +135,7 @@ void ioasid_set_for_each_ioasid(struct ioasid_set *sdata, void *data);int ioasid_register_notifier_mm (struct mm_struct *mm,struct notifier_block *nb);void ioasid_unregister_notifier_mm (struct mm_struct *mm,struct notifier_block *nb); +bool ioasid_queue_work (struct work_struct *work);#else /* !CONFIG_IOASID */ static inline void ioasid_install_capacity (ioasid_t total) {
Signature · 1 line
2.25.1
PATCH v4 14/18 iommu/vt-d: Listen to IOASID notifications
In reply to Jacob PanOn Intel Scalable I/O Virtualization (SIOV) enabled platforms, IOMMU driver is one of the users of IOASIDs. In normal flow, callers will perform IOASID allocation, bind, unbind, and free in order. However, for guest SVA, IOASID free could come before unbind as guest is untrusted. This patch registers IOASID notification handler such that IOMMU driver can perform PASID teardown upon receiving an unexpected IOASID free event.
--- drivers/iommu/intel/iommu.c | 2 + drivers/iommu/intel/svm.c | 109 +++++++++++++++++++++++++++++++++++- include/linux/intel-iommu.h | 2 + 3 files changed, 111 insertions(+), 2 deletions(-)
drivers/iommu/intel/iommu.c +2 -0
@@ -3313,6 +3313,8 @@ static int __init init_dmars(void)pr_err ("Failed to allocate host PASID set %lu\n" ,PTR_ERR (host_pasid_set)); intel_iommu_sm =0 ; + }else { +intel_svm_add_pasid_notifier (); } }
drivers/iommu/intel/svm.c +107 -2
135 lines not on this page · raw message
include/linux/intel-iommu.h +2 -0
@@ -757,6 +757,7 @@ void intel_svm_unbind(struct iommu_sva *handle);u32 intel_svm_get_pasid (struct iommu_sva *handle);int intel_svm_page_response (struct device *dev,struct iommu_fault_event *evt,struct iommu_page_response *msg); +void intel_svm_add_pasid_notifier (void );struct svm_dev_ops ;
@@ -783,6 +784,7 @@ struct intel_svm {int gpasid;/* In case that guest PASID is different from host PASID */ struct list_head devs;struct list_head list; + struct work_struct work; /* For deferred clean up */ };#else static inline void intel_svm_check (struct intel_iommu *iommu) {}
Signature · 1 line
2.25.1
RFC PATCH v4 15/18 cgroup: Introduce ioasids controller
In reply to Jacob PanIOASIDs are used to associate DMA requests with virtual address spaces. They are a system-wide limited resource made available to the userspace applications. Let it be VMs or user-space device drivers.
This RFC patch introduces a cgroup controller to address the following problems:
1. Some user applications exhaust all the available IOASIDs thus depriving others of the same host.
2. System admins need to provision VMs based on their needs for IOASIDs, e.g. the number of VMs with assigned devices that perform DMA requests with PASID.
This patch is nowhere near its completion, it merely provides the basic functionality for resource distribution and cgroup hierarchy organizational changes.
Since this is part of a greater effort to enable Shared Virtual Address (SVA) virtualization. We would like to have a direction check and collect feedback early. For details, please refer to the documentation:Documentation/admin-guide/cgroup-v1/ioasids.rst
--- include/linux/cgroup_subsys.h | 4 + include/linux/ioasid.h | 17 ++ init/Kconfig | 7 + kernel/cgroup/Makefile | 1 + kernel/cgroup/ioasids.c | 345 ++++++++++++++++++++++++++++++++++ 5 files changed, 374 insertions(+) create mode 100644 kernel/cgroup/ioasids.c
include/linux/cgroup_subsys.h +4 -0
@@ -57,6 +57,10 @@ SUBSYS(hugetlb) SUBSYS(pids)#endif +#if IS_ENABLED (CONFIG_CGROUP_IOASIDS ) +SUBSYS(ioasids) +#endif +#if IS_ENABLED (CONFIG_CGROUP_RDMA ) SUBSYS(rdma)#endif
include/linux/ioasid.h +17 -0
@@ -135,8 +135,25 @@ void ioasid_set_for_each_ioasid(struct ioasid_set *sdata, void *data);int ioasid_register_notifier_mm (struct mm_struct *mm,struct notifier_block *nb);void ioasid_unregister_notifier_mm (struct mm_struct *mm,struct notifier_block *nb); +#ifdef CONFIG_CGROUP_IOASIDS +int ioasid_cg_charge (struct ioasid_set *set); +void ioasid_cg_uncharge (struct ioasid_set *set); +#else +/* No cgroup control, allocation will proceed until run out total pool */ +static inline int ioasid_cg_charge (struct ioasid_set *set) +{ +return 0 ; +} + +static inline int ioasid_cg_uncharge (struct ioasid_set *set) +{ +return 0 ; +} +#endif /* CGROUP_IOASIDS */ bool ioasid_queue_work (struct work_struct *work); +#else /* !CONFIG_IOASID */ +static inline void ioasid_install_capacity (ioasid_t total) { }
init/Kconfig +7 -0
@@ -1017,6 +1017,13 @@ config CGROUP_PIDS since the PIDs limit only affects a process's ability to fork, not to attach to a cgroup. +config CGROUP_IOASIDS + bool "IOASIDs controller" + depends on IOASID + help + Provides enforcement of IO Address Space ID limits in the scope of a + cgroup. + config CGROUP_RDMA bool "RDMA controller" help
kernel/cgroup/Makefile +1 -0
@@ -3,6 +3,7 @@ obj-y := cgroup.o rstat.o namespace.o cgroup-v1.o freezer.o obj-$(CONFIG_CGROUP_FREEZER) += legacy_freezer.o obj-$(CONFIG_CGROUP_PIDS) += pids.o +obj-$(CONFIG_CGROUP_IOASIDS) += ioasids.o obj-$(CONFIG_CGROUP_RDMA) += rdma.o obj-$(CONFIG_CPUSETS) += cpuset.o obj-$(CONFIG_CGROUP_DEBUG) += debug.o
kernel/cgroup/ioasids.c new file +345 -0
345 lines not on this page · raw message
Signature · 1 line
2.25.1
Tejun Heo
maintainerRe: [RFC PATCH 15/18] cgroup: Introduce ioasids controller
Quoted 11 lines · Jacob Pan
IOASIDs are used to associate DMA requests with virtual address spaces. They are a system-wide limited resource made available to the userspace applications. Let it be VMs or user-space device drivers.
This RFC patch introduces a cgroup controller to address the following problems:
1. Some user applications exhaust all the available IOASIDs thus depriving others of the same host.
2. System admins need to provision VMs based on their needs for IOASIDs, e.g. the number of VMs with assigned devices that perform DMA requests with PASID.
Please take a look at the proposed misc controller:
http://lkml.kernel.org/r/[email protected]
Would that fit your bill?
Thanks.
Signature · 1 line
tejun
Re: [RFC PATCH 15/18] cgroup: Introduce ioasids controller
Hi Tejun,
Quoted 13 lines · Tejun Heo
Quoted 11 lines · Jacob Pan
IOASIDs are used to associate DMA requests with virtual address spaces. They are a system-wide limited resource made available to the userspace applications. Let it be VMs or user-space device drivers.
This RFC patch introduces a cgroup controller to address the following problems:
1. Some user applications exhaust all the available IOASIDs thus depriving others of the same host.
2. System admins need to provision VMs based on their needs for IOASIDs, e.g. the number of VMs with assigned devices that perform DMA requests with PASID.
Quoted 5 lines
Please take a look at the proposed misc controller:
http://lkml.kernel.org/r/[email protected]
Would that fit your bill?
The interface definitely can be reused. But IOASID has a different behavior in terms of migration and ownership checking. I guess SEV key IDs are not tied to a process whereas IOASIDs are. Perhaps this can be solved by adding
+ .can_attach = ioasids_can_attach,
+ .cancel_attach = ioasids_cancel_attach,
Let me give it a try and come back.
Thanks for the pointer.
Jacob
Quoted 3 lines
Thanks.
Thanks,
Jacob
_______________________________________________
iommu mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/iommu
Re: [RFC PATCH 15/18] cgroup: Introduce ioasids controller
Hi Jacob,
Quoted 28 lines · Jacob Pan
Hi Tejun,
Quoted 13 lines · Tejun Heo
Quoted 11 lines · Jacob Pan
IOASIDs are used to associate DMA requests with virtual address spaces. They are a system-wide limited resource made available to the userspace applications. Let it be VMs or user-space device drivers.
This RFC patch introduces a cgroup controller to address the following problems:
1. Some user applications exhaust all the available IOASIDs thus depriving others of the same host.
2. System admins need to provision VMs based on their needs for IOASIDs, e.g. the number of VMs with assigned devices that perform DMA requests with PASID.Quoted 5 lines
Please take a look at the proposed misc controller:
http://lkml.kernel.org/r/[email protected]
Would that fit your bill?
The interface definitely can be reused. But IOASID has a different behavior in terms of migration and ownership checking. I guess SEV key IDs are not tied to a process whereas IOASIDs are. Perhaps this can be solved by adding
kernel/cgroup/ioasids.c new file +2
-0+ .can_attach = ioasids_can_attach, + .cancel_attach = ioasids_cancel_attach,
Quoted 2 lines
Let me give it a try and come back.
While I am trying to fit the IOASIDs cgroup in to the misc cgroup proposal. I'd like to have a direction check on whether this idea of using cgroup for IOASID/PASID resource management is viable.
Alex/Jason/Jean and everyone, your feedback is much appreciated.
Quoted 5 lines
Thanks for the pointer.
Jacob
Quoted 1 line
Quoted 7 lines
Quoted 2 lines
Thanks.
Thanks,
Jacob
Thanks,
Jacob
_______________________________________________
iommu mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/iommu
Re: [RFC PATCH 15/18] cgroup: Introduce ioasids controller
Quoted 12 lines · Jacob Pan
Quoted 8 lines
The interface definitely can be reused. But IOASID has a different behavior in terms of migration and ownership checking. I guess SEV key IDs are not tied to a process whereas IOASIDs are. Perhaps this can be solved by adding
kernel/cgroup/ioasids.c new file +2
-0+ .can_attach = ioasids_can_attach, + .cancel_attach = ioasids_cancel_attach,Let me give it a try and come back.
While I am trying to fit the IOASIDs cgroup in to the misc cgroup proposal. I'd like to have a direction check on whether this idea of using cgroup for IOASID/PASID resource management is viable.
Quoted 1 line
Alex/Jason/Jean and everyone, your feedback is much appreciated.
IMHO I can't think of anything else to enforce some limit on a HW scarce resource that unpriv userspace can consume.
Jason
Re: [RFC PATCH 15/18] cgroup: Introduce ioasids controller
In reply to Jacob PanQuoted 27 lines · Jacob Pan
Hi Jacob,
Quoted 22 lines · Jacob Pan
Hi Tejun,
Quoted 18 lines · Tejun Heo
Quoted 11 lines · Jacob Pan
IOASIDs are used to associate DMA requests with virtual address spaces. They are a system-wide limited resource made available to the userspace applications. Let it be VMs or user-space device drivers.
This RFC patch introduces a cgroup controller to address the following problems:
1. Some user applications exhaust all the available IOASIDs thus depriving others of the same host.
2. System admins need to provision VMs based on their needs for IOASIDs, e.g. the number of VMs with assigned devices that perform DMA requests with PASID.Please take a look at the proposed misc controller:
http://lkml.kernel.org/r/20210302081705.1990283-2-vipinsh-hpIqsD4AKlfQT0dZR+AlfA@public.gmane.org
Would that fit your bill?
Quoted 11 lines
Quoted 8 lines
The interface definitely can be reused. But IOASID has a different behavior in terms of migration and ownership checking. I guess SEV key IDs are not tied to a process whereas IOASIDs are. Perhaps this can be solved by adding
kernel/cgroup/ioasids.c new file +2
-0+ .can_attach = ioasids_can_attach, + .cancel_attach = ioasids_cancel_attach,Let me give it a try and come back.
While I am trying to fit the IOASIDs cgroup in to the misc cgroup proposal. I'd like to have a direction check on whether this idea of using cgroup for IOASID/PASID resource management is viable.
Yes, even for host SVA it would be good to have a cgroup. Currently the number of shared address spaces is naturally limited by number of processes, which can be controlled with rlimit and cgroup. But on Arm the hardware limit on shared address spaces is 64k (number of ASIDs), easily exhausted with the default PASID and PID limits. So a cgroup for managing this resource is more than welcome.
It looks like your current implementation is very dependent on IOASID_SET_TYPE_MM? I'll need to do more reading about cgroup to see how easily it can be adapted to host SVA which uses IOASID_SET_TYPE_NULL.
Thanks,
Jean
Re: [RFC PATCH 15/18] cgroup: Introduce ioasids controller
Hi Jean-Philippe,
Quoted 48 lines · Jean-Philippe Brucker
Quoted 28 lines · Jacob Pan
Hi Jacob,
Quoted 23 lines · Jacob Pan
Hi Tejun,
Quoted 19 lines · Tejun Heo
Quoted 5 lines · Jacob Pan
IOASIDs are used to associate DMA requests with virtual address spaces. They are a system-wide limited resource made available to the userspace applications. Let it be VMs or user-space device drivers.
Quoted 7 lines
This RFC patch introduces a cgroup controller to address the following problems:
1. Some user applications exhaust all the available IOASIDs thus depriving others of the same host.
2. System admins need to provision VMs based on their needs for IOASIDs, e.g. the number of VMs with assigned devices that perform DMA requests with PASID.Please take a look at the proposed misc controller:
http://lkml.kernel.org/r/[email protected]
Would that fit your bill?
Quoted 11 lines
Quoted 8 lines
The interface definitely can be reused. But IOASID has a different behavior in terms of migration and ownership checking. I guess SEV key IDs are not tied to a process whereas IOASIDs are. Perhaps this can be solved by adding
kernel/cgroup/ioasids.c new file +2
-0+ .can_attach = ioasids_can_attach, + .cancel_attach = ioasids_cancel_attach,Let me give it a try and come back.
While I am trying to fit the IOASIDs cgroup in to the misc cgroup proposal. I'd like to have a direction check on whether this idea of using cgroup for IOASID/PASID resource management is viable.
Yes, even for host SVA it would be good to have a cgroup. Currently the number of shared address spaces is naturally limited by number of processes, which can be controlled with rlimit and cgroup. But on Arm the hardware limit on shared address spaces is 64k (number of ASIDs), easily exhausted with the default PASID and PID limits. So a cgroup for managing this resource is more than welcome.
Quoted 4 lines
It looks like your current implementation is very dependent on
IOASID_SET_TYPE_MM? I'll need to do more reading about cgroup to see how easily it can be adapted to host SVA which usesIOASID_SET_TYPE_NULL.
Right, I was assuming have three use cases of IOASIDs:
1. host supervisor SVA (not a concern, just one init_mm to bind)
2. host user SVA, either one IOASID per process or perhaps some private IOASID for private address space
3. VM use for guest SVA, each IOASID is bound to a guest process
My current cgroup proposal applies to #3 with IOASID_SET_TYPE_MM, which is allocated by the new /dev/ioasid interface.
For #2, I was thinking you can limit the host process via PIDs cgroup? i.e. limit fork. So the host IOASIDs are currently allocated from the system pool with quota of chosen by iommu_sva_init() in my patch, 0 means unlimited use whatever is available. https://lkml.org/lkml/2021/2/28/18
Quoted 2 lines
Thanks,
Jean
Thanks,
Jacob
_______________________________________________
iommu mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/iommu
Re: [RFC PATCH 15/18] cgroup: Introduce ioasids controller
Quoted 6 lines · Jacob Pan
Right, I was assuming have three use cases of IOASIDs:
1. host supervisor SVA (not a concern, just oneinit_mmto bind)
2. host user SVA, either one IOASID per process or perhaps some private IOASID for private address space
3. VM use for guest SVA, each IOASID is bound to a guest process
Quoted 7 lines
My current cgroup proposal applies to #3 with
IOASID_SET_TYPE_MM, which is allocated by the new/dev/ioasidinterface.For #2, I was thinking you can limit the host process via PIDs cgroup? i.e. limit fork. So the host IOASIDs are currently allocated from the system pool with quota of chosen by
iommu_sva_init()in my patch, 0 means unlimited use whatever is available. https://lkml.org/lkml/2021/2/28/18
Why do we need two pools?
If PASID's are limited then why does it matter how the PASID was allocated? Either the thing requesting it is below the limit, or it isn't.
For something like qemu I'd expect to put the qemu process in a cgroup with 1 PASID. Who cares what qemu uses the PASID for, or how it was allocated?
Jason
Re: [RFC PATCH 15/18] cgroup: Introduce ioasids controller
Hi Jason,
Quoted 17 lines · Jason Gunthorpe
Quoted 6 lines · Jacob Pan
Right, I was assuming have three use cases of IOASIDs:
1. host supervisor SVA (not a concern, just oneinit_mmto bind)
2. host user SVA, either one IOASID per process or perhaps some private IOASID for private address space
3. VM use for guest SVA, each IOASID is bound to a guest processQuoted 8 lines
My current cgroup proposal applies to #3 with
IOASID_SET_TYPE_MM, which is allocated by the new/dev/ioasidinterface.For #2, I was thinking you can limit the host process via PIDs cgroup? i.e. limit fork. So the host IOASIDs are currently allocated from the system pool with quota of chosen by
iommu_sva_init()in my patch, 0 means unlimited use whatever is available.
https://lkml.org/lkml/2021/2/28/18
Quoted 6 lines
Why do we need two pools?
If PASID's are limited then why does it matter how the PASID was allocated? Either the thing requesting it is below the limit, or it isn't.
you are right. it should be tracked based on the process regardless it is allocated by the user (/dev/ioasid) or indirectly by kernel drivers during iommu_sva_bind_device(). Need to consolidate both 2 and 3 and decouple cgroup and IOASID set.
Quoted 4 lines
For something like qemu I'd expect to put the qemu process in a cgroup with 1 PASID. Who cares what qemu uses the PASID for, or how it was allocated?
For vSVA, we will need one PASID per guest process. But that is up to the admin based on whether or how many SVA capable devices are directly assigned.
Quoted 1 line
Jason
Thanks,
Jacob
_______________________________________________
iommu mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/iommu
Re: [RFC PATCH 15/18] cgroup: Introduce ioasids controller
Quoted 7 lines · Jacob Pan
Quoted 3 lines
For something like qemu I'd expect to put the qemu process in a cgroup with 1 PASID. Who cares what qemu uses the PASID for, or how it was allocated?
For vSVA, we will need one PASID per guest process. But that is up to the admin based on whether or how many SVA capable devices are directly assigned.
I hope the virtual IOMMU driver can communicate the PASID limit and the cgroup machinery in the guest can know what the actual limit is.
I was thinking of a case where qemu is using a single PASID to setup the guest kVA or similar
Jason
Re: [RFC PATCH 15/18] cgroup: Introduce ioasids controller
Hi Jason,
Quoted 6 lines · Jason Gunthorpe
Quoted 4 lines · Jacob Pan
Quoted 3 lines
For something like qemu I'd expect to put the qemu process in a cgroup with 1 PASID. Who cares what qemu uses the PASID for, or how it was allocated?
Quoted 7 lines
Quoted 3 lines
For vSVA, we will need one PASID per guest process. But that is up to the admin based on whether or how many SVA capable devices are directly assigned.
I hope the virtual IOMMU driver can communicate the PASID limit and the cgroup machinery in the guest can know what the actual limit is.
For VT-d, emulated vIOMMU can communicate with the guest IOMMU driver on how many PASID bits are supported (extended cap reg PASID size fields). But it cannot communicate how many PASIDs are in the pool(host cgroup capacity).
The QEMU process may not be the only one in a cgroup so it cannot give hard guarantees. I don't see a good way to communicate accurately at runtime as the process migrates or limit changes.
We were thinking to adopt the "Limits" model as defined in the cgroup-v2 doc.
"
Limits
------
A child can only consume upto the configured amount of the resource. Limits can be over-committed - the sum of the limits of children can exceed the amount of resource available to the parent.
"
So the guest cgroup would still think it has full 20 bits of PASID at its disposal. But PASID allocation may fail before reaching the full 20 bits (2M).
Similar on the host side, we only enforce the limit set by the cgroup but not guarantee it.
Quoted 3 lines
I was thinking of a case where qemu is using a single PASID to setup the guest kVA or similar
got it.
Quoted 1 line
Jason
Thanks,
Jacob
_______________________________________________
iommu mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/iommu
Re: [RFC PATCH 15/18] cgroup: Introduce ioasids controller
In reply to Jacob PanQuoted 63 lines · Jacob Pan
Hi Jean-Philippe,
Quoted 48 lines · Jean-Philippe Brucker
Quoted 28 lines · Jacob Pan
Hi Jacob,
Quoted 23 lines · Jacob Pan
Hi Tejun,
Quoted 19 lines · Tejun Heo
Quoted 5 lines · Jacob Pan
IOASIDs are used to associate DMA requests with virtual address spaces. They are a system-wide limited resource made available to the userspace applications. Let it be VMs or user-space device drivers.
Quoted 7 lines
This RFC patch introduces a cgroup controller to address the following problems:
1. Some user applications exhaust all the available IOASIDs thus depriving others of the same host.
2. System admins need to provision VMs based on their needs for IOASIDs, e.g. the number of VMs with assigned devices that perform DMA requests with PASID.Please take a look at the proposed misc controller:
http://lkml.kernel.org/r/20210302081705.1990283-2-vipinsh-hpIqsD4AKlfQT0dZR+AlfA@public.gmane.org
Would that fit your bill?
Quoted 11 lines
Quoted 8 lines
The interface definitely can be reused. But IOASID has a different behavior in terms of migration and ownership checking. I guess SEV key IDs are not tied to a process whereas IOASIDs are. Perhaps this can be solved by adding
kernel/cgroup/ioasids.c new file +2
-0+ .can_attach = ioasids_can_attach, + .cancel_attach = ioasids_cancel_attach,Let me give it a try and come back.
While I am trying to fit the IOASIDs cgroup in to the misc cgroup proposal. I'd like to have a direction check on whether this idea of using cgroup for IOASID/PASID resource management is viable.
Yes, even for host SVA it would be good to have a cgroup. Currently the number of shared address spaces is naturally limited by number of processes, which can be controlled with rlimit and cgroup. But on Arm the hardware limit on shared address spaces is 64k (number of ASIDs), easily exhausted with the default PASID and PID limits. So a cgroup for managing this resource is more than welcome.
Quoted 4 lines
It looks like your current implementation is very dependent on
IOASID_SET_TYPE_MM? I'll need to do more reading about cgroup to see how easily it can be adapted to host SVA which usesIOASID_SET_TYPE_NULL.Right, I was assuming have three use cases of IOASIDs:
1. host supervisor SVA (not a concern, just oneinit_mmto bind)
2. host user SVA, either one IOASID per process or perhaps some private IOASID for private address space
3. VM use for guest SVA, each IOASID is bound to a guest process
Quoted 5 lines
My current cgroup proposal applies to #3 with
IOASID_SET_TYPE_MM, which is allocated by the new/dev/ioasidinterface.For #2, I was thinking you can limit the host process via PIDs cgroup? i.e. limit fork.
That works but isn't perfect, because the hardware resource of shared address spaces can be much lower that PID limit - 16k ASIDs on Arm. To allow an admin to fairly distribute that resource we could introduce another cgroup just to limit the number of shared address spaces, but limiting the number of IOASIDs does the trick.
Quoted 3 lines
So the host IOASIDs are currently allocated from the system pool with quota of chosen by
iommu_sva_init()in my patch, 0 means unlimited use whatever is available. https://lkml.org/lkml/2021/2/28/18
Yes that's sensible, but it would be good to plan the cgroup user interface to work for #2 as well, even if we don't implement it right away.
Thanks,
Jean
Re: [RFC PATCH 15/18] cgroup: Introduce ioasids controller
Quoted 2 lines · Jean-Philippe Brucker
That works but isn't perfect, because the hardware resource of shared address spaces can be much lower that PID limit - 16k ASIDs on Arm. To
Sorry I meant 16-bit here - 64k
Thanks,
Jean
Re: [RFC PATCH 15/18] cgroup: Introduce ioasids controller
In reply to Jean-Philippe BruckerHi Jean-Philippe,
Quoted 67 lines · Jean-Philippe Brucker
Quoted 66 lines · Jacob Pan
Hi Jean-Philippe,
Quoted 48 lines · Jean-Philippe Brucker
Quoted 28 lines · Jacob Pan
Hi Jacob,
Quoted 23 lines · Jacob Pan
Hi Tejun,
Quoted 19 lines · Tejun Heo
Quoted 5 lines · Jacob Pan
IOASIDs are used to associate DMA requests with virtual address spaces. They are a system-wide limited resource made available to the userspace applications. Let it be VMs or user-space device drivers.
Quoted 7 lines
This RFC patch introduces a cgroup controller to address the following problems:
1. Some user applications exhaust all the available IOASIDs thus depriving others of the same host.
2. System admins need to provision VMs based on their needs for IOASIDs, e.g. the number of VMs with assigned devices that perform DMA requests with PASID.Please take a look at the proposed misc controller:
http://lkml.kernel.org/r/[email protected]
Would that fit your bill?
Quoted 11 lines
Quoted 8 lines
The interface definitely can be reused. But IOASID has a different behavior in terms of migration and ownership checking. I guess SEV key IDs are not tied to a process whereas IOASIDs are. Perhaps this can be solved by adding
(quoted diff) +2
-0+ .can_attach = ioasids_can_attach, + .cancel_attach = ioasids_cancel_attach,Let me give it a try and come back.
While I am trying to fit the IOASIDs cgroup in to the misc cgroup proposal. I'd like to have a direction check on whether this idea of using cgroup for IOASID/PASID resource management is viable.
Yes, even for host SVA it would be good to have a cgroup. Currently the number of shared address spaces is naturally limited by number of processes, which can be controlled with rlimit and cgroup. But on Arm the hardware limit on shared address spaces is 64k (number of ASIDs), easily exhausted with the default PASID and PID limits. So a cgroup for managing this resource is more than welcome.
Quoted 4 lines
It looks like your current implementation is very dependent on
IOASID_SET_TYPE_MM? I'll need to do more reading about cgroup to see how easily it can be adapted to host SVA which usesIOASID_SET_TYPE_NULL.Right, I was assuming have three use cases of IOASIDs:
1. host supervisor SVA (not a concern, just oneinit_mmto bind)
2. host user SVA, either one IOASID per process or perhaps some private IOASID for private address space
3. VM use for guest SVA, each IOASID is bound to a guest processMy current cgroup proposal applies to #3 with
IOASID_SET_TYPE_MM, which is allocated by the new/dev/ioasidinterface.
Quoted 9 lines
Quoted 2 lines
For #2, I was thinking you can limit the host process via PIDs cgroup? i.e. limit fork.
That works but isn't perfect, because the hardware resource of shared address spaces can be much lower that PID limit - 16k ASIDs on Arm. To allow an admin to fairly distribute that resource we could introduce another cgroup just to limit the number of shared address spaces, but limiting the number of IOASIDs does the trick.
make sense. it would be cleaner to have a single approach to limit IOASIDs (as Jason asked).
Quoted 8 lines
Quoted 3 lines
So the host IOASIDs are currently allocated from the system pool with quota of chosen by
iommu_sva_init()in my patch, 0 means unlimited use whatever is available. https://lkml.org/lkml/2021/2/28/18Yes that's sensible, but it would be good to plan the cgroup user interface to work for #2 as well, even if we don't implement it right away.
will do it in the next version.
Quoted 2 lines
Thanks,
Jean
Thanks,
Jacob
_______________________________________________
iommu mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/iommu
RFC PATCH v4 16/18 iommu/ioasid: Consult IOASIDs cgroup for allocation
In reply to Jacob PanOnce IOASIDs cgroup is active, we must consult the limitation set up by the cgroups during allocation. Freeing IOASIDs also need to return the quota back to the cgroup.
--- drivers/iommu/ioasid.c | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-)
drivers/iommu/ioasid.c +6 -1
@@ -782,7 +782,10 @@ ioasid_t ioasid_alloc(struct ioasid_set *set, ioasid_t min, ioasid_t max,spin_lock (&ioasid_allocator_lock);/* Check if the IOASID set has been allocated and initialized */ -if (!ioasid_set_is_valid (set)) +if (!set || !ioasid_set_is_valid (set)) + goto done_unlock; + +if (set->type ==IOASID_SET_TYPE_MM &&ioasid_cg_charge (set)) goto done_unlock;if (set->quota <=atomic_read (&set->nr_ioasids)) {
@@ -832,6 +835,7 @@ ioasid_t ioasid_alloc(struct ioasid_set *set, ioasid_t min, ioasid_t max, goto done_unlock; exit_free:kfree (data); +ioasid_cg_uncharge (set); done_unlock:spin_unlock (&ioasid_allocator_lock);return id;
@@ -849,6 +853,7 @@ static void ioasid_do_free_locked(struct ioasid_data *data)kfree_rcu (ioasid_data, rcu); }atomic_dec (&data->set->nr_ioasids); +ioasid_cg_uncharge (data->set);xa_erase (&data->set->xa, data->id);/* Destroy the set if empty */ if (!atomic_read (&data->set->nr_ioasids))
Signature · 1 line
2.25.1
RFC PATCH v4 17/18 docs: cgroup-v1: Add IOASIDs controller
In reply to Jacob Pan--- Documentation/admin-guide/cgroup-v1/index.rst | 1 + .../admin-guide/cgroup-v1/ioasids.rst | 110 ++++++++++++++++++ 2 files changed, 111 insertions(+) create mode 100644 Documentation/admin-guide/cgroup-v1/ioasids.rst
Documentation/admin-guide/cgroup-v1/index.rst +1 -0
@@ -15,6 +15,7 @@ Control Groups version 1 devices freezer-subsystem hugetlb + ioasids memcg_test memory net_cls
Documentation/admin-guide/cgroup-v1/ioasids.rst new file +110 -0
110 lines not on this page · raw message
Signature · 6 lines
2.25.1 _______________________________________________ iommu mailing list [email protected] https://lists.linuxfoundation.org/mailman/listinfo/iommu
RFC PATCH v4 18/18 ioasid: Add /dev/ioasid for userspace
In reply to Jacob PanFrom: Liu Yi L <[email protected]>
I/O Address Space IDs (IOASIDs) is used to tag DMA requests to target multiple DMA address spaces for physical devices. Its PCI terminology is called PASID (Process Address Space ID). Platforms with PASID support can provide PASID granularity DMA isolation, which is very useful for efficient and secure device sharing (SVA, subdevice passthrough, etc.).
Today only kernel drivers are allowed to allocate IOASIDs [1]. This patch aims to extend this capability to userspace as required in device pass- through scenarios. For example, a userspace driver may want to create its own DMA address spaces besides the default IOVA address space established by the kernel on the assigned device (e.g. vDPA control vq [2] and guest SVA [3]), thus need to get IOASIDs from the kernel IOASID allocator for tagging. In concept, each device can have its own IOASID space, thus it's also possible for userspace driver to manage a private IOASID space itself, say, when PF/VF is assigned. However it doesn't work for subdevice pass- through, as multiple subdevices under the same parent device share a single IOASID space thus IOASIDs must be centrally managed by the kernel in such case.
This patch introduces a /dev/ioasid interface for this purpose (per discussion in [4]). An IOASID is just a number before it is tagged to a specific DMA address space. The actual IOASID tagging (to DMA requests) and association (with DMA address spaces) operations from userspace are scrutinized by specific device passthrough frameworks, which must ensure that a malicious driver cannot program arbitrary IOASIDs to its assigned device to access DMA address spaces that don't belong to it, this is out of the scope of this patch (a reference VFIO implementation will be posted soon).
Open:
PCIe PASID is 20bit implying a space with 1M IOASIDs. although it's plenty there was an open [4] on whether this user interface is open to all processes or only selective processes (e.g. with device assigned). In this patchseries, a cgroup controller is introduced to manage IOASID quota that a process is allowed to use. A cgroup-enabled system may by default set quota=0 to disallow IOASID allocation for most processes, and then having the virt management stack to adjust the quota for a process which gets device assigned. But yeah, we are also willing to hear more suggestions.
[1] https://lore.kernel.org/linux-iommu/1565900005-62508-8-git-send-email-jacob.jun.pan-VuQAYsv1563Yd54FQh9/[email protected]/
[2] https://lore.kernel.org/kvm/20201216064818.48239-1-jasowang-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org/
[3] https://lore.kernel.org/linux-iommu/1599734733-6431-1-git-send-email-yi.l.liu-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org/
[4] https://lore.kernel.org/kvm/[email protected]/
--- Documentation/userspace-api/index.rst | 1 + Documentation/userspace-api/ioasid.rst | 49 ++++ drivers/iommu/Kconfig | 5 + drivers/iommu/Makefile | 1 + drivers/iommu/intel/Kconfig | 1 + drivers/iommu/ioasid_user.c | 297 +++++++++++++++++++++++++ include/linux/ioasid.h | 26 +++ include/linux/miscdevice.h | 1 + include/uapi/linux/ioasid.h | 98 ++++++++ 9 files changed, 479 insertions(+) create mode 100644 Documentation/userspace-api/ioasid.rst create mode 100644 drivers/iommu/ioasid_user.c create mode 100644 include/uapi/linux/ioasid.h
Documentation/userspace-api/index.rst +1 -0
@@ -24,6 +24,7 @@ place where this information is gathered. ioctl/index iommu media/index + ioasid .. only:: subproject and html
Documentation/userspace-api/ioasid.rst new file +49 -0
49 lines not on this page · raw message
drivers/iommu/Kconfig +5 -0
@@ -7,6 +7,11 @@ config IOMMU_IOVA config IOASID tristate +config IOASID_USER + tristate + depends on IOASID + default n + # IOMMU_API always gets selected by whoever wants it. config IOMMU_API bool
drivers/iommu/Makefile +1 -0
@@ -9,6 +9,7 @@ obj-$(CONFIG_IOMMU_IO_PGTABLE) += io-pgtable.o obj-$(CONFIG_IOMMU_IO_PGTABLE_ARMV7S) += io-pgtable-arm-v7s.o obj-$(CONFIG_IOMMU_IO_PGTABLE_LPAE) += io-pgtable-arm.o obj-$(CONFIG_IOASID) += ioasid.o +obj-$(CONFIG_IOASID_USER) += ioasid_user.o obj-$(CONFIG_IOMMU_IOVA) += iova.o obj-$(CONFIG_OF_IOMMU) += of_iommu.o obj-$(CONFIG_MSM_IOMMU) += msm_iommu.o
drivers/iommu/intel/Kconfig +1 -0
@@ -13,6 +13,7 @@ config INTEL_IOMMU select DMAR_TABLE select SWIOTLB select IOASID + select IOASID_USER select IOMMU_DMA help DMA remapping (DMAR) devices support enables independent address
drivers/iommu/ioasid_user.c new file +297 -0
297 lines not on this page · raw message
include/linux/ioasid.h +26 -0
@@ -6,6 +6,7 @@#include <linux/errno.h> #include <linux/xarray.h> #include <linux/refcount.h> +#include <uapi/linux/ioasid.h> #define INVALID_IOASID ((ioasid_t)-1)typedef unsigned int ioasid_t ;
@@ -152,6 +153,31 @@ static inline int ioasid_cg_uncharge(struct ioasid_set *set)#endif /* CGROUP_IOASIDS */ bool ioasid_queue_work (struct work_struct *work); +/* IOASID userspace support */ +struct ioasid_user ; +#if IS_ENABLED (CONFIG_IOASID_USER ) +extern struct ioasid_user *ioasid_user_get_from_task (struct task_struct *task); +extern void ioasid_user_put (struct ioasid_user *iuser); +extern void ioasid_user_for_each_id (struct ioasid_user *iuser,void *data, +void (*fn)(ioasid_t id,void *data)); + +#else /* CONFIG_IOASID_USER */ +static inline struct ioasid_user * +ioasid_user_get_from_task (struct task_struct *task) +{ +return ERR_PTR (-ENOTTY ); +} + +static inline void ioasid_user_put (struct ioasid_user *iuser) +{ +} + +static inline void ioasid_user_for_each_id (struct ioasid_user *iuser,void *data, +void (*fn)(ioasid_t id,void *data)) +{ +} +#endif /* CONFIG_IOASID_USER */ +#else /* !CONFIG_IOASID */ static inline void ioasid_install_capacity (ioasid_t total)
include/linux/miscdevice.h +1 -0
@@ -21,6 +21,7 @@#define APOLLO_MOUSE_MINOR 7/* unused */ #define PC110PAD_MINOR 9/* unused */ /*#define ADB_MOUSE_MINOR 10 FIXME OBSOLETE */ +#define IOASID_MINOR 129/* /dev/ioasid */ #define WATCHDOG_MINOR 130/* Watchdog timer */ #define TEMP_MINOR 131/* Temperature Sensor */ #define APM_MINOR_DEV 134
include/uapi/linux/ioasid.h new file +98 -0
98 lines not on this page · raw message
Signature · 1 line
2.25.1
Re: [RFC PATCH 18/18] ioasid: Add /dev/ioasid for userspace
include/uapi/linux/ioasid.h line 42
Quoted 12 lines · Jacob Pan
include/uapi/linux/ioasid.h new file +12
-0+/* -------- IOCTLs for IOASID file descriptor (/dev/ioasid) -------- */ + +/** +* IOASID_GET_API_VERSION - _IO(IOASID_TYPE, IOASID_BASE + 0) +* +* Report the version of the IOASID API. This allows us to bump the entire +* API version should we later need to add or change features in incompatible +* ways. +* Return: IOASID_API_VERSION +* Availability: Always +*/ +#define IOASID_GET_API_VERSION _IO(IOASID_TYPE, IOASID_BASE + 0)
I think this is generally a bad idea, if you change the API later then also change the ioctl numbers and everything should work out
eg use the 4th argument to IOC to specify something about the ABI
Jason
Re: [RFC PATCH 18/18] ioasid: Add /dev/ioasid for userspace
Hi Jason,
Thanks for the review.
Quoted 21 lines · Jason Gunthorpe
Quoted 7 lines · Jacob Pan
include/uapi/linux/ioasid.h new file +6
-0+/* -------- IOCTLs for IOASID file descriptor (/dev/ioasid) -------- */ + +/** +* IOASID_GET_API_VERSION - _IO(IOASID_TYPE, IOASID_BASE + 0) +* +* Report the version of the IOASID API. This allows us to bump the entire
Quoted 8 lines
+ * API version should we later need to add or change features in incompatible
+ * ways.
+ * Return:IOASID_API_VERSION
+ * Availability: Always
+ */
+#defineIOASID_GET_API_VERSION_IO(IOASID_TYPE,IOASID_BASE+ 0)I think this is generally a bad idea, if you change the API later then also change the ioctl numbers and everything should work out
Quoted 2 lines
eg use the 4th argument to IOC to specify something about the ABI
Let me try to understand the idea, do you mean something like this? #define IOASID_GET_INFO _IOC(_IOC_NONE, IOASID_TYPE, IOASID_BASE + 1, sizeof(struct ioasid_info))
If we later change the size of struct ioasid_info, IOASID_GET_INFO would be a different ioctl number. Then we will break the existing user space that uses the old number. So I am guessing you meant we need to have a different name also. i.e.
#define IOASID_GET_INFO_V2 _IOC(_IOC_NONE, IOASID_TYPE, IOASID_BASE + 1, sizeof(struct ioasid_info_v2))
We can get rid of the API version, just have individual IOCTL version. Is that right?
Quoted 1 line
Jason
Thanks,
Jacob
_______________________________________________
iommu mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/iommu
Re: [RFC PATCH 18/18] ioasid: Add /dev/ioasid for userspace
Quoted 29 lines · Jacob Pan
Hi Jason,
Thanks for the review.
Quoted 21 lines · Jason Gunthorpe
Quoted 7 lines · Jacob Pan
include/uapi/linux/ioasid.h new file +6
-0+/* -------- IOCTLs for IOASID file descriptor (/dev/ioasid) -------- */ + +/** +* IOASID_GET_API_VERSION - _IO(IOASID_TYPE, IOASID_BASE + 0) +* +* Report the version of the IOASID API. This allows us to bump the entire
Quoted 8 lines
+ * API version should we later need to add or change features in incompatible
+ * ways.
+ * Return:IOASID_API_VERSION
+ * Availability: Always
+ */
+#defineIOASID_GET_API_VERSION_IO(IOASID_TYPE,IOASID_BASE+ 0)I think this is generally a bad idea, if you change the API later then also change the ioctl numbers and everything should work out
Quoted 2 lines
eg use the 4th argument to IOC to specify something about the ABI
Quoted 8 lines
Let me try to understand the idea, do you mean something like this? #define
IOASID_GET_INFO_IOC(_IOC_NONE, IOASID_TYPE, IOASID_BASE + 1, sizeof(struct ioasid_info))If we later change the size of struct
ioasid_info,IOASID_GET_INFOwould be a different ioctl number. Then we will break the existing user space that uses the old number. So I am guessing you meant we need to have a different name also. i.e.
Something like that is more appropriate. Generally we should not be planning to 'remove' IOCTLs. The kernel must always have backwards compat, so any new format you introduce down the road has to have new IOCTL number so the old format can continue to be supported.
Negotiation of support can usually by done by probing for ENOIOCTLCMD or similar on the new ioctls, not an API version
Jason
Quoted 53 lines
From: Jacob Pan <jacob.jun.pan-VuQAYsv1563Yd54FQh9/[email protected]>
Sent: Sunday, February 28, 2021 6:01 AMI/O Address Space ID (IOASID) core code was introduced in v5.5 as a generic kernel allocator service for both PCIe Process Address Space ID (PASID) and ARM SMMU's Substream ID. IOASIDs are used to associate DMA requests with
virtual address spaces, including both host and guest.In addition to providing basic ID allocation,
ioasid_setwas defined as a token that is shared by a group of IOASIDs. This set token can be used for permission checking, but lack some features to address the following needs by guest Shared Virtual Address (SVA).
- Manage IOASIDs by group, group ownership, quota, etc.
- State synchronization among IOASID users (e.g. IOMMU driver, KVM, device
drivers)
- Non-identity guest-host IOASID mapping
- Lifecycle managementThis patchset introduces the following extensions as solutions to the problems above.
- Redefine and extend IOASID set such that IOASIDs can be managed by groups/pools.
- Add notifications for IOASID state synchronization
- Extend reference counting for life cycle alignment among multiple users
- Supportioasid_setprivate IDs, which can be used as guest IOASIDs
- Add a new cgroup controller for resource distributionPlease refer to
Documentation/admin-guide/cgroup-v1/ioasids.rstandDocumentation/driver-api/ioasid.rstin the enclosed patches for more details.Based on discussions on LKML[1], a direction change was made in v4 such that
the user interfaces for IOASID allocation are extracted from VFIO subsystem. The proposed IOASID subsystem now consists of three components:
1. IOASID core[01-14]: provides APIs for allocation, pool management, notifications, and refcounting.
2. IOASID cgroup controller[RFC 15-17]: manage resource distribution[2].
3. IOASID user[RFC 18]: provides user allocation interface via/dev/ioasidThis patchset only included VT-d driver as users of some of the new APIs. VFIO and KVM patches are coming up to fully utilize the APIs introduced here.
[1] https://lore.kernel.org/linux-iommu/1599734733-6431-1-git-send-email-
[email protected]/
[2] Note that ioasid quota management code can be removed once the IOASIDs
cgroup is ratified.
Quoted 3 lines
You can find this series, VFIO, KVM, and IOASID user at:
https://github.com/jacobpan/linux.gitioasid_v4
(VFIO and KVM patches will be available at this branch when published.)
VFIO and QEMU series are listed below:
VFIO: https://lore.kernel.org/linux-iommu/20210302203545.436623-1-yi.l.liu-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org/
QEMU: https://lore.kernel.org/qemu-devel/20210302203827.437645-1-yi.l.liu-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org/T/#t
Regards,
Yi Liu