bind pasid table API
Hi Jean and All,
This is a follow-up on the LPC discussion we had last week.
(https://linuxplumbersconf.org/2017/ocw/proposals/4748)
My understanding is that the data structure below can satisfy the needs from Intel (pointer + size) and AMD (pointer only). But ARM pvIOMMU would need additional info to indicate the page table format. Could you share your idea of the right addition for ARM such that we can have a unified API?
/**
* PASID table data used to bind guest PASID table to the host IOMMU. This will
* enable guest managed first level page tables.
* @ptr: PASID table pointer
* @size_order: number of bits supported in the guest PASID table, must be less
* or equal than the host table size.
*/
struct pasid_table_info {
__u64 ptr;
__u64 size_order;
};Thanks,
Jacob
Hi Jacob,
[Adding Eric as he might need pasid_table_info for vSVM at some point]
Quoted 18 lines · Jacob Pan
Hi Jean and All,
This is a follow-up on the LPC discussion we had last week.
(https://linuxplumbersconf.org/2017/ocw/proposals/4748)My understanding is that the data structure below can satisfy the needs from Intel (pointer + size) and AMD (pointer only). But ARM pvIOMMU would need additional info to indicate the page table format. Could you share your idea of the right addition for ARM such that we can have a unified API?
/**
* PASID table data used to bind guest PASID table to the host IOMMU. This will
* enable guest managed first level page tables.
* @ptr: PASID table pointer
* @size_order: number of bits supported in the guest PASID table, must be less
* or equal than the host table size.
*/
Quoted 4 lines
struct pasid_table_info {
__u64 ptr;
__u64 size_order;
};For the PASID table, Arm SMMUv3 would need two additional fields:
* 'format' telling whether the table has 1 or 2 levels and their dimensions,
* 'default_substream' telling if PASID0 is reserved for non-pasid traffic.
I think that's it for the moment, but it does require to leave space for a vendor-specific structure at the end. It is one reason why I'd prefer having a 'model' field in the pasid_table_info structure telling what fields the whole structure actually contains.
Another reason is if some IOMMU is able to support multiple PASID table formats, it could advertise them all in sysfs and Qemu could tell which one it chose in 'model'. I'm not sure we'll ever see that in practice.
For binding page tables instead of PASID tables (e.g. virtio-iommu), the generic data would be:
struct pgtable_info {
__u32 pasid;
__u64 ptr;
__u32 model;
__u8 model_data[];
};Followed by a few arch-specific configuration values. For Arm we can summarize this to three registers, defined in the Armv8 Architecture Reference Manual:
struct arm_lpae_pgtable_info {
__u64 tcr; /* Translation Control Register */
__u64 mair; /* Memory Attributes Indirection Register */
__u64 asid; /* Address Space ID */
};
Some data packed in the TCR might be common to most architectures, like page granularity and max VA size. Most fields of the TCR won't be used but it provides a nice architected way to communicate Arm page table configuration.
Note that there might be an additional page directory in the arch-specific info, as we can split the address space in two. I'm not sure whether we should allow it yet.
Thanks,
Jean
Quoted 30 lines · On Wed, 20 Sep 2017 13:09:47 +0100 Jean-Philippe Brucker…
Hi Jacob,
[Adding Eric as he might need
pasid_table_infofor vSVM at some point]Quoted 20 lines · Jacob Pan
Hi Jean and All,
This is a follow-up on the LPC discussion we had last week.
(https://linuxplumbersconf.org/2017/ocw/proposals/4748)My understanding is that the data structure below can satisfy the needs from Intel (pointer + size) and AMD (pointer only). But ARM pvIOMMU would need additional info to indicate the page table format. Could you share your idea of the right addition for ARM such that we can have a unified API?
/**
* PASID table data used to bind guest PASID table to the host
IOMMU. This will
* enable guest managed first level page tables.
* @ptr: PASID table pointer
* @size_order: number of bits supported in the guest PASID
table, must be less
* or equal than the host table size.
*/Quoted 4 lines
struct pasid_table_info { __u64 ptr; __u64 size_order; };
Quoted 3 lines
For the PASID table, Arm SMMUv3 would need two additional fields:
* 'format' telling whether the table has 1 or 2 levels and their dimensions,
just wondering if the format will be different than whatever is used on the host? i.e. could there be a mismatch? I am ok with this field just wondering if this can be resolved via the sysfs query interface if that is a global thing.
Quoted 3 lines
* '
default_substream' telling if PASID0 is reserved for non-pasid traffic.
could it be part of a feature flag? i.e.
struct pasid_table_info {
__u64 ptr;
__u64 size_order; __u64 flags; #define PASID0_FOR_DMA_WITHOUT_PASID; enum pasid_table_format format; };
Quoted 5 lines
I think that's it for the moment, but it does require to leave space for a vendor-specific structure at the end. It is one reason why I'd prefer having a 'model' field in the
pasid_table_infostructure telling what fields the whole structure actually contains.
I think we have been there before. the downside is that model specific knowledge is required in the generic VFIO layer, if its content is to be inspected.
Quoted 5 lines
Another reason is if some IOMMU is able to support multiple PASID table formats, it could advertise them all in sysfs and Qemu could tell which one it chose in 'model'. I'm not sure we'll ever see that in practice.
I would expect when query interface between QEMU and sysfs would ensure matching format prior to issue bind_pasid_table call.
Quoted 22 lines
For binding page tables instead of PASID tables (e.g. virtio-iommu), the generic data would be:
struct pgtable_info { __u32 pasid; __u64 ptr; __u32 model; __u8 model_data[]; };Followed by a few arch-specific configuration values. For Arm we can summarize this to three registers, defined in the Armv8 Architecture Reference Manual:
struct
arm_lpae_pgtable_info{
__u64tcr; /* Translation Control Register */
__u64mair; /* Memory Attributes Indirection
Register */__u64asid; /* Address Space ID */
};
Quoted 9 lines
Some data packed in the TCR might be common to most architectures, like page granularity and max VA size. Most fields of the TCR won't be used but it provides a nice architected way to communicate Arm page table configuration.
Note that there might be an additional page directory in the arch-specific info, as we can split the address space in two. I'm not sure whether we should allow it yet.
This can be combined with bind mm with user tasks also (e.g. DPDK with SVM in native case), right? In that case the pasid would be allocated by the kernel instead of passdown in pgtable_info, and your 'ptr' filed is harvested from the current task mm? There is also complexity w.r.t. user task life cycle management. My immediate goal to enable vSVM does not require bind page tables, try not to think too far ahead.
Quoted 2 lines
Thanks,
Jean
[Jacob Pan]
Hi Jean,
Quoted 45 lines
-----Original Message-----
From: Jean-Philippe Brucker [mailto:[email protected]]
Sent: Wednesday, September 20, 2017 8:10 PM
To: Pan, Jacob jun <[email protected]>; iommu-cunTk1MwBs/[email protected] foundation.org
Cc: Liu, Yi L <[email protected]>; Raj, Ashok <[email protected]>; David Woodhouse <[email protected]>; Joerg Roedel <[email protected]>; Tian, Kevin <[email protected]>; Auger Eric <[email protected]>
Subject: Re: bind pasid table APIHi Jacob,
[Adding Eric as he might need
pasid_table_infofor vSVM at some point]Quoted 11 lines · Jacob Pan
Hi Jean and All,
This is a follow-up on the LPC discussion we had last week.
(https://linuxplumbersconf.org/2017/ocw/proposals/4748)My understanding is that the data structure below can satisfy the needs from Intel (pointer + size) and AMD (pointer only). But ARM pvIOMMU would need additional info to indicate the page table format. Could you share your idea of the right addition for ARM such that we can have a unified API?
Quoted 6 lines
/**
* PASID table data used to bind guest PASID table to the host IOMMU.
This will
* enable guest managed first level page tables.
* @ptr: PASID table pointer
* @size_order: number of bits supported in the guest PASID table, mustbe less
Quoted 6 lines
* or equal than the host table size.
*/struct pasid_table_info { __u64 ptr; __u64 size_order; };For the PASID table, Arm SMMUv3 would need two additional fields:
* 'format' telling whether the table has 1 or 2 levels and their dimensions,
* 'default_substream' telling if PASID0 is reserved for non-pasid traffic.
Quoted 8 lines
I think that's it for the moment, but it does require to leave space for a vendor- specific structure at the end. It is one reason why I'd prefer having a 'model' field in the
pasid_table_infostructure telling what fields the whole structure actually contains.Another reason is if some IOMMU is able to support multiple PASID table formats, it could advertise them all in sysfs and Qemu could tell which one it chose in 'model'. I'm not sure we'll ever see that in practice.
Regards to your idea, I think the whole flow may be:
* Qemu queries underlying IOMMU for the capability(through sysfs)
* Combined with requirement from user(the guy who starts the VM), Qemu chose a suitable model or exit if HW capability is incompatible with user's requirement
* In the coming bind_pasid_table() calling, Qemu pass the chosen model info to host
* Host checks the "model" and use the correct model specific structure to parse the model specific data. may be kind of structure intel_pasid_table_info, amd_pasid_table_info or arm_pasid_table_info_v#
Does this flow show what you want? This would be a "model" + "model specific data" proposal. And my concern is the model specific field may look like to be kind of "opaque".
Besides the comments above, is there also possibility for us to put all the possible info in a super-set just as what we plan to do for the tlb_invalidate() API?
Quoted 10 lines
For binding page tables instead of PASID tables (e.g. virtio-iommu), the generic data would be:
struct pgtable_info { __u32 pasid; __u64 ptr; __u32 model; __u8 model_data[]; };
Besides bind_pasid_table API, you would also want to propose an extra API which likely to be named as bind_pgtable() for this page table binding?
What would be the "model" field indicate? "vendor" or "vendor+version" ? You may want a length field to indicate the size of "model_data" field.
And same with the bind_pasid_table API, would model_data look like an "opaque"?
Quoted 8 lines
Followed by a few arch-specific configuration values. For Arm we can summarize this to three registers, defined in the Armv8 Architecture Reference Manual:
struct
arm_lpae_pgtable_info{
__u64tcr; /* Translation Control Register */
__u64mair; /* Memory Attributes Indirection Register */
__u64asid; /* Address Space ID */
};
Hmmm, just for curious. What is the difference between "pasid" and "asid"?
Thanks,
Yi L
Quoted 10 lines
Some data packed in the TCR might be common to most architectures, like page granularity and max VA size. Most fields of the TCR won't be used but it provides a nice architected way to communicate Arm page table configuration.
Note that there might be an additional page directory in the arch-specific info, as we can split the address space in two. I'm not sure whether we should allow it yet.
Thanks,
Jean
Quoted 33 lines · Jacob Pan
Quoted 30 lines · On Wed, 20 Sep 2017 13:09:47 +0100 Jean-Philippe Brucker…
Hi Jacob,
[Adding Eric as he might need
pasid_table_infofor vSVM at some point]Quoted 24 lines · Jacob Pan
Hi Jean and All,
This is a follow-up on the LPC discussion we had last week.
(https://linuxplumbersconf.org/2017/ocw/proposals/4748)My understanding is that the data structure below can satisfy the needs from Intel (pointer + size) and AMD (pointer only). But ARM pvIOMMU would need additional info to indicate the page table format. Could you share your idea of the right addition for ARM such that we can have a unified API?
/**
* PASID table data used to bind guest PASID table to the host
IOMMU. This will
* enable guest managed first level page tables.
* @ptr: PASID table pointer
* @size_order: number of bits supported in the guest PASID
table, must be less
* or equal than the host table size.
*/struct pasid_table_info { __u64 ptr; __u64 size_order; };
Quoted 7 lines
Quoted 3 lines
For the PASID table, Arm SMMUv3 would need two additional fields:
* 'format' telling whether the table has 1 or 2 levels and their dimensions,just wondering if the format will be different than whatever is used on the host? i.e. could there be a mismatch? I am ok with this field just wondering if this can be resolved via the sysfs query interface if that is a global thing.
This format tells how the guest organizes its PASID tables. Depending on 'format', the PASID table can be:
* A flat array of descriptors
* One array of 1st-level descriptors pointing to a 2nd level of descriptors. When translating an access, the SMMU splits the PASID at bit 6, uses the upper half to get the 1st level descriptor and the lower half to get the 2nd level one (4k leaf table).
* The same format, but instead split at PASID bit 10 (64k leaf table)
So the guest has to tell the host what format of PASID tables it chose, but the configuration might be too vendor-specific to go in the generic part of pasid_table_info.
It seems that AMD IOMMUv2 can have either one, two or three levels of PASID tables. See definition of GLX field in the 3.2.2.1 Device Table Entry Format.
We could try to come up with a generic definition to cover all these cases, but it might be more complicated than appending a few vendor-specific fields. Only AMD can have three levels, only ARM can split the PASID in two different places. And we can expect a future architecture to come up with a new colorful format...
I'm mostly concerned about VFIO though, as the IOMMU API doesn't have the same backward compatible restriction and is therefore easier to update.
Quoted 13 lines
Quoted 3 lines
* '
default_substream' telling if PASID0 is reserved for non-pasid traffic.could it be part of a feature flag? i.e.
struct
pasid_table_info{(quoted diff) +0
-0__u64 ptr; __u64 size_order;__u64 flags; #define PASID0_FOR_DMA_WITHOUT_PASID; enum pasid_table_format format; };
Quoted 8 lines
Quoted 5 lines
I think that's it for the moment, but it does require to leave space for a vendor-specific structure at the end. It is one reason why I'd prefer having a 'model' field in the
pasid_table_infostructure telling what fields the whole structure actually contains.I think we have been there before. the downside is that model specific knowledge is required in the generic VFIO layer, if its content is to be inspected.
Yes, all of these vendor-specific fields would have to be UAPI. VFIO doesn't necessarily have to deeply inspect each field, it can just check that the structure contains the right fields (is the right size) and let the IOMMU driver, that knows how to read these values, validate that they make sense.
I think VFIO should be able to trust the IOMMU driver to perform the sanity checks.
Quoted 38 lines
Quoted 5 lines
Another reason is if some IOMMU is able to support multiple PASID table formats, it could advertise them all in sysfs and Qemu could tell which one it chose in 'model'. I'm not sure we'll ever see that in practice.
I would expect when query interface between QEMU and sysfs would ensure matching format prior to issue
bind_pasid_tablecall.Quoted 22 lines
For binding page tables instead of PASID tables (e.g. virtio-iommu), the generic data would be:
struct pgtable_info { __u32 pasid; __u64 ptr; __u32 model; __u8 model_data[]; };Followed by a few arch-specific configuration values. For Arm we can summarize this to three registers, defined in the Armv8 Architecture Reference Manual:
struct
arm_lpae_pgtable_info{
__u64tcr; /* Translation Control Register */
__u64mair; /* Memory Attributes Indirection
Register */__u64asid; /* Address Space ID */
};Quoted 9 lines
Some data packed in the TCR might be common to most architectures, like page granularity and max VA size. Most fields of the TCR won't be used but it provides a nice architected way to communicate Arm page table configuration.
Note that there might be an additional page directory in the arch-specific info, as we can split the address space in two. I'm not sure whether we should allow it yet.
Quoted 6 lines
This can be combined with bind mm with user tasks also (e.g. DPDK with SVM in native case), right? In that case the pasid would be allocated by the kernel instead of passdown in
pgtable_info, and your 'ptr' filed is harvested from the current task mm? There is also complexity w.r.t. user task life cycle management. My immediate goal to enable vSVM does not require bind page tables, try not to think too far ahead.
Yes, I'm working on the native case at the moment. The PASID is allocated by the kernel and returned in 'pasid', and user can specify an additional 'pid' field to bind another process. There would be a 'flag' field telling if 'pid' is valid or if we should just bind the current process.
Thanks,
Jean
Quoted 65 lines · Yi L
Hi Jean,
Quoted 45 lines
-----Original Message-----
From: Jean-Philippe Brucker [mailto:[email protected]]
Sent: Wednesday, September 20, 2017 8:10 PM
To: Pan, Jacob jun <[email protected]>; iommu-cunTk1MwBs/[email protected] foundation.org
Cc: Liu, Yi L <[email protected]>; Raj, Ashok <[email protected]>; David Woodhouse <[email protected]>; Joerg Roedel <[email protected]>; Tian, Kevin <[email protected]>; Auger Eric <[email protected]>
Subject: Re: bind pasid table APIHi Jacob,
[Adding Eric as he might need
pasid_table_infofor vSVM at some point]Quoted 11 lines · Jacob Pan
Hi Jean and All,
This is a follow-up on the LPC discussion we had last week.
(https://linuxplumbersconf.org/2017/ocw/proposals/4748)My understanding is that the data structure below can satisfy the needs from Intel (pointer + size) and AMD (pointer only). But ARM pvIOMMU would need additional info to indicate the page table format. Could you share your idea of the right addition for ARM such that we can have a unified API?
Quoted 6 lines
/**
* PASID table data used to bind guest PASID table to the host IOMMU.
This will
* enable guest managed first level page tables.
* @ptr: PASID table pointer
* @size_order: number of bits supported in the guest PASID table, mustbe less
Quoted 6 lines
* or equal than the host table size.
*/struct pasid_table_info { __u64 ptr; __u64 size_order; };For the PASID table, Arm SMMUv3 would need two additional fields:
* 'format' telling whether the table has 1 or 2 levels and their dimensions,
* 'default_substream' telling if PASID0 is reserved for non-pasid traffic.Quoted 8 lines
I think that's it for the moment, but it does require to leave space for a vendor- specific structure at the end. It is one reason why I'd prefer having a 'model' field in the
pasid_table_infostructure telling what fields the whole structure actually contains.Another reason is if some IOMMU is able to support multiple PASID table formats, it could advertise them all in sysfs and Qemu could tell which one it chose in 'model'. I'm not sure we'll ever see that in practice.
Regards to your idea, I think the whole flow may be:
* Qemu queries underlying IOMMU for the capability(through sysfs)
* Combined with requirement from user(the guy who starts the VM), Qemu chose a suitable model or exit if HW capability is incompatible with user's requirement
* In the comingbind_pasid_table()calling, Qemu pass the chosen model info to host
* Host checks the "model" and use the correct model specific structure to parse the model specific data. may be kind of structureintel_pasid_table_info,amd_pasid_table_infoor arm_pasid_table_info_v#
Quoted 2 lines
Does this flow show what you want? This would be a "model" + "model specific data" proposal. And my concern is the model specific field may look like to be kind of "opaque".
It doesn't have to be opaque, it has to be defined in UAPI and an intermediate layer (VFIO) can verify that they are present. The layer can also inspect their value more deeply if it wants, since it is clear what they are supposed to contain. But (in VFIO's case) it doesn't have too, since the IOMMU driver will validate the values anyway.
Quoted 2 lines
Besides the comments above, is there also possibility for us to put all the possible info in a super-set just as what we plan to do for the
tlb_invalidate()API?
It's possible, but might require too much effort and might become more difficult to do once the number of IOMMU drivers supporting PASID increases. (See my other reply to Jacob)
Quoted 8 lines
Quoted 8 lines
For binding page tables instead of PASID tables (e.g. virtio-iommu), the generic data would be:
struct pgtable_info { __u32 pasid; __u64 ptr; __u32 model; __u8 model_data[];
This should be an union actually, since it's internal to the kernel. It's VFIO that uses this interface.
Quoted 4 lines
Quoted 1 line
};
Besides
bind_pasid_tableAPI, you would also want to propose an extra API which likely to be named asbind_pgtable()for this page table binding?
Yes, but we could use the same bind() function taking an enum {process, pgtable, pasid_table} and the parameter structure.
Quoted 2 lines
What would be the "model" field indicate? "vendor" or "vendor+version" ? You may want a length field to indicate the size of "
model_data" field.
I think it should be vendor+version.
Quoted 5 lines
And same with the
bind_pasid_tableAPI, wouldmodel_datalook like an "opaque"?Quoted 3 lines
Followed by a few arch-specific configuration values. For Arm we can summarize this to three registers, defined in the Armv8 Architecture Reference Manual:
Quoted 7 lines
Quoted 5 lines
struct
arm_lpae_pgtable_info{
__u64tcr; /* Translation Control Register */
__u64mair; /* Memory Attributes Indirection Register */
__u64asid; /* Address Space ID */
};Hmmm, just for curious. What is the difference between "pasid" and "asid"?
ASID is an ID used to tag TLB (but not ATC) entries in Arm systems. It's very similar to the "PCID" recently introduced in Linux for x86. In an ideal system we could use the same value for ASID and PASID, but I think it's safer to dissociate them for the moment.
Thanks,
Jean
On Mon, Sep 25, 2017 at 12:45:00PM +0100, Jean-Philippe Brucker wrote:
[snip]
Quoted 8 lines
This format tells how the guest organizes its PASID tables. Depending on 'format', the PASID table can be:
* A flat array of descriptors
* One array of 1st-level descriptors pointing to a 2nd level of descriptors. When translating an access, the SMMU splits the PASID at bit 6, uses the upper half to get the 1st level descriptor and the lower half to get the 2nd level one (4k leaf table).
* The same format, but instead split at PASID bit 10 (64k leaf table)
When in multilevel PASID table, do you shadow it in the host? or every level is all hosted in guest space? in Intel vt-d, when context is in a nested mode, PASID table is a gPA. Its a single level table.
Cheers,
Ashok
Quoted 12 lines · Ashok
On Mon, Sep 25, 2017 at 12:45:00PM +0100, Jean-Philippe Brucker wrote:
[snip]Quoted 8 lines
This format tells how the guest organizes its PASID tables. Depending on 'format', the PASID table can be:
* A flat array of descriptors
* One array of 1st-level descriptors pointing to a 2nd level of descriptors. When translating an access, the SMMU splits the PASID at bit 6, uses the upper half to get the 1st level descriptor and the lower half to get the 2nd level one (4k leaf table).
* The same format, but instead split at PASID bit 10 (64k leaf table)
Quoted 3 lines
When in multilevel PASID table, do you shadow it in the host? or every level is all hosted in guest space? in Intel vt-d, when context is in a nested mode, PASID table is a gPA. Its a single level table.
In the SMMU as well, both levels of PASID tables are accessed by gPA, so there's no need to shadow them.
Thanks,
Jean
Hi,
Quoted 9 lines · Jean-Philippe Brucker
For binding page tables instead of PASID tables (e.g. virtio-iommu), the generic data would be:
struct pgtable_info { __u32 pasid; __u64 ptr; __u32 model; __u8 model_data[]; };
I had a look again and at the AMD side there is no way to build a shadow pasid-table when more than 9-bit pasids are used, because all pointers in that multi-level table are GPA and thus translated.
So how about allowing the virtio-iommu to build pasid-tables in different formats? We can easily collect the existing code for that in the Intel and AMD drivers into a library that could be used by the virtio-iommu driver too. Then the virtio-iommu can create the correct table for the host-iommu and we don't have to rely on shadowing.
For binding whole pasid tables, reading through the thread, it looks like we can't get away with generic attributes that would fit everyone. For AMD and Intel it would suffice to have a base-ptr and the number of pasids-bits the table can map.
The Intel driver can then calculate the size of the table and the AMD driver can compute the GLX value;
In case we need the model-specific info, I'd like to have them explicitly stated in the struct:
enum pasid_table_model {
PASID_TABLE_INTEL,
PASID_TABLE_ARM,
PASID_TABLE_AMD,
/* ... */
};
struct pasid_table_config {
__u64 base_ptr;
__u8 pasid_bits;
union {
struct {
/* Intel specific fields */
} intel;
struct {
/* ARM specific fields */
} arm;
struct {
/* AMD specific fields */
} amd;
/* ... */
};
};How does that look?
Regards,
Joerg
Quoted 34 lines · On Wed, 27 Sep 2017 15:40:41 +0200 Joerg Roedel…
Hi,
Quoted 9 lines · Jean-Philippe Brucker
For binding page tables instead of PASID tables (e.g. virtio-iommu), the generic data would be:
struct pgtable_info { __u32 pasid; __u64 ptr; __u32 model; __u8 model_data[]; };I had a look again and at the AMD side there is no way to build a shadow pasid-table when more than 9-bit pasids are used, because all pointers in that multi-level table are GPA and thus translated.
So how about allowing the virtio-iommu to build pasid-tables in different formats? We can easily collect the existing code for that in the Intel and AMD drivers into a library that could be used by the virtio-iommu driver too. Then the virtio-iommu can create the correct table for the host-iommu and we don't have to rely on shadowing.
For binding whole pasid tables, reading through the thread, it looks like we can't get away with generic attributes that would fit everyone. For AMD and Intel it would suffice to have a base-ptr and the number of pasids-bits the table can map.
The Intel driver can then calculate the size of the table and the AMD driver can compute the GLX value;
In case we need the model-specific info, I'd like to have them explicitly stated in the struct:
Quoted 7 lines
enum pasid_table_model {
PASID_TABLE_INTEL,
PASID_TABLE_ARM,
PASID_TABLE_AMD,
/* ... */
};I guess one vendor could have multiple pasid table format. so perhaps the name could reflect the format as well?
Quoted 21 lines
struct pasid_table_config {
__u64 base_ptr;
__u8 pasid_bits;
union {
struct {
/* Intel specific fields */
} intel;
struct {
/* ARM specific fields */
} arm;
struct {
/* AMD specific fields */
} amd;
/* ... */
};
};Quoted 2 lines
How does that look?
It should work for us for now but I am not sure how stable the vendor specific fields will be, this is UAPI. BTW, do you also intend to include the pasid_table_model # in pasid_table_config?
Quoted 4 lines
Regards,
Joerg
[Jacob Pan]
Quoted 13 lines · Joerg Roedel
Hi,
Quoted 9 lines · Jean-Philippe Brucker
For binding page tables instead of PASID tables (e.g. virtio-iommu), the generic data would be:
struct pgtable_info { __u32 pasid; __u64 ptr; __u32 model; __u8 model_data[]; };
Quoted 3 lines
I had a look again and at the AMD side there is no way to build a shadow pasid-table when more than 9-bit pasids are used, because all pointers in that multi-level table are GPA and thus translated.
It's the same problem on SMMUv3, all pointers in the multi-level PASID tables are GPAs. Our solution was to grab chunks GPA space from the guest when necessary.
The host would maintain the PASID tables just like with native SVM. The difference is that they are stored in guest memory. In more details, it would go like this:
(1) host tells the guest the size of a memreserve (greatest size between all levels) and what pgtable format the host iommu supports.
(2) guest sends a memreserve request, giving up some of its GPA space for the host. There already are physical pages allocated for it since the GPA->PA mappings were created by the host with VFIO_MAP_DMA before booting the guest.
(3) guest sends a add_table request, asking to bind a pgd to a PASID
(4) the host installs the first-level PASID table. Instead of allocating it with kzalloc, it uses vmap on the existing memreserve pages.
(5) The host doesn't have any memreserve left for the second level so it returns an error code for add_table, asking the guest for more memreserve.
(5) guest sends another memreserve request
(6) guest sends the add_table request again, which succeeds
(7) host installs the second-level PASID table
(8) When the guest removes the page tables, it can ask to reclaim the memreserve
I know it's not particularly graceful, but it isn't too difficult to implement and there are a few reasons that make us prefer page table binding over PASID table binding for virtio-iommu:
* It also enables nested mode for IOMMUs that don't support PASIDs (but do support two translation stages), like the SMMUv2. They would require an add_table mechanism anyway.
* For a little more effort on the host side (handling memreserve pool), it removes a great deal of work in guests. We wouldn't have to factor all PASID code into a library in guests (currently battling with SMMUv3 support for PASID, and struggling to move a little bit of PASID code into the core, I'd prefer this to be last resort...)
The host driver already has code to manipulate PASID table entries in the right place, so hopefully it'd be less work doing it in the host.
* Even if we decide to bind PASID tables we'll still have to tell the guest what page table format it can use, since in SMMUv3 we have a lot of fields and knobs in the PASID descriptor itself, it's not just a PGD pointer :/
So the probe step is necessary. Supporting PASID tables adds an additional level of complexity to the device specification
So I'd like to try this solution first, since it seems less invasive, more flexible and a lot of the code would be necessary for both solutions. Both are difficult to specify and implement, though, so it's hard to say which is best for the moment. It shouldn't be too hard to fall back to PASID table binding if page table binding proves too cumbersome.
Thanks,
Jean
Quoted 11 lines · Jacob Pan
Quoted 7 lines · On Wed, 27 Sep 2017 15:40:41 +0200 Joerg Roedel…
enum pasid_table_model { PASID_TABLE_INTEL, PASID_TABLE_ARM, PASID_TABLE_AMD, /* ... */ };I guess one vendor could have multiple pasid table format. so perhaps the name could reflect the format as well?
Yes, that makes sense.
Quoted 21 lines
Quoted 21 lines
struct pasid_table_config { __u64 base_ptr; __u8 pasid_bits; union { struct { /* Intel specific fields */ } intel; struct { /* ARM specific fields */ } arm; struct { /* AMD specific fields */ } amd; /* ... */ }; };
Quoted 5 lines
Quoted 2 lines
How does that look?
It should work for us for now but I am not sure how stable the vendor specific fields will be, this is UAPI. BTW, do you also intend to include the
pasid_table_model# inpasid_table_config?
If its going to be UAPI then we probably also need a version and a length/size field. But since individual formats do not change, their sub-struct should be stable.
Regards,
Joerg
Hi Jean
Quoted 14 lines · Jean-Philippe Brucker
Quoted 13 lines · Joerg Roedel
Hi,
Quoted 9 lines · Jean-Philippe Brucker
For binding page tables instead of PASID tables (e.g. virtio-iommu), the generic data would be:
struct pgtable_info { __u32 pasid; __u64 ptr; __u32 model; __u8 model_data[]; };
Quoted 7 lines
Quoted 3 lines
I had a look again and at the AMD side there is no way to build a shadow pasid-table when more than 9-bit pasids are used, because all pointers in that multi-level table are GPA and thus translated.
It's the same problem on SMMUv3, all pointers in the multi-level PASID tables are GPAs. Our solution was to grab chunks GPA space from the guest when necessary.
If pasid table is gPA and built and managed by guest IOMMU driver in case of vIOMMU, or virtio-iommu. Is this complex interaction to reserve memory and setup required? wouldn't it be sufficient if we do this.
- Host advertises the capability in terms of what table format it it supports.
- guest does all the memory allocation and management.
- communicate to host via VFIO, gPA, format, size of table
- host now programs context entries appropriately with proper nested mode
Quoted 56 lines
The host would maintain the PASID tables just like with native SVM. The difference is that they are stored in guest memory. In more details, it would go like this:
(1) host tells the guest the size of a memreserve (greatest size between all levels) and what pgtable format the host iommu supports.
(2) guest sends a memreserve request, giving up some of its GPA space for the host. There already are physical pages allocated for it since theGPA->PAmappings were created by the host withVFIO_MAP_DMAbefore booting the guest.
(3) guest sends aadd_tablerequest, asking to bind a pgd to a PASID
(4) the host installs the first-level PASID table. Instead of allocating it with kzalloc, it uses vmap on the existing memreserve pages.
(5) The host doesn't have any memreserve left for the second level so it returns an error code foradd_table, asking the guest for more memreserve.
(5) guest sends another memreserve request
(6) guest sends theadd_tablerequest again, which succeeds
(7) host installs the second-level PASID table
(8) When the guest removes the page tables, it can ask to reclaim the memreserveI know it's not particularly graceful, but it isn't too difficult to implement and there are a few reasons that make us prefer page table binding over PASID table binding for virtio-iommu:
* It also enables nested mode for IOMMUs that don't support PASIDs (but do support two translation stages), like the SMMUv2. They would require an
add_tablemechanism anyway.* For a little more effort on the host side (handling memreserve pool), it removes a great deal of work in guests. We wouldn't have to factor all PASID code into a library in guests (currently battling with SMMUv3 support for PASID, and struggling to move a little bit of PASID code into the core, I'd prefer this to be last resort...)
The host driver already has code to manipulate PASID table entries in the right place, so hopefully it'd be less work doing it in the host.
* Even if we decide to bind PASID tables we'll still have to tell the guest what page table format it can use, since in SMMUv3 we have a lot of fields and knobs in the PASID descriptor itself, it's not just a PGD pointer :/
So the probe step is necessary. Supporting PASID tables adds an additional level of complexity to the device specification
So I'd like to try this solution first, since it seems less invasive, more flexible and a lot of the code would be necessary for both solutions. Both are difficult to specify and implement, though, so it's hard to say which is best for the moment. It shouldn't be too hard to fall back to PASID table binding if page table binding proves too cumbersome.
Thanks,
Jean
Quoted 38 lines · On Thu, 28 Sep 2017 14:07:05 +0200 Joerg Roedel…
Quoted 11 lines · Jacob Pan
Quoted 7 lines · On Wed, 27 Sep 2017 15:40:41 +0200 Joerg Roedel…
enum pasid_table_model { PASID_TABLE_INTEL, PASID_TABLE_ARM, PASID_TABLE_AMD, /* ... */ };I guess one vendor could have multiple pasid table format. so perhaps the name could reflect the format as well?
Yes, that makes sense.
Quoted 23 lines
Quoted 23 lines
struct pasid_table_config { __u64 base_ptr; __u8 pasid_bits; union { struct { /* Intel specific fields */ } intel; struct { /* ARM specific fields */ } arm; struct { /* AMD specific fields */ } amd; /* ... */ }; };How does that look?
Quoted 8 lines
Quoted 3 lines
It should work for us for now but I am not sure how stable the vendor specific fields will be, this is UAPI. BTW, do you also intend to include the
pasid_table_model# inpasid_table_config?If its going to be UAPI then we probably also need a version and a length/size field. But since individual formats do not change, their sub-struct should be stable.
just to confirm, we have two separate APIs for bind_pasid_table_ptr and bind_page_table. For the former, we need to passdown the following info from qemu/vfio. For now
struct pasid_table_config {
__u32 version;
#define PASID_TABLE_CONFIG_VERSION_1 (1)
__u32 bytes;
__u64 base_ptr;
__u8 pasid_bits;
enum pasid_table_model model;
union {
struct {
/* Intel specific fields */
} intel;
struct {
/* ARM specific fields */
} arm;
struct {
/* AMD specific fields */
} amd;
/* ... */
};
};Quoted 4 lines
Regards,
Joerg
[Jacob Pan]
Quoted 37 lines
From: Raj, Ashok
Sent: Friday, September 29, 2017 1:11 AMHi Jean
Quoted 4 lines · Jean-Philippe Brucker
Quoted 3 lines · Joerg Roedel
Hi,
On Wed, Sep 20, 2017 at 01:09:47PM +0100, Jean-Philippe Brucker
Quoted 11 lines · wrote:
Quoted 11 lines
Quoted 9 lines
For binding page tables instead of PASID tables (e.g. virtio-iommu), the generic data would be:
struct pgtable_info { __u32 pasid; __u64 ptr; __u32 model; __u8 model_data[]; };I had a look again and at the AMD side there is no way to build a
shadow
Quoted 5 lines
Quoted 2 lines
pasid-table when more than 9-bit pasids are used, because all pointers in that multi-level table are GPA and thus translated.
It's the same problem on SMMUv3, all pointers in the multi-level PASID tables are GPAs. Our solution was to grab chunks GPA space from the
guest
Quoted 1 line
when necessary.
If pasid table is gPA and built and managed by guest IOMMU driver in case of
vIOMMU, or virtio-iommu. Is this complex interaction to reserve memory and
setup required? wouldn't it be sufficient if we do this.
Quoted 6 lines
- Host advertises the capability in terms of what table format it it supports.
- guest does all the memory allocation and management.
- communicate to host via VFIO, gPA, format, size of table
- host now programs context entries appropriately with proper nested mode
I can see the motivation why Jean would like to go this way. As he explained below, it doesn't require factoring PASID codes (all formats on all vendors) into a library in guest. Guest just provides PGD and host IOMMU driver has all the logic to build PASID table in right form.
However I'm with Ashok here. I'm not sure whether host-side change is really less invasive. New memreserve APIs need to be introduced cross Qemu/VFIO/IOMMU driver. Also PASID logic in each host IOMMU driver needs change to cope with whether the table is built for a native usage or guest usage. In the latter case, memreserve must be considered which is not like a typical way of page table mgmt. (especially regarding to a multi-level PASID table).
Thinking about anyway we need to factor page table code in the guest side. It's not a big deal to further factor PASID code due to the same reason. Of course there are additional format info to be communicated back-and-forth, but the framework should be same for both page table and PASID purposes. Also such change is restricted within IOMMU sub-system. :-)
Thanks
Kevin
Quoted 19 lines · Jacob Pan
just to confirm, we have two separate APIs for
bind_pasid_table_ptrandbind_page_table. For the former, we need to passdown the following info from qemu/vfio. For nowstruct pasid_table_config { __u32 version; #define PASID_TABLE_CONFIG_VERSION_1 (1) __u32 bytes; __u64 base_ptr; __u8 pasid_bits; enum pasid_table_model model; union { struct { /* Intel specific fields */ } intel; struct { /* ARM specific fields */ } arm;
Quoted 7 lines
struct {
/* AMD specific fields */
} amd;
/* ... */
};
};Yes, this looks good to me.
Regards,
Joerg
Quoted 7 lines · Ashok
Hi Jean
Quoted 3 lines · Jean-Philippe Brucker
It's the same problem on SMMUv3, all pointers in the multi-level PASID tables are GPAs. Our solution was to grab chunks GPA space from the guest when necessary.
Quoted 8 lines
If pasid table is gPA and built and managed by guest IOMMU driver in case of vIOMMU, or virtio-iommu. Is this complex interaction to reserve memory and setup required? wouldn't it be sufficient if we do this.
- Host advertises the capability in terms of what table format it it supports.
- guest does all the memory allocation and management.
- communicate to host via VFIO, gPA, format, size of table
- host now programs context entries appropriately with proper nested mode
Yes, I think this is simpler than introducing a new memreserve API. The code to build the pasid-tables is mostly there already, it just needs some refactoring to make it usable for viommu.
The memreserve approach also implicates that the host can not trust the pasid tables it builds and need to re-verify them on every change because it is still accessible by the guest. We could write-protect the table or unmap it from the guest, but it would still be accessible by DMA. I think its tricky to get this right.
Regards,
Joerg
Quoted 6 lines · Kevin
Thinking about anyway we need to factor page table code in the guest side. It's not a big deal to further factor PASID code due to the same reason. Of course there are additional format info to be communicated back-and-forth, but the framework should be same for both page table and PASID purposes. Also such change is restricted within IOMMU sub-system. :-)
Right, and in the common case where the guest runs a distro-kernel which has the iommu-drivers compiled-in, the code to build the pasid-tables is also already available and vIOMMU just needs to use it. So this is the easiest approach.
Regards,
Joerg