Skip to contentSkip to conversation

bind pasid table API

DISCUSSION Jacob Pan 6 participants 18 messages Sep 19, 2017 – Sep 29, 2017 7 messages not in the archive

lore
Share & tools
lore

Jacob Pan

author
#0 lore raw

Hi Jean and All,

This is a follow-up on the LPC discussion we had last week.
(https://linuxplumbersconf.org/2017/ocw/proposals/4748)

My understanding is that the data structure below can satisfy the needs from Intel (pointer + size) and AMD (pointer only). But ARM pvIOMMU would need additional info to indicate the page table format. Could you share your idea of the right addition for ARM such that we can have a unified API?

/**
* PASID table data used to bind guest PASID table to the host IOMMU. This will
* enable guest managed first level page tables.
* @ptr: PASID table pointer
* @size_order: number of bits supported in the guest PASID table, must be less
* or equal than the host table size.
*/

struct pasid_table_info {
	__u64	ptr;
	__u64	size_order;
};

Thanks,

Jacob

#1 lore raw

Hi Jacob,

[Adding Eric as he might need pasid_table_info for vSVM at some point]

Quoted 18 lines · Jacob Pan

Hi Jean and All,

This is a follow-up on the LPC discussion we had last week.
(https://linuxplumbersconf.org/2017/ocw/proposals/4748)

My understanding is that the data structure below can satisfy the needs from Intel (pointer + size) and AMD (pointer only). But ARM pvIOMMU would need additional info to indicate the page table format. Could you share your idea of the right addition for ARM such that we can have a unified API?

/**
* PASID table data used to bind guest PASID table to the host IOMMU. This will
* enable guest managed first level page tables.
* @ptr: PASID table pointer
* @size_order: number of bits supported in the guest PASID table, must be less
* or equal than the host table size.
*/

Quoted 4 lines
struct pasid_table_info {
	__u64	ptr;
	__u64	size_order;
};

For the PASID table, Arm SMMUv3 would need two additional fields:
* 'format' telling whether the table has 1 or 2 levels and their dimensions,
* 'default_substream' telling if PASID0 is reserved for non-pasid traffic.

I think that's it for the moment, but it does require to leave space for a vendor-specific structure at the end. It is one reason why I'd prefer having a 'model' field in the pasid_table_info structure telling what fields the whole structure actually contains.

Another reason is if some IOMMU is able to support multiple PASID table formats, it could advertise them all in sysfs and Qemu could tell which one it chose in 'model'. I'm not sure we'll ever see that in practice.

For binding page tables instead of PASID tables (e.g. virtio-iommu), the generic data would be:

struct pgtable_info {
	__u32	pasid;
	__u64	ptr;
	__u32	model;
	__u8	model_data[];
};

Followed by a few arch-specific configuration values. For Arm we can summarize this to three registers, defined in the Armv8 Architecture Reference Manual:

struct arm_lpae_pgtable_info {
__u64 tcr; /* Translation Control Register */
__u64 mair; /* Memory Attributes Indirection Register */
__u64 asid; /* Address Space ID */
};

Some data packed in the TCR might be common to most architectures, like page granularity and max VA size. Most fields of the TCR won't be used but it provides a nice architected way to communicate Arm page table configuration.

Note that there might be an additional page directory in the arch-specific info, as we can split the address space in two. I'm not sure whether we should allow it yet.

Thanks,
Jean

author
#3 lore raw
In reply to a message not in the archive
Quoted 30 lines · On Wed, 20 Sep 2017 13:09:47 +0100 Jean-Philippe Brucker…

Hi Jacob,

[Adding Eric as he might need pasid_table_info for vSVM at some point]

Quoted 20 lines · Jacob Pan

Hi Jean and All,

This is a follow-up on the LPC discussion we had last week.
(https://linuxplumbersconf.org/2017/ocw/proposals/4748)

My understanding is that the data structure below can satisfy the needs from Intel (pointer + size) and AMD (pointer only). But ARM pvIOMMU would need additional info to indicate the page table format. Could you share your idea of the right addition for ARM such that we can have a unified API?

/**
* PASID table data used to bind guest PASID table to the host
IOMMU. This will
* enable guest managed first level page tables.
* @ptr: PASID table pointer
* @size_order: number of bits supported in the guest PASID
table, must be less
* or equal than the host table size.
*/

Quoted 4 lines
struct pasid_table_info {
	__u64	ptr;
	__u64	size_order;
};  
Quoted 3 lines

For the PASID table, Arm SMMUv3 would need two additional fields:
* 'format' telling whether the table has 1 or 2 levels and their dimensions,

just wondering if the format will be different than whatever is used on the host? i.e. could there be a mismatch? I am ok with this field just wondering if this can be resolved via the sysfs query interface if that is a global thing.

Quoted 3 lines

* 'default_substream' telling if PASID0 is reserved for non-pasid traffic.

could it be part of a feature flag? i.e.

struct pasid_table_info {
__u64 ptr;

 	__u64	size_order;
	__u64	flags;
#define PASID0_FOR_DMA_WITHOUT_PASID;
	enum pasid_table_format format;
};  
Quoted 5 lines

I think that's it for the moment, but it does require to leave space for a vendor-specific structure at the end. It is one reason why I'd prefer having a 'model' field in the pasid_table_info structure telling what fields the whole structure actually contains.

I think we have been there before. the downside is that model specific knowledge is required in the generic VFIO layer, if its content is to be inspected.

Quoted 5 lines

Another reason is if some IOMMU is able to support multiple PASID table formats, it could advertise them all in sysfs and Qemu could tell which one it chose in 'model'. I'm not sure we'll ever see that in practice.

I would expect when query interface between QEMU and sysfs would ensure matching format prior to issue bind_pasid_table call.

Quoted 22 lines

For binding page tables instead of PASID tables (e.g. virtio-iommu), the generic data would be:

struct pgtable_info {
	__u32	pasid;
	__u64	ptr;
	__u32	model;
	__u8	model_data[];
};

Followed by a few arch-specific configuration values. For Arm we can summarize this to three registers, defined in the Armv8 Architecture Reference Manual:

struct arm_lpae_pgtable_info {
__u64 tcr; /* Translation Control Register */
__u64 mair; /* Memory Attributes Indirection
Register */ __u64 asid; /* Address Space ID */
};

Quoted 9 lines

Some data packed in the TCR might be common to most architectures, like page granularity and max VA size. Most fields of the TCR won't be used but it provides a nice architected way to communicate Arm page table configuration.

Note that there might be an additional page directory in the arch-specific info, as we can split the address space in two. I'm not sure whether we should allow it yet.

This can be combined with bind mm with user tasks also (e.g. DPDK with SVM in native case), right? In that case the pasid would be allocated by the kernel instead of passdown in pgtable_info, and your 'ptr' filed is harvested from the current task mm? There is also complexity w.r.t. user task life cycle management. My immediate goal to enable vSVM does not require bind page tables, try not to think too far ahead.

Quoted 2 lines

Thanks,
Jean

[Jacob Pan]

#8 lore raw
In reply to a message not in the archive

Hi Jean,

Quoted 45 lines

-----Original Message-----
From: Jean-Philippe Brucker [mailto:[email protected]]
Sent: Wednesday, September 20, 2017 8:10 PM
To: Pan, Jacob jun <[email protected]>; iommu-cunTk1MwBs/[email protected] foundation.org
Cc: Liu, Yi L <[email protected]>; Raj, Ashok <[email protected]>; David Woodhouse <[email protected]>; Joerg Roedel <[email protected]>; Tian, Kevin <[email protected]>; Auger Eric <[email protected]>
Subject: Re: bind pasid table API

Hi Jacob,

[Adding Eric as he might need pasid_table_info for vSVM at some point]

Quoted 11 lines · Jacob Pan

Hi Jean and All,

This is a follow-up on the LPC discussion we had last week.
(https://linuxplumbersconf.org/2017/ocw/proposals/4748)

My understanding is that the data structure below can satisfy the needs from Intel (pointer + size) and AMD (pointer only). But ARM pvIOMMU would need additional info to indicate the page table format. Could you share your idea of the right addition for ARM such that we can have a unified API?

Quoted 6 lines

/**
* PASID table data used to bind guest PASID table to the host IOMMU.
This will
* enable guest managed first level page tables.
* @ptr: PASID table pointer
* @size_order: number of bits supported in the guest PASID table, must

be less

Quoted 6 lines

* or equal than the host table size.
*/

struct pasid_table_info {
	__u64	ptr;
	__u64	size_order;
};

For the PASID table, Arm SMMUv3 would need two additional fields:
* 'format' telling whether the table has 1 or 2 levels and their dimensions,
* 'default_substream' telling if PASID0 is reserved for non-pasid traffic.

Quoted 8 lines

I think that's it for the moment, but it does require to leave space for a vendor- specific structure at the end. It is one reason why I'd prefer having a 'model' field in the pasid_table_info structure telling what fields the whole structure actually contains.

Another reason is if some IOMMU is able to support multiple PASID table formats, it could advertise them all in sysfs and Qemu could tell which one it chose in 'model'. I'm not sure we'll ever see that in practice.

Regards to your idea, I think the whole flow may be:
* Qemu queries underlying IOMMU for the capability(through sysfs)
* Combined with requirement from user(the guy who starts the VM), Qemu chose a suitable model or exit if HW capability is incompatible with user's requirement
* In the coming bind_pasid_table() calling, Qemu pass the chosen model info to host
* Host checks the "model" and use the correct model specific structure to parse the model specific data. may be kind of structure intel_pasid_table_info, amd_pasid_table_info or arm_pasid_table_info_v#

Does this flow show what you want? This would be a "model" + "model specific data" proposal. And my concern is the model specific field may look like to be kind of "opaque".

Besides the comments above, is there also possibility for us to put all the possible info in a super-set just as what we plan to do for the tlb_invalidate() API?

Quoted 10 lines

For binding page tables instead of PASID tables (e.g. virtio-iommu), the generic data would be:

struct pgtable_info {
	__u32	pasid;
	__u64	ptr;
	__u32	model;
	__u8	model_data[];
};

Besides bind_pasid_table API, you would also want to propose an extra API which likely to be named as bind_pgtable() for this page table binding?

What would be the "model" field indicate? "vendor" or "vendor+version" ? You may want a length field to indicate the size of "model_data" field.

And same with the bind_pasid_table API, would model_data look like an "opaque"?

Quoted 8 lines

Followed by a few arch-specific configuration values. For Arm we can summarize this to three registers, defined in the Armv8 Architecture Reference Manual:

struct arm_lpae_pgtable_info {
__u64 tcr; /* Translation Control Register */
__u64 mair; /* Memory Attributes Indirection Register */
__u64 asid; /* Address Space ID */
};

Hmmm, just for curious. What is the difference between "pasid" and "asid"?

Thanks,
Yi L

Quoted 10 lines

Some data packed in the TCR might be common to most architectures, like page granularity and max VA size. Most fields of the TCR won't be used but it provides a nice architected way to communicate Arm page table configuration.

Note that there might be an additional page directory in the arch-specific info, as we can split the address space in two. I'm not sure whether we should allow it yet.

Thanks,
Jean

#4 lore raw
In reply to Jacob Pan
Quoted 33 lines · Jacob Pan
Quoted 30 lines · On Wed, 20 Sep 2017 13:09:47 +0100 Jean-Philippe Brucker…

Hi Jacob,

[Adding Eric as he might need pasid_table_info for vSVM at some point]

Quoted 24 lines · Jacob Pan

Hi Jean and All,

This is a follow-up on the LPC discussion we had last week.
(https://linuxplumbersconf.org/2017/ocw/proposals/4748)

My understanding is that the data structure below can satisfy the needs from Intel (pointer + size) and AMD (pointer only). But ARM pvIOMMU would need additional info to indicate the page table format. Could you share your idea of the right addition for ARM such that we can have a unified API?

/**
* PASID table data used to bind guest PASID table to the host
IOMMU. This will
* enable guest managed first level page tables.
* @ptr: PASID table pointer
* @size_order: number of bits supported in the guest PASID
table, must be less
* or equal than the host table size.
*/

struct pasid_table_info {
	__u64	ptr;
	__u64	size_order;
};  
Quoted 7 lines
Quoted 3 lines

For the PASID table, Arm SMMUv3 would need two additional fields:
* 'format' telling whether the table has 1 or 2 levels and their dimensions,

just wondering if the format will be different than whatever is used on the host? i.e. could there be a mismatch? I am ok with this field just wondering if this can be resolved via the sysfs query interface if that is a global thing.

This format tells how the guest organizes its PASID tables. Depending on 'format', the PASID table can be:
* A flat array of descriptors
* One array of 1st-level descriptors pointing to a 2nd level of descriptors. When translating an access, the SMMU splits the PASID at bit 6, uses the upper half to get the 1st level descriptor and the lower half to get the 2nd level one (4k leaf table).
* The same format, but instead split at PASID bit 10 (64k leaf table)

So the guest has to tell the host what format of PASID tables it chose, but the configuration might be too vendor-specific to go in the generic part of pasid_table_info.

It seems that AMD IOMMUv2 can have either one, two or three levels of PASID tables. See definition of GLX field in the 3.2.2.1 Device Table Entry Format.

We could try to come up with a generic definition to cover all these cases, but it might be more complicated than appending a few vendor-specific fields. Only AMD can have three levels, only ARM can split the PASID in two different places. And we can expect a future architecture to come up with a new colorful format...

I'm mostly concerned about VFIO though, as the IOMMU API doesn't have the same backward compatible restriction and is therefore easier to update.

Quoted 13 lines
Quoted 3 lines

* 'default_substream' telling if PASID0 is reserved for non-pasid traffic.

could it be part of a feature flag? i.e.

struct pasid_table_info {

(quoted diff) +0 -0
 	__u64	ptr;
 	__u64	size_order;
	__u64	flags;
#define PASID0_FOR_DMA_WITHOUT_PASID;
	enum pasid_table_format format;
};  
Quoted 8 lines
Quoted 5 lines

I think that's it for the moment, but it does require to leave space for a vendor-specific structure at the end. It is one reason why I'd prefer having a 'model' field in the pasid_table_info structure telling what fields the whole structure actually contains.

I think we have been there before. the downside is that model specific knowledge is required in the generic VFIO layer, if its content is to be inspected.

Yes, all of these vendor-specific fields would have to be UAPI. VFIO doesn't necessarily have to deeply inspect each field, it can just check that the structure contains the right fields (is the right size) and let the IOMMU driver, that knows how to read these values, validate that they make sense.

I think VFIO should be able to trust the IOMMU driver to perform the sanity checks.

Quoted 38 lines
Quoted 5 lines

Another reason is if some IOMMU is able to support multiple PASID table formats, it could advertise them all in sysfs and Qemu could tell which one it chose in 'model'. I'm not sure we'll ever see that in practice.

I would expect when query interface between QEMU and sysfs would ensure matching format prior to issue bind_pasid_table call.

Quoted 22 lines

For binding page tables instead of PASID tables (e.g. virtio-iommu), the generic data would be:

struct pgtable_info {
	__u32	pasid;
	__u64	ptr;
	__u32	model;
	__u8	model_data[];
};

Followed by a few arch-specific configuration values. For Arm we can summarize this to three registers, defined in the Armv8 Architecture Reference Manual:

struct arm_lpae_pgtable_info {
__u64 tcr; /* Translation Control Register */
__u64 mair; /* Memory Attributes Indirection
Register */ __u64 asid; /* Address Space ID */
};

Quoted 9 lines

Some data packed in the TCR might be common to most architectures, like page granularity and max VA size. Most fields of the TCR won't be used but it provides a nice architected way to communicate Arm page table configuration.

Note that there might be an additional page directory in the arch-specific info, as we can split the address space in two. I'm not sure whether we should allow it yet.

Quoted 6 lines

This can be combined with bind mm with user tasks also (e.g. DPDK with SVM in native case), right? In that case the pasid would be allocated by the kernel instead of passdown in pgtable_info, and your 'ptr' filed is harvested from the current task mm? There is also complexity w.r.t. user task life cycle management. My immediate goal to enable vSVM does not require bind page tables, try not to think too far ahead.

Yes, I'm working on the native case at the moment. The PASID is allocated by the kernel and returned in 'pasid', and user can specify an additional 'pid' field to bind another process. There would be a 'flag' field telling if 'pid' is valid or if we should just bind the current process.

Thanks,
Jean

#10 lore raw
In reply to a message not in the archive
Quoted 65 lines · Yi L

Hi Jean,

Quoted 45 lines

-----Original Message-----
From: Jean-Philippe Brucker [mailto:[email protected]]
Sent: Wednesday, September 20, 2017 8:10 PM
To: Pan, Jacob jun <[email protected]>; iommu-cunTk1MwBs/[email protected] foundation.org
Cc: Liu, Yi L <[email protected]>; Raj, Ashok <[email protected]>; David Woodhouse <[email protected]>; Joerg Roedel <[email protected]>; Tian, Kevin <[email protected]>; Auger Eric <[email protected]>
Subject: Re: bind pasid table API

Hi Jacob,

[Adding Eric as he might need pasid_table_info for vSVM at some point]

Quoted 11 lines · Jacob Pan

Hi Jean and All,

This is a follow-up on the LPC discussion we had last week.
(https://linuxplumbersconf.org/2017/ocw/proposals/4748)

My understanding is that the data structure below can satisfy the needs from Intel (pointer + size) and AMD (pointer only). But ARM pvIOMMU would need additional info to indicate the page table format. Could you share your idea of the right addition for ARM such that we can have a unified API?

Quoted 6 lines

/**
* PASID table data used to bind guest PASID table to the host IOMMU.
This will
* enable guest managed first level page tables.
* @ptr: PASID table pointer
* @size_order: number of bits supported in the guest PASID table, must

be less

Quoted 6 lines

* or equal than the host table size.
*/

struct pasid_table_info {
	__u64	ptr;
	__u64	size_order;
};

For the PASID table, Arm SMMUv3 would need two additional fields:
* 'format' telling whether the table has 1 or 2 levels and their dimensions,
* 'default_substream' telling if PASID0 is reserved for non-pasid traffic.

Quoted 8 lines

I think that's it for the moment, but it does require to leave space for a vendor- specific structure at the end. It is one reason why I'd prefer having a 'model' field in the pasid_table_info structure telling what fields the whole structure actually contains.

Another reason is if some IOMMU is able to support multiple PASID table formats, it could advertise them all in sysfs and Qemu could tell which one it chose in 'model'. I'm not sure we'll ever see that in practice.

Regards to your idea, I think the whole flow may be:
* Qemu queries underlying IOMMU for the capability(through sysfs)
* Combined with requirement from user(the guy who starts the VM), Qemu chose a suitable model or exit if HW capability is incompatible with user's requirement
* In the coming bind_pasid_table() calling, Qemu pass the chosen model info to host
* Host checks the "model" and use the correct model specific structure to parse the model specific data. may be kind of structure intel_pasid_table_info, amd_pasid_table_info or arm_pasid_table_info_v#

Quoted 2 lines

Does this flow show what you want? This would be a "model" + "model specific data" proposal. And my concern is the model specific field may look like to be kind of "opaque".

It doesn't have to be opaque, it has to be defined in UAPI and an intermediate layer (VFIO) can verify that they are present. The layer can also inspect their value more deeply if it wants, since it is clear what they are supposed to contain. But (in VFIO's case) it doesn't have too, since the IOMMU driver will validate the values anyway.

Quoted 2 lines

Besides the comments above, is there also possibility for us to put all the possible info in a super-set just as what we plan to do for the tlb_invalidate() API?

It's possible, but might require too much effort and might become more difficult to do once the number of IOMMU drivers supporting PASID increases. (See my other reply to Jacob)

Quoted 8 lines
Quoted 8 lines

For binding page tables instead of PASID tables (e.g. virtio-iommu), the generic data would be:

struct pgtable_info {
	__u32	pasid;
	__u64	ptr;
	__u32	model;
	__u8	model_data[];

This should be an union actually, since it's internal to the kernel. It's VFIO that uses this interface.

Quoted 4 lines
Quoted 1 line

};

Besides bind_pasid_table API, you would also want to propose an extra API which likely to be named as bind_pgtable() for this page table binding?

Yes, but we could use the same bind() function taking an enum {process, pgtable, pasid_table} and the parameter structure.

Quoted 2 lines

What would be the "model" field indicate? "vendor" or "vendor+version" ? You may want a length field to indicate the size of "model_data" field.

I think it should be vendor+version.

Quoted 5 lines

And same with the bind_pasid_table API, would model_data look like an "opaque"?

Quoted 3 lines

Followed by a few arch-specific configuration values. For Arm we can summarize this to three registers, defined in the Armv8 Architecture Reference Manual:

Quoted 7 lines
Quoted 5 lines

struct arm_lpae_pgtable_info {
__u64 tcr; /* Translation Control Register */
__u64 mair; /* Memory Attributes Indirection Register */
__u64 asid; /* Address Space ID */
};

Hmmm, just for curious. What is the difference between "pasid" and "asid"?

ASID is an ID used to tag TLB (but not ATC) entries in Arm systems. It's very similar to the "PCID" recently introduced in Linux for x86. In an ideal system we could use the same value for ASID and PASID, but I think it's safer to dissociate them for the moment.

Thanks,
Jean

#6 lore raw
In reply to a message not in the archive

On Mon, Sep 25, 2017 at 12:45:00PM +0100, Jean-Philippe Brucker wrote:
[snip]

Quoted 8 lines

This format tells how the guest organizes its PASID tables. Depending on 'format', the PASID table can be:
* A flat array of descriptors
* One array of 1st-level descriptors pointing to a 2nd level of descriptors. When translating an access, the SMMU splits the PASID at bit 6, uses the upper half to get the 1st level descriptor and the lower half to get the 2nd level one (4k leaf table).
* The same format, but instead split at PASID bit 10 (64k leaf table)

When in multilevel PASID table, do you shadow it in the host? or every level is all hosted in guest space? in Intel vt-d, when context is in a nested mode, PASID table is a gPA. Its a single level table.

Cheers,
Ashok

#7 lore raw
Quoted 12 lines · Ashok

On Mon, Sep 25, 2017 at 12:45:00PM +0100, Jean-Philippe Brucker wrote:
[snip]

Quoted 8 lines

This format tells how the guest organizes its PASID tables. Depending on 'format', the PASID table can be:
* A flat array of descriptors
* One array of 1st-level descriptors pointing to a 2nd level of descriptors. When translating an access, the SMMU splits the PASID at bit 6, uses the upper half to get the 1st level descriptor and the lower half to get the 2nd level one (4k leaf table).
* The same format, but instead split at PASID bit 10 (64k leaf table)

Quoted 3 lines

When in multilevel PASID table, do you shadow it in the host? or every level is all hosted in guest space? in Intel vt-d, when context is in a nested mode, PASID table is a gPA. Its a single level table.

In the SMMU as well, both levels of PASID tables are accessed by gPA, so there's no need to shadow them.

Thanks,
Jean

#11 lore raw
In reply to a message not in the archive

Hi,

Quoted 9 lines · Jean-Philippe Brucker

For binding page tables instead of PASID tables (e.g. virtio-iommu), the generic data would be:

struct pgtable_info {
	__u32	pasid;
	__u64	ptr;
	__u32	model;
	__u8	model_data[];
};

I had a look again and at the AMD side there is no way to build a shadow pasid-table when more than 9-bit pasids are used, because all pointers in that multi-level table are GPA and thus translated.

So how about allowing the virtio-iommu to build pasid-tables in different formats? We can easily collect the existing code for that in the Intel and AMD drivers into a library that could be used by the virtio-iommu driver too. Then the virtio-iommu can create the correct table for the host-iommu and we don't have to rely on shadowing.

For binding whole pasid tables, reading through the thread, it looks like we can't get away with generic attributes that would fit everyone. For AMD and Intel it would suffice to have a base-ptr and the number of pasids-bits the table can map.

The Intel driver can then calculate the size of the table and the AMD driver can compute the GLX value;

In case we need the model-specific info, I'd like to have them explicitly stated in the struct:

	enum pasid_table_model {
		PASID_TABLE_INTEL,
		PASID_TABLE_ARM,
		PASID_TABLE_AMD,
		/* ... */
	};

	struct pasid_table_config {
		__u64 base_ptr;
		__u8 pasid_bits;
		union {
			struct {
				/* Intel specific fields */
			} intel;

			struct {
				/* ARM specific fields */
			} arm;


			struct {
				/* AMD specific fields */
			} amd;

			/* ... */
		};
	};

How does that look?

Regards,

Joerg

author
#13 lore raw
In reply to a message not in the archive
Quoted 34 lines · On Wed, 27 Sep 2017 15:40:41 +0200 Joerg Roedel…

Hi,

Quoted 9 lines · Jean-Philippe Brucker

For binding page tables instead of PASID tables (e.g. virtio-iommu), the generic data would be:

struct pgtable_info {
	__u32	pasid;
	__u64	ptr;
	__u32	model;
	__u8	model_data[];
};  

I had a look again and at the AMD side there is no way to build a shadow pasid-table when more than 9-bit pasids are used, because all pointers in that multi-level table are GPA and thus translated.

So how about allowing the virtio-iommu to build pasid-tables in different formats? We can easily collect the existing code for that in the Intel and AMD drivers into a library that could be used by the virtio-iommu driver too. Then the virtio-iommu can create the correct table for the host-iommu and we don't have to rely on shadowing.

For binding whole pasid tables, reading through the thread, it looks like we can't get away with generic attributes that would fit everyone. For AMD and Intel it would suffice to have a base-ptr and the number of pasids-bits the table can map.

The Intel driver can then calculate the size of the table and the AMD driver can compute the GLX value;

In case we need the model-specific info, I'd like to have them explicitly stated in the struct:

Quoted 7 lines
	enum pasid_table_model {
		PASID_TABLE_INTEL,
		PASID_TABLE_ARM,
		PASID_TABLE_AMD,
		/* ... */
	};

I guess one vendor could have multiple pasid table format. so perhaps the name could reflect the format as well?

Quoted 21 lines
	struct pasid_table_config {
		__u64 base_ptr;
		__u8 pasid_bits;
		union {
			struct {
				/* Intel specific fields */
			} intel;

			struct {
				/* ARM specific fields */
			} arm;


			struct {
				/* AMD specific fields */
			} amd;

			/* ... */
		};
	};
Quoted 2 lines

How does that look?

It should work for us for now but I am not sure how stable the vendor specific fields will be, this is UAPI. BTW, do you also intend to include the pasid_table_model # in pasid_table_config?

Quoted 4 lines

Regards,

Joerg

[Jacob Pan]

#18 lore raw
In reply to a message not in the archive
Quoted 13 lines · Joerg Roedel

Hi,

Quoted 9 lines · Jean-Philippe Brucker

For binding page tables instead of PASID tables (e.g. virtio-iommu), the generic data would be:

struct pgtable_info {
	__u32	pasid;
	__u64	ptr;
	__u32	model;
	__u8	model_data[];
};
Quoted 3 lines

I had a look again and at the AMD side there is no way to build a shadow pasid-table when more than 9-bit pasids are used, because all pointers in that multi-level table are GPA and thus translated.

It's the same problem on SMMUv3, all pointers in the multi-level PASID tables are GPAs. Our solution was to grab chunks GPA space from the guest when necessary.

The host would maintain the PASID tables just like with native SVM. The difference is that they are stored in guest memory. In more details, it would go like this:

(1) host tells the guest the size of a memreserve (greatest size between all levels) and what pgtable format the host iommu supports.
(2) guest sends a memreserve request, giving up some of its GPA space for the host. There already are physical pages allocated for it since the GPA->PA mappings were created by the host with VFIO_MAP_DMA before booting the guest.
(3) guest sends a add_table request, asking to bind a pgd to a PASID
(4) the host installs the first-level PASID table. Instead of allocating it with kzalloc, it uses vmap on the existing memreserve pages.
(5) The host doesn't have any memreserve left for the second level so it returns an error code for add_table, asking the guest for more memreserve.
(5) guest sends another memreserve request
(6) guest sends the add_table request again, which succeeds
(7) host installs the second-level PASID table
(8) When the guest removes the page tables, it can ask to reclaim the memreserve

I know it's not particularly graceful, but it isn't too difficult to implement and there are a few reasons that make us prefer page table binding over PASID table binding for virtio-iommu:

* It also enables nested mode for IOMMUs that don't support PASIDs (but do support two translation stages), like the SMMUv2. They would require an add_table mechanism anyway.

* For a little more effort on the host side (handling memreserve pool), it removes a great deal of work in guests. We wouldn't have to factor all PASID code into a library in guests (currently battling with SMMUv3 support for PASID, and struggling to move a little bit of PASID code into the core, I'd prefer this to be last resort...)

The host driver already has code to manipulate PASID table entries in the right place, so hopefully it'd be less work doing it in the host.

* Even if we decide to bind PASID tables we'll still have to tell the guest what page table format it can use, since in SMMUv3 we have a lot of fields and knobs in the PASID descriptor itself, it's not just a PGD pointer :/

So the probe step is necessary. Supporting PASID tables adds an additional level of complexity to the device specification

So I'd like to try this solution first, since it seems less invasive, more flexible and a lot of the code would be necessary for both solutions. Both are difficult to specify and implement, though, so it's hard to say which is best for the moment. It shouldn't be too hard to fall back to PASID table binding if page table binding proves too cumbersome.

Thanks,
Jean

#14 lore raw
In reply to Jacob Pan
Quoted 11 lines · Jacob Pan
Quoted 7 lines · On Wed, 27 Sep 2017 15:40:41 +0200 Joerg Roedel…
	enum pasid_table_model {
		PASID_TABLE_INTEL,
		PASID_TABLE_ARM,
		PASID_TABLE_AMD,
		/* ... */
	};

I guess one vendor could have multiple pasid table format. so perhaps the name could reflect the format as well?

Yes, that makes sense.

Quoted 21 lines
Quoted 21 lines
	struct pasid_table_config {
		__u64 base_ptr;
		__u8 pasid_bits;
		union {
			struct {
				/* Intel specific fields */
			} intel;

			struct {
				/* ARM specific fields */
			} arm;


			struct {
				/* AMD specific fields */
			} amd;

			/* ... */
		};
	};
Quoted 5 lines
Quoted 2 lines

How does that look?

It should work for us for now but I am not sure how stable the vendor specific fields will be, this is UAPI. BTW, do you also intend to include the pasid_table_model # in pasid_table_config?

If its going to be UAPI then we probably also need a version and a length/size field. But since individual formats do not change, their sub-struct should be stable.

Regards,

Joerg

#20 lore raw
In reply to a message not in the archive

Hi Jean

Quoted 14 lines · Jean-Philippe Brucker
Quoted 13 lines · Joerg Roedel

Hi,

Quoted 9 lines · Jean-Philippe Brucker

For binding page tables instead of PASID tables (e.g. virtio-iommu), the generic data would be:

struct pgtable_info {
	__u32	pasid;
	__u64	ptr;
	__u32	model;
	__u8	model_data[];
};
Quoted 7 lines
Quoted 3 lines

I had a look again and at the AMD side there is no way to build a shadow pasid-table when more than 9-bit pasids are used, because all pointers in that multi-level table are GPA and thus translated.

It's the same problem on SMMUv3, all pointers in the multi-level PASID tables are GPAs. Our solution was to grab chunks GPA space from the guest when necessary.

If pasid table is gPA and built and managed by guest IOMMU driver in case of vIOMMU, or virtio-iommu. Is this complex interaction to reserve memory and setup required? wouldn't it be sufficient if we do this.

- Host advertises the capability in terms of what table format it it supports.
- guest does all the memory allocation and management.
- communicate to host via VFIO, gPA, format, size of table
- host now programs context entries appropriately with proper nested mode

Quoted 56 lines

The host would maintain the PASID tables just like with native SVM. The difference is that they are stored in guest memory. In more details, it would go like this:

(1) host tells the guest the size of a memreserve (greatest size between all levels) and what pgtable format the host iommu supports.
(2) guest sends a memreserve request, giving up some of its GPA space for the host. There already are physical pages allocated for it since the GPA->PA mappings were created by the host with VFIO_MAP_DMA before booting the guest.
(3) guest sends a add_table request, asking to bind a pgd to a PASID
(4) the host installs the first-level PASID table. Instead of allocating it with kzalloc, it uses vmap on the existing memreserve pages.
(5) The host doesn't have any memreserve left for the second level so it returns an error code for add_table, asking the guest for more memreserve.
(5) guest sends another memreserve request
(6) guest sends the add_table request again, which succeeds
(7) host installs the second-level PASID table
(8) When the guest removes the page tables, it can ask to reclaim the memreserve

I know it's not particularly graceful, but it isn't too difficult to implement and there are a few reasons that make us prefer page table binding over PASID table binding for virtio-iommu:

* It also enables nested mode for IOMMUs that don't support PASIDs (but do support two translation stages), like the SMMUv2. They would require an add_table mechanism anyway.

* For a little more effort on the host side (handling memreserve pool), it removes a great deal of work in guests. We wouldn't have to factor all PASID code into a library in guests (currently battling with SMMUv3 support for PASID, and struggling to move a little bit of PASID code into the core, I'd prefer this to be last resort...)

The host driver already has code to manipulate PASID table entries in the right place, so hopefully it'd be less work doing it in the host.

* Even if we decide to bind PASID tables we'll still have to tell the guest what page table format it can use, since in SMMUv3 we have a lot of fields and knobs in the PASID descriptor itself, it's not just a PGD pointer :/

So the probe step is necessary. Supporting PASID tables adds an additional level of complexity to the device specification

So I'd like to try this solution first, since it seems less invasive, more flexible and a lot of the code would be necessary for both solutions. Both are difficult to specify and implement, though, so it's hard to say which is best for the moment. It shouldn't be too hard to fall back to PASID table binding if page table binding proves too cumbersome.

Thanks,
Jean

author
#16 lore raw
In reply to a message not in the archive
Quoted 38 lines · On Thu, 28 Sep 2017 14:07:05 +0200 Joerg Roedel…
Quoted 11 lines · Jacob Pan
Quoted 7 lines · On Wed, 27 Sep 2017 15:40:41 +0200 Joerg Roedel…
	enum pasid_table_model {
		PASID_TABLE_INTEL,
		PASID_TABLE_ARM,
		PASID_TABLE_AMD,
		/* ... */
	};

I guess one vendor could have multiple pasid table format. so perhaps the name could reflect the format as well?

Yes, that makes sense.

Quoted 23 lines
Quoted 23 lines
	struct pasid_table_config {
		__u64 base_ptr;
		__u8 pasid_bits;
		union {
			struct {
				/* Intel specific fields */
			} intel;

			struct {
				/* ARM specific fields */
			} arm;


			struct {
				/* AMD specific fields */
			} amd;

			/* ... */
		};
	};

How does that look?

Quoted 8 lines
Quoted 3 lines

It should work for us for now but I am not sure how stable the vendor specific fields will be, this is UAPI. BTW, do you also intend to include the pasid_table_model # in pasid_table_config?

If its going to be UAPI then we probably also need a version and a length/size field. But since individual formats do not change, their sub-struct should be stable.

just to confirm, we have two separate APIs for bind_pasid_table_ptr and bind_page_table. For the former, we need to passdown the following info from qemu/vfio. For now

struct pasid_table_config {
	__u32 version;
#define PASID_TABLE_CONFIG_VERSION_1 (1)
	__u32 bytes;
	__u64 base_ptr;
	__u8 pasid_bits;
	enum pasid_table_model model;
	union {
		struct {
			/* Intel specific fields */
		} intel;

		struct {
			/* ARM specific fields */
		} arm;

		struct {
			/* AMD specific fields */
		} amd;

		/* ... */
	};
};
Quoted 4 lines

Regards,

Joerg

[Jacob Pan]

#21 lore raw
In reply to Raj, Ashok
Quoted 37 lines

From: Raj, Ashok
Sent: Friday, September 29, 2017 1:11 AM

Hi Jean

Quoted 4 lines · Jean-Philippe Brucker
Quoted 3 lines · Joerg Roedel

Hi,

On Wed, Sep 20, 2017 at 01:09:47PM +0100, Jean-Philippe Brucker

Quoted 11 lines · wrote:
Quoted 11 lines
Quoted 9 lines

For binding page tables instead of PASID tables (e.g. virtio-iommu), the generic data would be:

struct pgtable_info {
	__u32	pasid;
	__u64	ptr;
	__u32	model;
	__u8	model_data[];
};

I had a look again and at the AMD side there is no way to build a

shadow

Quoted 5 lines
Quoted 2 lines

pasid-table when more than 9-bit pasids are used, because all pointers in that multi-level table are GPA and thus translated.

It's the same problem on SMMUv3, all pointers in the multi-level PASID tables are GPAs. Our solution was to grab chunks GPA space from the

guest

Quoted 1 line

when necessary.

If pasid table is gPA and built and managed by guest IOMMU driver in case of
vIOMMU, or virtio-iommu. Is this complex interaction to reserve memory and
setup required? wouldn't it be sufficient if we do this.

Quoted 6 lines

- Host advertises the capability in terms of what table format it it supports.
- guest does all the memory allocation and management.
- communicate to host via VFIO, gPA, format, size of table
- host now programs context entries appropriately with proper nested mode

I can see the motivation why Jean would like to go this way. As he explained below, it doesn't require factoring PASID codes (all formats on all vendors) into a library in guest. Guest just provides PGD and host IOMMU driver has all the logic to build PASID table in right form.

However I'm with Ashok here. I'm not sure whether host-side change is really less invasive. New memreserve APIs need to be introduced cross Qemu/VFIO/IOMMU driver. Also PASID logic in each host IOMMU driver needs change to cope with whether the table is built for a native usage or guest usage. In the latter case, memreserve must be considered which is not like a typical way of page table mgmt. (especially regarding to a multi-level PASID table).

Thinking about anyway we need to factor page table code in the guest side. It's not a big deal to further factor PASID code due to the same reason. Of course there are additional format info to be communicated back-and-forth, but the framework should be same for both page table and PASID purposes. Also such change is restricted within IOMMU sub-system. :-)

Thanks
Kevin

#17 lore raw
In reply to Jacob Pan
Quoted 19 lines · Jacob Pan

just to confirm, we have two separate APIs for bind_pasid_table_ptr and bind_page_table. For the former, we need to passdown the following info from qemu/vfio. For now

struct pasid_table_config {
	__u32 version;
#define PASID_TABLE_CONFIG_VERSION_1 (1)
	__u32 bytes;
	__u64 base_ptr;
	__u8 pasid_bits;
	enum pasid_table_model model;
	union {
		struct {
			/* Intel specific fields */
		} intel;

		struct {
			/* ARM specific fields */
		} arm;
Quoted 7 lines
		struct {
			/* AMD specific fields */
		} amd;

		/* ... */
	};
};

Yes, this looks good to me.

Regards,

Joerg

#24 lore raw
In reply to Raj, Ashok
Quoted 7 lines · Ashok

Hi Jean

Quoted 3 lines · Jean-Philippe Brucker

It's the same problem on SMMUv3, all pointers in the multi-level PASID tables are GPAs. Our solution was to grab chunks GPA space from the guest when necessary.

Quoted 8 lines

If pasid table is gPA and built and managed by guest IOMMU driver in case of vIOMMU, or virtio-iommu. Is this complex interaction to reserve memory and setup required? wouldn't it be sufficient if we do this.

- Host advertises the capability in terms of what table format it it supports.
- guest does all the memory allocation and management.
- communicate to host via VFIO, gPA, format, size of table
- host now programs context entries appropriately with proper nested mode

Yes, I think this is simpler than introducing a new memreserve API. The code to build the pasid-tables is mostly there already, it just needs some refactoring to make it usable for viommu.

The memreserve approach also implicates that the host can not trust the pasid tables it builds and need to re-verify them on every change because it is still accessible by the guest. We could write-protect the table or unmap it from the guest, but it would still be accessible by DMA. I think its tricky to get this right.

Regards,

Joerg

#23 lore raw
In reply to a message not in the archive
Quoted 6 lines · Kevin

Thinking about anyway we need to factor page table code in the guest side. It's not a big deal to further factor PASID code due to the same reason. Of course there are additional format info to be communicated back-and-forth, but the framework should be same for both page table and PASID purposes. Also such change is restricted within IOMMU sub-system. :-)

Right, and in the common case where the guest runs a distro-kernel which has the iommu-drivers compiled-in, the code to build the pasid-tables is also already available and vIOMMU just needs to use it. So this is the easiest approach.

Regards,

Joerg