IOMMU user API enhancement
PATCH 1/3 iommu/uapi: Define uapi version and capabilities
In reply to Jacob PanDefine a unified UAPI version to be used for compatibility checks between user and kernel.
--- include/uapi/linux/iommu.h | 48 ++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 48 insertions(+)
include/uapi/linux/iommu.h +48 -0
@@ -8,6 +8,54 @@#include <linux/types.h> +/** +* Current version of the IOMMU user API. This is intended for query +* between user and kernel to determine compatible data structures. +* +* Having a single UAPI version to govern the user-kernel data structures +* makes compatibility check straightforward. On the contrary, supporting +* combinations of multiple versions of the data can be a nightmare. +* +* UAPI version can be bumped up with the following rules: +* 1. All data structures passed between user and kernel space share +* the same version number. i.e. any extension to to any structure +* results in version bump up. +* +* 2. Data structures are open to extension but closed to modification. +* New fields must be added at the end of each data structure with +* 64bit alignment. Flag bits can be added without size change but +* existing ones cannot be altered. +* +* 3. Versions are backward compatible. +* +* 4. Version to size lookup is supported by kernel internal API for each +* API function type. @version is mandatory for new data structures +* and must be at the beginning with type of __u32. +*/ +#define IOMMU_UAPI_VERSION 1 +static inline int iommu_get_uapi_version (void ) +{ +return IOMMU_UAPI_VERSION ; +} + +/* +* Supported UAPI features that can be reported to user space. +* These types represent the capability available in the kernel. +* +* REVISIT: UAPI version also implies the capabilities. Should we +* report them explicitly? +*/ +enum IOMMU_UAPI_DATA_TYPES { +IOMMU_UAPI_BIND_GPASID , +IOMMU_UAPI_CACHE_INVAL , +IOMMU_UAPI_PAGE_RESP , +NR_IOMMU_UAPI_TYPE , +}; + +#define IOMMU_UAPI_CAP_MASK ((1 << IOMMU_UAPI_BIND_GPASID) | \ + (1 << IOMMU_UAPI_CACHE_INVAL) | \ + (1 << IOMMU_UAPI_PAGE_RESP)) +#define IOMMU_FAULT_PERM_READ (1 << 0)/* read */ #define IOMMU_FAULT_PERM_WRITE (1 << 1)/* write */ #define IOMMU_FAULT_PERM_EXEC (1 << 2)/* exec */
Signature · 6 lines
2.7.4 _______________________________________________ iommu mailing list [email protected] https://lists.linuxfoundation.org/mailman/listinfo/iommu
cover IOMMU user API enhancement
IOMMU user API header was introduced to support nested DMA translation and related fault handling. The current UAPI data structures consist of three areas that cover the interactions between host kernel and guest:
- fault handling
- cache invalidation
- bind guest page tables, i.e. guest PASID
With future extension in mind, the UAPI structures passed from user to kernel always starts with a mandatory version field (u32). While this is flexible for extensions of individual structures, it is difficult to maintain support of combinations of different version numbers.
This patchset introduces a unified UAPI version number that governs all the UAPI data structure versions. When userspace query UAPI version for check on compatibility, a single match would be sufficient.
After UAPI version check, users such as VFIO can also retrieve the matching data structure size based on version and type. Kernel IOMMU UAPI support is always backward compatible. Data structures are also only open to extension and closed to modifications.
The introduction of UAPI version does not change the existing UAPI but rather simplify the data structure version and size matching.
Thanks,
Jacob
Jacob Pan (3):
iommu/uapi: Define uapi version and capabilities iommu/uapi: Use unified UAPI version iommu/uapi: Add helper function for size lookup
drivers/iommu/intel-iommu.c | 3 ++- drivers/iommu/intel-svm.c | 2 +- drivers/iommu/iommu.c | 25 +++++++++++++++++++- include/linux/iommu.h | 6 +++++ include/uapi/linux/iommu.h | 57 ++++++++++++++++++++++++++++++++++++++++----- 5 files changed, 84 insertions(+), 9 deletions(-)
Signature · 6 lines
2.7.4 _______________________________________________ iommu mailing list [email protected] https://lists.linuxfoundation.org/mailman/listinfo/iommu
PATCH 2/3 iommu/uapi: Use unified UAPI version
Reuse UAPI version for each UAPI data structure.
This is to avoid supporting multiple version combinations, simplify support model as we bump up the versions.
--- drivers/iommu/intel-iommu.c | 3 ++- drivers/iommu/intel-svm.c | 2 +- drivers/iommu/iommu.c | 3 ++- include/uapi/linux/iommu.h | 9 +++------ 4 files changed, 8 insertions(+), 9 deletions(-)
drivers/iommu/intel-iommu.c +2 -1
@@ -5763,8 +5763,9 @@ static int intel_iommu_sva_invalidate(struct iommu_domain *domain,int ret =0 ;u64 size; +/* Support current or older UAPI versions */ if (!inv_info || !dmar_domain || - inv_info->version !=IOMMU_CACHE_INVALIDATE_INFO_VERSION_1 ) + inv_info->version >IOMMU_UAPI_VERSION )return -EINVAL ;if (!dev || !dev_is_pci (dev))
drivers/iommu/intel-svm.c +1 -1
@@ -239,7 +239,7 @@ int intel_svm_bind_gpasid(struct iommu_domain *domain,if (WARN_ON (!iommu) || !data)return -EINVAL ; -if (data->version !=IOMMU_GPASID_BIND_VERSION_1 || +if (data->version >IOMMU_UAPI_VERSION || data->format !=IOMMU_PASID_FORMAT_INTEL_VTD )return -EINVAL ;
drivers/iommu/iommu.c +2 -1
@@ -1094,7 +1094,8 @@ int iommu_page_response(struct device *dev,if (!param || !param->fault_param)return -EINVAL ; -if (msg->version !=IOMMU_PAGE_RESP_VERSION_1 || +/* Support current or older UAPI versions */ +if (msg->version >IOMMU_UAPI_VERSION || msg->flags & ~IOMMU_PAGE_RESP_PASID_VALID )return -EINVAL ;
include/uapi/linux/iommu.h +3 -6
@@ -183,7 +183,7 @@ enum iommu_page_response_code {/** * struct iommu_page_response - Generic page response information -* @version: API version of this structure +* @version: IOMMU_UAPI_VERSION * @flags: encodes whether the corresponding fields are valid * (IOMMU_FAULT_PAGE_RESPONSE_* values) * @pasid: Process Address Space ID
@@ -191,7 +191,6 @@ enum iommu_page_response_code {* @code: response code from &enum iommu_page_response_code */ struct iommu_page_response { -#define IOMMU_PAGE_RESP_VERSION_1 1__u32 version;#define IOMMU_PAGE_RESP_PASID_VALID (1 << 0)__u32 flags;
@@ -266,7 +265,7 @@ struct iommu_inv_pasid_info {/** * struct iommu_cache_invalidate_info - First level/stage invalidation * information -* @version: API version of this structure +* @version: IOMMU_UAPI_VERSION * @cache: bitfield that allows to select which caches to invalidate * @granularity: defines the lowest granularity used for the invalidation: * domain > PASID > addr
@@ -294,7 +293,6 @@ struct iommu_inv_pasid_info {* must support the used granularity. */ struct iommu_cache_invalidate_info { -#define IOMMU_CACHE_INVALIDATE_INFO_VERSION_1 1__u32 version;/* IOMMU paging structure cache */ #define IOMMU_CACHE_INV_TYPE_IOTLB (1 << 0)/* IOMMU IOTLB */
@@ -338,7 +336,7 @@ struct iommu_gpasid_bind_data_vtd { IOMMU_SVA_VTD_GPASID_PWT)/** * struct iommu_gpasid_bind_data - Information about device and guest PASID binding -* @version: Version of this data structure +* @version: IOMMU_UAPI_VERSION * @format: PASID table entry format * @flags: Additional information on guest bind request * @gpgd: Guest page directory base of the guest mm to bind
@@ -355,7 +353,6 @@ struct iommu_gpasid_bind_data_vtd {* PASID to host PASID based on this bind data. */ struct iommu_gpasid_bind_data { -#define IOMMU_GPASID_BIND_VERSION_1 1__u32 version;#define IOMMU_PASID_FORMAT_INTEL_VTD 1__u32 format;
Signature · 6 lines
2.7.4 _______________________________________________ iommu mailing list [email protected] https://lists.linuxfoundation.org/mailman/listinfo/iommu
PATCH 3/3 iommu/uapi: Add helper function for size lookup
In reply to Jacob PanIOMMU UAPI can be extended in the future by adding new fields at the end of each user data structure. Since we use a unified UAPI version for compatibility checking, a lookup function is needed to find the correct user data size to copy from user.
This patch adds a helper function based on a 2D lookup with version and type as input arguments.
--- drivers/iommu/iommu.c | 22 ++++++++++++++++++++++ include/linux/iommu.h | 6 ++++++ 2 files changed, 28 insertions(+)
drivers/iommu/iommu.c +22 -0
@@ -1696,6 +1696,28 @@ int iommu_sva_unbind_gpasid(struct iommu_domain *domain, struct device *dev, }EXPORT_SYMBOL_GPL (iommu_sva_unbind_gpasid); + +/** +* Maintain a UAPI version to user data structure size lookup for each +* API function types we support. e.g. bind guest pasid, cache invalidation. +* As data structures being extended with new members, the offsetofend() +* will identify the new sizes. +*/ +const static int iommu_uapi_data_size[NR_IOMMU_UAPI_TYPE ][IOMMU_UAPI_VERSION ] = { +/* IOMMU_UAPI_BIND_GPASID */ + {offsetofend(struct iommu_gpasid_bind_data, vtd)}, +/* IOMMU_UAPI_CACHE_INVAL */ + {offsetofend(struct iommu_cache_invalidate_info, addr_info)}, +/* IOMMU_UAPI_PAGE_RESP */ + {offsetofend(struct iommu_page_response, code)}, +}; + +int iommu_uapi_get_data_size (int type,int version) +{ +return iommu_uapi_data_size[type][version -1 ]; +} +EXPORT_SYMBOL_GPL (iommu_uapi_get_data_size); +static void __iommu_detach_device (struct iommu_domain *domain,struct device *dev) {
include/linux/iommu.h +6 -0
@@ -500,6 +500,7 @@ extern int iommu_report_device_fault(struct device *dev, struct iommu_fault_event *evt);extern int iommu_page_response (struct device *dev,struct iommu_page_response *msg); +extern int iommu_uapi_get_data_size (int type,int version);extern int iommu_group_id (struct iommu_group *group);extern struct iommu_group *iommu_group_get_for_dev (struct device *dev);
@@ -885,6 +886,11 @@ static inline int iommu_page_response(struct device *dev,return -ENODEV ; } +static int iommu_uapi_get_data_size (int type,int version) +{ +return -ENODEV ; +} +static inline int iommu_group_id (struct iommu_group *group) {return -ENODEV ;
Signature · 6 lines
2.7.4 _______________________________________________ iommu mailing list [email protected] https://lists.linuxfoundation.org/mailman/listinfo/iommu
Re: [PATCH 3/3] iommu/uapi: Add helper function for size lookup
Quoted 32 lines · On Tue, 28 Jan 2020 22:02:04 -0800 Jacob Pan <[email protected]>…
IOMMU UAPI can be extended in the future by adding new fields at the end of each user data structure. Since we use a unified UAPI version for compatibility checking, a lookup function is needed to find the correct user data size to copy from user.
This patch adds a helper function based on a 2D lookup with version and type as input arguments.
Signed-off-by Liu Yi L <[email protected]>Signed-off-by Jacob Pan <[email protected]>--- drivers/iommu/iommu.c | 22 ++++++++++++++++++++++ include/linux/iommu.h | 6 ++++++ 2 files changed, 28 insertions(+)drivers/iommu/iommu.c +8
-0@@ -1696,6 +1696,28 @@ int iommu_sva_unbind_gpasid(struct iommu_domain *domain, struct device *dev, }EXPORT_SYMBOL_GPL (iommu_sva_unbind_gpasid); + +/** +* Maintain a UAPI version to user data structure size lookup for each +* API function types we support. e.g. bind guest pasid, cache invalidation. +* As data structures being extended with new members, the offsetofend() +* will identify the new sizes. +*/ +const static int iommu_uapi_data_size[NR_IOMMU_UAPI_TYPE ][IOMMU_UAPI_VERSION ] = {
drivers/iommu/iommu.c int iommu_sva_unbind_gpasid(struct iommu_domain *domain, struct device *dev, · line 1716
Quoted 10 lines
drivers/iommu/iommu.c +10
-0+/* IOMMU_UAPI_BIND_GPASID */ + {offsetofend(struct iommu_gpasid_bind_data, vtd)}, + /* IOMMU_UAPI_CACHE_INVAL */ + {offsetofend(struct iommu_cache_invalidate_info, addr_info)}, + /* IOMMU_UAPI_PAGE_RESP */ + {offsetofend(struct iommu_page_response, code)}, +}; + +int iommu_uapi_get_data_size(int type, int version) +{
Seems like this is asking for a bounds check,
if (type >= NR_IOMMU_UAPI_TYPE || version > IOMMU_UAPI_VERSION) return -EINVAL;
If we add new types in future versions, I assume we'd back fill the table with -EINVAL as well (rather than zero). Thanks,
Alex
Quoted 31 lines
drivers/iommu/iommu.c +4
-0+return iommu_uapi_data_size[type][version -1 ]; +} +EXPORT_SYMBOL_GPL (iommu_uapi_get_data_size); +static void __iommu_detach_device (struct iommu_domain *domain,struct device *dev) {include/linux/iommu.h +6
-0@@ -500,6 +500,7 @@ extern int iommu_report_device_fault(struct device *dev, struct iommu_fault_event *evt);extern int iommu_page_response (struct device *dev,struct iommu_page_response *msg); +extern int iommu_uapi_get_data_size (int type,int version);extern int iommu_group_id (struct iommu_group *group);extern struct iommu_group *iommu_group_get_for_dev (struct device *dev);@@ -885,6 +886,11 @@ static inline int iommu_page_response(struct device *dev,return -ENODEV ; } +static int iommu_uapi_get_data_size (int type,int version) +{ +return -ENODEV ; +} +static inline int iommu_group_id (struct iommu_group *group) {return -ENODEV ;
Re: [PATCH 3/3] iommu/uapi: Add helper function for size lookup
Quoted 27 lines · On Wed, 29 Jan 2020 14:40:46 -0700 Alex Williamson…
Quoted 24 lines · On Tue, 28 Jan 2020 22:02:04 -0800 Jacob Pan <[email protected]>…
IOMMU UAPI can be extended in the future by adding new fields at the end of each user data structure. Since we use a unified UAPI version for compatibility checking, a lookup function is needed to find the correct user data size to copy from user.
This patch adds a helper function based on a 2D lookup with version and type as input arguments.
Signed-off-by Liu Yi L <[email protected]>Signed-off-by Jacob Pan <[email protected]>--- drivers/iommu/iommu.c | 22 ++++++++++++++++++++++ include/linux/iommu.h | 6 ++++++ 2 files changed, 28 insertions(+)drivers/iommu/iommu.c +0
-0@@ -1696,6 +1696,28 @@ int iommu_sva_unbind_gpasid(struct iommu_domain *domain, struct device *dev, }EXPORT_SYMBOL_GPL (iommu_sva_unbind_gpasid);
Quoted 12 lines
drivers/iommu/iommu.c int iommu_sva_unbind_gpasid(struct iommu_domain *domain, struct device *dev, · line 1710
Quoted 12 lines
drivers/iommu/iommu.c +12
-0+ +/** +* Maintain a UAPI version to user data structure size lookup for each +* API function types we support. e.g. bind guest pasid, cache invalidation. +* As data structures being extended with new members, the offsetofend() +* will identify the new sizes. +*/ +const static int iommu_uapi_data_size[NR_IOMMU_UAPI_TYPE ][IOMMU_UAPI_VERSION ] = { +/* IOMMU_UAPI_BIND_GPASID */ + {offsetofend(struct iommu_gpasid_bind_data, vtd)}, +/* IOMMU_UAPI_CACHE_INVAL */ + {offsetofend(struct iommu_cache_invalidate_info, addr_info)},
This seems prone to errors in future revisions. Both of the above reference the end of fields within an anonymous union. When a new field is added, it's not necessarily the newest field that needs to be listed here, but the largest at the time. So should the current version always use sizeof instead (or name the union so we can reference it)? I'm not sure of an error proof way to make sure we keep the N-1 version consistent when we add a new version though. More comments?
Also, is the 12-bytes of padding in struct iommu_gpasid_bind_data excessive with this new versioning scheme? Per rule #2 I'm not sure if we're allowed to repurpose those padding bytes, but if we add fields to the end of the structure as the scheme suggests, we're stuck with not being able to expand the union for new fields.
Thanks,
Alex
Quoted 49 lines
drivers/iommu/iommu.c int iommu_sva_unbind_gpasid(struct iommu_domain *domain, struct device *dev, · line 1716
Quoted 6 lines
drivers/iommu/iommu.c +6
-0+/* IOMMU_UAPI_PAGE_RESP */ + {offsetofend(struct iommu_page_response, code)}, +}; + +int iommu_uapi_get_data_size (int type,int version) +{Seems like this is asking for a bounds check,
if (type >= NR_IOMMU_UAPI_TYPE || version > IOMMU_UAPI_VERSION) return -EINVAL;If we add new types in future versions, I assume we'd back fill the table with
-EINVALas well (rather than zero). Thanks,Alex
Quoted 31 lines
drivers/iommu/iommu.c +4
-0+return iommu_uapi_data_size[type][version -1 ]; +} +EXPORT_SYMBOL_GPL (iommu_uapi_get_data_size); +static void __iommu_detach_device (struct iommu_domain *domain,struct device *dev) {include/linux/iommu.h +6
-0@@ -500,6 +500,7 @@ extern int iommu_report_device_fault(struct device *dev, struct iommu_fault_event *evt);extern int iommu_page_response (struct device *dev,struct iommu_page_response *msg); +extern int iommu_uapi_get_data_size (int type,int version);extern int iommu_group_id (struct iommu_group *group);extern struct iommu_group *iommu_group_get_for_dev (struct device *dev);@@ -885,6 +886,11 @@ static inline int iommu_page_response(struct device *dev,return -ENODEV ; } +static int iommu_uapi_get_data_size (int type,int version) +{ +return -ENODEV ; +} +static inline int iommu_group_id (struct iommu_group *group) {return -ENODEV ;
Re: [PATCH 3/3] iommu/uapi: Add helper function for size lookup
In reply to Alex WilliamsonQuoted 53 lines · On Wed, 29 Jan 2020 14:40:46 -0700 Alex Williamson…
Quoted 41 lines · On Tue, 28 Jan 2020 22:02:04 -0800 Jacob Pan <[email protected]>…
IOMMU UAPI can be extended in the future by adding new fields at the end of each user data structure. Since we use a unified UAPI version for compatibility checking, a lookup function is needed to find the correct user data size to copy from user.
This patch adds a helper function based on a 2D lookup with version and type as input arguments.
Signed-off-by Liu Yi L <[email protected]>Signed-off-by Jacob Pan <[email protected]>--- drivers/iommu/iommu.c | 22 ++++++++++++++++++++++ include/linux/iommu.h | 6 ++++++ 2 files changed, 28 insertions(+)drivers/iommu/iommu.c +3
-0@@ -1696,6 +1696,28 @@ int iommu_sva_unbind_gpasid(struct iommu_domain *domain, struct device *dev, }EXPORT_SYMBOL_GPL (iommu_sva_unbind_gpasid ); + +/** +* Maintain a UAPI version to user data structure size lookup for each
+ * API function types we support. e.g. bind guest pasid, cache invalidation.
+ * As data structures being extended with new members, theoffsetofend()
+ * will identify the new sizes.
+ */
+const static int iommu_uapi_data_size[NR_IOMMU_UAPI_TYPE][IOMMU_UAPI_VERSION] = {drivers/iommu/iommu.c +4
-0+/* IOMMU_UAPI_BIND_GPASID */ + {offsetofend(struct iommu_gpasid_bind_data, vtd)}, +/* IOMMU_UAPI_CACHE_INVAL */ + {offsetofend(struct iommu_cache_invalidate_info,
addr_info)},drivers/iommu/iommu.c int iommu_sva_unbind_gpasid(struct iommu_domain *domain, struct device *dev, · line 1716
Quoted 6 lines
drivers/iommu/iommu.c +6
-0+/* IOMMU_UAPI_PAGE_RESP */ + {offsetofend(struct iommu_page_response, code)}, +}; + +int iommu_uapi_get_data_size (int type,int version) +{Seems like this is asking for a bounds check,
Quoted 3 lines
if (type >= NR_IOMMU_UAPI_TYPE || version > IOMMU_UAPI_VERSION) return -EINVAL;
yes, agreed.
Quoted 3 lines
If we add new types in future versions, I assume we'd back fill the table with
-EINVALas well (rather than zero). Thanks,
right, if the array increase due to new types, the older version with the new type should be filled with -EINVAL.
Let me document this in the rules of extensions.
Quoted 25 lines
Alex
Quoted 23 lines
drivers/iommu/iommu.c +4
-0+return iommu_uapi_data_size[type][version -1 ]; +} +EXPORT_SYMBOL_GPL (iommu_uapi_get_data_size); +static void __iommu_detach_device (struct iommu_domain *domain,struct device *dev) {include/linux/iommu.h +1
-0@@ -500,6 +500,7 @@ extern int iommu_report_device_fault(struct device *dev, struct iommu_fault_event *evt);extern int iommu_page_response (struct device *dev,struct iommu_page_response *msg); +extern int iommu_uapi_get_data_size (int type,int version);extern int iommu_group_id (struct iommu_group *group); extern struct iommu_group *iommu_group_get_for_dev(struct device*dev); @@ -885,6 +886,11 @@ static inline int iommu_page_response(struct device *dev, return
-ENODEV;
}
Quoted 9 lines
include/linux/iommu.h static inline int iommu_page_response(struct device *dev, · line 896
Quoted 8 lines
include/linux/iommu.h +5
-0+static int iommu_uapi_get_data_size (int type,int version) +{ +return -ENODEV ; +} +static inline int iommu_group_id (struct iommu_group *group) {return -ENODEV ;
[Jacob Pan]
Re: [PATCH 3/3] iommu/uapi: Add helper function for size lookup
In reply to Alex WilliamsonQuoted 49 lines · On Wed, 29 Jan 2020 15:19:51 -0700 Alex Williamson…
Quoted 36 lines · On Wed, 29 Jan 2020 14:40:46 -0700 Alex Williamson…
Quoted 33 lines · On Tue, 28 Jan 2020 22:02:04 -0800 Jacob Pan <[email protected]>…
IOMMU UAPI can be extended in the future by adding new fields at the end of each user data structure. Since we use a unified UAPI version for compatibility checking, a lookup function is needed to find the correct user data size to copy from user.
This patch adds a helper function based on a 2D lookup with version and type as input arguments.
Signed-off-by Liu Yi L <[email protected]>Signed-off-by Jacob Pan <[email protected]>--- drivers/iommu/iommu.c | 22 ++++++++++++++++++++++ include/linux/iommu.h | 6 ++++++ 2 files changed, 28 insertions(+)drivers/iommu/iommu.c +3
-0@@ -1696,6 +1696,28 @@ int iommu_sva_unbind_gpasid(struct iommu_domain *domain, struct device *dev, }EXPORT_SYMBOL_GPL (iommu_sva_unbind_gpasid ); + +/** +* Maintain a UAPI version to user data structure size lookup for each
+ * API function types we support. e.g. bind guest pasid, cache invalidation.
+ * As data structures being extended with new members, theoffsetofend()
+ * will identify the new sizes.Quoted 8 lines
Quoted 8 lines
+ */
+const static int iommu_uapi_data_size[NR_IOMMU_UAPI_TYPE][IOMMU_UAPI_VERSION] = {drivers/iommu/iommu.c +4
-0+/* IOMMU_UAPI_BIND_GPASID */ + {offsetofend(struct iommu_gpasid_bind_data, vtd)}, +/* IOMMU_UAPI_CACHE_INVAL */ + {offsetofend(struct iommu_cache_invalidate_info,
addr_info)},This seems prone to errors in future revisions. Both of the above
Quoted 9 lines
reference the end of fields within an anonymous union. When a new field is added, it's not necessarily the newest field that needs to be listed here, but the largest at the time.
So should the current
version always use sizeof instead (or name the union so we can reference it)? I'm not sure of an error proof way to make sure we keep the N-1 version consistent when we add a new version though. More comments?
yes. we must be very careful the new size has to be the largest of the union not the newest. I agree that using sizeof() would make current version size straightforward. But we have to find the size for N-1 with offsetofend, seems more risk to the existing N-1 code when bump up version.
I was thinking this array also serve as documentation of the revision history, but as you pointed out offsetofend may not be the newest member of the union. So it cannot document which member was added in which version. Seems more comments is the only way.
How about comments as below with example future extensions?
/**
* Maintain a UAPI version to user data structure size lookup for each
* API function types we support. e.g. bind guest pasid, cache invalidation.
* As data structures being extended with new members, offsetofend() is
* used to identify the size. In case of adding a new member to a union,
* offsetofend() applies to the largest member which may not be the newest. *
* When new types are introduced with new versions, the new types for older
* version must be filled with -EINVAL.
*
* The table below documents UAPI revision history with the name of the
* newest member of each data structure. The largest member of a union was
* used for the initial version of each type.
*
* Current version: V1
* +--------------+---------------+
* | Type / | V1 | * | UAPI Version | | * +==============+===============+ * | BIND_GPASID | vtd | * +--------------+---------------+ * | CACHE_INVAL | addr_info | * +--------------+---------------+ * | PAGE_RESP | code | * +--------------+---------------+ * * Future extension examples: * * V2 adds new members to bind_gpasid and cache_invalidate API but not * page response. * +--------------+---------------+---------------+ * | Type / | V1 | V2 | * | UAPI Version | | | * +==============+===============+===============+ * | BIND_GPASID | vtd | smmu_v3 | * +--------------+---------------+---------------+ * | CACHE_INVAL | addr_info | new_info | * +--------------+---------------+---------------+ * | PAGE_RESP | code | N/A | * +--------------+---------------+---------------+ * * V3 introduces a new UAPI data type: NEW_TYPE but with no new members * added to the existing types. * +--------------+---------------+---------------+---------------+ * | Type / | V1 | V2 | V3 | * | UAPI Version | | | |
* +==============+===============+===============+===============+
* | BIND_GPASID | vtd | smmu_v3 | N/A |
* +--------------+---------------+---------------+---------------+
* | CACHE_INVAL | addr_info | new_info | N/A |
* +--------------+---------------+---------------+---------------+
* | PAGE_RESP | code | N/A | N/A |
* +--------------+---------------+---------------+---------------+
* | NEW_TYPE | -EINVAL | -EINVAL | largest_member|
* +--------------+---------------+---------------+---------------+ *
Quoted 6 lines
Also, is the 12-bytes of padding in struct
iommu_gpasid_bind_dataexcessive with this new versioning scheme? Per rule #2 I'm not sure if we're allowed to repurpose those padding bytes, but if we add fields to the end of the structure as the scheme suggests, we're stuck with not being able to expand the union for new fields.
Quoted 53 lines
Thanks,
AlexQuoted 49 lines
drivers/iommu/iommu.c int iommu_sva_unbind_gpasid(struct iommu_domain *domain, struct device *dev, · line 1716
Quoted 6 lines
drivers/iommu/iommu.c +6
-0+/* IOMMU_UAPI_PAGE_RESP */ + {offsetofend(struct iommu_page_response, code)}, +}; + +int iommu_uapi_get_data_size (int type,int version) +{Seems like this is asking for a bounds check,
if (type >= NR_IOMMU_UAPI_TYPE || version > IOMMU_UAPI_VERSION) return -EINVAL;If we add new types in future versions, I assume we'd back fill the table with
-EINVALas well (rather than zero). Thanks,Alex
Quoted 31 lines
drivers/iommu/iommu.c +4
-0+return iommu_uapi_data_size[type][version -1 ]; +} +EXPORT_SYMBOL_GPL (iommu_uapi_get_data_size); +static void __iommu_detach_device (struct iommu_domain *domain,struct device *dev) {include/linux/iommu.h +1
-0@@ -500,6 +500,7 @@ extern int iommu_report_device_fault(struct device *dev, struct iommu_fault_event *evt);extern int iommu_page_response (struct device *dev,struct iommu_page_response *msg); +extern int iommu_uapi_get_data_size (int type,int version);extern int iommu_group_id (struct iommu_group *group); extern struct iommu_group *iommu_group_get_for_dev(struct device*dev); @@ -885,6 +886,11 @@ static inline int iommu_page_response(struct device *dev, return
-ENODEV;
}include/linux/iommu.h +5
-0+static int iommu_uapi_get_data_size (int type,int version) +{ +return -ENODEV ; +} +static inline int iommu_group_id (struct iommu_group *group) {return -ENODEV ;
[Jacob Pan]
Re: [PATCH 3/3] iommu/uapi: Add helper function for size lookup
In reply to Alex WilliamsonHi Alex,
Sorry I missed this part in the previous reply. Comments below.
Quoted 3 lines · On Wed, 29 Jan 2020 15:19:51 -0700 Alex Williamson…
Also, is the 12-bytes of padding in struct
iommu_gpasid_bind_dataexcessive with this new versioning scheme? Per rule #2 I'm not sure if we're allowed to repurpose those padding bytes,
We can still use the padding bytes as long as there is a new flag bit to indicate the validity of the new filed within the padding. I should have made it clear in rule #2 when mentioning the flags bits. Should define what extension constitutes.
How about this?
"
* 2. Data structures are open to extension but closed to modification.
* Extension should leverage the padding bytes first where a new * flag bit is required to indicate the validity of each new member. * The above rule for padding bytes also applies to adding new union * members. * After padding bytes are exhausted, new fields must be added at the * end of each data structure with 64bit alignment. Flag bits can be * added without size change but existing ones cannot be altered.
*
"
So if we add new field by doing re-purpose of padding bytes, size lookup result will remain the same. New code would recognize the new flag, old code stays the same.
VFIO layer checks for UAPI compatibility and size to copy, version sanity check and flag usage are done in the IOMMU code.
Quoted 3 lines
but if we add
fields to the end of the structure as the scheme suggests, we're stuck with not being able to expand the union for new fields.
Good point, it does sound contradictory. I hope the rewritten rule #2 address that.
Adding data after the union should be extremely rare. Do you see any issues with the example below?
offsetofend() can still find the right size.
e.g.
V1
struct iommu_gpasid_bind_data {
__u32 version;
#define IOMMU_PASID_FORMAT_INTEL_VTD 1
__u32 format;
#define IOMMU_SVA_GPASID_VAL (1 << 0) /* guest PASID valid */
__u64 flags;
__u64 gpgd;
__u64 hpasid;
__u64 gpasid;
__u32 addr_width;
__u8 padding[12];
/* Vendor specific data */
union {
struct iommu_gpasid_bind_data_vtd vtd;
};
};const static int iommu_uapi_data_size[NR_IOMMU_UAPI_TYPE][IOMMU_UAPI_VERSION] = { /* IOMMU_UAPI_BIND_GPASID */ {offsetofend(struct iommu_gpasid_bind_data, vtd)}, ...
};
V2, Add new_member at the end (forget padding for now).
struct iommu_gpasid_bind_data {
__u32 version;
#define IOMMU_PASID_FORMAT_INTEL_VTD 1
__u32 format;
#define IOMMU_SVA_GPASID_VAL (1 << 0) /* guest PASID valid */
#define IOMMU_NEW_MEMBER_VAL (1 << 1) /* new member added */
__u64 flags;
__u64 gpgd;
__u64 hpasid;
__u64 gpasid;
__u32 addr_width;
__u8 padding[12];
/* Vendor specific data */
union {
struct iommu_gpasid_bind_data_vtd vtd;
};
__u64 new_member;
};const static int iommu_uapi_data_size[NR_IOMMU_UAPI_TYPE][IOMMU_UAPI_VERSION] = { /* IOMMU_UAPI_BIND_GPASID */
{offsetofend(struct iommu_gpasid_bind_data,
vtd), offsetofend(struct iommu_gpasid_bind_data,new_member)},
};
V3, Add smmu to the union,larger than vtd
struct iommu_gpasid_bind_data {
__u32 version;
#define IOMMU_PASID_FORMAT_INTEL_VTD 1
#define IOMMU_PASID_FORMAT_INTEL_SMMU 2
__u32 format;
#define IOMMU_SVA_GPASID_VAL (1 << 0) /* guest PASID valid */
#define IOMMU_NEW_MEMBER_VAL (1 << 1) /* new member added */
#define IOMMU_SVA_SMMU_SUPP (1 << 2) /* SMMU data supported */
__u64 flags;
__u64 gpgd;
__u64 hpasid;
__u64 gpasid;
__u32 addr_width;
__u8 padding[12];
/* Vendor specific data */
union {
struct iommu_gpasid_bind_data_vtd vtd;
struct iommu_gpasid_bind_data_smmu smmu;
};
__u64 new_member;
};const static int iommu_uapi_data_size[NR_IOMMU_UAPI_TYPE][IOMMU_UAPI_VERSION] = {
/* IOMMU_UAPI_BIND_GPASID */ {offsetofend(struct iommu_gpasid_bind_data,vtd), offsetofend(struct iommu_gpasid_bind_data, new_member), offsetofend(struct iommu_gpasid_bind_data, new_member)},
...
};
Re: [PATCH 3/3] iommu/uapi: Add helper function for size lookup
Quoted 116 lines · On Fri, 31 Jan 2020 15:51:25 -0800 Jacob Pan <[email protected]>…
Hi Alex,
Sorry I missed this part in the previous reply. Comments below.Quoted 3 lines · On Wed, 29 Jan 2020 15:19:51 -0700 Alex Williamson…
Also, is the 12-bytes of padding in struct
iommu_gpasid_bind_dataexcessive with this new versioning scheme? Per rule #2 I'm not sure if we're allowed to repurpose those padding bytes,We can still use the padding bytes as long as there is a new flag bit to indicate the validity of the new filed within the padding. I should have made it clear in rule #2 when mentioning the flags bits. Should define what extension constitutes.
How about this?
"
* 2. Data structures are open to extension but closed to modification.* Extension should leverage the padding bytes first where a new * flag bit is required to indicate the validity of each new member. * The above rule for padding bytes also applies to adding new union * members. * After padding bytes are exhausted, new fields must be added at the * end of each data structure with 64bit alignment. Flag bits can be * added without size change but existing ones cannot be altered.*
"
So if we add new field by doing re-purpose of padding bytes, size lookup result will remain the same. New code would recognize the new flag, old code stays the same.VFIO layer checks for UAPI compatibility and size to copy, version sanity check and flag usage are done in the IOMMU code.
Quoted 3 lines
but if we add
fields to the end of the structure as the scheme suggests, we're stuck with not being able to expand the union for new fields.Good point, it does sound contradictory. I hope the rewritten rule #2 address that.
Adding data after the union should be extremely rare. Do you see any issues with the example below?
offsetofend()can still find the right size.
e.g.
V1struct iommu_gpasid_bind_data { __u32 version; #define IOMMU_PASID_FORMAT_INTEL_VTD 1 __u32 format; #define IOMMU_SVA_GPASID_VAL (1 << 0) /* guest PASID valid */ __u64 flags; __u64 gpgd; __u64 hpasid; __u64 gpasid; __u32 addr_width; __u8 padding[12]; /* Vendor specific data */ union { struct iommu_gpasid_bind_data_vtd vtd; }; };const static int iommu_uapi_data_size[NR_IOMMU_UAPI_TYPE][IOMMU_UAPI_VERSION] = { /*
IOMMU_UAPI_BIND_GPASID*/ {offsetofend(struct iommu_gpasid_bind_data, vtd)}, ...
};V2, Add
new_memberat the end (forget padding for now).struct iommu_gpasid_bind_data { __u32 version; #define IOMMU_PASID_FORMAT_INTEL_VTD 1 __u32 format; #define IOMMU_SVA_GPASID_VAL (1 << 0) /* guest PASID valid */ #define IOMMU_NEW_MEMBER_VAL (1 << 1) /* new member added */ __u64 flags; __u64 gpgd; __u64 hpasid; __u64 gpasid; __u32 addr_width; __u8 padding[12]; /* Vendor specific data */ union { struct iommu_gpasid_bind_data_vtd vtd; }; __u64 new_member; };const static int iommu_uapi_data_size[NR_IOMMU_UAPI_TYPE][IOMMU_UAPI_VERSION] = { /*
IOMMU_UAPI_BIND_GPASID*/
{offsetofend(structiommu_gpasid_bind_data,
vtd),offsetofend(struct iommu_gpasid_bind_data,new_member)},};
V3, Add smmu to the union,larger than vtd
struct iommu_gpasid_bind_data { __u32 version; #define IOMMU_PASID_FORMAT_INTEL_VTD 1 #define IOMMU_PASID_FORMAT_INTEL_SMMU 2 __u32 format; #define IOMMU_SVA_GPASID_VAL (1 << 0) /* guest PASID valid */ #define IOMMU_NEW_MEMBER_VAL (1 << 1) /* new member added */ #define IOMMU_SVA_SMMU_SUPP (1 << 2) /* SMMU data supported */ __u64 flags; __u64 gpgd; __u64 hpasid; __u64 gpasid; __u32 addr_width; __u8 padding[12]; /* Vendor specific data */ union { struct iommu_gpasid_bind_data_vtd vtd; struct iommu_gpasid_bind_data_smmu smmu; }; __u64 new_member; };
Quoted 9 lines
const static int iommu_uapi_data_size[NR_IOMMU_UAPI_TYPE][IOMMU_UAPI_VERSION] = {
/*IOMMU_UAPI_BIND_GPASID*/ {offsetofend(struct iommu_gpasid_bind_data,vtd),offsetofend(struct iommu_gpasid_bind_data, new_member),offsetofend(struct iommu_gpasid_bind_data, new_member)},
...
};
How are you not breaking rule #3, "Versions are backward compatible" with this? If the kernel is at version 3 and userspace is at version 2 then new_member exists at different offsets of the structure. The kernels iommu_uapi_data_size for V2 changed between version 2 and 3. Thanks,
Alex
Re: [PATCH 3/3] iommu/uapi: Add helper function for size lookup
Quoted 135 lines · On Mon, 3 Feb 2020 11:27:08 -0700 Alex Williamson <[email protected]>…
Quoted 122 lines · On Fri, 31 Jan 2020 15:51:25 -0800 Jacob Pan <[email protected]>…
Hi Alex,
Sorry I missed this part in the previous reply. Comments below.Quoted 3 lines · On Wed, 29 Jan 2020 15:19:51 -0700 Alex Williamson…
Also, is the 12-bytes of padding in struct
iommu_gpasid_bind_dataexcessive with this new versioning scheme? Per rule #2 I'm not sure if we're allowed to repurpose those padding bytes,We can still use the padding bytes as long as there is a new flag bit to indicate the validity of the new filed within the padding. I should have made it clear in rule #2 when mentioning the flags bits. Should define what extension constitutes.
How about this?
"
* 2. Data structures are open to extension but closed to
modification.
* Extension should leverage the padding bytes first where a new
* flag bit is required to indicate the validity of each new
member.
* The above rule for padding bytes also applies to adding new
union
* members.
* After padding bytes are exhausted, new fields must be added
at the
* end of each data structure with 64bit alignment. Flag bits
can be
* added without size change but existing ones cannot be altered.
*
"
So if we add new field by doing re-purpose of padding bytes, size lookup result will remain the same. New code would recognize the new flag, old code stays the same.VFIO layer checks for UAPI compatibility and size to copy, version sanity check and flag usage are done in the IOMMU code.
Quoted 3 lines
but if we add
fields to the end of the structure as the scheme suggests, we're stuck with not being able to expand the union for new fields.Good point, it does sound contradictory. I hope the rewritten rule #2 address that.
Adding data after the union should be extremely rare. Do you see any issues with the example below?
offsetofend()can still find the right size.
e.g.
V1struct iommu_gpasid_bind_data { __u32 version; #define IOMMU_PASID_FORMAT_INTEL_VTD 1 __u32 format; #define IOMMU_SVA_GPASID_VAL (1 << 0) /* guest PASID valid */ __u64 flags; __u64 gpgd; __u64 hpasid; __u64 gpasid; __u32 addr_width; __u8 padding[12]; /* Vendor specific data */ union { struct iommu_gpasid_bind_data_vtd vtd; }; };const static int iommu_uapi_data_size[NR_IOMMU_UAPI_TYPE][IOMMU_UAPI_VERSION] = { /*
IOMMU_UAPI_BIND_GPASID*/ {offsetofend(struct iommu_gpasid_bind_data, vtd)}, ...
};V2, Add
new_memberat the end (forget padding for now).struct iommu_gpasid_bind_data { __u32 version; #define IOMMU_PASID_FORMAT_INTEL_VTD 1 __u32 format; #define IOMMU_SVA_GPASID_VAL (1 << 0) /* guest PASID valid */ #define IOMMU_NEW_MEMBER_VAL (1 << 1) /* new member added */ __u64 flags; __u64 gpgd; __u64 hpasid; __u64 gpasid; __u32 addr_width; __u8 padding[12]; /* Vendor specific data */ union { struct iommu_gpasid_bind_data_vtd vtd; }; __u64 new_member; };const static int iommu_uapi_data_size[NR_IOMMU_UAPI_TYPE][IOMMU_UAPI_VERSION] = { /*
IOMMU_UAPI_BIND_GPASID*/
{offsetofend(struct iommu_gpasid_bind_data, vtd), offsetofend(struct
iommu_gpasid_bind_data,new_member)},};
V3, Add smmu to the union,larger than vtd
struct iommu_gpasid_bind_data { __u32 version; #define IOMMU_PASID_FORMAT_INTEL_VTD 1 #define IOMMU_PASID_FORMAT_INTEL_SMMU 2 __u32 format; #define IOMMU_SVA_GPASID_VAL (1 << 0) /* guest PASID valid */ #define IOMMU_NEW_MEMBER_VAL (1 << 1) /* new member added */ #define IOMMU_SVA_SMMU_SUPP (1 << 2) /* SMMU data supported */ __u64 flags; __u64 gpgd; __u64 hpasid; __u64 gpasid; __u32 addr_width; __u8 padding[12]; /* Vendor specific data */ union { struct iommu_gpasid_bind_data_vtd vtd; struct iommu_gpasid_bind_data_smmu smmu; }; __u64 new_member; };Quoted 9 lines
const static int iommu_uapi_data_size[NR_IOMMU_UAPI_TYPE][IOMMU_UAPI_VERSION] = {
/*IOMMU_UAPI_BIND_GPASID*/ {offsetofend(struct iommu_gpasid_bind_data,vtd),offsetofend(struct iommu_gpasid_bind_data, new_member),offsetofend(struct iommu_gpasid_bind_data, new_member)},
...
};
Quoted 6 lines
How are you not breaking rule #3, "Versions are backward compatible" with this? If the kernel is at version 3 and userspace is at version 2 then
new_memberexists at different offsets of the structure. The kernelsiommu_uapi_data_sizefor V2 changed between version 2 and 3. Thanks,
You are right. if we want to add new member to the end of the structure as well as expanding union, I think we have to fix the size of the union. Would this work? (just an example for one struct)
@@ -344,6 +348,11 @@ struct iommu_gpasid_bind_data_vtd {
* @gpasid: Process address space ID used for the guest mm in guest
IOMMU
* @addr_width: Guest virtual address width
* @padding: Reserved for future use (should be zero)
+ * @dummy Reserve space for vendor specific data in the union. New
+ * members added to the union cannot exceed the size of
dummy.
+ * The fixed size union is needed to allow further
expansion
+ * after the end of the union while still maintain backward
+ * compatibility.
* @vtd: Intel VT-d specific data
*
* Guest to host PASID mapping can be an identity or non-identity,
where guest @@ -365,6 +374,7 @@ struct iommu_gpasid_bind_data {
__u8 padding[12];
/* Vendor specific data */
union {
+ __u8 dummy[128];
struct iommu_gpasid_bind_data_vtd vtd;
};};
Quoted 2 lines
Alex
[Jacob Pan]
Re: [PATCH 3/3] iommu/uapi: Add helper function for size lookup
Quoted 171 lines · On Mon, 3 Feb 2020 12:41:43 -0800 Jacob Pan <[email protected]>…
Quoted 135 lines · On Mon, 3 Feb 2020 11:27:08 -0700 Alex Williamson <[email protected]>…
Quoted 122 lines · On Fri, 31 Jan 2020 15:51:25 -0800 Jacob Pan <[email protected]>…
Hi Alex,
Sorry I missed this part in the previous reply. Comments below.Quoted 3 lines · On Wed, 29 Jan 2020 15:19:51 -0700 Alex Williamson…
Also, is the 12-bytes of padding in struct
iommu_gpasid_bind_dataexcessive with this new versioning scheme? Per rule #2 I'm not sure if we're allowed to repurpose those padding bytes,We can still use the padding bytes as long as there is a new flag bit to indicate the validity of the new filed within the padding. I should have made it clear in rule #2 when mentioning the flags bits. Should define what extension constitutes.
How about this?
"
* 2. Data structures are open to extension but closed to
modification.
* Extension should leverage the padding bytes first where a new
* flag bit is required to indicate the validity of each new
member.
* The above rule for padding bytes also applies to adding new
union
* members.
* After padding bytes are exhausted, new fields must be added
at the
* end of each data structure with 64bit alignment. Flag bits
can be
* added without size change but existing ones cannot be altered.
*
"
So if we add new field by doing re-purpose of padding bytes, size lookup result will remain the same. New code would recognize the new flag, old code stays the same.VFIO layer checks for UAPI compatibility and size to copy, version sanity check and flag usage are done in the IOMMU code.
Quoted 3 lines
but if we add
fields to the end of the structure as the scheme suggests, we're stuck with not being able to expand the union for new fields.Good point, it does sound contradictory. I hope the rewritten rule #2 address that.
Adding data after the union should be extremely rare. Do you see any issues with the example below?
offsetofend()can still find the right size.
e.g.
V1struct iommu_gpasid_bind_data { __u32 version; #define IOMMU_PASID_FORMAT_INTEL_VTD 1 __u32 format; #define IOMMU_SVA_GPASID_VAL (1 << 0) /* guest PASID valid */ __u64 flags; __u64 gpgd; __u64 hpasid; __u64 gpasid; __u32 addr_width; __u8 padding[12]; /* Vendor specific data */ union { struct iommu_gpasid_bind_data_vtd vtd; }; };const static int iommu_uapi_data_size[NR_IOMMU_UAPI_TYPE][IOMMU_UAPI_VERSION] = { /*
IOMMU_UAPI_BIND_GPASID*/ {offsetofend(struct iommu_gpasid_bind_data, vtd)}, ...
};V2, Add
new_memberat the end (forget padding for now).struct iommu_gpasid_bind_data { __u32 version; #define IOMMU_PASID_FORMAT_INTEL_VTD 1 __u32 format; #define IOMMU_SVA_GPASID_VAL (1 << 0) /* guest PASID valid */ #define IOMMU_NEW_MEMBER_VAL (1 << 1) /* new member added */ __u64 flags; __u64 gpgd; __u64 hpasid; __u64 gpasid; __u32 addr_width; __u8 padding[12]; /* Vendor specific data */ union { struct iommu_gpasid_bind_data_vtd vtd; }; __u64 new_member; };const static int iommu_uapi_data_size[NR_IOMMU_UAPI_TYPE][IOMMU_UAPI_VERSION] = { /*
IOMMU_UAPI_BIND_GPASID*/
{offsetofend(struct iommu_gpasid_bind_data, vtd), offsetofend(struct
iommu_gpasid_bind_data,new_member)},};
V3, Add smmu to the union,larger than vtd
struct iommu_gpasid_bind_data { __u32 version; #define IOMMU_PASID_FORMAT_INTEL_VTD 1 #define IOMMU_PASID_FORMAT_INTEL_SMMU 2 __u32 format; #define IOMMU_SVA_GPASID_VAL (1 << 0) /* guest PASID valid */ #define IOMMU_NEW_MEMBER_VAL (1 << 1) /* new member added */ #define IOMMU_SVA_SMMU_SUPP (1 << 2) /* SMMU data supported */ __u64 flags; __u64 gpgd; __u64 hpasid; __u64 gpasid; __u32 addr_width; __u8 padding[12]; /* Vendor specific data */ union { struct iommu_gpasid_bind_data_vtd vtd; struct iommu_gpasid_bind_data_smmu smmu; }; __u64 new_member; };Quoted 9 lines
const static int iommu_uapi_data_size[NR_IOMMU_UAPI_TYPE][IOMMU_UAPI_VERSION] = {
/*IOMMU_UAPI_BIND_GPASID*/ {offsetofend(struct iommu_gpasid_bind_data,vtd),offsetofend(struct iommu_gpasid_bind_data, new_member),offsetofend(struct iommu_gpasid_bind_data, new_member)},
...
};Quoted 6 lines
How are you not breaking rule #3, "Versions are backward compatible" with this? If the kernel is at version 3 and userspace is at version 2 then
new_memberexists at different offsets of the structure. The kernelsiommu_uapi_data_sizefor V2 changed between version 2 and 3. Thanks,You are right. if we want to add new member to the end of the structure as well as expanding union, I think we have to fix the size of the union. Would this work? (just an example for one struct)
(quoted diff) +0
-0@@ -344,6 +348,11 @@ struct iommu_gpasid_bind_data_vtd { * @gpasid: Process address space ID used for the guest mm in guestIOMMU
* @addr_width: Guest virtual address width* @padding: Reserved for future use (should be zero) + * @dummy Reserve space for vendor specific data in the union. New + * members added to the union cannot exceed the size of dummy. + * The fixed size union is needed to allow further expansion + * after the end of the union while still maintain backward + * compatibility. * @vtd: Intel VT-d specific data * * Guest to host PASID mapping can be an identity or non-identity, where guest @@ -365,6 +374,7 @@ struct iommu_gpasid_bind_data { __u8 padding[12]; /* Vendor specific data */ union { + __u8 dummy[128]; struct iommu_gpasid_bind_data_vtd vtd; };
Quoted 1 line
};
It's not the most space efficient thing and we're just guessing at what might need to be added into that union in future, but it works... until it doesn't ;) One might also argue that we could inflate the padding field even further to serve the same purpose. The only other route I can think of would be to let the user specify the offset of the variable size data from the start of the structure, for example similar to how we're laying out vfio migration region or how we do capabilities in vfio ioctls. This is where passing an argsz for each ioctl comes in handy so we're not limited to a structure, we can link various structures together in a chain. Of course that requires work on both the user and kernel side to pack and unpack, but it leaves a lot of flexibility in extending it. Thanks,
Alex
Re: [PATCH 3/3] iommu/uapi: Add helper function for size lookup
Quoted 177 lines · On Mon, 3 Feb 2020 14:12:36 -0700 Alex Williamson <[email protected]>…
Quoted 174 lines · On Mon, 3 Feb 2020 12:41:43 -0800 Jacob Pan <[email protected]>…
Quoted 138 lines · On Mon, 3 Feb 2020 11:27:08 -0700 Alex Williamson <[email protected]>…
Quoted 125 lines · On Fri, 31 Jan 2020 15:51:25 -0800 Jacob Pan <[email protected]>…
Hi Alex,
Sorry I missed this part in the previous reply. Comments below.Quoted 4 lines · On Wed, 29 Jan 2020 15:19:51 -0700 Alex Williamson…
Also, is the 12-bytes of padding in struct
iommu_gpasid_bind_dataexcessive with this new versioning scheme? Per rule #2 I'm not sure if we're allowed to repurpose those padding bytes,We can still use the padding bytes as long as there is a new flag bit to indicate the validity of the new filed within the padding. I should have made it clear in rule #2 when mentioning the flags bits. Should define what extension constitutes. How about this?
"
* 2. Data structures are open to extension but closed to
modification.
* Extension should leverage the padding bytes first where a
new
* flag bit is required to indicate the validity of each new
member.
* The above rule for padding bytes also applies to adding
new union
* members.
* After padding bytes are exhausted, new fields must be
added at the
* end of each data structure with 64bit alignment. Flag bits
can be
* added without size change but existing ones cannot be
altered. *
"
So if we add new field by doing re-purpose of padding bytes, size lookup result will remain the same. New code would recognize the new flag, old code stays the same.VFIO layer checks for UAPI compatibility and size to copy, version sanity check and flag usage are done in the IOMMU code.
Quoted 4 lines
but if we add
fields to the end of the structure as the scheme suggests, we're stuck with not being able to expand the union for new fields.Good point, it does sound contradictory. I hope the rewritten rule #2 address that.
Adding data after the union should be extremely rare. Do you see any issues with the example below?
offsetofend()can still find the right size.
e.g.
V1struct iommu_gpasid_bind_data { __u32 version; #define IOMMU_PASID_FORMAT_INTEL_VTD 1 __u32 format; #define IOMMU_SVA_GPASID_VAL (1 << 0) /* guest PASID valid */ __u64 flags; __u64 gpgd; __u64 hpasid; __u64 gpasid; __u32 addr_width; __u8 padding[12]; /* Vendor specific data */ union { struct iommu_gpasid_bind_data_vtd vtd; }; };const static int iommu_uapi_data_size[NR_IOMMU_UAPI_TYPE][IOMMU_UAPI_VERSION] = { /*
IOMMU_UAPI_BIND_GPASID*/ {offsetofend(struct iommu_gpasid_bind_data, vtd)}, ...
};V2, Add
new_memberat the end (forget padding for now).struct iommu_gpasid_bind_data { __u32 version; #define IOMMU_PASID_FORMAT_INTEL_VTD 1 __u32 format; #define IOMMU_SVA_GPASID_VAL (1 << 0) /* guest PASID valid */ #define IOMMU_NEW_MEMBER_VAL (1 << 1) /* new member added */ __u64 flags; __u64 gpgd; __u64 hpasid; __u64 gpasid; __u32 addr_width; __u8 padding[12]; /* Vendor specific data */ union { struct iommu_gpasid_bind_data_vtd vtd; }; __u64 new_member; };const static int iommu_uapi_data_size[NR_IOMMU_UAPI_TYPE][IOMMU_UAPI_VERSION] = { /*
IOMMU_UAPI_BIND_GPASID*/
{offsetofend(struct iommu_gpasid_bind_data, vtd), offsetofend(struct
iommu_gpasid_bind_data,new_member)},};
V3, Add smmu to the union,larger than vtd
struct iommu_gpasid_bind_data { __u32 version; #define IOMMU_PASID_FORMAT_INTEL_VTD 1 #define IOMMU_PASID_FORMAT_INTEL_SMMU 2 __u32 format; #define IOMMU_SVA_GPASID_VAL (1 << 0) /* guest PASID valid */ #define IOMMU_NEW_MEMBER_VAL (1 << 1) /* new member added */ #define IOMMU_SVA_SMMU_SUPP (1 << 2) /* SMMU data supported */ __u64 flags; __u64 gpgd; __u64 hpasid; __u64 gpasid; __u32 addr_width; __u8 padding[12]; /* Vendor specific data */ union { struct iommu_gpasid_bind_data_vtd vtd; struct iommu_gpasid_bind_data_smmu smmu; }; __u64 new_member; };Quoted 9 lines
const static int iommu_uapi_data_size[NR_IOMMU_UAPI_TYPE][IOMMU_UAPI_VERSION] = {
/*IOMMU_UAPI_BIND_GPASID*/ {offsetofend(struct iommu_gpasid_bind_data,vtd),offsetofend(struct iommu_gpasid_bind_data, new_member),offsetofend(struct iommu_gpasid_bind_data, new_member)},
...
};Quoted 6 lines
How are you not breaking rule #3, "Versions are backward compatible" with this? If the kernel is at version 3 and userspace is at version 2 then
new_memberexists at different offsets of the structure. The kernelsiommu_uapi_data_sizefor V2 changed between version 2 and 3. Thanks,You are right. if we want to add new member to the end of the structure as well as expanding union, I think we have to fix the size of the union. Would this work? (just an example for one struct)
(quoted diff) +0
-0@@ -344,6 +348,11 @@ struct iommu_gpasid_bind_data_vtd { * @gpasid: Process address space ID used for the guest mm inguest IOMMU
* @addr_width: Guest virtual address width
* @padding: Reserved for future use (should be zero)
+ * @dummy Reserve space for vendor specific data in the
union. New
+ * members added to the union cannot exceed the size of
dummy.
+ * The fixed size union is needed to allow further
expansion
+ * after the end of the union while still maintain
backward
+ * compatibility.
* @vtd: Intel VT-d specific data
*
* Guest to host PASID mapping can be an identity or non-identity,where guest @@ -365,6 +374,7 @@ struct iommu_gpasid_bind_data { __u8 padding[12]; /* Vendor specific data */ union {+
__u8dummy[128];
Quoted 8 lines
Quoted 3 lines
struct iommu_gpasid_bind_data_vtd vtd; };};
It's not the most space efficient thing and we're just guessing at what might need to be added into that union in future, but it works... until it doesn't ;) One might also argue that we could inflate the padding field even further to serve the same purpose.
That was our initial intention, the padding field is already inflated to accommodate any foreseeable extensions :)
Extensions beyond union was deemed unlikely that is why we use the union at the end.
The intention of this patchset is not to change that, but rather clarify and simplify the version checking.
Quoted 10 lines
The only other route I can think of would be to let the user specify the offset of the variable size data from the start of the structure, for example similar to how we're laying out vfio migration region or how we do capabilities in vfio ioctls. This is where passing an argsz for each ioctl comes in handy so we're not limited to a structure, we can link various structures together in a chain. Of course that requires work on both the user and kernel side to pack and unpack, but it leaves a lot of flexibility in extending it. Thanks,
Yeah, that would work as well. I just feel IOMMU UAPI is unlikely to get updated frequently, should be much less than adding new capabilities. I think argsz could be viewed as the version field set by the user, minsz is what kernel current code supports.
So let me summarize the options we have
1. Disallow adding new members to each structure other than reuse padding bits or adding union members at the end.
2. Allow extension of the structures beyond union, but union size has to be fixed with reserved spaces
3. Adopt VFIO argsz scheme, I don't think we need version for each struct anymore. argsz implies the version that user is using assuming UAPI data is extension only.
Jean, Eric, any comments? My preference is #1. In the apocalyptic event when we run out of padding, perhaps we can introduce a new API_v2 :)
Quoted 2 lines
Alex
[Jacob Pan]
Re: [PATCH 3/3] iommu/uapi: Add helper function for size lookup
Hi Jacob,
Quoted 221 lines · Jacob Pan
Quoted 177 lines · On Mon, 3 Feb 2020 14:12:36 -0700 Alex Williamson <[email protected]>…
Quoted 174 lines · On Mon, 3 Feb 2020 12:41:43 -0800 Jacob Pan <[email protected]>…
Quoted 138 lines · On Mon, 3 Feb 2020 11:27:08 -0700 Alex Williamson <[email protected]>…
Quoted 125 lines · On Fri, 31 Jan 2020 15:51:25 -0800 Jacob Pan <[email protected]>…
Hi Alex,
Sorry I missed this part in the previous reply. Comments below.Quoted 4 lines · On Wed, 29 Jan 2020 15:19:51 -0700 Alex Williamson…
Also, is the 12-bytes of padding in struct
iommu_gpasid_bind_dataexcessive with this new versioning scheme? Per rule #2 I'm not sure if we're allowed to repurpose those padding bytes,We can still use the padding bytes as long as there is a new flag bit to indicate the validity of the new filed within the padding. I should have made it clear in rule #2 when mentioning the flags bits. Should define what extension constitutes. How about this?
"
* 2. Data structures are open to extension but closed to
modification.
* Extension should leverage the padding bytes first where a
new
* flag bit is required to indicate the validity of each new
member.
* The above rule for padding bytes also applies to adding
new union
* members.
* After padding bytes are exhausted, new fields must be
added at the
* end of each data structure with 64bit alignment. Flag bits
can be
* added without size change but existing ones cannot be
altered. *
"
So if we add new field by doing re-purpose of padding bytes, size lookup result will remain the same. New code would recognize the new flag, old code stays the same.VFIO layer checks for UAPI compatibility and size to copy, version sanity check and flag usage are done in the IOMMU code.
Quoted 4 lines
but if we add
fields to the end of the structure as the scheme suggests, we're stuck with not being able to expand the union for new fields.Good point, it does sound contradictory. I hope the rewritten rule #2 address that.
Adding data after the union should be extremely rare. Do you see any issues with the example below?
offsetofend()can still find the right size.
e.g.
V1struct iommu_gpasid_bind_data { __u32 version; #define IOMMU_PASID_FORMAT_INTEL_VTD 1 __u32 format; #define IOMMU_SVA_GPASID_VAL (1 << 0) /* guest PASID valid */ __u64 flags; __u64 gpgd; __u64 hpasid; __u64 gpasid; __u32 addr_width; __u8 padding[12]; /* Vendor specific data */ union { struct iommu_gpasid_bind_data_vtd vtd; }; };const static int iommu_uapi_data_size[NR_IOMMU_UAPI_TYPE][IOMMU_UAPI_VERSION] = { /*
IOMMU_UAPI_BIND_GPASID*/ {offsetofend(struct iommu_gpasid_bind_data, vtd)}, ...
};V2, Add
new_memberat the end (forget padding for now).struct iommu_gpasid_bind_data { __u32 version; #define IOMMU_PASID_FORMAT_INTEL_VTD 1 __u32 format; #define IOMMU_SVA_GPASID_VAL (1 << 0) /* guest PASID valid */ #define IOMMU_NEW_MEMBER_VAL (1 << 1) /* new member added */ __u64 flags; __u64 gpgd; __u64 hpasid; __u64 gpasid; __u32 addr_width; __u8 padding[12]; /* Vendor specific data */ union { struct iommu_gpasid_bind_data_vtd vtd; }; __u64 new_member; };const static int iommu_uapi_data_size[NR_IOMMU_UAPI_TYPE][IOMMU_UAPI_VERSION] = { /*
IOMMU_UAPI_BIND_GPASID*/
{offsetofend(struct iommu_gpasid_bind_data, vtd), offsetofend(struct
iommu_gpasid_bind_data,new_member)},};
V3, Add smmu to the union,larger than vtd
struct iommu_gpasid_bind_data { __u32 version; #define IOMMU_PASID_FORMAT_INTEL_VTD 1 #define IOMMU_PASID_FORMAT_INTEL_SMMU 2 __u32 format; #define IOMMU_SVA_GPASID_VAL (1 << 0) /* guest PASID valid */ #define IOMMU_NEW_MEMBER_VAL (1 << 1) /* new member added */ #define IOMMU_SVA_SMMU_SUPP (1 << 2) /* SMMU data supported */ __u64 flags; __u64 gpgd; __u64 hpasid; __u64 gpasid; __u32 addr_width; __u8 padding[12]; /* Vendor specific data */ union { struct iommu_gpasid_bind_data_vtd vtd; struct iommu_gpasid_bind_data_smmu smmu; }; __u64 new_member; };Quoted 9 lines
const static int iommu_uapi_data_size[NR_IOMMU_UAPI_TYPE][IOMMU_UAPI_VERSION] = {
/*IOMMU_UAPI_BIND_GPASID*/ {offsetofend(struct iommu_gpasid_bind_data,vtd),offsetofend(struct iommu_gpasid_bind_data, new_member),offsetofend(struct iommu_gpasid_bind_data, new_member)},
...
};Quoted 6 lines
How are you not breaking rule #3, "Versions are backward compatible" with this? If the kernel is at version 3 and userspace is at version 2 then
new_memberexists at different offsets of the structure. The kernelsiommu_uapi_data_sizefor V2 changed between version 2 and 3. Thanks,You are right. if we want to add new member to the end of the structure as well as expanding union, I think we have to fix the size of the union. Would this work? (just an example for one struct)
(quoted diff) +0
-0@@ -344,6 +348,11 @@ struct iommu_gpasid_bind_data_vtd { * @gpasid: Process address space ID used for the guest mm inguest IOMMU
* @addr_width: Guest virtual address width
* @padding: Reserved for future use (should be zero)
+ * @dummy Reserve space for vendor specific data in the
union. New
+ * members added to the union cannot exceed the size of
dummy.
+ * The fixed size union is needed to allow further
expansion
+ * after the end of the union while still maintain
backward
+ * compatibility.
* @vtd: Intel VT-d specific data
*
* Guest to host PASID mapping can be an identity or non-identity,where guest @@ -365,6 +374,7 @@ struct iommu_gpasid_bind_data { __u8 padding[12]; /* Vendor specific data */ union {+
__u8dummy[128];Quoted 8 lines
Quoted 3 lines
struct iommu_gpasid_bind_data_vtd vtd; };};
It's not the most space efficient thing and we're just guessing at what might need to be added into that union in future, but it works... until it doesn't ;) One might also argue that we could inflate the padding field even further to serve the same purpose.
That was our initial intention, the padding field is already inflated to accommodate any foreseeable extensions :)
Extensions beyond union was deemed unlikely that is why we use the union at the end.
The intention of this patchset is not to change that, but rather clarify and simplify the version checking.
Quoted 10 lines
The only other route I can think of would be to let the user specify the offset of the variable size data from the start of the structure, for example similar to how we're laying out vfio migration region or how we do capabilities in vfio ioctls. This is where passing an argsz for each ioctl comes in handy so we're not limited to a structure, we can link various structures together in a chain. Of course that requires work on both the user and kernel side to pack and unpack, but it leaves a lot of flexibility in extending it. Thanks,
Yeah, that would work as well. I just feel IOMMU UAPI is unlikely to get updated frequently, should be much less than adding new capabilities. I think argsz could be viewed as the version field set by the user, minsz is what kernel current code supports.
So let me summarize the options we have
1. Disallow adding new members to each structure other than reuse padding bits or adding union members at the end.
2. Allow extension of the structures beyond union, but union size has to be fixed with reserved spaces
3. Adopt VFIO argsz scheme, I don't think we need version for each struct anymore. argsz implies the version that user is using assuming UAPI data is extension only.
Quoted 2 lines
Jean, Eric, any comments? My preference is #1. In the apocalyptic event when we run out of padding, perhaps we can introduce a new
API_v2:)
I had #1 in mind too.
Thanks
Eric
Quoted 6 lines
Quoted 3 lines
Alex
[Jacob Pan]
Re: [PATCH 1/3] iommu/uapi: Define uapi version and capabilities
In reply to Jacob PanHi Jacob,
Quoted 14 lines · Jacob Pan
Define a unified UAPI version to be used for compatibility checks between user and kernel.
Signed-off-by Liu Yi L <[email protected]>Signed-off-by Jacob Pan <[email protected]>--- include/uapi/linux/iommu.h | 48 ++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 48 insertions(+)include/uapi/linux/iommu.h +0
-0
include/uapi/linux/iommu.h line 17
Quoted 10 lines
include/uapi/linux/iommu.h +7
-0@@ -8,6 +8,54 @@#include <linux/types.h> +/** +* Current version of the IOMMU user API. This is intended for query +* between user and kernel to determine compatible data structures. +* +* Having a single UAPI version to govern the user-kernel data structures +* makes compatibility check straightforward. On the contrary, supporting +* combinations of multiple versions of the data can be a nightmare.
I would rather put the above justification in the commit msg and not here.
include/uapi/linux/iommu.h line 21
Quoted 4 lines
include/uapi/linux/iommu.h +4
-0+* +* UAPI version can be bumped up with the following rules: +* 1. All data structures passed between user and kernel space share +* the same version number. i.e. any extension to to any structure
s/to to/to
include/uapi/linux/iommu.h line 22
Quoted 1 line
include/uapi/linux/iommu.h +1
-0+* results in version bump up.
in a version number increment?
Quoted 29 lines
include/uapi/linux/iommu.h +28
-0+* +* 2. Data structures are open to extension but closed to modification.> + * New fields must be added at the end of each data structure with +* 64bit alignment. Flag bits can be added without size change but +* existing ones cannot be altered. +* +* 3. Versions are backward compatible. +* +* 4. Version to size lookup is supported by kernel internal API for each +* API function type. @version is mandatory for new data structures +* and must be at the beginning with type of __u32. +*/ +#define IOMMU_UAPI_VERSION 1 +static inline int iommu_get_uapi_version (void ) +{ +return IOMMU_UAPI_VERSION ; +} + +/* +* Supported UAPI features that can be reported to user space. +* These types represent the capability available in the kernel. +* +* REVISIT: UAPI version also implies the capabilities. Should we +* report them explicitly? +*/ +enum IOMMU_UAPI_DATA_TYPES { +IOMMU_UAPI_BIND_GPASID , +IOMMU_UAPI_CACHE_INVAL , +IOMMU_UAPI_PAGE_RESP ,
include/uapi/linux/iommu.h line 61
Quoted 10 lines
include/uapi/linux/iommu.h +7
-0+ NR_IOMMU_UAPI_TYPE, +}; + +#define IOMMU_UAPI_CAP_MASK ((1 << IOMMU_UAPI_BIND_GPASID) | \ + (1 << IOMMU_UAPI_CACHE_INVAL) | \ + (1 << IOMMU_UAPI_PAGE_RESP)) +#define IOMMU_FAULT_PERM_READ (1 << 0)/* read */ #define IOMMU_FAULT_PERM_WRITE (1 << 1)/* write */ #define IOMMU_FAULT_PERM_EXEC (1 << 2)/* exec */
Thanks
Eric
Re: [PATCH 1/3] iommu/uapi: Define uapi version and capabilities
Quoted 20 lines · On Thu, 6 Feb 2020 11:14:27 +0100 Auger Eric <[email protected]> wrote:
Hi Jacob,
Quoted 18 lines · Jacob Pan
Define a unified UAPI version to be used for compatibility checks between user and kernel.
Signed-off-by Liu Yi L <[email protected]>Signed-off-by Jacob Pan <[email protected]>--- include/uapi/linux/iommu.h | 48++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 48 insertions(+)
include/uapi/linux/iommu.h +0
-0@@ -8,6 +8,54 @@#include <linux/types.h>
Quoted 13 lines
Quoted 11 lines
include/uapi/linux/iommu.h +2
-0+/** +* Current version of the IOMMU user API. This is intended for query
+ * between user and kernel to determine compatible data structures.
+ *
+ * Having a single UAPI version to govern the user-kernel data structures
+ * makes compatibility check straightforward. On the contrary, supporting
+ * combinations of multiple versions of the data can be a nightmare.I would rather put the above justification in the commit msg and not here.
make sense.
Quoted 7 lines
Quoted 6 lines
+ *
+ * UAPI version can be bumped up with the following rules:
+ * 1. All data structures passed between user and kernel space share
+ * the same version number. i.e. any extension to to any
structures/to to/to
will fix.
Quoted 2 lines
Quoted 1 line
+ * results in version bump up.
in a version number increment?
sounds good, more specific.
Quoted 50 lines
Quoted 41 lines
+ *
+ * 2. Data structures are open to extension but closed to
modification.> + * New fields must be added at the end of each
data structure with
+ * 64bit alignment. Flag bits can be added without size change
but
+ * existing ones cannot be altered.
+ *
+ * 3. Versions are backward compatible.
+ *
+ * 4. Version to size lookup is supported by kernel internal API for each
+ * API function type. @version is mandatory for new data
structures
+ * and must be at the beginning with type of__u32.
+ */include/uapi/linux/iommu.h +21
-0+#define IOMMU_UAPI_VERSION 1 +static inline int iommu_get_uapi_version (void ) +{ +return IOMMU_UAPI_VERSION ; +} + +/* +* Supported UAPI features that can be reported to user space. +* These types represent the capability available in the kernel. +* +* REVISIT: UAPI version also implies the capabilities. Should we +* report them explicitly? +*/ +enum IOMMU_UAPI_DATA_TYPES { +IOMMU_UAPI_BIND_GPASID , +IOMMU_UAPI_CACHE_INVAL , +IOMMU_UAPI_PAGE_RESP , +NR_IOMMU_UAPI_TYPE , +}; + +#define IOMMU_UAPI_CAP_MASK ((1 << IOMMU_UAPI_BIND_GPASID)| \
include/uapi/linux/iommu.h +1
-0+ (1 <<IOMMU_UAPI_CACHE_INVAL )| \
include/uapi/linux/iommu.h line 61
Quoted 5 lines
include/uapi/linux/iommu.h +2
-0+ (1 << IOMMU_UAPI_PAGE_RESP)) +#define IOMMU_FAULT_PERM_READ (1 << 0)/* read */ #define IOMMU_FAULT_PERM_WRITE (1 << 1)/* write */ #define IOMMU_FAULT_PERM_EXEC (1 << 2)/* exec */ Thanks
Eric
Re: [PATCH 3/3] iommu/uapi: Add helper function for size lookup
In reply to Jacob PanQuoted 14 lines · Jacob Pan
Yeah, that would work as well. I just feel IOMMU UAPI is unlikely to get updated frequently, should be much less than adding new capabilities. I think argsz could be viewed as the version field set by the user, minsz is what kernel current code supports.
So let me summarize the options we have
1. Disallow adding new members to each structure other than reuse padding bits or adding union members at the end.
2. Allow extension of the structures beyond union, but union size has to be fixed with reserved spaces
3. Adopt VFIO argsz scheme, I don't think we need version for each struct anymore. argsz implies the version that user is using assuming UAPI data is extension only.
Quoted 2 lines
Jean, Eric, any comments? My preference is #1. In the apocalyptic event when we run out of padding, perhaps we can introduce a new
API_v2:)
I agree, new extensions will most likely want to extend the vendor specific structures at the end rather than introduce new common fields, so I prefer #1 which avoids fixing the union size.
Thanks,
Jean