xen/memory: Introduce a hypercall to provide unallocated space
RFC PATCH 1/1 xen/memory: Introduce a hypercall to provide unallocated space
From: Oleksandr Tyshchenko <[email protected]>
Add XENMEM_get_unallocated_space hypercall which purpose is to query hypervisor to find regions of guest physical address space which are unused and can be used to create grant/foreign mappings instead of wasting real pages from the domain memory for establishing these mappings. The problem with the current Linux on Xen on Arm behaviour is if we want to map some guest memory regions in advance or to perform cache mappings in the backend we might run out of memory in the host (see XSA-300). This of course, depends on the both host and guest memory sizes.
The "unallocated space" can't be figured out precisely by the domain on Arm without hypervisor involvement:
- not all device I/O regions are known by the time domain starts creating grant/foreign mappings
- the Dom0 is not aware of memory regions used for the identity mappings needed for the PV drivers to work
In both cases we might end up re-using these regions by
a mistake. So, the hypervisor which maintains the P2M for the domain is in the best position to provide "unallocated space".
The arch code is in charge of finding these regions and filling in corresponding array in new helper arch_get_unallocated_space().
This patch implements common and Arm parts, the x86 specific bits are left uniplemented for now.
---
Current patch is based on the latest staging branch:
73c932d tools/libxc: use uint32_t for pirq in xc_domain_irq_permission and also available at:
https://github.com/otyshchenko1/xen/commits/map_opt_ml1
The corresponding Linux changes you can find at:
https://github.com/otyshchenko1/linux/commits/map_opt_ml1
I am going to publish them soon.
--- tools/flask/policy/modules/dom0.te | 2 +- xen/arch/arm/mm.c | 18 ++++++++++++ xen/common/memory.c | 56 +++++++++++++++++++++++++++++++++++++ xen/include/asm-arm/mm.h | 4 +++ xen/include/asm-x86/mm.h | 8 ++++++ xen/include/public/memory.h | 37 +++++++++++++++++++++++- xen/include/xsm/dummy.h | 6 ++++ xen/include/xsm/xsm.h | 6 ++++ xen/xsm/dummy.c | 1 + xen/xsm/flask/hooks.c | 6 ++++ xen/xsm/flask/policy/access_vectors | 2 ++ 11 files changed, 144 insertions(+), 2 deletions(-)
tools/flask/policy/modules/dom0.te +1 -1
@@ -39,7 +39,7 @@ allow dom0_t dom0_t:domain { }; allow dom0_t dom0_t:domain2 { set_cpu_policy gettsc settsc setscheduler set_vnumainfo - get_vnumainfo psr_cmt_op psr_alloc get_cpu_policy + get_vnumainfo psr_cmt_op psr_alloc get_cpu_policy get_unallocated_space }; allow dom0_t dom0_t:resource { add remove };
xen/arch/arm/mm.c +18 -0
@@ -1635,6 +1635,24 @@ unsigned long get_upper_mfn_bound(void)return max_page -1 ; } +#define GPFN_ALIGN_TO_GB (x) (((x) + (1UL << 18) - 1) & (~((1UL << 18) - 1))) + +int arch_get_unallocated_space (struct domain *d, +struct xen_unallocated_region *regions, +unsigned int *nr_regions) +{ +/* +* Everything not mapped into P2M could be treated as unused. Taking into +* the account that we have a lot of unallocated space above max_mapped_gfn +* and in order to simplify things, just provide a single 8GB memory +* region aligned to 1GB boundary for now. +*/ + regions->start_gpfn =GPFN_ALIGN_TO_GB (domain_get_maximum_gpfn (d) +1 ); + regions->nr_gpfns = (1UL <<18 ) *8 ; + *nr_regions =1 ; + +return 0 ; +}/* * Local variables: * mode: C
xen/common/memory.c +56 -0
62 lines not on this page · raw message
xen/include/asm-arm/mm.h +4 -0
@@ -359,6 +359,10 @@ void clear_and_clean_page(struct page_info *page);unsigned int arch_get_dma_bitsize (void ); +int arch_get_unallocated_space (struct domain *d, +struct xen_unallocated_region *regions, +unsigned int *nr_regions); +#endif /* __ARCH_ARM_MM__ */ /* * Local variables:
xen/include/asm-x86/mm.h +8 -0
@@ -652,4 +652,12 @@ static inline bool arch_mfn_in_directmap(unsigned long mfn) return mfn <= (virt_to_mfn(eva - 1) + 1); } +static inline +int arch_get_unallocated_space (struct domain *d, +struct xen_unallocated_region *regions, +unsigned int *nr_regions) +{ +return -EOPNOTSUPP ; +} +#endif /* __ASM_X86_MM_H__ */
xen/include/public/memory.h +36 -1
@@ -739,7 +739,42 @@ struct xen_vnuma_topology_info {typedef struct xen_vnuma_topology_info xen_vnuma_topology_info_t ;DEFINE_XEN_GUEST_HANDLE (xen_vnuma_topology_info_t); -/* Next available subop number is 29 */ +/* +* Get the unallocated space (regions of guest physical address space which +* are unused) and can be used to create grant/foreign mappings. +*/ +#define XENMEM_get_unallocated_space 29 +struct xen_unallocated_region { +xen_pfn_t start_gpfn; +xen_ulong_t nr_gpfns; +}; +typedef struct xen_unallocated_region xen_unallocated_region_t ; +DEFINE_XEN_GUEST_HANDLE (xen_unallocated_region_t); + +#define XEN_MAX_UNALLOCATED_REGIONS 32 + +struct xen_get_unallocated_space { +/* IN - Which domain to provide unallocated space for */ +domid_t domid; + +/* +* IN/OUT - As an IN parameter number of memory regions which +* can be written to the buffer (maximum size of the array) +* As OUT parameter number of memory regions which +* have been written to the buffer +*/ +unsigned int nr_regions; + +/* +* OUT - An array of memory regions, the regions must be placed in +* ascending order, there must be no overlap between them. +*/ +XEN_GUEST_HANDLE (xen_unallocated_region_t) buffer; +}; +typedef struct xen_get_unallocated_space xen_get_unallocated_space_t ; +DEFINE_XEN_GUEST_HANDLE (xen_get_unallocated_space_t); + +/* Next available subop number is 30 */ #endif /* __XEN_PUBLIC_MEMORY_H__ */
xen/include/xsm/dummy.h +6 -0
@@ -772,3 +772,9 @@ static XSM_INLINE int xsm_domain_resource_map(XSM_DEFAULT_ARG struct domain *d)XSM_ASSERT_ACTION (XSM_DM_PRIV );return xsm_default_action (action, current->domain, d); } + +static XSM_INLINE int xsm_get_unallocated_space(XSM_DEFAULT_ARG struct domain *d) +{ +XSM_ASSERT_ACTION (XSM_HOOK ); +return xsm_default_action (action, current->domain, d); +}
xen/include/xsm/xsm.h +6 -0
@@ -180,6 +180,7 @@ struct xsm_operations {int (*dm_op) (struct domain *d);int (*xen_version) (uint32_t cmd);int (*domain_resource_map) (struct domain *d); +int (*get_unallocated_space) (struct domain *d);#ifdef CONFIG_ARGO int (*argo_enable) (const struct domain *d); int (*argo_register_single_source) (const struct domain *d,
@@ -701,6 +702,11 @@ static inline int xsm_domain_resource_map(xsm_default_t def, struct domain *d)return xsm_ops->domain_resource_map (d); } +static inline int xsm_get_unallocated_space (xsm_default_t def,struct domain *d) +{ +return xsm_ops->get_unallocated_space (d); +} +#ifdef CONFIG_ARGO static inline int xsm_argo_enable (const struct domain *d) {
xen/xsm/dummy.c +1 -0
@@ -151,6 +151,7 @@ void __init xsm_fixup_ops (struct xsm_operations *ops)set_to_dummy_if_null (ops, dm_op);set_to_dummy_if_null (ops, xen_version);set_to_dummy_if_null (ops, domain_resource_map); +set_to_dummy_if_null (ops, get_unallocated_space);#ifdef CONFIG_ARGO set_to_dummy_if_null (ops, argo_enable);set_to_dummy_if_null (ops, argo_register_single_source);
xen/xsm/flask/hooks.c +6 -0
@@ -1715,6 +1715,11 @@ static int flask_domain_resource_map(struct domain *d)return current_has_perm (d,SECCLASS_DOMAIN2 ,DOMAIN2__RESOURCE_MAP ); } +static int flask_get_unallocated_space (struct domain *d) +{ +return current_has_perm (d,SECCLASS_DOMAIN2 ,DOMAIN2__GET_UNALLOCATED_SPACE ); +} +#ifdef CONFIG_ARGO static int flask_argo_enable (const struct domain *d) {
@@ -1875,6 +1880,7 @@ static struct xsm_operations flask_ops = { .dm_op = flask_dm_op, .xen_version = flask_xen_version, .domain_resource_map = flask_domain_resource_map, + .get_unallocated_space = flask_get_unallocated_space, #ifdef CONFIG_ARGO .argo_enable = flask_argo_enable, .argo_register_single_source = flask_argo_register_single_source,
xen/xsm/flask/policy/access_vectors +2 -0
@@ -245,6 +245,8 @@ class domain2 resource_map # XEN_DOMCTL_get_cpu_policy get_cpu_policy +# XENMEM_get_unallocated_space + get_unallocated_space } # Similar to class domain, but primarily contains domctls related to HVM domains
Signature · 1 line
2.7.4
Hello all
Quoted 35 lines · Oleksandr Tyshchenko
From: Oleksandr Tyshchenko <[email protected]>
Add
XENMEM_get_unallocated_spacehypercall which purpose is to query hypervisor to find regions of guest physical address space which are unused and can be used to create grant/foreign mappings instead of wasting real pages from the domain memory for establishing these mappings. The problem with the current Linux on Xen on Arm behaviour is if we want to map some guest memory regions in advance or to perform cache mappings in the backend we might run out of memory in the host (see XSA-300). This of course, depends on the both host and guest memory sizes.The "unallocated space" can't be figured out precisely by the domain on Arm without hypervisor involvement:
- not all device I/O regions are known by the time domain starts creating grant/foreign mappings
- the Dom0 is not aware of memory regions used for the identity mappings needed for the PV drivers to work
In both cases we might end up re-using these regions by
a mistake. So, the hypervisor which maintains the P2M for the domain is in the best position to provide "unallocated space".The arch code is in charge of finding these regions and filling in corresponding array in new helper
arch_get_unallocated_space().This patch implements common and Arm parts, the x86 specific bits are left uniplemented for now.
Signed-off-by Oleksandr Tyshchenko <[email protected]>---
Current patch is based on the latest staging branch:
73c932dtools/libxc: useuint32_tfor pirq inxc_domain_irq_permissionand also available at:
https://github.com/otyshchenko1/xen/commits/map_opt_ml1
Quoted 3 lines
The corresponding Linux changes you can find at:
https://github.com/otyshchenko1/linux/commits/map_opt_ml1
I am going to publish them soon.
The Linux changes are already pushed. You can find them at:
1. Small prereq patch
https://lore.kernel.org/lkml/[email protected]/
2. Main patch
https://lore.kernel.org/lkml/[email protected]/
Signature · 3 lines
Regards, Oleksandr Tyshchenko
Quoted 12 lines · Oleksandr Tyshchenko
From: Oleksandr Tyshchenko <[email protected]>
Add
XENMEM_get_unallocated_spacehypercall which purpose is to query hypervisor to find regions of guest physical address space which are unused and can be used to create grant/foreign mappings instead of wasting real pages from the domain memory for establishing these mappings. The problem with the current Linux on Xen on Arm behaviour is if we want to map some guest memory regions in advance or to perform cache mappings in the backend we might run out of memory in the host (see XSA-300). This of course, depends on the both host and guest memory sizes.
Quoted 9 lines
The "unallocated space" can't be figured out precisely by the domain on Arm without hypervisor involvement:
- not all device I/O regions are known by the time domain starts creating grant/foreign mappings
- the Dom0 is not aware of memory regions used for the identity mappings needed for the PV drivers to work
In both cases we might end up re-using these regions by
a mistake. So, the hypervisor which maintains the P2M for the domain is in the best position to provide "unallocated space".
I'm afraid this does not improve the situation.
If a guest follows the advice from XENMEM_get_unallocated_space, and subsequently a new IO or identity region appears, everything will explode, because the "safe area" wasn't actually safe.
The safe range *must* be chosen by the toolstack, because nothing else can do it safely or correctly.
Once a safe range (or ranges) has been chosen, any subsequent action which overlaps with the ranges must be rejected, as it will violate the guarantees provided.
Furthermore, the ranges should be made available to the guest via normal memory map means. On x86, this is via the E820 table, and on ARM I presume the DTB. There is no need for a new hypercall.
Quoted 6 lines
The arch code is in charge of finding these regions and filling in corresponding array in new helper
arch_get_unallocated_space().This patch implements common and Arm parts, the x86 specific bits are left uniplemented for now.
Leaving x86 for now is fine by me.
Quoted 11 lines
xen/arch/arm/mm.c +3
-0@@ -1635,6 +1635,24 @@ unsigned long get_upper_mfn_bound(void)return max_page -1 ; } +#define GPFN_ALIGN_TO_GB (x) (((x) + (1UL << 18) - 1) & (~((1UL << 18) - 1))) + +int arch_get_unallocated_space(struct domain *d,
xen/arch/arm/mm.c unsigned long get_upper_mfn_bound(void) · line 1650
Quoted 10 lines
xen/arch/arm/mm.c +10
-0+ struct xen_unallocated_region *regions, + unsigned int *nr_regions) +{ +/* +* Everything not mapped into P2M could be treated as unused. Taking into +* the account that we have a lot of unallocated space above max_mapped_gfn +* and in order to simplify things, just provide a single 8GB memory +* region aligned to 1GB boundary for now. +*/ + regions->start_gpfn =GPFN_ALIGN_TO_GB (domain_get_maximum_gpfn (d) +1 );
To drive home my earlier point, this is racy and cannot be fixed in Xen.
You don't (and cannot) hold the p2m lock until the guest has acted upon the information provided, so it is guaranteed stale by the time the guest can inspect the results. In the concurrent meantime, a wide range of operations can change max_gpfn.
The only way to get a value the guest can use is to choose it before hand, and have Xen enforce that the nominated range(s) remain safe.
~Andrew
Hi Andrew,
Quoted 23 lines · Andrew Cooper
Quoted 12 lines · Oleksandr Tyshchenko
From: Oleksandr Tyshchenko <[email protected]>
Add
XENMEM_get_unallocated_spacehypercall which purpose is to query hypervisor to find regions of guest physical address space which are unused and can be used to create grant/foreign mappings instead of wasting real pages from the domain memory for establishing these mappings. The problem with the current Linux on Xen on Arm behaviour is if we want to map some guest memory regions in advance or to perform cache mappings in the backend we might run out of memory in the host (see XSA-300). This of course, depends on the both host and guest memory sizes.Quoted 9 lines
The "unallocated space" can't be figured out precisely by the domain on Arm without hypervisor involvement:
- not all device I/O regions are known by the time domain starts creating grant/foreign mappings
- the Dom0 is not aware of memory regions used for the identity mappings needed for the PV drivers to work
In both cases we might end up re-using these regions by
a mistake. So, the hypervisor which maintains the P2M for the domain is in the best position to provide "unallocated space".
Quoted 8 lines
I'm afraid this does not improve the situation.
If a guest follows the advice from
XENMEM_get_unallocated_space, and subsequently a new IO or identity region appears, everything will explode, because the "safe area" wasn't actually safe.The safe range *must* be chosen by the toolstack, because nothing else can do it safely or correctly.
The problem is how do you size it? In particular, a backend may map multiple time the same page (for instance if the page is granted twice).
Quoted 8 lines
Once a safe range (or ranges) has been chosen, any subsequent action which overlaps with the ranges must be rejected, as it will violate the guarantees provided.
Furthermore, the ranges should be made available to the guest via normal memory map means. On x86, this is via the E820 table, and on ARM I presume the DTB. There is no need for a new hypercall.
Device-Tree only works if you have a guest using it. How about ACPI?
To me the hypercall solution at least:
1) Avoid to have to define the region on every single firmware table
2) Allow to easily extend after the guest run
Cheers,
Signature · 1 line
Julien Grall
Quoted 32 lines · Julien Grall
Hi Andrew,
Quoted 29 lines · Andrew Cooper
Quoted 12 lines · Oleksandr Tyshchenko
From: Oleksandr Tyshchenko <[email protected]>
Add
XENMEM_get_unallocated_spacehypercall which purpose is to query hypervisor to find regions of guest physical address space which are unused and can be used to create grant/foreign mappings instead of wasting real pages from the domain memory for establishing these mappings. The problem with the current Linux on Xen on Arm behaviour is if we want to map some guest memory regions in advance or to perform cache mappings in the backend we might run out of memory in the host (see XSA-300). This of course, depends on the both host and guest memory sizes.Quoted 9 lines
The "unallocated space" can't be figured out precisely by the domain on Arm without hypervisor involvement:
- not all device I/O regions are known by the time domain starts creating grant/foreign mappings
- the Dom0 is not aware of memory regions used for the identity mappings needed for the PV drivers to work
In both cases we might end up re-using these regions by
a mistake. So, the hypervisor which maintains the P2M for the domain is in the best position to provide "unallocated space".I'm afraid this does not improve the situation.
If a guest follows the advice from
XENMEM_get_unallocated_space, and subsequently a new IO or identity region appears, everything will explode, because the "safe area" wasn't actually safe.
Quoted 5 lines
Quoted 2 lines
The safe range *must* be chosen by the toolstack, because nothing else can do it safely or correctly.
The problem is how do you size it? In particular, a backend may map multiple time the same page (for instance if the page is granted twice).
The number of mapped grants is limited by the size of the maptrack table in Xen, which is a toolstack input to the domaincreate hypercall. Therefore, the amount of space required is known and bounded.
There are a handful of other frames required in the current ABI (shared info, vcpu info, etc).
The areas where things do become fuzzy is things like foreign mappings, acquire_resource, etc for the control domain, which are effectively unbounded from the domain's point of view.
For those, its entirely fine to say "here 128G of safe mapping space" or so. Even the quantity of mapping dom0 can make is limited by the shadow memory pool and the number of pagetables Xen is willing to expend on the second stage translation tables.
Quoted 11 lines
Quoted 9 lines
Once a safe range (or ranges) has been chosen, any subsequent action which overlaps with the ranges must be rejected, as it will violate the guarantees provided.
Furthermore, the ranges should be made available to the guest via normal memory map means. On x86, this is via the E820 table, and on ARM I presume the DTB. There is no need for a new hypercall.
Device-Tree only works if you have a guest using it. How about ACPI?
ACPI inherits E820 from x86 (its a trivial format), and UEFI was also based on it.
But whichever... All firmware interfaces have a memory map.
Quoted 2 lines
To me the hypercall solution at least:
1) Avoid to have to define the region on every single firmware table
There is only ever one.
Quoted 1 line
2) Allow to easily extend after the guest run
The safe ranges can't be changed (safely). This is the same problem as needing to know things like your PCI apertures ahead of time, or where the DIMM hotplug regions are.
Having the guest physmap be actually dynamic is the cause of so many bugs (inc security) and misfeatures in Xen. Guests cannot and do no cope with things being fully dynamic, because that's not how real hardware works. What you get is layers and layers of breakage on top of each other, rather than a working system.
The size of mapping space is a limit, just like maxphysaddr, or the PCI apertures, or MMCFG space, etc. You can make it large by default (as it doesn't consume resource when not being used), but any guest OS isn't going to tolerate it morphing during runtime.
~Andrew
Quoted 46 lines · Andrew Cooper
Quoted 32 lines · Julien Grall
Hi Andrew,
Quoted 29 lines · Andrew Cooper
Quoted 12 lines · Oleksandr Tyshchenko
From: Oleksandr Tyshchenko <[email protected]>
Add
XENMEM_get_unallocated_spacehypercall which purpose is to query hypervisor to find regions of guest physical address space which are unused and can be used to create grant/foreign mappings instead of wasting real pages from the domain memory for establishing these mappings. The problem with the current Linux on Xen on Arm behaviour is if we want to map some guest memory regions in advance or to perform cache mappings in the backend we might run out of memory in the host (see XSA-300). This of course, depends on the both host and guest memory sizes.Quoted 9 lines
The "unallocated space" can't be figured out precisely by the domain on Arm without hypervisor involvement:
- not all device I/O regions are known by the time domain starts creating grant/foreign mappings
- the Dom0 is not aware of memory regions used for the identity mappings needed for the PV drivers to work
In both cases we might end up re-using these regions by
a mistake. So, the hypervisor which maintains the P2M for the domain is in the best position to provide "unallocated space".I'm afraid this does not improve the situation.
If a guest follows the advice from
XENMEM_get_unallocated_space, and subsequently a new IO or identity region appears, everything will explode, because the "safe area" wasn't actually safe.Quoted 5 lines
Quoted 2 lines
The safe range *must* be chosen by the toolstack, because nothing else can do it safely or correctly.
The problem is how do you size it? In particular, a backend may map multiple time the same page (for instance if the page is granted twice).
The number of mapped grants is limited by the size of the maptrack table in Xen, which is a toolstack input to the domaincreate hypercall. Therefore, the amount of space required is known and bounded.
There are a handful of other frames required in the current ABI (shared info, vcpu info, etc).
Quoted 8 lines
The areas where things do become fuzzy is things like foreign mappings,
acquire_resource, etc for the control domain, which are effectively unbounded from the domain's point of view.For those, its entirely fine to say "here 128G of safe mapping space" or so. Even the quantity of mapping dom0 can make is limited by the shadow memory pool and the number of pagetables Xen is willing to expend on the second stage translation tables.
FWIW, on Arm, we don't have shadow memory pool.
Anyway, it should be possible to give a 128GB "safe range" and let Xen deal with it.
Quoted 11 lines
Quoted 11 lines
Quoted 10 lines
Once a safe range (or ranges) has been chosen, any subsequent action which overlaps with the ranges must be rejected, as it will violate the guarantees provided.
Furthermore, the ranges should be made available to the guest via normal memory map means. On x86, this is via the E820 table, and on ARM I presume the DTB. There is no need for a new hypercall.
Quoted 6 lines
Quoted 1 line
Device-Tree only works if you have a guest using it. How about ACPI?
ACPI inherits E820 from x86 (its a trivial format), and UEFI was also based on it.
But whichever... All firmware interfaces have a memory map.
This will be UEFI memory map. However, I am a bit confused how we can tell the OS the region will be used for grant/foreign mapping. Is it possible to reserved a new type?
Quoted 5 lines
Quoted 3 lines
To me the hypercall solution at least:
1) Avoid to have to define the region on every single firmware tableThere is only ever one.
Why? I could forsee an interest to use the host memory map and therefore we may need to find a few holes for safe regions to use.
Quoted 7 lines
Quoted 2 lines
2) Allow to easily extend after the guest run
The safe ranges can't be changed (safely). This is the same problem as needing to know things like your PCI apertures ahead of time, or where the DIMM hotplug regions are.
Quoted 5 lines
Having the guest physmap be actually dynamic is the cause of so many bugs (inc security) and misfeatures in Xen. Guests cannot and do no cope with things being fully dynamic, because that's not how real hardware works. What you get is layers and layers of breakage on top of each other, rather than a working system.
I would not call it "fully dynamic". Xen can easily know whether a region has ever be allocated before. So long the region has never been allocated, then it should be fine. In fact...
Quoted 5 lines
The size of mapping space is a limit, just like maxphysaddr, or the PCI apertures, or MMCFG space, etc. You can make it large by default (as it doesn't consume resource when not being used), but any guest OS isn't going to tolerate it morphing during runtime.
... I believe the OS may be not aware of the hotplug region until it is actually used.
Anyway, I suggested this approach a few months ago to Oleksandr (see [1]) which BTW you were CCed on ;). My take was that Xen knows about the unallocated space and therefore can make an informed decision without having to allocate insanely large region.
Now if you think that's fine (IIRC Stefano had a preference for that as well). Then we can use the firmware table (assuming we can solve the ACPI question).
At the end of the day, this is not really the interesting bit of the problem. What matter if the OS part where hopefully Linux will be able to use normally the RAM. We may even be able to fix XSA-300!
Cheers,
[1] <[email protected]>
Signature · 1 line
Julien Grall
Hi Andrew, Julien.
@Andrew, I think that arguments you provided in your first answer (why the proposed solution is a potentially racy and not a safe) are valid and reasonable. Thanks for the feedback.
Quoted 110 lines · Julien Grall
Quoted 46 lines · Andrew Cooper
Quoted 32 lines · Julien Grall
Hi Andrew,
Quoted 29 lines · Andrew Cooper
Quoted 12 lines · Oleksandr Tyshchenko
From: Oleksandr Tyshchenko <[email protected]>
Add
XENMEM_get_unallocated_spacehypercall which purpose is to query hypervisor to find regions of guest physical address space which are unused and can be used to create grant/foreign mappings instead of wasting real pages from the domain memory for establishing these mappings. The problem with the current Linux on Xen on Arm behaviour is if we want to map some guest memory regions in advance or to perform cache mappings in the backend we might run out of memory in the host (see XSA-300). This of course, depends on the both host and guest memory sizes.Quoted 9 lines
The "unallocated space" can't be figured out precisely by the domain on Arm without hypervisor involvement:
- not all device I/O regions are known by the time domain starts creating grant/foreign mappings
- the Dom0 is not aware of memory regions used for the identity mappings needed for the PV drivers to work
In both cases we might end up re-using these regions by
a mistake. So, the hypervisor which maintains the P2M for the domain is in the best position to provide "unallocated space".I'm afraid this does not improve the situation.
If a guest follows the advice from
XENMEM_get_unallocated_space, and subsequently a new IO or identity region appears, everything will explode, because the "safe area" wasn't actually safe.Quoted 5 lines
Quoted 2 lines
The safe range *must* be chosen by the toolstack, because nothing else can do it safely or correctly.
The problem is how do you size it? In particular, a backend may map multiple time the same page (for instance if the page is granted twice).
The number of mapped grants is limited by the size of the maptrack table in Xen, which is a toolstack input to the domaincreate hypercall. Therefore, the amount of space required is known and bounded.
There are a handful of other frames required in the current ABI (shared info, vcpu info, etc).
Quoted 8 lines
The areas where things do become fuzzy is things like foreign mappings,
acquire_resource, etc for the control domain, which are effectively unbounded from the domain's point of view.For those, its entirely fine to say "here 128G of safe mapping space" or so. Even the quantity of mapping dom0 can make is limited by the shadow memory pool and the number of pagetables Xen is willing to expend on the second stage translation tables.
FWIW, on Arm, we don't have shadow memory pool.
Anyway, it should be possible to give a 128GB "safe range" and let Xen deal with it.
Quoted 11 lines
Quoted 11 lines
Quoted 10 lines
Once a safe range (or ranges) has been chosen, any subsequent action which overlaps with the ranges must be rejected, as it will violate the guarantees provided.
Furthermore, the ranges should be made available to the guest via normal memory map means. On x86, this is via the E820 table, and on ARM I presume the DTB. There is no need for a new hypercall.
Quoted 6 lines
Quoted 1 line
Device-Tree only works if you have a guest using it. How about ACPI?
ACPI inherits E820 from x86 (its a trivial format), and UEFI was also based on it.
But whichever... All firmware interfaces have a memory map.
This will be UEFI memory map. However, I am a bit confused how we can tell the OS the region will be used for grant/foreign mapping. Is it possible to reserved a new type?
Quoted 5 lines
Quoted 3 lines
To me the hypercall solution at least:
1) Avoid to have to define the region on every single firmware tableThere is only ever one.
Why? I could forsee an interest to use the host memory map and therefore we may need to find a few holes for safe regions to use.
Quoted 7 lines
Quoted 2 lines
2) Allow to easily extend after the guest run
The safe ranges can't be changed (safely). This is the same problem as needing to know things like your PCI apertures ahead of time, or where the DIMM hotplug regions are.
Quoted 5 lines
Having the guest physmap be actually dynamic is the cause of so many bugs (inc security) and misfeatures in Xen. Guests cannot and do no cope with things being fully dynamic, because that's not how real hardware works. What you get is layers and layers of breakage on top of each other, rather than a working system.
I would not call it "fully dynamic". Xen can easily know whether a region has ever be allocated before. So long the region has never been allocated, then it should be fine. In fact...
Quoted 5 lines
The size of mapping space is a limit, just like maxphysaddr, or the PCI apertures, or MMCFG space, etc. You can make it large by default (as it doesn't consume resource when not being used), but any guest OS isn't going to tolerate it morphing during runtime.
Quoted 5 lines
... I believe the OS may be not aware of the hotplug region until it is actually used.
Anyway, I suggested this approach a few months ago to Oleksandr (see [1]) which BTW you were CCed on ;). My take was that Xen knows about the unallocated space and therefore can make an informed decision without having to allocate insanely large region.
Now if you think that's fine (IIRC Stefano had a preference for that as well). Then we can use the firmware table (assuming we can solve the ACPI question).
Well, if new hypercall and, what is more, "the querying hypervisor at runtime to find unused space" model itself is not welcome, I am ok, let's try to create a working system, may we please find a common ground to move this forward (at least on Arm for now, the foreign mapping is the most important question).
I got the proposed idea in general, but I haven't connected all dots yet, some points need clarification.
1. The safe range must be defined/allocated in advance and must remain const during the runtime. The safe range must be chosen by the toolstack. [For the initial implementation we can start with some large value (128GB) as discussed above]
Questions:
- Do we need to inform Xen about that range (via domain_create hypercall, etc)?
- What will be in charge of guaranteeing the safety of that range at runtime (reject new mapping requests with possible overlaps, etc), Xen, toolstack or both?
- Where that range should be located in guest address space, should that range be the same for all domains (how GUEST_GNTTAB_BASE(SIZE) for example) or it should be calculated based on actual guest_ram_base(size) for a particular domain?
- What about a safe range the Dom0 can use itself? Xen should choose it for Dom0 the same way how toolstack chooses it for other domains, correct?
2. The safe range must be provided to domain using the firmware table.
[We can start with the DTB and leave ACPI unimplemented for now, assuming we will be able to solve open questions as discussed above]
Questions:
- Do we need distinguish between foreign and grant mappings at the domain side at all? Can the same safe range be used for all types of mapping?
- How will domain recognize what range can be used for foreign/grant mappings? Do we need to introduce new device-tree bindings for describing the range or it is possible to re-use current bindings (expand the "reg" property under "hypervisor" node, currently it contains gnttab region)?
Quoted 7 lines
At the end of the day, this is not really the interesting bit of the problem. What matter if the OS part where hopefully Linux will be able to use normally the RAM. We may even be able to fix XSA-300!
Cheers,
[1] <[email protected]>
Signature · 3 lines
Regards, Oleksandr Tyshchenko
Quoted 157 lines · Oleksandr
Hi Andrew, Julien.
@Andrew, I think that arguments you provided in your first answer (why the proposed solution is a potentially racy and not a safe) are valid and reasonable.
Thanks for the feedback.Quoted 121 lines · Julien Grall
Quoted 46 lines · Andrew Cooper
Quoted 32 lines · Julien Grall
Hi Andrew,
Quoted 29 lines · Andrew Cooper
Quoted 12 lines · Oleksandr Tyshchenko
From: Oleksandr Tyshchenko <[email protected]>
Add
XENMEM_get_unallocated_spacehypercall which purpose is to query hypervisor to find regions of guest physical address space which are unused and can be used to create grant/foreign mappings instead of wasting real pages from the domain memory for establishing these mappings. The problem with the current Linux on Xen on Arm behaviour is if we want to map some guest memory regions in advance or to perform cache mappings in the backend we might run out of memory in the host (see XSA-300). This of course, depends on the both host and guest memory sizes.Quoted 9 lines
The "unallocated space" can't be figured out precisely by the domain on Arm without hypervisor involvement:
- not all device I/O regions are known by the time domain starts creating grant/foreign mappings
- the Dom0 is not aware of memory regions used for the identity mappings needed for the PV drivers to work
In both cases we might end up re-using these regions by
a mistake. So, the hypervisor which maintains the P2M for the domain is in the best position to provide "unallocated space".I'm afraid this does not improve the situation.
If a guest follows the advice from
XENMEM_get_unallocated_space, and subsequently a new IO or identity region appears, everything will explode, because the "safe area" wasn't actually safe.Quoted 5 lines
Quoted 2 lines
The safe range *must* be chosen by the toolstack, because nothing else can do it safely or correctly.
The problem is how do you size it? In particular, a backend may map multiple time the same page (for instance if the page is granted twice).
The number of mapped grants is limited by the size of the maptrack table in Xen, which is a toolstack input to the domaincreate hypercall. Therefore, the amount of space required is known and bounded.
There are a handful of other frames required in the current ABI (shared info, vcpu info, etc).
Quoted 8 lines
The areas where things do become fuzzy is things like foreign mappings,
acquire_resource, etc for the control domain, which are effectively unbounded from the domain's point of view.For those, its entirely fine to say "here 128G of safe mapping space" or so. Even the quantity of mapping dom0 can make is limited by the shadow memory pool and the number of pagetables Xen is willing to expend on the second stage translation tables.
FWIW, on Arm, we don't have shadow memory pool.
Anyway, it should be possible to give a 128GB "safe range" and let Xen deal with it.
Quoted 13 lines
Quoted 13 lines
Quoted 12 lines
Once a safe range (or ranges) has been chosen, any subsequent action which overlaps with the ranges must be rejected, as it will violate the
guarantees provided.Furthermore, the ranges should be made available to the guest via normal
memory map means. On x86, this is via the E820 table, and on ARM I presume the DTB. There is no need for a new hypercall.Quoted 6 lines
Quoted 1 line
Device-Tree only works if you have a guest using it. How about ACPI?
ACPI inherits E820 from x86 (its a trivial format), and UEFI was also based on it.
But whichever... All firmware interfaces have a memory map.
This will be UEFI memory map. However, I am a bit confused how we can tell the OS the region will be used for grant/foreign mapping. Is it possible to reserved a new type?
Quoted 5 lines
Quoted 3 lines
To me the hypercall solution at least:
1) Avoid to have to define the region on every single firmware tableThere is only ever one.
Why? I could forsee an interest to use the host memory map and therefore we may need to find a few holes for safe regions to use.
Quoted 7 lines
Quoted 2 lines
2) Allow to easily extend after the guest run
The safe ranges can't be changed (safely). This is the same problem as needing to know things like your PCI apertures ahead of time, or where the DIMM hotplug regions are.
Quoted 5 lines
Having the guest physmap be actually dynamic is the cause of so many bugs (inc security) and misfeatures in Xen. Guests cannot and do no cope with things being fully dynamic, because that's not how real hardware works. What you get is layers and layers of breakage on top of each other, rather than a working system.
I would not call it "fully dynamic". Xen can easily know whether a region has ever be allocated before. So long the region has never been allocated, then it should be fine. In fact...
Quoted 5 lines
The size of mapping space is a limit, just like maxphysaddr, or the PCI apertures, or MMCFG space, etc. You can make it large by default (as it doesn't consume resource when not being used), but any guest OS isn't going to tolerate it morphing during runtime.
... I believe the OS may be not aware of the hotplug region until it is actually used.
Quoted 8 lines
Anyway, I suggested this approach a few months ago to Oleksandr (see [1]) which BTW you were CCed on ;). My take was that Xen knows about the unallocated space and therefore can make an informed decision without having to allocate insanely large region.
Now if you think that's fine (IIRC Stefano had a preference for that as well). Then we can use the firmware table (assuming we can solve the ACPI question).
Well, if new hypercall and, what is more, "the querying hypervisor at runtime to find unused space" model itself is not welcome, I am ok, let's try to create a working system,
may we please find a common ground to move this forward (at least on Arm for now, the foreign mapping is the most important question).I got the proposed idea in general, but I haven't connected all dots yet, some points need clarification.
1. The safe range must be defined/allocated in advance and must remain const during the runtime. The safe range must be chosen by the toolstack. [For the initial implementation we can start with some large value (128GB) as discussed above]
Questions:
Quoted 11 lines
- Do we need to inform Xen about that range (via
domain_createhypercall, etc)?
- What will be in charge of guaranteeing the safety of that range at runtime (reject new mapping requests with possible overlaps, etc), Xen, toolstack or both?
- Where that range should be located in guest address space, should that range be the same for all domains (howGUEST_GNTTAB_BASE(SIZE)for example) or it should be calculated based on actualguest_ram_base(size)for a particular domain?
- What about a safe range the Dom0 can use itself? Xen should choose it for Dom0 the same way how toolstack chooses it for other domains, correct?
Brainstorming an idea to answer some of these questions.
Xen needs to be able to choose a safe range without the toolstack's help at least for dom0 and dom0less guests. Could we just have Xen choose the safe range for all domains?
Could it be just as simple as choosing a range 1GB-aligned above whatever is the highest address previously allocated for memory, MMIO, whatever?
We could also have the toolstack provide the info but I wonder if we can get away without it.
Quoted 8 lines
2. The safe range must be provided to domain using the firmware table. [We can start with the DTB and leave ACPI unimplemented for now, assuming we will be able to solve open questions as discussed above]
Questions:
- Do we need distinguish between foreign and grant mappings at the domain side at all? Can the same safe range be used for all types of mapping?
This is a good question. I can't think of a reason why we would need to distinguish between the two from Xen's point of view. Would it make the Linux implementation easier if we distinguished them?
Quoted 4 lines
- How will domain recognize what range can be used for foreign/grant mappings? Do we need to introduce new device-tree bindings for describing the range or it is possible to re-use current bindings (expand the "reg" property under "hypervisor" node, currently it contains gnttab region)?
Do we need a struct page* for the range? We needed it in the past and it is the reason why we currenly use ballooned-out pages: they have a proper struct page* associated with them. pte_special() caused problems.
So, continuing with the assumption that we need a struct page*, then the range needs to be advertised as "memory" to the DomU (part to of the /memory node). But we can reserve the range for foreign/grant-mapping use by adding a reserved-memory binding for it. If we go down this route, we should add a new binding as I don't think we should reuse reserved-memory/xen,shared-memory.txt. It would be very simply to add, just follow the model of xen,shared-memory.txt. (Note that just because the range is part of the DomU /memory node, it doesn't mean it needs to be backed by real memory.)
If we don't need a struct page* then we can do something different using bus ranges and/or MMIO regions.
Hi Stefano,
Thank you for the comments and ideas.
Quoted 150 lines · Stefano Stabellini
Quoted 138 lines · Oleksandr
Hi Andrew, Julien.
@Andrew, I think that arguments you provided in your first answer (why the proposed solution is a potentially racy and not a safe) are valid and reasonable.
Thanks for the feedback.Quoted 103 lines · Julien Grall
Quoted 43 lines · Andrew Cooper
Quoted 27 lines · Julien Grall
Hi Andrew,
Quoted 24 lines · Andrew Cooper
Quoted 12 lines · Oleksandr Tyshchenko
From: Oleksandr Tyshchenko <[email protected]>
Add
XENMEM_get_unallocated_spacehypercall which purpose is to query hypervisor to find regions of guest physical address space which are unused and can be used to create grant/foreign mappings instead of wasting real pages from the domain memory for establishing these mappings. The problem with the current Linux on Xen on Arm behaviour is if we want to map some guest memory regions in advance or to perform cache mappings in the backend we might run out of memory in the host (see XSA-300). This of course, depends on the both host and guest memory sizes.Quoted 9 lines
The "unallocated space" can't be figured out precisely by the domain on Arm without hypervisor involvement:
- not all device I/O regions are known by the time domain starts creating grant/foreign mappings
- the Dom0 is not aware of memory regions used for the identity mappings needed for the PV drivers to work
In both cases we might end up re-using these regions by
a mistake. So, the hypervisor which maintains the P2M for the domain is in the best position to provide "unallocated space".I'm afraid this does not improve the situation.
Quoted 8 lines
Quoted 6 lines
If a guest follows the advice from
XENMEM_get_unallocated_space, and subsequently a new IO or identity region appears, everything will explode, because the "safe area" wasn't actually safe.The safe range *must* be chosen by the toolstack, because nothing else can do it safely or correctly.
The problem is how do you size it? In particular, a backend may map multiple time the same page (for instance if the page is granted twice).
The number of mapped grants is limited by the size of the maptrack table in Xen, which is a toolstack input to the domaincreate hypercall. Therefore, the amount of space required is known and bounded.
There are a handful of other frames required in the current ABI (shared info, vcpu info, etc).
Quoted 8 lines
The areas where things do become fuzzy is things like foreign mappings,
acquire_resource, etc for the control domain, which are effectively unbounded from the domain's point of view.For those, its entirely fine to say "here 128G of safe mapping space" or so. Even the quantity of mapping dom0 can make is limited by the shadow memory pool and the number of pagetables Xen is willing to expend on the second stage translation tables.
FWIW, on Arm, we don't have shadow memory pool.
Anyway, it should be possible to give a 128GB "safe range" and let Xen deal with it.
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Once a safe range (or ranges) has been chosen, any subsequent action which overlaps with the ranges must be rejected, as it will violate the
guarantees provided.Furthermore, the ranges should be made available to the guest via normal
memory map means. On x86, this is via the E820 table, and on ARM I presume the DTB. There is no need for a new hypercall.Quoted 5 lines
Quoted 1 line
Device-Tree only works if you have a guest using it. How about ACPI?
ACPI inherits E820 from x86 (its a trivial format), and UEFI was also based on it.
But whichever... All firmware interfaces have a memory map.
This will be UEFI memory map. However, I am a bit confused how we can tell the OS the region will be used for grant/foreign mapping. Is it possible to reserved a new type?
Quoted 3 lines
Quoted 2 lines
To me the hypercall solution at least:
1) Avoid to have to define the region on every single firmware tableThere is only ever one.
Why? I could forsee an interest to use the host memory map and therefore we may need to find a few holes for safe regions to use.
Quoted 10 lines
Quoted 1 line
2) Allow to easily extend after the guest run
The safe ranges can't be changed (safely). This is the same problem as needing to know things like your PCI apertures ahead of time, or where the DIMM hotplug regions are.
Having the guest physmap be actually dynamic is the cause of so many bugs (inc security) and misfeatures in Xen. Guests cannot and do no cope with things being fully dynamic, because that's not how real hardware works. What you get is layers and layers of breakage on top of each other, rather than a working system.
I would not call it "fully dynamic". Xen can easily know whether a region has ever be allocated before. So long the region has never been allocated, then it should be fine. In fact...
Quoted 4 lines
The size of mapping space is a limit, just like maxphysaddr, or the PCI apertures, or MMCFG space, etc. You can make it large by default (as it doesn't consume resource when not being used), but any guest OS isn't going to tolerate it morphing during runtime.
... I believe the OS may be not aware of the hotplug region until it is actually used.
Quoted 8 lines
Anyway, I suggested this approach a few months ago to Oleksandr (see [1]) which BTW you were CCed on ;). My take was that Xen knows about the unallocated space and therefore can make an informed decision without having to allocate insanely large region.
Now if you think that's fine (IIRC Stefano had a preference for that as well). Then we can use the firmware table (assuming we can solve the ACPI question).
Well, if new hypercall and, what is more, "the querying hypervisor at runtime to find unused space" model itself is not welcome, I am ok, let's try to create a working system,
may we please find a common ground to move this forward (at least on Arm for now, the foreign mapping is the most important question).I got the proposed idea in general, but I haven't connected all dots yet, some points need clarification.
1. The safe range must be defined/allocated in advance and must remain const during the runtime. The safe range must be chosen by the toolstack. [For the initial implementation we can start with some large value (128GB) as discussed above]
Questions:
Quoted 11 lines
- Do we need to inform Xen about that range (via
domain_createhypercall, etc)?
- What will be in charge of guaranteeing the safety of that range at runtime (reject new mapping requests with possible overlaps, etc), Xen, toolstack or both?
- Where that range should be located in guest address space, should that range be the same for all domains (howGUEST_GNTTAB_BASE(SIZE)for example) or it should be calculated based on actualguest_ram_base(size)for a particular domain?
- What about a safe range the Dom0 can use itself? Xen should choose it for Dom0 the same way how toolstack chooses it for other domains, correct?
Quoted 4 lines
Brainstorming an idea to answer some of these questions.
Xen needs to be able to choose a safe range without the toolstack's help at least for dom0 and dom0less guests.
Completely agree.
Quoted 2 lines
Could we just have Xen choose the
safe range for all domains?
As I understand, the region info (base, size) should be known to whoever generates firmware tables (DTB) for the domain in order to insert that range. So, in case if Xen chooses that range, it should be queried by the toolstack anyway (either by new domctl or extending existing one which context would fit). This adds some complexity.
Quoted 4 lines
Could it be just as simple as choosing a range 1GB-aligned above whatever is the highest address previously allocated for memory, MMIO, whatever?
Probably, we could choose a range above max mapped gpfn at the domain creation time.
Currently, we are dealing with platform devices only on Arm and all IO are known in advance, so no new IO are expected at runtime. But, what to do when hotplug comes into play (and new conflicting IO might just appear)? On the other side, we might be faced with the same problem, if we used a predefined static range. With both options (calculated and predefined range) we would need to deny any new conflicting mapping.
Quoted 3 lines
We could also have the toolstack provide the info but I wonder if we can get away without it.
I may not see all pitfalls here, but it seems that having the same predefined range for all domains would simplify things. On Arm we have the following:
#define GUEST_RAM1_BASE xen_mk_ullong(0x0200000000) /* 1016GB of RAM @ 8GB */ #define GUEST_RAM1_SIZE xen_mk_ullong(0xfe00000000)
Now I am wondering, can we limit the GUEST_RAM1_SIZE by 128GB for example in order to reserve a range at the end, would this be acceptable?
So, both Xen and toolstack would see that definition and would be able to insert a range to the generated DTB, Xen's P2M code would be in change of keeping that range in a safe state.
Quoted 11 lines
Quoted 8 lines
2. The safe range must be provided to domain using the firmware table. [We can start with the DTB and leave ACPI unimplemented for now, assuming we will be able to solve open questions as discussed above]
Questions:
- Do we need distinguish between foreign and grant mappings at the domain side at all? Can the same safe range be used for all types of mapping?
This is a good question. I can't think of a reason why we would need to distinguish between the two from Xen's point of view. Would it make the Linux implementation easier if we distinguished them?
I may mistake, but I don't think it would make the Linux implementation easier, both foreign and grants mappings result in same action which is xen_alloc_unpopulated_pages (or alloc_xenballooned_pages). For both mappings the problem with wasting real domain memory on Arm is actual at the moment, although with foreign mapping
for the device emulator use case it becomes more evident.
Quoted 19 lines
Quoted 6 lines
- How will domain recognize what range can be used for foreign/grant mappings? Do we need to introduce new device-tree bindings for describing the range or it is possible to re-use current bindings (expand the "reg" property under "hypervisor" node, currently it contains gnttab region)?
Do we need a struct page* for the range? We needed it in the past and it is the reason why we currenly use ballooned-out pages: they have a proper struct page* associated with them.
pte_special()caused problems. So, continuing with the assumption that we need a struct page*, then the range needs to be advertised as "memory" to the DomU (part to of the /memory node). But we can reserve the range for foreign/grant-mapping use by adding a reserved-memory binding for it. If we go down this route, we should add a new binding as I don't think we should reusereserved-memory/xen,shared-memory.txt. It would be very simply to add, just follow the model of xen,shared-memory.txt. (Note that just because the range is part of the DomU /memory node, it doesn't mean it needs to be backed by real memory.)
Quoted 2 lines
If we don't need a struct page* then we can do something different using bus ranges and/or MMIO regions.
Yes, we need struct page* for the each pfn in the range. But, I think, this can be achieved even without describing the range as a part of memory/reserved-memory nodes. Fortunately, the Linux ZONE_DEVICE facility can help with that, actually the "unpopulated-alloc" is based on this feature, please see drivers/xen/unpopulated-alloc.c:fill_list()
and https://lore.kernel.org/lkml/[email protected]/ for details.
Signature · 3 lines
Regards, Oleksandr Tyshchenko
Hi Julien, Stefano.
Quoted 241 lines · Oleksandr
Hi Stefano,
Thank you for the comments and ideas.
Quoted 155 lines · Stefano Stabellini
Quoted 139 lines · Oleksandr
Hi Andrew, Julien.
@Andrew, I think that arguments you provided in your first answer (why the
proposed solution is a potentially racy and not a safe) are valid and reasonable.
Thanks for the feedback.Quoted 103 lines · Julien Grall
Quoted 43 lines · Andrew Cooper
Quoted 27 lines · Julien Grall
Hi Andrew,
Quoted 24 lines · Andrew Cooper
Quoted 12 lines · Oleksandr Tyshchenko
From: Oleksandr Tyshchenko <[email protected]>
Add
XENMEM_get_unallocated_spacehypercall which purpose is to query hypervisor to find regions of guest physical address space which are unused and can be used to create grant/foreign mappings instead of wasting real pages from the domain memory for establishing these mappings. The problem with the current Linux on Xen on Arm behaviour is if we want to map some guest memory regions in advance or to perform cache mappings in the backend we might run out of memory in the host (see XSA-300). This of course, depends on the both host and guest memory sizes.Quoted 9 lines
The "unallocated space" can't be figured out precisely by the domain on Arm without hypervisor involvement:
- not all device I/O regions are known by the time domain starts creating grant/foreign mappings
- the Dom0 is not aware of memory regions used for the identity mappings needed for the PV drivers to work
In both cases we might end up re-using these regions by
a mistake. So, the hypervisor which maintains the P2M for the domain is in the best position to provide "unallocated space".I'm afraid this does not improve the situation.
Quoted 8 lines
Quoted 6 lines
If a guest follows the advice from
XENMEM_get_unallocated_space, and subsequently a new IO or identity region appears, everything will explode, because the "safe area" wasn't actually safe.The safe range *must* be chosen by the toolstack, because nothing else can do it safely or correctly.
The problem is how do you size it? In particular, a backend may map multiple time the same page (for instance if the page is granted twice).
The number of mapped grants is limited by the size of the maptrack table in Xen, which is a toolstack input to the domaincreate hypercall. Therefore, the amount of space required is known and bounded.
There are a handful of other frames required in the current ABI (shared info, vcpu info, etc).
Quoted 8 lines
The areas where things do become fuzzy is things like foreign mappings,
acquire_resource, etc for the control domain, which are effectively unbounded from the domain's point of view.For those, its entirely fine to say "here 128G of safe mapping space" or so. Even the quantity of mapping dom0 can make is limited by the shadow memory pool and the number of pagetables Xen is willing to expend on the second stage translation tables.
FWIW, on Arm, we don't have shadow memory pool.
Anyway, it should be possible to give a 128GB "safe range" and let Xen deal with it.
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Once a safe range (or ranges) has been chosen, any subsequent action which overlaps with the ranges must be rejected, as it will violate the
guarantees provided.Furthermore, the ranges should be made available to the guest via normal
memory map means. On x86, this is via the E820 table, and on ARM I presume the DTB. There is no need for a new hypercall.Quoted 5 lines
Quoted 1 line
Device-Tree only works if you have a guest using it. How about ACPI?
ACPI inherits E820 from x86 (its a trivial format), and UEFI was also based on it.
But whichever... All firmware interfaces have a memory map.
This will be UEFI memory map. However, I am a bit confused how we can tell the OS the region will be used for grant/foreign mapping. Is it possible to reserved a new type?
Quoted 3 lines
Quoted 2 lines
To me the hypercall solution at least:
1) Avoid to have to define the region on every single firmware tableThere is only ever one.
Why? I could forsee an interest to use the host memory map and therefore we may need to find a few holes for safe regions to use.
Quoted 10 lines
Quoted 1 line
2) Allow to easily extend after the guest run
The safe ranges can't be changed (safely). This is the same problem as needing to know things like your PCI apertures ahead of time, or where the DIMM hotplug regions are.
Having the guest physmap be actually dynamic is the cause of so many bugs (inc security) and misfeatures in Xen. Guests cannot and do no cope with things being fully dynamic, because that's not how real hardware works. What you get is layers and layers of breakage on top of each other, rather than a working system.
I would not call it "fully dynamic". Xen can easily know whether a region has ever be allocated before. So long the region has never been allocated, then it should be fine. In fact...
Quoted 4 lines
The size of mapping space is a limit, just like maxphysaddr, or the PCI apertures, or MMCFG space, etc. You can make it large by default (as it doesn't consume resource when not being used), but any guest OS isn't going to tolerate it morphing during runtime.
... I believe the OS may be not aware of the hotplug region until it is actually used.
Quoted 8 lines
Anyway, I suggested this approach a few months ago to Oleksandr (see [1]) which BTW you were CCed on ;). My take was that Xen knows about the unallocated space and therefore can make an informed decision without having to allocate insanely large region.
Now if you think that's fine (IIRC Stefano had a preference for that as well). Then we can use the firmware table (assuming we can solve the ACPI question).
Well, if new hypercall and, what is more, "the querying hypervisor at runtime to find unused space" model itself is not welcome, I am ok, let's try to create a working system,
may we please find a common ground to move this forward (at least on Arm for now, the foreign mapping is the most important question).I got the proposed idea in general, but I haven't connected all dots yet, some points need clarification.
1. The safe range must be defined/allocated in advance and must remain const during the runtime. The safe range must be chosen by the toolstack. [For the initial implementation we can start with some large value (128GB) as discussed above]
Questions:
Quoted 15 lines
- Do we need to inform Xen about that range (via
domain_createhypercall,
etc)?
- What will be in charge of guaranteeing the safety of that range at runtime
(reject new mapping requests with possible overlaps, etc), Xen, toolstack or both?
- Where that range should be located in guest address space, should that range
be the same for all domains (howGUEST_GNTTAB_BASE(SIZE)for example) or it should be calculated based on actualguest_ram_base(size)for a particular domain?
- What about a safe range the Dom0 can use itself? Xen should choose it for
Dom0 the same way how toolstack chooses it for other domains, correct?Quoted 4 lines
Brainstorming an idea to answer some of these questions.
Xen needs to be able to choose a safe range without the toolstack's help at least for dom0 and dom0less guests.
Completely agree.
Quoted 2 lines
Could we just have Xen choose the
safe range for all domains?As I understand, the region info (base, size) should be known to whoever generates firmware tables (DTB) for the domain in order to insert that range. So, in case if Xen chooses that range, it should be queried by the toolstack anyway (either by new domctl or extending existing one which context would fit). This adds some complexity.
Quoted 4 lines
Could it be just as simple as choosing a range 1GB-aligned above whatever is the highest address previously allocated for memory, MMIO, whatever?
Probably, we could choose a range above max mapped gpfn at the domain creation time.
Currently, we are dealing with platform devices only on Arm and all IO are known in advance, so no new IO are expected at runtime. But, what to do when hotplug comes into play (and new conflicting IO might just appear)? On the other side, we might be faced with the same problem, if we used a predefined static range. With both options (calculated and predefined range) we would need to deny any new conflicting mapping.Quoted 3 lines
We could also have the toolstack provide the info but I wonder if we can get away without it.
I may not see all pitfalls here, but it seems that having the same predefined range for all domains would simplify things. On Arm we have the following:
#define GUEST_RAM1_BASE
xen_mk_ullong(0x0200000000)/* 1016GB of RAM @ 8GB */
#define GUEST_RAM1_SIZExen_mk_ullong(0xfe00000000)Now I am wondering, can we limit the
GUEST_RAM1_SIZEby 128GB for example in order to reserve a range at the end, would this be acceptable?
So, both Xen and toolstack would see that definition and would be able to insert a range to the generated DTB, Xen's P2M code would be in change of keeping that range in a safe state.Quoted 4 lines
Quoted 4 lines
2. The safe range must be provided to domain using the firmware table. [We can start with the DTB and leave ACPI unimplemented for now, assuming we will be able to solve open questions as discussed above]
Quoted 8 lines
Quoted 5 lines
Questions:
- Do we need distinguish between foreign and grant mappings at the domain side
at all? Can the same safe range be used for all types of mapping?This is a good question. I can't think of a reason why we would need to distinguish between the two from Xen's point of view. Would it make the Linux implementation easier if we distinguished them?
I may mistake, but I don't think it would make the Linux implementation easier, both foreign and grants mappings result in same action which is
xen_alloc_unpopulated_pages(oralloc_xenballooned_pages). For both mappings the problem with wasting real domain memory on Arm is actual at the moment, although with foreign mapping
for the device emulator use case it becomes more evident.Quoted 20 lines
Quoted 7 lines
- How will domain recognize what range can be used for foreign/grant mappings?
Do we need to introduce new device-tree bindings for describing the range or it is possible to re-use current bindings (expand the "reg" property under "hypervisor" node, currently it contains gnttab region)?Do we need a struct page* for the range? We needed it in the past and it is the reason why we currenly use ballooned-out pages: they have a proper struct page* associated with them.
pte_special()caused problems. So, continuing with the assumption that we need a struct page*, then the range needs to be advertised as "memory" to the DomU (part to of the /memory node). But we can reserve the range for foreign/grant-mapping use by adding a reserved-memory binding for it. If we go down this route, we should add a new binding as I don't think we should reusereserved-memory/xen,shared-memory.txt. It would be very simply to add, just follow the model of xen,shared-memory.txt. (Note that just because the range is part of the DomU /memory node, it doesn't mean it needs to be backed by real memory.)
Quoted 6 lines
Quoted 2 lines
If we don't need a struct page* then we can do something different using bus ranges and/or MMIO regions.
Yes, we need struct page* for the each pfn in the range. But, I think, this can be achieved even without describing the range as a part of memory/reserved-memory nodes. Fortunately, the Linux
ZONE_DEVICEfacility can help with that, actually the "unpopulated-alloc" is based on this feature, please see drivers/xen/unpopulated-alloc.c:fill_list()
and https://lore.kernel.org/lkml/[email protected]/ for details.
I have done some experiments with Xen and toolstack according to the discussion above. So, I re-used DTB to pass a safe range to the domain. For the range I borrowed some space from the second RAM bank.
-#define GUEST_RAM1_BASE xen_mk_ullong(0x0200000000) /* 1016GB of RAM @ 8GB */ -#define GUEST_RAM1_SIZE xen_mk_ullong(0xfe00000000)
+#define GUEST_RAM1_BASE xen_mk_ullong(0x0200000000) /* 888GB of RAM @ 8GB */ +#define GUEST_RAM1_SIZE xen_mk_ullong(0xDE00000000)
+
+#define GUEST_SAFE_RANGE_BASE xen_mk_ullong(0xDE00000000) /* 128GB */ +#define GUEST_SAFE_RANGE_SIZE xen_mk_ullong(0x0200000000)
While the possible new DT bindings has not been agreed yet, I re-used existing "reg" property under the hypervisor node to pass safe range as a second region,
https://elixir.bootlin.com/linux/v5.14-rc4/source/Documentation/devicetree/bindings/arm/xen.txt#L10:
tools/libs/light/libxl_arm.c +0 -0
@@ -735,9 +735,11 @@ static int make_hypervisor_node(libxl__gc *gc, void *fdt,
"xen,xen");
if (res) return res;
- /* reg 0 is grant table space */milat
+ /* reg 0 is grant table space, reg 1 is safe range */
res = fdt_property_regs(gc, fdt, GUEST_ROOT_ADDRESS_CELLS, GUEST_ROOT_SIZE_CELLS, - 1,GUEST_GNTTAB_BASE, GUEST_GNTTAB_SIZE);
+ 2,
+ GUEST_GNTTAB_BASE, GUEST_GNTTAB_SIZE,
+ GUEST_SAFE_RANGE_BASE, GUEST_SAFE_RANGE_SIZE); if (res) return res;
/*
/* Resulting hypervisor node */
hypervisor {
interrupts = <0x01 0x0f 0xf08>; interrupt-parent = <0xfde8>;
compatible = "xen,xen-4.16\0xen,xen";
reg = <0x00 0x38000000 0x00 0x1000000 0xde 0x00 0x02 0x00>; };
Near the same I did for the Xen itself to insert a range for Dom0. The Linux side change is just to retrieve a range from DTB instead of issuing a hypercall.
Sorry, I might miss some important bits here, but from what I wrote about the "reg" purpose, it seems it could be suitable for us, why actually not? Why do we need yet another binding? I noticed, Linux on Arm doesn't use it at all, probably it is used by other OSes, I don't know.
Now, I am wondering, would it be possible to update/clarify the current "reg" purpose and use it to pass a safe unallocated space for any Xen specific mappings (grant, foreign, whatever) instead of just for the grant table region. In case, it is not allowed for any reason (compatibility PoV, etc), would it be possible to extend a property by passing an extra range separately, something similar to how I described above?
Signature · 3 lines
Regards, Oleksandr Tyshchenko
Quoted 2 lines · Oleksandr
Hi Julien, Stefano.
Hi Oleksandr,
Review
Quoted 11 lines · Oleksandr
I have done some experiments with Xen and toolstack according to the discussion above. So, I re-used DTB to pass a safe range to the domain. For the range I borrowed some space from the second RAM bank.
-#define GUEST_RAM1_BASE
xen_mk_ullong(0x0200000000)/* 1016GB of RAM @ 8GB */(quoted diff) +1
-1-#define GUEST_RAM1_SIZE xen_mk_ullong(0xfe00000000) +#define GUEST_RAM1_BASE xen_mk_ullong(0x0200000000) /* 888GB of RAM @8GB */
(quoted diff) +2
-0+#define GUEST_RAM1_SIZE xen_mk_ullong(0xDE00000000) +
I am a bit split with reducing the amount of RAM. On one hand large guest is not unheard on the server side (at least in the x86 world). On the other hand, I am not aware of anyone using Xen on Arm in such setup.
So technically this will be a regression, but it may be OK.
Regarding the range, this will be a problem as Xen configure the number of the IPA bits based on the PA bits. The lowest possible address space ize on 64-bit is 4GB.
From my understanding, this is because the number of IPA bits supported is contrained by the PA bits. So the position and the size of the region would need to depend on the P2M configuration.
For simplicity, this could be the last few X bytes of the supported address space.
For 32-bit domain, we also need to make sure the address is usable for domain short page tables (not too long ago Debian was shipping the kernel with them rather than LPAE). I haven't yet checked what's the limit here.
Quoted 5 lines
(quoted diff) +2
-0+#define GUEST_SAFE_RANGE_BASE xen_mk_ullong(0xDE00000000) /* 128GB */ +#define GUEST_SAFE_RANGE_SIZE xen_mk_ullong(0x0200000000)While the possible new DT bindings has not been agreed yet, I re-used existing "reg" property under the hypervisor node to pass safe range as a second region,
https://elixir.bootlin.com/linux/v5.14-rc4/source/Documentation/devicetree/bindings/arm/xen.txt#L10:
So a single region works for a guest today, but for dom0 we will need multiple regions because it is may be difficult to find enough contiguous space for a single region.
That said, as dom0 is mapped 1:1 (including some guest mapping), there is also the question where to allocate the safe region. For grant table, we so far re-use the Xen address space because it is assumed it will space will always be bigger than the grant table.
I am not sure yet where we could allocate the safe regions. Stefano, do you have any ideas?
Quoted 34 lines
tools/libs/light/libxl_arm.c +4
-2@@ -735,9 +735,11 @@ static int make_hypervisor_node(libxl__gc *gc, void *fdt, "xen,xen"); if (res) return res; - /* reg 0 is grant table space */milat + /* reg 0 is grant table space, reg 1 is safe range */ res = fdt_property_regs(gc, fdt, GUEST_ROOT_ADDRESS_CELLS,GUEST_ROOT_SIZE_CELLS , - 1,GUEST_GNTTAB_BASE, GUEST_GNTTAB_SIZE); + 2, + GUEST_GNTTAB_BASE, GUEST_GNTTAB_SIZE, + GUEST_SAFE_RANGE_BASE, GUEST_SAFE_RANGE_SIZE); if (res) return res;/*
/* Resulting hypervisor node */
hypervisor {
interrupts = <0x010x0f0xf08>; interrupt-parent = <0xfde8>;
compatible = "xen,xen-4.16\0xen,xen";
reg = <0x000x380000000x000x10000000xde0x000x020x00>;
};
Quoted 4 lines
Near the same I did for the Xen itself to insert a range for Dom0. The Linux side change is just to retrieve a range from DTB instead of issuing a hypercall.
Sorry, I might miss some important bits here, but from what I wrote about the "reg" purpose, it seems it could be suitable for us, why actually not? Why do we need yet another binding? I noticed, Linux on Arm doesn't use it at all, probably it is used by other OSes, I don't know.
Linux used the range until 4.7. This was dropped by commit 3cf4095d7446efde28b48c26050b9db6f0bcb004 so the same code can be used by ACPI and DT. However, looking at this now, I think this was a bad decision because it means we are shattering superpages.
So ideally we should switch back the region to use the safe address space once this is in place.
Quoted 2 lines
Now, I am wondering, would it be possible to update/clarify the current "reg" purpose and use it to pass a safe unallocated space for any Xen specific mappings (grant, foreign, whatever) instead of just for the grant table region. In case, it is not allowed for any reason (compatibility PoV, etc), would it be possible to extend a property by passing an extra range separately, something similar to how I described above?
I think it should be fine to re-use the same region so long the size of the first bank is at least the size of the original region.
I also think we should be able to add extra regions as OSes are unlikely to enforce that the "reg" contains a single region.
That said, we need to be careful about new guests as the region may be quite small on older Xen. So we would need some heuristic to decide whether to stole some RAM or use the safe space.
Another possibility would be to add a new compatible in the DT that indicates the region is "big" enough.
Cheers,
Signature · 1 line
Julien Grall
Quoted 7 lines · Julien Grall
Quoted 2 lines · Oleksandr
Hi Julien, Stefano.
Hi Oleksandr,
Hi, Julien
Thank you for the prompt reply and explanations.
Quoted 15 lines
Review
Quoted 12 lines · Oleksandr
I have done some experiments with Xen and toolstack according to the discussion above. So, I re-used DTB to pass a safe range to the domain. For the range I borrowed some space from the second RAM bank.
-#define GUEST_RAM1_BASE
xen_mk_ullong(0x0200000000)/* 1016GB of RAM @ 8GB */(quoted diff) +1
-1-#define GUEST_RAM1_SIZE xen_mk_ullong(0xfe00000000) +#define GUEST_RAM1_BASE xen_mk_ullong(0x0200000000) /* 888GB ofRAM @ 8GB */
(quoted diff) +2
-0+#define GUEST_RAM1_SIZE xen_mk_ullong(0xDE00000000) +I am a bit split with reducing the amount of RAM. On one hand large guest is not unheard on the server side (at least in the x86 world). On the other hand, I am not aware of anyone using Xen on Arm in such setup.
Quoted 1 line
So technically this will be a regression, but it may be OK.
I got it.
Quoted 6 lines
Regarding the range, this will be a problem as Xen configure the number of the IPA bits based on the PA bits. The lowest possible address space ize on 64-bit is 4GB.
From my understanding, this is because the number of IPA bits supported is contrained by the PA bits. So the position and the size of the region would need to depend on the P2M configuration.
Indeed, I missed these bits that IPA bits on Arm64 might be < 40 bit, I remember, we select p2m_ipa_bits in setup_virt_paging() depending on pabits, moreover the p2m_ipa_bits might be even restricted by some external entity (IOMMU, if P2M is shared).
Quoted 2 lines
For simplicity, this could be the last few X bytes of the supported address space.
ok, agree. To summarize, so it sounds like we can't use the fixed safe range as in my example, it must be variable. Well, I hope, we will be able to achieve this without reducing the total amount of domain RAM in front (GUEST_RAM1_SIZE). After all, we know the IPA size and the domain RAM in advance, so we certainly can choose the start and size of the range. In case, we won't be able to find a suitable large chunk (for example, when IPA bits = 32, and domain has a lot of RAM assigned and as the result - almost all address space below 4GB is in use), we won't expose a safe range to a domain at all, and domain will just fall back to use real pages instead (actually, how it currently behaves on Arm).
A side note: we would likely need the toolstack (that generates device-tree for guests) to query IPA size, or similar.
Quoted 3 lines
For 32-bit domain, we also need to make sure the address is usable for domain short page tables (not too long ago Debian was shipping the kernel with them rather than LPAE). I haven't yet checked what's the limit here.
Hmm, I didn't take this use-case into the account. So, I assume we need the safe range to be located below 4GB if is_32bit_domain() returns true.
Quoted 5 lines
Quoted 5 lines
(quoted diff) +2
-0+#define GUEST_SAFE_RANGE_BASE xen_mk_ullong(0xDE00000000) /* 128GB */ +#define GUEST_SAFE_RANGE_SIZE xen_mk_ullong(0x0200000000)
Quoted 7 lines
Quoted 2 lines
While the possible new DT bindings has not been agreed yet, I re-used existing "reg" property under the hypervisor node to pass safe range as a second region,
https://elixir.bootlin.com/linux/v5.14-rc4/source/Documentation/devicetree/bindings/arm/xen.txt#L10:So a single region works for a guest today, but for dom0 we will need multiple regions because it is may be difficult to find enough contiguous space for a single region.
That said, as dom0 is mapped 1:1 (including some guest mapping), there is also the question where to allocate the safe region. For grant table, we so far re-use the Xen address space because it is assumed it will space will always be bigger than the grant table.
Oh, I see. Indeed, it is unclear at the moment. Agree, the possibility to provide multiple ranges should be envisaged.
Quoted 37 lines
I am not sure yet where we could allocate the safe regions. Stefano, do you have any ideas?
Quoted 33 lines
tools/libs/light/libxl_arm.c +4
-2@@ -735,9 +735,11 @@ static int make_hypervisor_node(libxl__gc *gc, void *fdt, "xen,xen"); if (res) return res; - /* reg 0 is grant table space */milat + /* reg 0 is grant table space, reg 1 is safe range */ res = fdt_property_regs(gc, fdt, GUEST_ROOT_ADDRESS_CELLS, GUEST_ROOT_SIZE_CELLS, - 1,GUEST_GNTTAB_BASE, GUEST_GNTTAB_SIZE); + 2, + GUEST_GNTTAB_BASE, GUEST_GNTTAB_SIZE, + GUEST_SAFE_RANGE_BASE,GUEST_SAFE_RANGE_SIZE ); if (res) return res;/*
/* Resulting hypervisor node */
hypervisor {
interrupts = <0x010x0f0xf08>; interrupt-parent = <0xfde8>;
compatible = "xen,xen-4.16\0xen,xen";
reg = <0x000x380000000x000x10000000xde0x000x020x00>; };
Quoted 6 lines
Quoted 4 lines
Near the same I did for the Xen itself to insert a range for Dom0. The Linux side change is just to retrieve a range from DTB instead of issuing a hypercall.
Sorry, I might miss some important bits here, but from what I wrote about the "reg" purpose, it seems it could be suitable for us, why actually not? Why do we need yet another binding? I noticed, Linux on Arm doesn't use it at all, probably it is used by other OSes, I don't know.
Linux used the range until 4.7. This was dropped by commit
3cf4095d7446efde28b48c26050b9db6f0bcb004so the same code can be used by ACPI and DT. However, looking at this now, I think this was a bad decision because it means we are shattering superpages.
Yes, the another benefit of this enabling work (besides avoid wasting real domain pages) would be avoid shattering superpages, and as the result the I/O performance for the DMA devices behind the IOMMU would be slightly better, I hope.
Quoted 3 lines
So ideally we should switch back the region to use the safe address space once this is in place.
Yes.
Quoted 6 lines
Quoted 4 lines
Now, I am wondering, would it be possible to update/clarify the current "reg" purpose and use it to pass a safe unallocated space for any Xen specific mappings (grant, foreign, whatever) instead of just for the grant table region. In case, it is not allowed for any reason (compatibility PoV, etc), would it be possible to extend a property by passing an extra range separately, something similar to how I described above?
I think it should be fine to re-use the same region so long the size of the first bank is at least the size of the original region.
ok
Quoted 4 lines
I also think we should be able to add extra regions as OSes are unlikely to enforce that the "reg" contains a single region.
That said, we need to be careful about new guests as the region may be quite small on older Xen.
Agree.
Quoted 2 lines
So we would need some heuristic to decide whether to stole some RAM or use the safe space. Another possibility would be to add a new compatible in the DT that indicates the region is "big" enough.
I like the last idea, did you perhaps mean new property (optional) rather than new compatible? Let's say "xen, safe-range" or "xen, extended-regions" ...
That sign would suggest us how to operate: either pick up safe range(s) or behave as usual (stole real pages) as existing region is indeed small - 16MB (if I am not mistaken), this would work with old and new Xen.
Quoted 4 lines
Cheers,
Signature · 3 lines
Regards, Oleksandr Tyshchenko
Hi Oleksandr,
Quoted 44 lines · Oleksandr
Quoted 7 lines · Julien Grall
Quoted 2 lines · Oleksandr
Hi Julien, Stefano.
Hi Oleksandr,
Hi, Julien
Thank you for the prompt reply and explanations.
Quoted 15 lines
Review
Quoted 12 lines · Oleksandr
I have done some experiments with Xen and toolstack according to the discussion above. So, I re-used DTB to pass a safe range to the domain. For the range I borrowed some space from the second RAM bank.
-#define GUEST_RAM1_BASE
xen_mk_ullong(0x0200000000)/* 1016GB of RAM @ 8GB */(quoted diff) +1
-1-#define GUEST_RAM1_SIZE xen_mk_ullong(0xfe00000000) +#define GUEST_RAM1_BASE xen_mk_ullong(0x0200000000) /* 888GB ofRAM @ 8GB */
(quoted diff) +2
-0+#define GUEST_RAM1_SIZE xen_mk_ullong(0xDE00000000) +I am a bit split with reducing the amount of RAM. On one hand large guest is not unheard on the server side (at least in the x86 world). On the other hand, I am not aware of anyone using Xen on Arm in such setup.
Quoted 1 line
So technically this will be a regression, but it may be OK.
I got it.
Quoted 6 lines
Regarding the range, this will be a problem as Xen configure the number of the IPA bits based on the PA bits. The lowest possible address space ize on 64-bit is 4GB.
From my understanding, this is because the number of IPA bits supported is contrained by the PA bits. So the position and the size of the region would need to depend on the P2M configuration.
Quoted 6 lines
Indeed, I missed these bits that IPA bits on Arm64 might be < 40 bit, I remember, we select
p2m_ipa_bitsinsetup_virt_paging()depending on pabits, moreover thep2m_ipa_bitsmight be even restricted by some external entity (IOMMU, if P2M is shared).Quoted 2 lines
For simplicity, this could be the last few X bytes of the supported address space.
ok, agree. To summarize, so it sounds like we can't use the fixed safe range as in my example, it must be variable. Well, I hope, we will be able to achieve this without reducing the total amount of domain RAM in front (
GUEST_RAM1_SIZE). After all, we know the IPA size and the domain RAM in advance, so we certainly can choose the start and size of the range. In case, we won't be able to find a suitable large chunk (for example, when IPA bits = 32, and domain has a lot of RAM assigned and as the result - almost all address space below 4GB is in use), we won't expose a safe range to a domain at all, and domain will just fall back to use real pages instead (actually, how it currently behaves on Arm).
I think it would be fine for a first approach. We can refine it in the future. What matters is that we correctly define the binding/hypercall.
Quoted 2 lines
A side note: we would likely need the toolstack (that generates device-tree for guests) to query IPA size, or similar.
I think we can use XEN_SYSCTL_* (or possibly hypfs) for that.
Quoted 5 lines
Quoted 3 lines
For 32-bit domain, we also need to make sure the address is usable for domain short page tables (not too long ago Debian was shipping the kernel with them rather than LPAE). I haven't yet checked what's the limit here.
Hmm, I didn't take this use-case into the account. So, I assume we need the safe range to be located below 4GB if
is_32bit_domain()returns true.
Yes. Or we can say that if you are using a 32-bit domain then we don't (yet) support a safe range for range.
Quoted 3 lines
Quoted 2 lines
So we would need some heuristic to decide whether to stole some RAM or use the safe space. Another possibility would be to add a new compatible in the DT that indicates the region is "big" enough.
I like the last idea, did you perhaps mean new property (optional) rather than new compatible? Let's say "xen, safe-range" or "xen, extended-regions" ...
I actually meant adding an extra compatible because this is technically a change of the binding (even though it is backward compatible).
Although, I would be OK with the property. You may first want to ask the Device-Tree folks how they expect a binding to be extended in a backward compatible way.
Cheers,
Signature · 1 line
Julien Grall
Quoted 1 line · Julien Grall
Hi Oleksandr,
Hi Julien
Quoted 46 lines · Oleksandr
Quoted 44 lines · Julien Grall
Quoted 7 lines · Oleksandr
Quoted 2 lines
Hi Julien, Stefano.
Hi Oleksandr,
Hi, Julien
Thank you for the prompt reply and explanations.
Quoted 15 lines
Review
Quoted 12 lines · Oleksandr
I have done some experiments with Xen and toolstack according to the discussion above. So, I re-used DTB to pass a safe range to the domain. For the range I borrowed some space from the second RAM bank.
-#define GUEST_RAM1_BASE
xen_mk_ullong(0x0200000000)/* 1016GB of RAM @ 8GB */(quoted diff) +1
-1-#define GUEST_RAM1_SIZE xen_mk_ullong(0xfe00000000) +#define GUEST_RAM1_BASE xen_mk_ullong(0x0200000000) /* 888GB ofRAM @ 8GB */
(quoted diff) +2
-0+#define GUEST_RAM1_SIZE xen_mk_ullong(0xDE00000000) +I am a bit split with reducing the amount of RAM. On one hand large guest is not unheard on the server side (at least in the x86 world). On the other hand, I am not aware of anyone using Xen on Arm in such setup.
Quoted 1 line
So technically this will be a regression, but it may be OK.
I got it.
Quoted 6 lines
Regarding the range, this will be a problem as Xen configure the number of the IPA bits based on the PA bits. The lowest possible address space ize on 64-bit is 4GB.
From my understanding, this is because the number of IPA bits supported is contrained by the PA bits. So the position and the size of the region would need to depend on the P2M configuration.
Quoted 8 lines
Quoted 6 lines
Indeed, I missed these bits that IPA bits on Arm64 might be < 40 bit, I remember, we select
p2m_ipa_bitsinsetup_virt_paging()depending on pabits, moreover thep2m_ipa_bitsmight be even restricted by some external entity (IOMMU, if P2M is shared).Quoted 2 lines
For simplicity, this could be the last few X bytes of the supported address space.
ok, agree. To summarize, so it sounds like we can't use the fixed safe range as in my example, it must be variable. Well, I hope, we will be able to achieve this without reducing the total amount of domain RAM in front (
GUEST_RAM1_SIZE). After all, we know the IPA size and the domain RAM in advance, so we certainly can choose the start and size of the range. In case, we won't be able to find a suitable large chunk (for example, when IPA bits = 32, and domain has a lot of RAM assigned and as the result - almost all address space below 4GB is in use), we won't expose a safe range to a domain at all, and domain will just fall back to use real pages instead (actually, how it currently behaves on Arm).I think it would be fine for a first approach. We can refine it in the future. What matters is that we correctly define the binding/hypercall.
ok
Quoted 5 lines
Quoted 3 lines
A side note: we would likely need the toolstack (that generates device-tree for guests) to query IPA size, or similar.
I think we can use XEN_SYSCTL_* (or possibly hypfs) for that.
probably yes, need to investigate
Quoted 9 lines
Quoted 7 lines
Quoted 5 lines
For 32-bit domain, we also need to make sure the address is usable for domain short page tables (not too long ago Debian was shipping the kernel with them rather than LPAE). I haven't yet checked what's the limit here.
Hmm, I didn't take this use-case into the account. So, I assume we need the safe range to be located below 4GB if
is_32bit_domain()returns true.Yes. Or we can say that if you are using a 32-bit domain then we don't (yet) support a safe range for range.
yes, as a variant
Quoted 8 lines
Quoted 4 lines
Quoted 3 lines
So we would need some heuristic to decide whether to stole some RAM or use the safe space. Another possibility would be to add a new compatible in the DT that indicates the region is "big" enough.
I like the last idea, did you perhaps mean new property (optional) rather than new compatible? Let's say "xen, safe-range" or "xen, extended-regions" ...
I actually meant adding an extra compatible because this is technically a change of the binding (even though it is backward compatible).
Although, I would be OK with the property. You may first want to ask the Device-Tree folks how they expect a binding to be extended in a backward compatible way.
makes sense, will clarify these bits
Quoted 3 lines
Cheers,
Signature · 3 lines
Regards, Oleksandr Tyshchenko
Quoted 74 lines · Julien Grall
Hi Oleksandr,
Quoted 60 lines · Oleksandr
Quoted 7 lines · Julien Grall
Quoted 2 lines · Oleksandr
Hi Julien, Stefano.
Hi Oleksandr,
Hi, Julien
Thank you for the prompt reply and explanations.
Quoted 19 lines
Quoted 12 lines · Oleksandr
I have done some experiments with Xen and toolstack according to the discussion above. So, I re-used DTB to pass a safe range to the domain. For the range I borrowed some space from the second RAM bank.
-#define GUEST_RAM1_BASE
xen_mk_ullong(0x0200000000)/* 1016GB of RAM @ 8GB */(quoted diff) +1
-1-#define GUEST_RAM1_SIZE xen_mk_ullong(0xfe00000000) +#define GUEST_RAM1_BASE xen_mk_ullong(0x0200000000) /* 888GB of RAM @8GB */
Review
Quoted 2 lines
(quoted diff) +2
-0+#define GUEST_RAM1_SIZE xen_mk_ullong(0xDE00000000) +I am a bit split with reducing the amount of RAM. On one hand large guest is not unheard on the server side (at least in the x86 world). On the other hand, I am not aware of anyone using Xen on Arm in such setup.
Quoted 1 line
So technically this will be a regression, but it may be OK.
I got it.
Quoted 9 lines
Regarding the range, this will be a problem as Xen configure the number of the IPA bits based on the PA bits. The lowest possible address space ize on 64-bit is 4GB.
From my understanding, this is because the number of IPA bits supported is contrained by the PA bits. So the position and the size of the region would need to depend on the P2M configuration.
Indeed, I missed these bits that IPA bits on Arm64 might be < 40 bit, I remember, we select
p2m_ipa_bitsinsetup_virt_paging()depending on pabits, moreover thep2m_ipa_bitsmight be even restricted by some external entity (IOMMU, if P2M is shared).Quoted 3 lines
For simplicity, this could be the last few X bytes of the supported address space.
Quoted 10 lines
ok, agree. To summarize, so it sounds like we can't use the fixed safe range as in my example, it must be variable. Well, I hope, we will be able to achieve this without reducing the total amount of domain RAM in front (
GUEST_RAM1_SIZE). After all, we know the IPA size and the domain RAM in advance, so we certainly can choose the start and size of the range. In case, we won't be able to find a suitable large chunk (for example, when IPA bits = 32, and domain has a lot of RAM assigned and as the result - almost all address space below 4GB is in use), we won't expose a safe range to a domain at all, and domain will just fall back to use real pages instead (actually, how it currently behaves on Arm).
Quoted 2 lines
I think it would be fine for a first approach. We can refine it in the future. What matters is that we correctly define the binding/hypercall.
I agree with Julien on both points. Looking at the existing device tree binding, it is almost exactly what we need, so I think it would be OK to use it.
Quoted 17 lines
Quoted 6 lines
Quoted 3 lines
For 32-bit domain, we also need to make sure the address is usable for domain short page tables (not too long ago Debian was shipping the kernel with them rather than LPAE). I haven't yet checked what's the limit here.
Hmm, I didn't take this use-case into the account. So, I assume we need the safe range to be located below 4GB if
is_32bit_domain()returns true.Yes. Or we can say that if you are using a 32-bit domain then we don't (yet) support a safe range for range.
Quoted 7 lines
Quoted 4 lines
So we would need some heuristic to decide whether to stole some RAM or use the safe space.
Another possibility would be to add a new compatible in the DT that indicates the region is "big" enough.I like the last idea, did you perhaps mean new property (optional) rather than new compatible? Let's say "xen, safe-range" or "xen, extended-regions"
...
Quoted 6 lines
I actually meant adding an extra compatible because this is technically a change of the binding (even though it is backward compatible).
Although, I would be OK with the property. You may first want to ask the Device-Tree folks how they expect a binding to be extended in a backward compatible way.
I think we should expand the description in Documentation/devicetree/bindings/arm/xen.txt to cover things other than the grant table.
I don't think we necessarely need a new compatible string because as you said the change is fully compatible. At the same time it is trivial to add compatible strings, so that could be done as well.
Hi Stefano
Quoted 62 lines · Stefano Stabellini
Quoted 61 lines · Julien Grall
Hi Oleksandr,
Quoted 58 lines · Oleksandr
Quoted 4 lines · Julien Grall
Quoted 1 line · Oleksandr
Hi Julien, Stefano.
Hi Oleksandr,
Hi, Julien
Thank you for the prompt reply and explanations.
Quoted 16 lines
Review
Quoted 12 lines · Oleksandr
I have done some experiments with Xen and toolstack according to the discussion above. So, I re-used DTB to pass a safe range to the domain. For the range I borrowed some space from the second RAM bank.
-#define GUEST_RAM1_BASE
xen_mk_ullong(0x0200000000)/* 1016GB of RAM @ 8GB */(quoted diff) +1
-1-#define GUEST_RAM1_SIZE xen_mk_ullong(0xfe00000000) +#define GUEST_RAM1_BASE xen_mk_ullong(0x0200000000) /* 888GB of RAM @8GB */
(quoted diff) +2
-0+#define GUEST_RAM1_SIZE xen_mk_ullong(0xDE00000000) +I am a bit split with reducing the amount of RAM. On one hand large guest is not unheard on the server side (at least in the x86 world). On the other hand, I am not aware of anyone using Xen on Arm in such setup.
Quoted 1 line
So technically this will be a regression, but it may be OK.
I got it.
Quoted 8 lines
Regarding the range, this will be a problem as Xen configure the number of the IPA bits based on the PA bits. The lowest possible address space ize on 64-bit is 4GB.
From my understanding, this is because the number of IPA bits supported is
contrained by the PA bits. So the position and the size of the region would need to depend on the P2M configuration.Indeed, I missed these bits that IPA bits on Arm64 might be < 40 bit, I remember, we select
p2m_ipa_bitsinsetup_virt_paging()depending on pabits, moreover thep2m_ipa_bitsmight be even restricted by some external entity (IOMMU, if P2M is shared).Quoted 2 lines
For simplicity, this could be the last few X bytes of the supported address space.
ok, agree. To summarize, so it sounds like we can't use the fixed safe range as in my example, it must be variable. Well, I hope, we will be able to achieve this without reducing the total amount of domain RAM in front (
GUEST_RAM1_SIZE). After all, we know the IPA size and the domain RAM in advance, so we certainly can choose the start and size of the range. In case, we won't be able to find a suitable large chunk (for example, when IPA bits = 32, and domain has a lot of RAM assigned and as the result - almost all address space below 4GB is in use), we won't expose a safe range to a domain at all, and domain will just fall back to use real pages instead (actually, how it currently behaves on Arm).
Quoted 5 lines
Quoted 2 lines
I think it would be fine for a first approach. We can refine it in the future. What matters is that we correctly define the binding/hypercall.
I agree with Julien on both points. Looking at the existing device tree binding, it is almost exactly what we need, so I think it would be OK to use it.
ok, great.
Quoted 19 lines
Quoted 19 lines
Quoted 7 lines
Quoted 5 lines
For 32-bit domain, we also need to make sure the address is usable for domain short page tables (not too long ago Debian was shipping the kernel with them rather than LPAE). I haven't yet checked what's the limit here.
Hmm, I didn't take this use-case into the account. So, I assume we need the safe range to be located below 4GB if
is_32bit_domain()returns true.Yes. Or we can say that if you are using a 32-bit domain then we don't (yet) support a safe range for range.
Quoted 7 lines
Quoted 4 lines
So we would need some heuristic to decide whether to stole some RAM or use the safe space.
Another possibility would be to add a new compatible in the DT that indicates the region is "big" enough.I like the last idea, did you perhaps mean new property (optional) rather than new compatible? Let's say "xen, safe-range" or "xen, extended-regions"
...I actually meant adding an extra compatible because this is technically a change of the binding (even though it is backward compatible).
Quoted 6 lines
Quoted 3 lines
Although, I would be OK with the property. You may first want to ask the Device-Tree folks how they expect a binding to be extended in a backward compatible way.
I think we should expand the description in
Documentation/devicetree/bindings/arm/xen.txtto cover things other than the grant table.
yes
Quoted 4 lines
I don't think we necessarely need a new compatible string because as you said the change is fully compatible. At the same time it is trivial to add compatible strings, so that could be done as well.
ok, please note, I have already asked DT folks for the clarification (how to make it properly), so let's wait for input.
Signature · 3 lines
Regards, Oleksandr Tyshchenko
Quoted 1 line · Julien Grall
Hi Oleksandr,
Hi Julien, all
Quoted 51 lines · Oleksandr
Quoted 49 lines · Julien Grall
Quoted 7 lines · Oleksandr
Quoted 2 lines
Hi Julien, Stefano.
Hi Oleksandr,
Hi, Julien
Thank you for the prompt reply and explanations.
Quoted 15 lines
Review
Quoted 12 lines · Oleksandr
I have done some experiments with Xen and toolstack according to the discussion above. So, I re-used DTB to pass a safe range to the domain. For the range I borrowed some space from the second RAM bank.
-#define GUEST_RAM1_BASE
xen_mk_ullong(0x0200000000)/* 1016GB of RAM @ 8GB */(quoted diff) +1
-1-#define GUEST_RAM1_SIZE xen_mk_ullong(0xfe00000000) +#define GUEST_RAM1_BASE xen_mk_ullong(0x0200000000) /* 888GB ofRAM @ 8GB */
(quoted diff) +2
-0+#define GUEST_RAM1_SIZE xen_mk_ullong(0xDE00000000) +I am a bit split with reducing the amount of RAM. On one hand large guest is not unheard on the server side (at least in the x86 world). On the other hand, I am not aware of anyone using Xen on Arm in such setup.
Quoted 1 line
So technically this will be a regression, but it may be OK.
I got it.
Quoted 6 lines
Regarding the range, this will be a problem as Xen configure the number of the IPA bits based on the PA bits. The lowest possible address space ize on 64-bit is 4GB.
From my understanding, this is because the number of IPA bits supported is contrained by the PA bits. So the position and the size of the region would need to depend on the P2M configuration.
Indeed, I missed these bits that IPA bits on Arm64 might be < 40 bit, I remember, we select
p2m_ipa_bitsinsetup_virt_paging()depending on pabits, moreover thep2m_ipa_bitsmight be even restricted by some external entity (IOMMU, if P2M is shared).Quoted 2 lines
For simplicity, this could be the last few X bytes of the supported address space.
Quoted 8 lines
Quoted 1 line
ok, agree. To summarize, so it sounds like we can't use the fixed safe range as in my example, it must be variable. Well, I hope, we will be able to achieve this without reducing the total amount of domain RAM in front (
GUEST_RAM1_SIZE). After all, we know the IPA size and the domain RAM in advance, so we certainly can choose the start and size of the range. In case, we won't be able to find a suitable large chunk (for example, when IPA bits = 32, and domain has a lot of RAM assigned and as the result - almost all address space below 4GB is in use), we won't expose a safe range to a domain at all, and domain will just fall back to use real pages instead (actually, how it currently behaves on Arm).I think it would be fine for a first approach. We can refine it in the future. What matters is that we correctly define the binding/hypercall.
Quoted 2 lines
A side note: we would likely need the toolstack (that generates device-tree for guests) to query IPA size, or similar.
I think we can use XEN_SYSCTL_* (or possibly hypfs) for that.
The RFC patch (which is one of the prereq patches for safe range) is already pushed:
https://lore.kernel.org/xen-devel/[email protected]/
What we need is to reach an agreement regarding the interface. We can discuss it there.
Thanks!
Quoted 18 lines
Quoted 6 lines
Quoted 4 lines
For 32-bit domain, we also need to make sure the address is usable for domain short page tables (not too long ago Debian was shipping the kernel with them rather than LPAE). I haven't yet checked what's the limit here.
Hmm, I didn't take this use-case into the account. So, I assume we need the safe range to be located below 4GB if
is_32bit_domain()returns true.Yes. Or we can say that if you are using a 32-bit domain then we don't (yet) support a safe range for range.
Quoted 3 lines
Quoted 2 lines
So we would need some heuristic to decide whether to stole some RAM or use the safe space. Another possibility would be to add a new compatible in the DT that indicates the region is "big" enough.
I like the last idea, did you perhaps mean new property (optional) rather than new compatible? Let's say "xen, safe-range" or "xen, extended-regions" ...
I actually meant adding an extra compatible because this is technically a change of the binding (even though it is backward compatible).
Although, I would be OK with the property. You may first want to ask the Device-Tree folks how they expect a binding to be extended in a backward compatible way.
Cheers,
Signature · 3 lines
Regards, Oleksandr Tyshchenko
Quoted 12 lines · Julien Grall
Quoted 7 lines
(quoted diff) +2
-0+#define GUEST_SAFE_RANGE_BASE xen_mk_ullong(0xDE00000000) /* 128GB */ +#define GUEST_SAFE_RANGE_SIZE xen_mk_ullong(0x0200000000)While the possible new DT bindings has not been agreed yet, I re-used existing "reg" property under the hypervisor node to pass safe range as a second region,
https://elixir.bootlin.com/linux/v5.14-rc4/source/Documentation/devicetree/bindings/arm/xen.txt#L10:So a single region works for a guest today, but for dom0 we will need multiple regions because it is may be difficult to find enough contiguous space for a single region.
Quoted 7 lines
That said, as dom0 is mapped 1:1 (including some guest mapping), there is also the question where to allocate the safe region. For grant table, we so far re-use the Xen address space because it is assumed it will space will always be bigger than the grant table.
I am not sure yet where we could allocate the safe regions. Stefano, do you have any ideas?
The safest choice would be the address range corresponding to memory (/memory) not already allocated to Dom0.
For instance from my last boot logs:
(XEN) Allocating 1:1 mappings totalling 1600MB for dom0:
(XEN) BANK[0] 0x00000010000000-0x00000070000000 (1536MB) (XEN) BANK[1] 0x00000078000000-0x0000007c000000 (64MB)
All the other ranges could be given as unallocated space:
- 0x0 - 0x10000000
- 0x70000000 - 0x78000000
- 0x8_0000_0000 - 0x8_8000_0000
The second best choice would be an hole: an address range not used by anybody else (no reg property) and also not even mappable by a bus (not covered by a ranges property). This is not the best choice because there can cases where physical resources appear afterwards.
On 06.08.21 03:30, Stefano Stabellini wrote:
Hi Stefano
Quoted 27 lines · Julien Grall
Quoted 11 lines
Quoted 7 lines
(quoted diff) +2
-0+#define GUEST_SAFE_RANGE_BASE xen_mk_ullong(0xDE00000000) /* 128GB */ +#define GUEST_SAFE_RANGE_SIZE xen_mk_ullong(0x0200000000)While the possible new DT bindings has not been agreed yet, I re-used existing "reg" property under the hypervisor node to pass safe range as a second region,
https://elixir.bootlin.com/linux/v5.14-rc4/source/Documentation/devicetree/bindings/arm/xen.txt#L10:So a single region works for a guest today, but for dom0 we will need multiple regions because it is may be difficult to find enough contiguous space for a single region.
Quoted 7 lines
That said, as dom0 is mapped 1:1 (including some guest mapping), there is also the question where to allocate the safe region. For grant table, we so far re-use the Xen address space because it is assumed it will space will always be bigger than the grant table.
I am not sure yet where we could allocate the safe regions. Stefano, do you have any ideas?
The safest choice would be the address range corresponding to memory (/memory) not already allocated to Dom0.
For instance from my last boot logs:
(XEN) Allocating 1:1 mappings totalling 1600MB for dom0:
(XEN) BANK[0] 0x00000010000000-0x00000070000000 (1536MB) (XEN) BANK[1] 0x00000078000000-0x0000007c000000 (64MB)
Quoted 5 lines
All the other ranges could be given as unallocated space:
-
0x0-0x10000000
-0x70000000-0x78000000
- 0x8_0000_0000 - 0x8_8000_0000
Thank you for the ideas.
If I got the idea correctly, yes, as these ranges represent the real RAM, so no I/O would be in conflict with them and as the result - no overlaps would be expected. But, I wonder, would this work if we have IOMMU enabled for Dom0 and need to establish 1:1 mapping for the DMA devices to work with grant mappings... In arm_iommu_map_page() we call guest_physmap_add_entry() with gfn = mfn, so the question is could we end up with this new gfn replacing the valid mapping (with gfn allocated from the safe region)?
Quoted 7 lines
The second best choice would be an hole: an address range not used by anybody else (no reg property) and also not even mappable by a bus (not covered by a ranges property). This is not the best choice because there can cases where physical resources appear afterwards.
Unfortunately, yes.
Signature · 3 lines
Regards, Oleksandr Tyshchenko
Hi Oleksandr,
Quoted 40 lines · Oleksandr
On 06.08.21 03:30, Stefano Stabellini wrote:
Hi Stefano
Quoted 29 lines · Julien Grall
Quoted 13 lines
Quoted 9 lines
+#define GUEST_SAFE_RANGE_BASE
xen_mk_ullong(0xDE00000000)/* 128GB */
+#define GUEST_SAFE_RANGE_SIZExen_mk_ullong(0x0200000000)While the possible new DT bindings has not been agreed yet, I re-used existing "reg" property under the hypervisor node to pass safe range as a second region,
https://elixir.bootlin.com/linux/v5.14-rc4/source/Documentation/devicetree/bindings/arm/xen.txt#L10:So a single region works for a guest today, but for dom0 we will need multiple regions because it is may be difficult to find enough contiguous space for a single region.
Quoted 7 lines
That said, as dom0 is mapped 1:1 (including some guest mapping), there is also the question where to allocate the safe region. For grant table, we so far re-use the Xen address space because it is assumed it will space will always be bigger than the grant table.
I am not sure yet where we could allocate the safe regions. Stefano, do you have any ideas?
The safest choice would be the address range corresponding to memory (/memory) not already allocated to Dom0.
For instance from my last boot logs:
(XEN) Allocating 1:1 mappings totalling 1600MB for dom0:
(XEN) BANK[0] 0x00000010000000-0x00000070000000 (1536MB) (XEN) BANK[1] 0x00000078000000-0x0000007c000000 (64MB)Quoted 5 lines
All the other ranges could be given as unallocated space:
-
0x0-0x10000000
-0x70000000-0x78000000
- 0x8_0000_0000 - 0x8_8000_0000
Quoted 6 lines
Thank you for the ideas.
If I got the idea correctly, yes, as these ranges represent the real RAM, so no I/O would be in conflict with them and as the result - no overlaps would be expected. But, I wonder, would this work if we have IOMMU enabled for Dom0 and need to establish 1:1 mapping for the DMA devices to work with grant mappings... In
arm_iommu_map_page()we callguest_physmap_add_entry()with gfn = mfn, so the question is could we end up with this new gfn replacing the valid mapping (with gfn allocated from the safe region)?
Right, when we enable the IOMMU for dom0, Xen will add an extra mapping with GFN == MFN for foreign and grant pages. This is because Linux is not aware that whether a device is protected by an IOMMU. Therefore it is assuming it is not and will use the MFN to configure for DMA transaction.
We can't remove the mapping without significant changes in Linux and Xen. I would not mandate them for this work.
That said, I think it would be acceptable to have different way to find the region depending on the dom0 configuration. So we could use the RAM not used by dom0 when the IOMMU is turned off.
Quoted 4 lines
Quoted 4 lines
The second best choice would be an hole: an address range not used by anybody else (no reg property) and also not even mappable by a bus (not covered by a ranges property). This is not the best choice because there can cases where physical resources appear afterwards.
Are you saying that the original device-tree doesn't even describe them in any way (i.e. reserved...)?
Quoted 2 lines
Unfortunately, yes.
So the decision where the safe region is located will be done by Xen. There is no involvement of the domain (it will discover the region from the DT). Therefore, I don't think we need to think about everything right now as we could adapt this is exact region is not part of the stable ABI.
The hotplug is one I would defer because this is not supported (and quite likely not working) in Xen upstream today.
Now regarding the case where dom0 is using the IOMMU. The assumption is Xen will be able to figure out all the regions used from the firmware table (ACPI or DT).
AFAIK, this assumption would be correct for DT. However, for ACPI, I remember we were not able to find all the MMIOs region in Xen (see [1] and [2]). So even this solution would not work for ACPI.
If I am not mistaken, we don't support IOMMU with ACPI yet. So we could defer the problem to when this is going to be supported.
Cheers,
[1] https://marc.info/?l=linux-arm-kernel&m=148469169210500&w=2
[2] Xen commit 80f9c316708400cea4417e36337267d3b26591db
Signature · 1 line
Julien Grall
Quoted 1 line · Julien Grall
Hi Oleksandr,
Hi Julien.
Thank you for the input.
Quoted 49 lines · Oleksandr
Quoted 40 lines · Stefano Stabellini
Hi Stefano
Quoted 29 lines · Julien Grall
Quoted 13 lines
Quoted 9 lines
+#define
GUEST_SAFE_RANGE_BASExen_mk_ullong(0xDE00000000)/* 128GB */
+#defineGUEST_SAFE_RANGE_SIZExen_mk_ullong(0x0200000000)While the possible new DT bindings has not been agreed yet, I re-used existing "reg" property under the hypervisor node to pass safe range as a second region,
https://elixir.bootlin.com/linux/v5.14-rc4/source/Documentation/devicetree/bindings/arm/xen.txt#L10:So a single region works for a guest today, but for dom0 we will need multiple regions because it is may be difficult to find enough contiguous space for a single region.
Quoted 7 lines
That said, as dom0 is mapped 1:1 (including some guest mapping), there is also the question where to allocate the safe region. For grant table, we so far re-use the Xen address space because it is assumed it will space will always be bigger than the grant table.
I am not sure yet where we could allocate the safe regions. Stefano, do you have any ideas?
The safest choice would be the address range corresponding to memory (/memory) not already allocated to Dom0.
For instance from my last boot logs:
(XEN) Allocating 1:1 mappings totalling 1600MB for dom0:
(XEN) BANK[0] 0x00000010000000-0x00000070000000 (1536MB) (XEN) BANK[1] 0x00000078000000-0x0000007c000000 (64MB)Quoted 5 lines
All the other ranges could be given as unallocated space:
-
0x0-0x10000000
-0x70000000-0x78000000
- 0x8_0000_0000 - 0x8_8000_0000Quoted 6 lines
Thank you for the ideas.
If I got the idea correctly, yes, as these ranges represent the real RAM, so no I/O would be in conflict with them and as the result - no overlaps would be expected. But, I wonder, would this work if we have IOMMU enabled for Dom0 and need to establish 1:1 mapping for the DMA devices to work with grant mappings... In
arm_iommu_map_page()we callguest_physmap_add_entry()with gfn = mfn, so the question is could we end up with this new gfn replacing the valid mapping (with gfn allocated from the safe region)?
Quoted 5 lines
Right, when we enable the IOMMU for dom0, Xen will add an extra mapping with GFN == MFN for foreign and grant pages. This is because Linux is not aware that whether a device is protected by an IOMMU. Therefore it is assuming it is not and will use the MFN to configure for DMA transaction.
We can't remove the mapping without significant changes in Linux and Xen. I would not mandate them for this work.
That said, I think it would be acceptable to have different way to find the region depending on the dom0 configuration. So we could use the RAM not used by dom0 when the IOMMU is turned off.
OK
Quoted 6 lines
Quoted 5 lines
Quoted 5 lines
The second best choice would be an hole: an address range not used by anybody else (no reg property) and also not even mappable by a bus (not covered by a ranges property). This is not the best choice because there can cases where physical resources appear afterwards.
Quoted 8 lines
Are you saying that the original device-tree doesn't even describe them in any way (i.e. reserved...)?
Quoted 2 lines
Unfortunately, yes.
So the decision where the safe region is located will be done by Xen. There is no involvement of the domain (it will discover the region from the DT). Therefore, I don't think we need to think about everything right now as we could adapt this is exact region is not part of the stable ABI.
The hotplug is one I would defer because this is not supported (and quite likely not working) in Xen upstream today.
Sounds reasonable.
Quoted 7 lines
Now regarding the case where dom0 is using the IOMMU. The assumption is Xen will be able to figure out all the regions used from the firmware table (ACPI or DT).
AFAIK, this assumption would be correct for DT. However, for ACPI, I remember we were not able to find all the MMIOs region in Xen (see [1] and [2]). So even this solution would not work for ACPI.
If I am not mistaken, we don't support IOMMU with ACPI yet. So we could defer the problem to when this is going to be supported.
Sounds reasonable.
To summarize:
0. Skip ACPI case for now, implement for DT case
1. If IOMMU is enabled for Dom0 -> provide holes found in Host DT as safe ranges
I would take into the account holes >= 1MB. I am wondering, do we need a special alignment here other than a PAGE_SIZE?
2. If IOMMU is disabled for Dom0 -> provide RAM which not assigned to Dom0 as safe ranges
We could even provide holes here as well.
Quoted 7 lines
Cheers,
[1] https://marc.info/?l=linux-arm-kernel&m=148469169210500&w=2
[2] Xen commit80f9c316708400cea4417e36337267d3b26591db
Signature · 3 lines
Regards, Oleksandr Tyshchenko
Quoted 93 lines · Oleksandr
Quoted 1 line · Julien Grall
Hi Oleksandr,
Hi Julien.
Thank you for the input.
Quoted 49 lines · Oleksandr
Quoted 40 lines · Stefano Stabellini
Hi Stefano
Quoted 29 lines · Julien Grall
Quoted 13 lines
Quoted 9 lines
+#define
GUEST_SAFE_RANGE_BASExen_mk_ullong(0xDE00000000)/* 128GB */
+#defineGUEST_SAFE_RANGE_SIZExen_mk_ullong(0x0200000000)While the possible new DT bindings has not been agreed yet, I re-used existing "reg" property under the hypervisor node to pass safe range as a second region,
https://elixir.bootlin.com/linux/v5.14-rc4/source/Documentation/devicetree/bindings/arm/xen.txt#L10:So a single region works for a guest today, but for dom0 we will need multiple regions because it is may be difficult to find enough contiguous space for a single region.
Quoted 7 lines
That said, as dom0 is mapped 1:1 (including some guest mapping), there is also the question where to allocate the safe region. For grant table, we so far re-use the Xen address space because it is assumed it will space will always be bigger than the grant table.
I am not sure yet where we could allocate the safe regions. Stefano, do you have any ideas?
The safest choice would be the address range corresponding to memory (/memory) not already allocated to Dom0.
For instance from my last boot logs:
(XEN) Allocating 1:1 mappings totalling 1600MB for dom0:
(XEN) BANK[0] 0x00000010000000-0x00000070000000 (1536MB) (XEN) BANK[1] 0x00000078000000-0x0000007c000000 (64MB)Quoted 5 lines
All the other ranges could be given as unallocated space:
-
0x0-0x10000000
-0x70000000-0x78000000
- 0x8_0000_0000 - 0x8_8000_0000Quoted 6 lines
Thank you for the ideas.
If I got the idea correctly, yes, as these ranges represent the real RAM, so no I/O would be in conflict with them and as the result - no overlaps would be expected. But, I wonder, would this work if we have IOMMU enabled for Dom0 and need to establish 1:1 mapping for the DMA devices to work with grant mappings... In
arm_iommu_map_page()we callguest_physmap_add_entry()with gfn = mfn, so the question is could we end up with this new gfn replacing the valid mapping (with gfn allocated from the safe region)?Quoted 5 lines
Right, when we enable the IOMMU for dom0, Xen will add an extra mapping with GFN == MFN for foreign and grant pages. This is because Linux is not aware that whether a device is protected by an IOMMU. Therefore it is assuming it is not and will use the MFN to configure for DMA transaction.
We can't remove the mapping without significant changes in Linux and Xen. I would not mandate them for this work.
That said, I think it would be acceptable to have different way to find the region depending on the dom0 configuration. So we could use the RAM not used by dom0 when the IOMMU is turned off.
OK
Quoted 6 lines
Quoted 5 lines
Quoted 5 lines
The second best choice would be an hole: an address range not used by anybody else (no reg property) and also not even mappable by a bus (not covered by a ranges property). This is not the best choice because there can cases where physical resources appear afterwards.
Quoted 8 lines
Are you saying that the original device-tree doesn't even describe them in any way (i.e. reserved...)?
Quoted 2 lines
Unfortunately, yes.
So the decision where the safe region is located will be done by Xen. There is no involvement of the domain (it will discover the region from the DT). Therefore, I don't think we need to think about everything right now as we could adapt this is exact region is not part of the stable ABI.
The hotplug is one I would defer because this is not supported (and quite likely not working) in Xen upstream today.
Sounds reasonable.
Quoted 6 lines
Now regarding the case where dom0 is using the IOMMU. The assumption is Xen will be able to figure out all the regions used from the firmware table (ACPI or DT).
AFAIK, this assumption would be correct for DT. However, for ACPI, I remember we were not able to find all the MMIOs region in Xen (see [1] and [2]). So even this solution would not work for ACPI.
Quoted 8 lines
Quoted 1 line
If I am not mistaken, we don't support IOMMU with ACPI yet. So we could defer the problem to when this is going to be supported.
Sounds reasonable.
To summarize:
0. Skip ACPI case for now, implement for DT case
Just to be clear, I suggested to skip it when the IOMMU is enabled with ACPI. We should still support the case without IOMMU. The implementation would be the same as 2.
Quoted 4 lines
1. If IOMMU is enabled for Dom0 -> provide holes found in Host DT as safe ranges
I would take into the account holes >= 1MB.
May I ask why 1MB?
Quoted 1 line
I am wondering, do we need a special alignment here other than a
PAGE_SIZE?
It needs to be 64KB aligned so a guest using 64KB pages can use it.
Quoted 4 lines
2. If IOMMU is disabled for Dom0 -> provide RAM which not assigned to Dom0 as safe ranges
We could even provide holes here as well.
I would rather not. We likely need hack for the hotplug case. So I want to keep them contained to IOMMU unless there is a strong reason to do it.
Cheers,
Signature · 1 line
Julien Grall
On 09.08.21 23:45, Julien Grall wrote:
Hi Julien
Quoted 98 lines · Oleksandr
Quoted 95 lines · Julien Grall
Quoted 1 line
Hi Oleksandr,
Hi Julien.
Thank you for the input.
Quoted 49 lines · Oleksandr
Quoted 40 lines · Stefano Stabellini
Hi Stefano
Quoted 29 lines · Julien Grall
Quoted 13 lines
Quoted 9 lines
+#define
GUEST_SAFE_RANGE_BASExen_mk_ullong(0xDE00000000)/* 128GB */
+#defineGUEST_SAFE_RANGE_SIZExen_mk_ullong(0x0200000000)While the possible new DT bindings has not been agreed yet, I re-used existing "reg" property under the hypervisor node to pass safe range as a second region,
https://elixir.bootlin.com/linux/v5.14-rc4/source/Documentation/devicetree/bindings/arm/xen.txt#L10:So a single region works for a guest today, but for dom0 we will need multiple regions because it is may be difficult to find enough contiguous space for a single region.
Quoted 7 lines
That said, as dom0 is mapped 1:1 (including some guest mapping), there is also the question where to allocate the safe region. For grant table, we so far re-use the Xen address space because it is assumed it will space will always be bigger than the grant table.
I am not sure yet where we could allocate the safe regions. Stefano, do you have any ideas?
The safest choice would be the address range corresponding to memory (/memory) not already allocated to Dom0.
For instance from my last boot logs:
(XEN) Allocating 1:1 mappings totalling 1600MB for dom0:
(XEN) BANK[0] 0x00000010000000-0x00000070000000 (1536MB) (XEN) BANK[1] 0x00000078000000-0x0000007c000000 (64MB)Quoted 5 lines
All the other ranges could be given as unallocated space:
-
0x0-0x10000000
-0x70000000-0x78000000
- 0x8_0000_0000 - 0x8_8000_0000Quoted 6 lines
Thank you for the ideas.
If I got the idea correctly, yes, as these ranges represent the real RAM, so no I/O would be in conflict with them and as the result - no overlaps would be expected. But, I wonder, would this work if we have IOMMU enabled for Dom0 and need to establish 1:1 mapping for the DMA devices to work with grant mappings... In
arm_iommu_map_page()we callguest_physmap_add_entry()with gfn = mfn, so the question is could we end up with this new gfn replacing the valid mapping (with gfn allocated from the safe region)?Quoted 5 lines
Right, when we enable the IOMMU for dom0, Xen will add an extra mapping with GFN == MFN for foreign and grant pages. This is because Linux is not aware that whether a device is protected by an IOMMU. Therefore it is assuming it is not and will use the MFN to configure for DMA transaction.
We can't remove the mapping without significant changes in Linux and Xen. I would not mandate them for this work.
That said, I think it would be acceptable to have different way to find the region depending on the dom0 configuration. So we could use the RAM not used by dom0 when the IOMMU is turned off.
OK
Quoted 6 lines
Quoted 5 lines
Quoted 5 lines
The second best choice would be an hole: an address range not used by anybody else (no reg property) and also not even mappable by a bus (not covered by a ranges property). This is not the best choice because there can cases where physical resources appear afterwards.
Quoted 8 lines
Are you saying that the original device-tree doesn't even describe them in any way (i.e. reserved...)?
Quoted 2 lines
Unfortunately, yes.
So the decision where the safe region is located will be done by Xen. There is no involvement of the domain (it will discover the region from the DT). Therefore, I don't think we need to think about everything right now as we could adapt this is exact region is not part of the stable ABI.
The hotplug is one I would defer because this is not supported (and quite likely not working) in Xen upstream today.
Sounds reasonable.
Quoted 7 lines
Now regarding the case where dom0 is using the IOMMU. The assumption is Xen will be able to figure out all the regions used from the firmware table (ACPI or DT).
AFAIK, this assumption would be correct for DT. However, for ACPI, I remember we were not able to find all the MMIOs region in Xen (see [1] and [2]). So even this solution would not work for ACPI.
If I am not mistaken, we don't support IOMMU with ACPI yet. So we could defer the problem to when this is going to be supported.
Quoted 8 lines
Quoted 6 lines
Sounds reasonable.
To summarize:
0. Skip ACPI case for now, implement for DT case
Just to be clear, I suggested to skip it when the IOMMU is enabled with ACPI. We should still support the case without IOMMU. The implementation would be the same as 2.
yes, sorry for not being precise
Quoted 8 lines
Quoted 6 lines
1. If IOMMU is enabled for Dom0 -> provide holes found in Host DT as safe ranges
I would take into the account holes >= 1MB.
May I ask why 1MB?
Nothing special, just thinking to not bother with small regions which would not be too useful overall, but could bloat resulting reg property.
Anyway, I would be ok with any sizes.
Quoted 4 lines
Quoted 2 lines
I am wondering, do we need a special alignment here other than a
PAGE_SIZE?It needs to be 64KB aligned so a guest using 64KB pages can use it.
ok, sounds reasonable
Quoted 7 lines
Quoted 5 lines
2. If IOMMU is disabled for Dom0 -> provide RAM which not assigned to Dom0 as safe ranges
We could even provide holes here as well.
I would rather not. We likely need hack for the hotplug case. So I want to keep them contained to IOMMU unless there is a strong reason to do it.
ok, I got it
Quoted 3 lines
Cheers,
Signature · 3 lines
Regards, Oleksandr Tyshchenko
Quoted 5 lines · Oleksandr
On 09.08.21 23:45, Julien Grall wrote:
Hi Julien
Hi Oleksandr,
Quoted 5 lines
Quoted 5 lines · Oleksandr
Quoted 4 lines · Julien Grall
1. If IOMMU is enabled for Dom0 -> provide holes found in Host DT as safe ranges
Quoted 7 lines
Quoted 3 lines
Quoted 1 line
I would take into the account holes >= 1MB.
May I ask why 1MB?
Nothing special, just thinking to not bother with small regions which would not be too useful overall, but could bloat resulting reg property.
Anyway, I would be ok with any sizes.
I was more interesting with the reasoning behind your choice rather than saying we want more.
I agree that we may want to skip smaller region. I am guess that an high number of small regions will also increase the bookeeping in Linux.
But I would probably suggest to align the safe region to a 2MB (superpage for 4KB page granularity) just because the backend (or even Xen) may be able to do some optimization there.
Cheers,
Signature · 1 line
Julien Grall
On 10.08.21 19:28, Julien Grall wrote:
Hi Julien.
Quoted 23 lines · Oleksandr
Quoted 5 lines · Julien Grall
Hi Julien
Hi Oleksandr,
Quoted 11 lines
Quoted 8 lines · Oleksandr
Quoted 5 lines · Julien Grall
1. If IOMMU is enabled for Dom0 -> provide holes found in Host DT as safe ranges
I would take into the account holes >= 1MB.
May I ask why 1MB?
Nothing special, just thinking to not bother with small regions which would not be too useful overall, but could bloat resulting reg property.
Quoted 7 lines
Quoted 1 line
Anyway, I would be ok with any sizes.
I was more interesting with the reasoning behind your choice rather than saying we want more.
I agree that we may want to skip smaller region. I am guess that an high number of small regions will also increase the bookeeping in Linux.
But I would probably suggest to align the safe region to a 2MB (superpage for 4KB page granularity) just because the backend (or even Xen) may be able to do some optimization there.
OK, agree regarding the alignment. But, what about the size? I assume, it should be a multiple of 2MB.
Quoted 4 lines
Cheers,
Signature · 3 lines
Regards, Oleksandr Tyshchenko
Hi Oleksandr,
Quoted 30 lines · Oleksandr
On 10.08.21 19:28, Julien Grall wrote:
Hi Julien.
Quoted 25 lines · Oleksandr
Quoted 5 lines · Julien Grall
Hi Julien
Hi Oleksandr,
Quoted 12 lines
Quoted 8 lines · Oleksandr
Quoted 5 lines · Julien Grall
1. If IOMMU is enabled for Dom0 -> provide holes found in Host DT as safe ranges
I would take into the account holes >= 1MB.
May I ask why 1MB?
Nothing special, just thinking to not bother with small regions which would not be too useful overall, but could bloat resulting reg property.
Anyway, I would be ok with any sizes.
Quoted 7 lines
Quoted 5 lines
I was more interesting with the reasoning behind your choice rather than saying we want more.
I agree that we may want to skip smaller region. I am guess that an high number of small regions will also increase the bookeeping in Linux.
But I would probably suggest to align the safe region to a 2MB (superpage for 4KB page granularity) just because the backend (or even Xen) may be able to do some optimization there.
OK, agree regarding the alignment. But, what about the size? I assume, it should be a multiple of 2MB.
I would say yes. In the documentation for the guest, I would just write "64KB" to give us some flexibility to reduce the alignment/size if we encounter platform where we can meet those limits (hopefully there is none...).
Cheers,
Signature · 1 line
Julien Grall
Quoted 35 lines · Julien Grall
Hi Oleksandr,
Quoted 32 lines · Oleksandr
On 10.08.21 19:28, Julien Grall wrote:
Hi Julien.
Quoted 27 lines · Oleksandr
Quoted 5 lines · Julien Grall
Hi Julien
Hi Oleksandr,
Quoted 5 lines
Quoted 5 lines · Oleksandr
Quoted 4 lines · Julien Grall
1. If IOMMU is enabled for Dom0 -> provide holes found in Host DT as safe ranges
Quoted 7 lines
Quoted 3 lines
Quoted 1 line
I would take into the account holes >= 1MB.
May I ask why 1MB?
Nothing special, just thinking to not bother with small regions which would not be too useful overall, but could bloat resulting reg property.
Anyway, I would be ok with any sizes.
I was more interesting with the reasoning behind your choice rather than saying we want more.
Quoted 7 lines
Quoted 5 lines
Quoted 3 lines
I agree that we may want to skip smaller region. I am guess that an high number of small regions will also increase the bookeeping in Linux.
But I would probably suggest to align the safe region to a 2MB (superpage for 4KB page granularity) just because the backend (or even Xen) may be able to do some optimization there.
OK, agree regarding the alignment. But, what about the size? I assume, it should be a multiple of 2MB.
I would say yes. In the documentation for the guest, I would just write "64KB" to give us some flexibility to reduce the alignment/size if we encounter platform where we can meet those limits (hopefully there is none...).
Whoops, I meant cannot rather than can.
Cheers,
Signature · 1 line
Julien Grall
On 17.08.21 20:54, Julien Grall wrote:
Hi Julien
Quoted 38 lines · Julien Grall
Quoted 37 lines
Hi Oleksandr,
Quoted 34 lines · Oleksandr
On 10.08.21 19:28, Julien Grall wrote:
Hi Julien.
Quoted 29 lines · Oleksandr
Quoted 5 lines · Julien Grall
Hi Julien
Hi Oleksandr,
Quoted 5 lines
Quoted 5 lines · Oleksandr
Quoted 4 lines · Julien Grall
1. If IOMMU is enabled for Dom0 -> provide holes found in Host DT as safe ranges
Quoted 7 lines
Quoted 3 lines
Quoted 1 line
I would take into the account holes >= 1MB.
May I ask why 1MB?
Nothing special, just thinking to not bother with small regions which would not be too useful overall, but could bloat resulting reg property.
Anyway, I would be ok with any sizes.
I was more interesting with the reasoning behind your choice rather than saying we want more.
I agree that we may want to skip smaller region. I am guess that an high number of small regions will also increase the bookeeping in Linux.
Quoted 7 lines
Quoted 5 lines
Quoted 3 lines
Quoted 1 line
But I would probably suggest to align the safe region to a 2MB (superpage for 4KB page granularity) just because the backend (or even Xen) may be able to do some optimization there.
OK, agree regarding the alignment. But, what about the size? I assume, it should be a multiple of 2MB.
I would say yes. In the documentation for the guest, I would just write "64KB" to give us some flexibility to reduce the alignment/size if we encounter platform where we can meet those limits (hopefully there is none...).
Whoops, I meant cannot rather than can.
ok, makes sense, thank you for the clarification.
Quoted 3 lines
Cheers,
Signature · 3 lines
Regards, Oleksandr Tyshchenko
Hi Oleksandr,
Quoted 149 lines
-----Original Message-----
From: Oleksandr <[email protected]>
Sent: 2021年8月10日 2:25
To: Julien Grall <[email protected]>
Cc: Stefano Stabellini <[email protected]>; Andrew Cooper <[email protected]>; [email protected]; Oleksandr Tyshchenko <[email protected]>; Daniel De Graaf <[email protected]>; Daniel P. Smith <[email protected]>; Ian Jackson <[email protected]>; Wei Liu <[email protected]>; George Dunlap <[email protected]>; Jan Beulich <[email protected]>; Volodymyr Babchuk <[email protected]>; Roger Pau Monné <[email protected]>; Bertrand Marquis <[email protected]>; Wei Chen <[email protected]>
Subject: Re: [RFC PATCH] xen/memory: Introduce a hypercall to provide unallocated spaceQuoted 1 line · Julien Grall
Hi Oleksandr,
Hi Julien.
Thank you for the input.
Quoted 13 lines · Oleksandr
Quoted 11 lines · Stefano Stabellini
Hi Stefano
Quoted 6 lines · Julien Grall
Quoted 5 lines
Quoted 5 lines
+#define
GUEST_SAFE_RANGE_BASExen_mk_ullong(0xDE00000000)/* 128GB */
+#defineGUEST_SAFE_RANGE_SIZExen_mk_ullong(0x0200000000)While the possible new DT bindings has not been agreed yet, I re-
used
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
existing "reg" property under the hypervisor node to pass safe range as a
second region,
https://elixir.bootlin.com/linux/v5.14-rc4/source/Documentation/devicetree/bindings/arm/xen.txt#L10:
Quoted 51 lines
Quoted 34 lines
Quoted 26 lines
Quoted 15 lines
Quoted 1 line
So a single region works for a guest today, but for dom0 we will need multiple
regions because it is may be difficult to find enough contiguous space for a
single region.That said, as dom0 is mapped 1:1 (including some guest mapping), there is also
the question where to allocate the safe region. For grant table, we so far
re-use the Xen address space because it is assumed it will space will always
be bigger than the grant table.Quoted 3 lines
I am not sure yet where we could allocate the safe regions. Stefano, do you
have any ideas?The safest choice would be the address range corresponding to memory (/memory) not already allocated to Dom0.
For instance from my last boot logs:
(XEN) Allocating 1:1 mappings totalling 1600MB for dom0:
(XEN) BANK[0] 0x00000010000000-0x00000070000000 (1536MB) (XEN) BANK[1] 0x00000078000000-0x0000007c000000 (64MB)Quoted 5 lines
All the other ranges could be given as unallocated space:
-
0x0-0x10000000
-0x70000000-0x78000000
- 0x8_0000_0000 - 0x8_8000_0000Thank you for the ideas.
Quoted 10 lines
If I got the idea correctly, yes, as these ranges represent the real RAM, so no I/O would be in conflict with them and as the result - no overlaps would be expected.
But, I wonder, would this work if we have IOMMU enabled for Dom0 and need to establish 1:1 mapping for the DMA devices to work with grant mappings...
Inarm_iommu_map_page()we callguest_physmap_add_entry()with gfn = mfn, so the question is could we end up with this new gfn replacing the valid mapping
(with gfn allocated from the safe region)?Right, when we enable the IOMMU for dom0, Xen will add an extra mapping with GFN == MFN for foreign and grant pages. This is because Linux is not aware that whether a device is protected by an IOMMU. Therefore it is assuming it is not and will use the MFN to configure for DMA transaction.
Quoted 6 lines
We can't remove the mapping without significant changes in Linux and Xen. I would not mandate them for this work.
That said, I think it would be acceptable to have different way to find the region depending on the dom0 configuration. So we could use the RAM not used by dom0 when the IOMMU is turned off.
OK
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
The second best choice would be an hole: an address range not used by anybody else (no reg property) and also not even mappable by a bus
(not
Quoted 10 lines
Quoted 3 lines
Quoted 3 lines
covered by a ranges property). This is not the best choice because there
can cases where physical resources appear afterwards.Are you saying that the original device-tree doesn't even describe them in any way (i.e. reserved...)?
Quoted 2 lines
Unfortunately, yes.
Quoted 8 lines
So the decision where the safe region is located will be done by Xen. There is no involvement of the domain (it will discover the region from the DT). Therefore, I don't think we need to think about everything right now as we could adapt this is exact region is not part of the stable ABI.
The hotplug is one I would defer because this is not supported (and quite likely not working) in Xen upstream today.
Sounds reasonable.
Quoted 6 lines
Now regarding the case where dom0 is using the IOMMU. The assumption is Xen will be able to figure out all the regions used from the firmware table (ACPI or DT).
Quoted 6 lines
AFAIK, this assumption would be correct for DT. However, for ACPI, I remember we were not able to find all the MMIOs region in Xen (see [1] and [2]). So even this solution would not work for ACPI.
If I am not mistaken, we don't support IOMMU with ACPI yet. So we could defer the problem to when this is going to be supported.
Sounds reasonable.
Quoted 7 lines
To summarize:
0. Skip ACPI case for now, implement for DT case
1. If IOMMU is enabled for Dom0 -> provide holes found in Host DT as safe ranges
Does static allocation and direct mapping driver domain can be treated as this case?
Quoted 20 lines
I would take into the account holes >= 1MB. I am wondering, do we need a special alignment here other than a
PAGE_SIZE?2. If IOMMU is disabled for Dom0 -> provide RAM which not assigned to Dom0 as safe ranges
We could even provide holes here as well.
Quoted 7 lines
Cheers,
[1] https://marc.info/?l=linux-arm-kernel&m=148469169210500&w=2
[2] Xen commit80f9c316708400cea4417e36337267d3b26591db--
Regards,Oleksandr Tyshchenko
Quoted 1 line · Wei Chen
Hi Oleksandr,
Hi Wei, Julien.
Quoted 138 lines
Quoted 138 lines
-----Original Message-----
From: Oleksandr <[email protected]>
Sent: 2021年8月10日 2:25
To: Julien Grall <[email protected]>
Cc: Stefano Stabellini <[email protected]>; Andrew Cooper <[email protected]>; [email protected]; Oleksandr Tyshchenko <[email protected]>; Daniel De Graaf <[email protected]>; Daniel P. Smith <[email protected]>; Ian Jackson <[email protected]>; Wei Liu <[email protected]>; George Dunlap <[email protected]>; Jan Beulich <[email protected]>; Volodymyr Babchuk <[email protected]>; Roger Pau Monné <[email protected]>; Bertrand Marquis <[email protected]>; Wei Chen <[email protected]>
Subject: Re: [RFC PATCH] xen/memory: Introduce a hypercall to provide unallocated spaceQuoted 1 line · Julien Grall
Hi Oleksandr,
Hi Julien.
Thank you for the input.
Quoted 11 lines · Oleksandr
Quoted 10 lines · Stefano Stabellini
Hi Stefano
Quoted 6 lines · Julien Grall
Quoted 5 lines
Quoted 5 lines
+#define
GUEST_SAFE_RANGE_BASExen_mk_ullong(0xDE00000000)/* 128GB */
+#defineGUEST_SAFE_RANGE_SIZExen_mk_ullong(0x0200000000)While the possible new DT bindings has not been agreed yet, I re-
used
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
existing "reg" property under the hypervisor node to pass safe range as a
second region,
https://elixir.bootlin.com/linux/v5.14-rc4/source/Documentation/devicetree/bindings/arm/xen.txt#L10:
Quoted 48 lines
Quoted 32 lines
Quoted 25 lines
Quoted 14 lines
So a single region works for a guest today, but for dom0 we will need multiple
regions because it is may be difficult to find enough contiguous space for a
single region.That said, as dom0 is mapped 1:1 (including some guest mapping), there is also
the question where to allocate the safe region. For grant table, we so far
re-use the Xen address space because it is assumed it will space will always
be bigger than the grant table.Quoted 3 lines
I am not sure yet where we could allocate the safe regions. Stefano, do you
have any ideas?The safest choice would be the address range corresponding to memory (/memory) not already allocated to Dom0.
For instance from my last boot logs:
(XEN) Allocating 1:1 mappings totalling 1600MB for dom0:
(XEN) BANK[0] 0x00000010000000-0x00000070000000 (1536MB) (XEN) BANK[1] 0x00000078000000-0x0000007c000000 (64MB)Quoted 5 lines
All the other ranges could be given as unallocated space:
-
0x0-0x10000000
-0x70000000-0x78000000
- 0x8_0000_0000 - 0x8_8000_0000Thank you for the ideas.
Quoted 10 lines
If I got the idea correctly, yes, as these ranges represent the real RAM, so no I/O would be in conflict with them and as the result - no overlaps would be expected.
But, I wonder, would this work if we have IOMMU enabled for Dom0 and need to establish 1:1 mapping for the DMA devices to work with grant mappings...
Inarm_iommu_map_page()we callguest_physmap_add_entry()with gfn = mfn, so the question is could we end up with this new gfn replacing the valid mapping
(with gfn allocated from the safe region)?Right, when we enable the IOMMU for dom0, Xen will add an extra mapping with GFN == MFN for foreign and grant pages. This is because Linux is not aware that whether a device is protected by an IOMMU. Therefore it is assuming it is not and will use the MFN to configure for DMA transaction.
Quoted 6 lines
We can't remove the mapping without significant changes in Linux and Xen. I would not mandate them for this work.
That said, I think it would be acceptable to have different way to find the region depending on the dom0 configuration. So we could use the RAM not used by dom0 when the IOMMU is turned off.
OK
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
The second best choice would be an hole: an address range not used by anybody else (no reg property) and also not even mappable by a bus
(not
Quoted 7 lines
Quoted 3 lines
Quoted 3 lines
covered by a ranges property). This is not the best choice because there
can cases where physical resources appear afterwards.Are you saying that the original device-tree doesn't even describe them in any way (i.e. reserved...)?
Quoted 1 line
Unfortunately, yes.
Quoted 8 lines
So the decision where the safe region is located will be done by Xen. There is no involvement of the domain (it will discover the region from the DT). Therefore, I don't think we need to think about everything right now as we could adapt this is exact region is not part of the stable ABI.
The hotplug is one I would defer because this is not supported (and quite likely not working) in Xen upstream today.
Sounds reasonable.
Quoted 11 lines
Now regarding the case where dom0 is using the IOMMU. The assumption is Xen will be able to figure out all the regions used from the firmware table (ACPI or DT).
AFAIK, this assumption would be correct for DT. However, for ACPI, I remember we were not able to find all the MMIOs region in Xen (see [1] and [2]). So even this solution would not work for ACPI.
If I am not mistaken, we don't support IOMMU with ACPI yet. So we could defer the problem to when this is going to be supported.
Sounds reasonable.
To summarize:
Quoted 7 lines
Quoted 5 lines
0. Skip ACPI case for now, implement for DT case
1. If IOMMU is enabled for Dom0 -> provide holes found in Host DT as safe ranges
Does static allocation and direct mapping driver domain can be treated as this case?
I am not sure I can answer this question correctly due to the limited knowledge of these features.
But, it feels to me that holes solution would work, at least I don't see why not.
I wonder, why these can't be treated as the case #2 (where we provide not assigned RAM), I also don't see why not, however there might be pitfalls with direct mapped driver domain. Julien, do you have any opinion on this?
Quoted 20 lines
Quoted 20 lines
I would take into the account holes >= 1MB. I am wondering, do we need a special alignment here other than a
PAGE_SIZE?2. If IOMMU is disabled for Dom0 -> provide RAM which not assigned to Dom0 as safe ranges
We could even provide holes here as well.
Quoted 6 lines
Cheers,
[1] https://marc.info/?l=linux-arm-kernel&m=148469169210500&w=2
[2] Xen commit80f9c316708400cea4417e36337267d3b26591db--
Regards,Oleksandr Tyshchenko
Signature · 3 lines
Regards, Oleksandr Tyshchenko
Quoted 5 lines · Oleksandr
Quoted 1 line · Wei Chen
Hi Oleksandr,
Hi Wei, Julien.
Hi,
Quoted 142 lines
Quoted 142 lines
Quoted 142 lines
-----Original Message-----
From: Oleksandr <[email protected]>
Sent: 2021年8月10日 2:25
To: Julien Grall <[email protected]>
Cc: Stefano Stabellini <[email protected]>; Andrew Cooper <[email protected]>; [email protected]; Oleksandr Tyshchenko <[email protected]>; Daniel De Graaf <[email protected]>; Daniel P. Smith <[email protected]>; Ian Jackson <[email protected]>; Wei Liu <[email protected]>; George Dunlap <[email protected]>; Jan Beulich <[email protected]>; Volodymyr Babchuk <[email protected]>; Roger Pau Monné <[email protected]>; Bertrand Marquis <[email protected]>; Wei Chen <[email protected]>
Subject: Re: [RFC PATCH] xen/memory: Introduce a hypercall to provide unallocated spaceQuoted 1 line · Julien Grall
Hi Oleksandr,
Hi Julien.
Thank you for the input.
Quoted 11 lines · Oleksandr
Quoted 10 lines · Stefano Stabellini
Hi Stefano
Quoted 6 lines · Julien Grall
Quoted 5 lines
Quoted 5 lines
+#define
GUEST_SAFE_RANGE_BASExen_mk_ullong(0xDE00000000)/* 128GB */
+#defineGUEST_SAFE_RANGE_SIZExen_mk_ullong(0x0200000000)While the possible new DT bindings has not been agreed yet, I re-
used
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
existing "reg" property under the hypervisor node to pass safe range as a
second region,
https://elixir.bootlin.com/linux/v5.14-rc4/source/Documentation/devicetree/bindings/arm/xen.txt#L10:
Quoted 48 lines
Quoted 32 lines
Quoted 25 lines
Quoted 14 lines
So a single region works for a guest today, but for dom0 we will need multiple
regions because it is may be difficult to find enough contiguous space for a
single region.That said, as dom0 is mapped 1:1 (including some guest mapping), there is also
the question where to allocate the safe region. For grant table, we so far
re-use the Xen address space because it is assumed it will space will always
be bigger than the grant table.Quoted 3 lines
I am not sure yet where we could allocate the safe regions. Stefano, do you
have any ideas?The safest choice would be the address range corresponding to memory (/memory) not already allocated to Dom0.
For instance from my last boot logs:
(XEN) Allocating 1:1 mappings totalling 1600MB for dom0:
(XEN) BANK[0] 0x00000010000000-0x00000070000000 (1536MB) (XEN) BANK[1] 0x00000078000000-0x0000007c000000 (64MB)Quoted 5 lines
All the other ranges could be given as unallocated space:
-
0x0-0x10000000
-0x70000000-0x78000000
- 0x8_0000_0000 - 0x8_8000_0000Thank you for the ideas.
Quoted 10 lines
If I got the idea correctly, yes, as these ranges represent the real RAM, so no I/O would be in conflict with them and as the result - no overlaps would be expected.
But, I wonder, would this work if we have IOMMU enabled for Dom0 and need to establish 1:1 mapping for the DMA devices to work with grant mappings...
Inarm_iommu_map_page()we callguest_physmap_add_entry()with gfn = mfn, so the question is could we end up with this new gfn replacing the valid mapping
(with gfn allocated from the safe region)?Right, when we enable the IOMMU for dom0, Xen will add an extra mapping with GFN == MFN for foreign and grant pages. This is because Linux is not aware that whether a device is protected by an IOMMU. Therefore it is assuming it is not and will use the MFN to configure for DMA transaction.
Quoted 6 lines
We can't remove the mapping without significant changes in Linux and Xen. I would not mandate them for this work.
That said, I think it would be acceptable to have different way to find the region depending on the dom0 configuration. So we could use the RAM not used by dom0 when the IOMMU is turned off.
OK
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
The second best choice would be an hole: an address range not used by anybody else (no reg property) and also not even mappable by a bus
(not
Quoted 7 lines
Quoted 3 lines
Quoted 3 lines
covered by a ranges property). This is not the best choice because there
can cases where physical resources appear afterwards.Are you saying that the original device-tree doesn't even describe them in any way (i.e. reserved...)?
Quoted 1 line
Unfortunately, yes.
Quoted 8 lines
So the decision where the safe region is located will be done by Xen. There is no involvement of the domain (it will discover the region from the DT). Therefore, I don't think we need to think about everything right now as we could adapt this is exact region is not part of the stable ABI.
The hotplug is one I would defer because this is not supported (and quite likely not working) in Xen upstream today.
Sounds reasonable.
Quoted 11 lines
Now regarding the case where dom0 is using the IOMMU. The assumption is Xen will be able to figure out all the regions used from the firmware table (ACPI or DT).
AFAIK, this assumption would be correct for DT. However, for ACPI, I remember we were not able to find all the MMIOs region in Xen (see [1] and [2]). So even this solution would not work for ACPI.
If I am not mistaken, we don't support IOMMU with ACPI yet. So we could defer the problem to when this is going to be supported.
Sounds reasonable.
To summarize:
0. Skip ACPI case for now, implement for DT case
1. If IOMMU is enabled for Dom0 -> provide holes found in Host DT as safe ranges
Quoted 8 lines
Quoted 2 lines
Does static allocation and direct mapping driver domain can be treated as this case?
I am not sure I can answer this question correctly due to the limited knowledge of these features.
But, it feels to me that holes solution would work, at least I don't see why not.
I wonder, why these can't be treated as the case #2 (where we provide not assigned RAM), I also don't see why not, however there might be pitfalls with direct mapped driver domain. Julien, do you have any opinion on this?
So whether the memory is statically allocated or dynamically allocated should not matter here. For direct mapped domain, then they should be treated the same way as dom0.
By that I mean if the IOMMU is not enabled for the domain, then we can use thew unallocated RAM. Otherwise, we would need to find some holes.
Cheers,
Signature · 1 line
Julien Grall
On 10.08.21 19:21, Julien Grall wrote:
Hi Julien
Quoted 156 lines · Oleksandr
Quoted 5 lines · Wei Chen
Quoted 1 line
Hi Oleksandr,
Hi Wei, Julien.
Hi,
Quoted 146 lines
Quoted 137 lines
Quoted 137 lines
-----Original Message-----
From: Oleksandr <[email protected]>
Sent: 2021年8月10日 2:25
To: Julien Grall <[email protected]>
Cc: Stefano Stabellini <[email protected]>; Andrew Cooper <[email protected]>; [email protected]; Oleksandr Tyshchenko <[email protected]>; Daniel De Graaf <[email protected]>; Daniel P. Smith <[email protected]>; Ian Jackson <[email protected]>; Wei Liu <[email protected]>; George Dunlap <[email protected]>; Jan Beulich <[email protected]>; Volodymyr Babchuk <[email protected]>; Roger Pau Monné <[email protected]>; Bertrand Marquis <[email protected]>; Wei Chen <[email protected]>
Subject: Re: [RFC PATCH] xen/memory: Introduce a hypercall to provide unallocated spaceQuoted 1 line · Julien Grall
Hi Oleksandr,
Hi Julien.
Thank you for the input.
Quoted 11 lines · Oleksandr
Quoted 10 lines · Stefano Stabellini
Hi Stefano
Quoted 6 lines · Julien Grall
Quoted 5 lines
Quoted 5 lines
+#define
GUEST_SAFE_RANGE_BASExen_mk_ullong(0xDE00000000)/* 128GB */
+#defineGUEST_SAFE_RANGE_SIZExen_mk_ullong(0x0200000000)While the possible new DT bindings has not been agreed yet, I re-
used
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
existing "reg" property under the hypervisor node to pass safe range as a
second region,
https://elixir.bootlin.com/linux/v5.14-rc4/source/Documentation/devicetree/bindings/arm/xen.txt#L10:
Quoted 48 lines
Quoted 32 lines
Quoted 25 lines
Quoted 14 lines
So a single region works for a guest today, but for dom0 we will need multiple
regions because it is may be difficult to find enough contiguous space for a
single region.That said, as dom0 is mapped 1:1 (including some guest mapping), there is also
the question where to allocate the safe region. For grant table, we so far
re-use the Xen address space because it is assumed it will space will always
be bigger than the grant table.Quoted 3 lines
I am not sure yet where we could allocate the safe regions. Stefano, do you
have any ideas?The safest choice would be the address range corresponding to memory (/memory) not already allocated to Dom0.
For instance from my last boot logs:
(XEN) Allocating 1:1 mappings totalling 1600MB for dom0:
(XEN) BANK[0] 0x00000010000000-0x00000070000000 (1536MB) (XEN) BANK[1] 0x00000078000000-0x0000007c000000 (64MB)Quoted 5 lines
All the other ranges could be given as unallocated space:
-
0x0-0x10000000
-0x70000000-0x78000000
- 0x8_0000_0000 - 0x8_8000_0000Thank you for the ideas.
Quoted 10 lines
If I got the idea correctly, yes, as these ranges represent the real RAM, so no I/O would be in conflict with them and as the result - no overlaps would be expected.
But, I wonder, would this work if we have IOMMU enabled for Dom0 and need to establish 1:1 mapping for the DMA devices to work with grant mappings...
Inarm_iommu_map_page()we callguest_physmap_add_entry()with gfn = mfn, so the question is could we end up with this new gfn replacing the valid mapping
(with gfn allocated from the safe region)?Right, when we enable the IOMMU for dom0, Xen will add an extra mapping with GFN == MFN for foreign and grant pages. This is because Linux is not aware that whether a device is protected by an IOMMU. Therefore it is assuming it is not and will use the MFN to configure for DMA transaction.
Quoted 6 lines
We can't remove the mapping without significant changes in Linux and Xen. I would not mandate them for this work.
That said, I think it would be acceptable to have different way to find the region depending on the dom0 configuration. So we could use the RAM not used by dom0 when the IOMMU is turned off.
OK
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
The second best choice would be an hole: an address range not used by anybody else (no reg property) and also not even mappable by a bus
(not
Quoted 7 lines
Quoted 3 lines
Quoted 3 lines
covered by a ranges property). This is not the best choice because there
can cases where physical resources appear afterwards.Are you saying that the original device-tree doesn't even describe them in any way (i.e. reserved...)?
Quoted 1 line
Unfortunately, yes.
Quoted 8 lines
So the decision where the safe region is located will be done by Xen. There is no involvement of the domain (it will discover the region from the DT). Therefore, I don't think we need to think about everything right now as we could adapt this is exact region is not part of the stable ABI.
The hotplug is one I would defer because this is not supported (and quite likely not working) in Xen upstream today.
Sounds reasonable.
Quoted 11 lines
Now regarding the case where dom0 is using the IOMMU. The assumption is Xen will be able to figure out all the regions used from the firmware table (ACPI or DT).
AFAIK, this assumption would be correct for DT. However, for ACPI, I remember we were not able to find all the MMIOs region in Xen (see [1] and [2]). So even this solution would not work for ACPI.
If I am not mistaken, we don't support IOMMU with ACPI yet. So we could defer the problem to when this is going to be supported.
Sounds reasonable.
To summarize:
Quoted 7 lines
Quoted 5 lines
0. Skip ACPI case for now, implement for DT case
1. If IOMMU is enabled for Dom0 -> provide holes found in Host DT as safe ranges
Does static allocation and direct mapping driver domain can be treated as this case?
I am not sure I can answer this question correctly due to the limited knowledge of these features.
Quoted 8 lines
Quoted 4 lines
But, it feels to me that holes solution would work, at least I don't see why not.
I wonder, why these can't be treated as the case #2 (where we provide not assigned RAM), I also don't see why not, however there might be pitfalls with direct mapped driver domain. Julien, do you have any opinion on this?
So whether the memory is statically allocated or dynamically allocated should not matter here. For direct mapped domain, then they should be treated the same way as dom0.
By that I mean if the IOMMU is not enabled for the domain, then we can use thew unallocated RAM. Otherwise, we would need to find some holes.
OK, thank you for the clarification.
Quoted 4 lines
Cheers,
Signature · 3 lines
Regards, Oleksandr Tyshchenko
Hi,
I am writing down here what we discussed on another thread and on IRC. This will be easier to track in a single thread.
Quoted 4 lines · Julien Grall
Quoted 1 line · Oleksandr
Now, I am wondering, would it be possible to update/clarify the current "reg" purpose and use it to pass a safe unallocated space for any Xen specific mappings (grant, foreign, whatever) instead of just for the grant table region. In case, it is not allowed for any reason (compatibility PoV, etc), would it be possible to extend a property by passing an extra range separately, something similar to how I described above?
I think it should be fine to re-use the same region so long the size of the first bank is at least the size of the original region.
While answering to the DT binding question on the DT ML, I realized that this is probably not going to be fine because there is a bug in Xen when mapping grant-table frame.
The function gnttab_map_frame() is used to map the grant table frame. If there is an old mapping, it will first remove it.
The function is using the helper gnttab_map_frame() to find the corresponding GFN or return INVALID_GFN if not mapped.
On Arm, gnttab_map_frame() is implementing using an array index by the grant table frame number. The trouble is we don't update the array when the page is unmapped. So if the GFN is re-used before the grant-table is remapped, then we will end up to remove whatever was mapped there (this could be a foreign page...).
This behavior already happens today as the toolstack will use the first GFN of the region if Linux doesn't support the acquire resource interface. We are getting away in the Linux because the toolstack only map the first grant table frame and:
- Newer Linux will not used the region provided by the DT and nothing
will be mapped there.
- Older Linux will use the region but still map the grant table frame
0 to the same GFN.
I am not sure about U-boot and other OSes here.
This is not new but it is going to be become a bigger source of problem (read more chance to hit it) as we try to re-use the first region.
This means the first region should exclusively used for the grant-table (in a specific order) until the issue is properly fixed.
A potential fix is to update the array in p2m_put_l3_page(). The default max size of the array is 1024, so it might be fine to just walk it (it would be simply a comparison).
Note that this is not a problem on x86 because the is using the M2P. So when a mapping is removed, the mapping MFN -> GFN will also be removed.
Cheers,
Signature · 1 line
Julien Grall
Quoted 1 line · Julien Grall
Hi,
Hi Julien.
Quoted 21 lines
I am writing down here what we discussed on another thread and on IRC. This will be easier to track in a single thread.
Quoted 4 lines · Julien Grall
Quoted 1 line · Oleksandr
Now, I am wondering, would it be possible to update/clarify the current "reg" purpose and use it to pass a safe unallocated space for any Xen specific mappings (grant, foreign, whatever) instead of just for the grant table region. In case, it is not allowed for any reason (compatibility PoV, etc), would it be possible to extend a property by passing an extra range separately, something similar to how I described above?
I think it should be fine to re-use the same region so long the size of the first bank is at least the size of the original region.
While answering to the DT binding question on the DT ML, I realized that this is probably not going to be fine because there is a bug in Xen when mapping grant-table frame.
The function
gnttab_map_frame()is used to map the grant table frame. If there is an old mapping, it will first remove it.The function is using the helper
gnttab_map_frame()to find the corresponding GFN or returnINVALID_GFNif not mapped.On Arm,
gnttab_map_frame()is implementing using an array index by the grant table frame number. The trouble is we don't update the array when the page is unmapped. So if the GFN is re-used before the grant-table is remapped, then we will end up to remove whatever was mapped there (this could be a foreign page...).This behavior already happens today as the toolstack will use the first GFN of the region if Linux doesn't support the acquire resource interface. We are getting away in the Linux because the toolstack only map the first grant table frame and:
- Newer Linux will not used the region provided by the DT and nothing will be mapped there.
- Older Linux will use the region but still map the grant table frame 0 to the same GFN.
Quoted 5 lines
I am not sure about U-boot and other OSes here.
This is not new but it is going to be become a bigger source of problem (read more chance to hit it) as we try to re-use the first region.
This means the first region should exclusively used for the grant-table (in a specific order) until the issue is properly fixed.
Thank you for the explanation, it is clear now.
Quoted 3 lines
A potential fix is to update the array in
p2m_put_l3_page(). The default max size of the array is 1024, so it might be fine to just walk it (it would be simply a comparison).
I think, this would work. Looks like we don't need to walk for each gfn which is being freed, we could just filter it by p2m_is_ram() ...
Quoted 6 lines
Note that this is not a problem on x86 because the is using the M2P. So when a mapping is removed, the mapping MFN -> GFN will also be removed.
Cheers,
Signature · 3 lines
Regards, Oleksandr Tyshchenko
Quoted 3 lines · Oleksandr
On 09.08.21 17:51, Julien Grall wrote:
Hi Julien.
Hi Oleksandr,
Quoted 26 lines
Quoted 20 lines
I am writing down here what we discussed on another thread and on IRC. This will be easier to track in a single thread.
Quoted 4 lines · Julien Grall
Quoted 1 line · Oleksandr
Now, I am wondering, would it be possible to update/clarify the current "reg" purpose and use it to pass a safe unallocated space for any Xen specific mappings (grant, foreign, whatever) instead of just for the grant table region. In case, it is not allowed for any reason (compatibility PoV, etc), would it be possible to extend a property by passing an extra range separately, something similar to how I described above?
I think it should be fine to re-use the same region so long the size of the first bank is at least the size of the original region.
While answering to the DT binding question on the DT ML, I realized that this is probably not going to be fine because there is a bug in Xen when mapping grant-table frame.
The function
gnttab_map_frame()is used to map the grant table frame. If there is an old mapping, it will first remove it.The function is using the helper
gnttab_map_frame()to find the corresponding GFN or returnINVALID_GFNif not mapped.On Arm,
gnttab_map_frame()is implementing using an array index by the grant table frame number. The trouble is we don't update the array when the page is unmapped. So if the GFN is re-used before the grant-table is remapped, then we will end up to remove whatever was mapped there (this could be a foreign page...).This behavior already happens today as the toolstack will use the first GFN of the region if Linux doesn't support the acquire resource interface. We are getting away in the Linux because the toolstack only map the first grant table frame and:
- Newer Linux will not used the region provided by the DT and nothing will be mapped there.
- Older Linux will use the region but still map the grant table frame 0 to the same GFN.Quoted 5 lines
I am not sure about U-boot and other OSes here.
This is not new but it is going to be become a bigger source of problem (read more chance to hit it) as we try to re-use the first region.
This means the first region should exclusively used for the grant-table (in a specific order) until the issue is properly fixed.
Quoted 8 lines
Thank you for the explanation, it is clear now.
Quoted 3 lines
A potential fix is to update the array in
p2m_put_l3_page(). The default max size of the array is 1024, so it might be fine to just walk it (it would be simply a comparison).I think, this would work. Looks like we don't need to walk for each gfn which is being freed, we could just filter it by
p2m_is_ram()...
Well. This would still potentially result to a few unnecessary walk. I would consider to introduce a new P2M type or possibly add a check if the page is in xenheap (grant-table are xenheap pages so far).
Cheers,
Signature · 1 line
Julien Grall
On 09.08.21 20:18, Julien Grall wrote:
Hi Julien
Quoted 40 lines · Oleksandr
Quoted 3 lines · Julien Grall
Hi Julien.
Hi Oleksandr,
Quoted 31 lines
Quoted 20 lines
I am writing down here what we discussed on another thread and on IRC. This will be easier to track in a single thread.
Quoted 4 lines · Julien Grall
Quoted 1 line · Oleksandr
Now, I am wondering, would it be possible to update/clarify the current "reg" purpose and use it to pass a safe unallocated space for any Xen specific mappings (grant, foreign, whatever) instead of just for the grant table region. In case, it is not allowed for any reason (compatibility PoV, etc), would it be possible to extend a property by passing an extra range separately, something similar to how I described above?
I think it should be fine to re-use the same region so long the size of the first bank is at least the size of the original region.
While answering to the DT binding question on the DT ML, I realized that this is probably not going to be fine because there is a bug in Xen when mapping grant-table frame.
The function
gnttab_map_frame()is used to map the grant table frame. If there is an old mapping, it will first remove it.The function is using the helper
gnttab_map_frame()to find the corresponding GFN or returnINVALID_GFNif not mapped.On Arm,
gnttab_map_frame()is implementing using an array index by the grant table frame number. The trouble is we don't update the array when the page is unmapped. So if the GFN is re-used before the grant-table is remapped, then we will end up to remove whatever was mapped there (this could be a foreign page...).This behavior already happens today as the toolstack will use the first GFN of the region if Linux doesn't support the acquire resource interface. We are getting away in the Linux because the toolstack only map the first grant table frame and:
- Newer Linux will not used the region provided by the DT and nothing will be mapped there.
- Older Linux will use the region but still map the grant table frame 0 to the same GFN.Quoted 5 lines
I am not sure about U-boot and other OSes here.
This is not new but it is going to be become a bigger source of problem (read more chance to hit it) as we try to re-use the first region.
This means the first region should exclusively used for the grant-table (in a specific order) until the issue is properly fixed.
Thank you for the explanation, it is clear now.
Quoted 2 lines
Quoted 5 lines
Quoted 3 lines
Quoted 1 line
A potential fix is to update the array in
p2m_put_l3_page(). The default max size of the array is 1024, so it might be fine to just walk it (it would be simply a comparison).I think, this would work. Looks like we don't need to walk for each gfn which is being freed, we could just filter it by
p2m_is_ram()...Well. This would still potentially result to a few unnecessary walk. I would consider to introduce a new P2M type or possibly add a check if the page is in xenheap (grant-table are xenheap pages so far).
Indeed, this would be better, personally I would prefer to check if page is in xenheap.
Quoted 4 lines
Cheers,
Signature · 3 lines
Regards, Oleksandr Tyshchenko
Quoted 1 line · Julien Grall
Hi,
Hi Julien, all
Quoted 25 lines
I am writing down here what we discussed on another thread and on IRC. This will be easier to track in a single thread.
Quoted 4 lines · Julien Grall
Quoted 1 line · Oleksandr
Now, I am wondering, would it be possible to update/clarify the current "reg" purpose and use it to pass a safe unallocated space for any Xen specific mappings (grant, foreign, whatever) instead of just for the grant table region. In case, it is not allowed for any reason (compatibility PoV, etc), would it be possible to extend a property by passing an extra range separately, something similar to how I described above?
I think it should be fine to re-use the same region so long the size of the first bank is at least the size of the original region.
While answering to the DT binding question on the DT ML, I realized that this is probably not going to be fine because there is a bug in Xen when mapping grant-table frame.
The function
gnttab_map_frame()is used to map the grant table frame. If there is an old mapping, it will first remove it.The function is using the helper
gnttab_map_frame()to find the corresponding GFN or returnINVALID_GFNif not mapped.On Arm,
gnttab_map_frame()is implementing using an array index by the grant table frame number. The trouble is we don't update the array when the page is unmapped. So if the GFN is re-used before the grant-table is remapped, then we will end up to remove whatever was mapped there (this could be a foreign page...).This behavior already happens today as the toolstack will use the first GFN of the region if Linux doesn't support the acquire resource interface. We are getting away in the Linux because the toolstack only map the first grant table frame and:
- Newer Linux will not used the region provided by the DT and nothing will be mapped there.
- Older Linux will use the region but still map the grant table frame 0 to the same GFN.I am not sure about U-boot and other OSes here.
This is not new but it is going to be become a bigger source of problem (read more chance to hit it) as we try to re-use the first region.
Quoted 7 lines
This means the first region should exclusively used for the grant-table (in a specific order) until the issue is properly fixed.
A potential fix is to update the array in
p2m_put_l3_page(). The default max size of the array is 1024, so it might be fine to just walk it (it would be simply a comparison).Note that this is not a problem on x86 because the is using the M2P. So when a mapping is removed, the mapping MFN -> GFN will also be removed.
Cheers,
The fix is already pushed:
https://lore.kernel.org/xen-devel/[email protected]/
Thanks!
Signature · 3 lines
Regards, Oleksandr Tyshchenko
Quoted 14 lines · Oleksandr
Well, if new hypercall and, what is more, "the querying hypervisor at runtime to find unused space" model itself is not welcome, I am ok, let's try to create a working system,
may we please find a common ground to move this forward (at least on Arm for now, the foreign mapping is the most important question).I got the proposed idea in general, but I haven't connected all dots yet, some points need clarification.
1. The safe range must be defined/allocated in advance and must remain const during the runtime. The safe range must be chosen by the toolstack. [For the initial implementation we can start with some large value (128GB) as discussed above]
Quoted 7 lines
Questions:
- Do we need to inform Xen about that range (via
domain_createhypercall, etc)?
- What will be in charge of guaranteeing the safety of that range at runtime (reject new mapping requests with possible overlaps, etc), Xen, toolstack or both?
Well, what other entity than Xen could enforce this? (By implication, the answer to the earlier question can imo only be "yes", unless it's Xen itself to establish the region.)
Quoted 4 lines
- Where that range should be located in guest address space, should that range be the same for all domains (how
GUEST_GNTTAB_BASE(SIZE)for example) or it should be calculated based on actualguest_ram_base(size)for a particular domain?
The default size may well be fixed or amount-of-memory-dependent, but I think there will need to be a way to enlarge the region for guests with particular needs.
Quoted 7 lines
- What about a safe range the Dom0 can use itself? Xen should choose it for Dom0 the same way how toolstack chooses it for other domains, correct?
2. The safe range must be provided to domain using the firmware table. [We can start with the DTB and leave ACPI unimplemented for now, assuming we will be able to solve open questions as discussed above]
Quoted 5 lines
Questions:
- Do we need distinguish between foreign and grant mappings at the domain side at all? Can the same safe range be used for all types of mapping?
Like Stefano I don't think so.
Jan
On 03.08.21 15:53, Jan Beulich wrote:
Hi, Jan
Thank you for the input.
Quoted 17 lines · Oleksandr
Quoted 16 lines
Well, if new hypercall and, what is more, "the querying hypervisor at runtime to find unused space" model itself is not welcome, I am ok, let's try to create a working system,
may we please find a common ground to move this forward (at least on Arm for now, the foreign mapping is the most important question).I got the proposed idea in general, but I haven't connected all dots yet, some points need clarification.
1. The safe range must be defined/allocated in advance and must remain const during the runtime. The safe range must be chosen by the toolstack. [For the initial implementation we can start with some large value (128GB) as discussed above]
Questions:
Quoted 8 lines
Quoted 5 lines
- Do we need to inform Xen about that range (via
domain_createhypercall, etc)?
- What will be in charge of guaranteeing the safety of that range at runtime (reject new mapping requests with possible overlaps, etc), Xen, toolstack or both?Well, what other entity than Xen could enforce this? (By implication, the answer to the earlier question can imo only be "yes", unless it's Xen itself to establish the region.)
Indeed, agree.
Quoted 8 lines
Quoted 5 lines
- Where that range should be located in guest address space, should that range be the same for all domains (how
GUEST_GNTTAB_BASE(SIZE)for example) or it should be calculated based on actualguest_ram_base(size)for a particular domain?The default size may well be fixed or amount-of-memory-dependent, but I think there will need to be a way to enlarge the region for guests with particular needs.
Well, but why we couldn't just make a large chunk by default which would
satisfy all guests, as it was mentioned earlier in this thread "as it doesn't consume resource when not being used" to avoid an extra configuration option, etc?
Quoted 8 lines
Quoted 8 lines
- What about a safe range the Dom0 can use itself? Xen should choose it for Dom0 the same way how toolstack chooses it for other domains, correct?
2. The safe range must be provided to domain using the firmware table. [We can start with the DTB and leave ACPI unimplemented for now, assuming we will be able to solve open questions as discussed above]
Quoted 6 lines
Quoted 5 lines
Questions:
- Do we need distinguish between foreign and grant mappings at the domain side at all? Can the same safe range be used for all types of mapping?
Like Stefano I don't think so.
Agree.
Signature · 3 lines
Regards, Oleksandr Tyshchenko
Quoted 6 lines · Oleksandr
Quoted 5 lines · Jan Beulich
Quoted 4 lines · Oleksandr
- Where that range should be located in guest address space, should that range be the same for all domains (how
GUEST_GNTTAB_BASE(SIZE)for example) or it should be calculated based on actualguest_ram_base(size)for a particular domain?
Quoted 7 lines
Quoted 3 lines
The default size may well be fixed or amount-of-memory-dependent, but I think there will need to be a way to enlarge the region for guests with particular needs.
Well, but why we couldn't just make a large chunk by default which would satisfy all guests, as it was mentioned earlier in this thread "as it doesn't consume resource when not being used"
to avoid an extra configuration option, etc?
Because experience tells me that no static upper limit will ever please everyone.
Jan
On 05.08.21 08:58, Jan Beulich wrote:
Hi Jan.
Quoted 10 lines · Oleksandr
Quoted 9 lines · Jan Beulich
Quoted 8 lines · Oleksandr
Quoted 4 lines
- Where that range should be located in guest address space, should that range be the same for all domains (how
GUEST_GNTTAB_BASE(SIZE)for example) or it should be calculated based on actualguest_ram_base(size)for a particular domain?The default size may well be fixed or amount-of-memory-dependent, but I think there will need to be a way to enlarge the region for guests with particular needs.
Quoted 6 lines
Quoted 4 lines
Well, but why we couldn't just make a large chunk by default which would satisfy all guests, as it was mentioned earlier in this thread "as it doesn't consume resource when not being used"
to avoid an extra configuration option, etc?Because experience tells me that no static upper limit will ever please everyone.
ok, thank you for the explanation.
Quoted 3 lines
Jan
Signature · 3 lines
Regards, Oleksandr Tyshchenko
Quoted 49 lines · Julien Grall
Quoted 48 lines · Andrew Cooper
Quoted 30 lines · Julien Grall
Quoted 29 lines · Andrew Cooper
Quoted 12 lines · Oleksandr Tyshchenko
From: Oleksandr Tyshchenko <[email protected]>
Add
XENMEM_get_unallocated_spacehypercall which purpose is to query hypervisor to find regions of guest physical address space which are unused and can be used to create grant/foreign mappings instead of wasting real pages from the domain memory for establishing these mappings. The problem with the current Linux on Xen on Arm behaviour is if we want to map some guest memory regions in advance or to perform cache mappings in the backend we might run out of memory in the host (see XSA-300). This of course, depends on the both host and guest memory sizes.Quoted 9 lines
The "unallocated space" can't be figured out precisely by the domain on Arm without hypervisor involvement:
- not all device I/O regions are known by the time domain starts creating grant/foreign mappings
- the Dom0 is not aware of memory regions used for the identity mappings needed for the PV drivers to work
In both cases we might end up re-using these regions by
a mistake. So, the hypervisor which maintains the P2M for the domain is in the best position to provide "unallocated space".I'm afraid this does not improve the situation.
If a guest follows the advice from
XENMEM_get_unallocated_space, and subsequently a new IO or identity region appears, everything will explode, because the "safe area" wasn't actually safe.Quoted 5 lines
Quoted 2 lines
The safe range *must* be chosen by the toolstack, because nothing else can do it safely or correctly.
The problem is how do you size it? In particular, a backend may map multiple time the same page (for instance if the page is granted twice).
The number of mapped grants is limited by the size of the maptrack table in Xen, which is a toolstack input to the domaincreate hypercall. Therefore, the amount of space required is known and bounded.
There are a handful of other frames required in the current ABI (shared info, vcpu info, etc).
The areas where things do become fuzzy is things like foreign mappings,
acquire_resource, etc for the control domain, which are effectively unbounded from the domain's point of view.
Quoted 6 lines
Quoted 4 lines
For those, its entirely fine to say "here 128G of safe mapping space" or so. Even the quantity of mapping dom0 can make is limited by the shadow memory pool and the number of pagetables Xen is willing to expend on the second stage translation tables.
FWIW, on Arm, we don't have shadow memory pool.
Where do you take 2nd level page table memory from then?
Quoted 10 lines
Quoted 10 lines
Quoted 9 lines
Quoted 7 lines
Once a safe range (or ranges) has been chosen, any subsequent action which overlaps with the ranges must be rejected, as it will violate the guarantees provided.
Furthermore, the ranges should be made available to the guest via normal memory map means. On x86, this is via the E820 table, and on ARM I presume the DTB. There is no need for a new hypercall.
Device-Tree only works if you have a guest using it. How about ACPI?
Quoted 8 lines
Quoted 4 lines
ACPI inherits E820 from x86 (its a trivial format), and UEFI was also based on it.
But whichever... All firmware interfaces have a memory map.
This will be UEFI memory map. However, I am a bit confused how we can tell the OS the region will be used for grant/foreign mapping. Is it possible to reserved a new type?
As with about any non-abandoned specification it is in principle possible to define/reserve new types. Question how practical it is, i.e. in particular how long it may take to get to the point where we have a firmly reserved type. Short of this I wonder whether you, Andrew, were thinking to re-use an existing type (in which case the question of disambiguation arises).
As a result I wonder whether a "middle" approach wouldn't be better:
Have the range be determined up front (by tool stack or Xen), but communicate it to the guest by PV means (hypercall, shared info, start info, or yet some other table).
Jan
Hi Jan,
Quoted 50 lines · Jan Beulich
Quoted 49 lines · Julien Grall
Quoted 48 lines · Andrew Cooper
Quoted 30 lines · Julien Grall
Quoted 29 lines · Andrew Cooper
Quoted 12 lines · Oleksandr Tyshchenko
From: Oleksandr Tyshchenko <[email protected]>
Add
XENMEM_get_unallocated_spacehypercall which purpose is to query hypervisor to find regions of guest physical address space which are unused and can be used to create grant/foreign mappings instead of wasting real pages from the domain memory for establishing these mappings. The problem with the current Linux on Xen on Arm behaviour is if we want to map some guest memory regions in advance or to perform cache mappings in the backend we might run out of memory in the host (see XSA-300). This of course, depends on the both host and guest memory sizes.Quoted 9 lines
The "unallocated space" can't be figured out precisely by the domain on Arm without hypervisor involvement:
- not all device I/O regions are known by the time domain starts creating grant/foreign mappings
- the Dom0 is not aware of memory regions used for the identity mappings needed for the PV drivers to work
In both cases we might end up re-using these regions by
a mistake. So, the hypervisor which maintains the P2M for the domain is in the best position to provide "unallocated space".I'm afraid this does not improve the situation.
If a guest follows the advice from
XENMEM_get_unallocated_space, and subsequently a new IO or identity region appears, everything will explode, because the "safe area" wasn't actually safe.Quoted 5 lines
Quoted 2 lines
The safe range *must* be chosen by the toolstack, because nothing else can do it safely or correctly.
The problem is how do you size it? In particular, a backend may map multiple time the same page (for instance if the page is granted twice).
The number of mapped grants is limited by the size of the maptrack table in Xen, which is a toolstack input to the domaincreate hypercall. Therefore, the amount of space required is known and bounded.
There are a handful of other frames required in the current ABI (shared info, vcpu info, etc).
The areas where things do become fuzzy is things like foreign mappings,
acquire_resource, etc for the control domain, which are effectively unbounded from the domain's point of view.
Quoted 8 lines
Quoted 6 lines
Quoted 4 lines
For those, its entirely fine to say "here 128G of safe mapping space" or so. Even the quantity of mapping dom0 can make is limited by the shadow memory pool and the number of pagetables Xen is willing to expend on the second stage translation tables.
FWIW, on Arm, we don't have shadow memory pool.
Where do you take 2nd level page table memory from then?
From xenheap directly. And yes, I know this may lead to other issues. But that's a known missing features that so far no-one tackled.
Cheers,
Signature · 1 line
Julien Grall
Hi Jan,
Quoted 19 lines · Jan Beulich
Quoted 10 lines
Quoted 10 lines
Quoted 9 lines
Quoted 7 lines
Once a safe range (or ranges) has been chosen, any subsequent action which overlaps with the ranges must be rejected, as it will violate the guarantees provided.
Furthermore, the ranges should be made available to the guest via normal memory map means. On x86, this is via the E820 table, and on ARM I presume the DTB. There is no need for a new hypercall.
Device-Tree only works if you have a guest using it. How about ACPI?
Quoted 8 lines
Quoted 4 lines
ACPI inherits E820 from x86 (its a trivial format), and UEFI was also based on it.
But whichever... All firmware interfaces have a memory map.
This will be UEFI memory map. However, I am a bit confused how we can tell the OS the region will be used for grant/foreign mapping. Is it possible to reserved a new type?
Quoted 6 lines
As with about any non-abandoned specification it is in principle possible to define/reserve new types. Question how practical it is, i.e. in particular how long it may take to get to the point where we have a firmly reserved type. Short of this I wonder whether you, Andrew, were thinking to re-use an existing type (in which case the question of disambiguation arises).
Copying/pasting the IRC discussion related to this:
[11:32:19] <Diziet> julieng: I have skimread the thread "[RFC PATCH] xen/memory: Introduce a hypercall to provide unallocated space"
[11:32:56] <Diziet> My impression is that it is converging on a workable solution but I am not sure. Does it need any help ?
[12:20:32] <julieng> Diziet: I think we have a solution for Arm and DT. We are waiting on andyhhp for the ACPI part. He suggested to use the memory map but it is unclear how we could describe the safe region. [13:01:49] <andyhhp> that's easy, seeing as we already have hypercall to convay that information, but feel free to skip the x86 side for v1 if that helps
[13:02:15] <andyhhp> it wants an extention to the PVH spec to define a new memory type
[13:04:09] <julieng> andyhhp: This doesn't really address the question to how we can define the memory type because this is not a spec we own. See [email protected] for more details.. [13:04:27] <andyhhp> after which it wants to appear in XENMEM_memory_map
[13:04:32] <andyhhp> this is a spec we own
[13:04:43] <julieng> We don't own the E820.
[13:05:19] <julieng> Nor the UEFI memory map.
[13:05:24] <andyhhp> no, we don't, but that's not the spec in question
[13:06:06] <andyhhp> the spec in question is the PVH start info and/or XENMEM_memory_map, both of which are "in the format of the E820" table, not "is an E820 table"
[13:06:27] <andyhhp> with almost 4 billion type identifiers avaialble
[13:07:03] <julieng> So what you are saying is let's pick up a random number and hope no-one will use it?
[13:07:34] <julieng> Because neither XENMEM_memory_map nor PVH start info exist for ACPI on Arm.
[13:08:17] <andyhhp> we (xen) are the source of this information, via a Xen specified API/ABI
[13:08:41] <andyhhp> we are entirely within our rights to document an extention, which has defined meaning under Xen
[13:09:03] <andyhhp> and yeah - you choose some Xen specific constant to go in the high bits of the type id or something
[13:09:04] <julieng> I agree for a domU. But for dom0, the memory map is the same as the host. So we have to make sure the number doesn't clash.
[13:09:49] <andyhhp> the chance of actually having a collision is 0, because in 30 years or so, about 10 types have been declared and handed out
[13:10:15] <andyhhp> and if a collision does occur, we add a second hypercall saying "this is a raw non-xenified E820 table, and here is the xenified one"
Quoted 8 lines
As a result I wonder whether a "middle" approach wouldn't be better:
Have the range be determined up front (by tool stack or Xen), but communicate it to the guest by PV means (hypercall, shared info, start info, or yet some other table).Jan
Signature · 1 line
Julien Grall
Quoted 1 line · Oleksandr Tyshchenko
From: Oleksandr Tyshchenko <[email protected]>
...
Quoted 15 lines
xen/common/memory.c +7
-0@@ -1811,6 +1811,62 @@ long do_memory_op(unsigned long cmd, XEN_GUEST_HANDLE_PARAM(void) arg) start_extent);break ; +case XENMEM_get_unallocated_space: + { +struct xen_get_unallocated_space xgus; +struct xen_unallocated_region *regions; + +if (unlikely (start_extent) ) +return -EINVAL ;
xen/common/memory.c long do_memory_op(unsigned long cmd, XEN_GUEST_HANDLE_PARAM(void) arg) · line 1830
Quoted 10 lines
xen/common/memory.c +10
-0+ +if (copy_from_guest (&xgus, arg,1 ) || + !guest_handle_okay (xgus.buffer, xgus.nr_regions) ) +return -EFAULT ; + + d =rcu_lock_domain_by_any_id (xgus.domid); +if ( d ==NULL ) +return -ESRCH ; + + rc =xsm_get_unallocated_space (XSM_HOOK , d);
Not sure if you are aware but XSM_HOOK is a no-op check, meaning that you are allowing any domain to do this operation on any other domain. In most cases there is an XSM check at the beginning of the hypercall processing to do an initial clamp down but I am pretty sure there is no prior XSM check on this path. Based on my understanding of how this is intended, which may be incorrect, but I think you would actually want XSM_TARGET.
Quoted 42 lines
xen/common/memory.c +39
-0+if ( rc ) + { +rcu_unlock_domain (d); +return rc; + } + +if ( !xgus.nr_regions || xgus.nr_regions >XEN_MAX_UNALLOCATED_REGIONS ) + { +rcu_unlock_domain (d); +return -EINVAL ; + } + + regions = xzalloc_array(struct xen_unallocated_region, xgus.nr_regions); +if ( !regions ) + { +rcu_unlock_domain (d); +return -ENOMEM ; + } + + rc =arch_get_unallocated_space (d, regions, &xgus.nr_regions); +if ( rc ) + goto unallocated_out; + +if (__copy_to_guest (xgus.buffer, regions, xgus.nr_regions) ) + { + rc = -EFAULT ; + goto unallocated_out; + } + +if (__copy_to_guest (arg, &xgus,1 ) ) + rc = -EFAULT ; + +unallocated_out: +rcu_unlock_domain (d); +xfree (regions); + +break ; + } +default : rc =arch_memory_op (cmd, arg);break ;
...
xen/include/xsm/dummy.h static XSM_INLINE int xsm_domain_resource_map(XSM_DEFAULT_ARG struct domain *d) · line 778
Quoted 12 lines
xen/include/xsm/dummy.h +4
-0@@ -772,3 +772,9 @@ static XSM_INLINE int xsm_domain_resource_map(XSM_DEFAULT_ARG struct domain *d)XSM_ASSERT_ACTION (XSM_DM_PRIV );return xsm_default_action (action, current->domain, d); } + +static XSM_INLINE int xsm_get_unallocated_space(XSM_DEFAULT_ARG struct domain *d) +{ +XSM_ASSERT_ACTION (XSM_HOOK );
For completeness, if you switch to XSM_TARGET at the call site, you will want to change it here as well.
xen/include/xsm/dummy.h static XSM_INLINE int xsm_domain_resource_map(XSM_DEFAULT_ARG struct domain *d) · line 780
Quoted 2 lines
xen/include/xsm/dummy.h +2
-0+return xsm_default_action (action, current->domain, d); +}
V/r,
Daniel P. Smith
On 05.08.21 18:03, Daniel P. Smith wrote:
Hi Daniel.
Quoted 28 lines · Oleksandr Tyshchenko
Quoted 1 line
From: Oleksandr Tyshchenko <[email protected]>
...
Quoted 24 lines
xen/common/memory.c +16
-0@@ -1811,6 +1811,62 @@ long do_memory_op(unsigned long cmd, XEN_GUEST_HANDLE_PARAM(void) arg) start_extent);break ; +case XENMEM_get_unallocated_space: + { +struct xen_get_unallocated_space xgus; +struct xen_unallocated_region *regions; + +if (unlikely (start_extent) ) +return -EINVAL ; + +if (copy_from_guest (&xgus, arg,1 ) || + !guest_handle_okay (xgus.buffer, xgus.nr_regions) ) +return -EFAULT ; + + d =rcu_lock_domain_by_any_id (xgus.domid); +if ( d ==NULL ) +return -ESRCH ; +
Quoted 8 lines
xen/common/memory.c long do_memory_op(unsigned long cmd, XEN_GUEST_HANDLE_PARAM(void) arg) · line 1830
Quoted 1 line
xen/common/memory.c +1
-0+ rc =xsm_get_unallocated_space (XSM_HOOK , d);Not sure if you are aware but
XSM_HOOKis a no-op check, meaning that you are allowing any domain to do this operation on any other domain. In most cases there is an XSM check at the beginning of the hypercall processing to do an initial clamp down but I am pretty sure there is no prior XSM check on this path. Based on my understanding of how this is intended, which may be incorrect, but I think you would actually want XSM_TARGET.the
Thank you for pointing this out.
I am aware what the XSM_HOOK is, but I was thinking what the default action would be better suited for that hypercall, and failed to think of a better alternative. I was going to choose XSM_TARGET, but the description "/* Can perform on self or your target domain */" confused me a bit, as there was no target domain involved as I thought, XSM_PRIV sounded too strictly to me, etc. So, I decided to leave a "hook" for the RFC version. But, now I see that XSM_TARGET might be indeed better choice across all possible variants.
Quoted 45 lines
Quoted 43 lines
xen/common/memory.c +39
-0+if ( rc ) + { +rcu_unlock_domain (d); +return rc; + } + +if ( !xgus.nr_regions || xgus.nr_regions >XEN_MAX_UNALLOCATED_REGIONS ) + { +rcu_unlock_domain (d); +return -EINVAL ; + } + + regions = xzalloc_array(struct xen_unallocated_region, xgus.nr_regions); +if ( !regions ) + { +rcu_unlock_domain (d); +return -ENOMEM ; + } + + rc =arch_get_unallocated_space (d, regions, &xgus.nr_regions); +if ( rc ) + goto unallocated_out; + +if (__copy_to_guest (xgus.buffer, regions, xgus.nr_regions) ) + { + rc = -EFAULT ; + goto unallocated_out; + } + +if (__copy_to_guest (arg, &xgus,1 ) ) + rc = -EFAULT ; + +unallocated_out: +rcu_unlock_domain (d); +xfree (regions); + +break ; + } +default : rc =arch_memory_op (cmd, arg);break ;...
Quoted 14 lines
xen/include/xsm/dummy.h static XSM_INLINE int xsm_domain_resource_map(XSM_DEFAULT_ARG struct domain *d) · line 778
Quoted 12 lines
xen/include/xsm/dummy.h +4
-0@@ -772,3 +772,9 @@ static XSM_INLINE int xsm_domain_resource_map(XSM_DEFAULT_ARG struct domain *d)XSM_ASSERT_ACTION (XSM_DM_PRIV );return xsm_default_action (action, current->domain, d); } + +static XSM_INLINE int xsm_get_unallocated_space(XSM_DEFAULT_ARG struct domain *d) +{ +XSM_ASSERT_ACTION (XSM_HOOK );For completeness, if you switch to
XSM_TARGETat the call site, you will want to change it here as well.
Yes.
Quoted 5 lines
xen/include/xsm/dummy.h static XSM_INLINE int xsm_domain_resource_map(XSM_DEFAULT_ARG struct domain *d) · line 780
Quoted 3 lines
xen/include/xsm/dummy.h +2
-0+return xsm_default_action (action, current->domain, d); +}V/r,
Daniel P. Smith
Signature · 3 lines
Regards, Oleksandr Tyshchenko
Hey Oleksandr!
Quoted 41 lines · Oleksandr
On 05.08.21 18:03, Daniel P. Smith wrote:
Hi Daniel.
Quoted 28 lines · Oleksandr Tyshchenko
Quoted 1 line
From: Oleksandr Tyshchenko <[email protected]>
...
Quoted 24 lines
xen/common/memory.c +0
-0@@ -1811,6 +1811,62 @@ long do_memory_op(unsigned long cmd, XEN_GUEST_HANDLE_PARAM(void) arg) start_extent); break;+ case
XENMEM_get_unallocated_space:
+ {
+ structxen_get_unallocated_spacexgus;
+ structxen_unallocated_region*regions;xen/common/memory.c +12
-0+ + if ( unlikely(start_extent) ) + return -EINVAL; + + if ( copy_from_guest(&xgus, arg, 1) || + !guest_handle_okay(xgus.buffer, xgus.nr_regions) ) + return -EFAULT; + + d = rcu_lock_domain_by_any_id(xgus.domid); + if ( d == NULL ) + return -ESRCH; +Quoted 8 lines
Quoted 1 line
+ rc =
xsm_get_unallocated_space(XSM_HOOK, d);Not sure if you are aware but
XSM_HOOKis a no-op check, meaning that you are allowing any domain to do this operation on any other domain. In most cases there is an XSM check at the beginning of the hypercall processing to do an initial clamp down but I am pretty sure there is no prior XSM check on this path. Based on my understanding of how this is intended, which may be incorrect, but I think you would actually want XSM_TARGET.the
Quoted 10 lines
Thank you for pointing this out.
I am aware what theXSM_HOOKis, but I was thinking what the default action would be better suited for that hypercall, and failed to think of a better alternative.
I was going to chooseXSM_TARGET, but the description "/* Can perform on self or your target domain */" confused me a bit, as there was no target domain involved as I thought,XSM_PRIV
sounded too strictly to me, etc. So, I decided to leave a "hook" for the RFC version. But, now I see thatXSM_TARGETmight be indeed better choice across all possible variants.
If you unravel the craftiness that is xsm_default_action, there is actually a bit of hierarchy there. If you set the default_action to XSM_TARGET, it will first check if calling domain(src) is the target, then falls into the XSM_DM_PRIV check which is if src->target == target, and then finally checks if is_control_domain(src). That will constrict the operation so that a domain can call it on itself, a device model domain (stubdom) can call it on the domain it is backing, and the control domain can make the call. I am not a 100% sure on this but I do not believe a hardware domain would be able to make the call with it set to XSM_TARGET and not employing Flask. Hope this helps.
Quoted 1 line · Daniel P. Smith
Hey Oleksandr!
Hi Daniel.
Quoted 52 lines · Oleksandr
Quoted 40 lines · Daniel P. Smith
Hi Daniel.
Quoted 28 lines · Oleksandr Tyshchenko
Quoted 1 line
From: Oleksandr Tyshchenko <[email protected]>
...
Quoted 24 lines
xen/common/memory.c +15
-0@@ -1811,6 +1811,62 @@ long do_memory_op(unsigned long cmd, XEN_GUEST_HANDLE_PARAM(void) arg) start_extent); break; + case XENMEM_get_unallocated_space: + { + struct xen_get_unallocated_space xgus; + struct xen_unallocated_region *regions; + + if ( unlikely(start_extent) ) + return -EINVAL; + + if ( copy_from_guest(&xgus, arg, 1) || + !guest_handle_okay(xgus.buffer, xgus.nr_regions) ) + return -EFAULT; + + d = rcu_lock_domain_by_any_id(xgus.domid); + if ( d == NULL ) + return -ESRCH; +Quoted 8 lines
xen/common/memory.c long do_memory_op(unsigned long cmd, XEN_GUEST_HANDLE_PARAM(void) arg) · line 1830
Quoted 1 line
xen/common/memory.c +1
-0+ rc = xsm_get_unallocated_space(XSM_HOOK, d);Not sure if you are aware but
XSM_HOOKis a no-op check, meaning that you are allowing any domain to do this operation on any other domain. In most cases there is an XSM check at the beginning of the hypercall processing to do an initial clamp down but I am pretty sure there is no prior XSM check on this path. Based on my understanding of how this is intended, which may be incorrect, but I think you would actually want XSM_TARGET.theQuoted 10 lines
Thank you for pointing this out.
I am aware what theXSM_HOOKis, but I was thinking what the default action would be better suited for that hypercall, and failed to think of a better alternative.
I was going to chooseXSM_TARGET, but the description "/* Can perform on self or your target domain */" confused me a bit, as there was no target domain involved as I thought,XSM_PRIV
sounded too strictly to me, etc. So, I decided to leave a "hook" for the RFC version. But, now I see thatXSM_TARGETmight be indeed better choice across all possible variants.
Quoted 10 lines
If you unravel the craftiness that is
xsm_default_action, there is actually a bit of hierarchy there. If you set thedefault_actiontoXSM_TARGET, it will first check if calling domain(src) is the target, then falls into theXSM_DM_PRIVcheck which is ifsrc->target== target, and then finally checks ifis_control_domain(src). That will constrict the operation so that a domain can call it on itself, a device model domain (stubdom) can call it on the domain it is backing, and the control domain can make the call. I am not a 100% sure on this but I do not believe a hardware domain would be able to make the call with it set toXSM_TARGETand not employing Flask. Hope this helps.
This definitely helps. Thank you for the explanation.
Quoted 3 lines
Signature · 3 lines
Regards, Oleksandr Tyshchenko
Quoted 44 lines · Daniel P. Smith
Quoted 32 lines · Oleksandr
Quoted 23 lines · Daniel P. Smith
Quoted 22 lines · Oleksandr Tyshchenko
xen/common/memory.c +0
-0@@ -1811,6 +1811,62 @@ long do_memory_op(unsigned long cmd, XEN_GUEST_HANDLE_PARAM(void) arg) start_extent); break;+ case
XENMEM_get_unallocated_space:
+ {
+ structxen_get_unallocated_spacexgus;
+ structxen_unallocated_region*regions;xen/common/memory.c +12
-0+ + if ( unlikely(start_extent) ) + return -EINVAL; + + if ( copy_from_guest(&xgus, arg, 1) || + !guest_handle_okay(xgus.buffer, xgus.nr_regions) ) + return -EFAULT; + + d = rcu_lock_domain_by_any_id(xgus.domid); + if ( d == NULL ) + return -ESRCH; +Quoted 8 lines
Quoted 1 line
+ rc =
xsm_get_unallocated_space(XSM_HOOK, d);Not sure if you are aware but
XSM_HOOKis a no-op check, meaning that you are allowing any domain to do this operation on any other domain. In most cases there is an XSM check at the beginning of the hypercall processing to do an initial clamp down but I am pretty sure there is no prior XSM check on this path. Based on my understanding of how this is intended, which may be incorrect, but I think you would actually want XSM_TARGET.theQuoted 10 lines
Thank you for pointing this out.
I am aware what theXSM_HOOKis, but I was thinking what the default action would be better suited for that hypercall, and failed to think of a better alternative.
I was going to chooseXSM_TARGET, but the description "/* Can perform on self or your target domain */" confused me a bit, as there was no target domain involved as I thought,XSM_PRIV
sounded too strictly to me, etc. So, I decided to leave a "hook" for the RFC version. But, now I see thatXSM_TARGETmight be indeed better choice across all possible variants.
Quoted 10 lines
If you unravel the craftiness that is
xsm_default_action, there is actually a bit of hierarchy there. If you set thedefault_actiontoXSM_TARGET, it will first check if calling domain(src) is the target, then falls into theXSM_DM_PRIVcheck which is ifsrc->target== target, and then finally checks ifis_control_domain(src). That will constrict the operation so that a domain can call it on itself, a device model domain (stubdom) can call it on the domain it is backing, and the control domain can make the call. I am not a 100% sure on this but I do not believe a hardware domain would be able to make the call with it set toXSM_TARGETand not employing Flask.
Afaict (perhaps leaving aside late-hwdom, which I have little knowledge of) right now we have is_control_domain(d) == is_hardware_domain(d).
Jan
Quoted 56 lines · Jan Beulich
Quoted 44 lines · Daniel P. Smith
Quoted 32 lines · Oleksandr
Quoted 23 lines · Daniel P. Smith
Quoted 22 lines · Oleksandr Tyshchenko
xen/common/memory.c +0
-0@@ -1811,6 +1811,62 @@ long do_memory_op(unsigned long cmd, XEN_GUEST_HANDLE_PARAM(void) arg) start_extent); break;+ case
XENMEM_get_unallocated_space:
+ {
+ structxen_get_unallocated_spacexgus;
+ structxen_unallocated_region*regions;xen/common/memory.c +12
-0+ + if ( unlikely(start_extent) ) + return -EINVAL; + + if ( copy_from_guest(&xgus, arg, 1) || + !guest_handle_okay(xgus.buffer, xgus.nr_regions) ) + return -EFAULT; + + d = rcu_lock_domain_by_any_id(xgus.domid); + if ( d == NULL ) + return -ESRCH; +Quoted 8 lines
Quoted 1 line
+ rc =
xsm_get_unallocated_space(XSM_HOOK, d);Not sure if you are aware but
XSM_HOOKis a no-op check, meaning that you are allowing any domain to do this operation on any other domain. In most cases there is an XSM check at the beginning of the hypercall processing to do an initial clamp down but I am pretty sure there is no prior XSM check on this path. Based on my understanding of how this is intended, which may be incorrect, but I think you would actually want XSM_TARGET.theQuoted 10 lines
Thank you for pointing this out.
I am aware what theXSM_HOOKis, but I was thinking what the default action would be better suited for that hypercall, and failed to think of a better alternative.
I was going to chooseXSM_TARGET, but the description "/* Can perform on self or your target domain */" confused me a bit, as there was no target domain involved as I thought,XSM_PRIV
sounded too strictly to me, etc. So, I decided to leave a "hook" for the RFC version. But, now I see thatXSM_TARGETmight be indeed better choice across all possible variants.Quoted 10 lines
If you unravel the craftiness that is
xsm_default_action, there is actually a bit of hierarchy there. If you set thedefault_actiontoXSM_TARGET, it will first check if calling domain(src) is the target, then falls into theXSM_DM_PRIVcheck which is ifsrc->target== target, and then finally checks ifis_control_domain(src). That will constrict the operation so that a domain can call it on itself, a device model domain (stubdom) can call it on the domain it is backing, and the control domain can make the call. I am not a 100% sure on this but I do not believe a hardware domain would be able to make the call with it set toXSM_TARGETand not employing Flask.
Quoted 2 lines
Afaict (perhaps leaving aside late-hwdom, which I have little knowledge of) right now we have
is_control_domain(d)==is_hardware_domain(d).
That is my fault for not being more explicit. When I refer to a "hardware domain" I am referring to what you call "late-hwdom". When a hardware domain that is not dom0 is constructed, it does not get the is_privileged flag set to true which will result in is_control_domain(d) to return false. Additionally there is currently no enum xsm_default for hardware domain access/privilege and thus there is no rule/access check defined in default_action() that allows any of the XSM hooks to be restricted to the hardware domain. Which is what I was referring to regarding the use of the hardware domain, aka late-hwdom, without Flask. With Flask it becomes possible for the hardware domain to be granted access to calls that are reserved to the control domain under the dummy/default access policy,thus allowing it to function fully.
dps
Hi Oleksandr,
Quoted 39 lines
-----Original Message-----
From: Xen-devel <[email protected]> On Behalf Of Oleksandr Tyshchenko
Sent: Thursday, July 29, 2021 12:19 AM
To: [email protected]
Cc: Oleksandr Tyshchenko <[email protected]>; Daniel De Graaf <[email protected]>; Daniel P. Smith <[email protected]>; Ian Jackson <[email protected]>; Wei Liu <[email protected]>; Andrew Cooper <[email protected]>; George Dunlap <[email protected]>; Jan Beulich <[email protected]>; Julien Grall <[email protected]>; Stefano Stabellini <[email protected]>; Volodymyr Babchuk <[email protected]>; Roger Pau Monné <[email protected]>; Bertrand Marquis <[email protected]>; Wei Chen <[email protected]>
Subject: [RFC PATCH] xen/memory: Introduce a hypercall to provide unallocated spaceFrom: Oleksandr Tyshchenko <[email protected]>
Add
XENMEM_get_unallocated_spacehypercall which purpose is to query hypervisor to find regions of guest physical address space which are unused and can be used to create grant/foreign mappings instead of wasting real pages from the domain memory for establishing these mappings. The problem with the current Linux on Xen on Arm behaviour is if we want to map some guest memory regions in advance or to perform cache mappings in the backend we might run out of memory in the host (see XSA-300). This of course, depends on the both host and guest memory sizes.The "unallocated space" can't be figured out precisely by the domain on Arm without hypervisor involvement:
- not all device I/O regions are known by the time domain starts creating grant/foreign mappings
- the Dom0 is not aware of memory regions used for the identity mappings needed for the PV drivers to work
In both cases we might end up re-using these regions by
a mistake. So, the hypervisor which maintains the P2M for the domain is in the best position to provide "unallocated space".
Quoted 7 lines
The arch code is in charge of finding these regions and filling in corresponding array in new helper
arch_get_unallocated_space().This patch implements common and Arm parts, the x86 specific bits are left uniplemented for now.
Signed-off-by Oleksandr Tyshchenko <[email protected]>
Kind regards,
Henry
Quoted 322 lines
---
Current patch is based on the latest staging branch:
73c932dtools/libxc: useuint32_tfor pirq inxc_domain_irq_permissionand also available at:
https://github.com/otyshchenko1/xen/commits/map_opt_ml1The corresponding Linux changes you can find at:
https://github.com/otyshchenko1/linux/commits/map_opt_ml1
I am going to publish them soon.--- tools/flask/policy/modules/dom0.te | 2 +- xen/arch/arm/mm.c | 18 ++++++++++++ xen/common/memory.c | 56 +++++++++++++++++++++++++++++++++++++ xen/include/asm-arm/mm.h | 4 +++ xen/include/asm-x86/mm.h | 8 ++++++ xen/include/public/memory.h | 37 +++++++++++++++++++++++- xen/include/xsm/dummy.h | 6 ++++ xen/include/xsm/xsm.h | 6 ++++ xen/xsm/dummy.c | 1 + xen/xsm/flask/hooks.c | 6 ++++ xen/xsm/flask/policy/access_vectors | 2 ++ 11 files changed, 144 insertions(+), 2 deletions(-)tools/flask/policy/modules/dom0.te +1
-1@@ -39,7 +39,7 @@ allow dom0_t dom0_t:domain { }; allow dom0_t dom0_t:domain2 { set_cpu_policy gettsc settsc setscheduler set_vnumainfo - get_vnumainfo psr_cmt_op psr_alloc get_cpu_policy + get_vnumainfo psr_cmt_op psr_alloc get_cpu_policy
get_unallocated_space
};
allowdom0_tdom0_t:resource { add remove };xen/arch/arm/mm.c +8
-0@@ -1635,6 +1635,24 @@ unsigned long get_upper_mfn_bound(void)return max_page -1 ; } +#define GPFN_ALIGN_TO_GB (x) (((x) + (1UL << 18) - 1) & (~((1UL << 18) - 1))) + +int arch_get_unallocated_space (struct domain *d, +struct xen_unallocated_region *regions, +unsigned int *nr_regions) +{ +/* +* Everything not mapped into P2M could be treated as unused. Taking into
+ * the account that we have a lot of unallocated space abovemax_mapped_gfnxen/arch/arm/mm.c +4
-0+* and in order to simplify things, just provide a single 8GB memory +* region aligned to 1GB boundary for now. +*/ + regions->start_gpfn =
GPFN_ALIGN_TO_GB(domain_get_maximum_gpfn(d) + 1);xen/arch/arm/mm.c +5
-0+ regions->nr_gpfns = (1UL <<18 ) *8 ; + *nr_regions =1 ; + +return 0 ; +}/* * Local variables: * mode: C xen/common/memory.c +56
-0@@ -1811,6 +1811,62 @@ long do_memory_op(unsigned long cmd, XEN_GUEST_HANDLE_PARAM(void) arg) start_extent);break ; +case XENMEM_get_unallocated_space: + { +struct xen_get_unallocated_space xgus; +struct xen_unallocated_region *regions; + +if (unlikely (start_extent) ) +return -EINVAL ; + +if (copy_from_guest (&xgus, arg,1 ) || + !guest_handle_okay (xgus.buffer, xgus.nr_regions) ) +return -EFAULT ; + + d =rcu_lock_domain_by_any_id (xgus.domid); +if ( d ==NULL ) +return -ESRCH ; + + rc =xsm_get_unallocated_space (XSM_HOOK , d); +if ( rc ) + { +rcu_unlock_domain (d); +return rc; + } + +if ( !xgus.nr_regions || xgus.nr_regions >XEN_MAX_UNALLOCATED_REGIONS ) + { +rcu_unlock_domain (d); +return -EINVAL ; + } + + regions = xzalloc_array(struct xen_unallocated_region, xgus.nr_regions); +if ( !regions ) + { +rcu_unlock_domain (d); +return -ENOMEM ; + } + + rc =arch_get_unallocated_space (d, regions, &xgus.nr_regions); +if ( rc ) + goto unallocated_out; + +if (__copy_to_guest (xgus.buffer, regions, xgus.nr_regions) ) + { + rc = -EFAULT ; + goto unallocated_out; + } + +if (__copy_to_guest (arg, &xgus,1 ) ) + rc = -EFAULT ; + +unallocated_out: +rcu_unlock_domain (d); +xfree (regions); + +break ; + } +default: rc = arch_memory_op(cmd, arg); break;xen/include/asm-arm/mm.h +4
-0@@ -359,6 +359,10 @@ void clear_and_clean_page(struct page_info *page);unsigned int arch_get_dma_bitsize (void ); +int arch_get_unallocated_space (struct domain *d, +struct xen_unallocated_region *regions, +unsigned int *nr_regions); +#endif /* __ARCH_ARM_MM__ */ /* * Local variables: xen/include/asm-x86/mm.h +8
-0@@ -652,4 +652,12 @@ static inline bool arch_mfn_in_directmap(unsigned long mfn)return mfn <= (virt_to_mfn (eva -1 ) +1 ); } +static inline +int arch_get_unallocated_space (struct domain *d, +struct xen_unallocated_region *regions, +unsigned int *nr_regions) +{ +return -EOPNOTSUPP ; +} +#endif /*
__ASM_X86_MM_H__*/xen/include/public/memory.h +36
-1@@ -739,7 +739,42 @@ struct xen_vnuma_topology_info {typedef struct xen_vnuma_topology_info xen_vnuma_topology_info_t ;DEFINE_XEN_GUEST_HANDLE (xen_vnuma_topology_info_t); -/* Next available subop number is 29 */ +/* +* Get the unallocated space (regions of guest physical address space which +* are unused) and can be used to create grant/foreign mappings. +*/ +#define XENMEM_get_unallocated_space 29 +struct xen_unallocated_region { +xen_pfn_t start_gpfn; +xen_ulong_t nr_gpfns; +}; +typedef struct xen_unallocated_region xen_unallocated_region_t ; +DEFINE_XEN_GUEST_HANDLE (xen_unallocated_region_t); + +#define XEN_MAX_UNALLOCATED_REGIONS 32 + +struct xen_get_unallocated_space { +/* IN - Which domain to provide unallocated space for */ +domid_t domid; + +/* +* IN/OUT - As an IN parameter number of memory regions which +* can be written to the buffer (maximum size of the array) +* As OUT parameter number of memory regions which +* have been written to the buffer +*/ +unsigned int nr_regions; + +/* +* OUT - An array of memory regions, the regions must be placed in +* ascending order, there must be no overlap between them. +*/ +XEN_GUEST_HANDLE (xen_unallocated_region_t) buffer; +}; +typedef struct xen_get_unallocated_space xen_get_unallocated_space_t ; +DEFINE_XEN_GUEST_HANDLE (xen_get_unallocated_space_t); + +/* Next available subop number is 30 */ #endif /* __XEN_PUBLIC_MEMORY_H__ */ xen/include/xsm/dummy.h +2
-0@@ -772,3 +772,9 @@ static XSM_INLINE int xsm_domain_resource_map(XSM_DEFAULT_ARG struct domain *d)XSM_ASSERT_ACTION (XSM_DM_PRIV);return xsm_default_action (action, current->domain, d); } + +static XSM_INLINE int xsm_get_unallocated_space(XSM_DEFAULT_ARGstruct domain *d)
xen/include/xsm/dummy.h +4
-0+{ +XSM_ASSERT_ACTION (XSM_HOOK ); +return xsm_default_action (action, current->domain, d); +}xen/include/xsm/xsm.h +6
-0@@ -180,6 +180,7 @@ struct xsm_operations {int (*dm_op) (struct domain *d);int (*xen_version) (uint32_t cmd);int (*domain_resource_map) (struct domain *d); +int (*get_unallocated_space) (struct domain *d);#ifdef CONFIG_ARGO int (*argo_enable) (const struct domain *d); int (*argo_register_single_source) (const struct domain *d,@@ -701,6 +702,11 @@ static inline int xsm_domain_resource_map(xsm_default_t def, struct domain *d) return xsm_ops->domain_resource_map(d); } +static inline int xsm_get_unallocated_space (xsm_default_t def,struct domain *d) +{ +return xsm_ops->get_unallocated_space (d); +} +#ifdef CONFIG_ARGO static inline int
xsm_argo_enable(const struct domain *d){xen/xsm/dummy.c +1
-0@@ -151,6 +151,7 @@ void __init xsm_fixup_ops (struct xsm_operations *ops)set_to_dummy_if_null (ops, dm_op);set_to_dummy_if_null (ops, xen_version);set_to_dummy_if_null (ops, domain_resource_map); +set_to_dummy_if_null (ops, get_unallocated_space);#ifdef CONFIG_ARGO set_to_dummy_if_null (ops, argo_enable);set_to_dummy_if_null(ops, argo_register_single_source);xen/xsm/flask/hooks.c +5
-0@@ -1715,6 +1715,11 @@ static int flask_domain_resource_map(struct domain *d)return current_has_perm (d,SECCLASS_DOMAIN2 ,DOMAIN2__RESOURCE_MAP ); } +static int flask_get_unallocated_space (struct domain *d) +{ +return current_has_perm (d,SECCLASS_DOMAIN2 ,DOMAIN2__GET_UNALLOCATED_SPACE ); +} +#ifdef
CONFIG_ARGO
static intflask_argo_enable(const struct domain *d){xen/xsm/flask/hooks.c +1
-0@@ -1875,6 +1880,7 @@ static struct xsm_operations flask_ops = { .dm_op = flask_dm_op, .xen_version = flask_xen_version, .domain_resource_map = flask_domain_resource_map, + .get_unallocated_space = flask_get_unallocated_space, #ifdef CONFIG_ARGO .argo_enable = flask_argo_enable, .argo_register_single_source = flask_argo_register_single_source,xen/xsm/flask/policy/access_vectors +2
-0@@ -245,6 +245,8 @@ class domain2 resource_map # XEN_DOMCTL_get_cpu_policy get_cpu_policy +# XENMEM_get_unallocated_space + get_unallocated_space } # Similar to class domain, but primarily contains domctls related to HVMdomains
--
2.7.4
Quoted 1 line · Henry Wang
Hi Oleksandr,
Hi Henry
Quoted 40 lines
Quoted 40 lines
-----Original Message-----
From: Xen-devel <[email protected]> On Behalf Of Oleksandr Tyshchenko
Sent: Thursday, July 29, 2021 12:19 AM
To: [email protected]
Cc: Oleksandr Tyshchenko <[email protected]>; Daniel De Graaf <[email protected]>; Daniel P. Smith <[email protected]>; Ian Jackson <[email protected]>; Wei Liu <[email protected]>; Andrew Cooper <[email protected]>; George Dunlap <[email protected]>; Jan Beulich <[email protected]>; Julien Grall <[email protected]>; Stefano Stabellini <[email protected]>; Volodymyr Babchuk <[email protected]>; Roger Pau Monné <[email protected]>; Bertrand Marquis <[email protected]>; Wei Chen <[email protected]>
Subject: [RFC PATCH] xen/memory: Introduce a hypercall to provide unallocated spaceFrom: Oleksandr Tyshchenko <[email protected]>
Add
XENMEM_get_unallocated_spacehypercall which purpose is to query hypervisor to find regions of guest physical address space which are unused and can be used to create grant/foreign mappings instead of wasting real pages from the domain memory for establishing these mappings. The problem with the current Linux on Xen on Arm behaviour is if we want to map some guest memory regions in advance or to perform cache mappings in the backend we might run out of memory in the host (see XSA-300). This of course, depends on the both host and guest memory sizes.The "unallocated space" can't be figured out precisely by the domain on Arm without hypervisor involvement:
- not all device I/O regions are known by the time domain starts creating grant/foreign mappings
- the Dom0 is not aware of memory regions used for the identity mappings needed for the PV drivers to work
In both cases we might end up re-using these regions by
a mistake. So, the hypervisor which maintains the P2M for the domain is in the best position to provide "unallocated space".
Quoted 8 lines
Quoted 7 lines
The arch code is in charge of finding these regions and filling in corresponding array in new helper
arch_get_unallocated_space().This patch implements common and Arm parts, the x86 specific bits are left uniplemented for now.
Signed-off-by Oleksandr Tyshchenko <[email protected]>Tested-by Henry Wang <[email protected]>
Thank you!
Please note, I have pushed new version (which is notably different) based on the discussion in current thread:
https://lore.kernel.org/xen-devel/[email protected]/
Quoted 5 lines
Kind regards,
Henry
Signature · 3 lines
Regards, Oleksandr Tyshchenko