Skip to contentSkip to conversation

Static heap follow-up doc changes

PATCH In review Henry Wang 4 participants 13 messages Oct 13, 2022 – Oct 17, 2022

lore
Share & tools
lore

Henry Wang

author
#0 lore raw

cover Static heap follow-up doc changes

The static heap feature requires user to know the minimal size of heap to make sure the system can work. Since the heap controlled by Xen is intended to provide memory for the whole system, not only the boot time memory allocation should be covered by the static heap region, but also the runtime allocation should be covered.

The main source of runtime allocation is the memory for the P2M. Currently, from XSA-409, the P2M memory is bounded by the P2M pool. So make this part as the minimal requirement of static heap. The amount of memory allocated after all the guests have been created should be quite limited and mostly predictable.

Also, with the P2M pages pool bounding the domain memory runtime allocation and the documented minimal size requirement of the static heap, it is safe to mark static heap feature as supported.

This series handles the above-mentioned doc changes respectively.

Henry Wang (2):
docs: Document the minimal requirement of static heap SUPPORT.md: Mark static heap feature as supported

 SUPPORT.md                            | 2 +-
 docs/misc/arm/device-tree/booting.txt | 7 +++++++
 2 files changed, 8 insertions(+), 1 deletion(-)
Signature · 1 line
2.17.1
author
#1 lore raw

PATCH 1/2 docs: Document the minimal requirement of static heap

The static heap feature requires user to know the minimal size of heap to make sure the system can work. Since the heap controlled by Xen is intended to provide memory for the whole system, not only the boot time memory allocation should be covered by the static heap region, but also the runtime allocation should be covered.

The main source of runtime allocation is the memory for the P2M. Currently, from XSA-409, the P2M memory is bounded by the P2M pool. So make this part as the minimal requirement of static heap. The amount of memory allocated after all the guests have been created should be quite limited and mostly predictable.

This commit adds documentation that explains how a user can size the static heap region.

---
 docs/misc/arm/device-tree/booting.txt | 7 +++++++
 1 file changed, 7 insertions(+)
docs/misc/arm/device-tree/booting.txt +7 -0
index 87eaa3e254..046f28ce31 100644
@@ -531,6 +531,13 @@ Below is an example on how to specify the static heap in device tree:
 RAM starting from the host physical address 0x30000000 of 1GB size will
 be reserved as static heap.
 
+Users should be mindful that the static heap should at least satisfy the
+allocation of the P2M maps for all guests. Currently, the minimal requirement
+of per-domain P2M pages pool is in-sync with function
+libxl__get_required_paging_memory() (for xl-created domUs) and
+domain_p2m_pages() (for dom0less domUs), that is, 1MB per vCPU, plus 4KiB per
+MiB of RAM for the P2M map, and plus 512KiB to cover extended regions.
+
 Static Shared Memory
 ====================
 
Signature · 1 line
2.17.1
Acked-by on v1 1/2
#2 lore raw

Re: [PATCH for-4.17 1/2] docs: Document the minimal requirement of static heap

Quoted 12 lines · Henry Wang

The static heap feature requires user to know the minimal size of heap to make sure the system can work. Since the heap controlled by Xen is intended to provide memory for the whole system, not only the boot time memory allocation should be covered by the static heap region, but also the runtime allocation should be covered.

The main source of runtime allocation is the memory for the P2M. Currently, from XSA-409, the P2M memory is bounded by the P2M pool. So make this part as the minimal requirement of static heap. The amount of memory allocated after all the guests have been created should be quite limited and mostly predictable.

Quoted 4 lines

This commit adds documentation that explains how a user can size the static heap region.

Quoted 25 lines
---
 docs/misc/arm/device-tree/booting.txt | 7 +++++++
 1 file changed, 7 insertions(+)
docs/misc/arm/device-tree/booting.txt +7 -0
index 87eaa3e254..046f28ce31 100644
@@ -531,6 +531,13 @@ Below is an example on how to specify the static heap in device tree:
 RAM starting from the host physical address 0x30000000 of 1GB size will
 be reserved as static heap.
 
+Users should be mindful that the static heap should at least satisfy the
+allocation of the P2M maps for all guests. Currently, the minimal requirement
+of per-domain P2M pages pool is in-sync with function
+libxl__get_required_paging_memory() (for xl-created domUs) and
+domain_p2m_pages() (for dom0less domUs), that is, 1MB per vCPU, plus 4KiB per
+MiB of RAM for the P2M map, and plus 512KiB to cover extended regions.
+
 Static Shared Memory
 ====================
 
Signature · 1 line
2.17.1
#3 lore raw

Re: [PATCH for-4.17 1/2] docs: Document the minimal requirement of static heap

In reply to Henry Wang

Hi Henry,

Quoted 25 lines · Henry Wang

The static heap feature requires user to know the minimal size of heap to make sure the system can work. Since the heap controlled by Xen is intended to provide memory for the whole system, not only the boot time memory allocation should be covered by the static heap region, but also the runtime allocation should be covered.

The main source of runtime allocation is the memory for the P2M. Currently, from XSA-409, the P2M memory is bounded by the P2M pool. So make this part as the minimal requirement of static heap. The amount of memory allocated after all the guests have been created should be quite limited and mostly predictable.

This commit adds documentation that explains how a user can size the static heap region.

---
  docs/misc/arm/device-tree/booting.txt | 7 +++++++
  1 file changed, 7 insertions(+)
docs/misc/arm/device-tree/booting.txt +0 -0
index 87eaa3e254..046f28ce31 100644

docs/misc/arm/device-tree/booting.txt Below is an example on how to specify the static heap in device tree: · line 539

Quoted 9 lines
docs/misc/arm/device-tree/booting.txt +6 -0
@@ -531,6 +531,13 @@ Below is an example on how to specify the static heap in device tree:
  RAM starting from the host physical address 0x30000000 of 1GB size will
  be reserved as static heap.
 
+Users should be mindful that the static heap should at least satisfy the
+allocation of the P2M maps for all guests. Currently, the minimal requirement
+of per-domain P2M pages pool is in-sync with function
+libxl__get_required_paging_memory() (for xl-created domUs) and
+domain_p2m_pages() (for dom0less domUs), that is, 1MB per vCPU, plus 4KiB per
+MiB of RAM for the P2M map, and plus 512KiB to cover extended regions.

I think this wording is OK if the feature is a tech preview. However, if this is security supported, we need to provide some more details about the size.

In particular, this doesn't tell a user how they can find the size that would fit them. Can this be decided with a formula?

If not, could a user configure his system without the reserved heap and find out the memory usage from dom0 (or a debug keys)?

Cheers,

Signature · 1 line
Julien Grall
author
#4 lore raw

RE: [PATCH for-4.17 1/2] docs: Document the minimal requirement of static heap

Hi Julien,

Quoted 14 lines

-----Original Message-----
From: Julien Grall <[email protected]>
Subject: Re: [PATCH for-4.17 1/2] docs: Document the minimal requirement of static heap

Hi Henry,

Review

Quoted 2 lines
(quoted diff) +2 -0
+Users should be mindful that the static heap should at least satisfy the
+allocation of the P2M maps for all guests. Currently, the minimal

requirement

docs/misc/arm/device-tree/booting.txt Below is an example on how to specify the static heap in device tree:

Quoted 3 lines
docs/misc/arm/device-tree/booting.txt +3 -0
+of per-domain P2M pages pool is in-sync with function
+libxl__get_required_paging_memory() (for xl-created domUs) and
+domain_p2m_pages() (for dom0less domUs), that is, 1MB per vCPU, plus

4KiB per

Quoted 8 lines
Quoted 1 line

+MiB of RAM for the P2M map, and plus 512KiB to cover extended regions.

I think this wording is OK if the feature is a tech preview. However, if this is security supported, we need to provide some more details about the size.

In particular, this doesn't tell a user how they can find the size that would fit them. Can this be decided with a formula?

My feeling of the formula would be:

My feeling of the formula would be:

Total heap size needed per guest = 1MB * num_guest_vcpu +

    4KB * guest_ram_size_in_mb + 512KB +
    the memory allocated from heap by xzalloc/xzalloc_array for	various uses
    for example alloc_domain_struct(), d->shared_info, evtchn_bucket, etc.

Is this formula somehow make sense to you? I think we need to have a rough estimation of the last part (boot time allocation) though.

Kind regards,
Henry

Quoted 9 lines

If not, could a user configure his system without the reserved heap and find out the memory usage from dom0 (or a debug keys)?

Cheers,

--
Julien Grall

#5 lore raw

Re: [PATCH for-4.17 1/2] docs: Document the minimal requirement of static heap

Quoted 25 lines · Henry Wang

Hi Julien,

Quoted 14 lines

-----Original Message-----
From: Julien Grall <[email protected]>
Subject: Re: [PATCH for-4.17 1/2] docs: Document the minimal requirement of static heap

Hi Henry,

Review

Quoted 2 lines
(quoted diff) +2 -0
+Users should be mindful that the static heap should at least satisfy the
+allocation of the P2M maps for all guests. Currently, the minimal

requirement

docs/misc/arm/device-tree/booting.txt Below is an example on how to specify the static heap in device tree:

Quoted 3 lines
docs/misc/arm/device-tree/booting.txt +3 -0
+of per-domain P2M pages pool is in-sync with function
+libxl__get_required_paging_memory() (for xl-created domUs) and
+domain_p2m_pages() (for dom0less domUs), that is, 1MB per vCPU, plus

4KiB per

Quoted 8 lines
Quoted 1 line

+MiB of RAM for the P2M map, and plus 512KiB to cover extended regions.

I think this wording is OK if the feature is a tech preview. However, if this is security supported, we need to provide some more details about the size.

In particular, this doesn't tell a user how they can find the size that would fit them. Can this be decided with a formula?

Quoted 8 lines

My feeling of the formula would be:

My feeling of the formula would be:

Total heap size needed per guest = 1MB * num_guest_vcpu +

     4KB * guest_ram_size_in_mb + 512KB +
     the memory allocated from heap by xzalloc/xzalloc_array for	various uses
     for example alloc_domain_struct(), d->shared_info, evtchn_bucket, etc.

There are also some pages allocated using alloc_{xen,dom}heap_pages(). We also need to take into account runtime allocation done by some hypercalls (I can't remember which one) or subsystem like OPTee.

In addition to that, you also have memory for the system. E.g frametables, Xen page-tables, various driver allocations...

Quoted 3 lines

Is this formula somehow make sense to you? I think we need to have a rough estimation of the last part (boot time allocation) though.

That's going to be hard. It will vary depending on your system and this could change in the future as we add more features. For instance, I expect the PCI passthrough will need some memory to keep track of all the devices.

I am worry the formula will become complex. Ideally we need to have a very simple formula. If that's not possible, then we need to provide a way for the user to estimate it at runtime (like what I suggested before).

Cheers,

Signature · 1 line
Julien Grall
author
#6 lore raw

RE: [PATCH for-4.17 1/2] docs: Document the minimal requirement of static heap

Hi Julien,

Quoted 38 lines

-----Original Message-----
From: Julien Grall <[email protected]>

Quoted 6 lines
Quoted 6 lines

Review

Quoted 2 lines
(quoted diff) +2 -0
+Users should be mindful that the static heap should at least satisfy the
+allocation of the P2M maps for all guests. Currently, the minimal

requirement

docs/misc/arm/device-tree/booting.txt Below is an example on how to specify the static heap in device tree:

Quoted 3 lines
docs/misc/arm/device-tree/booting.txt +3 -0
+of per-domain P2M pages pool is in-sync with function
+libxl__get_required_paging_memory() (for xl-created domUs) and
+domain_p2m_pages() (for dom0less domUs), that is, 1MB per vCPU,

plus

Quoted 2 lines
Quoted 2 lines

4KiB per

Quoted 1 line

+MiB of RAM for the P2M map, and plus 512KiB to cover extended

regions.

Quoted 5 lines
Quoted 5 lines

I think this wording is OK if the feature is a tech preview. However, if this is security supported, we need to provide some more details about the size.

Quoted 7 lines
Quoted 2 lines

In particular, this doesn't tell a user how they can find the size that would fit them. Can this be decided with a formula?

My feeling of the formula would be:

Total heap size needed per guest = 1MB * num_guest_vcpu +
4KB * guest_ram_size_in_mb + 512KB +
the memory allocated from heap by xzalloc/xzalloc_array for

various uses

Quoted 1 line
     for example alloc_domain_struct(), d->shared_info, evtchn_bucket, etc.

There are also some pages allocated using alloc_{xen,dom}heap_pages(). We also need to take into account runtime allocation done by some hypercalls (I can't remember which one) or subsystem like OPTee.

In addition to that, you also have memory for the system. E.g frametables, Xen page-tables, various driver allocations...

Quoted 3 lines

Is this formula somehow make sense to you? I think we need to have a rough estimation of the last part (boot time allocation) though.

Quoted 8 lines

That's going to be hard. It will vary depending on your system and this could change in the future as we add more features. For instance, I expect the PCI passthrough will need some memory to keep track of all the devices.

I am worry the formula will become complex. Ideally we need to have a very simple formula. If that's not possible, then we need to provide a way for the user to estimate it at runtime (like what I suggested before).

I agree, I think the simple formula can only be achieved is we have an estimation of the worst case scenario of those scattered memory usages. I remember I once had a try so let me try to find the results back that time...

I am also very interested in the method that you proposed to provide a mechanism for users to get the system memory allocation at runtime. But IIUC this needs some work in another series. Could you please confirm if I am understanding correctly? Or probably Xen has some mechanisms that I am likely unaware? Thanks!

Kind regards,
Henry

Quoted 5 lines

Cheers,

--
Julien Grall

#7 lore raw

Re: [PATCH for-4.17 1/2] docs: Document the minimal requirement of static heap

Quoted 1 line · Henry Wang

Hi Julien,

Hi,

Quoted 52 lines
Quoted 39 lines

-----Original Message-----
From: Julien Grall <[email protected]>

Quoted 6 lines
Quoted 6 lines

Review

Quoted 2 lines
(quoted diff) +2 -0
+Users should be mindful that the static heap should at least satisfy the
+allocation of the P2M maps for all guests. Currently, the minimal

requirement

docs/misc/arm/device-tree/booting.txt Below is an example on how to specify the static heap in device tree:

Quoted 3 lines
docs/misc/arm/device-tree/booting.txt +3 -0
+of per-domain P2M pages pool is in-sync with function
+libxl__get_required_paging_memory() (for xl-created domUs) and
+domain_p2m_pages() (for dom0less domUs), that is, 1MB per vCPU,

plus

Quoted 2 lines
Quoted 2 lines

4KiB per

Quoted 1 line

+MiB of RAM for the P2M map, and plus 512KiB to cover extended

regions.

Quoted 5 lines
Quoted 5 lines

I think this wording is OK if the feature is a tech preview. However, if this is security supported, we need to provide some more details about the size.

Quoted 7 lines
Quoted 2 lines

In particular, this doesn't tell a user how they can find the size that would fit them. Can this be decided with a formula?

My feeling of the formula would be:

Total heap size needed per guest = 1MB * num_guest_vcpu +
4KB * guest_ram_size_in_mb + 512KB +
the memory allocated from heap by xzalloc/xzalloc_array for

various uses

Quoted 1 line
      for example alloc_domain_struct(), d->shared_info, evtchn_bucket, etc.

There are also some pages allocated using alloc_{xen,dom}heap_pages(). We also need to take into account runtime allocation done by some hypercalls (I can't remember which one) or subsystem like OPTee.

In addition to that, you also have memory for the system. E.g frametables, Xen page-tables, various driver allocations...

Quoted 3 lines

Is this formula somehow make sense to you? I think we need to have a rough estimation of the last part (boot time allocation) though.

Quoted 8 lines

That's going to be hard. It will vary depending on your system and this could change in the future as we add more features. For instance, I expect the PCI passthrough will need some memory to keep track of all the devices.

I am worry the formula will become complex. Ideally we need to have a very simple formula. If that's not possible, then we need to provide a way for the user to estimate it at runtime (like what I suggested before).

I agree, I think the simple formula can only be achieved is we have an estimation of the worst case scenario of those scattered memory usages. I remember I once had a try so let me try to find the results back that time...

Quoted 5 lines

I am also very interested in the method that you proposed to provide a mechanism for users to get the system memory allocation at runtime. But IIUC this needs some work in another series. Could you please confirm if I am understanding correctly? Or probably Xen has some mechanisms that I am likely unaware? Thanks!

It will depend the way you account memory statically allocated to domains in Xen.

We already provide the total amount of memory in the system and how much is free. The values can be retrieved using ``xl info``.

* When not allocated, is this considered free or used?
* Are they included in the total memory?

If the answer is no for both (possibly just one), then we will need to provide extra hypercalls to expose the size of the xenheap and how much is free.

Cheers,

Signature · 1 line
Julien Grall
author
#8 lore raw

RE: [PATCH for-4.17 1/2] docs: Document the minimal requirement of static heap

Hi Julien,

Quoted 33 lines

-----Original Message-----
From: Julien Grall <[email protected]>
Subject: Re: [PATCH for-4.17 1/2] docs: Document the minimal requirement of static heap

Quoted 16 lines
Quoted 11 lines
Quoted 2 lines

Is this formula somehow make sense to you? I think we need to have a rough estimation of the last part (boot time allocation) though.

That's going to be hard. It will vary depending on your system and this could change in the future as we add more features. For instance, I expect the PCI passthrough will need some memory to keep track of all the devices.

I am worry the formula will become complex. Ideally we need to have a very simple formula. If that's not possible, then we need to provide a way for the user to estimate it at runtime (like what I suggested before).

I agree, I think the simple formula can only be achieved is we have an estimation of the worst case scenario of those scattered memory usages. I remember I once had a try so let me try to find the results back that time...

Quoted 5 lines

I am also very interested in the method that you proposed to provide a mechanism for users to get the system memory allocation at runtime. But IIUC this needs some work in another series. Could you please confirm if I am understanding correctly? Or probably Xen has some mechanisms that I am likely unaware? Thanks!

It will depend the way you account memory statically allocated to domains in Xen.

We already provide the total amount of memory in the system and how much
is free. The values can be retrieved using ``xl info``.

Quoted 6 lines

* When not allocated, is this considered free or used?
* Are they included in the total memory?

If the answer is no for both (possibly just one), then we will need to provide extra hypercalls to expose the size of the xenheap and how much is free.

Thanks for sharing your thoughts. I will add a follow-up series for this work. Before we have this mechanism, I think the patch 2/2 should be dropped.

Kind regards,
Henry

Quoted 5 lines

Cheers,

--
Julien Grall

author
#9 lore raw

PATCH 2/2 SUPPORT.md: Mark static heap feature as supported

In reply to Henry Wang

With the P2M pages pool bounding the domain memory runtime allocation and the documented minimal size requirement of the static heap, it is safe to mark static heap feature as supported.

---
 SUPPORT.md | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
SUPPORT.md +1 -1
index cf2ddfacaf..ddcf8a13a7 100644
@@ -298,7 +298,7 @@ pre-defined by configuration using physical address ranges.
 Allow reserving parts of RAM through the device tree using physical
 address ranges as heap.
 
-    Status, ARM: Tech Preview
+    Status, ARM: Supported
 
 ### Memory Sharing
 
Signature · 1 line
2.17.1
#10 lore raw

Re: [PATCH for-4.17 2/2] SUPPORT.md: Mark static heap feature as supported

Quoted 3 lines · Henry Wang

With the P2M pages pool bounding the domain memory runtime allocation and the documented minimal size requirement of the static heap, it is safe to mark static heap feature as supported.

Usually we wait at least one release cycle before marking a feature as supported. However, I can see that this case is a bit different because effectively static heap memory is a safety/security feature.

Julien, Bertrand, I'll let you know decide on this one

Quoted 21 lines
---
 SUPPORT.md | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
SUPPORT.md +1 -1
index cf2ddfacaf..ddcf8a13a7 100644
@@ -298,7 +298,7 @@ pre-defined by configuration using physical address ranges.
 Allow reserving parts of RAM through the device tree using physical
 address ranges as heap.
 
-    Status, ARM: Tech Preview
+    Status, ARM: Supported
 
 ### Memory Sharing
 
Signature · 1 line
2.17.1
#11 lore raw

Re: [PATCH for-4.17 2/2] SUPPORT.md: Mark static heap feature as supported

Hi Stefano,

Quoted 8 lines · Stefano Stabellini
Quoted 3 lines · Henry Wang

With the P2M pages pool bounding the domain memory runtime allocation and the documented minimal size requirement of the static heap, it is safe to mark static heap feature as supported.

Usually we wait at least one release cycle before marking a feature as supported. However, I can see that this case is a bit different because effectively static heap memory is a safety/security feature.

Even with patch #1 merged, we are still missing some information on how to size the heap. But see below...

Quoted 2 lines

Julien, Bertrand, I'll let you know decide on this one

It doesn't seem to make sense to security support the static heap when allocating statically memory for domains are still in tech preview.

That's because if you want to have your system security supported, then you will need all the allocations to come from the heap. At which point, restricting the heap seems pointless.

So I think it would be better to wait until the static memory is fully supported.

Cheers,

Signature · 1 line
Julien Grall
#12 lore raw

Re: [PATCH for-4.17 2/2] SUPPORT.md: Mark static heap feature as supported

Hi,

Quoted 15 lines · Julien Grall

Hi Stefano,

Quoted 7 lines · Stefano Stabellini
Quoted 3 lines · Henry Wang

With the P2M pages pool bounding the domain memory runtime allocation and the documented minimal size requirement of the static heap, it is safe to mark static heap feature as supported.

Usually we wait at least one release cycle before marking a feature as supported. However, I can see that this case is a bit different because effectively static heap memory is a safety/security feature.

Even with patch #1 merged, we are still missing some information on how to size the heap. But see below...

Quoted 8 lines
Quoted 1 line

Julien, Bertrand, I'll let you know decide on this one

It doesn't seem to make sense to security support the static heap when allocating statically memory for domains are still in tech preview.

That's because if you want to have your system security supported, then you will need all the allocations to come from the heap. At which point, restricting the heap seems pointless.

So I think it would be better to wait until the static memory is fully supported.

I think this should stay a tech preview in 4.17.

Cheers
Bertrand

Quoted 4 lines

Cheers,

Signature · 1 line
Julien Grall