Skip to contentSkip to conversation

static-memory: allow skipping the cache flush

RFC PATCH In review Jan Setje-Eilers 2 participants 5 messages Aug 29 – Oct 11

lore
Share & tools
lore
#0 lore raw

cover static-memory: allow skipping the cache flush

Hi all,

I would like feedback on whether Xen needs to flush every page in a dom0less xen,static-mem bank before assigning it to a guest.

Static-memory support is currently enabled only on Arm, where I am using and testing this change. Its implementation and acquisition path are in common code, so this RFC is about the generic static-memory path.

The current code cleans and invalidates every page in the bank. This is a safe default, but walking a large bank can add several seconds to boot even though Xen only writes the pages used for the guest kernel, initrd, and device tree.

Here, cleaning refers to cache maintenance, not boot scrubbing. With bootscrub=1 or the default bootscrub=idle, Xen assigns dom0less static banks before boot scrubbing starts. The pages are no longer free, so boot scrubbing does not clear them.

My starting point is that if a platform permits the untouched pages to be cleared before guest start, the guest cannot depend on their initial contents. This is a platform policy assumption, not something Xen's boot scrubbing establishes. Xen already cleans each page it writes during domain construction.

Would it therefore be reasonable to let a platform skip the full-bank flush, provided it can also guarantee that the untouched pages are absent from all caches and that firmware, DMA, EL3 services, and other CPUs do not write them before guest start?

The attached patch adds a default-on staticmem-cache-flush option. Specifying no-staticmem-cache-flush opts into the faster path. The normal flush remains the default, and static shared memory is unaffected.

Are these conditions enough to make the opt-out valid, or is there another reason the full-bank flush is required?

I am using no-staticmem-cache-flush with bootscrub=0 on a 64-bit Arm system with two dom0less guests. Both guests boot normally. Skipping the flush reduced Xen startup from about five seconds to about one second.

Thanks for any advice or help with this.

Jan Setje-Eilers (1):
static-memory: allow skipping the cache flush

 docs/misc/xen-command-line.pandoc      | 26 ++++++++++++++++++++++++++
 xen/common/device-tree/static-memory.c |  9 ++++++++-
 xen/common/page_alloc.c                |  6 ++++--
 xen/include/xen/mm.h                   |  2 ++
 4 files changed, 40 insertions(+), 3 deletions(-)

base-commit: 79225a0c77e13b693b4d2b903a88289704b79db6

Signature · 1 line
2.47.3
#1 lore raw

RFC PATCH 1/1 static-memory: allow skipping the cache flush

Static memory acquisition follows the same cache-coherency policy as ordinary heap allocation. Xen therefore cleans and invalidates every page in each xen,static-mem bank before assigning it to a domain.

Static-memory support is currently enabled only on Arm, but its implementation and this acquisition path are in common code.

This cache maintenance is independent of boot scrubbing, which clears memory. With bootscrub=1 or the default bootscrub=idle, create_domUs() acquires static banks before heap_init_late() starts boot scrubbing. The pages are in-use by then, while boot scrubbing processes only free pages. Neither mode therefore clears an assigned static bank.

Walking a large static bank can add several seconds to boot, including time spent flushing pages Xen does not touch during domain construction.

If a platform permits those untouched pages to be cleared before guest start, the guest cannot depend on their incoming contents. This is a platform policy assumption, not a property established by Xen's boot scrubbing. Xen need not make that initial data coherent with RAM. Xen still cleans every page it writes while loading the guest kernel, initrd, and device tree.

Add the default-on staticmem-cache-flush option. Specifying no-staticmem-cache-flush passes MEMF_no_cache_flush when acquiring a xen,static-mem bank. The existing behavior remains the default, and static shared memory is unaffected.

Skipping the flush is safe only if the platform also guarantees that the untouched pages are absent from all caches and no other agent writes them before guest start. Such agents include firmware, non-coherent DMA, and EL3 runtime services.

Assisted-by Codex:GPT-5
---
 docs/misc/xen-command-line.pandoc      | 26 ++++++++++++++++++++++++++
 xen/common/device-tree/static-memory.c |  9 ++++++++-
 xen/common/page_alloc.c                |  6 ++++--
 xen/include/xen/mm.h                   |  2 ++
 4 files changed, 40 insertions(+), 3 deletions(-)
docs/misc/xen-command-line.pandoc +26 -0
index 1c711fa980..3f5c27858c 100644
@@ -2647,6 +2647,32 @@ On Sappire and Emerald Rapids CPUs with May 2025 microcode or later, the
 `ibpb-alt=` option can be used to switch to the alternative mitigation for
 Intel SA-00982.  Intel suggest that some workloads will benefit from this.
 
+### staticmem-cache-flush (arm)
+> `= <boolean>`
+
+> Default: `true`
+
+Control the up-front per-page cache flush over dom0less `xen,static-mem`
+banks. Specify `no-staticmem-cache-flush` to skip it. Xen still cleans the
+guest kernel, initrd, and device tree pages it writes to guest RAM.
+
+Static-memory support is currently available only on Arm. Its implementation
+and this option live in common code.
+
+Here, cleaning means cache maintenance, not boot scrubbing. With
+`bootscrub=1` or the default `bootscrub=idle`, Xen assigns dom0less static
+banks before boot scrubbing starts. Assigned pages are no longer free, so
+boot scrubbing does not clear them. This option is therefore independent of
+`bootscrub`.
+
+Only disable this flush when the guest does not depend on the initial contents
+of pages Xen leaves untouched. For example, this may be true when the platform
+permits those pages to be cleared before guest start.
+
+The platform must also ensure that those pages are absent from all caches and
+rule out other writers before guest start. Such writers may include
+non-coherent DMA or EL3 runtime services active during domain construction.
+
 ### sync_console
 > `= <boolean>`
 
xen/common/device-tree/static-memory.c +8 -1
index ffbc12aa24..ed155d162d 100644
@@ -1,10 +1,15 @@
 /* SPDX-License-Identifier: GPL-2.0-only */
 
+#include <xen/init.h>
+#include <xen/param.h>
 #include <xen/sched.h>
 #include <xen/static-memory.h>
 
 #include <asm/setup.h>
 
+static bool __initdata opt_staticmem_cache_flush = true;
+boolean_param("staticmem-cache-flush", opt_staticmem_cache_flush);
+
 static bool __init append_static_memory_to_bank(struct domain *d,
                                                 struct membank *bank,
                                                 mfn_t smfn,
@@ -54,7 +59,9 @@ static mfn_t __init acquire_static_memory_bank(struct domain *d,
     }
 
     smfn = maddr_to_mfn(*pbase);
-    res = acquire_domstatic_pages(d, smfn, PFN_DOWN(*psize), 0);
+    res = acquire_domstatic_pages(d, smfn, PFN_DOWN(*psize),
+                                  opt_staticmem_cache_flush ? 0 :
+                                  MEMF_no_cache_flush);
     if ( res )
     {
         printk(XENLOG_ERR
xen/common/page_alloc.c +4 -2
index 1e47f38721..b2ede17bd2 100644
@@ -3094,8 +3094,10 @@ static struct page_info * __init acquire_staticmem_pages(mfn_t smfn,
      * Ensure cache and RAM are consistent for platforms where the guest
      * can control its own visibility of/through the cache.
      */
-    for ( i = 0; i < nr_mfns; i++ )
-        flush_page_to_ram(mfn_x(smfn) + i, !(memflags & MEMF_no_icache_flush));
+    if ( !(memflags & MEMF_no_cache_flush) )
+        for ( i = 0; i < nr_mfns; i++ )
+            flush_page_to_ram(mfn_x(smfn) + i,
+                              !(memflags & MEMF_no_icache_flush));
 
     return pg;
 }
xen/include/xen/mm.h +2 -0
index fd8b0ba3f5..511bbc56e0 100644
@@ -228,6 +228,8 @@ struct npfec {
 #define  MEMF_no_icache_flush (1U<<_MEMF_no_icache_flush)
 #define _MEMF_no_scrub    8
 #define  MEMF_no_scrub    (1U<<_MEMF_no_scrub)
+#define _MEMF_no_cache_flush 9
+#define  MEMF_no_cache_flush (1U<<_MEMF_no_cache_flush)
 #define _MEMF_node        16
 #define  MEMF_node_mask   ((1U << (8 * sizeof(nodeid_t))) - 1)
 #define  MEMF_node(n)     ((((n) + 1) & MEMF_node_mask) << _MEMF_node)
Signature · 1 line
2.47.3
#2 lore raw
In reply to Jan Setje-Eilers

Hi Jan,

Sorry for the late answer. The topic is a bit tricky.

Quoted 25 lines · Jan Setje-Eilers

Hi all,

I would like feedback on whether Xen needs to flush every page in a dom0less xen,static-mem bank before assigning it to a guest.

Static-memory support is currently enabled only on Arm, where I am using and testing this change. Its implementation and acquisition path are in common code, so this RFC is about the generic static-memory path.

The current code cleans and invalidates every page in the bank. This is a safe default, but walking a large bank can add several seconds to boot even though Xen only writes the pages used for the guest kernel, initrd, and device tree.

Here, cleaning refers to cache maintenance, not boot scrubbing. With bootscrub=1 or the default bootscrub=idle, Xen assigns dom0less static banks before boot scrubbing starts. The pages are no longer free, so boot scrubbing does not clear them.

My starting point is that if a platform permits the untouched pages to be cleared before guest start, the guest cannot depend on their initial contents. This is a platform policy assumption, not something Xen's boot scrubbing establishes. Xen already cleans each page it writes during domain construction.

Quoted 4 lines

Would it therefore be reasonable to let a platform skip the full-bank flush, provided it can also guarantee that the untouched pages are absent from all caches and that firmware, DMA, EL3 services, and other CPUs do not write them before guest start?

IIUC, the cache flush was mainly added to prevent memory leaking between two guests if they are the MMU off (see XSA-364). I think what you wrote makes sense, but I would prefer Arm to confirm this is fine. For Xen ...

Quoted 3 lines

I am using no-staticmem-cache-flush with bootscrub=0 on a 64-bit Arm system with two dom0less guests. Both guests boot normally. Skipping the flush reduced Xen startup from about five seconds to about one second.

... I think the slow down you describe could also happen when a dynamic VM is created (e.g. via xl).

A few years ago, Arm introduced FEAT_FWB to rework how stage-2 + stage-1 memory attributes works. Currently in Xen, we are using the "old" approach where the used attributes is the strongest of the two. For instance, the guest has the MMU off, then the access will bypass the cache. With FEAT_FWB, you can force the memory attributes to be cacheable even when the MMU is off.

I believe FEAT_FWB should solve the problem you have (assume your HW supports it). In term of effort, it is mostly updating the stage-2 memory attributes and then getting rid of some cache operations (include set/way emulation) when the feature is enabled.

BTW, how much memory do you give to the domain? 4 seconds difference seems quite a lot.

Cheers,

Signature · 1 line
Julien Grall
#3 lore raw

On 9/17/26 04:06, Grall, Julien wrote:

Hi!

Quoted 1 line

Sorry for the late answer. The topic is a bit tricky.

Thanks for replying regardless. FWIW I tried to get some feedback internally as well, and it seemed like no one wanted to get involved. :)

Quoted 26 lines · Jan Setje-Eilers
Quoted 25 lines

Hi all,

I would like feedback on whether Xen needs to flush every page in a dom0less xen,static-mem bank before assigning it to a guest.

Static-memory support is currently enabled only on Arm, where I am using and testing this change. Its implementation and acquisition path are in common code, so this RFC is about the generic static-memory path.

The current code cleans and invalidates every page in the bank. This is a safe default, but walking a large bank can add several seconds to boot even though Xen only writes the pages used for the guest kernel, initrd, and device tree.

Here, cleaning refers to cache maintenance, not boot scrubbing. With bootscrub=1 or the default bootscrub=idle, Xen assigns dom0less static banks before boot scrubbing starts. The pages are no longer free, so boot scrubbing does not clear them.

My starting point is that if a platform permits the untouched pages to be cleared before guest start, the guest cannot depend on their initial contents. This is a platform policy assumption, not something Xen's boot scrubbing establishes. Xen already cleans each page it writes during domain construction.

Quoted 8 lines
Quoted 4 lines

Would it therefore be reasonable to let a platform skip the full-bank flush, provided it can also guarantee that the untouched pages are absent from all caches and that firmware, DMA, EL3 services, and other CPUs do not write them before guest start?

IIUC, the cache flush was mainly added to prevent memory leaking between two guests if they are the MMU off (see XSA-364). I think what you wrote makes sense, but I would prefer Arm to confirm this is fine. For Xen ...

Yes, I think in a world where guests are coming and going, and smaller,
this flush seems relatively sensible.

Quoted 14 lines
Quoted 3 lines

I am using no-staticmem-cache-flush with bootscrub=0 on a 64-bit Arm system with two dom0less guests. Both guests boot normally. Skipping the flush reduced Xen startup from about five seconds to about one second.

... I think the slow down you describe could also happen when a dynamic VM is created (e.g. via xl).

A few years ago, Arm introduced FEAT_FWB to rework how stage-2 + stage-1 memory attributes works. Currently in Xen, we are using the "old" approach where the used attributes is the strongest of the two. For instance, the guest has the MMU off, then the access will bypass the cache. With FEAT_FWB, you can force the memory attributes to be cacheable even when the MMU is off.

Quoted 4 lines

I believe FEAT_FWB should solve the problem you have (assume your HW supports it). In term of effort, it is mostly updating the stage-2 memory attributes and then getting rid of some cache operations (include set/way emulation) when the feature is enabled.

I found that as well, but at least some of my hardware is from just
before that shows up.

Quoted 2 lines

BTW, how much memory do you give to the domain? 4 seconds difference seems quite a lot.

This is two domains: 15 1/4 G to one and 4 1/4 G to the other.

It really stood out when we did a boot time reduction exercise.
-jan

#4 lore raw

Hi Jan,

Quoted 8 lines · Jan Setje-Eilers
Quoted 7 lines · Julien

A few years ago, Arm introduced FEAT_FWB to rework how stage-2 + stage-1 memory attributes works. Currently in Xen, we are using the "old" approach where the used attributes is the strongest of the two. For instance, the guest has the MMU off, then the access will bypass the cache. With FEAT_FWB, you can force the memory attributes to be cacheable even when the MMU is off.

Quoted 6 lines
Quoted 4 lines

I believe FEAT_FWB should solve the problem you have (assume your HW supports it). In term of effort, it is mostly updating the stage-2 memory attributes and then getting rid of some cache operations (include set/way emulation) when the feature is enabled.

I found that as well, but at least some of my hardware is from just
before that shows up.

Thanks for clarification. I would have though most of such HW are pretty much gone :).

I am not entirely sure what's the best way forward for HW not supporting FEAT_FWB. With your approach, I am concerned we may be missing some cases after boot, so we could potentially skip the flush if the pages have not been touched for Xen (you said the pages are not scrubbed for static memory). I'd like some opinions from Bertrand, Michal and Stefano on this.

Cheers,

Signature · 1 line
Julien Grall