Skip to contentSkip to conversation

x86: memcpy() / memset() (non-)ERMS flavors plus fallout

PATCH Superseded Jan Beulich 3 participants 20 messages Nov 25, 2024 – Nov 28, 2024 RISC-V

v3 of 7 Superseded Latest is v7 →

lore
Share & tools
lore
#0 lore raw

v3 cover x86: memcpy() / memset() (non-)ERMS flavors plus fallout

While the performance varies quite a bit on older (pre-ERMS) and newer (ERMS) hardware, so far we've been going with just a single flavor of these two functions, and oddly enough with ones not consistent with one another. Using plain memcpy() / memset() on MMIO (video frame buffer) is generally okay, but the ERMS variant of memcpy() turned out to regress (boot) performance in a way easily visible to the human eye.

v3, apart from the new 1st patch, submitted mainly upon request, as otherwise there was just some re-basing over the 3.5 years since v2.

01: x86: suppress ERMS for internal use when MISC_ENABLE.FAST_STRING is clear 02: x86: re-work memset()
03: x86: re-work memcpy()
04: x86: control memset() and memcpy() inlining
05: x86: introduce "hot" and "cold" page clearing functions
06: page-alloc: make scrub_on_page() static
07: mm: allow page scrubbing routine(s) to be arch controlled

Jan

#1 lore raw

PATCH v3 1/7 x86: suppress ERMS for internal use when MISC_ENABLE.FAST_STRING is clear

Before we start actually adjusting behavior when ERMS is available, follow Linux commit 161ec53c702c ("x86, mem, intel: Initialize Enhanced REP MOVSB/STOSB") and zap the CPUID-derived feature flag when the MSR bit is clear. Don't extend the artificial clearing to guest view,
though: Guests can take their own decision in this regard, as they can read (most of) MISC_ENABLE.

---
TBD: Would be nice if "cpuid=no-erms" propagated to guest view (for
"cpuid=" generally meaning to affect guests as well as Xen), but since both disabling paths use setup_clear_cpu_cap() they're indistinguishable in guest_common_feature_adjustments(). A separate boolean could take care of this, but would look clumsy to me.
---
v3: New.

xen/arch/x86/cpu/intel.c +10 -0
@@ -337,8 +337,18 @@ static void cf_check early_init_intel(st
 		paddr_bits = 36;
 
 	if (c == &boot_cpu_data) {
+		uint64_t misc_enable;
+
 		check_memory_type_self_snoop_errata();
 
+		/*
+		 * If fast string is not enabled in IA32_MISC_ENABLE for any reason,
+		 * clear the enhanced fast string CPU capability.
+		 */
+		rdmsrl(MSR_IA32_MISC_ENABLE, misc_enable);
+		if (!(misc_enable & MSR_IA32_MISC_ENABLE_FAST_STRING))
+			setup_clear_cpu_cap(X86_FEATURE_ERMS);
+
 		intel_init_levelling();
 	}
 
xen/arch/x86/cpu-policy.c +9 -0
@@ -590,6 +590,15 @@ static void __init guest_common_feature_
      */
     if ( host_cpu_policy.feat.ibrsb )
         __set_bit(X86_FEATURE_IBPB, fs);
+
+    /*
+     * We expose MISC_ENABLE to guests, so our internal clearing of ERMS when
+     * FAST_STRING is not set should not propagate to guest view.  Guests can
+     * judge on their own whether to ignore the CPUID bit when the MSR bit is
+     * clear.
+     */
+    if ( raw_cpu_policy.feat.erms )
+        __set_bit(X86_FEATURE_ERMS, fs);
 }
 
 static void __init calculate_pv_max_policy(void)
xen/arch/x86/include/asm/msr-index.h +1 -0
@@ -489,6 +489,7 @@
 #define MSR_IA32_THERM_INTERRUPT	0x0000019b
 #define MSR_IA32_THERM_STATUS		0x0000019c
 #define MSR_IA32_MISC_ENABLE		0x000001a0
+#define MSR_IA32_MISC_ENABLE_FAST_STRING  (1<<0)
 #define MSR_IA32_MISC_ENABLE_PERF_AVAIL   (1<<7)
 #define MSR_IA32_MISC_ENABLE_BTS_UNAVAIL  (1<<11)
 #define MSR_IA32_MISC_ENABLE_PEBS_UNAVAIL (1<<12)
#2 lore raw

Re: [PATCH v3 1/7] x86: suppress ERMS for internal use when MISC_ENABLE.FAST_STRING is clear

Quoted 8 lines · Jan Beulich

Before we start actually adjusting behavior when ERMS is available, follow Linux commit 161ec53c702c ("x86, mem, intel: Initialize Enhanced REP MOVSB/STOSB") and zap the CPUID-derived feature flag when the MSR bit is clear. Don't extend the artificial clearing to guest view,
though: Guests can take their own decision in this regard, as they can read (most of) MISC_ENABLE.

Quoted 8 lines

---
TBD: Would be nice if "cpuid=no-erms" propagated to guest view (for
"cpuid=" generally meaning to affect guests as well as Xen), but since both disabling paths use setup_clear_cpu_cap() they're indistinguishable in guest_common_feature_adjustments(). A separate boolean could take care of this, but would look clumsy to me.
---
v3: New.

I'm not sure this is a terribly wise course of action.  First, ...

Quoted 8 lines
xen/arch/x86/cpu/intel.c +1 -0
@@ -337,8 +337,18 @@ static void cf_check early_init_intel(st
 		paddr_bits = 36;
 
 	if (c == &boot_cpu_data) {
+		uint64_t misc_enable;

xen/arch/x86/cpu/intel.c static void cf_check early_init_intel(st · line 350

Quoted 10 lines
xen/arch/x86/cpu/intel.c +8 -0
+
 		check_memory_type_self_snoop_errata();
 
+		/*
+		 * If fast string is not enabled in IA32_MISC_ENABLE for any reason,
+		 * clear the enhanced fast string CPU capability.
+		 */
+		rdmsrl(MSR_IA32_MISC_ENABLE, misc_enable);
+		if (!(misc_enable & MSR_IA32_MISC_ENABLE_FAST_STRING))
+			setup_clear_cpu_cap(X86_FEATURE_ERMS);

... this is a thread scope MSR, and commonly does have a BIOS option, and has been observed to be asymmetrically configured.  (On Intel at least.  AMD has no equivalent control that I'm aware of.)

It needs checking in init_intel() and on every CPU, and to use plain clear_cpu_cap().

But, we virtualise MSR_MISC_ENABLE (along with MSR_PLATFORM_INFO) to *all* guests in order to advertise CPUID Faulting (even on AMD systems which have an architectural CPUID faulting).  This means that all guests reliably see FAST_STRINGS disabled, even when it happens to be active.

It turns out that Linux will hide ERMS because of this, adversely affecting Linux's choices in the same way that PVShim is about to be impacted.

I see no option but to virtualise MSR_MISC_ENABLE more properly on Intel systems, but it's not without it's complexity.  One #MC errata workaround involves clearing FAST_STRING and leaving it disabled until the next warm reset (Xen has no knowledge about this, but Linux will if provoked).

The ability to modify the FAST_STRING bit is without enumeration; It's simply existed since the P4, and I'm unsure whether we want to honour a guest's choice to disable fast strings, or simply ignore what the guest wants and echo it's choice back to it.

Furthermore, ...

Quoted 9 lines
xen/arch/x86/cpu/intel.c +1 -0
+
 		intel_init_levelling();
 	}
 
xen/arch/x86/cpu-policy.c +0 -0
@@ -590,6 +590,15 @@ static void __init guest_common_feature_
      */
     if ( host_cpu_policy.feat.ibrsb )

xen/arch/x86/cpu-policy.c static void __init guest_common_feature_ · line 601

Quoted 10 lines
xen/arch/x86/cpu-policy.c +9 -0
         __set_bit(X86_FEATURE_IBPB, fs);
+
+    /*
+     * We expose MISC_ENABLE to guests, so our internal clearing of ERMS when
+     * FAST_STRING is not set should not propagate to guest view.  Guests can
+     * judge on their own whether to ignore the CPUID bit when the MSR bit is
+     * clear.
+     */
+    if ( raw_cpu_policy.feat.erms )
+        __set_bit(X86_FEATURE_ERMS, fs);

... this breaks migration of a VM between hosts with different Fast String settings.  ERMS is perf-hint bit, so wants to be set in the Max policy, with Default taking the host value.  There are several other examples of this pattern, but I've not made a magic string for it yet.

~Andrew

#3 lore raw

Re: [PATCH v3 1/7] x86: suppress ERMS for internal use when MISC_ENABLE.FAST_STRING is clear

Quoted 18 lines · Andrew Cooper
Quoted 8 lines · Jan Beulich

Before we start actually adjusting behavior when ERMS is available, follow Linux commit 161ec53c702c ("x86, mem, intel: Initialize Enhanced REP MOVSB/STOSB") and zap the CPUID-derived feature flag when the MSR bit is clear. Don't extend the artificial clearing to guest view,
though: Guests can take their own decision in this regard, as they can read (most of) MISC_ENABLE.

Quoted 8 lines

---
TBD: Would be nice if "cpuid=no-erms" propagated to guest view (for
"cpuid=" generally meaning to affect guests as well as Xen), but since both disabling paths use setup_clear_cpu_cap() they're indistinguishable in guest_common_feature_adjustments(). A separate boolean could take care of this, but would look clumsy to me.
---
v3: New.

Quoted 1 line

I'm not sure this is a terribly wise course of action.  First, ...

It's been a long time, but I wonder if it wasn't you who pointed me at that Linux change.

Quoted 23 lines
Quoted 7 lines
xen/arch/x86/cpu/intel.c +1 -0
@@ -337,8 +337,18 @@ static void cf_check early_init_intel(st
 		paddr_bits = 36;
 
 	if (c == &boot_cpu_data) {
+		uint64_t misc_enable;

xen/arch/x86/cpu/intel.c static void cf_check early_init_intel(st · line 350

Quoted 10 lines
xen/arch/x86/cpu/intel.c +8 -0
+
 		check_memory_type_self_snoop_errata();
 
+		/*
+		 * If fast string is not enabled in IA32_MISC_ENABLE for any reason,
+		 * clear the enhanced fast string CPU capability.
+		 */
+		rdmsrl(MSR_IA32_MISC_ENABLE, misc_enable);
+		if (!(misc_enable & MSR_IA32_MISC_ENABLE_FAST_STRING))
+			setup_clear_cpu_cap(X86_FEATURE_ERMS);

... this is a thread scope MSR, and commonly does have a BIOS option, and has been observed to be asymmetrically configured.  (On Intel at least.  AMD has no equivalent control that I'm aware of.)

Quoted 2 lines

It needs checking in init_intel() and on every CPU, and to use plain clear_cpu_cap().

Except that we have no plain clear_cpu_cap(), and for a good reason: What would be the use of it when invoked post-boot? All we'd get is an inconsistent view of the system, as cleared_caps[] is consumed only by identify_cpu() (and early_cpu_init()).

If we have to fear asymmetry, shouldn't we rather update APs to what the BSP has?

Quoted 3 lines

But, we virtualise MSR_MISC_ENABLE (along with MSR_PLATFORM_INFO) to *all* guests in order to advertise CPUID Faulting (even on AMD systems which have an architectural CPUID faulting).

How does CPUID faulting come into the picture here?

Quoted 2 lines

  This means that all guests
reliably see FAST_STRINGS disabled, even when it happens to be active.

DYM "reliably see FAST_STRINGS set to the host value"? For both PV and HVM we only make some adjustments to the value read from hardware. We don't fiddle with FAST_STRINGS there either way.

Plus that's nothing the patch here changes, so it's not clear to me what you expect me to do in this regard.

Quoted 10 lines

It turns out that Linux will hide ERMS because of this, adversely affecting Linux's choices in the same way that PVShim is about to be impacted.

I see no option but to virtualise MSR_MISC_ENABLE more properly on Intel systems, but it's not without it's complexity.  One #MC errata workaround involves clearing FAST_STRING and leaving it disabled until the next warm reset (Xen has no knowledge about this, but Linux will if provoked).

As per above, that's a separate change then. I'm neither changing MISC_ENABLE here, nor do I do anything affecting what guests would see there.

Quoted 4 lines

The ability to modify the FAST_STRING bit is without enumeration; It's simply existed since the P4, and I'm unsure whether we want to honour a guest's choice to disable fast strings, or simply ignore what the guest wants and echo it's choice back to it.

Perhaps the latter, at least as a first step. Otherwise we get into playing with the MSR on context switch.

Quoted 10 lines
Quoted 10 lines
xen/arch/x86/cpu-policy.c +4 -0
@@ -590,6 +590,15 @@ static void __init guest_common_feature_
      */
     if ( host_cpu_policy.feat.ibrsb )
         __set_bit(X86_FEATURE_IBPB, fs);
+
+    /*
+     * We expose MISC_ENABLE to guests, so our internal clearing of ERMS when
+     * FAST_STRING is not set should not propagate to guest view.  Guests can
Quoted 8 lines

xen/arch/x86/cpu-policy.c static void __init guest_common_feature_ · line 601

Quoted 5 lines
xen/arch/x86/cpu-policy.c +5 -0
+     * judge on their own whether to ignore the CPUID bit when the MSR bit is
+     * clear.
+     */
+    if ( raw_cpu_policy.feat.erms )
+        __set_bit(X86_FEATURE_ERMS, fs);

... this breaks migration of a VM between hosts with different Fast String settings.

Isn't this broken already? They'd observe each host's FAST_STRING setting. I'm extending that brokenness to CPUID, yes, and ...

Quoted 3 lines

  ERMS is perf-hint bit, so wants to be set in the Max policy, with Default taking the host value.  There are several other examples of this pattern, but I've not made a magic string for it yet.

... I will certainly switch to this model to avoid doing so. Using raw rather than host value though as the default, because of the clearing of the feature in the host policy (implicit from its clearing via setup_clear_cpu_cap()). I.e. keeping the hunk above as is, but additionally set the bit in the max policies (in guest_common_max_feature_adjustments()).

Jan

#4 lore raw

PATCH v3 2/7 x86: re-work memset()

In reply to Jan Beulich

Move the function to its own assembly file. Having it in C just for the entire body to be an asm() isn't really helpful. Then have two flavors:
A "basic" version using qword steps for the bulk of the operation, and an ERMS version for modern hardware, to be substituted in via alternatives patching.

---
We may want to consider branching over the REP STOSQ as well, if the number of qwords turns out to be zero.
We may also want to consider using non-REP STOS{L,W,B} for the tail.
---
v3: Re-base.

xen/arch/x86/Makefile +1 -0
@@ -48,6 +48,7 @@ obj-$(CONFIG_INDIRECT_THUNK) += indirect
 obj-$(CONFIG_PV) += ioport_emulate.o
 obj-y += irq.o
 obj-$(CONFIG_KEXEC) += machine_kexec.o
+obj-y += memset.o
 obj-y += mm.o x86_64/mm.o
 obj-$(CONFIG_HVM) += monitor.o
 obj-y += mpparse.o
xen/arch/x86/memset.S new file +30 -0
@@ -0,0 +1,30 @@
+#include <asm/asm_defns.h>
+
+.macro memset
+        and     $7, %edx
+        shr     $3, %rcx
+        movzbl  %sil, %esi
+        mov     $0x0101010101010101, %rax
+        imul    %rsi, %rax
+        mov     %rdi, %rsi
+        rep stosq
+        or      %edx, %ecx
+        jz      0f
+        rep stosb
+0:
+        mov     %rsi, %rax
+        ret
+.endm
+
+.macro memset_erms
+        mov     %esi, %eax
+        mov     %rdi, %rsi
+        rep stosb
+        mov     %rsi, %rax
+        ret
+.endm
+
+FUNC(memset)
+        mov     %rdx, %rcx
+        ALTERNATIVE memset, memset_erms, X86_FEATURE_ERMS
+END(memset)
xen/arch/x86/string.c +0 -13
@@ -22,19 +22,6 @@ void *(memcpy)(void *dest, const void *s
     return dest;
 }
 
-void *(memset)(void *s, int c, size_t n)
-{
-    long d0, d1;
-
-    asm volatile (
-        "rep stosb"
-        : "=&c" (d0), "=&D" (d1)
-        : "a" (c), "1" (s), "0" (n)
-        : "memory");
-
-    return s;
-}
-
 void *(memmove)(void *dest, const void *src, size_t n)
 {
     long d0, d1, d2;
#5 lore raw

Re: [PATCH v3 2/7] x86: re-work memset()

Quoted 7 lines · Jan Beulich

Move the function to its own assembly file. Having it in C just for the entire body to be an asm() isn't really helpful. Then have two flavors:
A "basic" version using qword steps for the bulk of the operation, and an ERMS version for modern hardware, to be substituted in via alternatives patching.

This is far nicer than previous versions with nested alternatives.

Quoted 3 lines

---
We may want to consider branching over the REP STOSQ as well, if the number of qwords turns out to be zero.

Until FSR{S,M} (Fast Short Rep {STO,MOV}SB), which is far newer than ERMS, passing 0 into any REP instruction is expensive.

I wonder how often we memset with a size less than 8.

Quoted 1 line

We may also want to consider using non-REP STOS{L,W,B} for the tail.

Probably, yes.  We use this form in non-ERMS cases, where we're advised to stay away from STOSB entirely.

Interestingly, Linux doesn't have a STOSQ case at all.  Or rather, it was deleted by Linus in 20f3337d350c last year.  It was also identified as causing a performance regression.
https://lore.kernel.org/lkml/CANn89iKUbyrJ=r2+_kK+sb2ZSSHifFZ7QkPLDpAtkJ8v4WUumA@mail.gmail.com/T/#u
although the memset() path was not reverted as part of the fix (47ee3f1dd93bcb eventually).

Yet ca96b162bfd2 shows that REP MOVSQ is still definitely a win on Rome CPUs.

I expect we probably do want some non-rep forms in here.

Do you have any benchmarks with this series?

Quoted 21 lines

---
v3: Re-base.

xen/arch/x86/Makefile +1 -0
@@ -48,6 +48,7 @@ obj-$(CONFIG_INDIRECT_THUNK) += indirect
 obj-$(CONFIG_PV) += ioport_emulate.o
 obj-y += irq.o
 obj-$(CONFIG_KEXEC) += machine_kexec.o
+obj-y += memset.o
 obj-y += mm.o x86_64/mm.o
 obj-$(CONFIG_HVM) += monitor.o
 obj-y += mpparse.o
xen/arch/x86/memset.S new file +5 -0
@@ -0,0 +1,30 @@
+#include <asm/asm_defns.h>
+
+.macro memset
+        and     $7, %edx
+        shr     $3, %rcx

xen/arch/x86/memset.S line 15

Quoted 10 lines
xen/arch/x86/memset.S new file +10 -0
+        movzbl  %sil, %esi
+        mov     $0x0101010101010101, %rax
+        imul    %rsi, %rax
+        mov     %rdi, %rsi
+        rep stosq
+        or      %edx, %ecx
+        jz      0f
+        rep stosb
+0:
+        mov     %rsi, %rax

Could you use %r8/9/etc instead of %rsi please?  This is deceptively close to looking like a bug, and it took me a while to figure out it's only correct because STOSB only edits %rdi.

Otherwise, I suspect this can go in.  It should be an improvement on plain REP STOSB on non-ERMS systems, even if there are other improvements to come.  I specifically wouldn't suggest blocking it until patch 1 is resolved.

~Andrew

#6 lore raw

Re: [PATCH v3 2/7] x86: re-work memset()

Quoted 7 lines · Andrew Cooper
Quoted 3 lines · Jan Beulich

---
We may want to consider branching over the REP STOSQ as well, if the number of qwords turns out to be zero.

Until FSR{S,M} (Fast Short Rep {STO,MOV}SB), which is far newer than ERMS, passing 0 into any REP instruction is expensive.

Is this a request to add such a conditional branch then, perhaps patched out when FSRS is available? And then perhaps also for the JZ that's already there?

Quoted 1 line

I wonder how often we memset with a size less than 8.

Hence why I raised the point, rather than putting the jump there directly.

Quoted 4 lines
Quoted 1 line

We may also want to consider using non-REP STOS{L,W,B} for the tail.

Probably, yes.  We use this form in non-ERMS cases, where we're advised to stay away from STOSB entirely.

Yet then we'll end up with three conditional branches - do we really want that?

Quoted 7 lines

Interestingly, Linux doesn't have a STOSQ case at all.  Or rather, it was deleted by Linus in 20f3337d350c last year.  It was also identified as causing a performance regression.
https://lore.kernel.org/lkml/CANn89iKUbyrJ=r2+_kK+sb2ZSSHifFZ7QkPLDpAtkJ8v4WUumA@mail.gmail.com/T/#u
although the memset() path was not reverted as part of the fix (47ee3f1dd93bcb eventually).

Quoted 6 lines

Yet ca96b162bfd2 shows that REP MOVSQ is still definitely a win on Rome CPUs.

I expect we probably do want some non-rep forms in here.

Do you have any benchmarks with this series?

What I specifically measured were the clear_page() variants. I didn't do any measurements for memset() (or memcpy()), first ad foremost because any selection of inputs is going to be arbitrary rather than representative.

Quoted 14 lines
Quoted 14 lines
xen/arch/x86/memset.S new file +11 -0
@@ -0,0 +1,30 @@
+#include <asm/asm_defns.h>
+
+.macro memset
+        and     $7, %edx
+        shr     $3, %rcx
+        movzbl  %sil, %esi
+        mov     $0x0101010101010101, %rax
+        imul    %rsi, %rax
+        mov     %rdi, %rsi
+        rep stosq
+        or      %edx, %ecx
Quoted 8 lines

xen/arch/x86/memset.S line 15

Quoted 4 lines
xen/arch/x86/memset.S new file +4 -0
+        jz      0f
+        rep stosb
+0:
+        mov     %rsi, %rax

Could you use %r8/9/etc instead of %rsi please?  This is deceptively close to looking like a bug, and it took me a while to figure out it's only correct because STOSB only edits %rdi.

Well, I can certainly switch (the number of REX prefixes will remain the same as it looks), but the fact that STOS, unlike MOVS, doesn't touch %rsi is a pretty basic one.

Does your request extend to all uses of %rsi (and %esi), or merely the latter two (across the REP STOS)?

Jan

#7 lore raw

PATCH v3 3/7 x86: re-work memcpy()

In reply to Jan Beulich

Move the function to its own assembly file. Having it in C just for the entire body to be an asm() isn't really helpful. Then have two flavors:
A "basic" version using qword steps for the bulk of the operation, and an ERMS version for modern hardware, to be substituted in via alternatives patching.

Alternatives patching, however, requires an extra precaution: It uses memcpy() itself, and hence the function may patch itself. Luckily the patched-in code only replaces the prolog of the original function. Make sure this remains this way.

Additionally alternatives patching, while supposedly safe via enforcing a control flow change when modifying already prefetched code, may not really be. Afaict a request is pending to drop the first of the two options in the SDM's "Handling Self- and Cross-Modifying Code" section. Insert a serializing instruction there.

---
We may want to consider branching over the REP MOVSQ as well, if the number of qwords turns out to be zero.
We may also want to consider using non-REP MOVS{L,W,B} for the tail.

TBD: We may further need a workaround similar to Linux'es 8ca97812c3c8
("x86/mce: Work around an erratum on fast string copy instructions").
---
v3: Re-base.

xen/arch/x86/Makefile +1 -0
@@ -48,6 +48,7 @@ obj-$(CONFIG_INDIRECT_THUNK) += indirect
 obj-$(CONFIG_PV) += ioport_emulate.o
 obj-y += irq.o
 obj-$(CONFIG_KEXEC) += machine_kexec.o
+obj-y += memcpy.o
 obj-y += memset.o
 obj-y += mm.o x86_64/mm.o
 obj-$(CONFIG_HVM) += monitor.o
xen/arch/x86/alternative.c +3 -1
@@ -153,12 +153,14 @@ void init_or_livepatch add_nops(void *in
  * executing.
  *
  * "noinline" to cause control flow change and thus invalidate I$ and
- * cause refetch after modification.
+ * cause refetch after modification.  While the SDM continues to suggest this
+ * is sufficient, it may not be - issue a serializing insn afterwards as well.
  */
 static void init_or_livepatch noinline
 text_poke(void *addr, const void *opcode, size_t len)
 {
     memcpy(addr, opcode, len);
+    cpuid_eax(0);
 }
 
 extern void *const __initdata_cf_clobber_start[];
xen/arch/x86/memcpy.S new file +20 -0
@@ -0,0 +1,20 @@
+#include <asm/asm_defns.h>
+
+FUNC(memcpy)
+        mov     %rdx, %rcx
+        mov     %rdi, %rax
+        /*
+         * We need to be careful here: memcpy() is involved in alternatives
+         * patching, so the code doing the actual copying (i.e. past setting
+         * up registers) may not be subject to patching (unless further
+         * precautions were taken).
+         */
+        ALTERNATIVE "and $7, %edx; shr $3, %rcx", \
+                    "rep movsb; ret", X86_FEATURE_ERMS
+        rep movsq
+        or      %edx, %ecx
+        jz      1f
+        rep movsb
+1:
+        ret
+END(memcpy)
xen/arch/x86/string.c +0 -15
@@ -7,21 +7,6 @@
 
 #include <xen/lib.h>
 
-void *(memcpy)(void *dest, const void *src, size_t n)
-{
-    long d0, d1, d2;
-
-    asm volatile (
-        "   rep ; movs"__OS" ; "
-        "   mov %k4,%k3      ; "
-        "   rep ; movsb        "
-        : "=&c" (d0), "=&D" (d1), "=&S" (d2)
-        : "0" (n/BYTES_PER_LONG), "r" (n%BYTES_PER_LONG), "1" (dest), "2" (src)
-        : "memory" );
-
-    return dest;
-}
-
 void *(memmove)(void *dest, const void *src, size_t n)
 {
     long d0, d1, d2;
#8 lore raw

Re: [PATCH v3 3/7] x86: re-work memcpy()

Quoted 17 lines · Jan Beulich

Move the function to its own assembly file. Having it in C just for the entire body to be an asm() isn't really helpful. Then have two flavors:
A "basic" version using qword steps for the bulk of the operation, and an ERMS version for modern hardware, to be substituted in via alternatives patching.

Alternatives patching, however, requires an extra precaution: It uses memcpy() itself, and hence the function may patch itself. Luckily the patched-in code only replaces the prolog of the original function. Make sure this remains this way.

Additionally alternatives patching, while supposedly safe via enforcing a control flow change when modifying already prefetched code, may not really be. Afaict a request is pending to drop the first of the two options in the SDM's "Handling Self- and Cross-Modifying Code" section. Insert a serializing instruction there.

Quoted 5 lines

---
We may want to consider branching over the REP MOVSQ as well, if the number of qwords turns out to be zero.
We may also want to consider using non-REP MOVS{L,W,B} for the tail.

My feedback for patch 2 is largely applicable here too.

Quoted 4 lines

TBD: We may further need a workaround similar to Linux'es 8ca97812c3c8
("x86/mce: Work around an erratum on fast string copy instructions").

Ah, so you found that erratum.  I'd say there's lower hanging fruit to go after in the MCE logic before we get to this.

Quoted 16 lines

---
v3: Re-base.

xen/arch/x86/Makefile +1 -0
@@ -48,6 +48,7 @@ obj-$(CONFIG_INDIRECT_THUNK) += indirect
 obj-$(CONFIG_PV) += ioport_emulate.o
 obj-y += irq.o
 obj-$(CONFIG_KEXEC) += machine_kexec.o
+obj-y += memcpy.o
 obj-y += memset.o
 obj-y += mm.o x86_64/mm.o
 obj-$(CONFIG_HVM) += monitor.o

xen/arch/x86/alternative.c void init_or_livepatch add_nops(void *in · line 157

Quoted 6 lines
xen/arch/x86/alternative.c +2 -1
@@ -153,12 +153,14 @@ void init_or_livepatch add_nops(void *in
  * executing.
  *
  * "noinline" to cause control flow change and thus invalidate I$ and
- * cause refetch after modification.
+ * cause refetch after modification.  While the SDM continues to suggest this
+ * is sufficient, it may not be - issue a serializing insn afterwards as well.

Did you find a problem in practice, or is this just in case?

I suspect if you are seeing problems, then it's non-atomicity of the stores into memcpy() rather than serialisation.

xen/arch/x86/alternative.c void init_or_livepatch add_nops(void *in · line 163

Quoted 6 lines
xen/arch/x86/alternative.c +1 -0
  */
 static void init_or_livepatch noinline
 text_poke(void *addr, const void *opcode, size_t len)
 {
     memcpy(addr, opcode, len);
+    cpuid_eax(0);

This whole function is buggy in a couple of ways, starting with the comments.

The comment about noinline and control flow changes is only really relevant to 32bit processors; we inherited that comment from Linux, and they're not applicable to Xen.

AMD64 (both the APM, and SDM) guarantee that Self Modifying Code will be dealt with on your behalf, with no serialisation needed.

Cross-modifying code needs far more severe serialisation than given here.  We get away with it because alternative_{instructions,branches}() are pre-SMP, and apply_alternatives() is on livepatches prior to them becoming live.

I happen to know there's an AMD CPU which has an erratum regarding Self Modifying Code and genuinely does need a serialising instruction, but I don't know which exact CPU it is.

If we're going to put a serialising instruction, it should be a write to CR2.  We don't care about 486 compatibility, and it's faster than CPUID and much much faster if virtualised because it's unlikely to be intercepted even under shadow paging.

But, it would be nice not to put serialisation in the general case to begin with, especially not into the livepatching case.

~Andrew

#9 lore raw

Re: [PATCH v3 3/7] x86: re-work memcpy()

Quoted 18 lines · Andrew Cooper
Quoted 17 lines · Jan Beulich

Move the function to its own assembly file. Having it in C just for the entire body to be an asm() isn't really helpful. Then have two flavors:
A "basic" version using qword steps for the bulk of the operation, and an ERMS version for modern hardware, to be substituted in via alternatives patching.

Alternatives patching, however, requires an extra precaution: It uses memcpy() itself, and hence the function may patch itself. Luckily the patched-in code only replaces the prolog of the original function. Make sure this remains this way.

Additionally alternatives patching, while supposedly safe via enforcing a control flow change when modifying already prefetched code, may not really be. Afaict a request is pending to drop the first of the two options in the SDM's "Handling Self- and Cross-Modifying Code" section. Insert a serializing instruction there.

Quoted 7 lines
Quoted 5 lines

---
We may want to consider branching over the REP MOVSQ as well, if the number of qwords turns out to be zero.
We may also want to consider using non-REP MOVS{L,W,B} for the tail.

My feedback for patch 2 is largely applicable here too.

Sure, and I'll apply here whatever we decide to do there.

Quoted 11 lines

xen/arch/x86/alternative.c void init_or_livepatch add_nops(void *in · line 157

Quoted 9 lines
xen/arch/x86/alternative.c +2 -1
@@ -153,12 +153,14 @@ void init_or_livepatch add_nops(void *in
  * executing.
  *
  * "noinline" to cause control flow change and thus invalidate I$ and
- * cause refetch after modification.
+ * cause refetch after modification.  While the SDM continues to suggest this
+ * is sufficient, it may not be - issue a serializing insn afterwards as well.

Did you find a problem in practice, or is this just in case?

It's been too long, so I can now only guess that it's just in case. The comment change, otoh, suggests otherwise.

Quoted 2 lines

I suspect if you are seeing problems, then it's non-atomicity of the stores into memcpy() rather than serialisation.

How would atomicity (or not) matter here? There shouldn't be any difference between a single and any number of stores into the (previously executed) insn stream.

Quoted 23 lines

xen/arch/x86/alternative.c void init_or_livepatch add_nops(void *in · line 163

Quoted 6 lines
xen/arch/x86/alternative.c +1 -0
  */
 static void init_or_livepatch noinline
 text_poke(void *addr, const void *opcode, size_t len)
 {
     memcpy(addr, opcode, len);
+    cpuid_eax(0);

This whole function is buggy in a couple of ways, starting with the comments.

The comment about noinline and control flow changes is only really relevant to 32bit processors; we inherited that comment from Linux, and they're not applicable to Xen.

AMD64 (both the APM, and SDM) guarantee that Self Modifying Code will be dealt with on your behalf, with no serialisation needed.

Cross-modifying code needs far more severe serialisation than given here.  We get away with it because alternative_{instructions,branches}() are pre-SMP, and apply_alternatives() is on livepatches prior to them becoming live.

Quoted 3 lines

I happen to know there's an AMD CPU which has an erratum regarding Self Modifying Code and genuinely does need a serialising instruction, but I don't know which exact CPU it is.

Maybe I ran into that on one of the two older AMD systems I routinely test on every once in a while?

Quoted 7 lines

If we're going to put a serialising instruction, it should be a write to CR2.  We don't care about 486 compatibility, and it's faster than CPUID and much much faster if virtualised because it's unlikely to be intercepted even under shadow paging.

But, it would be nice not to put serialisation in the general case to begin with, especially not into the livepatching case.

If you're aware of an erratum there, how can we get away without any serialization? I can surely switch to a CR2 write, and I can also make this dependent upon system_state (thus excluding the LP case).

I notice that arch_livepatch_{apply,revert}() indeed use plain memcpy() with just the noinline "protection". I wonder how well that works if a livepatch actually touched the tail of either of these functions (however unlikely that may be).

Jan

#10 lore raw

PATCH v3 4/7 x86: control memset() and memcpy() inlining

In reply to Jan Beulich

Stop the compiler from inlining non-trivial memset() and memcpy() (for memset() see e.g. map_vcpu_info() or kimage_load_segments() for examples). This way we even keep the compiler from using REP STOSQ / REP MOVSQ when we'd prefer REP STOSB / REP MOVSB (when ERMS is available).

With gcc10 this yields a modest .text size reduction (release build) of around 2k.

Unfortunately these options aren't understood by the clang versions I have readily available for testing with; I'm unaware of equivalents.

Note also that using cc-option-add is not an option here, or at least I couldn't make things work with it (in case the option was not supported by the compiler): The embedded comma in the option looks to be getting in the way.

---
v3: Re-base.
v2: New.
---
The boundary values are of course up for discussion - I wasn't really certain whether to use 16 or 32; I'd be less certain about using yet larger values.

Similarly whether to permit the compiler to emit REP STOSQ / REP MOVSQ for known size, properly aligned blocks is up for discussion.

xen/arch/x86/arch.mk +3 -0
@@ -65,6 +65,9 @@ endif
 $(call cc-option-add,CFLAGS_stack_boundary,CC,-mpreferred-stack-boundary=3)
 export CFLAGS_stack_boundary
 
+CFLAGS += $(call cc-option,$(CC),-mmemcpy-strategy=unrolled_loop:16:noalign$(comma)libcall:-1:noalign)
+CFLAGS += $(call cc-option,$(CC),-mmemset-strategy=unrolled_loop:16:noalign$(comma)libcall:-1:noalign)
+
 ifeq ($(CONFIG_UBSAN),y)
 # Don't enable alignment sanitisation.  x86 has efficient unaligned accesses,
 # and various things (ACPI tables, hypercall pages, stubs, etc) are wont-fix.
#11 lore raw

Re: [PATCH v3 4/7] x86: control memset() and memcpy() inlining

Quoted 27 lines · Jan Beulich

Stop the compiler from inlining non-trivial memset() and memcpy() (for memset() see e.g. map_vcpu_info() or kimage_load_segments() for examples). This way we even keep the compiler from using REP STOSQ / REP MOVSQ when we'd prefer REP STOSB / REP MOVSB (when ERMS is available).

With gcc10 this yields a modest .text size reduction (release build) of around 2k.

Unfortunately these options aren't understood by the clang versions I have readily available for testing with; I'm unaware of equivalents.

Note also that using cc-option-add is not an option here, or at least I couldn't make things work with it (in case the option was not supported by the compiler): The embedded comma in the option looks to be getting in the way.

---
v3: Re-base.
v2: New.
---
The boundary values are of course up for discussion - I wasn't really certain whether to use 16 or 32; I'd be less certain about using yet larger values.

Quoted 2 lines

Similarly whether to permit the compiler to emit REP STOSQ / REP MOVSQ for known size, properly aligned blocks is up for discussion.

I didn't realise there were any options like this.

The result is very different on GCC-12, with the following extremes:

add/remove: 0/0 grow/shrink: 83/71 up/down: 8764/-3913 (4851) Function                                     old     new   delta x86_emulate                               136966  139990   +3024 ptwr_emulated_cmpxchg                        555    1058    +503 hvm_emulate_cmpxchg                         1178    1648    +470 hvmemul_do_io                               1605    2059    +454 hvmemul_linear_mmio_access                  1060    1324    +264 hvmemul_write_cache                          655     890    +235
...
do_console_io                               1293    1170    -123 arch_get_info_guest                         2200    2072    -128 avtab_read_item                              821     692    -129 acpi_tb_create_local_fadt                    866     714    -152 xz_dec_lzma2_run                            2573    2272    -301 __hvm_copy                                  1085     737    -348
Total: Before=3902769, After=3907620, chg +0.12%

So there is a mix, but it's in a distinctly upward direction.

As a possibly-related tangent, something I did notice when playing with -fanalyzer was that even attr(alloc_size/align) helped the code generation for an inlined memcpy().

e.g. with _xmalloc() only getting __attribute__((alloc_size(1),alloc_align(2))), functions like init_domain_cpu_policy() go from:

48 8b 13                 mov    (%rbx),%rdx
48 8d 78 08              lea    0x8(%rax),%rdi 48 89 c1                 mov    %rax,%rcx
48 89 de                 mov    %rbx,%rsi
48 83 e7 f8              and    $0xfffffffffffffff8,%rdi 48 89 10                 mov    %rdx,(%rax)
48 29 f9                 sub    %rdi,%rcx
48 8b 93 b0 07 00 00     mov    0x7b0(%rbx),%rdx 48 29 ce                 sub    %rcx,%rsi
81 c1 b8 07 00 00        add    $0x7b8,%ecx
48 89 90 b0 07 00 00     mov    %rdx,0x7b0(%rax) c1 e9 03                 shr    $0x3,%ecx
f3 48 a5                 rep movsq %ds:(%rsi),%es:(%rdi)

down to simply

48 89 c7                 mov    %rax,%rdi
b9 f7 00 00 00           mov    $0xf7,%ecx
48 89 ee                 mov    %rbp,%rsi
f3 48 a5                 rep movsq %ds:(%rsi),%es:(%rdi)

which is removing the logic to cope with a misaligned destination pointer.

As a possibly unrelated tangent, even __attribute__((malloc)) seems to have some code gen changes.

In xenctl_bitmap_to_cpumask(), the change is simply to not align the -ENOMEM basic block, saving 8 bytes.  This is quite reasonable because xmalloc() genuinely failing is 0% of the time to many significant figures.

Mostly though, it's just basic block churn, which seems to be giving a "likely not NULL" on the return value, therefore shuffling the error paths.

~Andrew

#12 lore raw

Re: [PATCH v3 4/7] x86: control memset() and memcpy() inlining

Quoted 9 lines · Andrew Cooper
Quoted 8 lines · Jan Beulich

Stop the compiler from inlining non-trivial memset() and memcpy() (for memset() see e.g. map_vcpu_info() or kimage_load_segments() for examples). This way we even keep the compiler from using REP STOSQ / REP MOVSQ when we'd prefer REP STOSB / REP MOVSB (when ERMS is available).

With gcc10 this yields a modest .text size reduction (release build) of around 2k.

With this, ...

Quoted 42 lines
Quoted 18 lines

Unfortunately these options aren't understood by the clang versions I have readily available for testing with; I'm unaware of equivalents.

Note also that using cc-option-add is not an option here, or at least I couldn't make things work with it (in case the option was not supported by the compiler): The embedded comma in the option looks to be getting in the way.

---
v3: Re-base.
v2: New.
---
The boundary values are of course up for discussion - I wasn't really certain whether to use 16 or 32; I'd be less certain about using yet larger values.

Quoted 2 lines

Similarly whether to permit the compiler to emit REP STOSQ / REP MOVSQ for known size, properly aligned blocks is up for discussion.

I didn't realise there were any options like this.

The result is very different on GCC-12, with the following extremes:

add/remove: 0/0 grow/shrink: 83/71 up/down: 8764/-3913 (4851) Function                                     old     new   delta x86_emulate                               136966  139990   +3024 ptwr_emulated_cmpxchg                        555    1058    +503 hvm_emulate_cmpxchg                         1178    1648    +470 hvmemul_do_io                               1605    2059    +454 hvmemul_linear_mmio_access                  1060    1324    +264 hvmemul_write_cache                          655     890    +235
...
do_console_io                               1293    1170    -123 arch_get_info_guest                         2200    2072    -128 avtab_read_item                              821     692    -129 acpi_tb_create_local_fadt                    866     714    -152 xz_dec_lzma2_run                            2573    2272    -301 __hvm_copy                                  1085     737    -348
Total: Before=3902769, After=3907620, chg +0.12%

Quoted 1 line

So there is a mix, but it's in a distinctly upward direction.

... was this a release or a debug build? Of course I'm not surprised of there being differences between compiler versions, but the overall change being clearly in the opposite direction is still a little worrying.

Quoted 35 lines

As a possibly-related tangent, something I did notice when playing with -fanalyzer was that even attr(alloc_size/align) helped the code generation for an inlined memcpy().

e.g. with _xmalloc() only getting __attribute__((alloc_size(1),alloc_align(2))), functions like init_domain_cpu_policy() go from:

48 8b 13                 mov    (%rbx),%rdx
48 8d 78 08              lea    0x8(%rax),%rdi 48 89 c1                 mov    %rax,%rcx
48 89 de                 mov    %rbx,%rsi
48 83 e7 f8              and    $0xfffffffffffffff8,%rdi 48 89 10                 mov    %rdx,(%rax)
48 29 f9                 sub    %rdi,%rcx
48 8b 93 b0 07 00 00     mov    0x7b0(%rbx),%rdx 48 29 ce                 sub    %rcx,%rsi
81 c1 b8 07 00 00        add    $0x7b8,%ecx
48 89 90 b0 07 00 00     mov    %rdx,0x7b0(%rax) c1 e9 03                 shr    $0x3,%ecx
f3 48 a5                 rep movsq %ds:(%rsi),%es:(%rdi)

down to simply

48 89 c7                 mov    %rax,%rdi
b9 f7 00 00 00           mov    $0xf7,%ecx
48 89 ee                 mov    %rbp,%rsi
f3 48 a5                 rep movsq %ds:(%rsi),%es:(%rdi)

which is removing the logic to cope with a misaligned destination pointer.

As a possibly unrelated tangent, even __attribute__((malloc)) seems to have some code gen changes.

Quoted 6 lines

In xenctl_bitmap_to_cpumask(), the change is simply to not align the -ENOMEM basic block, saving 8 bytes.  This is quite reasonable because xmalloc() genuinely failing is 0% of the time to many significant figures.

Mostly though, it's just basic block churn, which seems to be giving a "likely not NULL" on the return value, therefore shuffling the error paths.

Could you clarify for me what of the above is the actionable part, for me to take care of?

Jan

#13 lore raw

PATCH v3 5/7 x86: introduce "hot" and "cold" page clearing functions

In reply to Jan Beulich

The present clear_page_sse2() is useful in case a page isn't going to get touched again soon, or if we want to limit churn on the caches. Amend it by alternatively using CLZERO, which has been found to be quite a bit faster on Zen2 hardware at least. Note that to use CLZERO, we need to know the cache line size, and hence a feature dependency on CLFLUSH gets introduced.

For cases where latency is the most important aspect, or when it is expected that sufficiently large parts of a page will get accessed again soon after the clearing, introduce a "hot" alternative. Again use alternatives patching to select between a "legacy" and an ERMS variant.

Don't switch any callers just yet - this will be the subject of subsequent changes.

---
v3: Re-base.
v2: New.
---
Note: Ankur indicates that for ~L3-size or larger regions MOVNT/CLZERO
is better even latency-wise.

xen/arch/x86/clear_page.S +43 -4
@@ -1,9 +1,9 @@
         .file __FILE__
 
-#include <xen/linkage.h>
-#include <asm/page.h>
+#include <xen/page-size.h>
+#include <asm/asm_defns.h>
 
-FUNC(clear_page_sse2)
+        .macro clear_page_sse2
         mov     $PAGE_SIZE/32, %ecx
         xor     %eax,%eax
 
@@ -17,4 +17,43 @@ FUNC(clear_page_sse2)
 
         sfence
         ret
-END(clear_page_sse2)
+        .endm
+
+        .macro clear_page_clzero
+        mov     %rdi, %rax
+        mov     $PAGE_SIZE/64, %ecx
+        .globl clear_page_clzero_post_count
+clear_page_clzero_post_count:
+
+0:      clzero
+        sub     $-64, %rax
+        .globl clear_page_clzero_post_neg_size
+clear_page_clzero_post_neg_size:
+        sub     $1, %ecx
+        jnz     0b
+
+        sfence
+        ret
+        .endm
+
+FUNC(clear_page_cold)
+        ALTERNATIVE clear_page_sse2, clear_page_clzero, X86_FEATURE_CLZERO
+END(clear_page_cold)
+
+        .macro clear_page_stosb
+        mov     $PAGE_SIZE, %ecx
+        xor     %eax,%eax
+        rep stosb
+        ret
+        .endm
+
+        .macro clear_page_stosq
+        mov     $PAGE_SIZE/8, %ecx
+        xor     %eax, %eax
+        rep stosq
+        ret
+        .endm
+
+FUNC(clear_page_hot)
+        ALTERNATIVE clear_page_stosq, clear_page_stosb, X86_FEATURE_ERMS
+END(clear_page_hot)
xen/arch/x86/cpu/common.c +35 -2
@@ -58,6 +58,9 @@ DEFINE_PER_CPU(bool, full_gdt_loaded);
 
 DEFINE_PER_CPU(uint32_t, pkrs);
 
+extern uint32_t clear_page_clzero_post_count[];
+extern int8_t clear_page_clzero_post_neg_size[];
+
 void __init setup_clear_cpu_cap(unsigned int cap)
 {
 	const uint32_t *dfs;
@@ -355,8 +358,38 @@ void __init early_cpu_init(bool verbose)
 
 	edx &= ~cleared_caps[FEATURESET_1d];
 	ecx &= ~cleared_caps[FEATURESET_1c];
-	if (edx & cpufeat_mask(X86_FEATURE_CLFLUSH))
-		c->x86_cache_alignment = ((ebx >> 8) & 0xff) * 8;
+	if (edx & cpufeat_mask(X86_FEATURE_CLFLUSH)) {
+		unsigned int size = ((ebx >> 8) & 0xff) * 8;
+
+		c->x86_cache_alignment = size;
+
+		/*
+		 * Patch in parameters of clear_page_cold()'s CLZERO
+		 * alternative. Note that for now we cap this at 128 bytes.
+		 * Larger cache line sizes would still be dealt with
+		 * correctly, but would cause redundant work done.
+		 */
+		if (size > 128)
+			size = 128;
+		if (size && !(size & (size - 1))) {
+			/*
+			 * Need to play some games to keep the compiler from
+			 * recognizing the negative array index as being out
+			 * of bounds. The labels in assembler code really are
+			 * _after_ the locations to be patched, so the
+			 * negative index is intentional.
+			 */
+			uint32_t *pcount = clear_page_clzero_post_count;
+			int8_t *neg_size = clear_page_clzero_post_neg_size;
+
+			OPTIMIZER_HIDE_VAR(pcount);
+			OPTIMIZER_HIDE_VAR(neg_size);
+			pcount[-1] = PAGE_SIZE / size;
+			neg_size[-1] = -size;
+		}
+		else
+			setup_clear_cpu_cap(X86_FEATURE_CLZERO);
+	}
 	/* Leaf 0x1 capabilities filled in early for Xen. */
 	c->x86_capability[FEATURESET_1d] = edx;
 	c->x86_capability[FEATURESET_1c] = ecx;
xen/arch/x86/include/asm/asm-defns.h +4 -0
@@ -20,6 +20,10 @@
     .byte 0x0f, 0x01, 0xdd
 .endm
 
+.macro clzero
+    .byte 0x0f, 0x01, 0xfc
+.endm
+
 /*
  * Call a noreturn function.  This could be JMP, but CALL results in a more
  * helpful backtrace.  BUG is to catch functions which do decide to return...
xen/arch/x86/include/asm/page.h +3 -2
@@ -219,10 +219,11 @@ typedef struct { u64 pfn; } pagetable_t;
 #define pagetable_from_paddr(p) pagetable_from_pfn((p)>>PAGE_SHIFT)
 #define pagetable_null()        pagetable_from_pfn(0)
 
-void clear_page_sse2(void *pg);
+void clear_page_hot(void *pg);
+void clear_page_cold(void *pg);
 void copy_page_sse2(void *to, const void *from);
 
-#define clear_page(_p)      clear_page_sse2(_p)
+#define clear_page(_p)      clear_page_cold(_p)
 #define copy_page(_t, _f)   copy_page_sse2(_t, _f)
 
 /* Convert between Xen-heap virtual addresses and machine addresses. */
xen/tools/gen-cpuid.py +4 -0
@@ -212,6 +212,10 @@ def crunch_numbers(state):
         # the first place.
         APIC: [X2APIC, TSC_DEADLINE, EXTAPIC],
 
+        # The CLZERO insn requires a means to determine the cache line size,
+        # which is tied to the CLFLUSH insn.
+        CLFLUSH: [CLZERO],
+
         # AMD built MMXExtentions and 3DNow as extentions to MMX.
         MMX: [MMXEXT, _3DNOW],
 
#14 lore raw

PATCH v3 6/7 page-alloc: make scrub_one_page() static

In reply to Jan Beulich

Before starting to alter its properties, restrict the function's visibility. The only external user is mem-paging, which we can accommodate by different means.

Also move the function up in its source file, so we won't need to forward-declare it. Constify its parameter at the same time.

---
v3: Re-base.
v2: New.

xen/arch/x86/mm/mem_paging.c +0 -3
@@ -304,9 +304,6 @@ static int evict(struct domain *d, gfn_t
     ret = p2m_set_entry(p2m, gfn, INVALID_MFN, PAGE_ORDER_4K,
                         p2m_ram_paged, a);
 
-    /* Clear content before returning the page to Xen */
-    scrub_one_page(page);
-
     /* Track number of paged gfns */
     atomic_inc(&d->paged_pages);
 
xen/common/page_alloc.c +21 -19
@@ -137,6 +137,7 @@
 #include <xen/sections.h>
 #include <xen/softirq.h>
 #include <xen/spinlock.h>
+#include <xen/vm_event.h>
 
 #include <asm/flushtlb.h>
 #include <asm/page.h>
@@ -774,6 +775,21 @@ static void page_list_add_scrub(struct p
 #endif
 #define SCRUB_BYTE_PATTERN   (SCRUB_PATTERN & 0xff)
 
+static void scrub_one_page(const struct page_info *pg)
+{
+    if ( unlikely(pg->count_info & PGC_broken) )
+        return;
+
+#ifndef NDEBUG
+    /* Avoid callers relying on allocations returning zeroed pages. */
+    unmap_domain_page(memset(__map_domain_page(pg),
+                             SCRUB_BYTE_PATTERN, PAGE_SIZE));
+#else
+    /* For a production build, clear_page() is the fastest way to scrub. */
+    clear_domain_page(_mfn(page_to_mfn(pg)));
+#endif
+}
+
 static void poison_one_page(struct page_info *pg)
 {
 #ifdef CONFIG_SCRUB_DEBUG
@@ -2548,10 +2564,12 @@ void free_domheap_pages(struct page_info
             /*
              * Normally we expect a domain to clear pages before freeing them,
              * if it cares about the secrecy of their contents. However, after
-             * a domain has died we assume responsibility for erasure. We do
-             * scrub regardless if option scrub_domheap is set.
+             * a domain has died or if it has mem-paging enabled we assume
+             * responsibility for erasure. We do scrub regardless if option
+             * scrub_domheap is set.
              */
-            scrub = d->is_dying || scrub_debug || opt_scrub_domheap;
+            scrub = d->is_dying || mem_paging_enabled(d) ||
+                    scrub_debug || opt_scrub_domheap;
         }
         else
         {
@@ -2635,22 +2653,6 @@ static __init int cf_check pagealloc_key
 }
 __initcall(pagealloc_keyhandler_init);
 
-
-void scrub_one_page(struct page_info *pg)
-{
-    if ( unlikely(pg->count_info & PGC_broken) )
-        return;
-
-#ifndef NDEBUG
-    /* Avoid callers relying on allocations returning zeroed pages. */
-    unmap_domain_page(memset(__map_domain_page(pg),
-                             SCRUB_BYTE_PATTERN, PAGE_SIZE));
-#else
-    /* For a production build, clear_page() is the fastest way to scrub. */
-    clear_domain_page(_mfn(page_to_mfn(pg)));
-#endif
-}
-
 static void cf_check dump_heap(unsigned char key)
 {
     s_time_t      now = NOW();
xen/arch/x86/include/asm/mem_paging.h +0 -6
@@ -12,12 +12,6 @@
 
 int mem_paging_memop(XEN_GUEST_HANDLE_PARAM(xen_mem_paging_op_t) arg);
 
-#ifdef CONFIG_MEM_PAGING
-# define mem_paging_enabled(d) vm_event_check_ring((d)->vm_event_paging)
-#else
-# define mem_paging_enabled(d) false
-#endif
-
 #endif /*__ASM_X86_MEM_PAGING_H__ */
 
 /*
xen/include/xen/mm.h +0 -2
@@ -532,8 +532,6 @@ static inline unsigned int get_order_fro
     return order;
 }
 
-void scrub_one_page(struct page_info *pg);
-
 #ifndef arch_free_heap_page
 #define arch_free_heap_page(d, pg) \
     page_list_del(pg, page_to_list(d, pg))
xen/include/xen/sched.h +6 -0
@@ -1203,6 +1203,12 @@ static always_inline bool is_iommu_enabl
     return evaluate_nospec(d->options & XEN_DOMCTL_CDF_iommu);
 }
 
+#ifdef CONFIG_MEM_PAGING
+# define mem_paging_enabled(d) vm_event_check_ring((d)->vm_event_paging)
+#else
+# define mem_paging_enabled(d) false
+#endif
+
 extern bool sched_smt_power_savings;
 extern bool sched_disable_smt_switching;
 
Acked-by on v3 6/7
#15 lore raw

Re: [PATCH v3 6/7] page-alloc: make scrub_one_page() static

Hi Jan,

Quoted 8 lines · Jan Beulich

Before starting to alter its properties, restrict the function's visibility. The only external user is mem-paging, which we can accommodate by different means.

Also move the function up in its source file, so we won't need to forward-declare it. Constify its parameter at the same time.

Cheers,

Signature · 1 line
Julien Grall
#16 lore raw

PATCH v3 7/7 mm: allow page scrubbing routine(s) to be arch controlled

In reply to Jan Beulich

Especially when dealing with large amounts of memory, memset() may not be very efficient; this can be bad enough that even for debug builds a custom function is warranted. We additionally want to distinguish "hot" and "cold" cases (with, as initial heuristic, "hot" being for any allocations a domain does for itself, assuming that in all other cases the page wouldn't be accessed [again] soon). The goal is for accesses of "cold" pages to not disturb caches (albeit finding a good balance between this and the higher latency looks to be difficult).

Keep the default fallback to clear_page_*() in common code; this may want to be revisited down the road.

---
v3: Re-base.
v2: New.
---
The choice between hot and cold in scrub_one_page()'s callers is certainly up for discussion / improvement.

xen/arch/arm/include/asm/page.h +6 -0
@@ -144,6 +144,12 @@ extern size_t dcache_line_bytes;
 
 #define copy_page(dp, sp) memcpy(dp, sp, PAGE_SIZE)
 
+#define clear_page_hot  clear_page
+#define clear_page_cold clear_page
+
+#define scrub_page_hot(page) memset(page, SCRUB_BYTE_PATTERN, PAGE_SIZE)
+#define scrub_page_cold      scrub_page_hot
+
 static inline size_t read_dcache_line_bytes(void)
 {
     register_t ctr;
xen/arch/ppc/include/asm/page.h +6 -0
@@ -190,6 +190,12 @@ static inline void invalidate_icache(voi
 #define clear_page(page) memset(page, 0, PAGE_SIZE)
 #define copy_page(dp, sp) memcpy(dp, sp, PAGE_SIZE)
 
+#define clear_page_hot  clear_page
+#define clear_page_cold clear_page
+
+#define scrub_page_hot(page) memset(page, SCRUB_BYTE_PATTERN, PAGE_SIZE)
+#define scrub_page_cold      scrub_page_hot
+
 /* TODO: Flush the dcache for an entire page. */
 static inline void flush_page_to_ram(unsigned long mfn, bool sync_icache)
 {
xen/arch/riscv/include/asm/page.h +6 -0
@@ -156,6 +156,12 @@ static inline void invalidate_icache(voi
 #define clear_page(page) memset((void *)(page), 0, PAGE_SIZE)
 #define copy_page(dp, sp) memcpy(dp, sp, PAGE_SIZE)
 
+#define clear_page_hot  clear_page
+#define clear_page_cold clear_page
+
+#define scrub_page_hot(page) memset(page, SCRUB_BYTE_PATTERN, PAGE_SIZE)
+#define scrub_page_cold      scrub_page_hot
+
 /* TODO: Flush the dcache for an entire page. */
 static inline void flush_page_to_ram(unsigned long mfn, bool sync_icache)
 {
xen/arch/x86/Makefile +1 -0
@@ -59,6 +59,7 @@ obj-y += pci.o
 obj-y += physdev.o
 obj-$(CONFIG_COMPAT) += x86_64/physdev.o
 obj-$(CONFIG_X86_PSR) += psr.o
+obj-bin-$(CONFIG_DEBUG) += scrub_page.o
 obj-y += setup.o
 obj-y += shutdown.o
 obj-y += smp.o
xen/arch/x86/include/asm/page.h +5 -0
@@ -226,6 +226,11 @@ void copy_page_sse2(void *to, const void
 #define clear_page(_p)      clear_page_cold(_p)
 #define copy_page(_t, _f)   copy_page_sse2(_t, _f)
 
+#ifdef CONFIG_DEBUG
+void scrub_page_hot(void *);
+void scrub_page_cold(void *);
+#endif
+
 /* Convert between Xen-heap virtual addresses and machine addresses. */
 #define __pa(x)             (virt_to_maddr(x))
 #define __va(x)             (maddr_to_virt(x))
xen/arch/x86/scrub_page.S new file +39 -0
@@ -0,0 +1,39 @@
+        .file __FILE__
+
+#include <asm/asm_defns.h>
+#include <xen/page-size.h>
+#include <xen/scrub.h>
+
+FUNC(scrub_page_cold)
+        mov     $PAGE_SIZE/32, %ecx
+        mov     $SCRUB_PATTERN, %rax
+
+0:      movnti  %rax,   (%rdi)
+        movnti  %rax,  8(%rdi)
+        movnti  %rax, 16(%rdi)
+        movnti  %rax, 24(%rdi)
+        add     $32, %rdi
+        sub     $1, %ecx
+        jnz     0b
+
+        sfence
+        ret
+END(scrub_page_cold)
+
+        .macro scrub_page_stosb
+        mov     $PAGE_SIZE, %ecx
+        mov     $SCRUB_BYTE_PATTERN, %eax
+        rep stosb
+        ret
+        .endm
+
+        .macro scrub_page_stosq
+        mov     $PAGE_SIZE/8, %ecx
+        mov     $SCRUB_PATTERN, %rax
+        rep stosq
+        ret
+        .endm
+
+FUNC(scrub_page_hot)
+        ALTERNATIVE scrub_page_stosq, scrub_page_stosb, X86_FEATURE_ERMS
+END(scrub_page_hot)
xen/common/page_alloc.c +26 -19

92 lines not on this page · raw message

xen/include/xen/scrub.h new file +24 -0
@@ -0,0 +1,24 @@
+#ifndef __XEN_SCRUB_H__
+#define __XEN_SCRUB_H__
+
+#include <xen/const.h>
+
+/* SCRUB_PATTERN needs to be a repeating series of bytes. */
+#ifdef CONFIG_DEBUG
+# define SCRUB_PATTERN       _AC(0xc2c2c2c2c2c2c2c2,ULL)
+#else
+# define SCRUB_PATTERN       _AC(0,ULL)
+#endif
+#define SCRUB_BYTE_PATTERN   (SCRUB_PATTERN & 0xff)
+
+#endif /* __XEN_SCRUB_H__ */
+
+/*
+ * Local variables:
+ * mode: C
+ * c-file-style: "BSD"
+ * c-basic-offset: 4
+ * tab-width: 4
+ * indent-tabs-mode: nil
+ * End:
+ */
#17 lore raw

Re: [PATCH v3 7/7] mm: allow page scrubbing routine(s) to be arch controlled

Hi Jan,

Quoted 23 lines · Jan Beulich

Especially when dealing with large amounts of memory, memset() may not be very efficient; this can be bad enough that even for debug builds a custom function is warranted. We additionally want to distinguish "hot" and "cold" cases (with, as initial heuristic, "hot" being for any allocations a domain does for itself, assuming that in all other cases the page wouldn't be accessed [again] soon). The goal is for accesses of "cold" pages to not disturb caches (albeit finding a good balance between this and the higher latency looks to be difficult).

Keep the default fallback to clear_page_*() in common code; this may want to be revisited down the road.

---
v3: Re-base.
v2: New.
---
The choice between hot and cold in scrub_one_page()'s callers is certainly up for discussion / improvement.

xen/arch/arm/include/asm/page.h extern size_t dcache_line_bytes; · line 151

Quoted 8 lines
xen/arch/arm/include/asm/page.h +5 -0
@@ -144,6 +144,12 @@ extern size_t dcache_line_bytes;
 
  #define copy_page(dp, sp) memcpy(dp, sp, PAGE_SIZE)
 
+#define clear_page_hot  clear_page
+#define clear_page_cold clear_page
+
+#define scrub_page_hot(page) memset(page, SCRUB_BYTE_PATTERN, PAGE_SIZE)
+#define scrub_page_cold      scrub_page_hot

This block seems to be common between all the arch but x86. Should we add an header in asm generic?

The rest looks fine to me.

Cheers,

Signature · 1 line
Julien Grall
#18 lore raw

Re: [PATCH v3 7/7] mm: allow page scrubbing routine(s) to be arch controlled

Quoted 6 lines · Julien Grall
Quoted 6 lines
xen/arch/arm/include/asm/page.h +0 -0
@@ -144,6 +144,12 @@ extern size_t dcache_line_bytes;
 
  #define copy_page(dp, sp) memcpy(dp, sp, PAGE_SIZE)
 
Quoted 8 lines

xen/arch/arm/include/asm/page.h extern size_t dcache_line_bytes; · line 151

Quoted 5 lines
xen/arch/arm/include/asm/page.h +5 -0
+#define clear_page_hot  clear_page
+#define clear_page_cold clear_page
+
+#define scrub_page_hot(page) memset(page, SCRUB_BYTE_PATTERN, PAGE_SIZE)
+#define scrub_page_cold      scrub_page_hot

This block seems to be common between all the arch but x86. Should we add an header in asm generic?

I'd say that largely depends on the intentions of Arm, RISC-V, and PPC. Personally I've always found it odd that memset() / memcpy() are used for page clearing / copying. Surely there are better ways, and pretty certainly about every arch also has distinct means to efficiently do "hot" and "cold" clearing. Therefore keeping these #define-s in per-arch headers imo serves as a reminder that something wants doing about them.

Jan

Acked-by
#19 lore raw

Re: [PATCH v3 7/7] mm: allow page scrubbing routine(s) to be arch controlled

Hi Jan,

Quoted 16 lines · Jan Beulich
Quoted 6 lines · Julien Grall
Quoted 6 lines
xen/arch/arm/include/asm/page.h +0 -0
@@ -144,6 +144,12 @@ extern size_t dcache_line_bytes;
 
   #define copy_page(dp, sp) memcpy(dp, sp, PAGE_SIZE)
 
Quoted 8 lines

xen/arch/arm/include/asm/page.h extern size_t dcache_line_bytes; · line 151

Quoted 5 lines
xen/arch/arm/include/asm/page.h +5 -0
+#define clear_page_hot  clear_page
+#define clear_page_cold clear_page
+
+#define scrub_page_hot(page) memset(page, SCRUB_BYTE_PATTERN, PAGE_SIZE)
+#define scrub_page_cold      scrub_page_hot

This block seems to be common between all the arch but x86. Should we add an header in asm generic?

Quoted 6 lines

I'd say that largely depends on the intentions of Arm, RISC-V, and PPC. Personally I've always found it odd that memset() / memcpy() are used for page clearing / copying. Surely there are better ways, and pretty certainly about every arch also has distinct means to efficiently do "hot" and "cold" clearing. Therefore keeping these #define-s in per-arch headers imo serves as a reminder that something wants doing about them.

Fair enough. For the code:

Cheers,

Signature · 1 line
Julien Grall