target/i386: add support for FRED
v3 cover target/i386: add support for FRED
This patch set adds support for the Intel flexible return and event delivery (FRED) architecture to allow Qemu to run KVM guests with FRED enabled.
The FRED architecture defines simple new transitions that change privilege level (ring transitions). The FRED architecture was designed with the following goals:
1) Improve overall performance and response time by replacing event delivery through the interrupt descriptor table (IDT event delivery) and event return by the IRET instruction with lower latency transitions.
2) Improve software robustness by ensuring that event delivery establishes the full supervisor context and that event return establishes the full user context.
Intel VMX architecture is extended to run FRED guests, and the changes are majorly:
1) New VMCS fields for FRED context management, which includes two new event data VMCS fields, eight new guest FRED context VMCS fields and eight new host FRED context VMCS fields.
2) VMX nested-Exception support for proper virtualization of stack levels introduced with FRED architecture.
Search for the latest FRED spec in most search engines with this search pattern:
site:intel.com FRED (flexible return and event delivery) specification
The counterpart KVM patch set is at:
https://lore.kernel.org/kvm/[email protected]/T/#m77876e22876f41c5ec677c0834a46113a4987d31
---
Changelog
v3:
- Add WRMSRNS as a baseline feature for FRED.
- Add the secondary VM exit controls MSR.
- Add FRED VMX controls to VM exit/entry feature words and scripts/kvm/vmxcap.
- Do not set/get FRED SSP0 MSR, i.e. PL0_SSP MSR, with FRED, leave it to KVM CET.
v2:
- Add VMX nested-exception support to scripts/kvm/vmxcap (Paolo Bonzini).
- Move FRED MSRs from basic x86_cpu part to .subsections part (Weijiang Yang).
Xin Li (6):
target/i386: add support for FRED in CPUID enumeration target/i386: mark CR4.FRED not reserved target/i386: add the secondary VM exit controls MSR target/i386: add support for VMX FRED controls target/i386: enumerate VMX nested-exception support target/i386: Add get/set/migrate support for FRED MSRs
scripts/kvm/vmxcap | 13 ++++++++++++ target/i386/cpu.c | 15 ++++++++++--- target/i386/cpu.h | 42 ++++++++++++++++++++++++++++++++++++- target/i386/kvm/kvm.c | 49 +++++++++++++++++++++++++++++++++++++++++++ target/i386/machine.c | 28 +++++++++++++++++++++++++ 5 files changed, 143 insertions(+), 4 deletions(-)
base-commit: a3c3aaa846ad61b801e7196482dcf4afb8ba34e4
Signature · 1 line
2.42.0
PATCH v3 1/6 target/i386: add support for FRED in CPUID enumeration
FRED, i.e., the Intel flexible return and event delivery architecture, defines simple new transitions that change privilege level (ring transitions).
The new transitions defined by the FRED architecture are FRED event delivery and, for returning from events, two FRED return instructions. FRED event delivery can effect a transition from ring 3 to ring 0, but it is used also to deliver events incident to ring 0. One FRED instruction (ERETU) effects a return from ring 0 to ring 3, while the other (ERETS) returns while remaining in ring 0. Collectively, FRED event delivery and the FRED return instructions are FRED transitions.
In addition to these transitions, the FRED architecture defines a new instruction (LKGS) for managing the state of the GS segment register. The LKGS instruction can be used by 64-bit operating systems that do not use the new FRED transitions.
WRMSRNS is an instruction that behaves exactly like WRMSR, with the only difference being that it is not a serializing instruction by default. Under certain conditions, WRMSRNS may replace WRMSR to improve performance. FRED uses it to switch RSP0 in a faster manner.
Search for the latest FRED spec in most search engines with this search pattern:
site:intel.com FRED (flexible return and event delivery) specification
The CPUID feature flag CPUID.(EAX=7,ECX=1):EAX[17] enumerates FRED, and the CPUID feature flag CPUID.(EAX=7,ECX=1):EAX[18] enumerates LKGS, and the CPUID feature flag CPUID.(EAX=7,ECX=1):EAX[19] enumerates WRMSRNS.
Add CPUID definitions for FRED/LKGS/WRMSRNS, and expose them to KVM guests.
Because FRED relies on LKGS and WRMSRNS, add that to feature dependency map.
--- target/i386/cpu.c | 10 +++++++++- target/i386/cpu.h | 6 ++++++ 2 files changed, 15 insertions(+), 1 deletion(-)
target/i386/cpu.c +9 -1
@@ -965,7 +965,7 @@ FeatureWordInfo feature_word_info[FEATURE_WORDS] = {"avx-vnni" ,"avx512-bf16" ,NULL ,"cmpccxadd" ,NULL ,NULL ,"fzrm" ,"fsrs" ,"fsrc" ,NULL ,NULL ,NULL , -NULL ,NULL ,NULL ,NULL , +NULL ,"fred" ,"lkgs" ,"wrmsrns" ,NULL ,"amx-fp16" ,NULL ,"avx-ifma" ,NULL ,NULL ,NULL ,NULL , NULL, NULL, NULL, NULL,
@@ -1552,6 +1552,14 @@ static FeatureDep feature_dependencies[] = { .from = { FEAT_VMX_SECONDARY_CTLS, VMX_SECONDARY_EXEC_ENABLE_USER_WAIT_PAUSE }, .to = { FEAT_7_0_ECX, CPUID_7_0_ECX_WAITPKG }, }, + { + .from = { FEAT_7_1_EAX, CPUID_7_1_EAX_FRED }, + .to = { FEAT_7_1_EAX, CPUID_7_1_EAX_LKGS }, + }, + { + .from = { FEAT_7_1_EAX, CPUID_7_1_EAX_FRED }, + .to = { FEAT_7_1_EAX, CPUID_7_1_EAX_WRMSRNS }, + }, }; typedef struct X86RegisterInfo32 {
target/i386/cpu.h +6 -0
@@ -934,6 +934,12 @@ uint64_t x86_cpu_get_supported_feature_word(FeatureWord w,#define CPUID_7_1_EDX_AMX_COMPLEX (1U << 8)/* PREFETCHIT0/1 Instructions */ #define CPUID_7_1_EDX_PREFETCHITI (1U << 14) +/* Flexible return and event delivery (FRED) */ +#define CPUID_7_1_EAX_FRED (1U << 17) +/* Load into IA32_KERNEL_GS_BASE (LKGS) */ +#define CPUID_7_1_EAX_LKGS (1U << 18) +/* Non-Serializing Write to Model Specific Register (WRMSRNS) */ +#define CPUID_7_1_EAX_WRMSRNS (1U << 19)/* Do not exhibit MXCSR Configuration Dependent Timing (MCDT) behavior */ #define CPUID_7_2_EDX_MCDT_NO (1U << 5)
Signature · 1 line
2.42.0
Re: [PATCH v3 1/6] target/i386: add support for FRED in CPUID enumeration
Quoted 58 lines · Xin Li
FRED, i.e., the Intel flexible return and event delivery architecture, defines simple new transitions that change privilege level (ring transitions).
The new transitions defined by the FRED architecture are FRED event delivery and, for returning from events, two FRED return instructions. FRED event delivery can effect a transition from ring 3 to ring 0, but it is used also to deliver events incident to ring 0. One FRED instruction (ERETU) effects a return from ring 0 to ring 3, while the other (ERETS) returns while remaining in ring 0. Collectively, FRED event delivery and the FRED return instructions are FRED transitions.
In addition to these transitions, the FRED architecture defines a new instruction (LKGS) for managing the state of the GS segment register. The LKGS instruction can be used by 64-bit operating systems that do not use the new FRED transitions.
WRMSRNS is an instruction that behaves exactly like WRMSR, with the only difference being that it is not a serializing instruction by default. Under certain conditions, WRMSRNS may replace WRMSR to improve performance. FRED uses it to switch RSP0 in a faster manner.
Search for the latest FRED spec in most search engines with this search pattern:
site:intel.com FRED (flexible return and event delivery) specification
The CPUID feature flag CPUID.(EAX=7,ECX=1):EAX[17] enumerates FRED, and the CPUID feature flag CPUID.(EAX=7,ECX=1):EAX[18] enumerates LKGS, and the CPUID feature flag CPUID.(EAX=7,ECX=1):EAX[19] enumerates WRMSRNS.
Add CPUID definitions for FRED/LKGS/WRMSRNS, and expose them to KVM guests.
Because FRED relies on LKGS and WRMSRNS, add that to feature dependency map.
Tested-by Shan Kang<[email protected]>Signed-off-by Xin Li<[email protected]>--- target/i386/cpu.c | 10 +++++++++- target/i386/cpu.h | 6 ++++++ 2 files changed, 15 insertions(+), 1 deletion(-)target/i386/cpu.c +1
-1@@ -965,7 +965,7 @@ FeatureWordInfo feature_word_info[FEATURE_WORDS] = {"avx-vnni" ,"avx512-bf16" ,NULL ,"cmpccxadd" ,NULL ,NULL ,"fzrm" ,"fsrs" ,"fsrc" ,NULL ,NULL ,NULL , -NULL ,NULL ,NULL ,NULL , +NULL ,"fred" ,"lkgs" ,"wrmsrns" ,NULL ,"amx-fp16" ,NULL ,"avx-ifma" ,NULL ,NULL ,NULL ,NULL , NULL, NULL, NULL, NULL,@@ -1552,6 +1552,14 @@ static FeatureDep feature_dependencies[] = { .from = { FEAT_VMX_SECONDARY_CTLS, VMX_SECONDARY_EXEC_ENABLE_USER_WAIT_PAUSE },
target/i386/cpu.c static FeatureDep feature_dependencies[] = { · line 1562
Quoted 10 lines
target/i386/cpu.c +8
-0.to = { FEAT_7_0_ECX, CPUID_7_0_ECX_WAITPKG }, }, + { + .from = { FEAT_7_1_EAX, CPUID_7_1_EAX_FRED }, + .to = { FEAT_7_1_EAX, CPUID_7_1_EAX_LKGS }, + }, + { + .from = { FEAT_7_1_EAX, CPUID_7_1_EAX_FRED }, + .to = { FEAT_7_1_EAX, CPUID_7_1_EAX_WRMSRNS }, + },
since FRED relies on LKGS and WRMSRNS, here should be as follow:
+ {
+ .from = { FEAT_7_1_EAX, CPUID_7_1_EAX_LKGS },
+ .to = { FEAT_7_1_EAX, CPUID_7_1_EAX_FRED },
+ },
+ {
+ .from = { FEAT_7_1_EAX, CPUID_7_1_EAX_WRMSRNS },
+ .to = { FEAT_7_1_EAX, CPUID_7_1_EAX_FRED },+ },
Quoted 20 lines
target/i386/cpu.c +0
-0}; typedef struct X86RegisterInfo32 {target/i386/cpu.h +6
-0@@ -934,6 +934,12 @@ uint64_t x86_cpu_get_supported_feature_word(FeatureWord w,#define CPUID_7_1_EDX_AMX_COMPLEX (1U << 8)/* PREFETCHIT0/1 Instructions */ #define CPUID_7_1_EDX_PREFETCHITI (1U << 14) +/* Flexible return and event delivery (FRED) */ +#define CPUID_7_1_EAX_FRED (1U << 17) +/* Load into IA32_KERNEL_GS_BASE (LKGS) */ +#define CPUID_7_1_EAX_LKGS (1U << 18) +/* Non-Serializing Write to Model Specific Register (WRMSRNS) */ +#define CPUID_7_1_EAX_WRMSRNS (1U << 19)/* Do not exhibit MXCSR Configuration Dependent Timing (MCDT) behavior */ #define CPUID_7_2_EDX_MCDT_NO (1U << 5)
RE: [PATCH v3 1/6] target/i386: add support for FRED in CPUID enumeration
Quoted 6 lines
Quoted 6 lines
target/i386/cpu.c +2
-0@@ -1552,6 +1552,14 @@ static FeatureDep feature_dependencies[] = { .from = { FEAT_VMX_SECONDARY_CTLS, VMX_SECONDARY_EXEC_ENABLE_USER_WAIT_PAUSE }, .to = { FEAT_7_0_ECX, CPUID_7_0_ECX_WAITPKG }, }, + { + .from = { FEAT_7_1_EAX, CPUID_7_1_EAX_FRED },
Quoted 9 lines
target/i386/cpu.c static FeatureDep feature_dependencies[] = { · line 1562
Quoted 6 lines
target/i386/cpu.c +6
-0+ .to = { FEAT_7_1_EAX, CPUID_7_1_EAX_LKGS }, + }, + { + .from = { FEAT_7_1_EAX, CPUID_7_1_EAX_FRED }, + .to = { FEAT_7_1_EAX, CPUID_7_1_EAX_WRMSRNS }, + },since FRED relies on LKGS and WRMSRNS, here should be as follow:
You're right, I messed it up in this version.
Thank!
Xin
Quoted 8 lines
target/i386/cpu.c +8
-0+ { + .from = { FEAT_7_1_EAX, CPUID_7_1_EAX_LKGS }, + .to = { FEAT_7_1_EAX, CPUID_7_1_EAX_FRED }, + }, + { + .from = { FEAT_7_1_EAX, CPUID_7_1_EAX_WRMSRNS }, + .to = { FEAT_7_1_EAX, CPUID_7_1_EAX_FRED }, + },
PATCH v3 1/6 target/i386: add support for FRED in CPUID enumeration
FRED, i.e., the Intel flexible return and event delivery architecture, defines simple new transitions that change privilege level (ring transitions).
The new transitions defined by the FRED architecture are FRED event delivery and, for returning from events, two FRED return instructions. FRED event delivery can effect a transition from ring 3 to ring 0, but it is used also to deliver events incident to ring 0. One FRED instruction (ERETU) effects a return from ring 0 to ring 3, while the other (ERETS) returns while remaining in ring 0. Collectively, FRED event delivery and the FRED return instructions are FRED transitions.
In addition to these transitions, the FRED architecture defines a new instruction (LKGS) for managing the state of the GS segment register. The LKGS instruction can be used by 64-bit operating systems that do not use the new FRED transitions.
WRMSRNS is an instruction that behaves exactly like WRMSR, with the only difference being that it is not a serializing instruction by default. Under certain conditions, WRMSRNS may replace WRMSR to improve performance. FRED uses it to switch RSP0 in a faster manner.
Search for the latest FRED spec in most search engines with this search pattern:
site:intel.com FRED (flexible return and event delivery) specification
The CPUID feature flag CPUID.(EAX=7,ECX=1):EAX[17] enumerates FRED, and the CPUID feature flag CPUID.(EAX=7,ECX=1):EAX[18] enumerates LKGS, and the CPUID feature flag CPUID.(EAX=7,ECX=1):EAX[19] enumerates WRMSRNS.
Add CPUID definitions for FRED/LKGS/WRMSRNS, and expose them to KVM guests.
Because FRED relies on LKGS and WRMSRNS, add that to feature dependency map.
---
Changelog
v3A:
- Fix reversed dependency (Wu Dan1).
--- target/i386/cpu.c | 10 +++++++++- target/i386/cpu.h | 6 ++++++ 2 files changed, 15 insertions(+), 1 deletion(-)
target/i386/cpu.c +9 -1
@@ -965,7 +965,7 @@ FeatureWordInfo feature_word_info[FEATURE_WORDS] = {"avx-vnni" ,"avx512-bf16" ,NULL ,"cmpccxadd" ,NULL ,NULL ,"fzrm" ,"fsrs" ,"fsrc" ,NULL ,NULL ,NULL , -NULL ,NULL ,NULL ,NULL , +NULL ,"fred" ,"lkgs" ,"wrmsrns" ,NULL ,"amx-fp16" ,NULL ,"avx-ifma" ,NULL ,NULL ,NULL ,NULL , NULL, NULL, NULL, NULL,
@@ -1552,6 +1552,14 @@ static FeatureDep feature_dependencies[] = { .from = { FEAT_VMX_SECONDARY_CTLS, VMX_SECONDARY_EXEC_ENABLE_USER_WAIT_PAUSE }, .to = { FEAT_7_0_ECX, CPUID_7_0_ECX_WAITPKG }, }, + { + .from = { FEAT_7_1_EAX, CPUID_7_1_EAX_LKGS }, + .to = { FEAT_7_1_EAX, CPUID_7_1_EAX_FRED }, + }, + { + .from = { FEAT_7_1_EAX, CPUID_7_1_EAX_WRMSRNS }, + .to = { FEAT_7_1_EAX, CPUID_7_1_EAX_FRED }, + }, }; typedef struct X86RegisterInfo32 {
target/i386/cpu.h +6 -0
@@ -934,6 +934,12 @@ uint64_t x86_cpu_get_supported_feature_word(FeatureWord w,#define CPUID_7_1_EDX_AMX_COMPLEX (1U << 8)/* PREFETCHIT0/1 Instructions */ #define CPUID_7_1_EDX_PREFETCHITI (1U << 14) +/* Flexible return and event delivery (FRED) */ +#define CPUID_7_1_EAX_FRED (1U << 17) +/* Load into IA32_KERNEL_GS_BASE (LKGS) */ +#define CPUID_7_1_EAX_LKGS (1U << 18) +/* Non-Serializing Write to Model Specific Register (WRMSRNS) */ +#define CPUID_7_1_EAX_WRMSRNS (1U << 19)/* Do not exhibit MXCSR Configuration Dependent Timing (MCDT) behavior */ #define CPUID_7_2_EDX_MCDT_NO (1U << 5)
Signature · 1 line
2.43.0
Re: [PATCH v3A 1/6] target/i386: add support for FRED in CPUID enumeration
Quoted 37 lines · Xin Li
Date: Thu, 21 Dec 2023 19:03:36 -0800
From: Xin Li <[email protected]>
Subject: [PATCH v3A 1/6] target/i386: add support for FRED in CPUID
enumeration
X-Mailer: git-send-email 2.43.0FRED, i.e., the Intel flexible return and event delivery architecture, defines simple new transitions that change privilege level (ring transitions).
The new transitions defined by the FRED architecture are FRED event delivery and, for returning from events, two FRED return instructions. FRED event delivery can effect a transition from ring 3 to ring 0, but it is used also to deliver events incident to ring 0. One FRED instruction (ERETU) effects a return from ring 0 to ring 3, while the other (ERETS) returns while remaining in ring 0. Collectively, FRED event delivery and the FRED return instructions are FRED transitions.
In addition to these transitions, the FRED architecture defines a new instruction (LKGS) for managing the state of the GS segment register. The LKGS instruction can be used by 64-bit operating systems that do not use the new FRED transitions.
WRMSRNS is an instruction that behaves exactly like WRMSR, with the only difference being that it is not a serializing instruction by default. Under certain conditions, WRMSRNS may replace WRMSR to improve performance. FRED uses it to switch RSP0 in a faster manner.
Search for the latest FRED spec in most search engines with this search pattern:
site:intel.com FRED (flexible return and event delivery) specification
The CPUID feature flag CPUID.(EAX=7,ECX=1):EAX[17] enumerates FRED, and the CPUID feature flag CPUID.(EAX=7,ECX=1):EAX[18] enumerates LKGS, and the CPUID feature flag CPUID.(EAX=7,ECX=1):EAX[19] enumerates WRMSRNS.
Quoted 8 lines
Add CPUID definitions for FRED/LKGS/WRMSRNS, and expose them to KVM guests.
Because FRED relies on LKGS and WRMSRNS, add that to feature dependency map.
Tested-by Shan Kang <[email protected]>Signed-off-by Xin Li <[email protected]>---
Quoted 58 lines
Changelog
v3A:
- Fix reversed dependency (Wu Dan1).--- target/i386/cpu.c | 10 +++++++++- target/i386/cpu.h | 6 ++++++ 2 files changed, 15 insertions(+), 1 deletion(-)target/i386/cpu.c +9
-1@@ -965,7 +965,7 @@ FeatureWordInfo feature_word_info[FEATURE_WORDS] = {"avx-vnni" ,"avx512-bf16" ,NULL ,"cmpccxadd" ,NULL ,NULL ,"fzrm" ,"fsrs" ,"fsrc" ,NULL ,NULL ,NULL , -NULL ,NULL ,NULL ,NULL , +NULL ,"fred" ,"lkgs" ,"wrmsrns" ,NULL ,"amx-fp16" ,NULL ,"avx-ifma" ,NULL ,NULL ,NULL ,NULL , NULL, NULL, NULL, NULL,@@ -1552,6 +1552,14 @@ static FeatureDep feature_dependencies[] = { .from = { FEAT_VMX_SECONDARY_CTLS, VMX_SECONDARY_EXEC_ENABLE_USER_WAIT_PAUSE }, .to = { FEAT_7_0_ECX, CPUID_7_0_ECX_WAITPKG }, }, + { + .from = { FEAT_7_1_EAX, CPUID_7_1_EAX_LKGS }, + .to = { FEAT_7_1_EAX, CPUID_7_1_EAX_FRED }, + }, + { + .from = { FEAT_7_1_EAX, CPUID_7_1_EAX_WRMSRNS }, + .to = { FEAT_7_1_EAX, CPUID_7_1_EAX_FRED }, + }, }; typedef struct X86RegisterInfo32 {target/i386/cpu.h +6
-0@@ -934,6 +934,12 @@ uint64_t x86_cpu_get_supported_feature_word(FeatureWord w,#define CPUID_7_1_EDX_AMX_COMPLEX (1U << 8)/* PREFETCHIT0/1 Instructions */ #define CPUID_7_1_EDX_PREFETCHITI (1U << 14) +/* Flexible return and event delivery (FRED) */ +#define CPUID_7_1_EAX_FRED (1U << 17) +/* Load into IA32_KERNEL_GS_BASE (LKGS) */ +#define CPUID_7_1_EAX_LKGS (1U << 18) +/* Non-Serializing Write to Model Specific Register (WRMSRNS) */ +#define CPUID_7_1_EAX_WRMSRNS (1U << 19)/* Do not exhibit MXCSR Configuration Dependent Timing (MCDT) behavior */ #define CPUID_7_2_EDX_MCDT_NO (1U << 5)Signature · 1 line
2.43.0
Re: [PATCH v3A 1/6] target/i386: add support for FRED in CPUID enumeration
Quoted 80 lines · Zhao Liu
Date: Fri, 22 Dec 2023 15:34:02 +0800
From: Zhao Liu <[email protected]>
Subject: Re: [PATCH v3A 1/6] target/i386: add support for FRED in CPUID
enumerationQuoted 37 lines · Xin Li
Date: Thu, 21 Dec 2023 19:03:36 -0800
From: Xin Li <[email protected]>
Subject: [PATCH v3A 1/6] target/i386: add support for FRED in CPUID
enumeration
X-Mailer: git-send-email 2.43.0FRED, i.e., the Intel flexible return and event delivery architecture, defines simple new transitions that change privilege level (ring transitions).
The new transitions defined by the FRED architecture are FRED event delivery and, for returning from events, two FRED return instructions. FRED event delivery can effect a transition from ring 3 to ring 0, but it is used also to deliver events incident to ring 0. One FRED instruction (ERETU) effects a return from ring 0 to ring 3, while the other (ERETS) returns while remaining in ring 0. Collectively, FRED event delivery and the FRED return instructions are FRED transitions.
In addition to these transitions, the FRED architecture defines a new instruction (LKGS) for managing the state of the GS segment register. The LKGS instruction can be used by 64-bit operating systems that do not use the new FRED transitions.
WRMSRNS is an instruction that behaves exactly like WRMSR, with the only difference being that it is not a serializing instruction by default. Under certain conditions, WRMSRNS may replace WRMSR to improve performance. FRED uses it to switch RSP0 in a faster manner.
Search for the latest FRED spec in most search engines with this search pattern:
site:intel.com FRED (flexible return and event delivery) specification
The CPUID feature flag CPUID.(EAX=7,ECX=1):EAX[17] enumerates FRED, and the CPUID feature flag CPUID.(EAX=7,ECX=1):EAX[18] enumerates LKGS, and the CPUID feature flag CPUID.(EAX=7,ECX=1):EAX[19] enumerates WRMSRNS.
Quoted 8 lines
Add CPUID definitions for FRED/LKGS/WRMSRNS, and expose them to KVM guests.
Because FRED relies on LKGS and WRMSRNS, add that to feature dependency map.
Tested-by Shan Kang <[email protected]>Signed-off-by Xin Li <[email protected]>---
Reviewed-by Zhao Liu <[email protected]>Quoted 26 lines
Changelog
v3A:
- Fix reversed dependency (Wu Dan1).--- target/i386/cpu.c | 10 +++++++++- target/i386/cpu.h | 6 ++++++ 2 files changed, 15 insertions(+), 1 deletion(-)target/i386/cpu.c +1
-1@@ -965,7 +965,7 @@ FeatureWordInfo feature_word_info[FEATURE_WORDS] = {"avx-vnni" ,"avx512-bf16" ,NULL ,"cmpccxadd" ,NULL ,NULL ,"fzrm" ,"fsrs" ,"fsrc" ,NULL ,NULL ,NULL , -NULL ,NULL ,NULL ,NULL , +NULL ,"fred" ,"lkgs" ,"wrmsrns" ,NULL ,"amx-fp16" ,NULL ,"avx-ifma" ,NULL ,NULL ,NULL ,NULL , NULL, NULL, NULL, NULL,@@ -1552,6 +1552,14 @@ static FeatureDep feature_dependencies[] = { .from = { FEAT_VMX_SECONDARY_CTLS, VMX_SECONDARY_EXEC_ENABLE_USER_WAIT_PAUSE }, .to = { FEAT_7_0_ECX, CPUID_7_0_ECX_WAITPKG }, },
Quoted 8 lines
target/i386/cpu.c static FeatureDep feature_dependencies[] = { · line 1562
Quoted 8 lines
target/i386/cpu.c +8
-0+ { + .from = { FEAT_7_1_EAX, CPUID_7_1_EAX_LKGS }, + .to = { FEAT_7_1_EAX, CPUID_7_1_EAX_FRED }, + }, + { + .from = { FEAT_7_1_EAX, CPUID_7_1_EAX_WRMSRNS }, + .to = { FEAT_7_1_EAX, CPUID_7_1_EAX_FRED }, + },
Oh, sorry, one thing that comes to mind, is this dependency required? Since the FRED spec (v3.0) is all about WRMSR as the example, without mentioning WRMSRNS, could there be other implementations that depend on WRMSR instead of WRMSRNS?
The dependencies of LKGS are clearly defined in spec.
-Zhao
Quoted 25 lines
Quoted 24 lines
target/i386/cpu.c +0
-0}; typedef struct X86RegisterInfo32 {target/i386/cpu.h +6
-0@@ -934,6 +934,12 @@ uint64_t x86_cpu_get_supported_feature_word(FeatureWord w,#define CPUID_7_1_EDX_AMX_COMPLEX (1U << 8)/* PREFETCHIT0/1 Instructions */ #define CPUID_7_1_EDX_PREFETCHITI (1U << 14) +/* Flexible return and event delivery (FRED) */ +#define CPUID_7_1_EAX_FRED (1U << 17) +/* Load into IA32_KERNEL_GS_BASE (LKGS) */ +#define CPUID_7_1_EAX_LKGS (1U << 18) +/* Non-Serializing Write to Model Specific Register (WRMSRNS) */ +#define CPUID_7_1_EAX_WRMSRNS (1U << 19)/* Do not exhibit MXCSR Configuration Dependent Timing (MCDT) behavior */ #define CPUID_7_2_EDX_MCDT_NO (1U << 5)Signature · 1 line
2.43.0
RE: [PATCH v3A 1/6] target/i386: add support for FRED in CPUID enumeration
Quoted 16 lines
Quoted 4 lines
Quoted 4 lines
NULL, NULL, NULL, NULL, @@ -1552,6 +1552,14 @@ staticFeatureDep feature_dependencies[] = {
.from = {FEAT_VMX_SECONDARY_CTLS,
VMX_SECONDARY_EXEC_ENABLE_USER_WAIT_PAUSE},Quoted 10 lines
target/i386/cpu.c static FeatureDep feature_dependencies[] = { · line 1562
Quoted 10 lines
.to = { FEAT_7_0_ECX, CPUID_7_0_ECX_WAITPKG }, },target/i386/cpu.c +8
-0+ { + .from = { FEAT_7_1_EAX, CPUID_7_1_EAX_LKGS }, + .to = { FEAT_7_1_EAX, CPUID_7_1_EAX_FRED }, + }, + { + .from = { FEAT_7_1_EAX, CPUID_7_1_EAX_WRMSRNS }, + .to = { FEAT_7_1_EAX, CPUID_7_1_EAX_FRED }, + },
Quoted 4 lines
Oh, sorry, one thing that comes to mind, is this dependency required? Since the FRED spec (v3.0) is all about WRMSR as the example, without mentioning WRMSRNS, could there be other implementations that depend on WRMSR instead of WRMSRNS?
This is a community ask from tglx:
https://lkml.kernel.org/kvm/87y1h81ht4.ffs@tglx/
Boris had the same question:
https://lore.kernel.org/lkml/20231114050201.GAZVL%2FSd%2FyLIdON9la@fat_crate.local/
But it needs to go through a formal approach, which takes time, to reach the FRED public spec.
Thanks!
Xin
Re: [PATCH v3A 1/6] target/i386: add support for FRED in CPUID enumeration
Quoted 26 lines · On Fri, Dec 22, 2023 at 08:24:52AM +0000, Li, Xin3 wrote:
Date: Fri, 22 Dec 2023 08:24:52 +0000
From: "Li, Xin3" <[email protected]>
Subject: RE: [PATCH v3A 1/6] target/i386: add support for FRED in CPUID
enumerationQuoted 15 lines
Quoted 3 lines
Quoted 3 lines
NULL, NULL, NULL, NULL, @@ -1552,6 +1552,14 @@ staticFeatureDep feature_dependencies[] = {
.from = {FEAT_VMX_SECONDARY_CTLS,
VMX_SECONDARY_EXEC_ENABLE_USER_WAIT_PAUSE},Quoted 10 lines
target/i386/cpu.c static FeatureDep feature_dependencies[] = { · line 1562
Quoted 10 lines
.to = { FEAT_7_0_ECX, CPUID_7_0_ECX_WAITPKG }, },target/i386/cpu.c +8
-0+ { + .from = { FEAT_7_1_EAX, CPUID_7_1_EAX_LKGS }, + .to = { FEAT_7_1_EAX, CPUID_7_1_EAX_FRED }, + }, + { + .from = { FEAT_7_1_EAX, CPUID_7_1_EAX_WRMSRNS }, + .to = { FEAT_7_1_EAX, CPUID_7_1_EAX_FRED }, + },Quoted 4 lines
Oh, sorry, one thing that comes to mind, is this dependency required? Since the FRED spec (v3.0) is all about WRMSR as the example, without mentioning WRMSRNS, could there be other implementations that depend on WRMSR instead of WRMSRNS?
Quoted 9 lines
This is a community ask from tglx:
https://lkml.kernel.org/kvm/87y1h81ht4.ffs@tglx/Boris had the same question:
https://lore.kernel.org/lkml/20231114050201.GAZVL%2FSd%2FyLIdON9la@fat_crate.local/But it needs to go through a formal approach, which takes time, to reach the FRED public spec.
Thanks Xin! You can add a simple note in the commit message, such as FRED's dependency on WRMSRNS will be documented, to avoid confusion for later reviewers interested in FRED.
Regards,
Zhao
PATCH v3 2/6 target/i386: mark CR4.FRED not reserved
In reply to Xin LiThe CR4.FRED bit, i.e., CR4[32], is no longer a reserved bit when FRED is exposed to guests, otherwise it is still a reserved bit.
--- target/i386/cpu.h | 12 +++++++++++- 1 file changed, 11 insertions(+), 1 deletion(-)
target/i386/cpu.h +11 -1
@@ -262,6 +262,12 @@ typedef enum X86Seg {#define CR4_PKE_MASK (1U << 22)#define CR4_PKS_MASK (1U << 24) +#ifdef TARGET_X86_64 +#define CR4_FRED_MASK (1ULL << 32) +#else +#define CR4_FRED_MASK 0 +#endif +#define CR4_RESERVED_MASK \ (~(target_ulong)(CR4_VME_MASK | CR4_PVI_MASK | CR4_TSD_MASK \ | CR4_DE_MASK | CR4_PSE_MASK | CR4_PAE_MASK \
@@ -269,7 +275,8 @@ typedef enum X86Seg { | CR4_OSFXSR_MASK | CR4_OSXMMEXCPT_MASK | CR4_UMIP_MASK \ | CR4_LA57_MASK \ | CR4_FSGSBASE_MASK | CR4_PCIDE_MASK | CR4_OSXSAVE_MASK \ - | CR4_SMEP_MASK | CR4_SMAP_MASK | CR4_PKE_MASK | CR4_PKS_MASK)) + | CR4_SMEP_MASK | CR4_SMAP_MASK | CR4_PKE_MASK | CR4_PKS_MASK \ + | CR4_FRED_MASK))#define DR6_BD (1 << 13)#define DR6_BS (1 << 14)
@@ -2520,6 +2527,9 @@ static inline uint64_t cr4_reserved_bits(CPUX86State *env)if (!(env->features[FEAT_7_0_ECX ] &CPUID_7_0_ECX_PKS )) { reserved_bits |=CR4_PKS_MASK ; } +if (!(env->features[FEAT_7_1_EAX ] &CPUID_7_1_EAX_FRED )) { + reserved_bits |=CR4_FRED_MASK ; + }return reserved_bits; }
Signature · 1 line
2.42.0
Re: [PATCH v3 2/6] target/i386: mark CR4.FRED not reserved
Quoted 11 lines · Xin Li
Date: Wed, 8 Nov 2023 23:20:08 -0800
From: Xin Li <[email protected]>
Subject: [PATCH v3 2/6] target/i386: mark CR4.FRED not reserved
X-Mailer: git-send-email 2.42.0The CR4.FRED bit, i.e., CR4[32], is no longer a reserved bit when FRED is exposed to guests, otherwise it is still a reserved bit.
Tested-by Shan Kang <[email protected]>Signed-off-by Xin Li <[email protected]>---
Quoted 44 lines
target/i386/cpu.h | 12 +++++++++++- 1 file changed, 11 insertions(+), 1 deletion(-)target/i386/cpu.h +11
-1@@ -262,6 +262,12 @@ typedef enum X86Seg {#define CR4_PKE_MASK (1U << 22)#define CR4_PKS_MASK (1U << 24) +#ifdef TARGET_X86_64 +#define CR4_FRED_MASK (1ULL << 32) +#else +#define CR4_FRED_MASK 0 +#endif +#define CR4_RESERVED_MASK \ (~(target_ulong)(CR4_VME_MASK | CR4_PVI_MASK | CR4_TSD_MASK \ | CR4_DE_MASK | CR4_PSE_MASK | CR4_PAE_MASK \@@ -269,7 +275,8 @@ typedef enum X86Seg { | CR4_OSFXSR_MASK | CR4_OSXMMEXCPT_MASK | CR4_UMIP_MASK \ | CR4_LA57_MASK \ | CR4_FSGSBASE_MASK | CR4_PCIDE_MASK | CR4_OSXSAVE_MASK \ - | CR4_SMEP_MASK | CR4_SMAP_MASK | CR4_PKE_MASK | CR4_PKS_MASK)) + | CR4_SMEP_MASK | CR4_SMAP_MASK | CR4_PKE_MASK | CR4_PKS_MASK \ + | CR4_FRED_MASK))#define DR6_BD (1 << 13)#define DR6_BS (1 << 14)@@ -2520,6 +2527,9 @@ static inline uint64_t cr4_reserved_bits(CPUX86State *env)if (!(env->features[FEAT_7_0_ECX ] &CPUID_7_0_ECX_PKS )) { reserved_bits |=CR4_PKS_MASK ; } +if (!(env->features[FEAT_7_1_EAX ] &CPUID_7_1_EAX_FRED )) { + reserved_bits |=CR4_FRED_MASK ; + }return reserved_bits; }Signature · 1 line
2.42.0
PATCH v3 3/6 target/i386: add the secondary VM exit controls MSR
In reply to Xin LiAdd the secondary VM exit controls MSR to prepare for KVM FRED enabling.
--- scripts/kvm/vmxcap | 9 +++++++++ target/i386/cpu.c | 2 +- target/i386/cpu.h | 1 + 3 files changed, 11 insertions(+), 1 deletion(-)
scripts/kvm/vmxcap +9 -0
@@ -24,6 +24,7 @@ MSR_IA32_VMX_TRUE_EXIT_CTLS = 0x48F MSR_IA32_VMX_TRUE_ENTRY_CTLS = 0x490 MSR_IA32_VMX_VMFUNC = 0x491 MSR_IA32_VMX_PROCBASED_CTLS3 = 0x492 +MSR_IA32_VMX_EXIT_CTLS2 = 0x493 class msr(object): def __init__(self):
@@ -219,11 +220,19 @@ controls = [ 23: 'Clear IA32_BNDCFGS', 24: 'Conceal VM exits from PT', 25: 'Clear IA32_RTIT_CTL', + 31: 'Activate secondary VM-exit controls', }, cap_msr = MSR_IA32_VMX_EXIT_CTLS, true_cap_msr = MSR_IA32_VMX_TRUE_EXIT_CTLS, ), + Allowed1Control( + name = 'secondary VM-Exit controls', + bits = { + }, + cap_msr = MSR_IA32_VMX_EXIT_CTLS2, + ), + Control( name = 'VM-Entry controls', bits = {
target/i386/cpu.c +1 -1
@@ -1270,7 +1270,7 @@ FeatureWordInfo feature_word_info[FEATURE_WORDS] = {"vmx-exit-save-efer" ,"vmx-exit-load-efer" ,"vmx-exit-save-preemption-timer" ,"vmx-exit-clear-bndcfgs" ,NULL ,"vmx-exit-clear-rtit-ctl" ,NULL ,NULL , - NULL, "vmx-exit-load-pkrs", NULL, NULL, + NULL, "vmx-exit-load-pkrs", NULL, "vmx-exit-secondary-ctls", }, .msr = { .index = MSR_IA32_VMX_TRUE_EXIT_CTLS,
target/i386/cpu.h +1 -0
@@ -551,6 +551,7 @@ typedef enum X86Seg {#define MSR_IA32_VMX_TRUE_EXIT_CTLS 0x0000048f#define MSR_IA32_VMX_TRUE_ENTRY_CTLS 0x00000490#define MSR_IA32_VMX_VMFUNC 0x00000491 +#define MSR_IA32_VMX_EXIT_CTLS2 0x00000493#define XSTATE_FP_BIT 0#define XSTATE_SSE_BIT 1
Signature · 1 line
2.42.0
PATCH v3 4/6 target/i386: add support for VMX FRED controls
In reply to Xin LiAdd VMX FRED controls used to enable save/load of FRED MSRs.
--- scripts/kvm/vmxcap | 3 +++ target/i386/cpu.c | 2 +- 2 files changed, 4 insertions(+), 1 deletion(-)
scripts/kvm/vmxcap +3 -0
@@ -229,6 +229,8 @@ controls = [ Allowed1Control( name = 'secondary VM-Exit controls', bits = { + 0: 'Save IA32 FRED MSRs', + 1: 'Load IA32 FRED MSRs', }, cap_msr = MSR_IA32_VMX_EXIT_CTLS2, ),
@@ -246,6 +248,7 @@ controls = [ 16: 'Load IA32_BNDCFGS', 17: 'Conceal VM entries from PT', 18: 'Load IA32_RTIT_CTL', + 23: 'Load IA32 FRED MSRs', }, cap_msr = MSR_IA32_VMX_ENTRY_CTLS, true_cap_msr = MSR_IA32_VMX_TRUE_ENTRY_CTLS,
target/i386/cpu.c +1 -1
@@ -1285,7 +1285,7 @@ FeatureWordInfo feature_word_info[FEATURE_WORDS] = {NULL ,"vmx-entry-ia32e-mode" ,NULL ,NULL ,NULL ,"vmx-entry-load-perf-global-ctrl" ,"vmx-entry-load-pat" ,"vmx-entry-load-efer" ,"vmx-entry-load-bndcfgs" ,NULL ,"vmx-entry-load-rtit-ctl" ,NULL , -NULL ,NULL ,"vmx-entry-load-pkrs" ,NULL , +NULL ,NULL ,"vmx-entry-load-pkrs" ,"vmx-entry-load-fred" ,NULL ,NULL ,NULL ,NULL , NULL, NULL, NULL, NULL, },
Signature · 1 line
2.42.0
Re: [PATCH v3 4/6] target/i386: add support for VMX FRED controls
Quoted 35 lines · Xin Li
Add VMX FRED controls used to enable save/load of FRED MSRs.
Tested-by Shan Kang <[email protected]>Signed-off-by Xin Li <[email protected]>--- scripts/kvm/vmxcap | 3 +++ target/i386/cpu.c | 2 +- 2 files changed, 4 insertions(+), 1 deletion(-)scripts/kvm/vmxcap +3
-0@@ -229,6 +229,8 @@ controls = [ Allowed1Control( name = 'secondary VM-Exit controls', bits = { + 0: 'Save IA32 FRED MSRs', + 1: 'Load IA32 FRED MSRs', }, cap_msr = MSR_IA32_VMX_EXIT_CTLS2, ),@@ -246,6 +248,7 @@ controls = [ 16: 'Load IA32_BNDCFGS', 17: 'Conceal VM entries from PT', 18: 'Load IA32_RTIT_CTL', + 23: 'Load IA32 FRED MSRs', }, cap_msr = MSR_IA32_VMX_ENTRY_CTLS, true_cap_msr = MSR_IA32_VMX_TRUE_ENTRY_CTLS,target/i386/cpu.c +0
-0
target/i386/cpu.c FeatureWordInfo feature_word_info[FEATURE_WORDS] = { · line 1291
Quoted 8 lines
target/i386/cpu.c +1
-1@@ -1285,7 +1285,7 @@ FeatureWordInfo feature_word_info[FEATURE_WORDS] = {NULL ,"vmx-entry-ia32e-mode" ,NULL ,NULL ,NULL ,"vmx-entry-load-perf-global-ctrl" ,"vmx-entry-load-pat" ,"vmx-entry-load-efer" ,"vmx-entry-load-bndcfgs" ,NULL ,"vmx-entry-load-rtit-ctl" ,NULL , -NULL ,NULL ,"vmx-entry-load-pkrs" ,NULL , +NULL ,NULL ,"vmx-entry-load-pkrs" ,"vmx-entry-load-fred" ,NULL ,NULL ,NULL ,NULL , NULL, NULL, NULL, NULL, },
The bits in the secondary vmexit controls are not supported, and in general the same is true for the secondary vmexit case. I think it's better to not include the vmx-entry-load-fred bit either, and only do the vmxcap changes.
Also, in patch 1 there should be a dependency from LM to FRED.
I applied these changes and queued the series, thanks.
Paolo
RE: [PATCH v3 4/6] target/i386: add support for VMX FRED controls
Quoted 3 lines
The bits in the secondary vmexit controls are not supported, and in general the same is true for the secondary vmexit case. I think it's better to not include the vmx-entry- load-fred bit either, and only do the vmxcap changes.
Right, we don't need it at all.
Quoted 2 lines
Also, in patch 1 there should be a dependency from LM to FRED.
Make sense.
I really hope we can simplify the hundreds, if not thousands, of x86 CPU features with just a few CPU feature baselines.
Quoted 2 lines
I applied these changes and queued the series, thanks.
A great news to me!
Thanks a lot!
Xin
PATCH v3 5/6 target/i386: enumerate VMX nested-exception support
In reply to Xin LiAllow VMX nested-exception support to be exposed in KVM guests, thus nested KVM guests can enumerate it.
--- scripts/kvm/vmxcap | 1 + target/i386/cpu.c | 1 + target/i386/cpu.h | 1 + 3 files changed, 3 insertions(+)
scripts/kvm/vmxcap +1 -0
@@ -117,6 +117,7 @@ controls = [ 54: 'INS/OUTS instruction information', 55: 'IA32_VMX_TRUE_*_CTLS support', 56: 'Skip checks on event error code', + 58: 'VMX nested exception support', }, msr = MSR_IA32_VMX_BASIC, ),
target/i386/cpu.c +1 -0
@@ -1343,6 +1343,7 @@ FeatureWordInfo feature_word_info[FEATURE_WORDS] = { [54] = "vmx-ins-outs", [55] = "vmx-true-ctls", [56] = "vmx-any-errcode", + [58 ] ="vmx-nested-exception" , }, .msr = { .index =MSR_IA32_VMX_BASIC ,
target/i386/cpu.h +1 -0
@@ -1054,6 +1054,7 @@ uint64_t x86_cpu_get_supported_feature_word(FeatureWord w,#define MSR_VMX_BASIC_INS_OUTS (1ULL << 54)#define MSR_VMX_BASIC_TRUE_CTLS (1ULL << 55)#define MSR_VMX_BASIC_ANY_ERRCODE (1ULL << 56) +#define MSR_VMX_BASIC_NESTED_EXCEPTION (1ULL << 58)#define MSR_VMX_MISC_PREEMPTION_TIMER_SHIFT_MASK 0x1Full#define MSR_VMX_MISC_STORE_LMA (1ULL << 5)
Signature · 1 line
2.42.0
PATCH v3 6/6 target/i386: Add get/set/migrate support for FRED MSRs
In reply to Xin LiFRED CPU states are managed in 9 new FRED MSRs, in addtion to a few existing CPU registers and MSRs, e.g., CR4.FRED and MSR_IA32_PL0_SSP.
Save/restore/migrate FRED MSRs if FRED is exposed to the guest.
--- target/i386/cpu.h | 22 +++++++++++++++++++ target/i386/kvm/kvm.c | 49 +++++++++++++++++++++++++++++++++++++++++++ target/i386/machine.c | 28 +++++++++++++++++++++++++ 3 files changed, 99 insertions(+)
target/i386/cpu.h +22 -0
@@ -529,6 +529,17 @@ typedef enum X86Seg {#define MSR_IA32_XFD 0x000001c4#define MSR_IA32_XFD_ERR 0x000001c5 +/* FRED MSRs */ +#define MSR_IA32_FRED_RSP0 0x000001cc/* Stack level 0 regular stack pointer */ +#define MSR_IA32_FRED_RSP1 0x000001cd/* Stack level 1 regular stack pointer */ +#define MSR_IA32_FRED_RSP2 0x000001ce/* Stack level 2 regular stack pointer */ +#define MSR_IA32_FRED_RSP3 0x000001cf/* Stack level 3 regular stack pointer */ +#define MSR_IA32_FRED_STKLVLS 0x000001d0/* FRED exception stack levels */ +#define MSR_IA32_FRED_SSP1 0x000001d1/* Stack level 1 shadow stack pointer in ring 0 */ +#define MSR_IA32_FRED_SSP2 0x000001d2/* Stack level 2 shadow stack pointer in ring 0 */ +#define MSR_IA32_FRED_SSP3 0x000001d3/* Stack level 3 shadow stack pointer in ring 0 */ +#define MSR_IA32_FRED_CONFIG 0x000001d4/* FRED Entrypoint and interrupt stack level */ +#define MSR_IA32_BNDCFGS 0x00000d90#define MSR_IA32_XSS 0x00000da0#define MSR_IA32_UMWAIT_CONTROL 0xe1
@@ -1687,6 +1698,17 @@ typedef struct CPUArchState {target_ulong cstar;target_ulong fmask;target_ulong kernelgsbase; + +/* FRED MSRs */ +uint64_t fred_rsp0; +uint64_t fred_rsp1; +uint64_t fred_rsp2; +uint64_t fred_rsp3; +uint64_t fred_stklvls; +uint64_t fred_ssp1; +uint64_t fred_ssp2; +uint64_t fred_ssp3; +uint64_t fred_config;#endif uint64_t tsc_adjust;
target/i386/kvm/kvm.c +49 -0
67 lines not on this page · raw message
target/i386/machine.c +28 -0
@@ -1544,6 +1544,33 @@ static const VMStateDescription vmstate_msr_xfd = { };#ifdef TARGET_X86_64 +static bool intel_fred_msrs_needed (void *opaque) +{ +X86CPU *cpu = opaque; +CPUX86State *env = &cpu->env; + +return !!(env->features[FEAT_7_1_EAX ] &CPUID_7_1_EAX_FRED ); +} + +static const VMStateDescription vmstate_msr_fred = { + .name ="cpu/fred" , + .version_id =1 , + .minimum_version_id =1 , + .needed = intel_fred_msrs_needed, + .fields = (VMStateField []) { +VMSTATE_UINT64 (env.fred_rsp0,X86CPU ), +VMSTATE_UINT64 (env.fred_rsp1,X86CPU ), +VMSTATE_UINT64 (env.fred_rsp2,X86CPU ), +VMSTATE_UINT64 (env.fred_rsp3,X86CPU ), +VMSTATE_UINT64 (env.fred_stklvls,X86CPU ), +VMSTATE_UINT64 (env.fred_ssp1,X86CPU ), +VMSTATE_UINT64 (env.fred_ssp2,X86CPU ), +VMSTATE_UINT64 (env.fred_ssp3,X86CPU ), +VMSTATE_UINT64 (env.fred_config,X86CPU ), +VMSTATE_END_OF_LIST () + } + }; +static bool amx_xtile_needed (void *opaque) {X86CPU *cpu = opaque;
@@ -1747,6 +1774,7 @@ const VMStateDescription vmstate_x86_cpu = { &vmstate_pdptrs, &vmstate_msr_xfd, #ifdef TARGET_X86_64 + &vmstate_msr_fred, &vmstate_amx_xtile, #endif &vmstate_arch_lbr,
Signature · 1 line
2.42.0