x86/svm: add pause filtering threshold for SVM
cover x86/svm: add pause filtering threshold for SVM
This patch series adds support and enablement of the pause filtering threshold. Once there's pause filtering threshold amount of cycles between pauses, the pause filtering counter resets to what was in the VMCB. This allows the pause filtering count to "reset" between pauses and keeps the guset from getting intercepted by the hypervisor. See AMD APM vol 2 section 15.14.4 for more details.
In reply to this will be an email with graphs showing some benchmark results of why the values of the counter and threshold were picked.
Brian Woods (2):
x86/svm: add support for pause filtering threshold x86/svm: enable pause filtering threshold
xen/arch/x86/hvm/svm/svm.c | 1 + xen/arch/x86/hvm/svm/vmcb.c | 3 +++ xen/include/asm-x86/hvm/svm/svm.h | 5 ++++- xen/include/asm-x86/hvm/svm/vmcb.h | 3 ++- 4 files changed, 10 insertions(+), 2 deletions(-)
Signature · 7 lines
2.11.0 _______________________________________________ Xen-devel mailing list [email protected] https://lists.xenproject.org/mailman/listinfo/xen-devel
PATCH 1/2 x86/svm: add support for pause filtering threshold
Add support for enabling the pause filtering threshold feature. This causes the pause filtering count to reset if there's pause filtering threshold cycles or greater between pauses. See AMD APM Vol 2 Section 15.14.4 for more details.
The values of the pause filtering count and threshold were found by iterating over different values of the count and threshold while running kernbench and a pi spigot algorithm with yields placed in it. A balanced setting for both variable provides:
(Using averaged elapsed time with kernbench)
old = 852.0
new = 848.8
improvement = .4%
For system without pause filtering threshold, the change, from 3000 to 4000 for the count, should not negatively effect system performance.
--- xen/include/asm-x86/hvm/svm/svm.h | 5 ++++- xen/include/asm-x86/hvm/svm/vmcb.h | 3 ++- 2 files changed, 6 insertions(+), 2 deletions(-)
xen/include/asm-x86/hvm/svm/svm.h +4 -1
@@ -64,6 +64,7 @@ extern u32 svm_feature_flags;#define SVM_FEATURE_FLUSHBYASID 6/* TLB flush by ASID support */ #define SVM_FEATURE_DECODEASSISTS 7/* Decode assists support */ #define SVM_FEATURE_PAUSEFILTER 10/* Pause intercept filter support */ +#define SVM_FEATURE_PAUSETHRESH 12/* Pause intercept filter support */ #define SVM_FEATURE_VLOADSAVE 15/* virtual vmload/vmsave */ #define SVM_FEATURE_VGIF 16/* Virtual GIF */
@@ -76,10 +77,12 @@ extern u32 svm_feature_flags;#define cpu_has_svm_decode cpu_has_svm_feature(SVM_FEATURE_DECODEASSISTS)#define cpu_has_svm_vgif cpu_has_svm_feature(SVM_FEATURE_VGIF)#define cpu_has_pause_filter cpu_has_svm_feature(SVM_FEATURE_PAUSEFILTER) +#define cpu_has_pause_thresh cpu_has_svm_feature(SVM_FEATURE_PAUSETHRESH)#define cpu_has_tsc_ratio cpu_has_svm_feature(SVM_FEATURE_TSCRATEMSR)#define cpu_has_svm_vloadsave cpu_has_svm_feature(SVM_FEATURE_VLOADSAVE) -#define SVM_PAUSEFILTER_INIT 3000 +#define SVM_PAUSEFILTER_INIT 4000 +#define SVM_PAUSETHRESH_INIT 1000/* TSC rate */ #define DEFAULT_TSC_RATIO 0x0000000100000000ULL
xen/include/asm-x86/hvm/svm/vmcb.h +2 -1
@@ -412,7 +412,7 @@ struct vmcb_struct {u64 res04;/* offset 0x28 */ u64 res05;/* offset 0x30 */ u32 res06;/* offset 0x38 */ -u16 res06a;/* offset 0x3C */ +u16 _pause_filter_thresh;/* offset 0x3C - cleanbit 0 */ u16 _pause_filter_count;/* offset 0x3E - cleanbit 0 */ u64 _iopm_base_pa;/* offset 0x40 - cleanbit 1 */ u64 _msrpm_base_pa;/* offset 0x48 - cleanbit 1 */
@@ -568,6 +568,7 @@ VMCB_ACCESSORS(exception_intercepts, intercepts) VMCB_ACCESSORS(general1_intercepts, intercepts) VMCB_ACCESSORS(general2_intercepts, intercepts) VMCB_ACCESSORS(pause_filter_count, intercepts) +VMCB_ACCESSORS(pause_filter_thresh, intercepts) VMCB_ACCESSORS(tsc_offset, intercepts) VMCB_ACCESSORS(iopm_base_pa, iopm)VMCB_ACCESSORS (msrpm_base_pa, iopm)
Signature · 7 lines
2.11.0 _______________________________________________ Xen-devel mailing list [email protected] https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [PATCH 1/2] x86/svm: add support for pause filtering threshold
Looks good.
Quoted 91 lines
-----Original Message-----
From: Woods, Brian
Sent: Tuesday, February 20, 2018 6:13 PM
To: [email protected]
Cc: Boris Ostrovsky <[email protected]>; Suthikulpanit, Suravee <[email protected]>; Jan Beulich <[email protected]>; Andrew Cooper <[email protected]>; Woods, Brian <[email protected]>
Subject: [PATCH 1/2] x86/svm: add support for pause filtering thresholdAdd support for enabling the pause filtering threshold feature. This causes the pause filtering count to reset if there's pause filtering threshold cycles or greater between pauses. See AMD APM Vol 2 Section 15.14.4 for more details.
The values of the pause filtering count and threshold were found by iterating over different values of the count and threshold while running kernbench and a pi spigot algorithm with yields placed in it. A balanced setting for both variable provides:
(Using averaged elapsed time with kernbench)
old = 852.0
new = 848.8
improvement = .4%For system without pause filtering threshold, the change, from 3000 to 4000 for the count, should not negatively effect system performance.
Signed-off-by Brian Woods <[email protected]>--- xen/include/asm-x86/hvm/svm/svm.h | 5 ++++- xen/include/asm-x86/hvm/svm/vmcb.h | 3 ++- 2 files changed, 6 insertions(+), 2 deletions(-)xen/include/asm- renamed from xen/include/asm-x86/hvm/svm/svm.h +0
-0
x86/hvm/svm/svm.h
index462cb89b7c..593546fb56100644xen/include/asm-x86/hvm/svm/svm.h +1
-0@@ -64,6 +64,7 @@ extern u32 svm_feature_flags;#define SVM_FEATURE_FLUSHBYASID 6/* TLB flush by ASID support */ #define SVM_FEATURE_DECODEASSISTS 7/* Decode assists support */ #define SVM_FEATURE_PAUSEFILTER 10/* Pause intercept filter support */ +#define SVM_FEATURE_PAUSETHRESH 12/* Pause intercept filter support */ #define SVM_FEATURE_VLOADSAVE 15/* virtual vmload/vmsave */ #define
SVM_FEATURE_VGIF16 /* Virtual GIF */xen/include/asm-x86/hvm/svm/svm.h +3
-1@@ -76,10 +77,12 @@ extern u32 svm_feature_flags;#define cpu_has_svm_decode cpu_has_svm_feature(SVM_FEATURE_DECODEASSISTS)#define cpu_has_svm_vgif cpu_has_svm_feature(SVM_FEATURE_VGIF)#define cpu_has_pause_filter cpu_has_svm_feature(SVM_FEATURE_PAUSEFILTER) +#define cpu_has_pause_thresh cpu_has_svm_feature(SVM_FEATURE_PAUSETHRESH)#define cpu_has_tsc_ratio cpu_has_svm_feature(SVM_FEATURE_TSCRATEMSR)#define cpu_has_svm_vloadsave cpu_has_svm_feature(SVM_FEATURE_VLOADSAVE) -#define SVM_PAUSEFILTER_INIT 3000 +#define SVM_PAUSEFILTER_INIT 4000 +#define SVM_PAUSETHRESH_INIT 1000/* TSC rate */ #define DEFAULT_TSC_RATIO 0x0000000100000000ULLxen/include/asm- renamed from xen/include/asm-x86/hvm/svm/vmcb.h +0
-0
x86/hvm/svm/vmcb.h
index9d5dfc58f2..de07429dff100644xen/include/asm-x86/hvm/svm/vmcb.h +2
-2@@ -412,7 +412,7 @@ struct vmcb_struct {u64 res04;/* offset 0x28 */ u64 res05;/* offset 0x30 */ u32 res06;/* offset 0x38 */ -u16 res06a;/* offset 0x3C */ +u16 _pause_filter_thresh;/* offset 0x3C - cleanbit 0 */ u16 _pause_filter_count;/* offset 0x3E - cleanbit 0 */ u64 _iopm_base_pa;/* offset 0x40 - cleanbit 1 */ u64 _msrpm_base_pa;/* offset 0x48 - cleanbit 1 */ @@ -568,6 +568,7 @@ VMCB_ACCESSORS(exception_intercepts, intercepts) VMCB_ACCESSORS(general1_intercepts, intercepts) VMCB_ACCESSORS(general2_intercepts, intercepts) VMCB_ACCESSORS(pause_filter_count, intercepts) +VMCB_ACCESSORS(pause_filter_thresh, intercepts) VMCB_ACCESSORS(tsc_offset, intercepts) VMCB_ACCESSORS(iopm_base_pa, iopm)VMCB_ACCESSORS (msrpm_base_pa, iopm) --
Quoted 1 line
2.11.0
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [PATCH 1/2] x86/svm: add support for pause filtering threshold
In reply to Brian WoodsQuoted 5 lines
Quoted 1 line
Quoted 1 line · On 21.02.18 at 01:12, <[email protected]> wrote:
Add support for enabling the pause filtering threshold feature. This causes the pause filtering count to reset if there's pause filtering threshold cycles or greater between pauses. See AMD APM Vol 2 Section 15.14.4 for more details.
FTR I'm going to commit this as is (since it has been acked), but in the future please don't refer to documentation sections by number, but instead by title (as the numbers tend to change over time).
Jan
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
PATCH 2/2 x86/svm: enable pause filtering threshold
In reply to Brian WoodsIf available, enable the pause filtering threshold feature. See the previous commit for more information.
--- xen/arch/x86/hvm/svm/svm.c | 1 + xen/arch/x86/hvm/svm/vmcb.c | 3 +++ 2 files changed, 4 insertions(+)
xen/arch/x86/hvm/svm/svm.c +1 -0
@@ -1694,6 +1694,7 @@ const struct hvm_function_table * __init start_svm(void)P (cpu_has_svm_vloadsave,"Virtual VMLOAD/VMSAVE" );P (cpu_has_svm_vgif,"Virtual GIF" );P (cpu_has_pause_filter,"Pause-Intercept Filter" ); +P (cpu_has_pause_thresh,"Pause-Intercept Filter Threshold" );P (cpu_has_tsc_ratio,"TSC Rate MSR" );#undef P
xen/arch/x86/hvm/svm/vmcb.c +3 -0
@@ -227,6 +227,9 @@ static int construct_vmcb(struct vcpu *v) { vmcb->_pause_filter_count =SVM_PAUSEFILTER_INIT ; vmcb->_general1_intercepts |=GENERAL1_INTERCEPT_PAUSE ; + +if ( cpu_has_pause_thresh ) + vmcb->_pause_filter_thresh =SVM_PAUSETHRESH_INIT ; } vmcb->cleanbits.bytes =0 ;
Signature · 7 lines
2.11.0 _______________________________________________ Xen-devel mailing list [email protected] https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [PATCH 2/2] x86/svm: enable pause filtering threshold
Looks good.
Quoted 48 lines
-----Original Message-----
From: Woods, Brian
Sent: Tuesday, February 20, 2018 6:13 PM
To: [email protected]
Cc: Boris Ostrovsky <[email protected]>; Suthikulpanit, Suravee <[email protected]>; Jan Beulich <[email protected]>; Andrew Cooper <[email protected]>; Woods, Brian <[email protected]>
Subject: [PATCH 2/2] x86/svm: enable pause filtering thresholdIf available, enable the pause filtering threshold feature. See the previous commit for more information.
Signed-off-by Brian Woods <[email protected]>--- xen/arch/x86/hvm/svm/svm.c | 1 + xen/arch/x86/hvm/svm/vmcb.c | 3 +++ 2 files changed, 4 insertions(+)xen/arch/x86/hvm/svm/svm.c +1
-0@@ -1694,6 +1694,7 @@ const struct hvm_function_table * __init start_svm(void) P(cpu_has_svm_vloadsave, "Virtual VMLOAD/VMSAVE");P (cpu_has_svm_vgif,"Virtual GIF" );P (cpu_has_pause_filter,"Pause-Intercept Filter" ); +P (cpu_has_pause_thresh,"Pause-Intercept Filter Threshold" );P (cpu_has_tsc_ratio,"TSC Rate MSR" );#undef Pxen/arch/x86/hvm/svm/vmcb.c +3
-1@@ -227,6 +227,9 @@ static int construct_vmcb(struct vcpu *v) { vmcb->_pause_filter_count =SVM_PAUSEFILTER_INIT ; vmcb->_general1_intercepts |=GENERAL1_INTERCEPT_PAUSE ; + +if ( cpu_has_pause_thresh ) + vmcb->_pause_filter_thresh =SVM_PAUSETHRESH_INIT ; } vmcb->cleanbits.bytes =0 ; --
Quoted 1 line
2.11.0
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
HW Setup:
CPU: AMD EPYC 7401 24c/48t x 2 RAM: 128GB RAM Storage: HDD for dom0, SSDs for VMs
SW Setup:
Xen base 4c7e478d597b0346eef3a256cfd6794ac778b608 32 VMs
6 VPCUs per VM
3.5GB RAM per VM
7.5GB disk per VM
Results:
kernbench:
half -> -j vpcu/2 = one job per host thread
quad -> -j vpcu*4 = two VMs per host thread,
four jobs per VM VCPU
pi:
03 -> 3 in parallel = one pi per host thread
24 -> 24 in parallel = two VMs per host thread,
four pis per VM VCPUKernbench: https://github.com/linux-test-project/ltp/blob/master/utils/benchmark/kernbench-0.42/kernbench
pi: based off of https://stackoverflow.com/questions/4084571/implementing-the-spigot-algorithm-for-%CF%80-pi
but with yields and 10ns sleeps added in.
Between the current usage of pause filtering counter, there's about an ~.5% improve in performance. While this isn't huge, all that's required is to write a value in the VMCB.
Attached:
elapsed_time_kernbenchhalf.pdf
elapsed_time_kernbenchquad.pdf
elapsed_time_pi03.pdf
elapsed_time_pi24.pdf
Signature · 1 line
Brian Woods
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Quoted 14 lines · Brian Woods
This patch series adds support and enablement of the pause filtering threshold. Once there's pause filtering threshold amount of cycles between pauses, the pause filtering counter resets to what was in the VMCB. This allows the pause filtering count to "reset" between pauses and keeps the guset from getting intercepted by the hypervisor. See AMD APM vol 2 section 15.14.4 for more details.
In reply to this will be an email with graphs showing some benchmark results of why the values of the counter and threshold were picked.
Brian Woods (2):
x86/svm: add support for pause filtering threshold x86/svm: enable pause filtering threshold
Quoted 6 lines
xen/arch/x86/hvm/svm/svm.c | 1 + xen/arch/x86/hvm/svm/vmcb.c | 3 +++ xen/include/asm-x86/hvm/svm/svm.h | 5 ++++- xen/include/asm-x86/hvm/svm/vmcb.h | 3 ++- 4 files changed, 10 insertions(+), 2 deletions(-)
IIRC previous count value (3000) was somewhat arbitrary so
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel