Skip to contentSkip to conversation

x86/svm: add pause filtering threshold for SVM

PATCH In review Brian Woods 4 participants 8 messages Feb 21, 2018 – Feb 23, 2018

lore
Share & tools
lore
#0 lore raw

cover x86/svm: add pause filtering threshold for SVM

This patch series adds support and enablement of the pause filtering threshold. Once there's pause filtering threshold amount of cycles between pauses, the pause filtering counter resets to what was in the VMCB. This allows the pause filtering count to "reset" between pauses and keeps the guset from getting intercepted by the hypervisor. See AMD APM vol 2 section 15.14.4 for more details.

In reply to this will be an email with graphs showing some benchmark results of why the values of the counter and threshold were picked.

Brian Woods (2):
x86/svm: add support for pause filtering threshold x86/svm: enable pause filtering threshold

 xen/arch/x86/hvm/svm/svm.c         | 1 +
 xen/arch/x86/hvm/svm/vmcb.c        | 3 +++
 xen/include/asm-x86/hvm/svm/svm.h  | 5 ++++-
 xen/include/asm-x86/hvm/svm/vmcb.h | 3 ++-
 4 files changed, 10 insertions(+), 2 deletions(-)
Signature · 7 lines
2.11.0


_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
#1 lore raw

PATCH 1/2 x86/svm: add support for pause filtering threshold

Add support for enabling the pause filtering threshold feature. This causes the pause filtering count to reset if there's pause filtering threshold cycles or greater between pauses. See AMD APM Vol 2 Section 15.14.4 for more details.

The values of the pause filtering count and threshold were found by iterating over different values of the count and threshold while running kernbench and a pi spigot algorithm with yields placed in it. A balanced setting for both variable provides:

(Using averaged elapsed time with kernbench)
old = 852.0
new = 848.8
improvement = .4%

For system without pause filtering threshold, the change, from 3000 to 4000 for the count, should not negatively effect system performance.

---
 xen/include/asm-x86/hvm/svm/svm.h  | 5 ++++-
 xen/include/asm-x86/hvm/svm/vmcb.h | 3 ++-
 2 files changed, 6 insertions(+), 2 deletions(-)
xen/include/asm-x86/hvm/svm/svm.h +4 -1
index 462cb89b7c..593546fb56 100644
@@ -64,6 +64,7 @@ extern u32 svm_feature_flags;
 #define SVM_FEATURE_FLUSHBYASID    6 /* TLB flush by ASID support */
 #define SVM_FEATURE_DECODEASSISTS  7 /* Decode assists support */
 #define SVM_FEATURE_PAUSEFILTER   10 /* Pause intercept filter support */
+#define SVM_FEATURE_PAUSETHRESH   12 /* Pause intercept filter support */
 #define SVM_FEATURE_VLOADSAVE     15 /* virtual vmload/vmsave */
 #define SVM_FEATURE_VGIF          16 /* Virtual GIF */
 
@@ -76,10 +77,12 @@ extern u32 svm_feature_flags;
 #define cpu_has_svm_decode    cpu_has_svm_feature(SVM_FEATURE_DECODEASSISTS)
 #define cpu_has_svm_vgif      cpu_has_svm_feature(SVM_FEATURE_VGIF)
 #define cpu_has_pause_filter  cpu_has_svm_feature(SVM_FEATURE_PAUSEFILTER)
+#define cpu_has_pause_thresh  cpu_has_svm_feature(SVM_FEATURE_PAUSETHRESH)
 #define cpu_has_tsc_ratio     cpu_has_svm_feature(SVM_FEATURE_TSCRATEMSR)
 #define cpu_has_svm_vloadsave cpu_has_svm_feature(SVM_FEATURE_VLOADSAVE)
 
-#define SVM_PAUSEFILTER_INIT    3000
+#define SVM_PAUSEFILTER_INIT    4000
+#define SVM_PAUSETHRESH_INIT    1000
 
 /* TSC rate */
 #define DEFAULT_TSC_RATIO       0x0000000100000000ULL
xen/include/asm-x86/hvm/svm/vmcb.h +2 -1
index 9d5dfc58f2..de07429dff 100644
@@ -412,7 +412,7 @@ struct vmcb_struct {
     u64 res04;                  /* offset 0x28 */
     u64 res05;                  /* offset 0x30 */
     u32 res06;                  /* offset 0x38 */
-    u16 res06a;                 /* offset 0x3C */
+    u16 _pause_filter_thresh;   /* offset 0x3C - cleanbit 0 */
     u16 _pause_filter_count;    /* offset 0x3E - cleanbit 0 */
     u64 _iopm_base_pa;          /* offset 0x40 - cleanbit 1 */
     u64 _msrpm_base_pa;         /* offset 0x48 - cleanbit 1 */
@@ -568,6 +568,7 @@ VMCB_ACCESSORS(exception_intercepts, intercepts)
 VMCB_ACCESSORS(general1_intercepts, intercepts)
 VMCB_ACCESSORS(general2_intercepts, intercepts)
 VMCB_ACCESSORS(pause_filter_count, intercepts)
+VMCB_ACCESSORS(pause_filter_thresh, intercepts)
 VMCB_ACCESSORS(tsc_offset, intercepts)
 VMCB_ACCESSORS(iopm_base_pa, iopm)
 VMCB_ACCESSORS(msrpm_base_pa, iopm)
Signature · 7 lines
2.11.0


_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Reviewed-by on v1 1/2
#2 lore raw

Re: [PATCH 1/2] x86/svm: add support for pause filtering threshold

Looks good.

Quoted 91 lines

-----Original Message-----
From: Woods, Brian
Sent: Tuesday, February 20, 2018 6:13 PM
To: [email protected]
Cc: Boris Ostrovsky <[email protected]>; Suthikulpanit, Suravee <[email protected]>; Jan Beulich <[email protected]>; Andrew Cooper <[email protected]>; Woods, Brian <[email protected]>
Subject: [PATCH 1/2] x86/svm: add support for pause filtering threshold

Add support for enabling the pause filtering threshold feature. This causes the pause filtering count to reset if there's pause filtering threshold cycles or greater between pauses. See AMD APM Vol 2 Section 15.14.4 for more details.

The values of the pause filtering count and threshold were found by iterating over different values of the count and threshold while running kernbench and a pi spigot algorithm with yields placed in it. A balanced setting for both variable provides:

(Using averaged elapsed time with kernbench)
old = 852.0
new = 848.8
improvement = .4%

For system without pause filtering threshold, the change, from 3000 to 4000 for the count, should not negatively effect system performance.

---
 xen/include/asm-x86/hvm/svm/svm.h  | 5 ++++-
 xen/include/asm-x86/hvm/svm/vmcb.h | 3 ++-
 2 files changed, 6 insertions(+), 2 deletions(-)
xen/include/asm- renamed from xen/include/asm-x86/hvm/svm/svm.h +0 -0

x86/hvm/svm/svm.h
index 462cb89b7c..593546fb56 100644

xen/include/asm-x86/hvm/svm/svm.h +1 -0
@@ -64,6 +64,7 @@ extern u32 svm_feature_flags;
 #define SVM_FEATURE_FLUSHBYASID    6 /* TLB flush by ASID support */
 #define SVM_FEATURE_DECODEASSISTS  7 /* Decode assists support */
 #define SVM_FEATURE_PAUSEFILTER   10 /* Pause intercept filter support
 */
+#define SVM_FEATURE_PAUSETHRESH   12 /* Pause intercept filter support
 */
 #define SVM_FEATURE_VLOADSAVE     15 /* virtual vmload/vmsave */

#define SVM_FEATURE_VGIF 16 /* Virtual GIF */

xen/include/asm-x86/hvm/svm/svm.h +3 -1
@@ -76,10 +77,12 @@ extern u32 svm_feature_flags;
 #define cpu_has_svm_decode
 cpu_has_svm_feature(SVM_FEATURE_DECODEASSISTS)
 #define cpu_has_svm_vgif      cpu_has_svm_feature(SVM_FEATURE_VGIF)
 #define cpu_has_pause_filter
 cpu_has_svm_feature(SVM_FEATURE_PAUSEFILTER)
+#define cpu_has_pause_thresh
 cpu_has_svm_feature(SVM_FEATURE_PAUSETHRESH)
 #define cpu_has_tsc_ratio
 cpu_has_svm_feature(SVM_FEATURE_TSCRATEMSR)
 #define cpu_has_svm_vloadsave
 cpu_has_svm_feature(SVM_FEATURE_VLOADSAVE)
 
-#define SVM_PAUSEFILTER_INIT    3000
+#define SVM_PAUSEFILTER_INIT    4000
+#define SVM_PAUSETHRESH_INIT    1000
 
 /* TSC rate */
 #define DEFAULT_TSC_RATIO       0x0000000100000000ULL
xen/include/asm- renamed from xen/include/asm-x86/hvm/svm/vmcb.h +0 -0

x86/hvm/svm/vmcb.h
index 9d5dfc58f2..de07429dff 100644

xen/include/asm-x86/hvm/svm/vmcb.h +2 -2
@@ -412,7 +412,7 @@ struct vmcb_struct {
     u64 res04;                  /* offset 0x28 */
     u64 res05;                  /* offset 0x30 */
     u32 res06;                  /* offset 0x38 */
-    u16 res06a;                 /* offset 0x3C */
+    u16 _pause_filter_thresh;   /* offset 0x3C - cleanbit 0 */
     u16 _pause_filter_count;    /* offset 0x3E - cleanbit 0 */
     u64 _iopm_base_pa;          /* offset 0x40 - cleanbit 1 */
     u64 _msrpm_base_pa;         /* offset 0x48 - cleanbit 1 */
@@ -568,6 +568,7 @@ VMCB_ACCESSORS(exception_intercepts, intercepts)
 VMCB_ACCESSORS(general1_intercepts, intercepts)
 VMCB_ACCESSORS(general2_intercepts, intercepts)
 VMCB_ACCESSORS(pause_filter_count, intercepts)
+VMCB_ACCESSORS(pause_filter_thresh, intercepts)
 VMCB_ACCESSORS(tsc_offset, intercepts)
 VMCB_ACCESSORS(iopm_base_pa, iopm)
 VMCB_ACCESSORS(msrpm_base_pa, iopm)
--
Quoted 1 line

2.11.0

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

#3 lore raw

Re: [PATCH 1/2] x86/svm: add support for pause filtering threshold

In reply to Brian Woods
Quoted 5 lines
Quoted 1 line
Quoted 1 line · On 21.02.18 at 01:12, <[email protected]> wrote:

Add support for enabling the pause filtering threshold feature. This causes the pause filtering count to reset if there's pause filtering threshold cycles or greater between pauses. See AMD APM Vol 2 Section 15.14.4 for more details.

FTR I'm going to commit this as is (since it has been acked), but in the future please don't refer to documentation sections by number, but instead by title (as the numbers tend to change over time).

Jan

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

#4 lore raw

PATCH 2/2 x86/svm: enable pause filtering threshold

In reply to Brian Woods

If available, enable the pause filtering threshold feature. See the previous commit for more information.

---
 xen/arch/x86/hvm/svm/svm.c  | 1 +
 xen/arch/x86/hvm/svm/vmcb.c | 3 +++
 2 files changed, 4 insertions(+)
xen/arch/x86/hvm/svm/svm.c +1 -0
index 9f58afc2d8..b6b92365bf 100644
@@ -1694,6 +1694,7 @@ const struct hvm_function_table * __init start_svm(void)
     P(cpu_has_svm_vloadsave, "Virtual VMLOAD/VMSAVE");
     P(cpu_has_svm_vgif, "Virtual GIF");
     P(cpu_has_pause_filter, "Pause-Intercept Filter");
+    P(cpu_has_pause_thresh, "Pause-Intercept Filter Threshold");
     P(cpu_has_tsc_ratio, "TSC Rate MSR");
 #undef P
 
xen/arch/x86/hvm/svm/vmcb.c +3 -0
index 0e6cba5b7b..ebe6f0c751 100644
@@ -227,6 +227,9 @@ static int construct_vmcb(struct vcpu *v)
     {
         vmcb->_pause_filter_count = SVM_PAUSEFILTER_INIT;
         vmcb->_general1_intercepts |= GENERAL1_INTERCEPT_PAUSE;
+
+        if ( cpu_has_pause_thresh )
+            vmcb->_pause_filter_thresh = SVM_PAUSETHRESH_INIT;
     }
 
     vmcb->cleanbits.bytes = 0;
Signature · 7 lines
2.11.0


_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Reviewed-by on v1 2/2
#5 lore raw

Re: [PATCH 2/2] x86/svm: enable pause filtering threshold

Looks good.

Quoted 48 lines

-----Original Message-----
From: Woods, Brian
Sent: Tuesday, February 20, 2018 6:13 PM
To: [email protected]
Cc: Boris Ostrovsky <[email protected]>; Suthikulpanit, Suravee <[email protected]>; Jan Beulich <[email protected]>; Andrew Cooper <[email protected]>; Woods, Brian <[email protected]>
Subject: [PATCH 2/2] x86/svm: enable pause filtering threshold

If available, enable the pause filtering threshold feature. See the previous commit for more information.

---
 xen/arch/x86/hvm/svm/svm.c  | 1 +
 xen/arch/x86/hvm/svm/vmcb.c | 3 +++
 2 files changed, 4 insertions(+)
xen/arch/x86/hvm/svm/svm.c +1 -0
index 9f58afc2d8..b6b92365bf 100644
@@ -1694,6 +1694,7 @@ const struct hvm_function_table * __init
 start_svm(void)
     P(cpu_has_svm_vloadsave, "Virtual VMLOAD/VMSAVE");
     P(cpu_has_svm_vgif, "Virtual GIF");
     P(cpu_has_pause_filter, "Pause-Intercept Filter");
+    P(cpu_has_pause_thresh, "Pause-Intercept Filter Threshold");
     P(cpu_has_tsc_ratio, "TSC Rate MSR");
 #undef P
xen/arch/x86/hvm/svm/vmcb.c +3 -1
index 0e6cba5b7b..ebe6f0c751 100644
@@ -227,6 +227,9 @@ static int construct_vmcb(struct vcpu *v)
     {
         vmcb->_pause_filter_count = SVM_PAUSEFILTER_INIT;
         vmcb->_general1_intercepts |= GENERAL1_INTERCEPT_PAUSE;
+
+        if ( cpu_has_pause_thresh )
+            vmcb->_pause_filter_thresh = SVM_PAUSETHRESH_INIT;
     }
 
     vmcb->cleanbits.bytes = 0;
--
Quoted 1 line

2.11.0

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

#6 lore raw
In reply to Brian Woods

HW Setup:

	CPU: AMD EPYC 7401 24c/48t x 2 
	RAM: 128GB RAM
	Storage: HDD for dom0, SSDs for VMs

SW Setup:
Xen base 4c7e478d597b0346eef3a256cfd6794ac778b608 32 VMs
6 VPCUs per VM
3.5GB RAM per VM
7.5GB disk per VM

Results:

	kernbench:
		half -> -j vpcu/2 = one job per host thread
		quad -> -j vpcu*4 = two VMs per host thread,
                                    four jobs per VM VCPU
	pi:
		03 ->  3 in parallel = one pi per host thread
		24 -> 24 in parallel = two VMs per host thread,
                                       four pis per VM VCPU

Kernbench: https://github.com/linux-test-project/ltp/blob/master/utils/benchmark/kernbench-0.42/kernbench
pi: based off of https://stackoverflow.com/questions/4084571/implementing-the-spigot-algorithm-for-%CF%80-pi
but with yields and 10ns sleeps added in.

Between the current usage of pause filtering counter, there's about an ~.5% improve in performance. While this isn't huge, all that's required is to write a value in the VMCB.

Attached:
elapsed_time_kernbenchhalf.pdf
elapsed_time_kernbenchquad.pdf
elapsed_time_pi03.pdf
elapsed_time_pi24.pdf

Signature · 1 line
Brian Woods

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

Reviewed-by on the v1 series
#7 lore raw
In reply to Brian Woods
Quoted 14 lines · Brian Woods

This patch series adds support and enablement of the pause filtering threshold. Once there's pause filtering threshold amount of cycles between pauses, the pause filtering counter resets to what was in the VMCB. This allows the pause filtering count to "reset" between pauses and keeps the guset from getting intercepted by the hypervisor. See AMD APM vol 2 section 15.14.4 for more details.

In reply to this will be an email with graphs showing some benchmark results of why the values of the counter and threshold were picked.

Brian Woods (2):
x86/svm: add support for pause filtering threshold x86/svm: enable pause filtering threshold

Quoted 6 lines
 xen/arch/x86/hvm/svm/svm.c         | 1 +
 xen/arch/x86/hvm/svm/vmcb.c        | 3 +++
 xen/include/asm-x86/hvm/svm/svm.h  | 5 ++++-
 xen/include/asm-x86/hvm/svm/vmcb.h | 3 ++-
 4 files changed, 10 insertions(+), 2 deletions(-)

IIRC previous count value (3000) was somewhat arbitrary so

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel