Skip to contentSkip to conversation

KVM: LAPIC: Per vCPU control over kvm_can_post_timer_interrupt

PATCH Superseded Aili Yao 4 participants 9 messages Nov 22, 2021 – Nov 29, 2021

Subsystems (3)

v1 of 2 Superseded Latest is v2 →

lore
Share & tools
lore

Aili Yao

author
#0 lore raw

PATCH 1/1 KVM: LAPIC: Per vCPU control over kvm_can_post_timer_interrupt

From: Aili Yao <[email protected]>

When we isolate some pyhiscal cores, We may not use them for kvm guests, We may use them for other purposes like DPDK, or we can make some kvm guests isolated and some not, the global judgement pi_inject_timer is not enough; We may make wrong decisions:

In such a scenario, the guests without isolated cores will not be permitted to use vmx preemption timer, and tscdeadline fastpath also be disabled, both will lead to performance penalty.

So check whether the vcpu->cpu is isolated, if not, don't post timer interrupt.

---
 arch/x86/kvm/lapic.c | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)
arch/x86/kvm/lapic.c +3 -1
index 759952dd1222..72dde5532101 100644
@@ -34,6 +34,7 @@
 #include <asm/delay.h>
 #include <linux/atomic.h>
 #include <linux/jump_label.h>
+#include <linux/sched/isolation.h>
 #include "kvm_cache_regs.h"
 #include "irq.h"
 #include "ioapic.h"
@@ -113,7 +114,8 @@ static inline u32 kvm_x2apic_id(struct kvm_lapic *apic)
 
 static bool kvm_can_post_timer_interrupt(struct kvm_vcpu *vcpu)
 {
-	return pi_inject_timer && kvm_vcpu_apicv_active(vcpu);
+	return pi_inject_timer && kvm_vcpu_apicv_active(vcpu) &&
+		!housekeeping_cpu(vcpu->cpu, HK_FLAG_TIMER);
 }
 
 bool kvm_can_use_hv_timer(struct kvm_vcpu *vcpu)
Signature · 1 line
2.25.1
#1 lore raw
Quoted 32 lines · Aili Yao

From: Aili Yao <[email protected]>

When we isolate some pyhiscal cores, We may not use them for kvm guests, We may use them for other purposes like DPDK, or we can make some kvm guests isolated and some not, the global judgement pi_inject_timer is not enough; We may make wrong decisions:

In such a scenario, the guests without isolated cores will not be permitted to use vmx preemption timer, and tscdeadline fastpath also be disabled, both will lead to performance penalty.

So check whether the vcpu->cpu is isolated, if not, don't post timer interrupt.

---
 arch/x86/kvm/lapic.c | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)
arch/x86/kvm/lapic.c +1 -0
index 759952dd1222..72dde5532101 100644
@@ -34,6 +34,7 @@
 #include <asm/delay.h>
 #include <linux/atomic.h>
 #include <linux/jump_label.h>
+#include <linux/sched/isolation.h>
 #include "kvm_cache_regs.h"
 #include "irq.h"
 #include "ioapic.h"

arch/x86/kvm/lapic.c static inline u32 kvm_x2apic_id(struct kvm_lapic *apic) · line 118

Quoted 6 lines
arch/x86/kvm/lapic.c +2 -1
@@ -113,7 +114,8 @@ static inline u32 kvm_x2apic_id(struct kvm_lapic *apic)
 
 static bool kvm_can_post_timer_interrupt(struct kvm_vcpu *vcpu)
 {
-	return pi_inject_timer && kvm_vcpu_apicv_active(vcpu);
+	return pi_inject_timer && kvm_vcpu_apicv_active(vcpu) &&
+		!housekeeping_cpu(vcpu->cpu, HK_FLAG_TIMER);

I don't think this is safe, vcpu->cpu will be -1 if the vCPU isn't scheduled in. This also doesn't play nice with the admin forcing pi_inject_timer=1. Not saying there's a reasonable use case for doing that, but it's supported today and this would break that behavior. It would also lead to weird behavior if a vCPU were migrated on/off a housekeeping vCPU. Again, probably not a reasonable use case, but I don't see anything that would outright prevent that behavior.

The existing behavior also feels a bit unsafe as pi_inject_timer is writable while KVM is running, though I supposed that's orthogonal to this discussion.

Rather than check vcpu->cpu, is there an existing vCPU flag that can be queried, e.g. KVM_HINTS_REALTIME?

Quoted 6 lines
arch/x86/kvm/lapic.c +0 -0
 }
 
 bool kvm_can_use_hv_timer(struct kvm_vcpu *vcpu)
Signature · 1 line
2.25.1
#2 lore raw
Quoted 48 lines · Sean Christopherson
Quoted 32 lines · Aili Yao

From: Aili Yao <[email protected]>

When we isolate some pyhiscal cores, We may not use them for kvm guests, We may use them for other purposes like DPDK, or we can make some kvm guests isolated and some not, the global judgement pi_inject_timer is not enough; We may make wrong decisions:

In such a scenario, the guests without isolated cores will not be permitted to use vmx preemption timer, and tscdeadline fastpath also be disabled, both will lead to performance penalty.

So check whether the vcpu->cpu is isolated, if not, don't post timer interrupt.

---
 arch/x86/kvm/lapic.c | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)
arch/x86/kvm/lapic.c +1 -0
index 759952dd1222..72dde5532101 100644
@@ -34,6 +34,7 @@
 #include <asm/delay.h>
 #include <linux/atomic.h>
 #include <linux/jump_label.h>
+#include <linux/sched/isolation.h>
 #include "kvm_cache_regs.h"
 #include "irq.h"
 #include "ioapic.h"

arch/x86/kvm/lapic.c static inline u32 kvm_x2apic_id(struct kvm_lapic *apic) · line 118

Quoted 6 lines
arch/x86/kvm/lapic.c +2 -1
@@ -113,7 +114,8 @@ static inline u32 kvm_x2apic_id(struct kvm_lapic *apic)
 
 static bool kvm_can_post_timer_interrupt(struct kvm_vcpu *vcpu)
 {
-     return pi_inject_timer && kvm_vcpu_apicv_active(vcpu);
+     return pi_inject_timer && kvm_vcpu_apicv_active(vcpu) &&
+             !housekeeping_cpu(vcpu->cpu, HK_FLAG_TIMER);

I don't think this is safe, vcpu->cpu will be -1 if the vCPU isn't scheduled in. This also doesn't play nice with the admin forcing pi_inject_timer=1. Not saying there's a reasonable use case for doing that, but it's supported today and this would break that behavior. It would also lead to weird behavior if a vCPU were migrated on/off a housekeeping vCPU. Again, probably not a reasonable use case, but I don't see anything that would outright prevent that behavior.

Quoted 5 lines

The existing behavior also feels a bit unsafe as pi_inject_timer is writable while KVM is running, though I supposed that's orthogonal to this discussion.

Rather than check vcpu->cpu, is there an existing vCPU flag that can be queried, e.g. KVM_HINTS_REALTIME?

How about something like below:

From 67f605120e212384cb3d5788ba8c83f15659503b Mon Sep 17 00:00:00 2001
From: Wanpeng Li <[email protected]>
Date: Tue, 23 Nov 2021 10:36:10 +0800
Subject: [PATCH] KVM: LAPIC: To keep the vCPUs in non-root mode for timer-pi

From: Wanpeng Li <[email protected]>

As commit 0c5f81dad46 (KVM: LAPIC: Inject timer interrupt via posted interrupt) mentioned that the host admin should well tune the guest setup, so that vCPUs are placed on isolated pCPUs, and with several pCPUs surplus for *busy* housekeeping.
It is better to disable mwait/hlt/pause vmexits to keep the vCPUs in non-root mode. However, we may isolate pCPUs for other purpose like DPDK or we can make some guests isolated and others not, Let's add the checking kvm_mwait_in_guest() to kvm_can_post_timer_interrupt() since we can't benefit from timer posted-interrupt
w/o keeping the vCPUs in non-root mode.

---
 arch/x86/kvm/lapic.c | 5 ++---
 1 file changed, 2 insertions(+), 3 deletions(-)
arch/x86/kvm/lapic.c +1 -1
index 759952dd1222..8257566d44c7 100644
@@ -113,14 +113,13 @@ static inline u32 kvm_x2apic_id(struct kvm_lapic *apic)
 
 static bool kvm_can_post_timer_interrupt(struct kvm_vcpu *vcpu)
 {
-    return pi_inject_timer && kvm_vcpu_apicv_active(vcpu);
+    return pi_inject_timer && kvm_mwait_in_guest(vcpu->kvm) &&

kvm_vcpu_apicv_active(vcpu);
}

bool kvm_can_use_hv_timer(struct kvm_vcpu *vcpu)
{
return kvm_x86_ops.set_hv_timer
- && !(kvm_mwait_in_guest(vcpu->kvm) ||
- kvm_can_post_timer_interrupt(vcpu));
+ && !kvm_mwait_in_guest(vcpu->kvm);
}
EXPORT_SYMBOL_GPL(kvm_can_use_hv_timer);

#3 lore raw
Quoted 38 lines
Quoted 36 lines · Sean Christopherson
Quoted 34 lines · Aili Yao

From: Aili Yao <[email protected]>

When we isolate some pyhiscal cores, We may not use them for kvm guests, We may use them for other purposes like DPDK, or we can make some kvm guests isolated and some not, the global judgement pi_inject_timer is not enough; We may make wrong decisions:

In such a scenario, the guests without isolated cores will not be permitted to use vmx preemption timer, and tscdeadline fastpath also be disabled, both will lead to performance penalty.

So check whether the vcpu->cpu is isolated, if not, don't post timer interrupt.

---
 arch/x86/kvm/lapic.c | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)
arch/x86/kvm/lapic.c index renamed from arch/x86/kvm/lapic.c +0 -0

759952dd1222..72dde5532101 100644

arch/x86/kvm/lapic.c +1 -0
@@ -34,6 +34,7 @@
 #include <asm/delay.h>
 #include <linux/atomic.h>
 #include <linux/jump_label.h>
+#include <linux/sched/isolation.h>
 #include "kvm_cache_regs.h"
 #include "irq.h"
 #include "ioapic.h"
@@ -113,7 +114,8 @@ static inline u32 kvm_x2apic_id(struct kvm_lapic
 *apic)
 
Quoted 6 lines
Quoted 6 lines

arch/x86/kvm/lapic.c static inline u32 kvm_x2apic_id(struct kvm_lapic · line 118

Quoted 4 lines
arch/x86/kvm/lapic.c +2 -1
 static bool kvm_can_post_timer_interrupt(struct kvm_vcpu *vcpu)  {
-     return pi_inject_timer && kvm_vcpu_apicv_active(vcpu);
+     return pi_inject_timer && kvm_vcpu_apicv_active(vcpu) &&
+             !housekeeping_cpu(vcpu->cpu, HK_FLAG_TIMER);

I don't think this is safe, vcpu->cpu will be -1 if the vCPU isn't scheduled in.

Yes, vcpu->cpu is -1 before vcpu create, but in my environments, it didn't trigger this issue. I need to dig more, Thanks!
Maybe I need one valid check here.

Quoted 6 lines
Quoted 5 lines

This also doesn't play nice with the admin forcing pi_inject_timer=1. Not saying there's a reasonable use case for doing that, but it's supported today and this would break that behavior. It would also lead to weird behavior if a vCPU were migrated on/off a housekeeping vCPU. Again, probably not a reasonable use case, but I don't see anything

that would outright prevent that behavior.

Yes, this is not one common operation, But I did do test some scenarios:

1. isolated cpu --> housekeeping cpu;
    isolated guest timer is in housekeeping CPU, for migration, kvm_can_post_timer_interrupt
    will return false, so the timer may be migrated to vcpu->cpu;
    This seems works in my test;

2. isolated --> isolated
Isolated guest timer is in housekeeping cpu, for migration,kvm_can_post_timer_interrupt return true, timer is not migrated
3. housekeeping CPU --> isolated CPU

    non-isolated CPU timer is usually in vcpu->cpu, for migration to isolated, kvm_can_post_timer_interrupt
    will be true,  the timer remain on the same CPU;
    This seems works in my test;  
4. housekeeping CPU --> housekeeping CPU
     timer migrated;

It seems this is not an affecting problem;

Quoted 47 lines
Quoted 3 lines

The existing behavior also feels a bit unsafe as pi_inject_timer is writable while KVM is running, though I supposed that's orthogonal to this

discussion.

Quoted 3 lines

Rather than check vcpu->cpu, is there an existing vCPU flag that can be queried, e.g. KVM_HINTS_REALTIME?

How about something like below:

From 67f605120e212384cb3d5788ba8c83f15659503b Mon Sep 17 00:00:00 2001
From: Wanpeng Li <[email protected]>
Date: Tue, 23 Nov 2021 10:36:10 +0800
Subject: [PATCH] KVM: LAPIC: To keep the vCPUs in non-root mode for timer- pi

From: Wanpeng Li <[email protected]>

As commit 0c5f81dad46 (KVM: LAPIC: Inject timer interrupt via posted interrupt) mentioned that the host admin should well tune the guest setup, so that vCPUs are placed on isolated pCPUs, and with several pCPUs surplus for
*busy* housekeeping.
It is better to disable mwait/hlt/pause vmexits to keep the vCPUs in non-root mode. However, we may isolate pCPUs for other purpose like DPDK or we can make some guests isolated and others not, Let's add the checking kvm_mwait_in_guest() to kvm_can_post_timer_interrupt() since we can't benefit from timer posted-interrupt w/o keeping the vCPUs in non-root mode.

---
 arch/x86/kvm/lapic.c | 5 ++---
 1 file changed, 2 insertions(+), 3 deletions(-)
arch/x86/kvm/lapic.c index renamed from arch/x86/kvm/lapic.c +0 -0

759952dd1222..8257566d44c7 100644

arch/x86/kvm/lapic.c +1 -1
@@ -113,14 +113,13 @@ static inline u32 kvm_x2apic_id(struct kvm_lapic
 *apic)
 
 static bool kvm_can_post_timer_interrupt(struct kvm_vcpu *vcpu)  {
-    return pi_inject_timer && kvm_vcpu_apicv_active(vcpu);
+    return pi_inject_timer && kvm_mwait_in_guest(vcpu->kvm) &&

arch/x86/kvm/lapic.c static inline u32 kvm_x2apic_id(struct kvm_lapic · line 124

Quoted 10 lines
arch/x86/kvm/lapic.c +1 -2
 kvm_vcpu_apicv_active(vcpu);
 }
 
 bool kvm_can_use_hv_timer(struct kvm_vcpu *vcpu)  {
     return kvm_x86_ops.set_hv_timer
-           && !(kvm_mwait_in_guest(vcpu->kvm) ||
-            kvm_can_post_timer_interrupt(vcpu));
+           && !kvm_mwait_in_guest(vcpu->kvm);
 }
 EXPORT_SYMBOL_GPL(kvm_can_use_hv_timer);

This method seems more quick and safe, but I have one question: Does this kvm_mwait_in_guest
can guarantee the CPU isolated, in some production environments and usually, MWAIT feature is disabled in host and even guests with isolated CPUs. And also we can set guests kvm_mwait_in_guest true with CPUs just pinned, not isolated.

Thanks

#4 lore raw
Quoted 123 lines · yaoaili [么爱利]
Quoted 39 lines
Quoted 36 lines · Sean Christopherson
Quoted 34 lines · Aili Yao

From: Aili Yao <[email protected]>

When we isolate some pyhiscal cores, We may not use them for kvm guests, We may use them for other purposes like DPDK, or we can make some kvm guests isolated and some not, the global judgement pi_inject_timer is not enough; We may make wrong decisions:

In such a scenario, the guests without isolated cores will not be permitted to use vmx preemption timer, and tscdeadline fastpath also be disabled, both will lead to performance penalty.

So check whether the vcpu->cpu is isolated, if not, don't post timer interrupt.

---
 arch/x86/kvm/lapic.c | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)
arch/x86/kvm/lapic.c index renamed from arch/x86/kvm/lapic.c +0 -0

759952dd1222..72dde5532101 100644

arch/x86/kvm/lapic.c +1 -0
@@ -34,6 +34,7 @@
 #include <asm/delay.h>
 #include <linux/atomic.h>
 #include <linux/jump_label.h>
+#include <linux/sched/isolation.h>
 #include "kvm_cache_regs.h"
 #include "irq.h"
 #include "ioapic.h"
@@ -113,7 +114,8 @@ static inline u32 kvm_x2apic_id(struct kvm_lapic
 *apic)
 
Quoted 6 lines
Quoted 6 lines

arch/x86/kvm/lapic.c static inline u32 kvm_x2apic_id(struct kvm_lapic · line 118

Quoted 4 lines
arch/x86/kvm/lapic.c +2 -1
 static bool kvm_can_post_timer_interrupt(struct kvm_vcpu *vcpu)  {
-     return pi_inject_timer && kvm_vcpu_apicv_active(vcpu);
+     return pi_inject_timer && kvm_vcpu_apicv_active(vcpu) &&
+             !housekeeping_cpu(vcpu->cpu, HK_FLAG_TIMER);

I don't think this is safe, vcpu->cpu will be -1 if the vCPU isn't scheduled in.

Yes, vcpu->cpu is -1 before vcpu create, but in my environments, it didn't trigger this issue. I need to dig more, Thanks!
Maybe I need one valid check here.

Quoted 6 lines
Quoted 5 lines

This also doesn't play nice with the admin forcing pi_inject_timer=1. Not saying there's a reasonable use case for doing that, but it's supported today and this would break that behavior. It would also lead to weird behavior if a vCPU were migrated on/off a housekeeping vCPU. Again, probably not a reasonable use case, but I don't see anything

that would outright prevent that behavior.

Yes, this is not one common operation, But I did do test some scenarios:

1. isolated cpu --> housekeeping cpu;
    isolated guest timer is in housekeeping CPU, for migration, kvm_can_post_timer_interrupt
    will return false, so the timer may be migrated to vcpu->cpu;
    This seems works in my test;

2. isolated --> isolated
Isolated guest timer is in housekeeping cpu, for migration,kvm_can_post_timer_interrupt return true, timer is not migrated
3. housekeeping CPU --> isolated CPU

    non-isolated CPU timer is usually in vcpu->cpu, for migration to isolated, kvm_can_post_timer_interrupt
    will be true,  the timer remain on the same CPU;
    This seems works in my test;
4. housekeeping CPU --> housekeeping CPU
     timer migrated;

It seems this is not an affecting problem;

Quoted 50 lines
Quoted 3 lines

The existing behavior also feels a bit unsafe as pi_inject_timer is writable while KVM is running, though I supposed that's orthogonal to this

discussion.

Quoted 3 lines

Rather than check vcpu->cpu, is there an existing vCPU flag that can be queried, e.g. KVM_HINTS_REALTIME?

How about something like below:

From 67f605120e212384cb3d5788ba8c83f15659503b Mon Sep 17 00:00:00 2001
From: Wanpeng Li <[email protected]>
Date: Tue, 23 Nov 2021 10:36:10 +0800
Subject: [PATCH] KVM: LAPIC: To keep the vCPUs in non-root mode for timer- pi

From: Wanpeng Li <[email protected]>

As commit 0c5f81dad46 (KVM: LAPIC: Inject timer interrupt via posted interrupt) mentioned that the host admin should well tune the guest setup, so that vCPUs are placed on isolated pCPUs, and with several pCPUs surplus for
*busy* housekeeping.
It is better to disable mwait/hlt/pause vmexits to keep the vCPUs in non-root mode. However, we may isolate pCPUs for other purpose like DPDK or we can make some guests isolated and others not, Let's add the checking kvm_mwait_in_guest() to kvm_can_post_timer_interrupt() since we can't benefit from timer posted-interrupt w/o keeping the vCPUs in non-root mode.

---
 arch/x86/kvm/lapic.c | 5 ++---
 1 file changed, 2 insertions(+), 3 deletions(-)
arch/x86/kvm/lapic.c index renamed from arch/x86/kvm/lapic.c +0 -0

759952dd1222..8257566d44c7 100644

arch/x86/kvm/lapic.c +1 -1
@@ -113,14 +113,13 @@ static inline u32 kvm_x2apic_id(struct kvm_lapic
 *apic)
 
 static bool kvm_can_post_timer_interrupt(struct kvm_vcpu *vcpu)  {
-    return pi_inject_timer && kvm_vcpu_apicv_active(vcpu);
+    return pi_inject_timer && kvm_mwait_in_guest(vcpu->kvm) &&
 kvm_vcpu_apicv_active(vcpu);
 }
 
Quoted 11 lines

arch/x86/kvm/lapic.c static inline u32 kvm_x2apic_id(struct kvm_lapic · line 124

Quoted 7 lines
arch/x86/kvm/lapic.c +1 -2
 bool kvm_can_use_hv_timer(struct kvm_vcpu *vcpu)  {
     return kvm_x86_ops.set_hv_timer
-           && !(kvm_mwait_in_guest(vcpu->kvm) ||
-            kvm_can_post_timer_interrupt(vcpu));
+           && !kvm_mwait_in_guest(vcpu->kvm);
 }
 EXPORT_SYMBOL_GPL(kvm_can_use_hv_timer);

This method seems more quick and safe, but I have one question: Does this kvm_mwait_in_guest
can guarantee the CPU isolated, in some production environments and usually, MWAIT feature is disabled in host and even guests with isolated CPUs. And also we can set guests kvm_mwait_in_guest true with CPUs just pinned, not isolated.

You won't benefit from timer posted-interrupt if mwait is not exposed to the guest since you can't keep CPU in non-root mode. kvm_mwait_in_guest() will not guarantee the CPU is isolated, but what's still bothering?

Wanpeng

author
#5 lore raw
Quoted 147 lines · On Tue, 23 Nov 2021 14:24:19 +0800 Wanpeng Li <[email protected]> wrote:
Quoted 130 lines · yaoaili [么爱利]
Quoted 41 lines
Quoted 38 lines · Sean Christopherson
Quoted 36 lines · Aili Yao

From: Aili Yao <[email protected]>

When we isolate some pyhiscal cores, We may not use them for kvm guests, We may use them for other purposes like DPDK, or we can make some kvm guests isolated and some not, the global judgement pi_inject_timer is not enough; We may make wrong decisions:

In such a scenario, the guests without isolated cores will not be permitted to use vmx preemption timer, and tscdeadline fastpath also be disabled, both will lead to performance penalty.

So check whether the vcpu->cpu is isolated, if not, don't post timer interrupt.

---
 arch/x86/kvm/lapic.c | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)
arch/x86/kvm/lapic.c index renamed from arch/x86/kvm/lapic.c +0 -0

759952dd1222..72dde5532101 100644

arch/x86/kvm/lapic.c +1 -0
@@ -34,6 +34,7 @@
 #include <asm/delay.h>
 #include <linux/atomic.h>
 #include <linux/jump_label.h>
+#include <linux/sched/isolation.h>
 #include "kvm_cache_regs.h"
 #include "irq.h"
 #include "ioapic.h"
@@ -113,7 +114,8 @@ static inline u32 kvm_x2apic_id(struct
 kvm_lapic *apic)
 
Quoted 8 lines
Quoted 8 lines

arch/x86/kvm/lapic.c static inline u32 kvm_x2apic_id(struct · line 119

Quoted 5 lines
arch/x86/kvm/lapic.c +2 -1
 static bool kvm_can_post_timer_interrupt(struct kvm_vcpu
 *vcpu)  {
-     return pi_inject_timer && kvm_vcpu_apicv_active(vcpu);
+     return pi_inject_timer && kvm_vcpu_apicv_active(vcpu) &&
+             !housekeeping_cpu(vcpu->cpu, HK_FLAG_TIMER);  

I don't think this is safe, vcpu->cpu will be -1 if the vCPU isn't scheduled in.

Yes, vcpu->cpu is -1 before vcpu create, but in my environments, it didn't trigger this issue. I need to dig more, Thanks! Maybe I need one valid check here.

Quoted 7 lines
Quoted 6 lines

This also doesn't play nice with the admin forcing pi_inject_timer=1. Not saying there's a reasonable use case for doing that, but it's supported today and this would break that behavior. It would also lead to weird behavior if a vCPU were migrated on/off a housekeeping vCPU. Again, probably not a reasonable use case, but I don't see anything

that would outright prevent that behavior.

Yes, this is not one common operation, But I did do test some
scenarios: 1. isolated cpu --> housekeeping cpu;
isolated guest timer is in housekeeping CPU, for migration,
kvm_can_post_timer_interrupt will return false, so the timer may be migrated to vcpu->cpu; This seems works in my test;
2. isolated --> isolated
Isolated guest timer is in housekeeping cpu, for
migration,kvm_can_post_timer_interrupt return true, timer is not migrated 3. housekeeping CPU --> isolated CPU

    non-isolated CPU timer is usually in vcpu->cpu, for migration
to isolated, kvm_can_post_timer_interrupt will be true,  the timer
remain on the same CPU; This seems works in my test;
4. housekeeping CPU --> housekeeping CPU
     timer migrated;

It seems this is not an affecting problem;

Quoted 52 lines
Quoted 4 lines

The existing behavior also feels a bit unsafe as pi_inject_timer is writable while KVM is running, though I supposed that's orthogonal to this

discussion.

Quoted 3 lines

Rather than check vcpu->cpu, is there an existing vCPU flag that can be queried, e.g. KVM_HINTS_REALTIME?

How about something like below:

From 67f605120e212384cb3d5788ba8c83f15659503b Mon Sep 17 00:00:00 2001
From: Wanpeng Li <[email protected]>
Date: Tue, 23 Nov 2021 10:36:10 +0800
Subject: [PATCH] KVM: LAPIC: To keep the vCPUs in non-root mode for timer- pi

From: Wanpeng Li <[email protected]>

As commit 0c5f81dad46 (KVM: LAPIC: Inject timer interrupt via posted interrupt) mentioned that the host admin should well tune the guest setup, so that vCPUs are placed on isolated pCPUs, and with several pCPUs surplus for
*busy* housekeeping.
It is better to disable mwait/hlt/pause vmexits to keep the vCPUs in non-root mode. However, we may isolate pCPUs for other purpose like DPDK or we can make some guests isolated and others not, Let's add the checking kvm_mwait_in_guest() to kvm_can_post_timer_interrupt() since we can't benefit from timer posted-interrupt w/o keeping the vCPUs in non-root mode.

---
 arch/x86/kvm/lapic.c | 5 ++---
 1 file changed, 2 insertions(+), 3 deletions(-)
arch/x86/kvm/lapic.c index renamed from arch/x86/kvm/lapic.c +0 -0

759952dd1222..8257566d44c7 100644

arch/x86/kvm/lapic.c +1 -1
@@ -113,14 +113,13 @@ static inline u32 kvm_x2apic_id(struct
 kvm_lapic *apic)
 
 static bool kvm_can_post_timer_interrupt(struct kvm_vcpu *vcpu)
 {
-    return pi_inject_timer && kvm_vcpu_apicv_active(vcpu);
+    return pi_inject_timer && kvm_mwait_in_guest(vcpu->kvm) &&
 kvm_vcpu_apicv_active(vcpu);
 }
 
Quoted 14 lines

arch/x86/kvm/lapic.c static inline u32 kvm_x2apic_id(struct · line 125

Quoted 7 lines
arch/x86/kvm/lapic.c +1 -2
 bool kvm_can_use_hv_timer(struct kvm_vcpu *vcpu)  {
     return kvm_x86_ops.set_hv_timer
-           && !(kvm_mwait_in_guest(vcpu->kvm) ||
-            kvm_can_post_timer_interrupt(vcpu));
+           && !kvm_mwait_in_guest(vcpu->kvm);
 }
 EXPORT_SYMBOL_GPL(kvm_can_use_hv_timer);  

This method seems more quick and safe, but I have one question:
Does this kvm_mwait_in_guest can guarantee the CPU isolated, in some production environments and usually, MWAIT feature is disabled in host and even guests with isolated CPUs. And also we can set guests kvm_mwait_in_guest true with CPUs just pinned, not isolated.

Quoted 4 lines

You won't benefit from timer posted-interrupt if mwait is not exposed to the guest since you can't keep CPU in non-root mode. kvm_mwait_in_guest() will not guarantee the CPU is isolated, but what's still bothering?

Sorry, Did I miss some thing?

What in my mind: MWait may be disabled in bios, so host will use halt instruction as one replacement for idle operation, in such a configuration, Mwait in guest will also be disabled even if you try to set kvm_mwait_in_guest true; As a result, halt,pause may not exit the guest, so the post interrupt still counts?

For current code, We can migrate guest between isolated and housekeeping or we can change the cpu pinning on the fly, we allow this even the operation is not usually used, right?

Thanks!
Aili Yao

#6 lore raw
Quoted 156 lines · Aili Yao
Quoted 147 lines · On Tue, 23 Nov 2021 14:24:19 +0800 Wanpeng Li <[email protected]> wrote:
Quoted 130 lines · yaoaili [么爱利]
Quoted 41 lines
Quoted 38 lines · Sean Christopherson
Quoted 36 lines · Aili Yao

From: Aili Yao <[email protected]>

When we isolate some pyhiscal cores, We may not use them for kvm guests, We may use them for other purposes like DPDK, or we can make some kvm guests isolated and some not, the global judgement pi_inject_timer is not enough; We may make wrong decisions:

In such a scenario, the guests without isolated cores will not be permitted to use vmx preemption timer, and tscdeadline fastpath also be disabled, both will lead to performance penalty.

So check whether the vcpu->cpu is isolated, if not, don't post timer interrupt.

---
 arch/x86/kvm/lapic.c | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)
arch/x86/kvm/lapic.c index renamed from arch/x86/kvm/lapic.c +0 -0

759952dd1222..72dde5532101 100644

arch/x86/kvm/lapic.c +1 -0
@@ -34,6 +34,7 @@
 #include <asm/delay.h>
 #include <linux/atomic.h>
 #include <linux/jump_label.h>
+#include <linux/sched/isolation.h>
 #include "kvm_cache_regs.h"
 #include "irq.h"
 #include "ioapic.h"
@@ -113,7 +114,8 @@ static inline u32 kvm_x2apic_id(struct
 kvm_lapic *apic)
 
Quoted 8 lines
Quoted 8 lines

arch/x86/kvm/lapic.c static inline u32 kvm_x2apic_id(struct · line 119

Quoted 5 lines
arch/x86/kvm/lapic.c +2 -1
 static bool kvm_can_post_timer_interrupt(struct kvm_vcpu
 *vcpu)  {
-     return pi_inject_timer && kvm_vcpu_apicv_active(vcpu);
+     return pi_inject_timer && kvm_vcpu_apicv_active(vcpu) &&
+             !housekeeping_cpu(vcpu->cpu, HK_FLAG_TIMER);

I don't think this is safe, vcpu->cpu will be -1 if the vCPU isn't scheduled in.

Yes, vcpu->cpu is -1 before vcpu create, but in my environments, it didn't trigger this issue. I need to dig more, Thanks! Maybe I need one valid check here.

Quoted 7 lines
Quoted 6 lines

This also doesn't play nice with the admin forcing pi_inject_timer=1. Not saying there's a reasonable use case for doing that, but it's supported today and this would break that behavior. It would also lead to weird behavior if a vCPU were migrated on/off a housekeeping vCPU. Again, probably not a reasonable use case, but I don't see anything

that would outright prevent that behavior.

Yes, this is not one common operation, But I did do test some
scenarios: 1. isolated cpu --> housekeeping cpu;
isolated guest timer is in housekeeping CPU, for migration,
kvm_can_post_timer_interrupt will return false, so the timer may be migrated to vcpu->cpu; This seems works in my test;
2. isolated --> isolated
Isolated guest timer is in housekeeping cpu, for
migration,kvm_can_post_timer_interrupt return true, timer is not migrated 3. housekeeping CPU --> isolated CPU

    non-isolated CPU timer is usually in vcpu->cpu, for migration
to isolated, kvm_can_post_timer_interrupt will be true,  the timer
remain on the same CPU; This seems works in my test;
4. housekeeping CPU --> housekeeping CPU
     timer migrated;

It seems this is not an affecting problem;

Quoted 52 lines
Quoted 4 lines

The existing behavior also feels a bit unsafe as pi_inject_timer is writable while KVM is running, though I supposed that's orthogonal to this

discussion.

Quoted 3 lines

Rather than check vcpu->cpu, is there an existing vCPU flag that can be queried, e.g. KVM_HINTS_REALTIME?

How about something like below:

From 67f605120e212384cb3d5788ba8c83f15659503b Mon Sep 17 00:00:00 2001
From: Wanpeng Li <[email protected]>
Date: Tue, 23 Nov 2021 10:36:10 +0800
Subject: [PATCH] KVM: LAPIC: To keep the vCPUs in non-root mode for timer- pi

From: Wanpeng Li <[email protected]>

As commit 0c5f81dad46 (KVM: LAPIC: Inject timer interrupt via posted interrupt) mentioned that the host admin should well tune the guest setup, so that vCPUs are placed on isolated pCPUs, and with several pCPUs surplus for
*busy* housekeeping.
It is better to disable mwait/hlt/pause vmexits to keep the vCPUs in non-root mode. However, we may isolate pCPUs for other purpose like DPDK or we can make some guests isolated and others not, Let's add the checking kvm_mwait_in_guest() to kvm_can_post_timer_interrupt() since we can't benefit from timer posted-interrupt w/o keeping the vCPUs in non-root mode.

---
 arch/x86/kvm/lapic.c | 5 ++---
 1 file changed, 2 insertions(+), 3 deletions(-)
arch/x86/kvm/lapic.c index renamed from arch/x86/kvm/lapic.c +0 -0

759952dd1222..8257566d44c7 100644

arch/x86/kvm/lapic.c +1 -1
@@ -113,14 +113,13 @@ static inline u32 kvm_x2apic_id(struct
 kvm_lapic *apic)
 
 static bool kvm_can_post_timer_interrupt(struct kvm_vcpu *vcpu)
 {
-    return pi_inject_timer && kvm_vcpu_apicv_active(vcpu);
+    return pi_inject_timer && kvm_mwait_in_guest(vcpu->kvm) &&
 kvm_vcpu_apicv_active(vcpu);
 }
 
Quoted 14 lines

arch/x86/kvm/lapic.c static inline u32 kvm_x2apic_id(struct · line 125

Quoted 7 lines
arch/x86/kvm/lapic.c +1 -2
 bool kvm_can_use_hv_timer(struct kvm_vcpu *vcpu)  {
     return kvm_x86_ops.set_hv_timer
-           && !(kvm_mwait_in_guest(vcpu->kvm) ||
-            kvm_can_post_timer_interrupt(vcpu));
+           && !kvm_mwait_in_guest(vcpu->kvm);
 }
 EXPORT_SYMBOL_GPL(kvm_can_use_hv_timer);

This method seems more quick and safe, but I have one question:
Does this kvm_mwait_in_guest can guarantee the CPU isolated, in some production environments and usually, MWAIT feature is disabled in host and even guests with isolated CPUs. And also we can set guests kvm_mwait_in_guest true with CPUs just pinned, not isolated.

Quoted 4 lines

You won't benefit from timer posted-interrupt if mwait is not exposed to the guest since you can't keep CPU in non-root mode. kvm_mwait_in_guest() will not guarantee the CPU is isolated, but what's still bothering?

Quoted 7 lines

Sorry, Did I miss some thing?

What in my mind: MWait may be disabled in bios, so host will use halt instruction as one replacement for idle operation, in such a configuration, Mwait in guest will also be disabled even if you try to set kvm_mwait_in_guest true; As a result, halt,pause may not exit the guest, so the post interrupt still counts?

I prefer to expose mwait/hlt/pause to the guest simultaneously, you don't need the ultra schedule latency/performance if you aren't exposing mwait. Then why do you care about latency from the timer?

Quoted 4 lines

For current code, We can migrate guest between isolated and housekeeping or we can change the cpu pinning on the fly, we allow this even the operation is not usually used, right?

My patch will not prevent using vmx preemption timer or tscdeadline fastpath.

Wanpeng

author
#7 lore raw
In reply to Sean Christopherson
Quoted 34 lines · On Mon, 22 Nov 2021 19:13:02 +0000 Sean Christopherson <[email protected]>…
Quoted 33 lines · Aili Yao

From: Aili Yao <[email protected]>

When we isolate some pyhiscal cores, We may not use them for kvm guests, We may use them for other purposes like DPDK, or we can make some kvm guests isolated and some not, the global judgement pi_inject_timer is not enough; We may make wrong decisions:

In such a scenario, the guests without isolated cores will not be permitted to use vmx preemption timer, and tscdeadline fastpath also be disabled, both will lead to performance penalty.

So check whether the vcpu->cpu is isolated, if not, don't post timer interrupt.

---
 arch/x86/kvm/lapic.c | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)
arch/x86/kvm/lapic.c +1 -0
index 759952dd1222..72dde5532101 100644
@@ -34,6 +34,7 @@
 #include <asm/delay.h>
 #include <linux/atomic.h>
 #include <linux/jump_label.h>
+#include <linux/sched/isolation.h>
 #include "kvm_cache_regs.h"
 #include "irq.h"
 #include "ioapic.h"
@@ -113,7 +114,8 @@ static inline u32 kvm_x2apic_id(struct
 kvm_lapic *apic) 
Quoted 8 lines

arch/x86/kvm/lapic.c static inline u32 kvm_x2apic_id(struct kvm_lapic *apic) · line 118

Quoted 5 lines
arch/x86/kvm/lapic.c +2 -1
 static bool kvm_can_post_timer_interrupt(struct kvm_vcpu *vcpu)
 {
-	return pi_inject_timer && kvm_vcpu_apicv_active(vcpu);
+	return pi_inject_timer && kvm_vcpu_apicv_active(vcpu) &&
+		!housekeeping_cpu(vcpu->cpu, HK_FLAG_TIMER);  

I don't think this is safe, vcpu->cpu will be -1 if the vCPU isn't scheduled in.

I checked this, It seems we will set vcpu->cpu to a valid value when we create vcpu( kvm_vm_ioctl_create_vcpu()), only after that we can configure lapic through vcpu fd and start the timer, this may not be one real problem.

Currently, the patch seems work as expected in my test, maybe one possible candidate for the issue listed above.

Thanks

Quoted 21 lines

This also doesn't play nice with the admin forcing pi_inject_timer=1. Not saying there's a reasonable use case for doing that, but it's supported today and this would break that behavior. It would also lead to weird behavior if a vCPU were migrated on/off a housekeeping vCPU. Again, probably not a reasonable use case, but I don't see anything that would outright prevent that behavior.

The existing behavior also feels a bit unsafe as pi_inject_timer is writable while KVM is running, though I supposed that's orthogonal to this discussion.

Rather than check vcpu->cpu, is there an existing vCPU flag that can be queried, e.g. KVM_HINTS_REALTIME?

Quoted 6 lines
arch/x86/kvm/lapic.c +0 -0
 }
 
 bool kvm_can_use_hv_timer(struct kvm_vcpu *vcpu)
Signature · 1 line
2.25.1
author
#8 lore raw
Quoted 46 lines · On Tue, 23 Nov 2021 16:18:34 +0800 Aili Yao <[email protected]> wrote:
Quoted 34 lines · On Mon, 22 Nov 2021 19:13:02 +0000 Sean Christopherson <[email protected]>…
Quoted 33 lines · Aili Yao

From: Aili Yao <[email protected]>

When we isolate some pyhiscal cores, We may not use them for kvm guests, We may use them for other purposes like DPDK, or we can make some kvm guests isolated and some not, the global judgement pi_inject_timer is not enough; We may make wrong decisions:

In such a scenario, the guests without isolated cores will not be permitted to use vmx preemption timer, and tscdeadline fastpath also be disabled, both will lead to performance penalty.

So check whether the vcpu->cpu is isolated, if not, don't post timer interrupt.

---
 arch/x86/kvm/lapic.c | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)
arch/x86/kvm/lapic.c +1 -0
index 759952dd1222..72dde5532101 100644
@@ -34,6 +34,7 @@
 #include <asm/delay.h>
 #include <linux/atomic.h>
 #include <linux/jump_label.h>
+#include <linux/sched/isolation.h>
 #include "kvm_cache_regs.h"
 #include "irq.h"
 #include "ioapic.h"
@@ -113,7 +114,8 @@ static inline u32 kvm_x2apic_id(struct
 kvm_lapic *apic) 
Quoted 8 lines

arch/x86/kvm/lapic.c static inline u32 kvm_x2apic_id(struct kvm_lapic *apic) · line 118

Quoted 5 lines
arch/x86/kvm/lapic.c +2 -1
 static bool kvm_can_post_timer_interrupt(struct kvm_vcpu *vcpu)
 {
-	return pi_inject_timer && kvm_vcpu_apicv_active(vcpu);
+	return pi_inject_timer && kvm_vcpu_apicv_active(vcpu) &&
+		!housekeeping_cpu(vcpu->cpu, HK_FLAG_TIMER);  

I don't think this is safe, vcpu->cpu will be -1 if the vCPU isn't scheduled in.

Quoted 2 lines

I checked this, It seems we will set vcpu->cpu to a valid value when we create vcpu( kvm_vm_ioctl_create_vcpu()),

Really Sorry, My code base is too old; This vcpu->cpu assignment has been deleted in latest code, And this housekeeping_cpu() check will result problem.

Thanks!

Quoted 31 lines

only after that we can
configure lapic through vcpu fd and start the timer, this may not be one real problem.

Currently, the patch seems work as expected in my test, maybe one possible candidate for the issue listed above.

Thanks

Quoted 21 lines

This also doesn't play nice with the admin forcing pi_inject_timer=1. Not saying there's a reasonable use case for doing that, but it's supported today and this would break that behavior. It would also lead to weird behavior if a vCPU were migrated on/off a housekeeping vCPU. Again, probably not a reasonable use case, but I don't see anything that would outright prevent that behavior.

The existing behavior also feels a bit unsafe as pi_inject_timer is writable while KVM is running, though I supposed that's orthogonal to this discussion.

Rather than check vcpu->cpu, is there an existing vCPU flag that can be queried, e.g. KVM_HINTS_REALTIME?

Quoted 6 lines
arch/x86/kvm/lapic.c +0 -0
 }
 
 bool kvm_can_use_hv_timer(struct kvm_vcpu *vcpu)
Signature · 1 line
2.25.1