KVM: s390: enable kvm_vpcu_kick/wake_up
Subsystems (10)
cover KVM: s390: enable kvm_vpcu_kick/wake_up
David, I have certainly forgotten something, but I think this should be good enough for all non-special cases that require specific CPUs outside of the guest context and only need to handle a request fast.
Christian Borntraeger (2):
s390/smp: export smp_send_reschedule
KVM: enable kvm_vcpu_kick/wake_up for s390
arch/s390/kernel/smp.c | 1 + arch/s390/kvm/kvm-s390.c | 12 ++++++++++-- virt/kvm/kvm_main.c | 2 -- 3 files changed, 11 insertions(+), 4 deletions(-)
Signature · 1 line
2.7.4
PATCH 1/2 s390/smp: export smp_send_reschedule
needed by KVM common code.
--- arch/s390/kernel/smp.c | 1 + 1 file changed, 1 insertion(+)
arch/s390/kernel/smp.c +1 -0
@@ -513,6 +513,7 @@ void smp_send_reschedule(int cpu) {pcpu_ec_call (pcpu_devices + cpu, ec_schedule); } +EXPORT_SYMBOL_GPL (smp_send_reschedule);/* * parameter area for the set/clear control bit callbacks
Signature · 1 line
2.7.4
PATCH 1/1 KVM: add kvm_arch_cpu_kick
2017-02-17 14:10+0100, Christian Borntraeger:
Quoted 12 lines
needed by KVM common code.
Signed-off-by Christian Borntraeger <[email protected]>--- arch/s390/kernel/smp.c | 1 + 1 file changed, 1 insertion(+)arch/s390/kernel/smp.c +0
-0
arch/s390/kernel/smp.c void smp_send_reschedule(int cpu) · line 516
Quoted 4 lines
arch/s390/kernel/smp.c +1
-0@@ -513,6 +513,7 @@ void smp_send_reschedule(int cpu) {pcpu_ec_call (pcpu_devices + cpu, ec_schedule); } +EXPORT_SYMBOL_GPL (smp_send_reschedule);
s390 doesn't want to use smp_send_reschedule() so I think the patch at the bottom is going in a better direction.
Btw. I think that we shouldn't be using smp_send_reschedule() for forcing VCPUs out of guest mode anyway -- the task we want to run is already scheduled and we don't want the schduler to do anything.
We could use smp_call_function() with an empty function instead, but that also has high overhead ... allocating a new IPI seems best.
But optimizing the current code is premature. :)
---8<---
s390 has a different mechanism from bringing the VCPU out of guest mode. Make the kick arch-specific.
--- arch/arm/kvm/arm.c | 5 +++++ arch/mips/kvm/mips.c | 5 +++++ arch/powerpc/kvm/powerpc.c | 5 +++++ arch/s390/kvm/kvm-s390.c | 6 ++++++ arch/x86/kvm/x86.c | 5 +++++ include/linux/kvm_host.h | 1 + virt/kvm/kvm_main.c | 2 +- 7 files changed, 28 insertions(+), 1 deletion(-)
arch/arm/kvm/arm.c +5 -0
@@ -97,6 +97,11 @@ int kvm_arch_vcpu_should_kick(struct kvm_vcpu *vcpu)return kvm_vcpu_exiting_guest_mode (vcpu) ==IN_GUEST_MODE ; } +void kvm_arch_cpu_kick (int cpu) +{ +smp_send_reschedule (cpu); +} +int kvm_arch_hardware_setup (void ) {return 0 ;
arch/mips/kvm/mips.c +5 -0
@@ -78,6 +78,11 @@ int kvm_arch_vcpu_should_kick(struct kvm_vcpu *vcpu)return 1 ; } +void kvm_arch_cpu_kick (int cpu) +{ +smp_send_reschedule (cpu); +} +int kvm_arch_hardware_enable (void ) {return 0 ;
arch/powerpc/kvm/powerpc.c +5 -0
@@ -60,6 +60,11 @@ int kvm_arch_vcpu_should_kick(struct kvm_vcpu *vcpu)return 1 ; } +void kvm_arch_cpu_kick (int cpu) +{ +smp_send_reschedule (cpu); +} +/* * Common checks before entering the guest world. Call with interrupts * disabled.
arch/s390/kvm/kvm-s390.c +6 -0
@@ -2131,6 +2131,12 @@ int kvm_arch_vcpu_should_kick(struct kvm_vcpu *vcpu)return 0 ; } +void kvm_arch_cpu_kick (int cpu) +{ +/* TODO: implement kicking with SIE */ +BUG (); +} +static int kvm_arch_vcpu_ioctl_get_one_reg (struct kvm_vcpu *vcpu,struct kvm_one_reg *reg) {
arch/x86/kvm/x86.c +5 -0
@@ -8405,6 +8405,11 @@ int kvm_arch_vcpu_should_kick(struct kvm_vcpu *vcpu)return kvm_vcpu_exiting_guest_mode (vcpu) ==IN_GUEST_MODE ; } +void kvm_arch_cpu_kick (int cpu) +{ +smp_send_reschedule (cpu); +} +int kvm_arch_interrupt_allowed (struct kvm_vcpu *vcpu) {return kvm_x86_ops->interrupt_allowed (vcpu);
include/linux/kvm_host.h +1 -0
@@ -766,6 +766,7 @@ void kvm_arch_hardware_unsetup(void);void kvm_arch_check_processor_compat (void *rtn);int kvm_arch_vcpu_runnable (struct kvm_vcpu *vcpu);int kvm_arch_vcpu_should_kick (struct kvm_vcpu *vcpu); +void kvm_arch_cpu_kick (int cpu);void *kvm_kvzalloc (unsigned long size);
virt/kvm/kvm_main.c +1 -1
@@ -2240,7 +2240,7 @@ void kvm_vcpu_kick(struct kvm_vcpu *vcpu) me =get_cpu ();if (cpu != me && (unsigned )cpu < nr_cpu_ids &&cpu_online (cpu))if (kvm_arch_vcpu_should_kick (vcpu)) -smp_send_reschedule (cpu); +kvm_arch_cpu_kick (cpu);put_cpu (); }EXPORT_SYMBOL_GPL (kvm_vcpu_kick);
Signature · 1 line
2.11.1
Re: [PATCH] KVM: add kvm_arch_cpu_kick
Quoted 25 lines · Radim Krčmář
2017-02-17 14:10+0100, Christian Borntraeger:
Quoted 12 lines
needed by KVM common code.
Signed-off-by Christian Borntraeger <[email protected]>--- arch/s390/kernel/smp.c | 1 + 1 file changed, 1 insertion(+)arch/s390/kernel/smp.c +0
-0arch/s390/kernel/smp.c void smp_send_reschedule(int cpu) · line 516
Quoted 4 lines
arch/s390/kernel/smp.c +1
-0@@ -513,6 +513,7 @@ void smp_send_reschedule(int cpu) {pcpu_ec_call (pcpu_devices + cpu, ec_schedule); } +EXPORT_SYMBOL_GPL (smp_send_reschedule);s390 doesn't want to use
smp_send_reschedule()so I think the patch at the bottom is going in a better direction.Btw. I think that we shouldn't be using
smp_send_reschedule()for forcing VCPUs out of guest mode anyway -- the task we want to run is already scheduled and we don't want the schduler to do anything.
Quoted 8 lines
We could use
smp_call_function()with an empty function instead, but that also has high overhead ... allocating a new IPI seems best.But optimizing the current code is premature. :)
---8<---
s390 has a different mechanism from bringing the VCPU out of guest mode. Make the kick arch-specific.
Looks good. The kick does not have to be synchronous and its ok if we reenter the guest as long as we execute the request in a timely manner, correct?
e.g.
- kick vcpu
- vcpu enters SIE
- vcpu exits SIE immediately
- vcpu handles request
- vcpu enters SIE
would be perfectly fine?
In that case the s390 implementation could be pretty simple.
Quoted 119 lines
Signed-off-by Radim Krčmář <[email protected]>--- arch/arm/kvm/arm.c | 5 +++++ arch/mips/kvm/mips.c | 5 +++++ arch/powerpc/kvm/powerpc.c | 5 +++++ arch/s390/kvm/kvm-s390.c | 6 ++++++ arch/x86/kvm/x86.c | 5 +++++ include/linux/kvm_host.h | 1 + virt/kvm/kvm_main.c | 2 +- 7 files changed, 28 insertions(+), 1 deletion(-)arch/arm/kvm/arm.c +5
-0@@ -97,6 +97,11 @@ int kvm_arch_vcpu_should_kick(struct kvm_vcpu *vcpu)return kvm_vcpu_exiting_guest_mode (vcpu) ==IN_GUEST_MODE ; } +void kvm_arch_cpu_kick (int cpu) +{ +smp_send_reschedule (cpu); +} +int kvm_arch_hardware_setup (void ) {return 0 ;arch/mips/kvm/mips.c +5
-0@@ -78,6 +78,11 @@ int kvm_arch_vcpu_should_kick(struct kvm_vcpu *vcpu)return 1 ; } +void kvm_arch_cpu_kick (int cpu) +{ +smp_send_reschedule (cpu); +} +int kvm_arch_hardware_enable (void ) {return 0 ;arch/powerpc/kvm/powerpc.c +5
-0@@ -60,6 +60,11 @@ int kvm_arch_vcpu_should_kick(struct kvm_vcpu *vcpu)return 1 ; } +void kvm_arch_cpu_kick (int cpu) +{ +smp_send_reschedule (cpu); +} +/* * Common checks before entering the guest world. Call with interrupts * disabled. arch/s390/kvm/kvm-s390.c +6
-0@@ -2131,6 +2131,12 @@ int kvm_arch_vcpu_should_kick(struct kvm_vcpu *vcpu)return 0 ; } +void kvm_arch_cpu_kick (int cpu) +{ +/* TODO: implement kicking with SIE */ +BUG (); +} +static int kvm_arch_vcpu_ioctl_get_one_reg (struct kvm_vcpu *vcpu,struct kvm_one_reg *reg) {arch/x86/kvm/x86.c +5
-0@@ -8405,6 +8405,11 @@ int kvm_arch_vcpu_should_kick(struct kvm_vcpu *vcpu)return kvm_vcpu_exiting_guest_mode (vcpu) ==IN_GUEST_MODE ; } +void kvm_arch_cpu_kick (int cpu) +{ +smp_send_reschedule (cpu); +} +int kvm_arch_interrupt_allowed (struct kvm_vcpu *vcpu) {return kvm_x86_ops->interrupt_allowed (vcpu);include/linux/kvm_host.h +1
-0@@ -766,6 +766,7 @@ void kvm_arch_hardware_unsetup(void);void kvm_arch_check_processor_compat (void *rtn);int kvm_arch_vcpu_runnable (struct kvm_vcpu *vcpu);int kvm_arch_vcpu_should_kick (struct kvm_vcpu *vcpu); +void kvm_arch_cpu_kick (int cpu);void *kvm_kvzalloc (unsigned long size);virt/kvm/kvm_main.c +1
-1@@ -2240,7 +2240,7 @@ void kvm_vcpu_kick(struct kvm_vcpu *vcpu) me =get_cpu ();if (cpu != me && (unsigned )cpu < nr_cpu_ids &&cpu_online (cpu))if (kvm_arch_vcpu_should_kick (vcpu)) -smp_send_reschedule (cpu); +kvm_arch_cpu_kick (cpu);put_cpu (); }EXPORT_SYMBOL_GPL (kvm_vcpu_kick);
Paolo Bonzini
maintainerRe: [PATCH] KVM: add kvm_arch_cpu_kick
Quoted 4 lines · Christian Borntraeger
Looks good. The kick does not have to be synchronous and its ok if we reenter the guest as long as we execute the request in a timely manner, correct?
Quoted 8 lines
e.g.
- kick vcpu
- vcpu enters SIE
- vcpu exits SIE immediately
- vcpu handles request
- vcpu enters SIEwould be perfectly fine?
Yes, it would. There's some parallel with QEMU's qemu_cpu_kick, where the signal would be processed immediately after entering KVM_RUN.
Paolo
Re: [PATCH] KVM: add kvm_arch_cpu_kick
Quoted 14 lines · Paolo Bonzini
Quoted 11 lines · Christian Borntraeger
Looks good. The kick does not have to be synchronous and its ok if we reenter the guest as long as we execute the request in a timely manner, correct?
e.g.
- kick vcpu
- vcpu enters SIE
- vcpu exits SIE immediately
- vcpu handles request
- vcpu enters SIE
Quoted 4 lines
Quoted 1 line
would be perfectly fine?
Yes, it would. There's some parallel with QEMU's
qemu_cpu_kick, where the signal would be processed immediately after enteringKVM_RUN.
Something like
---snip-----
struct kvm_s390_sie_block *scb = READ_ONCE(vcpu->arch.vsie_block);
atomic_or(CPUSTAT_STOP_INT, &vcpu->arch.sie_block->cpuflags);
if (scb)
atomic_or(CPUSTAT_STOP_INT, &scb->cpuflags);---snip-----
or
---snip-----
atomic_or(CPUSTAT_STOP_INT, &vcpu->arch.sie_block->cpuflags); kvm_s390_vsie_kick(vcpu);
---snip-----
should be enough then. The code will either delete that stop request when processing interrupts, but then the requests will be handled afterwards, or we enter the guest once, exit and process then the requests in the next loop iteration.
As I am on my way into the weekend this needs double checking.
Re: [PATCH] KVM: add kvm_arch_cpu_kick
Quoted 12 lines
Quoted 3 lines
Yes, it would. There's some parallel with QEMU's
qemu_cpu_kick, where the signal would be processed immediately after enteringKVM_RUN.Something like
---snip-----
struct kvm_s390_sie_block *scb = READ_ONCE(vcpu->arch.vsie_block); atomic_or(CPUSTAT_STOP_INT, &vcpu->arch.sie_block->cpuflags); if (scb) atomic_or(CPUSTAT_STOP_INT, &scb->cpuflags);
Quoted 7 lines
---snip-----
or
---snip-----atomic_or(CPUSTAT_STOP_INT, &vcpu->arch.sie_block->cpuflags); kvm_s390_vsie_kick(vcpu);---snip-----
I'd go for the latter one. Keep the vsie stuff isolated. Please note that the vsie loop currently doesn't look for vcpu->requests yet. (I left it out because its just racy either way and all current requests don't require to be processes immediately - and it doesn't harm).
Quoted 5 lines
should be enough then. The code will either delete that stop request when processing interrupts, but then the requests will be handled afterwards, or we enter the guest once, exit and process then the requests in the next loop iteration.
Signature · 3 lines
Thanks, David
Re: [PATCH] KVM: add kvm_arch_cpu_kick
Quoted 14 lines · David Hildenbrand
Quoted 14 lines
Quoted 3 lines
Yes, it would. There's some parallel with QEMU's
qemu_cpu_kick, where the signal would be processed immediately after enteringKVM_RUN.Something like
---snip-----
struct kvm_s390_sie_block *scb = READ_ONCE(vcpu->arch.vsie_block); atomic_or(CPUSTAT_STOP_INT, &vcpu->arch.sie_block->cpuflags); if (scb) atomic_or(CPUSTAT_STOP_INT, &scb->cpuflags);---snip-----
Quoted 7 lines
Quoted 5 lines
or
---snip-----atomic_or(CPUSTAT_STOP_INT, &vcpu->arch.sie_block->cpuflags); kvm_s390_vsie_kick(vcpu);---snip-----
I'd go for the latter one. Keep the vsie stuff isolated. Please note
Yes makes sense.
Radim, if you go with this patch something like this can be used as the s390 variant of kvm_arch_cpu_kick:
---snip---
/*
* The stop indication is reset in the interrupt code. As the CPU
* loop handles requests after interrupts, we will
* a: miss the request handler and enter the guest, but then the
* stop request will exit the CPU and handle the request in the next
* round or
* b: handle the request directly before entering the guest */
atomic_or(CPUSTAT_STOP_INT, &vcpu->arch.sie_block->cpuflags); kvm_s390_vsie_kick(vcpu);
---snip---
feel free to add that to your patch. I can also send a fixup patch later on if you prefer that.
Christian
Re: [PATCH] KVM: add kvm_arch_cpu_kick
Am 20.02.2017 um 12:12 schrieb Christian Borntraeger:
Quoted 40 lines · David Hildenbrand
Quoted 14 lines
Quoted 14 lines
Quoted 3 lines
Yes, it would. There's some parallel with QEMU's
qemu_cpu_kick, where the signal would be processed immediately after enteringKVM_RUN.Something like
---snip-----
struct kvm_s390_sie_block *scb = READ_ONCE(vcpu->arch.vsie_block); atomic_or(CPUSTAT_STOP_INT, &vcpu->arch.sie_block->cpuflags); if (scb) atomic_or(CPUSTAT_STOP_INT, &scb->cpuflags);---snip-----
Quoted 7 lines
Quoted 5 lines
or
---snip-----atomic_or(CPUSTAT_STOP_INT, &vcpu->arch.sie_block->cpuflags); kvm_s390_vsie_kick(vcpu);---snip-----
I'd go for the latter one. Keep the vsie stuff isolated. Please note
Yes makes sense.
Radim, if you go with this patch something like this can be used as the s390 variant of
kvm_arch_cpu_kick:---snip---
/*
* The stop indication is reset in the interrupt code. As the CPU
* loop handles requests after interrupts, we will
* a: miss the request handler and enter the guest, but then the
* stop request will exit the CPU and handle the request in the next
* round or
* b: handle the request directly before entering the guest */
atomic_or(CPUSTAT_STOP_INT, &vcpu->arch.sie_block->cpuflags);kvm_s390_vsie_kick(vcpu);
Quoted 3 lines
---snip---
feel free to add that to your patch. I can also send a fixup patch later on if you prefer that.
kvm_arch_vcpu_should_kick() then also has to be changed to return 1 for now.
An interesting thing to note is how vcpu->cpu is used.
Again, as s390x can preempt just before entering the guest, vcpu_kick() might see vcpu->cpu = -1. Therefore, kvm_arch_vcpu_should_kick() won't even be called. So our cpu might go into guest mode and stay there longer than expected (as we won't kick it).
On x86, it is the following way:
If vcpu->cpu is -1, no need to kick the VCPU. It will check for requests when preemption is disabled, therefore when rescheduled.
If vcpu->cpu is set, kvm_arch_vcpu_should_kick() remembers if the VCPU has already been kicked while in the critical section. It will get kicked by smp resched as soon as entering guest mode.
So here, disabled preemption + checks in the section with disabled preemption (for requests and EXITING_GUEST_MODE) make sure that the guest will leave guest mode and process requests in a timely fashion.
On s390x, this is not 100% true. vcpu->cpu cannot be used as an indicator whether a kick is necessary. Either that is ok for now, or the vcpu->cpu != -1 check has to be disabled for s390x, e.g. by moving the check into kvm_arch_vcpu_should_kick().
Signature · 3 lines
Thanks, David
Re: [PATCH] KVM: add kvm_arch_cpu_kick
2017-02-20 12:35+0100, David Hildenbrand:
Quoted 63 lines
Am 20.02.2017 um 12:12 schrieb Christian Borntraeger:
Quoted 40 lines · David Hildenbrand
Quoted 14 lines
Quoted 14 lines
Quoted 3 lines
Yes, it would. There's some parallel with QEMU's
qemu_cpu_kick, where the signal would be processed immediately after enteringKVM_RUN.Something like
---snip-----
struct kvm_s390_sie_block *scb = READ_ONCE(vcpu->arch.vsie_block); atomic_or(CPUSTAT_STOP_INT, &vcpu->arch.sie_block->cpuflags); if (scb) atomic_or(CPUSTAT_STOP_INT, &scb->cpuflags);---snip-----
Quoted 7 lines
Quoted 5 lines
or
---snip-----atomic_or(CPUSTAT_STOP_INT, &vcpu->arch.sie_block->cpuflags); kvm_s390_vsie_kick(vcpu);---snip-----
I'd go for the latter one. Keep the vsie stuff isolated. Please note
Yes makes sense.
Radim, if you go with this patch something like this can be used as the s390 variant of
kvm_arch_cpu_kick:---snip---
/*
* The stop indication is reset in the interrupt code. As the CPU
* loop handles requests after interrupts, we will
* a: miss the request handler and enter the guest, but then the
* stop request will exit the CPU and handle the request in the next
* round or
* b: handle the request directly before entering the guest */
atomic_or(CPUSTAT_STOP_INT, &vcpu->arch.sie_block->cpuflags);kvm_s390_vsie_kick(vcpu);Quoted 3 lines
---snip---
feel free to add that to your patch. I can also send a fixup patch later on if you prefer that.
kvm_arch_vcpu_should_kick()then also has to be changed to return 1 for now.An interesting thing to note is how
vcpu->cpuis used.Again, as s390x can preempt just before entering the guest,
vcpu_kick()might seevcpu->cpu= -1. Therefore,kvm_arch_vcpu_should_kick()won't even be called. So our cpu might go into guest mode and stay there longer than expected (as we won't kick it).On x86, it is the following way:
If
vcpu->cpuis -1, no need to kick the VCPU. It will check for requests when preemption is disabled, therefore when rescheduled.If
vcpu->cpuis set,kvm_arch_vcpu_should_kick()remembers if the VCPU has already been kicked while in the critical section. It will get kicked by smp resched as soon as entering guest mode.
Quoted 8 lines
So here, disabled preemption + checks in the section with disabled preemption (for requests and
EXITING_GUEST_MODE) make sure that the guest will leave guest mode and process requests in a timely fashion.On s390x, this is not 100% true.
vcpu->cpucannot be used as an indicator whether a kick is necessary. Either that is ok for now, or thevcpu->cpu!= -1 check has to be disabled for s390x, e.g. by moving the check intokvm_arch_vcpu_should_kick().
Good point.
So s390 doesn't need vcpu->cpu and only sets it because other arches do?
And do I understand it correctly that the s390 SIE block operations have no side-effect, apart from changed memory, when outside of guest-mode? (We have cpu->mode mostly because interrupts are expensive. :])
In the end, I'd like to use kvm_vcpu_kick() for kvm_s390_vcpu_wakeup(). s390 sets vcpu->preempted to get a performance boost, which makes touching it less than desirable ...
On s390, vcpu->preempted is only used in __diag_time_slice_end(), which seems to be a type of spinning-on-a-taken-lock hypercall -- any reason why that optimization shouldn't work on other architectures?
Thanks.
Re: [PATCH] KVM: add kvm_arch_cpu_kick
Quoted 68 lines · Radim Krčmář
2017-02-20 12:35+0100, David Hildenbrand:
Quoted 67 lines
Am 20.02.2017 um 12:12 schrieb Christian Borntraeger:
Quoted 40 lines · David Hildenbrand
Quoted 14 lines
Quoted 14 lines
Quoted 3 lines
Yes, it would. There's some parallel with QEMU's
qemu_cpu_kick, where the signal would be processed immediately after enteringKVM_RUN.Something like
---snip-----
struct kvm_s390_sie_block *scb = READ_ONCE(vcpu->arch.vsie_block); atomic_or(CPUSTAT_STOP_INT, &vcpu->arch.sie_block->cpuflags); if (scb) atomic_or(CPUSTAT_STOP_INT, &scb->cpuflags);---snip-----
Quoted 7 lines
Quoted 5 lines
or
---snip-----atomic_or(CPUSTAT_STOP_INT, &vcpu->arch.sie_block->cpuflags); kvm_s390_vsie_kick(vcpu);---snip-----
I'd go for the latter one. Keep the vsie stuff isolated. Please note
Yes makes sense.
Radim, if you go with this patch something like this can be used as the s390 variant of
kvm_arch_cpu_kick:---snip---
/*
* The stop indication is reset in the interrupt code. As the CPU
* loop handles requests after interrupts, we will
* a: miss the request handler and enter the guest, but then the
* stop request will exit the CPU and handle the request in the next
* round or
* b: handle the request directly before entering the guest */
atomic_or(CPUSTAT_STOP_INT, &vcpu->arch.sie_block->cpuflags);kvm_s390_vsie_kick(vcpu);Quoted 3 lines
---snip---
feel free to add that to your patch. I can also send a fixup patch later on if you prefer that.
kvm_arch_vcpu_should_kick()then also has to be changed to return 1 for now.An interesting thing to note is how
vcpu->cpuis used.Again, as s390x can preempt just before entering the guest,
vcpu_kick()might seevcpu->cpu= -1. Therefore,kvm_arch_vcpu_should_kick()won't even be called. So our cpu might go into guest mode and stay there longer than expected (as we won't kick it).On x86, it is the following way:
If
vcpu->cpuis -1, no need to kick the VCPU. It will check for requests when preemption is disabled, therefore when rescheduled.If
vcpu->cpuis set,kvm_arch_vcpu_should_kick()remembers if the VCPU has already been kicked while in the critical section. It will get kicked by smp resched as soon as entering guest mode.So here, disabled preemption + checks in the section with disabled preemption (for requests and
EXITING_GUEST_MODE) make sure that the guest will leave guest mode and process requests in a timely fashion.
Quoted 8 lines
Quoted 4 lines
On s390x, this is not 100% true.
vcpu->cpucannot be used as an indicator whether a kick is necessary. Either that is ok for now, or thevcpu->cpu!= -1 check has to be disabled for s390x, e.g. by moving the check intokvm_arch_vcpu_should_kick().Good point.
So s390 doesn't need
vcpu->cpuand only sets it because other arches do?
David added it as a sanity check for cpu time accounting while in host. But we do not need it, yes.
Quoted 2 lines
And do I understand it correctly that the s390 SIE block operations have no side-effect, apart from changed memory, when outside of guest-mode?
You mean accesses to the sie control block vcpu->arch.sie_block? Yes its just changed memory as long as the VCPU that is backed by this sie control block is not running.
Quoted 3 lines
(We have
cpu->modemostly because interrupts are expensive. :])In the end, I'd like to use
kvm_vcpu_kick()forkvm_s390_vcpu_wakeup().
something like this?
arch/s390/kvm/interrupt.c +1 -9
@@ -1067,15 +1067,7 @@ void kvm_s390_vcpu_wakeup(struct kvm_vcpu *vcpu)* in kvm_vcpu_block without having the waitqueue set (polling) */ vcpu->valid_wakeup = true; -if (swait_active (&vcpu->wq)) { -/* -* The vcpu gave up the cpu voluntarily, mark it as a good -* yield-candidate. -*/ - vcpu->preempted = true; -swake_up (&vcpu->wq); - vcpu->stat.halt_wakeup++; - } +kvm_vcpu_wake_up (vcpu);/* * The VCPU might not be sleeping but is executing the VSIE. Let's * kick it, so it leaves the SIE to process the request.
virt/kvm/kvm_main.c +1 -0
@@ -2213,6 +2213,7 @@ void kvm_vcpu_wake_up(struct kvm_vcpu *vcpu) wqp =kvm_arch_vcpu_wq (vcpu);if (swait_active (wqp)) {swake_up (wqp); + vcpu->preempted = true; ++vcpu->stat.halt_wakeup; }
Quoted 5 lines
s390 sets
vcpu->preemptedto get a performance boost, which makes touching it less than desirable ...
On s390,vcpu->preemptedis only used in__diag_time_slice_end(), which seems to be a type of spinning-on-a-taken-lock hypercall -- any reason why that optimization shouldn't work on other architectures?
We set preempted in kvm_s390_vcpu_wakeup because otherwise a cpu that sleeps (halted) would not be considered as a good candidate in kvm_vcpu_on_spin, even if we just decided to wakeup that CPU for an interrupt.
Yes, it certainly makes sense to do that in kvm_vcpu_wake_up as well.
Re: [PATCH] KVM: add kvm_arch_cpu_kick
2017-02-21 09:59+0100, Christian Borntraeger:
Quoted 81 lines · Radim Krčmář
Quoted 68 lines
2017-02-20 12:35+0100, David Hildenbrand:
Quoted 67 lines
Am 20.02.2017 um 12:12 schrieb Christian Borntraeger:
Quoted 40 lines · David Hildenbrand
Quoted 14 lines
Quoted 14 lines
Quoted 3 lines
Yes, it would. There's some parallel with QEMU's
qemu_cpu_kick, where the signal would be processed immediately after enteringKVM_RUN.Something like
---snip-----
struct kvm_s390_sie_block *scb = READ_ONCE(vcpu->arch.vsie_block); atomic_or(CPUSTAT_STOP_INT, &vcpu->arch.sie_block->cpuflags); if (scb) atomic_or(CPUSTAT_STOP_INT, &scb->cpuflags);---snip-----
Quoted 7 lines
Quoted 5 lines
or
---snip-----atomic_or(CPUSTAT_STOP_INT, &vcpu->arch.sie_block->cpuflags); kvm_s390_vsie_kick(vcpu);---snip-----
I'd go for the latter one. Keep the vsie stuff isolated. Please note
Yes makes sense.
Radim, if you go with this patch something like this can be used as the s390 variant of
kvm_arch_cpu_kick:---snip---
/*
* The stop indication is reset in the interrupt code. As the CPU
* loop handles requests after interrupts, we will
* a: miss the request handler and enter the guest, but then the
* stop request will exit the CPU and handle the request in the next
* round or
* b: handle the request directly before entering the guest */
atomic_or(CPUSTAT_STOP_INT, &vcpu->arch.sie_block->cpuflags);kvm_s390_vsie_kick(vcpu);Quoted 3 lines
---snip---
feel free to add that to your patch. I can also send a fixup patch later on if you prefer that.
kvm_arch_vcpu_should_kick()then also has to be changed to return 1 for now.An interesting thing to note is how
vcpu->cpuis used.Again, as s390x can preempt just before entering the guest,
vcpu_kick()might seevcpu->cpu= -1. Therefore,kvm_arch_vcpu_should_kick()won't even be called. So our cpu might go into guest mode and stay there longer than expected (as we won't kick it).On x86, it is the following way:
If
vcpu->cpuis -1, no need to kick the VCPU. It will check for requests when preemption is disabled, therefore when rescheduled.If
vcpu->cpuis set,kvm_arch_vcpu_should_kick()remembers if the VCPU has already been kicked while in the critical section. It will get kicked by smp resched as soon as entering guest mode.So here, disabled preemption + checks in the section with disabled preemption (for requests and
EXITING_GUEST_MODE) make sure that the guest will leave guest mode and process requests in a timely fashion.Quoted 8 lines
Quoted 4 lines
On s390x, this is not 100% true.
vcpu->cpucannot be used as an indicator whether a kick is necessary. Either that is ok for now, or thevcpu->cpu!= -1 check has to be disabled for s390x, e.g. by moving the check intokvm_arch_vcpu_should_kick().Good point.
So s390 doesn't need
vcpu->cpuand only sets it because other arches do?David added it as a sanity check for cpu time accounting while in host. But we do not need it, yes.
Quoted 6 lines
Quoted 2 lines
And do I understand it correctly that the s390 SIE block operations have no side-effect, apart from changed memory, when outside of guest-mode?
You mean accesses to the sie control block
vcpu->arch.sie_block? Yes its just changed memory as long as the VCPU that is backed by this sie control block is not running.
Great, thanks.
Quoted 15 lines
Quoted 3 lines
(We have
cpu->modemostly because interrupts are expensive. :])In the end, I'd like to use
kvm_vcpu_kick()forkvm_s390_vcpu_wakeup().something like this?
arch/s390/kvm/interrupt.c +0
-3@@ -1067,15 +1067,7 @@ void kvm_s390_vcpu_wakeup(struct kvm_vcpu *vcpu)* in kvm_vcpu_block without having the waitqueue set (polling) */ vcpu->valid_wakeup = true; -if (swait_active (&vcpu->wq)) { -/* -* The vcpu gave up the cpu voluntarily, mark it as a good
arch/s390/kvm/interrupt.c void kvm_s390_vcpu_wakeup(struct kvm_vcpu *vcpu) · line 1073
Quoted 10 lines
arch/s390/kvm/interrupt.c +1
-6-* yield-candidate. -*/ - vcpu->preempted = true; -swake_up (&vcpu->wq); - vcpu->stat.halt_wakeup++; - } +kvm_vcpu_wake_up (vcpu);/* * The VCPU might not be sleeping but is executing the VSIE. Let's * kick it, so it leaves the SIE to process the request.
Yes, and ideally also covering the SIE kick, so the result would be
{
vcpu->valid_wakeup = true;
+ kvm_vcpu_kick(vcpu);
}
Quoted 17 lines
virt/kvm/kvm_main.c +1
-0@@ -2213,6 +2213,7 @@ void kvm_vcpu_wake_up(struct kvm_vcpu *vcpu) wqp =kvm_arch_vcpu_wq (vcpu);if (swait_active (wqp)) {swake_up (wqp); + vcpu->preempted = true; ++vcpu->stat.halt_wakeup; }Quoted 6 lines
s390 sets
vcpu->preemptedto get a performance boost, which makes touching it less than desirable ...
On s390,vcpu->preemptedis only used in__diag_time_slice_end(), which seems to be a type of spinning-on-a-taken-lock hypercall -- any reason why that optimization shouldn't work on other architectures?
Quoted 5 lines
We set preempted in
kvm_s390_vcpu_wakeupbecause otherwise a cpu that sleeps (halted) would not be considered as a good candidate inkvm_vcpu_on_spin, even if we just decided to wakeup that CPU for an interrupt.Yes, it certainly makes sense to do that in
kvm_vcpu_wake_upas well.
I assume that s390 doesn't go to sleep while holding a spinlock, so it would mean that we need to update kvm_vcpu_on_spin(). (Ignoring a formerly sleeping VCPU as a candidate made sense: the VCPU
shouldn't have been holding a lock that is blocking the spinning VCPU.)
Boosting a VCPU that is likely not going to use the contended spinlock seems like a good thing to try. I think we don't need vcpu->preempted in its current form then. After the change, it it would be false only in two cases:
1) the VCPU task is currently scheduled on some CPU
2) the VCPU is sleeping
There might be some other way know (1) (or we can adapt vcpu->preempted to vcpu->scheduled) and (2) is done with swait_active().
Sadly, any change of kvm_vcpu_on_spin() is going to need many days of testing ...
Re: [PATCH] KVM: add kvm_arch_cpu_kick
Quoted 119 lines · Radim Krčmář
2017-02-21 09:59+0100, Christian Borntraeger:
Quoted 81 lines · Radim Krčmář
Quoted 68 lines
2017-02-20 12:35+0100, David Hildenbrand:
Quoted 67 lines
Am 20.02.2017 um 12:12 schrieb Christian Borntraeger:
Quoted 40 lines · David Hildenbrand
Quoted 14 lines
Quoted 14 lines
Quoted 3 lines
Yes, it would. There's some parallel with QEMU's
qemu_cpu_kick, where the signal would be processed immediately after enteringKVM_RUN.Something like
---snip-----
struct kvm_s390_sie_block *scb = READ_ONCE(vcpu->arch.vsie_block); atomic_or(CPUSTAT_STOP_INT, &vcpu->arch.sie_block->cpuflags); if (scb) atomic_or(CPUSTAT_STOP_INT, &scb->cpuflags);---snip-----
Quoted 7 lines
Quoted 5 lines
or
---snip-----atomic_or(CPUSTAT_STOP_INT, &vcpu->arch.sie_block->cpuflags); kvm_s390_vsie_kick(vcpu);---snip-----
I'd go for the latter one. Keep the vsie stuff isolated. Please note
Yes makes sense.
Radim, if you go with this patch something like this can be used as the s390 variant of
kvm_arch_cpu_kick:---snip---
/*
* The stop indication is reset in the interrupt code. As the CPU
* loop handles requests after interrupts, we will
* a: miss the request handler and enter the guest, but then the
* stop request will exit the CPU and handle the request in the next
* round or
* b: handle the request directly before entering the guest */
atomic_or(CPUSTAT_STOP_INT, &vcpu->arch.sie_block->cpuflags);kvm_s390_vsie_kick(vcpu);Quoted 3 lines
---snip---
feel free to add that to your patch. I can also send a fixup patch later on if you prefer that.
kvm_arch_vcpu_should_kick()then also has to be changed to return 1 for now.An interesting thing to note is how
vcpu->cpuis used.Again, as s390x can preempt just before entering the guest,
vcpu_kick()might seevcpu->cpu= -1. Therefore,kvm_arch_vcpu_should_kick()won't even be called. So our cpu might go into guest mode and stay there longer than expected (as we won't kick it).On x86, it is the following way:
If
vcpu->cpuis -1, no need to kick the VCPU. It will check for requests when preemption is disabled, therefore when rescheduled.If
vcpu->cpuis set,kvm_arch_vcpu_should_kick()remembers if the VCPU has already been kicked while in the critical section. It will get kicked by smp resched as soon as entering guest mode.So here, disabled preemption + checks in the section with disabled preemption (for requests and
EXITING_GUEST_MODE) make sure that the guest will leave guest mode and process requests in a timely fashion.Quoted 8 lines
Quoted 4 lines
On s390x, this is not 100% true.
vcpu->cpucannot be used as an indicator whether a kick is necessary. Either that is ok for now, or thevcpu->cpu!= -1 check has to be disabled for s390x, e.g. by moving the check intokvm_arch_vcpu_should_kick().Good point.
So s390 doesn't need
vcpu->cpuand only sets it because other arches do?David added it as a sanity check for cpu time accounting while in host. But we do not need it, yes.
Quoted 6 lines
Quoted 2 lines
And do I understand it correctly that the s390 SIE block operations have no side-effect, apart from changed memory, when outside of guest-mode?
You mean accesses to the sie control block
vcpu->arch.sie_block? Yes its just changed memory as long as the VCPU that is backed by this sie control block is not running.Great, thanks.
Quoted 15 lines
Quoted 3 lines
(We have
cpu->modemostly because interrupts are expensive. :])In the end, I'd like to use
kvm_vcpu_kick()forkvm_s390_vcpu_wakeup().something like this?
arch/s390/kvm/interrupt.c +0
-3@@ -1067,15 +1067,7 @@ void kvm_s390_vcpu_wakeup(struct kvm_vcpu *vcpu)* in kvm_vcpu_block without having the waitqueue set (polling) */ vcpu->valid_wakeup = true; -if (swait_active (&vcpu->wq)) { -/* -* The vcpu gave up the cpu voluntarily, mark it as a good arch/s390/kvm/interrupt.c void kvm_s390_vcpu_wakeup(struct kvm_vcpu *vcpu) · line 1073
Quoted 10 lines
arch/s390/kvm/interrupt.c +1
-6-* yield-candidate. -*/ - vcpu->preempted = true; -swake_up (&vcpu->wq); - vcpu->stat.halt_wakeup++; - } +kvm_vcpu_wake_up (vcpu);/* * The VCPU might not be sleeping but is executing the VSIE. Let's * kick it, so it leaves the SIE to process the request. Yes, and ideally also covering the SIE kick, so the result would be
Quoted 5 lines
{
vcpu->valid_wakeup= true;
+kvm_vcpu_kick(vcpu);
}
No we do not want to replace kvm_s390_vcpu_wakeup with the generic SIE kick. the stop exit will exit the guest unconditionally. As kvm_s390_vcpu_wakeup is also called after queuing interrupts we want to use the special exits for the different interrupt types that wait with the guest exits until that interrupt is enabled by the guest. (e.g. not when running in local_irq_disable)
So I think it does not make sense to replace this with the generic kvm_vcpu_wake_up if we need the kick.
Quoted 17 lines
Quoted 17 lines
virt/kvm/kvm_main.c +1
-0@@ -2213,6 +2213,7 @@ void kvm_vcpu_wake_up(struct kvm_vcpu *vcpu) wqp =kvm_arch_vcpu_wq (vcpu);if (swait_active (wqp)) {swake_up (wqp); + vcpu->preempted = true; ++vcpu->stat.halt_wakeup; }Quoted 6 lines
s390 sets
vcpu->preemptedto get a performance boost, which makes touching it less than desirable ...
On s390,vcpu->preemptedis only used in__diag_time_slice_end(), which seems to be a type of spinning-on-a-taken-lock hypercall -- any reason why that optimization shouldn't work on other architectures?
Quoted 7 lines
Quoted 5 lines
We set preempted in
kvm_s390_vcpu_wakeupbecause otherwise a cpu that sleeps (halted) would not be considered as a good candidate inkvm_vcpu_on_spin, even if we just decided to wakeup that CPU for an interrupt.Yes, it certainly makes sense to do that in
kvm_vcpu_wake_upas well.I assume that s390 doesn't go to sleep while holding a spinlock, so it
Yes x86 and s390 do not sleep in atomic contexts. Both spin for a while and exit after some time into kvm_vcpu_on_on (*) We try to find a good candidate for yielding and then sleep due to the yield. But a normal spinlock does not cause a halt (or a wait PSW).
(*) all modern s390 Linuxes now use the directed yield hypercall for spinlocks (see diag 9c) so the diag 44 is only relevant for stop machine run.
Quoted 3 lines
would mean that we need to update
kvm_vcpu_on_spin(). (Ignoring a formerly sleeping VCPU as a candidate made sense: the VCPU
shouldn't have been holding a lock that is blocking the spinning VCPU.)
Yes, if the yielding wants to boost the lock holder, indeed ignoring a sleeping cpu sounds like a good idea. In reality things are more complicated and lock holder preemption is especially nasty with many VCPUs. In that case we want to also boost these CPUs that are delivering interrupts. We had several performance issues with some workloads that were fixed by this in commit 9cac38dd5dc41c943d711b96f9755a29c8b854ea
KVM/s390: Set preempted flag during vcpu wakeup and interrupt delivery
so it turned out to be better this way for a semop intense workload back then.
Quoted 7 lines
Boosting a VCPU that is likely not going to use the contended spinlock seems like a good thing to try. I think we don't need
vcpu->preemptedin its current form then. After the change, it it would be false only in two cases:
1) the VCPU task is currently scheduled on some CPU
2) the VCPU is sleeping
Quoted 5 lines
There might be some other way know (1) (or we can adapt
vcpu->preemptedtovcpu->scheduled) and (2) is done withswait_active().Sadly, any change of
kvm_vcpu_on_spin()is going to need many days of testing ...
Yes, this requires a lot of tuning. Maybe we should just stick with the minimal changes then?
Re: [PATCH] KVM: add kvm_arch_cpu_kick
2017-02-21 20:08+0100, Christian Borntraeger:
Quoted 126 lines · Radim Krčmář
Quoted 119 lines
2017-02-21 09:59+0100, Christian Borntraeger:
Quoted 81 lines · Radim Krčmář
Quoted 68 lines
2017-02-20 12:35+0100, David Hildenbrand:
Quoted 67 lines
Am 20.02.2017 um 12:12 schrieb Christian Borntraeger:
Quoted 40 lines · David Hildenbrand
Quoted 14 lines
Quoted 14 lines
Quoted 3 lines
Yes, it would. There's some parallel with QEMU's
qemu_cpu_kick, where the signal would be processed immediately after enteringKVM_RUN.Something like
---snip-----
struct kvm_s390_sie_block *scb = READ_ONCE(vcpu->arch.vsie_block); atomic_or(CPUSTAT_STOP_INT, &vcpu->arch.sie_block->cpuflags); if (scb) atomic_or(CPUSTAT_STOP_INT, &scb->cpuflags);---snip-----
Quoted 7 lines
Quoted 5 lines
or
---snip-----atomic_or(CPUSTAT_STOP_INT, &vcpu->arch.sie_block->cpuflags); kvm_s390_vsie_kick(vcpu);---snip-----
I'd go for the latter one. Keep the vsie stuff isolated. Please note
Yes makes sense.
Radim, if you go with this patch something like this can be used as the s390 variant of
kvm_arch_cpu_kick:---snip---
/*
* The stop indication is reset in the interrupt code. As the CPU
* loop handles requests after interrupts, we will
* a: miss the request handler and enter the guest, but then the
* stop request will exit the CPU and handle the request in the next
* round or
* b: handle the request directly before entering the guest */
atomic_or(CPUSTAT_STOP_INT, &vcpu->arch.sie_block->cpuflags);kvm_s390_vsie_kick(vcpu);Quoted 3 lines
---snip---
feel free to add that to your patch. I can also send a fixup patch later on if you prefer that.
kvm_arch_vcpu_should_kick()then also has to be changed to return 1 for now.An interesting thing to note is how
vcpu->cpuis used.Again, as s390x can preempt just before entering the guest,
vcpu_kick()might seevcpu->cpu= -1. Therefore,kvm_arch_vcpu_should_kick()won't even be called. So our cpu might go into guest mode and stay there longer than expected (as we won't kick it).On x86, it is the following way:
If
vcpu->cpuis -1, no need to kick the VCPU. It will check for requests when preemption is disabled, therefore when rescheduled.If
vcpu->cpuis set,kvm_arch_vcpu_should_kick()remembers if the VCPU has already been kicked while in the critical section. It will get kicked by smp resched as soon as entering guest mode.So here, disabled preemption + checks in the section with disabled preemption (for requests and
EXITING_GUEST_MODE) make sure that the guest will leave guest mode and process requests in a timely fashion.Quoted 8 lines
Quoted 4 lines
On s390x, this is not 100% true.
vcpu->cpucannot be used as an indicator whether a kick is necessary. Either that is ok for now, or thevcpu->cpu!= -1 check has to be disabled for s390x, e.g. by moving the check intokvm_arch_vcpu_should_kick().Good point.
So s390 doesn't need
vcpu->cpuand only sets it because other arches do?David added it as a sanity check for cpu time accounting while in host. But we do not need it, yes.
Quoted 6 lines
Quoted 2 lines
And do I understand it correctly that the s390 SIE block operations have no side-effect, apart from changed memory, when outside of guest-mode?
You mean accesses to the sie control block
vcpu->arch.sie_block? Yes its just changed memory as long as the VCPU that is backed by this sie control block is not running.Great, thanks.
Quoted 15 lines
Quoted 3 lines
(We have
cpu->modemostly because interrupts are expensive. :])In the end, I'd like to use
kvm_vcpu_kick()forkvm_s390_vcpu_wakeup().something like this?
arch/s390/kvm/interrupt.c +0
-3@@ -1067,15 +1067,7 @@ void kvm_s390_vcpu_wakeup(struct kvm_vcpu *vcpu)* in kvm_vcpu_block without having the waitqueue set (polling) */ vcpu->valid_wakeup = true; -if (swait_active (&vcpu->wq)) { -/* -* The vcpu gave up the cpu voluntarily, mark it as a good arch/s390/kvm/interrupt.c void kvm_s390_vcpu_wakeup(struct kvm_vcpu *vcpu) · line 1073
Quoted 10 lines
arch/s390/kvm/interrupt.c +1
-6-* yield-candidate. -*/ - vcpu->preempted = true; -swake_up (&vcpu->wq); - vcpu->stat.halt_wakeup++; - } +kvm_vcpu_wake_up (vcpu);/* * The VCPU might not be sleeping but is executing the VSIE. Let's * kick it, so it leaves the SIE to process the request. Yes, and ideally also covering the SIE kick, so the result would be
Quoted 5 lines
{
vcpu->valid_wakeup= true;
+kvm_vcpu_kick(vcpu);
}
Quoted 7 lines
No we do not want to replace
kvm_s390_vcpu_wakeupwith the generic SIE kick. the stop exit will exit the guest unconditionally. Askvm_s390_vcpu_wakeupis also called after queuing interrupts we want to use the special exits for the different interrupt types that wait with the guest exits until that interrupt is enabled by the guest. (e.g. not when running inlocal_irq_disable)
So I think it does not make sense to replace this with the generickvm_vcpu_wake_upif we need the kick.
Makes sense, thanks. I was misunderstanding the vsie kick and its users.
Quoted 56 lines
Quoted 17 lines
Quoted 17 lines
virt/kvm/kvm_main.c +1
-0@@ -2213,6 +2213,7 @@ void kvm_vcpu_wake_up(struct kvm_vcpu *vcpu) wqp =kvm_arch_vcpu_wq (vcpu);if (swait_active (wqp)) {swake_up (wqp); + vcpu->preempted = true; ++vcpu->stat.halt_wakeup; }Quoted 6 lines
s390 sets
vcpu->preemptedto get a performance boost, which makes touching it less than desirable ...
On s390,vcpu->preemptedis only used in__diag_time_slice_end(), which seems to be a type of spinning-on-a-taken-lock hypercall -- any reason why that optimization shouldn't work on other architectures?Quoted 7 lines
Quoted 5 lines
We set preempted in
kvm_s390_vcpu_wakeupbecause otherwise a cpu that sleeps (halted) would not be considered as a good candidate inkvm_vcpu_on_spin, even if we just decided to wakeup that CPU for an interrupt.Yes, it certainly makes sense to do that in
kvm_vcpu_wake_upas well.I assume that s390 doesn't go to sleep while holding a spinlock, so it
Yes x86 and s390 do not sleep in atomic contexts. Both spin for a while and exit after some time into
kvm_vcpu_on_on(*) We try to find a good candidate for yielding and then sleep due to the yield. But a normal spinlock does not cause a halt (or a wait PSW).(*) all modern s390 Linuxes now use the directed yield hypercall for spinlocks (see diag 9c) so the diag 44 is only relevant for stop machine run.
Quoted 3 lines
would mean that we need to update
kvm_vcpu_on_spin(). (Ignoring a formerly sleeping VCPU as a candidate made sense: the VCPU
shouldn't have been holding a lock that is blocking the spinning VCPU.)Yes, if the yielding wants to boost the lock holder, indeed ignoring a sleeping cpu sounds like a good idea. In reality things are more complicated and lock holder preemption is especially nasty with many VCPUs. In that case we want to also boost these CPUs that are delivering interrupts. We had several performance issues with some workloads that were fixed by this in commit
9cac38dd5dc41c943d711b96f9755a29c8b854ea
KVM/s390: Set preempted flag during vcpu wakeup and interrupt deliveryso it turned out to be better this way for a semop intense workload back then.
Quoted 7 lines
Boosting a VCPU that is likely not going to use the contended spinlock seems like a good thing to try. I think we don't need
vcpu->preemptedin its current form then. After the change, it it would be false only in two cases:
1) the VCPU task is currently scheduled on some CPU
2) the VCPU is sleeping
Quoted 8 lines
Quoted 5 lines
There might be some other way know (1) (or we can adapt
vcpu->preemptedtovcpu->scheduled) and (2) is done withswait_active().Sadly, any change of
kvm_vcpu_on_spin()is going to need many days of testing ...Yes, this requires a lot of tuning. Maybe we should just stick with the minimal changes then?
Any changes would need the same testing ... I could start with no changes :)
if (kvm_vcpu_wake_up(vcpu)) vcpu->preempted = true;
Finding this has made the rewrite useful much earlier than I hoped, though you mentioned that s390 doesn't use it all that much and x86 is also using directed kicks for blocked locks.
Re: [PATCH] KVM: add kvm_arch_cpu_kick
In reply to Christian Borntraeger2017-02-20 12:12+0100, Christian Borntraeger:
Quoted 40 lines · David Hildenbrand
Quoted 14 lines
Quoted 14 lines
Quoted 3 lines
Yes, it would. There's some parallel with QEMU's
qemu_cpu_kick, where the signal would be processed immediately after enteringKVM_RUN.Something like
---snip-----
struct kvm_s390_sie_block *scb = READ_ONCE(vcpu->arch.vsie_block); atomic_or(CPUSTAT_STOP_INT, &vcpu->arch.sie_block->cpuflags); if (scb) atomic_or(CPUSTAT_STOP_INT, &scb->cpuflags);---snip-----
Quoted 7 lines
Quoted 5 lines
or
---snip-----atomic_or(CPUSTAT_STOP_INT, &vcpu->arch.sie_block->cpuflags); kvm_s390_vsie_kick(vcpu);---snip-----
I'd go for the latter one. Keep the vsie stuff isolated. Please note
Yes makes sense.
Radim, if you go with this patch something like this can be used as the s390 variant of
kvm_arch_cpu_kick:---snip---
/*
* The stop indication is reset in the interrupt code. As the CPU
* loop handles requests after interrupts, we will
* a: miss the request handler and enter the guest, but then the
* stop request will exit the CPU and handle the request in the next
* round or
* b: handle the request directly before entering the guest */
atomic_or(CPUSTAT_STOP_INT, &vcpu->arch.sie_block->cpuflags);kvm_s390_vsie_kick(vcpu);
Quoted 3 lines
---snip---
feel free to add that to your patch. I can also send a fixup patch later on if you prefer that.
I'll probably use it. Just looking at the hunk made me realize that the interface will need to pass the vcpu as well, so I'd be best to have the whole picture.
Thanks.
Re: [PATCH] KVM: add kvm_arch_cpu_kick
In reply to Radim KrčmářAm 17.02.2017 um 16:12 schrieb Radim Krčmář:
Quoted 50 lines
2017-02-17 14:10+0100, Christian Borntraeger:
Quoted 12 lines
needed by KVM common code.
Signed-off-by Christian Borntraeger <[email protected]>--- arch/s390/kernel/smp.c | 1 + 1 file changed, 1 insertion(+)arch/s390/kernel/smp.c +0
-0arch/s390/kernel/smp.c void smp_send_reschedule(int cpu) · line 516
Quoted 4 lines
arch/s390/kernel/smp.c +1
-0@@ -513,6 +513,7 @@ void smp_send_reschedule(int cpu) {pcpu_ec_call (pcpu_devices + cpu, ec_schedule); } +EXPORT_SYMBOL_GPL (smp_send_reschedule);s390 doesn't want to use
smp_send_reschedule()so I think the patch at the bottom is going in a better direction.Btw. I think that we shouldn't be using
smp_send_reschedule()for forcing VCPUs out of guest mode anyway -- the task we want to run is already scheduled and we don't want the schduler to do anything.We could use
smp_call_function()with an empty function instead, but that also has high overhead ... allocating a new IPI seems best.But optimizing the current code is premature. :)
---8<---
s390 has a different mechanism from bringing the VCPU out of guest mode. Make the kick arch-specific.Signed-off-by Radim Krčmář <[email protected]>--- arch/arm/kvm/arm.c | 5 +++++ arch/mips/kvm/mips.c | 5 +++++ arch/powerpc/kvm/powerpc.c | 5 +++++ arch/s390/kvm/kvm-s390.c | 6 ++++++ arch/x86/kvm/x86.c | 5 +++++ include/linux/kvm_host.h | 1 + virt/kvm/kvm_main.c | 2 +- 7 files changed, 28 insertions(+), 1 deletion(-)arch/arm/kvm/arm.c +0
-0
arch/arm/kvm/arm.c int kvm_arch_vcpu_should_kick(struct kvm_vcpu *vcpu) · line 99
Quoted 3 lines
arch/arm/kvm/arm.c +0
-0@@ -97,6 +97,11 @@ int kvm_arch_vcpu_should_kick(struct kvm_vcpu *vcpu)return kvm_vcpu_exiting_guest_mode (vcpu) ==IN_GUEST_MODE ; }
That's what I had in mind. Looks good.
Signature · 3 lines
Thanks, David
PATCH 2/2 KVM: enable kvm_vcpu_kick/wake_up for s390
In reply to Christian Borntraegerwhile we still need to provide our private version of kick and wakeup to synchronize with some special cases, we can still enable the common code variant.
--- arch/s390/kvm/kvm-s390.c | 12 ++++++++++-- virt/kvm/kvm_main.c | 2 -- 2 files changed, 10 insertions(+), 4 deletions(-)
arch/s390/kvm/kvm-s390.c +10 -2
@@ -2485,8 +2485,14 @@ static void kvm_gmap_notifier(struct gmap *gmap, unsigned long start,int kvm_arch_vcpu_should_kick (struct kvm_vcpu *vcpu) { -/* kvm common code refers to this, but never calls it */ -BUG (); +/* +* STOP indication is resetted when delivering interrupts. This +* is done before we handle requests, so we only "loose" this +* when we are still going to handle requests. In that case +* we no longer need that STOP indication. +*/ +__set_cpuflag (vcpu,CPUSTAT_STOP_INT ); +kvm_s390_vsie_kick (vcpu);return 0 ; }
@@ -3064,6 +3070,7 @@ static int __vcpu_run(struct kvm_vcpu *vcpu)*/ vcpu->srcu_idx =srcu_read_lock (&vcpu->kvm->srcu); + vcpu->mode =IN_GUEST_MODE ; do { rc = vcpu_pre_run(vcpu); if (rc)
@@ -3088,6 +3095,7 @@ static int __vcpu_run(struct kvm_vcpu *vcpu) rc =vcpu_post_run (vcpu, exit_reason); }while (!signal_pending (current) && !guestdbg_exit_pending (vcpu) && !rc); + vcpu->mode =OUTSIDE_GUEST_MODE ;srcu_read_unlock (&vcpu->kvm->srcu, vcpu->srcu_idx);return rc;
virt/kvm/kvm_main.c +0 -2
@@ -2205,7 +2205,6 @@ void kvm_vcpu_block(struct kvm_vcpu *vcpu) }EXPORT_SYMBOL_GPL (kvm_vcpu_block); -#ifndef CONFIG_S390 void kvm_vcpu_wake_up (struct kvm_vcpu *vcpu) {struct swait_queue_head *wqp;
@@ -2235,7 +2234,6 @@ void kvm_vcpu_kick(struct kvm_vcpu *vcpu)put_cpu (); }EXPORT_SYMBOL_GPL (kvm_vcpu_kick); -#endif /* !CONFIG_S390 */ int kvm_vcpu_yield_to (struct kvm_vcpu *target) {
Signature · 1 line
2.7.4
Re: [PATCH/RFC 2/2] KVM: enable kvm_vcpu_kick/wake_up for s390
2017-02-17 14:10+0100, Christian Borntraeger:
Quoted 18 lines
while we still need to provide our private version of kick and wakeup to synchronize with some special cases, we can still enable the common code variant.
Signed-off-by Christian Borntraeger <[email protected]>--- arch/s390/kvm/kvm-s390.c | 12 ++++++++++-- virt/kvm/kvm_main.c | 2 -- 2 files changed, 10 insertions(+), 4 deletions(-)arch/s390/kvm/kvm-s390.c +0
-0@@ -2485,8 +2485,14 @@ static void kvm_gmap_notifier(struct gmap *gmap, unsigned long start,int kvm_arch_vcpu_should_kick (struct kvm_vcpu *vcpu) {
arch/s390/kvm/kvm-s390.c static void kvm_gmap_notifier(struct gmap *gmap, unsigned long start, · line 2495
Quoted 10 lines
arch/s390/kvm/kvm-s390.c +8
-2-/* kvm common code refers to this, but never calls it */ -BUG (); +/* +* STOP indication is resetted when delivering interrupts. This +* is done before we handle requests, so we only "loose" this +* when we are still going to handle requests. In that case +* we no longer need that STOP indication. +*/ +__set_cpuflag (vcpu,CPUSTAT_STOP_INT ); +kvm_s390_vsie_kick (vcpu);
I wouldn't put the kick into that function. On other architectures, the function decides whether we should call an operation that actually forces the exit. e.g. x86 and arm do
return kvm_vcpu_exiting_guest_mode(vcpu) == IN_GUEST_MODE
(which also contains an optimization, because they will not kick a VCPU
twice or do a kick for a VCPU outside the guest mode.)
There little point in having vcpu->mode if we do not use it here. And we'd also like to use kvm_arch_vcpu_should_kick() for the same decision in kvm_make_all_cpus_request(); basic idea:
virt/kvm/kvm_main.c +3 -7
@@ -163,10 +163,6 @@ void vcpu_put(struct kvm_vcpu *vcpu) }EXPORT_SYMBOL_GPL (vcpu_put); -static void ack_flush (void *_completed) -{ -} -bool kvm_make_all_cpus_request (struct kvm *kvm,unsigned int req) {int i, cpu, me;
@@ -194,13 +190,13 @@ bool kvm_make_all_cpus_request(struct kvm *kvm, unsigned int req)smp_mb__after_atomic ();if (cpus !=NULL && cpu !=-1 && cpu != me && -kvm_vcpu_exiting_guest_mode (vcpu) !=OUTSIDE_GUEST_MODE ) +kvm_arch_vcpu_should_kick (vcpu))cpumask_set_cpu (cpu, cpus); }if (unlikely (cpus ==NULL )) -smp_call_function_many (cpu_online_mask, ack_flush,NULL ,1 ); +kvm_arch_cpu_kick_many (cpu_online_mask);else if (!cpumask_empty (cpus)) -smp_call_function_many (cpus, ack_flush,NULL ,1 ); +kvm_arch_cpu_kick_many (cpus);else called = false;put_cpu ();