kvm-s390: infrastructure to kick vcpus out of guest state - v3
PATCH 1/4 kvm-s390: infrastructure to kick vcpus out of guest state - v3
From: Christian Ehrhardt <[email protected]>
*updates in v3*
- ensure allocations (might_sleep) are out of atomic context
*updates in v2*
instead of a "kick to level" behaviour the patch now implements a kick to the lowest level. The check there bails out to upper levels if not all outstanding vcpu->requests could be handled internally (it could still support an explicit kick to level if ever needed).
To ensure vcpu's come out of guest context in certain cases this patch adds a s390 specific way to kick them out of guest context. Currently it kicks them out to rerun the vcpu_run path in the s390 code, but the mechanism itself is expandable and with a new flag we could also add e.g. kicks to userspace etc.
---
[diffstat]
include/asm/kvm_host.h | 5 ++-- kvm/intercept.c | 14 +++++++++-- kvm/kvm-s390.c | 6 ++++ kvm/kvm-s390.h | 17 ++++++++++++++ kvm/sigp.c | 59 ++++++++++++++++++++++++++++++++++--------------- 5 files changed, 79 insertions(+), 22 deletions(-)
[diff]
arch/s390/kvm/intercept.c +11 -3
@@ -128,7 +128,7 @@ static int handle_noop(struct kvm_vcpu *static int handle_stop (struct kvm_vcpu *vcpu) { -int rc; +int rc =0 ; vcpu->stat.exit_stop_request++;atomic_clear_mask (CPUSTAT_RUNNING , &vcpu->arch.sie_block->cpuflags);
@@ -141,12 +141,20 @@ static int handle_stop(struct kvm_vcpu * rc = -ENOTSUPP ; } +if (vcpu->arch.local_int.action_bits &ACTION_VCPUREQUEST_ON_STOP ) { + vcpu->arch.local_int.action_bits &= ~ACTION_VCPUREQUEST_ON_STOP ; +if (kvm_s390_handle_vcpu_requests (vcpu,VCPUREQUESTLVL_SIGP )) { + rc =SIE_INTERCEPT_CHECKREQUESTS ; + vcpu->run->exit_reason =KVM_EXIT_INTR ; + } + } +if (vcpu->arch.local_int.action_bits &ACTION_STOP_ON_STOP ) { vcpu->arch.local_int.action_bits &= ~ACTION_STOP_ON_STOP ;VCPU_EVENT (vcpu,3 ,"%s" ,"cpu stopped" ); rc = -ENOTSUPP ; - }else - rc =0 ; + } +spin_unlock_bh (&vcpu->arch.local_int.lock);return rc; }
arch/s390/kvm/kvm-s390.c +6 -0
@@ -487,6 +487,9 @@ int kvm_arch_vcpu_ioctl_run(struct kvm_vvcpu_load (vcpu); +rerun_vcpu: +kvm_s390_handle_vcpu_requests (vcpu,VCPUREQUESTLVL_VCPURUN ); +/* verify, that memory has been registered */ if (!vcpu->kvm->arch.guest_memsize) {vcpu_put (vcpu);
@@ -519,6 +522,9 @@ int kvm_arch_vcpu_ioctl_run(struct kvm_v rc =kvm_handle_sie_intercept (vcpu); }while (!signal_pending (current) && !rc); +if (rc ==SIE_INTERCEPT_CHECKREQUESTS ) + goto rerun_vcpu; +if (signal_pending (current) && !rc) rc = -EINTR ;
arch/s390/kvm/kvm-s390.h +17 -0
@@ -20,6 +20,8 @@typedef int (*intercept_handler_t )(struct kvm_vcpu *vcpu); +/* negativ values are error codes, positive values for internal conditions */ +#define SIE_INTERCEPT_CHECKREQUESTS (1<<0)int kvm_handle_sie_intercept (struct kvm_vcpu *vcpu); #define VM_EVENT(d_kvm, d_loglevel, d_string, d_args...)\
@@ -50,6 +52,21 @@ int kvm_s390_inject_vm(struct kvm *kvm,int kvm_s390_inject_vcpu (struct kvm_vcpu *vcpu,struct kvm_s390_interrupt *s390int);int kvm_s390_inject_program_int (struct kvm_vcpu *vcpu,u16 code); +int kvm_s390_inject_sigp_stop (struct kvm_vcpu *vcpu,int action); + +/* interception levels from which handle vcpu requests can be called */ +#define VCPUREQUESTLVL_SIGP 1 +#define VCPUREQUESTLVL_VCPURUN 2 +static inline unsigned long kvm_s390_handle_vcpu_requests (struct kvm_vcpu *vcpu, +int level) +{ +BUG_ON (!level); + +if (!vcpu->requests) +return 0 ; + +return vcpu->requests; +}/* implemented in priv.c */ int kvm_s390_handle_b2 (struct kvm_vcpu *vcpu);
arch/s390/include/asm/kvm_host.h +3 -2
@@ -180,8 +180,9 @@ struct kvm_s390_interrupt_info { };/* for local_interrupt.action_flags */ -#define ACTION_STORE_ON_STOP 1 -#define ACTION_STOP_ON_STOP 2 +#define ACTION_STORE_ON_STOP (1<<0) +#define ACTION_STOP_ON_STOP (1<<1) +#define ACTION_VCPUREQUEST_ON_STOP (1<<2)struct kvm_s390_local_interrupt {spinlock_t lock;
arch/s390/kvm/sigp.c +42 -17
106 lines not on this page · raw message
PATCH 2/4 kvm-s390: fix interruption caused by signal - v2
From: Christian Ehrhardt <[email protected]>
*updates in v2*
merged a small piece of code from patch 1/1 that belongs here themtically
If signal pending is true we exit without updating kvm_run, userspace currently just does nothing and jumps to kvm_run again. Since we did not set an exit_reason we might end up with a random one (whatever was the last exit). Therefore it was possible to e.g. jump to the psw position the last real interruption set.
Setting the INTR exit reason ensures that no old psw data is swapped in on reentry.
---
[diffstat]
kvm-s390.c | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-)
[diff]
arch/s390/kvm/kvm-s390.c +4 -1
@@ -509,6 +509,7 @@ rerun_vcpu: vcpu->arch.sie_block->gpsw.addr = kvm_run->s390_sieic.addr;break ;case KVM_EXIT_UNKNOWN : +case KVM_EXIT_INTR :case KVM_EXIT_S390_RESET :break ;default :
@@ -525,8 +526,10 @@ rerun_vcpu:if (rc ==SIE_INTERCEPT_CHECKREQUESTS ) goto rerun_vcpu; -if (signal_pending (current) && !rc) +if (signal_pending (current) && !rc) { + kvm_run->exit_reason =KVM_EXIT_INTR ; rc = -EINTR ; + }if (rc == -ENOTSUPP ) {/* intercept cannot be handled in-kernel, prepare kvm-run */
PATCH 3/4 kvm-s390: update vcpu->cpu
In reply to [email protected]From: Christian Ehrhardt <[email protected]>
kvm on s390 formerly ignored vcpu->cpu.
This patch adds set/unset vcpu->cpu in kvm_arch_vcpu_load/put to allow further architecture unification e.g. let generic code not find -1 on currently scheduled vcpus.
---
[diffstat]
kvm-s390.c | 2 ++ 1 file changed, 2 insertions(+)
[diff]
arch/s390/kvm/kvm-s390.c +2 -0
@@ -243,6 +243,7 @@ void kvm_arch_vcpu_uninit(struct kvm_vcpvoid kvm_arch_vcpu_load (struct kvm_vcpu *vcpu,int cpu) { + vcpu->cpu = cpu;save_fp_regs (&vcpu->arch.host_fpregs);save_access_regs (vcpu->arch.host_acrs); vcpu->arch.guest_fpregs.fpc &=FPC_VALID_MASK ;
@@ -252,6 +253,7 @@ void kvm_arch_vcpu_load(struct kvm_vcpuvoid kvm_arch_vcpu_put (struct kvm_vcpu *vcpu) { + vcpu->cpu =-1 ;save_fp_regs (&vcpu->arch.guest_fpregs);save_access_regs (vcpu->arch.guest_acrs);restore_fp_regs (&vcpu->arch.host_fpregs);
PATCH 4/4 kvm-s390: streamline memslot handling - v6
In reply to [email protected]From: Christian Ehrhardt <[email protected]>
*updates in v6*
- ensure the wait_on_bit waiter is notified
- move the reset of requests to kvm_vcpu_release to drop them early
*updates in v5*
- ensure dropping vcpu all requests while freeing a vcpu
*updates in v4*
- kickout only scheduled vcpus (its superfluous and wait might hang forever on not running vcpus)
*updates in v3*
- handling the mmu reload vcpu request can now be handled inside the sigp handling avoiding an addtional exit
- kvm_arch_set_memory_region now waits for kicked vcpu's to consume the request bit it set to ensure that after the kvm_arch_set_memory_region call all vcpus use the updated memory information
*updates in v2*
- added optimization to skip (addtional) kickout of vcpu's that had the request already set.
This patch relocates the variables kvm-s390 uses to track guest mem addr/size. As discussed dropping the variables at struct kvm_arch level allows to use the common vcpu->request based mechanism to reload guest memory if e.g. changes via set_memory_region.
The kick mechanism introduced in this series is used to ensure running vcpus leave guest state to catch the update.
---
[diffstat]
arch/s390/include/asm/kvm_host.h | 4 -- arch/s390/kvm/gaccess.h | 23 ++++++++------- arch/s390/kvm/intercept.c | 6 ++-- arch/s390/kvm/kvm-s390.c | 57 ++++++++++++++++----------------------- arch/s390/kvm/kvm-s390.h | 36 +++++++++++++++++++++++- arch/s390/kvm/sigp.c | 4 +- virt/kvm/kvm_main.c | 4 ++ 7 files changed, 81 insertions(+), 53 deletions(-)
[diff]
arch/s390/kvm/gaccess.h +12 -11
@@ -1,7 +1,7 @@/* * gaccess.h - access guest memory * -* Copyright IBM Corp. 2008 +* Copyright IBM Corp. 2008,2009 * * This program is free software; you can redistribute it and/or modify * it under the terms of the GNU General Public License (version 2 only)
@@ -16,13 +16,14 @@#include <linux/compiler.h> #include <linux/kvm_host.h> #include <asm/uaccess.h> +#include "kvm-s390.h" static inline void __user *__guestaddr_to_user (struct kvm_vcpu *vcpu,unsigned long guestaddr) {unsigned long prefix = vcpu->arch.sie_block->prefix; -unsigned long origin = vcpu->kvm->arch.guest_origin; -unsigned long memsize = vcpu->kvm->arch.guest_memsize; +unsigned long origin = vcpu->arch.sie_block->gmsor; +unsigned long memsize =kvm_s390_vcpu_get_memsize (vcpu);if (guestaddr <2 *PAGE_SIZE ) guestaddr += prefix;
@@ -158,8 +159,8 @@ static inline int copy_to_guest(struct k const void *from, unsigned long n) { unsigned long prefix = vcpu->arch.sie_block->prefix; -unsigned long origin = vcpu->kvm->arch.guest_origin; -unsigned long memsize = vcpu->kvm->arch.guest_memsize; +unsigned long origin = vcpu->arch.sie_block->gmsor; +unsigned long memsize =kvm_s390_vcpu_get_memsize (vcpu);if ((guestdest <2 *PAGE_SIZE ) && (guestdest + n >2 *PAGE_SIZE )) goto slowpath;
@@ -209,8 +210,8 @@ static inline int copy_from_guest(struct unsigned long guestsrc, unsigned long n) {unsigned long prefix = vcpu->arch.sie_block->prefix; -unsigned long origin = vcpu->kvm->arch.guest_origin; -unsigned long memsize = vcpu->kvm->arch.guest_memsize; +unsigned long origin = vcpu->arch.sie_block->gmsor; +unsigned long memsize =kvm_s390_vcpu_get_memsize (vcpu);if ((guestsrc <2 *PAGE_SIZE ) && (guestsrc + n >2 *PAGE_SIZE )) goto slowpath;
@@ -244,8 +245,8 @@ static inline int copy_to_guest_absoluteunsigned long guestdest, const void *from, unsigned long n) { - unsigned long origin = vcpu->kvm->arch.guest_origin; -unsigned long memsize = vcpu->kvm->arch.guest_memsize; + unsigned long origin = vcpu->arch.sie_block->gmsor; +unsigned long memsize =kvm_s390_vcpu_get_memsize (vcpu);if (guestdest + n > memsize)return -EFAULT ;
@@ -262,8 +263,8 @@ static inline int copy_from_guest_absolu unsigned long guestsrc, unsigned long n) { -unsigned long origin = vcpu->kvm->arch.guest_origin; -unsigned long memsize = vcpu->kvm->arch.guest_memsize; +unsigned long origin = vcpu->arch.sie_block->gmsor; +unsigned long memsize =kvm_s390_vcpu_get_memsize (vcpu);if (guestsrc + n > memsize)return -EFAULT ;
arch/s390/kvm/intercept.c +3 -3
@@ -1,7 +1,7 @@/* * intercept.c - in-kernel handling for sie intercepts * -* Copyright IBM Corp. 2008 +* Copyright IBM Corp. 2008,2009 * * This program is free software; you can redistribute it and/or modify * it under the terms of the GNU General Public License (version 2 only)
@@ -166,9 +166,9 @@ static int handle_validity(struct kvm_vc vcpu->stat.exit_validity++;if ((viwhy ==0x37 ) && (vcpu->arch.sie_block->prefix - <= vcpu->kvm->arch.guest_memsize -2 *PAGE_SIZE )){ + <=kvm_s390_vcpu_get_memsize (vcpu) -2 *PAGE_SIZE )) { rc =fault_in_pages_writeable ((char __user *) - vcpu->kvm->arch.guest_origin + + vcpu->arch.sie_block->gmsor + vcpu->arch.sie_block->prefix,2 *PAGE_SIZE );if (rc)
arch/s390/kvm/kvm-s390.c +24 -33
116 lines not on this page · raw message
arch/s390/kvm/kvm-s390.h +35 -1
@@ -1,7 +1,7 @@/* * kvm_s390.h - definition for kvm on s390 * -* Copyright IBM Corp. 2008 +* Copyright IBM Corp. 2008,2009 * * This program is free software; you can redistribute it and/or modify * it under the terms of the GNU General Public License (version 2 only)
@@ -9,6 +9,7 @@* * Author(s): Carsten Otte <[email protected]> * Christian Borntraeger <[email protected]> +* Christian Ehrhardt <[email protected]> */ #ifndef ARCH_S390_KVM_S390_H
@@ -18,6 +19,9 @@#include <linux/kvm.h> #include <linux/kvm_host.h> +/* The current code can have up to 256 pages for virtio */ +#define VIRTIODESCSPACE (256ul * 4096ul) +typedef int (*intercept_handler_t )(struct kvm_vcpu *vcpu);/* negativ values are error codes, positive values for internal conditions */
@@ -54,6 +58,29 @@ int kvm_s390_inject_vcpu(struct kvm_vcpuint kvm_s390_inject_program_int (struct kvm_vcpu *vcpu,u16 code);int kvm_s390_inject_sigp_stop (struct kvm_vcpu *vcpu,int action); +static inline int kvm_s390_vcpu_get_memsize (struct kvm_vcpu *vcpu) +{ +return vcpu->arch.sie_block->gmslm + - vcpu->arch.sie_block->gmsor + -VIRTIODESCSPACE +1ul ; +} + +static inline void kvm_s390_vcpu_set_mem (struct kvm_vcpu *vcpu) +{ +struct kvm_memory_slot *mem; + +down_read (&vcpu->kvm->slots_lock); + mem = &vcpu->kvm->memslots[0 ]; + + vcpu->arch.sie_block->gmsor = mem->userspace_addr; + vcpu->arch.sie_block->gmslm = + mem->userspace_addr + + (mem->npages <<PAGE_SHIFT ) + +VIRTIODESCSPACE -1ul ; + +up_read (&vcpu->kvm->slots_lock); +} +/* interception levels from which handle vcpu requests can be called */ #define VCPUREQUESTLVL_SIGP 1#define VCPUREQUESTLVL_VCPURUN 2
@@ -65,6 +92,13 @@ static inline unsigned long kvm_s390_hanif (!vcpu->requests)return 0 ; +/* requests that can be handled at all levels */ +if (test_and_clear_bit (KVM_REQ_MMU_RELOAD , &vcpu->requests)) { +smp_mb__after_clear_bit (); +wake_up_bit (vcpu->requests,KVM_REQ_MMU_RELOAD ); +kvm_s390_vcpu_set_mem (vcpu); + } +return vcpu->requests; }
arch/s390/include/asm/kvm_host.h +1 -3
@@ -1,7 +1,7 @@/* * asm-s390/kvm_host.h - definition for kernel virtual machines on s390 * -* Copyright IBM Corp. 2008 +* Copyright IBM Corp. 2008,2009 * * This program is free software; you can redistribute it and/or modify * it under the terms of the GNU General Public License (version 2 only)
@@ -226,8 +226,6 @@ struct kvm_vm_stat { };struct kvm_arch { -unsigned long guest_origin; -unsigned long guest_memsize;struct sca_block *sca;debug_info_t *dbf;struct kvm_s390_float_interrupt float_int;
arch/s390/kvm/sigp.c +2 -2
@@ -202,9 +202,9 @@ static int __sigp_set_prefix(struct kvm_/* make sure that the new value is valid memory */ address = address &0x7fffe000u ;if ((copy_from_guest (vcpu, &tmp, - (u64) (address + vcpu->kvm->arch.guest_origin) ,1 )) || + (u64) (address + vcpu->arch.sie_block->gmsor) ,1 )) || (copy_from_guest (vcpu, &tmp, (u64) (address + - vcpu->kvm->arch.guest_origin +PAGE_SIZE ),1 ))) { + vcpu->arch.sie_block->gmsor +PAGE_SIZE ),1 ))) { *reg |=SIGP_STAT_INVALID_PARAMETER ;return 1 ;/* invalid parameter */ }
virt/kvm/kvm_main.c +4 -0
@@ -1686,6 +1686,10 @@ static int kvm_vcpu_release(struct inode {struct kvm_vcpu *vcpu = filp->private_data; +clear_bit (vcpu->requests,KVM_REQ_MMU_RELOAD ); +smp_mb__after_clear_bit (); +wake_up_bit (vcpu->requests,KVM_REQ_MMU_RELOAD ); +kvm_put_kvm (vcpu->kvm);return 0 ; }
Re: [PATCH 4/4] kvm-s390: streamline memslot handling - v6
Quoted 6 lines · [email protected] wrote:
From: Christian Ehrhardt <[email protected]>
*updates in v6*
- ensure thewait_on_bitwaiter is notified
- move the reset of requests tokvm_vcpu_releaseto drop them early
Quoted 8 lines
*updates in v5*
- ensure dropping vcpu all requests while freeing a vcpu*updates in v4*
- kickout only scheduled vcpus (its superfluous and wait might hang forever on not running vcpus)
v3 is already in (and pushed so I can't unapply), so please rebase on top of current git.
Signature · 1 line
error compiling committee.c: too many arguments to function
Re: [PATCH 4/4] kvm-s390: streamline memslot handling - v6
Quoted 10 lines · Avi Kivity
Quoted 9 lines · [email protected] wrote:
From: Christian Ehrhardt <[email protected]>
*updates in v6*
- ensure thewait_on_bitwaiter is notified
- move the reset of requests tokvm_vcpu_releaseto drop them early*updates in v5*
- ensure dropping vcpu all requests while freeing a vcpu
Quoted 8 lines
Quoted 5 lines
*updates in v4*
- kickout only scheduled vcpus (its superfluous and wait might hang forever on not running vcpus)v3 is already in (and pushed so I can't unapply), so please rebase on top of current git.
Christian,
Seems a good step toward further unification. Can you please rebase as requested?
Re: [PATCH 4/4] kvm-s390: streamline memslot handling - v6
Quoted 24 lines · Marcelo Tosatti
Quoted 11 lines · Avi Kivity
Quoted 9 lines · [email protected] wrote:
From: Christian Ehrhardt <[email protected]>
*updates in v6*
- ensure thewait_on_bitwaiter is notified
- move the reset of requests tokvm_vcpu_releaseto drop them early*updates in v5*
- ensure dropping vcpu all requests while freeing a vcpuQuoted 8 lines
Quoted 6 lines
*updates in v4*
- kickout only scheduled vcpus (its superfluous and wait might hang forever on not running vcpus)v3 is already in (and pushed so I can't unapply), so please rebase on top of current git.
Christian,
Quoted 8 lines
Seems a good step toward further unification. Can you please rebase as requested?
--
To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
The missing updates are mostly fixes or code merges due to our discussions and should be on the list in a few minutes.
Signature · 2 lines
Grüsse / regards, Christian Ehrhardt IBM Linux Technology Center, Open Virtualization