KVM: s390: interrupt: Fix stepping into interrupt handlers
v4 cover KVM: s390: interrupt: Fix stepping into interrupt handlers
v3: https://lore.kernel.org/kvm/[email protected]/
v3 -> v4: Restore the per_event() macro (Claudio).
v2: https://lore.kernel.org/lkml/[email protected]/
v2 -> v3: Add comments, improve the commit messages (Christian, David).
Add R-bs.
Patches that need review: [4/6], [6/6].v1: https://lore.kernel.org/lkml/[email protected]/
v1 -> v2: Fix three more issues.
Add selftests (Claudio).
Hi,
I tried to compare the behavior of KVM and TCG by diffing instruction traces, and found five issues in KVM related to stepping into interrupt handlers.
Best regards,
Ilya
Ilya Leoshkevich (6):
KVM: s390: interrupt: Fix single-stepping into interrupt handlers
KVM: s390: interrupt: Fix single-stepping into program interrupt
handlers
KVM: s390: interrupt: Fix single-stepping kernel-emulated instructions
KVM: s390: interrupt: Fix single-stepping userspace-emulated
instructions
KVM: s390: interrupt: Fix single-stepping keyless mode exits
KVM: s390: selftests: Add selftest for single-stepping
arch/s390/kvm/intercept.c | 38 ++++- arch/s390/kvm/interrupt.c | 14 ++ arch/s390/kvm/kvm-s390.c | 27 ++- tools/testing/selftests/kvm/Makefile | 1 + .../testing/selftests/kvm/s390x/debug_test.c | 160 ++++++++++++++++++ 5 files changed, 229 insertions(+), 11 deletions(-) create mode 100644 tools/testing/selftests/kvm/s390x/debug_test.c
Signature · 1 line
2.41.0
PATCH v4 1/6 KVM: s390: interrupt: Fix single-stepping into interrupt handlers
After single-stepping an instruction that generates an interrupt, GDB ends up on the second instruction of the respective interrupt handler.
The reason is that vcpu_pre_run() manually delivers the interrupt, and then __vcpu_run() runs the first handler instruction using the CPUSTAT_P flag. This causes a KVM_SINGLESTEP exit on the second handler instruction.
Fix by delaying the KVM_SINGLESTEP exit until after the manual interrupt delivery.
--- arch/s390/kvm/interrupt.c | 14 ++++++++++++++ arch/s390/kvm/kvm-s390.c | 4 ++-- 2 files changed, 16 insertions(+), 2 deletions(-)
arch/s390/kvm/interrupt.c +14 -0
@@ -1392,6 +1392,7 @@ int __must_check kvm_s390_deliver_pending_interrupts(struct kvm_vcpu *vcpu) {struct kvm_s390_local_interrupt *li = &vcpu->arch.local_int;int rc =0 ; +bool delivered = false;unsigned long irq_type;unsigned long irqs;
@@ -1465,6 +1466,19 @@ int __must_check kvm_s390_deliver_pending_interrupts(struct kvm_vcpu *vcpu)WARN_ONCE (1 ,"Unknown pending irq type %ld" , irq_type);clear_bit (irq_type, &li->pending_irqs); } + delivered |= !rc; + } + +/* +* We delivered at least one interrupt and modified the PC. Force a +* singlestep event now. +*/ +if (delivered &&guestdbg_sstep_enabled (vcpu)) { +struct kvm_debug_exit_arch *debug_exit = &vcpu->run->debug.arch; + + debug_exit->addr = vcpu->arch.sie_block->gpsw.addr; + debug_exit->type =KVM_SINGLESTEP ; + vcpu->guest_debug |=KVM_GUESTDBG_EXIT_PENDING ; }set_intercept_indicators (vcpu);
arch/s390/kvm/kvm-s390.c +2 -2
@@ -4611,7 +4611,7 @@ static int vcpu_pre_run(struct kvm_vcpu *vcpu)if (!kvm_is_ucontrol (vcpu->kvm)) { rc =kvm_s390_deliver_pending_interrupts (vcpu); -if (rc) +if (rc ||guestdbg_exit_pending (vcpu))return rc; }
@@ -4738,7 +4738,7 @@ static int __vcpu_run(struct kvm_vcpu *vcpu) do { rc = vcpu_pre_run(vcpu); - if (rc) + if (rc || guestdbg_exit_pending(vcpu)) break; kvm_vcpu_srcu_read_unlock(vcpu);
Signature · 1 line
2.41.0
PATCH v4 2/6 KVM: s390: interrupt: Fix single-stepping into program interrupt handlers
In reply to Ilya LeoshkevichCurrently, after single-stepping an instruction that generates a specification exception, GDB ends up on the instruction immediately following it.
The reason is that vcpu_post_run() injects the interrupt and sets KVM_GUESTDBG_EXIT_PENDING, causing a KVM_SINGLESTEP exit. The interrupt is not delivered, however, therefore userspace sees the address of the next instruction.
Fix by letting the __vcpu_run() loop go into the next iteration, where vcpu_pre_run() delivers the interrupt and sets KVM_GUESTDBG_EXIT_PENDING.
--- arch/s390/kvm/intercept.c | 17 ++++++++++++++++- 1 file changed, 16 insertions(+), 1 deletion(-)
arch/s390/kvm/intercept.c +16 -1
@@ -228,6 +228,21 @@ static int handle_itdb(struct kvm_vcpu *vcpu)#define per_event (vcpu) (vcpu->arch.sie_block->iprcc & PGM_PER) +static bool should_handle_per_event (const struct kvm_vcpu *vcpu) +{ +if (!guestdbg_enabled (vcpu) || !per_event (vcpu)) +return false; +if (guestdbg_sstep_enabled (vcpu) && + vcpu->arch.sie_block->iprcc !=PGM_PER ) { +/* +* __vcpu_run() will exit after delivering the concurrently +* indicated condition. +*/ +return false; + } +return true; +} +static int handle_prog (struct kvm_vcpu *vcpu) {psw_t psw;
@@ -242,7 +257,7 @@ static int handle_prog(struct kvm_vcpu *vcpu)if (kvm_s390_pv_cpu_is_protected (vcpu))return -EOPNOTSUPP ; -if (guestdbg_enabled (vcpu) &&per_event (vcpu)) { +if (should_handle_per_event (vcpu)) { rc =kvm_s390_handle_per_event (vcpu);if (rc)return rc;
Signature · 1 line
2.41.0
Re: [PATCH v4 2/6] KVM: s390: interrupt: Fix single-stepping into program interrupt handlers
Quoted 9 lines · On Tue, 25 Jul 2023 16:37:17 +0200 Ilya Leoshkevich <[email protected]> wrote:
Currently, after single-stepping an instruction that generates a specification exception, GDB ends up on the instruction immediately following it.
The reason is that
vcpu_post_run()injects the interrupt and setsKVM_GUESTDBG_EXIT_PENDING, causing aKVM_SINGLESTEPexit. The interrupt is not delivered, however, therefore userspace sees the address of the next instruction.
Quoted 6 lines
Fix by letting the
__vcpu_run()loop go into the next iteration, wherevcpu_pre_run()delivers the interrupt and setsKVM_GUESTDBG_EXIT_PENDING.Reviewed-by David Hildenbrand <[email protected]>Signed-off-by Ilya Leoshkevich <[email protected]>
Quoted 39 lines
--- arch/s390/kvm/intercept.c | 17 ++++++++++++++++- 1 file changed, 16 insertions(+), 1 deletion(-)arch/s390/kvm/intercept.c +16
-1@@ -228,6 +228,21 @@ static int handle_itdb(struct kvm_vcpu *vcpu)#define per_event (vcpu) (vcpu->arch.sie_block->iprcc & PGM_PER) +static bool should_handle_per_event (const struct kvm_vcpu *vcpu) +{ +if (!guestdbg_enabled (vcpu) || !per_event (vcpu)) +return false; +if (guestdbg_sstep_enabled (vcpu) && + vcpu->arch.sie_block->iprcc !=PGM_PER ) { +/* +* __vcpu_run() will exit after delivering the concurrently +* indicated condition. +*/ +return false; + } +return true; +} +static int handle_prog (struct kvm_vcpu *vcpu) {psw_t psw;@@ -242,7 +257,7 @@ static int handle_prog(struct kvm_vcpu *vcpu)if (kvm_s390_pv_cpu_is_protected (vcpu))return -EOPNOTSUPP ; -if (guestdbg_enabled (vcpu) &&per_event (vcpu)) { +if (should_handle_per_event (vcpu)) { rc =kvm_s390_handle_per_event (vcpu);if (rc)return rc;
PATCH v4 3/6 KVM: s390: interrupt: Fix single-stepping kernel-emulated instructions
In reply to Ilya LeoshkevichSingle-stepping a kernel-emulated instruction that generates an interrupt causes GDB to land on the instruction following it instead of the respective interrupt handler.
The reason is that kvm_handle_sie_intercept(), after injecting the interrupt, also processes the PER event and arranges a KVM_SINGLESTEP exit. The interrupt is not yet delivered, however, so the userspace sees the next instruction.
Fix by avoiding the KVM_SINGLESTEP exit when there is a pending interrupt. The next __vcpu_run() loop iteration will arrange a KVM_SINGLESTEP exit after delivering the interrupt.
--- arch/s390/kvm/intercept.c | 17 ++++++++++++++--- 1 file changed, 14 insertions(+), 3 deletions(-)
arch/s390/kvm/intercept.c +14 -3
@@ -583,6 +583,19 @@ static int handle_pv_notification(struct kvm_vcpu *vcpu)return handle_instruction (vcpu); } +static bool should_handle_per_ifetch (const struct kvm_vcpu *vcpu,int rc) +{ +/* Process PER, also if the instruction is processed in user space. */ +if (!(vcpu->arch.sie_block->icptstatus &0x02 )) +return false; +if (rc !=0 && rc != -EOPNOTSUPP ) +return false; +if (guestdbg_sstep_enabled (vcpu) && vcpu->arch.local_int.pending_irqs) +/* __vcpu_run() will exit after delivering the interrupt. */ +return false; +return true; +} +int kvm_handle_sie_intercept (struct kvm_vcpu *vcpu) {int rc, per_rc =0 ;
@@ -645,9 +658,7 @@ int kvm_handle_sie_intercept(struct kvm_vcpu *vcpu)return -EOPNOTSUPP ; } -/* process PER, also if the instruction is processed in user space */ -if (vcpu->arch.sie_block->icptstatus &0x02 && - (!rc || rc == -EOPNOTSUPP )) +if (should_handle_per_ifetch (vcpu, rc)) per_rc =kvm_s390_handle_per_ifetch_icpt (vcpu);return per_rc ? per_rc : rc; }
Signature · 1 line
2.41.0
PATCH v4 4/6 KVM: s390: interrupt: Fix single-stepping userspace-emulated instructions
In reply to Ilya LeoshkevichSingle-stepping a userspace-emulated instruction that generates an interrupt causes GDB to land on the instruction following it instead of the respective interrupt handler.
The reason is that after arranging a KVM_EXIT_S390_SIEIC exit, kvm_handle_sie_intercept() calls kvm_s390_handle_per_ifetch_icpt(), which sets KVM_GUESTDBG_EXIT_PENDING. This bit, however, is not processed immediately, but rather persists until the next ioctl(), causing a spurious single-step exit.
Fix by clearing this bit in ioctl().
--- arch/s390/kvm/kvm-s390.c | 23 ++++++++++++++++++++--- 1 file changed, 20 insertions(+), 3 deletions(-)
arch/s390/kvm/kvm-s390.c +20 -3
@@ -5383,6 +5383,7 @@ long kvm_arch_vcpu_async_ioctl(struct file *filp, {struct kvm_vcpu *vcpu = filp->private_data;void __user *argp = (void __user *)arg; +int rc;switch (ioctl) {case KVM_S390_IRQ : {
@@ -5390,7 +5391,8 @@ long kvm_arch_vcpu_async_ioctl(struct file *filp,if (copy_from_user (&s390irq, argp,sizeof (s390irq)))return -EFAULT ; -return kvm_s390_inject_vcpu (vcpu, &s390irq); + rc =kvm_s390_inject_vcpu (vcpu, &s390irq); +break ; }case KVM_S390_INTERRUPT : {struct kvm_s390_interrupt s390int;
@@ -5400,10 +5402,25 @@ long kvm_arch_vcpu_async_ioctl(struct file *filp,return -EFAULT ;if (s390int_to_s390irq (&s390int, &s390irq))return -EINVAL ; -return kvm_s390_inject_vcpu (vcpu, &s390irq); + rc =kvm_s390_inject_vcpu (vcpu, &s390irq); +break ; } +default : + rc = -ENOIOCTLCMD ; +break ; } -return -ENOIOCTLCMD ; + +/* +* To simplify single stepping of userspace-emulated instructions, +* KVM_EXIT_S390_SIEIC exit sets KVM_GUESTDBG_EXIT_PENDING (see +* should_handle_per_ifetch()). However, if userspace emulation injects +* an interrupt, it needs to be cleared, so that KVM_EXIT_DEBUG happens +* after (and not before) the interrupt delivery. +*/ +if (!rc) + vcpu->guest_debug &= ~KVM_GUESTDBG_EXIT_PENDING ; + +return rc; } static int kvm_s390_handle_pv_vcpu_dump(struct kvm_vcpu *vcpu,
Signature · 1 line
2.41.0
PATCH v4 5/6 KVM: s390: interrupt: Fix single-stepping keyless mode exits
In reply to Ilya Leoshkevichkvm_s390_skey_check_enable() does not emulate any instructions, rather, it clears CPUSTAT_KSS and arranges the instruction that caused the exit (e.g., ISKE, SSKE, RRBE or LPSWE with a keyed PSW) to run again.
Therefore, skip the PER check and let the instruction execution happen. Otherwise, a debugger will see two single-step events on the same instruction.
--- arch/s390/kvm/intercept.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
arch/s390/kvm/intercept.c +2 -2
@@ -630,8 +630,8 @@ int kvm_handle_sie_intercept(struct kvm_vcpu *vcpu) rc =handle_partial_execution (vcpu);break ;case ICPT_KSS : - rc =kvm_s390_skey_check_enable (vcpu); -break ; +/* Instruction will be redriven, skip the PER check. */ +return kvm_s390_skey_check_enable (vcpu);case ICPT_MCHKREQ :case ICPT_INT_ENABLE :/*
Signature · 1 line
2.41.0
PATCH v4 6/6 KVM: s390: selftests: Add selftest for single-stepping
In reply to Ilya LeoshkevichTest different variations of single-stepping into interrupts:
- SVC and PGM interrupts;
- Interrupts generated by ISKE;
- Interrupts generated by instructions emulated by KVM;
- Interrupts generated by instructions emulated by userspace.
--- tools/testing/selftests/kvm/Makefile | 1 + .../testing/selftests/kvm/s390x/debug_test.c | 160 ++++++++++++++++++ 2 files changed, 161 insertions(+) create mode 100644 tools/testing/selftests/kvm/s390x/debug_test.c
tools/testing/selftests/kvm/Makefile +1 -0
@@ -166,6 +166,7 @@ TEST_GEN_PROGS_s390x += s390x/resets TEST_GEN_PROGS_s390x += s390x/sync_regs_test TEST_GEN_PROGS_s390x += s390x/tprot TEST_GEN_PROGS_s390x += s390x/cmma_test +TEST_GEN_PROGS_s390x += s390x/debug_test TEST_GEN_PROGS_s390x += demand_paging_test TEST_GEN_PROGS_s390x += dirty_log_test TEST_GEN_PROGS_s390x += kvm_create_max_vcpus
tools/testing/selftests/kvm/s390x/debug_test.c new file +160 -0
160 lines not on this page · raw message
Signature · 1 line
2.41.0
Sean Christopherson
reviewerQuoted 9 lines · Ilya Leoshkevich
Ilya Leoshkevich (6):
KVM: s390: interrupt: Fix single-stepping into interrupt handlers
KVM: s390: interrupt: Fix single-stepping into program interrupt
handlers
KVM: s390: interrupt: Fix single-stepping kernel-emulated instructions
KVM: s390: interrupt: Fix single-stepping userspace-emulated
instructions
KVM: s390: interrupt: Fix single-stepping keyless mode exits
KVM: s390: selftests: Add selftest for single-stepping
FYI, the selftests change silently conflicts with a global s/ASSERT_EQ/TEST_ASSERT_EQ rename[1], but the conflicts are very straightforward to resolve (just prepend TEST_). If we want to proactively avoid mild pain in linux-next, one option would be to merge the full kvm-x86/selftests branch/tag once I've made that immutable[2] (will be done Friday if there are no fireworks). Though we can probably just get away with doing nothing other than letting Paolo know there's a silent conflict.
[1] https://lore.kernel.org/all/[email protected]
[2] https://lore.kernel.org/all/[email protected]
Christian Borntraeger
maintainerAm 03.08.23 um 00:24 schrieb Sean Christopherson:
Quoted 18 lines · Ilya Leoshkevich
Quoted 9 lines
Ilya Leoshkevich (6):
KVM: s390: interrupt: Fix single-stepping into interrupt handlers
KVM: s390: interrupt: Fix single-stepping into program interrupt
handlers
KVM: s390: interrupt: Fix single-stepping kernel-emulated instructions
KVM: s390: interrupt: Fix single-stepping userspace-emulated
instructions
KVM: s390: interrupt: Fix single-stepping keyless mode exits
KVM: s390: selftests: Add selftest for single-steppingFYI, the selftests change silently conflicts with a global s/ASSERT_EQ/TEST_ASSERT_EQ rename[1], but the conflicts are very straightforward to resolve (just prepend
TEST_). If we want to proactively avoid mild pain in linux-next, one option would be to merge the full kvm-x86/selftests branch/tag once I've made that immutable[2] (will be done Friday if there are no fireworks). Though we can probably just get away with doing nothing other than letting Paolo know there's a silent conflict.
Quoted 2 lines
[1] https://lore.kernel.org/all/[email protected]
[2] https://lore.kernel.org/all/[email protected]
Thanks for telling. Paolo, do you have a preference?
Janosch, Claudio I think this series is good to go otherwise.