arm: warning at virt/kvm/arm/vgic.c:1468

Hi,

after fixing the VM_BUG_ON, my QEMU guest on the Jetson TK1 generally refuses to boot. Once in a while it does, but quickly gets stuck again. In one case I found this in the kernel log (never happened again so far):

[  762.022874] WARNING: CPU: 1 PID: 972 at ../arch/arm/kvm/../../../virt/kvm/arm/vgic.c:1468 kvm_vgic_sync_hwstate+0x314/0x344()
[  762.022884] Modules linked in:
[  762.022902] CPU: 1 PID: 972 Comm: qemu-system-arm Not tainted 3.19.0-rc7-00221-gfd7a168-dirty #13
[  762.022911] Hardware name: NVIDIA Tegra SoC (Flattened Device Tree)
[  762.022937] [<c0025d80>] (unwind_backtrace) from [<c00215d8>] (show_stack+0x10/0x14)
[  762.022958] [<c00215d8>] (show_stack) from [<c065f494>] (dump_stack+0x98/0xd8)
[  762.022976] [<c065f494>] (dump_stack) from [<c0035f38>] (warn_slowpath_common+0x80/0xb0)
[  762.022991] [<c0035f38>] (warn_slowpath_common) from [<c0036004>] (warn_slowpath_null+0x1c/0x24)
[  762.023007] [<c0036004>] (warn_slowpath_null) from [<c001c3c4>] (kvm_vgic_sync_hwstate+0x314/0x344)
[  762.023024] [<c001c3c4>] (kvm_vgic_sync_hwstate) from [<c00147c0>] (kvm_arch_vcpu_ioctl_run+0x210/0x400)
[  762.023041] [<c00147c0>] (kvm_arch_vcpu_ioctl_run) from [<c001063c>] (kvm_vcpu_ioctl+0x2e4/0x6ec)
[  762.023059] [<c001063c>] (kvm_vcpu_ioctl) from [<c01098a4>] (do_vfs_ioctl+0x40c/0x600)
[  762.023076] [<c01098a4>] (do_vfs_ioctl) from [<c0109acc>] (SyS_ioctl+0x34/0x5c)
[  762.023091] [<c0109acc>] (SyS_ioctl) from [<c001e320>] (ret_fast_syscall+0x0/0x34)

BTW, KVM tracing support on ARM seems like it requires some care. E.g.:
kvm_exit does not report an exit reason. The in-kernel vgic also seems to lack instrumentation. Unfortunate. Tracing is usually the first stop when KVM is stuck on a guest.

Hi Jan,

Quoted 7 lines · Jan Kiszka

Hi,

after fixing the VM_BUG_ON, my QEMU guest on the Jetson TK1 generally refuses to boot. Once in a while it does, but quickly gets stuck again. In one case I found this in the kernel log (never happened again so far):

Quoted 14 lines

[  762.022874] WARNING: CPU: 1 PID: 972 at ../arch/arm/kvm/../../../virt/kvm/arm/vgic.c:1468 kvm_vgic_sync_hwstate+0x314/0x344()
[  762.022884] Modules linked in:
[  762.022902] CPU: 1 PID: 972 Comm: qemu-system-arm Not tainted 3.19.0-rc7-00221-gfd7a168-dirty #13
[  762.022911] Hardware name: NVIDIA Tegra SoC (Flattened Device Tree)
[  762.022937] [<c0025d80>] (unwind_backtrace) from [<c00215d8>] (show_stack+0x10/0x14)
[  762.022958] [<c00215d8>] (show_stack) from [<c065f494>] (dump_stack+0x98/0xd8)
[  762.022976] [<c065f494>] (dump_stack) from [<c0035f38>] (warn_slowpath_common+0x80/0xb0)
[  762.022991] [<c0035f38>] (warn_slowpath_common) from [<c0036004>] (warn_slowpath_null+0x1c/0x24)
[  762.023007] [<c0036004>] (warn_slowpath_null) from [<c001c3c4>] (kvm_vgic_sync_hwstate+0x314/0x344)
[  762.023024] [<c001c3c4>] (kvm_vgic_sync_hwstate) from [<c00147c0>] (kvm_arch_vcpu_ioctl_run+0x210/0x400)
[  762.023041] [<c00147c0>] (kvm_arch_vcpu_ioctl_run) from [<c001063c>] (kvm_vcpu_ioctl+0x2e4/0x6ec)
[  762.023059] [<c001063c>] (kvm_vcpu_ioctl) from [<c01098a4>] (do_vfs_ioctl+0x40c/0x600)
[  762.023076] [<c01098a4>] (do_vfs_ioctl) from [<c0109acc>] (SyS_ioctl+0x34/0x5c)
[  762.023091] [<c0109acc>] (SyS_ioctl) from [<c001e320>] (ret_fast_syscall+0x0/0x34)

so this means your guest caused a maintenance interrupt and the bit is set in the GICH_EISR for the LR in question but the link register state is not 0, which is in direct violation of the GIC spec. Hmmmm.

You're not doing any IRQ forwarding stuff or device passthrough here are you?

Quoted 7 lines

BTW, KVM tracing support on ARM seems like it requires some care. E.g.:
kvm_exit does not report an exit reason. The in-kernel vgic also seems to lack instrumentation. Unfortunate. Tracing is usually the first stop when KVM is stuck on a guest.

I know, the exit reason is on my todo list, and Alex B is sitting on trace patches for the gic. Coming soon to a git repo near your.

Hi Christoffer,

Quoted 25 lines · Christoffer Dall

Hi Jan,

Quoted 7 lines · Jan Kiszka

Hi,

after fixing the VM_BUG_ON, my QEMU guest on the Jetson TK1 generally refuses to boot. Once in a while it does, but quickly gets stuck again. In one case I found this in the kernel log (never happened again so far):

Quoted 14 lines

[  762.022874] WARNING: CPU: 1 PID: 972 at ../arch/arm/kvm/../../../virt/kvm/arm/vgic.c:1468 kvm_vgic_sync_hwstate+0x314/0x344()
[  762.022884] Modules linked in:
[  762.022902] CPU: 1 PID: 972 Comm: qemu-system-arm Not tainted 3.19.0-rc7-00221-gfd7a168-dirty #13
[  762.022911] Hardware name: NVIDIA Tegra SoC (Flattened Device Tree)
[  762.022937] [<c0025d80>] (unwind_backtrace) from [<c00215d8>] (show_stack+0x10/0x14)
[  762.022958] [<c00215d8>] (show_stack) from [<c065f494>] (dump_stack+0x98/0xd8)
[  762.022976] [<c065f494>] (dump_stack) from [<c0035f38>] (warn_slowpath_common+0x80/0xb0)
[  762.022991] [<c0035f38>] (warn_slowpath_common) from [<c0036004>] (warn_slowpath_null+0x1c/0x24)
[  762.023007] [<c0036004>] (warn_slowpath_null) from [<c001c3c4>] (kvm_vgic_sync_hwstate+0x314/0x344)
[  762.023024] [<c001c3c4>] (kvm_vgic_sync_hwstate) from [<c00147c0>] (kvm_arch_vcpu_ioctl_run+0x210/0x400)
[  762.023041] [<c00147c0>] (kvm_arch_vcpu_ioctl_run) from [<c001063c>] (kvm_vcpu_ioctl+0x2e4/0x6ec)
[  762.023059] [<c001063c>] (kvm_vcpu_ioctl) from [<c01098a4>] (do_vfs_ioctl+0x40c/0x600)
[  762.023076] [<c01098a4>] (do_vfs_ioctl) from [<c0109acc>] (SyS_ioctl+0x34/0x5c)
[  762.023091] [<c0109acc>] (SyS_ioctl) from [<c001e320>] (ret_fast_syscall+0x0/0x34)

Quoted 6 lines

so this means your guest caused a maintenance interrupt and the bit is set in the GICH_EISR for the LR in question but the link register state is not 0, which is in direct violation of the GIC spec. Hmmmm.

You're not doing any IRQ forwarding stuff or device passthrough here are you?

No, just boring emulation. The command line is

qemu-system-ar -machine vexpress-a15 -kernel zImage -serial mon:stdio -append 'console=ttyAMA0 root=/dev/mmcblk0 rw' -snapshot -sd OpenSuse13-1_arm.img -dtb vexpress-v2p-ca15-tc1.dtb -s -enable-kvm

Quoted 8 lines

Quoted 7 lines

BTW, KVM tracing support on ARM seems like it requires some care. E.g.:
kvm_exit does not report an exit reason. The in-kernel vgic also seems to lack instrumentation. Unfortunate. Tracing is usually the first stop when KVM is stuck on a guest.

Quoted 2 lines

I know, the exit reason is on my todo list, and Alex B is sitting on trace patches for the gic. Coming soon to a git repo near your.

Cool, looking forward.

Next thing I noticed is that guest debugging via qemu causes troubles in kvm mode. For some reason, qemu is unable to write soft-breakpoints, thus not even a single-step works. Also known?

Hi Jan,

Quoted 7 lines · Jan Kiszka

Hi,

after fixing the VM_BUG_ON, my QEMU guest on the Jetson TK1 generally refuses to boot. Once in a while it does, but quickly gets stuck again. In one case I found this in the kernel log (never happened again so far):

Quoted 14 lines

[  762.022874] WARNING: CPU: 1 PID: 972 at ../arch/arm/kvm/../../../virt/kvm/arm/vgic.c:1468 kvm_vgic_sync_hwstate+0x314/0x344()
[  762.022884] Modules linked in:
[  762.022902] CPU: 1 PID: 972 Comm: qemu-system-arm Not tainted 3.19.0-rc7-00221-gfd7a168-dirty #13
[  762.022911] Hardware name: NVIDIA Tegra SoC (Flattened Device Tree)
[  762.022937] [<c0025d80>] (unwind_backtrace) from [<c00215d8>] (show_stack+0x10/0x14)
[  762.022958] [<c00215d8>] (show_stack) from [<c065f494>] (dump_stack+0x98/0xd8)
[  762.022976] [<c065f494>] (dump_stack) from [<c0035f38>] (warn_slowpath_common+0x80/0xb0)
[  762.022991] [<c0035f38>] (warn_slowpath_common) from [<c0036004>] (warn_slowpath_null+0x1c/0x24)
[  762.023007] [<c0036004>] (warn_slowpath_null) from [<c001c3c4>] (kvm_vgic_sync_hwstate+0x314/0x344)
[  762.023024] [<c001c3c4>] (kvm_vgic_sync_hwstate) from [<c00147c0>] (kvm_arch_vcpu_ioctl_run+0x210/0x400)
[  762.023041] [<c00147c0>] (kvm_arch_vcpu_ioctl_run) from [<c001063c>] (kvm_vcpu_ioctl+0x2e4/0x6ec)
[  762.023059] [<c001063c>] (kvm_vcpu_ioctl) from [<c01098a4>] (do_vfs_ioctl+0x40c/0x600)
[  762.023076] [<c01098a4>] (do_vfs_ioctl) from [<c0109acc>] (SyS_ioctl+0x34/0x5c)
[  762.023091] [<c0109acc>] (SyS_ioctl) from [<c001e320>] (ret_fast_syscall+0x0/0x34)

so this means your guest caused a maintenance interrupt and the bit is set in the GICH_EISR for the LR in question but the link register state is not 0, which is in direct violation of the GIC spec. Hmmmm.

You're not doing any IRQ forwarding stuff or device passthrough here are you?

Quoted 2 lines

Quoted 4 lines

BTW, KVM tracing support on ARM seems like it requires some care. E.g.:
kvm_exit does not report an exit reason. The in-kernel vgic also seems to lack instrumentation. Unfortunate. Tracing is usually the first stop when KVM is stuck on a guest.

I know, the exit reason is on my todo list, and Alex B is sitting on trace patches for the gic. Coming soon to a git repo near your.

-Christoffer
_______________________________________________
kvmarm mailing list
[email protected]
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm

Christoffer Dall <[email protected]> writes:

Quoted 1 line · Jan Kiszka

Quoted 5 lines

Quoted 4 lines

BTW, KVM tracing support on ARM seems like it requires some care. E.g.:
kvm_exit does not report an exit reason. The in-kernel vgic also seems to lack instrumentation. Unfortunate. Tracing is usually the first stop when KVM is stuck on a guest.

Quoted 2 lines

I know, the exit reason is on my todo list, and Alex B is sitting on trace patches for the gic. Coming soon to a git repo near your.

For the impatient the raw patches are in:

git.linaro.org/people/alex.bennee/linux.git
migration/v3.19-rc7-improve-tracing

Quoted 1 line · Alex Bennée

Christoffer Dall <[email protected]> writes:

<snip>

Quoted 1 line

Quoted 1 line · Jan Kiszka

<snip>

Quoted 8 lines

Quoted 7 lines

Quoted 4 lines

BTW, KVM tracing support on ARM seems like it requires some care. E.g.:
kvm_exit does not report an exit reason. The in-kernel vgic also seems to lack instrumentation. Unfortunate. Tracing is usually the first stop when KVM is stuck on a guest.

I know, the exit reason is on my todo list, and Alex B is sitting on trace patches for the gic. Coming soon to a git repo near your.

Quoted 4 lines

For the impatient the raw patches are in:

git.linaro.org/people/alex.bennee/linux.git
migration/v3.19-rc7-improve-tracing

OK try tracing/kvm-exit-entry for something cleaner.

Quoted 15 lines · Alex Bennée

Quoted 1 line · Alex Bennée

Christoffer Dall <[email protected]> writes:

<snip>

Quoted 1 line

Quoted 1 line · Jan Kiszka

<snip>

Quoted 8 lines

Quoted 7 lines

Quoted 4 lines

BTW, KVM tracing support on ARM seems like it requires some care. E.g.:
kvm_exit does not report an exit reason. The in-kernel vgic also seems to lack instrumentation. Unfortunate. Tracing is usually the first stop when KVM is stuck on a guest.

I know, the exit reason is on my todo list, and Alex B is sitting on trace patches for the gic. Coming soon to a git repo near your.

Quoted 6 lines

Quoted 4 lines

For the impatient the raw patches are in:

git.linaro.org/people/alex.bennee/linux.git
migration/v3.19-rc7-improve-tracing

OK try tracing/kvm-exit-entry for something cleaner.

Doesn't build for ARM (vcpu_sys_reg is ARM64-only so far).

But the values traced seem useful. Wei Huang's patch in kvm.git queue traces the exception class, but unfortunately nothing else. When would we need that class? Do we need it at all?

In any case, please add symbolic printing of the magic values whenever possible, just like on x86.

I'm now throwing trace_printk at my broken KVM. Already found out that I get ARM_EXCEPTION_IRQ every few 10 µs. Not seeing any irq_* traces, though. Weird.

Quoted 3 lines · Jan Kiszka

I'm now throwing trace_printk at my broken KVM. Already found out that I get ARM_EXCEPTION_IRQ every few 10 µs. Not seeing any irq_* traces, though. Weird.

This very much looks like a screaming interrupt. At such a rate, no wonder your VM make much progress. Can you find out which interrupt is screaming like this? Looking at GICC_HPPIR should help, but you'll have to map the CPU interface in HYP before being able to access it there.

Do you have an form of power-management on this system?

Quoted 5 lines · Marc Zyngier

Quoted 3 lines · Jan Kiszka

I'm now throwing trace_printk at my broken KVM. Already found out that I get ARM_EXCEPTION_IRQ every few 10 µs. Not seeing any irq_* traces, though. Weird.

Quoted 4 lines

This very much looks like a screaming interrupt. At such a rate, no wonder your VM make much progress. Can you find out which interrupt is screaming like this? Looking at GICC_HPPIR should help, but you'll have to map the CPU interface in HYP before being able to access it there.

OK... let me figure this out. I had this suspect as well - the host gets a VM exit for each injected guest IRQ?

BTW, I also tried with in-kernel GIC disabled (in the kernel config), but I guess that's pointless. Linux seems to be stuck on a non-functional architectural timer then, right?

Quoted 2 lines

Do you have an form of power-management on this system?

Just killed every config that has PM for FREQ in its name, but that makes no difference.

Quoted 11 lines · Jan Kiszka

Quoted 9 lines · Marc Zyngier

Quoted 3 lines · Jan Kiszka

I'm now throwing trace_printk at my broken KVM. Already found out that I get ARM_EXCEPTION_IRQ every few 10 µs. Not seeing any irq_* traces, though. Weird.

This very much looks like a screaming interrupt. At such a rate, no wonder your VM make much progress. Can you find out which interrupt is screaming like this? Looking at GICC_HPPIR should help, but you'll have to map the CPU interface in HYP before being able to access it there.

Quoted 2 lines

OK... let me figure this out. I had this suspect as well - the host gets a VM exit for each injected guest IRQ?

Not exactly. There is a VM exit for each physical interrupt that fires while the guest is running. Injecting an interrupt also causes a VM exit, as we force the vcpu to reload its context.

Quoted 4 lines

BTW, I also tried with in-kernel GIC disabled (in the kernel config), but I guess that's pointless. Linux seems to be stuck on a non-functional architectural timer then, right?

Yes. Useful for bringup, but nothing more.

Quoted 6 lines

Quoted 3 lines

Do you have an form of power-management on this system?

Just killed every config that has PM for FREQ in its name, but that makes no difference.

I still wonder if the 4+1 design on the K1 is not playing tricks behind our back. Having talked to Ian Campbell earlier this week, he also can't manage to run guests in Xen on this platform, so there's something rather fishy here.

Quoted 16 lines · Marc Zyngier

Quoted 14 lines · Jan Kiszka

Quoted 10 lines · Marc Zyngier

Quoted 3 lines · Jan Kiszka

I'm now throwing trace_printk at my broken KVM. Already found out that I get ARM_EXCEPTION_IRQ every few 10 µs. Not seeing any irq_* traces, though. Weird.

This very much looks like a screaming interrupt. At such a rate, no wonder your VM make much progress. Can you find out which interrupt is screaming like this? Looking at GICC_HPPIR should help, but you'll have to map the CPU interface in HYP before being able to access it there.

OK... let me figure this out. I had this suspect as well - the host gets a VM exit for each injected guest IRQ?

Quoted 3 lines

Not exactly. There is a VM exit for each physical interrupt that fires while the guest is running. Injecting an interrupt also causes a VM exit, as we force the vcpu to reload its context.

Ah, GICC != GICV - you are referring to host-side pending IRQs. Any hints on how to get access to that register would accelerate the analysis (ARM KVM code is still new to me).

Quoted 5 lines

Quoted 3 lines

BTW, I also tried with in-kernel GIC disabled (in the kernel config), but I guess that's pointless. Linux seems to be stuck on a non-functional architectural timer then, right?

Yes. Useful for bringup, but nothing more.

Maybe we should perform a feature check and issue a warning from QEMU?

Quoted 4 lines

I still wonder if the 4+1 design on the K1 is not playing tricks behind our back. Having talked to Ian Campbell earlier this week, he also can't manage to run guests in Xen on this platform, so there's something rather fishy here.

Interesting. The announcements of his PSCI patches [1] sounded more promising. Maybe it was only referring to getting the hypervisor itself running...

To my current (still limited understanding) of that platform would say that this little core is parked after power-up of the main APs. And as we do not power them down, there is no reason to perform a cluster switch or anything similarly nasty, no?

Quoted 21 lines · Jan Kiszka

Quoted 19 lines · Marc Zyngier

Quoted 7 lines · Jan Kiszka

Quoted 6 lines · Marc Zyngier

Quoted 3 lines · Jan Kiszka

I'm now throwing trace_printk at my broken KVM. Already found out that I get ARM_EXCEPTION_IRQ every few 10 µs. Not seeing any irq_* traces, though. Weird.

Quoted 7 lines

Quoted 4 lines

This very much looks like a screaming interrupt. At such a rate, no wonder your VM make much progress. Can you find out which interrupt is screaming like this? Looking at GICC_HPPIR should help, but you'll have to map the CPU interface in HYP before being able to access it there.

OK... let me figure this out. I had this suspect as well - the host gets a VM exit for each injected guest IRQ?

Not exactly. There is a VM exit for each physical interrupt that fires while the guest is running. Injecting an interrupt also causes a VM exit, as we force the vcpu to reload its context.

Quoted 3 lines

Ah, GICC != GICV - you are referring to host-side pending IRQs. Any hints on how to get access to that register would accelerate the analysis (ARM KVM code is still new to me).

Map the GICC region in HYP using create_hyp_io_mapping (see vgic_v2_probe for an example of how we map GICH), and stash the read of GICC_HPPIR before leaving HYP mode (and before saving the guest timer).

BTW, when you look at /proc/interrupts on the host, don't you see an interrupt that's a bit too eager to fire?

Quoted 8 lines

Quoted 6 lines

Quoted 4 lines

BTW, I also tried with in-kernel GIC disabled (in the kernel config), but I guess that's pointless. Linux seems to be stuck on a non-functional architectural timer then, right?

Yes. Useful for bringup, but nothing more.

Maybe we should perform a feature check and issue a warning from QEMU?

I'd assume this is already in place (but I almost never run QEMU, so I could be wrong here).

Quoted 9 lines

Quoted 5 lines

I still wonder if the 4+1 design on the K1 is not playing tricks behind our back. Having talked to Ian Campbell earlier this week, he also can't manage to run guests in Xen on this platform, so there's something rather fishy here.

Interesting. The announcements of his PSCI patches [1] sounded more promising. Maybe it was only referring to getting the hypervisor itself running...

This is my understanding so far.

Quoted 4 lines

To my current (still limited understanding) of that platform would say that this little core is parked after power-up of the main APs. And as we do not power them down, there is no reason to perform a cluster switch or anything similarly nasty, no?

I can't see why this would happen, but I've learned not to assume anything when it come to braindead creativity on the HW side...

Quoted 22 lines · Marc Zyngier

Quoted 21 lines · Jan Kiszka

Quoted 19 lines · Marc Zyngier

Quoted 7 lines · Jan Kiszka

Quoted 6 lines · Marc Zyngier

Quoted 3 lines · Jan Kiszka

I'm now throwing trace_printk at my broken KVM. Already found out that I get ARM_EXCEPTION_IRQ every few 10 µs. Not seeing any irq_* traces, though. Weird.

Quoted 7 lines

Quoted 4 lines

This very much looks like a screaming interrupt. At such a rate, no wonder your VM make much progress. Can you find out which interrupt is screaming like this? Looking at GICC_HPPIR should help, but you'll have to map the CPU interface in HYP before being able to access it there.

OK... let me figure this out. I had this suspect as well - the host gets a VM exit for each injected guest IRQ?

Not exactly. There is a VM exit for each physical interrupt that fires while the guest is running. Injecting an interrupt also causes a VM exit, as we force the vcpu to reload its context.

Quoted 7 lines

Quoted 3 lines

Ah, GICC != GICV - you are referring to host-side pending IRQs. Any hints on how to get access to that register would accelerate the analysis (ARM KVM code is still new to me).

Map the GICC region in HYP using create_hyp_io_mapping (see vgic_v2_probe for an example of how we map GICH), and stash the read of GICC_HPPIR before leaving HYP mode (and before saving the guest timer).

OK.

Quoted 3 lines

BTW, when you look at /proc/interrupts on the host, don't you see an interrupt that's a bit too eager to fire?

No - but that makes sense given that we do not enter any interrupt handler according to ftrace, thus there can't be any counter incrementation.

Quoted 7 lines

Quoted 7 lines

Quoted 6 lines

Quoted 4 lines

BTW, I also tried with in-kernel GIC disabled (in the kernel config), but I guess that's pointless. Linux seems to be stuck on a non-functional architectural timer then, right?

Yes. Useful for bringup, but nothing more.

Quoted 4 lines

Quoted 1 line

Maybe we should perform a feature check and issue a warning from QEMU?

I'd assume this is already in place (but I almost never run QEMU, so I could be wrong here).

Nope, QEMU starts up fine, just lets the guest starve while waiting for jiffies to increase.

Quoted 52 lines · Jan Kiszka

Quoted 22 lines · Marc Zyngier

Quoted 21 lines · Jan Kiszka

Quoted 19 lines · Marc Zyngier

Quoted 7 lines · Jan Kiszka

Quoted 6 lines · Marc Zyngier

Quoted 3 lines · Jan Kiszka

I'm now throwing trace_printk at my broken KVM. Already found out that I get ARM_EXCEPTION_IRQ every few 10 µs. Not seeing any irq_* traces, though. Weird.

Quoted 7 lines

Quoted 4 lines

This very much looks like a screaming interrupt. At such a rate, no wonder your VM make much progress. Can you find out which interrupt is screaming like this? Looking at GICC_HPPIR should help, but you'll have to map the CPU interface in HYP before being able to access it there.

OK... let me figure this out. I had this suspect as well - the host gets a VM exit for each injected guest IRQ?

Not exactly. There is a VM exit for each physical interrupt that fires while the guest is running. Injecting an interrupt also causes a VM exit, as we force the vcpu to reload its context.

Quoted 7 lines

Quoted 3 lines

Ah, GICC != GICV - you are referring to host-side pending IRQs. Any hints on how to get access to that register would accelerate the analysis (ARM KVM code is still new to me).

Map the GICC region in HYP using create_hyp_io_mapping (see vgic_v2_probe for an example of how we map GICH), and stash the read of GICC_HPPIR before leaving HYP mode (and before saving the guest timer).

OK.

Quoted 3 lines

BTW, when you look at /proc/interrupts on the host, don't you see an interrupt that's a bit too eager to fire?

No - but that makes sense given that we do not enter any interrupt handler according to ftrace, thus there can't be any counter incrementation.

Quoted 11 lines

Quoted 8 lines

Quoted 6 lines

Quoted 4 lines

BTW, I also tried with in-kernel GIC disabled (in the kernel config), but I guess that's pointless. Linux seems to be stuck on a non-functional architectural timer then, right?

Yes. Useful for bringup, but nothing more.

Maybe we should perform a feature check and issue a warning from QEMU?

I'd assume this is already in place (but I almost never run QEMU, so I could be wrong here).

Quoted 3 lines

Nope, QEMU starts up fine, just lets the guest starve while waiting for jiffies to increase.

you should be able to turn the in-kernel irqchip off with a QEMU command-line option and the that should prevent the kernel from adding an arch-timer. This would only work on the vexpress guest model though, since the virt-board doesn't provide an emulated timer as a replacement.

Quoted 21 lines · Jan Kiszka

Quoted 19 lines · Marc Zyngier

Quoted 7 lines · Jan Kiszka

Quoted 6 lines · Marc Zyngier

Quoted 3 lines · Jan Kiszka

I'm now throwing trace_printk at my broken KVM. Already found out that I get ARM_EXCEPTION_IRQ every few 10 µs. Not seeing any irq_* traces, though. Weird.

Quoted 7 lines

Quoted 4 lines

This very much looks like a screaming interrupt. At such a rate, no wonder your VM make much progress. Can you find out which interrupt is screaming like this? Looking at GICC_HPPIR should help, but you'll have to map the CPU interface in HYP before being able to access it there.

OK... let me figure this out. I had this suspect as well - the host gets a VM exit for each injected guest IRQ?

Not exactly. There is a VM exit for each physical interrupt that fires while the guest is running. Injecting an interrupt also causes a VM exit, as we force the vcpu to reload its context.

Quoted 3 lines

Ah, GICC != GICV - you are referring to host-side pending IRQs. Any hints on how to get access to that register would accelerate the analysis (ARM KVM code is still new to me).

Map the GICC region in HYP using create_hyp_io_mapping (see vgic_v2_probe for an example of how we map GICH), and stash the read of GICC_HPPIR before leaving HYP mode (and before saving the guest timer).

Quoted 26 lines · Marc Zyngier

Quoted 24 lines · Jan Kiszka

Quoted 22 lines · Marc Zyngier

Quoted 7 lines · Jan Kiszka

Quoted 6 lines · Marc Zyngier

Quoted 3 lines · Jan Kiszka

I'm now throwing trace_printk at my broken KVM. Already found out that I get ARM_EXCEPTION_IRQ every few 10 µs. Not seeing any irq_* traces, though. Weird.

Quoted 8 lines

Quoted 5 lines

This very much looks like a screaming interrupt. At such a rate, no wonder your VM make much progress. Can you find out which interrupt is screaming like this? Looking at GICC_HPPIR should help, but you'll have to map the CPU interface in HYP before being able to access it there.

OK... let me figure this out. I had this suspect as well - the host gets a VM exit for each injected guest IRQ?

Not exactly. There is a VM exit for each physical interrupt that fires while the guest is running. Injecting an interrupt also causes a VM exit, as we force the vcpu to reload its context.

Quoted 8 lines

Quoted 3 lines

Ah, GICC != GICV - you are referring to host-side pending IRQs. Any hints on how to get access to that register would accelerate the analysis (ARM KVM code is still new to me).

Map the GICC region in HYP using create_hyp_io_mapping (see vgic_v2_probe for an example of how we map GICH), and stash the read of GICC_HPPIR before leaving HYP mode (and before saving the guest timer).

Hacked on it until it started to work. The result delivered initially are 0x002 or 0x01e. Then, when the guest gets stuck, I have 0x01b most of the time (a few 0x01e arrive when there is a real host irq). The virtual timer on speed?

Wait, there is also early printk for ARM, but it was off in my guest! Turning it on confirms we have some problems here:

Architected timer frequency not available Division by zero in kernel.

When in emulation mode, I get:

Architected cp15 timer(s) running at 62.50MHz (virt).

Digging deeper.

Quoted 48 lines · Jan Kiszka

Quoted 26 lines · Marc Zyngier

Quoted 24 lines · Jan Kiszka

Quoted 22 lines · Marc Zyngier

Quoted 7 lines · Jan Kiszka

Quoted 6 lines · Marc Zyngier

Quoted 3 lines · Jan Kiszka

I'm now throwing trace_printk at my broken KVM. Already found out that I get ARM_EXCEPTION_IRQ every few 10 µs. Not seeing any irq_* traces, though. Weird.

Quoted 8 lines

Quoted 5 lines

This very much looks like a screaming interrupt. At such a rate, no wonder your VM make much progress. Can you find out which interrupt is screaming like this? Looking at GICC_HPPIR should help, but you'll have to map the CPU interface in HYP before being able to access it there.

OK... let me figure this out. I had this suspect as well - the host gets a VM exit for each injected guest IRQ?

Not exactly. There is a VM exit for each physical interrupt that fires while the guest is running. Injecting an interrupt also causes a VM exit, as we force the vcpu to reload its context.

Quoted 8 lines

Quoted 3 lines

Ah, GICC != GICV - you are referring to host-side pending IRQs. Any hints on how to get access to that register would accelerate the analysis (ARM KVM code is still new to me).

Map the GICC region in HYP using create_hyp_io_mapping (see vgic_v2_probe for an example of how we map GICH), and stash the read of GICC_HPPIR before leaving HYP mode (and before saving the guest timer).

Hacked on it until it started to work. The result delivered initially are 0x002 or 0x01e. Then, when the guest gets stuck, I have 0x01b most of the time (a few 0x01e arrive when there is a real host irq). The virtual timer on speed?

Wait, there is also early printk for ARM, but it was off in my guest! Turning it on confirms we have some problems here:

Architected timer frequency not available Division by zero in kernel.

When in emulation mode, I get:

Quoted 3 lines

Architected cp15 timer(s) running at 62.50MHz (virt).

Digging deeper.

U-Boot didn't initialize CNTFRQ on cores 1..3. Fixing this, the guest passes early boot reliably, now hangs much later (RCU stalls are detected by the guest).

Quoted 52 lines · Jan Kiszka

Quoted 50 lines · Jan Kiszka

Quoted 26 lines · Marc Zyngier

Quoted 24 lines · Jan Kiszka

Quoted 22 lines · Marc Zyngier

Quoted 7 lines · Jan Kiszka

Quoted 6 lines · Marc Zyngier

Quoted 3 lines · Jan Kiszka

I'm now throwing trace_printk at my broken KVM. Already found out that I get ARM_EXCEPTION_IRQ every few 10 µs. Not seeing any irq_* traces, though. Weird.

Quoted 8 lines

Quoted 5 lines

This very much looks like a screaming interrupt. At such a rate, no wonder your VM make much progress. Can you find out which interrupt is screaming like this? Looking at GICC_HPPIR should help, but you'll have to map the CPU interface in HYP before being able to access it there.

OK... let me figure this out. I had this suspect as well - the host gets a VM exit for each injected guest IRQ?

Not exactly. There is a VM exit for each physical interrupt that fires while the guest is running. Injecting an interrupt also causes a VM exit, as we force the vcpu to reload its context.

Quoted 8 lines

Quoted 3 lines

Ah, GICC != GICV - you are referring to host-side pending IRQs. Any hints on how to get access to that register would accelerate the analysis (ARM KVM code is still new to me).

Map the GICC region in HYP using create_hyp_io_mapping (see vgic_v2_probe for an example of how we map GICH), and stash the read of GICC_HPPIR before leaving HYP mode (and before saving the guest timer).

Hacked on it until it started to work. The result delivered initially are 0x002 or 0x01e. Then, when the guest gets stuck, I have 0x01b most of the time (a few 0x01e arrive when there is a real host irq). The virtual timer on speed?

Wait, there is also early printk for ARM, but it was off in my guest! Turning it on confirms we have some problems here:

Architected timer frequency not available Division by zero in kernel.

When in emulation mode, I get:

Architected cp15 timer(s) running at 62.50MHz (virt).

Quoted 5 lines

Quoted 1 line

Digging deeper.

U-Boot didn't initialize CNTFRQ on cores 1..3. Fixing this, the guest passes early boot reliably, now hangs much later (RCU stalls are detected by the guest).

Right, that explains a lot of things. Can you describe a bit more what you're seeing now?