KVM crash on Jetson TK1

Hi,

using Ian Campbell's PSCI patches for U-Boot (ported to master, see [1]), I managed to enable KVM on the Jetson TK1 board. Unfortunately, I'm getting a crash on the host when starting QEMU:

[   59.164674] kernel BUG at ../arch/arm/include/asm/kvm_mmu.h:189!
[   59.173139] Internal error: Oops - BUG: 0 [#1] PREEMPT SMP ARM
[   59.181442] Modules linked in:
[   59.186960] CPU: 0 PID: 820 Comm: qemu-system-arm Not tainted 3.19.0-rc7-00050-gdf75905-dirty #6
[   59.198353] Hardware name: NVIDIA Tegra SoC (Flattened Device Tree)
[   59.207249] task: ed33b340 ti: ed15e000 task.ti: ed15e000
[   59.215288] PC is at kvm_handle_guest_abort+0x2c0/0x320
[   59.223161] LR is at kvm_handle_guest_abort+0x2c0/0x320
[   59.230996] pc : [<c00175c4>]    lr : [<c00175c4>]    psr: 600f0013
[   59.230996] sp : ed15fe70  ip : 00000000  fp : 00000000
[   59.247782] r10: 00000000  r9 : 00000000  r8 : 000aafa6
[   59.255678] r7 : ed117000  r6 : 0000016f  r5 : 00080000  r4 : ed2e5630
[   59.264912] r3 : 00000010  r2 : 0008027c  r1 : eed564c0  r0 : 00000000
[   59.274146] Flags: nZCv  IRQs on  FIQs on  Mode SVC_32  ISA ARM  Segment user
[   59.284044] Control: 30c5387d  Table: ad9d0ac0  DAC: fffffffd
[   59.292558] Process qemu-system-arm (pid: 820, stack limit = 0xed15e238)
[   59.302114] Stack: (0xed15fe70 to 0xed160000)
[   59.309340] fe60:                                     00000000 ed15fe87 00000000 73800000
[   59.320462] fe80: 0000000f 01010000 00000504 ed2e5630 c0917450 00000007 00010000 00000001
[   59.331630] fea0: edac4000 c0960164 c0917450 c0014710 00000000 00010000 ed91a580 7ffbfeff
[   59.342833] fec0: fffffffe fffffffb 00000001 00000000 0000ae80 ed2e5630 ed114040 00000000
[   59.354049] fee0: 00000000 ed114040 ed2e5630 00000000 ed15e000 00000000 737fed4c c001063c
[   59.365272] ff00: 737fea4c c0042688 00000240 00000000 00000000 00000000 0000000a 00000000
[   59.376493] ff20: edc0de78 00000000 ed114040 0000000b 00000000 c0109678 00000001 b749ba00
[   59.387737] ff40: 000000f0 00000000 00000001 00000081 b7032dc0 c009627c 00000002 00000000
[   59.399006] ff60: ffffffff 00000000 00000000 737feb4c ed114040 ed114041 0000000b ed114040
[   59.410271] ff80: 0000ae80 00000000 ed15e000 c01098a0 b6a62eb0 00004254 b749ba00 00000036
[   59.421558] ffa0: c001e2e4 c001e160 b6a62eb0 00004254 0000000b 0000ae80 00000000 cc628500
[   59.432884] ffc0: b6a62eb0 00004254 b749ba00 00000036 b76baba8 bebc9da8 00000000 737fed4c
[   59.444244] ffe0: b6fc33f8 737feb74 b6b3afcc b57f3abc 600f0010 0000000b 00000000 00000000
[   59.455661] [<c00175c4>] (kvm_handle_guest_abort) from [<c0014710>] (kvm_arch_vcpu_ioctl_run+0x160/0x400)
[   59.468552] [<c0014710>] (kvm_arch_vcpu_ioctl_run) from [<c001063c>] (kvm_vcpu_ioctl+0x2e4/0x6ec)
[   59.480803] [<c001063c>] (kvm_vcpu_ioctl) from [<c0109678>] (do_vfs_ioctl+0x40c/0x600)
[   59.492142] [<c0109678>] (do_vfs_ioctl) from [<c01098a0>] (SyS_ioctl+0x34/0x5c)
[   59.502909] [<c01098a0>] (SyS_ioctl) from [<c001e160>] (ret_fast_syscall+0x0/0x34)
[   59.513944] Code: 0a000002 e1a00008 e1a01009 ebffe2c7 (e7f001f2) 

Host kernel is from Linus tree. I can try a different one if there are pending fixes somewhere.

Any ideas? Anything I should look at in particular?

Jan

[1] https://github.com/siemens/u-boot/tree/jetson-tk1

On Sat, 7 Feb 2015 20:09:14 +0000
Jan Kiszka <[email protected]> wrote:

Hi Jan,

That crash doesn't make much sense. If you look at the BUG_ON you're hitting, we check that the size to flush is a multiple of PAGE_SIZE.

The only two call sites do pass either PMD_SIZE or PAGE_SIZE, so it is hard to imagine how this can trigger.

What compiler are you using? Can you force the compiler not to inline this function and see if that makes a difference? I'd be very interested in seeing the assembly code generated for this particular function.

Thanks,

	M.

Jazz is not dead. It just smells funny.

Linaro 4.9-2014.09

Here we go:

[   41.973321] kernel BUG at ../arch/arm/include/asm/kvm_mmu.h:189!
[   41.981747] Internal error: Oops - BUG: 0 [#1] PREEMPT SMP ARM
[   41.990011] Modules linked in:
[   41.995498] CPU: 2 PID: 807 Comm: qemu-system-arm Not tainted 3.19.0-rc7-00221-gfd7a168-dirty #8
[   42.006850] Hardware name: NVIDIA Tegra SoC (Flattened Device Tree)
[   42.015707] task: ec354d80 ti: ed31a000 task.ti: ed31a000
[   42.023728] PC is at __coherent_cache_guest_page.constprop.37+0x0/0x4
[   42.032808] LR is at kvm_handle_guest_abort+0x2e0/0x340
[   42.040678] pc : [<c0016bc0>]    lr : [<c00175e8>]    psr: 600f0013
[   42.040678] sp : ed31be68  ip : 00000000  fp : 73800000
[   42.057517] r10: 00000000  r9 : 00000000  r8 : 000f7ede
[   42.065465] r7 : 00000170  r6 : eda3e800  r5 : 00000000  r4 : edb4b630
[   42.074730] r3 : 00000000  r2 : 000f7ede  r1 : ef6f4bc0  r0 : edb4b630
[   42.083989] Flags: nZCv  IRQs on  FIQs on  Mode SVC_32  ISA ARM  Segment user
[   42.093932] Control: 30c5387d  Table: ad1ae700  DAC: 55555555
[   42.102470] Process qemu-system-arm (pid: 807, stack limit = 0xed31a238)
[   42.112069] Stack: (0xed31be68 to 0xed31c000)
[   42.119326] be60:                   00000000 ed31be87 c0013fa0 00000000 ec390000 ed00b008
[   42.130477] be80: 0000000f 01010000 00000504 edb4b630 c0915450 00000007 00010000 00000001
[   42.141659] bea0: ec390000 c095e164 c0915450 c0014710 00000000 00010000 edb8ec80 7ffbfeff
[   42.152875] bec0: fffffffe fffffffb 00000001 00000000 0000ae80 edb4b630 edaf6b80 00000000
[   42.164115] bee0: 00000000 edaf6b80 edb4b630 00000000 ed31a000 00000000 737fed4c c001063c
[   42.175356] bf00: 737fea4c c0042688 00000240 00000000 00000000 00000000 0000000a 00000000
[   42.186607] bf20: edc0de78 00000000 edaf6b80 0000000b 00000000 c0109724 00000001 b74e1a80
[   42.197866] bf40: 000000f0 00000000 00000001 00000081 b7078dc0 c0096310 00000002 00000000
[   42.209184] bf60: ffffffff 00000000 00000000 737feb4c edaf6b80 edaf6b81 0000000b edaf6b80
[   42.220531] bf80: 0000ae80 00000000 ed31a000 c010994c b6aa8eb0 00004254 b74e1a80 00000036
[   42.231880] bfa0: c001e324 c001e1a0 b6aa8eb0 00004254 0000000b 0000ae80 00000000 622b3f00
[   42.243254] bfc0: b6aa8eb0 00004254 b74e1a80 00000036 b7700c48 bedaad98 00000000 737fed4c
[   42.254662] bfe0: b70093f8 737feb74 b6b80fcc b5839abc 600f0010 0000000b 00000000 00000000
[   42.266123] [<c0016bc0>] (__coherent_cache_guest_page.constprop.37) from [<c00175e8>] (kvm_handle_guest_abort+0x2e0/0x340)
[   42.280587] [<c00175e8>] (kvm_handle_guest_abort) from [<c0014710>] (kvm_arch_vcpu_ioctl_run+0x160/0x400)
[   42.293631] [<c0014710>] (kvm_arch_vcpu_ioctl_run) from [<c001063c>] (kvm_vcpu_ioctl+0x2e4/0x6ec)
[   42.306056] [<c001063c>] (kvm_vcpu_ioctl) from [<c0109724>] (do_vfs_ioctl+0x40c/0x600)
[   42.317561] [<c0109724>] (do_vfs_ioctl) from [<c010994c>] (SyS_ioctl+0x34/0x5c)
[   42.328482] [<c010994c>] (SyS_ioctl) from [<c001e1a0>] (ret_fast_syscall+0x0/0x34)
[   42.339686] Code: e3e0000b e8bd8010 e3a00000 e8bd8010 (e7f001f2) 

And that functions looks, well, simple:

000084c0 <__coherent_cache_guest_page.constprop.37>:
static inline bool vcpu_has_cache_enabled(struct kvm_vcpu *vcpu)

{
	return (vcpu->arch.cp15[c1_SCTLR] & 0b101) == 0b101;
}

static noinline void __coherent_cache_guest_page(struct kvm_vcpu *vcpu, pfn_t pfn,

    84c0:	e7f001f2 	.word	0xe7f001f2

000084c4 <free_boot_hyp_pgd>:
...

Seems the size parameter is constant and fulfills the BUG_ON condition.

Jan

From: Jan Kiszka <[email protected]>

The check is supposed to catch page-unaligned sizes, not the inverse.

---

That resolves the crash, but the guest still doesn't boot (it does when removing -enable-kvm). Examining later.

 arch/arm/include/asm/kvm_mmu.h | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

index 1bca8f8..90052de 100644
@@ -186,7 +186,7 @@ static inline void __coherent_cache_guest_page(struct kvm_vcpu *vcpu, pfn_t pfn,
 
 	bool need_flush = !vcpu_has_cache_enabled(vcpu) || ipa_uncached;
 
-	VM_BUG_ON(size & PAGE_MASK);
+	VM_BUG_ON(size & ~PAGE_MASK);
 
 	if (!need_flush && !icache_is_pipt())
 		goto vipt_cache;

2.1.4

Thanks, applied.

-Christoffer

Thanks, can you send a pull request as soon as possible with this and "arm/arm64: KVM: Add exit reaons to kvm_exit event tracing"?

Paolo

Yep, done.

Thanks,
-Christoffer