Skip to contentSkip to conversation

Patch "x86: Store a per-cpu shadow copy of CR4" gives VirtualBox 4.3.24 indigestion.

DISCUSSION Valdis Kletnieks 2 participants 3 messages Mar 4, 2015

lore
Share & tools
lore
#0 lore raw

Just a heads up, as it's a out-of-tree module, so this is mostly to get it out for the benefit of others googling for this..

I'm seeing a kernel panic with VirtualBox 4.3.24 with a weird but consistent traceback:

[  359.370165] created IOMMU domain ffff8801ffeecd80
[  395.739820] general protection fault: 0000 [#1] PREEMPT SMP
[  395.739917] task: ffff88022cab4310 ti: ffff88022cab8000 task.ti: ffff88022cab8000
[  395.739921] RIP: 0010:[<ffffffffa803cd02>]  [<ffffffffa803cd02>] do_flush_tlb_all+0x26/0x4f
[  395.739932] RSP: 0018:ffff88022dd03f68  EFLAGS: 00010006
[  395.739935] RAX: 00000000001407e0 RBX: ffff88022dd1ec80 RCX: 0000000000140760
[  395.739939] RDX: 0000000000000002 RSI: 0000000000000000 RDI: 0000000000000000
[  395.739942] RBP: ffff88022dd03f68 R08: 0000000000000001 R09: 0000000000000001
[  395.739946] R10: ffff88022cab4310 R11: ffffffffffffffdb R12: ffff88022dd15580
[  395.739949] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000014500
[  395.739954] FS:  0000000000000000(0000) GS:ffff88022dd00000(0000) knlGS:0000000000000000
[  395.739957] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  395.739961] CR2: 0000000094e7de6f CR3: 0000000028c0b000 CR4: 00000000001427e0
[  395.739964] Stack:
[  395.739967]  ffff88022dd03f98 ffffffffa80b83d6 ffffe8ffffd08ed8 0000000000000003
[  395.739976]  0000005c037d19a5 0000005c037dfc49 ffff88022dd03fa8 ffffffffa80297bb
[  395.739985]  ffff88022cabbe88 ffffffffa871de70 ffff88022cabbdc8 <EOI>  ffff88022cabbe88
[  395.739996] Call Trace:
[  395.740000]  <IRQ>
[  395.740003]  [<ffffffffa80b83d6>] generic_smp_call_function_single_interrupt+0x108/0x1ca
[  395.740018]  [<ffffffffa80297bb>] smp_call_function_interrupt+0x18/0x26
[  395.740025]  [<ffffffffa871de70>] call_function_interrupt+0x70/0x80
[  395.740028]  <EOI>
[  395.740031]  [<ffffffffa8085e8d>] ? trace_hardirqs_off_caller+0x1f/0xac
[  395.740046]  [<ffffffffa84a23fa>] ? cpuidle_enter_state+0x60/0xb3
[  395.740052]  [<ffffffffa80885fb>] ? trace_hardirqs_on+0xd/0xf
[  395.740058]  [<ffffffffa84a23fe>] ? cpuidle_enter_state+0x64/0xb3
[  395.740065]  [<ffffffffa84a23fa>] ? cpuidle_enter_state+0x60/0xb3
[  395.740071]  [<ffffffffa84a2479>] cpuidle_enter+0x17/0x19
[  395.740078]  [<ffffffffa8080853>] cpu_startup_entry+0x286/0x3e7
[  395.740084]  [<ffffffffa80b0119>] ? clockevents_register_device+0x1e6/0x207
[  395.740091]  [<ffffffffa802a168>] start_secondary+0x14e/0x17b
[  395.740097] Code: 00 eb d9 c3 c3 0f 1f 44 00 00 48 8b 05 ec 24 d2 00 55 48 89 e5 f6 c4 20 74 1b 9c 5a fa 65 48 8b 05 54 6d fd 57 48 89 c1 80 e1 7f <0f> 22 e1 0f 22 e0 52 9d eb 05 e8 da fb ff ff 65 8b 05 30 6d fd

Bisection points at:

commit 1e02ce4cccdcb9688386e5b8d2c9fa4660b45389

Author: Andy Lutomirski <[email protected]>
Date:   Fri Oct 24 15:58:08 2014 -0700

x86: Store a per-cpu shadow copy of CR4

Context switches and TLB flushes can change individual bits of CR4. CR4 reads take several cycles, so store a shadow copy of CR4 in a per-cpu variable.

I'm reasonably sure that the problem is that VirtualBox isn't onboard with the whole shadow copy thing, and hilarity and hijinks result when it twiddles CR4 and doesn't update the shadow copy, and they get out of sync.

Am also poking the VirtualBox people.

#1 lore raw
Quoted 52 lines · Valdis Kletnieks

Just a heads up, as it's a out-of-tree module, so this is mostly to get it out for the benefit of others googling for this..

I'm seeing a kernel panic with VirtualBox 4.3.24 with a weird but consistent traceback:

[  359.370165] created IOMMU domain ffff8801ffeecd80
[  395.739820] general protection fault: 0000 [#1] PREEMPT SMP
[  395.739917] task: ffff88022cab4310 ti: ffff88022cab8000 task.ti: ffff88022cab8000
[  395.739921] RIP: 0010:[<ffffffffa803cd02>]  [<ffffffffa803cd02>] do_flush_tlb_all+0x26/0x4f
[  395.739932] RSP: 0018:ffff88022dd03f68  EFLAGS: 00010006
[  395.739935] RAX: 00000000001407e0 RBX: ffff88022dd1ec80 RCX: 0000000000140760
[  395.739939] RDX: 0000000000000002 RSI: 0000000000000000 RDI: 0000000000000000
[  395.739942] RBP: ffff88022dd03f68 R08: 0000000000000001 R09: 0000000000000001
[  395.739946] R10: ffff88022cab4310 R11: ffffffffffffffdb R12: ffff88022dd15580
[  395.739949] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000014500
[  395.739954] FS:  0000000000000000(0000) GS:ffff88022dd00000(0000) knlGS:0000000000000000
[  395.739957] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  395.739961] CR2: 0000000094e7de6f CR3: 0000000028c0b000 CR4: 00000000001427e0
[  395.739964] Stack:
[  395.739967]  ffff88022dd03f98 ffffffffa80b83d6 ffffe8ffffd08ed8 0000000000000003
[  395.739976]  0000005c037d19a5 0000005c037dfc49 ffff88022dd03fa8 ffffffffa80297bb
[  395.739985]  ffff88022cabbe88 ffffffffa871de70 ffff88022cabbdc8 <EOI>  ffff88022cabbe88
[  395.739996] Call Trace:
[  395.740000]  <IRQ>
[  395.740003]  [<ffffffffa80b83d6>] generic_smp_call_function_single_interrupt+0x108/0x1ca
[  395.740018]  [<ffffffffa80297bb>] smp_call_function_interrupt+0x18/0x26
[  395.740025]  [<ffffffffa871de70>] call_function_interrupt+0x70/0x80
[  395.740028]  <EOI>
[  395.740031]  [<ffffffffa8085e8d>] ? trace_hardirqs_off_caller+0x1f/0xac
[  395.740046]  [<ffffffffa84a23fa>] ? cpuidle_enter_state+0x60/0xb3
[  395.740052]  [<ffffffffa80885fb>] ? trace_hardirqs_on+0xd/0xf
[  395.740058]  [<ffffffffa84a23fe>] ? cpuidle_enter_state+0x64/0xb3
[  395.740065]  [<ffffffffa84a23fa>] ? cpuidle_enter_state+0x60/0xb3
[  395.740071]  [<ffffffffa84a2479>] cpuidle_enter+0x17/0x19
[  395.740078]  [<ffffffffa8080853>] cpu_startup_entry+0x286/0x3e7
[  395.740084]  [<ffffffffa80b0119>] ? clockevents_register_device+0x1e6/0x207
[  395.740091]  [<ffffffffa802a168>] start_secondary+0x14e/0x17b
[  395.740097] Code: 00 eb d9 c3 c3 0f 1f 44 00 00 48 8b 05 ec 24 d2 00 55 48 89 e5 f6 c4 20 74 1b 9c 5a fa 65 48 8b 05 54 6d fd 57 48 89 c1 80 e1 7f <0f> 22 e1 0f 22 e0 52 9d eb 05 e8 da fb ff ff 65 8b 05 30 6d fd

Bisection points at:

commit 1e02ce4cccdcb9688386e5b8d2c9fa4660b45389

Author: Andy Lutomirski <[email protected]>
Date:   Fri Oct 24 15:58:08 2014 -0700

x86: Store a per-cpu shadow copy of CR4

Context switches and TLB flushes can change individual bits of CR4. CR4 reads take several cycles, so store a shadow copy of CR4 in a per-cpu variable.

Quoted 6 lines

I'm reasonably sure that the problem is that VirtualBox isn't onboard with the whole shadow copy thing, and hilarity and hijinks result when it twiddles CR4 and doesn't update the shadow copy, and they get out of sync.

Am also poking the VirtualBox people.

Are they poking cr4 directly with mov? Let's see:

    /* Enable the VMX bit in CR4 if necessary. */
    RTCCUINTREG uCr4 = ASMGetCR4();
    if (!(uCr4 & X86_CR4_VMXE))
        ASMSetCR4(uCr4 | X86_CR4_VMXE);

ASMSetCR4 is about what you'd expect.

I deliberately renamed write_cr4 to __write_cr4 to catch this kind of thing in anything that uses the asm/ headers, but I can't do anything about asm code in out-of-tree modules.

--Andy

Signature · 2 lines
Andy Lutomirski
AMA Capital Management, LLC
#2 lore raw
Quoted 8 lines · Andy Lutomirski

Are they poking cr4 directly with mov? Let's see:

    /* Enable the VMX bit in CR4 if necessary. */
    RTCCUINTREG uCr4 = ASMGetCR4();
    if (!(uCr4 & X86_CR4_VMXE))
        ASMSetCR4(uCr4 | X86_CR4_VMXE);
Quoted 5 lines

ASMSetCR4 is about what you'd expect.

I deliberately renamed write_cr4 to __write_cr4 to catch this kind of thing in anything that uses the asm/ headers, but I can't do anything about asm code in out-of-tree modules.

Thanks for the confirmation, I've already updated the tracking bug they were using for 4.0 issues, and will chase it further with them...