BUG: unable to handle kernel NULL pointer dereference in handle_external_interrupt_irqoff
Subsystems (5)
syzbot
syzbotHello,
syzbot found the following crash on:
HEAD commit: b74b991f Merge tag 'block-5.6-20200320' of git://git.kerne.. git tree: upstream console output: https://syzkaller.appspot.com/x/log.txt?x=16403223e00000 kernel config: https://syzkaller.appspot.com/x/.config?x=6dfa02302d6db985 dashboard link: https://syzkaller.appspot.com/bug?extid=3f29ca2efb056a761e38 compiler: clang version 10.0.0 (https://github.com/llvm/llvm-project/ c2443155a0fb245c8f17f2c1c72b6ea391e86e81)
Unfortunately, I don't have any reproducer for this crash yet.
IMPORTANT: if you fix the bug, please add the following tag to the commit:
BUG: kernel NULL pointer dereference, address: 0000000000000086 #PF: supervisor instruction fetch in kernel mode #PF: error_code(0x0010) - not-present page PGD a63a4067 P4D a63a4067 PUD a7627067 PMD 0 Oops: 0010 [#1] PREEMPT SMP KASAN CPU: 0 PID: 9785 Comm: syz-executor.2 Not tainted 5.6.0-rc6-syzkaller #0 Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011 RIP: 0010:0x86 Code: Bad RIP value. RSP: 0018:ffffc90001ac7998 EFLAGS: 00010086 RAX: ffffc90001ac79c8 RBX: fffffe0000000000 RCX: 0000000000040000 RDX: ffffc9000e20f000 RSI: 000000000000b452 RDI: 000000000000b453 RBP: 0000000000000ec0 R08: ffffffff83987523 R09: ffffffff811c7eca R10: ffff8880a4e94200 R11: 0000000000000002 R12: dffffc0000000000 R13: fffffe0000000ec8 R14: ffffffff880016f0 R15: fffffe0000000ecb FS: 00007fb50e370700(0000) GS:ffff8880ae800000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 000000000000005c CR3: 0000000092fc7000 CR4: 00000000001426f0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 Call Trace: handle_external_interrupt_irqoff+0x154/0x280 arch/x86/kvm/vmx/vmx.c:6274 kvm_before_interrupt arch/x86/kvm/x86.h:343 [inline] handle_external_interrupt_irqoff+0x132/0x280 arch/x86/kvm/vmx/vmx.c:6272 __irqentry_text_start+0x8/0x8 vcpu_enter_guest+0x6c77/0x9290 arch/x86/kvm/x86.c:8405 save_stack mm/kasan/common.c:72 [inline] set_track mm/kasan/common.c:80 [inline] kasan_set_free_info mm/kasan/common.c:337 [inline] __kasan_slab_free+0x12e/0x1e0 mm/kasan/common.c:476 __cache_free mm/slab.c:3426 [inline] kfree+0x10a/0x220 mm/slab.c:3757 tomoyo_path_number_perm+0x525/0x690 security/tomoyo/file.c:736 security_file_ioctl+0x55/0xb0 security/security.c:1441 entry_SYSCALL_64_after_hwframe+0x49/0xbe __lock_acquire+0xc5a/0x1bc0 kernel/locking/lockdep.c:3954 test_bit include/asm-generic/bitops/instrumented-non-atomic.h:110 [inline] hlock_class kernel/locking/lockdep.c:163 [inline] mark_lock+0x107/0x1650 kernel/locking/lockdep.c:3642 lock_acquire+0x154/0x250 kernel/locking/lockdep.c:4484 rcu_lock_acquire+0x9/0x30 include/linux/rcupdate.h:208 kvm_check_async_pf_completion+0x34e/0x360 arch/x86/kvm/../../../virt/kvm/async_pf.c:137 vcpu_run+0x3a3/0xd50 arch/x86/kvm/x86.c:8513 kvm_arch_vcpu_ioctl_run+0x419/0x880 arch/x86/kvm/x86.c:8735 kvm_vcpu_ioctl+0x67c/0xa80 arch/x86/kvm/../../../virt/kvm/kvm_main.c:2932 kvm_vm_release+0x50/0x50 arch/x86/kvm/../../../virt/kvm/kvm_main.c:858 vfs_ioctl fs/ioctl.c:47 [inline] ksys_ioctl fs/ioctl.c:763 [inline] __do_sys_ioctl fs/ioctl.c:772 [inline] __se_sys_ioctl+0xf9/0x160 fs/ioctl.c:770 do_syscall_64+0xf3/0x1b0 arch/x86/entry/common.c:294 entry_SYSCALL_64_after_hwframe+0x49/0xbe Modules linked in: CR2: 0000000000000086 ---[ end trace 4da75c292cd7e3e8 ]--- RIP: 0010:0x86 Code: Bad RIP value. RSP: 0018:ffffc90001ac7998 EFLAGS: 00010086 RAX: ffffc90001ac79c8 RBX: fffffe0000000000 RCX: 0000000000040000 RDX: ffffc9000e20f000 RSI: 000000000000b452 RDI: 000000000000b453 RBP: 0000000000000ec0 R08: ffffffff83987523 R09: ffffffff811c7eca R10: ffff8880a4e94200 R11: 0000000000000002 R12: dffffc0000000000 R13: fffffe0000000ec8 R14: ffffffff880016f0 R15: fffffe0000000ecb FS: 00007fb50e370700(0000) GS:ffff8880ae800000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 000000000000005c CR3: 0000000092fc7000 CR4: 00000000001426f0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
---
This bug is generated by a bot. It may contain errors. See https://goo.gl/tpsmEJ for more information about syzbot. syzbot engineers can be reached at [email protected].
syzbot will keep track of this bug report. See:
https://goo.gl/tpsmEJ#status for how to communicate with syzbot.
Quoted 12 lines · syzbot
Hello,
syzbot found the following crash on:
HEAD commit: b74b991f Merge tag 'block-5.6-20200320' of git://git.kerne.. git tree: upstream console output: https://syzkaller.appspot.com/x/log.txt?x=16403223e00000 kernel config: https://syzkaller.appspot.com/x/.config?x=6dfa02302d6db985 dashboard link: https://syzkaller.appspot.com/bug?extid=3f29ca2efb056a761e38 compiler: clang version 10.0.0 (https://github.com/llvm/llvm-project/ c2443155a0fb245c8f17f2c1c72b6ea391e86e81)
Quoted 4 lines
Unfortunately, I don't have any reproducer for this crash yet.
IMPORTANT: if you fix the bug, please add the following tag to the commit:
Reported-by [email protected]
+clang-built-linux
This only happens on the instance that uses clang. So potentially this is related to clang. The instance also uses smack lsm, but it's less likely to be involved I think.
This actually started happening around Mar 6, but the ORC unwinder somehow fails to unwind stack and prints only questionable frames, so the reports were classified as "corrupted" and all thrown in the "corrupted reports" bucket:
https://syzkaller.appspot.com/bug?id=d5bc3e0c66d200d72216ab343a67c4327e4a3452
There is already some discussion about this on the clang-built-linux list:
https://groups.google.com/d/msg/clang-built-linux/Cm3VojRK69I/cfDGxIlTAwAJ
The handle_external_interrupt_irqoff has some inline asm and the special STACK_FRAME_NON_STANDARD. So it has some potential for bad interaction with compilers...
The commit range is presumably fb279f4e238617417b132a550f24c1e86d922558..63849c8f410717eb2e6662f3953ff674727303e7 But I don't see anything that says "it's me". The only commit that does non-trivial changes to x86/vmx seems to be "KVM: VMX: check descriptor table exits on instruction emulation":
$ git log --oneline fb279f4e238617417b132a550f24c1e86d922558..63849c8f410717eb2e6662f3953ff674727303e7 virt/kvm/ arch/x86/kvm/ 86f7e90ce840a KVM: VMX: check descriptor table exits on instruction emulation e951445f4d3b5 Merge tag 'kvmarm-fixes-5.6-1' of git://git.kernel.org/pub/scm/linux/kernel/git/kvmarm/kvmarm into HEAD ef935c25fd648 kvm: x86: Limit the number of "kvm: disabled by bios" messages aaec7c03de92c KVM: x86: avoid useless copy of cpufreq policy 4f337faf1c55e KVM: allow disabling -Werror 575b255c1663c KVM: x86: allow compiling as non-module with W=1 7943f4acea3ca KVM: SVM: allocate AVIC data structures based on kvm_amd module parameter b3f15ec3d809c kvm: arm/arm64: Fold VHE entry/exit work into kvm_vcpu_run_vhe() 51b2569402a38 KVM: arm/arm64: Fix up includes for trace.h
Quoted 82 lines
BUG: kernel NULL pointer dereference, address: 0000000000000086 #PF: supervisor instruction fetch in kernel mode #PF: error_code(0x0010) - not-present page PGD a63a4067 P4D a63a4067 PUD a7627067 PMD 0 Oops: 0010 [#1] PREEMPT SMP KASAN CPU: 0 PID: 9785 Comm: syz-executor.2 Not tainted 5.6.0-rc6-syzkaller #0 Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011 RIP: 0010:0x86 Code: Bad RIP value. RSP: 0018:ffffc90001ac7998 EFLAGS: 00010086 RAX: ffffc90001ac79c8 RBX: fffffe0000000000 RCX: 0000000000040000 RDX: ffffc9000e20f000 RSI: 000000000000b452 RDI: 000000000000b453 RBP: 0000000000000ec0 R08: ffffffff83987523 R09: ffffffff811c7eca R10: ffff8880a4e94200 R11: 0000000000000002 R12: dffffc0000000000 R13: fffffe0000000ec8 R14: ffffffff880016f0 R15: fffffe0000000ecb FS: 00007fb50e370700(0000) GS:ffff8880ae800000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 000000000000005c CR3: 0000000092fc7000 CR4: 00000000001426f0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 Call Trace: handle_external_interrupt_irqoff+0x154/0x280 arch/x86/kvm/vmx/vmx.c:6274 kvm_before_interrupt arch/x86/kvm/x86.h:343 [inline] handle_external_interrupt_irqoff+0x132/0x280 arch/x86/kvm/vmx/vmx.c:6272 __irqentry_text_start+0x8/0x8 vcpu_enter_guest+0x6c77/0x9290 arch/x86/kvm/x86.c:8405 save_stack mm/kasan/common.c:72 [inline] set_track mm/kasan/common.c:80 [inline] kasan_set_free_info mm/kasan/common.c:337 [inline] __kasan_slab_free+0x12e/0x1e0 mm/kasan/common.c:476 __cache_free mm/slab.c:3426 [inline] kfree+0x10a/0x220 mm/slab.c:3757 tomoyo_path_number_perm+0x525/0x690 security/tomoyo/file.c:736 security_file_ioctl+0x55/0xb0 security/security.c:1441 entry_SYSCALL_64_after_hwframe+0x49/0xbe __lock_acquire+0xc5a/0x1bc0 kernel/locking/lockdep.c:3954 test_bit include/asm-generic/bitops/instrumented-non-atomic.h:110 [inline] hlock_class kernel/locking/lockdep.c:163 [inline] mark_lock+0x107/0x1650 kernel/locking/lockdep.c:3642 lock_acquire+0x154/0x250 kernel/locking/lockdep.c:4484 rcu_lock_acquire+0x9/0x30 include/linux/rcupdate.h:208 kvm_check_async_pf_completion+0x34e/0x360 arch/x86/kvm/../../../virt/kvm/async_pf.c:137 vcpu_run+0x3a3/0xd50 arch/x86/kvm/x86.c:8513 kvm_arch_vcpu_ioctl_run+0x419/0x880 arch/x86/kvm/x86.c:8735 kvm_vcpu_ioctl+0x67c/0xa80 arch/x86/kvm/../../../virt/kvm/kvm_main.c:2932 kvm_vm_release+0x50/0x50 arch/x86/kvm/../../../virt/kvm/kvm_main.c:858 vfs_ioctl fs/ioctl.c:47 [inline] ksys_ioctl fs/ioctl.c:763 [inline] __do_sys_ioctl fs/ioctl.c:772 [inline] __se_sys_ioctl+0xf9/0x160 fs/ioctl.c:770 do_syscall_64+0xf3/0x1b0 arch/x86/entry/common.c:294 entry_SYSCALL_64_after_hwframe+0x49/0xbe Modules linked in: CR2: 0000000000000086 ---[ end trace 4da75c292cd7e3e8 ]--- RIP: 0010:0x86 Code: Bad RIP value. RSP: 0018:ffffc90001ac7998 EFLAGS: 00010086 RAX: ffffc90001ac79c8 RBX: fffffe0000000000 RCX: 0000000000040000 RDX: ffffc9000e20f000 RSI: 000000000000b452 RDI: 000000000000b453 RBP: 0000000000000ec0 R08: ffffffff83987523 R09: ffffffff811c7eca R10: ffff8880a4e94200 R11: 0000000000000002 R12: dffffc0000000000 R13: fffffe0000000ec8 R14: ffffffff880016f0 R15: fffffe0000000ecb FS: 00007fb50e370700(0000) GS:ffff8880ae800000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 000000000000005c CR3: 0000000092fc7000 CR4: 00000000001426f0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400---
This bug is generated by a bot. It may contain errors. See https://goo.gl/tpsmEJ for more information about syzbot. syzbot engineers can be reached at [email protected].syzbot will keep track of this bug report. See:
https://goo.gl/tpsmEJ#status for how to communicate with syzbot.--
You received this message because you are subscribed to the Google Groups "syzkaller-bugs" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/syzkaller-bugs/000000000000277a0405a16bd5c9%40google.com.
Quoted 44 lines · Dmitry Vyukov
Quoted 12 lines · syzbot
Hello,
syzbot found the following crash on:
HEAD commit: b74b991f Merge tag 'block-5.6-20200320' of git://git.kerne.. git tree: upstream console output: https://syzkaller.appspot.com/x/log.txt?x=16403223e00000 kernel config: https://syzkaller.appspot.com/x/.config?x=6dfa02302d6db985 dashboard link: https://syzkaller.appspot.com/bug?extid=3f29ca2efb056a761e38 compiler: clang version 10.0.0 (https://github.com/llvm/llvm-project/ c2443155a0fb245c8f17f2c1c72b6ea391e86e81)Quoted 4 lines
Unfortunately, I don't have any reproducer for this crash yet.
IMPORTANT: if you fix the bug, please add the following tag to the commit:
Reported-by [email protected]+clang-built-linux
This only happens on the instance that uses clang. So potentially this is related to clang. The instance also uses smack lsm, but it's less likely to be involved I think.
This actually started happening around Mar 6, but the ORC unwinder somehow fails to unwind stack and prints only questionable frames, so the reports were classified as "corrupted" and all thrown in the "corrupted reports" bucket:
https://syzkaller.appspot.com/bug?id=d5bc3e0c66d200d72216ab343a67c4327e4a3452There is already some discussion about this on the clang-built-linux list:
https://groups.google.com/d/msg/clang-built-linux/Cm3VojRK69I/cfDGxIlTAwAJThe
handle_external_interrupt_irqoffhas some inline asm and the specialSTACK_FRAME_NON_STANDARD. So it has some potential for bad interaction with compilers...The commit range is presumably
fb279f4e238617417b132a550f24c1e86d922558..63849c8f410717eb2e6662f3953ff674727303e7But I don't see anything that says "it's me". The only commit that does non-trivial changes to x86/vmx seems to be "KVM: VMX: check descriptor table exits on instruction emulation":
Quoted 14 lines
$ git log --oneline fb279f4e238617417b132a550f24c1e86d922558..63849c8f410717eb2e6662f3953ff674727303e7 virt/kvm/ arch/x86/kvm/ 86f7e90ce840a KVM: VMX: check descriptor table exits on instruction emulation e951445f4d3b5 Merge tag 'kvmarm-fixes-5.6-1' of git://git.kernel.org/pub/scm/linux/kernel/git/kvmarm/kvmarm into HEAD ef935c25fd648 kvm: x86: Limit the number of "kvm: disabled by bios" messages aaec7c03de92c KVM: x86: avoid useless copy of cpufreq policy 4f337faf1c55e KVM: allow disabling -Werror 575b255c1663c KVM: x86: allow compiling as non-module with W=1 7943f4acea3ca KVM: SVM: allocate AVIC data structures based on kvm_amd module parameter b3f15ec3d809c kvm: arm/arm64: Fold VHE entry/exit work into kvm_vcpu_run_vhe() 51b2569402a38 KVM: arm/arm64: Fix up includes for trace.h
And the problem with this crash is that it happens all the time, basically the only crash that now happens on the instance. So effectively all kernel testing of all subsystems has stalled due to this.
Quoted 82 lines
Quoted 82 lines
BUG: kernel NULL pointer dereference, address: 0000000000000086 #PF: supervisor instruction fetch in kernel mode #PF: error_code(0x0010) - not-present page PGD a63a4067 P4D a63a4067 PUD a7627067 PMD 0 Oops: 0010 [#1] PREEMPT SMP KASAN CPU: 0 PID: 9785 Comm: syz-executor.2 Not tainted 5.6.0-rc6-syzkaller #0 Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011 RIP: 0010:0x86 Code: Bad RIP value. RSP: 0018:ffffc90001ac7998 EFLAGS: 00010086 RAX: ffffc90001ac79c8 RBX: fffffe0000000000 RCX: 0000000000040000 RDX: ffffc9000e20f000 RSI: 000000000000b452 RDI: 000000000000b453 RBP: 0000000000000ec0 R08: ffffffff83987523 R09: ffffffff811c7eca R10: ffff8880a4e94200 R11: 0000000000000002 R12: dffffc0000000000 R13: fffffe0000000ec8 R14: ffffffff880016f0 R15: fffffe0000000ecb FS: 00007fb50e370700(0000) GS:ffff8880ae800000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 000000000000005c CR3: 0000000092fc7000 CR4: 00000000001426f0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 Call Trace: handle_external_interrupt_irqoff+0x154/0x280 arch/x86/kvm/vmx/vmx.c:6274 kvm_before_interrupt arch/x86/kvm/x86.h:343 [inline] handle_external_interrupt_irqoff+0x132/0x280 arch/x86/kvm/vmx/vmx.c:6272 __irqentry_text_start+0x8/0x8 vcpu_enter_guest+0x6c77/0x9290 arch/x86/kvm/x86.c:8405 save_stack mm/kasan/common.c:72 [inline] set_track mm/kasan/common.c:80 [inline] kasan_set_free_info mm/kasan/common.c:337 [inline] __kasan_slab_free+0x12e/0x1e0 mm/kasan/common.c:476 __cache_free mm/slab.c:3426 [inline] kfree+0x10a/0x220 mm/slab.c:3757 tomoyo_path_number_perm+0x525/0x690 security/tomoyo/file.c:736 security_file_ioctl+0x55/0xb0 security/security.c:1441 entry_SYSCALL_64_after_hwframe+0x49/0xbe __lock_acquire+0xc5a/0x1bc0 kernel/locking/lockdep.c:3954 test_bit include/asm-generic/bitops/instrumented-non-atomic.h:110 [inline] hlock_class kernel/locking/lockdep.c:163 [inline] mark_lock+0x107/0x1650 kernel/locking/lockdep.c:3642 lock_acquire+0x154/0x250 kernel/locking/lockdep.c:4484 rcu_lock_acquire+0x9/0x30 include/linux/rcupdate.h:208 kvm_check_async_pf_completion+0x34e/0x360 arch/x86/kvm/../../../virt/kvm/async_pf.c:137 vcpu_run+0x3a3/0xd50 arch/x86/kvm/x86.c:8513 kvm_arch_vcpu_ioctl_run+0x419/0x880 arch/x86/kvm/x86.c:8735 kvm_vcpu_ioctl+0x67c/0xa80 arch/x86/kvm/../../../virt/kvm/kvm_main.c:2932 kvm_vm_release+0x50/0x50 arch/x86/kvm/../../../virt/kvm/kvm_main.c:858 vfs_ioctl fs/ioctl.c:47 [inline] ksys_ioctl fs/ioctl.c:763 [inline] __do_sys_ioctl fs/ioctl.c:772 [inline] __se_sys_ioctl+0xf9/0x160 fs/ioctl.c:770 do_syscall_64+0xf3/0x1b0 arch/x86/entry/common.c:294 entry_SYSCALL_64_after_hwframe+0x49/0xbe Modules linked in: CR2: 0000000000000086 ---[ end trace 4da75c292cd7e3e8 ]--- RIP: 0010:0x86 Code: Bad RIP value. RSP: 0018:ffffc90001ac7998 EFLAGS: 00010086 RAX: ffffc90001ac79c8 RBX: fffffe0000000000 RCX: 0000000000040000 RDX: ffffc9000e20f000 RSI: 000000000000b452 RDI: 000000000000b453 RBP: 0000000000000ec0 R08: ffffffff83987523 R09: ffffffff811c7eca R10: ffff8880a4e94200 R11: 0000000000000002 R12: dffffc0000000000 R13: fffffe0000000ec8 R14: ffffffff880016f0 R15: fffffe0000000ecb FS: 00007fb50e370700(0000) GS:ffff8880ae800000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 000000000000005c CR3: 0000000092fc7000 CR4: 00000000001426f0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400---
This bug is generated by a bot. It may contain errors. See https://goo.gl/tpsmEJ for more information about syzbot. syzbot engineers can be reached at [email protected].syzbot will keep track of this bug report. See:
https://goo.gl/tpsmEJ#status for how to communicate with syzbot.--
You received this message because you are subscribed to the Google Groups "syzkaller-bugs" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/syzkaller-bugs/000000000000277a0405a16bd5c9%40google.com.
Paolo Bonzini
maintainerQuoted 6 lines · Dmitry Vyukov
The commit range is presumably
fb279f4e238617417b132a550f24c1e86d922558..63849c8f410717eb2e6662f3953ff674727303e7But I don't see anything that says "it's me". The only commit that does non-trivial changes to x86/vmx seems to be "KVM: VMX: check descriptor table exits on instruction emulation":
That seems unlikely, it's a completely different file and it would only affect the outside (non-nested) environment rather than your own kernel.
The only instance of "0x86" in the registers is in the flags:
Quoted 11 lines
RSP: 0018:ffffc90001ac7998 EFLAGS: 00010086 RAX: ffffc90001ac79c8 RBX: fffffe0000000000 RCX: 0000000000040000 RDX: ffffc9000e20f000 RSI: 000000000000b452 RDI: 000000000000b453 RBP: 0000000000000ec0 R08: ffffffff83987523 R09: ffffffff811c7eca R10: ffff8880a4e94200 R11: 0000000000000002 R12: dffffc0000000000 R13: fffffe0000000ec8 R14: ffffffff880016f0 R15: fffffe0000000ecb FS: 00007fb50e370700(0000) GS:ffff8880ae800000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 000000000000005c CR3: 0000000092fc7000 CR4: 00000000001426f0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
That would suggest a miscompilation of the inline assembly, which does push the flags:
#ifdef CONFIG_X86_64
"mov %%" _ASM_SP ", %[sp]\n\t"
"and $0xfffffffffffffff0, %%" _ASM_SP "\n\t"
"push $%c[ss]\n\t"
"push %[sp]\n\t"
#endif
"pushf\n\t"
__ASM_SIZE(push) " $%c[cs]\n\t"
CALL_NOSPECIt would not explain why it suddenly started to break, unless the clang version also changed, but it would be easy to ascertain and fix (in either KVM or clang). Dmitry, can you send me the vmx.o and kvm-intel.ko files?
Thanks,
Paolo
Quoted 40 lines · Paolo Bonzini
Quoted 6 lines · Dmitry Vyukov
The commit range is presumably
fb279f4e238617417b132a550f24c1e86d922558..63849c8f410717eb2e6662f3953ff674727303e7But I don't see anything that says "it's me". The only commit that does non-trivial changes to x86/vmx seems to be "KVM: VMX: check descriptor table exits on instruction emulation":That seems unlikely, it's a completely different file and it would only affect the outside (non-nested) environment rather than your own kernel.
The only instance of "
0x86" in the registers is in the flags:Quoted 11 lines
RSP: 0018:ffffc90001ac7998 EFLAGS: 00010086 RAX: ffffc90001ac79c8 RBX: fffffe0000000000 RCX: 0000000000040000 RDX: ffffc9000e20f000 RSI: 000000000000b452 RDI: 000000000000b453 RBP: 0000000000000ec0 R08: ffffffff83987523 R09: ffffffff811c7eca R10: ffff8880a4e94200 R11: 0000000000000002 R12: dffffc0000000000 R13: fffffe0000000ec8 R14: ffffffff880016f0 R15: fffffe0000000ecb FS: 00007fb50e370700(0000) GS:ffff8880ae800000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 000000000000005c CR3: 0000000092fc7000 CR4: 00000000001426f0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400That would suggest a miscompilation of the inline assembly, which does push the flags:
#ifdef CONFIG_X86_64 "mov %%" _ASM_SP ", %[sp]\n\t" "and $0xfffffffffffffff0, %%" _ASM_SP "\n\t" "push $%c[ss]\n\t" "push %[sp]\n\t" #endif "pushf\n\t" __ASM_SIZE(push) " $%c[cs]\n\t" CALL_NOSPEC
Quoted 4 lines
It would not explain why it suddenly started to break, unless the clang version also changed, but it would be easy to ascertain and fix (in either KVM or clang). Dmitry, can you send me the vmx.o and kvm-intel.ko files?
On a quick glance, Clang does not miscompile this part.
Attached is the disassembly of handle_external_interrupt_irqoff() from v5.4 (where the problem seems to also reproduce) with Clang and GCC. They do virtually the same (look for asm blob after kvm_before_interrupt()).
vmlinux.gcc: file format elf64-x86-64
Disassembly of section .text:
ffffffff8118dfe0 <handle_external_interrupt_irqoff>:handle_external_interrupt_irqoff():/usr/local/google/src/linux-trunk/arch/x86/kvm/vmx/vmx.c:6230
kvm_after_interrupt(&vmx->vcpu); } }
static void handle_external_interrupt_irqoff(struct kvm_vcpu *vcpu) {
ffffffff8118dfe0: 41 55 push %r13 ffffffff8118dfe2: 41 54 push %r12 ffffffff8118dfe4: 55 push %rbp ffffffff8118dfe5: 48 89 fd mov %rdi,%rbp ffffffff8118dfe8: 53 push %rbx
arch_static_branch():/usr/local/google/src/linux-trunk/./arch/x86/include/asm/jump_label.h:25 #include <linux/stringify.h>
#include <linux/types.h>
static __always_inline bool arch_static_branch(struct static_key *key, bool branch)
{
asm_volatile_goto("1:"
ffffffff8118dfe9: e8 42 79 57 00 callq ffffffff81705930 <__sanitizer_cov_trace_pc>
ffffffff8118dfee: 0f 1f 44 00 00 nopl 0x0(%rax,%rax,1)
ffffffff8118dff3: e8 38 79 57 00 callq ffffffff81705930 <__sanitizer_cov_trace_pc>__vmcs_readl():/usr/local/google/src/linux-trunk/arch/x86/kvm/vmx/ops.h:70
static __always_inline unsigned long __vmcs_readl(unsigned long field)
{
unsigned long value;
asm volatile("1: vmread %2, %1\n\t"
ffffffff8118dff8: bb 04 44 00 00 mov $0x4404,%ebx
ffffffff8118dffd: 0f 78 db vmread %rbx,%rbx
ffffffff8118e000: 3e 77 0d ja,pt ffffffff8118e010 <handle_external_interrupt_irqoff+0x30>
ffffffff8118e003: 48 89 df mov %rbx,%rdi
ffffffff8118e006: 48 31 f6 xor %rsi,%rsi
ffffffff8118e009: e8 62 93 00 00 callq ffffffff81197370 <vmread_error>
ffffffff8118e00e: 31 db xor %ebx,%ebx
is_external_intr():
/usr/local/google/src/linux-trunk/arch/x86/kvm/vmx/vmcs.h:129
== (INTR_TYPE_NMI_INTR | INTR_INFO_VALID_MASK);
}static inline bool is_external_intr(u32 intr_info)
{
return (intr_info & (INTR_INFO_VALID_MASK | INTR_INFO_INTR_TYPE_MASK))ffffffff8118e010: 41 89 dc mov %ebx,%r12dhandle_external_interrupt_irqoff():/usr/local/google/src/linux-trunk/arch/x86/kvm/vmx/vmx.c:6240
#endif gate_desc *desc; u32 intr_info; intr_info = vmcs_read32(VM_EXIT_INTR_INFO); if (WARN_ONCE(!is_external_intr(intr_info),
ffffffff8118e013: bf 00 00 00 80 mov $0x80000000,%edivmcs_read32():/usr/local/google/src/linux-trunk/arch/x86/kvm/vmx/ops.h:102 static __always_inline u32 vmcs_read32(unsigned long field)
{
vmcs_check32(field);
if (static_branch_unlikely(&enable_evmcs))
return evmcs_read32(field);
return __vmcs_readl(field);ffffffff8118e018: 41 89 dd mov %ebx,%r13dis_external_intr():/usr/local/google/src/linux-trunk/arch/x86/kvm/vmx/vmcs.h:129ffffffff8118e01b: 41 81 e4 00 07 00 80 and $0x80000700,%r12dhandle_external_interrupt_irqoff():/usr/local/google/src/linux-trunk/arch/x86/kvm/vmx/vmx.c:6240
ffffffff8118e022: 44 89 e6 mov %r12d,%esi ffffffff8118e025: e8 76 7a 57 00 callq ffffffff81705aa0 <__sanitizer_cov_trace_const_cmp4> ffffffff8118e02a: 41 81 fc 00 00 00 80 cmp $0x80000000,%r12d ffffffff8118e031: 0f 85 7a 01 00 00 jne ffffffff8118e1b1 <handle_external_interrupt_irqoff+0x1d1> /usr/local/google/src/linux-trunk/arch/x86/kvm/vmx/vmx.c:6244 "KVM: unexpected VM-Exit interrupt info: 0x%x", intr_info)) return; vector = intr_info & INTR_INFO_VECTOR_MASK; ffffffff8118e037: e8 f4 78 57 00 callq ffffffff81705930 <__sanitizer_cov_trace_pc> /usr/local/google/src/linux-trunk/arch/x86/kvm/vmx/vmx.c:6245 desc = (gate_desc *)host_idt_base + vector;
ffffffff8118e03c: 0f b6 db movzbl %bl,%ebxgate_offset():/usr/local/google/src/linux-trunk/./arch/x86/include/asm/desc_defs.h:93 typedef struct gate_struct gate_desc;
static inline unsigned long gate_offset(const gate_desc *g) {
#ifdef CONFIG_X86_64 return g->offset_low | ((unsigned long)g->offset_middle << 16) |
ffffffff8118e03f: 48 b9 00 00 00 00 00 movabs $0xdffffc0000000000,%rcxffffffff8118e046: fc ff dfhandle_external_interrupt_irqoff():/usr/local/google/src/linux-trunk/arch/x86/kvm/vmx/vmx.c:6245ffffffff8118e049: 48 c1 e3 04 shl $0x4,%rbxffffffff8118e04d: 48 03 1d 2c 68 2f 0a add 0xa2f682c(%rip),%rbx # ffffffff8b484880 <host_idt_base>gate_offset():/usr/local/google/src/linux-trunk/./arch/x86/include/asm/desc_defs.h:93
ffffffff8118e054: 48 89 d8 mov %rbx,%rax ffffffff8118e057: 48 c1 e8 03 shr $0x3,%rax ffffffff8118e05b: 0f b6 14 08 movzbl (%rax,%rcx,1),%edx ffffffff8118e05f: 48 8d 43 01 lea 0x1(%rbx),%rax ffffffff8118e063: 48 89 c6 mov %rax,%rsi ffffffff8118e066: 48 c1 ee 03 shr $0x3,%rsi ffffffff8118e06a: 0f b6 0c 0e movzbl (%rsi,%rcx,1),%ecx ffffffff8118e06e: 48 89 de mov %rbx,%rsi ffffffff8118e071: 83 e6 07 and $0x7,%esi ffffffff8118e074: 40 38 f2 cmp %sil,%dl ffffffff8118e077: 40 0f 9e c6 setle %sil ffffffff8118e07b: 84 d2 test %dl,%dl ffffffff8118e07d: 0f 95 c2 setne %dl ffffffff8118e080: 40 84 d6 test %dl,%sil ffffffff8118e083: 0f 85 7e 01 00 00 jne ffffffff8118e207 <handle_external_interrupt_irqoff+0x227> ffffffff8118e089: 83 e0 07 and $0x7,%eax ffffffff8118e08c: 38 c1 cmp %al,%cl ffffffff8118e08e: 0f 9e c2 setle %dl ffffffff8118e091: 84 c9 test %cl,%cl ffffffff8118e093: 0f 95 c0 setne %al ffffffff8118e096: 84 c2 test %al,%dl ffffffff8118e098: 0f 85 69 01 00 00 jne ffffffff8118e207 <handle_external_interrupt_irqoff+0x227> ffffffff8118e09e: 48 8d 7b 06 lea 0x6(%rbx),%rdi ffffffff8118e0a2: 44 0f b7 2b movzwl (%rbx),%r13d ffffffff8118e0a6: 48 b9 00 00 00 00 00 movabs $0xdffffc0000000000,%rcx ffffffff8118e0ad: fc ff df ffffffff8118e0b0: 48 89 f8 mov %rdi,%rax ffffffff8118e0b3: 48 c1 e8 03 shr $0x3,%rax ffffffff8118e0b7: 0f b6 14 08 movzbl (%rax,%rcx,1),%edx ffffffff8118e0bb: 48 8d 43 07 lea 0x7(%rbx),%rax ffffffff8118e0bf: 48 89 c6 mov %rax,%rsi ffffffff8118e0c2: 48 c1 ee 03 shr $0x3,%rsi ffffffff8118e0c6: 0f b6 0c 0e movzbl (%rsi,%rcx,1),%ecx ffffffff8118e0ca: 48 89 fe mov %rdi,%rsi ffffffff8118e0cd: 83 e6 07 and $0x7,%esi ffffffff8118e0d0: 40 38 f2 cmp %sil,%dl ffffffff8118e0d3: 40 0f 9e c6 setle %sil ffffffff8118e0d7: 84 d2 test %dl,%dl ffffffff8118e0d9: 0f 95 c2 setne %dl ffffffff8118e0dc: 40 84 d6 test %dl,%sil ffffffff8118e0df: 0f 85 13 01 00 00 jne ffffffff8118e1f8 <handle_external_interrupt_irqoff+0x218> ffffffff8118e0e5: 83 e0 07 and $0x7,%eax ffffffff8118e0e8: 38 c1 cmp %al,%cl ffffffff8118e0ea: 0f 9e c2 setle %dl ffffffff8118e0ed: 84 c9 test %cl,%cl ffffffff8118e0ef: 0f 95 c0 setne %al ffffffff8118e0f2: 84 c2 test %al,%dl ffffffff8118e0f4: 0f 85 fe 00 00 00 jne ffffffff8118e1f8 <handle_external_interrupt_irqoff+0x218> /usr/local/google/src/linux-trunk/./arch/x86/include/asm/desc_defs.h:94 ((unsigned long) g->offset_high << 32); ffffffff8118e0fa: 48 8d 7b 08 lea 0x8(%rbx),%rdi /usr/local/google/src/linux-trunk/./arch/x86/include/asm/desc_defs.h:93 return g->offset_low | ((unsigned long)g->offset_middle << 16) | ffffffff8118e0fe: 44 0f b7 63 06 movzwl 0x6(%rbx),%r12d /usr/local/google/src/linux-trunk/./arch/x86/include/asm/desc_defs.h:94 ((unsigned long) g->offset_high << 32); ffffffff8118e103: 48 b9 00 00 00 00 00 movabs $0xdffffc0000000000,%rcx ffffffff8118e10a: fc ff df ffffffff8118e10d: 48 89 f8 mov %rdi,%rax ffffffff8118e110: 48 c1 e8 03 shr $0x3,%rax ffffffff8118e114: 0f b6 14 08 movzbl (%rax,%rcx,1),%edx ffffffff8118e118: 48 8d 43 0b lea 0xb(%rbx),%rax ffffffff8118e11c: 48 89 c6 mov %rax,%rsi /usr/local/google/src/linux-trunk/./arch/x86/include/asm/desc_defs.h:93 return g->offset_low | ((unsigned long)g->offset_middle << 16) | ffffffff8118e11f: 49 c1 e4 10 shl $0x10,%r12 /usr/local/google/src/linux-trunk/./arch/x86/include/asm/desc_defs.h:94 ((unsigned long) g->offset_high << 32); ffffffff8118e123: 48 c1 ee 03 shr $0x3,%rsi ffffffff8118e127: 0f b6 0c 0e movzbl (%rsi,%rcx,1),%ecx ffffffff8118e12b: 48 89 fe mov %rdi,%rsi ffffffff8118e12e: 83 e6 07 and $0x7,%esi ffffffff8118e131: 40 38 f2 cmp %sil,%dl ffffffff8118e134: 40 0f 9e c6 setle %sil ffffffff8118e138: 84 d2 test %dl,%dl ffffffff8118e13a: 0f 95 c2 setne %dl ffffffff8118e13d: 40 84 d6 test %dl,%sil ffffffff8118e140: 0f 85 a3 00 00 00 jne ffffffff8118e1e9 <handle_external_interrupt_irqoff+0x209> ffffffff8118e146: 83 e0 07 and $0x7,%eax ffffffff8118e149: 38 c1 cmp %al,%cl ffffffff8118e14b: 0f 9e c2 setle %dl ffffffff8118e14e: 84 c9 test %cl,%cl ffffffff8118e150: 0f 95 c0 setne %al ffffffff8118e153: 84 c2 test %al,%dl ffffffff8118e155: 0f 85 8e 00 00 00 jne ffffffff8118e1e9 <handle_external_interrupt_irqoff+0x209> ffffffff8118e15b: 8b 5b 08 mov 0x8(%rbx),%ebx
kvm_before_interrupt():/usr/local/google/src/linux-trunk/arch/x86/kvm/x86.h:352
DECLARE_PER_CPU(struct kvm_vcpu *, current_vcpu);
static inline void kvm_before_interrupt(struct kvm_vcpu *vcpu)
{
__this_cpu_write(current_vcpu, vcpu);
ffffffff8118e15e: 48 c7 c7 80 fd e3 87 mov $0xffffffff87e3fd80,%rdi
ffffffff8118e165: e8 26 21 6a 02 callq ffffffff83830290 <__this_cpu_preempt_check>
ffffffff8118e16a: 65 48 89 2d 5e ff e8 mov %rbp,%gs:0x7ee8ff5e(%rip) # 1e0d0 <current_vcpu>ffffffff8118e171: 7egate_offset():/usr/local/google/src/linux-trunk/./arch/x86/include/asm/desc_defs.h:94ffffffff8118e172: 48 c1 e3 20 shl $0x20,%rbx/usr/local/google/src/linux-trunk/./arch/x86/include/asm/desc_defs.h:93
return g->offset_low | ((unsigned long)g->offset_middle << 16) | ffffffff8118e176: 4c 09 e3 or %r12,%rbx ffffffff8118e179: 4c 09 eb or %r13,%rbx handle_external_interrupt_irqoff(): /usr/local/google/src/linux-trunk/arch/x86/kvm/vmx/vmx.c:6250 entry = gate_offset(desc); kvm_before_interrupt(vcpu); asm volatile( ffffffff8118e17c: 48 89 e0 mov %rsp,%rax ffffffff8118e17f: 48 83 e4 f0 and $0xfffffffffffffff0,%rsp ffffffff8118e183: 6a 18 pushq $0x18 ffffffff8118e185: 50 push %rax ffffffff8118e186: 9c pushfq ffffffff8118e187: 6a 10 pushq $0x10 ffffffff8118e189: ff d3 callq *%rbx
kvm_after_interrupt():/usr/local/google/src/linux-trunk/arch/x86/kvm/x86.h:357 }
static inline void kvm_after_interrupt(struct kvm_vcpu *vcpu)
{
__this_cpu_write(current_vcpu, NULL);
ffffffff8118e18b: 48 c7 c7 80 fd e3 87 mov $0xffffffff87e3fd80,%rdi
ffffffff8118e192: e8 f9 20 6a 02 callq ffffffff83830290 <__this_cpu_preempt_check>
ffffffff8118e197: 65 48 c7 05 2d ff e8 movq $0x0,%gs:0x7ee8ff2d(%rip) # 1e0d0 <current_vcpu>
ffffffff8118e19e: 7e 00 00 00 00
handle_external_interrupt_irqoff():
/usr/local/google/src/linux-trunk/arch/x86/kvm/vmx/vmx.c:6272
[ss]"i"(__KERNEL_DS),
[cs]"i"(__KERNEL_CS)
);
kvm_after_interrupt(vcpu);
}
ffffffff8118e1a3: 5b pop %rbx
ffffffff8118e1a4: 5d pop %rbp
ffffffff8118e1a5: 41 5c pop %r12
ffffffff8118e1a7: 41 5d pop %r13
ffffffff8118e1a9: e9 82 77 57 00 jmpq ffffffff81705930 <__sanitizer_cov_trace_pc>
vmcs_read32():
/usr/local/google/src/linux-trunk/arch/x86/kvm/vmx/ops.h:101
return evmcs_read32(field);ffffffff8118e1ae: 45 31 ed xor %r13d,%r13dhandle_external_interrupt_irqoff():/usr/local/google/src/linux-trunk/arch/x86/kvm/vmx/vmx.c:6240 (discriminator 1)
if (WARN_ONCE(!is_external_intr(intr_info),ffffffff8118e1b1: e8 7a 77 57 00 callq ffffffff81705930 <__sanitizer_cov_trace_pc>ffffffff8118e1b6: 0f b6 1d cf 2f fa 08 movzbl 0x8fa2fcf(%rip),%ebx # ffffffff8a13118c <__warned.77930>
ffffffff8118e1bd: 31 ff xor %edi,%edi ffffffff8118e1bf: 89 de mov %ebx,%esi ffffffff8118e1c1: e8 9a 78 57 00 callq ffffffff81705a60 <__sanitizer_cov_trace_const_cmp1> ffffffff8118e1c6: 84 db test %bl,%bl ffffffff8118e1c8: 75 d9 jne ffffffff8118e1a3 <handle_external_interrupt_irqoff+0x1c3> /usr/local/google/src/linux-trunk/arch/x86/kvm/vmx/vmx.c:6240 (discriminator 3) ffffffff8118e1ca: e8 61 77 57 00 callq ffffffff81705930 <__sanitizer_cov_trace_pc> ffffffff8118e1cf: 44 89 ee mov %r13d,%esi ffffffff8118e1d2: 48 c7 c7 e0 fc e3 87 mov $0xffffffff87e3fce0,%rdi ffffffff8118e1d9: c6 05 ac 2f fa 08 01 movb $0x1,0x8fa2fac(%rip) # ffffffff8a13118c <__warned.77930> ffffffff8118e1e0: e8 10 1a 2a 00 callq ffffffff8142fbf5 <__warn_printk>
ffffffff8118e1e5: 0f 0b ud2ffffffff8118e1e7: eb ba jmp ffffffff8118e1a3 <handle_external_interrupt_irqoff+0x1c3>gate_offset():/usr/local/google/src/linux-trunk/./arch/x86/include/asm/desc_defs.h:94
((unsigned long) g->offset_high << 32); ffffffff8118e1e9: be 04 00 00 00 mov $0x4,%esi ffffffff8118e1ee: e8 4d 80 91 00 callq ffffffff81aa6240 <__asan_report_load_n_noabort> ffffffff8118e1f3: e9 63 ff ff ff jmpq ffffffff8118e15b <handle_external_interrupt_irqoff+0x17b> /usr/local/google/src/linux-trunk/./arch/x86/include/asm/desc_defs.h:93 return g->offset_low | ((unsigned long)g->offset_middle << 16) | ffffffff8118e1f8: be 02 00 00 00 mov $0x2,%esi ffffffff8118e1fd: e8 3e 80 91 00 callq ffffffff81aa6240 <__asan_report_load_n_noabort> ffffffff8118e202: e9 f3 fe ff ff jmpq ffffffff8118e0fa <handle_external_interrupt_irqoff+0x11a> ffffffff8118e207: be 02 00 00 00 mov $0x2,%esi ffffffff8118e20c: 48 89 df mov %rbx,%rdi ffffffff8118e20f: e8 2c 80 91 00 callq ffffffff81aa6240 <__asan_report_load_n_noabort> ffffffff8118e214: e9 85 fe ff ff jmpq ffffffff8118e09e <handle_external_interrupt_irqoff+0xbe>
handle_external_interrupt_irqoff():/usr/local/google/src/linux-trunk/./arch/x86/include/asm/desc_defs.h:93ffffffff8118e219: 0f 1f 80 00 00 00 00 nopl 0x0(%rax)
vmlinux.clang: file format elf64-x86-64
Disassembly of section .text:
ffffffff811b7850 <handle_external_interrupt_irqoff>:handle_external_interrupt_irqoff():/usr/local/google/src/linux-trunk/arch/x86/kvm/vmx/vmx.c:6230
kvm_after_interrupt(&vmx->vcpu); } }
static void handle_external_interrupt_irqoff(struct kvm_vcpu *vcpu) {
ffffffff811b7850: 55 push %rbp ffffffff811b7851: 41 57 push %r15 ffffffff811b7853: 41 56 push %r14 ffffffff811b7855: 41 55 push %r13 ffffffff811b7857: 41 54 push %r12 ffffffff811b7859: 53 push %rbx ffffffff811b785a: 48 83 ec 10 sub $0x10,%rsp ffffffff811b785e: 49 89 fe mov %rdi,%r14 ffffffff811b7861: e8 6a 06 56 00 callq ffffffff81717ed0 <__sanitizer_cov_trace_pc> ffffffff811b7866: 31 db xor %ebx,%ebx
arch_static_branch():/usr/local/google/src/linux-trunk/arch/x86/kvm/vmx/vmx.c:6230ffffffff811b7868: 0f 1f 44 00 00 nopl 0x0(%rax,%rax,1)vmcs_read32():/usr/local/google/src/linux-trunk/./arch/x86/include/asm/jump_label.h:25 #include <linux/stringify.h>
#include <linux/types.h>
static __always_inline bool arch_static_branch(struct static_key *key, bool branch)
{
asm_volatile_goto("1:"
ffffffff811b786d: e8 5e 06 56 00 callq ffffffff81717ed0 <__sanitizer_cov_trace_pc>
ffffffff811b7872: b8 04 44 00 00 mov $0x4404,%eax__vmcs_readl():/usr/local/google/src/linux-trunk/arch/x86/kvm/vmx/ops.h:70
static __always_inline unsigned long __vmcs_readl(unsigned long field)
{
unsigned long value;
asm volatile("1: vmread %2, %1\n\t"
ffffffff811b7877: 0f 78 c3 vmread %rax,%rbx
ffffffff811b787a: 3e 77 0d ja,pt ffffffff811b788a <handle_external_interrupt_irqoff+0x3a>
ffffffff811b787d: 48 89 c7 mov %rax,%rdi
ffffffff811b7880: 48 31 f6 xor %rsi,%rsi
ffffffff811b7883: e8 d8 2f ff ff callq ffffffff811aa860 <vmread_error>
ffffffff811b7888: 31 db xor %ebx,%ebx
is_external_intr():
/usr/local/google/src/linux-trunk/arch/x86/kvm/vmx/vmcs.h:129
== (INTR_TYPE_NMI_INTR | INTR_INFO_VALID_MASK);
}static inline bool is_external_intr(u32 intr_info)
{
return (intr_info & (INTR_INFO_VALID_MASK | INTR_INFO_INTR_TYPE_MASK))
ffffffff811b788a: 89 dd mov %ebx,%ebp ffffffff811b788c: 81 e5 00 07 00 80 and $0x80000700,%ebp /usr/local/google/src/linux-trunk/arch/x86/kvm/vmx/vmcs.h:130 == (INTR_INFO_VALID_MASK | INTR_TYPE_EXT_INTR); ffffffff811b7892: bf 00 00 00 80 mov $0x80000000,%edi ffffffff811b7897: 89 ee mov %ebp,%esi ffffffff811b7899: e8 d2 09 56 00 callq ffffffff81718270 <__sanitizer_cov_trace_const_cmp4> ffffffff811b789e: 81 fd 00 00 00 80 cmp $0x80000000,%ebp handle_external_interrupt_irqoff(): /usr/local/google/src/linux-trunk/arch/x86/kvm/vmx/vmx.c:6240 #endif gate_desc *desc; u32 intr_info; intr_info = vmcs_read32(VM_EXIT_INTR_INFO); if (WARN_ONCE(!is_external_intr(intr_info),
ffffffff811b78a4: 0f 85 14 01 00 00 jne ffffffff811b79be <handle_external_interrupt_irqoff+0x16e>ffffffff811b78aa: 49 bc 00 00 00 00 00 movabs $0xdffffc0000000000,%r12ffffffff811b78b1: fc ff df/usr/local/google/src/linux-trunk/arch/x86/kvm/vmx/vmx.c:6244
"KVM: unexpected VM-Exit interrupt info: 0x%x", intr_info)) return; vector = intr_info & INTR_INFO_VECTOR_MASK; ffffffff811b78b4: e8 17 06 56 00 callq ffffffff81717ed0 <__sanitizer_cov_trace_pc> ffffffff811b78b9: 0f b6 eb movzbl %bl,%ebp /usr/local/google/src/linux-trunk/arch/x86/kvm/vmx/vmx.c:6245 desc = (gate_desc *)host_idt_base + vector;
ffffffff811b78bc: 48 8b 1d 7d e3 2a 09 mov 0x92ae37d(%rip),%rbx # ffffffff8a465c40 <host_idt_base>gate_offset():/usr/local/google/src/linux-trunk/./arch/x86/include/asm/desc_defs.h:93 typedef struct gate_struct gate_desc;
static inline unsigned long gate_offset(const gate_desc *g) {
#ifdef CONFIG_X86_64 return g->offset_low | ((unsigned long)g->offset_middle << 16) | ffffffff811b78c3: 48 c1 e5 04 shl $0x4,%rbp ffffffff811b78c7: 4c 8d 2c 2b lea (%rbx,%rbp,1),%r13 ffffffff811b78cb: 4c 8d 7c 2b 01 lea 0x1(%rbx,%rbp,1),%r15 ffffffff811b78d0: 4c 89 e8 mov %r13,%rax ffffffff811b78d3: 48 c1 e8 03 shr $0x3,%rax ffffffff811b78d7: 42 8a 04 20 mov (%rax,%r12,1),%al ffffffff811b78db: 84 c0 test %al,%al ffffffff811b78dd: 0f 85 18 01 00 00 jne ffffffff811b79fb <handle_external_interrupt_irqoff+0x1ab> ffffffff811b78e3: 4c 89 f8 mov %r15,%rax ffffffff811b78e6: 48 c1 e8 03 shr $0x3,%rax ffffffff811b78ea: 42 8a 04 20 mov (%rax,%r12,1),%al ffffffff811b78ee: 84 c0 test %al,%al ffffffff811b78f0: 0f 85 25 01 00 00 jne ffffffff811b7a1b <handle_external_interrupt_irqoff+0x1cb> ffffffff811b78f6: 41 0f b7 55 00 movzwl 0x0(%r13),%edx ffffffff811b78fb: 4c 8d 6c 2b 06 lea 0x6(%rbx,%rbp,1),%r13 ffffffff811b7900: 4c 8d 7c 2b 07 lea 0x7(%rbx,%rbp,1),%r15 ffffffff811b7905: 4c 89 e8 mov %r13,%rax ffffffff811b7908: 48 c1 e8 03 shr $0x3,%rax ffffffff811b790c: 42 8a 04 20 mov (%rax,%r12,1),%al ffffffff811b7910: 84 c0 test %al,%al ffffffff811b7912: 0f 85 23 01 00 00 jne ffffffff811b7a3b <handle_external_interrupt_irqoff+0x1eb> ffffffff811b7918: 4c 89 34 24 mov %r14,(%rsp) ffffffff811b791c: 4c 89 f8 mov %r15,%rax ffffffff811b791f: 48 c1 e8 03 shr $0x3,%rax ffffffff811b7923: 42 8a 04 20 mov (%rax,%r12,1),%al ffffffff811b7927: 84 c0 test %al,%al ffffffff811b7929: 0f 85 34 01 00 00 jne ffffffff811b7a63 <handle_external_interrupt_irqoff+0x213> ffffffff811b792f: 45 0f b7 75 00 movzwl 0x0(%r13),%r14d ffffffff811b7934: 49 c1 e6 10 shl $0x10,%r14 /usr/local/google/src/linux-trunk/./arch/x86/include/asm/desc_defs.h:94 ((unsigned long) g->offset_high << 32); ffffffff811b7938: 4c 8d 6c 2b 08 lea 0x8(%rbx,%rbp,1),%r13 ffffffff811b793d: 4c 8d 7c 2b 0b lea 0xb(%rbx,%rbp,1),%r15 ffffffff811b7942: 4c 89 e8 mov %r13,%rax ffffffff811b7945: 48 c1 e8 03 shr $0x3,%rax ffffffff811b7949: 42 8a 04 20 mov (%rax,%r12,1),%al ffffffff811b794d: 84 c0 test %al,%al ffffffff811b794f: 0f 85 34 01 00 00 jne ffffffff811b7a89 <handle_external_interrupt_irqoff+0x239> ffffffff811b7955: 49 09 d6 or %rdx,%r14 ffffffff811b7958: 4c 89 f8 mov %r15,%rax ffffffff811b795b: 48 c1 e8 03 shr $0x3,%rax ffffffff811b795f: 42 8a 04 20 mov (%rax,%r12,1),%al ffffffff811b7963: 84 c0 test %al,%al ffffffff811b7965: 0f 85 44 01 00 00 jne ffffffff811b7aaf <handle_external_interrupt_irqoff+0x25f> ffffffff811b796b: 41 8b 45 00 mov 0x0(%r13),%eax ffffffff811b796f: 48 c1 e0 20 shl $0x20,%rax /usr/local/google/src/linux-trunk/./arch/x86/include/asm/desc_defs.h:93 return g->offset_low | ((unsigned long)g->offset_middle << 16) |
ffffffff811b7973: 49 09 c6 or %rax,%r14kvm_before_interrupt():/usr/local/google/src/linux-trunk/arch/x86/kvm/x86.h:352
DECLARE_PER_CPU(struct kvm_vcpu *, current_vcpu);
static inline void kvm_before_interrupt(struct kvm_vcpu *vcpu)
{
__this_cpu_write(current_vcpu, vcpu);
ffffffff811b7976: 48 c7 c7 d8 93 84 88 mov $0xffffffff888493d8,%rdi
ffffffff811b797d: e8 ce f3 69 02 callq ffffffff83856d50 <__this_cpu_preempt_check>
ffffffff811b7982: 48 8b 04 24 mov (%rsp),%rax
ffffffff811b7986: 65 48 89 05 2a 67 e6 mov %rax,%gs:0x7ee6672a(%rip) # 1e0b8 <current_vcpu>
ffffffff811b798d: 7e
handle_external_interrupt_irqoff():
/usr/local/google/src/linux-trunk/arch/x86/kvm/vmx/vmx.c:6250
entry = gate_offset(desc);
kvm_before_interrupt(vcpu);
asm volatile(
ffffffff811b798e: 4c 89 74 24 08 mov %r14,0x8(%rsp)
ffffffff811b7993: 48 89 e0 mov %rsp,%rax
ffffffff811b7996: 48 83 e4 f0 and $0xfffffffffffffff0,%rsp
ffffffff811b799a: 6a 18 pushq $0x18
ffffffff811b799c: 50 push %rax
ffffffff811b799d: 9c pushfq
ffffffff811b799e: 6a 10 pushq $0x10
ffffffff811b79a0: ff 54 24 08 callq *0x8(%rsp)kvm_after_interrupt():/usr/local/google/src/linux-trunk/arch/x86/kvm/x86.h:357 }
static inline void kvm_after_interrupt(struct kvm_vcpu *vcpu)
{
__this_cpu_write(current_vcpu, NULL);
ffffffff811b79a4: 48 c7 c7 d8 93 84 88 mov $0xffffffff888493d8,%rdi
ffffffff811b79ab: e8 a0 f3 69 02 callq ffffffff83856d50 <__this_cpu_preempt_check>
ffffffff811b79b0: 65 48 c7 05 fc 66 e6 movq $0x0,%gs:0x7ee666fc(%rip) # 1e0b8 <current_vcpu>ffffffff811b79b7: 7e 00 00 00 00ffffffff811b79bc: eb 0e jmp ffffffff811b79cc <handle_external_interrupt_irqoff+0x17c>handle_external_interrupt_irqoff():/usr/local/google/src/linux-trunk/arch/x86/kvm/vmx/vmx.c:6240
if (WARN_ONCE(!is_external_intr(intr_info),
ffffffff811b79be: 80 3d 80 3a e4 07 01 cmpb $0x1,0x7e43a80(%rip) # ffffffff88ffb445 <handle_external_interrupt_irqoff.__warned> ffffffff811b79c5: 75 14 jne ffffffff811b79db <handle_external_interrupt_irqoff+0x18b> ffffffff811b79c7: e8 04 05 56 00 callq ffffffff81717ed0 <__sanitizer_cov_trace_pc> /usr/local/google/src/linux-trunk/arch/x86/kvm/vmx/vmx.c:6272 [ss]"i"(__KERNEL_DS), [cs]"i"(__KERNEL_CS) ); kvm_after_interrupt(vcpu); } ffffffff811b79cc: 48 83 c4 10 add $0x10,%rsp ffffffff811b79d0: 5b pop %rbx ffffffff811b79d1: 41 5c pop %r12 ffffffff811b79d3: 41 5d pop %r13 ffffffff811b79d5: 41 5e pop %r14 ffffffff811b79d7: 41 5f pop %r15 ffffffff811b79d9: 5d pop %rbp ffffffff811b79da: c3 retq /usr/local/google/src/linux-trunk/arch/x86/kvm/vmx/vmx.c:6240 if (WARN_ONCE(!is_external_intr(intr_info), ffffffff811b79db: e8 f0 04 56 00 callq ffffffff81717ed0 <__sanitizer_cov_trace_pc> ffffffff811b79e0: c6 05 5e 3a e4 07 01 movb $0x1,0x7e43a5e(%rip) # ffffffff88ffb445 <handle_external_interrupt_irqoff.__warned> ffffffff811b79e7: 48 c7 c7 47 88 7b 88 mov $0xffffffff887b8847,%rdi ffffffff811b79ee: 89 de mov %ebx,%esi ffffffff811b79f0: 31 c0 xor %eax,%eax ffffffff811b79f2: e8 e9 76 29 00 callq ffffffff8144f0e0 <__warn_printk>
ffffffff811b79f7: 0f 0b ud2ffffffff811b79f9: eb d1 jmp ffffffff811b79cc <handle_external_interrupt_irqoff+0x17c>gate_offset():/usr/local/google/src/linux-trunk/./arch/x86/include/asm/desc_defs.h:93
ffffffff811b79fb: 44 89 e9 mov %r13d,%ecx ffffffff811b79fe: 80 e1 07 and $0x7,%cl ffffffff811b7a01: 38 c1 cmp %al,%cl ffffffff811b7a03: 0f 8c da fe ff ff jl ffffffff811b78e3 <handle_external_interrupt_irqoff+0x93> ffffffff811b7a09: be 02 00 00 00 mov $0x2,%esi ffffffff811b7a0e: 4c 89 ef mov %r13,%rdi ffffffff811b7a11: e8 aa 8f 8f 00 callq ffffffff81ab09c0 <__asan_report_load_n_noabort> ffffffff811b7a16: e9 c8 fe ff ff jmpq ffffffff811b78e3 <handle_external_interrupt_irqoff+0x93> ffffffff811b7a1b: 44 89 f9 mov %r15d,%ecx ffffffff811b7a1e: 80 e1 07 and $0x7,%cl ffffffff811b7a21: 38 c1 cmp %al,%cl ffffffff811b7a23: 0f 8c cd fe ff ff jl ffffffff811b78f6 <handle_external_interrupt_irqoff+0xa6> ffffffff811b7a29: be 02 00 00 00 mov $0x2,%esi ffffffff811b7a2e: 4c 89 ff mov %r15,%rdi ffffffff811b7a31: e8 8a 8f 8f 00 callq ffffffff81ab09c0 <__asan_report_load_n_noabort> ffffffff811b7a36: e9 bb fe ff ff jmpq ffffffff811b78f6 <handle_external_interrupt_irqoff+0xa6> ffffffff811b7a3b: 44 89 e9 mov %r13d,%ecx ffffffff811b7a3e: 80 e1 07 and $0x7,%cl ffffffff811b7a41: 38 c1 cmp %al,%cl ffffffff811b7a43: 0f 8c cf fe ff ff jl ffffffff811b7918 <handle_external_interrupt_irqoff+0xc8> ffffffff811b7a49: be 02 00 00 00 mov $0x2,%esi ffffffff811b7a4e: 4c 89 ef mov %r13,%rdi ffffffff811b7a51: 48 89 14 24 mov %rdx,(%rsp) ffffffff811b7a55: e8 66 8f 8f 00 callq ffffffff81ab09c0 <__asan_report_load_n_noabort> ffffffff811b7a5a: 48 8b 14 24 mov (%rsp),%rdx ffffffff811b7a5e: e9 b5 fe ff ff jmpq ffffffff811b7918 <handle_external_interrupt_irqoff+0xc8> ffffffff811b7a63: 44 89 f9 mov %r15d,%ecx ffffffff811b7a66: 80 e1 07 and $0x7,%cl ffffffff811b7a69: 38 c1 cmp %al,%cl ffffffff811b7a6b: 0f 8c be fe ff ff jl ffffffff811b792f <handle_external_interrupt_irqoff+0xdf> ffffffff811b7a71: be 02 00 00 00 mov $0x2,%esi ffffffff811b7a76: 4c 89 ff mov %r15,%rdi ffffffff811b7a79: 49 89 d6 mov %rdx,%r14 ffffffff811b7a7c: e8 3f 8f 8f 00 callq ffffffff81ab09c0 <__asan_report_load_n_noabort> ffffffff811b7a81: 4c 89 f2 mov %r14,%rdx ffffffff811b7a84: e9 a6 fe ff ff jmpq ffffffff811b792f <handle_external_interrupt_irqoff+0xdf> /usr/local/google/src/linux-trunk/./arch/x86/include/asm/desc_defs.h:94 ((unsigned long) g->offset_high << 32); ffffffff811b7a89: 44 89 e9 mov %r13d,%ecx ffffffff811b7a8c: 80 e1 07 and $0x7,%cl ffffffff811b7a8f: 38 c1 cmp %al,%cl ffffffff811b7a91: 0f 8c be fe ff ff jl ffffffff811b7955 <handle_external_interrupt_irqoff+0x105> ffffffff811b7a97: be 04 00 00 00 mov $0x4,%esi ffffffff811b7a9c: 4c 89 ef mov %r13,%rdi ffffffff811b7a9f: 48 89 d3 mov %rdx,%rbx ffffffff811b7aa2: e8 19 8f 8f 00 callq ffffffff81ab09c0 <__asan_report_load_n_noabort> ffffffff811b7aa7: 48 89 da mov %rbx,%rdx ffffffff811b7aaa: e9 a6 fe ff ff jmpq ffffffff811b7955 <handle_external_interrupt_irqoff+0x105> ffffffff811b7aaf: 44 89 f9 mov %r15d,%ecx ffffffff811b7ab2: 80 e1 07 and $0x7,%cl ffffffff811b7ab5: 38 c1 cmp %al,%cl ffffffff811b7ab7: 0f 8c ae fe ff ff jl ffffffff811b796b <handle_external_interrupt_irqoff+0x11b> ffffffff811b7abd: be 04 00 00 00 mov $0x4,%esi ffffffff811b7ac2: 4c 89 ff mov %r15,%rdi ffffffff811b7ac5: e8 f6 8e 8f 00 callq ffffffff81ab09c0 <__asan_report_load_n_noabort> ffffffff811b7aca: e9 9c fe ff ff jmpq ffffffff811b796b <handle_external_interrupt_irqoff+0x11b>
handle_external_interrupt_irqoff():/usr/local/google/src/linux-trunk/./arch/x86/include/asm/desc_defs.h:94ffffffff811b7acf: 90 nop
Quoted 41 lines · Alexander Potapenko
Quoted 40 lines · Paolo Bonzini
Quoted 6 lines · Dmitry Vyukov
The commit range is presumably
fb279f4e238617417b132a550f24c1e86d922558..63849c8f410717eb2e6662f3953ff674727303e7But I don't see anything that says "it's me". The only commit that does non-trivial changes to x86/vmx seems to be "KVM: VMX: check descriptor table exits on instruction emulation":That seems unlikely, it's a completely different file and it would only affect the outside (non-nested) environment rather than your own kernel.
The only instance of "
0x86" in the registers is in the flags:Quoted 11 lines
RSP: 0018:ffffc90001ac7998 EFLAGS: 00010086 RAX: ffffc90001ac79c8 RBX: fffffe0000000000 RCX: 0000000000040000 RDX: ffffc9000e20f000 RSI: 000000000000b452 RDI: 000000000000b453 RBP: 0000000000000ec0 R08: ffffffff83987523 R09: ffffffff811c7eca R10: ffff8880a4e94200 R11: 0000000000000002 R12: dffffc0000000000 R13: fffffe0000000ec8 R14: ffffffff880016f0 R15: fffffe0000000ecb FS: 00007fb50e370700(0000) GS:ffff8880ae800000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 000000000000005c CR3: 0000000092fc7000 CR4: 00000000001426f0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400That would suggest a miscompilation of the inline assembly, which does push the flags:
#ifdef CONFIG_X86_64 "mov %%" _ASM_SP ", %[sp]\n\t" "and $0xfffffffffffffff0, %%" _ASM_SP "\n\t" "push $%c[ss]\n\t" "push %[sp]\n\t" #endif "pushf\n\t" __ASM_SIZE(push) " $%c[cs]\n\t" CALL_NOSPEC
Quoted 6 lines
Quoted 4 lines
It would not explain why it suddenly started to break, unless the clang version also changed, but it would be easy to ascertain and fix (in either KVM or clang). Dmitry, can you send me the vmx.o and kvm-intel.ko files?
On a quick glance, Clang does not miscompile this part.
Clang definitely miscompiles the asm, the indirect call operates on the EFLAGS value, not on @entry as expected. It looks like clang doesn't honor ASM_CALL_CONSTRAINT, which effectively tells the compiler that %rsp is getting clobbered, e.g. the "mov %r14,0x8(%rsp)" is loading @entry for "callq *0x8(%rsp)", which breaks because of asm's pushes.
clang:
kvm_before_interrupt(vcpu); asm volatile( ffffffff811b798e: 4c 89 74 24 08 mov %r14,0x8(%rsp) ffffffff811b7993: 48 89 e0 mov %rsp,%rax ffffffff811b7996: 48 83 e4 f0 and $0xfffffffffffffff0,%rsp ffffffff811b799a: 6a 18 pushq $0x18 ffffffff811b799c: 50 push %rax ffffffff811b799d: 9c pushfq ffffffff811b799e: 6a 10 pushq $0x10 ffffffff811b79a0: ff 54 24 08 callq *0x8(%rsp) <--------- calls the EFLAGS value
kvm_after_interrupt():
gcc:
kvm_before_interrupt(vcpu); asm volatile( ffffffff8118e17c: 48 89 e0 mov %rsp,%rax ffffffff8118e17f: 48 83 e4 f0 and $0xfffffffffffffff0,%rsp ffffffff8118e183: 6a 18 pushq $0x18 ffffffff8118e185: 50 push %rax ffffffff8118e186: 9c pushfq ffffffff8118e187: 6a 10 pushq $0x10 ffffffff8118e189: ff d3 callq *%rbx <-------- calls @entry
kvm_after_interrupt():
Quoted 48 lines · Sean Christopherson
Quoted 46 lines · Alexander Potapenko
Quoted 44 lines · Paolo Bonzini
Quoted 6 lines · Dmitry Vyukov
The commit range is presumably
fb279f4e238617417b132a550f24c1e86d922558..63849c8f410717eb2e6662f3953ff674727303e7But I don't see anything that says "it's me". The only commit that does non-trivial changes to x86/vmx seems to be "KVM: VMX: check descriptor table exits on instruction emulation":That seems unlikely, it's a completely different file and it would only affect the outside (non-nested) environment rather than your own kernel.
The only instance of "
0x86" in the registers is in the flags:Quoted 11 lines
RSP: 0018:ffffc90001ac7998 EFLAGS: 00010086 RAX: ffffc90001ac79c8 RBX: fffffe0000000000 RCX: 0000000000040000 RDX: ffffc9000e20f000 RSI: 000000000000b452 RDI: 000000000000b453 RBP: 0000000000000ec0 R08: ffffffff83987523 R09: ffffffff811c7eca R10: ffff8880a4e94200 R11: 0000000000000002 R12: dffffc0000000000 R13: fffffe0000000ec8 R14: ffffffff880016f0 R15: fffffe0000000ecb FS: 00007fb50e370700(0000) GS:ffff8880ae800000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 000000000000005c CR3: 0000000092fc7000 CR4: 00000000001426f0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400That would suggest a miscompilation of the inline assembly, which does push the flags:
#ifdef CONFIG_X86_64 "mov %%" _ASM_SP ", %[sp]\n\t" "and $0xfffffffffffffff0, %%" _ASM_SP "\n\t" "push $%c[ss]\n\t" "push %[sp]\n\t" #endif "pushf\n\t" __ASM_SIZE(push) " $%c[cs]\n\t" CALL_NOSPECIt would not explain why it suddenly started to break, unless the clang version also changed, but it would be easy to ascertain and fix (in either KVM or clang). Dmitry, can you send me the vmx.o and kvm-intel.ko files?
Quoted 7 lines
Quoted 1 line
On a quick glance, Clang does not miscompile this part.
Clang definitely miscompiles the asm, the indirect call operates on the EFLAGS value, not on @entry as expected. It looks like clang doesn't honor
ASM_CALL_CONSTRAINT, which effectively tells the compiler that %rsp is getting clobbered, e.g. the "mov %r14,0x8(%rsp)" is loading @entry for "callq *0x8(%rsp)", which breaks because of asm's pushes.
Ugh, I completely overlooked this. Right, this is something to work this on the Clang side.
Quoted 33 lines
clang:
kvm_before_interrupt(vcpu); asm volatile( ffffffff811b798e: 4c 89 74 24 08 mov %r14,0x8(%rsp) ffffffff811b7993: 48 89 e0 mov %rsp,%rax ffffffff811b7996: 48 83 e4 f0 and $0xfffffffffffffff0,%rsp ffffffff811b799a: 6a 18 pushq $0x18 ffffffff811b799c: 50 push %rax ffffffff811b799d: 9c pushfq ffffffff811b799e: 6a 10 pushq $0x10 ffffffff811b79a0: ff 54 24 08 callq *0x8(%rsp) <--------- calls the EFLAGS value
kvm_after_interrupt():gcc:
kvm_before_interrupt(vcpu); asm volatile( ffffffff8118e17c: 48 89 e0 mov %rsp,%rax ffffffff8118e17f: 48 83 e4 f0 and $0xfffffffffffffff0,%rsp ffffffff8118e183: 6a 18 pushq $0x18 ffffffff8118e185: 50 push %rax ffffffff8118e186: 9c pushfq ffffffff8118e187: 6a 10 pushq $0x10 ffffffff8118e189: ff d3 callq *%rbx <-------- calls @entry
kvm_after_interrupt():--
You received this message because you are subscribed to the Google Groups "syzkaller-bugs" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/syzkaller-bugs/20200323163925.GP28711%40linux.intel.com.
Signature · 10 lines
Alexander Potapenko Software Engineer Google Germany GmbH Erika-Mann-Straße, 33 80636 München Geschäftsführer: Paul Manicle, Halimah DeLaine Prado Registergericht und -nummer: Hamburg, HRB 86891 Sitz der Gesellschaft: Hamburg
Quoted 83 lines · Sean Christopherson
Quoted 41 lines · Alexander Potapenko
Quoted 40 lines · Paolo Bonzini
Quoted 6 lines · Dmitry Vyukov
The commit range is presumably
fb279f4e238617417b132a550f24c1e86d922558..63849c8f410717eb2e6662f3953ff674727303e7But I don't see anything that says "it's me". The only commit that does non-trivial changes to x86/vmx seems to be "KVM: VMX: check descriptor table exits on instruction emulation":That seems unlikely, it's a completely different file and it would only affect the outside (non-nested) environment rather than your own kernel.
The only instance of "
0x86" in the registers is in the flags:Quoted 11 lines
RSP: 0018:ffffc90001ac7998 EFLAGS: 00010086 RAX: ffffc90001ac79c8 RBX: fffffe0000000000 RCX: 0000000000040000 RDX: ffffc9000e20f000 RSI: 000000000000b452 RDI: 000000000000b453 RBP: 0000000000000ec0 R08: ffffffff83987523 R09: ffffffff811c7eca R10: ffff8880a4e94200 R11: 0000000000000002 R12: dffffc0000000000 R13: fffffe0000000ec8 R14: ffffffff880016f0 R15: fffffe0000000ecb FS: 00007fb50e370700(0000) GS:ffff8880ae800000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 000000000000005c CR3: 0000000092fc7000 CR4: 00000000001426f0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400That would suggest a miscompilation of the inline assembly, which does push the flags:
#ifdef CONFIG_X86_64 "mov %%" _ASM_SP ", %[sp]\n\t" "and $0xfffffffffffffff0, %%" _ASM_SP "\n\t" "push $%c[ss]\n\t" "push %[sp]\n\t" #endif "pushf\n\t" __ASM_SIZE(push) " $%c[cs]\n\t" CALL_NOSPECQuoted 6 lines
Quoted 4 lines
It would not explain why it suddenly started to break, unless the clang version also changed, but it would be easy to ascertain and fix (in either KVM or clang). Dmitry, can you send me the vmx.o and kvm-intel.ko files?
On a quick glance, Clang does not miscompile this part.
Clang definitely miscompiles the asm, the indirect call operates on the EFLAGS value, not on @entry as expected. It looks like clang doesn't honor
ASM_CALL_CONSTRAINT, which effectively tells the compiler that %rsp is getting clobbered, e.g. the "mov %r14,0x8(%rsp)" is loading @entry for "callq *0x8(%rsp)", which breaks because of asm's pushes.clang:
kvm_before_interrupt(vcpu); asm volatile( ffffffff811b798e: 4c 89 74 24 08 mov %r14,0x8(%rsp) ffffffff811b7993: 48 89 e0 mov %rsp,%rax ffffffff811b7996: 48 83 e4 f0 and $0xfffffffffffffff0,%rsp ffffffff811b799a: 6a 18 pushq $0x18 ffffffff811b799c: 50 push %rax ffffffff811b799d: 9c pushfq ffffffff811b799e: 6a 10 pushq $0x10 ffffffff811b79a0: ff 54 24 08 callq *0x8(%rsp) <--------- calls the EFLAGS value
kvm_after_interrupt():gcc:
kvm_before_interrupt(vcpu); asm volatile( ffffffff8118e17c: 48 89 e0 mov %rsp,%rax ffffffff8118e17f: 48 83 e4 f0 and $0xfffffffffffffff0,%rsp ffffffff8118e183: 6a 18 pushq $0x18 ffffffff8118e185: 50 push %rax ffffffff8118e186: 9c pushfq ffffffff8118e187: 6a 10 pushq $0x10 ffffffff8118e189: ff d3 callq *%rbx <-------- calls @entry
Quoted 1 line
kvm_after_interrupt():
Thanks for this analysis, it looks like this is dependent on some particular configuration; here's clang+defconfig+CONFIG_KVM_INTEL=y:
0x000000000000528f <+127>: pushq $0x18 0x0000000000005291 <+129>: push %rcx 0x0000000000005292 <+130>: pushfq 0x0000000000005293 <+131>: pushq $0x10 0x0000000000005295 <+133>: callq *%rax
Signature · 2 lines
Thanks, ~Nick Desaulniers
Quoted 51 lines · Nick Desaulniers
Quoted 48 lines · Sean Christopherson
Quoted 46 lines · Alexander Potapenko
Quoted 44 lines · Paolo Bonzini
Quoted 6 lines · Dmitry Vyukov
The commit range is presumably
fb279f4e238617417b132a550f24c1e86d922558..63849c8f410717eb2e6662f3953ff674727303e7But I don't see anything that says "it's me". The only commit that does non-trivial changes to x86/vmx seems to be "KVM: VMX: check descriptor table exits on instruction emulation":That seems unlikely, it's a completely different file and it would only affect the outside (non-nested) environment rather than your own kernel.
The only instance of "
0x86" in the registers is in the flags:Quoted 11 lines
RSP: 0018:ffffc90001ac7998 EFLAGS: 00010086 RAX: ffffc90001ac79c8 RBX: fffffe0000000000 RCX: 0000000000040000 RDX: ffffc9000e20f000 RSI: 000000000000b452 RDI: 000000000000b453 RBP: 0000000000000ec0 R08: ffffffff83987523 R09: ffffffff811c7eca R10: ffff8880a4e94200 R11: 0000000000000002 R12: dffffc0000000000 R13: fffffe0000000ec8 R14: ffffffff880016f0 R15: fffffe0000000ecb FS: 00007fb50e370700(0000) GS:ffff8880ae800000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 000000000000005c CR3: 0000000092fc7000 CR4: 00000000001426f0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400That would suggest a miscompilation of the inline assembly, which does push the flags:
#ifdef CONFIG_X86_64 "mov %%" _ASM_SP ", %[sp]\n\t" "and $0xfffffffffffffff0, %%" _ASM_SP "\n\t" "push $%c[ss]\n\t" "push %[sp]\n\t" #endif "pushf\n\t" __ASM_SIZE(push) " $%c[cs]\n\t" CALL_NOSPECIt would not explain why it suddenly started to break, unless the clang version also changed, but it would be easy to ascertain and fix (in either KVM or clang). Dmitry, can you send me the vmx.o and kvm-intel.ko files?
Quoted 5 lines
Quoted 5 lines
Quoted 1 line
On a quick glance, Clang does not miscompile this part.
Clang definitely miscompiles the asm, the indirect call operates on the EFLAGS value, not on @entry as expected. It looks like clang doesn't honor
ASM_CALL_CONSTRAINT, which effectively tells the compiler that %rsp is
I noticed that in the syzcaller config I have, that CONFIG_RETPOLINE is not set. I'm more reliably able to reproduce this with clang+defconfig+CONFIG_KVM=y+CONFIG_KVM_INTEL=y+CONFIG_RETPOLINE=n, ie. by manually disabling retpoline.
Quoted 44 lines
Quoted 30 lines
getting clobbered, e.g. the "mov %r14,0x8(%rsp)" is loading @entry for "callq *0x8(%rsp)", which breaks because of asm's pushes.
clang:
kvm_before_interrupt(vcpu); asm volatile( ffffffff811b798e: 4c 89 74 24 08 mov %r14,0x8(%rsp) ffffffff811b7993: 48 89 e0 mov %rsp,%rax ffffffff811b7996: 48 83 e4 f0 and $0xfffffffffffffff0,%rsp ffffffff811b799a: 6a 18 pushq $0x18 ffffffff811b799c: 50 push %rax ffffffff811b799d: 9c pushfq ffffffff811b799e: 6a 10 pushq $0x10 ffffffff811b79a0: ff 54 24 08 callq *0x8(%rsp) <--------- calls the EFLAGS value
kvm_after_interrupt():gcc:
kvm_before_interrupt(vcpu); asm volatile( ffffffff8118e17c: 48 89 e0 mov %rsp,%rax ffffffff8118e17f: 48 83 e4 f0 and $0xfffffffffffffff0,%rsp ffffffff8118e183: 6a 18 pushq $0x18 ffffffff8118e185: 50 push %rax ffffffff8118e186: 9c pushfq ffffffff8118e187: 6a 10 pushq $0x10 ffffffff8118e189: ff d3 callq *%rbx <-------- calls @entryQuoted 1 line
kvm_after_interrupt():Thanks for this analysis, it looks like this is dependent on some particular configuration; here's clang+defconfig+CONFIG_KVM_INTEL=y:
0x000000000000528f <+127>: pushq $0x18 0x0000000000005291 <+129>: push %rcx 0x0000000000005292 <+130>: pushfq 0x0000000000005293 <+131>: pushq $0x10 0x0000000000005295 <+133>: callq *%rax--
Thanks,
~Nick Desaulniers
Signature · 2 lines
Thanks, ~Nick Desaulniers
I've reduced the faulty test case to the following code:
=================================
a;
long b;
register unsigned long current_stack_pointer asm("rsp"); handle_external_interrupt_irqoff() {
asm("and $0xfffffffffffffff0, %%rsp\n\tpush $%c[ss]\n\tpush "
"%[sp]\n\tpushf\n\tpushq $%c[cs]\n\tcall *%[thunk_target]\n"
: [ sp ] "=&r"(b), "+r" (current_stack_pointer)
: [ thunk_target ] "rm"(a), [ ss ] "i"(3 * 8), [ cs ] "i"(2 * 8) );
}=================================
(in fact creduce even throws away current_stack_pointer, but we probably want to keep it to prove the point).
Clang generates the following code for it:
$ clang vmx.i -O2 -c -w -o vmx.o $ objdump -d vmx.o ... 0000000000000000 <handle_external_interrupt_irqoff>: 0: 8b 05 00 00 00 00 mov 0x0(%rip),%eax # 6 <handle_external_interrupt_irqoff+0x6> 6: 89 44 24 fc mov %eax,-0x4(%rsp) a: 48 83 e4 f0 and $0xfffffffffffffff0,%rsp e: 6a 18 pushq $0x18 10: 50 push %rax 11: 9c pushfq 12: 6a 10 pushq $0x10 14: ff 54 24 fc callq *-0x4(%rsp) 18: 48 89 05 00 00 00 00 mov %rax,0x0(%rip) # 1f <handle_external_interrupt_irqoff+0x1f> 1f: c3 retq
The question is whether using current_stack_pointer as an output is actually a valid way to tell the compiler it should not clobber RSP. Intuitively it is, but explicitly adding RSP to the clobber list sounds a bit more bulletproof.
Quoted 36 lines · Alexander Potapenko
I've reduced the faulty test case to the following code:
=================================
a;
long b;
register unsigned longcurrent_stack_pointerasm("rsp");handle_external_interrupt_irqoff(){
asm("and $0xfffffffffffffff0, %%rsp\n\tpush $%c[ss]\n\tpush ""%[sp]\n\tpushf\n\tpushq $%c[cs]\n\tcall *%[thunk_target]\n" : [ sp ] "=&r"(b), "+r" (current_stack_pointer) : [ thunk_target ] "rm"(a), [ ss ] "i"(3 * 8), [ cs ] "i"(2 * 8) ); }=================================
(in fact creduce even throws awaycurrent_stack_pointer, but we probably want to keep it to prove the point).Clang generates the following code for it:
$ clang vmx.i -O2 -c -w -o vmx.o $ objdump -d vmx.o ... 0000000000000000 <handle_external_interrupt_irqoff>: 0: 8b 05 00 00 00 00 mov 0x0(%rip),%eax # 6 <handle_external_interrupt_irqoff+0x6> 6: 89 44 24 fc mov %eax,-0x4(%rsp) a: 48 83 e4 f0 and $0xfffffffffffffff0,%rsp e: 6a 18 pushq $0x18 10: 50 push %rax 11: 9c pushfq 12: 6a 10 pushq $0x10 14: ff 54 24 fc callq *-0x4(%rsp) 18: 48 89 05 00 00 00 00 mov %rax,0x0(%rip) # 1f <handle_external_interrupt_irqoff+0x1f> 1f: c3 retq
Quoted 4 lines
The question is whether using
current_stack_pointeras an output is actually a valid way to tell the compiler it should not clobber RSP. Intuitively it is, but explicitly adding RSP to the clobber list sounds a bit more bulletproof.
Ok, I think this reproducer demonstrates the issue:
https://godbolt.org/z/jAafjz
I *think* what's happening is that we're not specifying correctly that the stack is being modified by inline asm, so using variable references against the stack pointer is not correct.
commit f5caf621ee357 ("x86/asm: Fix inline asm call constraints for Clang") has more context about ASM_CALL_CONSTRAINT.
It seems that specifying "rsp" in the clobber list is a -Wdeprecated warning in GCC, and an error in Clang (unless you remove current_stack_pointer as an output, but will get Clang to produce the correct code).
Signature · 2 lines
Thanks, ~Nick Desaulniers
Quoted 36 lines · Alexander Potapenko
I've reduced the faulty test case to the following code:
=================================
a;
long b;
register unsigned longcurrent_stack_pointerasm("rsp");handle_external_interrupt_irqoff(){
asm("and $0xfffffffffffffff0, %%rsp\n\tpush $%c[ss]\n\tpush ""%[sp]\n\tpushf\n\tpushq $%c[cs]\n\tcall *%[thunk_target]\n" : [ sp ] "=&r"(b), "+r" (current_stack_pointer) : [ thunk_target ] "rm"(a), [ ss ] "i"(3 * 8), [ cs ] "i"(2 * 8) ); }=================================
(in fact creduce even throws awaycurrent_stack_pointer, but we probably want to keep it to prove the point).Clang generates the following code for it:
$ clang vmx.i -O2 -c -w -o vmx.o $ objdump -d vmx.o ... 0000000000000000 <handle_external_interrupt_irqoff>: 0: 8b 05 00 00 00 00 mov 0x0(%rip),%eax # 6 <handle_external_interrupt_irqoff+0x6> 6: 89 44 24 fc mov %eax,-0x4(%rsp) a: 48 83 e4 f0 and $0xfffffffffffffff0,%rsp e: 6a 18 pushq $0x18 10: 50 push %rax 11: 9c pushfq 12: 6a 10 pushq $0x10 14: ff 54 24 fc callq *-0x4(%rsp) 18: 48 89 05 00 00 00 00 mov %rax,0x0(%rip) # 1f <handle_external_interrupt_irqoff+0x1f> 1f: c3 retq
Quoted 4 lines
The question is whether using
current_stack_pointeras an output is actually a valid way to tell the compiler it should not clobber RSP. Intuitively it is, but explicitly adding RSP to the clobber list sounds a bit more bulletproof.
Ok, I am wrong: according to
https://gcc.gnu.org/onlinedocs/gcc/Extended-Asm.html it's incorrect to list RSP in the clobber list.
Quoted 38 lines · Alexander Potapenko
Quoted 36 lines · Alexander Potapenko
I've reduced the faulty test case to the following code:
=================================
a;
long b;
register unsigned longcurrent_stack_pointerasm("rsp");handle_external_interrupt_irqoff(){
asm("and $0xfffffffffffffff0, %%rsp\n\tpush $%c[ss]\n\tpush ""%[sp]\n\tpushf\n\tpushq $%c[cs]\n\tcall *%[thunk_target]\n" : [ sp ] "=&r"(b), "+r" (current_stack_pointer) : [ thunk_target ] "rm"(a), [ ss ] "i"(3 * 8), [ cs ] "i"(2 * 8) ); }=================================
(in fact creduce even throws awaycurrent_stack_pointer, but we probably want to keep it to prove the point).Clang generates the following code for it:
$ clang vmx.i -O2 -c -w -o vmx.o $ objdump -d vmx.o ... 0000000000000000 <handle_external_interrupt_irqoff>: 0: 8b 05 00 00 00 00 mov 0x0(%rip),%eax # 6 <handle_external_interrupt_irqoff+0x6> 6: 89 44 24 fc mov %eax,-0x4(%rsp) a: 48 83 e4 f0 and $0xfffffffffffffff0,%rsp e: 6a 18 pushq $0x18 10: 50 push %rax 11: 9c pushfq 12: 6a 10 pushq $0x10 14: ff 54 24 fc callq *-0x4(%rsp) 18: 48 89 05 00 00 00 00 mov %rax,0x0(%rip) # 1f <handle_external_interrupt_irqoff+0x1f> 1f: c3 retq
Quoted 8 lines
Quoted 4 lines
The question is whether using
current_stack_pointeras an output is actually a valid way to tell the compiler it should not clobber RSP. Intuitively it is, but explicitly adding RSP to the clobber list sounds a bit more bulletproof.Ok, I am wrong: according to
https://gcc.gnu.org/onlinedocs/gcc/Extended-Asm.html it's incorrect to list RSP in the clobber list.
You could force entry into a register:
arch/x86/kvm/vmx/vmx.c +0 -0
@@ -6277,7 +6277,7 @@ static void
handle_external_interrupt_irqoff(struct kvm_vcpu *vcpu)
#endif
ASM_CALL_CONSTRAINT
:
- THUNK_TARGET(entry),
+ [thunk_target] "a"(entry),
[ss]"i"(__KERNEL_DS),
[cs]"i"(__KERNEL_CS)
);(https://stackoverflow.com/a/48877683/1027966 had some interesting feedback to this problem)
Signature · 2 lines
Thanks, ~Nick Desaulniers
Quoted 67 lines · Nick Desaulniers
Quoted 38 lines · Alexander Potapenko
Quoted 36 lines · Alexander Potapenko
I've reduced the faulty test case to the following code:
=================================
a;
long b;
register unsigned longcurrent_stack_pointerasm("rsp");handle_external_interrupt_irqoff(){
asm("and $0xfffffffffffffff0, %%rsp\n\tpush $%c[ss]\n\tpush ""%[sp]\n\tpushf\n\tpushq $%c[cs]\n\tcall *%[thunk_target]\n" : [ sp ] "=&r"(b), "+r" (current_stack_pointer) : [ thunk_target ] "rm"(a), [ ss ] "i"(3 * 8), [ cs ] "i"(2 * 8) ); }=================================
(in fact creduce even throws awaycurrent_stack_pointer, but we probably want to keep it to prove the point).Clang generates the following code for it:
$ clang vmx.i -O2 -c -w -o vmx.o $ objdump -d vmx.o ... 0000000000000000 <handle_external_interrupt_irqoff>: 0: 8b 05 00 00 00 00 mov 0x0(%rip),%eax # 6 <handle_external_interrupt_irqoff+0x6> 6: 89 44 24 fc mov %eax,-0x4(%rsp) a: 48 83 e4 f0 and $0xfffffffffffffff0,%rsp e: 6a 18 pushq $0x18 10: 50 push %rax 11: 9c pushfq 12: 6a 10 pushq $0x10 14: ff 54 24 fc callq *-0x4(%rsp) 18: 48 89 05 00 00 00 00 mov %rax,0x0(%rip) # 1f <handle_external_interrupt_irqoff+0x1f> 1f: c3 retqQuoted 8 lines
Quoted 4 lines
The question is whether using
current_stack_pointeras an output is actually a valid way to tell the compiler it should not clobber RSP. Intuitively it is, but explicitly adding RSP to the clobber list sounds a bit more bulletproof.Ok, I am wrong: according to
https://gcc.gnu.org/onlinedocs/gcc/Extended-Asm.html it's incorrect to list RSP in the clobber list.You could force
entryinto a register:arch/x86/kvm/vmx/vmx.c +1
-1@@ -6277,7 +6277,7 @@ static void handle_external_interrupt_irqoff(struct kvm_vcpu *vcpu)#endif ASM_CALL_CONSTRAINT : - THUNK_TARGET(entry), + [thunk_target] "a"(entry), [ss]"i"(__KERNEL_DS), [cs]"i"(__KERNEL_CS));(https://stackoverflow.com/a/48877683/1027966 had some interesting feedback to this problem)
Quoted 4 lines · Sean
It looks like clang doesn't honor
ASM_CALL_CONSTRAINT, which effectively tells the compiler that %rsp is getting clobbered, e.g. the "mov %r14,0x8(%rsp)" is loading @entry for "callq *0x8(%rsp)", which breaks because of asm's pushes.
I'm not sure about this, I think ASM_CALL_CONSTRAINT may be a red herring, based on the commit message that added it (commit f5caf621ee357 ("x86/asm: Fix inline asm call constraints for Clang")).
Further, it seems the "m" in "rm" in THUNK_TARGET for CONFIG_RETPOLINE=n is problematic.
THUNK_TARGET defines [thunk_target] as "rm" when CONFIG_RETPOLINE is not set, which isn't constrained enough for this specific case; if entry winds up at the bottom of the stack where rsp points to, then %rsp is good enough to satisfy the constraints for using entry as an input. For inline assembly that modifies the the stack pointer before using this input, the underspecification of constraints is dangerous, and results in an indirect call to a previously pushed flags register.
So maybe we can find why
commit 76b043848fd2 ("x86/retpoline: Add initial retpoline support") added THUNK_TARGET with and without "m" constraint, and either:
- remove "m" from THUNK_TARGET. (Maybe this doesn't compile somewhere) or
- use my above recommendation locally avoiding THUNK_TARGET. We can use "r" rather than "a" (what Clang would have picked) or "b (what GCC would have picked) to give the compilers maximal flexibility.
Signature · 2 lines
Thanks, ~Nick Desaulniers
PATCH 1/1 KVM: VMX: don't allow memory operands for inline asm that modifies SP
THUNK_TARGET defines [thunk_target] as having "rm" input constraints when CONFIG_RETPOLINE is not set, which isn't constrained enough for this specific case.
For inline assembly that modifies the stack pointer before using this input, the underspecification of constraints is dangerous, and results in an indirect call to a previously pushed flags register.
In this case entry's stack slot is good enough to satisfy the "m" constraint in "rm", but the inline assembly in handle_external_interrupt_irqoff() modifies the stack pointer via push+pushf before using this input, which in this case results in calling what was the previous state of the flags register, rather than entry.
Be more specific in the constraints by requiring entry be in a register, and not a memory operand.
--- arch/x86/kvm/vmx/vmx.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
arch/x86/kvm/vmx/vmx.c +1 -1
@@ -6277,7 +6277,7 @@ static void handle_external_interrupt_irqoff(struct kvm_vcpu *vcpu)#endif ASM_CALL_CONSTRAINT : - THUNK_TARGET(entry), + [thunk_target]"r"(entry), [ss]"i"(__KERNEL_DS), [cs]"i"(__KERNEL_CS) );
Signature · 1 line
2.25.1.696.g5e7596f4ac-goog
Quoted 9 lines · Nick Desaulniers
So maybe we can find why
commit76b043848fd2("x86/retpoline: Add initial retpoline support") addedTHUNK_TARGETwith and without "m" constraint, and either:
- remove "m" fromTHUNK_TARGET. (Maybe this doesn't compile somewhere) or
- use my above recommendation locally avoidingTHUNK_TARGET. We can use "r" rather than "a" (what Clang would have picked) or "b (what GCC would have picked) to give the compilers maximal flexibility.
So I've sent a patch for the latter; my reason for not pursuing the former is:
1. I assume that the thunk target could be spilled, or a pointer, and we'd like to keep flexibility for the general case of inline asm that doesn't modify the stack pointer.
2. entry is local to handle_external_interrupt_irqoff; it's not being passed in via pointer as a function parameter.
3. register pressure is irrelevant if the resulting code is incorrect.
Signature · 2 lines
Thanks, ~Nick Desaulniers
Paolo Bonzini
maintainerQuoted 9 lines · Nick Desaulniers
Quoted 8 lines · <[email protected]> wrote:
So maybe we can find why
commit76b043848fd2("x86/retpoline: Add initial retpoline support") addedTHUNK_TARGETwith and without "m" constraint, and either:
- remove "m" fromTHUNK_TARGET. (Maybe this doesn't compile somewhere) or
- use my above recommendation locally avoidingTHUNK_TARGET. We can use "r" rather than "a" (what Clang would have picked) or "b (what GCC would have picked) to give the compilers maximal flexibility.
Quoted 7 lines
So I've sent a patch for the latter; my reason for not pursuing the former is:
1. I assume that the thunk target could be spilled, or a pointer, and we'd like to keep flexibility for the general case of inline asm that doesn't modify the stack pointer.
2.entryis local tohandle_external_interrupt_irqoff; it's not being passed in via pointer as a function parameter.
3. register pressure is irrelevant if the resulting code is incorrect.
Yes, this is fair enough. I've queued your patch and will send it shortly to Linus.
Paolo
syzbot
syzbotsyzbot has found a reproducer for the following crash on:
syzbot has found a reproducer for the following crash on:
HEAD commit: b74b991f Merge tag 'block-5.6-20200320' of git://git.kerne.. git tree: upstream console output: https://syzkaller.appspot.com/x/log.txt?x=13059373e00000 kernel config: https://syzkaller.appspot.com/x/.config?x=6dfa02302d6db985 dashboard link: https://syzkaller.appspot.com/bug?extid=3f29ca2efb056a761e38 compiler: clang version 10.0.0 (https://github.com/llvm/llvm-project/ c2443155a0fb245c8f17f2c1c72b6ea391e86e81) syz repro: https://syzkaller.appspot.com/x/repro.syz?x=1199c0c5e00000 C reproducer: https://syzkaller.appspot.com/x/repro.c?x=15097373e00000
IMPORTANT: if you fix the bug, please add the following tag to the commit:
BUG: kernel NULL pointer dereference, address: 0000000000000086 #PF: supervisor instruction fetch in kernel mode #PF: error_code(0x0010) - not-present page PGD 9330b067 P4D 9330b067 PUD 9e66f067 PMD 0 Oops: 0010 [#1] PREEMPT SMP KASAN CPU: 1 PID: 8439 Comm: syz-executor724 Not tainted 5.6.0-rc6-syzkaller #0 Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011 RIP: 0010:0x86 Code: Bad RIP value. RSP: 0018:ffffc900022e7998 EFLAGS: 00010086 RAX: ffffc900022e79c8 RBX: fffffe0000000000 RCX: ffff88809dcf2500 RDX: 0000000000000000 RSI: 0000000000000001 RDI: 0000000000000000 RBP: 0000000000000ec0 R08: ffffffff83987523 R09: ffffffff811c7eca R10: ffff88809dcf2500 R11: 0000000000000002 R12: dffffc0000000000 R13: fffffe0000000ec8 R14: ffffffff880016f0 R15: fffffe0000000ecb FS: 0000000001d0d880(0000) GS:ffff8880ae900000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 000000000000005c CR3: 00000000978c3000 CR4: 00000000001426e0 Call Trace: handle_external_interrupt_irqoff+0x154/0x280 arch/x86/kvm/vmx/vmx.c:6274 kvm_before_interrupt arch/x86/kvm/x86.h:343 [inline] handle_external_interrupt_irqoff+0x132/0x280 arch/x86/kvm/vmx/vmx.c:6272 __irqentry_text_start+0x8/0x8 vcpu_enter_guest+0x6c77/0x9290 arch/x86/kvm/x86.c:8405 save_stack mm/kasan/common.c:72 [inline] set_track mm/kasan/common.c:80 [inline] kasan_set_free_info mm/kasan/common.c:337 [inline] __kasan_slab_free+0x12e/0x1e0 mm/kasan/common.c:476 __cache_free mm/slab.c:3426 [inline] kfree+0x10a/0x220 mm/slab.c:3757 tomoyo_path_number_perm+0x525/0x690 security/tomoyo/file.c:736 security_file_ioctl+0x55/0xb0 security/security.c:1441 entry_SYSCALL_64_after_hwframe+0x49/0xbe __lock_acquire+0xc5a/0x1bc0 kernel/locking/lockdep.c:3954 paravirt_write_msr arch/x86/include/asm/paravirt.h:167 [inline] wrmsrl arch/x86/include/asm/paravirt.h:200 [inline] native_x2apic_icr_write arch/x86/include/asm/apic.h:249 [inline] __x2apic_send_IPI_dest arch/x86/kernel/apic/x2apic_phys.c:112 [inline] x2apic_send_IPI+0x96/0xc0 arch/x86/kernel/apic/x2apic_phys.c:41 test_bit include/asm-generic/bitops/instrumented-non-atomic.h:110 [inline] hlock_class kernel/locking/lockdep.c:163 [inline] mark_lock+0x107/0x1650 kernel/locking/lockdep.c:3642 lock_acquire+0x154/0x250 kernel/locking/lockdep.c:4484 rcu_lock_acquire+0x9/0x30 include/linux/rcupdate.h:208 vcpu_run+0x3a3/0xd50 arch/x86/kvm/x86.c:8513 kvm_arch_vcpu_ioctl_run+0x419/0x880 arch/x86/kvm/x86.c:8735 kvm_vcpu_ioctl+0x67c/0xa80 arch/x86/kvm/../../../virt/kvm/kvm_main.c:2932 lock_is_held include/linux/lockdep.h:361 [inline] rcu_read_lock_sched_held+0x106/0x170 kernel/rcu/update.c:121 kvm_vm_release+0x50/0x50 arch/x86/kvm/../../../virt/kvm/kvm_main.c:858 vfs_ioctl fs/ioctl.c:47 [inline] ksys_ioctl fs/ioctl.c:763 [inline] __do_sys_ioctl fs/ioctl.c:772 [inline] __se_sys_ioctl+0xf9/0x160 fs/ioctl.c:770 do_syscall_64+0xf3/0x1b0 arch/x86/entry/common.c:294 entry_SYSCALL_64_after_hwframe+0x49/0xbe Modules linked in: CR2: 0000000000000086 ---[ end trace 480d6b60d5a9226d ]--- RIP: 0010:0x86 Code: Bad RIP value. RSP: 0018:ffffc900022e7998 EFLAGS: 00010086 RAX: ffffc900022e79c8 RBX: fffffe0000000000 RCX: ffff88809dcf2500 RDX: 0000000000000000 RSI: 0000000000000001 RDI: 0000000000000000 RBP: 0000000000000ec0 R08: ffffffff83987523 R09: ffffffff811c7eca R10: ffff88809dcf2500 R11: 0000000000000002 R12: dffffc0000000000 R13: fffffe0000000ec8 R14: ffffffff880016f0 R15: fffffe0000000ecb FS: 0000000001d0d880(0000) GS:ffff8880ae900000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 000000000000005c CR3: 00000000978c3000 CR4: 00000000001426e0
syzbot
syzbotsyzbot has bisected this bug to:
syzbot has bisected this bug to:
commit f71dbf2fb28489a79bde0dca1c8adfb9cdb20a6b
Author: David Howells <[email protected]> Date: Thu Jan 30 21:50:36 2020 +0000
rxrpc: Fix insufficient receive notification generation
bisection log: https://syzkaller.appspot.com/x/bisect.txt?x=1483bb19e00000 start commit: b74b991f Merge tag 'block-5.6-20200320' of git://git.kerne.. git tree: upstream final crash: https://syzkaller.appspot.com/x/report.txt?x=1683bb19e00000 console output: https://syzkaller.appspot.com/x/log.txt?x=1283bb19e00000 kernel config: https://syzkaller.appspot.com/x/.config?x=6dfa02302d6db985 dashboard link: https://syzkaller.appspot.com/bug?extid=3f29ca2efb056a761e38 syz repro: https://syzkaller.appspot.com/x/repro.syz?x=1199c0c5e00000 C reproducer: https://syzkaller.appspot.com/x/repro.c?x=15097373e00000
For information about bisection process see: https://goo.gl/tpsmEJ#bisection
Quoted 8 lines · syzbot
syzbot has bisected this bug to:
commit
f71dbf2fb28489a79bde0dca1c8adfb9cdb20a6bAuthor: David Howells <[email protected]> Date: Thu Jan 30 21:50:36 2020 +0000rxrpc: Fix insufficient receive notification generation
This is unrelated.
Somehow the crash wasn't reproduced again on the same commit. Can it depend on host CPU type maybe?
Quoted 14 lines
bisection log: https://syzkaller.appspot.com/x/bisect.txt?x=1483bb19e00000 start commit: b74b991f Merge tag 'block-5.6-20200320' of git://git.kerne.. git tree: upstream final crash: https://syzkaller.appspot.com/x/report.txt?x=1683bb19e00000 console output: https://syzkaller.appspot.com/x/log.txt?x=1283bb19e00000 kernel config: https://syzkaller.appspot.com/x/.config?x=6dfa02302d6db985 dashboard link: https://syzkaller.appspot.com/bug?extid=3f29ca2efb056a761e38 syz repro: https://syzkaller.appspot.com/x/repro.syz?x=1199c0c5e00000 C reproducer: https://syzkaller.appspot.com/x/repro.c?x=15097373e00000Reported-by [email protected]Fixes f71dbf2fb284 ("rxrpc: Fix insufficient receive notification generation")For information about bisection process see: https://goo.gl/tpsmEJ#bisection