3.10.X kernel/jump_label kvm

DISCUSSION Stefan Priebe 4 participants 9 messages Feb 28, 2014 – Mar 4, 2014

Subsystems (3)

x86 KVM KVM x86

lore

Share & tools

lore

Andrey Korolyov

2014-03-01 14:28 UTC #1 lore raw

Quoted 5 lines · Stefan Priebe

Hello,
i got this stack trace multiple times while using a vanilla 3.10.32 kernel and already sent it to the list in december but got no replies.

Hi,

What kind of workload the host system is experiencing at same time? Does this event correlate with high memory pressure?

Quoted 57 lines

[78136.551061] WARNING: at kernel/jump_label.c:80
__static_key_slow_dec+0xb6/0xc0()
[78136.551062] jump label: negative count!
[78136.551063] Modules linked in: sch_htb act_police cls_u32 sch_ingress

vhost_net tun macvtap macvlan netconsole ipt_REJECT dlm sctp iptable_filter ip_tables x_tables iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi nfsd auth_rpcgss oid_registry bonding ext2 8021q garp fuse mperf coretemp kvm_intel kvm crc32_pclmul ghash_clmulni_intel microcode i2c_i801 button dm_mod raid1 md_mod usbhid usb_storage ohci_hcd sg sd_mod ehci_pci ahci ehci_hcd igb libahci i2c_algo_bit isci usbcore i2c_core usb_common libsas ptp ixgbe(O) scsi_transport_sas pps_core

[78136.551080] CPU: 21 PID: 47183 Comm: kvm Tainted: G        W  O
3.10.32+68-ph #1
[78136.551081] Hardware name: Supermicro
X9DRW-3LN4F+/X9DRW-3TF+/X9DRW-3LN4F+/X9DRW-3TF+, BIOS 3.00 07/05/2013
[78136.551081]  0000000000000009 ffff882f4a669be8 ffffffff81524606
ffff882f4a669c28
[78136.551085]  ffffffff8104853b 000000004a669c08 ffffffffa045cc40
00000000000000fa
[78136.551088]  ffffffffa045cc60 ffff882f51460160 ffff882f74ab8110
ffff882f4a669c88
[78136.551091] Call Trace:
[78136.551093]  [<ffffffff81524606>] dump_stack+0x19/0x1b
[78136.551095]  [<ffffffff8104853b>] warn_slowpath_common+0x6b/0xa0
[78136.551098]  [<ffffffff81048611>] warn_slowpath_fmt+0x41/0x50
[78136.551100]  [<ffffffff810e05f6>] __static_key_slow_dec+0xb6/0xc0
[78136.551102]  [<ffffffff810e0631>] static_key_slow_dec_deferred+0x11/0x20
[78136.551110]  [<ffffffffa043ff60>] kvm_free_lapic+0x90/0xa0 [kvm]
[78136.551116]  [<ffffffffa0429ef3>] kvm_arch_vcpu_uninit+0x23/0x90 [kvm]
[78136.551122]  [<ffffffffa0410a20>] kvm_vcpu_uninit+0x20/0x40 [kvm]
[78136.551125]  [<ffffffffa021fc12>] vmx_free_vcpu+0x52/0x70 [kvm_intel]
[78136.551132]  [<ffffffffa04295ef>] kvm_arch_vcpu_free+0x4f/0x60 [kvm]
[78136.551138]  [<ffffffffa042a112>] kvm_arch_destroy_vm+0xf2/0x1f0 [kvm]
[78136.551141]  [<ffffffff81071048>] ? synchronize_srcu+0x18/0x20
[78136.551143]  [<ffffffff8112677a>] ? mmu_notifier_unregister+0xaa/0xe0
[78136.551149]  [<ffffffffa041380e>] kvm_put_kvm+0x10e/0x1b0 [kvm]
[78136.551155]  [<ffffffffa0413a33>] kvm_vcpu_release+0x13/0x20 [kvm]
[78136.551157]  [<ffffffff811452d1>] __fput+0xe1/0x230
[78136.551160]  [<ffffffff81145429>] ____fput+0x9/0x10
[78136.551162]  [<ffffffff81068de5>] task_work_run+0xb5/0xd0
[78136.551164]  [<ffffffff8104da1c>] do_exit+0x2ac/0xa30
[78136.551166]  [<ffffffff8107a89b>] ? wake_up_state+0xb/0x10
[78136.551169]  [<ffffffff81059fad>] ? signal_wake_up_state+0x1d/0x30
[78136.551171]  [<ffffffff8105b1c3>] ? zap_other_threads+0x83/0xa0
[78136.551173]  [<ffffffff8104e21a>] do_group_exit+0x3a/0xa0
[78136.551175]  [<ffffffff8104e292>] SyS_exit_group+0x12/0x20
[78136.551177]  [<ffffffff81529fd2>] system_call_fastpath+0x16/0x1b
[78136.551178] ---[ end trace b9ebb6de9753ef4c ]---

Thanks!

Stefan

Signature · 3 lines

To unsubscribe from this list: send the line "unsubscribe kvm" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Stefan Priebe

author

2014-03-01 18:44 UTC #2 lore raw

Am 01.03.2014 15:27, schrieb Andrey Korolyov:

Quoted 11 lines · Stefan Priebe

Quoted 5 lines
Hello,
i got this stack trace multiple times while using a vanilla 3.10.32 kernel and already sent it to the list in december but got no replies.
Hi,
What kind of workload the host system is experiencing at same time? Does this event correlate with high memory pressure?

Running around 40 virtual machines.

It had 240GB of 378GB free so i think no memory pressure.

Stefan

Quoted 58 lines

Quoted 57 lines

[78136.551061] WARNING: at kernel/jump_label.c:80
__static_key_slow_dec+0xb6/0xc0()
[78136.551062] jump label: negative count!
[78136.551063] Modules linked in: sch_htb act_police cls_u32 sch_ingress

[78136.551080] CPU: 21 PID: 47183 Comm: kvm Tainted: G        W  O
3.10.32+68-ph #1
[78136.551081] Hardware name: Supermicro
X9DRW-3LN4F+/X9DRW-3TF+/X9DRW-3LN4F+/X9DRW-3TF+, BIOS 3.00 07/05/2013
[78136.551081]  0000000000000009 ffff882f4a669be8 ffffffff81524606
ffff882f4a669c28
[78136.551085]  ffffffff8104853b 000000004a669c08 ffffffffa045cc40
00000000000000fa
[78136.551088]  ffffffffa045cc60 ffff882f51460160 ffff882f74ab8110
ffff882f4a669c88
[78136.551091] Call Trace:
[78136.551093]  [<ffffffff81524606>] dump_stack+0x19/0x1b
[78136.551095]  [<ffffffff8104853b>] warn_slowpath_common+0x6b/0xa0
[78136.551098]  [<ffffffff81048611>] warn_slowpath_fmt+0x41/0x50
[78136.551100]  [<ffffffff810e05f6>] __static_key_slow_dec+0xb6/0xc0
[78136.551102]  [<ffffffff810e0631>] static_key_slow_dec_deferred+0x11/0x20
[78136.551110]  [<ffffffffa043ff60>] kvm_free_lapic+0x90/0xa0 [kvm]
[78136.551116]  [<ffffffffa0429ef3>] kvm_arch_vcpu_uninit+0x23/0x90 [kvm]
[78136.551122]  [<ffffffffa0410a20>] kvm_vcpu_uninit+0x20/0x40 [kvm]
[78136.551125]  [<ffffffffa021fc12>] vmx_free_vcpu+0x52/0x70 [kvm_intel]
[78136.551132]  [<ffffffffa04295ef>] kvm_arch_vcpu_free+0x4f/0x60 [kvm]
[78136.551138]  [<ffffffffa042a112>] kvm_arch_destroy_vm+0xf2/0x1f0 [kvm]
[78136.551141]  [<ffffffff81071048>] ? synchronize_srcu+0x18/0x20
[78136.551143]  [<ffffffff8112677a>] ? mmu_notifier_unregister+0xaa/0xe0
[78136.551149]  [<ffffffffa041380e>] kvm_put_kvm+0x10e/0x1b0 [kvm]
[78136.551155]  [<ffffffffa0413a33>] kvm_vcpu_release+0x13/0x20 [kvm]
[78136.551157]  [<ffffffff811452d1>] __fput+0xe1/0x230
[78136.551160]  [<ffffffff81145429>] ____fput+0x9/0x10
[78136.551162]  [<ffffffff81068de5>] task_work_run+0xb5/0xd0
[78136.551164]  [<ffffffff8104da1c>] do_exit+0x2ac/0xa30
[78136.551166]  [<ffffffff8107a89b>] ? wake_up_state+0xb/0x10
[78136.551169]  [<ffffffff81059fad>] ? signal_wake_up_state+0x1d/0x30
[78136.551171]  [<ffffffff8105b1c3>] ? zap_other_threads+0x83/0xa0
[78136.551173]  [<ffffffff8104e21a>] do_group_exit+0x3a/0xa0
[78136.551175]  [<ffffffff8104e292>] SyS_exit_group+0x12/0x20
[78136.551177]  [<ffffffff81529fd2>] system_call_fastpath+0x16/0x1b
[78136.551178] ---[ end trace b9ebb6de9753ef4c ]---

Thanks!

Stefan

--
To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

Paolo Bonzini

maintainer

2014-03-03 16:36 UTC #3 lore raw

In reply to Stefan Priebe

Il 28/02/2014 20:47, Stefan Priebe ha scritto:

Quoted 4 lines

Hello,
i got this stack trace multiple times while using a vanilla 3.10.32 kernel and already sent it to the list in december but got no replies.

Please try the patch of commit 0dce7cd67fd9055c4a2ff278f8af1431e646d346:

arch/x86/kvm/lapic.c +1 -1

index 1673940cf9c3..775702f649ca 100644
@@ -1355,7 +1355,7 @@ void kvm_lapic_set_base(struct kvm_vcpu *vcpu, u64 value)
        vcpu->arch.apic_base = value;
 
        /* update jump label if enable bit changes */
-       if ((vcpu->arch.apic_base ^ value) & MSR_IA32_APICBASE_ENABLE) {
+       if ((old_value ^ value) & MSR_IA32_APICBASE_ENABLE) {
                if (value & MSR_IA32_APICBASE_ENABLE)
                        static_key_slow_dec_deferred(&apic_hw_disabled);
                else

Paolo

Stefan Priebe

author

2014-03-03 18:17 UTC #4 lore raw

Am 03.03.2014 17:36, schrieb Paolo Bonzini:

Quoted 7 lines

Il 28/02/2014 20:47, Stefan Priebe ha scritto:
Quoted 4 lines
Hello,
i got this stack trace multiple times while using a vanilla 3.10.32 kernel and already sent it to the list in december but got no replies.
Please try the patch of commit 0dce7cd67fd9055c4a2ff278f8af1431e646d346:

Thanks - will try it. So this one should also go into 3.10.X

Stefan

Quoted 16 lines

arch/x86/kvm/lapic.c +1 -1

index 1673940cf9c3..775702f649ca 100644
@@ -1355,7 +1355,7 @@ void kvm_lapic_set_base(struct kvm_vcpu *vcpu, u64
 value)
         vcpu->arch.apic_base = value;
 
         /* update jump label if enable bit changes */
-       if ((vcpu->arch.apic_base ^ value) & MSR_IA32_APICBASE_ENABLE) {
+       if ((old_value ^ value) & MSR_IA32_APICBASE_ENABLE) {
                 if (value & MSR_IA32_APICBASE_ENABLE)
                         static_key_slow_dec_deferred(&apic_hw_disabled);

else

Paolo

Paolo Bonzini

maintainer

2014-03-03 18:20 UTC #5 lore raw

Il 03/03/2014 19:17, Stefan Priebe ha scritto:

Quoted 11 lines

Am 03.03.2014 17:36, schrieb Paolo Bonzini:
Quoted 7 lines
Il 28/02/2014 20:47, Stefan Priebe ha scritto:
Quoted 4 lines
Hello,
i got this stack trace multiple times while using a vanilla 3.10.32 kernel and already sent it to the list in december but got no replies.
Please try the patch of commit 0dce7cd67fd9055c4a2ff278f8af1431e646d346:
Thanks - will try it. So this one should also go into 3.10.X

Yes, I've sent it to [email protected] already.

Paolo

Jan Kiszka

2014-03-04 10:01 UTC #6 lore raw

In reply to Stefan Priebe

Quoted 11 lines · Stefan Priebe

Am 03.03.2014 17:36, schrieb Paolo Bonzini:
Quoted 7 lines
Il 28/02/2014 20:47, Stefan Priebe ha scritto:
Quoted 4 lines
Hello,
i got this stack trace multiple times while using a vanilla 3.10.32 kernel and already sent it to the list in december but got no replies.
Please try the patch of commit 0dce7cd67fd9055c4a2ff278f8af1431e646d346:
Thanks - will try it. So this one should also go into 3.10.X

It's part of 3.10.29. Only .26..28 were affected by the regression below.

Jan

Quoted 16 lines

arch/x86/kvm/lapic.c +1 -1

index 1673940cf9c3..775702f649ca 100644
@@ -1355,7 +1355,7 @@ void kvm_lapic_set_base(struct kvm_vcpu *vcpu, u64
 value)
         vcpu->arch.apic_base = value;
 
         /* update jump label if enable bit changes */
-       if ((vcpu->arch.apic_base ^ value) & MSR_IA32_APICBASE_ENABLE) {
+       if ((old_value ^ value) & MSR_IA32_APICBASE_ENABLE) {
                 if (value & MSR_IA32_APICBASE_ENABLE)
                         static_key_slow_dec_deferred(&apic_hw_disabled);

else

Paolo

Signature · 2 lines

Siemens AG, Corporate Technology, CT RTC ITP SES-DE
Corporate Competence Center Embedded Linux

Stefan Priebe - Profihost AG

author

2014-03-04 10:26 UTC #7 lore raw

Am 04.03.2014 11:01, schrieb Jan Kiszka:

Quoted 9 lines · Stefan Priebe

Quoted 8 lines
Am 03.03.2014 17:36, schrieb Paolo Bonzini:
Quoted 6 lines
Il 28/02/2014 20:47, Stefan Priebe ha scritto:
Quoted 4 lines
Hello,
i got this stack trace multiple times while using a vanilla 3.10.32 kernel and already sent it to the list in december but got no replies.

Quoted 7 lines

Quoted 3 lines
Quoted 1 line
Please try the patch of commit 0dce7cd67fd9055c4a2ff278f8af1431e646d346:
Thanks - will try it. So this one should also go into 3.10.X
It's part of 3.10.29. Only .26..28 were affected by the regression below.
Jan

HI Jan,

but i'm using 3.10.32. So it is another bug i'm seeing?

Stefan

Quoted 16 lines

arch/x86/kvm/lapic.c +1 -1

index 1673940cf9c3..775702f649ca 100644
@@ -1355,7 +1355,7 @@ void kvm_lapic_set_base(struct kvm_vcpu *vcpu, u64
 value)
         vcpu->arch.apic_base = value;
 
         /* update jump label if enable bit changes */
-       if ((vcpu->arch.apic_base ^ value) & MSR_IA32_APICBASE_ENABLE) {
+       if ((old_value ^ value) & MSR_IA32_APICBASE_ENABLE) {
                 if (value & MSR_IA32_APICBASE_ENABLE)
                         static_key_slow_dec_deferred(&apic_hw_disabled);

else

Paolo

Jan Kiszka

2014-03-04 11:21 UTC #8 lore raw

Quoted 14 lines · Stefan Priebe - Profihost AG

Am 04.03.2014 11:01, schrieb Jan Kiszka:
Quoted 13 lines · Stefan Priebe
Quoted 11 lines
Am 03.03.2014 17:36, schrieb Paolo Bonzini:
Quoted 7 lines
Il 28/02/2014 20:47, Stefan Priebe ha scritto:
Quoted 4 lines
Hello,
i got this stack trace multiple times while using a vanilla 3.10.32 kernel and already sent it to the list in december but got no replies.
Please try the patch of commit 0dce7cd67fd9055c4a2ff278f8af1431e646d346:
Thanks - will try it. So this one should also go into 3.10.X

Quoted 7 lines

Quoted 3 lines
It's part of 3.10.29. Only .26..28 were affected by the regression below.
Jan
HI Jan,
but i'm using 3.10.32. So it is another bug i'm seeing?

Yes, that's the logical consequence.

Jan

Signature · 2 lines

Siemens AG, Corporate Technology, CT RTC ITP SES-DE
Corporate Competence Center Embedded Linux