list corruption in the last few days. (block ? crypto ?)

Sometime in the last week, something was merged which causes my laptop to lock up occasionally. I can trigger it most of the time just by doing a kernel build. When it gets to the final linking stage, it locks up hard.

I finally managed to coax something out of usb console to get the traces below, which seem to implicate something in the block layer ?

my root device is an lvm volume on an dmcrypt'd block dev, which might be relevant, as I don't see this happening on other machines with simpler setups.

I'm going to try bisecting, but it might take me a few days, because it's such a pain in the ass to reproduce this reliably.

Dave

[ 5913.233035] ------------[ cut here ]------------
[ 5913.233097] WARNING: at lib/list_debug.c:53 __list_del_entry+0x8d/0x98()
[ 5913.233101] Hardware name: Adamo 13   
[ 5913.233105] list_del corruption. prev->next should be ffffea000434fd20, but was ffffea0004199520

Quoted 13 lines · Dave Jones

Sometime in the last week, something was merged which causes my laptop to lock up occasionally.  I can trigger it most of the time just by doing a kernel build. When it gets to the final linking stage, it locks up hard.

I finally managed to coax something out of usb console to get the traces below, which seem to implicate something in the block layer ?

my root device is an lvm volume on an dmcrypt'd block dev, which might be relevant, as I don't see this happening on other machines with simpler setups.

I'm going to try bisecting, but it might take me a few days, because it's such a pain in the ass to reproduce this reliably.

Quoted 4 lines

[ 5913.233035] ------------[ cut here ]------------
[ 5913.233097] WARNING: at lib/list_debug.c:53 __list_del_entry+0x8d/0x98()
[ 5913.233101] Hardware name: Adamo 13
[ 5913.233105] list_del corruption. prev->next should be ffffea000434fd20, but was ffffea0004199520

See also: http://lkml.org/lkml/2011/8/3/37

Quoted 19 lines · Markus Trippelsdorf

Quoted 17 lines · Dave Jones

Sometime in the last week, something was merged which causes my laptop to lock up occasionally. I can trigger it most of the time just by doing a kernel build. When it gets to the final linking stage, it locks up hard.

I finally managed to coax something out of usb console to get the traces below, which seem to implicate something in the block layer ?

my root device is an lvm volume on an dmcrypt'd block dev, which might be relevant, as I don't see this happening on other machines with simpler setups.

I'm going to try bisecting, but it might take me a few days, because it's such a pain in the ass to reproduce this reliably.

[ 5913.233035] ------------[ cut here ]------------
[ 5913.233097] WARNING: at lib/list_debug.c:53 __list_del_entry+0x8d/0x98()
[ 5913.233101] Hardware name: Adamo 13
[ 5913.233105] list_del corruption. prev->next should be ffffea000434fd20, but was ffffea0004199520

Quoted 1 line

See also: http://lkml.org/lkml/2011/8/3/37

That's in networking so SLUB lockless patches are almost certainly the issue here. Is this with SLUB debugging enabled or not? Christoph, it like the partial lists are getting corrupted somehow.

Quoted 19 lines · Markus Trippelsdorf

Quoted 17 lines · Dave Jones

Sometime in the last week, something was merged which causes my laptop to lock up occasionally.  I can trigger it most of the time just by doing a kernel build. When it gets to the final linking stage, it locks up hard.

I finally managed to coax something out of usb console to get the traces below, which seem to implicate something in the block layer ?

my root device is an lvm volume on an dmcrypt'd block dev, which might be relevant, as I don't see this happening on other machines with simpler setups.

I'm going to try bisecting, but it might take me a few days, because it's such a pain in the ass to reproduce this reliably.

[ 5913.233035] ------------[ cut here ]------------
[ 5913.233097] WARNING: at lib/list_debug.c:53 __list_del_entry+0x8d/0x98()
[ 5913.233101] Hardware name: Adamo 13
[ 5913.233105] list_del corruption. prev->next should be ffffea000434fd20, but was ffffea0004199520

Quoted 1 line

See also: http://lkml.org/lkml/2011/8/3/37

That's in networking so SLUB lockless patches are almost certainly the issue here. Is this with SLUB debugging enabled or not? Christoph, it like the partial lists are getting corrupted somehow.

Quoted 19 lines · Markus Trippelsdorf

Quoted 17 lines · Dave Jones

Sometime in the last week, something was merged which causes my laptop to lock up occasionally.  I can trigger it most of the time just by doing a kernel build. When it gets to the final linking stage, it locks up hard.

I finally managed to coax something out of usb console to get the traces below, which seem to implicate something in the block layer ?

my root device is an lvm volume on an dmcrypt'd block dev, which might be relevant, as I don't see this happening on other machines with simpler setups.

I'm going to try bisecting, but it might take me a few days, because it's such a pain in the ass to reproduce this reliably.

[ 5913.233035] ------------[ cut here ]------------
[ 5913.233097] WARNING: at lib/list_debug.c:53 __list_del_entry+0x8d/0x98()
[ 5913.233101] Hardware name: Adamo 13
[ 5913.233105] list_del corruption. prev->next should be ffffea000434fd20, but was ffffea0004199520

Quoted 1 line

See also: http://lkml.org/lkml/2011/8/3/37

That's in networking so SLUB lockless patches are almost certainly the issue here. Is this with SLUB debugging enabled or not? Christoph, it like the partial lists are getting corrupted somehow.

Quoted 21 lines · Pekka Enberg

Quoted 19 lines · Markus Trippelsdorf

Quoted 17 lines · Dave Jones

Sometime in the last week, something was merged which causes my laptop to lock up occasionally.  I can trigger it most of the time just by doing a kernel build. When it gets to the final linking stage, it locks up hard.

I finally managed to coax something out of usb console to get the traces below, which seem to implicate something in the block layer ?

my root device is an lvm volume on an dmcrypt'd block dev, which might be relevant, as I don't see this happening on other machines with simpler setups.

I'm going to try bisecting, but it might take me a few days, because it's such a pain in the ass to reproduce this reliably.

[ 5913.233035] ------------[ cut here ]------------
[ 5913.233097] WARNING: at lib/list_debug.c:53 __list_del_entry+0x8d/0x98()
[ 5913.233101] Hardware name: Adamo 13
[ 5913.233105] list_del corruption. prev->next should be ffffea000434fd20, but was ffffea0004199520

Quoted 5 lines

Quoted 1 line

See also: http://lkml.org/lkml/2011/8/3/37

That's in networking so SLUB lockless patches are almost certainly the issue here. Is this with SLUB debugging enabled or not? Christoph, it like the partial lists are getting corrupted somehow.

This is occurring in __slab_free when we are freeing the last object from a slab page that is on the partial list. It is not frozen so it is not in use by a processor and thus deactivate_slab cannot be run on it.

The logical race would be with acquire_slab() but both take the node lock before doing anything with the lists.

Do you have CONFIG_DEBUG_VM on? If not please do so. This will check if the frozen state is managed correctly.

Not sure if you were addressing my original report (quoted above), or the networking related report in this part of the thread. But anyway, I had it turned on (as well as just about every other debugging option). I didn't see anything output from it though, just the LIST_DEBUG warnings.

Quoted 4 lines · Dave Jones

Not sure if you were addressing my original report (quoted above), or the networking related report in this part of the thread. But anyway, I had it turned on (as well as just about every other debugging option). I didn't see anything output from it though, just the LIST_DEBUG warnings.

But CONFIG_SLUB_DEBUG_ON was off? If slab debugging was on then this could be an issue with full slab list management instead of partial lists.

Quoted 7 lines · Christoph Lameter

Quoted 4 lines · Dave Jones

Not sure if you were addressing my original report (quoted above), or the networking related report in this part of the thread. But anyway, I had it turned on (as well as just about every other debugging option). I didn't see anything output from it though, just the LIST_DEBUG warnings.

Quoted 2 lines

But CONFIG_SLUB_DEBUG_ON was off? If slab debugging was on then this could be an issue with full slab list management instead of partial lists.

$ grep SLUB_DEBUG_ON .config
CONFIG_SLUB_DEBUG_ON=y

Quoted 11 lines · Dave Jones

Quoted 9 lines · Christoph Lameter

Quoted 4 lines · Dave Jones

Not sure if you were addressing my original report (quoted above), or the networking related report in this part of the thread. But anyway, I had it turned on (as well as just about every other debugging option). I didn't see anything output from it though, just the LIST_DEBUG warnings.

But CONFIG_SLUB_DEBUG_ON was off? If slab debugging was on then this could be an issue with full slab list management instead of partial lists.

Quoted 2 lines

$ grep SLUB_DEBUG_ON .config
CONFIG_SLUB_DEBUG_ON=y

Argh another debugging code issue. Anyone who sees this please verify that it also occurs with debugging off. I ran this code for 9 months in various configs but only sporadically with debugging support since that switches the lockless stuff.

Quoted 4 lines

Argh another debugging code issue. Anyone who sees this please verify that it also occurs with debugging off. I ran this code for 9 months in various configs but only sporadically with debugging support since that switches the lockless stuff.

This started turning up immediately as soon as it got merged to mainline. Which leads me to believe no-one is running -next with debugging on either. Perhaps we should force debugging on for a week once per release cycle.

Quoted 8 lines · Dave Jones

Quoted 4 lines

Argh another debugging code issue. Anyone who sees this please verify that it also occurs with debugging off. I ran this code for 9 months in various configs but only sporadically with debugging support since that switches the lockless stuff.

This started turning up immediately as soon as it got merged to mainline. Which leads me to believe no-one is running -next with debugging on either. Perhaps we should force debugging on for a week once per release cycle.

I have run some tests over the last hours with kvm and CONFIG_SLUB_DEBUG_ON as well as CONFIG_DEBUG_LIST set but (aside from other unrelated backtraces in other subsystems) did not see this issue.

Can I get a .config? Maybe that will help.

Anybody has any ideas on this one? I'm back home, no more diving :(, ready to make -rc1, but I'd *prefer* to have a handle on this one.

Of course, the fact that it apparently only happens with SLUB debugging on means that the impact is less, but on the other hand I really like all the people who enable debug features and help us test with those on. So..

Anybody has any ideas on this one? I'm back home, no more diving :(, ready to make -rc1, but I'd *prefer* to have a handle on this one.

Of course, the fact that it apparently only happens with SLUB debugging on means that the impact is less, but on the other hand I really like all the people who enable debug features and help us test with those on. So..

Christoph, I've been reading the code and spotted two potential issues in __slab_free(). The first one seems like an off-by-one where our comparison in deactivate_slab() doesn't match __slab_free.

The other one is remove_full() call in __slab_free() that can get called even if cache debugging is not enabled.

Hmm?

Quoted 9 lines

Christoph, I've been reading the code and spotted two potential issues in __slab_free(). The first one seems like an off-by-one where our comparison in deactivate_slab() doesn't match __slab_free.

The other one is remove_full() call in __slab_free() that can get called even if cache debugging is not enabled.

Hmm?

I'd like to do -rc1 today, regardless of whether this fixes things or not (-rc1 is already a few days delayed).

The patch seems to be a good fix, and a likely candidate for the corruption. Commit log and sign-off? I assume you've given it some testing, even if you couldn't reproduce the original issue?

From 85380c605764927576d6ef54e4e8a3354df05d47 Mon Sep 17 00:00:00 2001

No, I haven't tested the patch myself but here's one in proper format in case someone wants to test it.

Quoted 3 lines · Pekka Enberg

No, I haven't tested the patch myself but here's one in proper format in case someone wants to test it.

Dave? It sounded like it was reasonably easily reproducible in your setup? Xiaotian?

I obviously already released -rc1, but it would be nice to get this one behind us regardless. Or, alternatively, know that what Pekka found isn't the root cause of your problems..

                     Linus

Hi Linus,

Quoted 10 lines · Pekka Enberg

Quoted 10 lines

Christoph, I've been reading the code and spotted two potential issues in __slab_free(). The first one seems like an off-by-one where our comparison
in deactivate_slab() doesn't match __slab_free.

The other one is remove_full() call in __slab_free() that can get called even if cache debugging is not enabled.

Hmm?

Quoted 4 lines · Linus Torvalds

I'd like to do -rc1 today, regardless of whether this fixes things or not (-rc1 is already a few days delayed).

Quoted 3 lines

The patch seems to be a good fix, and a likely candidate for the corruption. Commit log and sign-off? I assume you've given it some testing, even if you couldn't reproduce the original issue?

No, I haven't tested the patch myself but here's one in proper format in case someone wants to test it.

                       Pekka

From 85380c605764927576d6ef54e4e8a3354df05d47 Mon Sep 17 00:00:00 2001
From: Pekka Enberg <[email protected]>
Date: Mon, 8 Aug 2011 07:56:49 +0300
Subject: [PATCH] slub: Fix partial and full list handling in __slab_free

Dave Jones and Xiaotian Feng reported SLUB list corruption:

 https://lkml.org/lkml/2011/8/4/375

 https://lkml.org/lkml/2011/8/3/37

While I haven't able to reproduce the issue, I spotted two problems in __slab_free() during code review:

 - The ->nr_partial check in __slab_free() has an off-by-one bug when compared to similar check in deactivate_slab()

 - remove_full() is called even if cache debugging has not been enabled

Reported-by Dave Jones <[email protected]>

Reported-by Xiaotian Feng <[email protected]>

Signed-off-by Pekka Enberg <[email protected]>

---
 mm/slub.c |    5 +++--

 1 files changed, 3 insertions(+), 2 deletions(-)

mm/slub.c +1 -1

index eb5a8f9..cee8c20 100644
@@ -2368,7 +2368,7 @@ static void __slab_free(struct kmem_cache *s, struct
 page *page,
        if (was_frozen)
                stat(s, FREE_FROZEN);
        else {
-               if (unlikely(!inuse && n->nr_partial > s->min_partial))
+               if (unlikely(!inuse && n->nr_partial >= s->min_partial))
                         goto slab_empty;
 

               /*

mm/slub.c +2 -1

@@ -2376,7 +2376,8 @@ static void __slab_free(struct kmem_cache *s, struct
 page *page,
                 * then add it.
                 */
                if (unlikely(!prior)) {
-                       remove_full(s, page);
+                       if (kmem_cache_debug(s))
+                               remove_full(s, page);
                        add_partial(n, page, 0);
                        stat(s, FREE_ADD_PARTIAL);

               }
--
1.7.0.4

Quoted 19 lines · Pekka Enberg

Hi Linus,

Quoted 10 lines · Pekka Enberg

Quoted 10 lines

Christoph, I've been reading the code and spotted two potential issues in __slab_free(). The first one seems like an off-by-one where our comparison
in deactivate_slab() doesn't match __slab_free.

The other one is remove_full() call in __slab_free() that can get called even if cache debugging is not enabled.

Hmm?

Quoted 4 lines · Linus Torvalds

I'd like to do -rc1 today, regardless of whether this fixes things or not (-rc1 is already a few days delayed).

Quoted 6 lines

Quoted 3 lines

The patch seems to be a good fix, and a likely candidate for the corruption. Commit log and sign-off? I assume you've given it some testing, even if you couldn't reproduce the original issue?

No, I haven't tested the patch myself but here's one in proper format in case someone wants to test it.

I applied it and I'm still seeing the corruption warning start from this :(

Quoted 24 lines · Xiaotian Feng

Quoted 23 lines · Pekka Enberg

Hi Linus,

Quoted 10 lines · Pekka Enberg

Quoted 10 lines

Christoph, I've been reading the code and spotted two potential issues in __slab_free(). The first one seems like an off-by-one where our comparison
in deactivate_slab() doesn't match __slab_free.

The other one is remove_full() call in __slab_free() that can get called even if cache debugging is not enabled.

Hmm?

Quoted 7 lines · Linus Torvalds

I'd like to do -rc1 today, regardless of whether this fixes things or not (-rc1 is already a few days delayed).

The patch seems to be a good fix, and a likely candidate for the corruption. Commit log and sign-off? I assume you've given it some testing, even if you couldn't reproduce the original issue?

Quoted 4 lines

Quoted 2 lines

No, I haven't tested the patch myself but here's one in proper format in case someone wants to test it.

I applied it and I'm still seeing the corruption warning start from this :(

Thanks for testing. Can you share your .config, please?

Quoted 19 lines · Pekka Enberg

Hi Linus,

Quoted 10 lines · Pekka Enberg

Quoted 10 lines

Christoph, I've been reading the code and spotted two potential issues in __slab_free(). The first one seems like an off-by-one where our comparison
in deactivate_slab() doesn't match __slab_free.

The other one is remove_full() call in __slab_free() that can get called even if cache debugging is not enabled.

Hmm?

Quoted 4 lines · Linus Torvalds

I'd like to do -rc1 today, regardless of whether this fixes things or not (-rc1 is already a few days delayed).

Quoted 6 lines

Quoted 3 lines

The patch seems to be a good fix, and a likely candidate for the corruption. Commit log and sign-off? I assume you've given it some testing, even if you couldn't reproduce the original issue?

No, I haven't tested the patch myself but here's one in proper format in case someone wants to test it.

I applied it and I'm still seeing the corruption warning start from this :(

 [ 3674.255030] ------------[ cut here ]------------
 [ 3674.255040] WARNING: at lib/list_debug.c:53 __list_del_entry+0xa1/0xd0()
 [ 3674.255042] Hardware name: 42424XC
 [ 3674.255045] list_del corruption. prev->next should be
ffffea0000493420, but was ffffea0001ab0520
 [ 3674.255047] Modules linked in: ip6table_filter ip6_tables

ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack ipt_REJECT xt_CHECKSUM iptable_mangle xt_tcpudp iptable_filter ip_tables x_tables binfmt_misc bridge stp parport_pc ppdev snd_hda_codec_conexant snd_hda_intel snd_hda_codec snd_hwdep snd_pcm thinkpad_acpi arc4 cryptd aes_x86_64 aes_generic snd_seq_midi i915 snd_rawmidi snd_seq_midi_event snd_seq joydev iwlagn btusb snd_timer snd_seq_device drm_kms_helper bluetooth uvcvideo snd videodev drm mac80211 cfg80211 i2c_algo_bit soundcore tpm_tis tpm tpm_bios lp snd_page_alloc psmouse serio_raw v4l2_compat_ioctl32 nvram video parport usbhid hid sdhci_pci firewire_ohci ahci libahci sdhci firewire_core crc_itu_t e1000e

 [ 3674.255129] Pid: 3, comm: ksoftirqd/0 Not tainted 3.1.0-rc1+ #38
 [ 3674.255131] Call Trace:
 [ 3674.255138]  [<ffffffff8106db3f>] warn_slowpath_common+0x7f/0xc0
 [ 3674.255143]  [<ffffffff8106dc36>] warn_slowpath_fmt+0x46/0x50
 [ 3674.255147]  [<ffffffff81331851>] __list_del_entry+0xa1/0xd0
 [ 3674.255151]  [<ffffffff81331891>] list_del+0x11/0x40
 [ 3674.255156]  [<ffffffff81179631>] __slab_free+0x3d1/0x3e0
 [ 3674.255162]  [<ffffffff811c5b06>] ? bvec_free_bs+0x26/0x40
 [ 3674.255166]  [<ffffffff8117ab27>] ? kmem_cache_free+0x97/0x220
 [ 3674.255170]  [<ffffffff811c5b06>] ? bvec_free_bs+0x26/0x40
 [ 3674.255174]  [<ffffffff811c5b06>] ? bvec_free_bs+0x26/0x40
 [ 3674.255179]  [<ffffffff8117ac9f>] kmem_cache_free+0x20f/0x220

Quoted 46 lines · Xiaotian Feng

Quoted 19 lines · Pekka Enberg

Hi Linus,

Quoted 10 lines · Pekka Enberg

Quoted 10 lines

Christoph, I've been reading the code and spotted two potential issues in __slab_free(). The first one seems like an off-by-one where our comparison
in deactivate_slab() doesn't match __slab_free.

The other one is remove_full() call in __slab_free() that can get called even if cache debugging is not enabled.

Hmm?

Quoted 4 lines · Linus Torvalds

I'd like to do -rc1 today, regardless of whether this fixes things or not (-rc1 is already a few days delayed).

Quoted 6 lines

Quoted 3 lines

The patch seems to be a good fix, and a likely candidate for the corruption. Commit log and sign-off? I assume you've given it some testing, even if you couldn't reproduce the original issue?

No, I haven't tested the patch myself but here's one in proper format in case someone wants to test it.

I applied it and I'm still seeing the corruption warning start from this :(

 [ 3674.255030] ------------[ cut here ]------------
 [ 3674.255040] WARNING: at lib/list_debug.c:53 __list_del_entry+0xa1/0xd0()
 [ 3674.255042] Hardware name: 42424XC
 [ 3674.255045] list_del corruption. prev->next should be
ffffea0000493420, but was ffffea0001ab0520
 [ 3674.255047] Modules linked in: ip6table_filter ip6_tables

ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack ipt_REJECT xt_CHECKSUM iptable_mangle xt_tcpudp iptable_filter ip_tables x_tables binfmt_misc bridge stp parport_pc ppdev snd_hda_codec_conexant snd_hda_intel snd_hda_codec snd_hwdep snd_pcm thinkpad_acpi arc4 cryptd aes_x86_64 aes_generic snd_seq_midi i915 snd_rawmidi snd_seq_midi_event snd_seq joydev iwlagn btusb snd_timer snd_seq_device drm_kms_helper bluetooth uvcvideo snd videodev drm mac80211 cfg80211 i2c_algo_bit soundcore tpm_tis tpm tpm_bios lp snd_page_alloc psmouse serio_raw v4l2_compat_ioctl32 nvram video parport usbhid hid sdhci_pci firewire_ohci ahci libahci sdhci firewire_core crc_itu_t e1000e

Quoted 12 lines

 [ 3674.255129] Pid: 3, comm: ksoftirqd/0 Not tainted 3.1.0-rc1+ #38
 [ 3674.255131] Call Trace:
 [ 3674.255138]  [<ffffffff8106db3f>] warn_slowpath_common+0x7f/0xc0
 [ 3674.255143]  [<ffffffff8106dc36>] warn_slowpath_fmt+0x46/0x50
 [ 3674.255147]  [<ffffffff81331851>] __list_del_entry+0xa1/0xd0
 [ 3674.255151]  [<ffffffff81331891>] list_del+0x11/0x40
 [ 3674.255156]  [<ffffffff81179631>] __slab_free+0x3d1/0x3e0
 [ 3674.255162]  [<ffffffff811c5b06>] ? bvec_free_bs+0x26/0x40
 [ 3674.255166]  [<ffffffff8117ab27>] ? kmem_cache_free+0x97/0x220
 [ 3674.255170]  [<ffffffff811c5b06>] ? bvec_free_bs+0x26/0x40
 [ 3674.255174]  [<ffffffff811c5b06>] ? bvec_free_bs+0x26/0x40
 [ 3674.255179]  [<ffffffff8117ac9f>] kmem_cache_free+0x20f/0x220

Can you please also run

 addr2line -e vmlinux
 0xffffffff81179631

so that we know whether the corruption happens in the full list or the partial list.

                        Pekka

Quoted 60 lines · Pekka Enberg

Quoted 46 lines · Xiaotian Feng

Quoted 19 lines · Pekka Enberg

Hi Linus,

Quoted 10 lines · Pekka Enberg

Quoted 10 lines

Christoph, I've been reading the code and spotted two potential issues in __slab_free(). The first one seems like an off-by-one where our comparison
in deactivate_slab() doesn't match __slab_free.

The other one is remove_full() call in __slab_free() that can get called even if cache debugging is not enabled.

Hmm?

Quoted 4 lines · Linus Torvalds

I'd like to do -rc1 today, regardless of whether this fixes things or not (-rc1 is already a few days delayed).

Quoted 6 lines

Quoted 3 lines

The patch seems to be a good fix, and a likely candidate for the corruption. Commit log and sign-off? I assume you've given it some testing, even if you couldn't reproduce the original issue?

No, I haven't tested the patch myself but here's one in proper format in case someone wants to test it.

I applied it and I'm still seeing the corruption warning start from this :(

 [ 3674.255030] ------------[ cut here ]------------
 [ 3674.255040] WARNING: at lib/list_debug.c:53 __list_del_entry+0xa1/0xd0()
 [ 3674.255042] Hardware name: 42424XC
 [ 3674.255045] list_del corruption. prev->next should be
ffffea0000493420, but was ffffea0001ab0520
 [ 3674.255047] Modules linked in: ip6table_filter ip6_tables

ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack ipt_REJECT xt_CHECKSUM iptable_mangle xt_tcpudp iptable_filter ip_tables x_tables binfmt_misc bridge stp parport_pc ppdev snd_hda_codec_conexant snd_hda_intel snd_hda_codec snd_hwdep snd_pcm thinkpad_acpi arc4 cryptd aes_x86_64 aes_generic snd_seq_midi i915 snd_rawmidi snd_seq_midi_event snd_seq joydev iwlagn btusb snd_timer snd_seq_device drm_kms_helper bluetooth uvcvideo snd videodev drm mac80211 cfg80211 i2c_algo_bit soundcore tpm_tis tpm tpm_bios lp snd_page_alloc psmouse serio_raw v4l2_compat_ioctl32 nvram video parport usbhid hid sdhci_pci firewire_ohci ahci libahci sdhci firewire_core crc_itu_t e1000e

Quoted 12 lines

 [ 3674.255129] Pid: 3, comm: ksoftirqd/0 Not tainted 3.1.0-rc1+ #38
 [ 3674.255131] Call Trace:
 [ 3674.255138]  [<ffffffff8106db3f>] warn_slowpath_common+0x7f/0xc0
 [ 3674.255143]  [<ffffffff8106dc36>] warn_slowpath_fmt+0x46/0x50
 [ 3674.255147]  [<ffffffff81331851>] __list_del_entry+0xa1/0xd0
 [ 3674.255151]  [<ffffffff81331891>] list_del+0x11/0x40
 [ 3674.255156]  [<ffffffff81179631>] __slab_free+0x3d1/0x3e0
 [ 3674.255162]  [<ffffffff811c5b06>] ? bvec_free_bs+0x26/0x40
 [ 3674.255166]  [<ffffffff8117ab27>] ? kmem_cache_free+0x97/0x220
 [ 3674.255170]  [<ffffffff811c5b06>] ? bvec_free_bs+0x26/0x40
 [ 3674.255174]  [<ffffffff811c5b06>] ? bvec_free_bs+0x26/0x40
 [ 3674.255179]  [<ffffffff8117ac9f>] kmem_cache_free+0x20f/0x220

Quoted 5 lines

Can you please also run

 addr2line -e vmlinux
 0xffffffff81179631

/opt/upstream/linux-3.0/mm/slub.c:1061

/*
 * list_lock must be held.
 */
static void remove_full(struct kmem_cache *s, struct page *page) {
       if (!(s->flags & SLAB_STORE_USER))
               return;

       list_del(&page->lru);       <=== here
}

Quoted 78 lines · Xiaotian Feng

Quoted 60 lines · Pekka Enberg

Quoted 46 lines · Xiaotian Feng

Quoted 19 lines · Pekka Enberg

Hi Linus,

Quoted 10 lines · Pekka Enberg

Quoted 10 lines

Christoph, I've been reading the code and spotted two potential issues in __slab_free(). The first one seems like an off-by-one where our comparison
in deactivate_slab() doesn't match __slab_free.

The other one is remove_full() call in __slab_free() that can get called even if cache debugging is not enabled.

Hmm?

Quoted 4 lines · Linus Torvalds

I'd like to do -rc1 today, regardless of whether this fixes things or not (-rc1 is already a few days delayed).

Quoted 6 lines

Quoted 3 lines

The patch seems to be a good fix, and a likely candidate for the corruption. Commit log and sign-off? I assume you've given it some testing, even if you couldn't reproduce the original issue?

No, I haven't tested the patch myself but here's one in proper format in case someone wants to test it.

I applied it and I'm still seeing the corruption warning start from this :(

 [ 3674.255030] ------------[ cut here ]------------
 [ 3674.255040] WARNING: at lib/list_debug.c:53 __list_del_entry+0xa1/0xd0()
 [ 3674.255042] Hardware name: 42424XC
 [ 3674.255045] list_del corruption. prev->next should be
ffffea0000493420, but was ffffea0001ab0520
 [ 3674.255047] Modules linked in: ip6table_filter ip6_tables

ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack ipt_REJECT xt_CHECKSUM iptable_mangle xt_tcpudp iptable_filter ip_tables x_tables binfmt_misc bridge stp parport_pc ppdev snd_hda_codec_conexant snd_hda_intel snd_hda_codec snd_hwdep snd_pcm thinkpad_acpi arc4 cryptd aes_x86_64 aes_generic snd_seq_midi i915 snd_rawmidi snd_seq_midi_event snd_seq joydev iwlagn btusb snd_timer snd_seq_device drm_kms_helper bluetooth uvcvideo snd videodev drm mac80211 cfg80211 i2c_algo_bit soundcore tpm_tis tpm tpm_bios lp snd_page_alloc psmouse serio_raw v4l2_compat_ioctl32 nvram video parport usbhid hid sdhci_pci firewire_ohci ahci libahci sdhci firewire_core crc_itu_t e1000e

Quoted 12 lines

 [ 3674.255129] Pid: 3, comm: ksoftirqd/0 Not tainted 3.1.0-rc1+ #38
 [ 3674.255131] Call Trace:
 [ 3674.255138]  [<ffffffff8106db3f>] warn_slowpath_common+0x7f/0xc0
 [ 3674.255143]  [<ffffffff8106dc36>] warn_slowpath_fmt+0x46/0x50
 [ 3674.255147]  [<ffffffff81331851>] __list_del_entry+0xa1/0xd0
 [ 3674.255151]  [<ffffffff81331891>] list_del+0x11/0x40
 [ 3674.255156]  [<ffffffff81179631>] __slab_free+0x3d1/0x3e0
 [ 3674.255162]  [<ffffffff811c5b06>] ? bvec_free_bs+0x26/0x40
 [ 3674.255166]  [<ffffffff8117ab27>] ? kmem_cache_free+0x97/0x220
 [ 3674.255170]  [<ffffffff811c5b06>] ? bvec_free_bs+0x26/0x40
 [ 3674.255174]  [<ffffffff811c5b06>] ? bvec_free_bs+0x26/0x40
 [ 3674.255179]  [<ffffffff8117ac9f>] kmem_cache_free+0x20f/0x220

Quoted 5 lines

Can you please also run

 addr2line -e vmlinux
 0xffffffff81179631

/opt/upstream/linux-3.0/mm/slub.c:1061

/*
 * list_lock must be held.
 */
static void remove_full(struct kmem_cache *s, struct page *page) {
       if (!(s->flags & SLAB_STORE_USER))
               return;

Quoted 2 lines

       list_del(&page->lru);       <=== here
}

OK, so it's the full list. Looking at deactivate_slab() we actually can access the list without n->list_lock held. Does the attached patch help?

                       Pekka

Christoph, I've been reading the code and spotted two potential issues in __slab_free(). The first one seems like an off-by-one where our comparison in deactivate_slab() doesn't match __slab_free.

The other one is remove_full() call in __slab_free() that can get called even if cache debugging is not enabled.

Quoted 8 lines

 > Argh another debugging code issue. Anyone who sees this please verify that > it also occurs with debugging off. I ran this code for 9 months in > various configs but only sporadically with debugging support since that > switches the lockless stuff.

This started turning up immediately as soon as it got merged to mainline. Which leads me to believe no-one is running -next with debugging on either. Perhaps we should force debugging on for a week once per release cycle.

I have run some tests over the last hours with kvm and CONFIG_SLUB_DEBUG_ON as well as CONFIG_DEBUG_LIST set but (aside from other unrelated backtraces in other subsystems) did not see this issue.

Can I get a .config? Maybe that will help.

Sometime in the last week, something was merged which causes my laptop to lock up occasionally.  I can trigger it most of the time just by doing a kernel build. When it gets to the final linking stage, it locks up hard.

I finally managed to coax something out of usb console to get the traces below, which seem to implicate something in the block layer ?

my root device is an lvm volume on an dmcrypt'd block dev, which might be relevant, as I don't see this happening on other machines with simpler setups.

I'm going to try bisecting, but it might take me a few days, because it's such a pain in the ass to reproduce this reliably.

       Dave

[ 5913.233035] ------------[ cut here ]------------
[ 5913.233097] WARNING: at lib/list_debug.c:53 __list_del_entry+0x8d/0x98()
[ 5913.233101] Hardware name: Adamo 13
[ 5913.233105] list_del corruption. prev->next should be ffffea000434fd20, but was ffffea0004199520
[ 5913.233108] Modules linked in: nfs fscache fuse ebtable_nat ebtables ppdev parport_pc lp parport ipt_MASQUERADE iptable_nat nf_nat nfsd lockd nfs_acl auth_rpcgss xt_CHECKSUM sunrpc iptable_mangle bridge stp llc cpufreq_ondemand acpi_cpufreq freq_table mperf ip6t_REJECT nf_conntrack_ipv6 nf_defrag_ipv6 ip6table_filter ip6_tables rfcomm bnep arc4 iwlagn snd_hda_codec_hdmi snd_hda_codec_idt snd_hda_intel btusb mac80211 snd_hda_codec bluetooth snd_hwdep snd_seq snd_seq_device snd_pcm usb_debug dell_wmi sparse_keymap cdc_ether usbnet cdc_acm uvcvideo cdc_wdm mii cfg80211 snd_timer dell_laptop videodev dcdbas snd microcode v4l2_compat_ioctl32 soundcore joydev tg3 pcspkr snd_page_alloc iTCO_wdt i2c_i801 rfkill iTCO_vendor_support wmi virtio_net kvm_intel kvm ipv6 xts gf128mul dm_crypt i915 drm_kms_helper drm i2c_algo_bit i2c_core video [last unloaded: scsi_wait_scan]
[ 5913.233213] Pid: 0, comm: swapper Not tainted 3.0.0+ #127
[ 5913.233213] Call Trace:
[ 5913.233213]  <IRQ>  [<ffffffff8105df18>] warn_slowpath_common+0x83/0x9b
[ 5913.233213]  [<ffffffff8105dfd3>] warn_slowpath_fmt+0x46/0x48
[ 5913.233213]  [<ffffffff8127e7c1>] __list_del_entry+0x8d/0x98
[ 5913.233213]  [<ffffffff8127e7da>] list_del+0xe/0x2d
[ 5913.233213]  [<ffffffff814e0430>] __slab_free+0x1db/0x235
[ 5913.233213]  [<ffffffff811706ab>] ? bvec_free_bs+0x35/0x37
[ 5913.233213]  [<ffffffff811706ab>] ? bvec_free_bs+0x35/0x37
[ 5913.233213]  [<ffffffff811706ab>] ? bvec_free_bs+0x35/0x37
[ 5913.233213]  [<ffffffff81133085>] kmem_cache_free+0x88/0x102
[ 5913.233213]  [<ffffffff811706ab>] bvec_free_bs+0x35/0x37
[ 5913.233213]  [<ffffffff811706e1>] bio_free+0x34/0x64
[ 5913.233213]  [<ffffffff813dc390>] dm_bio_destructor+0x12/0x14
[ 5913.233213]  [<ffffffff8116fef6>] bio_put+0x2b/0x2d
[ 5913.233213]  [<ffffffff813dccab>] clone_endio+0x9e/0xb4
[ 5913.233213]  [<ffffffff8116f7dd>] bio_endio+0x2d/0x2f
[ 5913.233213]  [<ffffffffa00148da>] crypt_dec_pending+0x5c/0x8b [dm_crypt]
[ 5913.233213]  [<ffffffffa00150a9>] crypt_endio+0x78/0x81 [dm_crypt]
[ 5913.233213]  [<ffffffff8116f7dd>] bio_endio+0x2d/0x2f
[ 5913.233213]  [<ffffffff81257f60>] req_bio_endio+0x91/0x99
[ 5913.233213]  [<ffffffff81259fc9>] blk_update_request+0x1f2/0x395
[ 5913.233213]  [<ffffffff8125a18d>] blk_update_bidi_request+0x21/0x6d
[ 5913.233213]  [<ffffffff8125a2e0>] blk_end_bidi_request+0x1f/0x5d
[ 5913.233213]  [<ffffffff8125a358>] blk_end_request+0x10/0x12
[ 5913.233213]  [<ffffffff8135126c>] scsi_io_completion+0x1dc/0x4d7
[ 5913.233213]  [<ffffffff813491ac>] scsi_finish_command+0xe4/0xed
[ 5913.233213]  [<ffffffff81350fd4>] scsi_softirq_done+0x109/0x112
[ 5913.233213]  [<ffffffff8125f359>] blk_done_softirq+0x7f/0x93
[ 5913.233213]  [<ffffffff810646e5>] __do_softirq+0x10f/0x257
[ 5913.233213]  [<ffffffff810dc4e4>] ? time_hardirqs_off+0x1b/0x2f
[ 5913.233213]  [<ffffffff814f075c>] call_softirq+0x1c/0x30
[ 5913.233213]  [<ffffffff81011c7a>] do_softirq+0x4b/0xa1
[ 5913.233213]  [<ffffffff81064ac8>] irq_exit+0x5d/0xcd
[ 5913.233213]  [<ffffffff814f105d>] do_IRQ+0x8d/0xa4
[ 5913.233213]  [<ffffffff814e86f3>] common_interrupt+0x73/0x73
[ 5913.233213]  <EOI>  [<ffffffff81015f0c>] ? __cycles_2_ns+0xe/0x3a
[ 5913.233213]  [<ffffffff812eb3ff>] ? arch_local_irq_enable+0x8/0xd
[ 5913.233213]  [<ffffffff810918c2>] ? trace_hardirqs_on+0xd/0xf
[ 5913.233213]  [<ffffffff812ec20f>] acpi_idle_enter_bm+0x235/0x26d
[ 5913.233213]  [<ffffffff813ef81c>] cpuidle_idle_call+0x11c/0x1fe
[ 5913.233213]  [<ffffffff8100f31d>] cpu_idle+0xe2/0x14c
[ 5913.233213]  [<ffffffff81d48bad>] ? start_kernel+0x3ec/0x3f7
[ 5913.233213]  [<ffffffff814c4316>] rest_init+0xda/0xe1
[ 5913.233213]  [<ffffffff814c423c>] ? csum_partial_copy_generic+0x16c/0x16c
[ 5913.233213]  [<ffffffff81d48bad>] start_kernel+0x3ec/0x3f7
[ 5913.233213]  [<ffffffff81d482c4>] x86_64_start_reservations+0xaf/0[ 5913.239397] ------------[ cut here ]------------
[ 5913.239397] WARNING: at lib/list_debug.c:53 __list_del_entry+0x8d/0x98()
[ 5913.239397] Hardware name: Adamo 13
[ 5913.239397] list_del corruption. prev->next should be ffffea00041a82a0, but was ffffea0004677e60
[ 5913.239397] Modules linked in: nfs fscache fuse ebtable_nat ebtables ppdev parport_pc lp parport ipt_MASQUERADE iptable_nat nf_nat nfsd lockd nfs_acl auth_rpcgss xt_CHECKSUM sunrpc iptable_mangle bridge stp llc cpufreq_ondemand acpi_cpufreq freq_table mperf ip6t_REJECT nf_conntrack_ cdc_eth[ 5913.317698] ------------[ cut here ]------------
[ 5913.317761] WARNING: at lib/list_debug.c:53 __list_del_entry+0x8d/0x98()
[ 5913.317766] Hardware name: Adamo 13
[ 5913.317769] list_del corruption. prev->next should be ffffea0004145720, but was ffffea000415ee20
[ 5913.317773] Modules linked in: nfs fscache fuse ebtable_nat ebtables ppdev parport_pc lp parport ipt_MASQUERADE iptable_nat nf_nat nfsd lockd nfs_acl auth_rpcgss xt_CHECKSUM sunrpc iptable_mangle bridge stp llc cpufreq_ondemand acpi_cpufreq freq_table mperf ip6t_REJECT nf_conntrack_ipv6 nf_defrag_ipv6 ip6table_filter ip6_tables rfcomm bnep arc4 iwlagn snd_hda_codec_hdmi snd_hda_codec_idt snd_hda_intel btusb mac80211 snd_hda_codec bluetooth snd_hwdep snd_seq snd_seq_device snd_pcm usb_debug dell_wmi sparse_keymap cdc_ether usbnet cdc_acm uvcvideo cdc_wdm mii cfg80211 snd_timer dell_laptop videodev dcdbas snd microcode v4l2_compat_ioctl32 soundcore joydev tg3 pcspkr snd_page_alloc iTCO_wdt i2c_i801 rfkill iTCO_vendor_support wmi virtio_net kvm_intel kvm ipv6 xts gf128mul dm_crypt i915 drm_kms_helper drm i2c_algo_bit i2c_core video [last unloaded: scsi_wait_scan]
[ 5913.317926] Pid: 0, comm: kworker/0:0 Tainted: G        W   3.0.0+ #127
[ 5913.317930] Call Trace:
[ 5913.317933]  <IRQ>  [<ffffffff8105df18>] warn_slowpath_common+0x83/0x9b
[ 5913.317944]  [<ffffffff8105dfd3>] warn_slowpath_fmt+0x46/0x48
[ 5913.317950]  [<ffffffff8127e7c1>] __list_del_entry+0x8d/0x98
[ 5913.317955]  [<ffffffff8127e7da>] list_del+0xe/0x2d
[ 5913.317961]  [<ffffffff814e0430>] __slab_free+0x1db/0x235
[ 5913.317968]  [<ffffffff811706ab>] ? bvec_free_bs+0x35/0x37
[ 5913.317973]  [<ffffffff811706ab>] ? bvec_free_bs+0x35/0x37
[ 5913.317978]  [<ffffffff811706ab>] ? bvec_free_bs+0x35/0x37
[ 5913.317985]  [<ffffffff81133085>] kmem_cache_free+0x88/0x102
[ 5913.317990]  [<ffffffff811706ab>] bvec_free_bs+0x35/0x37
[ 5913.317995]  [<ffffffff811706e1>] bio_free+0x34/0x64
[ 5913.318002]  [<ffffffff813dc390>] dm_bio_destructor+0x12/0x14
[ 5913.318007]  [<ffffffff8116fef6>] bio_put+0x2b/0x2d
[ 5913.318012]  [<ffffffff813dccab>] clone_endio+0x9e/0xb4
[ 5913.318017]  [<ffffffff8116f7dd>] bio_endio+0x2d/0x2f
[ 5913.318030]  [<ffffffffa00148da>] crypt_dec_pending+0x5c/0x8b [dm_crypt]
[ 5913.318041]  [<ffffffffa00150a9>] crypt_endio+0x78/0x81 [dm_crypt]
[ 5913.318047]  [<ffffffff8116f7dd>] bio_endio+0x2d/0x2f
[ 5913.318053]  [<ffffffff81257f60>] req_bio_endio+0x91/0x99
[ 5913.318059]  [<ffffffff81259fc9>] blk_update_request+0x1f2/0x395
[ 5913.318064]  [<ffffffff8125a18d>] blk_update_bidi_request+0x21/0x6d
[ 5913.318070]  [<ffffffff8125a2e0>] blk_end_bidi_request+0x1f/0x5d
[ 5913.318075]  [<ffffffff8125a358>] blk_end_request+0x10/0x12
[ 5913.318081]  [<ffffffff8135126c>] scsi_io_completion+0x1dc/0x4d7
[ 5913.318087]  [<ffffffff813491ac>] scsi_finish_command+0xe4/0xed
[ 5913.318092]  [<ffffffff81350fd4>] scsi_softirq_done+0x109/0x112
[ 5913.318098]  [<ffffffff8125f359>] blk_done_softirq+0x7f/0x93
[ 5913.318104]  [<ffffffff810646e5>] __do_softirq+0x10f/0x257
[ 5913.318110]  [<ffffffff810dc4e4>] ? time_hardirqs_off+0x1b/0x2f
[ 5913.318116]  [<ffffffff814f075c>] call_softirq+0x1c/0x30
[ 5913.318123]  [<ffffffff81011c7a>] do_softirq+0x4b/0xa1
[ 5913.318127]  [<ffffffff81064ac8>] irq_exit+0x5d/0xcd
[ 5913.318133]  [<ffffffff814f105d>] do_IRQ+0x8d/0xa4
[ 5913.318138]  [<ffffffff814e86f3>] common_interrupt+0x73/0x73
[ 5913.318142]  <EOI>  [<ffffffff81015f0c>] ? __cycles_2_ns+0xe/0x3a
[ 5913.318152]  [<ffffffff812eb3ff>] ? arch_local_irq_enable+0x8/0xd
[ 5913.318158]  [<ffffffff810918c2>] ? trace_hardirqs_on+0xd/0xf
[ 5913.318164]  [<ffffffff812ec325>] acpi_idle_enter_simple+0xde/0x114
[ 5913.318170]  [<ffffffff813ef81c>] cpuidle_idle_call+0x11c/0x1fe
[ 5913.318176]  [<ffffffff8100f31d>] cpu_idle+0xe2/0x14c
[ 5913.318181]  [<ffffffff814d451d>] start_secondary+0x20d/0x218
[ 5913.318186] ---[ end trace ff2d5fa66f3f892e ]---
[ 5913.422416] ------------[ cut here ]------------
[ 5913.422501] WARNING: at lib/list_debug.c:53 __list_del_entry+0x8d/0x98()
[ 5913.422505] Hardware name: Adamo 13
[ 5913.422509] list_del corruption. prev->next should be ffffea00045acda0, but was dead000000100100
[ 5913.422513] Modules linked in: nfs fscache fuse ebtable_nat ebtables ppdev parport_pc lp parport ipt_MASQUERADE iptable_nat nf_nat nfsd lockd nfs_acl auth_rpcgss xt_CHECKSUM sunrpc iptable_mangle bridge stp llc cpufreq_ondemand acpi_cpufreq freq_table mperf ip6t_REJECT nf_conntrack_ipv6 nf_defrag_ipv6 ip6table_filter ip6_tables rfcomm bnep arc4 iwlagn snd_hda_codec_hdmi snd_hda_codec_idt snd_hda_intel btusb mac80211 snd_hda_codec bluetooth snd_hwdep snd_seq snd_seq_device snd_pcm usb_debug dell_wmi sparse_keymap cdc_ether usbnet cdc_acm uvcvideo cdc_wdm mii cfg80211 snd_timer dell_laptop videodev dcdbas snd microcode v4l2_compat_ioctl32 soundcore joydev tg3 pcspkr snd_page_alloc iTCO_wdt i2c_i801 rfkill iTCO_vendor_support wmi virtio_net kvm_intel kvm ipv6 xts gf128mul dm_crypt i915 drm_kms_helper drm i2c_algo_bit i2c_core video [last unloaded: scsi_wait_scan]
[ 5913.422667] Pid: 17555, comm: kworker/1:3 Tainted: G        W   3.0.0+ #127
[ 5913.422670] Call Trace:
[ 5913.422674]  <IRQ>  [<ffffffff8105df18>] warn_slowpath_common+0x83/0x9b
[ 5913.422685]  [<ffffffff8105dfd3>] warn_slowpath_fmt+0x46/0x48
[ 5913.422691]  [<ffffffff8127e7c1>] __list_del_entry+0x8d/0x98
[ 5913.422696]  [<ffffffff8127e7da>] list_del+0xe/0x2d
[ 5913.422702]  [<ffffffff814e0430>] __slab_free+0x1db/0x235
[ 5913.422709]  [<ffffffff811706ab>] ? bvec_free_bs+0x35/0x37
[ 5913.422715]  [<ffffffff811706ab>] ? bvec_free_bs+0x35/0x37
[ 5913.422720]  [<ffffffff811706ab>] ? bvec_free_bs+0x35/0x37
[ 5913.422726]  [<ffffffff81133085>] kmem_cache_free+0x88/0x102
[ 5913.422732]  [<ffffffff811706ab>] bvec_free_bs+0x35/0x37
[ 5913.422737]  [<ffffffff811706e1>] bio_free+0x34/0x64
[ 5913.422743]  [<ffffffff813dc390>] dm_bio_destructor+0x12/0x14
[ 5913.422749]  [<ffffffff8116fef6>] bio_put+0x2b/0x2d
[ 5913.422754]  [<ffffffff813dccab>] clone_endio+0x9e/0xb4
[ 5913.422759]  [<ffffffff8116f7dd>] bio_endio+0x2d/0x2f
[ 5913.422773]  [<ffffffffa00148da>] crypt_dec_pending+0x5c/0x8b [dm_crypt]
[ 5913.422786]  [<ffffffffa00150a9>] crypt_endio+0x78/0x81 [dm_crypt]
[ 5913.422791]  [<ffffffff8116f7dd>] bio_endio+0x2d/0x2f
[ 5913.422797]  [<ffffffff81257f60>] req_bio_endio+0x91/0x99
[ 5913.422803]  [<ffffffff81259fc9>] blk_update_request+0x1f2/0x395
[ 5913.422808]  [<ffffffff8125a18d>] blk_update_bidi_request+0x21/0x6d
[ 5913.422814]  [<ffffffff8125a2e0>] blk_end_bidi_request+0x1f/0x5d
[ 5913.422819]  [<ffffffff8125a358>] blk_end_request+0x10/0x12
[ 5913.422825]  [<ffffffff8135126c>] scsi_io_completion+0x1dc/0x4d7
[ 5913.422831]  [<ffffffff813491ac>] scsi_finish_command+0xe4/0xed
[ 5913.422836]  [<ffffffff81350fd4>] scsi_softirq_done+0x109/0x112
[ 5913.422842]  [<ffffffff8125f359>] blk_done_softirq+0x7f/0x93
[ 5913.422848]  [<ffffffff810646e5>] __do_softirq+0x10f/0x257
[ 5913.422854]  [<ffffffff810dc4e4>] ? time_hardirqs_off+0x1b/0x2f
[ 5913.422867]  [<ffffffffa00156fa>] ? crypt_convert+0x2b7/0x2b7 [dm_crypt]
[ 5913.422873]  [<ffffffff814f075c>] call_softirq+0x1c/0x30
[ 5913.422879]  [<ffffffff81011c7a>] do_softirq+0x4b/0xa1
[ 5913.422884]  [<ffffffff81064ac8>] irq_exit+0x5d/0xcd
[ 5913.422889]  [<ffffffff814f105d>] do_IRQ+0x8d/0xa4
[ 5913.422895]  [<ffffffff814e86f3>] common_interrupt+0x73/0x73
[ 5913.422898]  <EOI>  [<ffffffff8108d090>] ? debug_mutex_init+0x3e/0x3e
[ 5913.422909]  [<ffffffff8108d0b3>] ? arch_local_irq_save+0x9/0x18
[ 5913.422915]  [<ffffffff8108e10b>] lock_is_held+0x2c/0x95
[ 5913.422930]  [<ffffffff81099aea>] rcu_read_lock_held+0x25/0x27
[ 5913.422935]  [<ffffffff81099b4d>] __module_address+0x61/0xa0
[ 5913.422949]  [<ffffffff8109d41a>] is_module_address+0x20/0x59
[ 5913.422960]  [<ffffffff8108dbd9>] static_obj+0x3c/0x54
[ 5913.422970]  [<ffffffff8108e5a2>] lockdep_init_map+0xa0/0x115
[ 5913.422982]  [<ffffffff8107c41b>] __init_waitqueue_head+0x3a/0x4b
[ [ 5913.555121] ------------[ cut here ]------------
[ 5913.555121] WARNING: at lib/list_debug.c:53 __list_del_entry+0x8d/0x98()
[ 5913.555121] Hardware name: Adamo 13
[ 5913.555121] list_del corruption. prev->next should be ffffea000451a320, but was ffffea000480eda0
[ 5913.555121] Modules linked in: nfs fscache fuse ebtable_nat ebtables ppdev parport_pc lp parport ipt_MASQUERADE iptable_nat nf_nat nfsd lockd nfs_acl auth_rpcgss xt_CHECKSUM sunrpc iptable_mangle bridge stp llc cpufreq_ondemand acpi_cpufreq freq_table mperf ip6t_REJECT nf_conntrack_ipv6 nf_defrag_ipv6 ip6table_filter ip6_tables rfcomm bnep arc4 iwlagn snd_hda_codec_hdmi snd_hda_codec_idt snd_hda_intel btusb mac80211 snd_hda_codec bluetooth snd_hwdep snd_seq snd_seq_device snd_pcm usb_debug dell_wmi sparse_keymap cdc_ether usbnet cdc_acm uvcvideo cdc_wdm mii cfg80211 snd_timer dell_laptop videodev dcdbas snd microcode v4l2_compat_ioctl32 soundcore joydev tg3 pcspkr snd_page_alloc iTCO_wdt i2c_i801 rfkill iTCO_vendor_support wmi virtio_net kvm_intel kvm ipv6 xts gf128mul dm_crypt i915 drm_kms_helper drm i2c_algo_bit i2c_core video [last unloaded: scsi_wait_scan]
[ 5913.555121] Pid: 0, comm: swapper Tainted: G        W   3.0.0+ #127
[ 5913.555121] Call Trace:
[ 5913.555121]  <IRQ>  [<ffffffff8105df18>] warn_slowpath_common+0x83/0x9b
[ 5913.555121]  [<ffffffff8105dfd3>] warn_slowpath_fmt+0x46/0x48
[ 5913.555121]  [<ffffffff8127e7c1>] __list_del_entry+0x8d/0x98
[ 5913.555121]  [<ffffffff8127e7da>] list_del+0xe/0x2d
[ 5913.555121]  [<ffffffff814e0430>] __slab_free+0x1db/0x235
[ 5913.555121]  [<ffffffff811706ab>] ? bvec_free_bs+0x35/0x37
[ 5913.555121]  [<ffffffff811706ab>] ? bvec_free_bs+0x35/0x37
[ 5913.555121]  [<ffffffff811706ab>] ? bvec_free_bs+0x35/0x37
[ 5913.555121]  [<ffffffff81133085>] kmem_cache_free+0x88/0x102
[ 5913.555121]  [<ffffffff811706ab>] bvec_free_bs+0x35/0x37
[ 5913.555121]  [<ffffffff811706e1>] bio_free+0x34/0x64
[ 5913.555121]  [<ffffffff813dc390>] dm_bio_destructor+0x12/0x14
[ 5913.555121]  [<ffffffff8116fef6>] bio_put+0x2b/0x2d
[ 5913.555121]  [<ffffffff813dccab>] clone_endio+0x9e/0xb4
[ 5913.555121]  [<ffffffff8116f7dd>] bio_endio+0x2d/0x2f
[ 5913.555121]  [<ffffffffa00148da>] crypt_dec_pending+0x5c/0x8b [dm_crypt]
[ 5913.555121]  [<ffffffffa00150a9>] crypt_endio+0x78/0x81 [dm_crypt]
[ 5913.555121]  [<ffffffff8116f7dd>] bio_endio+0x2d/0x2f
[ 5913.555121]  [<ffffffff81257f60>] req_bio_endio+0x91/0x99
[ 5913.555121]  [<ffffffff81259fc9>] blk_update_request+0x1f2/0x395
[ 5913.555121]  [<ffffffff8125a18d>] blk_update_bidi_request+0x21/0x6d
[ 5913.555121]  [<ffffffff8125a2e0>] blk_end_bidi_request+0x1f/0x5d
[ 5913.555121]  [<ffffffff8125a358>] blk_end_request+0x10/0x12
[ 5913.555121]  [<ffffffff8135126c>] scsi_io_completion+0x1dc/0x4d7
[ 5913.555121]  [<ffffffff813491ac>] scsi_finish_command+0xe4/0xed
[ 5913.555121]  [<ffffffff81350fd4>] scsi_softirq_done+0x109/0x112
[ 5913.555121]  [<ffffffff8125f359>] blk_done_softirq+0x7f/0x93
[ 5913.555121]  [<ffffffff810646e5>] __do_softirq+0x10f/0x257
[ 5913.555121]  [<ffffffff810dc4e4>] ? time_hardirqs_off+0x1b/0x2f
[ 5913.555121]  [<ffffffff814f075c>] call_softirq+0x1c/0x30
[ 5913.555121]  [<ffffffff81011c7a>] do_softirq+0x4b/0xa1
[ 5913.555121]  [<ffffffff81064ac8>] irq_exit+0x5d/0xcd
[ 5913.555121]  [<ffffffff814f105d>] do_IRQ+0x8d/0xa4
[ 5913.555121]  [<ffffffff814e86f3>] common_interrupt+0x73/0x73
[ 5913.555121]  <EOI>  [<ffffffff81015f0c>] ? __cycles_2_ns+0xe/0x3a
[ 5913.555121]  [<ffffffff812eb3ff>] ? arch_local_irq_enable+0x8/0xd
[ 5913.555121]  [<ffffffff810918c2>] ? trace_hardirqs_on+0xd/0xf
[ 5913.555121]  [<ffffffff812ec325>] acpi_idle_enter_simple+0xde/0x114
[ 5913.555121]  [<ffffffff813ef81c>] cpuidle_idle_call+0x11c/0x1fe
[ 5913.555121]  [<ffffffff8100f31d>] cpu_idle+0xe2/0x14c
[ 5913.555121]  [<ffffffff81d48bad>] ? start_kernel+0x3ec/0x3f7
[ 5913.555121]  [<ffffffff814c4316>] rest_init+0xda/0xe1
[ 5913.555121]  [<ffffffff814c423c>] ? csum_partial_copy_generic+0x16c/0x16c
[ 5913.555121]  [<ffffffff81d48bad>] start_kernel+0x3ec/0x3f7
[ 5913.555121]  [ 5913.555121] ------------[ cut here ]------------
[ 5913.555121] WARNING: at lib/list_debug.c:53 __list_del_entry+0x8d/0x98()
[ 5913.555121] Hardware name: Adamo 13
[ 5913.555121] list_del corruption. prev->next should be ffffea00046e6c20, but was ffffea00043ae720
[ 5913.555121] Modules linked in: nfs fscache fuse ebtable_nat ebtables ppdev parport_pc lp parport ipt_MASQUERADE iptable_nat nf_nat nfsd lockd nfs_acl auth_rpcgss xt_CHECKSUM sunrpc iptable_mangle bridge stp llc cpufreq_ondemand acpi_cpufreq freq_table mperf ip6t_REJECT nf_conntrack_ipv6 nf_defrag_ipv6 ip6table_filter ip6_tables rfcomm bnep arc4 iwlagn snd_hda_codec_hdmi snd_hda_codec_idt snd_hda_intel btusb mac80211 snd_hda_codec bluetooth wmi vir[ 5913.5[ 5913.5[ 5913.5[ 5913.593105] ------------[ cut here ]------------
[ 5913.593114] WARNING: at lib/list_debug.c:26 __list_add+0x3d/0x7f()
[ 5913.593118] Hardware name: Adamo 13
[ 5913.593122] list_add corruption. next->prev should be prev (ffff88013825fe70), but was ffffea000439a120. (next=ffffea00045873e0).
[ 5913.593127] Modules linked in: nfs fscache fuse ebtable_nat ebtables ppdev parport_pc lp parport ipt_MASQUERADE iptable_nat nf_nat nfsd lockd nfs_acl auth_rpcgss xt_CHECKSUM sunrpc iptable_mangle bridge stp llc cpufreq_ondemand acpi_cpufreq freq_table mperf ip6t_REJECT nf_conntrack_ipv6 nf_defrag_ipv6 ip6table_filter ip6_tables rfcomm bnep arc4 iwlagn snd_hda_codec_hdmi snd_hda_codec_idt snd_hda_intel btusb mac80211 snd_hda_codec bluetooth snd_hwdep snd_seq snd_seq_device snd_pcm usb_debug dell_wmi sparse_keymap cdc_ether usbnet cdc_acm uvcvideo cdc_wdm mii cfg80211 snd_timer dell_laptop videodev dcdbas snd microcode v4l2_compat_ioctl32 soundcore joydev tg3 pcspkr snd_page_alloc iTCO_wdt i2c_i801 rfkill iTCO_vendor_support wmi virtio_net kvm_intel kvm ipv6 xts gf128mul dm_crypt i915 drm_kms_helper drm i2c_algo_bit i2c_core video [last unloaded: scsi_wait_scan]
[ 5913.593285] Pid: 900, comm: flush-253:3 Tainted: G        W   3.0.0+ #127
[ 5913.593289] Call Trace:
[ 5913.593295]  [<ffffffff8105df18>] warn_slowpath_common+0x83/0x9b
[ 5913.593301]  [<ffffffff8105dfd3>] warn_slowpath_fmt+0x46/0x48
[ 5913.593307]  [<ffffffff811335e1>] ? deactivate_slab+0x1b4/0x2b9
[ 5913.593313]  [<ffffffff8127e836>] __list_add+0x3d/0x7f
[ 5913.593318]  [<ffffffff8113367c>] deactivate_slab+0x24f/0x2b9
[ 5913.593325]  [<ffffffff811707d6>] ? bvec_[ 5914.004821] -

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/