BUG: lock held at task exit time!
Hi,
I think somehere might be interested in this, though I'm not sure who. I do not have the knowledge to say whether it originates within ext3, VFS, or elsewhere.
Anyway, I discovered an OOPS spammed into my ssh sessions to this machine, and kern.log contained:
Jul 22 06:26:55 localhost kernel: EXT3-fs error (device sda2): ext3_readdir: bad entry in directory #691212: directory entry across blocks - offset=12, inode=691211, rec_len=12320, name_len=2
Jul 22 06:26:55 localhost kernel: Remounting filesystem read-only
Jul 22 06:27:47 localhost kernel: Unable to handle kernel paging request at virtual address 0017e95a
Jul 22 06:27:47 localhost kernel: printing eip:
Jul 22 06:27:47 localhost kernel: c01502c1
Jul 22 06:27:47 localhost kernel: *pde = 00000000
Jul 22 06:27:47 localhost kernel: Oops: 0000 [#1]
Jul 22 06:27:47 localhost kernel: Modules linked in: i2c_via dm_mod
Jul 22 06:27:47 localhost kernel: CPU: 0
Jul 22 06:27:47 localhost kernel: EIP: 0060:[<c01502c1>] Not tainted VLI
Jul 22 06:27:47 localhost kernel: EFLAGS: 00010203 (2.6.16.18 #1)
Jul 22 06:27:47 localhost kernel: EIP is at __d_find_alias+0x14/0x9a
Jul 22 06:27:47 localhost kernel: eax: 00008000 ebx: 0017e95a ecx: 0017e95a edx: c73ed128
Jul 22 06:27:47 localhost kernel: esi: 00000000 edi: c73ec0ec ebp: c73ed110 esp: c4d0ede4
Jul 22 06:27:47 localhost kernel: ds: 007b es: 007b ss: 0068
Jul 22 06:27:47 localhost kernel: Process find (pid: 27598, threadinfo=c4d0e000 task=c63eba90)
Jul 22 06:27:47 localhost kernel: Stack: <0>00000001 c73ed110 c6c0b878 c6c0b878 c73ed364 c0150743 c73ed110 c13eb600
Jul 22 06:27:47 localhost kernel: c6c0b878 c017145b c3c67818 c03d65e0 c6c0b878 c73ed2f4 c0148d04 c4d0ee70
Jul 22 06:27:47 localhost kernel: c4d0ee64 c4d0ef1c c1145da0 95ca2dfe c73ed2f4 c67f2000 c4d0ef1c c0149435
Jul 22 06:27:47 localhost kernel: Call Trace:
Jul 22 06:27:47 localhost kernel: [<c0150743>] d_splice_alias+0x19/0xb2
Jul 22 06:27:47 localhost kernel: [<c017145b>] ext3_lookup+0x72/0x77
Jul 22 06:27:47 localhost kernel: [<c0148d04>] do_lookup+0xa3/0x137
Jul 22 06:27:47 localhost kernel: [<c0149435>] __link_path_walk+0x69d/0xa77
Jul 22 06:27:47 localhost kernel: [<c01525ef>] mntput_no_expire+0x11/0x52
Jul 22 06:27:47 localhost kernel: [<c01498be>] link_path_walk+0xaf/0xb9
Jul 22 06:27:47 localhost kernel: [<c0359f7c>] __mutex_lock_slowpath+0x1d0/0x276
Jul 22 06:27:47 localhost kernel: [<c0149856>] link_path_walk+0x47/0xb9
Jul 22 06:27:47 localhost kernel: [<c0149c74>] do_path_lookup+0x17f/0x19f
Jul 22 06:27:47 localhost kernel: [<c014a15a>] __user_walk_fd+0x2a/0x3f
Jul 22 06:27:47 localhost kernel: [<c0144f65>] vfs_lstat_fd+0x12/0x39
Jul 22 06:27:47 localhost kernel: [<c01455e9>] sys_lstat64+0xf/0x23
Jul 22 06:27:47 localhost kernel: [<c0102409>] syscall_call+0x7/0xb
Jul 22 06:27:47 localhost kernel: Code: 8d 4b c4 8b 59 3c 8d 74 26 00 8d 51 3c 8d 46 18 39 c2 75 96 5b 5e c3 55 89 c5 57 56 31 f6 53 51 89 14 24 8b 48 18 8d 50 18 eb 53 <8b> 19 8d 74 26 00 0f b7 45 28 8d 79 c4 25 00 f0 00 00 3d 00 40
Jul 22 06:27:47 localhost kernel: BUG: find/27598, lock held at task exit time!
Jul 22 06:27:47 localhost kernel: [c73ed364] {inode_init_once}
Jul 22 06:27:47 localhost kernel: .. held by: find:27598 [c63eba90, 126]
Jul 22 06:27:47 localhost kernel: ... acquired at: do_lookup+0x69/0x137/dev/sda2 is my root partition. Fortunately /var was on a different partition. Unsurprisingly the root partition contains errors:
Pass 1: Checking inodes, blocks, and sizes
Inode 114510 has illegal block(s). Clear? no
Illegal block #8 (3342783228) in inode 114510. IGNORED. Inodes that were part of a corrupted orphan linked list found. Fix? no
Inode 318876 was part of the orphaned inode list. IGNORED. Inode 351606 was part of the orphaned inode list. IGNORED. Inode 491835 was part of the orphaned inode list. IGNORED. Deleted inode 556073 has zero dtime. Fix? no
I am of course assuming that the mere presence of filesystem errors shouldn't cause the kernel to oops.
Output of ver_linux (keeping in mind I can't tell what has been apt-get upgraded since the kernel was compiled):
Linux plugh 2.6.16.18 #1 Sun May 28 01:17:17 WST 2006 i586 GNU/Linux
Gnu C 4.0.4
Gnu make 3.81
binutils 2.17
util-linux 2.12r
mount 2.12r
module-init-tools 3.2.2
e2fsprogs 1.39
reiserfsprogs line
reiser4progs line
PPP 2.4.4b1
Linux C Library 2.3.6
Dynamic linker (ldd) 2.3.6
Procps 3.2.7
Net-tools 1.60
Console-tools 0.2.3
Sh-utils 5.96
Modules Loaded i2c_via dm_mod
The machine is my household webserver (128MiB K6II-500, Debian testing/etch). It is still performing normally, despite a read only root fs (including /tmp). I'm happy to keep the machine in this state if further diagnostics are required; otherwise I'll eventually just build a new kernel and reboot it.
I'm not on the list, so please CC replies (though I'll probably check the archives from time to time anyway).
Thanks,
-MD
Signature · 3 lines
------------------------------------------------------------------------------- Michael Deegan Hugaholic http://wibble.darktech.org/gallery/ ------------------------- Nyy Tybel Gb Gur Ulcabgbnq! -------------------------
BUG: __d_find_alias went POP! (was: BUG: lock held at task exit time!)
Actually the lock held at exit time was caused by the BUG, it wasn't the bug itself. Seems you got a bad pointer which killed a task that happened to be holding a lock. And that's why you got the bug from your subject.
It looks like there was something fishy going on in __d_find_alias (like a corrupted inode?). Don't know for sure but since this looks like it's splice related or something wrong with general VFS, I CC'd Al Viro, and since it came from ext3, I CC'd Stephen Tweedie and the ext2-devel list.
Could a corrupted filesystem cause this oops?
-- Steve
Quoted 100 lines · Michael Deegan
Hi,
I think somehere might be interested in this, though I'm not sure who. I do not have the knowledge to say whether it originates within ext3, VFS, or elsewhere.
Anyway, I discovered an OOPS spammed into my ssh sessions to this machine, and kern.log contained:
Jul 22 06:26:55 localhost kernel: EXT3-fs error (device sda2): ext3_readdir: bad entry in directory #691212: directory entry across blocks - offset=12, inode=691211, rec_len=12320, name_len=2 Jul 22 06:26:55 localhost kernel: Remounting filesystem read-only Jul 22 06:27:47 localhost kernel: Unable to handle kernel paging request at virtual address 0017e95a Jul 22 06:27:47 localhost kernel: printing eip: Jul 22 06:27:47 localhost kernel: c01502c1 Jul 22 06:27:47 localhost kernel: *pde = 00000000 Jul 22 06:27:47 localhost kernel: Oops: 0000 [#1] Jul 22 06:27:47 localhost kernel: Modules linked in: i2c_via dm_mod Jul 22 06:27:47 localhost kernel: CPU: 0 Jul 22 06:27:47 localhost kernel: EIP: 0060:[<c01502c1>] Not tainted VLI Jul 22 06:27:47 localhost kernel: EFLAGS: 00010203 (2.6.16.18 #1) Jul 22 06:27:47 localhost kernel: EIP is at __d_find_alias+0x14/0x9a Jul 22 06:27:47 localhost kernel: eax: 00008000 ebx: 0017e95a ecx: 0017e95a edx: c73ed128 Jul 22 06:27:47 localhost kernel: esi: 00000000 edi: c73ec0ec ebp: c73ed110 esp: c4d0ede4 Jul 22 06:27:47 localhost kernel: ds: 007b es: 007b ss: 0068 Jul 22 06:27:47 localhost kernel: Process find (pid: 27598, threadinfo=c4d0e000 task=c63eba90) Jul 22 06:27:47 localhost kernel: Stack: <0>00000001 c73ed110 c6c0b878 c6c0b878 c73ed364 c0150743 c73ed110 c13eb600 Jul 22 06:27:47 localhost kernel: c6c0b878 c017145b c3c67818 c03d65e0 c6c0b878 c73ed2f4 c0148d04 c4d0ee70 Jul 22 06:27:47 localhost kernel: c4d0ee64 c4d0ef1c c1145da0 95ca2dfe c73ed2f4 c67f2000 c4d0ef1c c0149435 Jul 22 06:27:47 localhost kernel: Call Trace: Jul 22 06:27:47 localhost kernel: [<c0150743>] d_splice_alias+0x19/0xb2 Jul 22 06:27:47 localhost kernel: [<c017145b>] ext3_lookup+0x72/0x77 Jul 22 06:27:47 localhost kernel: [<c0148d04>] do_lookup+0xa3/0x137 Jul 22 06:27:47 localhost kernel: [<c0149435>] __link_path_walk+0x69d/0xa77 Jul 22 06:27:47 localhost kernel: [<c01525ef>] mntput_no_expire+0x11/0x52 Jul 22 06:27:47 localhost kernel: [<c01498be>] link_path_walk+0xaf/0xb9 Jul 22 06:27:47 localhost kernel: [<c0359f7c>] __mutex_lock_slowpath+0x1d0/0x276 Jul 22 06:27:47 localhost kernel: [<c0149856>] link_path_walk+0x47/0xb9 Jul 22 06:27:47 localhost kernel: [<c0149c74>] do_path_lookup+0x17f/0x19f Jul 22 06:27:47 localhost kernel: [<c014a15a>] __user_walk_fd+0x2a/0x3f Jul 22 06:27:47 localhost kernel: [<c0144f65>] vfs_lstat_fd+0x12/0x39 Jul 22 06:27:47 localhost kernel: [<c01455e9>] sys_lstat64+0xf/0x23 Jul 22 06:27:47 localhost kernel: [<c0102409>] syscall_call+0x7/0xb Jul 22 06:27:47 localhost kernel: Code: 8d 4b c4 8b 59 3c 8d 74 26 00 8d 51 3c 8d 46 18 39 c2 75 96 5b 5e c3 55 89 c5 57 56 31 f6 53 51 89 14 24 8b 48 18 8d 50 18 eb 53 <8b> 19 8d 74 26 00 0f b7 45 28 8d 79 c4 25 00 f0 00 00 3d 00 40 Jul 22 06:27:47 localhost kernel: BUG: find/27598, lock held at task exit time! Jul 22 06:27:47 localhost kernel: [c73ed364] {inode_init_once} Jul 22 06:27:47 localhost kernel: .. held by: find:27598 [c63eba90, 126] Jul 22 06:27:47 localhost kernel: ... acquired at: do_lookup+0x69/0x137
/dev/sda2is my root partition. Fortunately /var was on a different partition. Unsurprisingly the root partition contains errors:Pass 1: Checking inodes, blocks, and sizes
Inode 114510 has illegal block(s). Clear? noIllegal block #8 (3342783228) in inode 114510. IGNORED. Inodes that were part of a corrupted orphan linked list found. Fix? no
Inode 318876 was part of the orphaned inode list. IGNORED. Inode 351606 was part of the orphaned inode list. IGNORED. Inode 491835 was part of the orphaned inode list. IGNORED. Deleted inode 556073 has zero dtime. Fix? no
I am of course assuming that the mere presence of filesystem errors shouldn't cause the kernel to oops.
Output of
ver_linux(keeping in mind I can't tell what has been apt-get upgraded since the kernel was compiled):Linux plugh 2.6.16.18 #1 Sun May 28 01:17:17 WST 2006 i586 GNU/Linux
Gnu C 4.0.4
Gnu make 3.81
binutils 2.17
util-linux 2.12r
mount 2.12r
module-init-tools 3.2.2
e2fsprogs 1.39
reiserfsprogs line
reiser4progs line
PPP 2.4.4b1
Linux C Library 2.3.6
Dynamic linker (ldd) 2.3.6
Procps 3.2.7
Net-tools 1.60
Console-tools 0.2.3
Sh-utils 5.96
Modules Loadedi2c_viadm_modThe machine is my household webserver (128MiB K6II-500, Debian testing/etch). It is still performing normally, despite a read only root fs (including /tmp). I'm happy to keep the machine in this state if further diagnostics are required; otherwise I'll eventually just build a new kernel and reboot it.
I'm not on the list, so please CC replies (though I'll probably check the archives from time to time anyway).
Thanks,
-MD
Re: BUG: __d_find_alias went POP! (was: BUG: lock held at task exit time!)
Quoted 4 lines · Steven Rostedt
Actually the lock held at exit time was caused by the BUG, it wasn't the bug itself. Seems you got a bad pointer which killed a task that happened to be holding a lock. And that's why you got the bug from your subject.
I had initially been wondering if something in an error path forgot to clean things up correctly, but I guess there are valid circumstances where such cleanups don't get to happen...
Quoted 6 lines
It looks like there was something fishy going on in
__d_find_alias(like a corrupted inode?). Don't know for sure but since this looks like it's splice related or something wrong with general VFS, I CC'd Al Viro, and since it came from ext3, I CC'd Stephen Tweedie and the ext2-devel list.Could a corrupted filesystem cause this oops?
I'm afraid the problem was likely caused by bad RAM, as some time yesterday the machine hung hard, and upon restart failed POST. The machine was fine after removing the dud 32M SIMM.
Sorry for the bother,
-MD
Signature · 3 lines
------------------------------------------------------------------------------- Michael Deegan Hugaholic http://wibble.darktech.org/gallery/ ------------------------- Nyy Tybel Gb Gur Ulcabgbnq! -------------------------
Re: [Ext2-devel] BUG: __d_find_alias went POP! (was: BUG: lock held at task exit time!)
In reply to Steven RostedtQuoted 11 lines · Steven Rostedt
Actually the lock held at exit time was caused by the BUG, it wasn't the bug itself. Seems you got a bad pointer which killed a task that happened to be holding a lock. And that's why you got the bug from your subject.
It looks like there was something fishy going on in
__d_find_alias(like a corrupted inode?). Don't know for sure but since this looks like it's splice related or something wrong with general VFS, I CC'd Al Viro, and since it came from ext3, I CC'd Stephen Tweedie and the ext2-devel list.Could a corrupted filesystem cause this oops?
The filesystem is definitely corrupted a bit. At offset 12 in a directory is normally the ".." entry, which would match the name_len=2 value, and the fact that the (presumably parent) inode is one lower than the bad directory. What the actual value should be isn't clear, but it needs to be an integer between 12 and 4096, not 12320. The hex value of 12320 is 0x3020, which is also ASCII for " 0", though I'm not sure if this is relevant or not.
If "find" is doing a filesystem traversal it is definitely going to detect any such corruption. Whether the oops is directly related to this error, or only some later fallout because the filesystem is read-only is unclear.
Quoted 104 lines · Michael Deegan
Quoted 94 lines
Hi,
I think somehere might be interested in this, though I'm not sure who. I do not have the knowledge to say whether it originates within ext3, VFS, or elsewhere.
Anyway, I discovered an OOPS spammed into my ssh sessions to this machine, and kern.log contained:
Jul 22 06:26:55 localhost kernel: EXT3-fs error (device sda2): ext3_readdir: bad entry in directory #691212: directory entry across blocks - offset=12, inode=691211, rec_len=12320, name_len=2 Jul 22 06:26:55 localhost kernel: Remounting filesystem read-only Jul 22 06:27:47 localhost kernel: Unable to handle kernel paging request at virtual address 0017e95a Jul 22 06:27:47 localhost kernel: printing eip: Jul 22 06:27:47 localhost kernel: c01502c1 Jul 22 06:27:47 localhost kernel: *pde = 00000000 Jul 22 06:27:47 localhost kernel: Oops: 0000 [#1] Jul 22 06:27:47 localhost kernel: Modules linked in: i2c_via dm_mod Jul 22 06:27:47 localhost kernel: CPU: 0 Jul 22 06:27:47 localhost kernel: EIP: 0060:[<c01502c1>] Not tainted VLI Jul 22 06:27:47 localhost kernel: EFLAGS: 00010203 (2.6.16.18 #1) Jul 22 06:27:47 localhost kernel: EIP is at __d_find_alias+0x14/0x9a Jul 22 06:27:47 localhost kernel: eax: 00008000 ebx: 0017e95a ecx: 0017e95a edx: c73ed128 Jul 22 06:27:47 localhost kernel: esi: 00000000 edi: c73ec0ec ebp: c73ed110 esp: c4d0ede4 Jul 22 06:27:47 localhost kernel: ds: 007b es: 007b ss: 0068 Jul 22 06:27:47 localhost kernel: Process find (pid: 27598, threadinfo=c4d0e000 task=c63eba90) Jul 22 06:27:47 localhost kernel: Stack: <0>00000001 c73ed110 c6c0b878 c6c0b878 c73ed364 c0150743 c73ed110 c13eb600 Jul 22 06:27:47 localhost kernel: c6c0b878 c017145b c3c67818 c03d65e0 c6c0b878 c73ed2f4 c0148d04 c4d0ee70 Jul 22 06:27:47 localhost kernel: c4d0ee64 c4d0ef1c c1145da0 95ca2dfe c73ed2f4 c67f2000 c4d0ef1c c0149435 Jul 22 06:27:47 localhost kernel: Call Trace: Jul 22 06:27:47 localhost kernel: [<c0150743>] d_splice_alias+0x19/0xb2 Jul 22 06:27:47 localhost kernel: [<c017145b>] ext3_lookup+0x72/0x77 Jul 22 06:27:47 localhost kernel: [<c0148d04>] do_lookup+0xa3/0x137 Jul 22 06:27:47 localhost kernel: [<c0149435>] __link_path_walk+0x69d/0xa77 Jul 22 06:27:47 localhost kernel: [<c01525ef>] mntput_no_expire+0x11/0x52 Jul 22 06:27:47 localhost kernel: [<c01498be>] link_path_walk+0xaf/0xb9 Jul 22 06:27:47 localhost kernel: [<c0359f7c>] __mutex_lock_slowpath+0x1d0/0x276 Jul 22 06:27:47 localhost kernel: [<c0149856>] link_path_walk+0x47/0xb9 Jul 22 06:27:47 localhost kernel: [<c0149c74>] do_path_lookup+0x17f/0x19f Jul 22 06:27:47 localhost kernel: [<c014a15a>] __user_walk_fd+0x2a/0x3f Jul 22 06:27:47 localhost kernel: [<c0144f65>] vfs_lstat_fd+0x12/0x39 Jul 22 06:27:47 localhost kernel: [<c01455e9>] sys_lstat64+0xf/0x23 Jul 22 06:27:47 localhost kernel: [<c0102409>] syscall_call+0x7/0xb Jul 22 06:27:47 localhost kernel: Code: 8d 4b c4 8b 59 3c 8d 74 26 00 8d 51 3c 8d 46 18 39 c2 75 96 5b 5e c3 55 89 c5 57 56 31 f6 53 51 89 14 24 8b 48 18 8d 50 18 eb 53 <8b> 19 8d 74 26 00 0f b7 45 28 8d 79 c4 25 00 f0 00 00 3d 00 40 Jul 22 06:27:47 localhost kernel: BUG: find/27598, lock held at task exit time! Jul 22 06:27:47 localhost kernel: [c73ed364] {inode_init_once} Jul 22 06:27:47 localhost kernel: .. held by: find:27598 [c63eba90, 126] Jul 22 06:27:47 localhost kernel: ... acquired at: do_lookup+0x69/0x137
/dev/sda2is my root partition. Fortunately /var was on a different partition. Unsurprisingly the root partition contains errors:Pass 1: Checking inodes, blocks, and sizes
Inode 114510 has illegal block(s). Clear? noIllegal block #8 (3342783228) in inode 114510. IGNORED. Inodes that were part of a corrupted orphan linked list found. Fix? no
Inode 318876 was part of the orphaned inode list. IGNORED. Inode 351606 was part of the orphaned inode list. IGNORED. Inode 491835 was part of the orphaned inode list. IGNORED. Deleted inode 556073 has zero dtime. Fix? no
I am of course assuming that the mere presence of filesystem errors shouldn't cause the kernel to oops.
Output of
ver_linux(keeping in mind I can't tell what has been apt-get upgraded since the kernel was compiled):Linux plugh 2.6.16.18 #1 Sun May 28 01:17:17 WST 2006 i586 GNU/Linux
Gnu C 4.0.4
Gnu make 3.81
binutils 2.17
util-linux 2.12r
mount 2.12r
module-init-tools 3.2.2
e2fsprogs 1.39
reiserfsprogs line
reiser4progs line
PPP 2.4.4b1
Linux C Library 2.3.6
Dynamic linker (ldd) 2.3.6
Procps 3.2.7
Net-tools 1.60
Console-tools 0.2.3
Sh-utils 5.96
Modules Loadedi2c_viadm_modThe machine is my household webserver (128MiB K6II-500, Debian testing/etch). It is still performing normally, despite a read only root fs (including /tmp). I'm happy to keep the machine in this state if further diagnostics are required; otherwise I'll eventually just build a new kernel and reboot it.
Quoted 6 lines
I'm not on the list, so please CC replies (though I'll probably check the archives from time to time anyway).
Thanks,
-MD
Quoted 9 lines
-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT & business topics through brief surveys -- and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
_______________________________________________
Ext2-devel mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/ext2-devel
Cheers, Andreas
--
Andreas Dilger
Principal Software Engineer
Cluster File Systems, Inc.