Skip to contentSkip to conversation

2.4.29-pre2 Oops at find_inode/reiserfs_find_actor

DISCUSSION Manfred Schwarb 4 participants 8 messages Dec 20, 2004 – Dec 28, 2004 VFS

lore
Share & tools
lore
#0 lore raw

Hi,
yesterday, I got at the start of a rsync backup over nfs an Oops in find_inode/reiserfs_find_actor.
Some googling revealed that this seems to be an
inode list corruption, but how can this happen?
There seem to be quite some reports about similar problems, so doing rsync over nfs-mounted partitions might be a bad idea? Is there anything I can do? Any help is appreciated.

This machine is an athlon i386 with vanilla 2.4.29-pre2. Below you may find the ksymoopsified Oops:

Dec 19 21:00:01  kernel: Unable to handle kernel NULL pointer dereference at
virtual address 00000028
Dec 19 21:00:01  kernel: c0153b09
Dec 19 21:00:01  kernel: *pde = 1d678067
Dec 19 21:00:01  kernel: Oops: 0000
Dec 19 21:00:01  kernel: CPU:    0
Dec 19 21:00:01  kernel: EIP:    0010:[find_inode+25/112]    Not tainted
Dec 19 21:00:01  kernel: EIP:    0010:[<c0153b09>]    Not tainted
Using defaults from ksymoops -t elf32-i386 -a i386
Dec 19 21:00:01  kernel: EFLAGS: 00213213
Dec 19 21:00:01  kernel: eax: e0b3c9e0   ebx: 00000000   ecx: 0000000f  
edx: dff80000
Dec 19 21:00:01  kernel: esi: 00000000   edi: dffa2a58   ebp: 000000fa  
esp: d7f05d60
Dec 19 21:00:01  kernel: ds: 0018   es: 0018   ss: 0018
Dec 19 21:00:01  kernel: Process nfsd (pid: 2372, stackpage=d7f05000)
Dec 19 21:00:01  kernel: Stack: 0efe42ff 00000000 00000002 d7f05de4 dffa2a58
000000fa da2f6000 c0153f4e
Dec 19 21:00:01  kernel:        da2f6000 000000fa dffa2a58 e0b3c9e0 d7f05db8
d7f05de4 d7f05e48 d7f05db8
Dec 19 21:00:01  kernel:        da2f6000 e0b3ca60 da2f6000 000000fa e0b3c9e0
d7f05db8 00000011 d7f05de4
Dec 19 21:00:01  kernel: Call Trace:    [iget4_locked+94/272]

[keybdev:__insmod_keybdev_O/lib/modules/2.4.29-pre2/kernel/drivers/i+4288752096/96] [keybdev:__insmod_keybdev_O/lib/modules/2.4.29-pre2/kernel/drivers/i+4288752224/96] [keybdev:__insmod_keybdev_O/lib/modules/2.4.29-pre2/kernel/drivers/i+4288752096/96] [keybdev:__insmod_keybdev_O/lib/modules/2.4.29-pre2/kernel/drivers/i+4288731921/96]

Dec 19 21:00:01  kernel: Call Trace:    [<c0153f4e>] [<e0b3c9e0>]
[<e0b3ca60>] [<e0b3c9e0>] [<e0b37b11>]
Dec 19 21:00:01  kernel:   [<c0151d1c>] [<c01491cf>] [<c0149279>]
[<e118fd71>] [<e1196d39>] [<e119de8c>]
Dec 19 21:00:01  kernel:   [<e118c68d>] [<c027ab3e>] [<e119de8c>]
[<e119d758>] [<e119d778>] [<e118c3cb>]
Dec 19 21:00:01  kernel:   [<c010729b>] [<e118c210>]
Dec 19 21:00:01  kernel: Code: 39 6b 28 89 de 75 f1 8b 44 24 20 39 83 a0 00

00 00 75 e5 8b

Quoted 6 lines
Quoted 6 lines

EIP; c0153b09 <find_inode+19/70> <=====

eax; e0b3c9e0 <[reiserfs]reiserfs_find_actor+0/40> edx; dff80000 <_end+1fbfd7f4/20792854> edi; dffa2a58 <_end+1fc2024c/20792854> esp; d7f05d60 <_end+17b83554/20792854>

Trace; c0153f4e <iget4_locked+5e/110>
Trace; e0b3c9e0 <[reiserfs]reiserfs_find_actor+0/40> Trace; e0b3ca60 <[reiserfs]reiserfs_iget+40/c0> Trace; e0b3c9e0 <[reiserfs]reiserfs_find_actor+0/40> Trace; e0b37b11 <[reiserfs]reiserfs_lookup+101/120> Trace; c0151d1c <d_alloc+1c/1d0>
Trace; c01491cf <lookup_hash+9f/d0>
Trace; c0149279 <lookup_one_len+79/90> Trace; e118fd71 <[nfsd]nfsd_lookup+d1/490> Trace; e1196d39 <[nfsd]nfsd3_proc_lookup+a9/140> Trace; e119de8c <[nfsd]nfsd_procedures3+6c/320> Trace; e118c68d <[nfsd]nfsd_dispatch+14d/220> Trace; c027ab3e <svc_process+3de/590>
Trace; e119de8c <[nfsd]nfsd_procedures3+6c/320> Trace; e119d758 <[nfsd]nfsd_version3+0/10> Trace; e119d778 <[nfsd]nfsd_program+0/28> Trace; e118c3cb <[nfsd]nfsd+1bb/330>
Trace; c010729b <arch_kernel_thread+2b/40> Trace; e118c210 <[nfsd]nfsd+0/330>

Code; c0153b09 <find_inode+19/70>
00000000 <_EIP>:
Code; c0153b09 <find_inode+19/70> <=====
0: 39 6b 28 cmp %ebp,0x28(%ebx) <=====
Code; c0153b0c <find_inode+1c/70>
3: 89 de mov %ebx,%esi
Code; c0153b0e <find_inode+1e/70>
5: 75 f1 jne fffffff8 <_EIP+0xfffffff8>
Code; c0153b10 <find_inode+20/70>
7: 8b 44 24 20 mov 0x20(%esp,1),%eax
Code; c0153b14 <find_inode+24/70>
b: 39 83 a0 00 00 00 cmp %eax,0xa0(%ebx)
Code; c0153b1a <find_inode+2a/70>
11: 75 e5 jne fffffff8 <_EIP+0xfffffff8>
Code; c0153b1c <find_inode+2c/70>
13: 8b 00 mov (%eax),%eax

Signature · 2 lines
Psssst! Mit GMX Handyrechnung senken: http://www.gmx.net/de/go/mail
100 FreeSMS/Monat (GMX TopMail), 50 (GMX ProMail), 10 (GMX FreeMail)
#1 lore raw

hmm, I just saw that I forgot to add reiserfs-list to my report, I just sent it to lkml (fingers faster than brain, sorry). So here again, to the reiserfs folks.

Hi,
yesterday, I got at the start of a rsync backup over nfs an Oops in find_inode/reiserfs_find_actor.
Some googling revealed that this seems to be an
inode list corruption, but how can this happen?
There seem to be quite some reports about similar problems, so doing rsync over nfs-mounted partitions might be a bad idea? Is there anything I can do? Any help is appreciated.

This machine is an athlon i386 with vanilla 2.4.29-pre2. Below you may find the ksymoopsified Oops:

Dec 19 21:00:01  kernel: Unable to handle kernel NULL pointer dereference at
virtual address 00000028
Dec 19 21:00:01  kernel: c0153b09
Dec 19 21:00:01  kernel: *pde = 1d678067
Dec 19 21:00:01  kernel: Oops: 0000
Dec 19 21:00:01  kernel: CPU:    0
Dec 19 21:00:01  kernel: EIP:    0010:[find_inode+25/112]    Not tainted
Dec 19 21:00:01  kernel: EIP:    0010:[<c0153b09>]    Not tainted
Using defaults from ksymoops -t elf32-i386 -a i386
Dec 19 21:00:01  kernel: EFLAGS: 00213213
Dec 19 21:00:01  kernel: eax: e0b3c9e0   ebx: 00000000   ecx: 0000000f  
edx: dff80000
Dec 19 21:00:01  kernel: esi: 00000000   edi: dffa2a58   ebp: 000000fa  
esp: d7f05d60
Dec 19 21:00:01  kernel: ds: 0018   es: 0018   ss: 0018
Dec 19 21:00:01  kernel: Process nfsd (pid: 2372, stackpage=d7f05000)
Dec 19 21:00:01  kernel: Stack: 0efe42ff 00000000 00000002 d7f05de4 dffa2a58
000000fa da2f6000 c0153f4e
Dec 19 21:00:01  kernel:        da2f6000 000000fa dffa2a58 e0b3c9e0 d7f05db8
d7f05de4 d7f05e48 d7f05db8
Dec 19 21:00:01  kernel:        da2f6000 e0b3ca60 da2f6000 000000fa e0b3c9e0
d7f05db8 00000011 d7f05de4
Dec 19 21:00:01  kernel: Call Trace:   

[iget4_locked+94/272] [keybdev:__insmod_keybdev_O/lib/modules/2.4.29-pre2/kernel/drivers/i+4288752096/96] [keybdev:__insmod_keybdev_O/lib/modules/2.4.29-pre2/kernel/drivers/i+4288752224/96] [keybdev:__insmod_keybdev_O/lib/modules/2.4.29-pre2/kernel/drivers/i+4288752096/96] [keybdev:__insmod_keybdev_O/lib/modules/2.4.29-pre2/kernel/drivers/i+4288731921/96]

Dec 19 21:00:01  kernel: Call Trace:    [<c0153f4e>] [<e0b3c9e0>]
[<e0b3ca60>] [<e0b3c9e0>] [<e0b37b11>]
Dec 19 21:00:01  kernel:   [<c0151d1c>] [<c01491cf>] [<c0149279>]
[<e118fd71>] [<e1196d39>] [<e119de8c>]
Dec 19 21:00:01  kernel:   [<e118c68d>] [<c027ab3e>] [<e119de8c>]
[<e119d758>] [<e119d778>] [<e118c3cb>]
Dec 19 21:00:01  kernel:   [<c010729b>] [<e118c210>]
Dec 19 21:00:01  kernel: Code: 39 6b 28 89 de 75 f1 8b 44 24 20 39 83 a0 00

00 00 75 e5 8b

Quoted 6 lines
Quoted 6 lines

EIP; c0153b09 <find_inode+19/70> <=====

eax; e0b3c9e0 <[reiserfs]reiserfs_find_actor+0/40> edx; dff80000 <_end+1fbfd7f4/20792854> edi; dffa2a58 <_end+1fc2024c/20792854> esp; d7f05d60 <_end+17b83554/20792854>

Trace; c0153f4e <iget4_locked+5e/110>
Trace; e0b3c9e0 <[reiserfs]reiserfs_find_actor+0/40> Trace; e0b3ca60 <[reiserfs]reiserfs_iget+40/c0> Trace; e0b3c9e0 <[reiserfs]reiserfs_find_actor+0/40> Trace; e0b37b11 <[reiserfs]reiserfs_lookup+101/120> Trace; c0151d1c <d_alloc+1c/1d0>
Trace; c01491cf <lookup_hash+9f/d0>
Trace; c0149279 <lookup_one_len+79/90> Trace; e118fd71 <[nfsd]nfsd_lookup+d1/490> Trace; e1196d39 <[nfsd]nfsd3_proc_lookup+a9/140> Trace; e119de8c <[nfsd]nfsd_procedures3+6c/320> Trace; e118c68d <[nfsd]nfsd_dispatch+14d/220> Trace; c027ab3e <svc_process+3de/590>
Trace; e119de8c <[nfsd]nfsd_procedures3+6c/320> Trace; e119d758 <[nfsd]nfsd_version3+0/10> Trace; e119d778 <[nfsd]nfsd_program+0/28> Trace; e118c3cb <[nfsd]nfsd+1bb/330>
Trace; c010729b <arch_kernel_thread+2b/40> Trace; e118c210 <[nfsd]nfsd+0/330>

Code; c0153b09 <find_inode+19/70>
00000000 <_EIP>:
Code; c0153b09 <find_inode+19/70> <=====
0: 39 6b 28 cmp %ebp,0x28(%ebx) <=====
Code; c0153b0c <find_inode+1c/70>
3: 89 de mov %ebx,%esi
Code; c0153b0e <find_inode+1e/70>
5: 75 f1 jne fffffff8 <_EIP+0xfffffff8>
Code; c0153b10 <find_inode+20/70>
7: 8b 44 24 20 mov 0x20(%esp,1),%eax
Code; c0153b14 <find_inode+24/70>
b: 39 83 a0 00 00 00 cmp %eax,0xa0(%ebx)
Code; c0153b1a <find_inode+2a/70>
11: 75 e5 jne fffffff8 <_EIP+0xfffffff8>
Code; c0153b1c <find_inode+2c/70>
13: 8b 00 mov (%eax),%eax

Signature · 2 lines
Psssst! Mit GMX Handyrechnung senken: http://www.gmx.net/de/go/mail
100 FreeSMS/Monat (GMX TopMail), 50 (GMX ProMail), 10 (GMX FreeMail)
#2 lore raw
In reply to Manfred Schwarb

CCing the ReiserFS devel list because they might have some clue.

Quoted 7 lines · Manfred Schwarb

Hi,
yesterday, I got at the start of a rsync backup over nfs an Oops in find_inode/reiserfs_find_actor.
Some googling revealed that this seems to be an
inode list corruption, but how can this happen?
There seem to be quite some reports about similar problems, so doing rsync over nfs-mounted partitions might be a bad idea?

No, it should work.

Quoted 71 lines

Is there anything I can do? Any help is appreciated.

This machine is an athlon i386 with vanilla 2.4.29-pre2. Below you may find the ksymoopsified Oops:

Dec 19 21:00:01  kernel: Unable to handle kernel NULL pointer dereference at
virtual address 00000028
Dec 19 21:00:01  kernel: c0153b09
Dec 19 21:00:01  kernel: *pde = 1d678067
Dec 19 21:00:01  kernel: Oops: 0000
Dec 19 21:00:01  kernel: CPU:    0
Dec 19 21:00:01  kernel: EIP:    0010:[find_inode+25/112]    Not tainted
Dec 19 21:00:01  kernel: EIP:    0010:[<c0153b09>]    Not tainted
Using defaults from ksymoops -t elf32-i386 -a i386
Dec 19 21:00:01  kernel: EFLAGS: 00213213
Dec 19 21:00:01  kernel: eax: e0b3c9e0   ebx: 00000000   ecx: 0000000f  
edx: dff80000
Dec 19 21:00:01  kernel: esi: 00000000   edi: dffa2a58   ebp: 000000fa  
esp: d7f05d60
Dec 19 21:00:01  kernel: ds: 0018   es: 0018   ss: 0018
Dec 19 21:00:01  kernel: Process nfsd (pid: 2372, stackpage=d7f05000)
Dec 19 21:00:01  kernel: Stack: 0efe42ff 00000000 00000002 d7f05de4 dffa2a58
000000fa da2f6000 c0153f4e
Dec 19 21:00:01  kernel:        da2f6000 000000fa dffa2a58 e0b3c9e0 d7f05db8
d7f05de4 d7f05e48 d7f05db8
Dec 19 21:00:01  kernel:        da2f6000 e0b3ca60 da2f6000 000000fa e0b3c9e0
d7f05db8 00000011 d7f05de4
Dec 19 21:00:01  kernel: Call Trace:    [iget4_locked+94/272]

[keybdev:__insmod_keybdev_O/lib/modules/2.4.29-pre2/kernel/drivers/i+4288752096/96] [keybdev:__insmod_keybdev_O/lib/modules/2.4.29-pre2/kernel/drivers/i+4288752224/96] [keybdev:__insmod_keybdev_O/lib/modules/2.4.29-pre2/kernel/drivers/i+4288752096/96] [keybdev:__insmod_keybdev_O/lib/modules/2.4.29-pre2/kernel/drivers/i+4288731921/96]

Dec 19 21:00:01  kernel: Call Trace:    [<c0153f4e>] [<e0b3c9e0>]
[<e0b3ca60>] [<e0b3c9e0>] [<e0b37b11>]
Dec 19 21:00:01  kernel:   [<c0151d1c>] [<c01491cf>] [<c0149279>]
[<e118fd71>] [<e1196d39>] [<e119de8c>]
Dec 19 21:00:01  kernel:   [<e118c68d>] [<c027ab3e>] [<e119de8c>]
[<e119d758>] [<e119d778>] [<e118c3cb>]
Dec 19 21:00:01  kernel:   [<c010729b>] [<e118c210>]
Dec 19 21:00:01  kernel: Code: 39 6b 28 89 de 75 f1 8b 44 24 20 39 83 a0 00

00 00 75 e5 8b

Quoted 6 lines
Quoted 6 lines

EIP; c0153b09 <find_inode+19/70> <=====

eax; e0b3c9e0 <[reiserfs]reiserfs_find_actor+0/40> edx; dff80000 <_end+1fbfd7f4/20792854> edi; dffa2a58 <_end+1fc2024c/20792854> esp; d7f05d60 <_end+17b83554/20792854>

Trace; c0153f4e <iget4_locked+5e/110>
Trace; e0b3c9e0 <[reiserfs]reiserfs_find_actor+0/40> Trace; e0b3ca60 <[reiserfs]reiserfs_iget+40/c0> Trace; e0b3c9e0 <[reiserfs]reiserfs_find_actor+0/40> Trace; e0b37b11 <[reiserfs]reiserfs_lookup+101/120> Trace; c0151d1c <d_alloc+1c/1d0>
Trace; c01491cf <lookup_hash+9f/d0>
Trace; c0149279 <lookup_one_len+79/90> Trace; e118fd71 <[nfsd]nfsd_lookup+d1/490> Trace; e1196d39 <[nfsd]nfsd3_proc_lookup+a9/140> Trace; e119de8c <[nfsd]nfsd_procedures3+6c/320> Trace; e118c68d <[nfsd]nfsd_dispatch+14d/220> Trace; c027ab3e <svc_process+3de/590>
Trace; e119de8c <[nfsd]nfsd_procedures3+6c/320> Trace; e119d758 <[nfsd]nfsd_version3+0/10> Trace; e119d778 <[nfsd]nfsd_program+0/28> Trace; e118c3cb <[nfsd]nfsd+1bb/330>
Trace; c010729b <arch_kernel_thread+2b/40> Trace; e118c210 <[nfsd]nfsd+0/330>

Quoted 16 lines

Code; c0153b09 <find_inode+19/70>
00000000 <_EIP>:
Code; c0153b09 <find_inode+19/70> <=====
0: 39 6b 28 cmp %ebp,0x28(%ebx) <=====
Code; c0153b0c <find_inode+1c/70>
3: 89 de mov %ebx,%esi
Code; c0153b0e <find_inode+1e/70>
5: 75 f1 jne fffffff8 <_EIP+0xfffffff8>
Code; c0153b10 <find_inode+20/70>
7: 8b 44 24 20 mov 0x20(%esp,1),%eax
Code; c0153b14 <find_inode+24/70>
b: 39 83 a0 00 00 00 cmp %eax,0xa0(%ebx)
Code; c0153b1a <find_inode+2a/70>
11: 75 e5 jne fffffff8 <_EIP+0xfffffff8>
Code; c0153b1c <find_inode+2c/70>
13: 8b 00 mov (%eax),%eax

This is indeed corruption - an inode in this hash bucket has "->next" as NULL, so find_inode goes boom.

Something is leaving this hash bucket list corrupt.

Have you ever seen this crash before ? Can you reproduce it?

The only inode corruption case that was reliable was from Chris Caputo and we end up agreeing that it was most likely a hardware issue, because it was hard to reproduce and strange in several ways. Can you point me at the "quite some similar reports" you have found, please ?

The inode number being looked up is in %ebp, which is 250.

Trond, the following issue noted by you could not generate such corruption could it?

Here's one question:

Given the fact that in iput(), the inode remains hashed and the inode->i_state does not change until after we've dropped the inode_lock, called write_inode_now(), and then retaken the inode_lock, exactly what is preventing a third party task from grabbing that inode?

(Better still: write_inode_now() itself actually calls __iget(), which could cause that inode to be plonked right back onto the "inode_in_use" list if ever refile_inode() gets called.)

So does the following patch help?

Cheers,
Trond

fs/inode.c renamed from fs/inode.c.orig +10 -4
@@ -1200,6 +1200,7 @@ void iput(struct inode *inode)
 		struct super_block *sb = inode->i_sb;
 		struct super_operations *op = NULL;
+again:
 		if (inode->i_state == I_CLEAR)
 			BUG();
@@ -1241,11 +1242,16 @@ void iput(struct inode *inode)
 				if (!(inode->i_state & (I_DIRTY|I_LOCK)))
 					__refile_inode(inode);
 				inodes_stat.nr_unused++;
-				spin_unlock(&inode_lock);
-				if (!sb || (sb->s_flags & MS_ACTIVE))
+				if (!sb || (sb->s_flags & MS_ACTIVE)) {
+					spin_unlock(&inode_lock);
 					return;
-				write_inode_now(inode, 1);
-				spin_lock(&inode_lock);
+				}
+				if (inode->i_state & I_DIRTY) {
+					__iget(inode);
+					spin_unlock(&inode_lock);
+					write_inode_now(inode, 1);
+					goto again;
+				}
 				inodes_stat.nr_unused--;
 				list_del_init(&inode->i_hash);
 			}
#3 lore raw

ty den 21.12.2004 Klokka 14:46 (-0200) skreiv Marcelo Tosatti:

Quoted 3 lines

Trond, the following issue noted by you could not generate such corruption could it?

I doubt it. That bug requires the MS_ACTIVE flag to have been cleared. IOW, you would have to be in the middle of unmounting the filesystem, and that doesn't really square with the fact that it is being exported by knfsd.

Cheers,
Trond

Signature · 1 line
Trond Myklebust <[email protected]>
#4 lore raw
In reply to Marcelo Tosatti
Quoted 40 lines

CCing the ReiserFS devel list because they might have some clue.

Quoted 7 lines · Manfred Schwarb

Hi,
yesterday, I got at the start of a rsync backup over nfs an Oops in find_inode/reiserfs_find_actor.
Some googling revealed that this seems to be an
inode list corruption, but how can this happen?
There seem to be quite some reports about similar problems, so doing rsync over nfs-mounted partitions might be a bad idea?

No, it should work.

Quoted 7 lines

Is there anything I can do? Any help is appreciated.

This machine is an athlon i386 with vanilla 2.4.29-pre2. Below you may find the ksymoopsified Oops:

Dec 19 21:00:01  kernel: Unable to handle kernel NULL pointer

dereference at

Quoted 16 lines

virtual address 00000028

Dec 19 21:00:01  kernel: c0153b09
Dec 19 21:00:01  kernel: *pde = 1d678067
Dec 19 21:00:01  kernel: Oops: 0000
Dec 19 21:00:01  kernel: CPU:    0
Dec 19 21:00:01  kernel: EIP:    0010:[find_inode+25/112]    Not tainted
Dec 19 21:00:01  kernel: EIP:    0010:[<c0153b09>]    Not tainted
Using defaults from ksymoops -t elf32-i386 -a i386
Dec 19 21:00:01  kernel: EFLAGS: 00213213
Dec 19 21:00:01  kernel: eax: e0b3c9e0   ebx: 00000000   ecx: 0000000f  
edx: dff80000
Dec 19 21:00:01  kernel: esi: 00000000   edi: dffa2a58   ebp: 000000fa  
esp: d7f05d60
Dec 19 21:00:01  kernel: ds: 0018   es: 0018   ss: 0018
Dec 19 21:00:01  kernel: Process nfsd (pid: 2372, stackpage=d7f05000)
Dec 19 21:00:01  kernel: Stack: 0efe42ff 00000000 00000002 d7f05de4
Quoted 11 lines

dffa2a58

Quoted 2 lines

000000fa da2f6000 c0153f4e

Dec 19 21:00:01  kernel:        da2f6000 000000fa dffa2a58 e0b3c9e0

d7f05db8

Quoted 2 lines

d7f05de4 d7f05e48 d7f05db8

Dec 19 21:00:01  kernel:        da2f6000 e0b3ca60 da2f6000 000000fa

e0b3c9e0

Quoted 3 lines

d7f05db8 00000011 d7f05de4

Dec 19 21:00:01  kernel: Call Trace:    [iget4_locked+94/272]

[keybdev:__insmod_keybdev_O/lib/modules/2.4.29-pre2/kernel/drivers/i+4288752096/96]

Quoted 2 lines
Quoted 1 line

[keybdev:__insmod_keybdev_O/lib/modules/2.4.29-pre2/kernel/drivers/i+4288752224/96]

Quoted 2 lines
Quoted 1 line

[keybdev:__insmod_keybdev_O/lib/modules/2.4.29-pre2/kernel/drivers/i+4288752096/96]

Quoted 2 lines
Quoted 1 line

[keybdev:__insmod_keybdev_O/lib/modules/2.4.29-pre2/kernel/drivers/i+4288731921/96]

Quoted 56 lines
Quoted 8 lines
Dec 19 21:00:01  kernel: Call Trace:    [<c0153f4e>] [<e0b3c9e0>]
[<e0b3ca60>] [<e0b3c9e0>] [<e0b37b11>]
Dec 19 21:00:01  kernel:   [<c0151d1c>] [<c01491cf>] [<c0149279>]
[<e118fd71>] [<e1196d39>] [<e119de8c>]
Dec 19 21:00:01  kernel:   [<e118c68d>] [<c027ab3e>] [<e119de8c>]
[<e119d758>] [<e119d778>] [<e118c3cb>]
Dec 19 21:00:01  kernel:   [<c010729b>] [<e118c210>]
Dec 19 21:00:01  kernel: Code: 39 6b 28 89 de 75 f1 8b 44 24 20 39 83 a0

00

Quoted 30 lines

00 00 75 e5 8b

Quoted 6 lines
Quoted 6 lines

EIP; c0153b09 <find_inode+19/70> <=====

eax; e0b3c9e0 <[reiserfs]reiserfs_find_actor+0/40> edx; dff80000 <_end+1fbfd7f4/20792854> edi; dffa2a58 <_end+1fc2024c/20792854> esp; d7f05d60 <_end+17b83554/20792854>

Trace; c0153f4e <iget4_locked+5e/110>
Trace; e0b3c9e0 <[reiserfs]reiserfs_find_actor+0/40> Trace; e0b3ca60 <[reiserfs]reiserfs_iget+40/c0> Trace; e0b3c9e0 <[reiserfs]reiserfs_find_actor+0/40> Trace; e0b37b11 <[reiserfs]reiserfs_lookup+101/120> Trace; c0151d1c <d_alloc+1c/1d0>
Trace; c01491cf <lookup_hash+9f/d0>
Trace; c0149279 <lookup_one_len+79/90> Trace; e118fd71 <[nfsd]nfsd_lookup+d1/490> Trace; e1196d39 <[nfsd]nfsd3_proc_lookup+a9/140> Trace; e119de8c <[nfsd]nfsd_procedures3+6c/320> Trace; e118c68d <[nfsd]nfsd_dispatch+14d/220> Trace; c027ab3e <svc_process+3de/590>
Trace; e119de8c <[nfsd]nfsd_procedures3+6c/320> Trace; e119d758 <[nfsd]nfsd_version3+0/10> Trace; e119d778 <[nfsd]nfsd_program+0/28> Trace; e118c3cb <[nfsd]nfsd+1bb/330>
Trace; c010729b <arch_kernel_thread+2b/40> Trace; e118c210 <[nfsd]nfsd+0/330>

Quoted 16 lines

Code; c0153b09 <find_inode+19/70>
00000000 <_EIP>:
Code; c0153b09 <find_inode+19/70> <=====
0: 39 6b 28 cmp %ebp,0x28(%ebx) <=====
Code; c0153b0c <find_inode+1c/70>
3: 89 de mov %ebx,%esi
Code; c0153b0e <find_inode+1e/70>
5: 75 f1 jne fffffff8 <_EIP+0xfffffff8>
Code; c0153b10 <find_inode+20/70>
7: 8b 44 24 20 mov 0x20(%esp,1),%eax
Code; c0153b14 <find_inode+24/70>
b: 39 83 a0 00 00 00 cmp %eax,0xa0(%ebx)
Code; c0153b1a <find_inode+2a/70>
11: 75 e5 jne fffffff8 <_EIP+0xfffffff8>
Code; c0153b1c <find_inode+2c/70>
13: 8b 00 mov (%eax),%eax

Quoted 7 lines

This is indeed corruption - an inode in this hash bucket has "->next" as NULL, so find_inode goes boom.

Something is leaving this hash bucket list corrupt.

Have you ever seen this crash before ? Can you reproduce it?

No, not at all. This machine was running for 10 months with 2.4.xx kernels without any problems. Since this oops, I tried to reproduce the particular situation (rsync over nfs, and some additional load), put I had no success in crashing the box:
I mirrored the box (ca. 1 million files) 5 times, no problem.

Quoted 6 lines

The only inode corruption case that was reliable was from Chris Caputo and we end up agreeing that it was most likely a hardware issue, because it was
hard to reproduce and strange in several ways. Can you point me at the "quite some similar reports" you have found, please ?

Just my first impression, a closer look showed that most of the cases group around 2.4.1[789] because of lacking reiserfs_find_actor. Sorry for the overstatement.

A recent oops report shows some similarity, using a 2.6.7 kernel:
http://marc.theaimsgroup.com/?l=linux-kernel&m=109278828905885&w=2

Hardware issue: you mean memory? Last winter I ran memtest86 during a weekend, everything was fine. At the moment I can't take this box offline for a longer period to test again, so I tend to belive memory is ok, and knock on wood...

Signature · 2 lines
+++ Sparen Sie mit GMX DSL +++ http://www.gmx.net/de/go/dsl
AKTION für Wechsler: DSL-Tarife ab 3,99 EUR/Monat + Startguthaben
#5 lore raw

Hi,

Quoted 5 lines · Manfred Schwarb

No, not at all. This machine was running for 10 months with 2.4.xx kernels without any problems. Since this oops, I tried to reproduce the particular situation (rsync over nfs, and some additional load), put I had no success in crashing the box:
I mirrored the box (ca. 1 million files) 5 times, no problem.

(...)

Quoted 4 lines

Hardware issue: you mean memory? Last winter I ran memtest86 during a weekend, everything was fine. At the moment I can't take this box offline for a longer period to test again, so I tend to belive memory is ok, and knock on wood...

Does this machine have ECC memory ? If so, have you tried to check the error counters ? And if not, well, it might be some random bit flips in memory.

Last year, I encountered a really strange situation. During about one week, I had 3 or 4 machines that have panic'd several times each, and sometimes only some multi-threaded apps would freeze (eg: pdnsd). These machines were installed at different customers' in different locations, and at different times. They had never crashed before, nor did they after. Most other similar machines at other customer's, or some at the sames did not experience this. This was during the first week of november, when there were the very strong solar flares. I never told the customers about my thoughts on the subject, they would have definitely made fun of me. So we replaced the RAM sticks, to report something more believable...

Regards,
Willy

#6 lore raw
In reply to Marcelo Tosatti
Quoted 40 lines

CCing the ReiserFS devel list because they might have some clue.

Quoted 7 lines · Manfred Schwarb

Hi,
yesterday, I got at the start of a rsync backup over nfs an Oops in find_inode/reiserfs_find_actor.
Some googling revealed that this seems to be an
inode list corruption, but how can this happen?
There seem to be quite some reports about similar problems, so doing rsync over nfs-mounted partitions might be a bad idea?

No, it should work.

Quoted 7 lines

Is there anything I can do? Any help is appreciated.

This machine is an athlon i386 with vanilla 2.4.29-pre2. Below you may find the ksymoopsified Oops:

Dec 19 21:00:01  kernel: Unable to handle kernel NULL pointer

dereference at

Quoted 16 lines

virtual address 00000028

Dec 19 21:00:01  kernel: c0153b09
Dec 19 21:00:01  kernel: *pde = 1d678067
Dec 19 21:00:01  kernel: Oops: 0000
Dec 19 21:00:01  kernel: CPU:    0
Dec 19 21:00:01  kernel: EIP:    0010:[find_inode+25/112]    Not tainted
Dec 19 21:00:01  kernel: EIP:    0010:[<c0153b09>]    Not tainted
Using defaults from ksymoops -t elf32-i386 -a i386
Dec 19 21:00:01  kernel: EFLAGS: 00213213
Dec 19 21:00:01  kernel: eax: e0b3c9e0   ebx: 00000000   ecx: 0000000f  
edx: dff80000
Dec 19 21:00:01  kernel: esi: 00000000   edi: dffa2a58   ebp: 000000fa  
esp: d7f05d60
Dec 19 21:00:01  kernel: ds: 0018   es: 0018   ss: 0018
Dec 19 21:00:01  kernel: Process nfsd (pid: 2372, stackpage=d7f05000)
Dec 19 21:00:01  kernel: Stack: 0efe42ff 00000000 00000002 d7f05de4
Quoted 11 lines

dffa2a58

Quoted 2 lines

000000fa da2f6000 c0153f4e

Dec 19 21:00:01  kernel:        da2f6000 000000fa dffa2a58 e0b3c9e0

d7f05db8

Quoted 2 lines

d7f05de4 d7f05e48 d7f05db8

Dec 19 21:00:01  kernel:        da2f6000 e0b3ca60 da2f6000 000000fa

e0b3c9e0

Quoted 3 lines

d7f05db8 00000011 d7f05de4

Dec 19 21:00:01  kernel: Call Trace:    [iget4_locked+94/272]

[keybdev:__insmod_keybdev_O/lib/modules/2.4.29-pre2/kernel/drivers/i+4288752096/96]

Quoted 2 lines
Quoted 1 line

[keybdev:__insmod_keybdev_O/lib/modules/2.4.29-pre2/kernel/drivers/i+4288752224/96]

Quoted 2 lines
Quoted 1 line

[keybdev:__insmod_keybdev_O/lib/modules/2.4.29-pre2/kernel/drivers/i+4288752096/96]

Quoted 2 lines
Quoted 1 line

[keybdev:__insmod_keybdev_O/lib/modules/2.4.29-pre2/kernel/drivers/i+4288731921/96]

Quoted 56 lines
Quoted 8 lines
Dec 19 21:00:01  kernel: Call Trace:    [<c0153f4e>] [<e0b3c9e0>]
[<e0b3ca60>] [<e0b3c9e0>] [<e0b37b11>]
Dec 19 21:00:01  kernel:   [<c0151d1c>] [<c01491cf>] [<c0149279>]
[<e118fd71>] [<e1196d39>] [<e119de8c>]
Dec 19 21:00:01  kernel:   [<e118c68d>] [<c027ab3e>] [<e119de8c>]
[<e119d758>] [<e119d778>] [<e118c3cb>]
Dec 19 21:00:01  kernel:   [<c010729b>] [<e118c210>]
Dec 19 21:00:01  kernel: Code: 39 6b 28 89 de 75 f1 8b 44 24 20 39 83 a0

00

Quoted 30 lines

00 00 75 e5 8b

Quoted 6 lines
Quoted 6 lines

EIP; c0153b09 <find_inode+19/70> <=====

eax; e0b3c9e0 <[reiserfs]reiserfs_find_actor+0/40> edx; dff80000 <_end+1fbfd7f4/20792854> edi; dffa2a58 <_end+1fc2024c/20792854> esp; d7f05d60 <_end+17b83554/20792854>

Trace; c0153f4e <iget4_locked+5e/110>
Trace; e0b3c9e0 <[reiserfs]reiserfs_find_actor+0/40> Trace; e0b3ca60 <[reiserfs]reiserfs_iget+40/c0> Trace; e0b3c9e0 <[reiserfs]reiserfs_find_actor+0/40> Trace; e0b37b11 <[reiserfs]reiserfs_lookup+101/120> Trace; c0151d1c <d_alloc+1c/1d0>
Trace; c01491cf <lookup_hash+9f/d0>
Trace; c0149279 <lookup_one_len+79/90> Trace; e118fd71 <[nfsd]nfsd_lookup+d1/490> Trace; e1196d39 <[nfsd]nfsd3_proc_lookup+a9/140> Trace; e119de8c <[nfsd]nfsd_procedures3+6c/320> Trace; e118c68d <[nfsd]nfsd_dispatch+14d/220> Trace; c027ab3e <svc_process+3de/590>
Trace; e119de8c <[nfsd]nfsd_procedures3+6c/320> Trace; e119d758 <[nfsd]nfsd_version3+0/10> Trace; e119d778 <[nfsd]nfsd_program+0/28> Trace; e118c3cb <[nfsd]nfsd+1bb/330>
Trace; c010729b <arch_kernel_thread+2b/40> Trace; e118c210 <[nfsd]nfsd+0/330>

Quoted 16 lines

Code; c0153b09 <find_inode+19/70>
00000000 <_EIP>:
Code; c0153b09 <find_inode+19/70> <=====
0: 39 6b 28 cmp %ebp,0x28(%ebx) <=====
Code; c0153b0c <find_inode+1c/70>
3: 89 de mov %ebx,%esi
Code; c0153b0e <find_inode+1e/70>
5: 75 f1 jne fffffff8 <_EIP+0xfffffff8>
Code; c0153b10 <find_inode+20/70>
7: 8b 44 24 20 mov 0x20(%esp,1),%eax
Code; c0153b14 <find_inode+24/70>
b: 39 83 a0 00 00 00 cmp %eax,0xa0(%ebx)
Code; c0153b1a <find_inode+2a/70>
11: 75 e5 jne fffffff8 <_EIP+0xfffffff8>
Code; c0153b1c <find_inode+2c/70>
13: 8b 00 mov (%eax),%eax

Quoted 7 lines

This is indeed corruption - an inode in this hash bucket has "->next" as NULL, so find_inode goes boom.

Something is leaving this hash bucket list corrupt.

Have you ever seen this crash before ? Can you reproduce it?

No, not at all. This machine was running for 10 months with 2.4.xx kernels without any problems. Since this oops, I tried to reproduce the particular situation (rsync over nfs, and some additional load), put I had no success in crashing the box:
I mirrored the box (ca. 1 million files) 5 times, no problem.

Quoted 6 lines

The only inode corruption case that was reliable was from Chris Caputo and we end up agreeing that it was most likely a hardware issue, because it was
hard to reproduce and strange in several ways. Can you point me at the "quite some similar reports" you have found, please ?

Just my first impression, a closer look showed that most of the cases group around 2.4.1[789] because of lacking reiserfs_find_actor. Sorry for the overstatement.

A recent oops report shows some similarity, using a 2.6.7 kernel:
http://marc.theaimsgroup.com/?l=linux-kernel&m=109278828905885&w=2

Hardware issue: you mean memory? Last winter I ran memtest86 during a weekend, everything was fine. At the moment I can't take this box offline for a longer period to test again, so I tend to belive memory is ok, and knock on wood...

Signature · 2 lines
Psssst! Mit GMX Handyrechnung senken: http://www.gmx.net/de/go/mail
100 FreeSMS/Monat (GMX TopMail), 50 (GMX ProMail), 10 (GMX FreeMail)
#7 lore raw
Quoted 70 lines · Manfred Schwarb
Quoted 44 lines
Quoted 27 lines
Quoted 6 lines
Quoted 6 lines

EIP; c0153b09 <find_inode+19/70> <=====

eax; e0b3c9e0 <[reiserfs]reiserfs_find_actor+0/40> edx; dff80000 <_end+1fbfd7f4/20792854> edi; dffa2a58 <_end+1fc2024c/20792854> esp; d7f05d60 <_end+17b83554/20792854>

Trace; c0153f4e <iget4_locked+5e/110>
Trace; e0b3c9e0 <[reiserfs]reiserfs_find_actor+0/40> Trace; e0b3ca60 <[reiserfs]reiserfs_iget+40/c0> Trace; e0b3c9e0 <[reiserfs]reiserfs_find_actor+0/40> Trace; e0b37b11 <[reiserfs]reiserfs_lookup+101/120> Trace; c0151d1c <d_alloc+1c/1d0>
Trace; c01491cf <lookup_hash+9f/d0>
Trace; c0149279 <lookup_one_len+79/90> Trace; e118fd71 <[nfsd]nfsd_lookup+d1/490> Trace; e1196d39 <[nfsd]nfsd3_proc_lookup+a9/140> Trace; e119de8c <[nfsd]nfsd_procedures3+6c/320> Trace; e118c68d <[nfsd]nfsd_dispatch+14d/220> Trace; c027ab3e <svc_process+3de/590>
Trace; e119de8c <[nfsd]nfsd_procedures3+6c/320> Trace; e119d758 <[nfsd]nfsd_version3+0/10> Trace; e119d778 <[nfsd]nfsd_program+0/28> Trace; e118c3cb <[nfsd]nfsd+1bb/330>
Trace; c010729b <arch_kernel_thread+2b/40> Trace; e118c210 <[nfsd]nfsd+0/330>

Quoted 16 lines

Code; c0153b09 <find_inode+19/70>
00000000 <_EIP>:
Code; c0153b09 <find_inode+19/70> <=====
0: 39 6b 28 cmp %ebp,0x28(%ebx) <=====
Code; c0153b0c <find_inode+1c/70>
3: 89 de mov %ebx,%esi
Code; c0153b0e <find_inode+1e/70>
5: 75 f1 jne fffffff8 <_EIP+0xfffffff8>
Code; c0153b10 <find_inode+20/70>
7: 8b 44 24 20 mov 0x20(%esp,1),%eax
Code; c0153b14 <find_inode+24/70>
b: 39 83 a0 00 00 00 cmp %eax,0xa0(%ebx)
Code; c0153b1a <find_inode+2a/70>
11: 75 e5 jne fffffff8 <_EIP+0xfffffff8>
Code; c0153b1c <find_inode+2c/70>
13: 8b 00 mov (%eax),%eax

Quoted 7 lines

This is indeed corruption - an inode in this hash bucket has "->next" as NULL, so find_inode goes boom.

Something is leaving this hash bucket list corrupt.

Have you ever seen this crash before ? Can you reproduce it?

No, not at all. This machine was running for 10 months with 2.4.xx kernels without any problems. Since this oops, I tried to reproduce the particular situation (rsync over nfs, and some additional load), put I had no success in crashing the box:
I mirrored the box (ca. 1 million files) 5 times, no problem.

Quoted 6 lines

The only inode corruption case that was reliable was from Chris Caputo and we end up agreeing that it was most likely a hardware issue, because it was
hard to reproduce and strange in several ways. Can you point me at the "quite some similar reports" you have found, please ?

Just my first impression, a closer look showed that most of the cases group around 2.4.1[789] because of lacking reiserfs_find_actor. Sorry for the overstatement.

Quoted 8 lines

A recent oops report shows some similarity, using a 2.6.7 kernel:
http://marc.theaimsgroup.com/?l=linux-kernel&m=109278828905885&w=2

Hardware issue: you mean memory? Last winter I ran memtest86 during a weekend, everything was fine. At the moment I can't take this box offline for a longer period to test again, so I tend to belive memory is ok, and knock on wood...

Yes, what I'm saying is that no reliable inode corruption case has been reported recently, except when hardware was flaky (usually memory errors).

I'm not saying that this is your case - its just one explanation to the problem. It might well be a software problem.

Hans, did any of your developers see similar inode cache hashtable corruption in v2.4.x?