Skip to contentSkip to conversation

XFS crashed twice, once in 2.6.16.20, next in 2.6.17, reproducable

DISCUSSION Avuton Olrich 6 participants 17 messages Jun 19, 2006 – Jun 25, 2006

lore
Share & tools
lore
#0 lore raw

Didn't make it to the linux-xfs list, it's not working so I'll try by sending it both to the LKML and linux-xfs again.

Hello, when trying to recursively delete a directory (same directory twice) from my 500gb hard drive I get a problem. It crashed first in 2.6.16.20, then I upgraded to try to get rid of the issue. This one is from 2.6.17:

xfs_da_do_buf: bno 16777216
dir: inode 1507133580
Filesystem "sda1": XFS internal error xfs_da_do_buf(1) at line 2119 of file /usr/src/linux-stable-cold/fs/xfs/xfs_da_btree.c. Caller 0xb01 d9b63
<b01d9720> xfs_da_do_buf+0x40e/0x7c7 <b01d9b63> xfs_da_read_buf+0x30/0x35 <b01e43d5> xfs_dir2_leafn_lookup_int+0x2f3/0x453 <b01d9b63>

xfs_da_read_buf+0x30/0x35
<b01e2ba5> xfs_dir2_node_removename+0x288/0x47f  <b01e2ba5>
xfs_dir2_node_removename+0x288/0x47f
<b01ddbd3> xfs_dir2_removename+0xce/0xd5  <b020ff5d> kmem_zone_alloc+0x4d/0x98
<b020d0ef> xfs_remove+0x2ac/0x444  <b0215e7f> xfs_vn_unlink+0x17/0x3b
<b020a32b> xfs_lookup+0x6e/0x78  <b011e734> __capable+0xc/0x1f
<b0155827> generic_permission+0x93/0xcc  <b01558f8> permission+0x98/0xa4
<b0155da0> may_delete+0x32/0xe9  <b0156243> vfs_unlink+0x6d/0xa3
<b0157c7a> do_unlinkat+0x92/0x125  <b0159a0d> sys_getdents64+0x9c/0xa6
<b0102b67> sysenter_past_esp+0x54/0x75

Filesystem "sda1": XFS internal error xfs_trans_cancel at line 1150 of file /usr/src/linux-stable-cold/fs/xfs/xfs_trans.c. Caller 0xb020d2 5e
<b0204b48> xfs_trans_cancel+0x59/0xe5 <b020d25e> xfs_remove+0x41b/0x444 <b020d25e> xfs_remove+0x41b/0x444 <b0215e7f> xfs_vn_unlink+0x17/0x3b <b020a32b> xfs_lookup+0x6e/0x78 <b011e734> __capable+0xc/0x1f <b0155827> generic_permission+0x93/0xcc <b01558f8> permission+0x98/0xa4 <b0155da0> may_delete+0x32/0xe9 <b0156243> vfs_unlink+0x6d/0xa3 <b0157c7a> do_unlinkat+0x92/0x125 <b0159a0d> sys_getdents64+0x9c/0xa6 <b0102b67> sysenter_past_esp+0x54/0x75 xfs_force_shutdown(sda1,0x8) called from line 1151 of file /usr/src/linux-stable-cold/fs/xfs/xfs_trans.c. Return address = 0xb0218b68
Filesystem "sda1": Corruption of in-memory data detected. Shutting down filesystem: sda1

While trying to xfs_repair I get the following:
fatal error -- can't read block 16777216 for directory inode 1507133580

Badblocks has been run on this machine and it was sucessful.

I did find an old thread with this, but no solution:
http://oss.sgi.com/archives/xfs/2005-02/msg00067.html

config:
http://olricha.homelinux.net:8080/config.gz

Thanks for any help. If I can help at all please let me know.

Signature · 3 lines
avuton
--
Anyone who quotes me in their sig is an idiot. -- Rusty Russell.
#1 lore raw

"Avuton Olrich" <[email protected]> writes:

The same here.
after a complete mkfs.xfs under 2.6.17-rc6 it was solved.

Same if i boot 2.6.8 mk.xfs, boot into 2.6.16 the xfs get "shreddered" a directly boot from .8 to .17-rc6 works. so i think there was a bug in .16 in the transition of the xfs wich got solved somewhere in the .17.rc? time.

Quoted 2 lines

Filesystem "sda1": Corruption of in-memory data detected. Shutting down filesystem: sda1

Daniel

#2 lore raw
In reply to Avuton Olrich
Quoted 5 lines · Avuton Olrich

..
Hello, when trying to recursively delete a directory (same directory twice) from my 500gb hard drive I get a problem. It crashed first in 2.6.16.20, then I upgraded to try to get rid of the issue. This one is from 2.6.17:

How reproducible is it? Is it reproducible even after xfs_repair?

If so, can you try Mandy's patch below, to see if it is addressing the root cause of your problem? If problems persist, a reproducible test case would be wonderful, if one can be found..

cheers.

Signature · 24 lines
Nathan


Fix nused counter.  It's currently getting set to -1 rather than getting
decremented by 1.  Since nused never reaches 0, the "if (!free->hdr.nused)"
check in xfs_dir2_leafn_remove() fails every time and xfs_dir2_shrink_inode()
doesn't get called when it should.  This causes extra blocks to be left on
an empty directory and the directory in unable to be converted back to
inline extent mode.

Signed-off-by: Mandy Kirkconnell <[email protected]>
Signed-off-by: Nathan Scott <[email protected]>

--- a/fs/xfs/xfs_dir2_node.c	2006-06-20 16:00:45.000000000 +1000
+++ b/fs/xfs/xfs_dir2_node.c	2006-06-20 16:00:45.000000000 +1000
@@ -972,7 +972,7 @@ xfs_dir2_leafn_remove(
 			/*
 			 * One less used entry in the free table.
 			 */
-			free->hdr.nused = cpu_to_be32(-1);
+			be32_add(&free->hdr.nused, -1);
 			xfs_dir2_free_log_header(tp, fbp);
 			/*
 			 * If this was the last entry in the table, we can
#3 lore raw
Quoted 8 lines · Nathan Scott
Quoted 5 lines · Avuton Olrich

..
Hello, when trying to recursively delete a directory (same directory twice) from my 500gb hard drive I get a problem. It crashed first in 2.6.16.20, then I upgraded to try to get rid of the issue. This one is from 2.6.17:

How reproducible is it? Is it reproducible even after xfs_repair?

Happens every time I try to remove that inode (directory). xfs_repair ends with a fatal error:

Phase 6 - check inode connectivity...
- resetting contents of realtime bitmap and summary inodes
- ensuring existence of lost+found directory
- traversing filesystem starting at / ...
rebuilding directory inode 128

fatal error -- can't read block 16777216 for directory inode 1507133580

Quoted 3 lines

If so, can you try Mandy's patch below, to see if it is addressing the root cause of your problem? If problems persist, a reproducible test case would be wonderful, if one can be found..

I'm sorry, the patch doesn't change anything. It never makes it though the xfs_repair due to the above error. If there's any information I can get for you please let me know.

I'm not sure if it changes anything, but here's the message after the patch:
xfs_da_do_buf: bno 16777216
dir: inode 1507133580
Filesystem "sda1": XFS internal error xfs_da_do_buf(1) at line 2119 of file /usr/src/linux-stable-cold/fs/xfs/xfs_da_btree.c. Caller 0xb01d9b63
<b01d9720> xfs_da_do_buf+0x40e/0x7c7 <b01d9b63> xfs_da_read_buf+0x30/0x35 <b01e43d9> xfs_dir2_leafn_lookup_int+0x2f3/0x453 <b01d9b63>

xfs_da_read_buf+0x30/0x35
<b01e2ba5> xfs_dir2_node_removename+0x288/0x483  <b01e2ba5>
xfs_dir2_node_removename+0x288/0x483
<b01ddbd3> xfs_dir2_removename+0xce/0xd5  <b020ff61> kmem_zone_alloc+0x4d/0x98
<b020d0f3> xfs_remove+0x2ac/0x444  <b0215e83> xfs_vn_unlink+0x17/0x3b
<b016190c> mntput_no_expire+0x11/0x7e  <b01575f1> link_path_walk+0xaf/0xb9
<b011e734> __capable+0xc/0x1f  <b0155827> generic_permission+0x93/0xcc
<b01558f8> permission+0x98/0xa4  <b0155da0> may_delete+0x32/0xe9
<b0156243> vfs_unlink+0x6d/0xa3  <b0157c7a> do_unlinkat+0x92/0x125
<b0159a0d> sys_getdents64+0x9c/0xa6  <b0102b67> sysenter_past_esp+0x54/0x75

Filesystem "sda1": XFS internal error xfs_trans_cancel at line 1150 of file /usr/src/linux-stable-cold/fs/xfs/xfs_trans.c. Caller 0xb020d262 <b0204b4c> xfs_trans_cancel+0x59/0xe5 <b020d262> xfs_remove+0x41b/0x444 <b020d262> xfs_remove+0x41b/0x444 <b0215e83> xfs_vn_unlink+0x17/0x3b <b016190c> mntput_no_expire+0x11/0x7e <b01575f1> link_path_walk+0xaf/0xb9 <b011e734> __capable+0xc/0x1f <b0155827> generic_permission+0x93/0xcc <b01558f8> permission+0x98/0xa4 <b0155da0> may_delete+0x32/0xe9 <b0156243> vfs_unlink+0x6d/0xa3 <b0157c7a> do_unlinkat+0x92/0x125 <b0159a0d> sys_getdents64+0x9c/0xa6 <b0102b67> sysenter_past_esp+0x54/0x75 xfs_force_shutdown(sda1,0x8) called from line 1151 of file /usr/src/linux-stable-cold/fs/xfs/xfs_trans.c. Return address = 0xb0218b6c
Filesystem "sda1": Corruption of in-memory data detected. Shutting down filesystem: sda1
Please umount the filesystem, and rectify the problem(s) xfs_force_shutdown(sda1,0x1) called from line 338 of file /usr/src/linux-stable-cold/fs/xfs/xfs_rw.c. Return address = 0xb0218b6c

Signature · 3 lines
avuton
--
Anyone who quotes me in their sig is an idiot. -- Rusty Russell.
#4 lore raw
Quoted 8 lines · Avuton Olrich
Quoted 3 lines · Nathan Scott

If so, can you try Mandy's patch below, to see if it is addressing the root cause of your problem? If problems persist, a reproducible test case would be wonderful, if one can be found..

I'm sorry, the patch doesn't change anything. It never makes it though the xfs_repair due to the above error. If there's any information I can get for you please let me know.

Oh - thats a kernel patch, not a repair patch, I was more interested in whether the initial corruption could be reproduced. Which version of xfs_repair are you running? (xfs_repair -V) xfsprogs-2.7.18 will resolve your problem, I suspect.

cheers.

Signature · 1 line
Nathan
#5 lore raw
Quoted 4 lines · Nathan Scott

Oh - thats a kernel patch, not a repair patch, I was more interested in whether the initial corruption could be reproduced. Which version of xfs_repair are you running? (xfs_repair -V) xfsprogs-2.7.18 will resolve your problem, I suspect.

OK, I'm running Gentoo's latest: 2.7.11, I can't find 2.7.18 _anywhere_ although 2.7.13 is in the pre directory on the ftp, is that the one you're referring to?

Signature · 3 lines
avuton
--
Anyone who quotes me in their sig is an idiot. -- Rusty Russell.
#6 lore raw
Quoted 9 lines · Avuton Olrich
Quoted 4 lines · Nathan Scott

Oh - thats a kernel patch, not a repair patch, I was more interested in whether the initial corruption could be reproduced. Which version of xfs_repair are you running? (xfs_repair -V) xfsprogs-2.7.18 will resolve your problem, I suspect.

OK, I'm running Gentoo's latest: 2.7.11, I can't find 2.7.18 _anywhere_ although 2.7.13 is in the pre directory on the ftp, is that the one you're referring to?

No - its in CVS (for a long time); I'll go get the ftp area updated, looks like thats been forgotten about again.

cheers.

Signature · 1 line
Nathan
#7 lore raw
Quoted 7 lines · Nathan Scott
Quoted 6 lines · Avuton Olrich
Quoted 4 lines · Nathan Scott

Oh - thats a kernel patch, not a repair patch, I was more interested in whether the initial corruption could be reproduced. Which version of xfs_repair are you running? (xfs_repair -V) xfsprogs-2.7.18 will resolve your problem, I suspect.

Quoted 6 lines
Quoted 3 lines

OK, I'm running Gentoo's latest: 2.7.11, I can't find 2.7.18 _anywhere_ although 2.7.13 is in the pre directory on the ftp, is that the one you're referring to?

No - its in CVS (for a long time); I'll go get the ftp area updated, looks like thats been forgotten about again.

OK, just compiled from CVS HEAD (xfs_repair 2.8.2) and it still fails:

If this fix is not yet in the 2.8.x I will wait for 2.7.18 to get on the ftp.

Phase 1 - find and verify superblock...
Phase 2 - using internal log
- zero log...
- scan filesystem freespace and inode maps...
- found root inode chunk
Phase 3 - for each AG...
- scan and clear agi unlinked lists...
- process known inodes and perform inode discovery...
- agno = 0
entry "/ost+found" at block 0 offset 448 in directory inode 128 references invalid inode 18374686479671623679
clearing inode number in entry at offset 448...
entry at block 0 offset 448 in directory inode 128 has illegal name "/ost+found": imap claims a free inode 859505 is in use, correcting imap and clearing inode
- agno = 1
- agno = 2
- agno = 3
- agno = 4
- agno = 5
- agno = 6
- agno = 7
- agno = 8
- agno = 9
- agno = 10
- agno = 11
- agno = 12
- agno = 13
- agno = 14
- agno = 15
- process newly discovered inodes...
Phase 4 - check for duplicate blocks...
- setting up duplicate extent list...
- clear lost+found (if it exists) ...
- check for inodes claiming duplicate blocks...
- agno = 0
- agno = 1
- agno = 2
- agno = 3
- agno = 4
- agno = 5
- agno = 6
- agno = 7
- agno = 8
- agno = 9
- agno = 10
- agno = 11
- agno = 12
- agno = 13
- agno = 14
- agno = 15
Phase 5 - rebuild AG headers and trees...
- reset superblock...
Phase 6 - check inode connectivity...
- resetting contents of realtime bitmap and summary inodes
- ensuring existence of lost+found directory
- traversing filesystem starting at / ...
rebuilding directory inode 128

fatal error -- can't read block 16777216 for directory inode 1507133580

Signature · 3 lines
avuton
--
Anyone who quotes me in their sig is an idiot. -- Rusty Russell.
#8 lore raw
Quoted 1 line

fatal error -- can't read block 16777216 for directory inode 1507133580

This looks to be the same problem as http://oss.sgi.com/bugzilla/show_bug.cgi?id=631
Note that the block numbers are identical in both reports: 16777216 = 0x1000000. A very suspicious block number, wouldn't you say?

Best wishes,

Duncan.

#9 lore raw
In reply to Avuton Olrich
Quoted 3 lines · Avuton Olrich
Quoted 2 lines · Nathan Scott

No - its in CVS (for a long time); I'll go get the ftp area updated, looks like thats been forgotten about again.

FWIW, I've updated the ftp area now.

Quoted 1 line

OK, just compiled from CVS HEAD (xfs_repair 2.8.2) and it still fails:

Is this a large filesystem? Any chance we can get access to it somehow (e.g. xfs_copy to a sparse file, then send me a pointer to it) to reproduce the problem locally?

Quoted 2 lines

fatal error -- can't read block 16777216 for directory inode 1507133580

Once you save a copy of it for further analysis of xfs_repair, if you can, you can clear out this problem by directly poking at the device using xfs_db in expert mode. "xfs_db -x /dev/xxx"; then "inode 1507133580"; then "write core.mode 0"; and then try another xfs_repair run. Please try capture the fs for us first though (if possible) else we're going to struggle to improve on this aspect of xfs_repair. Send me some private mail if you do manage to grab the fs and put it someplace for me.

thanks.

Signature · 1 line
Nathan
#10 lore raw
In reply to Avuton Olrich

I just got a new XFS crash running 2.6.17, again with problems at block 16777216 - I'll try to make a copy of the corrupted filesystem available. Interestingly enough, I'm also seeing ext3 corruption. The usual manifestation is that a program fails to run, with a message about it not being in executable format (if it happens again I will take a note of the exact message). I've had no problems at all with 2.6.17. It seems to be happening randomly, which makes me suspect a race condition (uniprocessor machine, but preemptable kernel), or memory corruption. I will rebuild the kernel with all kernel debugging options turned on, once I recover the filesystem.

Ciao,

D.

#11 lore raw
Quoted 10 lines · Duncan Sands

I just got a new XFS crash running 2.6.17, again with problems at block 16777216 - I'll try to make a copy of the corrupted filesystem available. Interestingly enough, I'm also seeing ext3 corruption. The usual manifestation is that a program fails to run, with a message about it not being in executable format (if it happens again I will take a note of the exact message). I've had no problems at all with 2.6.17. It seems to be happening randomly, which makes me suspect a race condition (uniprocessor machine, but preemptable kernel), or memory corruption. I will rebuild the kernel with all kernel debugging options turned on, once I recover the filesystem.

Sorry, that should say: "I've had no problems at all with 2.6.15". Also, xfs_repair successfully repaired the filesystem this time. I've kept a copy of the filesystem in case anyone is interested.

Duncan.

#12 lore raw
In reply to Nathan Scott
Quoted 1 line · Nathan Scott

How reproducible is it? Is it reproducible even after xfs_repair?

It happens everytime I try to delete the directory.

Also, forgot to mention I ran xfs_check on it and it gave me more information than I had before:
More information, ran xfs_check and got the following:
missing free index for data block 0 in dir ino 1507133580 missing free index for data block 2 in dir ino 1507133580 missing free index for data block 3 in dir ino 1507133580 missing free index for data block 4 in dir ino 1507133580 missing free index for data block 5 in dir ino 1507133580 missing free index for data block 6 in dir ino 1507133580 missing free index for data block 7 in dir ino 1507133580 missing free index for data block 8 in dir ino 1507133580 missing free index for data block 9 in dir ino 1507133580

Signature · 3 lines
avuton
--
Anyone who quotes me in their sig is an idiot. -- Rusty Russell.
#13 lore raw

Have you checked to make sure you don't have a bad disk?

Quoted 23 lines · Avuton Olrich
Quoted 1 line · Nathan Scott

How reproducible is it? Is it reproducible even after xfs_repair?

It happens everytime I try to delete the directory.

Also, forgot to mention I ran xfs_check on it and it gave me more information than I had before:
More information, ran xfs_check and got the following:
missing free index for data block 0 in dir ino 1507133580 missing free index for data block 2 in dir ino 1507133580 missing free index for data block 3 in dir ino 1507133580 missing free index for data block 4 in dir ino 1507133580 missing free index for data block 5 in dir ino 1507133580 missing free index for data block 6 in dir ino 1507133580 missing free index for data block 7 in dir ino 1507133580 missing free index for data block 8 in dir ino 1507133580 missing free index for data block 9 in dir ino 1507133580

Signature · 3 lines
avuton
--
Anyone who quotes me in their sig is an idiot. -- Rusty Russell.
#14 lore raw
Quoted 1 line · Justin Piszcz

Have you checked to make sure you don't have a bad disk?

In the initial email I do state that I have run badblocks on this disk sucessfully.

Signature · 3 lines
avuton
--
Anyone who quotes me in their sig is an idiot. -- Rusty Russell.
#15 lore raw

WHat options did you pass to bad blocks?

Quoted 10 lines · Avuton Olrich
Quoted 1 line · Justin Piszcz

Have you checked to make sure you don't have a bad disk?

In the initial email I do state that I have run badblocks on this disk sucessfully.

Signature · 3 lines
avuton
--
Anyone who quotes me in their sig is an idiot. -- Rusty Russell.
#16 lore raw
Quoted 1 line · Justin Piszcz

WHat options did you pass to bad blocks?

Just the defaults, but it doesn't matter, someone else is having the same exact issue I am, from the bugzilla entry earlier in this thread.

Signature · 3 lines
avuton
--
Anyone who quotes me in their sig is an idiot. -- Rusty Russell.