ReiserFS phenomenon with 2.4.2 ac24/ac12
Quoted 28 lines · Chris Mason
On Monday, March 26, 2001 03:21:29 PM -0800 Christoph Lameter
Quoted 27 lines · Chris Mason
Quoted 24 lines
Quoted 15 lines · On Saturday, March 24, 2001 11:56:08 AM -0800 Christoph Lameter…
I got a directory /a/yy that I tried to erase with rm -rf /a/yy.
rm hangs...
ls gives the following output:
ls: /a/yy/cache3A0F94EA0A00557.html: No such file or directory
ls: /a/yy/cache3A0F94EA0A00557.html: No such file or directory
ls: /a/yy/cache3A8CCC6A0490B05.gifcache393C2B6A2CD2DF1.crumb: No such file or directory
ls: /a/yy/cache3A0F94EA0A00557.html: No such file or directory
ls: /a/yy/cache3A0F94EA0A00557.html: No such file or directory
ls: /a/yy/cache3A8CCC6A0490B05.gifcache393C2B6A2CD2DF1.crumb: No such file or directoryQuoted 4 lines
and so on and so on....
I tried a reiserfscheck -x on the /a volume but the strangeness still persists. What is going on here?
There should be more messages on the console that will help identify the problem. Please check through
/var/log/messagesand send along anything that looks odd.
Quoted 7 lines
Quoted 3 lines
Yes I see messages like this in the log:
Mar 26 06:47:50 k2-400 kernel: Out of Memory: Killed process 421 (mysqld).Hmmm, I was expecting something from reiserfs. The file not found messages from ls usually go along with warnings from reiserfs about an i/o error. What does a readonly reiserfsck say (without the -x).
Sorry the HD is ok. Is there any way to get rid of the mess?
output of reiserfsck:
grow_id_map: objectid map expanded: used 1024, 1 blocksbad_leaf: block 9454 has invalid item 4: 1928 5204 0x1 DIR, len 184, entry count 5, fsck need 0, format oldgrow_id_map: objectid map expanded: used 2048, 2 blocksgrow_id_map: objectid map expanded: used 3072, 3 blocksgrow_id_map: objectid map expanded: used 4096, 4 blocksshrink_id_map: objectid map shrinked: used 4096, 5 blocksgrow_id_map: objectid map expanded: used 4096, 4 blocksshrink_id_map: objectid map shrinked: used 4096, 5 blocksgrow_id_map: objectid map expanded: used 4096, 4 blocksgrow_id_map: objectid map expanded: used 5120, 5 blocksgrow_id_map: objectid map expanded: used 6144, 6 blocksgrow_id_map: objectid map expanded: used 7168, 7 blocksgrow_id_map: objectid map expanded: used 8192, 8 blocksgrow_id_map: objectid map expanded: used 9216, 9 blocksgrow_id_map: objectid map expanded: used 10240, 10 blocksgrow_id_map: objectid map expanded: used 11264, 11 blocksgrow_id_map: objectid map expanded: used 12288, 12 blocksshrink_id_map: objectid map shrinked: used 12288, 13 blocksgrow_id_map: objectid map expanded: used 12288, 12 blocksshrink_id_map: objectid map shrinked: used 12288, 13 blocksgrow_id_map: objectid map expanded: used 12288, 12 blocksgrow_id_map: objectid map expanded: used 13312, 13 blocksshrink_id_map: objectid map shrinked: used 13312, 14 blocksgrow_id_map: objectid map expanded: used 13312, 13 blocksshrink_id_map: objectid map shrinked: used 13312, 14 blocksgrow_id_map: objectid map expanded: used 13312, 13 blocksshrink_id_map: objectid map shrinked: used 13312, 14 blocksgrow_id_map: objectid map expanded: used 13312, 13 blocksshrink_id_map: objectid map shrinked: used 13312, 14 blocksshrink_id_map: objectid map shrinked: used 12288, 13 blocksshrink_id_map: objectid map shrinked: used 11264, 12 blocksgrow_id_map: objectid map expanded: used 11264, 11 blocksshrink_id_map: objectid map shrinked: used 11264, 12 blocksgrow_id_map: objectid map expanded: used 11264, 11 blocksshrink_id_map: objectid map shrinked: used 11264, 12 blocksgrow_id_map: objectid map expanded: used 11264, 11 blocksshrink_id_map: objectid map shrinked: used 11264, 12 blocksgrow_id_map: objectid map expanded: used 11264, 11 blocksgrow_id_map: objectid map expanded: used 12288, 12 blocksshrink_id_map: objectid map shrinked: used 12288, 13 blocksgrow_id_map: objectid map expanded: used 12288, 12 blocksshrink_id_map: objectid map shrinked: used 12288, 13 blocksgrow_id_map: objectid map expanded: used 12288, 12 blocksshrink_id_map: objectid map shrinked: used 12288, 13 blocksgrow_id_map: objectid map expanded: used 12288, 12 blocksshrink_id_map: objectid map shrinked: used 12288, 13 blocksgrow_id_map: objectid map expanded: used 12288, 12 blocksshrink_id_map: objectid map shrinked: used 12288, 13 blocksgrow_id_map: objectid map expanded: used 12288, 12 blocksshrink_id_map: objectid map shrinked: used 12288, 13 blocksgrow_id_map: objectid map expanded: used 12288, 12 blocksshrink_id_map: objectid map shrinked: used 12288, 13 blocksgrow_id_map: objectid map expanded: used 12288, 12 blocksshrink_id_map: objectid map shrinked: used 12288, 13 blocks
free block count 447444 mismatches with a correct one 447840.
on-disk bitmap does not match to the correct one.check_semantic_tree: hash mismatch detected (cache3A0F94EA0A00557.html)check_semantic_tree: name "cache3A0F94EA0A00557.html" in directory 1928 5204 points to nowherecheck_semantic_tree: hash mismatch detected (cache3A8CCC6A0490B05.gifcache393C2B6A2CD2DF1.crumb)check_semantic_tree: name "cache3A8CCC6A0490B05.gifcache393C2B6A2CD2DF1.crumb" in directory 1928 5204 points to nowhere
Quoted 21 lines · On Monday, March 26, 2001 06:57:37 PM -0800 Christoph Lameter…
Quoted 20 lines · Chris Mason
On Monday, March 26, 2001 03:21:29 PM -0800 Christoph Lameter
Quoted 19 lines · Chris Mason
Quoted 18 lines
Quoted 16 lines · On Saturday, March 24, 2001 11:56:08 AM -0800 Christoph Lameter…
I got a directory /a/yy that I tried to erase with rm -rf /a/yy.
rm hangs...
ls gives the following output:
ls: /a/yy/cache3A0F94EA0A00557.html: No such file or directory
ls: /a/yy/cache3A0F94EA0A00557.html: No such file or directory
ls: /a/yy/cache3A8CCC6A0490B05.gifcache393C2B6A2CD2DF1.crumb: No such file or directory
ls: /a/yy/cache3A0F94EA0A00557.html: No such file or directory
ls: /a/yy/cache3A0F94EA0A00557.html: No such file or directory
ls: /a/yy/cache3A8CCC6A0490B05.gifcache393C2B6A2CD2DF1.crumb: No such file or directory
Quoted 3 lines
output of reiserfsck:
grow_id_map: objectid map expanded: used 1024, 1 blocks
These are harmless and should have been removed from fsck forever ago.
Quoted 1 line
bad_leaf: block 9454 has invalid item 4: 1928 52040x1DIR, len 184,
This is probably the cause of the errors. hopefully /a is inode number 1928 and /a/yy is inode number 5204 ( you can check with ls -i ). Please send me (privately) the output of debugreiserfs -b 9454, and we'll figure out the best way to fix it.
Quoted 6 lines
check_semantic_tree: hash mismatch detected (cache3A0F94EA0A00557.html)check_semantic_tree: name "cache3A0F94EA0A00557.html" in directory 1928 5204 points to nowherecheck_semantic_tree: hash mismatch detected (cache3A8CCC6A0490B05.gifcache393C2B6A2CD2DF1.crumb)check_semantic_tree:
name "cache3A8CCC6A0490B05.gifcache393C2B6A2CD2DF1.crumb" in directory 1928 5204 points to nowhere
And these are in the bad directory mentioned above.
-chris
Quoted 23 lines · Chris Mason
Quoted 21 lines · On Monday, March 26, 2001 06:57:37 PM -0800 Christoph Lameter…
Quoted 20 lines · Chris Mason
On Monday, March 26, 2001 03:21:29 PM -0800 Christoph Lameter
Quoted 19 lines · Chris Mason
Quoted 18 lines
Quoted 16 lines · On Saturday, March 24, 2001 11:56:08 AM -0800 Christoph Lameter…
I got a directory /a/yy that I tried to erase with rm -rf /a/yy.
rm hangs...
ls gives the following output:
ls: /a/yy/cache3A0F94EA0A00557.html: No such file or directory
ls: /a/yy/cache3A0F94EA0A00557.html: No such file or directory
ls: /a/yy/cache3A8CCC6A0490B05.gifcache393C2B6A2CD2DF1.crumb: No such file or directory
ls: /a/yy/cache3A0F94EA0A00557.html: No such file or directory
ls: /a/yy/cache3A0F94EA0A00557.html: No such file or directory
ls: /a/yy/cache3A8CCC6A0490B05.gifcache393C2B6A2CD2DF1.crumb: No such file or directory
Quoted 7 lines
Quoted 2 lines
output of reiserfsck:
bad_leaf: block 9454 has invalid item 4: 1928 52040x1DIR, len 184,This is probably the cause of the errors. hopefully /a is inode number 1928 and /a/yy is inode number 5204 ( you can check with ls -i ). Please send me (privately) the output of debugreiserfs -b 9454, and we'll figure out the best way to fix it.
root@k2-400:/a/christoph# debugreiserfs -b 9454 /dev/hda6
<-------------debugreiserfs, 2000------------->
reiserfsprogs 3.x.0h
9454 is free in true bitmap
===================================================================
LEAF NODE (9454) contains level=1, nr_items=11, free_space=16 rdkey
-------------------------------------------------------------------------------
|###|type|ilen|f/sp| loc|fmt|fsck| key | | | | |e/cn| | |need| | ------------------------------------------------------------------------------- | 0|1928 1929 0x6d1 DRCT, len 8, entry count 65535, fsck need 0, format new|
-------------------------------------------------------------------------------
| 1|1928 1930 0x0 SD, len 44, entry count 65535, fsck need 0, format new| (NEW SD), mode -rw-------, size 47353, nlink 1, mtime 03/24/2001 07:15:21
-------------------------------------------------------------------------------
| 2|1928 1930 0x1 IND, len 48, entry count 0, fsck need 0, format new| 12 pointers
[ 101228(4) 101377 101379 101450 101585 101590(4)]
-------------------------------------------------------------------------------
| 3|1928 5204 0x0 SD, len 44, entry count 65535, fsck need 0, format new| (NEW SD), mode d---------, size 96, nlink 2, mtime 03/23/2001 20:49:37
-------------------------------------------------------------------------------
| 4|1928 5204 0x1 DIR, len 184, entry count 5, fsck need 0, format old| ###: Name length Object key Hash Gen
number
0: ". "( 1) 1928 5204 0
1, loc b0, state 4 ()
1: ".. "( 2) 1 2 0
2, loc a8, state 4 ()
2: "cache3A0F94EA0A00557.html"( 25) 5204 5334 1942043776
0, loc 88, state 4 ()
3: "cache3A0F94EA0A00557.html"( -6) 263760 5334 5248
86, loc 88, state 4 (BROKEN)
4: "cache3A8CCC6A0490B05.gifcache393C2B6A2CD2DF1.crumb"( 50) 263760
64136 5248 86, loc 50, state 4 (BROKEN)
-------------------------------------------------------------------------------
| 5|1928 20843 0x0 SD, len 44, entry count 65535, fsck need 0, format new|
(NEW SD), mode -rw-------, size 21439, nlink 1, mtime 03/25/2001 19:44:34
-------------------------------------------------------------------------------
| 6|1928 20843 0x1 IND, len 24, entry count 0, fsck need 0, format new| 6 pointers
[ 100720 100724 100942 101486 102588 102614]
-------------------------------------------------------------------------------
| 7|1928 42413 0x0 SD, len 44, entry count 65535, fsck need 0, format new|
(NEW SD), mode -rw-------, size 3978, nlink 1, mtime 03/25/2001 20:00:32
-------------------------------------------------------------------------------
| 8|1928 42413 0x1 IND, len 4, entry count 0, fsck need 0, format new| 1 pointers
[ 102864]
-------------------------------------------------------------------------------
| 9|1931 1933 0x0 SD, len 44, entry count 65535, fsck need 0, format new| (NEW SD), mode -rw-------, size 3448, nlink 1, mtime 03/24/2001 07:15:14
-------------------------------------------------------------------------------
| 10|1931 1933 0x1 DRCT, len 3304, entry count 65535, fsck need 0, format new|
===================================================================
Quoted 17 lines · On Tuesday, March 27, 2001 08:21:07 AM -0800 Christoph Lameter…
<-------------debugreiserfs, 2000------------->
reiserfsprogs 3.x.0h
9454 is free in true bitmap===================================================================
LEAF NODE (9454) contains level=1, nr_items=11, free_space=16 rdkey
-------------------------------------------------------------------------
------ | 3|1928 52040x0SD, len 44, entry count 65535, fsck need 0, format new| (NEW SD), mode d---------, size 96, nlink 2, mtime 03/23/2001 20:49:37
-------------------------------------------------------------------------
------ | 4|1928 52040x1DIR, len 184, entry count 5, fsck need 0,
format old| ###: Name length Object key
Hash Gen number
0: ". "( 1) 1928 5204 0
1, loc b0, state 4 ()
Quoted 8 lines
1: ".. "( 2) 1 2 0
2, loc a8, state 4 ()
2: "cache3A0F94EA0A00557.html"( 25) 5204 5334 1942043776
0, loc 88, state 4 ()
3: "cache3A0F94EA0A00557.html"( -6) 263760 5334 5248
86, loc 88, state 4 (BROKEN)
4: "cache3A8CCC6A0490B05.gifcache393C2B6A2CD2DF1.crumb"( 50) 263760
64136 5248 86, loc 50, state 4 (BROKEN)
Ok, notice how entry 2 and 3 are the same file name? That is a big part of your problem, and it should never happen with the normal kernel code. The two lines that show up as (BROKEN) mean their hash values are incorrect.
So, were there errors present before you ran reiserfsck -x? Had you run any version of reiserfsck (with -x or --rebuild-tree) before that?
I'm guessing these problems were caused by reiserfsck, things caused by kernel bug would tend towards much more random errors. The solution will probably be an upgrade to the latest fsck version, but I'd like to make sure we've got the problem nailed down.
-chris
Quoted 29 lines · Chris Mason
Quoted 17 lines · On Tuesday, March 27, 2001 08:21:07 AM -0800 Christoph Lameter…
<-------------debugreiserfs, 2000------------->
reiserfsprogs 3.x.0h
9454 is free in true bitmap===================================================================
LEAF NODE (9454) contains level=1, nr_items=11, free_space=16 rdkey
-------------------------------------------------------------------------
------ | 3|1928 52040x0SD, len 44, entry count 65535, fsck need 0, format new| (NEW SD), mode d---------, size 96, nlink 2, mtime 03/23/2001 20:49:37
-------------------------------------------------------------------------
------ | 4|1928 52040x1DIR, len 184, entry count 5, fsck need 0,
format old| ###: Name length Object key
Hash Gen number
0: ". "( 1) 1928 5204 0
1, loc b0, state 4 ()Quoted 8 lines
1: ".. "( 2) 1 2 0
2, loc a8, state 4 ()
2: "cache3A0F94EA0A00557.html"( 25) 5204 5334 1942043776
0, loc 88, state 4 ()
3: "cache3A0F94EA0A00557.html"( -6) 263760 5334 5248
86, loc 88, state 4 (BROKEN)
4: "cache3A8CCC6A0490B05.gifcache393C2B6A2CD2DF1.crumb"( 50) 263760
64136 5248 86, loc 50, state 4 (BROKEN)
Quoted 6 lines
Ok, notice how entry 2 and 3 are the same file name? That is a big part of your problem, and it should never happen with the normal kernel code. The two lines that show up as (BROKEN) mean their hash values are incorrect.
So, were there errors present before you ran reiserfsck -x? Had you run any version of reiserfsck (with -x or --rebuild-tree) before that?
The problems were present before I ran reiserfsck. I never ran --rebuild-tree
Quoted 4 lines
I'm guessing these problems were caused by reiserfsck, things caused by kernel bug would tend towards much more random errors. The solution will probably be an upgrade to the latest fsck version, but I'd like to make sure we've got the problem nailed down.
I think this is a problem with the reiserfs code in the kernel. I never ran reiserfsck before this problem surfaced. The problem arose in the netscape cache directory with lots of small files. Guess the tail handling is not that stable yet?
How do I get rid of the /a/yy directory now?
Quoted 11 lines · On Tuesday, March 27, 2001 09:50:17 AM -0800 Christoph Lameter…
Quoted 8 lines
Ok, notice how entry 2 and 3 are the same file name? That is a big part of your problem, and it should never happen with the normal kernel code. The two lines that show up as (BROKEN) mean their hash values are incorrect.
So, were there errors present before you ran reiserfsck -x? Had you run any version of reiserfsck (with -x or --rebuild-tree) before that?
The problems were present before I ran reiserfsck. I never ran --rebuild-tree
Just to make sure I understand, you had the exact same errors before running fsck? Same files could not be deleted?
Quoted 10 lines
Quoted 5 lines
I'm guessing these problems were caused by reiserfsck, things caused by kernel bug would tend towards much more random errors. The solution will probably be an upgrade to the latest fsck version, but I'd like to make sure we've got the problem nailed down.
I think this is a problem with the reiserfs code in the kernel. I never ran reiserfsck before this problem surfaced. The problem arose in the netscape cache directory with lots of small files. Guess the tail handling is not that stable yet?
I wish I could blame it on the tail code ;-) None of the bugs fixed there would have caused this, and they should be completely unrelated. I'll try some tests in oom situations to try and reproduce. It could also be caused by hashing errors. If you formatted with r5 hash, was the partition ever incorrectly detected as tea hash?
Quoted 2 lines
How do I get rid of the /a/yy directory now?
With fsck. I'll grab the latest today and make sure it can fix this bug. Until then, mv /a/yy /a/yy.broken and mkdir /a/yy.
-chris
Quoted 2 lines · Chris Mason
Just to make sure I understand, you had the exact same errors before running fsck? Same files could not be deleted?
Correct.
Quoted 10 lines
Quoted 4 lines
I think this is a problem with the reiserfs code in the kernel. I never ran reiserfsck before this problem surfaced. The problem arose in the netscape cache directory with lots of small files. Guess the tail handling is not that stable yet?
I wish I could blame it on the tail code ;-) None of the bugs fixed there would have caused this, and they should be completely unrelated. I'll try some tests in oom situations to try and reproduce. It could also be caused by hashing errors. If you formatted with r5 hash, was the partition ever incorrectly detected as tea hash?
No idea. I never got that deep into reiser.
Quoted 4 lines
Quoted 1 line
How do I get rid of the /a/yy directory now?
With fsck. I'll grab the latest today and make sure it can fix this bug. Until then, mv /a/yy /a/yy.broken and mkdir /a/yy.
/a/yy is already a "broken" dir moved out of the way.
Quoted 11 lines · On Tuesday, March 27, 2001 11:14:57 AM -0800 Christoph Lameter…
Quoted 2 lines · Chris Mason
Just to make sure I understand, you had the exact same errors before running fsck? Same files could not be deleted?
Correct.
Quoted 5 lines
Quoted 4 lines
I think this is a problem with the reiserfs code in the kernel. I never ran reiserfsck before this problem surfaced. The problem arose in the netscape cache directory with lots of small files. Guess the tail handling is not that stable yet?
Quoted 7 lines
Quoted 5 lines
I wish I could blame it on the tail code ;-) None of the bugs fixed there would have caused this, and they should be completely unrelated. I'll try some tests in oom situations to try and reproduce. It could also be caused by hashing errors. If you formatted with r5 hash, was the partition ever incorrectly detected as tea hash?
No idea. I never got that deep into reiser.
I should have been more clear. Everyt ime you mount the filesystem, a it prints the hash used. This is probably recorded in your log files, either 'Using r5 hash to sort names' or 'Using tea hash to sort names'. For any given filesystem, it should never switch from one to the other.
Quoted 8 lines
Quoted 5 lines
Quoted 2 lines
How do I get rid of the /a/yy directory now?
With fsck. I'll grab the latest today and make sure it can fix this bug. Until then, mv /a/yy /a/yy.broken and mkdir /a/yy.
/a/yy is already a "broken" dir moved out of the way.
Ok, I'll get back to you later on fsck.
-chris
Quoted 6 lines · Chris Mason
On Tuesday, March 27, 2001 11:14:57 AM -0800 Christoph Lameter <[email protected]> wrote:
I should have been more clear. Everyt ime you mount the filesystem, a it prints the hash used. This is probably recorded in your log files, either 'Using r5 hash to sort names' or 'Using tea hash to sort names'. For any given filesystem, it should never switch from one to the other.
reiserfs: checking transaction log (device 03:06) ... Using r5 hash to sort names
ReiserFS version 3.6.25
reiserfs: checking transaction log (device 03:05) ... Using r5 hash to sort names
ReiserFS version 3.6.25
Undo loss 199.183.24.194/2530 c2 l0 ss2/65535 p0 Undo partial loss 216.234.231.6/1936 c1 l1 ss2/65535 p1 Out of Memory: Killed process 408 (mysqld).
Out of Memory: Killed process 411 (mysqld).
Out of Memory: Killed process 412 (mysqld).
Out of Memory: Killed process 422 (mysqld).
Out of Memory: Killed process 524 (apache).
Out of Memory: Killed process 525 (apache).
Out of Memory: Killed process 527 (apache).
Out of Memory: Killed process 528 (apache).
Out of Memory: Killed process 529 (apache).
Out of Memory: Killed process 529 (apache).
Out of Memory: Killed process 647 (kdeinit). Out of Memory: Killed process 647 (kdeinit). Out of Memory: Killed process 1485 (kdevelop). Out of Memory: Killed process 643 (kdeinit). Out of Memory: Killed process 652 (kdeinit). Out of Memory: Killed process 650 (kdeinit). Out of Memory: Killed process 650 (kdeinit). Out of Memory: Killed process 650 (kdeinit). Out of Memory: Killed process 636 (kdeinit). Out of Memory: Killed process 639 (kdeinit). Out of Memory: Killed process 5310 (find).
Undo partial loss 216.234.231.6/3011 c1 l1 ss2/65535 p1 Undo partial loss 209.167.177.93/1974 c1 l1 ss2/2 p1
parport0: Timed out
Undo partial loss 24.169.102.121/3177 c1 l1 ss2/65535 p1 Undo partial loss 216.234.231.6/3384 c1 l1 ss2/2 p1 Undo loss 207.33.153.2/61745 c2 l0 ss2/65535 p0 Undo partial loss 207.33.153.2/61745 c2 l1 ss2/65535 p1 Undo partial loss 207.33.153.2/61745 c1 l1 ss2/65535 p1 Undo loss 207.33.153.2/61745 c2 l0 ss2/65535 p0 Undo loss 207.33.153.2/61921 c2 l0 ss2/65535 p0 Undo loss 207.33.153.2/61921 c2 l0 ss2/65535 p0 Undo partial loss 216.234.231.6/2008 c1 l1 ss2/2 p1 Undo partial loss 24.169.102.121/3423 c1 l1 ss2/65535 p1 Undo partial loss 216.234.231.6/4252 c1 l1 ss2/2 p1 Undo partial loss 216.234.231.5/55504 c1 l1 ss2/65535 p1 Undo loss 207.33.153.2/61448 c2 l0 ss2/65535 p0 Undo partial loss 207.33.153.2/61448 c1 l1 ss2/65535 p1 Undo loss 207.33.153.2/61448 c2 l0 ss2/65535 p0 Undo loss 207.33.153.2/61448 c2 l0 ss2/65535 p0