Skip to contentSkip to conversation

problems with ext3 fs, kernels up to 2.6.6-rc2

DISCUSSION Paul P Komkoff Jr 2 participants 5 messages May 19, 2004 – May 25, 2004

lore
Share & tools
lore
#0 lore raw

Hi.

For a long time I'm sorta have the following problem.

I have ext3 partition with dir_index turned on. I have programs, which store many files on it (for example, Maildir mailboxes for 500+ users, about 200k files).

Sometimes something going wrong. I am noticing it by rdiff-backup on this partition producing the following output:
ListError goloub/Maildir/cur/1082623479.1763_0.ns:2,S [Errno 5] Input/output error:
+'/mnt/mail/goloub/Maildir/cur/1082623479.1763_0.ns:2,S'

Yes, when I am doing strace ls -al (failed file), I am seeing -EIO

lstat64("/mnt/mail/goloub/Maildir/cur/1082623479.1763_0.ns:2,S", 0x806408c) = -1 EIO (Input/output error)

I know, so when I will e2fsck it it will be repaired. But how I can help debug it?

Which on-disk structs I need to examine, maybe extract, and send to someone?

Signature · 2 lines
Paul P 'Stingray' Komkoff Jr // http://stingr.net/key <- my pgp key
 This message represents the official view of the voices in my head
#1 lore raw
Quoted 14 lines · Paul P Komkoff Jr

For a long time I'm sorta have the following problem.

I have ext3 partition with dir_index turned on. I have programs, which store many files on it (for example, Maildir mailboxes for 500+ users, about 200k files).

Sometimes something going wrong. I am noticing it by rdiff-backup on this partition producing the following output:
ListError goloub/Maildir/cur/1082623479.1763_0.ns:2,S [Errno 5] Input/output error:
+'/mnt/mail/goloub/Maildir/cur/1082623479.1763_0.ns:2,S'

Yes, when I am doing strace ls -al (failed file), I am seeing -EIO

Quoted 8 lines

lstat64("/mnt/mail/goloub/Maildir/cur/1082623479.1763_0.ns:2,S", 0x806408c) = -1 EIO (Input/output error)

I know, so when I will e2fsck it it will be repaired. But how I can help debug it?

Which on-disk structs I need to examine, maybe extract, and send to someone?

A problem with htree was recently discovered during Lustre testing when files were being renamed within the same directory. In some cases the addition of the new name caused a directory block split and the old dir_entry was pointing at the wrong entry, and the wrong entry was removed. This would seem entirely possible in a Maildir directory, since the MTA will be doing a lot of renames within the same directory.

This seems to fix the majority of the problems, although there are still some rare failures in the rename test.

===== fs/ext3/namei.c 1.52 vs edited =====

fs/ext3/namei.c Wed May 19 10:59:39 2004 renamed from fs/ext3/namei.c Mon May 10 05:25:34 2004 +3 -1
@@ -2265,7 +2265,9 @@
 	 * ok, that's it
 	 */
 	retval = ext3_delete_entry(handle, old_dir, old_de, old_bh);
-	if (retval == -ENOENT) {
+	if (le32_to_cpu(old_de->inode) != old_inode->i_ino ||
+	    (retval = ext3_delete_entry(handle, old_dir, old_de, old_bh)) ==
+	    -ENOENT) {
 		/*
 		 * old_de could have moved out from under us.
 		 */

Cheers, Andreas
--
Andreas Dilger
http://sourceforge.net/projects/ext2resize/
http://www-mddsp.enel.ucalgary.ca/People/adilger/

#2 lore raw

Replying to Andreas Dilger:

Quoted 2 lines

This seems to fix the majority of the problems, although there are still some rare failures in the rename test.

Just curious. Is it really doing what it should? Is there are cases where ext3_delete_entry(handle, old_dir, old_de, old_bh) will be called twice with the same set of parameters? :()

Signature · 2 lines
Paul P 'Stingray' Komkoff Jr // http://stingr.net/key <- my pgp key
 This message represents the official view of the voices in my head
#3 lore raw
Quoted 7 lines · Paul P Komkoff Jr

Replying to Andreas Dilger:

Quoted 2 lines

This seems to fix the majority of the problems, although there are still some rare failures in the rename test.

Just curious. Is it really doing what it should? Is there are cases where ext3_delete_entry(handle, old_dir, old_de, old_bh) will be called twice with the same set of parameters? :()

Grr, my bad. I had moved this by hand from 2.4 (where this was discovered and where I'm testing) to 2.6.current just to make sure the context and everything was right for submissions and of course botched it. The right patch removes the old call to ext3_delete_entry():

===== fs/ext3/namei.c 1.52 vs edited =====

fs/ext3/namei.c +3 -2
@@ -2264,8 +2264,9 @@
 	/*
 	 * ok, that's it
 	 */
-	retval = ext3_delete_entry(handle, old_dir, old_de, old_bh);
-	if (retval == -ENOENT) {
+	if (le32_to_cpu(old_de->inode) != old_inode->i_ino ||
+	    (retval = ext3_delete_entry(handle, old_dir,
+					old_de, old_bh)) ==  -ENOENT) {
 		/*
 		 * old_de could have moved out from under us.
 		 */

Cheers, Andreas
--
Andreas Dilger
http://sourceforge.net/projects/ext2resize/
http://www-mddsp.enel.ucalgary.ca/People/adilger/

#4 lore raw
Quoted 8 lines · Andreas Dilger

This seems to fix the majority of the problems, although there are still some rare failures in the rename test.

===== fs/ext3/namei.c 1.52 vs edited =====

fs/ext3/namei.c +0 -0
@@ -2264,8 +2264,9 @@
 	/*

fs/ext3/namei.c line 2272

Quoted 10 lines
fs/ext3/namei.c +3 -2
 	 * ok, that's it
 	 */
-	retval = ext3_delete_entry(handle, old_dir, old_de, old_bh);
-	if (retval == -ENOENT) {
+	if (le32_to_cpu(old_de->inode) != old_inode->i_ino ||
+	    (retval = ext3_delete_entry(handle, old_dir,
+					old_de, old_bh)) ==  -ENOENT) {
 		/*
 		 * old_de could have moved out from under us.
 		 */

I isolated the source of the remaining problems. In very rare cases even with the above patch we could still do the wrong thing. If old_de is pointing to the newly-added entry (i_ino is the same) we end up deleting the new entry instead of the old one. It looks as if the rename never happened. We need to verify that the name we are unlinking is what we expect.

If is also possible that old_de is pointing to the now-unused space at the end of a newly-split leaf block, so we still need to try ext3_delete_entry() (which will skip the stale entry and return ENOENT) instead of just relying on the inum + name check.

===== fs/ext3/namei.c 1.52 vs edited =====

fs/ext3/namei.c +9 -5
@@ -2264,11 +2264,15 @@
 	/*
 	 * ok, that's it
 	 */
-	retval = ext3_delete_entry(handle, old_dir, old_de, old_bh);
-	if (retval == -ENOENT) {
-		/*
-		 * old_de could have moved out from under us.
-		 */
+	if (le32_to_cpu(old_de->inode) != old_inode->i_ino ||
+	    old_de->name_len != old_dentry->d_name.len ||
+	    strncmp(old_de->name, old_dentry->d_name.name, old_de->name_len) ||
+	    (retval = ext3_delete_entry(handle, old_dir,
+					old_de, old_bh)) == -ENOENT) {
+		/* old_de could have moved from under us during htree split, so
+		 * make sure that we are deleting the right entry.  We might
+		 * also be pointing to a stale entry in the unused part of
+		 * old_bh so just checking inum and the name isn't enough. */
 		struct buffer_head *old_bh2;
 		struct ext3_dir_entry_2 *old_de2;
 

Cheers, Andreas
--
Andreas Dilger
http://sourceforge.net/projects/ext2resize/
http://www-mddsp.enel.ucalgary.ca/People/adilger/