Skip to contentSkip to conversation

What will happen when disk(ext3) is full while i continue to operate files ?

DISCUSSION Seaman Hu 3 participants 7 messages Sep 20, 2002

lore
Share & tools
lore

Seaman Hu

author
#0 lore raw

I am sorry if you receive it twice. since the mail-list said that i can't send mail to it unless i become one its member.

hi,
My system will crash when the disk(ext3) is full
while i continue to launch 50 proceses to operate files(such as create, rm, mv, ...). Does ext3 have such a capability to stop journaling the changes when it finds there is no space left? or which source file of ext3 do i need to check this function?

My System info:

    Redhat 7.3    Linux Kernel: 2.4.18-17
    / : ext3 1.6G   /boot : ext3 60M

I am sorry that I will attach the oops next time
since i am still restoring my poor system. :(

Any opinions will be highly appriciated. Thanks in
advance. :)

Seaman

__________________________________________________
Do You Yahoo!?
Yahoo! Finance - Get real-time stock quotes
http://finance.yahoo.com

author
#1 lore raw

Sorry. I probably didn't make it clear.
My system is ok when first msg "EXT3-fs error (device sd(8,2)) in ext3_new_inode: error 28" appears. However, it will crash after millions of the same msg. Is there some kind of buffer full to cause the crash?

thanks in advance.

Seaman

Quoted 34 lines · --- Seaman Hu

I am sorry if you receive it twice. since the mail-list said that i can't send mail to it unless i become one its member.

hi,
My system will crash when the disk(ext3) is full
while i continue to launch 50 proceses to operate files(such as create, rm, mv, ...). Does ext3 have such a capability to stop journaling the changes when
it finds there is no space left? or which source file
of ext3 do i need to check this function?

My System info:

    Redhat 7.3    Linux Kernel: 2.4.18-17
    / : ext3 1.6G   /boot : ext3 60M

I am sorry that I will attach the oops next time
since i am still restoring my poor system. :(

Any opinions will be highly appriciated. Thanks
in
advance. :)

Seaman

__________________________________________________
Do You Yahoo!?
Yahoo! Finance - Get real-time stock quotes
http://finance.yahoo.com

Quoted 4 lines

_______________________________________________
Ext3-users mailing list
[email protected]

https://listman.redhat.com/mailman/listinfo/ext3-users

__________________________________________________
Do You Yahoo!?
Yahoo! Finance - Get real-time stock quotes
http://finance.yahoo.com

#2 lore raw

Hi,

Quoted 5 lines · Seaman Hu

Sorry. I probably didn't make it clear.
My system is ok when first msg "EXT3-fs error (device sd(8,2)) in ext3_new_inode: error 28" appears. However, it will crash after millions of the same msg. Is there some kind of buffer full to cause the crash?

Ah, that's a known problem when you run out of inodes. Ext3 incorrectly treated it as a full fs error. That's been fixed in -ac, ext3 CVS and the Red Hat kernels for a while, and it's in Marcelo's post-2.4.19 tree.

Cheers,
Stephen

#3 lore raw
Quoted 4 lines

Ah, that's a known problem when you run out of inodes. Ext3 incorrectly treated it as a full fs error. That's been fixed in -ac, ext3 CVS and the Red Hat kernels for a while, and it's in Marcelo's post-2.4.19 tree.

Do you know a good way to recover from this when it happens? The problem being that when you reboot it immediately happens again... I managed to recover from this but it required some time and tricks (see other email to list) - perhaps you know an easy way?

All the best,

Duncan.

#4 lore raw

Hi,

Quoted 9 lines · Duncan Sands
Quoted 4 lines

Ah, that's a known problem when you run out of inodes. Ext3 incorrectly treated it as a full fs error. That's been fixed in -ac, ext3 CVS and the Red Hat kernels for a while, and it's in Marcelo's post-2.4.19 tree.

Do you know a good way to recover from this when it happens? The problem being that when you reboot it immediately happens again... I managed to recover from this but it required some time and tricks (see other email to list) - perhaps you know an easy way?

Mounting with "errors=continue" will make the kernel ignore the error when it occurs.

Cheers,
Stephen

#5 lore raw
In reply to Seaman Hu
Quoted 5 lines · Seaman Hu

Sorry. I probably didn't make it clear.
My system is ok when first msg "EXT3-fs error (device sd(8,2)) in ext3_new_inode: error 28" appears. However, it will crash after millions of the same msg. Is there some kind of buffer full to cause the crash?

It looks like you have run out of inodes (df -i will tell you how many inodes you have left). When you format your disk as ext2 or ext3, you specify the number of inodes to be allocated on the disk. This number does not change later. The inodes are used for storing file information. It is possible to run out of inodes while still having plenty of space free on the disk. The ext3 code in 2.4 up to and including 2.4.19 contains a bug: it fails nastily if you run out of inodes (it fails gracefully if you run out of free space). This has been fixed in the latest 2.5 kernels and in the -ac kernels I think.

The problem is that it is quite tricky to recover from this. What you need to do is delete files on the disk in order to have some free inodes. Then you can apply a kernel patch to fix the bug (Andrew Morton sent me this patch against 2.5.20:

fs/ext3/ialloc.c Mon Jun 3 16:05:36 2002 renamed from fs/ext3/ialloc.c Wed May 29 12:48:15 2002 +2 -1
@@ -534,7 +534,8 @@ repeat:
 fail:
        unlock_super(sb);
        iput(inode);
-       ext3_std_error(sb, err);
+       if (err != -ENOSPC)
+               ext3_std_error(sb, err);
        return ERR_PTR(err);
 }
 

Thanks Andrew!)

Now, how to recover? When this happened to me I did something along these lines (I don't remember too well):
(1) force an fsck on reboot
(2) reboot quickly after the fsck and before any files get created (otherwise you are back where you started)
(3) mount the disk as ext2 via the rootfstype=ext2 boot option
(4) delete lots of files.

Probably there is a better way...

I hope this helps,

Duncan.

#6 lore raw

Hi,

Quoted 1 line · Duncan Sands

The problem is that it is quite tricky to recover from this.

Actually, mounting with "errors=continue" should let the filesystem ignore the failure.

Quoted 4 lines

What
you need to do is delete files on the disk in order to have some free inodes. Then you can apply a kernel patch to fix the bug (Andrew Morton sent me this patch against 2.5.20:

The official patch in 2.4 is attached.

Cheers,
Stephen

---------------------
PatchSet 827
Date: 2002/04/10 18:02:19
Author: sct
Log:
Don't consider ENOSPC as a fatal error when allocating an inode. Otherwise running out of inodes marks the fs as having an error, potentially taking the kernel down if we are in panic-on-error fs mode.

Members:

	fs/ext3/ialloc.c:1.19.4.4->1.19.4.5 [ext3-1_0-branch]
fs/ext3/ialloc.c renamed from fs/ext3/ialloc.c.=K0001=.orig +3 -2
@@ -392,7 +392,7 @@
 
 	err = -ENOSPC;
 	if (!gdp)
-		goto fail;
+		goto out;
 
 	err = -EIO;
 	bitmap_nr = load_inode_bitmap (sb, i);
@@ -523,9 +523,10 @@
 	return inode;
 
 fail:
+	ext3_std_error(sb, err);
+out:
 	unlock_super(sb);
 	iput(inode);
-	ext3_std_error(sb, err);
 	return ERR_PTR(err);
 }