Oops while writing on a newly ext4 formated 1TB partition on a external USB drive
http://bugzilla.kernel.org/show_bug.cgi?id=11688
------- Comment #2 from [email protected] 2008-11-06 12:27 ------- Can you replicate this on a newer kernel? This looks like a failure in ext3/jbd code, actually, triggered when the kernel was trying to free pages to get back some memory.
Signature · 3 lines
Configure bugmail: http://bugzilla.kernel.org/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are the assignee for the bug, or are watching the assignee.
http://bugzilla.kernel.org/show_bug.cgi?id=11688
------- Comment #3 from [email protected] 2009-01-17 16:24 ------- Hi, any more information about this bug? How much memory do you have?
What ext3 filesystems do you have mounted? The oops is in ext3 code:
handle_t *journal_start(journal_t *journal, int nblocks)
{
handle_t *handle = journal_current_handle();
int err;
if (!journal)
return ERR_PTR(-EROFS);
if (handle) {
J_ASSERT(handle->h_transaction->t_journal == journal); <----
handle->h_ref++;
return handle;
}It seems to be related ext3's quota code -- and again it seems to be related to an attempt to free memory when the system is under memory pressure. This looks very much like a quota and/or ext3 bug.
Signature · 3 lines
Configure bugmail: http://bugzilla.kernel.org/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are the assignee for the bug, or are watching the assignee.
http://bugzilla.kernel.org/show_bug.cgi?id=11688
------- Comment #4 from [email protected] 2009-01-18 15:39 ------- Sure, I'm waiting for 2.6.28 to enter debian so I can test. I have 1.7GB of memory, of that more than 1GB is free (excluding cache). All other filesystems are ext3 including the source (ext3->ext4). Quota is enabled but not set for any user. I disabled it (init.d/quota stop) and tried writing (ext3->ext4) again and oops didn't happen. Will test more, but this could be the issue. What I don't understand is why it doesn't happen when I'm writing to ext3.
Signature · 3 lines
Configure bugmail: http://bugzilla.kernel.org/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are the assignee for the bug, or are watching the assignee.
http://bugzilla.kernel.org/show_bug.cgi?id=11688
------- Comment #5 from [email protected] 2009-01-18 16:56 ------- Ext4's delayed allocation will cause more memory pressure, because blocks
aren't as aggressively pushed out to disk. How much memory do you have in
your system, how many ext3/ext4 filesystems do you have mounted, how many of them are active (and how active) versus totally quiescent, and how big is the journal for those filesystems?
Signature · 3 lines
Configure bugmail: http://bugzilla.kernel.org/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are the assignee for the bug, or are watching the assignee.
http://bugzilla.kernel.org/show_bug.cgi?id=11688
------- Comment #6 from [email protected] 2009-01-18 17:18 -------
total used free shared buffers cached
Mem: 1773 1730 43 0 23 1185
-/+ buffers/cache: 521 1252
Swap: 0 0 0
/dev/mapper/vg-root on / type ext3 (rw,errors=remount-ro) /dev/mapper/vg-home on /home type ext3 (rw,usrquota) /dev/mapper/vg-var on /var type ext3 (rw)/dev/sda3 on /media/disk type ext4dev (rw,nosuid,nodev,uhelper=hal) /dev/sda2 on /media/disk-1 type ext3 (rw,nosuid,nodev,uhelper=hal,data=ordered)
Copying from /home to /media/disk so only these are active if I understand you correctly.
# debugfs -R "stat <8>" /dev/mapper/vg-home debugfs 1.41.3 (12-Oct-2008) Inode: 8 Type: regular Mode: 0600 Flags: 0x0 Generation: 0 Version: 0x00000000 User: 0 Group: 0 Size: 134217728 File ACL: 0 Directory ACL: 0 Links: 1 Blockcount: 262416 Fragment: Address: 0 Number: 0 Size: 0 ctime: 0x487a96ce -- Mon Jul 14 01:59:10 2008 atime: 0x00000000 -- Thu Jan 1 01:00:00 1970 mtime: 0x487a96ce -- Mon Jul 14 01:59:10 2008 Size of extra inode fields: 0
# debugfs -R "stat <8>" /dev/sda3 debugfs 1.41.3 (12-Oct-2008) Inode: 8 Type: regular Mode: 0600 Flags: 0x80000 Generation: 0 Version: 0x00000000 User: 0 Group: 0 Size: 134217728 File ACL: 0 Directory ACL: 0 Links: 1 Blockcount: 262144 Fragment: Address: 0 Number: 0 Size: 0 ctime: 0x48ee6d44 -- Thu Oct 9 22:44:52 2008 atime: 0x00000000 -- Thu Jan 1 01:00:00 1970 mtime: 0x48ee6d44 -- Thu Oct 9 22:44:52 2008 Size of extra inode fields: 0
Signature · 3 lines
Configure bugmail: http://bugzilla.kernel.org/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are the assignee for the bug, or are watching the assignee.
http://bugzilla.kernel.org/show_bug.cgi?id=11688
[email protected] changed:
What |Removed |Added
----------------------------------------------------------------------------
AssignedTo|[email protected]|[email protected] Component|ext4 |ext3
------- Comment #7 from [email protected] 2009-01-18 18:28 -------
So you only have 2 megabytes of memory.... Yeah, this sounds like a low-memory
pressure situation. Ext4's delayed allocation will definitely use more
memory, and it looks like you are tickling an ext3 memory pressure bug. Is
there a reason why you aren't mounting your filesystems with noatime? That's
undoubtedly responsible for the ext3 journal activity, and I suspect mounting /home with noatime will make the problem go away. It might also be interesting disabling quotas on /home, to see if the problem goes away or moves to something else.
What is happening is that when ext4 starts writing, it needs memory for the page cache. To get that space, the VM system is trying to get space, so it calls among other things, shrink_icache_memory, to shrink the inode cache, which then calls clear_inode, which ended up calling ext3_dquot_drop, through the quota code. Ext3_dquot_drop() calls journal_start(), which then trips over
a sanity check. In any case, given what you've told me, I'm even more
convinced this is an ext3 and/or a quota system bug, which is getting triggered to severe memory pressure caused by ext4 and the fact that you have a very tiny amount of memory.
Signature · 3 lines
Configure bugmail: http://bugzilla.kernel.org/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are the assignee for the bug, or are watching the assignee.
http://bugzilla.kernel.org/show_bug.cgi?id=11688
------- Comment #8 from [email protected] 2009-01-19 00:50 ------- Errr, that was output of $ free -m, so it's ~1.7GB of memory. Regarding noatime... I didn't change debian defaults.
Disabling quotas (/etc/init.d/quota stop) seems to fix the problem. I copied some 70GB and oops didn't happen.
I need to test it more, but I'm 90% sure it's quota related bug. You can assign this bug to someone who is in charge of ext3 and quota.
Signature · 3 lines
Configure bugmail: http://bugzilla.kernel.org/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are the assignee for the bug, or are watching the assignee.
http://bugzilla.kernel.org/show_bug.cgi?id=11688
------- Comment #9 from [email protected] 2009-01-20 09:18 ------- Ted, I've looked into this and it is a bug in ext4. It should call grab_cache_page_write_begin() with AOP_FLAG_NOFS since it cannot handle recursion into the filesystem for freeing memory. The problem is that at the time grab_cache_page_write_begin() is called, a transaction is already started
- this is what quota code complains about (because it also needs to start a transaction but current->journal_info is already occupied by ext4).
If we wanted to avoid AOP_FLAG_NOFS we could get the page, start a transaction, lock and check the page but that's going to be a bit nasty.
Signature · 3 lines
Configure bugmail: http://bugzilla.kernel.org/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are the assignee for the bug, or are watching the assignee.
http://bugzilla.kernel.org/show_bug.cgi?id=11688
------- Comment #10 from [email protected] 2009-02-05 08:30 ------- Created an attachment (id=20124)
--> (http://bugzilla.kernel.org/attachment.cgi?id=20124&action=view)
Fix for the bug in ext4_write_begin and ext4_da_write_begin
This patch should fix the problem. Testing welcome ;)
Signature · 3 lines
Configure bugmail: http://bugzilla.kernel.org/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are the assignee for the bug, or are watching the assignee.
http://bugzilla.kernel.org/show_bug.cgi?id=11688
[email protected] changed:
What |Removed |Added
----------------------------------------------------------------------------
AssignedTo|[email protected]|[email protected] Status|NEW |ASSIGNED
------- Comment #11 from [email protected] 2009-02-24 08:22 ------- OK, the patch was merged into ext4-tree. Closing the bug.
Signature · 4 lines
Configure bugmail: http://bugzilla.kernel.org/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are the assignee for the bug, or are watching the assignee. You are the assignee for the bug, or are watching the assignee.