New: Oops in ext4 driver
http://bugzilla.kernel.org/show_bug.cgi?id=15025
Summary: Oops in ext4 driver
Product: File System
Version: 2.5
Kernel Version: 2.6.33-rc3
Platform: All
OS/Version: Linux
Tree: Mainline
Status: NEW
Severity: high
Priority: P1
Component: ext4
AssignedTo: [email protected]
ReportedBy: [email protected]
Regression: Yes
This may or may not be related to my previous bug report (15018); it's from the same machine against the same filesystem. My kernel oopsed on me today:
[141480.485154] ------------[ cut here ]------------ [141480.489001] kernel BUG at fs/jbd2/transaction.c:1027! [141480.489001] invalid opcode: 0000 [#1] SMP [141480.489001] last sysfs file: /sys/devices/system/cpu/cpu3/cache/index2/shared_cpu_map [141480.489001] CPU 2 [141480.489001] Pid: 28573, comm: rsync Not tainted 2.6.33-rc3 #2 C2SBC-Q/C2SBC-Q [141480.489001] RIP: 0010:[<ffffffffa022f6d8>] [<ffffffffa022f6d8>] jbd2_journal_dirty_metadata+0x69/0x100 [jbd2] [141480.489001] RSP: 0018:ffff88022ead11c8 EFLAGS: 00010246 [141480.489001] RAX: 0000000000000000 RBX: ffff880090f054a8 RCX: ffff880160c88540 [141480.489001] RDX: ffff8801655f44c0 RSI: ffff880160c88540 RDI: ffff880038a0ce70 [141480.489001] RBP: ffff88022ead11f8 R08: ffff880160c88540 R09: ffff88022ead1178 [141480.489001] R10: 0000000000000000 R11: ffffffffa0280b60 R12: ffff8802272e1828 [141480.489001] R13: ffff880160c88540 R14: ffff880038a0ce70 R15: ffff88023e9c7800 [141480.489001] FS: 00007fd4064996e0(0000) GS:ffff880028300000(0000) knlGS:0000000000000000 [141480.489001] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b [141480.489001] CR2: 00000000004a2071 CR3: 000000010670f000 CR4: 00000000000006e0 [141480.489001] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [141480.489001] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [141480.489001] Process rsync (pid: 28573, threadinfo ffff88022ead0000, task ffff88016fc927c0) [141480.489001] Stack: [141480.489001] ffffffffa027f890 ffff880090f054a8 ffff880160c88540 ffff880038a0ce70 [141480.489001] <0> ffffffffa027f890 ffff88022ead1300 ffff88022ead1228 ffffffffa0272aa0 [141480.489001] <0> 000000000000000e 0000000000000000 000000000000000e 0000000000000001 [141480.489001] Call Trace: [141480.489001] [<ffffffffa0272aa0>] __ext4_handle_dirty_metadata+0xcc/0xd7 [ext4] [141480.645267] [<ffffffffa0255285>] ext4_ind_get_blocks+0x78d/0x910 [ext4] [141480.645267] [<ffffffff81303833>] ? __down_write_nested+0x1a/0xc5 [141480.645267] [<ffffffffa0255573>] ext4_get_blocks+0x16b/0x3ef [ext4] [141480.645267] [<ffffffff8102b165>] ? select_task_rq_fair+0x4ce/0x89b [141480.645267] [<ffffffffa0255cc4>] mpage_da_map_blocks+0xb4/0x613 [ext4] [141480.645267] [<ffffffff8102be86>] ? enqueue_task_fair+0x3f/0x82 [141480.645267] [<ffffffff81024676>] ? enqueue_task+0x5b/0x66 [141480.645267] [<ffffffff81024785>] ? activate_task+0x28/0x31 [141480.645267] [<ffffffff8115fef1>] ? cpumask_next_and+0x2b/0x3c [141480.645267] [<ffffffff81163802>] ? radix_tree_gang_lookup_tag_slot+0x85/0xaa [141480.645267] [<ffffffffa02569bb>] mpage_add_bh_to_extent+0x94/0xab [ext4] [141480.645267] [<ffffffff810a3078>] ? __dec_zone_page_state+0x1e/0x20 [141480.645267] [<ffffffffa0256ad7>] __mpage_da_writepage+0x105/0x14d [ext4] [141480.645267] [<ffffffff81097a7b>] write_cache_pages+0x1ee/0x2f3 [141480.645267] [<ffffffffa02569d2>] ? __mpage_da_writepage+0x0/0x14d [ext4] [141480.645267] [<ffffffffa0256701>] ext4_da_writepages+0x4de/0x704 [ext4] [141480.645267] [<ffffffff81301829>] ? schedule+0x632/0xba3 [141480.645267] [<ffffffff81097bc1>] do_writepages+0x1c/0x25 [141480.645267] [<ffffffff810dabac>] writeback_single_inode+0xea/0x2df [141480.645267] [<ffffffff810dae48>] write_inode_now+0x78/0xba [141480.645267] [<ffffffffa0255b2d>] ext4_da_get_block_prep+0x28b/0x36e [ext4] [141480.645267] [<ffffffff810e1b69>] __block_prepare_write+0x1f3/0x3c0 [141480.645267] [<ffffffffa02558a2>] ? ext4_da_get_block_prep+0x0/0x36e [ext4] [141480.645267] [<ffffffff810e1eb1>] block_write_begin+0x80/0xcf [141480.645267] [<ffffffffa02587be>] ext4_da_write_begin+0x196/0x218 [ext4] [141480.645267] [<ffffffffa02558a2>] ? ext4_da_get_block_prep+0x0/0x36e [ext4] [141480.645267] [<ffffffff810911c1>] ? iov_iter_copy_from_user_atomic+0x84/0xf6 [141480.645267] [<ffffffff8109133f>] generic_file_buffered_write+0x10c/0x230 [141480.645267] [<ffffffffa025375b>] ? ext4_dirty_inode+0x40/0x45 [ext4] [141480.645267] [<ffffffff81092b9c>] __generic_file_aio_write+0x347/0x37b [141480.645267] [<ffffffff81302709>] ? __mutex_lock_slowpath+0x22d/0x23c [141480.645267] [<ffffffff81092c2b>] generic_file_aio_write+0x5b/0xa2 [141480.645267] [<ffffffffa024fa93>] ext4_file_write+0x8e/0x97 [ext4] [141480.645267] [<ffffffff810bfbc8>] do_sync_write+0xc0/0x106 [141480.645267] [<ffffffff810cdb68>] ? poll_select_copy_remaining+0xc5/0xe9 [141480.645267] [<ffffffff81125b50>] ? security_file_permission+0x11/0x13 [141480.645267] [<ffffffff810c04df>] vfs_write+0xae/0x14a [141480.645267] [<ffffffff810c063f>] sys_write+0x47/0x6e [141480.645267] [<ffffffff81001feb>] system_call_fastpath+0x16/0x1b [141480.645267] Code: 00 40 00 75 f3 f0 41 0f ba 6d 00 16 19 c0 85 c0 75 e8 41 83 7c 24 10 00 75 1b 41 c7 44 24 10 01 00 00 00 41 8b 46 08 85 c0 7f 04 <0f> 0b eb fe ff c8 41 89 46 08 49 39 54 24 28 75 12 41 83 7c 24 [141480.645267] RIP [<ffffffffa022f6d8>] jbd2_journal_dirty_metadata+0x69/0x100 [jbd2] [141480.645267] RSP <ffff88022ead11c8> [141480.937330] ---[ end trace 4eacf5860ca2410a ]---
Note that the file system was fsck-ed before last mount, so it should be okay unless ext4 ate it itself somehow.
Signature · 3 lines
Configure bugmail: http://bugzilla.kernel.org/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching the assignee of the bug.
[Bug 15025] Oops in ext4 driver
http://bugzilla.kernel.org/show_bug.cgi?id=15025
Rafael J. Wysocki <[email protected]> changed:
What |Removed |Added
----------------------------------------------------------------------------
CC| |[email protected] Blocks| |14885
--- Comment #1 from Rafael J. Wysocki <[email protected]> 2010-01-13 22:09:30 --- Since you marked it as a regression, what was the last working kernel?
Signature · 3 lines
Configure bugmail: http://bugzilla.kernel.org/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching the assignee of the bug.
[Bug 15025] Oops in ext4 driver
In reply to [email protected]http://bugzilla.kernel.org/show_bug.cgi?id=15025
--- Comment #2 from Steinar H. Gunderson <[email protected]> 2010-01-13 22:16:04 ---
Quoted 2 lines · On Wed, Jan 13, 2010 at 10:09:32PM +0000, [email protected]…
--- Comment #1 from Rafael J. Wysocki <[email protected]> 2010-01-13 22:09:30 --- Since you marked it as a regression, what was the last working kernel?
I upgraded from 2.6.32-rc5, where I didn't see this issue. 2.6.32.3 works fine on the same machine and filesystem.
/* Steinar */
Signature · 3 lines
Configure bugmail: http://bugzilla.kernel.org/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching the assignee of the bug.
[Bug 15025] Oops in ext4 driver
In reply to [email protected]http://bugzilla.kernel.org/show_bug.cgi?id=15025
--- Comment #3 from Rafael J. Wysocki <[email protected]> 2010-01-24 23:07:23 ---
Quoted 7 lines · Steinar H. Gunderson
Quoted 3 lines · Rafael J. Wysocki
The following bug entry is on the current list of known regressions from 2.6.32. Please verify if it still should be listed and let me know (either way).
I'm not using 2.6.33 anymore since this bug is a showstopper to me (it's on a production system), so I'm unable to check if it's fixed or not.
Signature · 3 lines
Configure bugmail: http://bugzilla.kernel.org/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching the assignee of the bug.
[Bug 15025] Oops in ext4 driver
In reply to [email protected]http://bugzilla.kernel.org/show_bug.cgi?id=15025
Theodore Tso <[email protected]> changed:
What |Removed |Added
----------------------------------------------------------------------------
CC| |[email protected]
--- Comment #4 from Theodore Tso <[email protected]> 2010-01-27 19:35:09 --- Hi Steiner,
Sorry for not getting back to you right away; I've been doing a huge amount of
travel right during January. Can you tell me something about the file system
workload on your machine? What does it do? NFS, rsync server, backups, ...?
And do you know what it might be doing right before it crashed? How easily
can you reproduce this? I take it since you had to stop using 2.6.33-rcX you could reproduce it easily?
If you are willing to try a 2.6.33-rcX kernel, I'd suggest seeing if "echo 0 > /sys/fs/ext4/<dev>/max_writeback_mb_bump" makes the crashes go away.
Signature · 3 lines
Configure bugmail: http://bugzilla.kernel.org/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching the assignee of the bug.
[Bug 15025] Oops in ext4 driver
In reply to [email protected]http://bugzilla.kernel.org/show_bug.cgi?id=15025
--- Comment #5 from Steinar H. Gunderson <[email protected]> 2010-01-27 20:23:43 ---
Quoted 3 lines · On Wed, Jan 27, 2010 at 07:35:11PM +0000, [email protected]…
Sorry for not getting back to you right away; I've been doing a huge amount of
travel right during January. Can you tell me something about the file system
workload on your machine? What does it do? NFS, rsync server, backups, ...?
IIRC this was a file system that was mainly used for video storage and transcoding -- I think I was encoding a video with x264 to it when it crashed. Apart from that the machine spends most of its I/O time doing web serving from relatively large (1-2TB) data sets, and occasionally rtorrent.
It was recently online expanded, so I thought that might be related, but the problem persisted after a reboot and a forced fsck, so there was no on-disk corruption involved.
Quoted 3 lines
And do you know what it might be doing right before it crashed? How easily
can you reproduce this? I take it since you had to stop using 2.6.33-rcX you could reproduce it easily?
It crashed two times in two days or something after I upgraded to 2.6.33-rcX. Not a statistically huge sample, I'm afraid.
Quoted 2 lines
If you are willing to try a 2.6.33-rcX kernel, I'd suggest seeing if "echo 0 > /sys/fs/ext4/<dev>/max_writeback_mb_bump" makes the crashes go away.
I'm afraid it's not so easy for me to do reboots into new kernels on this machine; kernel upgrades generally happen when the machine is booted for some other reason. :-/
/* Steinar */
Signature · 3 lines
Configure bugmail: http://bugzilla.kernel.org/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching the assignee of the bug.
[Bug 15025] Oops in ext4 driver
In reply to [email protected]http://bugzilla.kernel.org/show_bug.cgi?id=15025
--- Comment #6 from Steinar H. Gunderson <[email protected]> 2010-02-16 13:44:37 --- There was a power drop (too long for the UPS), so I've now run 2.6.33-rc8 on this same machine for about 24 hours without seeing any ext4 errors. The load is probably different, though, but at least it doesn't seem to bite me anymore.
Signature · 3 lines
Configure bugmail: http://bugzilla.kernel.org/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching the assignee of the bug.
[Bug 15025] Oops in ext4 driver
In reply to [email protected]http://bugzilla.kernel.org/show_bug.cgi?id=15025
Dmitry Monakhov <[email protected]> changed:
What |Removed |Added
----------------------------------------------------------------------------
CC| |[email protected]
--- Comment #7 from Dmitry Monakhov <[email protected]> 2010-02-16 14:08:29 ---
The issue was fixed by following commit
http://git.kernel.org/?p=linux/kernel/git/tytso/ext4.git;a=commit;h=1db913823c0f8360fccbd24ca67eb073966a5ffd
Test case:
dmon$ sudo mount /dev/sdd /mnt -oquota
dmon$ set-quota-limit /mnt id=dmon --bsoft=1000 --bsoft=1000 dmon$ dd if=/dev/zefo of=/mnt/file
Please close the bug
Signature · 3 lines
Configure bugmail: http://bugzilla.kernel.org/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching the assignee of the bug.
[Bug 15025] Oops in ext4 driver
In reply to [email protected]http://bugzilla.kernel.org/show_bug.cgi?id=15025
Rafael J. Wysocki <[email protected]> changed:
What |Removed |Added
----------------------------------------------------------------------------
Status|NEW |RESOLVED
Resolution| |CODE_FIXSignature · 3 lines
Configure bugmail: http://bugzilla.kernel.org/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching the assignee of the bug.
[Bug 15025] Oops in ext4 driver
In reply to [email protected]http://bugzilla.kernel.org/show_bug.cgi?id=15025
Rafael J. Wysocki <[email protected]> changed:
What |Removed |Added
----------------------------------------------------------------------------
Status|RESOLVED |CLOSED
--- Comment #8 from Rafael J. Wysocki <[email protected]> 2010-02-16 21:00:35 --- Thanks, closing.
Signature · 3 lines
Configure bugmail: http://bugzilla.kernel.org/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching the assignee of the bug.
[Bug 15025] Oops in ext4 driver
In reply to [email protected]http://bugzilla.kernel.org/show_bug.cgi?id=15025
Steinar H. Gunderson <[email protected]> changed:
What |Removed |Added
----------------------------------------------------------------------------
Resolution|CODE_FIX |UNREPRODUCIBLE
--- Comment #9 from Steinar H. Gunderson <[email protected]> 2010-02-16 23:56:59 ---
Dmitry: That seems impossible, as I'm not using quota on the machine in question (it's not even compiled into the kernel).
Signature · 3 lines
Configure bugmail: http://bugzilla.kernel.org/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching the assignee of the bug.
[Bug 15025] Oops in ext4 driver
In reply to [email protected]http://bugzilla.kernel.org/show_bug.cgi?id=15025
--- Comment #10 from Steinar H. Gunderson <[email protected]> 2010-02-17 00:21:40 ---
Quoted 2 lines · On Tue, Feb 16, 2010 at 02:08:33PM +0000, [email protected]…
The issue was fixed by following commit
http://git.kernel.org/?p=linux/kernel/git/tytso/ext4.git;a=commit;h=1db913823c0f8360fccbd24ca67eb073966a5ffd
This cannot be, as I don't use quota.
/* Steinar */
Signature · 3 lines
Configure bugmail: http://bugzilla.kernel.org/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching the assignee of the bug.
[Bug 15025] Oops in ext4 driver
In reply to [email protected]http://bugzilla.kernel.org/show_bug.cgi?id=15025
--- Comment #11 from Dmitry Monakhov <[email protected]> 2010-02-17 13:47:27 --- After the patch i can not trigger the bug (In reply to comment #10)
Quoted 6 lines
Quoted 2 lines · On Tue, Feb 16, 2010 at 02:08:33PM +0000, [email protected]…
The issue was fixed by following commit
http://git.kernel.org/?p=linux/kernel/git/tytso/ext4.git;a=commit;h=1db913823c0f8360fccbd24ca67eb073966a5ffdThis cannot be, as I don't use quota.
It i is posible to triger the bug without quota.
Untill the patch it we have following codefs/ext4/inode.c:
1850 if (ext4_claim_free_blocks(sbi, md_needed + 1)) {
1851 vfs_dq_release_reservation_block(inode, md_needed + 1);
1852 if (ext4_should_retry_alloc(inode->i_sb, &retries)) {
1853 retry:1854 if (md_reserved)
1855 write_inode_now(inode, (retries == 3));
^^^^^^^^^^ Here we goes in to lack of journal credits. ^^^^^^^^^^^^^^^^^^^^^^
1856 yield();
1857 goto repeat;
1858 }
1859 return -ENOSPC;
1860 }
You have failed exactly here.
So the bug happens even in case of ENOSPC (try following testase):
dd if=/dev/zero /mnt/BIG_FILE bs=1M
But it takes longer if partition is really huge.
Signature · 3 lines
Configure bugmail: http://bugzilla.kernel.org/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching the assignee of the bug.
[Bug 15025] Oops in ext4 driver
In reply to [email protected]http://bugzilla.kernel.org/show_bug.cgi?id=15025
--- Comment #12 from Dmitry Monakhov <[email protected]> 2010-02-17 13:51:25 --- Since calling "write_inode_now" from ext4_da_get_block_prep was the core of the issue. And the patch move it to an upper level. So the issue was completely fixed. Please close the bug as CODE_FIXED
Signature · 3 lines
Configure bugmail: http://bugzilla.kernel.org/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching the assignee of the bug.