2.5.60-BK reproducible oops, during LTP run
I have reproduced the BUG in fs/buffer.c:2533 twice now. Test conditions exactly the same, fsx-linux in one window, LTP in another window.
The machine stays alive for pings and sysrq's, but I cannot ssh into it nor login at the console. sysrq-s initiates a sync of the root filesystem on the ATA disk but never finishes. other sysrq's and pings continues to work after the sysrq-s invocation.
Call trace from BUG, each time I hit it:
EIP in submit_bhll_rw_blockjournal_commit_transaction
scheduledefault_wake_function
kjournaldcommit_timeout
kjournaldkernel_thread_helper
sysrq-t bits (I note truncate and sys_sync as common elements):
* shows that LTP test "ftest08" is running
* fsx-linux stack trace: io_schedule, __wait_on_buffer, free_hot_cold_page, autoremove_wake_function, autoremovce_wake_function, journal_invalidatepage, ext2_invalidatepage, do_invalidatepage, truncate_complete_page, truncate_inode_pages, vmtruncate, inode_setattr, ext3_setattr, notify_change, do_truncate, do_sys_ftruncate, sys_ftruncate, syscall_call
* ftest08 stack trace, process 0: sys_wait4, do_fork, default_wake_function, default_wake_function, sycall_call
* ftest08 trace, proc 1: __down, default_wake_function, __down_failed, .text.ock.read_write, sys_llseek, sys_sync, syscall_call
* ftest08 trace, proc 2: do_readv_writev, __down, default_wake_function, __down_failed, .text.lock.read_write, sys_llseek, syscall_call
* ftest08 trace, proc 3: sleep_on, default_wake_function, log_wait_commit, journal_stop, journal_force_commit, write_inode, __sync_single_inode, sync_sb_inodes, sync_inodes_sb, sync_inodes, sys_sync, syscall_call
* ftest08 trace, proc 4: do_writepages, __down, default_wake_function, __down_failed, .text.lock.read_write, sync_blockdev, sys_llseek, sys_sync, syscall_call
* ftest08 trace, proc 5: sleep_on, default_wake_function, log_wait_commit, journal_stop, journal_force_commit, ext3_force_commit, ext3_sync_file, sys_fsync, syscall_call
sysrq-m bits:
free pages: 5552kB (0kB highmem)
active: 48451, inactive: 9478 dirty:10 writeback:0 free:1388
DMA free: 2368kB, min 128kB, low:256kB, high:384kB, active:3776kB, inactive:6688kB Normal free: 3184kB min:1020 low:2040 high:3060 active:190028 inactive:31224 swap cache: add 93, delete 92, find 24/26, race 0+0
free swap: 2096776kB
63472 pages of RAM
1471 reserved pages
27697 pages shared
1 pages swap cached
hardware bits:
via c3 cpu
gcc 3.2.2
kernel 2.5.60-bk2
one 40GB ATA/100 drive, running at ATA/100
256 MB RAM
Quoted 4 lines · Jeff Garzik
I have reproduced the BUG in
fs/buffer.c:2533twice now. Test conditions exactly the same, fsx-linux in one window, LTP in another window.
Thanks. It's a long-standing but benign bogon which was exposed by recent
ext3 simplifications. This needs lots of testing.
when ext3_writepage races with truncate, block_write_full_page() will see that the page is outside i_size and will bale out with -EIO. But ext3_writepage() will ignore this and will proceed to add the buffers to the transaction.
Later, kjournald tries to write them out and goes BUG() because those buffers are not mapped to disk.
The fix is to not attach the buffers to the transaction in ext3_writepage() if block_write_full_page() failed.
So far so good, but that page now has dirty, unmapped buffers (the buffers were attached in a dirty state by ext3_writepage()). So teach block_write_full_page() to clean the buffers against the page if it is wholly outside i_size.
(A simpler fix to all of this might be to just bale out of ext3_writepage() if the page is outside i_size. But that is racy against block_write_full_page()'s subsequent execution of the same comparison).
buffer.c | 6 ++++++ ext3/inode.c | 13 ++++++++++--- 2 files changed, 16 insertions(+), 3 deletions(-)
fs/ext3/inode.c renamed from fs/ext3/inode.c~ext3-eio-fix +10 -3
@@ -1357,10 +1357,17 @@ static int ext3_writepage(struct page *p handle =ext3_journal_current_handle ();lock_kernel (); -/* And attach them to the current transaction */ +/* +* And attach them to the current transaction. But only if +* block_write_full_page() succeeded. Otherwise they are unmapped, +* and generally junk. +*/ if (order_data) { - err =walk_page_buffers (handle, page_bufs, -0 ,PAGE_CACHE_SIZE ,NULL , ext3_journal_dirty_data); +if (ret ==0 ) { + err =walk_page_buffers (handle, page_bufs, +0 ,PAGE_CACHE_SIZE ,NULL , + ext3_journal_dirty_data); + }walk_page_buffers (handle, page_bufs,0 ,PAGE_CACHE_SIZE ,NULL , bput_one);if (!ret)
fs/buffer.c renamed from fs/buffer.c~ext3-eio-fix +6 -0
@@ -2502,6 +2502,12 @@ int block_write_full_page(struct page *p/* Is the page fully outside i_size? (truncate in progress) */ offset = inode->i_size & (PAGE_CACHE_SIZE -1 );if (page->index >= end_index+1 || !offset) { +/* +* The page may have dirty, unmapped buffers. For example, +* they may have been added in ext3_writepage(). Make them +* freeable here, so the page does not leak. +*/ +block_invalidatepage (page,0 );unlock_page (page);return -EIO ; }
_
Thanks for chasing the bug down. I'll beat it up overnight.