Skip to contentSkip to conversation

Ext3 and/or SCSI bug?

DISCUSSION Gertjan van Wingerde 2 participants 2 messages Feb 19, 2004

lore
Share & tools
lore
#0 lore raw

Hi all,

(Please CC: me on replies, I'm not subscribed to the list)

Below is an part of my dmesg output that I captured when all of a sudden one of my filesystems became mounted read-only (while some heavy reading/ writing was being performed to that filesystem).

Can anyone explain to me what is going on here, and why this is happening to me about every 10 days.
Please note that this happened twice now in the last 10 days, both while running kernel version 2.6.2 and kernel 2.6.3 (for the last day). After the first time I've fsck'ed the file-system thoroughly.

System details:
Dual Athlon MP2000+, Asus A7M266-D motherboard.
QLA12160 based SCSI card connected to 3 Maxtor Atlas IV 10K RPM drives.

Best regards,

Gertjan.

Kernel output:

EXT3-fs error (device sdb8): ext3_free_blocks: bit already cleared for block 1142880 Aborting journal on device sdb8.
bad: scheduling while atomic!

 Call Trace:
  [<c012049e>] schedule+0x68e/0x6a0
  [<c021cce0>] scsi_request_fn+0x240/0x390
  [<c020d49d>] generic_unplug_device+0x8d/0x90
  [<c0121776>] io_schedule+0x26/0x40
  [<c0163574>] __wait_on_buffer+0xc4/0xd0
  [<c01229d0>] autoremove_wake_function+0x0/0x50
  [<c01229d0>] autoremove_wake_function+0x0/0x50
  [<c016754b>] sync_dirty_buffer+0x4b/0xb0
  [<c01bb5d5>] journal_update_superblock+0x95/0x130
  [<c01ac5f9>] ext3_handle_error+0x59/0xb0
  [<c01ac6a5>] ext3_error+0x55/0x60
  [<c01a0d03>] ext3_free_blocks+0x523/0x5d0
  [<c01a6958>] ext3_free_data+0x128/0x150
  [<c01a6a6e>] ext3_free_branches+0xee/0x270
  [<c01a715e>] ext3_truncate+0x56e/0x670
  [<c01b34b9>] journal_start+0xa9/0xd0
  [<c01a4203>] start_transaction+0x23/0x60
  [<c01a4396>] ext3_delete_inode+0xc6/0x110
  [<c01a42d0>] ext3_delete_inode+0x0/0x110
  [<c017efcb>] generic_delete_inode+0xab/0x190
  [<c017f2b2>] iput+0x62/0x90
  [<c017b855>] dput+0x115/0x250
  [<c0174a6e>] sys_rename+0x1fe/0x220
  [<c0154e5f>] do_munmap+0x16f/0x1e0
  [<c0154f15>] sys_munmap+0x45/0x70
  [<c010959b>] syscall_call+0x7/0xb

bad: scheduling while atomic!

 Call Trace:
  [<c012049e>] schedule+0x68e/0x6a0
  [<c020ec2a>] generic_make_request+0x16a/0x1f0
  [<c0121776>] io_schedule+0x26/0x40
  [<c0163574>] __wait_on_buffer+0xc4/0xd0
  [<c01229d0>] autoremove_wake_function+0x0/0x50
  [<c01229d0>] autoremove_wake_function+0x0/0x50
  [<c016754b>] sync_dirty_buffer+0x4b/0xb0
  [<c01ac60d>] ext3_handle_error+0x6d/0xb0
  [<c01ac6a5>] ext3_error+0x55/0x60
  [<c01a0d03>] ext3_free_blocks+0x523/0x5d0
  [<c01a6958>] ext3_free_data+0x128/0x150
  [<c01a6a6e>] ext3_free_branches+0xee/0x270
  [<c01a715e>] ext3_truncate+0x56e/0x670
  [<c01b34b9>] journal_start+0xa9/0xd0
  [<c01a4203>] start_transaction+0x23/0x60
  [<c01a4396>] ext3_delete_inode+0xc6/0x110
  [<c01a42d0>] ext3_delete_inode+0x0/0x110
  [<c017efcb>] generic_delete_inode+0xab/0x190
  [<c017f2b2>] iput+0x62/0x90
  [<c017b855>] dput+0x115/0x250
  [<c0174a6e>] sys_rename+0x1fe/0x220
  [<c0154e5f>] do_munmap+0x16f/0x1e0
  [<c0154f15>] sys_munmap+0x45/0x70
  [<c010959b>] syscall_call+0x7/0xb

EXT3-fs error (device sdb8): ext3_free_blocks: bit already cleared for block 1142881
bad: scheduling while atomic!

 Call Trace:
  [<c012049e>] schedule+0x68e/0x6a0
  [<c021cce0>] scsi_request_fn+0x240/0x390
  [<c020d49d>] generic_unplug_device+0x8d/0x90
  [<c0121776>] io_schedule+0x26/0x40
  [<c0163574>] __wait_on_buffer+0xc4/0xd0
  [<c01229d0>] autoremove_wake_function+0x0/0x50
  [<c01229d0>] autoremove_wake_function+0x0/0x50
  [<c016754b>] sync_dirty_buffer+0x4b/0xb0
  [<c01ac60d>] ext3_handle_error+0x6d/0xb0
  [<c01ac6a5>] ext3_error+0x55/0x60
  [<c01a0d03>] ext3_free_blocks+0x523/0x5d0
  [<c01a6958>] ext3_free_data+0x128/0x150
  [<c01a6a6e>] ext3_free_branches+0xee/0x270
  [<c01a715e>] ext3_truncate+0x56e/0x670
  [<c01b34b9>] journal_start+0xa9/0xd0
  [<c01a4203>] start_transaction+0x23/0x60
  [<c01a4396>] ext3_delete_inode+0xc6/0x110
  [<c01a42d0>] ext3_delete_inode+0x0/0x110
  [<c017efcb>] generic_delete_inode+0xab/0x190
  [<c017f2b2>] iput+0x62/0x90
  [<c017b855>] dput+0x115/0x250
  [<c0174a6e>] sys_rename+0x1fe/0x220
  [<c0154e5f>] do_munmap+0x16f/0x1e0
  [<c0154f15>] sys_munmap+0x45/0x70
  [<c010959b>] syscall_call+0x7/0xb

EXT3-fs error (device sdb8): ext3_free_blocks: bit already cleared for block 1142882
bad: scheduling while atomic!

 Call Trace:
  [<c012049e>] schedule+0x68e/0x6a0
  [<c020ec2a>] generic_make_request+0x16a/0x1f0
  [<c0121776>] io_schedule+0x26/0x40
  [<c0163574>] __wait_on_buffer+0xc4/0xd0
  [<c01229d0>] autoremove_wake_function+0x0/0x50
  [<c01229d0>] autoremove_wake_function+0x0/0x50
  [<c016754b>] sync_dirty_buffer+0x4b/0xb0
  [<c01ac60d>] ext3_handle_error+0x6d/0xb0
  [<c01ac6a5>] ext3_error+0x55/0x60
  [<c01a0d03>] ext3_free_blocks+0x523/0x5d0
  [<c01a6958>] ext3_free_data+0x128/0x150
  [<c01a6a6e>] ext3_free_branches+0xee/0x270
  [<c01a715e>] ext3_truncate+0x56e/0x670
  [<c01b34b9>] journal_start+0xa9/0xd0
  [<c01a4203>] start_transaction+0x23/0x60
  [<c01a4396>] ext3_delete_inode+0xc6/0x110
  [<c01a42d0>] ext3_delete_inode+0x0/0x110
  [<c017efcb>] generic_delete_inode+0xab/0x190
  [<c017f2b2>] iput+0x62/0x90
  [<c017b855>] dput+0x115/0x250
  [<c0174a6e>] sys_rename+0x1fe/0x220
  [<c0154e5f>] do_munmap+0x16f/0x1e0
  [<c0154f15>] sys_munmap+0x45/0x70
  [<c010959b>] syscall_call+0x7/0xb

EXT3-fs error (device sdb8): ext3_free_blocks: bit already cleared for block 1142883
bad: scheduling while atomic!

 Call Trace:
  [<c012049e>] schedule+0x68e/0x6a0
  [<c0214488>] as_next_request+0x38/0x50
  [<c020b7c6>] elv_next_request+0x16/0x100
  [<c020d490>] generic_unplug_device+0x80/0x90
  [<c0121776>] io_schedule+0x26/0x40
  [<c0163574>] __wait_on_buffer+0xc4/0xd0
  [<c01229d0>] autoremove_wake_function+0x0/0x50
  [<c01229d0>] autoremove_wake_function+0x0/0x50
  [<c016754b>] sync_dirty_buffer+0x4b/0xb0
  [<c01ac60d>] ext3_handle_error+0x6d/0xb0
  [<c01ac6a5>] ext3_error+0x55/0x60
  [<c01a0d03>] ext3_free_blocks+0x523/0x5d0
  [<c01a6958>] ext3_free_data+0x128/0x150
  [<c01a6a6e>] ext3_free_branches+0xee/0x270
  [<c01a715e>] ext3_truncate+0x56e/0x670
  [<c01b34b9>] journal_start+0xa9/0xd0
  [<c01a4203>] start_transaction+0x23/0x60
  [<c01a4396>] ext3_delete_inode+0xc6/0x110
  [<c01a42d0>] ext3_delete_inode+0x0/0x110
  [<c017efcb>] generic_delete_inode+0xab/0x190
  [<c017f2b2>] iput+0x62/0x90
  [<c017b855>] dput+0x115/0x250
  [<c0174a6e>] sys_rename+0x1fe/0x220
  [<c0154e5f>] do_munmap+0x16f/0x1e0
  [<c0154f15>] sys_munmap+0x45/0x70
  [<c010959b>] syscall_call+0x7/0xb

EXT3-fs error (device sdb8): ext3_free_blocks: bit already cleared for block 1142884
bad: scheduling while atomic!

 Call Trace:
  [<c012049e>] schedule+0x68e/0x6a0
  [<c020ec2a>] generic_make_request+0x16a/0x1f0
  [<c0121776>] io_schedule+0x26/0x40
  [<c0163574>] __wait_on_buffer+0xc4/0xd0
  [<c01229d0>] autoremove_wake_function+0x0/0x50
  [<c01229d0>] autoremove_wake_function+0x0/0x50
  [<c016754b>] sync_dirty_buffer+0x4b/0xb0
  [<c01ac60d>] ext3_handle_error+0x6d/0xb0
  [<c01ac6a5>] ext3_error+0x55/0x60
  [<c01a0d03>] ext3_free_blocks+0x523/0x5d0
  [<c01a6958>] ext3_free_data+0x128/0x150
  [<c01a6a6e>] ext3_free_branches+0xee/0x270
  [<c01a715e>] ext3_truncate+0x56e/0x670
  [<c01b34b9>] journal_start+0xa9/0xd0
  [<c01a4203>] start_transaction+0x23/0x60
  [<c01a4396>] ext3_delete_inode+0xc6/0x110
  [<c01a42d0>] ext3_delete_inode+0x0/0x110
  [<c017efcb>] generic_delete_inode+0xab/0x190
  [<c017f2b2>] iput+0x62/0x90
  [<c017b855>] dput+0x115/0x250
  [<c0174a6e>] sys_rename+0x1fe/0x220
  [<c0154e5f>] do_munmap+0x16f/0x1e0
  [<c0154f15>] sys_munmap+0x45/0x70
  [<c010959b>] syscall_call+0x7/0xb

EXT3-fs error (device sdb8): ext3_free_blocks: bit already cleared for block 1142885
bad: scheduling while atomic!

 Call Trace:
  [<c012049e>] schedule+0x68e/0x6a0
  [<c0214488>] as_next_request+0x38/0x50
  [<c020b7c6>] elv_next_request+0x16/0x100
  [<c020d490>] generic_unplug_device+0x80/0x90
  [<c0121776>] io_schedule+0x26/0x40
  [<c0163574>] __wait_on_buffer+0xc4/0xd0
  [<c01229d0>] autoremove_wake_function+0x0/0x50
  [<c01229d0>] autoremove_wake_function+0x0/0x50
  [<c016754b>] sync_dirty_buffer+0x4b/0xb0
  [<c01ac60d>] ext3_handle_error+0x6d/0xb0
  [<c01ac6a5>] ext3_error+0x55/0x60
  [<c01a0d03>] ext3_free_blocks+0x523/0x5d0
  [<c01a6958>] ext3_free_data+0x128/0x150
  [<c01a6a6e>] ext3_free_branches+0xee/0x270
  [<c01a715e>] ext3_truncate+0x56e/0x670
  [<c01b34b9>] journal_start+0xa9/0xd0
  [<c01a4203>] start_transaction+0x23/0x60
  [<c01a4396>] ext3_delete_inode+0xc6/0x110
  [<c01a42d0>] ext3_delete_inode+0x0/0x110
  [<c017efcb>] generic_delete_inode+0xab/0x190
  [<c017f2b2>] iput+0x62/0x90
  [<c017b855>] dput+0x115/0x250
  [<c0174a6e>] sys_rename+0x1fe/0x220
  [<c0154e5f>] do_munmap+0x16f/0x1e0
  [<c0154f15>] sys_munmap+0x45/0x70
  [<c010959b>] syscall_call+0x7/0xb

EXT3-fs error (device sdb8): ext3_free_blocks: bit already cleared for block 1142886
bad: scheduling while atomic!

 Call Trace:
  [<c012049e>] schedule+0x68e/0x6a0
  [<c0214488>] as_next_request+0x38/0x50
  [<c020b7c6>] elv_next_request+0x16/0x100
  [<c020d490>] generic_unplug_device+0x80/0x90
  [<c0121776>] io_schedule+0x26/0x40
  [<c0163574>] __wait_on_buffer+0xc4/0xd0
  [<c01229d0>] autoremove_wake_function+0x0/0x50
  [<c01229d0>] autoremove_wake_function+0x0/0x50
  [<c016754b>] sync_dirty_buffer+0x4b/0xb0
  [<c01ac60d>] ext3_handle_error+0x6d/0xb0
  [<c01ac6a5>] ext3_error+0x55/0x60
  [<c01a0d03>] ext3_free_blocks+0x523/0x5d0
  [<c01a6958>] ext3_free_data+0x128/0x150
  [<c01a6a6e>] ext3_free_branches+0xee/0x270
  [<c01a715e>] ext3_truncate+0x56e/0x670
  [<c01b34b9>] journal_start+0xa9/0xd0
  [<c01a4203>] start_transaction+0x23/0x60
  [<c01a4396>] ext3_delete_inode+0xc6/0x110
  [<c01a42d0>] ext3_delete_inode+0x0/0x110
  [<c017efcb>] generic_delete_inode+0xab/0x190
  [<c017f2b2>] iput+0x62/0x90
  [<c017b855>] dput+0x115/0x250
  [<c0174a6e>] sys_rename+0x1fe/0x220
  [<c0154e5f>] do_munmap+0x16f/0x1e0
  [<c0154f15>] sys_munmap+0x45/0x70
  [<c010959b>] syscall_call+0x7/0xb

EXT3-fs error (device sdb8): ext3_free_blocks: bit already cleared for block 1142887
bad: scheduling while atomic!

 Call Trace:
  [<c012049e>] schedule+0x68e/0x6a0
  [<c0214488>] as_next_request+0x38/0x50
  [<c020b7c6>] elv_next_request+0x16/0x100
  [<c020d490>] generic_unplug_device+0x80/0x90
  [<c0121776>] io_schedule+0x26/0x40
  [<c0163574>] __wait_on_buffer+0xc4/0xd0
  [<c01229d0>] autoremove_wake_function+0x0/0x50
  [<c01229d0>] autoremove_wake_function+0x0/0x50
  [<c016754b>] sync_dirty_buffer+0x4b/0xb0
  [<c01ac60d>] ext3_handle_error+0x6d/0xb0
  [<c01ac6a5>] ext3_error+0x55/0x60
  [<c01a0d03>] ext3_free_blocks+0x523/0x5d0
  [<c01a6958>] ext3_free_data+0x128/0x150
  [<c01a6a6e>] ext3_free_branches+0xee/0x270
  [<c01a715e>] ext3_truncate+0x56e/0x670
  [<c01b34b9>] journal_start+0xa9/0xd0
  [<c01a4203>] start_transaction+0x23/0x60
  [<c01a4396>] ext3_delete_inode+0xc6/0x110
  [<c01a42d0>] ext3_delete_inode+0x0/0x110
  [<c017efcb>] generic_delete_inode+0xab/0x190
  [<c017f2b2>] iput+0x62/0x90
  [<c017b855>] dput+0x115/0x250
  [<c0174a6e>] sys_rename+0x1fe/0x220
  [<c0154e5f>] do_munmap+0x16f/0x1e0
  [<c0154f15>] sys_munmap+0x45/0x70
  [<c010959b>] syscall_call+0x7/0xb

ext3_abort called.
EXT3-fs abort (device sdb8): ext3_journal_start: Detected aborted journal
Remounting filesystem read-only
ext3_reserve_inode_write: aborting transaction: Journal has aborted in __ext3_journal_get_write_access<2>EXT3-fs error (device sdb8) in ext3_reserve_inode_write: Journal has aborted EXT3-fs error (device sdb8) in ext3_truncate: Journal has aborted
ext3_reserve_inode_write: aborting transaction: Journal has aborted in __ext3_journal_get_write_access<2>EXT3-fs error (device sdb8) in ext3_reserve_inode_write: Journal has aborted EXT3-fs error (device sdb8) in ext3_orphan_del: Journal has aborted
ext3_reserve_inode_write: aborting transaction: Journal has aborted in __ext3_journal_get_write_access<2>EXT3-fs error (device sdb8) in ext3_reserve_inode_write: Journal has aborted EXT3-fs error (device sdb8) in ext3_delete_inode: Journal has aborted
EXT3-fs error (device sdb8) in start_transaction: Journal has aborted

Feb 19 19:46:05 wingerd last message repeated 30 times
 __journal_remove_journal_head: freeing b_committed_data
#1 lore raw
Quoted 7 lines · Gertjan van Wingerde

Below is an part of my dmesg output that I captured when all of a sudden one of my filesystems became mounted read-only (while some heavy reading/ writing was being performed to that filesystem).

Can anyone explain to me what is going on here, and why this is happening to me about every 10 days.

I don't know why this:

Quoted 2 lines

EXT3-fs error (device sdb8): ext3_free_blocks: bit already cleared for block 1142880 Aborting journal on device sdb8.

is occurring. It indicates that the filesystem metadata is corrupted. You should force a fsck to fix it up. It could be a hardware problem (disk, cable, controller, power supply, or even main memory).

Quoted 3 lines

bad: scheduling while atomic!

 Call Trace:
  [<c012049e>] schedule+0x68e/0x6a0

This part is caused by a minor bug.

ext3_error() sleeps, so don't call it with the lock held.

---

 25-akpm/fs/ext3/balloc.c |    7 ++++---
 1 files changed, 4 insertions(+), 3 deletions(-)
fs/ext3/balloc.c renamed from fs/ext3/balloc.c~ext3-schedule-inside-lock-fix +4 -3
diff -puN fs/ext3/balloc.c~ext3-schedule-inside-lock-fix fs/ext3/balloc.c
@@ -239,9 +239,10 @@ do_more:
 		BUFFER_TRACE(bitmap_bh, "clear bit");
 		if (!ext3_clear_bit_atomic(sb_bgl_lock(sbi, block_group),
 						bit + i, bitmap_bh->b_data)) {
-			ext3_error (sb, __FUNCTION__,
-				      "bit already cleared for block %lu", 
-				      block + i);
+			jbd_unlock_bh_state(bitmap_bh);
+			ext3_error(sb, __FUNCTION__,
+				"bit already cleared for block %lu", block + i);
+			jbd_lock_bh_state(bitmap_bh);
 			BUFFER_TRACE(bitmap_bh, "bit already cleared");
 		} else {
 			dquot_freed_blocks++;

_