Skip to contentSkip to conversation

ext3_orphan_del may double-decrement bh->b_count

DISCUSSION Jeff Mahoney 3 participants 7 messages Jun 2, 2004 – Jun 3, 2004 VFS

lore
Share & tools
lore
#0 lore raw

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Andrew -

Chris Mason and I ran across this one while hunting down another bug.

If ext3_mark_iloc_dirty() fails in ext3_orphan_del() on the outer buffer, bh->b_count will be decremented twice. ext3_mark_iloc_dirty() will brelse the buffer, even on error. ext3_orphan_del() is explicity brelse'ing the buffer on error. Prior to calling ext3_mark_iloc_dirty(), this is the correct behavior.

Fix attached.

- -Jeff

- --
Jeff Mahoney
SuSE Labs
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.4 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org

iD8DBQFAvjI1LPWxlyuTD7IRAht5AJ9sM8mN2TiLb+RFCqHF/Fj/pVsuCgCfWXse izGrI5bgD1zhoM5sqXgkM5Q=
=Td4v
-----END PGP SIGNATURE-----

#

# ext3_mark_iloc_dirty brelse's the buffer even on error,
# jumping to the out_brelse label causes a double decrement to occur.
#
--- linux-2.5.orig/fs/ext3/namei.c.orig	2004-06-02 11:46:52.903565552 -0400
+++ linux-2.5/fs/ext3/namei.c	2004-06-02 11:47:00.494411568 -0400
@@ -1963,7 +1963,7 @@
 	NEXT_ORPHAN(inode) = 0;
 	err = ext3_mark_iloc_dirty(handle, inode, &iloc);
 	if (err)
-		goto out_brelse;
+		goto out_err;

out_err:
ext3_std_error(inode->i_sb, err);

#1 lore raw
Quoted 2 lines · Jeff Mahoney

Chris Mason and I ran across this one while hunting down another bug.

Thanks. Can be simplified ;)

fs/ext3/namei.c renamed from fs/ext3/namei.c~ext3_orphan_del-may-double-decrement-bh-b_count +0 -2
@@ -1971,8 +1971,6 @@ int ext3_orphan_del(handle_t *handle, st
 		goto out_brelse;
 	NEXT_ORPHAN(inode) = 0;
 	err = ext3_mark_iloc_dirty(handle, inode, &iloc);
-	if (err)
-		goto out_brelse;
 
 out_err:
 	ext3_std_error(inode->i_sb, err);

_

What was the "other bug"?

#3 lore raw
Quoted 10 lines · Chris Mason
Quoted 6 lines · Andrew Morton
Quoted 2 lines · Jeff Mahoney

Chris Mason and I ran across this one while hunting down another bug.

What was the "other bug"?

We've got many names for it, but none that could be posted here ;-)

intrigued.

Quoted 3 lines

Looks like HP came up with a simplified test case:

http://sourceforge.net/mailarchive/forum.php?thread_id=4536665&forum_id=6379

hm, I never received that. If I'd known I wouldn't have removed the buffer_error() stuff.

Quoted 1 line

I've got machines trying to reproduce now.

You need the buffer-tracing patch. This is against 2.6.7-rc2. It should spit a nice trace when you hit the problem. It'll tell us how that buffer got itself not uptodate.

 25-akpm/drivers/ide/ide-io.c         |   22 +++
 25-akpm/fs/Kconfig                   |    4 
 25-akpm/fs/Makefile                  |    2 
 25-akpm/fs/buffer.c                  |   76 +++++++++-
 25-akpm/fs/ext3/inode.c              |    7 
 25-akpm/fs/ext3/namei.c              |   40 +++++
 25-akpm/fs/ext3/super.c              |   81 +++++++++++
 25-akpm/fs/jbd-kernel.c              |  255 +++++++++++++++++++++++++++++++++++
 25-akpm/fs/jbd/commit.c              |    9 -
 25-akpm/fs/jbd/transaction.c         |   15 +-
 25-akpm/fs/mpage.c                   |    2 
 25-akpm/include/linux/buffer-trace.h |   85 +++++++++++
 25-akpm/include/linux/buffer_head.h  |    5 
 25-akpm/include/linux/jbd.h          |   12 -
 14 files changed, 597 insertions(+), 18 deletions(-)
drivers/ide/ide-io.c renamed from drivers/ide/ide-io.c~ext3-debug +22 -0
diff -puN drivers/ide/ide-io.c~ext3-debug drivers/ide/ide-io.c
@@ -591,6 +591,28 @@ ide_startstop_t start_request (ide_drive
 	ide_startstop_t startstop;
 	sector_t block;
 
+#ifdef CONFIG_JBD_DEBUG
+	/*
+	 * Silently stop writing to this disk to simulate a crash.
+	 */
+	extern struct block_device *journal_no_write[2];
+	int i;
+
+	if (!(rq->flags & REQ_RW))
+		goto its_a_read;
+
+	for (i = 0; i < 2; i++) {
+		struct block_device *bdev = journal_no_write[i];
+		if (bdev && bdev_get_queue(bdev) == rq->q) {
+			printk("%s: write suppressed\n", __FUNCTION__);
+			ide_end_request(drive, 1, rq->nr_sectors);
+			return ide_stopped;
+		}
+	}
+its_a_read:
+	;
+#endif
+
 	BUG_ON(!(rq->flags & REQ_STARTED));
 
 #ifdef DEBUG
fs/buffer.c renamed from fs/buffer.c~ext3-debug +72 -4

288 lines not on this page · raw message

fs/ext3/inode.c renamed from fs/ext3/inode.c~ext3-debug +7 -0
diff -puN fs/ext3/inode.c~ext3-debug fs/ext3/inode.c
@@ -914,6 +914,7 @@ struct buffer_head *ext3_getblk(handle_t
 
 	dummy.b_state = 0;
 	dummy.b_blocknr = -1000;
+	dummy.b_page = NULL;
 	buffer_trace_init(&dummy.b_history);
 	*errp = ext3_get_block_handle(handle, inode, block, &dummy, create, 1);
 	if (!*errp && buffer_mapped(&dummy)) {
@@ -1377,6 +1378,11 @@ static int ext3_ordered_writepage(struct
 	walk_page_buffers(handle, page_bufs, 0,
 			PAGE_CACHE_SIZE, NULL, bget_one);
 
+	{
+		struct buffer_head *b = page_buffers(page);
+		BUFFER_TRACE(b, "call block_write_full_page");
+	}
+
 	ret = block_write_full_page(page, ext3_get_block, wbc);
 
 	/*
@@ -1744,6 +1750,7 @@ static int ext3_block_truncate_page(hand
 	} else {
 		if (ext3_should_order_data(inode))
 			err = ext3_journal_dirty_data(handle, bh);
+		BUFFER_TRACE(bh, "mark dirty");
 		mark_buffer_dirty(bh);
 	}
 
fs/ext3/namei.c renamed from fs/ext3/namei.c~ext3-debug +37 -3
diff -puN fs/ext3/namei.c~ext3-debug fs/ext3/namei.c
@@ -60,6 +60,7 @@ static struct buffer_head *ext3_append(h
 		EXT3_I(inode)->i_disksize = inode->i_size;
 		ext3_journal_get_write_access(handle,bh);
 	}
+	BUFFER_TRACE(bh, "exit");
 	return bh;
 }
 
@@ -343,6 +344,7 @@ dx_probe(struct dentry *dentry, struct i
 		dir = dentry->d_parent->d_inode;
 	if (!(bh = ext3_bread (NULL,dir, 0, 0, err)))
 		goto fail;
+	BUFFER_TRACE(bh, "bread it");
 	root = (struct dx_root *) bh->b_data;
 	if (root->info.hash_version != DX_HASH_TEA &&
 	    root->info.hash_version != DX_HASH_HALF_MD4 &&
@@ -417,18 +419,21 @@ dx_probe(struct dentry *dentry, struct i
 
 		at = p - 1;
 		dxtrace(printk(" %x->%u\n", at == entries? 0: dx_get_hash(at), dx_get_block(at)));
+		BUFFER_TRACE(bh, "add to frame");
 		frame->bh = bh;
 		frame->entries = entries;
 		frame->at = at;
 		if (!indirect--) return frame;
 		if (!(bh = ext3_bread (NULL,dir, dx_get_block(at), 0, err)))
 			goto fail2;
+		BUFFER_TRACE(bh, "read it");
 		at = entries = ((struct dx_node *) bh->b_data)->entries;
 		assert (dx_get_limit(entries) == dx_node_limit (dir));
 		frame++;
 	}
 fail2:
 	while (frame >= frame_in) {
+		BUFFER_TRACE(bh, "brelse it");
 		brelse(frame->bh);
 		frame--;
 	}
@@ -441,8 +446,11 @@ static void dx_release (struct dx_frame 
 	if (frames[0].bh == NULL)
 		return;
 
-	if (((struct dx_root *) frames[0].bh->b_data)->info.indirect_levels)
+	if (((struct dx_root *) frames[0].bh->b_data)->info.indirect_levels) {
+		BUFFER_TRACE(frames[1].bh, "brelse it");
 		brelse(frames[1].bh);
+	}
+	BUFFER_TRACE(frames[0].bh, "brelse it");
 	brelse(frames[0].bh);
 }
 
@@ -951,6 +959,8 @@ static struct buffer_head * ext3_dx_find
 			dx_release (frames);
 			return bh;
 		}
+		BUFFER_TRACE(bh, "brelse it");
+		BUFFER_TRACE(bh, "brelse it");
 		brelse (bh);
 		/* Check to see if we should continue to search */
 		retval = ext3_htree_next_block(dir, hash, frame,
@@ -985,6 +995,7 @@ static struct dentry *ext3_lookup(struct
 	inode = NULL;
 	if (bh) {
 		unsigned long ino = le32_to_cpu(de->inode);
+		BUFFER_TRACE(bh, "brelse it");
 		brelse (bh);
 		inode = iget(dir->i_sb, ino);
 
@@ -1106,16 +1117,20 @@ static struct ext3_dir_entry_2 *do_split
 
 	bh2 = ext3_append (handle, dir, &newblock, error);
 	if (!(bh2)) {
+		BUFFER_TRACE(*bh, "brelse it");
 		brelse(*bh);
 		*bh = NULL;
 		goto errout;
 	}
 
+	BUFFER_TRACE(bh2, "using it");
 	BUFFER_TRACE(*bh, "get_write_access");
 	err = ext3_journal_get_write_access(handle, *bh);
 	if (err) {
 	journal_error:
+		BUFFER_TRACE(*bh, "brelse");
 		brelse(*bh);
+		BUFFER_TRACE(bh2, "brelse");
 		brelse(bh2);
 		*bh = NULL;
 		ext3_std_error(dir->i_sb, err);
@@ -1151,6 +1166,8 @@ static struct ext3_dir_entry_2 *do_split
 	/* Which block gets the new entry? */
 	if (hinfo->hash >= hash2)
 	{
+		BUFFER_TRACE(*bh, "swapping");
+		BUFFER_TRACE(bh2, "swapping");
 		swap(*bh, bh2);
 		de = de2;
 	}
@@ -1159,8 +1176,11 @@ static struct ext3_dir_entry_2 *do_split
 	if (err)
 		goto journal_error;
 	err = ext3_journal_dirty_metadata (handle, frame->bh);
-	if (err)
+	if (err) {
+		BUFFER_TRACE(bh2, "error");
 		goto journal_error;
+	}
+	BUFFER_TRACE(*bh, "brelse");
 	brelse (bh2);
 	dxtrace(dx_show_index ("frame", frame->entries));
 errout:
@@ -1203,6 +1223,7 @@ static int add_dirent_to_buf(handle_t *h
 				return -EIO;
 			}
 			if (ext3_match (namelen, name, de)) {
+				BUFFER_TRACE(bh, "brelse");
 				brelse (bh);
 				return -EEXIST;
 			}
@@ -1260,6 +1281,7 @@ static int add_dirent_to_buf(handle_t *h
 	err = ext3_journal_dirty_metadata(handle, bh);
 	if (err)
 		ext3_std_error(dir->i_sb, err);
+	BUFFER_TRACE(bh, "brelse");
 	brelse(bh);
 	return 0;
 }
@@ -1290,6 +1312,7 @@ static int make_indexed_dir(handle_t *ha
 
 	blocksize =  dir->i_sb->s_blocksize;
 	dxtrace(printk("Creating index\n"));
+	BUFFER_TRACE(bh, "get write access");
 	retval = ext3_journal_get_write_access(handle, bh);
 	if (retval) {
 		ext3_std_error(dir->i_sb, retval);
@@ -1334,6 +1357,7 @@ static int make_indexed_dir(handle_t *ha
 	frame = frames;
 	frame->entries = entries;
 	frame->at = entries;
+	BUFFER_TRACE(bh, "adding to frame");
 	frame->bh = bh;
 	bh = bh2;
 	de = do_split(handle,dir, &bh, frame, &hinfo, &retval);
@@ -1390,6 +1414,7 @@ static int ext3_add_entry (handle_t *han
 		bh = ext3_bread(handle, dir, block, 0, &retval);
 		if(!bh)
 			return retval;
+		BUFFER_TRACE(bh, "read it");
 		retval = add_dirent_to_buf(handle, dentry, inode, 0, bh);
 		if (retval != -ENOSPC)
 			return retval;
@@ -1404,6 +1429,7 @@ static int ext3_add_entry (handle_t *han
 	bh = ext3_append(handle, dir, &block, &retval);
 	if (!bh)
 		return retval;
+	BUFFER_TRACE(bh, "append returned it");
 	de = (struct ext3_dir_entry_2 *) bh->b_data;
 	de->inode = 0;
 	de->rec_len = cpu_to_le16(rlen = blocksize);
@@ -1469,6 +1495,7 @@ static int ext3_dx_add_entry(handle_t *h
 		bh2 = ext3_append (handle, dir, &newblock, &err);
 		if (!(bh2))
 			goto cleanup;
+		BUFFER_TRACE(bh2, "append returned it");
 		node2 = (struct dx_node *)(bh2->b_data);
 		entries2 = node2->entries;
 		node2->fake.rec_len = cpu_to_le16(sb->s_blocksize);
@@ -1498,6 +1525,8 @@ static int ext3_dx_add_entry(handle_t *h
 			if (at - entries >= icount1) {
 				frame->at = at = at - entries - icount1 + entries2;
 				frame->entries = entries = entries2;
+				BUFFER_TRACE(frame->bh, "swap");
+				BUFFER_TRACE(bh2, "swap");
 				swap(frame->bh, bh2);
 			}
 			dx_insert_block (frames + 0, hash2, newblock);
@@ -1507,6 +1536,7 @@ static int ext3_dx_add_entry(handle_t *h
 			err = ext3_journal_dirty_metadata(handle, bh2);
 			if (err)
 				goto journal_error;
+			BUFFER_TRACE(bh2, "brelse");
 			brelse (bh2);
 		} else {
 			dxtrace(printk("Creating second level index...\n"));
@@ -1523,6 +1553,8 @@ static int ext3_dx_add_entry(handle_t *h
 			frame = frames + 1;
 			frame->at = at = at - entries + entries2;
 			frame->entries = entries = entries2;
+			BUFFER_TRACE(frame->bh, "overwrite");
+			BUFFER_TRACE(bh2, "add to frame");
 			frame->bh = bh2;
 			err = ext3_journal_get_write_access(handle,
 							     frame->bh);
@@ -1541,8 +1573,10 @@ static int ext3_dx_add_entry(handle_t *h
 journal_error:
 	ext3_std_error(dir->i_sb, err);
 cleanup:
-	if (bh)
+	if (bh) {
+		BUFFER_TRACE(bh, "brelse");
 		brelse(bh);
+	}
 	dx_release(frames);
 	return err;
 }
fs/ext3/super.c renamed from fs/ext3/super.c~ext3-debug +81 -0

147 lines not on this page · raw message

fs/jbd/commit.c renamed from fs/jbd/commit.c~ext3-debug +7 -2
diff -puN fs/jbd/commit.c~ext3-debug fs/jbd/commit.c
@@ -28,10 +28,12 @@
 static void journal_end_buffer_io_sync(struct buffer_head *bh, int uptodate)
 {
 	BUFFER_TRACE(bh, "");
-	if (uptodate)
+	if (uptodate) {
 		set_buffer_uptodate(bh);
-	else
+	} else {
+		BUFFER_TRACE(bh, "clear uptodate");
 		clear_buffer_uptodate(bh);
+	}
 	unlock_buffer(bh);
 }
 
@@ -743,6 +745,7 @@ skip_commit: /* The journal should be un
 		 * behind for writeback and gets reallocated for another
 		 * use in a different page. */
 		if (buffer_freed(bh)) {
+			BUFFER_TRACE(bh, "buffer_freed");
 			clear_buffer_freed(bh);
 			clear_buffer_jbddirty(bh);
 		}
@@ -752,8 +755,10 @@ skip_commit: /* The journal should be un
 			__journal_insert_checkpoint(jh, commit_transaction);
 			JBUFFER_TRACE(jh, "refile for checkpoint writeback");
 			__journal_refile_buffer(jh);
+			JBUFFER_TRACE(jh, "did writepage hack");
 			jbd_unlock_bh_state(bh);
 		} else {
+			BUFFER_TRACE(bh, "not jbddirty");
 			J_ASSERT_BH(bh, !buffer_dirty(bh));
 			J_ASSERT_JH(jh, jh->b_next_transaction == NULL);
 			__journal_unfile_buffer(jh);
fs/jbd-kernel.c new file +255 -0

255 lines not on this page · raw message

fs/jbd/transaction.c renamed from fs/jbd/transaction.c~ext3-debug +13 -2
diff -puN fs/jbd/transaction.c~ext3-debug fs/jbd/transaction.c
@@ -1503,6 +1503,7 @@ void __journal_unfile_buffer(struct jour
 	transaction_t *transaction;
 	struct buffer_head *bh = jh2bh(jh);
 
+	JBUFFER_TRACE(jh, "entry");
 	J_ASSERT_JH(jh, jbd_is_locked_bh_state(bh));
 	transaction = jh->b_transaction;
 	if (transaction)
@@ -1545,8 +1546,10 @@ void __journal_unfile_buffer(struct jour
 
 	__blist_del_buffer(list, jh);
 	jh->b_jlist = BJ_None;
-	if (test_clear_buffer_jbddirty(bh))
+	if (test_clear_buffer_jbddirty(bh)) {
+		BUFFER_TRACE(bh, "marking dirty");
 		mark_buffer_dirty(bh);	/* Expose it to the VM */
+	}
 out:
 	jh->b_transaction = NULL;
 }
@@ -1688,6 +1691,7 @@ static int __dispose_buffer(struct journ
 	int may_free = 1;
 	struct buffer_head *bh = jh2bh(jh);
 
+	JBUFFER_TRACE(jh, "unfile");
 	__journal_unfile_buffer(jh);
 
 	if (jh->b_cp_transaction) {
@@ -1934,6 +1938,7 @@ void __journal_file_buffer(struct journa
 	int was_dirty = 0;
 	struct buffer_head *bh = jh2bh(jh);
 
+	BUFFER_TRACE(bh, "entry");
 	J_ASSERT_JH(jh, jbd_is_locked_bh_state(bh));
 	assert_spin_locked(&transaction->t_journal->j_list_lock);
 
@@ -1955,8 +1960,10 @@ void __journal_file_buffer(struct journa
 			was_dirty = 1;
 	}
 
-	if (jh->b_transaction)
+	if (jh->b_transaction) {
+		BUFFER_TRACE(bh, "unfile");
 		__journal_unfile_buffer(jh);
+	}
 	jh->b_transaction = transaction;
 
 	switch (jlist) {
@@ -1996,6 +2003,7 @@ void __journal_file_buffer(struct journa
 
 	if (was_dirty)
 		set_buffer_jbddirty(bh);
+	BUFFER_TRACE(bh, "filed");
 }
 
 void journal_file_buffer(struct journal_head *jh,
@@ -2023,12 +2031,14 @@ void __journal_refile_buffer(struct jour
 	int was_dirty;
 	struct buffer_head *bh = jh2bh(jh);
 
+	JBUFFER_TRACE(jh, "entry");
 	J_ASSERT_JH(jh, jbd_is_locked_bh_state(bh));
 	if (jh->b_transaction)
 		assert_spin_locked(&jh->b_transaction->t_journal->j_list_lock);
 
 	/* If the buffer is now unused, just drop it. */
 	if (jh->b_next_transaction == NULL) {
+		BUFFER_TRACE(bh, "unfile");
 		__journal_unfile_buffer(jh);
 		return;
 	}
@@ -2039,6 +2049,7 @@ void __journal_refile_buffer(struct jour
 	 */
 
 	was_dirty = test_clear_buffer_jbddirty(bh);
+	BUFFER_TRACE(bh, "unfile");
 	__journal_unfile_buffer(jh);
 	jh->b_transaction = jh->b_next_transaction;
 	jh->b_next_transaction = NULL;
fs/Kconfig renamed from fs/Kconfig~ext3-debug +4 -0
diff -puN fs/Kconfig~ext3-debug fs/Kconfig
@@ -186,6 +186,10 @@ config JBD_DEBUG
 	  generated.  To turn debugging off again, do
 	  "echo 0 > /proc/sys/fs/jbd-debug".
 
+config BUFFER_DEBUG
+	bool "buffer-layer tracing"
+	depends on JBD
+
 config FS_MBCACHE
 # Meta block cache for Extended Attributes (ext2/ext3)
 	tristate
fs/Makefile renamed from fs/Makefile~ext3-debug +2 -0
diff -puN fs/Makefile~ext3-debug fs/Makefile
@@ -22,6 +22,8 @@ obj-$(CONFIG_BINFMT_AOUT)	+= binfmt_aout
 obj-$(CONFIG_BINFMT_EM86)	+= binfmt_em86.o
 obj-$(CONFIG_BINFMT_MISC)	+= binfmt_misc.o
 
+obj-$(CONFIG_BUFFER_DEBUG)	+= jbd-kernel.o
+
 # binfmt_script is always there
 obj-y				+= binfmt_script.o
 
fs/mpage.c renamed from fs/mpage.c~ext3-debug +2 -0
diff -puN fs/mpage.c~ext3-debug fs/mpage.c
@@ -233,6 +233,7 @@ do_mpage_readpage(struct bio *bio, struc
 	for (page_block = 0; page_block < blocks_per_page;
 				page_block++, block_in_file++) {
 		bh.b_state = 0;
+		bh.b_page = page;
 		if (block_in_file < last_block) {
 			if (get_block(inode, block_in_file, &bh, 0))
 				goto confused;
@@ -465,6 +466,7 @@ mpage_writepage(struct bio *bio, struct 
 	for (page_block = 0; page_block < blocks_per_page; ) {
 
 		map_bh.b_state = 0;
+		map_bh.b_page = page;
 		if (get_block(inode, block_in_file, &map_bh, 1))
 			goto confused;
 		if (buffer_new(&map_bh))
include/linux/buffer_head.h renamed from include/linux/buffer_head.h~ext3-debug +5 -0
diff -puN include/linux/buffer_head.h~ext3-debug include/linux/buffer_head.h
@@ -12,6 +12,7 @@
 #include <linux/linkage.h>
 #include <linux/wait.h>
 #include <asm/atomic.h>
+#include <linux/buffer-trace.h>
 
 enum bh_state_bits {
 	BH_Uptodate,	/* Contains valid data */
@@ -59,6 +60,10 @@ struct buffer_head {
 	bh_end_io_t *b_end_io;		/* I/O completion */
  	void *b_private;		/* reserved for b_end_io */
 	struct list_head b_assoc_buffers; /* associated with another mapping */
+
+#ifdef CONFIG_BUFFER_DEBUG
+	struct buffer_history b_history;
+#endif
 };
 
 /*
include/linux/buffer-trace.h new file +85 -0

85 lines not on this page · raw message

include/linux/jbd.h renamed from include/linux/jbd.h~ext3-debug +5 -7
diff -puN include/linux/jbd.h~ext3-debug include/linux/jbd.h
@@ -54,7 +54,9 @@
  * CONFIG_JBD_DEBUG is on.
  */
 #define JBD_EXPENSIVE_CHECKING
+
 extern int journal_enable_debug;
+extern struct block_device *journal_no_write[2];
 
 #define jbd_debug(n, f, a...)						\
 	do {								\
@@ -265,6 +267,7 @@ void buffer_assertion_failure(struct buf
 #else
 #define J_ASSERT_BH(bh, expr)	J_ASSERT(expr)
 #define J_ASSERT_JH(jh, expr)	J_ASSERT(expr)
+#define buffer_assertion_failure(bh) do { } while (0)
 #endif
 
 #else
@@ -1085,6 +1088,8 @@ static inline int jbd_space_needed(journ
  * Definitions which augment the buffer_head layer
  */
 
+/* JBD additions */
+
 /* journaling buffer types */
 #define BJ_None		0	/* Not journaled */
 #define BJ_SyncData	1	/* Normal data: flush before commit */
@@ -1107,13 +1112,6 @@ extern int jbd_blocks_per_page(struct in
 #define assert_spin_locked(lock)	do {} while(0)
 #endif
 
-#define buffer_trace_init(bh)	do {} while (0)
-#define print_buffer_fields(bh)	do {} while (0)
-#define print_buffer_trace(bh)	do {} while (0)
-#define BUFFER_TRACE(bh, info)	do {} while (0)
-#define BUFFER_TRACE2(bh, bh2, info)	do {} while (0)
-#define JBUFFER_TRACE(jh, info)	do {} while (0)
-
 #endif	/* __KERNEL__ */
 
 #endif	/* CONFIG_JBD || CONFIG_JBD_MODULE || !__KERNEL__ */

_

#4 lore raw
Quoted 8 lines · Andrew Morton
Quoted 3 lines · Chris Mason
Quoted 1 line

What was the "other bug"?

We've got many names for it, but none that could be posted here ;-)

intrigued.

Nah, just a string of curses unfit for the general public, or even lkml.

Quoted 7 lines
Quoted 3 lines

Looks like HP came up with a simplified test case:

http://sourceforge.net/mailarchive/forum.php?thread_id=4536665&forum_id=6379

hm, I never received that. If I'd known I wouldn't have removed the buffer_error() stuff.

I've been trying to blame this one on various other bugs until recently. The test case posted above is the first simple test that is supposed to be able to reproduce. Since ext2 also sees data corruption issues, it's either not ext3 specific or two different bugs.

Quoted 5 lines
Quoted 1 line

I've got machines trying to reproduce now.

You need the buffer-tracing patch. This is against 2.6.7-rc2. It should spit a nice trace when you hit the problem. It'll tell us how that buffer got itself not uptodate.

Thanks. jeffm had worked out something similar that stored the EIP of each bit operation, the uptodate bit seems to have turned off all on its own. Once we can reproduce reliably on local boxes, we'll start layering on the debugging code.

No triggers yet, I might have to grab a bigger machine in the morning.

-chris

#5 lore raw
Quoted 9 lines · Chris Mason
Quoted 4 lines

You need the buffer-tracing patch. This is against 2.6.7-rc2. It should spit a nice trace when you hit the problem. It'll tell us how that buffer got itself not uptodate.

Thanks. jeffm had worked out something similar that stored the EIP of each bit operation, the uptodate bit seems to have turned off all on its own. Once we can reproduce reliably on local boxes, we'll start layering on the debugging code.

buffer-trace code is what you need - it records the bh's internal state in its trace buffer too, replays it all when you hit an assertion failure.

Quoted 1 line

No triggers yet, I might have to grab a bigger machine in the morning.

Is direct-io involved? I just discovered that clean_blockdev_aliases() is invalidating too many blocks. It tends to munch those indirect blocks. (What does bonnie++'s -f option do?)

fs/direct-io.c renamed from fs/direct-io.c~direct-io-invalidation-fix +4 -1
diff -puN fs/direct-io.c~direct-io-invalidation-fix fs/direct-io.c
@@ -690,8 +690,11 @@ out:
 static void clean_blockdev_aliases(struct dio *dio)
 {
 	unsigned i;
+	unsigned nblocks;
 
-	for (i = 0; i < dio->blocks_available; i++) {
+	nblocks = dio->map_bh.b_size >> dio->inode->i_blkbits;
+
+	for (i = 0; i < nblocks; i++) {
 		unmap_underlying_metadata(dio->map_bh.b_bdev,
 					dio->map_bh.b_blocknr + i);
 	}

_

#6 lore raw
Quoted 6 lines · Andrew Morton
Quoted 5 lines · Chris Mason
Quoted 4 lines

You need the buffer-tracing patch. This is against 2.6.7-rc2. It should spit a nice trace when you hit the problem. It'll tell us how that buffer got itself not uptodate.

Quoted 8 lines
Quoted 4 lines

Thanks. jeffm had worked out something similar that stored the EIP of each bit operation, the uptodate bit seems to have turned off all on its own. Once we can reproduce reliably on local boxes, we'll start layering on the debugging code.

buffer-trace code is what you need - it records the bh's internal state in its trace buffer too, replays it all when you hit an assertion failure.

I think the buffers are just victims of slab corruption. With slab poisoning on I managed to get proof that 128 byte slabs are getting stomped on. I'm still searching for a reliable trigger though, tests after that haven't failed at all.

Quoted 6 lines
Quoted 1 line

No triggers yet, I might have to grab a bigger machine in the morning.

Is direct-io involved? I just discovered that clean_blockdev_aliases() is invalidating too many blocks. It tends to munch those indirect blocks. (What does bonnie++'s -f option do?)

skips the per char test. There's no O_DIRECT. Thanks for the patch though ;-)

-chris