Skip to contentSkip to conversation

__block_write_full_page bug

PATCH In review Nick Piggin 2 participants 6 messages Apr 25, 2005 – Apr 27, 2005 VFS

lore
Share & tools
lore

Nick Piggin

author
#0 lore raw

PATCH 1/1 __block_write_full_page bug

Another buffer bug :(

Signature · 1 line
SUSE Labs, Novell Inc.

When running

	fsstress -v -d $DIR/tmp -n 1000 -p 1000 -l 2
on an ext2 filesystem with 1024 byte block size, on SMP i386 with 4096 byte
page size over loopback to an image file on a tmpfs filesystem, I would
very quickly hit
	BUG_ON(!buffer_async_write(bh));

in fs/buffer.c:end_buffer_async_write

It seems that more than one request would be submitted for a given bh at a time. __block_write_full_page looks like the culprit - with the following patch things are very stable.

Signed-off-by Nick Piggin <[email protected]>
fs/buffer.c +23 -15
Index: linux-2.6/fs/buffer.c
===================================================================
@@ -1750,8 +1750,9 @@ static int __block_write_full_page(struc
 	int err;
 	sector_t block;
 	sector_t last_block;
-	struct buffer_head *bh, *head;
-	int nr_underway = 0;
+	struct buffer_head *bh, *head, *arr[MAX_BUF_PER_PAGE];
+	int idx = 0;
+	int nr_underway;
 
 	BUG_ON(!PageLocked(page));
 

@@ -1826,6 +1827,7 @@ static int __block_write_full_page(struc
 		}
 		if (test_clear_buffer_dirty(bh)) {
 			mark_buffer_async_write(bh);
+			arr[idx++] = bh;
 		} else {
 			unlock_buffer(bh);
 		}

@@ -1839,15 +1841,12 @@ static int __block_write_full_page(struc
 	set_page_writeback(page);
 	unlock_page(page);
 
-	do {
-		struct buffer_head *next = bh->b_this_page;
-		if (buffer_async_write(bh)) {
+	for (nr_underway = 0; nr_underway < idx; nr_underway++) {
+		bh = arr[nr_underway];
+		if (buffer_async_write(bh))
 			submit_bh(WRITE, bh);
-			nr_underway++;
-		}
 		put_bh(bh);
-		bh = next;
-	} while (bh != head);
+	}
 
 	err = 0;
 done:

@@ -1890,6 +1889,7 @@ recover:
 		if (buffer_mapped(bh) && buffer_dirty(bh)) {
 			lock_buffer(bh);
 			mark_buffer_async_write(bh);
+			arr[idx++] = bh;
 		} else {
 			/*
 			 * The buffer may have been set dirty during

@@ -1902,16 +1902,14 @@ recover:
 	BUG_ON(PageWriteback(page));
 	set_page_writeback(page);
 	unlock_page(page);
-	do {
-		struct buffer_head *next = bh->b_this_page;
+	for (nr_underway = 0; nr_underway < idx; nr_underway++) {
+		bh = arr[nr_underway];
 		if (buffer_async_write(bh)) {
 			clear_buffer_dirty(bh);
 			submit_bh(WRITE, bh);
-			nr_underway++;
 		}
 		put_bh(bh);
-		bh = next;
-	} while (bh != head);
+	}
 	goto done;
 }
 

@@ -2741,6 +2739,7 @@ sector_t generic_block_bmap(struct addre
 static int end_bio_bh_io_sync(struct bio *bio, unsigned int bytes_done, int err)
 {
 	struct buffer_head *bh = bio->bi_private;
+	bh_end_io_t *end_fn;
 
 	if (bio->bi_size)
 		return 1;

@@ -2750,7 +2749,16 @@ static int end_bio_bh_io_sync(struct bio
 		set_bit(BH_Eopnotsupp, &bh->b_state);
 	}
 
-	bh->b_end_io(bh, test_bit(BIO_UPTODATE, &bio->bi_flags));
+	end_fn = bh->b_end_io;
+
+	/* 
+	 * These two lines are debugging only - make sure b_end_io
+	 * isn't run twice for the same io request.
+	 */
+	BUG_ON(!end_fn);
+	bh->b_end_io = NULL;
+	
+	end_fn(bh, test_bit(BIO_UPTODATE, &bio->bi_flags));
 	bio_put(bio);
 	return 0;
 }
#1 lore raw

Review

Quoted 12 lines · Nick Piggin

When running

(quoted diff) +0 -0
 	fsstress -v -d $DIR/tmp -n 1000 -p 1000 -l 2
 on an ext2 filesystem with 1024 byte block size, on SMP i386 with 4096 byte
 page size over loopback to an image file on a tmpfs filesystem, I would
 very quickly hit
 	BUG_ON(!buffer_async_write(bh));
 in fs/buffer.c:end_buffer_async_write
 
 It seems that more than one request would be submitted for a given bh
 at a time. __block_write_full_page looks like the culprit - with the
 following patch things are very stable.

What's the bug? I don't see it.

Was an ENOSPC involved?

Those tests for buffer_async_write(bh) are redundant now, aren't they?

author
#2 lore raw
Quoted 10 lines · Andrew Morton
Quoted 9 lines · Nick Piggin

When running

(quoted diff) +0 -0
 	fsstress -v -d $DIR/tmp -n 1000 -p 1000 -l 2
 on an ext2 filesystem with 1024 byte block size, on SMP i386 with 4096 byte
 page size over loopback to an image file on a tmpfs filesystem, I would
 very quickly hit
 	BUG_ON(!buffer_async_write(bh));
 in fs/buffer.c:end_buffer_async_write
Quoted 6 lines
Quoted 3 lines

It seems that more than one request would be submitted for a given bh at a time. __block_write_full_page looks like the culprit - with the following patch things are very stable.

What's the bug? I don't see it.

Ah, the bug is that end_buffer_async_write first does

	BUG_ON(!buffer_async_write(bh));
then a bit later does
	clear_buffer_async_write(bh);

That's where it was blowing up for me, because end_buffer_async_write was being run twice for that buffer.

Or did you mean *how* is it being run twice? I didn't exactly find the stack traces involved, but I imagine that simply testing buffer_async_write catches other requests in flight - ie. we've lost track of exactly which ones we own.

Quoted 2 lines

Was an ENOSPC involved?

No.

Quoted 1 line

Those tests for buffer_async_write(bh) are redundant now, aren't they?

They are, yes. Sorry I noticed that earlier too - they should probably be BUG_ON(!buffer_async_write(bh)) instead. Would make things clearer.

Nick

#3 lore raw
Quoted 22 lines · Nick Piggin
Quoted 10 lines · Andrew Morton
Quoted 9 lines · Nick Piggin

When running

(quoted diff) +0 -0
 	fsstress -v -d $DIR/tmp -n 1000 -p 1000 -l 2
 on an ext2 filesystem with 1024 byte block size, on SMP i386 with 4096 byte
 page size over loopback to an image file on a tmpfs filesystem, I would
 very quickly hit
 	BUG_ON(!buffer_async_write(bh));
 in fs/buffer.c:end_buffer_async_write
Quoted 6 lines
Quoted 3 lines

It seems that more than one request would be submitted for a given bh at a time. __block_write_full_page looks like the culprit - with the following patch things are very stable.

What's the bug? I don't see it.

Ah, the bug is that end_buffer_async_write first does

(quoted diff) +0 -0
 	BUG_ON(!buffer_async_write(bh));

Review

Quoted 10 lines
(quoted diff) +0 -0
 then a bit later does
 	clear_buffer_async_write(bh);
 
 That's where it was blowing up for me, because end_buffer_async_write
 was being run twice for that buffer.
 
 Or did you mean *how* is it being run twice? I didn't exactly find
 the stack traces involved, but I imagine that simply testing
 buffer_async_write catches other requests in flight - ie. we've
 lost track of exactly which ones we own.

How can such a thing come about? Both PageLocked() and PageWriteback() are supposed to stop new writeback being started against the page.

<looks>

Were you using nobh? I guess not. What's to stop the new mpage_writepage() from trying to write a page which is already under PageWriteback()?

I don't think we understand this bug yet.

author
#4 lore raw
Quoted 35 lines · Andrew Morton
Quoted 23 lines · Nick Piggin
Quoted 10 lines · Andrew Morton
Quoted 9 lines · Nick Piggin

When running

(quoted diff) +0 -0
 	fsstress -v -d $DIR/tmp -n 1000 -p 1000 -l 2
 on an ext2 filesystem with 1024 byte block size, on SMP i386 with 4096 byte
 page size over loopback to an image file on a tmpfs filesystem, I would
 very quickly hit
 	BUG_ON(!buffer_async_write(bh));
 in fs/buffer.c:end_buffer_async_write
Quoted 6 lines
Quoted 3 lines

It seems that more than one request would be submitted for a given bh at a time. __block_write_full_page looks like the culprit - with the following patch things are very stable.

What's the bug? I don't see it.

Ah, the bug is that end_buffer_async_write first does

	BUG_ON(!buffer_async_write(bh));
then a bit later does
	clear_buffer_async_write(bh);
Quoted 8 lines

That's where it was blowing up for me, because end_buffer_async_write was being run twice for that buffer.

Or did you mean *how* is it being run twice? I didn't exactly find the stack traces involved, but I imagine that simply testing buffer_async_write catches other requests in flight - ie. we've lost track of exactly which ones we own.

Quoted 3 lines

How can such a thing come about? Both PageLocked() and PageWriteback() are supposed to stop new writeback being started against the page.

You have a point.

Quoted 3 lines

<looks>

Were you using nobh? I guess not. What's to stop the new

No

Quoted 3 lines

mpage_writepage() from trying to write a page which is already under PageWriteback()?

Don't know... I didn't think mapping->writepage should be called for a PageWriteback page?

Quoted 2 lines

I don't think we understand this bug yet.

It appears not. I'll look into it further.

Signature · 1 line
SUSE Labs, Novell Inc.
author
#5 lore raw
In reply to Andrew Morton
Quoted 2 lines · Andrew Morton

Those tests for buffer_async_write(bh) are redundant now, aren't they?

Like this?

Signature · 1 line
SUSE Labs, Novell Inc.

Previous patch made buffer_async_write tests superfluous. As suggested by Andrew Morton.

Signed-off-by Nick Piggin <[email protected]>
fs/buffer.c +5 -6
Index: linux-2.6/fs/buffer.c
===================================================================
@@ -1843,8 +1843,8 @@ static int __block_write_full_page(struc
 
 	for (nr_underway = 0; nr_underway < idx; nr_underway++) {
 		bh = arr[nr_underway];
-		if (buffer_async_write(bh))
-			submit_bh(WRITE, bh);
+		BUG_ON(!buffer_async_write(bh));
+		submit_bh(WRITE, bh);
 		put_bh(bh);
 	}
 

@@ -1904,10 +1904,9 @@ recover:
 	unlock_page(page);
 	for (nr_underway = 0; nr_underway < idx; nr_underway++) {
 		bh = arr[nr_underway];
-		if (buffer_async_write(bh)) {
-			clear_buffer_dirty(bh);
-			submit_bh(WRITE, bh);
-		}
+		BUG_ON(!buffer_async_write(bh));
+		clear_buffer_dirty(bh);
+		submit_bh(WRITE, bh);
 		put_bh(bh);
 	}
 	goto done;