Skip to contentSkip to conversation

Mount ext3 with barrier=1 doesn't send real barrier bio?

PATCH Milan Broz 5 participants 8 messages Aug 15, 2008 – Aug 22, 2008 VFS

lore
Share & tools
lore

Milan Broz

author
#0 lore raw

Hi,

I run some barrier tests over device-mapper (which currently doesn't support barrier bio at all) and even if I set barrier=1 in ext3 mount, there is never any bio with barrier flag... (in 2.6.27-rc)

How is the barrier=1 flag supposed to work in ext3 (JBD) now?

See:
If you specify barrier=1, JFS_BARRIER flag is set in ext3_init_journal_params

	journal->j_flags |= JFS_BARRIER;

Now, journal_write_commit_record is called and this happens:

	if (journal->j_flags & JFS_BARRIER) {
		set_buffer_ordered(bh);
		barrier_done = 1;
	}
	ret = sync_dirty_buffer(bh);

	if (barrier_done)
		clear_buffer_ordered(bh);

	if (ret == -EOPNOTSUPP && barrier_done) {
	...
Quoted 1 line

From this code I expect that EOPNOTSUPP is returned if barrier is not

supported (yes, that exactly does device-mapper now without barrier patches).

But it *never* happens because:

sync_dirty_buffer always calls
submit_bh(WRITE_SYNC, bh)

and in submit_bh is this test:

	if (buffer_ordered(bh) && (rw == WRITE))
		rw = WRITE_BARRIER;

but there is rw == WRITE_SYNC, not WRITE !

So the barrier flag for bio is never set and normal sync write is performed.

Why it isn't done like in attached patch? Is it intentional or it is bug?

I think it was caused by change in this commit:

commit 18ce3751ccd488c78d3827e9f6bf54e6322676fb

Author: Jens Axboe <[email protected]>
Date:   Tue Jul 1 09:07:34 2008 +0200

Properly notify block layer of sync writes

Milan
--

Set BIO_RW_BARRIER flag even for submit_bh sync write request.

---
 fs/buffer.c |    8 ++++----
 1 files changed, 4 insertions(+), 4 deletions(-)
fs/buffer.c +4 -4
@@ -2926,16 +2926,16 @@ int submit_bh(int rw, struct buffer_head * bh)
 	BUG_ON(!buffer_mapped(bh));
 	BUG_ON(!bh->b_end_io);
 
-	if (buffer_ordered(bh) && (rw == WRITE))
-		rw = WRITE_BARRIER;
-
 	/*
 	 * Only clear out a write error when rewriting, should this
 	 * include WRITE_SYNC as well?
 	 */
-	if (test_set_buffer_req(bh) && (rw == WRITE || rw == WRITE_BARRIER))
+	if (test_set_buffer_req(bh) && rw == WRITE)
 		clear_buffer_write_io_error(bh);
 
+	if (buffer_ordered(bh) && ((rw & RW_MASK) == WRITE))
+		rw |= (1 << BIO_RW_BARRIER);
+
 	/*
 	 * from here on down, it's all bio -- do the initial mapping,
 	 * submit_bio -> generic_make_request may further map this bio around
#1 lore raw
Quoted 7 lines · Milan Broz

Hi,

I run some barrier tests over device-mapper (which currently doesn't support barrier bio at all) and even if I set barrier=1 in ext3 mount, there is never any bio with barrier flag... (in 2.6.27-rc)

How is the barrier=1 flag supposed to work in ext3 (JBD) now?

Milan, you're right. Ric saw this same strange behavior when doing some benchmarking with and without barriers; Chris noticed the change in submit_bh; I was about to write up a similar patch to what you've sent already. Jens, does Milan's fix look good to you?

Incidentally, I ran Ric's test on ext3 on a sata drive:

# fs_mark -d /mnt/test -n 1600 -t 1 -s 20480

cfq:

                        files/s
           2.6.25   2.6.26.2  2.6.26.2+patch

barrier=0 169 127 126
barrier=1 33 126 33

noop:

                        files/s
           2.6.25   2.6.26.2  2.6.26.2+patch

barrier=0 191 184 185
barrier=1 33 180 33

deadline:

                        files/s
           2.6.25   2.6.26.2  2.6.26.2+patch

barrier=0 181 182 185
barrier=1 33 185 33

anticipatory:

                        files/s
           2.6.25   2.6.26.2  2.6.26.2+patch

barrier=0 187 133 132
barrier=1 34 134 33

-Eric

Quoted 89 lines

See:
If you specify barrier=1, JFS_BARRIER flag is set in ext3_init_journal_params

	journal->j_flags |= JFS_BARRIER;

Now, journal_write_commit_record is called and this happens:

	if (journal->j_flags & JFS_BARRIER) {
		set_buffer_ordered(bh);
		barrier_done = 1;
	}
	ret = sync_dirty_buffer(bh);

	if (barrier_done)
		clear_buffer_ordered(bh);

	if (ret == -EOPNOTSUPP && barrier_done) {
	...

From this code I expect that EOPNOTSUPP is returned if barrier is not supported (yes, that exactly does device-mapper now without barrier patches).

But it *never* happens because:

sync_dirty_buffer always calls
submit_bh(WRITE_SYNC, bh)

and in submit_bh is this test:

	if (buffer_ordered(bh) && (rw == WRITE))
		rw = WRITE_BARRIER;

but there is rw == WRITE_SYNC, not WRITE !

So the barrier flag for bio is never set and normal sync write is performed.

Why it isn't done like in attached patch? Is it intentional or it is bug?

I think it was caused by change in this commit:

commit 18ce3751ccd488c78d3827e9f6bf54e6322676fb

Author: Jens Axboe <[email protected]>
Date:   Tue Jul 1 09:07:34 2008 +0200

Properly notify block layer of sync writes

Milan
--

Set BIO_RW_BARRIER flag even for submit_bh sync write request.

---
 fs/buffer.c |    8 ++++----
 1 files changed, 4 insertions(+), 4 deletions(-)
fs/buffer.c +4 -5
@@ -2926,16 +2926,16 @@ int submit_bh(int rw, struct buffer_head * bh)
 	BUG_ON(!buffer_mapped(bh));
 	BUG_ON(!bh->b_end_io);
 
-	if (buffer_ordered(bh) && (rw == WRITE))
-		rw = WRITE_BARRIER;
-
 	/*
 	 * Only clear out a write error when rewriting, should this
 	 * include WRITE_SYNC as well?
 	 */
-	if (test_set_buffer_req(bh) && (rw == WRITE || rw == WRITE_BARRIER))
+	if (test_set_buffer_req(bh) && rw == WRITE)
 		clear_buffer_write_io_error(bh);
 
+	if (buffer_ordered(bh) && ((rw & RW_MASK) == WRITE))
+		rw |= (1 << BIO_RW_BARRIER);
+
 	/*
 	 * from here on down, it's all bio -- do the initial mapping,
 	 * submit_bio -> generic_make_request may further map this bio around
 
 
 
 
--

To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

#2 lore raw
Quoted 7 lines · Eric Sandeen
Quoted 6 lines · Milan Broz

Hi,

I run some barrier tests over device-mapper (which currently doesn't support barrier bio at all) and even if I set barrier=1 in ext3 mount, there is never any bio with barrier flag... (in 2.6.27-rc)

Quoted 6 lines
Quoted 1 line

How is the barrier=1 flag supposed to work in ext3 (JBD) now?

Milan, you're right. Ric saw this same strange behavior when doing some benchmarking with and without barriers; Chris noticed the change in submit_bh; I was about to write up a similar patch to what you've sent already. Jens, does Milan's fix look good to you?

Yep looks good, thanks a lot Milan! I'll send in the patch, unless I'm badly mistaken we need it for 2.6.26-stable as well.

Signature · 1 line
Jens Axboe
#3 lore raw
Quoted 12 lines · Jens Axboe
Quoted 10 lines · Eric Sandeen
Quoted 8 lines · Milan Broz

Hi,

I run some barrier tests over device-mapper (which currently doesn't support barrier bio at all) and even if I set barrier=1 in ext3 mount, there is never any bio with barrier flag... (in 2.6.27-rc)

How is the barrier=1 flag supposed to work in ext3 (JBD) now?

Quoted 10 lines
Quoted 5 lines

Milan, you're right. Ric saw this same strange behavior when doing some benchmarking with and without barriers; Chris noticed the change in submit_bh; I was about to write up a similar patch to what you've sent already. Jens, does Milan's fix look good to you?

Yep looks good, thanks a lot Milan! I'll send in the patch, unless I'm badly mistaken we need it for 2.6.26-stable as well.

I think we definitely need it there as well, thanks!

Ric

#4 lore raw
In reply to Eric Sandeen
Quoted 6 lines · Eric Sandeen
Quoted 6 lines
fs/buffer.c +0 -0
@@ -2926,16 +2926,16 @@ int submit_bh(int rw, struct buffer_head * bh)
 	BUG_ON(!buffer_mapped(bh));
 	BUG_ON(!bh->b_end_io);
 
Quoted 10 lines

fs/buffer.c int submit_bh(int rw, struct buffer_head * bh) · line 2934

Quoted 10 lines
fs/buffer.c +1 -4
-	if (buffer_ordered(bh) && (rw == WRITE))
-		rw = WRITE_BARRIER;
-
 	/*
 	 * Only clear out a write error when rewriting, should this
 	 * include WRITE_SYNC as well?
 	 */
-	if (test_set_buffer_req(bh) && (rw == WRITE || rw == WRITE_BARRIER))
+	if (test_set_buffer_req(bh) && rw == WRITE)
 		clear_buffer_write_io_error(bh);

This should be ((rw & RW_MASK) == WRITE) too? Anyway, this seems change behavior of submit_bh(WRITE_BARRIER) (maybe reiserfs only), it wouldn't be your intent...

Quoted 6 lines
Quoted 6 lines
fs/buffer.c +3 -0
+	if (buffer_ordered(bh) && ((rw & RW_MASK) == WRITE))
+		rw |= (1 << BIO_RW_BARRIER);
+
 	/*
 	 * from here on down, it's all bio -- do the initial mapping,
 	 * submit_bio -> generic_make_request may further map this bio around
Signature · 1 line
OGAWA Hirofumi <[email protected]>
#5 lore raw
Quoted 8 lines · OGAWA Hirofumi
Quoted 6 lines · Eric Sandeen
Quoted 6 lines
fs/buffer.c +0 -0
@@ -2926,16 +2926,16 @@ int submit_bh(int rw, struct buffer_head * bh)
 	BUG_ON(!buffer_mapped(bh));
 	BUG_ON(!bh->b_end_io);
 
Quoted 14 lines
Quoted 10 lines

fs/buffer.c int submit_bh(int rw, struct buffer_head * bh) · line 2934

Quoted 10 lines
fs/buffer.c +1 -4
-	if (buffer_ordered(bh) && (rw == WRITE))
-		rw = WRITE_BARRIER;
-
 	/*
 	 * Only clear out a write error when rewriting, should this
 	 * include WRITE_SYNC as well?
 	 */
-	if (test_set_buffer_req(bh) && (rw == WRITE || rw == WRITE_BARRIER))
+	if (test_set_buffer_req(bh) && rw == WRITE)
 		clear_buffer_write_io_error(bh);

This should be ((rw & RW_MASK) == WRITE) too? Anyway, this seems change behavior of submit_bh(WRITE_BARRIER) (maybe reiserfs only), it wouldn't be your intent...

Yes, I believe the simpler and more correct fix is:

fs/buffer.c +3 -4
index 38653e3..16b2263 100644
@@ -2926,14 +2926,13 @@ int submit_bh(int rw, struct buffer_head * bh)
 	BUG_ON(!buffer_mapped(bh));
 	BUG_ON(!bh->b_end_io);
 
-	if (buffer_ordered(bh) && (rw == WRITE))
+	if (buffer_ordered(bh) && (rw & WRITE))
 		rw = WRITE_BARRIER;
 
 	/*
-	 * Only clear out a write error when rewriting, should this
-	 * include WRITE_SYNC as well?
+	 * Only clear out a write error when rewriting
 	 */
-	if (test_set_buffer_req(bh) && (rw == WRITE || rw == WRITE_BARRIER))
+	if (test_set_buffer_req(bh) && (rw & WRITE))
 		clear_buffer_write_io_error(bh);
 
 	/*
Signature · 1 line
Jens Axboe
#6 lore raw
Quoted 11 lines · Jens Axboe
Quoted 3 lines

This should be ((rw & RW_MASK) == WRITE) too? Anyway, this seems change behavior of submit_bh(WRITE_BARRIER) (maybe reiserfs only), it wouldn't be your intent...

Yes, I believe the simpler and more correct fix is:

fs/buffer.c +0 -0
index 38653e3..16b2263 100644

fs/buffer.c int submit_bh(int rw, struct buffer_head * bh) · line 2930

Quoted 6 lines
fs/buffer.c +1 -1
@@ -2926,14 +2926,13 @@ int submit_bh(int rw, struct buffer_head * bh)
 	BUG_ON(!buffer_mapped(bh));
 	BUG_ON(!bh->b_end_io);
 
-	if (buffer_ordered(bh) && (rw == WRITE))
+	if (buffer_ordered(bh) && (rw & WRITE))
 		rw = WRITE_BARRIER;

I see. But, umm..., this means WRITE_SYNC with barrier was deprecated? Or typo?

Signature · 1 line
OGAWA Hirofumi <[email protected]>
#7 lore raw
Quoted 16 lines · OGAWA Hirofumi
Quoted 14 lines · Jens Axboe
Quoted 3 lines

This should be ((rw & RW_MASK) == WRITE) too? Anyway, this seems change behavior of submit_bh(WRITE_BARRIER) (maybe reiserfs only), it wouldn't be your intent...

Yes, I believe the simpler and more correct fix is:

fs/buffer.c +0 -0
index 38653e3..16b2263 100644
@@ -2926,14 +2926,13 @@ int submit_bh(int rw, struct buffer_head * bh)
 	BUG_ON(!buffer_mapped(bh));
 	BUG_ON(!bh->b_end_io);
 
Quoted 6 lines

fs/buffer.c int submit_bh(int rw, struct buffer_head * bh) · line 2930

Quoted 3 lines
fs/buffer.c +1 -1
-	if (buffer_ordered(bh) && (rw == WRITE))
+	if (buffer_ordered(bh) && (rw & WRITE))
 		rw = WRITE_BARRIER;

I see. But, umm..., this means WRITE_SYNC with barrier was deprecated? Or typo?

It was supposed to read rw |= WRITE_BARRIER :-)

Signature · 1 line
Jens Axboe