Skip to contentSkip to conversation

Avoid bio_endio recursion

PATCH In review Mikulas Patocka 4 participants 20 messages Jun 24, 2008 – Jul 4, 2008

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

PATCH 1/2 Avoid bio_endio recursion

Hi

bio_endio calls bi_end_io callback. In case of stacked devices (raid, dm), bio_end_io may call bio_endio again, up to an unspecified length.

The crash because of stack overflow was really observed on sparc64. And this recursion was one of the contributing factors (using 9 stack frames --- that is 1728 bytes).

This patch removes the recursion.

Mikulas

--

Avoid recursion on bio_endio. bio_endio calls bio->bi_end_io which may in turn call bio_endio again. When this recursion happens, put the new bio to the queue and process it later, from the top-level bio_endio.

fs/bio.c +32 -0
Index: linux-2.6.26-rc5-devel/fs/bio.c
===================================================================
@@ -1168,6 +1168,27 @@
  **/
 void bio_endio(struct bio *bio, int error)
 {
+	static DEFINE_PER_CPU(struct bio **, bio_end_queue) = { NULL };
+	struct bio ***bio_end_queue_ptr;
+	struct bio *bio_queue;
+
+	unsigned long flags;
+
+	local_irq_save(flags);
+	bio_end_queue_ptr = &__get_cpu_var(bio_end_queue);
+
+	if (*bio_end_queue_ptr) {
+		**bio_end_queue_ptr = bio;
+		*bio_end_queue_ptr = &bio->bi_next;
+		bio->bi_next = NULL;
+		goto ret;
+	}
+
+	bio_queue = NULL;
+queue_empty_next_bio:
+	*bio_end_queue_ptr = &bio_queue;
+next_bio:
+
 	if (error)
 		clear_bit(BIO_UPTODATE, &bio->bi_flags);
 	else if (!test_bit(BIO_UPTODATE, &bio->bi_flags))
@@ -1175,6 +1196,17 @@
 
 	if (bio->bi_end_io)
 		bio->bi_end_io(bio, error);
+
+	if (bio_queue) {
+		bio = bio_queue;
+		bio_queue = bio->bi_next;
+		if (!bio_queue) goto queue_empty_next_bio;
+		goto next_bio;
+	}
+	*bio_end_queue_ptr = NULL;
+
+ret:
+	local_irq_restore(flags);
 }
 
 void bio_pair_release(struct bio_pair *bp)
Reviewed-by on v1 1/2
#1 lore raw
Quoted 10 lines · On Tuesday June 24, [email protected] wrote:

Hi

bio_endio calls bi_end_io callback. In case of stacked devices (raid, dm), bio_end_io may call bio_endio again, up to an unspecified length.

The crash because of stack overflow was really observed on sparc64. And this recursion was one of the contributing factors (using 9 stack frames --- that is 1728 bytes).

This patch removes the recursion.

This is very cool, thanks! A close mirror of the recursion avoidance in generic_make_request.

You use a per-cpu queue were generic_make_request uses a per-task queue. This is fitting as bi_end_io doesn't have a process context, but is supposed to be fast and able to run with interrupts disabled, so tying to a cpu is no problem.

Quoted 10 lines
fs/bio.c +6 -0
+
+	bio_queue = NULL;
+queue_empty_next_bio:
+	*bio_end_queue_ptr = &bio_queue;
+next_bio:
+
  	if (error)
  		clear_bit(BIO_UPTODATE, &bio->bi_flags);
  	else if (!test_bit(BIO_UPTODATE, &bio->bi_flags))

fs/bio.c line 1204

Quoted 9 lines
fs/bio.c +6 -0
@@ -1175,6 +1196,17 @@
 
  	if (bio->bi_end_io)
  		bio->bi_end_io(bio, error);
+
+	if (bio_queue) {
+		bio = bio_queue;
+		bio_queue = bio->bi_next;
+		if (!bio_queue) goto queue_empty_next_bio;
+		goto next_bio;

checkpatch.pl doesn't like that:
ERROR: trailing statements should be on next line

and I don't either. I would not bother with the mini-optimisation at all.
Discard the queue_empty_next_bio label and replace the "if () goto" with

       if (!bio_queue)
		*bio_end_queue_ptr = &bio_queue;

and leave gcc to optimise the assignment if it wants to.

Thanks,
NeilBrown

#2 lore raw
Quoted 12 lines · Neil Brown
Quoted 10 lines · On Tuesday June 24, [email protected] wrote:

Hi

bio_endio calls bi_end_io callback. In case of stacked devices (raid, dm), bio_end_io may call bio_endio again, up to an unspecified length.

The crash because of stack overflow was really observed on sparc64. And this recursion was one of the contributing factors (using 9 stack frames --- that is 1728 bytes).

This patch removes the recursion.

Quoted 7 lines

This is very cool, thanks! A close mirror of the recursion avoidance in generic_make_request.

You use a per-cpu queue were generic_make_request uses a per-task queue. This is fitting as bi_end_io doesn't have a process context, but is supposed to be fast and able to run with interrupts disabled, so tying to a cpu is no problem.

Yes. I think "current" variable can't be used in irq context, it would blow with irq-stacks (or access some weird unknown memory).

I had another version of the patch that doesn't disable interrupts and only disables preempt and uses local_t atomic cpu-local variables. It is somehow more tricky, because interrupt can be triggered any time while processing the queue and it can add anything to the queue. Then I realized that bio_endio runs most time with disabled interrupts anyway, so it'd be better to just disable interrupts and don't do that local_cmpxchg trickery.

Mikulas

Quoted 37 lines
Quoted 10 lines
fs/bio.c +6 -0
+
+	bio_queue = NULL;
+queue_empty_next_bio:
+	*bio_end_queue_ptr = &bio_queue;
+next_bio:
+
  	if (error)
  		clear_bit(BIO_UPTODATE, &bio->bi_flags);
  	else if (!test_bit(BIO_UPTODATE, &bio->bi_flags))

fs/bio.c line 1204

Quoted 9 lines
fs/bio.c +6 -0
@@ -1175,6 +1196,17 @@
 
  	if (bio->bi_end_io)
  		bio->bi_end_io(bio, error);
+
+	if (bio_queue) {
+		bio = bio_queue;
+		bio_queue = bio->bi_next;
+		if (!bio_queue) goto queue_empty_next_bio;
+		goto next_bio;

checkpatch.pl doesn't like that:
ERROR: trailing statements should be on next line

and I don't either. I would not bother with the mini-optimisation at all.
Discard the queue_empty_next_bio label and replace the "if () goto" with

      if (!bio_queue)
		*bio_end_queue_ptr = &bio_queue;

and leave gcc to optimise the assignment if it wants to.

Thanks,
NeilBrown

#3 lore raw
In reply to Mikulas Patocka
Quoted 8 lines · Mikulas Patocka

Hi

bio_endio calls bi_end_io callback. In case of stacked devices (raid, dm), bio_end_io may call bio_endio again, up to an unspecified length.

The crash because of stack overflow was really observed on sparc64. And this recursion was one of the contributing factors (using 9 stack frames --- that is 1728 bytes).

Looks good, I like the concept. Can you please make it a little less goto driven, though? The next_bio and goto next_bio could just be a while().

Signature · 1 line
Jens Axboe
#4 lore raw
Quoted 6 lines · Jens Axboe
Quoted 5 lines · Mikulas Patocka

Hi

bio_endio calls bi_end_io callback. In case of stacked devices (raid, dm), bio_end_io may call bio_endio again, up to an unspecified length.

Quoted 11 lines
Quoted 3 lines

The crash because of stack overflow was really observed on sparc64. And this recursion was one of the contributing factors (using 9 stack frames --- that is 1728 bytes).

Looks good, I like the concept. Can you please make it a little less goto driven, though? The next_bio and goto next_bio could just be a while().

Signature · 1 line
Jens Axboe

Hi.

This is the patch, slightly de-goto-ized. (it still contains one, I think that while (1) { ... break ... } is no better readable than goto).

I found another problem in my previous patch, I forgot about the "error" variable (it would cause misbehavior for example if disk fails, submits an error and raid driver turns this failure into success). We need to save the error variable somewhere in the bio, there is no other place where it could be placed. I temporarily saved it to bi_idx, because it's unused at this place.

Mikulas

--

Avoid recursion on bio_endio. bio_endio calls bio->bi_end_io which may in turn call bio_endio again. When this recursion happens, put the new bio to the queue and process it later, from the top-level bio_endio.

fs/bio.c +34 -3
Index: linux-2.6.26-rc7-devel/fs/bio.c
===================================================================
@@ -1168,13 +1168,44 @@
  **/
 void bio_endio(struct bio *bio, int error)
 {
+	static DEFINE_PER_CPU(struct bio **, bio_end_queue) = { NULL };
+	struct bio ***bio_end_queue_ptr;
+	struct bio *bio_queue;
+
+	unsigned long flags;
+
+	bio->bi_idx = error;
 	if (error)
 		clear_bit(BIO_UPTODATE, &bio->bi_flags);
 	else if (!test_bit(BIO_UPTODATE, &bio->bi_flags))
-		error = -EIO;
+		bio->bi_idx = -EIO;
+
+	local_irq_save(flags);
+	bio_end_queue_ptr = &__get_cpu_var(bio_end_queue);
+
+	if (*bio_end_queue_ptr) {
+		**bio_end_queue_ptr = bio;
+		*bio_end_queue_ptr = &bio->bi_next;
+		bio->bi_next = NULL;
+	} else {
+		bio_queue = NULL;
+		*bio_end_queue_ptr = &bio_queue;
+
+next_bio:
+		if (bio->bi_end_io)
+			bio->bi_end_io(bio, (short)bio->bi_idx);
+
+		if (bio_queue) {
+			bio = bio_queue;
+			bio_queue = bio->bi_next;
+			if (!bio_queue)
+				*bio_end_queue_ptr = &bio_queue;
+			goto next_bio;
+		}
+		*bio_end_queue_ptr = NULL;
+	}
 
-	if (bio->bi_end_io)
-		bio->bi_end_io(bio, error);
+	local_irq_restore(flags);
 }
 
 void bio_pair_release(struct bio_pair *bp)
#5 lore raw
Quoted 18 lines · Mikulas Patocka
Quoted 14 lines · Jens Axboe
Quoted 8 lines · Mikulas Patocka

Hi

bio_endio calls bi_end_io callback. In case of stacked devices (raid, dm), bio_end_io may call bio_endio again, up to an unspecified length.

The crash because of stack overflow was really observed on sparc64. And this recursion was one of the contributing factors (using 9 stack frames --- that is 1728 bytes).

Looks good, I like the concept. Can you please make it a little less goto driven, though? The next_bio and goto next_bio could just be a while().

Quoted 8 lines
Quoted 3 lines
Signature · 1 line
Jens Axboe

Hi.

This is the patch, slightly de-goto-ized. (it still contains one, I think that while (1) { ... break ... } is no better readable than goto).

Sure, that looks better.

Quoted 6 lines

I found another problem in my previous patch, I forgot about the "error" variable (it would cause misbehavior for example if disk fails, submits an error and raid driver turns this failure into success). We need to save the error variable somewhere in the bio, there is no other place where it could be placed. I temporarily saved it to bi_idx, because it's unused at this place.

I don't think bi_idx is a fantastic idea, I could easily imagine the bi_end_io function wanting to do a segment loop on the bio. Use bi_phys_segments instead (or bi_hw_segemnts, no difference), they should only be used when queuing and building IO, not for completion purposes. And put a big fat comment there explaining the overload. Plus they are just a cache, so if you use either of those and at the same time clear BIO_SEG_VALID in bi_flags, then it's guarenteed to be safe.

Also please put the per-cpu definition outside of bio_endio(). And I don't think you need to disable interrupts, a plain preempt_disable() / preempt_enable() should be enough.

Signature · 1 line
Jens Axboe
#6 lore raw
Quoted 37 lines · Jens Axboe
Quoted 18 lines · Mikulas Patocka
Quoted 14 lines · Jens Axboe
Quoted 8 lines · Mikulas Patocka

Hi

bio_endio calls bi_end_io callback. In case of stacked devices (raid, dm), bio_end_io may call bio_endio again, up to an unspecified length.

The crash because of stack overflow was really observed on sparc64. And this recursion was one of the contributing factors (using 9 stack frames --- that is 1728 bytes).

Looks good, I like the concept. Can you please make it a little less goto driven, though? The next_bio and goto next_bio could just be a while().

Quoted 8 lines
Quoted 3 lines

--
Jens Axboe

Hi.

This is the patch, slightly de-goto-ized. (it still contains one, I think that while (1) { ... break ... } is no better readable than goto).

Sure, that looks better.

Quoted 6 lines

I found another problem in my previous patch, I forgot about the "error" variable (it would cause misbehavior for example if disk fails, submits an error and raid driver turns this failure into success). We need to save the error variable somewhere in the bio, there is no other place where it could be placed. I temporarily saved it to bi_idx, because it's unused at this place.

Quoted 7 lines

I don't think bi_idx is a fantastic idea, I could easily imagine the bi_end_io function wanting to do a segment loop on the bio. Use bi_phys_segments instead (or bi_hw_segemnts, no difference), they should only be used when queuing and building IO, not for completion purposes. And put a big fat comment there explaining the overload. Plus they are just a cache, so if you use either of those and at the same time clear BIO_SEG_VALID in bi_flags, then it's guarenteed to be safe.

Here is new patch that uses bi_phys_segments.

Quoted 3 lines

Also please put the per-cpu definition outside of bio_endio(). And I don't think you need to disable interrupts, a plain preempt_disable() / preempt_enable() should be enough.

Disabling interrupts is needed. If you disable only preempt, interrupt can come at any point and add anything to the queue. You'd have to use local_t and local_cmpxchg tricks then and the code would be much more ugly. I found that the code is called with interrupts disabled most time, so it doesn't matter if I disable them again.

Mikulas

Quoted 2 lines
Signature · 1 line
Jens Axboe

Avoid recursion on bio_endio. bio_endio calls bio->bi_end_io which may in turn call bio_endio again. When this recursion happens, put the new bio to the queue and process it later, from the top-level bio_endio.

---
 fs/bio.c            |   38 +++++++++++++++++++++++++++++++++++---
 include/linux/bio.h |    3 +++
 2 files changed, 38 insertions(+), 3 deletions(-)
fs/bio.c +35 -3
Index: linux-2.6.26-rc7-devel/fs/bio.c
===================================================================
@@ -1166,15 +1166,47 @@ void bio_check_pages_dirty(struct bio *b
  *   bio unless they own it and thus know that it has an end_io
  *   function.
  **/
+static DEFINE_PER_CPU(struct bio **, bio_end_queue) = { NULL };
+
 void bio_endio(struct bio *bio, int error)
 {
+	struct bio ***bio_end_queue_ptr;
+	struct bio *bio_queue;
+
+	unsigned long flags;
+
+	bio->bi_phys_segments = error;
 	if (error)
 		clear_bit(BIO_UPTODATE, &bio->bi_flags);
 	else if (!test_bit(BIO_UPTODATE, &bio->bi_flags))
-		error = -EIO;
+		bio->bi_phys_segments = -EIO;
+
+	local_irq_save(flags);
+	bio_end_queue_ptr = &__get_cpu_var(bio_end_queue);
+
+	if (*bio_end_queue_ptr) {
+		**bio_end_queue_ptr = bio;
+		*bio_end_queue_ptr = &bio->bi_next;
+		bio->bi_next = NULL;
+	} else {
+		bio_queue = NULL;
+		*bio_end_queue_ptr = &bio_queue;
+
+next_bio:
+		if (bio->bi_end_io)
+			bio->bi_end_io(bio, (short)bio->bi_phys_segments);
+
+		if (bio_queue) {
+			bio = bio_queue;
+			bio_queue = bio->bi_next;
+			if (!bio_queue)
+				*bio_end_queue_ptr = &bio_queue;
+			goto next_bio;
+		}
+		*bio_end_queue_ptr = NULL;
+	}
 
-	if (bio->bi_end_io)
-		bio->bi_end_io(bio, error);
+	local_irq_restore(flags);
 }
 
 void bio_pair_release(struct bio_pair *bp)
include/linux/bio.h +3 -0
Index: linux-2.6.26-rc7-devel/include/linux/bio.h
===================================================================
@@ -86,6 +86,9 @@ struct bio {
 
 	/* Number of segments in this BIO after
 	 * physical address coalescing is performed.
+	 *
+	 * When ending a bio request in bio_endio, this field is temporarily
+	 * (ab)used to keep the error code.
 	 */
 	unsigned short		bi_phys_segments;
 
#7 lore raw
Quoted 54 lines · Mikulas Patocka
Quoted 38 lines · Jens Axboe
Quoted 19 lines · Mikulas Patocka
Quoted 15 lines · Jens Axboe
Quoted 9 lines · Mikulas Patocka

Hi

bio_endio calls bi_end_io callback. In case of stacked devices (raid, dm),
bio_end_io may call bio_endio again, up to an unspecified length.

The crash because of stack overflow was really observed on sparc64. And this recursion was one of the contributing factors (using 9 stack frames --- that is 1728 bytes).

Looks good, I like the concept. Can you please make it a little less goto driven, though? The next_bio and goto next_bio could just be a while().

Quoted 8 lines
Quoted 3 lines

--
Jens Axboe

Hi.

This is the patch, slightly de-goto-ized. (it still contains one, I think that while (1) { ... break ... } is no better readable than goto).

Sure, that looks better.

Quoted 6 lines

I found another problem in my previous patch, I forgot about the "error" variable (it would cause misbehavior for example if disk fails, submits an error and raid driver turns this failure into success). We need to save the error variable somewhere in the bio, there is no other place where it could be placed. I temporarily saved it to bi_idx, because it's unused at this place.

Quoted 7 lines

I don't think bi_idx is a fantastic idea, I could easily imagine the bi_end_io function wanting to do a segment loop on the bio. Use bi_phys_segments instead (or bi_hw_segemnts, no difference), they should only be used when queuing and building IO, not for completion purposes. And put a big fat comment there explaining the overload. Plus they are just a cache, so if you use either of those and at the same time clear BIO_SEG_VALID in bi_flags, then it's guarenteed to be safe.

Here is new patch that uses bi_phys_segments.

Quoted 3 lines

Also please put the per-cpu definition outside of bio_endio(). And I don't think you need to disable interrupts, a plain preempt_disable() / preempt_enable() should be enough.

Quoted 5 lines

Disabling interrupts is needed. If you disable only preempt, interrupt can come at any point and add anything to the queue. You'd have to use local_t and local_cmpxchg tricks then and the code would be much more ugly. I found that the code is called with interrupts disabled most time, so it doesn't matter if I disable them again.

Right, that wont work of course. Completions are typically done through a softirq, so it is not currently done with hard interrupts disabled. So it's a bit of a shame to impose this extra restriction, bio unrolling is necessarily a really short operation. We could reenable around the bi_end_io() call, but then we'd need to save and restore for each bio.

Quoted 28 lines

Mikulas

Quoted 2 lines
Signature · 1 line
Jens Axboe

Avoid recursion on bio_endio. bio_endio calls bio->bi_end_io which may in turn
call bio_endio again. When this recursion happens, put the new bio to the queue
and process it later, from the top-level bio_endio.

---
 fs/bio.c            |   38 +++++++++++++++++++++++++++++++++++---
 include/linux/bio.h |    3 +++
 2 files changed, 38 insertions(+), 3 deletions(-)

Index: linux-2.6.26-rc7-devel/fs/bio.c
===================================================================
--- linux-2.6.26-rc7-devel.orig/fs/bio.c 2008-06-23
13:49:45.000000000 +0200
+++ linux-2.6.26-rc7-devel/fs/bio.c 2008-06-25 18:09:48.000000000 +0200

fs/bio.c +0 -0
@@ -1166,15 +1166,47 @@ void bio_check_pages_dirty(struct bio *b
  *   bio unless they own it and thus know that it has an end_io
  *   function.
  **/

fs/bio.c void bio_check_pages_dirty(struct bio *b · line 1178

Quoted 10 lines
fs/bio.c +8 -0
+static DEFINE_PER_CPU(struct bio **, bio_end_queue) = { NULL };
+
 void bio_endio(struct bio *bio, int error)
 {
+	struct bio ***bio_end_queue_ptr;
+	struct bio *bio_queue;
+
+	unsigned long flags;
+
+	bio->bi_phys_segments = error;

Please make that

        bio->bi_flags &= ~(1 << BIO_SEG_VALID);
        bio->bi_phys_segments = error;

as I mentioned, since then we have nothing to worry about. Otherwise bio_phys_segments() could return a completely wrong count.

Quoted 53 lines
fs/bio.c +27 -3
 	if (error)
 		clear_bit(BIO_UPTODATE, &bio->bi_flags);
 	else if (!test_bit(BIO_UPTODATE, &bio->bi_flags))
-		error = -EIO;
+		bio->bi_phys_segments = -EIO;
+
+	local_irq_save(flags);
+	bio_end_queue_ptr = &__get_cpu_var(bio_end_queue);
+
+	if (*bio_end_queue_ptr) {
+		**bio_end_queue_ptr = bio;
+		*bio_end_queue_ptr = &bio->bi_next;
+		bio->bi_next = NULL;
+	} else {
+		bio_queue = NULL;
+		*bio_end_queue_ptr = &bio_queue;
+
+next_bio:
+		if (bio->bi_end_io)
+			bio->bi_end_io(bio, (short)bio->bi_phys_segments);
+
+		if (bio_queue) {
+			bio = bio_queue;
+			bio_queue = bio->bi_next;
+			if (!bio_queue)
+				*bio_end_queue_ptr = &bio_queue;
+			goto next_bio;
+		}
+		*bio_end_queue_ptr = NULL;
+	}
 
-	if (bio->bi_end_io)
-		bio->bi_end_io(bio, error);
+	local_irq_restore(flags);
 }
 
 void bio_pair_release(struct bio_pair *bp)

Index: linux-2.6.26-rc7-devel/include/linux/bio.h
===================================================================
--- linux-2.6.26-rc7-devel.orig/include/linux/bio.h 2008-06-25
18:06:59.000000000 +0200
+++ linux-2.6.26-rc7-devel/include/linux/bio.h 2008-06-25
18:08:08.000000000 +0200

include/linux/bio.h +3 -0
@@ -86,6 +86,9 @@ struct bio {
 
 	/* Number of segments in this BIO after
 	 * physical address coalescing is performed.
+	 *
+	 * When ending a bio request in bio_endio, this field is temporarily
+	 * (ab)used to keep the error code.
 	 */
 	unsigned short		bi_phys_segments;
 
Signature · 1 line
Jens Axboe
#8 lore raw
Quoted 2 lines

Right, that wont work of course. Completions are typically done through a softirq, so it is not currently done with hard interrupts disabled.

I thought, from hardirq - that's what IDE is doing. And they are called with interrupts disabled (maybe unless you specify unmaskirq, which is not default). What block driver does completions with softirq? ... and why?

Quoted 1 line

So it's a bit of a shame to impose this extra restriction, bio unrolling is necessarily a really short operation. We could reenable around the bi_end_io() call, but then we'd need to save and restore for each bio.

I found another weird thing --- why does local_irq_save() execute that microcoded "cli" instruction even if interrupts are already disabled? That could be definitely optimized.

And does local_irq_restore() need to execute even more costy "popf" when it makes a transition from disabled to disabled state? What's local_irq_restore semantics --- is it allowed to use local_irq_restore for transition from interrupt-enabled state into interrupt-disabled state?

Quoted 4 lines

Please make that

       bio->bi_flags &= ~(1 << BIO_SEG_VALID);
       bio->bi_phys_segments = error;

OK, here is the updated patch:

Avoid recursion on bio_endio. bio_endio calls bio->bi_end_io which may in turn call bio_endio again. When this recursion happens, put the new bio to the queue and process it later, from the top-level bio_endio.

---
 fs/bio.c            |   39 ++++++++++++++++++++++++++++++++++++---
 include/linux/bio.h |    3 +++
 2 files changed, 39 insertions(+), 3 deletions(-)
fs/bio.c +36 -3
Index: linux-2.6.26-rc8-devel/fs/bio.c
===================================================================
@@ -1166,15 +1166,48 @@ void bio_check_pages_dirty(struct bio *b
  *   bio unless they own it and thus know that it has an end_io
  *   function.
  **/
+static DEFINE_PER_CPU(struct bio **, bio_end_queue) = { NULL };
+
 void bio_endio(struct bio *bio, int error)
 {
+	struct bio ***bio_end_queue_ptr;
+	struct bio *bio_queue;
+
+	unsigned long flags;
+
+	bio->bi_flags &= ~(1 << BIO_SEG_VALID);
+	bio->bi_phys_segments = error;
 	if (error)
 		clear_bit(BIO_UPTODATE, &bio->bi_flags);
 	else if (!test_bit(BIO_UPTODATE, &bio->bi_flags))
-		error = -EIO;
+		bio->bi_phys_segments = -EIO;
+
+	local_irq_save(flags);
+	bio_end_queue_ptr = &__get_cpu_var(bio_end_queue);
+
+	if (*bio_end_queue_ptr) {
+		**bio_end_queue_ptr = bio;
+		*bio_end_queue_ptr = &bio->bi_next;
+		bio->bi_next = NULL;
+	} else {
+		bio_queue = NULL;
+		*bio_end_queue_ptr = &bio_queue;
+
+next_bio:
+		if (bio->bi_end_io)
+			bio->bi_end_io(bio, (short)bio->bi_phys_segments);
+
+		if (bio_queue) {
+			bio = bio_queue;
+			bio_queue = bio->bi_next;
+			if (!bio_queue)
+				*bio_end_queue_ptr = &bio_queue;
+			goto next_bio;
+		}
+		*bio_end_queue_ptr = NULL;
+	}
 
-	if (bio->bi_end_io)
-		bio->bi_end_io(bio, error);
+	local_irq_restore(flags);
 }
 
 void bio_pair_release(struct bio_pair *bp)
include/linux/bio.h +3 -0
Index: linux-2.6.26-rc8-devel/include/linux/bio.h
===================================================================
@@ -86,6 +86,9 @@ struct bio {
 
 	/* Number of segments in this BIO after
 	 * physical address coalescing is performed.
+	 *
+	 * When ending a bio request in bio_endio, this field is temporarily
+	 * (ab)used to keep the error code.
 	 */
 	unsigned short		bi_phys_segments;
 
#9 lore raw
Quoted 4 lines · On Wed, 2 Jul 2008 00:09:22 -0400 (EDT) Mikulas Patocka <[email protected]>…
Quoted 2 lines

Right, that wont work of course. Completions are typically done through a softirq, so it is not currently done with hard interrupts disabled.

I thought, from hardirq - that's what IDE is doing. And they are called

Even IDE will sometimes complete from a timer on an error.

Quoted 4 lines

And does local_irq_restore() need to execute even more costy "popf" when it makes a transition from disabled to disabled state? What's local_irq_restore semantics --- is it allowed to use local_irq_restore for transition from interrupt-enabled state into interrupt-disabled state?

If you are worried about performance the network layer has _irq variants of various functions that are faster and can only be called from the right context (eg kfree_skb_irq), so you could do two versions of that code.

Alan

#10 lore raw
Quoted 15 lines · Alan Cox
Quoted 4 lines · On Wed, 2 Jul 2008 00:09:22 -0400 (EDT) Mikulas Patocka <[email protected]>…
Quoted 2 lines

Right, that wont work of course. Completions are typically done through a softirq, so it is not currently done with hard interrupts disabled.

I thought, from hardirq - that's what IDE is doing. And they are called

Even IDE will sometimes complete from a timer on an error.

Quoted 4 lines

And does local_irq_restore() need to execute even more costy "popf" when it makes a transition from disabled to disabled state? What's local_irq_restore semantics --- is it allowed to use local_irq_restore for transition from interrupt-enabled state into interrupt-disabled state?

Quoted 6 lines

If you are worried about performance the network layer has _irq variants of various functions that are faster and can only be called from the right context (eg kfree_skb_irq), so you could do two versions of that code.

Alan

I was thinking about a general optimization --- something like this. On Core2 it doesn't make much sense --- cli is fast there (just 11 ticks). On Pentium 4 it would make sense, because cli is much more costy there.

Mikulas

---
 include/asm-x86/irqflags.h |    3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
include/asm-x86/irqflags.h +2 -1
Index: linux-2.6.26-rc8/include/asm-x86/irqflags.h
===================================================================
@@ -101,7 +101,8 @@ static inline unsigned long __raw_local_
 {
 	unsigned long flags = __raw_local_save_flags();
 
-	raw_local_irq_disable();
+	if (flags & X86_EFLAGS_IF)
+		raw_local_irq_disable();
 
 	return flags;
 }
#11 lore raw
In reply to Mikulas Patocka
Quoted 6 lines · Mikulas Patocka
Quoted 2 lines

Right, that wont work of course. Completions are typically done through a softirq, so it is not currently done with hard interrupts disabled.

I thought, from hardirq - that's what IDE is doing. And they are called with interrupts disabled (maybe unless you specify unmaskirq, which is not default). What block driver does completions with softirq? ... and why?

The key word is 'typically', the old IDE driver really isn't used very much. The SCSI layer and eg cciss uses the block layer softirq completions, so that is what 99% of the uses will be.

Quoted 4 lines
Quoted 3 lines

So it's a bit of a shame to impose this extra restriction, bio unrolling is necessarily a really short operation. We could reenable around the bi_end_io() call, but then we'd need to save and restore for each bio.

Quoted 8 lines

I found another weird thing --- why does local_irq_save() execute that microcoded "cli" instruction even if interrupts are already disabled? That could be definitely optimized.

And does local_irq_restore() need to execute even more costy "popf" when it makes a transition from disabled to disabled state? What's local_irq_restore semantics --- is it allowed to use local_irq_restore for transition from interrupt-enabled state into interrupt-disabled state?

That's a question for the arch people. Probably nested _save and _restore aren't optimized.

Quoted 6 lines
Quoted 5 lines

Please make that

      bio->bi_flags &= ~(1 << BIO_SEG_VALID);
      bio->bi_phys_segments = error;
Quoted 7 lines

OK, here is the updated patch:

Avoid recursion on bio_endio. bio_endio calls bio->bi_end_io which may in turn
call bio_endio again. When this recursion happens, put the new bio to the queue
and process it later, from the top-level bio_endio.

Thanks that looks good, I'll throw this into the testing mix and see what happens.

Quoted 8 lines

void bio_pair_release(struct bio_pair *bp)
Index: linux-2.6.26-rc8-devel/include/linux/bio.h
===================================================================
--- linux-2.6.26-rc8-devel.orig/include/linux/bio.h 2008-07-02
04:20:43.000000000 +0200
+++ linux-2.6.26-rc8-devel/include/linux/bio.h 2008-07-02
04:21:16.000000000 +0200

include/linux/bio.h struct bio { · line 93

Quoted 8 lines
include/linux/bio.h +3 -0
@@ -86,6 +86,9 @@ struct bio {
 
 	/* Number of segments in this BIO after
 	 * physical address coalescing is performed.
+	 *
+	 * When ending a bio request in bio_endio, this field is temporarily
+	 * (ab)used to keep the error code.
 	 */
 	unsigned short		bi_phys_segments;

I'll drop this comment, we don't really need it. Users must use bio_phys_segments() to retrieve the segment count anyway, in which case the count is ALWAYS valid.

Signature · 1 line
Jens Axboe
#12 lore raw
Quoted 11 lines · Jens Axboe
Quoted 6 lines · Mikulas Patocka
Quoted 2 lines

Right, that wont work of course. Completions are typically done through a softirq, so it is not currently done with hard interrupts disabled.

I thought, from hardirq - that's what IDE is doing. And they are called with interrupts disabled (maybe unless you specify unmaskirq, which is not default). What block driver does completions with softirq? ... and why?

The key word is 'typically', the old IDE driver really isn't used very much. The SCSI layer and eg cciss uses the block layer softirq completions, so that is what 99% of the uses will be.

I use the old IDE driver, I don't see a reason why driver should create SCSI requests and lower layer translate them to ATA commands.

Just to look, this is the first version of the patch without disabling interrupts. If you are concerned about disabling interrupts, you can go this way.

Mikulas

fs/bio.c +32 -0
Index: linux-2.6.26-rc5-devel/fs/bio.c
===================================================================
@@ -1168,6 +1168,25 @@
  **/
 void bio_endio(struct bio *bio, int error)
 {
+	static DEFINE_PER_CPU(local_t, bio_end_queue) = LOCAL_INIT(0);
+	local_t *bio_end_queue_ptr;
+	unsigned long l;
+
+	bio->bi_next = NULL;
+
+	bio_end_queue_ptr = &get_cpu_var(bio_end_queue);
+	if (local_read(bio_end_queue_ptr)) {
+insert_race:
+		bio->bi_next = (void *)local_read(bio_end_queue_ptr);
+		if (unlikely(local_cmpxchg(bio_end_queue_ptr, (unsigned long)bio->bi_next, (unsigned long)bio) != (unsigned long)bio->bi_next))
+			goto insert_race;
+		put_cpu_var(bio_end_queue);
+		return;
+	}
+
+	local_set(bio_end_queue_ptr, 1);
+
+process_bio:
 	if (error)
 		clear_bit(BIO_UPTODATE, &bio->bi_flags);
 	else if (!test_bit(BIO_UPTODATE, &bio->bi_flags))
@@ -1175,6 +1194,19 @@
 
 	if (bio->bi_end_io)
 		bio->bi_end_io(bio, error);
+
+remove_race:
+	l = local_read(bio_end_queue_ptr);
+	if (l != 1) {
+		bio = (void *)l;
+		if (unlikely(local_cmpxchg(bio_end_queue_ptr, (unsigned long)bio, (unsigned long)bio->bi_next) != (unsigned long)bio))
+			goto remove_race;
+		goto process_bio;
+	}
+	if (unlikely(local_cmpxchg(bio_end_queue_ptr, 1, 0) != 1))
+		goto remove_race;
+
+	put_cpu_var(bio_end_queue);
 }
 
 void bio_pair_release(struct bio_pair *bp)
#13 lore raw
Quoted 2 lines

I use the old IDE driver, I don't see a reason why driver should create SCSI requests and lower layer translate them to ATA commands.

Because modern drives are basically SCSI devices. Once you turn on NCQ and stuff you need all the midlayer queueing magic ...

Either way the bio thing wants fixing.

#14 lore raw
Quoted 5 lines · Alan Cox
Quoted 2 lines

I use the old IDE driver, I don't see a reason why driver should create SCSI requests and lower layer translate them to ATA commands.

Because modern drives are basically SCSI devices. Once you turn on NCQ and stuff you need all the midlayer queueing magic ...

There is no piece of SCSI protocol in SATA disks. The command and response FIS of SATA standard contain legacy IDE registers, there's nothing about SCSI. Even the NCQ commands are done via IDE registers, not SCSI command block. It seems like someone wanted to save few weeks of coding by reusing the SCSI disk queuing ... and created a lot of other problems (example: if you have disk error with IDE driver, it dumps IDE registers into log ... if you have disk error with SATA driver, it dumps sense key ... but there's no sense key in SATA standard ... the driver just had to make up one because it pretends to be SCSI).

Mikulas

#15 lore raw
Quoted 2 lines

SCSI. Even the NCQ commands are done via IDE registers, not SCSI command block. It seems like someone wanted to save few weeks of coding by reusing

Thats really a trivial question of encoding, in all but actual byte encoding it is SCSI. Once you look at SAS and ATAPI it's all the more clear.

Quoted 3 lines

the SCSI disk queuing ... and created a lot of other problems (example: if you have disk error with IDE driver, it dumps IDE registers into log ... if you have disk error with SATA driver, it dumps sense key ... but

Actually it dumps both - and that behaviour was chosen to match existing drivers in a large part (eg 3ware).

#16 lore raw
Quoted 6 lines · Alan Cox
Quoted 2 lines

SCSI. Even the NCQ commands are done via IDE registers, not SCSI command block. It seems like someone wanted to save few weeks of coding by reusing

Thats really a trivial question of encoding, in all but actual byte encoding it is SCSI. Once you look at SAS and ATAPI it's all the more clear.

For SAS and ATAPI you're right - that's SCSI.

But not for SATA disks. The only thing SCSI and SATA disk have common is that they may process concurrently more requests. There's nothing else.

Mikulas

#17 lore raw
Quoted 3 lines

But not for SATA disks. The only thing SCSI and SATA disk have common is that they may process concurrently more requests. There's nothing else.

The only material difference between SATA and SCSI is the encoding of the command blocks and responses.

Alan

#18 lore raw
Quoted 8 lines · Alan Cox
Quoted 3 lines

But not for SATA disks. The only thing SCSI and SATA disk have common is that they may process concurrently more requests. There's nothing else.

The only material difference between SATA and SCSI is the encoding of the command blocks and responses.

Alan

That statement is true for any two block device protocol. Then, why doesn't Linux use SCSI for all block devices? --- add scsi command block to struct bio and we can pass them directly to controller driver :)

Mikulas

#19 lore raw
Quoted 1 line

That statement is true for any two block device protocol. Then, why

Not it isn't. The mentality of quite a few devices is very different to SCSI in the way the express queueing and caching. You cannot for example map I2O Block onto SCSI at all.

Quoted 2 lines

doesn't Linux use SCSI for all block devices? --- add scsi command block to struct bio and we can pass them directly to controller driver

That was discussed, along with continuing to split scsi and block aspects of queueing, tagging and error recovery apart. Windows btw does generally follow that 'everything is SCSI' approach