Skip to contentSkip to conversation

blk-mq: sync the update nr_hw_queues with blk_mq_queue_tag_busy_iter

PATCH In review Giuliano Procida 2 participants 10 messages Jun 8, 2020 – Jun 18, 2020

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

PATCH 1/1 blk-mq: sync the update nr_hw_queues with blk_mq_queue_tag_busy_iter

From: Jianchao Wang <[email protected]>

commit f5bbbbe4d63577026f908a809f22f5fd5a90ea1f upstream.

For blk-mq, part_in_flight/rw will invoke blk_mq_in_flight/rw to account the inflight requests. It will access the queue_hw_ctx and nr_hw_queues w/o any protection. When updating nr_hw_queues and blk_mq_in_flight/rw occur concurrently, panic comes up.

Before update nr_hw_queues, the q will be frozen. So we could use q_usage_counter to avoid the race. percpu_ref_is_zero is used here so that we will not miss any in-flight request. The access to nr_hw_queues and queue_hw_ctx in blk_mq_queue_tag_busy_iter are under rcu critical section, __blk_mq_update_nr_hw_queues could use synchronize_rcu to ensure the zeroed q_usage_counter to be globally visible.

Backporting Notes

This is a re-backport, landing synchronize_rcu in the right place.

---
 block/blk-mq.c | 8 ++++----
 1 file changed, 4 insertions(+), 4 deletions(-)
block/blk-mq.c +4 -4
index 9d53f476c517..cf56bdad2e06 100644
@@ -2738,6 +2738,10 @@ static void __blk_mq_update_nr_hw_queues(struct blk_mq_tag_set *set,
 
 	list_for_each_entry(q, &set->tag_list, tag_set_list)
 		blk_mq_freeze_queue(q);
+	/*
+	 * Sync with blk_mq_queue_tag_busy_iter.
+	 */
+	synchronize_rcu();
 
 	set->nr_hw_queues = nr_hw_queues;
 	blk_mq_update_queue_map(set);
@@ -2748,10 +2752,6 @@ static void __blk_mq_update_nr_hw_queues(struct blk_mq_tag_set *set,
 
 	list_for_each_entry(q, &set->tag_list, tag_set_list)
 		blk_mq_unfreeze_queue(q);
-	/*
-	 * Sync with blk_mq_queue_tag_busy_iter.
-	 */
-	synchronize_rcu();
 }
 
 void blk_mq_update_nr_hw_queues(struct blk_mq_tag_set *set, int nr_hw_queues)
Signature · 1 line
2.27.0.278.ge193c7cf3a9-goog
#1 lore raw

-bouncing mail.

This one is for 4.14. Apologies.

Giuliano.

Quoted 59 lines · Giuliano Procida

From: Jianchao Wang <[email protected]>

commit f5bbbbe4d63577026f908a809f22f5fd5a90ea1f upstream.

For blk-mq, part_in_flight/rw will invoke blk_mq_in_flight/rw to account the inflight requests. It will access the queue_hw_ctx and nr_hw_queues w/o any protection. When updating nr_hw_queues and blk_mq_in_flight/rw occur concurrently, panic comes up.

Before update nr_hw_queues, the q will be frozen. So we could use q_usage_counter to avoid the race. percpu_ref_is_zero is used here so that we will not miss any in-flight request. The access to nr_hw_queues and queue_hw_ctx in blk_mq_queue_tag_busy_iter are under rcu critical section, __blk_mq_update_nr_hw_queues could use synchronize_rcu to ensure the zeroed q_usage_counter to be globally visible.

Backporting Notes

This is a re-backport, landing synchronize_rcu in the right place.

---
 block/blk-mq.c | 8 ++++----
 1 file changed, 4 insertions(+), 4 deletions(-)
block/blk-mq.c +4 -5
index 9d53f476c517..cf56bdad2e06 100644
@@ -2738,6 +2738,10 @@ static void __blk_mq_update_nr_hw_queues(struct blk_mq_tag_set *set,
 
        list_for_each_entry(q, &set->tag_list, tag_set_list)
                blk_mq_freeze_queue(q);
+       /*
+        * Sync with blk_mq_queue_tag_busy_iter.
+        */
+       synchronize_rcu();
 
        set->nr_hw_queues = nr_hw_queues;
        blk_mq_update_queue_map(set);
@@ -2748,10 +2752,6 @@ static void __blk_mq_update_nr_hw_queues(struct blk_mq_tag_set *set,
 
        list_for_each_entry(q, &set->tag_list, tag_set_list)
                blk_mq_unfreeze_queue(q);
-       /*
-        * Sync with blk_mq_queue_tag_busy_iter.
-        */
-       synchronize_rcu();
 }
 
 void blk_mq_update_nr_hw_queues(struct blk_mq_tag_set *set, int nr_hw_queues)
--

2.27.0.278.ge193c7cf3a9-goog

#2 lore raw

Hi Greg.

Is this patch (and the similar one for 4.9) queued?

Giuliano.

Quoted 67 lines · Giuliano Procida

-bouncing mail.

This one is for 4.14. Apologies.

Giuliano.

Quoted 59 lines · Giuliano Procida

From: Jianchao Wang <[email protected]>

commit f5bbbbe4d63577026f908a809f22f5fd5a90ea1f upstream.

For blk-mq, part_in_flight/rw will invoke blk_mq_in_flight/rw to account the inflight requests. It will access the queue_hw_ctx and nr_hw_queues w/o any protection. When updating nr_hw_queues and blk_mq_in_flight/rw occur concurrently, panic comes up.

Before update nr_hw_queues, the q will be frozen. So we could use q_usage_counter to avoid the race. percpu_ref_is_zero is used here so that we will not miss any in-flight request. The access to nr_hw_queues and queue_hw_ctx in blk_mq_queue_tag_busy_iter are under rcu critical section, __blk_mq_update_nr_hw_queues could use synchronize_rcu to ensure the zeroed q_usage_counter to be globally visible.

Backporting Notes

This is a re-backport, landing synchronize_rcu in the right place.

---
 block/blk-mq.c | 8 ++++----
 1 file changed, 4 insertions(+), 4 deletions(-)
block/blk-mq.c +4 -5
index 9d53f476c517..cf56bdad2e06 100644
@@ -2738,6 +2738,10 @@ static void __blk_mq_update_nr_hw_queues(struct blk_mq_tag_set *set,
 
        list_for_each_entry(q, &set->tag_list, tag_set_list)
                blk_mq_freeze_queue(q);
+       /*
+        * Sync with blk_mq_queue_tag_busy_iter.
+        */
+       synchronize_rcu();
 
        set->nr_hw_queues = nr_hw_queues;
        blk_mq_update_queue_map(set);
@@ -2748,10 +2752,6 @@ static void __blk_mq_update_nr_hw_queues(struct blk_mq_tag_set *set,
 
        list_for_each_entry(q, &set->tag_list, tag_set_list)
                blk_mq_unfreeze_queue(q);
-       /*
-        * Sync with blk_mq_queue_tag_busy_iter.
-        */
-       synchronize_rcu();
 }
 
 void blk_mq_update_nr_hw_queues(struct blk_mq_tag_set *set, int nr_hw_queues)
--

2.27.0.278.ge193c7cf3a9-goog

#3 lore raw

A: http://en.wikipedia.org/wiki/Top_post
Q: Were do I find info about this thing called top-posting?
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?
A: Top-posting.
Q: What is the most annoying thing in e-mail?

A: No.
Q: Should I include quotations after my reply?

http://daringfireball.net/2007/07/on_top

:)

Quoted 3 lines · Giuliano Procida

Hi Greg.

Is this patch (and the similar one for 4.9) queued?

f5bbbbe4d635 ("blk-mq: sync the update nr_hw_queues with blk_mq_queue_tag_busy_iter") is in the following stable releases:

	4.4.224 4.9.219 4.14.176 4.19

Do you not see it there?

Or is there some other upstream commit you are referring to here?

thanks,

greg k-h

#4 lore raw

Hi.

Quoted 9 lines · Greg KH

A: http://en.wikipedia.org/wiki/Top_post
Q: Were do I find info about this thing called top-posting?
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?
A: Top-posting.
Q: What is the most annoying thing in e-mail?

Quoted 6 lines

A: No.
Q: Should I include quotations after my reply?

http://daringfireball.net/2007/07/on_top

:)

I'm well aware of the above.
Alas, I haven't used mutt properly in about 15 years and I'm still doing everything with Gmail.
Given that I was referring to the entire email thread, I punted on finding a place to insert a comment.
BTW, there's a typo in the Q&A above. s/Were/Where/
:)

Quoted 10 lines · Giuliano Procida
Quoted 3 lines

Hi Greg.

Is this patch (and the similar one for 4.9) queued?

f5bbbbe4d635 ("blk-mq: sync the update nr_hw_queues with blk_mq_queue_tag_busy_iter") is in the following stable releases:

        4.4.224 4.9.219 4.14.176 4.19

Do you not see it there?

We are referring to different "it"s.

Yours: f5bbbbe4d635 is the upstream patch that went into v4.19-rc1 and which you back-ported to at least some of these kernels. This is clearly there.

Mine: the commit sent earlier in this email thread - it's a re-back-port, as I think the original back-port for 4.14 (and similarly for 4.9) is incorrect. This has clearly not reached public git, hence my question about whether the change was queued.

These are the ids of messages containing my commits:

4.14: [email protected]
4.9: [email protected]

Quoted 1 line

Or is there some other upstream commit you are referring to here?

No.

Quoted 3 lines

thanks,

greg k-h

I hope this clears things up.

Regards,
Giuliano.

#5 lore raw
Quoted 15 lines · Giuliano Procida

Hi.

Quoted 12 lines · Greg KH

A: http://en.wikipedia.org/wiki/Top_post
Q: Were do I find info about this thing called top-posting?
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?
A: Top-posting.
Q: What is the most annoying thing in e-mail?

A: No.
Q: Should I include quotations after my reply?

Quoted 7 lines
Quoted 3 lines

http://daringfireball.net/2007/07/on_top

:)

I'm well aware of the above.
Alas, I haven't used mutt properly in about 15 years and I'm still doing everything with Gmail.

gmail can handle proper quoting, if you are stuck with that :)

Quoted 3 lines

Given that I was referring to the entire email thread, I punted on finding a place to insert a comment.
BTW, there's a typo in the Q&A above. s/Were/Where/

Ah, nice catch, first one to notice that in years!

Quoted 9 lines
Quoted 9 lines · Giuliano Procida
Quoted 3 lines

Hi Greg.

Is this patch (and the similar one for 4.9) queued?

f5bbbbe4d635 ("blk-mq: sync the update nr_hw_queues with blk_mq_queue_tag_busy_iter") is in the following stable releases:

        4.4.224 4.9.219 4.14.176 4.19
Quoted 7 lines
Quoted 1 line

Do you not see it there?

We are referring to different "it"s.

Yours: f5bbbbe4d635 is the upstream patch that went into v4.19-rc1 and which you back-ported to at least some of these kernels. This is clearly there.

Great.

Quoted 4 lines

Mine: the commit sent earlier in this email thread - it's a re-back-port, as I think the original back-port for 4.14 (and similarly for 4.9) is incorrect. This has clearly not reached public git, hence my question about whether the change was queued.

I don't know what the git commit id you are looking for here, sorry. I don't have the whole thread anywhere.

Quoted 4 lines

These are the ids of messages containing my commits:

4.14: [email protected]
4.9: [email protected]

Pointers to this on lore.kernel.org perhaps?

Remember, some of us get thousands of patches a week to handle, remembering old email thread, or even keeping them around, is impossible...

thanks,

greg k-h

#6 lore raw

Hi.

Quoted 60 lines · Greg KH
Quoted 15 lines · Giuliano Procida

Hi.

Quoted 12 lines · Greg KH

A: http://en.wikipedia.org/wiki/Top_post
Q: Were do I find info about this thing called top-posting?
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?
A: Top-posting.
Q: What is the most annoying thing in e-mail?

A: No.
Q: Should I include quotations after my reply?

Quoted 7 lines
Quoted 3 lines

http://daringfireball.net/2007/07/on_top

:)

I'm well aware of the above.
Alas, I haven't used mutt properly in about 15 years and I'm still doing everything with Gmail.

gmail can handle proper quoting, if you are stuck with that :)

Quoted 3 lines

Given that I was referring to the entire email thread, I punted on finding a place to insert a comment.
BTW, there's a typo in the Q&A above. s/Were/Where/

Ah, nice catch, first one to notice that in years!

Quoted 9 lines
Quoted 9 lines · Giuliano Procida
Quoted 3 lines

Hi Greg.

Is this patch (and the similar one for 4.9) queued?

f5bbbbe4d635 ("blk-mq: sync the update nr_hw_queues with blk_mq_queue_tag_busy_iter") is in the following stable releases:

        4.4.224 4.9.219 4.14.176 4.19
Quoted 7 lines
Quoted 1 line

Do you not see it there?

We are referring to different "it"s.

Yours: f5bbbbe4d635 is the upstream patch that went into v4.19-rc1 and which you back-ported to at least some of these kernels. This is clearly there.

Great.

Quoted 4 lines

Mine: the commit sent earlier in this email thread - it's a re-back-port, as I think the original back-port for 4.14 (and similarly for 4.9) is incorrect. This has clearly not reached public git, hence my question about whether the change was queued.

I don't know what the git commit id you are looking for here, sorry. I don't have the whole thread anywhere.

Quoted 7 lines
Quoted 4 lines

These are the ids of messages containing my commits:

4.14: [email protected]
4.9: [email protected]

Pointers to this on lore.kernel.org perhaps?

I wasn't aware stable was publicly archived. Here you go:

4.14: https://lore.kernel.org/stable/[email protected]/
4.9: https://lore.kernel.org/stable/[email protected]/

Regards,
Giuliano.

Quoted 7 lines

thanks

Remember, some of us get thousands of patches a week to handle, remembering old email thread, or even keeping them around, is impossible...

greg k-h

#7 lore raw
Quoted 68 lines · Giuliano Procida

Hi.

Quoted 65 lines · Greg KH
Quoted 15 lines · Giuliano Procida

Hi.

Quoted 12 lines · Greg KH

A: http://en.wikipedia.org/wiki/Top_post
Q: Were do I find info about this thing called top-posting?
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?
A: Top-posting.
Q: What is the most annoying thing in e-mail?

A: No.
Q: Should I include quotations after my reply?

Quoted 7 lines
Quoted 3 lines

http://daringfireball.net/2007/07/on_top

:)

I'm well aware of the above.
Alas, I haven't used mutt properly in about 15 years and I'm still doing everything with Gmail.

gmail can handle proper quoting, if you are stuck with that :)

Quoted 3 lines

Given that I was referring to the entire email thread, I punted on finding a place to insert a comment.
BTW, there's a typo in the Q&A above. s/Were/Where/

Ah, nice catch, first one to notice that in years!

Quoted 9 lines
Quoted 9 lines · Giuliano Procida
Quoted 3 lines

Hi Greg.

Is this patch (and the similar one for 4.9) queued?

f5bbbbe4d635 ("blk-mq: sync the update nr_hw_queues with blk_mq_queue_tag_busy_iter") is in the following stable releases:

        4.4.224 4.9.219 4.14.176 4.19
Quoted 7 lines
Quoted 1 line

Do you not see it there?

We are referring to different "it"s.

Yours: f5bbbbe4d635 is the upstream patch that went into v4.19-rc1 and which you back-ported to at least some of these kernels. This is clearly there.

Great.

Quoted 4 lines

Mine: the commit sent earlier in this email thread - it's a re-back-port, as I think the original back-port for 4.14 (and similarly for 4.9) is incorrect. This has clearly not reached public git, hence my question about whether the change was queued.

I don't know what the git commit id you are looking for here, sorry. I don't have the whole thread anywhere.

Quoted 4 lines

These are the ids of messages containing my commits:

4.14: [email protected]
4.9: [email protected]

Quoted 7 lines
Quoted 2 lines

Pointers to this on lore.kernel.org perhaps?

I wasn't aware stable was publicly archived. Here you go:

4.14: https://lore.kernel.org/stable/[email protected]/
4.9: https://lore.kernel.org/stable/[email protected]/

As per the git commit id in those emails, you should be able to determine the same thing that I can, namely that that patch is in the following stable releases:

	4.4.224 4.9.219 4.14.176 4.19

thanks,

greg k-h

#8 lore raw

Hi.

Quoted 75 lines · Greg KH
Quoted 73 lines · Giuliano Procida

Hi.

Quoted 60 lines · Greg KH
Quoted 15 lines · Giuliano Procida

Hi.

Quoted 12 lines · Greg KH

A: http://en.wikipedia.org/wiki/Top_post
Q: Were do I find info about this thing called top-posting?
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?
A: Top-posting.
Q: What is the most annoying thing in e-mail?

A: No.
Q: Should I include quotations after my reply?

Quoted 7 lines
Quoted 3 lines

http://daringfireball.net/2007/07/on_top

:)

I'm well aware of the above.
Alas, I haven't used mutt properly in about 15 years and I'm still doing everything with Gmail.

gmail can handle proper quoting, if you are stuck with that :)

Quoted 3 lines

Given that I was referring to the entire email thread, I punted on finding a place to insert a comment.
BTW, there's a typo in the Q&A above. s/Were/Where/

Ah, nice catch, first one to notice that in years!

Quoted 9 lines
Quoted 9 lines · Giuliano Procida
Quoted 3 lines

Hi Greg.

Is this patch (and the similar one for 4.9) queued?

f5bbbbe4d635 ("blk-mq: sync the update nr_hw_queues with blk_mq_queue_tag_busy_iter") is in the following stable releases:

        4.4.224 4.9.219 4.14.176 4.19
Quoted 7 lines
Quoted 1 line

Do you not see it there?

We are referring to different "it"s.

Yours: f5bbbbe4d635 is the upstream patch that went into v4.19-rc1 and which you back-ported to at least some of these kernels. This is clearly there.

Great.

Quoted 4 lines

Mine: the commit sent earlier in this email thread - it's a re-back-port, as I think the original back-port for 4.14 (and similarly for 4.9) is incorrect. This has clearly not reached public git, hence my question about whether the change was queued.

I don't know what the git commit id you are looking for here, sorry. I don't have the whole thread anywhere.

Quoted 7 lines
Quoted 4 lines

These are the ids of messages containing my commits:

4.14: [email protected]
4.9: [email protected]

Pointers to this on lore.kernel.org perhaps?

I wasn't aware stable was publicly archived. Here you go:

Quoted 7 lines
Quoted 2 lines

4.14: https://lore.kernel.org/stable/[email protected]/
4.9: https://lore.kernel.org/stable/[email protected]/

As per the git commit id in those emails, you should be able to determine the same thing that I can, namely that that patch is in the following stable releases:

        4.4.224 4.9.219 4.14.176 4.19

Both those commits contain the text:

"Backporting Notes

This is a re-backport, landing synchronize_rcu in the right place."

If there's a special procedure for fixing back-ports, let me know, 'cos I haven't found one!

Cheers,
Giuliano.

Quoted 3 lines

thanks,

greg k-h

#9 lore raw
Quoted 86 lines · Giuliano Procida

Hi.

Quoted 75 lines · Greg KH
Quoted 73 lines · Giuliano Procida

Hi.

Quoted 60 lines · Greg KH
Quoted 15 lines · Giuliano Procida

Hi.

Quoted 12 lines · Greg KH

A: http://en.wikipedia.org/wiki/Top_post
Q: Were do I find info about this thing called top-posting?
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?
A: Top-posting.
Q: What is the most annoying thing in e-mail?

A: No.
Q: Should I include quotations after my reply?

Quoted 7 lines
Quoted 3 lines

http://daringfireball.net/2007/07/on_top

:)

I'm well aware of the above.
Alas, I haven't used mutt properly in about 15 years and I'm still doing everything with Gmail.

gmail can handle proper quoting, if you are stuck with that :)

Quoted 3 lines

Given that I was referring to the entire email thread, I punted on finding a place to insert a comment.
BTW, there's a typo in the Q&A above. s/Were/Where/

Ah, nice catch, first one to notice that in years!

Quoted 9 lines
Quoted 9 lines · Giuliano Procida
Quoted 3 lines

Hi Greg.

Is this patch (and the similar one for 4.9) queued?

f5bbbbe4d635 ("blk-mq: sync the update nr_hw_queues with blk_mq_queue_tag_busy_iter") is in the following stable releases:

        4.4.224 4.9.219 4.14.176 4.19
Quoted 7 lines
Quoted 1 line

Do you not see it there?

We are referring to different "it"s.

Yours: f5bbbbe4d635 is the upstream patch that went into v4.19-rc1 and which you back-ported to at least some of these kernels. This is clearly there.

Great.

Quoted 4 lines

Mine: the commit sent earlier in this email thread - it's a re-back-port, as I think the original back-port for 4.14 (and similarly for 4.9) is incorrect. This has clearly not reached public git, hence my question about whether the change was queued.

I don't know what the git commit id you are looking for here, sorry. I don't have the whole thread anywhere.

Quoted 7 lines
Quoted 4 lines

These are the ids of messages containing my commits:

4.14: [email protected]
4.9: [email protected]

Pointers to this on lore.kernel.org perhaps?

I wasn't aware stable was publicly archived. Here you go:

Quoted 7 lines
Quoted 2 lines

4.14: https://lore.kernel.org/stable/[email protected]/
4.9: https://lore.kernel.org/stable/[email protected]/

As per the git commit id in those emails, you should be able to determine the same thing that I can, namely that that patch is in the following stable releases:

        4.4.224 4.9.219 4.14.176 4.19
Quoted 8 lines

Both those commits contain the text:

"Backporting Notes

This is a re-backport, landing synchronize_rcu in the right place."

If there's a special procedure for fixing back-ports, let me know, 'cos I haven't found one!

Ah, I missed that, as it was not obvious.

I can't take a patch that is a redo of a previous patch as it will not apply, right?

Send a patch saying, specifically, something like "This fixes a broken backport that was commit XXX which was commit YYY upstream..." and so on.

otherwise it gets lost in the noise as you are seeing here...

thansk,

greg k-h