net: sched: gred: prevent races when adding offloads to stats
Subsystems (3)
Jakub Kicinski
maintainerPATCH 1/1 net: sched: gred: prevent races when adding offloads to stats
Naresh reports seeing a warning that gred is calling u64_stats_update_begin() with preemption enabled. Arnd points out it's coming from _bstats_update().
We should be holding the qdisc lock when writing to stats, they are also updated from the datapath.
---
CC: [email protected]
CC: [email protected]
CC: [email protected]
CC: [email protected]
--- net/sched/sch_gred.c | 2 ++ 1 file changed, 2 insertions(+)
net/sched/sch_gred.c +2 -0
@@ -377,6 +377,7 @@ static int gred_offload_dump_stats(struct Qdisc *sch)/* Even if driver returns failure adjust the stats - in case offload * ended but driver still wants to adjust the values. */ +sch_tree_lock (sch);for (i =0 ; i < MAX_DPs; i++) {if (!table->tab[i])continue ;
@@ -393,6 +394,7 @@ static int gred_offload_dump_stats(struct Qdisc *sch) sch->qstats.overlimits += hw_stats->stats.qstats[i].overlimits; }_bstats_update (&sch->bstats, bytes, packets); +sch_tree_unlock (sch);kfree (hw_stats);return ret;
Signature · 1 line
2.38.1
Quoted 3 lines · Jakub Kicinski
Naresh reports seeing a warning that gred is calling
u64_stats_update_begin()with preemption enabled. Arnd points out it's coming from_bstats_update().
The stack trace looks confusing to me without further decoding.
Are you sure we use sch->qstats/bstats in __dev_queue_xmit() there not any netdev stats? It may be a false positive one as they may end up with the same lockdep class.
Thanks.
Paolo Abeni
maintainerHello,
Quoted 11 lines · Cong Wang
Quoted 3 lines · Jakub Kicinski
Naresh reports seeing a warning that gred is calling
u64_stats_update_begin()with preemption enabled. Arnd points out it's coming from_bstats_update().The stack trace looks confusing to me without further decoding.
Are you sure we use sch->qstats/bstats in
__dev_queue_xmit()there not any netdev stats? It may be a false positive one as they may end up with the same lockdep class.
I'm unsure I read you comment correctly. Please note that the referenced message includes several splats. The first one - arguably the most relevant - points to the lack of locking in the gred control path.
The posted patch LGTM, could you please re-phrase your doubts?
Thanks,
Paolo
Jakub Kicinski
maintainerQuoted 11 lines · On Tue, 17 Jan 2023 10:00:56 +0100 Paolo Abeni wrote:
Quoted 10 lines · Cong Wang
Quoted 3 lines · Jakub Kicinski
Naresh reports seeing a warning that gred is calling
u64_stats_update_begin()with preemption enabled. Arnd points out it's coming from_bstats_update().The stack trace looks confusing to me without further decoding.
Are you sure we use sch->qstats/bstats in
__dev_queue_xmit()there not any netdev stats? It may be a false positive one as they may end up with the same lockdep class.
I didn't repro this myself, TBH, but there is u64_stats_update_begin() inside _bstats_update(). Pretty sure it will trigger the warning that preemption is not disabled on non-SMP systems.
Quoted 4 lines
I'm unsure I read you comment correctly. Please note that the referenced message includes several splats. The first one - arguably the most relevant - points to the lack of locking in the gred control path.
Yup, I'm not really sure if we're fixing the right splat for the bug. But I am fairly confident we should be holding a lock while writing bstats from the dump path, enqueue/dequeue may run concurrently.
Quoted 1 line
The posted patch LGTM, could you please re-phrase your doubts?
This patch was applied to netdev/net.git (master) by Jakub Kicinski <[email protected]>:
Hello:
This patch was applied to netdev/net.git (master)
by Jakub Kicinski <[email protected]>:
Quoted 8 lines · On Thu, 12 Jan 2023 20:41:37 -0800 you wrote:
Naresh reports seeing a warning that gred is calling
u64_stats_update_begin()with preemption enabled. Arnd points out it's coming from_bstats_update().We should be holding the qdisc lock when writing to stats, they are also updated from the datapath.
[...]
Here is the summary with links:
- [net] net: sched: gred: prevent races when adding offloads to stats
https://git.kernel.org/netdev/net/c/339346d49ae0
You are awesome, thank you!
Signature · 2 lines
Deet-doot-dot, I am a bot. https://korg.docs.kernel.org/patchwork/pwbot.html
Quoted 11 lines · Paolo Abeni
Hello,
Quoted 8 lines · Cong Wang
Quoted 3 lines · Jakub Kicinski
Naresh reports seeing a warning that gred is calling
u64_stats_update_begin()with preemption enabled. Arnd points out it's coming from_bstats_update().The stack trace looks confusing to me without further decoding.
Quoted 8 lines
Quoted 3 lines
Are you sure we use sch->qstats/bstats in
__dev_queue_xmit()there not any netdev stats? It may be a false positive one as they may end up with the same lockdep class.I'm unsure I read you comment correctly. Please note that the referenced message includes several splats. The first one - arguably the most relevant - points to the lack of locking in the gred control path.
I think it means lack of BH disable, not necessarily locking.
Quoted 2 lines
The posted patch LGTM, could you please re-phrase your doubts?
Yes, because the key in lockdep ("&syncp->seq#14") may be same for other dev stats (aka, not qdisc stats).
Also, HTB does not accquire this lock when dumping (htb_dump_class_stats()) stats. Why don't we have a problem there? ;)
Thanks.
Quoted 17 lines · Jakub Kicinski
Quoted 11 lines · On Tue, 17 Jan 2023 10:00:56 +0100 Paolo Abeni wrote:
Quoted 10 lines · Cong Wang
Quoted 3 lines · Jakub Kicinski
Naresh reports seeing a warning that gred is calling
u64_stats_update_begin()with preemption enabled. Arnd points out it's coming from_bstats_update().The stack trace looks confusing to me without further decoding.
Are you sure we use sch->qstats/bstats in
__dev_queue_xmit()there not any netdev stats? It may be a false positive one as they may end up with the same lockdep class.I didn't repro this myself, TBH, but there is
u64_stats_update_begin()inside_bstats_update(). Pretty sure it will trigger the warning that preemption is not disabled on non-SMP systems.
Quoted 8 lines
Quoted 4 lines
I'm unsure I read you comment correctly. Please note that the referenced message includes several splats. The first one - arguably the most relevant - points to the lack of locking in the gred control path.
Yup, I'm not really sure if we're fixing the right splat for the bug. But I am fairly confident we should be holding a lock while writing bstats from the dump path, enqueue/dequeue may run concurrently.
Explain htb_dump_class_stats()? :) I see two _bstats_update() calls but I don't see any tree lock there.
Thanks.
Jakub Kicinski
maintainerQuoted 11 lines · On Thu, 19 Jan 2023 12:19:53 -0800 Cong Wang wrote:
Quoted 8 lines
Quoted 4 lines
I'm unsure I read you comment correctly. Please note that the referenced message includes several splats. The first one - arguably the most relevant - points to the lack of locking in the gred control path.
Yup, I'm not really sure if we're fixing the right splat for the bug. But I am fairly confident we should be holding a lock while writing bstats from the dump path, enqueue/dequeue may run concurrently.
Explain
htb_dump_class_stats()? :) I see two_bstats_update()calls but I don't see any tree lock there.
I don't know the HTB offload, if the datapath writes to the same cl->bstats then it's buggy in the same way. But maybe it doesn't - the HTB offload is local (as in offloads normal host egress) while the RED offloads are for switches so the data flows both thru the "representor" netdevs and therefore the SW instance and the offload on the HW forwarding path.