net/sched: flag inconsistent qdisc dumps
Subsystems (4)
cover net/sched: flag inconsistent qdisc dumps
tc_dump_qdisc() walks every netdev in the netns and spans many netlink batches on any busy host. RTNL is only held while a single batch is filled, so the device list can change in between and the reply can miss entries -- but nothing under net/sched sets cb->seq, so NLM_F_DUMP_INTR is never raised and userspace cannot tell a partial listing from a complete one. rtnl_dump_ifinfo() already pairs for_each_netdev_dump() with cb->seq = dev_base_seq and nl_dump_check_consistent(); patch 1 does the same for the qdisc dump.
Patch 2 adds a selftest. Netlink dumps are driven by the reader, so it unregisters a device at a known point between two batches instead of racing against one, and a second case requires the flag to stay clear on an undisturbed dump. It fails on net/main and passes with patch 1.
Build coverage: allmodconfig builds clean with W=1 -- vmlinux and all modules link, no new warnings, none in net/sched. allyesconfig compiles clean with W=1, including net/sched/sch_api.o with no diagnostics in net/sched, but the final LD of vmlinux.o exceeds the memory of the machine used here, so that link was not completed.
Reshma Sreekumar (2):
net/sched: flag inconsistent qdisc dumps
selftests: net: check that a disturbed qdisc dump is flagged
net/sched/sch_api.c | 16 ++- tools/testing/selftests/net/Makefile | 1 + .../testing/selftests/net/qdisc_dump_intr.py | 123 ++++++++++++++++++ 3 files changed, 139 insertions(+), 1 deletion(-) create mode 100755 tools/testing/selftests/net/qdisc_dump_intr.py
base-commit: 51b093a7ba27476e1f639455f005e8d2e75390e4
Signature · 1 line
2.43.0
PATCH 1/2 net/sched: flag inconsistent qdisc dumps
tc_dump_qdisc() walks every netdev in the netns and can span many netlink batches. RTNL is only held while a single batch is filled, so devices can be registered or unregistered in between. Since 748bbef5fc6a ("net/sched:
switch tc_dump_qdisc() to for_each_netdev_dump()") the walk resumes on an ifindex rather than a list position, but ifindexes are reused: a device created between two batches can be given an ifindex the dump has already walked past, and is then missed entirely. Before that commit the resume used a position in the device list, so any unregister shifted every later device and silently dropped a run of qdiscs from the reply.
Unlike rtnl_dump_ifinfo(), which pairs for_each_netdev_dump() with cb->seq = dev_base_seq and nl_dump_check_consistent(), nothing under net/sched sets cb->seq at all -- and never has. NLM_F_DUMP_INTR is therefore never raised for a qdisc dump, an incomplete reply is indistinguishable from a complete one, and userspace has no way to tell it is reconciling against a partial listing. Callers already handle the flag by redoing the dump (iproute2 warns, netlink libraries retry), so raising it is enough to make this self-correcting.
Use the same idiom as the link dump. Both return paths are covered, as the -EMSGSIZE path is the normal one for every batch but the last.
Tested by dumping the qdiscs of 400 dummy devices (mq + 16 fq + clsact each) while unregistering devices ahead of the dump: NLM_F_DUMP_INTR is raised on 482 of 972 dumps with this patch and on 0 of 1089 without it.
--- net/sched/sch_api.c | 16 +++++++++++++++- 1 file changed, 15 insertions(+), 1 deletion(-)
net/sched/sch_api.c +15 -1
@@ -1949,12 +1949,26 @@ static int tc_dump_qdisc(struct sk_buff *skb, struct netlink_callback *cb)netdev_unlock_ops (dev); s_q_idx =0 ; } -return skb->len; + err = skb->len; + goto out; error_unlock:netdev_unlock_ops (dev); ctx->q_idx = q_idx; +out: +/* RTNL is dropped between dump batches, so devices may have been +* registered or unregistered while this dump was in flight. Because +* ifindexes get reused, a device created in that window can be given an +* ifindex this dump has already walked past and be missed entirely. +* Advertise that, as rtnl_dump_ifinfo() does for RTM_GETLINK, so that +* userspace redoes the dump instead of acting on a listing that silently +* omits entries. +*/ + cb->seq =READ_ONCE (net->dev_base_seq); +if (skb->len) +nl_dump_check_consistent (cb,nlmsg_hdr (skb)); +return err; }
Signature · 1 line
2.43.0
Quoted 10 lines · Reshma Sreekumar
tc_dump_qdisc()walks every netdev in the netns and can span many netlink batches. RTNL is only held while a single batch is filled, so devices can be registered or unregistered in between. Since748bbef5fc6a("net/sched:
switchtc_dump_qdisc()tofor_each_netdev_dump()") the walk resumes on an ifindex rather than a list position, but ifindexes are reused: a device created between two batches can be given an ifindex the dump has already walked past, and is then missed entirely. Before that commit the resume used a position in the device list, so any unregister shifted every later device and silently dropped a run of qdiscs from the reply.
I dislike this patch; it will force dump operations to restart on busy netns.
Most devices are created with a new ifindex (dev_index_reserve() is called with ifindex == 0)
Forcing a given ifindex is rather unusual.
I suggest detecting this specific use case instead of slowing down everything.
Quoted 1 line · Eric Dumazet
I dislike this patch; it will force dump operations to restart on
busy netns.
Quoted 5 lines
Most devices are created with a new ifindex (
dev_index_reserve()is called with ifindex == 0)Forcing a given ifindex is rather unusual.
Agreed.
For context on why I went looking: we hit this on production hosts running kernels predating 748bbef5fc6a. An agent that reconciles tc configuration against a qdisc dump saw qdiscs that did exist as absent, tried to create them again (-EEXIST), and tore down configuration it believed had gone stale. Because nothing sets cb->seq there, a truncated reply is indistinguishable from a complete one, so there was no way to detect it and retry.
748bbef5fc6a makes that far less likely, and I have not been able to reproduce the missing-qdisc case on 7.2-rc5. One case does remain, which this patch would not have fixed either: ctx->q_idx is a position within the per-device qdisc_hash walk, so a qdisc added or removed on the device at a batch boundary misaligns the skip count.
Is that worth reporting to userspace, and how would you want it detected?
Quoted 21 lines · Eric Dumazet
Quoted 10 lines · Reshma Sreekumar
tc_dump_qdisc()walks every netdev in the netns and can span many netlink batches. RTNL is only held while a single batch is filled, so devices can be registered or unregistered in between. Since748bbef5fc6a("net/sched:
switchtc_dump_qdisc()tofor_each_netdev_dump()") the walk resumes on an ifindex rather than a list position, but ifindexes are reused: a device created between two batches can be given an ifindex the dump has already walked past, and is then missed entirely. Before that commit the resume used a position in the device list, so any unregister shifted every later device and silently dropped a run of qdiscs from the reply.I dislike this patch; it will force dump operations to restart on busy netns.
Most devices are created with a new ifindex (
dev_index_reserve()is called with ifindex == 0)Forcing a given ifindex is rather unusual.
I suggest detecting this specific use case instead of slowing down everything.
PATCH 2/2 selftests: net: check that a disturbed qdisc dump is flagged
In reply to Reshma SreekumarAdd a test for NLM_F_DUMP_INTR on RTM_GETQDISC. A qdisc dump spans several netlink batches once a few devices carry a full set of qdiscs, and RTNL is only held while a single batch is filled, so the device list can change in between and entries can be missed.
Netlink dumps are driven by the reader, so rather than racing against a dump the test unregisters a device at a known point: it receives the first batch, unregisters a dummy, then drains the rest. This is deterministic -- one flagged message, every run.
A second case dumps without disturbing anything and requires that the flag stays clear, so the test cannot be satisfied by a kernel that raises it unconditionally.
Four dummy devices with mq + 16 fq + clsact each, read with a 4KiB buffer, give 12 batches and run in about a second. The test skips if dummy, mq, fq or clsact are unavailable.
--- tools/testing/selftests/net/Makefile | 1 + .../testing/selftests/net/qdisc_dump_intr.py | 123 ++++++++++++++++++ 2 files changed, 124 insertions(+) create mode 100755 tools/testing/selftests/net/qdisc_dump_intr.py
tools/testing/selftests/net/Makefile +1 -0
@@ -72,6 +72,7 @@ TEST_PROGS := \ pmtu.sh \ protodown.sh \ psock_snd.sh \ + qdisc_dump_intr.py \ reuseaddr_ports_exhausted.sh \ reuseport_addr_any.sh \ route_hint.sh \
tools/testing/selftests/net/qdisc_dump_intr.py new file +123 -0
123 lines not on this page · raw message
Signature · 1 line
2.43.0