Possible race with xsk_flush
Subsystems (3)
Hello,
I've been trying to track down a problem we've been seeing with occasional corrupt packets using AF_XDP. As you probably expect this occurs because we end up with the same descriptor in ring multiple times. However, after lots of debugging and analyzing I think this is actually caused by a kernel bug, though I still don't fully understand what is happening. This is currently being seen on kernel 5.15.77 and I haven't done any testing to see if we see the same issue on newer versions.
Inside my application I've written code to track and log as descriptors are placed into the fill and tx queues, and when they are pulled out of the rx and completion queues. The transitions are logged to /sys/kernel/debug/tracing/trace_marker. Additionally I keep my own VecDeque which mirrors the order descriptors are placed in the fill queue and I verify that they come out of the rx queue in the same order. I do realize there are some cases where they might not come out in the same order but I do not appear to be hitting that.
I then add several kprobes to track the kernel side with ftrace. Most importantly are these probes:
This adds a probe on the call to xskq_prod_reserve_desc() this actually creates two probes so you see two prints everytime it is hit:
perf probe -a '__xsk_rcv_zc:7 addr len xs xs->pool->fq'
This adds a probe on xsk_flush():
perf probe -a 'xsk_flush:0 xs'
My AF_XDP application is bound to two multi-queue veth interfaces (called 'ingress' and 'egress' in my prints) in a networking namespace. I'm then generating traffic with iperf and hping3 through these interfaces. When I see an out-of-order descriptor in my application I dump the state of my internal VecDeque to the trace_marker. Here is an example of what I'm seeing which looks like a kernel bug:
// On CPU 0 we've removed descriptor 0xff0900 from the fill queue,
// copied a packet, but have not put it in the rx queue yet
flowtrackd-9chS-142014 [000] d.Z1. 609766.698512: __xsk_rcv_zc_L7: (__xsk_rcv_zc+0x9b/0x250) addr=0xff0900 len=0x42 xs=0xffff90fd32693c00 fq=0xffff90fd03d66380 flowtrackd-9chS-142014 [000] d.Z1. 609766.698513: __xsk_rcv_zc_L7: (__xsk_rcv_zc+0xa7/0x250) addr=0xff0900 len=0x42 xs=0xffff90fd32693c00 fq=0xffff90fd03d66380 // On CPU 2 we've removed descriptor 0x1000900 from the fill queue,
// copied a packet, but have not put it in the rx queue yet
iperf2-1217 [002] d.Z1. 609766.698523: __xsk_rcv_zc_L7: (__xsk_rcv_zc+0x9b/0x250) addr=0x1000900 len=0x42 xs=0xffff90fd32693c00 fq=0xffff90fd03d66380
iperf2-1217 [002] d.Z1. 609766.698524: __xsk_rcv_zc_L7: (__xsk_rcv_zc+0xa7/0x250) addr=0x1000900 len=0x42 xs=0xffff90fd32693c00 fq=0xffff90fd03d66380
// On CPU 0 xsk_flush is called on the socket
flowtrackd-9chS-142014 [000] d.Z1. 609766.698528: xsk_flush: (__xsk_map_flush+0x4e/0x180) xs=0xffff90fd32693c00
// My application receives 0xff0900 on the ingress interface queue 1
flowtrackd-9chS-142014 [000] ..... 609766.698540: tracing_mark_write: ingress q:1 0xff0900 FILL -> RX
// On CPU 2 xsk_flush is called on the socket
iperf2-1217 [002] d.Z1. 609766.698545: xsk_flush: (__xsk_map_flush+0x4e/0x180) xs=0xffff90fd32693c00 // My application receives 0xf61900 this is unexpected. We expected
// to receive 0x1000900 which is what you saw in the previous
// __xsk_rcv_zc print. 0xf61900 is in the fill queue but it is far
// away from our current position in the ring and I've trimmed the
// print slightly to show that.
flowtrackd-9chS-142014 [000] ..... 609766.698617: tracing_mark_write: ingress q:1 0xf61900 FILL -> RX: expected 0x1000900 remaining: [fe4100, f9c100, f8a100, ..., f61900
From reading the code I believe that the call to xskq_prod_reserve_desc() inside __xsk_rcv_zc is the only place descriptors are placed into the RX queue. To me this means I should see a print from my probe for the mystery 0xf61900 but we do not see this print. Instead we see the expected 0x1000900. One theory I have is that there could be a race where CPU 2 increments the cached_prod pointer but has not yet updated the addr and len, CPU 0 calls xsk_flush(), and now my application reads the old descriptor entry from that location in the RX ring. This would explain everything, but the problem with this theory is that __xsk_rcv_zc() and xsk_flush() are getting called from within napi_poll() and this appears to hold the netpoll_poll_lock() for the whole time which I think should prevent the race I just described.
A couple more notes:
* The ftrace print order and timestamps seem to indicate that the CPU 2 napi_poll is running before the CPU 0 xsk_flush(). I don't know if these timestamps can be trusted but it does imply that maybe this can race as I described. I've triggered this twice with xsk_flush probes and both show the order above.
* In the 3 times I've triggered this it has occurred right when the softirq processing switches CPUs
* I've also triggered this before I added the xsk_flush() probe and in that case saw the kernel side additionally fill in the next expected descriptor, which in the example above would be 0xfe4100. This seems to indicate that my tracking is all still sane.
* This is fairly reproducible, but I've got 10 test boxes running and I only get maybe bug a day.
Any thoughts on if the bug I described is actually possible, alternative theories, or other things to test/try would be welcome.
Thanks,
Shawn Bohrer
Quoted 60 lines · Shawn Bohrer
Hello,
I've been trying to track down a problem we've been seeing with occasional corrupt packets using
AF_XDP. As you probably expect this occurs because we end up with the same descriptor in ring multiple times. However, after lots of debugging and analyzing I think this is actually caused by a kernel bug, though I still don't fully understand what is happening. This is currently being seen on kernel 5.15.77 and I haven't done any testing to see if we see the same issue on newer versions.Inside my application I've written code to track and log as descriptors are placed into the fill and tx queues, and when they are pulled out of the rx and completion queues. The transitions are logged to
/sys/kernel/debug/tracing/trace_marker. Additionally I keep my own VecDeque which mirrors the order descriptors are placed in the fill queue and I verify that they come out of the rx queue in the same order. I do realize there are some cases where they might not come out in the same order but I do not appear to be hitting that.I then add several kprobes to track the kernel side with ftrace. Most importantly are these probes:
This adds a probe on the call to
xskq_prod_reserve_desc()this actually creates two probes so you see two prints everytime it is hit:
perf probe -a '__xsk_rcv_zc:7 addr len xsxs->pool->fq'This adds a probe on
xsk_flush():
perf probe -a 'xsk_flush:0 xs'My
AF_XDPapplication is bound to two multi-queue veth interfaces (called 'ingress' and 'egress' in my prints) in a networking namespace. I'm then generating traffic with iperf and hping3 through these interfaces. When I see an out-of-order descriptor in my application I dump the state of my internal VecDeque to thetrace_marker. Here is an example of what I'm seeing which looks like a kernel bug:// On CPU 0 we've removed descriptor
0xff0900from the fill queue,
// copied a packet, but have not put it in the rx queue yet
flowtrackd-9chS-142014 [000] d.Z1. 609766.698512:__xsk_rcv_zc_L7: (__xsk_rcv_zc+0x9b/0x250) addr=0xff0900 len=0x42 xs=0xffff90fd32693c00 fq=0xffff90fd03d66380 flowtrackd-9chS-142014 [000] d.Z1. 609766.698513:__xsk_rcv_zc_L7: (__xsk_rcv_zc+0xa7/0x250) addr=0xff0900 len=0x42 xs=0xffff90fd32693c00 fq=0xffff90fd03d66380 // On CPU 2 we've removed descriptor0x1000900from the fill queue,
// copied a packet, but have not put it in the rx queue yetiperf2-1217 [002] d.Z1. 609766.698523: __xsk_rcv_zc_L7: (__xsk_rcv_zc+0x9b/0x250) addr=0x1000900 len=0x42 xs=0xffff90fd32693c00 fq=0xffff90fd03d66380 iperf2-1217 [002] d.Z1. 609766.698524: __xsk_rcv_zc_L7: (__xsk_rcv_zc+0xa7/0x250) addr=0x1000900 len=0x42 xs=0xffff90fd32693c00 fq=0xffff90fd03d66380 // On CPU 0 xsk_flush is called on the socket flowtrackd-9chS-142014 [000] d.Z1. 609766.698528: xsk_flush: (__xsk_map_flush+0x4e/0x180) xs=0xffff90fd32693c00 // My application receives 0xff0900 on the ingress interface queue 1 flowtrackd-9chS-142014 [000] ..... 609766.698540: tracing_mark_write: ingress q:1 0xff0900 FILL -> RX // On CPU 2 xsk_flush is called on the socket iperf2-1217 [002] d.Z1. 609766.698545: xsk_flush: (__xsk_map_flush+0x4e/0x180) xs=0xffff90fd32693c00// My application receives
0xf61900this is unexpected. We expected
// to receive0x1000900which is what you saw in the previous
//__xsk_rcv_zcprint.0xf61900is in the fill queue but it is far
// away from our current position in the ring and I've trimmed the
// print slightly to show that.
flowtrackd-9chS-142014 [000] ..... 609766.698617:tracing_mark_write: ingress q:10xf61900FILL -> RX: expected0x1000900remaining: [fe4100, f9c100, f8a100, ..., f61900
Quoted 13 lines
From reading the code I believe that the call to
xskq_prod_reserve_desc()inside__xsk_rcv_zcis the only place descriptors are placed into the RX queue. To me this means I should see a print from my probe for the mystery0xf61900but we do not see this print. Instead we see the expected0x1000900. One theory I have is that there could be a race where CPU 2 increments thecached_prodpointer but has not yet updated the addr and len, CPU 0 callsxsk_flush(), and now my application reads the old descriptor entry from that location in the RX ring. This would explain everything, but the problem with this theory is that__xsk_rcv_zc()andxsk_flush()are getting called from withinnapi_poll()and this appears to hold thenetpoll_poll_lock()for the whole time which I think should prevent the race I just described.
Thanks Shawn for your detailed bug report. The rings between user space and kernel space are single-producer/single-consumer, so in your case when both CPU0 and CPU2 are working on the fill ring and the Rx ring at the same time, this will indeed produce races. The intended design principle to protect against this is that only one NAPI context can access any given ring at any point in time and that the NAPI logic should prevent two instances of the same NAPI instance from running at the same time. So if that is not true for some reason, we would get a race like this. Another option is that one of the CPUs should really process another fill ring instead of the same.
Do you see the second socket being worked on when this happens?
Could you please share how you set up the two AF_XDP sockets?
Are you using XDP_DRV mode in your tests?
Quoted 8 lines
A couple more notes:
* The ftrace print order and timestamps seem to indicate that the CPU 2napi_pollis running before the CPU 0xsk_flush(). I don't know if these timestamps can be trusted but it does imply that maybe this can race as I described. I've triggered this twice withxsk_flushprobes and both show the order above.
* In the 3 times I've triggered this it has occurred right when the softirq processing switches CPUs
This is interesting. Could you check, in some way, if you only have one core working on the fill ring before the softirq switching and then after that you have two? And if you have two, is that period transient?
Quoted 9 lines
* I've also triggered this before I added the
xsk_flush()probe and in that case saw the kernel side additionally fill in the next expected descriptor, which in the example above would be0xfe4100. This seems to indicate that my tracking is all still sane.
* This is fairly reproducible, but I've got 10 test boxes running and I only get maybe bug a day.Any thoughts on if the bug I described is actually possible, alternative theories, or other things to test/try would be welcome.
I thought this would be impossible, but apparently not :-). We are apparently doing something wrong in the AF_XDP code or have the wrong assumptions in some situation, but I just do not know what at this point in time. Maybe it is veth that breaks some of our assumptions, who knows. But let us dig into it. I need your help here, because I think it will be hard for me to reproduce the issue.
Thanks: Magnus
Quoted 2 lines
Thanks,
Shawn Bohrer
Hi Magnus,
Quoted 1 line
Could you please share how you set up the two
AF_XDPsockets?
Our architecture is pretty unique:
outside of │ inside of
namespace │ namespace
│
┌───────┐ │ ┌───────┐
│ outer │ │ │ inner │
│ veth │ │ │ veth │
└──┬─▲──┘ │ └──┬─▲──┘
│ │ │ │ │
┌──▼─┴────┴────▼─┴──┐
│ shared umem │
└───────────────────┘The goal is to position ourselves in the middle of a veth pair so that we can perform bidirectional traffic inspection and manipulation. To do this, we attach AF_XDP to both veth interfaces and share a umem between them. This allows us to forward packets between the veth interfaces without copying in userspace.
These interfaces are both multi-queue, with AF_XDP sockets attached to each queue. The queues are each managed on their own (unpinned) threads and have their own rx/tx/fill/completion rings. We also enable threaded NAPI on both of these interfaces, which may or may not be an important detail to note, since the problem appears much harder (though not impossible) to reproduce with threaded NAPI enabled.
Here’s a script that configures a namespace and veth pair that closely resembles production, except for enabling threaded NAPI:
```
#!/bin/bash
set -e -u -x -o pipefail
QUEUES=${QUEUES:=$(($(grep -c ^processor /proc/cpuinfo)))}
OUTER_CUSTOMER_VETH=${OUTER_CUSTOMER_VETH:=outer-veth}
INNER_CUSTOMER_VETH=${INNER_CUSTOMER_VETH:=inner-veth}
CUSTOMER_NAMESPACE=${CUSTOMER_NAMESPACE:=customer-namespace}
ip netns add $CUSTOMER_NAMESPACE
ip netns exec $CUSTOMER_NAMESPACE bash <<EOF
set -e -u -x -o pipefail
ip addr add 127.0.0.1/8 dev lo
ip link set dev lo up
EOF
ip link add \
name $OUTER_CUSTOMER_VETH \
numrxqueues $QUEUES numtxqueues $QUEUES type veth \
peer name $INNER_CUSTOMER_VETH netns $CUSTOMER_NAMESPACE \
numrxqueues $QUEUES numtxqueues $QUEUES
ethtool -K $OUTER_CUSTOMER_VETH \
gro off gso off tso off tx off rxvlan off txvlan off
ip link set dev $OUTER_CUSTOMER_VETH up
ip addr add 169.254.10.1/30 dev $OUTER_CUSTOMER_VETH
ip netns exec $CUSTOMER_NAMESPACE bash <<EOF
set -e -u -x -o pipefail
ethtool -K $INNER_CUSTOMER_VETH \
gro off gso off tso off tx off rxvlan off txvlan off
ip link set dev $INNER_CUSTOMER_VETH up
ip addr add 169.254.10.2/30 dev $INNER_CUSTOMER_VETH
EOF
```Quoted 1 line
Are you using
XDP_DRVmode in your tests?
Yes.
Quoted 11 lines · Magnus Karlsson
Thanks Shawn for your detailed bug report. The rings between user space and kernel space are single-producer/single-consumer, so in your case when both CPU0 and CPU2 are working on the fill ring and the Rx ring at the same time, this will indeed produce races. The intended design principle to protect against this is that only one NAPI context can access any given ring at any point in time and that the NAPI logic should prevent two instances of the same NAPI instance from running at the same time. So if that is not true for some reason, we would get a race like this. Another option is that one of the CPUs should really process another fill ring instead of the same.
Quoted 3 lines
Do you see the second socket being worked on when this happens?
Could you please share how you set up the two
AF_XDPsockets?
Alex Forster sent more details on the configuration but just to reiterate there are actually 8 AF_XDP sockets in this test setup. There are two veth interfaces and each interface has four receive queues. We create one socket per interface/queue pair. Our XDP program redirects each packet to the correct AF_XDP socket based on the queue number.
Yes there is often activity on other sockets near the time when the bug occurs. This is why I'm printing xs/fq, the socket address and fill queue address, and printing the ingress/egress device name and queue number in my prints. This allows to match up the user space and kernel space prints. Additionally we are using a shared UMEM so descriptors could move around between sockets though I've tried to minimize this and in every case I've seen so far the mystery descriptor was last used on the same socket and has also been in the fill queue just not next in line.
Quoted 11 lines
Are you using
XDP_DRVmode in your tests?Quoted 8 lines
A couple more notes:
* The ftrace print order and timestamps seem to indicate that the CPU 2napi_pollis running before the CPU 0xsk_flush(). I don't know if these timestamps can be trusted but it does imply that maybe this can race as I described. I've triggered this twice withxsk_flushprobes and both show the order above.
* In the 3 times I've triggered this it has occurred right when the softirq processing switches CPUs
Quoted 4 lines
This is interesting. Could you check, in some way, if you only have one core working on the fill ring before the softirq switching and then after that you have two? And if you have two, is that period transient?
I think what you are asking is why does the softirq processing switch CPUs? There is still a lot I don't fully understand here but I've tried to understand this, if only to try to make it happen more frequently and make this easier to reproduce.
In this test setup there is no hardware IRQ. iperf2 sends the packet and the CPU where iperf is running runs the veth softirq. I'm not sure how it picks which veth receive queue receives the packets, but they end up distributed across the veth qeueus. Additionally __veth_xdp_flush() calls __napi_schedule(). This is called from veth_xdp_xmit() which I think means that transmitting packets from AF_XDP also schedules the softirq on the current CPU for that veth queue. What I definitely see is that if I pin both iperf and my application to a single CPU all softirqs of all queues run on that single CPU. If I pin iperf2 to one core and my application to another core I get softirqs for all veth queues on both cores.
In our test setup we aren't applying any cpu affinity. iperf2 is multi-threaded and can run on all 4 cores, and our application is multithreaded and can run on all 4 cores. The napi scheduling seems to be per veth queue and yes I see those softirqs move and switch between CPUs. I don't however have anything that clearly shows it running concurrently on two CPUs (The stretches of __xsk_rcv_zc are all on one core before it switches). The closest I have is the several microseconds where it appears xsk_flush() overlaps at the end of my traces. I would think that if the napi locking didn't work at all you'd see clear overlap.
From my experiments with CPU affinity I've updated my test setup to frequently change the CPU affinity of iperf and our application on one of my test boxes with hopes that it helps to reproduce but I have no results so far.
Quoted 10 lines
Quoted 9 lines
* I've also triggered this before I added the
xsk_flush()probe and in that case saw the kernel side additionally fill in the next expected descriptor, which in the example above would be0xfe4100. This seems to indicate that my tracking is all still sane.
* This is fairly reproducible, but I've got 10 test boxes running and I only get maybe bug a day.Any thoughts on if the bug I described is actually possible, alternative theories, or other things to test/try would be welcome.
Quoted 6 lines
I thought this would be impossible, but apparently not :-). We are apparently doing something wrong in the
AF_XDPcode or have the wrong assumptions in some situation, but I just do not know what at this point in time. Maybe it is veth that breaks some of our assumptions, who knows. But let us dig into it. I need your help here, because I think it will be hard for me to reproduce the issue.
Yeah if you have ideas on what to test I'll do my best to try them.
I've additionally updated my application to put a bad "cookie" descriptor address back in the RX ring before updating the consumer pointer. My hope is that if we then ever receive that cookie it proves the kernel raced and failed to update the correct address.
Thanks,
Shawn Bohrer
Quoted 55 lines · Shawn Bohrer
Quoted 11 lines · Magnus Karlsson
Thanks Shawn for your detailed bug report. The rings between user space and kernel space are single-producer/single-consumer, so in your case when both CPU0 and CPU2 are working on the fill ring and the Rx ring at the same time, this will indeed produce races. The intended design principle to protect against this is that only one NAPI context can access any given ring at any point in time and that the NAPI logic should prevent two instances of the same NAPI instance from running at the same time. So if that is not true for some reason, we would get a race like this. Another option is that one of the CPUs should really process another fill ring instead of the same.
Quoted 3 lines
Do you see the second socket being worked on when this happens?
Could you please share how you set up the two
AF_XDPsockets?Alex Forster sent more details on the configuration but just to reiterate there are actually 8
AF_XDPsockets in this test setup. There are two veth interfaces and each interface has four receive queues. We create one socket per interface/queue pair. Our XDP program redirects each packet to the correctAF_XDPsocket based on the queue number.Yes there is often activity on other sockets near the time when the bug occurs. This is why I'm printing xs/fq, the socket address and fill queue address, and printing the ingress/egress device name and queue number in my prints. This allows to match up the user space and kernel space prints. Additionally we are using a shared UMEM so descriptors could move around between sockets though I've tried to minimize this and in every case I've seen so far the mystery descriptor was last used on the same socket and has also been in the fill queue just not next in line.
Quoted 11 lines
Are you using
XDP_DRVmode in your tests?Quoted 8 lines
A couple more notes:
* The ftrace print order and timestamps seem to indicate that the CPU 2napi_pollis running before the CPU 0xsk_flush(). I don't know if these timestamps can be trusted but it does imply that maybe this can race as I described. I've triggered this twice withxsk_flushprobes and both show the order above.
* In the 3 times I've triggered this it has occurred right when the softirq processing switches CPUsQuoted 4 lines
This is interesting. Could you check, in some way, if you only have one core working on the fill ring before the softirq switching and then after that you have two? And if you have two, is that period transient?
I think what you are asking is why does the softirq processing switch CPUs? There is still a lot I don't fully understand here but I've tried to understand this, if only to try to make it happen more frequently and make this easier to reproduce.
Quoted 11 lines
In this test setup there is no hardware IRQ. iperf2 sends the packet and the CPU where iperf is running runs the veth softirq. I'm not sure how it picks which veth receive queue receives the packets, but they end up distributed across the veth qeueus. Additionally
__veth_xdp_flush()calls__napi_schedule(). This is called fromveth_xdp_xmit()which I think means that transmitting packets fromAF_XDPalso schedules the softirq on the current CPU for that veth queue. What I definitely see is that if I pin both iperf and my application to a single CPU all softirqs of all queues run on that single CPU. If I pin iperf2 to one core and my application to another core I get softirqs for all veth queues on both cores.
To summarize, we are expecting this ordering:
CPU 0 __xsk_rcv_zc()
CPU 0 __xsk_map_flush()
CPU 2 __xsk_rcv_zc()
CPU 2 __xsk_map_flush()
But we are seeing this order:
CPU 0 __xsk_rcv_zc()
CPU 2 __xsk_rcv_zc()
CPU 0 __xsk_map_flush()
CPU 2 __xsk_map_flush()
Here is the veth NAPI poll loop:
static int veth_poll(struct napi_struct *napi, int budget)
{
struct veth_rq *rq =
container_of(napi, struct veth_rq, xdp_napi);
struct veth_stats stats = {};
struct veth_xdp_tx_bq bq;
int done;
bq.count = 0;
xdp_set_return_frame_no_direct();
done = veth_xdp_rcv(rq, budget, &bq, &stats);
if (done < budget && napi_complete_done(napi, done)) {
/* Write rx_notify_masked before reading ptr_ring */
smp_store_mb(rq->rx_notify_masked, false);
if (unlikely(!__ptr_ring_empty(&rq->xdp_ring))) {
if (napi_schedule_prep(&rq->xdp_napi)) {
WRITE_ONCE(rq->rx_notify_masked, true);
__napi_schedule(&rq->xdp_napi);
}
}
}
if (stats.xdp_tx > 0)
veth_xdp_flush(rq, &bq);
if (stats.xdp_redirect > 0)
xdp_do_flush();
xdp_clear_return_frame_no_direct();
return done;
}Something I have never seen before is that there is napi_complete_done() and a __napi_schedule() before xdp_do_flush(). Let us check if this has something to do with it. So two suggestions to be executed separately:
* Put a probe at the __napi_schedule() above and check if it gets triggered before this problem
* Move the "if (stats.xdp_redirect > 0) xdp_do_flush();" to just before "if (done < budget && napi_complete_done(napi, done)) {"
This might provide us some hints on what is going on.
Thanks: Magnus
Quoted 42 lines
In our test setup we aren't applying any cpu affinity. iperf2 is multi-threaded and can run on all 4 cores, and our application is multithreaded and can run on all 4 cores. The napi scheduling seems to be per veth queue and yes I see those softirqs move and switch between CPUs. I don't however have anything that clearly shows it running concurrently on two CPUs (The stretches of
__xsk_rcv_zcare all on one core before it switches). The closest I have is the several microseconds where it appearsxsk_flush()overlaps at the end of my traces. I would think that if the napi locking didn't work at all you'd see clear overlap.From my experiments with CPU affinity I've updated my test setup to frequently change the CPU affinity of iperf and our application on one of my test boxes with hopes that it helps to reproduce but I have no results so far.
Quoted 10 lines
Quoted 9 lines
* I've also triggered this before I added the
xsk_flush()probe and in that case saw the kernel side additionally fill in the next expected descriptor, which in the example above would be0xfe4100. This seems to indicate that my tracking is all still sane.
* This is fairly reproducible, but I've got 10 test boxes running and I only get maybe bug a day.Any thoughts on if the bug I described is actually possible, alternative theories, or other things to test/try would be welcome.
Quoted 6 lines
I thought this would be impossible, but apparently not :-). We are apparently doing something wrong in the
AF_XDPcode or have the wrong assumptions in some situation, but I just do not know what at this point in time. Maybe it is veth that breaks some of our assumptions, who knows. But let us dig into it. I need your help here, because I think it will be hard for me to reproduce the issue.Yeah if you have ideas on what to test I'll do my best to try them.
I've additionally updated my application to put a bad "cookie" descriptor address back in the RX ring before updating the consumer pointer. My hope is that if we then ever receive that cookie it proves the kernel raced and failed to update the correct address.
Thanks,
Shawn Bohrer
Quoted 7 lines · Magnus Karlsson
To summarize, we are expecting this ordering:
CPU 0
__xsk_rcv_zc()
CPU 0__xsk_map_flush()
CPU 2__xsk_rcv_zc()
CPU 2__xsk_map_flush()
Quoted 6 lines
But we are seeing this order:
CPU 0
__xsk_rcv_zc()
CPU 2__xsk_rcv_zc()
CPU 0__xsk_map_flush()
CPU 2__xsk_map_flush()
Yes exactly, and I think I've proved that this really is the order, and the race is occurring. See my cookie/poisoning below.
Quoted 40 lines
Here is the veth NAPI poll loop:
static int
veth_poll(struct napi_struct *napi, int budget){ struct veth_rq *rq = container_of(napi, struct veth_rq, xdp_napi); struct veth_stats stats = {}; struct veth_xdp_tx_bq bq; int done; bq.count = 0; xdp_set_return_frame_no_direct(); done = veth_xdp_rcv(rq, budget, &bq, &stats); if (done < budget && napi_complete_done(napi, done)) { /* Write rx_notify_masked before reading ptr_ring */ smp_store_mb(rq->rx_notify_masked, false); if (unlikely(!__ptr_ring_empty(&rq->xdp_ring))) { if (napi_schedule_prep(&rq->xdp_napi)) { WRITE_ONCE(rq->rx_notify_masked, true); __napi_schedule(&rq->xdp_napi); } } } if (stats.xdp_tx > 0) veth_xdp_flush(rq, &bq); if (stats.xdp_redirect > 0) xdp_do_flush(); xdp_clear_return_frame_no_direct(); return done; }Something I have never seen before is that there is
napi_complete_done()and a__napi_schedule()beforexdp_do_flush(). Let us check if this has something to do with it. So two suggestions to be executed separately:
Quoted 6 lines
* Put a probe at the
__napi_schedule()above and check if it gets triggered before this problem
* Move the "if (stats.xdp_redirect > 0)xdp_do_flush();" to just before "if (done < budget &&napi_complete_done(napi, done)) {"This might provide us some hints on what is going on.
Excellent observation, I haven't really looked at what napi_complete_done() does yet. I did notice it could call __napi_schedule() and that seemed like it might be fine. I'll also note that veth_xdp_flush() can also ultimately call __napi_schedule(). I'll see what I can do to explore these ideas.
Quoted 4 lines
Quoted 4 lines
I've additionally updated my application to put a bad "cookie" descriptor address back in the RX ring before updating the consumer pointer. My hope is that if we then ever receive that cookie it proves the kernel raced and failed to update the correct address.
I guess this is more like poisoning the old descriptors rather than a cookie. This ran last night and one of my machines read back my 0xdeadbeefdeadbeef poisoned cookie value:
iperf2-125483 [003] d.Z1. 792878.867088: __xsk_rcv_zc_L7: (__xsk_rcv_zc+0xa7/0x250) addr=0x8d4900 len=0x42 xs=0xffff8bbc542a5000 fq=0xffff8bbc1c464e40
iperf2-125483 [003] d.Z1. 792878.867093: xsk_flush: (__xsk_map_flush+0x4e/0x180) xs=0xffff8bbc542a5000
iperf2-125491 [001] d.Z1. 792878.867219: __xsk_rcv_zc_L7: (__xsk_rcv_zc+0xa7/0x250) addr=0xc79900 len=0x42 xs=0xffff8bbc542a5000 fq=0xffff8bbc1c464e40
iperf2-125491 [001] d.Z1. 792878.867229: xsk_flush: (__xsk_map_flush+0x4e/0x180) xs=0xffff8bbc542a5000
iperf2-125491 [001] d.Z1. 792878.867291: __xsk_rcv_zc_L7: (__xsk_rcv_zc+0xa7/0x250) addr=0x18e1900 len=0x42 xs=0xffff8bbc542a5000 fq=0xffff8bbc1c464e40
iperf2-125483 [003] d.Z1. 792878.867441: __xsk_rcv_zc_L7: (__xsk_rcv_zc+0xa7/0x250) addr=0xc0a900 len=0x42 xs=0xffff8bbc542a5000 fq=0xffff8bbc1c464e40
iperf2-125491 [001] d.Z1. 792878.867457: xsk_flush: (__xsk_map_flush+0x4e/0x180) xs=0xffff8bbc542a5000
flowtrackd-zjTA-201813 [001] ..... 792878.867496: tracing_mark_write: ingress q:2 0x8d4900 FILL -> RX
flowtrackd-zjTA-201813 [001] ..... 792878.867503: tracing_mark_write: ingress q:2 0xc79900 FILL -> RX
flowtrackd-zjTA-201813 [001] ..... 792878.867506: tracing_mark_write: ingress q:2 0x18e1900 FILL -> RX
flowtrackd-zjTA-201813 [001] ..... 792878.867524: tracing_mark_write: read invalid descriptor cookie: 0xdeadbeefdeadbeefThis shows what I've seen before where the xsk_flush() of CPU 1 runs after (during?) __xsk_rcv_zc() of CPU 3. In this trace we never see the xsk_flush() from CPU 3 but I stop tracing when the bug occurs so it probably just hasn't happened yet.
So at least to me this does confirm there is definitely a race here where we can flush an updated producer pointer before the descriptor address has been filled in.
--
Shawn
Quoted 54 lines · Magnus Karlsson
To summarize, we are expecting this ordering:
CPU 0
__xsk_rcv_zc()
CPU 0__xsk_map_flush()
CPU 2__xsk_rcv_zc()
CPU 2__xsk_map_flush()But we are seeing this order:
CPU 0
__xsk_rcv_zc()
CPU 2__xsk_rcv_zc()
CPU 0__xsk_map_flush()
CPU 2__xsk_map_flush()Here is the veth NAPI poll loop:
static int
veth_poll(struct napi_struct *napi, int budget){ struct veth_rq *rq = container_of(napi, struct veth_rq, xdp_napi); struct veth_stats stats = {}; struct veth_xdp_tx_bq bq; int done; bq.count = 0; xdp_set_return_frame_no_direct(); done = veth_xdp_rcv(rq, budget, &bq, &stats); if (done < budget && napi_complete_done(napi, done)) { /* Write rx_notify_masked before reading ptr_ring */ smp_store_mb(rq->rx_notify_masked, false); if (unlikely(!__ptr_ring_empty(&rq->xdp_ring))) { if (napi_schedule_prep(&rq->xdp_napi)) { WRITE_ONCE(rq->rx_notify_masked, true); __napi_schedule(&rq->xdp_napi); } } } if (stats.xdp_tx > 0) veth_xdp_flush(rq, &bq); if (stats.xdp_redirect > 0) xdp_do_flush(); xdp_clear_return_frame_no_direct(); return done; }Something I have never seen before is that there is
napi_complete_done()and a__napi_schedule()beforexdp_do_flush(). Let us check if this has something to do with it. So two suggestions to be executed separately:
Quoted 4 lines
* Put a probe at the
__napi_schedule()above and check if it gets triggered before this problem
* Move the "if (stats.xdp_redirect > 0)xdp_do_flush();" to just before "if (done < budget &&napi_complete_done(napi, done)) {"
I've built a kernel moving xdp_do_flush() before napi_complete_done() and will leave this running over the weekend. I also spent a while trying to understand the napi code to see if that reordering seemed like it might cause the bug. I'll admit I still don't fully undestand the napi code or how the bug can happen. We'll see if this appears to fix the issue, but since it is somewhat hard to reproduce I'd love for someone to be able to explain why it fixes it.
Thanks,
Shawn Bohrer
Quoted 54 lines · Magnus Karlsson
To summarize, we are expecting this ordering:
CPU 0
__xsk_rcv_zc()
CPU 0__xsk_map_flush()
CPU 2__xsk_rcv_zc()
CPU 2__xsk_map_flush()But we are seeing this order:
CPU 0
__xsk_rcv_zc()
CPU 2__xsk_rcv_zc()
CPU 0__xsk_map_flush()
CPU 2__xsk_map_flush()Here is the veth NAPI poll loop:
static int
veth_poll(struct napi_struct *napi, int budget){ struct veth_rq *rq = container_of(napi, struct veth_rq, xdp_napi); struct veth_stats stats = {}; struct veth_xdp_tx_bq bq; int done; bq.count = 0; xdp_set_return_frame_no_direct(); done = veth_xdp_rcv(rq, budget, &bq, &stats); if (done < budget && napi_complete_done(napi, done)) { /* Write rx_notify_masked before reading ptr_ring */ smp_store_mb(rq->rx_notify_masked, false); if (unlikely(!__ptr_ring_empty(&rq->xdp_ring))) { if (napi_schedule_prep(&rq->xdp_napi)) { WRITE_ONCE(rq->rx_notify_masked, true); __napi_schedule(&rq->xdp_napi); } } } if (stats.xdp_tx > 0) veth_xdp_flush(rq, &bq); if (stats.xdp_redirect > 0) xdp_do_flush(); xdp_clear_return_frame_no_direct(); return done; }Something I have never seen before is that there is
napi_complete_done()and a__napi_schedule()beforexdp_do_flush(). Let us check if this has something to do with it. So two suggestions to be executed separately:
Quoted 6 lines
* Put a probe at the
__napi_schedule()above and check if it gets triggered before this problem
* Move the "if (stats.xdp_redirect > 0)xdp_do_flush();" to just before "if (done < budget &&napi_complete_done(napi, done)) {"This might provide us some hints on what is going on.
After staring at this code for way too long I finally made a breakthrough! I could not understand how this race could occur when napi_poll() calls netpoll_poll_lock(). Here is netpoll_poll_lock():
```
static inline void *netpoll_poll_lock(struct napi_struct *napi)
{
struct net_device *dev = napi->dev;
if (dev && dev->npinfo) {
int owner = smp_processor_id();
while (cmpxchg(&napi->poll_owner, -1, owner) != -1)
cpu_relax();
return napi;
}
return NULL;
}
```If dev or dev->npinfo are NULL then it doesn't acquire a lock at all! Adding some more trace points I see:
``` iperf2-1325 [002] ..s1. 264246.626880: __napi_poll: (__napi_poll+0x0/0x150) n=0xffff91c885bff000 poll_owner=-1 dev=0xffff91c881d4e000 npinfo=0x0 iperf2-1325 [002] d.Z1. 264246.626882: __xsk_rcv_zc_L7: (__xsk_rcv_zc+0x3b/0xc0) addr=0x1503100 len=0x42 xs=0xffff91c8bfe77000 fq=0xffff91c8c1a43f80 dev=0xffff91c881d4e000 iperf2-1325 [002] d.Z1. 264246.626883: __xsk_rcv_zc_L7: (__xsk_rcv_zc+0x42/0xc0) addr=0x1503100 len=0x42 xs=0xffff91c8bfe77000 fq=0xffff91c8c1a43f80 dev=0xffff91c881d4e000 iperf2-1325 [002] d.Z1. 264246.626884: xsk_flush: (__xsk_map_flush+0x32/0xb0) xs=0xffff91c8bfe77000 ```
Here you can see that poll_owner=-1 meaning the lock was never acquired because npinfo is NULL. This means that the same veth rx queue can be napi_polled from multiple CPU and nothing stops it from running concurrently. They all look like this, just most of the time there aren't concurrent napi_polls running for the same queue. They do however move around CPUs as I explained earlier.
I'll note that I've ran with your suggested change of moving xdp_do_flush() before napi_complete_done() all weekend and I have not reproduced the issue. I don't know if that truly means the issue is fixed by that change or not. I suspect it does fix the issue because it prevents the napi_struct from being scheduled again before the first poll has completed, and nap_schedule_prep() ensures that only one instance is ever running.
If we think this is the correct fix I'll let it run for another day or two and prepare a patch.
Thanks,
Shawn Bohrer
Quoted 100 lines · Shawn Bohrer
Quoted 54 lines · Magnus Karlsson
To summarize, we are expecting this ordering:
CPU 0
__xsk_rcv_zc()
CPU 0__xsk_map_flush()
CPU 2__xsk_rcv_zc()
CPU 2__xsk_map_flush()But we are seeing this order:
CPU 0
__xsk_rcv_zc()
CPU 2__xsk_rcv_zc()
CPU 0__xsk_map_flush()
CPU 2__xsk_map_flush()Here is the veth NAPI poll loop:
static int
veth_poll(struct napi_struct *napi, int budget){ struct veth_rq *rq = container_of(napi, struct veth_rq, xdp_napi); struct veth_stats stats = {}; struct veth_xdp_tx_bq bq; int done; bq.count = 0; xdp_set_return_frame_no_direct(); done = veth_xdp_rcv(rq, budget, &bq, &stats); if (done < budget && napi_complete_done(napi, done)) { /* Write rx_notify_masked before reading ptr_ring */ smp_store_mb(rq->rx_notify_masked, false); if (unlikely(!__ptr_ring_empty(&rq->xdp_ring))) { if (napi_schedule_prep(&rq->xdp_napi)) { WRITE_ONCE(rq->rx_notify_masked, true); __napi_schedule(&rq->xdp_napi); } } } if (stats.xdp_tx > 0) veth_xdp_flush(rq, &bq); if (stats.xdp_redirect > 0) xdp_do_flush(); xdp_clear_return_frame_no_direct(); return done; }Something I have never seen before is that there is
napi_complete_done()and a__napi_schedule()beforexdp_do_flush(). Let us check if this has something to do with it. So two suggestions to be executed separately:Quoted 6 lines
* Put a probe at the
__napi_schedule()above and check if it gets triggered before this problem
* Move the "if (stats.xdp_redirect > 0)xdp_do_flush();" to just before "if (done < budget &&napi_complete_done(napi, done)) {"This might provide us some hints on what is going on.
After staring at this code for way too long I finally made a breakthrough! I could not understand how this race could occur when
napi_poll()callsnetpoll_poll_lock(). Here isnetpoll_poll_lock():``` static inline void *netpoll_poll_lock(struct napi_struct *napi) { struct net_device *dev = napi->dev; if (dev && dev->npinfo) { int owner = smp_processor_id(); while (cmpxchg(&napi->poll_owner, -1, owner) != -1) cpu_relax(); return napi; } return NULL; } ```If dev or
dev->npinfoare NULL then it doesn't acquire a lock at all! Adding some more trace points I see:``` iperf2-1325 [002] ..s1. 264246.626880: __napi_poll: (__napi_poll+0x0/0x150) n=0xffff91c885bff000 poll_owner=-1 dev=0xffff91c881d4e000 npinfo=0x0 iperf2-1325 [002] d.Z1. 264246.626882: __xsk_rcv_zc_L7: (__xsk_rcv_zc+0x3b/0xc0) addr=0x1503100 len=0x42 xs=0xffff91c8bfe77000 fq=0xffff91c8c1a43f80 dev=0xffff91c881d4e000 iperf2-1325 [002] d.Z1. 264246.626883: __xsk_rcv_zc_L7: (__xsk_rcv_zc+0x42/0xc0) addr=0x1503100 len=0x42 xs=0xffff91c8bfe77000 fq=0xffff91c8c1a43f80 dev=0xffff91c881d4e000 iperf2-1325 [002] d.Z1. 264246.626884: xsk_flush: (__xsk_map_flush+0x32/0xb0) xs=0xffff91c8bfe77000 ```Here you can see that poll_owner=-1 meaning the lock was never acquired because npinfo is NULL. This means that the same veth rx queue can be
napi_polledfrom multiple CPU and nothing stops it from running concurrently. They all look like this, just most of the time there aren't concurrentnapi_pollsrunning for the same queue. They do however move around CPUs as I explained earlier.
Quoted 7 lines
I'll note that I've ran with your suggested change of moving
xdp_do_flush()beforenapi_complete_done()all weekend and I have not reproduced the issue. I don't know if that truly means the issue is fixed by that change or not. I suspect it does fix the issue because it prevents thenapi_structfrom being scheduled again before the first poll has completed, andnap_schedule_prep()ensures that only one instance is ever running.
Thanks Shawn! Good news that the patch seems to fix the problem. To me, napi_schedule_prep() makes sure that only one NAPI instance is running. Netpoll is an optional feature and I do not even have it compiled into my kernel. At least I do not have it defined in my .config and I cannot find any netpoll symbols with a readelf command. If netpoll is not used, I would suspect that npinfo == NULL. So to me, it is still a mystery why this is happening.
To validate or disprove that there are two instances of the same napi running, could you please put a probe in veth_poll() right after veth_xdp_rcv() and one at xdp_do_flush() and dump the napi id in both cases? And run the original code with the bug :-). It would be good to know what exactly happens in the code between these points when this bug occurs. Maybe we could do this by enabling the trace buffer and dumping it when the bug occurs.
Quoted 2 lines
If we think this is the correct fix I'll let it run for another day or two and prepare a patch.
There is one more advantage to this bug fix and that is performance. By calling __xdp_map_flush() immediately after the receive function, user space can start to work on the packets quicker so performance will improve.
Quoted 2 lines
Thanks,
Shawn Bohrer
Quoted 109 lines · Magnus Karlsson
Quoted 100 lines · Shawn Bohrer
Quoted 54 lines · Magnus Karlsson
To summarize, we are expecting this ordering:
CPU 0
__xsk_rcv_zc()
CPU 0__xsk_map_flush()
CPU 2__xsk_rcv_zc()
CPU 2__xsk_map_flush()But we are seeing this order:
CPU 0
__xsk_rcv_zc()
CPU 2__xsk_rcv_zc()
CPU 0__xsk_map_flush()
CPU 2__xsk_map_flush()Here is the veth NAPI poll loop:
static int
veth_poll(struct napi_struct *napi, int budget){ struct veth_rq *rq = container_of(napi, struct veth_rq, xdp_napi); struct veth_stats stats = {}; struct veth_xdp_tx_bq bq; int done; bq.count = 0; xdp_set_return_frame_no_direct(); done = veth_xdp_rcv(rq, budget, &bq, &stats); if (done < budget && napi_complete_done(napi, done)) { /* Write rx_notify_masked before reading ptr_ring */ smp_store_mb(rq->rx_notify_masked, false); if (unlikely(!__ptr_ring_empty(&rq->xdp_ring))) { if (napi_schedule_prep(&rq->xdp_napi)) { WRITE_ONCE(rq->rx_notify_masked, true); __napi_schedule(&rq->xdp_napi); } } } if (stats.xdp_tx > 0) veth_xdp_flush(rq, &bq); if (stats.xdp_redirect > 0) xdp_do_flush(); xdp_clear_return_frame_no_direct(); return done; }Something I have never seen before is that there is
napi_complete_done()and a__napi_schedule()beforexdp_do_flush(). Let us check if this has something to do with it. So two suggestions to be executed separately:Quoted 6 lines
* Put a probe at the
__napi_schedule()above and check if it gets triggered before this problem
* Move the "if (stats.xdp_redirect > 0)xdp_do_flush();" to just before "if (done < budget &&napi_complete_done(napi, done)) {"This might provide us some hints on what is going on.
After staring at this code for way too long I finally made a breakthrough! I could not understand how this race could occur when
napi_poll()callsnetpoll_poll_lock(). Here isnetpoll_poll_lock():``` static inline void *netpoll_poll_lock(struct napi_struct *napi) { struct net_device *dev = napi->dev; if (dev && dev->npinfo) { int owner = smp_processor_id(); while (cmpxchg(&napi->poll_owner, -1, owner) != -1) cpu_relax(); return napi; } return NULL; } ```If dev or
dev->npinfoare NULL then it doesn't acquire a lock at all! Adding some more trace points I see:``` iperf2-1325 [002] ..s1. 264246.626880: __napi_poll: (__napi_poll+0x0/0x150) n=0xffff91c885bff000 poll_owner=-1 dev=0xffff91c881d4e000 npinfo=0x0 iperf2-1325 [002] d.Z1. 264246.626882: __xsk_rcv_zc_L7: (__xsk_rcv_zc+0x3b/0xc0) addr=0x1503100 len=0x42 xs=0xffff91c8bfe77000 fq=0xffff91c8c1a43f80 dev=0xffff91c881d4e000 iperf2-1325 [002] d.Z1. 264246.626883: __xsk_rcv_zc_L7: (__xsk_rcv_zc+0x42/0xc0) addr=0x1503100 len=0x42 xs=0xffff91c8bfe77000 fq=0xffff91c8c1a43f80 dev=0xffff91c881d4e000 iperf2-1325 [002] d.Z1. 264246.626884: xsk_flush: (__xsk_map_flush+0x32/0xb0) xs=0xffff91c8bfe77000 ```Here you can see that poll_owner=-1 meaning the lock was never acquired because npinfo is NULL. This means that the same veth rx queue can be
napi_polledfrom multiple CPU and nothing stops it from running concurrently. They all look like this, just most of the time there aren't concurrentnapi_pollsrunning for the same queue. They do however move around CPUs as I explained earlier.Quoted 7 lines
I'll note that I've ran with your suggested change of moving
xdp_do_flush()beforenapi_complete_done()all weekend and I have not reproduced the issue. I don't know if that truly means the issue is fixed by that change or not. I suspect it does fix the issue because it prevents thenapi_structfrom being scheduled again before the first poll has completed, andnap_schedule_prep()ensures that only one instance is ever running.
Quoted 7 lines
Thanks Shawn! Good news that the patch seems to fix the problem. To me,
napi_schedule_prep()makes sure that only one NAPI instance is running. Netpoll is an optional feature and I do not even have it compiled into my kernel. At least I do not have it defined in my .config and I cannot find any netpoll symbols with a readelf command. If netpoll is not used, I would suspect that npinfo == NULL. So to me, it is still a mystery why this is happening.
Oh I don't think it is a mystery anymore. The napi_complete_done() signals that this instance of of the napi_poll is complete. As you said nap_schedule_prep() checks to ensure that only one instance of napi_poll is running at a time, but we just indicated it was done with napi_complete_done(). This allows this CPU or more importantly any other CPU to reschedule napi polling for this receive queue, but we haven't called xdp_do_flush() yet so the flush can race. I'll note that the napi_schedule_prep()/__napi_schedule() in veth_poll really isn't the problem since it will schedule itself back on the same CPU. The problem is simply that another CPU is free to call napi_scheulde_prep()/__napi_schedule() in that window after napi_complete_done() and before xdp_do_flush(). The veth driver can schedule a napi_poll from the transmit path which is what starts the poll on a second CPU.
I was simply blinded by that stupid netpoll_poll_lock() but it is completely unnecessary.
Quoted 7 lines
To validate or disprove that there are two instances of the same napi running, could you please put a probe in
veth_poll()right afterveth_xdp_rcv()and one atxdp_do_flush()and dump the napi id in both cases? And run the original code with the bug :-). It would be good to know what exactly happens in the code between these points when this bug occurs. Maybe we could do this by enabling the trace buffer and dumping it when the bug occurs.
The two instances that race are definitely different napi_polls, the second one is scheduled from the veth transmit path.
Quoted 7 lines
Quoted 2 lines
If we think this is the correct fix I'll let it run for another day or two and prepare a patch.
There is one more advantage to this bug fix and that is performance. By calling
__xdp_map_flush()immediately after the receive function, user space can start to work on the packets quicker so performance will improve.
Yes definitely.
Thanks,
Shawn Bohrer
Quoted 119 lines · Shawn Bohrer
Quoted 109 lines · Magnus Karlsson
Quoted 100 lines · Shawn Bohrer
Quoted 54 lines · Magnus Karlsson
To summarize, we are expecting this ordering:
CPU 0
__xsk_rcv_zc()
CPU 0__xsk_map_flush()
CPU 2__xsk_rcv_zc()
CPU 2__xsk_map_flush()But we are seeing this order:
CPU 0
__xsk_rcv_zc()
CPU 2__xsk_rcv_zc()
CPU 0__xsk_map_flush()
CPU 2__xsk_map_flush()Here is the veth NAPI poll loop:
static int
veth_poll(struct napi_struct *napi, int budget){ struct veth_rq *rq = container_of(napi, struct veth_rq, xdp_napi); struct veth_stats stats = {}; struct veth_xdp_tx_bq bq; int done; bq.count = 0; xdp_set_return_frame_no_direct(); done = veth_xdp_rcv(rq, budget, &bq, &stats); if (done < budget && napi_complete_done(napi, done)) { /* Write rx_notify_masked before reading ptr_ring */ smp_store_mb(rq->rx_notify_masked, false); if (unlikely(!__ptr_ring_empty(&rq->xdp_ring))) { if (napi_schedule_prep(&rq->xdp_napi)) { WRITE_ONCE(rq->rx_notify_masked, true); __napi_schedule(&rq->xdp_napi); } } } if (stats.xdp_tx > 0) veth_xdp_flush(rq, &bq); if (stats.xdp_redirect > 0) xdp_do_flush(); xdp_clear_return_frame_no_direct(); return done; }Something I have never seen before is that there is
napi_complete_done()and a__napi_schedule()beforexdp_do_flush(). Let us check if this has something to do with it. So two suggestions to be executed separately:Quoted 6 lines
* Put a probe at the
__napi_schedule()above and check if it gets triggered before this problem
* Move the "if (stats.xdp_redirect > 0)xdp_do_flush();" to just before "if (done < budget &&napi_complete_done(napi, done)) {"This might provide us some hints on what is going on.
After staring at this code for way too long I finally made a breakthrough! I could not understand how this race could occur when
napi_poll()callsnetpoll_poll_lock(). Here isnetpoll_poll_lock():``` static inline void *netpoll_poll_lock(struct napi_struct *napi) { struct net_device *dev = napi->dev; if (dev && dev->npinfo) { int owner = smp_processor_id(); while (cmpxchg(&napi->poll_owner, -1, owner) != -1) cpu_relax(); return napi; } return NULL; } ```If dev or
dev->npinfoare NULL then it doesn't acquire a lock at all! Adding some more trace points I see:``` iperf2-1325 [002] ..s1. 264246.626880: __napi_poll: (__napi_poll+0x0/0x150) n=0xffff91c885bff000 poll_owner=-1 dev=0xffff91c881d4e000 npinfo=0x0 iperf2-1325 [002] d.Z1. 264246.626882: __xsk_rcv_zc_L7: (__xsk_rcv_zc+0x3b/0xc0) addr=0x1503100 len=0x42 xs=0xffff91c8bfe77000 fq=0xffff91c8c1a43f80 dev=0xffff91c881d4e000 iperf2-1325 [002] d.Z1. 264246.626883: __xsk_rcv_zc_L7: (__xsk_rcv_zc+0x42/0xc0) addr=0x1503100 len=0x42 xs=0xffff91c8bfe77000 fq=0xffff91c8c1a43f80 dev=0xffff91c881d4e000 iperf2-1325 [002] d.Z1. 264246.626884: xsk_flush: (__xsk_map_flush+0x32/0xb0) xs=0xffff91c8bfe77000 ```Here you can see that poll_owner=-1 meaning the lock was never acquired because npinfo is NULL. This means that the same veth rx queue can be
napi_polledfrom multiple CPU and nothing stops it from running concurrently. They all look like this, just most of the time there aren't concurrentnapi_pollsrunning for the same queue. They do however move around CPUs as I explained earlier.Quoted 7 lines
I'll note that I've ran with your suggested change of moving
xdp_do_flush()beforenapi_complete_done()all weekend and I have not reproduced the issue. I don't know if that truly means the issue is fixed by that change or not. I suspect it does fix the issue because it prevents thenapi_structfrom being scheduled again before the first poll has completed, andnap_schedule_prep()ensures that only one instance is ever running.Quoted 7 lines
Thanks Shawn! Good news that the patch seems to fix the problem. To me,
napi_schedule_prep()makes sure that only one NAPI instance is running. Netpoll is an optional feature and I do not even have it compiled into my kernel. At least I do not have it defined in my .config and I cannot find any netpoll symbols with a readelf command. If netpoll is not used, I would suspect that npinfo == NULL. So to me, it is still a mystery why this is happening.
Quoted 14 lines
Oh I don't think it is a mystery anymore. The
napi_complete_done()signals that this instance of of thenapi_pollis complete. As you saidnap_schedule_prep()checks to ensure that only one instance ofnapi_pollis running at a time, but we just indicated it was done withnapi_complete_done(). This allows this CPU or more importantly any other CPU to reschedule napi polling for this receive queue, but we haven't calledxdp_do_flush()yet so the flush can race. I'll note that the napi_schedule_prep()/__napi_schedule() inveth_pollreally isn't the problem since it will schedule itself back on the same CPU. The problem is simply that another CPU is free to call napi_scheulde_prep()/__napi_schedule() in that window afternapi_complete_done()and beforexdp_do_flush(). The veth driver can schedule anapi_pollfrom the transmit path which is what starts the poll on a second CPU.
Bingo! Would you like to prepare a patch or would you like me to do it? This has been broken since the support was put in veth in 4.19.
Quoted 26 lines
I was simply blinded by that stupid
netpoll_poll_lock()but it is completely unnecessary.Quoted 7 lines
To validate or disprove that there are two instances of the same napi running, could you please put a probe in
veth_poll()right afterveth_xdp_rcv()and one atxdp_do_flush()and dump the napi id in both cases? And run the original code with the bug :-). It would be good to know what exactly happens in the code between these points when this bug occurs. Maybe we could do this by enabling the trace buffer and dumping it when the bug occurs.The two instances that race are definitely different
napi_polls, the second one is scheduled from the veth transmit path.Quoted 7 lines
Quoted 2 lines
If we think this is the correct fix I'll let it run for another day or two and prepare a patch.
There is one more advantage to this bug fix and that is performance. By calling
__xdp_map_flush()immediately after the receive function, user space can start to work on the packets quicker so performance will improve.Yes definitely.
Thanks,
Shawn Bohrer
Quoted 135 lines · Magnus Karlsson
Quoted 119 lines · Shawn Bohrer
Quoted 109 lines · Magnus Karlsson
Quoted 100 lines · Shawn Bohrer
Quoted 54 lines · Magnus Karlsson
To summarize, we are expecting this ordering:
CPU 0
__xsk_rcv_zc()
CPU 0__xsk_map_flush()
CPU 2__xsk_rcv_zc()
CPU 2__xsk_map_flush()But we are seeing this order:
CPU 0
__xsk_rcv_zc()
CPU 2__xsk_rcv_zc()
CPU 0__xsk_map_flush()
CPU 2__xsk_map_flush()Here is the veth NAPI poll loop:
static int
veth_poll(struct napi_struct *napi, int budget){ struct veth_rq *rq = container_of(napi, struct veth_rq, xdp_napi); struct veth_stats stats = {}; struct veth_xdp_tx_bq bq; int done; bq.count = 0; xdp_set_return_frame_no_direct(); done = veth_xdp_rcv(rq, budget, &bq, &stats); if (done < budget && napi_complete_done(napi, done)) { /* Write rx_notify_masked before reading ptr_ring */ smp_store_mb(rq->rx_notify_masked, false); if (unlikely(!__ptr_ring_empty(&rq->xdp_ring))) { if (napi_schedule_prep(&rq->xdp_napi)) { WRITE_ONCE(rq->rx_notify_masked, true); __napi_schedule(&rq->xdp_napi); } } } if (stats.xdp_tx > 0) veth_xdp_flush(rq, &bq); if (stats.xdp_redirect > 0) xdp_do_flush(); xdp_clear_return_frame_no_direct(); return done; }Something I have never seen before is that there is
napi_complete_done()and a__napi_schedule()beforexdp_do_flush(). Let us check if this has something to do with it. So two suggestions to be executed separately:Quoted 6 lines
* Put a probe at the
__napi_schedule()above and check if it gets triggered before this problem
* Move the "if (stats.xdp_redirect > 0)xdp_do_flush();" to just before "if (done < budget &&napi_complete_done(napi, done)) {"This might provide us some hints on what is going on.
After staring at this code for way too long I finally made a breakthrough! I could not understand how this race could occur when
napi_poll()callsnetpoll_poll_lock(). Here isnetpoll_poll_lock():``` static inline void *netpoll_poll_lock(struct napi_struct *napi) { struct net_device *dev = napi->dev; if (dev && dev->npinfo) { int owner = smp_processor_id(); while (cmpxchg(&napi->poll_owner, -1, owner) != -1) cpu_relax(); return napi; } return NULL; } ```If dev or
dev->npinfoare NULL then it doesn't acquire a lock at all! Adding some more trace points I see:``` iperf2-1325 [002] ..s1. 264246.626880: __napi_poll: (__napi_poll+0x0/0x150) n=0xffff91c885bff000 poll_owner=-1 dev=0xffff91c881d4e000 npinfo=0x0 iperf2-1325 [002] d.Z1. 264246.626882: __xsk_rcv_zc_L7: (__xsk_rcv_zc+0x3b/0xc0) addr=0x1503100 len=0x42 xs=0xffff91c8bfe77000 fq=0xffff91c8c1a43f80 dev=0xffff91c881d4e000 iperf2-1325 [002] d.Z1. 264246.626883: __xsk_rcv_zc_L7: (__xsk_rcv_zc+0x42/0xc0) addr=0x1503100 len=0x42 xs=0xffff91c8bfe77000 fq=0xffff91c8c1a43f80 dev=0xffff91c881d4e000 iperf2-1325 [002] d.Z1. 264246.626884: xsk_flush: (__xsk_map_flush+0x32/0xb0) xs=0xffff91c8bfe77000 ```Here you can see that poll_owner=-1 meaning the lock was never acquired because npinfo is NULL. This means that the same veth rx queue can be
napi_polledfrom multiple CPU and nothing stops it from running concurrently. They all look like this, just most of the time there aren't concurrentnapi_pollsrunning for the same queue. They do however move around CPUs as I explained earlier.Quoted 7 lines
I'll note that I've ran with your suggested change of moving
xdp_do_flush()beforenapi_complete_done()all weekend and I have not reproduced the issue. I don't know if that truly means the issue is fixed by that change or not. I suspect it does fix the issue because it prevents thenapi_structfrom being scheduled again before the first poll has completed, andnap_schedule_prep()ensures that only one instance is ever running.Quoted 7 lines
Thanks Shawn! Good news that the patch seems to fix the problem. To me,
napi_schedule_prep()makes sure that only one NAPI instance is running. Netpoll is an optional feature and I do not even have it compiled into my kernel. At least I do not have it defined in my .config and I cannot find any netpoll symbols with a readelf command. If netpoll is not used, I would suspect that npinfo == NULL. So to me, it is still a mystery why this is happening.Quoted 14 lines
Oh I don't think it is a mystery anymore. The
napi_complete_done()signals that this instance of of thenapi_pollis complete. As you saidnap_schedule_prep()checks to ensure that only one instance ofnapi_pollis running at a time, but we just indicated it was done withnapi_complete_done(). This allows this CPU or more importantly any other CPU to reschedule napi polling for this receive queue, but we haven't calledxdp_do_flush()yet so the flush can race. I'll note that the napi_schedule_prep()/__napi_schedule() inveth_pollreally isn't the problem since it will schedule itself back on the same CPU. The problem is simply that another CPU is free to call napi_scheulde_prep()/__napi_schedule() in that window afternapi_complete_done()and beforexdp_do_flush(). The veth driver can schedule anapi_pollfrom the transmit path which is what starts the poll on a second CPU.
Quoted 2 lines
Bingo! Would you like to prepare a patch or would you like me to do it? This has been broken since the support was put in veth in 4.19.
I'll prepare the patch today. I feel like I've spent enough of my life chasing this down and appreciate your help!
Thanks,
Shawn Bohrer
PATCH 1/1 veth: Fix race with AF_XDP exposing old or uninitialized descriptors
In reply to Shawn BohrerWhen AF_XDP is used on on a veth interface the RX ring is updated in two steps. veth_xdp_rcv() removes packet descriptors from the FILL ring fills them and places them in the RX ring updating the cached_prod pointer. Later xdp_do_flush() syncs the RX ring prod pointer with the cached_prod pointer allowing user-space to see the recently filled in descriptors. The rings are intended to be SPSC, however the existing order in veth_poll allows the xdp_do_flush() to run concurrently with another CPU creating a race condition that allows user-space to see old or uninitialized descriptors in the RX ring. This bug has been observed in production systems.
To summarize, we are expecting this ordering:
CPU 0 __xsk_rcv_zc()
CPU 0 __xsk_map_flush()
CPU 2 __xsk_rcv_zc()
CPU 2 __xsk_map_flush()
But we are seeing this order:
CPU 0 __xsk_rcv_zc()
CPU 2 __xsk_rcv_zc()
CPU 0 __xsk_map_flush()
CPU 2 __xsk_map_flush()
This occurs because we rely on NAPI to ensure that only one napi_poll handler is running at a time for the given veth receive queue. napi_schedule_prep() will prevent multiple instances from getting scheduled. However calling napi_complete_done() signals that this napi_poll is complete and allows subsequent calls to napi_schedule_prep() and __napi_schedule() to succeed in scheduling a concurrent napi_poll before the xdp_do_flush() has been called. For the veth driver a concurrent call to napi_schedule_prep() and __napi_schedule() can occur on a different CPU because the veth xmit path can additionally schedule a napi_poll creating the race.
The fix as suggested by Magnus Karlsson, is to simply move the xdp_do_flush() call before napi_complete_done(). This syncs the producer ring pointers before another instance of napi_poll can be scheduled on another CPU. It will also slightly improve performance by moving the flush closer to when the descriptors were placed in the RX ring.
--- drivers/net/veth.c | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-)
drivers/net/veth.c +3 -2
@@ -974,6 +974,9 @@ static int veth_poll(struct napi_struct *napi, int budget)xdp_set_return_frame_no_direct (); done =veth_xdp_rcv (rq, budget, &bq, &stats); +if (stats.xdp_redirect >0 ) +xdp_do_flush (); +if (done < budget &&napi_complete_done (napi, done)) {/* Write rx_notify_masked before reading ptr_ring */ smp_store_mb (rq->rx_notify_masked, false);
@@ -987,8 +990,6 @@ static int veth_poll(struct napi_struct *napi, int budget)if (stats.xdp_tx >0 )veth_xdp_flush (rq, &bq); -if (stats.xdp_redirect >0 ) -xdp_do_flush ();xdp_clear_return_frame_no_direct ();return done;
Signature · 1 line
2.38.1
Jakub Kicinski
maintainerRe: [PATCH] veth: Fix race with AF_XDP exposing old or uninitialized descriptors
drivers/net/veth.c static int veth_poll(struct napi_struct *napi, int budget) · line 992
Quoted 2 lines · On Tue, 20 Dec 2022 12:59:03 -0600 Shawn Bohrer wrote:
drivers/net/veth.c +0
-0if (stats.xdp_tx >0 )veth_xdp_flush (rq, &bq);
This one does not need similar treatment, right? Only thing I could spot is potential races in updating stats, but that seems like a larger issue with veth xdp.
Quoted 2 lines
drivers/net/veth.c +0
-2-if (stats.xdp_redirect >0 ) -xdp_do_flush ();
Paolo Abeni
maintainerRe: [PATCH] veth: Fix race with AF_XDP exposing old or uninitialized descriptors
Quoted 7 lines · Jakub Kicinski
drivers/net/veth.c static int veth_poll(struct napi_struct *napi, int budget) · line 992
Quoted 2 lines · On Tue, 20 Dec 2022 12:59:03 -0600 Shawn Bohrer wrote:
drivers/net/veth.c +0
-0if (stats.xdp_tx >0 )veth_xdp_flush (rq, &bq);This one does not need similar treatment, right? Only thing I could spot is potential races in updating stats, but that seems like a larger issue with veth xdp.
Agreed, I think stats would deserve a separate patch (and this one looks correct).
Cheers,
Paolo
Paolo Abeni
maintainerRe: [PATCH] veth: Fix race with AF_XDP exposing old or uninitialized descriptors
In reply to Shawn BohrerQuoted 25 lines · Shawn Bohrer
When
AF_XDPis used on on a veth interface the RX ring is updated in two steps.veth_xdp_rcv()removes packet descriptors from the FILL ring fills them and places them in the RX ring updating thecached_prodpointer. Laterxdp_do_flush()syncs the RX ring prod pointer with thecached_prodpointer allowing user-space to see the recently filled in descriptors. The rings are intended to be SPSC, however the existing order inveth_pollallows thexdp_do_flush()to run concurrently with another CPU creating a race condition that allows user-space to see old or uninitialized descriptors in the RX ring. This bug has been observed in production systems.To summarize, we are expecting this ordering:
CPU 0
__xsk_rcv_zc()
CPU 0__xsk_map_flush()
CPU 2__xsk_rcv_zc()
CPU 2__xsk_map_flush()But we are seeing this order:
CPU 0
__xsk_rcv_zc()
CPU 2__xsk_rcv_zc()
CPU 0__xsk_map_flush()
CPU 2__xsk_map_flush()
Quoted 10 lines
This occurs because we rely on NAPI to ensure that only one
napi_pollhandler is running at a time for the given veth receive queue.napi_schedule_prep()will prevent multiple instances from getting scheduled. However callingnapi_complete_done()signals that thisnapi_pollis complete and allows subsequent calls tonapi_schedule_prep()and__napi_schedule()to succeed in scheduling a concurrentnapi_pollbefore thexdp_do_flush()has been called. For the veth driver a concurrent call tonapi_schedule_prep()and__napi_schedule()can occur on a different CPU because the veth xmit path can additionally schedule anapi_pollcreating the race.
The above looks like a generic problem that other drivers could hit. Perhaps it could be worthy updating the xdp_do_flush() doc text to explicitly mention it must be called before napi_complete_done().
(in a separate, net-next patch)
Thanks!
Paolo
Quoted 38 lines
The fix as suggested by Magnus Karlsson, is to simply move the
xdp_do_flush()call beforenapi_complete_done(). This syncs the producer ring pointers before another instance ofnapi_pollcan be scheduled on another CPU. It will also slightly improve performance by moving the flush closer to when the descriptors were placed in the RX ring.Fixes d1396004dd86 ("veth: Add XDP TX and REDIRECT")Suggested-by Magnus Karlsson <[email protected]>Signed-off-by Shawn Bohrer <[email protected]>--- drivers/net/veth.c | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-)drivers/net/veth.c +3
-2@@ -974,6 +974,9 @@ static int veth_poll(struct napi_struct *napi, int budget)xdp_set_return_frame_no_direct (); done =veth_xdp_rcv (rq, budget, &bq, &stats); +if (stats.xdp_redirect >0 ) +xdp_do_flush (); +if (done < budget &&napi_complete_done (napi, done)) {/* Write rx_notify_masked before reading ptr_ring */ smp_store_mb (rq->rx_notify_masked, false);@@ -987,8 +990,6 @@ static int veth_poll(struct napi_struct *napi, int budget)if (stats.xdp_tx >0 )veth_xdp_flush (rq, &bq); -if (stats.xdp_redirect >0 ) -xdp_do_flush ();xdp_clear_return_frame_no_direct ();return done;
Re: [PATCH] veth: Fix race with AF_XDP exposing old or uninitialized descriptors
Quoted 38 lines · Paolo Abeni
Quoted 25 lines · Shawn Bohrer
When
AF_XDPis used on on a veth interface the RX ring is updated in two steps.veth_xdp_rcv()removes packet descriptors from the FILL ring fills them and places them in the RX ring updating thecached_prodpointer. Laterxdp_do_flush()syncs the RX ring prod pointer with thecached_prodpointer allowing user-space to see the recently filled in descriptors. The rings are intended to be SPSC, however the existing order inveth_pollallows thexdp_do_flush()to run concurrently with another CPU creating a race condition that allows user-space to see old or uninitialized descriptors in the RX ring. This bug has been observed in production systems.To summarize, we are expecting this ordering:
CPU 0
__xsk_rcv_zc()
CPU 0__xsk_map_flush()
CPU 2__xsk_rcv_zc()
CPU 2__xsk_map_flush()But we are seeing this order:
CPU 0
__xsk_rcv_zc()
CPU 2__xsk_rcv_zc()
CPU 0__xsk_map_flush()
CPU 2__xsk_map_flush()Quoted 10 lines
This occurs because we rely on NAPI to ensure that only one
napi_pollhandler is running at a time for the given veth receive queue.napi_schedule_prep()will prevent multiple instances from getting scheduled. However callingnapi_complete_done()signals that thisnapi_pollis complete and allows subsequent calls tonapi_schedule_prep()and__napi_schedule()to succeed in scheduling a concurrentnapi_pollbefore thexdp_do_flush()has been called. For the veth driver a concurrent call tonapi_schedule_prep()and__napi_schedule()can occur on a different CPU because the veth xmit path can additionally schedule anapi_pollcreating the race.
Quoted 3 lines
The above looks like a generic problem that other drivers could hit. Perhaps it could be worthy updating the
xdp_do_flush()doc text to explicitly mention it must be called beforenapi_complete_done().
Good observation. I took a quick peek at this and it seems there are at least 5 more drivers that can call napi_complete_done() before xdp_do_flush():
drivers/net/ethernet/qlogic/qede/drivers/net/ethernet/freescale/dpaa2drivers/net/ethernet/freescale/dpaadrivers/net/ethernet/microchip/lan966xdrivers/net/virtio_net.c
The question is then if this race can occur on these five drivers. Dpaa2 has AF_XDP zero-copy support implemented, so it can suffer from this race as the Tx zero-copy path is basically just a napi_schedule() and it can be called/invoked from multiple processes at the same time. In regards to the others, I do not know.
Would it be prudent to just switch the order of xdp_do_flush() and napi_complete_done() in all these drivers, or would that be too defensive?
Quoted 45 lines
(in a separate, net-next patch)
Thanks!
Paolo
Quoted 38 lines
The fix as suggested by Magnus Karlsson, is to simply move the
xdp_do_flush()call beforenapi_complete_done(). This syncs the producer ring pointers before another instance ofnapi_pollcan be scheduled on another CPU. It will also slightly improve performance by moving the flush closer to when the descriptors were placed in the RX ring.Fixes d1396004dd86 ("veth: Add XDP TX and REDIRECT")Suggested-by Magnus Karlsson <[email protected]>Signed-off-by Shawn Bohrer <[email protected]>--- drivers/net/veth.c | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-)drivers/net/veth.c +3
-2@@ -974,6 +974,9 @@ static int veth_poll(struct napi_struct *napi, int budget)xdp_set_return_frame_no_direct (); done =veth_xdp_rcv (rq, budget, &bq, &stats); +if (stats.xdp_redirect >0 ) +xdp_do_flush (); +if (done < budget &&napi_complete_done (napi, done)) {/* Write rx_notify_masked before reading ptr_ring */ smp_store_mb (rq->rx_notify_masked, false);@@ -987,8 +990,6 @@ static int veth_poll(struct napi_struct *napi, int budget)if (stats.xdp_tx >0 )veth_xdp_flush (rq, &bq); -if (stats.xdp_redirect >0 ) -xdp_do_flush ();xdp_clear_return_frame_no_direct ();return done;
Re: [PATCH] veth: Fix race with AF_XDP exposing old or uninitialized descriptors
Quoted 59 lines · Magnus Karlsson
Quoted 38 lines · Paolo Abeni
Quoted 25 lines · Shawn Bohrer
When
AF_XDPis used on on a veth interface the RX ring is updated in two steps.veth_xdp_rcv()removes packet descriptors from the FILL ring fills them and places them in the RX ring updating thecached_prodpointer. Laterxdp_do_flush()syncs the RX ring prod pointer with thecached_prodpointer allowing user-space to see the recently filled in descriptors. The rings are intended to be SPSC, however the existing order inveth_pollallows thexdp_do_flush()to run concurrently with another CPU creating a race condition that allows user-space to see old or uninitialized descriptors in the RX ring. This bug has been observed in production systems.To summarize, we are expecting this ordering:
CPU 0
__xsk_rcv_zc()
CPU 0__xsk_map_flush()
CPU 2__xsk_rcv_zc()
CPU 2__xsk_map_flush()But we are seeing this order:
CPU 0
__xsk_rcv_zc()
CPU 2__xsk_rcv_zc()
CPU 0__xsk_map_flush()
CPU 2__xsk_map_flush()Quoted 10 lines
This occurs because we rely on NAPI to ensure that only one
napi_pollhandler is running at a time for the given veth receive queue.napi_schedule_prep()will prevent multiple instances from getting scheduled. However callingnapi_complete_done()signals that thisnapi_pollis complete and allows subsequent calls tonapi_schedule_prep()and__napi_schedule()to succeed in scheduling a concurrentnapi_pollbefore thexdp_do_flush()has been called. For the veth driver a concurrent call tonapi_schedule_prep()and__napi_schedule()can occur on a different CPU because the veth xmit path can additionally schedule anapi_pollcreating the race.Quoted 3 lines
The above looks like a generic problem that other drivers could hit. Perhaps it could be worthy updating the
xdp_do_flush()doc text to explicitly mention it must be called beforenapi_complete_done().Good observation. I took a quick peek at this and it seems there are at least 5 more drivers that can call
napi_complete_done()beforexdp_do_flush():
drivers/net/ethernet/qlogic/qede/drivers/net/ethernet/freescale/dpaa2drivers/net/ethernet/freescale/dpaadrivers/net/ethernet/microchip/lan966xdrivers/net/virtio_net.cThe question is then if this race can occur on these five drivers. Dpaa2 has
AF_XDPzero-copy support implemented, so it can suffer from this race as the Tx zero-copy path is basically just anapi_schedule()and it can be called/invoked from multiple processes at the same time. In regards to the others, I do not know.
Quoted 3 lines
Would it be prudent to just switch the order of
xdp_do_flush()andnapi_complete_done()in all these drivers, or would that be too defensive?
We rely on being inside a single NAPI instance trough to the xdp_do_flush() call for RCU protection of all in-kernel data structures as well[0]. Not sure if this leads to actual real-world bugs for the in-kernel path, but conceptually it's wrong at least. So yeah, I think we should definitely swap the order everywhere and document this!
-Toke
Re: [PATCH] veth: Fix race with AF_XDP exposing old or uninitialized descriptors
Quoted 66 lines · Toke Høiland-Jørgensen
Quoted 59 lines · Magnus Karlsson
Quoted 38 lines · Paolo Abeni
Quoted 25 lines · Shawn Bohrer
When
AF_XDPis used on on a veth interface the RX ring is updated in two steps.veth_xdp_rcv()removes packet descriptors from the FILL ring fills them and places them in the RX ring updating thecached_prodpointer. Laterxdp_do_flush()syncs the RX ring prod pointer with thecached_prodpointer allowing user-space to see the recently filled in descriptors. The rings are intended to be SPSC, however the existing order inveth_pollallows thexdp_do_flush()to run concurrently with another CPU creating a race condition that allows user-space to see old or uninitialized descriptors in the RX ring. This bug has been observed in production systems.To summarize, we are expecting this ordering:
CPU 0
__xsk_rcv_zc()
CPU 0__xsk_map_flush()
CPU 2__xsk_rcv_zc()
CPU 2__xsk_map_flush()But we are seeing this order:
CPU 0
__xsk_rcv_zc()
CPU 2__xsk_rcv_zc()
CPU 0__xsk_map_flush()
CPU 2__xsk_map_flush()Quoted 10 lines
This occurs because we rely on NAPI to ensure that only one
napi_pollhandler is running at a time for the given veth receive queue.napi_schedule_prep()will prevent multiple instances from getting scheduled. However callingnapi_complete_done()signals that thisnapi_pollis complete and allows subsequent calls tonapi_schedule_prep()and__napi_schedule()to succeed in scheduling a concurrentnapi_pollbefore thexdp_do_flush()has been called. For the veth driver a concurrent call tonapi_schedule_prep()and__napi_schedule()can occur on a different CPU because the veth xmit path can additionally schedule anapi_pollcreating the race.Quoted 3 lines
The above looks like a generic problem that other drivers could hit. Perhaps it could be worthy updating the
xdp_do_flush()doc text to explicitly mention it must be called beforenapi_complete_done().Good observation. I took a quick peek at this and it seems there are at least 5 more drivers that can call
napi_complete_done()beforexdp_do_flush():
drivers/net/ethernet/qlogic/qede/drivers/net/ethernet/freescale/dpaa2drivers/net/ethernet/freescale/dpaadrivers/net/ethernet/microchip/lan966xdrivers/net/virtio_net.cThe question is then if this race can occur on these five drivers. Dpaa2 has
AF_XDPzero-copy support implemented, so it can suffer from this race as the Tx zero-copy path is basically just anapi_schedule()and it can be called/invoked from multiple processes at the same time. In regards to the others, I do not know.Quoted 3 lines
Would it be prudent to just switch the order of
xdp_do_flush()andnapi_complete_done()in all these drivers, or would that be too defensive?
Quoted 5 lines
We rely on being inside a single NAPI instance trough to the
xdp_do_flush()call for RCU protection of all in-kernel data structures as well[0]. Not sure if this leads to actual real-world bugs for the in-kernel path, but conceptually it's wrong at least. So yeah, I think we should definitely swap the order everywhere and document this!
OK, let me take a stab at it. For at least the first four, it will be compilation tested only from my side since I do not own any of those SoCs/cards. Will need the help of those maintainers for sure.
Quoted 4 lines
-Toke
Re: [PATCH] veth: Fix race with AF_XDP exposing old or uninitialized descriptors
Quoted 73 lines · Magnus Karlsson
Quoted 66 lines · Toke Høiland-Jørgensen
Quoted 59 lines · Magnus Karlsson
Quoted 38 lines · Paolo Abeni
Quoted 25 lines · Shawn Bohrer
When
AF_XDPis used on on a veth interface the RX ring is updated in two steps.veth_xdp_rcv()removes packet descriptors from the FILL ring fills them and places them in the RX ring updating thecached_prodpointer. Laterxdp_do_flush()syncs the RX ring prod pointer with thecached_prodpointer allowing user-space to see the recently filled in descriptors. The rings are intended to be SPSC, however the existing order inveth_pollallows thexdp_do_flush()to run concurrently with another CPU creating a race condition that allows user-space to see old or uninitialized descriptors in the RX ring. This bug has been observed in production systems.To summarize, we are expecting this ordering:
CPU 0
__xsk_rcv_zc()
CPU 0__xsk_map_flush()
CPU 2__xsk_rcv_zc()
CPU 2__xsk_map_flush()But we are seeing this order:
CPU 0
__xsk_rcv_zc()
CPU 2__xsk_rcv_zc()
CPU 0__xsk_map_flush()
CPU 2__xsk_map_flush()Quoted 10 lines
This occurs because we rely on NAPI to ensure that only one
napi_pollhandler is running at a time for the given veth receive queue.napi_schedule_prep()will prevent multiple instances from getting scheduled. However callingnapi_complete_done()signals that thisnapi_pollis complete and allows subsequent calls tonapi_schedule_prep()and__napi_schedule()to succeed in scheduling a concurrentnapi_pollbefore thexdp_do_flush()has been called. For the veth driver a concurrent call tonapi_schedule_prep()and__napi_schedule()can occur on a different CPU because the veth xmit path can additionally schedule anapi_pollcreating the race.Quoted 3 lines
The above looks like a generic problem that other drivers could hit. Perhaps it could be worthy updating the
xdp_do_flush()doc text to explicitly mention it must be called beforenapi_complete_done().Good observation. I took a quick peek at this and it seems there are at least 5 more drivers that can call
napi_complete_done()beforexdp_do_flush():
drivers/net/ethernet/qlogic/qede/drivers/net/ethernet/freescale/dpaa2drivers/net/ethernet/freescale/dpaadrivers/net/ethernet/microchip/lan966xdrivers/net/virtio_net.cThe question is then if this race can occur on these five drivers. Dpaa2 has
AF_XDPzero-copy support implemented, so it can suffer from this race as the Tx zero-copy path is basically just anapi_schedule()and it can be called/invoked from multiple processes at the same time. In regards to the others, I do not know.Quoted 3 lines
Would it be prudent to just switch the order of
xdp_do_flush()andnapi_complete_done()in all these drivers, or would that be too defensive?Quoted 5 lines
We rely on being inside a single NAPI instance trough to the
xdp_do_flush()call for RCU protection of all in-kernel data structures as well[0]. Not sure if this leads to actual real-world bugs for the in-kernel path, but conceptually it's wrong at least. So yeah, I think we should definitely swap the order everywhere and document this!
Quoted 3 lines
OK, let me take a stab at it. For at least the first four, it will be compilation tested only from my side since I do not own any of those SoCs/cards. Will need the help of those maintainers for sure.
Sounds good, thanks! :)
-Toke
Re: [PATCH] veth: Fix race with AF_XDP exposing old or uninitialized descriptors
In reply to Shawn BohrerThis patch was applied to netdev/net.git (master) by Paolo Abeni <[email protected]>:
Hello:
This patch was applied to netdev/net.git (master)
by Paolo Abeni <[email protected]>:
Quoted 11 lines · On Tue, 20 Dec 2022 12:59:03 -0600 you wrote:
When
AF_XDPis used on on a veth interface the RX ring is updated in two steps.veth_xdp_rcv()removes packet descriptors from the FILL ring fills them and places them in the RX ring updating thecached_prodpointer. Laterxdp_do_flush()syncs the RX ring prod pointer with thecached_prodpointer allowing user-space to see the recently filled in descriptors. The rings are intended to be SPSC, however the existing order inveth_pollallows thexdp_do_flush()to run concurrently with another CPU creating a race condition that allows user-space to see old or uninitialized descriptors in the RX ring. This bug has been observed in production systems.
Quoted 1 line
[...]
Here is the summary with links:
- veth: Fix race with AF_XDP exposing old or uninitialized descriptors
https://git.kernel.org/netdev/net/c/fa349e396e48
You are awesome, thank you!
Signature · 2 lines
Deet-doot-dot, I am a bot. https://korg.docs.kernel.org/patchwork/pwbot.html