Pass correct tun device's RX queue number to XDP program
Subsystems (3)
cover Pass correct tun device's RX queue number to XDP program
Hi,
I'm trying to run XDP on top of a tun device in multi queue mode as that allows me to easily play on my laptop with an interface that supports multiple queues.
While testing it I found an issue: regardless of the the tun fd (i.e. RX queue) I'm writing packets to, the XDP program thinks all packets are coming from RX queue 0.
I'd like to propose a fix for that (see next mail) but let me first share some context.
Here's what I'm using to setup a tun device with multiple RX/TX queues and write some test packets into all of its RX queues:
struct ifreq ifr = {
.ifr_flags = IFF_TAP | IFF_NO_PI | IFF_MULTI_QUEUE, .ifr_name = "mytun0"
};
int fds[4];
for (int i = 0; i < 4; i++) {
if ((fds[i] = open("/dev/net/tun", O_RDWR)) < 0)
return -1;
if (ioctl(fds[i], TUNSETIFF, (void *)&ifr))
return -1;
}
while (1) {
for (int i = 0 ; i < 4; i++)
write(fds[i], "\x12\x34\x56\x78\x90\xab\x12\x34\x56\x67\x90\xab\x08\x00", 14);
sleep(2);
}Next I reproduced the problem with this XDP program attached to the tun device created with the previous code snippet:
$ cat xdp_kern.c
#include <linux/bpf.h>
#include <bpf_helpers.h> SEC("prog")
unsigned int xdp_prog(struct xdp_md *xdp) {
bpf_printk("rx queue: %d\n", xdp->rx_queue_index);
return XDP_PASS;
}
char _license[] SEC("license") = "GPL";
$ sudo ip link set dev mytun0 xdpdrv obj ./xdp_kern.o
$ sudo bpftool prog tracelog
a.out-22189 [003] .... 14444.594609: 0: rx queue: 0
a.out-22189 [003] .... 14444.594645: 0: rx queue: 0
a.out-22189 [003] .... 14444.594653: 0: rx queue: 0
a.out-22189 [003] .... 14444.594661: 0: rx queue: 0Here the XDP program is reporting all packets coming from RX queue 0 when they should have been coming from RX queues 0..3.
After looking into the tun driver I ended up writing this stap script which reveals the issue:
$ cat rx_queue.stap
probe kernel.function("tun_get_user") {
printf("tun_get_user(): tfile->xdp_rxq.queue_index: %d\n", $tfile->xdp_rxq->queue_index);
} probe kernel.function("netif_receive_generic_xdp") {
printf("netif_receive_generic_xdp(): skb->queue_mapping: %d\n", $skb->queue_mapping);
printf("\n");
}
$ sudo stap ./rx_queue.stap
tun_get_user(): tfile->xdp_rxq.queue_index: 0
netif_receive_generic_xdp(): skb->queue_mapping: 0
tun_get_user(): tfile->xdp_rxq.queue_index: 1
netif_receive_generic_xdp(): skb->queue_mapping: 0
tun_get_user(): tfile->xdp_rxq.queue_index: 2
netif_receive_generic_xdp(): skb->queue_mapping: 0
tun_get_user(): tfile->xdp_rxq.queue_index: 3
netif_receive_generic_xdp(): skb->queue_mapping: 0It looks like the tun driver (specifically tun_get_user()) is not passing correctly the information about the RX queue to do_xdp_generic().
Setting the queue_mapping field in the skb with skb_record_rx_queue() seems to fix the issue.
Same test with the patched kernel:
sudo bpftool prog tracelog
a.out-791 [004] .... 4826.499356: 0: rx queue: 0
a.out-791 [004] .... 4826.499532: 0: rx queue: 1
a.out-791 [004] .... 4826.499541: 0: rx queue: 2
a.out-791 [004] .... 4826.499549: 0: rx queue: 3While at it, I also fixed tun_xdp_one(), which was calling skb_record_rx_queue() after invoking do_xdp_generic().
Gilberto Bertin (1):
net: tun: record RX queue in skb before do_xdp_generic()
drivers/net/tun.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)
Signature · 1 line
2.20.1
PATCH 1/1 net: tun: record RX queue in skb before do_xdp_generic()
This allows netif_receive_generic_xdp() to correctly determine the RX queue from which the skb is coming, so that the context passed to the XDP program will contain the correct RX queue index.
--- drivers/net/tun.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)
drivers/net/tun.c +2 -1
@@ -1886,6 +1886,7 @@ static ssize_t tun_get_user(struct tun_struct *tun, struct tun_file *tfile,skb_reset_network_header (skb);skb_probe_transport_header (skb); +skb_record_rx_queue (skb, tfile->queue_index);if (skb_xdp) {struct bpf_prog *xdp_prog;
@@ -2457,6 +2458,7 @@ static int tun_xdp_one(struct tun_struct *tun, skb->protocol =eth_type_trans (skb, tun->dev);skb_reset_network_header (skb);skb_probe_transport_header (skb); +skb_record_rx_queue (skb, tfile->queue_index);if (skb_xdp) { err =do_xdp_generic (xdp_prog, skb);
@@ -2468,7 +2470,6 @@ static int tun_xdp_one(struct tun_struct *tun, !tfile->detached) rxhash =__skb_get_hash_symmetric (skb); -skb_record_rx_queue (skb, tfile->queue_index);netif_receive_skb (skb);/* No need for get_cpu_ptr() here since this function is
Signature · 1 line
2.20.1
David Miller
maintainerRe: [PATCH 1/1] net: tun: record RX queue in skb before do_xdp_generic()
From: Gilberto Bertin <[email protected]>
Date: Fri, 10 Apr 2020 18:20:59 +0200
Quoted 5 lines
This allows
netif_receive_generic_xdp()to correctly determine the RX queue from which the skb is coming, so that the context passed to the XDP program will contain the correct RX queue index.Signed-off-by Gilberto Bertin <[email protected]>
Applied and queued up for -stable, thanks.