Skip to contentSkip to conversation

Pass correct tun device's RX queue number to XDP program

PATCH In review Gilberto Bertin 2 participants 3 messages Apr 10, 2020 – Apr 13, 2020

Subsystems (3)
lore
Share & tools
lore
#0 lore raw

cover Pass correct tun device's RX queue number to XDP program

Hi,

I'm trying to run XDP on top of a tun device in multi queue mode as that allows me to easily play on my laptop with an interface that supports multiple queues.

While testing it I found an issue: regardless of the the tun fd (i.e. RX queue) I'm writing packets to, the XDP program thinks all packets are coming from RX queue 0.

I'd like to propose a fix for that (see next mail) but let me first share some context.

Here's what I'm using to setup a tun device with multiple RX/TX queues and write some test packets into all of its RX queues:

        struct ifreq ifr = {
                .ifr_flags = IFF_TAP | IFF_NO_PI | IFF_MULTI_QUEUE, .ifr_name = "mytun0"
        };

        int fds[4];
        for (int i = 0; i < 4; i++) {
                if ((fds[i] = open("/dev/net/tun", O_RDWR)) < 0)
                        return -1;
                if (ioctl(fds[i], TUNSETIFF, (void *)&ifr))
                        return -1;
        }

	while (1) {
		for (int i = 0 ; i < 4; i++)
			write(fds[i], "\x12\x34\x56\x78\x90\xab\x12\x34\x56\x67\x90\xab\x08\x00", 14);
		sleep(2);
	}

Next I reproduced the problem with this XDP program attached to the tun device created with the previous code snippet:

        $ cat xdp_kern.c
        #include <linux/bpf.h>
        #include <bpf_helpers.h>
        SEC("prog")
        unsigned int xdp_prog(struct xdp_md *xdp) {
                bpf_printk("rx queue: %d\n", xdp->rx_queue_index);
                return XDP_PASS;
        }

        char _license[] SEC("license") = "GPL";

        $ sudo ip link set dev mytun0 xdpdrv obj ./xdp_kern.o

        $ sudo bpftool prog tracelog
                   a.out-22189 [003] .... 14444.594609: 0: rx queue: 0
                   a.out-22189 [003] .... 14444.594645: 0: rx queue: 0
                   a.out-22189 [003] .... 14444.594653: 0: rx queue: 0
                   a.out-22189 [003] .... 14444.594661: 0: rx queue: 0

Here the XDP program is reporting all packets coming from RX queue 0 when they should have been coming from RX queues 0..3.

After looking into the tun driver I ended up writing this stap script which reveals the issue:

        $ cat rx_queue.stap
        probe kernel.function("tun_get_user") {
                printf("tun_get_user(): tfile->xdp_rxq.queue_index: %d\n", $tfile->xdp_rxq->queue_index);
        }
        probe kernel.function("netif_receive_generic_xdp") {
                printf("netif_receive_generic_xdp(): skb->queue_mapping: %d\n", $skb->queue_mapping);
                printf("\n");
        }

        $ sudo stap ./rx_queue.stap
        tun_get_user(): tfile->xdp_rxq.queue_index: 0
        netif_receive_generic_xdp(): skb->queue_mapping: 0

        tun_get_user(): tfile->xdp_rxq.queue_index: 1
        netif_receive_generic_xdp(): skb->queue_mapping: 0

        tun_get_user(): tfile->xdp_rxq.queue_index: 2
        netif_receive_generic_xdp(): skb->queue_mapping: 0

        tun_get_user(): tfile->xdp_rxq.queue_index: 3
        netif_receive_generic_xdp(): skb->queue_mapping: 0

It looks like the tun driver (specifically tun_get_user()) is not passing correctly the information about the RX queue to do_xdp_generic().

Setting the queue_mapping field in the skb with skb_record_rx_queue() seems to fix the issue.

Same test with the patched kernel:

        sudo bpftool prog tracelog
                   a.out-791   [004] ....  4826.499356: 0: rx queue: 0
                   a.out-791   [004] ....  4826.499532: 0: rx queue: 1
                   a.out-791   [004] ....  4826.499541: 0: rx queue: 2
                   a.out-791   [004] ....  4826.499549: 0: rx queue: 3

While at it, I also fixed tun_xdp_one(), which was calling skb_record_rx_queue() after invoking do_xdp_generic().

Gilberto Bertin (1):
net: tun: record RX queue in skb before do_xdp_generic()

 drivers/net/tun.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
Signature · 1 line
2.20.1
#1 lore raw

PATCH 1/1 net: tun: record RX queue in skb before do_xdp_generic()

This allows netif_receive_generic_xdp() to correctly determine the RX queue from which the skb is coming, so that the context passed to the XDP program will contain the correct RX queue index.

---
 drivers/net/tun.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
drivers/net/tun.c +2 -1
index 228fe449dc6d..42b5d8740987 100644
@@ -1886,6 +1886,7 @@ static ssize_t tun_get_user(struct tun_struct *tun, struct tun_file *tfile,
 
 	skb_reset_network_header(skb);
 	skb_probe_transport_header(skb);
+	skb_record_rx_queue(skb, tfile->queue_index);
 
 	if (skb_xdp) {
 		struct bpf_prog *xdp_prog;
@@ -2457,6 +2458,7 @@ static int tun_xdp_one(struct tun_struct *tun,
 	skb->protocol = eth_type_trans(skb, tun->dev);
 	skb_reset_network_header(skb);
 	skb_probe_transport_header(skb);
+	skb_record_rx_queue(skb, tfile->queue_index);
 
 	if (skb_xdp) {
 		err = do_xdp_generic(xdp_prog, skb);
@@ -2468,7 +2470,6 @@ static int tun_xdp_one(struct tun_struct *tun,
 	    !tfile->detached)
 		rxhash = __skb_get_hash_symmetric(skb);
 
-	skb_record_rx_queue(skb, tfile->queue_index);
 	netif_receive_skb(skb);
 
 	/* No need for get_cpu_ptr() here since this function is
Signature · 1 line
2.20.1
maintainer
#2 lore raw

Re: [PATCH 1/1] net: tun: record RX queue in skb before do_xdp_generic()

From: Gilberto Bertin <[email protected]>
Date: Fri, 10 Apr 2020 18:20:59 +0200

Quoted 5 lines

This allows netif_receive_generic_xdp() to correctly determine the RX queue from which the skb is coming, so that the context passed to the XDP program will contain the correct RX queue index.

Applied and queued up for -stable, thanks.