BPF metadata for direct access

Implement support for transferring XDP meta data into skb for nfp driver; before calling into the program, xdp.data_meta points to xdp.data, where on program return with pass verdict, we call into skb_metadata_set().

Signed-off-by Daniel Borkmann <[email protected]>

Acked-by Alexei Starovoitov <[email protected]>

Acked-by John Fastabend <[email protected]>

This work enables generic transfer of metadata from XDP into skb. The basic idea is that we can make use of the fact that the resulting skb must be linear and already comes with a larger headroom for supporting bpf_xdp_adjust_head(), which mangles xdp->data. Here, we base our work on a similar principle and introduce a small helper bpf_xdp_adjust_meta() for adjusting a new pointer called xdp->data_meta. Thus, the packet has a flexible and programmable room for meta data, followed by the actual packet data. struct xdp_buff is therefore laid out that we first point to data_hard_start, then data_meta directly prepended to data followed by data_end marking the end of packet. bpf_xdp_adjust_head() takes into account whether we have meta data already prepended and if so, memmove()s this along with the given offset provided there's enough room.

xdp->data_meta is optional and programs are not required to use it. The rationale is that when we process the packet in XDP (e.g. as DoS filter), we can push further meta data along with it for the XDP_PASS case, and give the guarantee that a clsact ingress BPF program on the same device can pick this up for further post-processing. Since we work with skb there, we can also set skb->mark, skb->priority or other skb meta data out of BPF, thus having this scratch space generic and programmable allows for more flexibility than defining a direct 1:1 transfer of potentially new XDP members into skb (it's also more efficient as we don't need to initialize/handle each of such new members). The facility also works together with GRO aggregation. The scratch space at the head of the packet can be multiple of 4 byte up to 32 byte large. Drivers not yet supporting xdp->data_meta can simply be set up with xdp->data_meta as xdp->data + 1 as bpf_xdp_adjust_meta() will detect this and bail out, such that the subsequent match against xdp->data for later access is guaranteed to fail.

The verifier treats xdp->data_meta/xdp->data the same way as we treat xdp->data/xdp->data_end pointer comparisons. The requirement for doing the compare against xdp->data is that it hasn't been modified from it's original address we got from ctx access. It may have a range marking already from prior successful xdp->data/xdp->data_end pointer comparisons though.

Signed-off-by Daniel Borkmann <[email protected]>

Acked-by Alexei Starovoitov <[email protected]>

Acked-by John Fastabend <[email protected]>

---
 drivers/net/ethernet/broadcom/bnxt/bnxt_xdp.c      |   1 +
 drivers/net/ethernet/cavium/thunder/nicvf_main.c   |   1 +
 drivers/net/ethernet/intel/i40e/i40e_txrx.c        |   1 +
 drivers/net/ethernet/intel/ixgbe/ixgbe_main.c      |   1 +
 drivers/net/ethernet/mellanox/mlx4/en_rx.c         |   1 +
 drivers/net/ethernet/mellanox/mlx5/core/en_rx.c    |   1 +
 .../net/ethernet/netronome/nfp/nfp_net_common.c    |   1 +
 drivers/net/ethernet/qlogic/qede/qede_fp.c         |   1 +
 drivers/net/tun.c                                  |   1 +
 drivers/net/virtio_net.c                           |   2 +
 include/linux/bpf.h                                |   1 +
 include/linux/filter.h                             |  21 +++-
 include/linux/skbuff.h                             |  68 +++++++++++-
 include/uapi/linux/bpf.h                           |  13 ++-
 kernel/bpf/verifier.c                              | 114 ++++++++++++++++-----
 net/bpf/test_run.c                                 |   1 +
 net/core/dev.c                                     |  31 +++++-
 net/core/filter.c                                  |  77 +++++++++++++-
 net/core/skbuff.c                                  |   2 +
 19 files changed, 297 insertions(+), 42 deletions(-)

First, thanks for this detailed description. It was helpful to read along with the patches.

My only concern about this area being generic is that you are now in a state where any bpf program must know about all the bpf programs in the receive pipeline before it can properly parse what is stored in the meta-data and add it to an skb (or perform any other action). Especially if each program adds it's own meta-data along the way.

Maybe this isn't a big concern based on the number of users of this today, but it just starts to seem like a concern as there are these hints being passed between layers that are challenging to track due to a lack of a standard format for passing data between.

The main reason I bring this up is that Michael and I had discussed and designed a way for drivers to communicate between each other that rx resources could be freed after a tx completion on an XDP_REDIRECT action. Much like this code, it involved adding an new element to struct xdp_md that could point to the important information. Now that there is a generic way to handle this, it would seem nice to be able to leverage it, but I'm not sure how reliable this meta-data area would be without the ability to mark it in some manner.

For additional background, the minimum amount of data needed in the case Michael and I were discussing was really 2 words. One to serve as a pointer to an rx_ring structure and one to have a counter to the rx producer entry. This data could be acessed by the driver processing the tx completions and callback to the driver that received the frame off the wire to perform any needed processing. (For those curious this would also require a new callback/netdev op to act on this data stored in the XDP buffer.)

On 09/25/2017 08:10 PM, Andy Gospodarek wrote:
[...]

Quoted 9 lines

First, thanks for this detailed description.  It was helpful to read along with the patches.

My only concern about this area being generic is that you are now in a state where any bpf program must know about all the bpf programs in the receive pipeline before it can properly parse what is stored in the meta-data and add it to an skb (or perform any other action). Especially if each program adds it's own meta-data along the way.

Quoted 4 lines

Maybe this isn't a big concern based on the number of users of this today, but it just starts to seem like a concern as there are these hints being passed between layers that are challenging to track due to a lack of a standard format for passing data between.

Btw, we do have similar kind of programmable scratch buffer also today wrt skb cb[] that you can program from tc side, the perf ring buffer, which doesn't have any fixed layout for the slots, or a per-cpu map where you can transfer data between tail calls for example, then tail calls themselves that need to coordinate, or simply mangling of packets itself if you will, but more below to your use case ...

Quoted 9 lines

The main reason I bring this up is that Michael and I had discussed and designed a way for drivers to communicate between each other that rx resources could be freed after a tx completion on an XDP_REDIRECT action.  Much like this code, it involved adding an new element to struct xdp_md that could point to the important information.  Now that there is a generic way to handle this, it would seem nice to be able to leverage it, but I'm not sure how reliable this meta-data area would be without the ability to mark it in some manner.

Quoted 9 lines

For additional background, the minimum amount of data needed in the case Michael and I were discussing was really 2 words.  One to serve as a pointer to an rx_ring structure and one to have a counter to the rx producer entry.  This data could be acessed by the driver processing the tx completions and callback to the driver that received the frame off the wire
to perform any needed processing.  (For those curious this would also require a
new callback/netdev op to act on this data stored in the XDP buffer.)

What you describe above doesn't seem to be fitting to the use-case of this set, meaning the area here is fully programmable out of the BPF program, the infrastructure you're describing is some sort of means of communication between drivers for the XDP_REDIRECT, and should be outside of the control of the BPF program to mangle.

You could probably reuse the base infra here and make a part of that inaccessible for the program with some sort of a fixed layout, but I haven't seen your code yet to be able to fully judge. Intention here is to allow for programmability within the BPF prog in a generic way, such that based on the use-case it can be populated in specific ways and propagated to the skb w/o having to define a fixed layout and bloat xdp_buff all the way to an skb while still retaining all the flexibility.

Thanks,
Daniel

On 09/25/2017 08:10 PM, Andy Gospodarek wrote:
[...]

Quoted 9 lines

First, thanks for this detailed description. It was helpful to read along with the patches.

My only concern about this area being generic is that you are now in a state where any bpf program must know about all the bpf programs in the receive pipeline before it can properly parse what is stored in the meta-data and add it to an skb (or perform any other action). Especially if each program adds it's own meta-data along the way.

Quoted 4 lines

Maybe this isn't a big concern based on the number of users of this today, but it just starts to seem like a concern as there are these hints being passed between layers that are challenging to track due to a lack of a standard format for passing data between.

Btw, we do have similar kind of programmable scratch buffer also today wrt skb cb[] that you can program from tc side, the perf ring buffer, which doesn't have any fixed layout for the slots, or a per-cpu map where you can transfer data between tail calls for example, then tail calls themselves that need to coordinate, or simply mangling of packets itself if you will, but more below to your use case ...

Quoted 9 lines

The main reason I bring this up is that Michael and I had discussed and designed a way for drivers to communicate between each other that rx resources could be freed after a tx completion on an XDP_REDIRECT action. Much like this code, it involved adding an new element to struct xdp_md that could point to the important information. Now that there is a generic way to handle this, it would seem nice to be able to leverage it, but I'm not sure how reliable this meta-data area would be without the ability to mark it in some manner.

Quoted 7 lines

For additional background, the minimum amount of data needed in the case Michael and I were discussing was really 2 words. One to serve as a pointer to an rx_ring structure and one to have a counter to the rx producer entry. This data could be acessed by the driver processing the tx completions and callback to the driver that received the frame off the wire to perform any needed processing. (For those curious this would also require a new callback/netdev op to act on this data stored in the XDP buffer.)

What you describe above doesn't seem to be fitting to the use-case of this set, meaning the area here is fully programmable out of the BPF program, the infrastructure you're describing is some sort of means of communication between drivers for the XDP_REDIRECT, and should be outside of the control of the BPF program to mangle.

You could probably reuse the base infra here and make a part of that inaccessible for the program with some sort of a fixed layout, but I haven't seen your code yet to be able to fully judge. Intention here is to allow for programmability within the BPF prog in a generic way, such that based on the use-case it can be populated in specific ways and propagated to the skb w/o having to define a fixed layout and bloat xdp_buff all the way to an skb while still retaining all the flexibility.

This work enables generic transfer of metadata from XDP into skb. The basic idea is that we can make use of the fact that the resulting skb must be linear and already comes with a larger headroom for supporting bpf_xdp_adjust_head(), which mangles xdp->data. Here, we base our work on a similar principle and introduce a small helper bpf_xdp_adjust_meta() for adjusting a new pointer called xdp->data_meta. Thus, the packet has a flexible and programmable room for meta data, followed by the actual packet data. struct xdp_buff is therefore laid out that we first point to data_hard_start, then data_meta directly prepended to data followed by data_end marking the end of packet. bpf_xdp_adjust_head() takes into account whether we have meta data already prepended and if so, memmove()s this along with the given offset provided there's enough room.

[...] The scratch space at the head
of the packet can be multiple of 4 byte up to 32 byte large. Drivers not yet supporting xdp->data_meta can simply be set up with xdp->data_meta as xdp->data + 1 as bpf_xdp_adjust_meta() will detect this and bail out, such that the subsequent match against xdp->data for later access is guaranteed to fail.

I'm currently implementing a cpumap type, that transfers raw XDP frames to another CPU, and the SKB is allocated on the remote CPU. (It actually works extremely well).

For transferring info I need, I'm currently using xdp->data_hard_start (the top/start of the xdp page). Which should be compatible with your approach, right?

The info I need:

  struct xdp_pkt {
	void *data;
	u16 len;
	u16 headroom;
	struct net_device *dev_rx;

};

When I enqueue the xdp packet I do the following:

int cpu_map_enqueue(struct bpf_cpu_map_entry *rcpu, struct xdp_buff *xdp,
struct net_device *dev_rx)

  {
	struct xdp_pkt *xdp_pkt;
	int headroom;

	/* Convert xdp_buff to xdp_pkt */
	headroom = xdp->data - xdp->data_hard_start;
	if (headroom < sizeof(*xdp_pkt))
		return -EOVERFLOW;
	xdp_pkt = xdp->data_hard_start;
	xdp_pkt->data = xdp->data;
	xdp_pkt->len  = xdp->data_end - xdp->data;
	xdp_pkt->headroom = headroom - sizeof(*xdp_pkt);

	/* Info needed when constructing SKB on remote CPU */
	xdp_pkt->dev_rx = dev_rx;

	bq_enqueue(rcpu, xdp_pkt);
	return 0;

}

On the remote CPU dequeueing the packet, I'm doing the following. As you can see I'm still lacking some meta-data, that would be nice to also transfer. Could I use your infrastructure for that?

This work enables generic transfer of metadata from XDP into skb, meaning the packet has a flexible and programmable room for meta data, which can later be used by BPF to set various skb members when passing up the stack. For details, please see second patch. Support has been implemented and tested with two drivers, and should be straight forward to add to other drivers as well which properly support head adjustment already.

From: Daniel Borkmann <[email protected]>
Date: Mon, 25 Sep 2017 02:25:49 +0200

Quoted 7 lines

This work enables generic transfer of metadata from XDP into skb, meaning the packet has a flexible and programmable room for meta data, which can later be used by BPF to set various skb members when passing up the stack. For details, please see second patch. Support has been implemented and tested with two drivers, and should be straight forward to add to other drivers as well which properly support head adjustment already.

Series applied, thanks.

We're accumulating features again and leaving some drivers behind, let's not allow this to get out of hand for XDP_REDIRECT and this stuff.

Looks like a couple of updates missed to get carried into tools/include/uapi/, so copy the bpf.h header as usual to pull in latest updates.

Signed-off-by Daniel Borkmann <[email protected]>

Acked-by Alexei Starovoitov <[email protected]>

Acked-by John Fastabend <[email protected]>

---
 tools/include/uapi/linux/bpf.h | 45 ++++++++++++++++++++++++++++++------------
 1 file changed, 32 insertions(+), 13 deletions(-)

tools/include/uapi/linux/bpf.h +0 -0

index 461811e..e43491a 100644

Quoted 15 lines · On Mon, 25 Sep 2017 02:25:52 +0200 Daniel Borkmann <[email protected]> wrote:

Looks like a couple of updates missed to get carried into tools/include/uapi/, so copy the bpf.h header as usual to pull in latest updates.

Signed-off-by Daniel Borkmann <[email protected]>

Acked-by Alexei Starovoitov <[email protected]>

Acked-by John Fastabend <[email protected]>

---
 tools/include/uapi/linux/bpf.h | 45 ++++++++++++++++++++++++++++++------------
 1 file changed, 32 insertions(+), 13 deletions(-)

tools/include/uapi/linux/bpf.h +0 -0

index 461811e..e43491a 100644

tools/include/uapi/linux/bpf.h enum bpf_attach_type { · line 151

Quoted 9 lines

tools/include/uapi/linux/bpf.h +0 -6

@@ -143,12 +143,6 @@ enum bpf_attach_type {
 
 #define MAX_BPF_ATTACH_TYPE __MAX_BPF_ATTACH_TYPE
 
-enum bpf_sockmap_flags {
-	BPF_SOCKMAP_UNSPEC,
-	BPF_SOCKMAP_STRPARSER,
-	__MAX_BPF_SOCKMAP_FLAG
-};
-  

Why does "bpf_sockmap_flags" gets removed?

It got added in: commit 69e8cc134bcb ("bpf: sockmap sample program"). But I cannot find any users of these flags, so I guess it is okay.

On 09/26/2017 09:13 PM, Jesper Dangaard Brouer wrote:
[...]

Quoted 3 lines

I'm currently implementing a cpumap type, that transfers raw XDP frames to another CPU, and the SKB is allocated on the remote CPU. (It actually works extremely well).

Meaning you let all the XDP_PASS packets get processed on a different CPU, so you can reserve the whole CPU just for prefiltering, right?

Do you have some numbers to share at this point, just curious when you mention it works extremely well.

Another test

Quoted 9 lines · On Tue, 26 Sep 2017 21:58:53 +0200 Daniel Borkmann <[email protected]> wrote:

On 09/26/2017 09:13 PM, Jesper Dangaard Brouer wrote:
[...]

Quoted 3 lines

I'm currently implementing a cpumap type, that transfers raw XDP frames to another CPU, and the SKB is allocated on the remote CPU. (It actually works extremely well).

Meaning you let all the XDP_PASS packets get processed on a different CPU, so you can reserve the whole CPU just for prefiltering, right?

Yes, exactly. Except I use the XDP_REDIRECT action to steer packets. The trick is using the map-flush point, to transfer packets in bulk to the remote CPU (single call IPC is too slow), but at the same time flush single packets if NAPI didn't see a bulk.

Quoted 2 lines

Do you have some numbers to share at this point, just curious when you mention it works extremely well.

Sure... I've done a lot of benchmarking on this patchset ;-) I have a benchmark program called xdp_redirect_cpu [1][2], that collect stats via tracepoints (atm I'm limiting bulking 8 packets, and have tracepoints at bulk spots, to amortize tracepoint cost 25ns/8=3.125ns)

[1] https://github.com/netoptimizer/prototype-kernel/blob/master/kernel/samples/bpf/xdp_redirect_cpu_kern.c
[2] https://github.com/netoptimizer/prototype-kernel/blob/master/kernel/samples/bpf/xdp_redirect_cpu_user.c

Here I'm installing a DDoS program that drops UDP port 9 (pktgen packets) on RX CPU=0. I'm forcing my netperf to hit the same CPU, that the 11.9Mpps DDoS attack is hitting.

Running XDP/eBPF prog_num:4

XDP-cpumap      CPU:to  pps            drop-pps    extra-info
XDP-RX          0       12,030,471     11,966,982  0          
XDP-RX          total   12,030,471     11,966,982 
cpumap-enqueue    0:2   63,488         0           0          
cpumap-enqueue  sum:2   63,488         0           0          
cpumap_kthread  2       63,488         0           3          time_exceed
cpumap_kthread  total   63,488         0           0          
redirect_err    total   0              0          

$ netperf -H 172.16.0.2 -t TCP_CRR  -l 10 -D1 -T5,5 -- -r 1024,1024
Local /Remote
Socket Size   Request  Resp.   Elapsed  Trans.
Send   Recv   Size     Size    Time     Rate         
bytes  Bytes  bytes    bytes   secs.    per sec   

16384 87380 1024 1024 10.00 12735.97
16384 87380

The netperf TCP_CRR performance is the same, without XDP loaded.

Quoted 52 lines · Jesper Dangaard Brouer

Quoted 9 lines · On Tue, 26 Sep 2017 21:58:53 +0200 Daniel Borkmann <[email protected]> wrote:

On 09/26/2017 09:13 PM, Jesper Dangaard Brouer wrote:
[...]

Quoted 3 lines

I'm currently implementing a cpumap type, that transfers raw XDP frames to another CPU, and the SKB is allocated on the remote CPU. (It actually works extremely well).

Meaning you let all the XDP_PASS packets get processed on a different CPU, so you can reserve the whole CPU just for prefiltering, right?

Yes, exactly. Except I use the XDP_REDIRECT action to steer packets. The trick is using the map-flush point, to transfer packets in bulk to the remote CPU (single call IPC is too slow), but at the same time flush single packets if NAPI didn't see a bulk.

Quoted 2 lines

Do you have some numbers to share at this point, just curious when you mention it works extremely well.

Sure... I've done a lot of benchmarking on this patchset ;-) I have a benchmark program called xdp_redirect_cpu [1][2], that collect stats via tracepoints (atm I'm limiting bulking 8 packets, and have tracepoints at bulk spots, to amortize tracepoint cost 25ns/8=3.125ns)

[1] https://github.com/netoptimizer/prototype-kernel/blob/master/kernel/samples/bpf/xdp_redirect_cpu_kern.c
[2] https://github.com/netoptimizer/prototype-kernel/blob/master/kernel/samples/bpf/xdp_redirect_cpu_user.c

Here I'm installing a DDoS program that drops UDP port 9 (pktgen packets) on RX CPU=0. I'm forcing my netperf to hit the same CPU, that the 11.9Mpps DDoS attack is hitting.

Running XDP/eBPF prog_num:4

XDP-cpumap      CPU:to  pps            drop-pps    extra-info
XDP-RX          0       12,030,471     11,966,982  0          
XDP-RX          total   12,030,471     11,966,982 
cpumap-enqueue    0:2   63,488         0           0          
cpumap-enqueue  sum:2   63,488         0           0          
cpumap_kthread  2       63,488         0           3          time_exceed
cpumap_kthread  total   63,488         0           0          
redirect_err    total   0              0          

$ netperf -H 172.16.0.2 -t TCP_CRR  -l 10 -D1 -T5,5 -- -r 1024,1024
Local /Remote
Socket Size   Request  Resp.   Elapsed  Trans.
Send   Recv   Size     Size    Time     Rate         
bytes  Bytes  bytes    bytes   secs.    per sec   

16384 87380 1024 1024 10.00 12735.97
16384 87380

Quoted 2 lines

The netperf TCP_CRR performance is the same, without XDP loaded.

Just curious could you also try this with RPS enabled (or does this have RPS enabled). RPS should effectively do the same thing but higher in the stack. I'm curious what the delta would be. Might be another interesting case and fairly easy to setup if you already have the above scripts.

From mpstat it looks like it is the RX-RPS CPUs that are the bottleneck.

Quoted 53 lines · On Wed, 27 Sep 2017 06:35:40 -0700 John Fastabend <[email protected]>…

Quoted 52 lines · Jesper Dangaard Brouer

Quoted 9 lines · On Tue, 26 Sep 2017 21:58:53 +0200 Daniel Borkmann <[email protected]> wrote:

On 09/26/2017 09:13 PM, Jesper Dangaard Brouer wrote:
[...]

Quoted 3 lines

I'm currently implementing a cpumap type, that transfers raw XDP frames to another CPU, and the SKB is allocated on the remote CPU. (It actually works extremely well).

Meaning you let all the XDP_PASS packets get processed on a different CPU, so you can reserve the whole CPU just for prefiltering, right?

Yes, exactly. Except I use the XDP_REDIRECT action to steer packets. The trick is using the map-flush point, to transfer packets in bulk to the remote CPU (single call IPC is too slow), but at the same time flush single packets if NAPI didn't see a bulk.

Quoted 2 lines

Do you have some numbers to share at this point, just curious when you mention it works extremely well.

Sure... I've done a lot of benchmarking on this patchset ;-) I have a benchmark program called xdp_redirect_cpu [1][2], that collect stats via tracepoints (atm I'm limiting bulking 8 packets, and have tracepoints at bulk spots, to amortize tracepoint cost 25ns/8=3.125ns)

[1] https://github.com/netoptimizer/prototype-kernel/blob/master/kernel/samples/bpf/xdp_redirect_cpu_kern.c
[2] https://github.com/netoptimizer/prototype-kernel/blob/master/kernel/samples/bpf/xdp_redirect_cpu_user.c

Here I'm installing a DDoS program that drops UDP port 9 (pktgen packets) on RX CPU=0. I'm forcing my netperf to hit the same CPU, that the 11.9Mpps DDoS attack is hitting.

Running XDP/eBPF prog_num:4

XDP-cpumap      CPU:to  pps            drop-pps    extra-info
XDP-RX          0       12,030,471     11,966,982  0          
XDP-RX          total   12,030,471     11,966,982 
cpumap-enqueue    0:2   63,488         0           0          
cpumap-enqueue  sum:2   63,488         0           0          
cpumap_kthread  2       63,488         0           3          time_exceed
cpumap_kthread  total   63,488         0           0          
redirect_err    total   0              0          

$ netperf -H 172.16.0.2 -t TCP_CRR  -l 10 -D1 -T5,5 -- -r 1024,1024
Local /Remote
Socket Size   Request  Resp.   Elapsed  Trans.
Send   Recv   Size     Size    Time     Rate         
bytes  Bytes  bytes    bytes   secs.    per sec   

16384 87380 1024 1024 10.00 12735.97
16384 87380

Quoted 7 lines

Quoted 2 lines

The netperf TCP_CRR performance is the same, without XDP loaded.

Just curious could you also try this with RPS enabled (or does this have RPS enabled). RPS should effectively do the same thing but higher in the stack. I'm curious what the delta would be. Might be another interesting case and fairly easy to setup if you already have the above scripts.

Yes, I'm essentially competing with RSP, thus such a comparison is very relevant...

This is only a 6 CPUs system. Allocate 2 CPUs to RPS receive and let other 4 CPUS process packet.

Summary of RPS (Receive Packet Steering) performance:
* End result is 6.3 Mpps max performance
* netperf TCP_CRR is 1 trans/sec.
* Each RX-RPS CPU stall at ~3.2Mpps.

The full test report below with setup:

The mask needed::

perl -e 'printf "%b\n",0x3C'
111100

RPS setup::

sudo sh -c 'echo 32768 > /proc/sys/net/core/rps_sock_flow_entries'

 for N in $(seq 0 5) ; do \
   sudo sh -c "echo 8192 > /sys/class/net/ixgbe1/queues/rx-$N/rps_flow_cnt" ; \
   sudo sh -c "echo 3c > /sys/class/net/ixgbe1/queues/rx-$N/rps_cpus" ; \
   grep -H . /sys/class/net/ixgbe1/queues/rx-$N/rps_cpus ; \

done

Reduce RX queues to two ::

ethtool -L ixgbe1 combined 2

IRQ align to CPU numbers::

 $ ~/setup01.sh
 Not root, running with sudo
  --- Disable Ethernet flow-control ---
 rx unmodified, ignoring
 tx unmodified, ignoring
 no pause parameters changed, aborting
 rx unmodified, ignoring
 tx unmodified, ignoring
 no pause parameters changed, aborting
  --- Align IRQs ---
 /proc/irq/54/ixgbe1-TxRx-0/../smp_affinity_list:0
 /proc/irq/55/ixgbe1-TxRx-1/../smp_affinity_list:1
 /proc/irq/56/ixgbe1/../smp_affinity_list:0-5

$ grep -H . /sys/class/net/ixgbe1/queues/rx-*/rps_cpus
/sys/class/net/ixgbe1/queues/rx-0/rps_cpus:3c
/sys/class/net/ixgbe1/queues/rx-1/rps_cpus:3c

Generator is sending: 12,715,782 tx_packets /sec

 ./pktgen_sample04_many_flows.sh -vi ixgbe2 -m 00:1b:21:bb:9a:84 \
    -d 172.16.0.2 -t8

$ nstat > /dev/null && sleep 1 && nstat
#kernel
IpInReceives                    6346544            0.0
IpInDelivers                    6346544            0.0
IpOutRequests                   1020               0.0
IcmpOutMsgs                     1020               0.0
IcmpOutDestUnreachs             1020               0.0
IcmpMsgOutType3                 1020               0.0
UdpNoPorts                      6346898            0.0
IpExtInOctets                   291964714          0.0
IpExtOutOctets                  73440              0.0
IpExtInNoECTPkts                6347063            0.0

$ mpstat -P ALL -u -I SCPU -I SUM

Average:     CPU    %usr   %nice    %sys   %irq   %soft  %idle
Average:     all    0.00    0.00    0.00   0.42   72.97  26.61
Average:       0    0.00    0.00    0.00   0.17   99.83   0.00
Average:       1    0.00    0.00    0.00   0.17   99.83   0.00
Average:       2    0.00    0.00    0.00   0.67   60.37  38.96
Average:       3    0.00    0.00    0.00   0.67   58.70  40.64
Average:       4    0.00    0.00    0.00   0.67   59.53  39.80
Average:       5    0.00    0.00    0.00   0.67   58.93  40.40

Average:     CPU    intr/s
Average:     all 152067.22
Average:       0  50064.73
Average:       1  50089.35
Average:       2  45095.17
Average:       3  44875.04
Average:       4  44906.32
Average:       5  45152.08

Average:     CPU     TIMER/s   NET_TX/s   NET_RX/s TASKLET/s  SCHED/s     RCU/s
Average:       0      609.48       0.17   49431.28      0.00     2.66     21.13
Average:       1      567.55       0.00   49498.00      0.00     2.66     21.13
Average:       2      998.34       0.00   43941.60      4.16    82.86     68.22
Average:       3      540.60       0.17   44140.27      0.00    85.52    108.49
Average:       4      537.27       0.00   44219.63      0.00    84.53     64.89
Average:       5      530.78       0.17   44445.59      0.00    85.02     90.52

From mpstat it looks like it is the RX-RPS CPUs that are the bottleneck.

Show adapter(s) (ixgbe1) statistics (ONLY that changed!)

Ethtool(ixgbe1) stat:     11109531 (   11,109,531) <= fdir_miss /sec
Ethtool(ixgbe1) stat:    380632356 (  380,632,356) <= rx_bytes /sec
Ethtool(ixgbe1) stat:    812792611 (  812,792,611) <= rx_bytes_nic /sec
Ethtool(ixgbe1) stat:      1753550 (    1,753,550) <= rx_missed_errors /sec
Ethtool(ixgbe1) stat:      4602487 (    4,602,487) <= rx_no_dma_resources /sec
Ethtool(ixgbe1) stat:      6343873 (    6,343,873) <= rx_packets /sec
Ethtool(ixgbe1) stat:     10946441 (   10,946,441) <= rx_pkts_nic /sec
Ethtool(ixgbe1) stat:    190287853 (  190,287,853) <= rx_queue_0_bytes /sec
Ethtool(ixgbe1) stat:      3171464 (    3,171,464) <= rx_queue_0_packets /sec
Ethtool(ixgbe1) stat:    190344503 (  190,344,503) <= rx_queue_1_bytes /sec
Ethtool(ixgbe1) stat:      3172408 (    3,172,408) <= rx_queue_1_packets /sec

Notice, each RX-CPU can only process 3.1Mpps.

RPS RX-CPU(0):

 # Overhead  CPU  Symbol
 # ........  ...  .......................................
 #
    11.72%  000  [k] ixgbe_poll
    11.29%  000  [k] _raw_spin_lock
    10.35%  000  [k] dev_gro_receive
     8.36%  000  [k] __build_skb
     7.35%  000  [k] __skb_get_hash
     6.22%  000  [k] enqueue_to_backlog
     5.89%  000  [k] __skb_flow_dissect
     4.43%  000  [k] inet_gro_receive
     4.19%  000  [k] ___slab_alloc
     3.90%  000  [k] queued_spin_lock_slowpath
     3.85%  000  [k] kmem_cache_alloc
     3.06%  000  [k] build_skb
     2.66%  000  [k] get_rps_cpu
     2.57%  000  [k] napi_gro_receive
     2.34%  000  [k] eth_type_trans
     1.81%  000  [k] __cmpxchg_double_slab.isra.61
     1.47%  000  [k] ixgbe_alloc_rx_buffers
     1.43%  000  [k] get_partial_node.isra.81
     0.84%  000  [k] swiotlb_sync_single
     0.74%  000  [k] udp4_gro_receive
     0.73%  000  [k] netif_receive_skb_internal
     0.72%  000  [k] udp_gro_receive
     0.63%  000  [k] skb_gro_reset_offset
     0.49%  000  [k] __skb_flow_get_ports
     0.48%  000  [k] llist_add_batch
     0.36%  000  [k] swiotlb_sync_single_for_cpu
     0.34%  000  [k] __slab_alloc

Remote RPS-CPU(3) getting packets::

 # Overhead  CPU  Symbol
 # ........  ...  ..............................................
 #
    33.02%  003  [k] poll_idle
    10.99%  003  [k] __netif_receive_skb_core
    10.45%  003  [k] page_frag_free
     8.49%  003  [k] ip_rcv
     4.19%  003  [k] fib_table_lookup
     2.84%  003  [k] __udp4_lib_rcv
     2.81%  003  [k] __slab_free
     2.23%  003  [k] __udp4_lib_lookup
     2.09%  003  [k] ip_route_input_rcu
     2.07%  003  [k] kmem_cache_free
     2.06%  003  [k] udp_v4_early_demux
     1.73%  003  [k] ip_rcv_finish

Quoted 40 lines · Daniel Borkmann

On 09/25/2017 08:10 PM, Andy Gospodarek wrote:
[...]

Quoted 9 lines

First, thanks for this detailed description. It was helpful to read along with the patches.

My only concern about this area being generic is that you are now in a state where any bpf program must know about all the bpf programs in the receive pipeline before it can properly parse what is stored in the meta-data and add it to an skb (or perform any other action). Especially if each program adds it's own meta-data along the way.

Quoted 4 lines

Maybe this isn't a big concern based on the number of users of this today, but it just starts to seem like a concern as there are these hints being passed between layers that are challenging to track due to a lack of a standard format for passing data between.

Btw, we do have similar kind of programmable scratch buffer also today wrt skb cb[] that you can program from tc side, the perf ring buffer, which doesn't have any fixed layout for the slots, or a per-cpu map where you can transfer data between tail calls for example, then tail calls themselves that need to coordinate, or simply mangling of packets itself if you will, but more below to your use case ...

Quoted 9 lines

The main reason I bring this up is that Michael and I had discussed and designed a way for drivers to communicate between each other that rx resources could be freed after a tx completion on an XDP_REDIRECT action. Much like this code, it involved adding an new element to struct xdp_md that could point to the important information. Now that there is a generic way to handle this, it would seem nice to be able to leverage it, but I'm not sure how reliable this meta-data area would be without the ability to mark it in some manner.

Quoted 7 lines

For additional background, the minimum amount of data needed in the case Michael and I were discussing was really 2 words. One to serve as a pointer to an rx_ring structure and one to have a counter to the rx producer entry. This data could be acessed by the driver processing the tx completions and callback to the driver that received the frame off the wire to perform any needed processing. (For those curious this would also require a new callback/netdev op to act on this data stored in the XDP buffer.)

Quoted 5 lines

What you describe above doesn't seem to be fitting to the use-case of this set, meaning the area here is fully programmable out of the BPF program, the infrastructure you're describing is some sort of means of communication between drivers for the XDP_REDIRECT, and should be outside of the control of the BPF program to mangle.

OK, I understand that perspective. I think saying this is really meant as a BPF<->BPF communication channel for now is fine.

Quoted 8 lines

You could probably reuse the base infra here and make a part of that inaccessible for the program with some sort of a fixed layout, but I haven't seen your code yet to be able to fully judge. Intention here is to allow for programmability within the BPF prog in a generic way, such that based on the use-case it can be populated in specific ways and propagated to the skb w/o having to define a fixed layout and bloat xdp_buff all the way to an skb while still retaining all the flexibility.

Some level of reuse might be proper, but I'd rather it be explicit for my use since it's not exclusively something that will need to be used by a BPF prog, but rather the driver. I'll produce some patches this week for reference.

Quoted 59 lines · Andy Gospodarek

Quoted 40 lines · Daniel Borkmann

On 09/25/2017 08:10 PM, Andy Gospodarek wrote:
[...]

Quoted 9 lines

First, thanks for this detailed description. It was helpful to read along with the patches.

My only concern about this area being generic is that you are now in a state where any bpf program must know about all the bpf programs in the receive pipeline before it can properly parse what is stored in the meta-data and add it to an skb (or perform any other action). Especially if each program adds it's own meta-data along the way.

Quoted 4 lines

Maybe this isn't a big concern based on the number of users of this today, but it just starts to seem like a concern as there are these hints being passed between layers that are challenging to track due to a lack of a standard format for passing data between.

Btw, we do have similar kind of programmable scratch buffer also today wrt skb cb[] that you can program from tc side, the perf ring buffer, which doesn't have any fixed layout for the slots, or a per-cpu map where you can transfer data between tail calls for example, then tail calls themselves that need to coordinate, or simply mangling of packets itself if you will, but more below to your use case ...

Quoted 9 lines

The main reason I bring this up is that Michael and I had discussed and designed a way for drivers to communicate between each other that rx resources could be freed after a tx completion on an XDP_REDIRECT action. Much like this code, it involved adding an new element to struct xdp_md that could point to the important information. Now that there is a generic way to handle this, it would seem nice to be able to leverage it, but I'm not sure how reliable this meta-data area would be without the ability to mark it in some manner.

Quoted 7 lines

For additional background, the minimum amount of data needed in the case Michael and I were discussing was really 2 words. One to serve as a pointer to an rx_ring structure and one to have a counter to the rx producer entry. This data could be acessed by the driver processing the tx completions and callback to the driver that received the frame off the wire to perform any needed processing. (For those curious this would also require a new callback/netdev op to act on this data stored in the XDP buffer.)

Quoted 5 lines

What you describe above doesn't seem to be fitting to the use-case of this set, meaning the area here is fully programmable out of the BPF program, the infrastructure you're describing is some sort of means of communication between drivers for the XDP_REDIRECT, and should be outside of the control of the BPF program to mangle.

OK, I understand that perspective. I think saying this is really meant as a BPF<->BPF communication channel for now is fine.

Quoted 8 lines

You could probably reuse the base infra here and make a part of that inaccessible for the program with some sort of a fixed layout, but I haven't seen your code yet to be able to fully judge. Intention here is to allow for programmability within the BPF prog in a generic way, such that based on the use-case it can be populated in specific ways and propagated to the skb w/o having to define a fixed layout and bloat xdp_buff all the way to an skb while still retaining all the flexibility.

Quoted 4 lines

Some level of reuse might be proper, but I'd rather it be explicit for my use since it's not exclusively something that will need to be used by a BPF prog, but rather the driver. I'll produce some patches this week for reference.

Sorry for chiming in late, I've been offline.

We're looking to add some functionality from driver to XDP inside this xdp_buff->data_meta region. We want to assign it to an opaque structure, that would be specific per driver (think of a flex descriptor coming out of the hardware). We'd like to pass these offloaded computations into XDP programs to help accelerate them, such as packet type, where headers are located, etc. It's similar to Jesper's RFC patches back in May when passing through the mlx Rx descriptor to XDP.

This is actually what a few of us are planning to present at NetDev 2.2 in November. If you're hoping to restrict this headroom in the xdp_buff for an exclusive use case with XDP_REDIRECT, then I'd like to discuss that further.

Quoted 75 lines · Peter

Quoted 59 lines · Andy Gospodarek

Quoted 40 lines · Daniel Borkmann

On 09/25/2017 08:10 PM, Andy Gospodarek wrote:
[...]

Quoted 9 lines

First, thanks for this detailed description. It was helpful to read along with the patches.

My only concern about this area being generic is that you are now in a state where any bpf program must know about all the bpf programs in the receive pipeline before it can properly parse what is stored in the meta-data and add it to an skb (or perform any other action). Especially if each program adds it's own meta-data along the way.

Quoted 4 lines

Maybe this isn't a big concern based on the number of users of this today, but it just starts to seem like a concern as there are these hints being passed between layers that are challenging to track due to a lack of a standard format for passing data between.

Btw, we do have similar kind of programmable scratch buffer also today wrt skb cb[] that you can program from tc side, the perf ring buffer, which doesn't have any fixed layout for the slots, or a per-cpu map where you can transfer data between tail calls for example, then tail calls themselves that need to coordinate, or simply mangling of packets itself if you will, but more below to your use case ...

Quoted 9 lines

The main reason I bring this up is that Michael and I had discussed and designed a way for drivers to communicate between each other that rx resources could be freed after a tx completion on an XDP_REDIRECT action. Much like this code, it involved adding an new element to struct xdp_md that could point to the important information. Now that there is a generic way to handle this, it would seem nice to be able to leverage it, but I'm not sure how reliable this meta-data area would be without the ability to mark it in some manner.

Quoted 7 lines

For additional background, the minimum amount of data needed in the case Michael and I were discussing was really 2 words. One to serve as a pointer to an rx_ring structure and one to have a counter to the rx producer entry. This data could be acessed by the driver processing the tx completions and callback to the driver that received the frame off the wire to perform any needed processing. (For those curious this would also require a new callback/netdev op to act on this data stored in the XDP buffer.)

Quoted 5 lines

What you describe above doesn't seem to be fitting to the use-case of this set, meaning the area here is fully programmable out of the BPF program, the infrastructure you're describing is some sort of means of communication between drivers for the XDP_REDIRECT, and should be outside of the control of the BPF program to mangle.

OK, I understand that perspective. I think saying this is really meant as a BPF<->BPF communication channel for now is fine.

Quoted 8 lines

You could probably reuse the base infra here and make a part of that inaccessible for the program with some sort of a fixed layout, but I haven't seen your code yet to be able to fully judge. Intention here is to allow for programmability within the BPF prog in a generic way, such that based on the use-case it can be populated in specific ways and propagated to the skb w/o having to define a fixed layout and bloat xdp_buff all the way to an skb while still retaining all the flexibility.

Quoted 4 lines

Some level of reuse might be proper, but I'd rather it be explicit for my use since it's not exclusively something that will need to be used by a BPF prog, but rather the driver. I'll produce some patches this week for reference.

Sorry for chiming in late, I've been offline.

We're looking to add some functionality from driver to XDP inside this xdp_buff->data_meta region. We want to assign it to an opaque structure, that would be specific per driver (think of a flex descriptor coming out of the hardware). We'd like to pass these offloaded computations into XDP programs to help accelerate them, such as packet type, where headers are located, etc. It's similar to Jesper's RFC patches back in May when passing through the mlx Rx descriptor to XDP.

Quoted 5 lines

This is actually what a few of us are planning to present at NetDev 2.2 in November. If you're hoping to restrict this headroom in the xdp_buff for an exclusive use case with XDP_REDIRECT, then I'd like to discuss that further.

No sweat, PJ, thanks for replying. I saw the notes for your accepted session and I'm looking forward to it.

John's suggestion earlier in the thread was actually similar to the conclusion I reached when thinking about Daniel's patch a bit more. (I like John's better though as it doesn't get constrained by UAPI.) Since redirect actions happen at a point where no other programs will run on the buffer, that space can be used for this redirect data and there are no conflicts.

It sounds like the idea behind your proposal includes populating some data into the buffer before the XDP program is executed so that it can be used by the program. Would this data be useful later in the driver or stack or are you just hoping to accelerate processing of frames in the BPF program?

If the headroom needed for redirect info was only added after it was clear the redirect action was needed, would this conflict with the information you are trying to provide? I had planned to add this just after the action was XDP_REDIRECT was selected or at the end of the driver's ndo_xdp_xmit function -- it seems like it would not conflict.

(There's also Jesper's series from today -- I've seen it but have not had time to fully grok all of those changes.)

I'm pretty sure I misunderstood what you were going after with XDP_REDIRECT reserving the headroom. Our use case (patches coming in a few weeks) will populate the headroom coming out of the driver to XDP, and then once the XDP program extracts whatever hints it wants via helpers, I fully expect that area in the headroom to get stomped by something else. If we want to send any of that hint data up farther, we'll already have it extracted via the helpers, and the eBPF program can happily assign it to wherever in the outbound metadata area.

Quoted 3 lines

(There's also Jesper's series from today -- I've seen it but have not had time to fully grok all of those changes.)

I'm also working through my inbox to get to that series. I have some email to catch up on...

Thanks Andy,
-PJ

Quoted 85 lines · Andy Gospodarek

Quoted 75 lines · Peter

Quoted 59 lines · Andy Gospodarek

Quoted 40 lines · Daniel Borkmann

On 09/25/2017 08:10 PM, Andy Gospodarek wrote:
[...]

Quoted 9 lines

First, thanks for this detailed description. It was helpful to read along with the patches.

My only concern about this area being generic is that you are now in a state where any bpf program must know about all the bpf programs in the receive pipeline before it can properly parse what is stored in the meta-data and add it to an skb (or perform any other action). Especially if each program adds it's own meta-data along the way.

Quoted 4 lines

Maybe this isn't a big concern based on the number of users of this today, but it just starts to seem like a concern as there are these hints being passed between layers that are challenging to track due to a lack of a standard format for passing data between.

Btw, we do have similar kind of programmable scratch buffer also today wrt skb cb[] that you can program from tc side, the perf ring buffer, which doesn't have any fixed layout for the slots, or a per-cpu map where you can transfer data between tail calls for example, then tail calls themselves that need to coordinate, or simply mangling of packets itself if you will, but more below to your use case ...

Quoted 9 lines

The main reason I bring this up is that Michael and I had discussed and designed a way for drivers to communicate between each other that rx resources could be freed after a tx completion on an XDP_REDIRECT action. Much like this code, it involved adding an new element to struct xdp_md that could point to the important information. Now that there is a generic way to handle this, it would seem nice to be able to leverage it, but I'm not sure how reliable this meta-data area would be without the ability to mark it in some manner.

Quoted 7 lines

For additional background, the minimum amount of data needed in the case Michael and I were discussing was really 2 words. One to serve as a pointer to an rx_ring structure and one to have a counter to the rx producer entry. This data could be acessed by the driver processing the tx completions and callback to the driver that received the frame off the wire to perform any needed processing. (For those curious this would also require a new callback/netdev op to act on this data stored in the XDP buffer.)

Quoted 5 lines

What you describe above doesn't seem to be fitting to the use-case of this set, meaning the area here is fully programmable out of the BPF program, the infrastructure you're describing is some sort of means of communication between drivers for the XDP_REDIRECT, and should be outside of the control of the BPF program to mangle.

OK, I understand that perspective. I think saying this is really meant as a BPF<->BPF communication channel for now is fine.

Quoted 8 lines

You could probably reuse the base infra here and make a part of that inaccessible for the program with some sort of a fixed layout, but I haven't seen your code yet to be able to fully judge. Intention here is to allow for programmability within the BPF prog in a generic way, such that based on the use-case it can be populated in specific ways and propagated to the skb w/o having to define a fixed layout and bloat xdp_buff all the way to an skb while still retaining all the flexibility.

Quoted 4 lines

Some level of reuse might be proper, but I'd rather it be explicit for my use since it's not exclusively something that will need to be used by a BPF prog, but rather the driver. I'll produce some patches this week for reference.

Sorry for chiming in late, I've been offline.

We're looking to add some functionality from driver to XDP inside this xdp_buff->data_meta region. We want to assign it to an opaque structure, that would be specific per driver (think of a flex descriptor coming out of the hardware). We'd like to pass these offloaded computations into XDP programs to help accelerate them, such as packet type, where headers are located, etc. It's similar to Jesper's RFC patches back in May when passing through the mlx Rx descriptor to XDP.

Quoted 4 lines

This is actually what a few of us are planning to present at NetDev 2.2 in November. If you're hoping to restrict this headroom in the xdp_buff for an exclusive use case with XDP_REDIRECT, then I'd like to discuss that further.

No sweat, PJ, thanks for replying. I saw the notes for your accepted session and I'm looking forward to it.

Quoted 6 lines

John's suggestion earlier in the thread was actually similar to the conclusion I reached when thinking about Daniel's patch a bit more. (I like John's better though as it doesn't get constrained by UAPI.) Since redirect actions happen at a point where no other programs will run on the buffer, that space can be used for this redirect data and there are no conflicts.

Ah, yes, John and I spoke about this at Plumber's and this is basically what we came to as well. A set of helpers that won't have to be in UAPI, but they will be potentially vendor-specific to extract the meta-data hints out for the XDP program to use.

Quoted 5 lines

It sounds like the idea behind your proposal includes populating some data into the buffer before the XDP program is executed so that it can be used by the program. Would this data be useful later in the driver or stack or are you just hoping to accelerate processing of frames in the BPF program?

Right now we're thinking it would only be useful for XDP programs to execute things quicker, i.e. not have to compute things that are already computed by the hardware (rxhash, ptype, header locations, etc.). I don't have any plans to pass this data off elsewhere in the stack or back to the driver at this point.

Quoted 5 lines

If the headroom needed for redirect info was only added after it was clear the redirect action was needed, would this conflict with the information you are trying to provide? I had planned to add this just after the action was XDP_REDIRECT was selected or at the end of the driver's ndo_xdp_xmit function -- it seems like it would not conflict.

I'm pretty sure I misunderstood what you were going after with XDP_REDIRECT reserving the headroom. Our use case (patches coming in a few weeks) will populate the headroom coming out of the driver to XDP, and then once the XDP program extracts whatever hints it wants via helpers, I fully expect that area in the headroom to get stomped by something else. If we want to send any of that hint data up farther, we'll already have it extracted via the helpers, and the eBPF program can happily assign it to wherever in the outbound metadata area.

[...]

Quoted 8 lines

I'm pretty sure I misunderstood what you were going after with XDP_REDIRECT reserving the headroom. Our use case (patches coming in a few weeks) will populate the headroom coming out of the driver to XDP, and then once the XDP program extracts whatever hints it wants via helpers, I fully expect that area in the headroom to get stomped by something else. If we want to send any of that hint data up farther, we'll already have it extracted via the helpers, and the eBPF program can happily assign it to wherever in the outbound metadata area.

In case its not obvious with the latest xdp metadata patches the outbound metadata can then be pushed into skb fields via a tc_cls program if needed.

Quoted 215 lines · Jesper Dangaard Brouer

Quoted 53 lines · On Wed, 27 Sep 2017 06:35:40 -0700 John Fastabend <[email protected]>…

Quoted 52 lines · Jesper Dangaard Brouer

Quoted 9 lines · On Tue, 26 Sep 2017 21:58:53 +0200 Daniel Borkmann <[email protected]> wrote:

On 09/26/2017 09:13 PM, Jesper Dangaard Brouer wrote:
[...]

Quoted 3 lines

I'm currently implementing a cpumap type, that transfers raw XDP frames to another CPU, and the SKB is allocated on the remote CPU. (It actually works extremely well).

Meaning you let all the XDP_PASS packets get processed on a different CPU, so you can reserve the whole CPU just for prefiltering, right?

Yes, exactly. Except I use the XDP_REDIRECT action to steer packets. The trick is using the map-flush point, to transfer packets in bulk to the remote CPU (single call IPC is too slow), but at the same time flush single packets if NAPI didn't see a bulk.

Quoted 2 lines

Do you have some numbers to share at this point, just curious when you mention it works extremely well.

Sure... I've done a lot of benchmarking on this patchset ;-) I have a benchmark program called xdp_redirect_cpu [1][2], that collect stats via tracepoints (atm I'm limiting bulking 8 packets, and have tracepoints at bulk spots, to amortize tracepoint cost 25ns/8=3.125ns)

[1] https://github.com/netoptimizer/prototype-kernel/blob/master/kernel/samples/bpf/xdp_redirect_cpu_kern.c
[2] https://github.com/netoptimizer/prototype-kernel/blob/master/kernel/samples/bpf/xdp_redirect_cpu_user.c

Here I'm installing a DDoS program that drops UDP port 9 (pktgen packets) on RX CPU=0. I'm forcing my netperf to hit the same CPU, that the 11.9Mpps DDoS attack is hitting.

Running XDP/eBPF prog_num:4

XDP-cpumap      CPU:to  pps            drop-pps    extra-info
XDP-RX          0       12,030,471     11,966,982  0          
XDP-RX          total   12,030,471     11,966,982 
cpumap-enqueue    0:2   63,488         0           0          
cpumap-enqueue  sum:2   63,488         0           0          
cpumap_kthread  2       63,488         0           3          time_exceed
cpumap_kthread  total   63,488         0           0          
redirect_err    total   0              0          

$ netperf -H 172.16.0.2 -t TCP_CRR  -l 10 -D1 -T5,5 -- -r 1024,1024
Local /Remote
Socket Size   Request  Resp.   Elapsed  Trans.
Send   Recv   Size     Size    Time     Rate         
bytes  Bytes  bytes    bytes   secs.    per sec   

16384 87380 1024 1024 10.00 12735.97
16384 87380

Quoted 7 lines

Quoted 2 lines

The netperf TCP_CRR performance is the same, without XDP loaded.

Just curious could you also try this with RPS enabled (or does this have RPS enabled). RPS should effectively do the same thing but higher in the stack. I'm curious what the delta would be. Might be another interesting case and fairly easy to setup if you already have the above scripts.

Yes, I'm essentially competing with RSP, thus such a comparison is very relevant...

This is only a 6 CPUs system. Allocate 2 CPUs to RPS receive and let other 4 CPUS process packet.

Summary of RPS (Receive Packet Steering) performance:
* End result is 6.3 Mpps max performance
* netperf TCP_CRR is 1 trans/sec.
* Each RX-RPS CPU stall at ~3.2Mpps.

The full test report below with setup:

The mask needed::

perl -e 'printf "%b\n",0x3C'
111100

RPS setup::

sudo sh -c 'echo 32768 > /proc/sys/net/core/rps_sock_flow_entries'

 for N in $(seq 0 5) ; do \
   sudo sh -c "echo 8192 > /sys/class/net/ixgbe1/queues/rx-$N/rps_flow_cnt" ; \
   sudo sh -c "echo 3c > /sys/class/net/ixgbe1/queues/rx-$N/rps_cpus" ; \
   grep -H . /sys/class/net/ixgbe1/queues/rx-$N/rps_cpus ; \

done

Reduce RX queues to two ::

ethtool -L ixgbe1 combined 2

IRQ align to CPU numbers::

 $ ~/setup01.sh
 Not root, running with sudo
  --- Disable Ethernet flow-control ---
 rx unmodified, ignoring
 tx unmodified, ignoring
 no pause parameters changed, aborting
 rx unmodified, ignoring
 tx unmodified, ignoring
 no pause parameters changed, aborting
  --- Align IRQs ---
 /proc/irq/54/ixgbe1-TxRx-0/../smp_affinity_list:0
 /proc/irq/55/ixgbe1-TxRx-1/../smp_affinity_list:1
 /proc/irq/56/ixgbe1/../smp_affinity_list:0-5

$ grep -H . /sys/class/net/ixgbe1/queues/rx-*/rps_cpus
/sys/class/net/ixgbe1/queues/rx-0/rps_cpus:3c
/sys/class/net/ixgbe1/queues/rx-1/rps_cpus:3c

Generator is sending: 12,715,782 tx_packets /sec

 ./pktgen_sample04_many_flows.sh -vi ixgbe2 -m 00:1b:21:bb:9a:84 \
    -d 172.16.0.2 -t8

$ nstat > /dev/null && sleep 1 && nstat
#kernel
IpInReceives                    6346544            0.0
IpInDelivers                    6346544            0.0
IpOutRequests                   1020               0.0
IcmpOutMsgs                     1020               0.0
IcmpOutDestUnreachs             1020               0.0
IcmpMsgOutType3                 1020               0.0
UdpNoPorts                      6346898            0.0
IpExtInOctets                   291964714          0.0
IpExtOutOctets                  73440              0.0
IpExtInNoECTPkts                6347063            0.0

$ mpstat -P ALL -u -I SCPU -I SUM

Average:     CPU    %usr   %nice    %sys   %irq   %soft  %idle
Average:     all    0.00    0.00    0.00   0.42   72.97  26.61
Average:       0    0.00    0.00    0.00   0.17   99.83   0.00
Average:       1    0.00    0.00    0.00   0.17   99.83   0.00
Average:       2    0.00    0.00    0.00   0.67   60.37  38.96
Average:       3    0.00    0.00    0.00   0.67   58.70  40.64
Average:       4    0.00    0.00    0.00   0.67   59.53  39.80
Average:       5    0.00    0.00    0.00   0.67   58.93  40.40

Average:     CPU    intr/s
Average:     all 152067.22
Average:       0  50064.73
Average:       1  50089.35
Average:       2  45095.17
Average:       3  44875.04
Average:       4  44906.32
Average:       5  45152.08

Average:     CPU     TIMER/s   NET_TX/s   NET_RX/s TASKLET/s  SCHED/s     RCU/s
Average:       0      609.48       0.17   49431.28      0.00     2.66     21.13
Average:       1      567.55       0.00   49498.00      0.00     2.66     21.13
Average:       2      998.34       0.00   43941.60      4.16    82.86     68.22
Average:       3      540.60       0.17   44140.27      0.00    85.52    108.49
Average:       4      537.27       0.00   44219.63      0.00    84.53     64.89
Average:       5      530.78       0.17   44445.59      0.00    85.02     90.52

From mpstat it looks like it is the RX-RPS CPUs that are the bottleneck.

Show adapter(s) (ixgbe1) statistics (ONLY that changed!)

Ethtool(ixgbe1) stat:     11109531 (   11,109,531) <= fdir_miss /sec
Ethtool(ixgbe1) stat:    380632356 (  380,632,356) <= rx_bytes /sec
Ethtool(ixgbe1) stat:    812792611 (  812,792,611) <= rx_bytes_nic /sec
Ethtool(ixgbe1) stat:      1753550 (    1,753,550) <= rx_missed_errors /sec
Ethtool(ixgbe1) stat:      4602487 (    4,602,487) <= rx_no_dma_resources /sec
Ethtool(ixgbe1) stat:      6343873 (    6,343,873) <= rx_packets /sec
Ethtool(ixgbe1) stat:     10946441 (   10,946,441) <= rx_pkts_nic /sec
Ethtool(ixgbe1) stat:    190287853 (  190,287,853) <= rx_queue_0_bytes /sec
Ethtool(ixgbe1) stat:      3171464 (    3,171,464) <= rx_queue_0_packets /sec
Ethtool(ixgbe1) stat:    190344503 (  190,344,503) <= rx_queue_1_bytes /sec
Ethtool(ixgbe1) stat:      3172408 (    3,172,408) <= rx_queue_1_packets /sec

Notice, each RX-CPU can only process 3.1Mpps.

RPS RX-CPU(0):

 # Overhead  CPU  Symbol
 # ........  ...  .......................................
 #
    11.72%  000  [k] ixgbe_poll
    11.29%  000  [k] _raw_spin_lock
    10.35%  000  [k] dev_gro_receive
     8.36%  000  [k] __build_skb
     7.35%  000  [k] __skb_get_hash
     6.22%  000  [k] enqueue_to_backlog
     5.89%  000  [k] __skb_flow_dissect
     4.43%  000  [k] inet_gro_receive
     4.19%  000  [k] ___slab_alloc
     3.90%  000  [k] queued_spin_lock_slowpath
     3.85%  000  [k] kmem_cache_alloc
     3.06%  000  [k] build_skb
     2.66%  000  [k] get_rps_cpu
     2.57%  000  [k] napi_gro_receive
     2.34%  000  [k] eth_type_trans
     1.81%  000  [k] __cmpxchg_double_slab.isra.61
     1.47%  000  [k] ixgbe_alloc_rx_buffers
     1.43%  000  [k] get_partial_node.isra.81
     0.84%  000  [k] swiotlb_sync_single
     0.74%  000  [k] udp4_gro_receive
     0.73%  000  [k] netif_receive_skb_internal
     0.72%  000  [k] udp_gro_receive
     0.63%  000  [k] skb_gro_reset_offset
     0.49%  000  [k] __skb_flow_get_ports
     0.48%  000  [k] llist_add_batch
     0.36%  000  [k] swiotlb_sync_single_for_cpu
     0.34%  000  [k] __slab_alloc

Remote RPS-CPU(3) getting packets::

Quoted 10 lines

 # Overhead  CPU  Symbol
 # ........  ...  ..............................................
 #
    33.02%  003  [k] poll_idle
    10.99%  003  [k] __netif_receive_skb_core
    10.45%  003  [k] page_frag_free
     8.49%  003  [k] ip_rcv
     4.19%  003  [k] fib_table_lookup
     2.84%  003  [k] __udp4_lib_rcv
     2.81%  003  [k] __slab_free

Quoted 5 lines

     2.23%  003  [k] __udp4_lib_lookup
     2.09%  003  [k] ip_route_input_rcu
     2.07%  003  [k] kmem_cache_free
     2.06%  003  [k] udp_v4_early_demux
     1.73%  003  [k] ip_rcv_finish  

Very interesting data.

So above perf report compares to xdp-redirect-cpu this one:
Perf top on a CPU(3) that have to alloc and free SKBs etc.

# Overhead  CPU  Symbol
# ........  ...  .......................................
#
    15.51%  003  [k] fib_table_lookup
     8.91%  003  [k] cpu_map_kthread_run
     8.04%  003  [k] build_skb
     7.88%  003  [k] page_frag_free
     5.13%  003  [k] kmem_cache_alloc
     4.76%  003  [k] ip_route_input_rcu
     4.59%  003  [k] kmem_cache_free
     4.02%  003  [k] __udp4_lib_rcv
     3.20%  003  [k] fib_validate_source
     3.02%  003  [k] __netif_receive_skb_core
     3.02%  003  [k] udp_v4_early_demux
     2.90%  003  [k] ip_rcv
     2.80%  003  [k] ip_rcv_finish

right?
and in RPS case the consumer cpu is 33% idle whereas in redirect-cpu you can load it up all the way.
Am I interpreting all this correctly that with RPS cpu0 cannot distributed the packets to other cpus fast enough and that's a bottleneck?

whereas in redirect-cpu you're doing early packet distribution before skb alloc?

So in other words with redirect-cpu all consumer cpus are doing skb alloc and in RPS cpu0 is allocating skbs for all ?

and that's where 6M->12M performance gain comes from?