XDP support for ixgbe

Basic XDP drop support for ixgbe. Uses READ_ONCE/xchg semantics on XDP
programs instead of rcu primitives as suggested by Daniel Borkmann and
Alex Duyck.

Signed-off-by John Fastabend <[email protected]>

Acked-by Alexander Duyck <[email protected]>

---

?drivers/net/ethernet/intel/ixgbe/ixgbe.h???????? |??? 4 -
?drivers/net/ethernet/intel/ixgbe/ixgbe_ethtool.c |??? 4 -
?drivers/net/ethernet/intel/ixgbe/ixgbe_main.c??? |? 163

+++++++++++++++++++---
?3 files changed, 144 insertions(+), 27 deletions(-)

Quoted 6 lines · John Fastabend

Basic XDP drop support for ixgbe. Uses READ_ONCE/xchg semantics on XDP
programs instead of rcu primitives as suggested by Daniel Borkmann and
Alex Duyck.

Quoted 8 lines

Signed-off-by John Fastabend <[email protected]>

Acked-by Alexander Duyck <[email protected]>

---
 drivers/net/ethernet/intel/ixgbe/ixgbe.h         |    4 -
 drivers/net/ethernet/intel/ixgbe/ixgbe_ethtool.c |    4 -
 drivers/net/ethernet/intel/ixgbe/ixgbe_main.c    |  163
+++++++++++++++++++---
 3 files changed, 144 insertions(+), 27 deletions(-)

Does not apply cleanly to my dev-queue branch, what tree did you base these changes on?

Quoted 18 lines · Jeff Kirsher

Quoted 16 lines · John Fastabend

Basic XDP drop support for ixgbe. Uses READ_ONCE/xchg semantics on
XDP
programs instead of rcu primitives as suggested by Daniel Borkmann
and
Alex Duyck.

Signed-off-by John Fastabend <[email protected]>

Acked-by Alexander Duyck <[email protected]>

---

? drivers/net/ethernet/intel/ixgbe/ixgbe.h???????? |??? 4 -
? drivers/net/ethernet/intel/ixgbe/ixgbe_ethtool.c |??? 4 -
? drivers/net/ethernet/intel/ixgbe/ixgbe_main.c??? |? 163

+++++++++++++++++++---
? 3 files changed, 144 insertions(+), 27 deletions(-)

Quoted 4 lines

Does not apply cleanly to my dev-queue branch, what tree did you base
these changes on?

Did you pop off the v6 series? I see its still in your tree :/ Or do you
just want fixes to the initial series?

-----Original Message-----
From: Intel-wired-lan [mailto:intel-wired-lan-bounces at lists.osuosl.org] On Behalf Of John Fastabend
Sent: Thursday, March 23, 2017 9:31 PM
To: john.fastabend at gmail.com; alexander.duyck at gmail.com
Cc: intel-wired-lan at lists.osuosl.org; u9012063 at gmail.com
Subject: [Intel-wired-lan] [PATCH v7 1/2] ixgbe: add XDP support for pass and drop actions

Basic XDP drop support for ixgbe. Uses READ_ONCE/xchg semantics on XDP programs instead of rcu primitives as suggested by Daniel Borkmann and Alex Duyck.

Signed-off-by John Fastabend <[email protected]>

Acked-by Alexander Duyck <[email protected]>

---
 drivers/net/ethernet/intel/ixgbe/ixgbe.h         |    4 -
 drivers/net/ethernet/intel/ixgbe/ixgbe_ethtool.c |    4 -
 drivers/net/ethernet/intel/ixgbe/ixgbe_main.c    |  163
+++++++++++++++++++---
 3 files changed, 144 insertions(+), 27 deletions(-)

-----Original Message-----
From: Intel-wired-lan [mailto:intel-wired-lan-bounces at lists.osuosl.org] On Behalf Of John Fastabend
Sent: Thursday, March 23, 2017 9:31 PM

Cc: intel-wired-lan at lists.osuosl.org; u9012063 at gmail.com
Subject: [Intel-wired-lan] [PATCH v7 1/2] ixgbe: add XDP support for pass and drop actions

Basic XDP drop support for ixgbe. Uses READ_ONCE/xchg semantics on XDP programs instead of rcu primitives as suggested by Daniel Borkmann and Alex Duyck.

Signed-off-by John Fastabend <[email protected]>

Acked-by Alexander Duyck <[email protected]>

---

drivers/net/ethernet/intel/ixgbe/ixgbe.h         |    4 -
drivers/net/ethernet/intel/ixgbe/ixgbe_ethtool.c |    4 -
drivers/net/ethernet/intel/ixgbe/ixgbe_main.c    |  163
+++++++++++++++++++---
3 files changed, 144 insertions(+), 27 deletions(-)

drivers/net/ethernet/intel/ixgbe/ixgbe.h +3 -1

index 656ca8f..cb14813 100644
@@ -318,6 +318,7 @@ struct ixgbe_ring {
 	struct ixgbe_ring *next;	/* pointer to next ring in q_vector */
 	struct ixgbe_q_vector *q_vector; /* backpointer to host q_vector */
 	struct net_device *netdev;	/* netdev ring belongs to */
+	struct bpf_prog *xdp_prog;
 	struct device *dev;		/* device for DMA mapping */
 	struct ixgbe_fwd_adapter *l2_accel_priv;
 	void *desc;			/* descriptor ring memory */

@@ -555,6 +556,7 @@ struct ixgbe_adapter {
 	unsigned long active_vlans[BITS_TO_LONGS(VLAN_N_VID)];
 	/* OS defined structs */
 	struct net_device *netdev;
+	struct bpf_prog *xdp_prog;
 	struct pci_dev *pdev;
 
 	unsigned long state;

@@ -835,7 +837,7 @@ enum ixgbe_boards {
 void ixgbe_reinit_locked(struct ixgbe_adapter *adapter);
 void ixgbe_reset(struct ixgbe_adapter *adapter);
 void ixgbe_set_ethtool_ops(struct net_device *netdev);
-int ixgbe_setup_rx_resources(struct ixgbe_ring *);
+int ixgbe_setup_rx_resources(struct ixgbe_adapter *, struct ixgbe_ring *);
 int ixgbe_setup_tx_resources(struct ixgbe_ring *);
 void ixgbe_free_rx_resources(struct ixgbe_ring *);
 void ixgbe_free_tx_resources(struct ixgbe_ring *);

drivers/net/ethernet/intel/ixgbe/ixgbe_ethtool.c +1 -1

index 59730ed..79a126d 100644
@@ -1128,7 +1128,7 @@ static int ixgbe_set_ringparam(struct net_device
 *netdev,
 			       sizeof(struct ixgbe_ring));
 
 			temp_ring[i].count = new_rx_count;
-			err = ixgbe_setup_rx_resources(&temp_ring[i]);
+			err = ixgbe_setup_rx_resources(adapter, &temp_ring[i]);
 			if (err) {
 				while (i) {

					i--;

drivers/net/ethernet/intel/ixgbe/ixgbe_ethtool.c +1 -1

@@ -1761,7 +1761,7 @@ static int ixgbe_setup_desc_rings(struct
 ixgbe_adapter *adapter)
 	rx_ring->netdev = adapter->netdev;
 	rx_ring->reg_idx = adapter->rx_ring[0]->reg_idx;
 
-	err = ixgbe_setup_rx_resources(rx_ring);
+	err = ixgbe_setup_rx_resources(adapter, rx_ring);
 	if (err) {
 		ret_val = 4;

		goto err_nomem;

drivers/net/ethernet/intel/ixgbe/ixgbe_main.c +7 -0

index 91c1cda..ddb75ba 100644
@@ -49,6 +49,9 @@
 #include <linux/if_macvlan.h>
 #include <linux/if_bridge.h>
 #include <linux/prefetch.h>
+#include <linux/bpf.h>
+#include <linux/bpf_trace.h>
+#include <linux/atomic.h>
 #include <scsi/fc/fc_fcoe.h>
 #include <net/udp_tunnel.h>
 #include <net/pkt_cls.h>

@@ -1858,6 +1861,10 @@ static void ixgbe_dma_sync_frag(struct ixgbe_ring
 *rx_ring,
 * @rx_desc: pointer to the EOP Rx descriptor
 * @skb: pointer to current skb being fixed
 *
+ * Check if the skb is valid in the XDP case it will be an error pointer.
+ * Return true in this case to abort processing and advance to next
+ * descriptor.
+ *
 * Check for corrupted packet headers caused by senders on the local L2
 * embedded NIC switch not setting up their Tx Descriptors right.  These

* should be very rare.

drivers/net/ethernet/intel/ixgbe/ixgbe_main.c +4 -0

@@ -1876,6 +1883,10 @@ static bool ixgbe_cleanup_headers(struct ixgbe_ring
 *rx_ring,
 {
 	struct net_device *netdev = rx_ring->netdev;
 
+	/* XDP packets use error pointer so abort at this point */
+	if (IS_ERR(skb))
+		return true;
+
 	/* verify that the packet does not have any known errors */
 	if (unlikely(ixgbe_test_staterr(rx_desc,

					IXGBE_RXDADV_ERR_FRAME_ERR_MASK) &&

drivers/net/ethernet/intel/ixgbe/ixgbe_main.c +1 -1

@@ -2051,7 +2062,7 @@ static void ixgbe_put_rx_buffer(struct ixgbe_ring
 *rx_ring,
 		/* hand second half of page back to the ring */
 		ixgbe_reuse_rx_page(rx_ring, rx_buffer);
 	} else {
-		if (IXGBE_CB(skb)->dma == rx_buffer->dma) {
+		if (!IS_ERR(skb) && IXGBE_CB(skb)->dma == rx_buffer->dma) {
 			/* the page has been released from the ring */
 			IXGBE_CB(skb)->page_released = true;

		} else {

drivers/net/ethernet/intel/ixgbe/ixgbe_main.c +3 -3

@@ -2072,10 +2083,10 @@ static void ixgbe_put_rx_buffer(struct ixgbe_ring
 *rx_ring,
 
 static struct sk_buff *ixgbe_construct_skb(struct ixgbe_ring *rx_ring,
 					   struct ixgbe_rx_buffer *rx_buffer,
-					   union ixgbe_adv_rx_desc *rx_desc,
-					   unsigned int size)
+					   struct xdp_buff *xdp,
+					   union ixgbe_adv_rx_desc *rx_desc)
 {
-	void *va = page_address(rx_buffer->page) + rx_buffer->page_offset;
+	unsigned int size = xdp->data_end - xdp->data;
 #if (PAGE_SIZE < 8192)
 	unsigned int truesize = ixgbe_rx_pg_size(rx_ring) / 2;

#else

drivers/net/ethernet/intel/ixgbe/ixgbe_main.c +2 -2

@@ -2084,9 +2095,9 @@ static struct sk_buff *ixgbe_construct_skb(struct
 ixgbe_ring *rx_ring,
 	struct sk_buff *skb;
 
 	/* prefetch first cache line of first page */
-	prefetch(va);
+	prefetch(xdp->data);
 #if L1_CACHE_BYTES < 128
-	prefetch(va + L1_CACHE_BYTES);
+	prefetch(xdp->data + L1_CACHE_BYTES);
 #endif
 

	/* allocate a skb to store the frags */

drivers/net/ethernet/intel/ixgbe/ixgbe_main.c +1 -1

@@ -2099,7 +2110,7 @@ static struct sk_buff *ixgbe_construct_skb(struct
 ixgbe_ring *rx_ring,
 			IXGBE_CB(skb)->dma = rx_buffer->dma;
 
 		skb_add_rx_frag(skb, 0, rx_buffer->page,
-				rx_buffer->page_offset,
+				xdp->data - page_address(rx_buffer->page),
 				size, truesize);
 #if (PAGE_SIZE < 8192)

		rx_buffer->page_offset ^= truesize;

drivers/net/ethernet/intel/ixgbe/ixgbe_main.c +4 -4

@@ -2107,7 +2118,8 @@ static struct sk_buff *ixgbe_construct_skb(struct
 ixgbe_ring *rx_ring,
 		rx_buffer->page_offset += truesize;
 #endif
 	} else {
-		memcpy(__skb_put(skb, size), va, ALIGN(size, sizeof(long)));
+		memcpy(__skb_put(skb, size),
+		       xdp->data, ALIGN(size, sizeof(long)));
 		rx_buffer->pagecnt_bias++;
 	}

@@ -2116,10 +2128,9 @@ static struct sk_buff *ixgbe_construct_skb(struct
 ixgbe_ring *rx_ring,
 
 static struct sk_buff *ixgbe_build_skb(struct ixgbe_ring *rx_ring,
 				       struct ixgbe_rx_buffer *rx_buffer,
-				       union ixgbe_adv_rx_desc *rx_desc,
-				       unsigned int size)
+				       struct xdp_buff *xdp,
+				       union ixgbe_adv_rx_desc *rx_desc)
 {
-	void *va = page_address(rx_buffer->page) + rx_buffer->page_offset;
 #if (PAGE_SIZE < 8192)
 	unsigned int truesize = ixgbe_rx_pg_size(rx_ring) / 2;

#else

-----Original Message-----
From: Intel-wired-lan [mailto:intel-wired-lan-bounces at lists.osuosl.org] On Behalf Of John Fastabend
Sent: Thursday, March 23, 2017 9:32 PM
To: john.fastabend at gmail.com; alexander.duyck at gmail.com
Cc: intel-wired-lan at lists.osuosl.org; u9012063 at gmail.com
Subject: [Intel-wired-lan] [PATCH v7 2/2] ixgbe: add support for XDP_TX action

Add support for XDP_TX action.

A couple design choices were made here. First I use a new ring pointer structure xdp_ring[] in the adapter struct instead of pushing the newly allocated xdp TX rings into the tx_ring[] structure. This means we have to duplicate loops around rings in places we want to initialize both TX rings and XDP rings.
But by making it explicit it is obvious when we are using XDP rings and when we are using TX rings. Further we don't have to do ring arithmatic which is error prone. As a proof point for doing this my first patches used only a single ring structure and introduced bugs in FCoE code and macvlan code paths.

Second I am aware this is not the most optimized version of this code possible. I want to get baseline support in using the most readable format possible and then once this series is included I will optimize the TX path in another series of patches.

Signed-off-by John Fastabend <[email protected]>

---
 drivers/net/ethernet/intel/ixgbe/ixgbe.h         |   19 +
 drivers/net/ethernet/intel/ixgbe/ixgbe_ethtool.c |   25 ++
 drivers/net/ethernet/intel/ixgbe/ixgbe_lib.c     |   78 +++++-
 drivers/net/ethernet/intel/ixgbe/ixgbe_main.c    |  279
+++++++++++++++++++---
 4 files changed, 350 insertions(+), 51 deletions(-)

Add support for XDP_TX action.

A couple design choices were made here. First I use a new ring pointer structure xdp_ring[] in the adapter struct instead of pushing the newly allocated xdp TX rings into the tx_ring[] structure. This means we have to duplicate loops around rings in places we want to initialize both TX rings and XDP rings. But by making it explicit it is obvious when we are using XDP rings and when we are using TX rings. Further we don't have to do ring arithmatic which is error prone. As a proof point for doing this my first patches used only a single ring structure and introduced bugs in FCoE code and macvlan code paths.

Second I am aware this is not the most optimized version of this code possible. I want to get baseline support in using the most readable format possible and then once this series is included I will optimize the TX path in another series of patches.

Signed-off-by John Fastabend <[email protected]>

---
 drivers/net/ethernet/intel/ixgbe/ixgbe.h         |   19 +
 drivers/net/ethernet/intel/ixgbe/ixgbe_ethtool.c |   25 ++
 drivers/net/ethernet/intel/ixgbe/ixgbe_lib.c     |   78 +++++-
 drivers/net/ethernet/intel/ixgbe/ixgbe_main.c    |  279 +++++++++++++++++++---
 4 files changed, 350 insertions(+), 51 deletions(-)

This version fixes a couple issues with q_vector allocation and query on XDP program.

---

John Fastabend (2):
ixgbe: add XDP support for pass and drop actions
ixgbe: add support for XDP_TX action

 drivers/net/ethernet/intel/ixgbe/ixgbe.h         |   23 +
 drivers/net/ethernet/intel/ixgbe/ixgbe_ethtool.c |   29 +
 drivers/net/ethernet/intel/ixgbe/ixgbe_lib.c     |   78 +++-
 drivers/net/ethernet/intel/ixgbe/ixgbe_main.c    |  434 +++++++++++++++++++---
 4 files changed, 490 insertions(+), 74 deletions(-)

--
Signature

Quoted 10 lines · John Fastabend

This version fixes a couple issues with q_vector allocation and query on XDP program.

---

John Fastabend (2):
ixgbe: add XDP support for pass and drop actions
ixgbe: add support for XDP_TX action

Quoted 8 lines

 drivers/net/ethernet/intel/ixgbe/ixgbe.h         |   23 +
 drivers/net/ethernet/intel/ixgbe/ixgbe_ethtool.c |   29 +
 drivers/net/ethernet/intel/ixgbe/ixgbe_lib.c     |   78 +++-
 drivers/net/ethernet/intel/ixgbe/ixgbe_main.c    |  434 +++++++++++++++++++---
 4 files changed, 490 insertions(+), 74 deletions(-)

--
Signature

Thanks. I tested using samples/bpf/xdp1 and xdp2 using ping and dpdk pktgen. Both work OK.

When enabling xdp, dmesg shows:

[12270.610462] pps pps0: new PPS source ptp6
[12270.610465] ixgbe 0000:05:00.0: registered PHC device on enp5s0f0
[12271.619018] ixgbe 0000:05:00.0: removed PHC on enp5s0f0
[12271.943334] ixgbe 0000:05:00.0: Multiqueue Enabled: Rx Queue count
= 16, Tx Queue count = 16 XDP Queue count = 0
[12272.011324] pps pps0: new PPS source ptp6
[12272.011326] ixgbe 0000:05:00.0: registered PHC device on enp5s0f0
[12276.477066] ixgbe 0000:05:00.0 enp5s0f0: NIC Link is Up 10 Gbps,

Flow Control: None

I'm not sure it's important or not.
Regards,
William

Quoted 33 lines · William Tu

Quoted 10 lines · John Fastabend

This version fixes a couple issues with q_vector allocation and query on XDP program.

---

John Fastabend (2):
ixgbe: add XDP support for pass and drop actions
ixgbe: add support for XDP_TX action

Quoted 8 lines

 drivers/net/ethernet/intel/ixgbe/ixgbe.h         |   23 +
 drivers/net/ethernet/intel/ixgbe/ixgbe_ethtool.c |   29 +
 drivers/net/ethernet/intel/ixgbe/ixgbe_lib.c     |   78 +++-
 drivers/net/ethernet/intel/ixgbe/ixgbe_main.c    |  434 +++++++++++++++++++---
 4 files changed, 490 insertions(+), 74 deletions(-)

--
Signature

Thanks. I tested using samples/bpf/xdp1 and xdp2 using ping and dpdk pktgen. Both work OK.

When enabling xdp, dmesg shows:

[12270.610462] pps pps0: new PPS source ptp6
[12270.610465] ixgbe 0000:05:00.0: registered PHC device on enp5s0f0
[12271.619018] ixgbe 0000:05:00.0: removed PHC on enp5s0f0
[12271.943334] ixgbe 0000:05:00.0: Multiqueue Enabled: Rx Queue count
= 16, Tx Queue count = 16 XDP Queue count = 0
[12272.011324] pps pps0: new PPS source ptp6
[12272.011326] ixgbe 0000:05:00.0: registered PHC device on enp5s0f0
[12276.477066] ixgbe 0000:05:00.0 enp5s0f0: NIC Link is Up 10 Gbps,

Quoted 6 lines

Flow Control: None

I'm not sure it's important or not.
Regards,
William

Well that is a bit odd. When you load the XDP program we should see a "XDP Queue count" equal to the cpu count from /proc/cpuinfo. But if this is from either after unloading a XDP program or from initial loading the module before an XDP program is loaded than zero is expected. The queues are only added after the XDP program is loaded.

Thanks,
John