Skip to contentSkip to conversation

xen networking: add XDP support to xen-netfront

PATCH Superseded Denis Kirjanov 4 participants 12 messages Jun 22, 2020

Subsystems (4)

v10 of 14 Superseded Latest is v14 →

lore
Share & tools
lore
#0 lore raw

v10 cover xen networking: add XDP support to xen-netfront

The first patch adds a new extra type to enable proper synchronization between an RX request/response pair.
The second patch implements BFP interface for xen-netfront. The third patch enables extra space for XDP processing.

v10:
- add a new xen_netif_extra_info type to enable proper synchronization between an RX request/response pair.
- order local variable declarations

v9:
- assign an xdp program before switching to Reconfiguring
- minor cleanups
- address checkpatch issues

v8:
- add PAGE_POOL config dependency
- keep the state of XDP processing in netfront_xdp_enabled
- fixed allocator type in xdp_rxq_info_reg_mem_model()
- minor cleanups in xen-netback

v7:
- use page_pool_dev_alloc_pages() on page allocation
- remove the leftover break statement from netback_changed

v6:
- added the missing SOB line
- fixed subject

v5:
- split netfront/netback changes
- added a sync point between backend/frontend on switching to XDP
- added pagepool API

v4:
- added verbose patch descriprion
- don't expose the XDP headroom offset to the domU guest
- add a modparam to netback to toggle XDP offset
- don't process jumbo frames for now

v3:
- added XDP_TX support (tested with xdping echoserver)
- added XDP_REDIRECT support (tested with modified xdp_redirect_kern)
- moved xdp negotiation to xen-netback

v2:
- avoid data copying while passing to XDP
- tell xen-netback that we need the headroom space

Denis Kirjanov (3):
xen: netif.h: add a new extra type for XDP
xen networking: add basic XDP support for xen-netfront xen networking: add XDP offset adjustment to xen-netback

 drivers/net/Kconfig                 |   1 +
 drivers/net/xen-netback/common.h    |   4 +
 drivers/net/xen-netback/interface.c |   2 +
 drivers/net/xen-netback/netback.c   |   7 +
 drivers/net/xen-netback/rx.c        |  15 +-
 drivers/net/xen-netback/xenbus.c    |  32 ++++
 drivers/net/xen-netfront.c          | 332 ++++++++++++++++++++++++++++++++++--
 include/xen/interface/io/netif.h    |  18 +-
 8 files changed, 399 insertions(+), 12 deletions(-)
Signature · 1 line
1.8.3.1
#1 lore raw

PATCH v10 1/3 xen: netif.h: add a new extra type for XDP

The patch adds a new extra type to be able to diffirentiate between RX responses on xen-netfront side with the adjusted offset required for XDP processing.

The offset value from a guest is passed via xenstore.

---
 include/xen/interface/io/netif.h | 18 +++++++++++++++++-
 1 file changed, 17 insertions(+), 1 deletion(-)
include/xen/interface/io/netif.h +17 -1
index 4f20dbc..c35a5165 100644
@@ -161,6 +161,17 @@
  */
 
 /*
+ * "xdp-headroom" is used to request that extra space is added
+ * for XDP processing.  The value is measured in bytes and passed by
+ * the frontend to be consistent between both ends.
+ * If the value is greater than zero that means that
+ * an RX response is going to be passed to an XDP program for processing.
+ *
+ * "feature-xdp-headroom" is set to "1" by the netback side like other features
+ * so a guest can check if an XDP program can be processed.
+ */
+
+/*
  * Control ring
  * ============
  *
@@ -846,7 +857,8 @@ struct xen_netif_tx_request {
 #define XEN_NETIF_EXTRA_TYPE_MCAST_ADD (2)	/* u.mcast */
 #define XEN_NETIF_EXTRA_TYPE_MCAST_DEL (3)	/* u.mcast */
 #define XEN_NETIF_EXTRA_TYPE_HASH      (4)	/* u.hash */
-#define XEN_NETIF_EXTRA_TYPE_MAX       (5)
+#define XEN_NETIF_EXTRA_TYPE_XDP       (5)	/* u.xdp */
+#define XEN_NETIF_EXTRA_TYPE_MAX       (6)
 
 /* xen_netif_extra_info_t flags. */
 #define _XEN_NETIF_EXTRA_FLAG_MORE (0)
@@ -879,6 +891,10 @@ struct xen_netif_extra_info {
 			uint8_t algorithm;
 			uint8_t value[4];
 		} hash;
+		struct {
+			uint16_t headroom;
+			uint16_t pad[2];
+		} xdp;
 		uint16_t pad[3];
 	} u;
 };
Signature · 1 line
1.8.3.1
#2 lore raw

PATCH v10 2/3 xen networking: add basic XDP support for xen-netfront

In reply to Denis Kirjanov

The patch adds a basic XDP processing to xen-netfront driver.

We ran an XDP program for an RX response received from netback driver. Also we request xen-netback to adjust data offset for bpf_xdp_adjust_head() header space for custom headers.

synchronization between frontend and backend parts is done by using xenbus state switching:
Reconfiguring -> Reconfigured- > Connected

UDP packets drop rate using xdp program is around 310 kpps using ./pktgen_sample04_many_flows.sh and 160 kpps without the patch.

---
 drivers/net/Kconfig        |   1 +
 drivers/net/xen-netfront.c | 332 +++++++++++++++++++++++++++++++++++++++++++--
 2 files changed, 323 insertions(+), 10 deletions(-)
drivers/net/Kconfig +1 -0
index 25a8f93..45918ce 100644
@@ -479,6 +479,7 @@ config XEN_NETDEV_FRONTEND
 	tristate "Xen network device frontend driver"
 	depends on XEN
 	select XEN_XENBUS_FRONTEND
+	select PAGE_POOL
 	default y
 	help
 	  This driver provides support for Xen paravirtual network
drivers/net/xen-netfront.c +322 -10

459 lines not on this page · raw message

Signature · 1 line
1.8.3.1
#8 lore raw

PATCH v10 3/3 xen networking: add XDP offset adjustment to xen-netback

In reply to Denis Kirjanov

the patch basically adds the offset adjustment and netfront state reading to make XDP work on netfront side.

---
 drivers/net/xen-netback/common.h    |  4 ++++
 drivers/net/xen-netback/interface.c |  2 ++
 drivers/net/xen-netback/netback.c   |  7 +++++++
 drivers/net/xen-netback/rx.c        | 15 ++++++++++++++-
 drivers/net/xen-netback/xenbus.c    | 32 ++++++++++++++++++++++++++++++++
 5 files changed, 59 insertions(+), 1 deletion(-)
drivers/net/xen-netback/common.h +4 -0
index 05847eb..f14dc10 100644
@@ -281,6 +281,9 @@ struct xenvif {
 	u8 ipv6_csum:1;
 	u8 multicast_control:1;
 
+	/* headroom requested by xen-netfront */
+	u16 netfront_xdp_headroom;
+
 	/* Is this interface disabled? True when backend discovers
 	 * frontend is rogue.
 	 */
@@ -395,6 +398,7 @@ static inline pending_ring_idx_t nr_pending_reqs(struct xenvif_queue *queue)
 irqreturn_t xenvif_interrupt(int irq, void *dev_id);
 
 extern bool separate_tx_rx_irq;
+extern bool provides_xdp_headroom;
 
 extern unsigned int rx_drain_timeout_msecs;
 extern unsigned int rx_stall_timeout_msecs;
drivers/net/xen-netback/interface.c +2 -0
index 0c8a02a..fc16edd 100644
@@ -483,6 +483,8 @@ struct xenvif *xenvif_alloc(struct device *parent, domid_t domid,
 	vif->queues = NULL;
 	vif->num_queues = 0;
 
+	vif->netfront_xdp_headroom = 0;
+
 	spin_lock_init(&vif->lock);
 	INIT_LIST_HEAD(&vif->fe_mcast_addr);
 
drivers/net/xen-netback/netback.c +7 -0
index 315dfc6..6dfca72 100644
@@ -96,6 +96,13 @@
 module_param_named(hash_cache_size, xenvif_hash_cache_size, uint, 0644);
 MODULE_PARM_DESC(hash_cache_size, "Number of flows in the hash cache");
 
+/* The module parameter tells that we have to put data
+ * for xen-netfront with the XDP_PACKET_HEADROOM offset
+ * needed for XDP processing
+ */
+bool provides_xdp_headroom = true;
+module_param(provides_xdp_headroom, bool, 0644);
+
 static void xenvif_idx_release(struct xenvif_queue *queue, u16 pending_idx,
 			       u8 status);
 
drivers/net/xen-netback/rx.c +14 -1
index ef58870..c5e9e14 100644
@@ -258,6 +258,19 @@ static void xenvif_rx_next_skb(struct xenvif_queue *queue,
 		pkt->extra_count++;
 	}
 
+	if (queue->vif->netfront_xdp_headroom) {
+		struct xen_netif_extra_info *extra;
+
+		extra = &pkt->extras[XEN_NETIF_EXTRA_TYPE_XDP - 1];
+
+		memset(extra, 0, sizeof(struct xen_netif_extra_info));
+		extra->u.xdp.headroom = queue->vif->netfront_xdp_headroom;
+		extra->type = XEN_NETIF_EXTRA_TYPE_XDP;
+		extra->flags = 0;
+
+		pkt->extra_count++;
+	}
+
 	if (skb->sw_hash) {
 		struct xen_netif_extra_info *extra;
 
@@ -356,7 +369,7 @@ static void xenvif_rx_data_slot(struct xenvif_queue *queue,
 				struct xen_netif_rx_request *req,
 				struct xen_netif_rx_response *rsp)
 {
-	unsigned int offset = 0;
+	unsigned int offset = queue->vif->netfront_xdp_headroom;
 	unsigned int flags;
 
 	do {
drivers/net/xen-netback/xenbus.c +32 -0
index 286054b..c67abc5 100644
@@ -393,6 +393,22 @@ static void set_backend_state(struct backend_info *be,
 	}
 }
 
+static void read_xenbus_frontend_xdp(struct backend_info *be,
+				      struct xenbus_device *dev)
+{
+	struct xenvif *vif = be->vif;
+	u16 headroom;
+	int err;
+
+	err = xenbus_scanf(XBT_NIL, dev->otherend,
+			   "netfront-xdp-headroom", "%hu", &headroom);
+	if (err < 0) {
+		vif->netfront_xdp_headroom = 0;
+		return;
+	}
+	vif->netfront_xdp_headroom = headroom;
+}
+
 /**
  * Callback received when the frontend's state changes.
  */
@@ -417,6 +433,11 @@ static void frontend_changed(struct xenbus_device *dev,
 		set_backend_state(be, XenbusStateConnected);
 		break;
 
+	case XenbusStateReconfiguring:
+		read_xenbus_frontend_xdp(be, dev);
+		xenbus_switch_state(dev, XenbusStateReconfigured);
+		break;
+
 	case XenbusStateClosing:
 		set_backend_state(be, XenbusStateClosing);
 		break;
@@ -947,6 +968,8 @@ static int read_xenbus_vif_flags(struct backend_info *be)
 	vif->ipv6_csum = !!xenbus_read_unsigned(dev->otherend,
 						"feature-ipv6-csum-offload", 0);
 
+	read_xenbus_frontend_xdp(be, dev);
+
 	return 0;
 }
 
@@ -1036,6 +1059,15 @@ static int netback_probe(struct xenbus_device *dev,
 			goto abort_transaction;
 		}
 
+		/* we can adjust a headroom for netfront XDP processing */
+		err = xenbus_printf(xbt, dev->nodename,
+				    "feature-xdp-headroom", "%d",
+				    provides_xdp_headroom);
+		if (err) {
+			message = "writing feature-xdp-headroom";
+			goto abort_transaction;
+		}
+
 		/* We don't support rx-flip path (except old guests who
 		 * don't grok this feature flag).
 		 */
Signature · 1 line
1.8.3.1
#3 lore raw

Re: [PATCH net-next v10 2/3] xen networking: add basic XDP support for xen-netfront

In reply to Denis Kirjanov
Quoted 4 lines · On Mon, 22 Jun 2020 12:21:11 +0300 Denis Kirjanov <[email protected]> wrote:
drivers/net/xen-netfront.c +0 -0
index 482c6c8..1b9f49e 100644

[...]

Quoted 7 lines
drivers/net/xen-netfront.c +3 -0
@@ -560,6 +572,65 @@ static u16 xennet_select_queue(struct net_device *dev, struct sk_buff *skb,
 	return queue_idx;
 }
 
+static int xennet_xdp_xmit_one(struct net_device *dev, struct xdp_frame *xdpf)
+{
+	struct netfront_info *np = netdev_priv(dev);

drivers/net/xen-netfront.c static u16 xennet_select_queue(struct net_device *dev, struct sk_buff *skb, · line 587

Quoted 10 lines
drivers/net/xen-netfront.c +10 -0
+	struct netfront_stats *tx_stats = this_cpu_ptr(np->tx_stats);
+	unsigned int num_queues = dev->real_num_tx_queues;
+	struct netfront_queue *queue = NULL;
+	struct xen_netif_tx_request *tx;
+	unsigned long flags;
+	int notify;
+
+	queue = &np->queues[smp_processor_id() % num_queues];
+
+	spin_lock_irqsave(&queue->tx_lock, flags);

Why are you taking a lock per packet (xdp_frame)?

Quoted 8 lines
drivers/net/xen-netfront.c +8 -0
+
+	tx = xennet_make_first_txreq(queue, NULL,
+				     virt_to_page(xdpf->data),
+				     offset_in_page(xdpf->data),
+				     xdpf->len);
+
+	RING_PUSH_REQUESTS_AND_CHECK_NOTIFY(&queue->tx, notify);
+	if (notify)

drivers/net/xen-netfront.c static u16 xennet_select_queue(struct net_device *dev, struct sk_buff *skb, · line 605

Quoted 10 lines
drivers/net/xen-netfront.c +10 -0
+		notify_remote_via_irq(queue->tx_irq);
+
+	u64_stats_update_begin(&tx_stats->syncp);
+	tx_stats->bytes += xdpf->len;
+	tx_stats->packets++;
+	u64_stats_update_end(&tx_stats->syncp);
+
+	xennet_tx_buf_gc(queue);
+
+	spin_unlock_irqrestore(&queue->tx_lock, flags);

Is the irqsave/irqrestore variant really needed here?

Quoted 26 lines
drivers/net/xen-netfront.c +26 -0
+	return 0;
+}
+
+static int xennet_xdp_xmit(struct net_device *dev, int n,
+			   struct xdp_frame **frames, u32 flags)
+{
+	int drops = 0;
+	int i, err;
+
+	if (unlikely(flags & ~XDP_XMIT_FLAGS_MASK))
+		return -EINVAL;
+
+	for (i = 0; i < n; i++) {
+		struct xdp_frame *xdpf = frames[i];
+
+		if (!xdpf)
+			continue;
+		err = xennet_xdp_xmit_one(dev, xdpf);
+		if (err) {
+			xdp_return_frame_rx_napi(xdpf);
+			drops++;
+		}
+	}
+
+	return n - drops;
+}
Signature · 4 lines
Best regards,
  Jesper Dangaard Brouer
  MSc.CS, Principal Kernel Engineer at Red Hat
  LinkedIn: http://www.linkedin.com/in/brouer
#9 lore raw

RE: [PATCH net-next v10 3/3] xen networking: add XDP offset adjustment to xen-netback

In reply to Denis Kirjanov
Quoted 46 lines

-----Original Message-----
From: Denis Kirjanov <[email protected]>
Sent: 22 June 2020 10:21
To: [email protected]
Cc: [email protected]; [email protected]; [email protected]; [email protected]; [email protected]
Subject: [PATCH net-next v10 3/3] xen networking: add XDP offset adjustment to xen-netback

the patch basically adds the offset adjustment and netfront state reading to make XDP work on netfront side.

---
 drivers/net/xen-netback/common.h    |  4 ++++
 drivers/net/xen-netback/interface.c |  2 ++
 drivers/net/xen-netback/netback.c   |  7 +++++++
 drivers/net/xen-netback/rx.c        | 15 ++++++++++++++-
 drivers/net/xen-netback/xenbus.c    | 32 ++++++++++++++++++++++++++++++++
 5 files changed, 59 insertions(+), 1 deletion(-)
drivers/net/xen-netback/common.h +4 -0
index 05847eb..f14dc10 100644
@@ -281,6 +281,9 @@ struct xenvif {
 	u8 ipv6_csum:1;
 	u8 multicast_control:1;
 
+	/* headroom requested by xen-netfront */
+	u16 netfront_xdp_headroom;
+
 	/* Is this interface disabled? True when backend discovers
 	 * frontend is rogue.
 	 */
@@ -395,6 +398,7 @@ static inline pending_ring_idx_t nr_pending_reqs(struct xenvif_queue *queue)
 irqreturn_t xenvif_interrupt(int irq, void *dev_id);
 
 extern bool separate_tx_rx_irq;
+extern bool provides_xdp_headroom;
 
 extern unsigned int rx_drain_timeout_msecs;
 extern unsigned int rx_stall_timeout_msecs;
drivers/net/xen-netback/interface.c +0 -0
index 0c8a02a..fc16edd 100644

drivers/net/xen-netback/interface.c struct xenvif *xenvif_alloc(struct device *parent, domid_t domid, · line 487

Quoted 5 lines
drivers/net/xen-netback/interface.c +2 -0
@@ -483,6 +483,8 @@ struct xenvif *xenvif_alloc(struct device *parent, domid_t domid,
 	vif->queues = NULL;
 	vif->num_queues = 0;
 
+	vif->netfront_xdp_headroom = 0;
+

How about just 'xdp_headroom'? It's shorter to type :-)

Quoted 61 lines
drivers/net/xen-netback/interface.c +0 -0
 	spin_lock_init(&vif->lock);
 	INIT_LIST_HEAD(&vif->fe_mcast_addr);
 
drivers/net/xen-netback/netback.c +7 -0
index 315dfc6..6dfca72 100644
@@ -96,6 +96,13 @@
 module_param_named(hash_cache_size, xenvif_hash_cache_size, uint, 0644);
 MODULE_PARM_DESC(hash_cache_size, "Number of flows in the hash cache");
 
+/* The module parameter tells that we have to put data
+ * for xen-netfront with the XDP_PACKET_HEADROOM offset
+ * needed for XDP processing
+ */
+bool provides_xdp_headroom = true;
+module_param(provides_xdp_headroom, bool, 0644);
+
 static void xenvif_idx_release(struct xenvif_queue *queue, u16 pending_idx,
 			       u8 status);
 
drivers/net/xen-netback/rx.c +14 -1
index ef58870..c5e9e14 100644
@@ -258,6 +258,19 @@ static void xenvif_rx_next_skb(struct xenvif_queue *queue,
 		pkt->extra_count++;
 	}
 
+	if (queue->vif->netfront_xdp_headroom) {
+		struct xen_netif_extra_info *extra;
+
+		extra = &pkt->extras[XEN_NETIF_EXTRA_TYPE_XDP - 1];
+
+		memset(extra, 0, sizeof(struct xen_netif_extra_info));
+		extra->u.xdp.headroom = queue->vif->netfront_xdp_headroom;
+		extra->type = XEN_NETIF_EXTRA_TYPE_XDP;
+		extra->flags = 0;
+
+		pkt->extra_count++;
+	}
+
 	if (skb->sw_hash) {
 		struct xen_netif_extra_info *extra;
 
@@ -356,7 +369,7 @@ static void xenvif_rx_data_slot(struct xenvif_queue *queue,
 				struct xen_netif_rx_request *req,
 				struct xen_netif_rx_response *rsp)
 {
-	unsigned int offset = 0;
+	unsigned int offset = queue->vif->netfront_xdp_headroom;
 	unsigned int flags;
 
 	do {
drivers/net/xen-netback/xenbus.c +0 -0
index 286054b..c67abc5 100644
@@ -393,6 +393,22 @@ static void set_backend_state(struct backend_info *be,
 	}
 }

drivers/net/xen-netback/xenbus.c static void set_backend_state(struct backend_info *be, · line 404

Quoted 10 lines
drivers/net/xen-netback/xenbus.c +9 -0
 
+static void read_xenbus_frontend_xdp(struct backend_info *be,
+				      struct xenbus_device *dev)
+{
+	struct xenvif *vif = be->vif;
+	u16 headroom;
+	int err;
+
+	err = xenbus_scanf(XBT_NIL, dev->otherend,
+			   "netfront-xdp-headroom", "%hu", &headroom);

Isn't it just "xdp-headroom"? That's what the comments in netif.h state.

drivers/net/xen-netback/xenbus.c static void set_backend_state(struct backend_info *be, · line 408

Quoted 4 lines
drivers/net/xen-netback/xenbus.c +4 -0
+	if (err < 0) {
+		vif->netfront_xdp_headroom = 0;
+		return;
+	}

What is a reasonable value for maximum headroom? Do we really want to allow values all the way up to 65535?

Paul

Quoted 45 lines
drivers/net/xen-netback/xenbus.c +19 -1
+	vif->netfront_xdp_headroom = headroom;
+}
+
 /**
  * Callback received when the frontend's state changes.
  */
@@ -417,6 +433,11 @@ static void frontend_changed(struct xenbus_device *dev,
 		set_backend_state(be, XenbusStateConnected);
 		break;
 
+	case XenbusStateReconfiguring:
+		read_xenbus_frontend_xdp(be, dev);
+		xenbus_switch_state(dev, XenbusStateReconfigured);
+		break;
+
 	case XenbusStateClosing:
 		set_backend_state(be, XenbusStateClosing);
 		break;
@@ -947,6 +968,8 @@ static int read_xenbus_vif_flags(struct backend_info *be)
 	vif->ipv6_csum = !!xenbus_read_unsigned(dev->otherend,
 						"feature-ipv6-csum-offload", 0);
 
+	read_xenbus_frontend_xdp(be, dev);
+
 	return 0;
 }
 
@@ -1036,6 +1059,15 @@ static int netback_probe(struct xenbus_device *dev,
 			goto abort_transaction;
 		}
 
+		/* we can adjust a headroom for netfront XDP processing */
+		err = xenbus_printf(xbt, dev->nodename,
+				    "feature-xdp-headroom", "%d",
+				    provides_xdp_headroom);
+		if (err) {
+			message = "writing feature-xdp-headroom";
+			goto abort_transaction;
+		}
+
 		/* We don't support rx-flip path (except old guests who
 		 * don't grok this feature flag).
 		 */
--

1.8.3.1

#7 lore raw

Re: [PATCH net-next v10 2/3] xen networking: add basic XDP support for xen-netfront

In reply to Denis Kirjanov
I love your patch! Yet something to improve:

Hi Denis,

I love your patch! Yet something to improve:

[auto build test ERROR on net-next/master]

url:    https://github.com/0day-ci/linux/commits/Denis-Kirjanov/xen-networking-add-XDP-support-to-xen-netfront/20200622-172512
base:   https://git.kernel.org/pub/scm/linux/kernel/git/davem/net-next.git 29a720c1042f469c8fea317cb5e7f496b116e07d
config: i386-debian-10.3 (attached as .config)
compiler: gcc-9 (Debian 9.3.0-13) 9.3.0
reproduce (this is a W=1 build):
        # save the attached .config to linux build tree
        make W=1 ARCH=i386 

If you fix the issue, kindly add following tag as appropriate

All error/warnings (new ones prefixed by >>):

drivers/net/xen-netfront.c: In function 'xennet_xdp_xmit_one':
drivers/net/xen-netfront.c:581:31: warning: variable 'tx' set but not used [-Wunused-but-set-variable]

     581 |  struct xen_netif_tx_request *tx;
         |                               ^~
   drivers/net/xen-netfront.c: In function 'xennet_run_xdp':
>> drivers/net/xen-netfront.c:871:10: error: implicit declaration of function 'convert_to_xdp_frame'; did you mean 'xdp_convert_zc_to_xdp_frame'? [-Werror=implicit-function-declaration]
     871 |   xdpf = convert_to_xdp_frame(xdp);
         |          ^~~~~~~~~~~~~~~~~~~~
         |          xdp_convert_zc_to_xdp_frame
>> drivers/net/xen-netfront.c:871:8: warning: assignment to 'struct xdp_frame *' from 'int' makes pointer from integer without a cast [-Wint-conversion]
     871 |   xdpf = convert_to_xdp_frame(xdp);
         |        ^
   drivers/net/xen-netfront.c: In function 'store_rxbuf':
   drivers/net/xen-netfront.c:2424:16: warning: variable 'target' set but not used [-Wunused-but-set-variable]
    2424 |  unsigned long target;
         |                ^~~~~~

cc1: some warnings being treated as errors

vim +871 drivers/net/xen-netfront.c

851
852 static u32 xennet_run_xdp(struct netfront_queue *queue, struct page *pdata,
853 struct xen_netif_rx_response *rx, struct bpf_prog *prog,
854 struct xdp_buff *xdp, bool *need_xdp_flush)
855 {
856 struct xdp_frame *xdpf;
857 u32 len = rx->status;
858 u32 act = XDP_PASS;
859 int err;
860
861 xdp->data_hard_start = page_address(pdata);
862 xdp->data = xdp->data_hard_start + XDP_PACKET_HEADROOM;
863 xdp_set_data_meta_invalid(xdp);
864 xdp->data_end = xdp->data + len;
865 xdp->rxq = &queue->xdp_rxq;
866
867 act = bpf_prog_run_xdp(prog, xdp);
868 switch (act) {
869 case XDP_TX:
870 get_page(pdata);

 > 871			xdpf = convert_to_xdp_frame(xdp);

872 err = xennet_xdp_xmit(queue->info->netdev, 1, &xdpf, 0);
873 if (unlikely(err < 0))
874 trace_xdp_exception(queue->info->netdev, prog, act);
875 break;
876 case XDP_REDIRECT:
877 get_page(pdata);
878 err = xdp_do_redirect(queue->info->netdev, xdp, prog);
879 *need_xdp_flush = true;
880 if (unlikely(err))
881 trace_xdp_exception(queue->info->netdev, prog, act);
882 break;
883 case XDP_PASS:
884 case XDP_DROP:
885 break;
886
887 case XDP_ABORTED:
888 trace_xdp_exception(queue->info->netdev, prog, act);
889 break;
890
891 default:
892 bpf_warn_invalid_xdp_action(act);
893 }
894
895 return act;
896 }
897

---
0-DAY CI Kernel Test Service, Intel Corporation
https://lists.01.org/hyperkitty/list/kbuild-all(a)lists.01.org

#4 lore raw

Re: [PATCH net-next v10 2/3] xen networking: add basic XDP support for xen-netfront

In reply to Jesper Dangaard Brouer
Quoted 14 lines · Jesper Dangaard Brouer
Quoted 4 lines · On Mon, 22 Jun 2020 12:21:11 +0300 Denis Kirjanov <[email protected]> wrote:
drivers/net/xen-netfront.c +0 -0
index 482c6c8..1b9f49e 100644

[...]

Quoted 5 lines
drivers/net/xen-netfront.c +0 -0
@@ -560,6 +572,65 @@ static u16 xennet_select_queue(struct net_device
 *dev, struct sk_buff *skb,
 	return queue_idx;
 }
 
Quoted 16 lines
Quoted 4 lines
drivers/net/xen-netfront.c +3 -0
+static int xennet_xdp_xmit_one(struct net_device *dev, struct xdp_frame
 *xdpf)
+{
+	struct netfront_info *np = netdev_priv(dev);

drivers/net/xen-netfront.c static u16 xennet_select_queue(struct net_device *dev, struct sk_buff *skb, · line 587

Quoted 10 lines
drivers/net/xen-netfront.c +10 -0
+	struct netfront_stats *tx_stats = this_cpu_ptr(np->tx_stats);
+	unsigned int num_queues = dev->real_num_tx_queues;
+	struct netfront_queue *queue = NULL;
+	struct xen_netif_tx_request *tx;
+	unsigned long flags;
+	int notify;
+
+	queue = &np->queues[smp_processor_id() % num_queues];
+
+	spin_lock_irqsave(&queue->tx_lock, flags);

Why are you taking a lock per packet (xdp_frame)?

Hi Jesper,

We have to protect shared ring indices.

Quoted 13 lines
Quoted 13 lines
drivers/net/xen-netfront.c +12 -0
+
+	tx = xennet_make_first_txreq(queue, NULL,
+				     virt_to_page(xdpf->data),
+				     offset_in_page(xdpf->data),
+				     xdpf->len);
+
+	RING_PUSH_REQUESTS_AND_CHECK_NOTIFY(&queue->tx, notify);
+	if (notify)
+		notify_remote_via_irq(queue->tx_irq);
+
+	u64_stats_update_begin(&tx_stats->syncp);
+	tx_stats->bytes += xdpf->len;
Quoted 8 lines

drivers/net/xen-netfront.c static u16 xennet_select_queue(struct net_device *dev, struct sk_buff *skb, · line 605

Quoted 6 lines
drivers/net/xen-netfront.c +6 -0
+	tx_stats->packets++;
+	u64_stats_update_end(&tx_stats->syncp);
+
+	xennet_tx_buf_gc(queue);
+
+	spin_unlock_irqrestore(&queue->tx_lock, flags);

Is the irqsave/irqrestore variant really needed here?

netpoll also invokes the tx completion handler.

Quoted 37 lines
Quoted 17 lines
drivers/net/xen-netfront.c +16 -0
+	return 0;
+}
+
+static int xennet_xdp_xmit(struct net_device *dev, int n,
+			   struct xdp_frame **frames, u32 flags)
+{
+	int drops = 0;
+	int i, err;
+
+	if (unlikely(flags & ~XDP_XMIT_FLAGS_MASK))
+		return -EINVAL;
+
+	for (i = 0; i < n; i++) {
+		struct xdp_frame *xdpf = frames[i];
+
+		if (!xdpf)

drivers/net/xen-netfront.c static u16 xennet_select_queue(struct net_device *dev, struct sk_buff *skb, · line 631

Quoted 10 lines
drivers/net/xen-netfront.c +10 -0
+			continue;
+		err = xennet_xdp_xmit_one(dev, xdpf);
+		if (err) {
+			xdp_return_frame_rx_napi(xdpf);
+			drops++;
+		}
+	}
+
+	return n - drops;
+}

--
Best regards,
Jesper Dangaard Brouer
MSc.CS, Principal Kernel Engineer at Red Hat
LinkedIn: http://www.linkedin.com/in/brouer

#10 lore raw

Re: [PATCH net-next v10 3/3] xen networking: add XDP offset adjustment to xen-netback

In reply to Paul Durrant
Quoted 56 lines · Paul Durrant
Quoted 56 lines

-----Original Message-----
From: Denis Kirjanov <[email protected]>
Sent: 22 June 2020 10:21
To: [email protected]
Cc: [email protected]; [email protected]; [email protected]; [email protected]; [email protected]
Subject: [PATCH net-next v10 3/3] xen networking: add XDP offset adjustment to xen-netback

the patch basically adds the offset adjustment and netfront state reading to make XDP work on netfront side.

---
 drivers/net/xen-netback/common.h    |  4 ++++
 drivers/net/xen-netback/interface.c |  2 ++
 drivers/net/xen-netback/netback.c   |  7 +++++++
 drivers/net/xen-netback/rx.c        | 15 ++++++++++++++-
 drivers/net/xen-netback/xenbus.c    | 32
++++++++++++++++++++++++++++++++
 5 files changed, 59 insertions(+), 1 deletion(-)
drivers/net/xen-netback/common.h +4 -0
index 05847eb..f14dc10 100644
@@ -281,6 +281,9 @@ struct xenvif {
 	u8 ipv6_csum:1;
 	u8 multicast_control:1;
 
+	/* headroom requested by xen-netfront */
+	u16 netfront_xdp_headroom;
+
 	/* Is this interface disabled? True when backend discovers
 	 * frontend is rogue.
 	 */
@@ -395,6 +398,7 @@ static inline pending_ring_idx_t
 nr_pending_reqs(struct xenvif_queue *queue)
 irqreturn_t xenvif_interrupt(int irq, void *dev_id);
 
 extern bool separate_tx_rx_irq;
+extern bool provides_xdp_headroom;
 
 extern unsigned int rx_drain_timeout_msecs;

extern unsigned int rx_stall_timeout_msecs;

drivers/net/xen-netback/interface.c +0 -0
index 0c8a02a..fc16edd 100644
@@ -483,6 +483,8 @@ struct xenvif *xenvif_alloc(struct device *parent,
 domid_t domid,
 	vif->queues = NULL;
 	vif->num_queues = 0;
 
Quoted 3 lines

drivers/net/xen-netback/interface.c struct xenvif *xenvif_alloc(struct device *parent, · line 488

Quoted 2 lines
drivers/net/xen-netback/interface.c +2 -0
+	vif->netfront_xdp_headroom = 0;
+

Hi Paul,

Quoted 1 line

How about just 'xdp_headroom'? It's shorter to type :-)

makes sense.

Quoted 81 lines
Quoted 68 lines
drivers/net/xen-netback/interface.c +0 -0
 	spin_lock_init(&vif->lock);
 	INIT_LIST_HEAD(&vif->fe_mcast_addr);
drivers/net/xen-netback/netback.c +7 -0
index 315dfc6..6dfca72 100644
@@ -96,6 +96,13 @@
 module_param_named(hash_cache_size, xenvif_hash_cache_size, uint, 0644);
 MODULE_PARM_DESC(hash_cache_size, "Number of flows in the hash cache");
 
+/* The module parameter tells that we have to put data
+ * for xen-netfront with the XDP_PACKET_HEADROOM offset
+ * needed for XDP processing
+ */
+bool provides_xdp_headroom = true;
+module_param(provides_xdp_headroom, bool, 0644);
+
 static void xenvif_idx_release(struct xenvif_queue *queue, u16
 pending_idx,
 			       u8 status);
drivers/net/xen-netback/rx.c +14 -1
index ef58870..c5e9e14 100644
@@ -258,6 +258,19 @@ static void xenvif_rx_next_skb(struct xenvif_queue
 *queue,
 		pkt->extra_count++;
 	}
 
+	if (queue->vif->netfront_xdp_headroom) {
+		struct xen_netif_extra_info *extra;
+
+		extra = &pkt->extras[XEN_NETIF_EXTRA_TYPE_XDP - 1];
+
+		memset(extra, 0, sizeof(struct xen_netif_extra_info));
+		extra->u.xdp.headroom = queue->vif->netfront_xdp_headroom;
+		extra->type = XEN_NETIF_EXTRA_TYPE_XDP;
+		extra->flags = 0;
+
+		pkt->extra_count++;
+	}
+
 	if (skb->sw_hash) {
 		struct xen_netif_extra_info *extra;
@@ -356,7 +369,7 @@ static void xenvif_rx_data_slot(struct xenvif_queue
 *queue,
 				struct xen_netif_rx_request *req,
 				struct xen_netif_rx_response *rsp)
 {
-	unsigned int offset = 0;
+	unsigned int offset = queue->vif->netfront_xdp_headroom;
 	unsigned int flags;
 
drivers/net/xen-netback/rx.c +0 -0
 	do {
drivers/net/xen-netback/xenbus.c +0 -0
index 286054b..c67abc5 100644
@@ -393,6 +393,22 @@ static void set_backend_state(struct backend_info
 *be,
 	}
 }

drivers/net/xen-netback/xenbus.c static void set_backend_state(struct backend_info · line 405

Quoted 10 lines
drivers/net/xen-netback/xenbus.c +9 -0
 
+static void read_xenbus_frontend_xdp(struct backend_info *be,
+				      struct xenbus_device *dev)
+{
+	struct xenvif *vif = be->vif;
+	u16 headroom;
+	int err;
+
+	err = xenbus_scanf(XBT_NIL, dev->otherend,
+			   "netfront-xdp-headroom", "%hu", &headroom);

Isn't it just "xdp-headroom"? That's what the comments in netif.h state.

Quoted 7 lines

drivers/net/xen-netback/xenbus.c static void set_backend_state(struct backend_info *be, · line 408

Quoted 4 lines
drivers/net/xen-netback/xenbus.c +4 -0
+	if (err < 0) {
+		vif->netfront_xdp_headroom = 0;
+		return;
+	}

What is a reasonable value for maximum headroom? Do we really want to allow values all the way up to 65535?

Since the headroom is used for encapsulation I think we definitely don't need more than 65535
but more that 255

Quoted 54 lines

Paul

Quoted 48 lines
drivers/net/xen-netback/xenbus.c +3 -0
+	vif->netfront_xdp_headroom = headroom;
+}
+
 /**
  * Callback received when the frontend's state changes.

*/

drivers/net/xen-netback/xenbus.c +5 -0
@@ -417,6 +433,11 @@ static void frontend_changed(struct xenbus_device
 *dev,
 		set_backend_state(be, XenbusStateConnected);
 		break;
 
+	case XenbusStateReconfiguring:
+		read_xenbus_frontend_xdp(be, dev);
+		xenbus_switch_state(dev, XenbusStateReconfigured);
+		break;
+
 	case XenbusStateClosing:
 		set_backend_state(be, XenbusStateClosing);
drivers/net/xen-netback/xenbus.c +11 -0
 		break;
@@ -947,6 +968,8 @@ static int read_xenbus_vif_flags(struct backend_info
 *be)
 	vif->ipv6_csum = !!xenbus_read_unsigned(dev->otherend,
 						"feature-ipv6-csum-offload", 0);
 
+	read_xenbus_frontend_xdp(be, dev);
+
 	return 0;
 }
@@ -1036,6 +1059,15 @@ static int netback_probe(struct xenbus_device
 *dev,
 			goto abort_transaction;
 		}
 
+		/* we can adjust a headroom for netfront XDP processing */
+		err = xenbus_printf(xbt, dev->nodename,
+				    "feature-xdp-headroom", "%d",
+				    provides_xdp_headroom);
+		if (err) {
+			message = "writing feature-xdp-headroom";
+			goto abort_transaction;
+		}
+
 		/* We don't support rx-flip path (except old guests who
 		 * don't grok this feature flag).

*/
--
1.8.3.1

#5 lore raw

Re: [PATCH net-next v10 2/3] xen networking: add basic XDP support for xen-netfront

In reply to Denis Kirjanov
Quoted 31 lines · On Mon, 22 Jun 2020 15:45:46 +0300 Denis Kirjanov <[email protected]> wrote:
Quoted 14 lines · Jesper Dangaard Brouer
Quoted 4 lines · On Mon, 22 Jun 2020 12:21:11 +0300 Denis Kirjanov <[email protected]> wrote:
drivers/net/xen-netfront.c +0 -0
index 482c6c8..1b9f49e 100644

[...]

Quoted 5 lines
drivers/net/xen-netfront.c +0 -0
@@ -560,6 +572,65 @@ static u16 xennet_select_queue(struct net_device
 *dev, struct sk_buff *skb,
 	return queue_idx;
 }
 
Quoted 16 lines
Quoted 4 lines
drivers/net/xen-netfront.c +3 -0
+static int xennet_xdp_xmit_one(struct net_device *dev, struct xdp_frame
 *xdpf)
+{
+	struct netfront_info *np = netdev_priv(dev);

drivers/net/xen-netfront.c static u16 xennet_select_queue(struct net_device *dev, struct sk_buff *skb, · line 587

Quoted 10 lines
drivers/net/xen-netfront.c +10 -0
+	struct netfront_stats *tx_stats = this_cpu_ptr(np->tx_stats);
+	unsigned int num_queues = dev->real_num_tx_queues;
+	struct netfront_queue *queue = NULL;
+	struct xen_netif_tx_request *tx;
+	unsigned long flags;
+	int notify;
+
+	queue = &np->queues[smp_processor_id() % num_queues];
+
+	spin_lock_irqsave(&queue->tx_lock, flags);  

Why are you taking a lock per packet (xdp_frame)?

Quoted 3 lines

Hi Jesper,

We have to protect shared ring indices.

Sure, I understand we need to protect the rings.

What I'm asking is why are doing this per-packet, and not once for the entire bulk of packets?

(notice how xennet_xdp_xmit gets a bulk of packets)

Quoted 22 lines
Quoted 13 lines
Quoted 13 lines
drivers/net/xen-netfront.c +12 -0
+
+	tx = xennet_make_first_txreq(queue, NULL,
+				     virt_to_page(xdpf->data),
+				     offset_in_page(xdpf->data),
+				     xdpf->len);
+
+	RING_PUSH_REQUESTS_AND_CHECK_NOTIFY(&queue->tx, notify);
+	if (notify)
+		notify_remote_via_irq(queue->tx_irq);
+
+	u64_stats_update_begin(&tx_stats->syncp);
+	tx_stats->bytes += xdpf->len;
Quoted 8 lines

drivers/net/xen-netfront.c static u16 xennet_select_queue(struct net_device *dev, struct sk_buff *skb, · line 605

Quoted 6 lines
drivers/net/xen-netfront.c +6 -0
+	tx_stats->packets++;
+	u64_stats_update_end(&tx_stats->syncp);
+
+	xennet_tx_buf_gc(queue);
+
+	spin_unlock_irqrestore(&queue->tx_lock, flags);  

Is the irqsave/irqrestore variant really needed here?

Quoted 1 line

netpoll also invokes the tx completion handler.

I forgot about netpoll.

The netpoll code cannot call this code path xennet_xdp_xmit / xennet_xdp_xmit_one, right?

Are the per-CPU ring queue's shared with normal network stack, that can be called from netpoll code path?

  queue = &np->queues[smp_processor_id() % num_queues];
Quoted 27 lines
Quoted 27 lines
Quoted 27 lines
drivers/net/xen-netfront.c +26 -0
+	return 0;
+}
+
+static int xennet_xdp_xmit(struct net_device *dev, int n,
+			   struct xdp_frame **frames, u32 flags)
+{
+	int drops = 0;
+	int i, err;
+
+	if (unlikely(flags & ~XDP_XMIT_FLAGS_MASK))
+		return -EINVAL;
+
+	for (i = 0; i < n; i++) {
+		struct xdp_frame *xdpf = frames[i];
+
+		if (!xdpf)
+			continue;
+		err = xennet_xdp_xmit_one(dev, xdpf);
+		if (err) {
+			xdp_return_frame_rx_napi(xdpf);
+			drops++;
+		}
+	}
+
+	return n - drops;
+}  
Signature · 4 lines
Best regards,
  Jesper Dangaard Brouer
  MSc.CS, Principal Kernel Engineer at Red Hat
  LinkedIn: http://www.linkedin.com/in/brouer
#6 lore raw

Re: [PATCH net-next v10 2/3] xen networking: add basic XDP support for xen-netfront

Quoted 36 lines · Jesper Dangaard Brouer
Quoted 33 lines · On Mon, 22 Jun 2020 15:45:46 +0300 Denis Kirjanov <[email protected]> wrote:
Quoted 32 lines · Jesper Dangaard Brouer
Quoted 4 lines · On Mon, 22 Jun 2020 12:21:11 +0300 Denis Kirjanov <[email protected]> wrote:
drivers/net/xen-netfront.c +0 -0
index 482c6c8..1b9f49e 100644

[...]

Quoted 10 lines
drivers/net/xen-netfront.c +1 -0
@@ -560,6 +572,65 @@ static u16 xennet_select_queue(struct net_device
 *dev, struct sk_buff *skb,
 	return queue_idx;
 }
 
+static int xennet_xdp_xmit_one(struct net_device *dev, struct

xdp_frame
*xdpf)

drivers/net/xen-netfront.c +2 -0
+{
+	struct netfront_info *np = netdev_priv(dev);

drivers/net/xen-netfront.c static u16 xennet_select_queue(struct net_device *dev, struct sk_buff *skb, · line 587

Quoted 10 lines
drivers/net/xen-netfront.c +10 -0
+	struct netfront_stats *tx_stats = this_cpu_ptr(np->tx_stats);
+	unsigned int num_queues = dev->real_num_tx_queues;
+	struct netfront_queue *queue = NULL;
+	struct xen_netif_tx_request *tx;
+	unsigned long flags;
+	int notify;
+
+	queue = &np->queues[smp_processor_id() % num_queues];
+
+	spin_lock_irqsave(&queue->tx_lock, flags);

Why are you taking a lock per packet (xdp_frame)?

Quoted 8 lines
Quoted 3 lines

Hi Jesper,

We have to protect shared ring indices.

Sure, I understand we need to protect the rings.

What I'm asking is why are doing this per-packet, and not once for the entire bulk of packets?

Now I see. I believe we can. Do you think it will give performance from the cache perspective?

Quoted 27 lines

(notice how xennet_xdp_xmit gets a bulk of packets)

Quoted 22 lines
Quoted 13 lines
Quoted 13 lines
drivers/net/xen-netfront.c +12 -0
+
+	tx = xennet_make_first_txreq(queue, NULL,
+				     virt_to_page(xdpf->data),
+				     offset_in_page(xdpf->data),
+				     xdpf->len);
+
+	RING_PUSH_REQUESTS_AND_CHECK_NOTIFY(&queue->tx, notify);
+	if (notify)
+		notify_remote_via_irq(queue->tx_irq);
+
+	u64_stats_update_begin(&tx_stats->syncp);
+	tx_stats->bytes += xdpf->len;
Quoted 8 lines

drivers/net/xen-netfront.c static u16 xennet_select_queue(struct net_device *dev, struct sk_buff *skb, · line 605

Quoted 6 lines
drivers/net/xen-netfront.c +6 -0
+	tx_stats->packets++;
+	u64_stats_update_end(&tx_stats->syncp);
+
+	xennet_tx_buf_gc(queue);
+
+	spin_unlock_irqrestore(&queue->tx_lock, flags);

Is the irqsave/irqrestore variant really needed here?

Quoted 1 line

netpoll also invokes the tx completion handler.

Quoted 7 lines

I forgot about netpoll.

The netpoll code cannot call this code path xennet_xdp_xmit / xennet_xdp_xmit_one, right?

Are the per-CPU ring queue's shared with normal network stack, that can be called from netpoll code path?

I meant that both xennet_start_xmit and xennet_poll_controller call xennet_tx_buf_gc

Quoted 39 lines
  queue = &np->queues[smp_processor_id() % num_queues];
Quoted 19 lines
Quoted 19 lines
Quoted 19 lines
drivers/net/xen-netfront.c +18 -0
+	return 0;
+}
+
+static int xennet_xdp_xmit(struct net_device *dev, int n,
+			   struct xdp_frame **frames, u32 flags)
+{
+	int drops = 0;
+	int i, err;
+
+	if (unlikely(flags & ~XDP_XMIT_FLAGS_MASK))
+		return -EINVAL;
+
+	for (i = 0; i < n; i++) {
+		struct xdp_frame *xdpf = frames[i];
+
+		if (!xdpf)
+			continue;
+		err = xennet_xdp_xmit_one(dev, xdpf);
Quoted 8 lines
Quoted 8 lines

drivers/net/xen-netfront.c static u16 xennet_select_queue(struct net_device *dev, struct sk_buff *skb, · line 631

Quoted 8 lines
drivers/net/xen-netfront.c +8 -0
+		if (err) {
+			xdp_return_frame_rx_napi(xdpf);
+			drops++;
+		}
+	}
+
+	return n - drops;
+}

--
Best regards,
Jesper Dangaard Brouer
MSc.CS, Principal Kernel Engineer at Red Hat
LinkedIn: http://www.linkedin.com/in/brouer

#11 lore raw

RE: [PATCH net-next v10 3/3] xen networking: add XDP offset adjustment to xen-netback

In reply to Denis Kirjanov
Quoted 163 lines

-----Original Message-----
From: Denis Kirjanov <[email protected]>
Sent: 22 June 2020 13:51
To: [email protected]
Cc: [email protected]; [email protected]; [email protected]; [email protected]; [email protected]
Subject: Re: [PATCH net-next v10 3/3] xen networking: add XDP offset adjustment to xen-netback

Quoted 56 lines · Paul Durrant
Quoted 56 lines

-----Original Message-----
From: Denis Kirjanov <[email protected]>
Sent: 22 June 2020 10:21
To: [email protected]
Cc: [email protected]; [email protected]; [email protected]; [email protected]; [email protected]
Subject: [PATCH net-next v10 3/3] xen networking: add XDP offset adjustment to xen-netback

the patch basically adds the offset adjustment and netfront state reading to make XDP work on netfront side.

---
 drivers/net/xen-netback/common.h    |  4 ++++
 drivers/net/xen-netback/interface.c |  2 ++
 drivers/net/xen-netback/netback.c   |  7 +++++++
 drivers/net/xen-netback/rx.c        | 15 ++++++++++++++-
 drivers/net/xen-netback/xenbus.c    | 32
++++++++++++++++++++++++++++++++
 5 files changed, 59 insertions(+), 1 deletion(-)
drivers/net/xen-netback/common.h +4 -0
index 05847eb..f14dc10 100644
@@ -281,6 +281,9 @@ struct xenvif {
 	u8 ipv6_csum:1;
 	u8 multicast_control:1;
 
+	/* headroom requested by xen-netfront */
+	u16 netfront_xdp_headroom;
+
 	/* Is this interface disabled? True when backend discovers
 	 * frontend is rogue.
 	 */
@@ -395,6 +398,7 @@ static inline pending_ring_idx_t
 nr_pending_reqs(struct xenvif_queue *queue)
 irqreturn_t xenvif_interrupt(int irq, void *dev_id);
 
 extern bool separate_tx_rx_irq;
+extern bool provides_xdp_headroom;
 
 extern unsigned int rx_drain_timeout_msecs;

extern unsigned int rx_stall_timeout_msecs;

drivers/net/xen-netback/interface.c +0 -0
index 0c8a02a..fc16edd 100644
@@ -483,6 +483,8 @@ struct xenvif *xenvif_alloc(struct device *parent,
 domid_t domid,
 	vif->queues = NULL;
 	vif->num_queues = 0;
 
Quoted 3 lines

drivers/net/xen-netback/interface.c struct xenvif *xenvif_alloc(struct device *parent, · line 488

Quoted 2 lines
drivers/net/xen-netback/interface.c +2 -0
+	vif->netfront_xdp_headroom = 0;
+

Hi Paul,

Quoted 1 line

How about just 'xdp_headroom'? It's shorter to type :-)

makes sense.

Quoted 81 lines
Quoted 68 lines
drivers/net/xen-netback/interface.c +0 -0
 	spin_lock_init(&vif->lock);
 	INIT_LIST_HEAD(&vif->fe_mcast_addr);
drivers/net/xen-netback/netback.c +7 -0
index 315dfc6..6dfca72 100644
@@ -96,6 +96,13 @@
 module_param_named(hash_cache_size, xenvif_hash_cache_size, uint, 0644);
 MODULE_PARM_DESC(hash_cache_size, "Number of flows in the hash cache");
 
+/* The module parameter tells that we have to put data
+ * for xen-netfront with the XDP_PACKET_HEADROOM offset
+ * needed for XDP processing
+ */
+bool provides_xdp_headroom = true;
+module_param(provides_xdp_headroom, bool, 0644);
+
 static void xenvif_idx_release(struct xenvif_queue *queue, u16
 pending_idx,
 			       u8 status);
drivers/net/xen-netback/rx.c +14 -1
index ef58870..c5e9e14 100644
@@ -258,6 +258,19 @@ static void xenvif_rx_next_skb(struct xenvif_queue
 *queue,
 		pkt->extra_count++;
 	}
 
+	if (queue->vif->netfront_xdp_headroom) {
+		struct xen_netif_extra_info *extra;
+
+		extra = &pkt->extras[XEN_NETIF_EXTRA_TYPE_XDP - 1];
+
+		memset(extra, 0, sizeof(struct xen_netif_extra_info));
+		extra->u.xdp.headroom = queue->vif->netfront_xdp_headroom;
+		extra->type = XEN_NETIF_EXTRA_TYPE_XDP;
+		extra->flags = 0;
+
+		pkt->extra_count++;
+	}
+
 	if (skb->sw_hash) {
 		struct xen_netif_extra_info *extra;
@@ -356,7 +369,7 @@ static void xenvif_rx_data_slot(struct xenvif_queue
 *queue,
 				struct xen_netif_rx_request *req,
 				struct xen_netif_rx_response *rsp)
 {
-	unsigned int offset = 0;
+	unsigned int offset = queue->vif->netfront_xdp_headroom;
 	unsigned int flags;
 
drivers/net/xen-netback/rx.c +0 -0
 	do {
drivers/net/xen-netback/xenbus.c +0 -0
index 286054b..c67abc5 100644
@@ -393,6 +393,22 @@ static void set_backend_state(struct backend_info
 *be,
 	}
 }

drivers/net/xen-netback/xenbus.c static void set_backend_state(struct backend_info · line 405

Quoted 10 lines
drivers/net/xen-netback/xenbus.c +9 -0
 
+static void read_xenbus_frontend_xdp(struct backend_info *be,
+				      struct xenbus_device *dev)
+{
+	struct xenvif *vif = be->vif;
+	u16 headroom;
+	int err;
+
+	err = xenbus_scanf(XBT_NIL, dev->otherend,
+			   "netfront-xdp-headroom", "%hu", &headroom);

Isn't it just "xdp-headroom"? That's what the comments in netif.h state.

Quoted 7 lines

drivers/net/xen-netback/xenbus.c static void set_backend_state(struct backend_info *be, · line 408

Quoted 4 lines
drivers/net/xen-netback/xenbus.c +4 -0
+	if (err < 0) {
+		vif->netfront_xdp_headroom = 0;
+		return;
+	}

What is a reasonable value for maximum headroom? Do we really want to allow values all the way up to 65535?

Quoted 3 lines

Since the headroom is used for encapsulation I think we definitely don't need more than 65535
but more that 255

Ok, I suggest documenting (and defining) the max in netif.h then and then sanity checking it here. Also I just noticed that your check on xenbus_scanf's return value is not correct since its return semantics are the same as for normal scanf(3).

Paul

Quoted 56 lines
Quoted 56 lines

Paul

Quoted 48 lines
drivers/net/xen-netback/xenbus.c +3 -0
+	vif->netfront_xdp_headroom = headroom;
+}
+
 /**
  * Callback received when the frontend's state changes.

*/

drivers/net/xen-netback/xenbus.c +5 -0
@@ -417,6 +433,11 @@ static void frontend_changed(struct xenbus_device
 *dev,
 		set_backend_state(be, XenbusStateConnected);
 		break;
 
+	case XenbusStateReconfiguring:
+		read_xenbus_frontend_xdp(be, dev);
+		xenbus_switch_state(dev, XenbusStateReconfigured);
+		break;
+
 	case XenbusStateClosing:
 		set_backend_state(be, XenbusStateClosing);
drivers/net/xen-netback/xenbus.c +11 -0
 		break;
@@ -947,6 +968,8 @@ static int read_xenbus_vif_flags(struct backend_info
 *be)
 	vif->ipv6_csum = !!xenbus_read_unsigned(dev->otherend,
 						"feature-ipv6-csum-offload", 0);
 
+	read_xenbus_frontend_xdp(be, dev);
+
 	return 0;
 }
@@ -1036,6 +1059,15 @@ static int netback_probe(struct xenbus_device
 *dev,
 			goto abort_transaction;
 		}
 
+		/* we can adjust a headroom for netfront XDP processing */
+		err = xenbus_printf(xbt, dev->nodename,
+				    "feature-xdp-headroom", "%d",
+				    provides_xdp_headroom);
+		if (err) {
+			message = "writing feature-xdp-headroom";
+			goto abort_transaction;
+		}
+
 		/* We don't support rx-flip path (except old guests who
 		 * don't grok this feature flag).

*/
--
1.8.3.1