net: Add support for XDP in egress path

PATCH Superseded David Ahern 4 participants 24 messages Apr 24, 2020 – Apr 27, 2020

Subsystems (10)

Networking Network drivers Kselftest BPF +6 more subsystems

lore

Share & tools

lore

David Ahern

author

2020-04-24 20:14 UTC #0 lore raw

v3 cover net: Add support for XDP in egress path

From: David Ahern <[email protected]>

This series adds support for XDP in the egress path by introducing a new XDP attachment type, BPF_XDP_EGRESS, and adding a UAPI to if_link.h for attaching the program to a netdevice and reporting the program. bpf programs can be run on all packets in the Tx path - skbs or redirected xdp frames. The intent is to emulate the current RX path for XDP as much as possible to maintain consistency and symmetry in the 2 paths with their APIs.

This is a missing primitive for XDP allowing solutions to build small, targeted programs properly distributed in the networking path allowing, for example, an egress firewall/ACL/traffic verification or packet manipulation and encapping an entire ethernet frame whether it is locally generated traffic, forwarded via the slow path (ie., full stack processing) or xdp redirected frames.

Nothing about running a program in the Tx path requires driver specific resources like the Rx path has. Thus, programs can be run in core code and attached to the net_device struct similar to skb mode. The egress attach is done using the new XDP_FLAGS_EGRESS_MODE flag, and is reported by the kernel using the XDP_ATTACHED_EGRESS_CORE attach flag with IFLA_XDP_EGRESS_PROG_ID making the api similar to existing APIs for XDP.

The locations chosen to run the egress program - __netdev_start_xmit before the call to ndo_start_xmit and bq_xmit_all before invoking ndo_xdp_xmit - allow follow on patch sets to handle tx queueing and setting the queue index if multi-queue with consistency in handling both packet formats.

A few of the patches trace back to work done on offloading programs from a VM by Jason Wang and Prashant Bole.

v3:
- removed IFLA_XDP_EGRESS and dropped back to XDP_FLAGS_EGRESS_MODE as the uapi to specify the attach. This caused the ordering of the patches to change with the uapi now introduced in the second patch and 2 refactoring patches are dropped. Samples and test programs updated to use the new API.

v2:
- changed rx checks in xdp_is_valid_access to any expected_attach_type
- add xdp_egress argument to bpftool prog rst document
- do not allow IFLA_XDP and IFLA_XDP_EGRESS in the same config. There is no way to rollback IFLA_XDP if IFLA_XDP_EGRESS fails.
- comments from Andrii on libbpf

v1:
- add selftests
- flip the order of xdp generic patches as requested by Toke
- fixed the count arg to do_xdp_egress_frame - Toke
- remove meta data invalidate in __xdp_egress_frame - Toke
- fixed data_hard_start in __xdp_egress_frame - Jesper
- refactored convert_to_xdp_frame to reuse buf to frame code - Jesper
- added missed refactoring patch when generating patch set

RFC v5:
- updated cover letter
- moved running of ebpf program to from ndo_{start,xdp}_xmit to core code. Dropped all tun and vhost related changes.
- added egress support to bpftool

RFC v4:
- updated cover letter
- patches related to code movement between tuntap, headers and vhost are dropped; previous RFC ran the XDP program in vhost context vs this set which runs them before queueing to vhost. As a part of this moved invocation of egress program to tun_net_xmit and tun_xdp_xmit.
- renamed do_xdp_generic to do_xdp_generic_rx to emphasize is called in the Rx path; added rx argument to do_xdp_generic_core since it is used for both directions and needs to know which queue values to set in xdp_buff

RFC v3:
- reworked the patches - splitting patch 1 from RFC v2 into 3, combining patch 2 from RFC v2 into the first 3, combining patches 6 and 7 from RFC v2 into 1 since both did a trivial rename and export. Reordered the patches such that kernel changes are first followed by libbpf and an enhancement to a sample.

- moved small xdp related helper functions from tun.c to tun.h to make tun_ptr_free usable from the tap code. This is needed to handle the case of tap builtin and tun built as a module.

- pkt_ptrs added to struct tun_file and passed to tun_consume_packets rather than declaring pkts as an array on the stack.

RFC v2:
- New XDP attachment type: Jesper, Toke and Alexei discussed whether to introduce a new program type. Since this set adds a way to attach regular XDP program to the tx path, as per Alexei's suggestion, a new attachment type BPF_XDP_EGRESS is introduced.

- libbpf API changes:
Alexei had suggested _opts() style of API extension. Considering it two new libbpf APIs are introduced which are equivalent to existing APIs. New ones can be extended easily. Please see individual patches for details. xdp1 sample program is modified to use new APIs.

- tun: Some patches from previous set are removed as they are irrelevant in this series. They will in introduced later.

David Ahern (15):
net: Refactor convert_to_xdp_frame
net: uapi for XDP programs in the egress path
net: Add XDP setup and query commands for Tx programs
net: Add BPF_XDP_EGRESS as a bpf_attach_type
xdp: Add xdp_txq_info to xdp_buff
net: Rename do_xdp_generic to do_xdp_generic_rx
net: rename netif_receive_generic_xdp to do_generic_xdp_core
net: set XDP egress program on netdevice
net: Support xdp in the Tx path for packets as an skb
net: Support xdp in the Tx path for xdp_frames
libbpf: Add egress XDP support
bpftool: Add support for XDP egress
selftest: Add test for xdp_egress
selftest: Add xdp_egress attach tests
samples/bpf: add XDP egress support to xdp1

 drivers/net/tun.c                             |   4 +-
 include/linux/netdevice.h                     |  21 +-
 include/net/xdp.h                             |  35 ++-
 include/uapi/linux/bpf.h                      |   3 +
 include/uapi/linux/if_link.h                  |   6 +-
 kernel/bpf/devmap.c                           |  19 +-
 net/core/dev.c                                | 241 ++++++++++++++----
 net/core/filter.c                             |  26 ++
 net/core/rtnetlink.c                          |  23 +-
 samples/bpf/xdp1_user.c                       |  11 +-
 .../bpf/bpftool/Documentation/bpftool-net.rst |   4 +-
 .../bpftool/Documentation/bpftool-prog.rst    |   2 +-
 tools/bpf/bpftool/bash-completion/bpftool     |   4 +-
 tools/bpf/bpftool/net.c                       |   6 +-
 tools/bpf/bpftool/netlink_dumper.c            |   5 +
 tools/bpf/bpftool/prog.c                      |   2 +-
 tools/include/uapi/linux/bpf.h                |   3 +
 tools/include/uapi/linux/if_link.h            |   6 +-
 tools/lib/bpf/libbpf.c                        |   2 +
 tools/lib/bpf/libbpf.h                        |   1 +
 tools/lib/bpf/netlink.c                       |   6 +
 tools/testing/selftests/bpf/Makefile          |   1 +
 .../bpf/prog_tests/xdp_egress_attach.c        |  56 ++++
 .../selftests/bpf/progs/test_xdp_egress.c     |  12 +
 .../bpf/progs/test_xdp_egress_fail.c          |  16 ++
 tools/testing/selftests/bpf/progs/xdp_drop.c  |  25 ++
 .../testing/selftests/bpf/test_xdp_egress.sh  | 160 ++++++++++++
 27 files changed, 623 insertions(+), 77 deletions(-)
 create mode 100644 tools/testing/selftests/bpf/prog_tests/xdp_egress_attach.c
 create mode 100644 tools/testing/selftests/bpf/progs/test_xdp_egress.c
 create mode 100644 tools/testing/selftests/bpf/progs/test_xdp_egress_fail.c
 create mode 100644 tools/testing/selftests/bpf/progs/xdp_drop.c
 create mode 100755 tools/testing/selftests/bpf/test_xdp_egress.sh

Signature · 1 line

2.21.1 (Apple Git-122.3)

David Ahern

author

2020-04-24 20:14 UTC #1 lore raw

PATCH v3 1/15 net: Refactor convert_to_xdp_frame

From: David Ahern <[email protected]>

Move the guts of convert_to_xdp_frame to a new helper, update_xdp_frame so it can be reused in a later patch.

Signed-off-by David Ahern <[email protected]>

Acked-by Jesper Dangaard Brouer <[email protected]>

---
 include/net/xdp.h | 30 ++++++++++++++++++++----------
 1 file changed, 20 insertions(+), 10 deletions(-)

include/net/xdp.h +20 ~~-10~~

index 40c6d3398458..779313862073 100644
@@ -93,32 +93,42 @@ static inline void xdp_scrub_frame(struct xdp_frame *frame)
 
 struct xdp_frame *xdp_convert_zc_to_xdp_frame(struct xdp_buff *xdp);
 
-/* Convert xdp_buff to xdp_frame */
 static inline
-struct xdp_frame *convert_to_xdp_frame(struct xdp_buff *xdp)
+bool update_xdp_frame(struct xdp_buff *xdp, struct xdp_frame *xdp_frame)
 {
-	struct xdp_frame *xdp_frame;
 	int metasize;
 	int headroom;
 
-	if (xdp->rxq->mem.type == MEM_TYPE_ZERO_COPY)
-		return xdp_convert_zc_to_xdp_frame(xdp);
-
 	/* Assure headroom is available for storing info */
 	headroom = xdp->data - xdp->data_hard_start;
 	metasize = xdp->data - xdp->data_meta;
 	metasize = metasize > 0 ? metasize : 0;
 	if (unlikely((headroom - metasize) < sizeof(*xdp_frame)))
-		return NULL;
-
-	/* Store info in top of packet */
-	xdp_frame = xdp->data_hard_start;
+		return false;
 
 	xdp_frame->data = xdp->data;
 	xdp_frame->len  = xdp->data_end - xdp->data;
 	xdp_frame->headroom = headroom - sizeof(*xdp_frame);
 	xdp_frame->metasize = metasize;
 
+	return true;
+}
+
+/* Convert xdp_buff to xdp_frame */
+static inline
+struct xdp_frame *convert_to_xdp_frame(struct xdp_buff *xdp)
+{
+	struct xdp_frame *xdp_frame;
+
+	if (xdp->rxq->mem.type == MEM_TYPE_ZERO_COPY)
+		return xdp_convert_zc_to_xdp_frame(xdp);
+
+	/* Store info in top of packet */
+	xdp_frame = xdp->data_hard_start;
+
+	if (unlikely(!update_xdp_frame(xdp, xdp_frame)))
+		return NULL;
+
 	/* rxq only valid until napi_schedule ends, convert to xdp_mem_info */
 	xdp_frame->mem = xdp->rxq->mem;

Signature · 1 line

2.21.1 (Apple Git-122.3)

John Fastabend

maintainer Acked-by on v3 1/15

2020-04-27 16:19 UTC #2 lore raw

RE: [PATCH v3 bpf-next 01/15] net: Refactor convert_to_xdp_frame

Quoted 9 lines · David Ahern

From: David Ahern <[email protected]>
Move the guts of convert_to_xdp_frame to a new helper, update_xdp_frame so it can be reused in a later patch.
Suggested-by Jesper Dangaard Brouer <[email protected]>
Signed-off-by David Ahern <[email protected]>
Acked-by Jesper Dangaard Brouer <[email protected]>
---

Acked-by John Fastabend <[email protected]>

David Ahern

author

2020-04-24 20:14 UTC #3 lore raw

PATCH v3 2/15 net: uapi for XDP programs in the egress path

In reply to David Ahern

From: David Ahern <[email protected]>

Running programs in the egress path, on skbs or xdp_frames, does not require driver specific resources like Rx path. Accordingly, the programs can be run in core code, so add xdp_egress_prog to net_device to hold a reference to an attached program.

For UAPI, add XDP_FLAGS_EGRESS_MODE to specify attach is at egress, add a new attach flag, XDP_ATTACHED_EGRESS_CORE, for reporting the attach point at the core, egress level and add IFLA_XDP_EGRESS_PROG_ID for reporting the program id. Add rtnl_xdp_prog_egress to fill in link message with egress data.

Signed-off-by David Ahern <[email protected]>

---
 include/linux/netdevice.h          |  1 +
 include/uapi/linux/if_link.h       |  6 +++++-
 net/core/rtnetlink.c               | 23 +++++++++++++++++++++--
 tools/include/uapi/linux/if_link.h |  6 +++++-
 4 files changed, 32 insertions(+), 4 deletions(-)

include/linux/netdevice.h +1 -0

index 0750b54b3765..a7b4777ddf9b 100644
@@ -1994,6 +1994,7 @@ struct net_device {
 	unsigned int		real_num_rx_queues;
 
 	struct bpf_prog __rcu	*xdp_prog;
+	struct bpf_prog __rcu	*xdp_egress_prog;
 	unsigned long		gro_flush_timeout;
 	rx_handler_func_t __rcu	*rx_handler;
 	void __rcu		*rx_handler_data;

include/uapi/linux/if_link.h +5 -1

index 127c704eeba9..4a33ff26ef62 100644
@@ -975,9 +975,11 @@ enum {
 #define XDP_FLAGS_DRV_MODE		(1U << 2)
 #define XDP_FLAGS_HW_MODE		(1U << 3)
 #define XDP_FLAGS_REPLACE		(1U << 4)
+#define XDP_FLAGS_EGRESS_MODE		(1U << 5)
 #define XDP_FLAGS_MODES			(XDP_FLAGS_SKB_MODE | \
 					 XDP_FLAGS_DRV_MODE | \
-					 XDP_FLAGS_HW_MODE)
+					 XDP_FLAGS_HW_MODE | \
+					 XDP_FLAGS_EGRESS_MODE)
 #define XDP_FLAGS_MASK			(XDP_FLAGS_UPDATE_IF_NOEXIST | \
 					 XDP_FLAGS_MODES | XDP_FLAGS_REPLACE)

@@ -988,6 +990,7 @@ enum {
 	XDP_ATTACHED_SKB,
 	XDP_ATTACHED_HW,
 	XDP_ATTACHED_MULTI,
+	XDP_ATTACHED_EGRESS_CORE,
 };
 
 enum {

@@ -1000,6 +1003,7 @@ enum {
 	IFLA_XDP_SKB_PROG_ID,
 	IFLA_XDP_HW_PROG_ID,
 	IFLA_XDP_EXPECTED_FD,
+	IFLA_XDP_EGRESS_PROG_ID,
 	__IFLA_XDP_MAX,
 };

net/core/rtnetlink.c +21 -2

index d6f4f4a9e8ba..c8f6cc67a070 100644
@@ -982,7 +982,7 @@ static size_t rtnl_xdp_size(void)
 	size_t xdp_size = nla_total_size(0) +	/* nest IFLA_XDP */
 			  nla_total_size(1) +	/* XDP_ATTACHED */
 			  nla_total_size(4) +	/* XDP_PROG_ID (or 1st mode) */
-			  nla_total_size(4);	/* XDP_<mode>_PROG_ID */
+			  nla_total_size(4) * 2; /* XDP_<mode>_PROG_ID */
 
 	return xdp_size;
 }

@@ -1402,6 +1402,18 @@ static int rtnl_fill_link_ifmap(struct sk_buff *skb, struct net_device *dev)
 	return 0;
 }
 
+static u32 rtnl_xdp_prog_egress(struct net_device *dev)
+{
+	const struct bpf_prog *prog;
+
+	ASSERT_RTNL();
+
+	prog = rtnl_dereference(dev->xdp_egress_prog);
+	if (!prog)
+		return 0;
+	return prog->aux->id;
+}
+
 static u32 rtnl_xdp_prog_skb(struct net_device *dev)
 {
 	const struct bpf_prog *generic_xdp_prog;

@@ -1474,6 +1486,12 @@ static int rtnl_xdp_fill(struct sk_buff *skb, struct net_device *dev)
 				  IFLA_XDP_HW_PROG_ID, rtnl_xdp_prog_hw);
 	if (err)
 		goto err_cancel;
+	err = rtnl_xdp_report_one(skb, dev, &prog_id, &mode,
+				  XDP_ATTACHED_EGRESS_CORE,
+				  IFLA_XDP_EGRESS_PROG_ID,
+				  rtnl_xdp_prog_egress);
+	if (err)
+		goto err_cancel;
 
 	err = nla_put_u8(skb, IFLA_XDP_ATTACHED, mode);
 	if (err)

@@ -2790,7 +2808,8 @@ static int do_setlink(const struct sk_buff *skb,
 		if (err < 0)
 			goto errout;
 
-		if (xdp[IFLA_XDP_ATTACHED] || xdp[IFLA_XDP_PROG_ID]) {
+		if (xdp[IFLA_XDP_ATTACHED] || xdp[IFLA_XDP_PROG_ID] ||
+		    xdp[IFLA_XDP_EGRESS_PROG_ID]) {
 			err = -EINVAL;
 			goto errout;
 		}

tools/include/uapi/linux/if_link.h +5 -1

index ca6665ea758a..41b1c5ff4875 100644
@@ -963,9 +963,11 @@ enum {
 #define XDP_FLAGS_DRV_MODE		(1U << 2)
 #define XDP_FLAGS_HW_MODE		(1U << 3)
 #define XDP_FLAGS_REPLACE		(1U << 4)
+#define XDP_FLAGS_EGRESS_MODE		(1U << 5)
 #define XDP_FLAGS_MODES			(XDP_FLAGS_SKB_MODE | \
 					 XDP_FLAGS_DRV_MODE | \
-					 XDP_FLAGS_HW_MODE)
+					 XDP_FLAGS_HW_MODE | \
+					 XDP_FLAGS_EGRESS_MODE)
 #define XDP_FLAGS_MASK			(XDP_FLAGS_UPDATE_IF_NOEXIST | \
 					 XDP_FLAGS_MODES | XDP_FLAGS_REPLACE)

@@ -976,6 +978,7 @@ enum {
 	XDP_ATTACHED_SKB,
 	XDP_ATTACHED_HW,
 	XDP_ATTACHED_MULTI,
+	XDP_ATTACHED_EGRESS_CORE,
 };
 
 enum {

@@ -988,6 +991,7 @@ enum {
 	IFLA_XDP_SKB_PROG_ID,
 	IFLA_XDP_HW_PROG_ID,
 	IFLA_XDP_EXPECTED_FD,
+	IFLA_XDP_EGRESS_PROG_ID,
 	__IFLA_XDP_MAX,
 };

Signature · 1 line

2.21.1 (Apple Git-122.3)

David Ahern

author

2020-04-24 20:15 UTC #4 lore raw

PATCH v3 3/15 net: Add XDP setup and query commands for Tx programs

In reply to David Ahern

From: David Ahern <[email protected]>

Add new netdev commands, XDP_SETUP_PROG_EGRESS and XDP_QUERY_PROG_EGRESS, to query and setup egress programs.

Update dev_change_xdp_fd and dev_xdp_install to check for egress mode via XDP_FLAGS_EGRESS_MODE in the flags. If egress bool is set, then use XDP_SETUP_PROG_EGRESS in dev_xdp_install and XDP_QUERY_PROG_EGRESS in dev_change_xdp_fd.

Signed-off-by David Ahern <[email protected]>

Co-developed-by Prashant Bhole <[email protected]>

Signed-off-by Prashant Bhole <[email protected]>

---
 include/linux/netdevice.h |  2 ++
 net/core/dev.c            | 20 +++++++++++++++-----
 2 files changed, 17 insertions(+), 5 deletions(-)

include/linux/netdevice.h +2 -0

index a7b4777ddf9b..7f9cc088aa64 100644
@@ -872,8 +872,10 @@ enum bpf_netdev_command {
 	 */
 	XDP_SETUP_PROG,
 	XDP_SETUP_PROG_HW,
+	XDP_SETUP_PROG_EGRESS,
 	XDP_QUERY_PROG,
 	XDP_QUERY_PROG_HW,
+	XDP_QUERY_PROG_EGRESS,
 	/* BPF program for offload callbacks, invoked at program load time. */
 	BPF_OFFLOAD_MAP_ALLOC,
 	BPF_OFFLOAD_MAP_FREE,

net/core/dev.c +15 -5

index fb61522b1ce1..726a93a5cb25 100644
@@ -8593,13 +8593,16 @@ static int dev_xdp_install(struct net_device *dev, bpf_op_t bpf_op,
 			   struct bpf_prog *prog)
 {
 	bool non_hw = !(flags & XDP_FLAGS_HW_MODE);
+	bool egress = flags & XDP_FLAGS_EGRESS_MODE;
 	struct bpf_prog *prev_prog = NULL;
 	struct netdev_bpf xdp;
 	int err;
 
 	if (non_hw) {
-		prev_prog = bpf_prog_by_id(__dev_xdp_query(dev, bpf_op,
-							   XDP_QUERY_PROG));
+		enum bpf_netdev_command cmd;
+
+		cmd = egress ? XDP_QUERY_PROG_EGRESS : XDP_QUERY_PROG;
+		prev_prog = bpf_prog_by_id(__dev_xdp_query(dev, bpf_op, cmd));
 		if (IS_ERR(prev_prog))
 			prev_prog = NULL;
 	}

@@ -8608,7 +8611,7 @@ static int dev_xdp_install(struct net_device *dev, bpf_op_t bpf_op,
 	if (flags & XDP_FLAGS_HW_MODE)
 		xdp.command = XDP_SETUP_PROG_HW;
 	else
-		xdp.command = XDP_SETUP_PROG;
+		xdp.command = egress ? XDP_SETUP_PROG_EGRESS : XDP_SETUP_PROG;
 	xdp.extack = extack;
 	xdp.flags = flags;
 	xdp.prog = prog;

@@ -8670,12 +8673,18 @@ int dev_change_xdp_fd(struct net_device *dev, struct netlink_ext_ack *extack,
 	bpf_op_t bpf_op, bpf_chk;
 	struct bpf_prog *prog;
 	bool offload;
+	bool egress;
 	int err;
 
 	ASSERT_RTNL();
 
 	offload = flags & XDP_FLAGS_HW_MODE;
-	query = offload ? XDP_QUERY_PROG_HW : XDP_QUERY_PROG;
+	egress = flags & XDP_FLAGS_EGRESS_MODE;
+	if (egress)
+		query = XDP_QUERY_PROG_EGRESS;
+	else
+		query = offload ? XDP_QUERY_PROG_HW : XDP_QUERY_PROG;
+
 
 	bpf_op = bpf_chk = ops->ndo_bpf;
 	if (!bpf_op && (flags & (XDP_FLAGS_DRV_MODE | XDP_FLAGS_HW_MODE))) {

@@ -8705,7 +8714,8 @@ int dev_change_xdp_fd(struct net_device *dev, struct netlink_ext_ack *extack,
 		}
 	}
 	if (fd >= 0) {
-		if (!offload && __dev_xdp_query(dev, bpf_chk, XDP_QUERY_PROG)) {
+		if (!offload && !egress &&
+		    __dev_xdp_query(dev, bpf_chk, XDP_QUERY_PROG)) {
 			NL_SET_ERR_MSG(extack, "native and generic XDP can't be active at the same time");
 			return -EEXIST;
 		}

Signature · 1 line

2.21.1 (Apple Git-122.3)

David Ahern

author

2020-04-24 20:14 UTC #5 lore raw

PATCH v3 4/15 net: Add BPF_XDP_EGRESS as a bpf_attach_type

In reply to David Ahern

From: David Ahern <[email protected]>

Add new bpf_attach_type, BPF_XDP_EGRESS, for BPF programs attached at the XDP layer, but the egress path.

Since egress path will not have ingress_ifindex and rx_queue_index set, update xdp_is_valid_access to block access to these entries in the xdp context when a program is attached with expected_attach_type set.

Update dev_change_xdp_fd to verify expected_attach_type for a program is BPF_XDP_EGRESS if egress argument is set.

The next patch adds support for the egress ifindex.

Signed-off-by Prashant Bhole <[email protected]>

Co-developed-by David Ahern <[email protected]>

Signed-off-by David Ahern <[email protected]>

---
 include/uapi/linux/bpf.h       |  1 +
 net/core/dev.c                 | 11 +++++++++++
 net/core/filter.c              | 11 +++++++++++
 tools/include/uapi/linux/bpf.h |  1 +
 4 files changed, 24 insertions(+)

include/uapi/linux/bpf.h +1 -0

index 2e29a671d67e..a9d384998e8b 100644
@@ -215,6 +215,7 @@ enum bpf_attach_type {
 	BPF_TRACE_FEXIT,
 	BPF_MODIFY_RETURN,
 	BPF_LSM_MAC,
+	BPF_XDP_EGRESS,
 	__MAX_BPF_ATTACH_TYPE
 };

net/core/dev.c +11 -0

index 726a93a5cb25..2f6e18fc02ed 100644
@@ -8730,6 +8730,17 @@ int dev_change_xdp_fd(struct net_device *dev, struct netlink_ext_ack *extack,
 		if (IS_ERR(prog))
 			return PTR_ERR(prog);
 
+		if (egress && prog->expected_attach_type != BPF_XDP_EGRESS) {
+			NL_SET_ERR_MSG(extack, "XDP program in Tx path must use BPF_XDP_EGRESS attach type");
+			bpf_prog_put(prog);
+			return -EINVAL;
+		}
+		if (!egress && prog->expected_attach_type == BPF_XDP_EGRESS) {
+			NL_SET_ERR_MSG(extack, "XDP program in Rx path can not use BPF_XDP_EGRESS attach type");
+			bpf_prog_put(prog);
+			return -EINVAL;
+		}
+
 		if (!offload && bpf_prog_is_dev_bound(prog->aux)) {
 			NL_SET_ERR_MSG(extack, "using device-bound program without HW_MODE flag is not supported");
 			bpf_prog_put(prog);

net/core/filter.c +11 -0

index a943df3ad8b0..b4d064c7fdec 100644
@@ -6935,6 +6935,17 @@ static bool xdp_is_valid_access(int off, int size,
 				const struct bpf_prog *prog,
 				struct bpf_insn_access_aux *info)
 {
+	/* Rx data is only accessible from original XDP where
+	 * expected_attach_type is not set
+	 */
+	if (prog->expected_attach_type) {
+		switch (off) {
+		case offsetof(struct xdp_md, ingress_ifindex):
+		case offsetof(struct xdp_md, rx_queue_index):
+			return false;
+		}
+	}
+
 	if (type == BPF_WRITE) {
 		if (bpf_prog_is_dev_bound(prog->aux)) {
 			switch (off) {

tools/include/uapi/linux/bpf.h +1 -0

index 2e29a671d67e..a9d384998e8b 100644
@@ -215,6 +215,7 @@ enum bpf_attach_type {
 	BPF_TRACE_FEXIT,
 	BPF_MODIFY_RETURN,
 	BPF_LSM_MAC,
+	BPF_XDP_EGRESS,
 	__MAX_BPF_ATTACH_TYPE
 };

Signature · 1 line

2.21.1 (Apple Git-122.3)

David Ahern

author

2020-04-24 20:14 UTC #6 lore raw

PATCH v3 5/15 xdp: Add xdp_txq_info to xdp_buff

In reply to David Ahern

From: David Ahern <[email protected]>

Add xdp_txq_info as the Tx counterpart to xdp_rxq_info. At the moment only the device is added. Other fields (queue_index) can be added as use cases arise.

From a UAPI perspective, add egress_ifindex to xdp context.

Update the verifier to reject accesses to egress_ifindex by rx programs.

Signed-off-by David Ahern <[email protected]>

---
 include/net/xdp.h              |  5 +++++
 include/uapi/linux/bpf.h       |  2 ++
 net/core/filter.c              | 15 +++++++++++++++
 tools/include/uapi/linux/bpf.h |  2 ++
 4 files changed, 24 insertions(+)

include/net/xdp.h +5 -0

index 779313862073..7e4ca99def77 100644
@@ -63,6 +63,10 @@ struct xdp_rxq_info {
 	struct xdp_mem_info mem;
 } ____cacheline_aligned; /* perf critical, avoid false-sharing */
 
+struct xdp_txq_info {
+	struct net_device *dev;
+};
+
 struct xdp_buff {
 	void *data;
 	void *data_end;

@@ -70,6 +74,7 @@ struct xdp_buff {
 	void *data_hard_start;
 	unsigned long handle;
 	struct xdp_rxq_info *rxq;
+	struct xdp_txq_info *txq;
 };
 
 struct xdp_frame {

include/uapi/linux/bpf.h +2 -0

index a9d384998e8b..35e3aab97dd4 100644
@@ -3487,6 +3487,8 @@ struct xdp_md {
 	/* Below access go through struct xdp_rxq_info */
 	__u32 ingress_ifindex; /* rxq->dev->ifindex */
 	__u32 rx_queue_index;  /* rxq->queue_index  */
+
+	__u32 egress_ifindex;  /* txq->dev->ifindex */
 };
 
 enum sk_action {

net/core/filter.c +15 -0

index b4d064c7fdec..60351295baa7 100644
@@ -6944,6 +6944,11 @@ static bool xdp_is_valid_access(int off, int size,
 		case offsetof(struct xdp_md, rx_queue_index):
 			return false;
 		}
+	} else if (prog->expected_attach_type != BPF_XDP_EGRESS) {
+		switch (off) {
+		case offsetof(struct xdp_md, egress_ifindex):
+			return false;
+		}
 	}
 
 	if (type == BPF_WRITE) {

@@ -7893,6 +7898,16 @@ static u32 xdp_convert_ctx_access(enum bpf_access_type type,
 				      offsetof(struct xdp_rxq_info,
 					       queue_index));
 		break;
+	case offsetof(struct xdp_md, egress_ifindex):
+		*insn++ = BPF_LDX_MEM(BPF_FIELD_SIZEOF(struct xdp_buff, txq),
+				      si->dst_reg, si->src_reg,
+				      offsetof(struct xdp_buff, txq));
+		*insn++ = BPF_LDX_MEM(BPF_FIELD_SIZEOF(struct xdp_txq_info, dev),
+				      si->dst_reg, si->dst_reg,
+				      offsetof(struct xdp_txq_info, dev));
+		*insn++ = BPF_LDX_MEM(BPF_W, si->dst_reg, si->dst_reg,
+				      offsetof(struct net_device, ifindex));
+		break;
 	}
 
 	return insn - insn_buf;

tools/include/uapi/linux/bpf.h +2 -0

index a9d384998e8b..35e3aab97dd4 100644
@@ -3487,6 +3487,8 @@ struct xdp_md {
 	/* Below access go through struct xdp_rxq_info */
 	__u32 ingress_ifindex; /* rxq->dev->ifindex */
 	__u32 rx_queue_index;  /* rxq->queue_index  */
+
+	__u32 egress_ifindex;  /* txq->dev->ifindex */
 };
 
 enum sk_action {

Signature · 1 line

2.21.1 (Apple Git-122.3)

David Ahern

author

2020-04-24 20:14 UTC #7 lore raw

PATCH v3 6/15 net: Rename do_xdp_generic to do_xdp_generic_rx

In reply to David Ahern

From: David Ahern <[email protected]>

Rename do_xdp_generic to do_xdp_generic_rx to emphasize its use in the Rx path.

Signed-off-by Jason Wang <[email protected]>

Signed-off-by Prashant Bhole <[email protected]>

Signed-off-by David Ahern <[email protected]>

---
 drivers/net/tun.c         | 4 ++--
 include/linux/netdevice.h | 2 +-
 net/core/dev.c            | 7 ++++---
 3 files changed, 7 insertions(+), 6 deletions(-)

drivers/net/tun.c +2 -2

index 44889eba1dbc..efe655d27661 100644
@@ -1898,7 +1898,7 @@ static ssize_t tun_get_user(struct tun_struct *tun, struct tun_file *tfile,
 		rcu_read_lock();
 		xdp_prog = rcu_dereference(tun->xdp_prog);
 		if (xdp_prog) {
-			ret = do_xdp_generic(xdp_prog, skb);
+			ret = do_xdp_generic_rx(xdp_prog, skb);
 			if (ret != XDP_PASS) {
 				rcu_read_unlock();
 				local_bh_enable();

@@ -2463,7 +2463,7 @@ static int tun_xdp_one(struct tun_struct *tun,
 	skb_record_rx_queue(skb, tfile->queue_index);
 
 	if (skb_xdp) {
-		err = do_xdp_generic(xdp_prog, skb);
+		err = do_xdp_generic_rx(xdp_prog, skb);
 		if (err != XDP_PASS)
 			goto out;
 	}

include/linux/netdevice.h +1 -1

index 7f9cc088aa64..809e1a83b413 100644
@@ -3712,7 +3712,7 @@ static inline void dev_consume_skb_any(struct sk_buff *skb)
 }
 
 void generic_xdp_tx(struct sk_buff *skb, struct bpf_prog *xdp_prog);
-int do_xdp_generic(struct bpf_prog *xdp_prog, struct sk_buff *skb);
+int do_xdp_generic_rx(struct bpf_prog *xdp_prog, struct sk_buff *skb);
 int netif_rx(struct sk_buff *skb);
 int netif_rx_ni(struct sk_buff *skb);
 int netif_receive_skb(struct sk_buff *skb);

net/core/dev.c +4 -3

index 2f6e18fc02ed..50e07b7967c9 100644
@@ -4642,7 +4642,7 @@ void generic_xdp_tx(struct sk_buff *skb, struct bpf_prog *xdp_prog)
 
 static DEFINE_STATIC_KEY_FALSE(generic_xdp_needed_key);
 
-int do_xdp_generic(struct bpf_prog *xdp_prog, struct sk_buff *skb)
+int do_xdp_generic_rx(struct bpf_prog *xdp_prog, struct sk_buff *skb)
 {
 	if (xdp_prog) {
 		struct xdp_buff xdp;

@@ -4670,7 +4670,7 @@ int do_xdp_generic(struct bpf_prog *xdp_prog, struct sk_buff *skb)
 	kfree_skb(skb);
 	return XDP_DROP;
 }
-EXPORT_SYMBOL_GPL(do_xdp_generic);
+EXPORT_SYMBOL_GPL(do_xdp_generic_rx);
 
 static int netif_rx_internal(struct sk_buff *skb)
 {

@@ -5020,7 +5020,8 @@ static int __netif_receive_skb_core(struct sk_buff *skb, bool pfmemalloc,
 		int ret2;
 
 		preempt_disable();
-		ret2 = do_xdp_generic(rcu_dereference(skb->dev->xdp_prog), skb);
+		ret2 = do_xdp_generic_rx(rcu_dereference(skb->dev->xdp_prog),
+					 skb);
 		preempt_enable();
 
 		if (ret2 != XDP_PASS)

Signature · 1 line

2.21.1 (Apple Git-122.3)

David Ahern

author

2020-04-24 20:15 UTC #8 lore raw

PATCH v3 7/15 net: rename netif_receive_generic_xdp to do_generic_xdp_core

In reply to David Ahern

From: David Ahern <[email protected]>

In skb generic path, we need a way to run XDP program on skb but to have customized handling of XDP actions. netif_receive_generic_xdp will be more helpful in such cases than do_xdp_generic.

This patch prepares netif_receive_generic_xdp() to be used as general purpose function for running xdp programs on skbs by renaming it to do_xdp_generic_core, moving skb_is_redirected and rxq settings as well as XDP return code checks to the callers.

This allows this core function to be used from both Rx and Tx paths with rxq and txq set based on context.

Signed-off-by Jason Wang <[email protected]>

Signed-off-by Prashant Bhole <[email protected]>

Signed-off-by David Ahern <[email protected]>

---
 net/core/dev.c | 52 ++++++++++++++++++++++++--------------------------
 1 file changed, 25 insertions(+), 27 deletions(-)

net/core/dev.c +25 ~~-27~~

index 50e07b7967c9..0a56862a32e5 100644
@@ -4501,25 +4501,17 @@ static struct netdev_rx_queue *netif_get_rxqueue(struct sk_buff *skb)
 	return rxqueue;
 }
 
-static u32 netif_receive_generic_xdp(struct sk_buff *skb,
-				     struct xdp_buff *xdp,
-				     struct bpf_prog *xdp_prog)
+static u32 do_xdp_generic_core(struct sk_buff *skb, struct xdp_buff *xdp,
+			       struct bpf_prog *xdp_prog)
 {
-	struct netdev_rx_queue *rxqueue;
 	void *orig_data, *orig_data_end;
-	u32 metalen, act = XDP_DROP;
 	__be16 orig_eth_type;
 	struct ethhdr *eth;
+	u32 metalen, act;
 	bool orig_bcast;
 	int hlen, off;
 	u32 mac_len;
 
-	/* Reinjected packets coming from act_mirred or similar should
-	 * not get XDP generic processing.
-	 */
-	if (skb_is_redirected(skb))
-		return XDP_PASS;
-
 	/* XDP packets must be linear and must have sufficient headroom
 	 * of XDP_PACKET_HEADROOM bytes. This is the guarantee that also
 	 * native XDP provides, thus we need to do it here as well.

@@ -4535,9 +4527,9 @@ static u32 netif_receive_generic_xdp(struct sk_buff *skb,
 		if (pskb_expand_head(skb,
 				     hroom > 0 ? ALIGN(hroom, NET_SKB_PAD) : 0,
 				     troom > 0 ? troom + 128 : 0, GFP_ATOMIC))
-			goto do_drop;
+			return XDP_DROP;
 		if (skb_linearize(skb))
-			goto do_drop;
+			return XDP_DROP;
 	}
 
 	/* The XDP program wants to see the packet starting at the MAC

@@ -4555,9 +4547,6 @@ static u32 netif_receive_generic_xdp(struct sk_buff *skb,
 	orig_bcast = is_multicast_ether_addr_64bits(eth->h_dest);
 	orig_eth_type = eth->h_proto;
 
-	rxqueue = netif_get_rxqueue(skb);
-	xdp->rxq = &rxqueue->xdp_rxq;
-
 	act = bpf_prog_run_xdp(xdp_prog, xdp);
 
 	/* check if bpf_xdp_adjust_head was used */

@@ -4600,16 +4589,6 @@ static u32 netif_receive_generic_xdp(struct sk_buff *skb,
 		if (metalen)
 			skb_metadata_set(skb, metalen);
 		break;
-	default:
-		bpf_warn_invalid_xdp_action(act);
-		/* fall through */
-	case XDP_ABORTED:
-		trace_xdp_exception(skb->dev, xdp_prog, act);
-		/* fall through */
-	case XDP_DROP:
-	do_drop:
-		kfree_skb(skb);
-		break;
 	}
 
 	return act;

@@ -4644,12 +4623,22 @@ static DEFINE_STATIC_KEY_FALSE(generic_xdp_needed_key);
 
 int do_xdp_generic_rx(struct bpf_prog *xdp_prog, struct sk_buff *skb)
 {
+	/* Reinjected packets coming from act_mirred or similar should
+	 * not get XDP generic processing.
+	 */
+	if (skb_is_redirected(skb))
+		return XDP_PASS;
+
 	if (xdp_prog) {
+		struct netdev_rx_queue *rxqueue;
 		struct xdp_buff xdp;
 		u32 act;
 		int err;
 
-		act = netif_receive_generic_xdp(skb, &xdp, xdp_prog);
+		rxqueue = netif_get_rxqueue(skb);
+		xdp.rxq = &rxqueue->xdp_rxq;
+
+		act = do_xdp_generic_core(skb, &xdp, xdp_prog);
 		if (act != XDP_PASS) {
 			switch (act) {
 			case XDP_REDIRECT:

@@ -4661,6 +4650,15 @@ int do_xdp_generic_rx(struct bpf_prog *xdp_prog, struct sk_buff *skb)
 			case XDP_TX:
 				generic_xdp_tx(skb, xdp_prog);
 				break;
+			default:
+				bpf_warn_invalid_xdp_action(act);
+				/* fall through */
+			case XDP_ABORTED:
+				trace_xdp_exception(skb->dev, xdp_prog, act);
+				/* fall through */
+			case XDP_DROP:
+				kfree_skb(skb);
+				break;
 			}
 			return XDP_DROP;
 		}

Signature · 1 line

2.21.1 (Apple Git-122.3)

David Ahern

author

2020-04-24 20:14 UTC #9 lore raw

PATCH v3 8/15 net: set XDP egress program on netdevice

In reply to David Ahern

From: David Ahern <[email protected]>

This patch handles the plumbing for installing an XDP egress program on a net_device by handling XDP_SETUP_PROG_EGRESS and XDP_QUERY_PROG_EGRESS in generic_xdp_install handler. New static key is added to signal when an egress program has been installed.

Update dev_xdp_uninstall to remove egress programs.

Signed-off-by David Ahern <[email protected]>

---
 include/linux/netdevice.h |  2 ++
 net/core/dev.c            | 48 +++++++++++++++++++++++++++------------
 2 files changed, 36 insertions(+), 14 deletions(-)

include/linux/netdevice.h +2 -0

index 809e1a83b413..bfd8b6c8bf70 100644
@@ -751,6 +751,8 @@ struct netdev_rx_queue {
 #endif
 } ____cacheline_aligned_in_smp;
 
+extern struct static_key_false xdp_egress_needed_key;
+
 /*
  * RX queue sysfs structures and functions.
  */

net/core/dev.c +34 ~~-14~~

90 lines not on this page · raw message

Signature · 1 line

2.21.1 (Apple Git-122.3)

David Ahern

author

2020-04-24 20:15 UTC #10 lore raw

PATCH v3 9/15 net: Support xdp in the Tx path for packets as an skb

In reply to David Ahern

From: David Ahern <[email protected]>

Add support to run Tx path program on packets about to hit the ndo_start_xmit function for a device. Only XDP_DROP and XDP_PASS are supported now. Conceptually, XDP_REDIRECT for this path can work the same as it does for the Rx path, but that support is left for a follow on series.

Signed-off-by David Ahern <[email protected]>

---
 include/linux/netdevice.h | 11 +++++++++
 net/core/dev.c            | 52 ++++++++++++++++++++++++++++++++++++++-
 2 files changed, 62 insertions(+), 1 deletion(-)

include/linux/netdevice.h +11 -0

index bfd8b6c8bf70..5c771a3ee921 100644
@@ -3715,6 +3715,7 @@ static inline void dev_consume_skb_any(struct sk_buff *skb)
 
 void generic_xdp_tx(struct sk_buff *skb, struct bpf_prog *xdp_prog);
 int do_xdp_generic_rx(struct bpf_prog *xdp_prog, struct sk_buff *skb);
+u32 do_xdp_egress_skb(struct net_device *dev, struct sk_buff *skb);
 int netif_rx(struct sk_buff *skb);
 int netif_rx_ni(struct sk_buff *skb);
 int netif_receive_skb(struct sk_buff *skb);

@@ -4575,6 +4576,16 @@ static inline netdev_tx_t __netdev_start_xmit(const struct net_device_ops *ops,
 					      struct sk_buff *skb, struct net_device *dev,
 					      bool more)
 {
+	if (static_branch_unlikely(&xdp_egress_needed_key)) {
+		u32 act;
+
+		rcu_read_lock();
+		act = do_xdp_egress_skb(dev, skb);
+		rcu_read_unlock();
+		if (act == XDP_DROP)
+			return NET_XMIT_DROP;
+	}
+
 	__this_cpu_write(softnet_data.xmit.more, more);
 	return ops->ndo_start_xmit(skb, dev);
 }

net/core/dev.c +51 -1

64 lines not on this page · raw message

Signature · 1 line

2.21.1 (Apple Git-122.3)

David Ahern

author

2020-04-24 20:14 UTC #11 lore raw

PATCH v3 10/15 net: Support xdp in the Tx path for xdp_frames

In reply to David Ahern

From: David Ahern <[email protected]>

Add support to run Tx path program on xdp_frames by adding a hook to bq_xmit_all before xdp_frames are passed to ndo_xdp_xmit for the device.

If an xdp_frame is dropped by the program, it is removed from the xdp_frames array with subsequent entries moved up.

Signed-off-by David Ahern <[email protected]>

---
 include/linux/netdevice.h |  3 ++
 kernel/bpf/devmap.c       | 19 +++++++++----
 net/core/dev.c            | 59 +++++++++++++++++++++++++++++++++++++++
 3 files changed, 76 insertions(+), 5 deletions(-)

include/linux/netdevice.h +3 -0

index 5c771a3ee921..f656d902fa2c 100644
@@ -3716,6 +3716,9 @@ static inline void dev_consume_skb_any(struct sk_buff *skb)
 void generic_xdp_tx(struct sk_buff *skb, struct bpf_prog *xdp_prog);
 int do_xdp_generic_rx(struct bpf_prog *xdp_prog, struct sk_buff *skb);
 u32 do_xdp_egress_skb(struct net_device *dev, struct sk_buff *skb);
+unsigned int do_xdp_egress_frame(struct net_device *dev,
+				 struct xdp_frame **frames,
+				 unsigned int count);
 int netif_rx(struct sk_buff *skb);
 int netif_rx_ni(struct sk_buff *skb);
 int netif_receive_skb(struct sk_buff *skb);

kernel/bpf/devmap.c +14 -5

index 58bdca5d978a..c038c8c3ccdb 100644
@@ -322,24 +322,33 @@ static int bq_xmit_all(struct xdp_dev_bulk_queue *bq, u32 flags)
 {
 	struct net_device *dev = bq->dev;
 	int sent = 0, drops = 0, err = 0;
+	unsigned int count = bq->count;
 	int i;
 
-	if (unlikely(!bq->count))
+	if (unlikely(!count))
 		return 0;
 
-	for (i = 0; i < bq->count; i++) {
+	for (i = 0; i < count; i++) {
 		struct xdp_frame *xdpf = bq->q[i];
 
 		prefetch(xdpf);
 	}
 
-	sent = dev->netdev_ops->ndo_xdp_xmit(dev, bq->count, bq->q, flags);
+	if (static_branch_unlikely(&xdp_egress_needed_key)) {
+		count = do_xdp_egress_frame(dev, bq->q, count);
+		drops += bq->count - count;
+		/* all frames consumed by the xdp program? */
+		if (!count)
+			goto out;
+	}
+
+	sent = dev->netdev_ops->ndo_xdp_xmit(dev, count, bq->q, flags);
 	if (sent < 0) {
 		err = sent;
 		sent = 0;
 		goto error;
 	}
-	drops = bq->count - sent;
+	drops += count - sent;
 out:
 	bq->count = 0;

@@ -351,7 +360,7 @@ static int bq_xmit_all(struct xdp_dev_bulk_queue *bq, u32 flags)
 	/* If ndo_xdp_xmit fails with an errno, no frames have been
 	 * xmit'ed and it's our responsibility to them free all.
 	 */
-	for (i = 0; i < bq->count; i++) {
+	for (i = 0; i < count; i++) {
 		struct xdp_frame *xdpf = bq->q[i];
 
 		xdp_return_frame_rx_napi(xdpf);

net/core/dev.c +59 -0

65 lines not on this page · raw message

Signature · 1 line

2.21.1 (Apple Git-122.3)

David Ahern

author

2020-04-24 20:15 UTC #12 lore raw

PATCH v3 11/15 libbpf: Add egress XDP support

In reply to David Ahern

From: David Ahern <[email protected]>

New section name hint, xdp_egress, is added to set expected attach type at program load. Programs can use xdp_egress as the prefix in the SEC statement to load the program with the BPF_XDP_EGRESS attach type set.

egress_prog_id is added to xdp_link_info to report the program id.

Signed-off-by David Ahern <[email protected]>

---
 tools/lib/bpf/libbpf.c  | 2 ++
 tools/lib/bpf/libbpf.h  | 1 +
 tools/lib/bpf/netlink.c | 6 ++++++
 3 files changed, 9 insertions(+)

tools/lib/bpf/libbpf.c +2 -0

index 8f480e29a6b0..32fc970495d9 100644
@@ -6366,6 +6366,8 @@ static const struct bpf_sec_def section_defs[] = {
 		.is_attach_btf = true,
 		.expected_attach_type = BPF_LSM_MAC,
 		.attach_fn = attach_lsm),
+	BPF_EAPROG_SEC("xdp_egress",		BPF_PROG_TYPE_XDP,
+						BPF_XDP_EGRESS),
 	BPF_PROG_SEC("xdp",			BPF_PROG_TYPE_XDP),
 	BPF_PROG_SEC("perf_event",		BPF_PROG_TYPE_PERF_EVENT),
 	BPF_PROG_SEC("lwt_in",			BPF_PROG_TYPE_LWT_IN),

tools/lib/bpf/libbpf.h +1 -0

index f1dacecb1619..445c3789faa4 100644
@@ -454,6 +454,7 @@ struct xdp_link_info {
 	__u32 hw_prog_id;
 	__u32 skb_prog_id;
 	__u8 attach_mode;
+	__u32 egress_prog_id;
 };
 
 struct bpf_xdp_set_link_opts {

tools/lib/bpf/netlink.c +6 -0

index 0b709fd10bba..072d3552248b 100644
@@ -280,6 +280,10 @@ static int get_xdp_info(void *cookie, void *msg, struct nlattr **tb)
 		xdp_id->info.hw_prog_id = libbpf_nla_getattr_u32(
 			xdp_tb[IFLA_XDP_HW_PROG_ID]);
 
+	if (xdp_tb[IFLA_XDP_EGRESS_PROG_ID])
+		xdp_id->info.egress_prog_id = libbpf_nla_getattr_u32(
+			xdp_tb[IFLA_XDP_EGRESS_PROG_ID]);
+
 	return 0;
 }

@@ -329,6 +333,8 @@ static __u32 get_xdp_id(struct xdp_link_info *info, __u32 flags)
 		return info->hw_prog_id;
 	if (flags & XDP_FLAGS_SKB_MODE)
 		return info->skb_prog_id;
+	if (flags & XDP_FLAGS_EGRESS_MODE)
+		return info->egress_prog_id;
 
 	return 0;
 }

Signature · 1 line

2.21.1 (Apple Git-122.3)

David Ahern

author

2020-04-24 20:14 UTC #13 lore raw

PATCH v3 12/15 bpftool: Add support for XDP egress

In reply to David Ahern

From: David Ahern <[email protected]>

Add xdp_egress as a program type since it requires a new attach type. This follows suit with other program type + attach type combintations and leverages the SEC name in libbpf.

Add NET_ATTACH_TYPE_XDP_EGRESS and update attach_type_strings to allow a user to specify 'xdp_egress' as the attach or detach point.

Update do_attach_detach_xdp to set XDP_FLAGS_EGRESS_MODE if egress is selected.

Update do_xdp_dump_one to show egress program ids.

Update the documentation and help output.

Signed-off-by David Ahern <[email protected]>

---
 tools/bpf/bpftool/Documentation/bpftool-net.rst  | 4 +++-
 tools/bpf/bpftool/Documentation/bpftool-prog.rst | 2 +-
 tools/bpf/bpftool/bash-completion/bpftool        | 4 ++--
 tools/bpf/bpftool/net.c                          | 6 +++++-
 tools/bpf/bpftool/netlink_dumper.c               | 5 +++++
 tools/bpf/bpftool/prog.c                         | 2 +-
 6 files changed, 17 insertions(+), 6 deletions(-)

tools/bpf/bpftool/Documentation/bpftool-net.rst +3 -1

index 8651b00b81ea..d7398fb00ec4 100644
@@ -26,7 +26,8 @@ NET COMMANDS
 |	**bpftool** **net help**
 |
 |	*PROG* := { **id** *PROG_ID* | **pinned** *FILE* | **tag** *PROG_TAG* }
-|	*ATTACH_TYPE* := { **xdp** | **xdpgeneric** | **xdpdrv** | **xdpoffload** }
+|	*ATTACH_TYPE* :=
+|       { **xdp** | **xdpgeneric** | **xdpdrv** | **xdpoffload** | **xdp_egress** }
 
 DESCRIPTION
 ===========

@@ -63,6 +64,7 @@ DESCRIPTION
                   **xdpgeneric** - Generic XDP. runs at generic XDP hook when packet already enters receive path as skb;
                   **xdpdrv** - Native XDP. runs earliest point in driver's receive path;
                   **xdpoffload** - Offload XDP. runs directly on NIC on each packet reception;
+                  **xdp_egress** - XDP in egress path. runs at core networking level;
 
 	**bpftool** **net detach** *ATTACH_TYPE* **dev** *NAME*
                   Detach bpf program attached to network interface *NAME* with

tools/bpf/bpftool/Documentation/bpftool-prog.rst +1 -1

index 9f19404f470e..ab0a8846a8e3 100644
@@ -44,7 +44,7 @@ PROG COMMANDS
 |		**cgroup/connect4** | **cgroup/connect6** | **cgroup/sendmsg4** | **cgroup/sendmsg6** |
 |		**cgroup/recvmsg4** | **cgroup/recvmsg6** | **cgroup/sysctl** |
 |		**cgroup/getsockopt** | **cgroup/setsockopt** |
-|		**struct_ops** | **fentry** | **fexit** | **freplace**
+|		**struct_ops** | **fentry** | **fexit** | **freplace** | **xdp_egress**
 |	}
 |       *ATTACH_TYPE* := {
 |		**msg_verdict** | **stream_verdict** | **stream_parser** | **flow_dissector**

tools/bpf/bpftool/bash-completion/bpftool +2 -2

index 45ee99b159e2..ab20696c20c6 100644
@@ -471,7 +471,7 @@ _bpftool()
                                 cgroup/post_bind4 cgroup/post_bind6 \
                                 cgroup/sysctl cgroup/getsockopt \
                                 cgroup/setsockopt struct_ops \
-                                fentry fexit freplace" -- \
+                                fentry fexit freplace xdp_egress" -- \
                                                    "$cur" ) )
                             return 0
                             ;;

@@ -1003,7 +1003,7 @@ _bpftool()
             ;;
         net)
             local PROG_TYPE='id pinned tag name'
-            local ATTACH_TYPES='xdp xdpgeneric xdpdrv xdpoffload'
+            local ATTACH_TYPES='xdp xdpgeneric xdpdrv xdpoffload xdp_egress'
             case $command in
                 show|list)
                     [[ $prev != "$command" ]] && return 0

tools/bpf/bpftool/net.c +5 -1

index c5e3895b7c8b..dbace14e5484 100644
@@ -61,6 +61,7 @@ enum net_attach_type {
 	NET_ATTACH_TYPE_XDP_GENERIC,
 	NET_ATTACH_TYPE_XDP_DRIVER,
 	NET_ATTACH_TYPE_XDP_OFFLOAD,
+	NET_ATTACH_TYPE_XDP_EGRESS,
 };
 
 static const char * const attach_type_strings[] = {

@@ -68,6 +69,7 @@ static const char * const attach_type_strings[] = {
 	[NET_ATTACH_TYPE_XDP_GENERIC]	= "xdpgeneric",
 	[NET_ATTACH_TYPE_XDP_DRIVER]	= "xdpdrv",
 	[NET_ATTACH_TYPE_XDP_OFFLOAD]	= "xdpoffload",
+	[NET_ATTACH_TYPE_XDP_EGRESS]	= "xdp_egress",
 };
 
 const size_t net_attach_type_size = ARRAY_SIZE(attach_type_strings);

@@ -286,6 +288,8 @@ static int do_attach_detach_xdp(int progfd, enum net_attach_type attach_type,
 		flags |= XDP_FLAGS_DRV_MODE;
 	if (attach_type == NET_ATTACH_TYPE_XDP_OFFLOAD)
 		flags |= XDP_FLAGS_HW_MODE;
+	if (attach_type == NET_ATTACH_TYPE_XDP_EGRESS)
+		flags |= XDP_FLAGS_EGRESS_MODE;
 
 	return bpf_set_link_xdp_fd(ifindex, progfd, flags);
 }

@@ -464,7 +468,7 @@ static int do_help(int argc, char **argv)
 		"       %s %s help\n"
 		"\n"
 		"       " HELP_SPEC_PROGRAM "\n"
-		"       ATTACH_TYPE := { xdp | xdpgeneric | xdpdrv | xdpoffload }\n"
+		"       ATTACH_TYPE := { xdp | xdpgeneric | xdpdrv | xdpoffload | xdp_egress}\n"
 		"\n"
 		"Note: Only xdp and tc attachments are supported now.\n"
 		"      For progs attached to cgroups, use \"bpftool cgroup\"\n"

tools/bpf/bpftool/netlink_dumper.c +5 -0

index 5f65140b003b..68e4909b6073 100644
@@ -55,6 +55,8 @@ static int do_xdp_dump_one(struct nlattr *attr, unsigned int ifindex,
 		xdp_dump_prog_id(tb, IFLA_XDP_SKB_PROG_ID, "generic", true);
 		xdp_dump_prog_id(tb, IFLA_XDP_DRV_PROG_ID, "driver", true);
 		xdp_dump_prog_id(tb, IFLA_XDP_HW_PROG_ID, "offload", true);
+		xdp_dump_prog_id(tb, IFLA_XDP_EGRESS_PROG_ID,
+				 "egress", true);
 		if (json_output)
 			jsonw_end_array(json_wtr);
 	} else if (mode == XDP_ATTACHED_DRV) {

@@ -63,6 +65,9 @@ static int do_xdp_dump_one(struct nlattr *attr, unsigned int ifindex,
 		xdp_dump_prog_id(tb, IFLA_XDP_PROG_ID, "generic", false);
 	} else if (mode == XDP_ATTACHED_HW) {
 		xdp_dump_prog_id(tb, IFLA_XDP_PROG_ID, "offload", false);
+	} else if (mode == XDP_ATTACHED_EGRESS_CORE) {
+		xdp_dump_prog_id(tb, IFLA_XDP_EGRESS_PROG_ID,
+				 "egress", false);
 	}
 
 	NET_END_OBJECT_FINAL;

tools/bpf/bpftool/prog.c +1 -1

index f6a5974a7b0a..64695ccdcf9d 100644
@@ -2014,7 +2014,7 @@ static int do_help(int argc, char **argv)
 		"                 cgroup/post_bind6 | cgroup/connect4 | cgroup/connect6 |\n"
 		"                 cgroup/sendmsg4 | cgroup/sendmsg6 | cgroup/recvmsg4 |\n"
 		"                 cgroup/recvmsg6 | cgroup/getsockopt | cgroup/setsockopt |\n"
-		"                 struct_ops | fentry | fexit | freplace }\n"
+		"                 struct_ops | fentry | fexit | freplace | xdp_egress }\n"
 		"       ATTACH_TYPE := { msg_verdict | stream_verdict | stream_parser |\n"
 		"                        flow_dissector }\n"
 		"       METRIC := { cycles | instructions | l1d_loads | llc_misses }\n"

Signature · 1 line

2.21.1 (Apple Git-122.3)

Toke Høiland-Jørgensen

2020-04-27 15:13 UTC #14 lore raw

Re: [PATCH v3 bpf-next 12/15] bpftool: Add support for XDP egress

Quoted 113 lines · David Ahern

From: David Ahern <[email protected]>

Add xdp_egress as a program type since it requires a new attach type. This follows suit with other program type + attach type combintations and leverages the SEC name in libbpf.

Add NET_ATTACH_TYPE_XDP_EGRESS and update attach_type_strings to allow a user to specify 'xdp_egress' as the attach or detach point.

Update do_attach_detach_xdp to set XDP_FLAGS_EGRESS_MODE if egress is selected.

Update do_xdp_dump_one to show egress program ids.

Update the documentation and help output.

Signed-off-by David Ahern <[email protected]>

---
 tools/bpf/bpftool/Documentation/bpftool-net.rst  | 4 +++-
 tools/bpf/bpftool/Documentation/bpftool-prog.rst | 2 +-
 tools/bpf/bpftool/bash-completion/bpftool        | 4 ++--
 tools/bpf/bpftool/net.c                          | 6 +++++-
 tools/bpf/bpftool/netlink_dumper.c               | 5 +++++
 tools/bpf/bpftool/prog.c                         | 2 +-
 6 files changed, 17 insertions(+), 6 deletions(-)

tools/bpf/bpftool/Documentation/bpftool-net.rst +3 -1

index 8651b00b81ea..d7398fb00ec4 100644
@@ -26,7 +26,8 @@ NET COMMANDS
 |	**bpftool** **net help**
 |
 |	*PROG* := { **id** *PROG_ID* | **pinned** *FILE* | **tag** *PROG_TAG* }
-|	*ATTACH_TYPE* := { **xdp** | **xdpgeneric** | **xdpdrv** | **xdpoffload** }
+|	*ATTACH_TYPE* :=
+|       { **xdp** | **xdpgeneric** | **xdpdrv** | **xdpoffload** | **xdp_egress** }
 
 DESCRIPTION
 ===========

@@ -63,6 +64,7 @@ DESCRIPTION
                   **xdpgeneric** - Generic XDP. runs at generic XDP hook when packet already enters receive path as skb;
                   **xdpdrv** - Native XDP. runs earliest point in driver's receive path;
                   **xdpoffload** - Offload XDP. runs directly on NIC on each packet reception;
+                  **xdp_egress** - XDP in egress path. runs at core networking level;
 
 	**bpftool** **net detach** *ATTACH_TYPE* **dev** *NAME*
                   Detach bpf program attached to network interface *NAME* with

tools/bpf/bpftool/Documentation/bpftool-prog.rst +1 -1

index 9f19404f470e..ab0a8846a8e3 100644
@@ -44,7 +44,7 @@ PROG COMMANDS
 |		**cgroup/connect4** | **cgroup/connect6** | **cgroup/sendmsg4** | **cgroup/sendmsg6** |
 |		**cgroup/recvmsg4** | **cgroup/recvmsg6** | **cgroup/sysctl** |
 |		**cgroup/getsockopt** | **cgroup/setsockopt** |
-|		**struct_ops** | **fentry** | **fexit** | **freplace**
+|		**struct_ops** | **fentry** | **fexit** | **freplace** | **xdp_egress**
 |	}
 |       *ATTACH_TYPE* := {
 |		**msg_verdict** | **stream_verdict** | **stream_parser** | **flow_dissector**

tools/bpf/bpftool/bash-completion/bpftool +2 -2

index 45ee99b159e2..ab20696c20c6 100644
@@ -471,7 +471,7 @@ _bpftool()
                                 cgroup/post_bind4 cgroup/post_bind6 \
                                 cgroup/sysctl cgroup/getsockopt \
                                 cgroup/setsockopt struct_ops \
-                                fentry fexit freplace" -- \
+                                fentry fexit freplace xdp_egress" -- \
                                                    "$cur" ) )
                             return 0
                             ;;

@@ -1003,7 +1003,7 @@ _bpftool()
             ;;
         net)
             local PROG_TYPE='id pinned tag name'
-            local ATTACH_TYPES='xdp xdpgeneric xdpdrv xdpoffload'
+            local ATTACH_TYPES='xdp xdpgeneric xdpdrv xdpoffload xdp_egress'
             case $command in
                 show|list)
                     [[ $prev != "$command" ]] && return 0

tools/bpf/bpftool/net.c +4 -0

index c5e3895b7c8b..dbace14e5484 100644
@@ -61,6 +61,7 @@ enum net_attach_type {
 	NET_ATTACH_TYPE_XDP_GENERIC,
 	NET_ATTACH_TYPE_XDP_DRIVER,
 	NET_ATTACH_TYPE_XDP_OFFLOAD,
+	NET_ATTACH_TYPE_XDP_EGRESS,
 };
 
 static const char * const attach_type_strings[] = {

@@ -68,6 +69,7 @@ static const char * const attach_type_strings[] = {
 	[NET_ATTACH_TYPE_XDP_GENERIC]	= "xdpgeneric",
 	[NET_ATTACH_TYPE_XDP_DRIVER]	= "xdpdrv",
 	[NET_ATTACH_TYPE_XDP_OFFLOAD]	= "xdpoffload",
+	[NET_ATTACH_TYPE_XDP_EGRESS]	= "xdp_egress",
 };
 
 const size_t net_attach_type_size = ARRAY_SIZE(attach_type_strings);

@@ -286,6 +288,8 @@ static int do_attach_detach_xdp(int progfd, enum net_attach_type attach_type,
 		flags |= XDP_FLAGS_DRV_MODE;
 	if (attach_type == NET_ATTACH_TYPE_XDP_OFFLOAD)
 		flags |= XDP_FLAGS_HW_MODE;
+	if (attach_type == NET_ATTACH_TYPE_XDP_EGRESS)
+		flags |= XDP_FLAGS_EGRESS_MODE;
 
 	return bpf_set_link_xdp_fd(ifindex, progfd, flags);
 }

tools/bpf/bpftool/net.c static int do_help(int argc, char **argv) · line 471

Quoted 5 lines

tools/bpf/bpftool/net.c +1 -1

@@ -464,7 +468,7 @@ static int do_help(int argc, char **argv)
 		"       %s %s help\n"
 		"\n"
 		"       " HELP_SPEC_PROGRAM "\n"
-		"       ATTACH_TYPE := { xdp | xdpgeneric | xdpdrv | xdpoffload }\n"
+		"       ATTACH_TYPE := { xdp | xdpgeneric | xdpdrv | xdpoffload | xdp_egress}\n"

Nit: Missing space before }

David Ahern

author

2020-04-24 20:14 UTC #15 lore raw

PATCH v3 13/15 selftest: Add test for xdp_egress

In reply to David Ahern

From: David Ahern <[email protected]>

Add selftest for xdp_egress. Add xdp_drop program to veth connecting a namespace to drop packets and break connectivity.

Signed-off-by David Ahern <[email protected]>

---
 tools/testing/selftests/bpf/Makefile          |   1 +
 tools/testing/selftests/bpf/progs/xdp_drop.c  |  25 +++
 .../testing/selftests/bpf/test_xdp_egress.sh  | 160 ++++++++++++++++++
 3 files changed, 186 insertions(+)
 create mode 100644 tools/testing/selftests/bpf/progs/xdp_drop.c
 create mode 100755 tools/testing/selftests/bpf/test_xdp_egress.sh

tools/testing/selftests/bpf/Makefile +1 -0

index 7729892e0b04..5dae18ebac13 100644
@@ -50,6 +50,7 @@ TEST_PROGS := test_kmod.sh \
 	test_xdp_redirect.sh \
 	test_xdp_meta.sh \
 	test_xdp_veth.sh \
+	test_xdp_egress.sh \
 	test_offload.py \
 	test_sock_addr.sh \
 	test_tunnel.sh \

tools/testing/selftests/bpf/progs/xdp_drop.c new file +25 -0

new file mode 100644
index 000000000000..cffabc53a5e1
@@ -0,0 +1,25 @@
+// SPDX-License-Identifier: GPL-2.0
+
+#include <linux/bpf.h>
+#include <linux/if_ether.h>
+#include <bpf/bpf_helpers.h>
+
+SEC("drop")
+int xdp_drop(struct xdp_md *ctx)
+{
+	void *data_end = (void *)(long)ctx->data_end;
+	void *data = (void *)(long)ctx->data;
+	struct ethhdr *eth = data;
+	void *nh;
+
+	nh = data + sizeof(*eth);
+	if (nh > data_end)
+		return XDP_DROP;
+
+	if (eth->h_proto == 0x0008)
+		return XDP_DROP;
+
+	return XDP_PASS;
+}
+
+char _license[] SEC("license") = "GPL";

tools/testing/selftests/bpf/test_xdp_egress.sh new file +160 -0

160 lines not on this page · raw message

Signature · 1 line

2.21.1 (Apple Git-122.3)

John Fastabend

maintainer

2020-04-27 17:48 UTC #16 lore raw

RE: [PATCH v3 bpf-next 13/15] selftest: Add test for xdp_egress

Quoted 7 lines · David Ahern

From: David Ahern <[email protected]>
Add selftest for xdp_egress. Add xdp_drop program to veth connecting a namespace to drop packets and break connectivity.
Signed-off-by David Ahern <[email protected]>
---

[...]

Quoted 12 lines

tools/testing/selftests/bpf/test_xdp_egress.sh new file +12 -0

+################################################################################
+# main
+
+if [ $(id -u) -ne 0 ]; then
+	echo "selftests: $TESTNAME [SKIP] Need root privileges"
+	exit $ksft_skip
+fi
+
+if ! ip link set dev lo xdp off > /dev/null 2>&1; then
+	echo "selftests: $TESTNAME [SKIP] Could not run test without the ip xdp support"
+	exit $ksft_skip
+fi

tools/testing/selftests/bpf/test_xdp_egress.sh line 128

Quoted 10 lines

tools/testing/selftests/bpf/test_xdp_egress.sh new file +10 -0

+
+if [ -z "$BPF_FS" ]; then
+	echo "selftests: $TESTNAME [SKIP] Could not run test without bpffs mounted"
+	exit $ksft_skip
+fi
+
+if ! bpftool version > /dev/null 2>&1; then
+	echo "selftests: $TESTNAME [SKIP] Could not run test without bpftool"
+	exit $ksft_skip
+fi

This is consistent with test_xdp_veth.sh so it is probably ok for this series but I think it would be nice to go back and make these fail on errors. Or at least fail on bpffs mount. Seems other tests fail on bpffs mount failures so would be OK I think.

Otherwise LGTM.

John Fastabend

maintainer

2020-04-27 17:52 UTC #17 lore raw

RE: [PATCH v3 bpf-next 13/15] selftest: Add test for xdp_egress

Quoted 39 lines · John Fastabend

Quoted 7 lines · David Ahern
From: David Ahern <[email protected]>
Add selftest for xdp_egress. Add xdp_drop program to veth connecting a namespace to drop packets and break connectivity.
Signed-off-by David Ahern <[email protected]>
---
[...]
Quoted 12 lines
tools/testing/selftests/bpf/test_xdp_egress.sh new file +12 -0
+################################################################################
+# main
+
+if [ $(id -u) -ne 0 ]; then
+	echo "selftests: $TESTNAME [SKIP] Need root privileges"
+	exit $ksft_skip
+fi
+
+if ! ip link set dev lo xdp off > /dev/null 2>&1; then
+	echo "selftests: $TESTNAME [SKIP] Could not run test without the ip xdp support"
+	exit $ksft_skip
+fi
tools/testing/selftests/bpf/test_xdp_egress.sh line 128
Quoted 10 lines
tools/testing/selftests/bpf/test_xdp_egress.sh new file +10 -0
+
+if [ -z "$BPF_FS" ]; then
+	echo "selftests: $TESTNAME [SKIP] Could not run test without bpffs mounted"
+	exit $ksft_skip
+fi
+
+if ! bpftool version > /dev/null 2>&1; then
+	echo "selftests: $TESTNAME [SKIP] Could not run test without bpftool"
+	exit $ksft_skip
+fi
This is consistent with test_xdp_veth.sh so it is probably ok for this series but I think it would be nice to go back and make these fail on errors. Or at least fail on bpffs mount. Seems other tests fail on bpffs mount failures so would be OK I think.

Quoted 1 line

Otherwise LGTM.

Also would be nice to have a test case for the xdp redirect into tx case. I think this is only going to test the skb path?

Thanks,
John

David Ahern

2020-04-27 17:58 UTC #18 lore raw

Re: [PATCH v3 bpf-next 13/15] selftest: Add test for xdp_egress

Quoted 2 lines · John Fastabend

Also would be nice to have a test case for the xdp redirect into tx case. I think this is only going to test the skb path?

I have tests for the xdp_frame case, but it requires multiple nodes - not namespaces. Multiple node tests no longer fall into the 'selftest' category. I am not aware of any way to do XDP_REDIRECT testing within a single node; general shortcoming with all xdp_frame specific tests.

David Ahern

author

2020-04-24 20:14 UTC #19 lore raw

PATCH v3 14/15 selftest: Add xdp_egress attach tests

In reply to David Ahern

From: David Ahern <[email protected]>

Add xdp_egress attach tests:
1. verify egress programs cannot access ingress entries in xdp context
2. verify ability to load, attach, and detach xdp egress to a device.

Signed-off-by David Ahern <[email protected]>

---
 .../bpf/prog_tests/xdp_egress_attach.c        | 56 +++++++++++++++++++
 .../selftests/bpf/progs/test_xdp_egress.c     | 12 ++++
 .../bpf/progs/test_xdp_egress_fail.c          | 16 ++++++
 3 files changed, 84 insertions(+)
 create mode 100644 tools/testing/selftests/bpf/prog_tests/xdp_egress_attach.c
 create mode 100644 tools/testing/selftests/bpf/progs/test_xdp_egress.c
 create mode 100644 tools/testing/selftests/bpf/progs/test_xdp_egress_fail.c

tools/testing/selftests/bpf/prog_tests/xdp_egress_attach.c new file +56 -0

new file mode 100644
index 000000000000..5253754b27de
@@ -0,0 +1,56 @@
+// SPDX-License-Identifier: GPL-2.0
+#include <linux/if_link.h>
+#include <test_progs.h>
+
+#define IFINDEX_LO 1
+
+void test_xdp_egress_attach(void)
+{
+	struct bpf_prog_load_attr attr = {
+		.prog_type = BPF_PROG_TYPE_XDP,
+		.expected_attach_type = BPF_XDP_EGRESS,
+	};
+	struct bpf_prog_info info = {};
+	__u32 id, len = sizeof(info);
+	struct bpf_object *obj;
+	__u32 duration = 0;
+	int err, fd = -1;
+
+	/* should fail - accesses rx queue info */
+	attr.file = "./test_xdp_egress_fail.o",
+	err = bpf_prog_load_xattr(&attr, &obj, &fd);
+	if (CHECK(err == 0 && fd >= 0, "xdp_egress with rx failed to load",
+		 "load of xdp_egress with rx succeeded instead of failed"))
+		return;
+
+	attr.file = "./test_xdp_egress.o",
+	err = bpf_prog_load_xattr(&attr, &obj, &fd);
+	if (CHECK_FAIL(err))
+		return;
+
+	err = bpf_obj_get_info_by_fd(fd, &info, &len);
+	if (CHECK_FAIL(err))
+		goto out_close;
+
+	err = bpf_set_link_xdp_fd(IFINDEX_LO, fd, XDP_FLAGS_EGRESS_MODE);
+	if (CHECK(err, "xdp attach", "xdp attach failed"))
+		goto out_close;
+
+	err = bpf_get_link_xdp_id(IFINDEX_LO, &id, XDP_FLAGS_EGRESS_MODE);
+	if (CHECK(err || id != info.id, "id_check",
+		  "loaded prog id %u != id %u, err %d", info.id, id, err))
+		goto out;
+
+out:
+	err = bpf_set_link_xdp_fd(IFINDEX_LO, -1, XDP_FLAGS_EGRESS_MODE);
+	if (CHECK(err, "xdp detach", "xdp detach failed"))
+		goto out_close;
+
+	err = bpf_get_link_xdp_id(IFINDEX_LO, &id, XDP_FLAGS_EGRESS_MODE);
+	if (CHECK(err || id, "id_check",
+		  "failed to detach program %u", id))
+		goto out;
+
+out_close:
+	bpf_object__close(obj);
+}

tools/testing/selftests/bpf/progs/test_xdp_egress.c new file +12 -0

new file mode 100644
index 000000000000..0477e8537b7f
@@ -0,0 +1,12 @@
+// SPDX-License-Identifier: GPL-2.0
+
+#include <linux/bpf.h>
+#include <bpf/bpf_helpers.h>
+
+SEC("xdp_egress")
+int xdp_egress_good(struct xdp_md *ctx)
+{
+	__u32 idx = ctx->egress_ifindex;
+
+	return idx == 1 ? XDP_DROP : XDP_PASS;
+}

tools/testing/selftests/bpf/progs/test_xdp_egress_fail.c new file +16 -0

new file mode 100644
index 000000000000..76b47b1d3bc3
@@ -0,0 +1,16 @@
+// SPDX-License-Identifier: GPL-2.0
+
+#include <linux/bpf.h>
+#include <bpf/bpf_helpers.h>
+
+SEC("xdp_egress")
+int xdp_egress_fail(struct xdp_md *ctx)
+{
+	__u32 rxq = ctx->rx_queue_index;
+	__u32 idx = ctx->ingress_ifindex;
+
+	if (idx == 1)
+		return XDP_DROP;
+
+	return rxq ? XDP_DROP : XDP_PASS;
+}

Signature · 1 line

2.21.1 (Apple Git-122.3)

David Ahern

author

2020-04-24 20:14 UTC #20 lore raw

PATCH v3 15/15 samples/bpf: add XDP egress support to xdp1

In reply to David Ahern

From: David Ahern <[email protected]>

xdp1 and xdp2 now accept -E flag to set XDP program in the egress path.

Signed-off-by Prashant Bhole <[email protected]>

Signed-off-by David Ahern <[email protected]>

---
 samples/bpf/xdp1_user.c | 11 ++++++++---
 1 file changed, 8 insertions(+), 3 deletions(-)

samples/bpf/xdp1_user.c +8 -3

index c447ad9e3a1d..bb104f4d8c5e 100644
@@ -73,7 +73,8 @@ static void usage(const char *prog)
 		"OPTS:\n"
 		"    -S    use skb-mode\n"
 		"    -N    enforce native mode\n"
-		"    -F    force loading prog\n",
+		"    -F    force loading prog\n"
+		"    -E	   egress path program\n",
 		prog);
 }

@@ -85,7 +86,7 @@ int main(int argc, char **argv)
 	};
 	struct bpf_prog_info info = {};
 	__u32 info_len = sizeof(info);
-	const char *optstr = "FSN";
+	const char *optstr = "FSNE";
 	int prog_fd, map_fd, opt;
 	struct bpf_object *obj;
 	struct bpf_map *map;

@@ -103,13 +104,17 @@ int main(int argc, char **argv)
 		case 'F':
 			xdp_flags &= ~XDP_FLAGS_UPDATE_IF_NOEXIST;
 			break;
+		case 'E':
+			xdp_flags |= XDP_FLAGS_EGRESS_MODE;
+			prog_load_attr.expected_attach_type = BPF_XDP_EGRESS;
+			break;
 		default:
 			usage(basename(argv[0]));
 			return 1;
 		}
 	}
 
-	if (!(xdp_flags & XDP_FLAGS_SKB_MODE))
+	if (!(xdp_flags & (XDP_FLAGS_SKB_MODE | XDP_FLAGS_EGRESS_MODE)))
 		xdp_flags |= XDP_FLAGS_DRV_MODE;
 
 	if (optind == argc) {

Signature · 1 line

2.21.1 (Apple Git-122.3)

Toke Høiland-Jørgensen

Acked-by on the v3 series

2020-04-27 15:21 UTC #21 lore raw

In reply to David Ahern

Quoted 4 lines · David Ahern

From: David Ahern <[email protected]>
This series adds support for XDP in the egress path by introducing a new XDP attachment type, BPF_XDP_EGRESS

Can't find anything more to complain about :)

Ran a quick performance test: On a test using xdp_redirect_map from samples/bpf, which gets 8.15 Mpps normally, loading an XDP egress program on the target interface drops performance to 7.55 Mpps. So ~600k pps, or ~9.5ns overhead for the egress program. So other than the nit in the bpftool patch, please consider the series:

Acked-by Toke Høiland-Jørgensen <[email protected]>

John Fastabend

maintainer

2020-04-27 18:25 UTC #22 lore raw

In reply to David Ahern

Quoted 10 lines · David Ahern

From: David Ahern <[email protected]>
This series adds support for XDP in the egress path by introducing a new XDP attachment type, BPF_XDP_EGRESS, and adding a UAPI to if_link.h for attaching the program to a netdevice and reporting the program. bpf programs can be run on all packets in the Tx path - skbs or redirected xdp frames. The intent is to emulate the current RX path for XDP as much as possible to maintain consistency and symmetry in the 2 paths with their APIs.

Quoted 6 lines

This is a missing primitive for XDP allowing solutions to build small, targeted programs properly distributed in the networking path allowing, for example, an egress firewall/ACL/traffic verification or packet manipulation and encapping an entire ethernet frame whether it is locally generated traffic, forwarded via the slow path (ie., full stack processing) or xdp redirected frames.

I'm still a bit unsure why the BPF programs would not push logic into ingress XDP program + skb egress. Is there a case where that does not work or is it mostly about ease of use for some use case?

Do we have overhead performance numbers? I'm wondering how close the redirect case with these TX hooks are vs redirect without TX hooks. The main reason I ask is if it slows performance down by more than say 5% (sort of made up number, but point is some N%) then I don't think we would recommend using it.

Quoted 9 lines

Nothing about running a program in the Tx path requires driver specific resources like the Rx path has. Thus, programs can be run in core code and attached to the net_device struct similar to skb mode. The egress attach is done using the new XDP_FLAGS_EGRESS_MODE flag, and is reported by the kernel using the XDP_ATTACHED_EGRESS_CORE attach flag with IFLA_XDP_EGRESS_PROG_ID making the api similar to existing APIs for XDP.

Quoted 8 lines

The locations chosen to run the egress program - __netdev_start_xmit before the call to ndo_start_xmit and bq_xmit_all before invoking ndo_xdp_xmit - allow follow on patch sets to handle tx queueing and setting the queue index if multi-queue with consistency in handling both packet formats.
A few of the patches trace back to work done on offloading programs from a VM by Jason Wang and Prashant Bole.

The idea for offloading VM programs would be to take a BPF program from the VM somehow out of band or over mgmt interface and load it into the egress hook of virtio?

Code LGTM other than a couple suggestions on the test side but I'm missing something on the use case picture.

Thanks,
John

David Ahern

2020-04-27 18:58 UTC #23 lore raw

Quoted 18 lines · John Fastabend

Quoted 10 lines · David Ahern
From: David Ahern <[email protected]>
This series adds support for XDP in the egress path by introducing a new XDP attachment type, BPF_XDP_EGRESS, and adding a UAPI to if_link.h for attaching the program to a netdevice and reporting the program. bpf programs can be run on all packets in the Tx path - skbs or redirected xdp frames. The intent is to emulate the current RX path for XDP as much as possible to maintain consistency and symmetry in the 2 paths with their APIs.
Quoted 6 lines
This is a missing primitive for XDP allowing solutions to build small, targeted programs properly distributed in the networking path allowing, for example, an egress firewall/ACL/traffic verification or packet manipulation and encapping an entire ethernet frame whether it is locally generated traffic, forwarded via the slow path (ie., full stack processing) or xdp redirected frames.

Quoted 3 lines

I'm still a bit unsure why the BPF programs would not push logic into ingress XDP program + skb egress. Is there a case where that does not work or is it mostly about ease of use for some use case?

host and VMs.

Some packets take the XDP fast path (known unicast traffic redirected from host ingress to VM tap or from one VM tap to another VM tap); some packets take the slow path. Regardless of path, each VM can have its own per-VM data (e.g., ingress ACL). With XDP egress programs getting both packet formats, that per-VM ACL config only needs to be in 1 place - egress program map.

Quoted 6 lines

Do we have overhead performance numbers? I'm wondering how close the redirect case with these TX hooks are vs redirect without TX hooks. The main reason I ask is if it slows performance down by more than say 5% (sort of made up number, but point is some N%) then I don't think we would recommend using it.

Toke ran some tests:
"On a test using xdp_redirect_map from samples/bpf, which gets 8.15 Mpps normally, loading an XDP egress program on the target interface drops performance to 7.55 Mpps. So ~600k pps, or ~9.5ns overhead for the egress program."

###

If XDP redirect gives a 2-4x speedup over skb path but the existence of an egress program takes away ~8-10% of that speedup, it is still an overall huge win in performance with a much simpler design, architecture and lifecycle management of per-VM data.

Quoted 16 lines

Quoted 16 lines
Nothing about running a program in the Tx path requires driver specific resources like the Rx path has. Thus, programs can be run in core code and attached to the net_device struct similar to skb mode. The egress attach is done using the new XDP_FLAGS_EGRESS_MODE flag, and is reported by the kernel using the XDP_ATTACHED_EGRESS_CORE attach flag with IFLA_XDP_EGRESS_PROG_ID making the api similar to existing APIs for XDP.
The locations chosen to run the egress program - __netdev_start_xmit before the call to ndo_start_xmit and bq_xmit_all before invoking ndo_xdp_xmit - allow follow on patch sets to handle tx queueing and setting the queue index if multi-queue with consistency in handling both packet formats.

Quoted 6 lines

Quoted 2 lines
A few of the patches trace back to work done on offloading programs from a VM by Jason Wang and Prashant Bole.
The idea for offloading VM programs would be to take a BPF program from the VM somehow out of band or over mgmt interface and load it into the egress hook of virtio?

The latest thought is for programs to run in the vhost thread. Offloaded programs for a guest should run in process context where the cycles can be associated with the VM.

Quoted 3 lines

Code LGTM other than a couple suggestions on the test side but I'm missing something on the use case picture.

thanks for the review.