Support defragmenting IPv(4|6) packets in BPF
Subsystems (6)
v3 cover Support defragmenting IPv(4|6) packets in BPF
=== Context ===
In the context of a middlebox, fragmented packets are tricky to handle. The full 5-tuple of a packet is often only available in the first fragment which makes enforcing consistent policy difficult. There are really only two stateless options, neither of which are very nice:
1. Enforce policy on first fragment and accept all subsequent fragments. This works but may let in certain attacks or allow data exfiltration.
2. Enforce policy on first fragment and drop all subsequent fragments. This does not really work b/c some protocols may rely on fragmentation. For example, DNS may rely on oversized UDP packets for large responses.
So stateful tracking is the only sane option. RFC 8900 [0] calls this out as well in section 6.3:
Middleboxes [...] should process IP fragments in a manner that is consistent with [RFC0791] and [RFC8200]. In many cases, middleboxes must maintain state in order to achieve this goal.
=== BPF related bits ===
Policy has traditionally been enforced from XDP/TC hooks. Both hooks run before kernel reassembly facilities. However, with the new BPF_PROG_TYPE_NETFILTER, we can rather easily hook into existing netfilter reassembly infra.
The basic idea is we bump a refcnt on the netfilter defrag module and then run the bpf prog after the defrag module runs. This allows bpf progs to transparently see full, reassembled packets. The nice thing about this is that progs don't have to carry around logic to detect fragments.
=== Changelog ===
Changes from v2:
* module_put() if ->enable() fails
* Fix CI build errors
Changes from v1:
* Drop bpf_program__attach_netfilter() patches
* static -> static const where appropriate
* Fix callback assignment order during registration
* Only request_module() if callbacks are missing
* Fix retval when modprobe fails in userspace
* Fix v6 defrag module name (nf_defrag_ipv6_hooks -> nf_defrag_ipv6)
* Simplify priority checking code
* Add warning if module doesn't assign callbacks in the future
* Take refcnt on module while defrag link is active
[0]: https://datatracker.ietf.org/doc/html/rfc8900
Daniel Xu (6):
netfilter: defrag: Add glue hooks for enabling/disabling defrag
netfilter: bpf: Support BPF_F_NETFILTER_IP_DEFRAG in netfilter link
netfilter: bpf: Prevent defrag module unload while link active
bpf: selftests: Support not connecting client socket
bpf: selftests: Support custom type and proto for client sockets
bpf: selftests: Add defrag selftests
include/linux/netfilter.h | 15 + include/uapi/linux/bpf.h | 5 + net/ipv4/netfilter/nf_defrag_ipv4.c | 17 +- net/ipv6/netfilter/nf_defrag_ipv6_hooks.c | 11 + net/netfilter/core.c | 6 + net/netfilter/nf_bpf_link.c | 150 +++++++++- tools/include/uapi/linux/bpf.h | 5 + tools/testing/selftests/bpf/Makefile | 4 +- .../selftests/bpf/generate_udp_fragments.py | 90 ++++++ .../selftests/bpf/ip_check_defrag_frags.h | 57 ++++ tools/testing/selftests/bpf/network_helpers.c | 26 +- tools/testing/selftests/bpf/network_helpers.h | 3 + .../bpf/prog_tests/ip_check_defrag.c | 282 ++++++++++++++++++ .../selftests/bpf/progs/ip_check_defrag.c | 104 +++++++ 14 files changed, 753 insertions(+), 22 deletions(-) create mode 100755 tools/testing/selftests/bpf/generate_udp_fragments.py create mode 100644 tools/testing/selftests/bpf/ip_check_defrag_frags.h create mode 100644 tools/testing/selftests/bpf/prog_tests/ip_check_defrag.c create mode 100644 tools/testing/selftests/bpf/progs/ip_check_defrag.c
Signature · 1 line
2.41.0
PATCH v3 1/6 netfilter: defrag: Add glue hooks for enabling/disabling defrag
We want to be able to enable/disable IP packet defrag from core bpf/netfilter code. In other words, execute code from core that could possibly be built as a module.
To help avoid symbol resolution errors, use glue hooks that the modules will register callbacks with during module init.
--- include/linux/netfilter.h | 12 ++++++++++++ net/ipv4/netfilter/nf_defrag_ipv4.c | 16 +++++++++++++++- net/ipv6/netfilter/nf_defrag_ipv6_hooks.c | 10 ++++++++++ net/netfilter/core.c | 6 ++++++ 4 files changed, 43 insertions(+), 1 deletion(-)
include/linux/netfilter.h +12 -0
@@ -481,6 +481,18 @@ struct nfnl_ct_hook { };extern const struct nfnl_ct_hook __rcu *nfnl_ct_hook; +struct nf_defrag_v4_hook { +int (*enable)(struct net *net); +void (*disable)(struct net *net); +}; +extern const struct nf_defrag_v4_hook __rcu *nf_defrag_v4_hook; + +struct nf_defrag_v6_hook { +int (*enable)(struct net *net); +void (*disable)(struct net *net); +}; +extern const struct nf_defrag_v6_hook __rcu *nf_defrag_v6_hook; +/* * nf_skb_duplicated - TEE target has sent a packet *
net/ipv4/netfilter/nf_defrag_ipv4.c +15 -1
@@ -7,6 +7,7 @@#include <linux/ip.h> #include <linux/netfilter.h> #include <linux/module.h> +#include <linux/rcupdate.h> #include <linux/skbuff.h> #include <net/netns/generic.h> #include <net/route.h>
@@ -113,17 +114,30 @@ static void __net_exit defrag4_net_exit(struct net *net) } } +static const struct nf_defrag_v4_hook defrag_hook = { + .enable = nf_defrag_ipv4_enable, + .disable = nf_defrag_ipv4_disable, +}; +static struct pernet_operations defrag4_net_ops = { .exit = defrag4_net_exit, };static int __init nf_defrag_init (void ) { -return register_pernet_subsys (&defrag4_net_ops); +int err; + + err =register_pernet_subsys (&defrag4_net_ops); +if (err) +return err; + +rcu_assign_pointer (nf_defrag_v4_hook, &defrag_hook); +return err; } static void __exit nf_defrag_fini(void) { +rcu_assign_pointer (nf_defrag_v4_hook,NULL );unregister_pernet_subsys (&defrag4_net_ops); }
net/ipv6/netfilter/nf_defrag_ipv6_hooks.c +10 -0
@@ -10,6 +10,7 @@#include <linux/module.h> #include <linux/skbuff.h> #include <linux/icmp.h> +#include <linux/rcupdate.h> #include <linux/sysctl.h> #include <net/ipv6_frag.h>
@@ -96,6 +97,11 @@ static void __net_exit defrag6_net_exit(struct net *net) } } +static const struct nf_defrag_v6_hook defrag_hook = { + .enable = nf_defrag_ipv6_enable, + .disable = nf_defrag_ipv6_disable, +}; +static struct pernet_operations defrag6_net_ops = { .exit = defrag6_net_exit, };
@@ -114,6 +120,9 @@ static int __init nf_defrag_init(void)pr_err ("nf_defrag_ipv6: can't register pernet ops\n" ); goto cleanup_frag6; } + +rcu_assign_pointer (nf_defrag_v6_hook, &defrag_hook); +return ret; cleanup_frag6:
@@ -124,6 +133,7 @@ static int __init nf_defrag_init(void) static void __exit nf_defrag_fini(void) { +rcu_assign_pointer (nf_defrag_v6_hook,NULL );unregister_pernet_subsys (&defrag6_net_ops);nf_ct_frag6_cleanup (); }
net/netfilter/core.c +6 -0
@@ -680,6 +680,12 @@ EXPORT_SYMBOL_GPL(nfnl_ct_hook);const struct nf_ct_hook __rcu *nf_ct_hook__read_mostly ;EXPORT_SYMBOL_GPL (nf_ct_hook); +const struct nf_defrag_v4_hook __rcu *nf_defrag_v4_hook__read_mostly ; +EXPORT_SYMBOL_GPL (nf_defrag_v4_hook); + +const struct nf_defrag_v6_hook __rcu *nf_defrag_v6_hook__read_mostly ; +EXPORT_SYMBOL_GPL (nf_defrag_v6_hook); +#if IS_ENABLED (CONFIG_NF_CONNTRACK )u8 nf_ctnetlink_has_listener;EXPORT_SYMBOL_GPL (nf_ctnetlink_has_listener);
Signature · 1 line
2.41.0
PATCH v3 2/6 netfilter: bpf: Support BPF_F_NETFILTER_IP_DEFRAG in netfilter link
In reply to Daniel XuThis commit adds support for enabling IP defrag using pre-existing netfilter defrag support. Basically all the flag does is bump a refcnt while the link the active. Checks are also added to ensure the prog requesting defrag support is run _after_ netfilter defrag hooks.
--- include/uapi/linux/bpf.h | 5 ++ net/netfilter/nf_bpf_link.c | 129 ++++++++++++++++++++++++++++++--- tools/include/uapi/linux/bpf.h | 5 ++ 3 files changed, 128 insertions(+), 11 deletions(-)
include/uapi/linux/bpf.h +5 -0
@@ -1170,6 +1170,11 @@ enum bpf_link_type {*/ #define BPF_F_KPROBE_MULTI_RETURN (1U << 0) +/* link_create.netfilter.flags used in LINK_CREATE command for +* BPF_PROG_TYPE_NETFILTER to enable IP packet defragmentation. +*/ +#define BPF_F_NETFILTER_IP_DEFRAG (1U << 0) +/* When BPF ldimm64's insn[0].src_reg != 0 then this can have * the following extensions: *
net/netfilter/nf_bpf_link.c +118 -11
169 lines not on this page · raw message
tools/include/uapi/linux/bpf.h +5 -0
@@ -1170,6 +1170,11 @@ enum bpf_link_type {*/ #define BPF_F_KPROBE_MULTI_RETURN (1U << 0) +/* link_create.netfilter.flags used in LINK_CREATE command for +* BPF_PROG_TYPE_NETFILTER to enable IP packet defragmentation. +*/ +#define BPF_F_NETFILTER_IP_DEFRAG (1U << 0) +/* When BPF ldimm64's insn[0].src_reg != 0 then this can have * the following extensions: *
Signature · 1 line
2.41.0
PATCH v3 3/6 netfilter: bpf: Prevent defrag module unload while link active
In reply to Daniel XuWhile in practice we could handle the module being unloaded while a netfilter link (that requested defrag) was active, it's a better user experience to prevent the defrag module from going away. It would violate user expectations if fragmented packets started showing up if some other part of the system tried to unload defrag module.
--- include/linux/netfilter.h | 3 +++ net/ipv4/netfilter/nf_defrag_ipv4.c | 1 + net/ipv6/netfilter/nf_defrag_ipv6_hooks.c | 1 + net/netfilter/nf_bpf_link.c | 25 +++++++++++++++++++++-- 4 files changed, 28 insertions(+), 2 deletions(-)
include/linux/netfilter.h +3 -0
@@ -11,6 +11,7 @@#include <linux/wait.h> #include <linux/list.h> #include <linux/static_key.h> +#include <linux/module.h> #include <linux/netfilter_defs.h> #include <linux/netdevice.h> #include <linux/sockptr.h>
@@ -482,12 +483,14 @@ struct nfnl_ct_hook {extern const struct nfnl_ct_hook __rcu *nfnl_ct_hook;struct nf_defrag_v4_hook { +struct module *owner;int (*enable)(struct net *net);void (*disable)(struct net *net); };extern const struct nf_defrag_v4_hook __rcu *nf_defrag_v4_hook;struct nf_defrag_v6_hook { +struct module *owner;int (*enable)(struct net *net);void (*disable)(struct net *net); };
net/ipv4/netfilter/nf_defrag_ipv4.c +1 -0
@@ -115,6 +115,7 @@ static void __net_exit defrag4_net_exit(struct net *net) }static const struct nf_defrag_v4_hook defrag_hook = { + .owner =THIS_MODULE , .enable = nf_defrag_ipv4_enable, .disable = nf_defrag_ipv4_disable, };
net/ipv6/netfilter/nf_defrag_ipv6_hooks.c +1 -0
@@ -98,6 +98,7 @@ static void __net_exit defrag6_net_exit(struct net *net) }static const struct nf_defrag_v6_hook defrag_hook = { + .owner =THIS_MODULE , .enable = nf_defrag_ipv6_enable, .disable = nf_defrag_ipv6_disable, };
net/netfilter/nf_bpf_link.c +23 -2
@@ -2,6 +2,7 @@#include <linux/bpf.h> #include <linux/filter.h> #include <linux/kmod.h> +#include <linux/module.h> #include <linux/netfilter.h> #include <net/netfilter/nf_bpf_link.h>
@@ -53,7 +54,15 @@ static int bpf_nf_enable_defrag(struct bpf_nf_link *link) } } +/* Prevent defrag module from going away while in use */ +if (!try_module_get (v4_hook->owner)) { + err = -ENOENT ; + goto out_v4; + } + err = v4_hook->enable (link->net); +if (err) +module_put (v4_hook->owner); out_v4:rcu_read_unlock ();return err;
@@ -77,7 +86,15 @@ static int bpf_nf_enable_defrag(struct bpf_nf_link *link) } } +/* Prevent defrag module from going away while in use */ +if (!try_module_get (v6_hook->owner)) { + err = -ENOENT ; + goto out_v6; + } + err = v6_hook->enable (link->net); +if (err) +module_put (v6_hook->owner); out_v6:rcu_read_unlock ();return err;
@@ -97,8 +114,10 @@ static void bpf_nf_disable_defrag(struct bpf_nf_link *link)case NFPROTO_IPV4 :rcu_read_lock (); v4_hook =rcu_dereference (nf_defrag_v4_hook); -if (v4_hook) +if (v4_hook) { v4_hook->disable (link->net); +module_put (v4_hook->owner); + }rcu_read_unlock ();break ;
@@ -107,8 +126,10 @@ static void bpf_nf_disable_defrag(struct bpf_nf_link *link)case NFPROTO_IPV6 :rcu_read_lock (); v6_hook =rcu_dereference (nf_defrag_v6_hook); -if (v6_hook) +if (v6_hook) { v6_hook->disable (link->net); +module_put (v6_hook->owner); + }rcu_read_unlock ();break ;
Signature · 1 line
2.41.0
PATCH v3 4/6 bpf: selftests: Support not connecting client socket
In reply to Daniel XuFor connectionless protocols or raw sockets we do not want to actually connect() to the server.
--- tools/testing/selftests/bpf/network_helpers.c | 5 +++-- tools/testing/selftests/bpf/network_helpers.h | 1 + 2 files changed, 4 insertions(+), 2 deletions(-)
tools/testing/selftests/bpf/network_helpers.c +3 -2
@@ -301,8 +301,9 @@ int connect_to_fd_opts(int server_fd, const struct network_helper_opts *opts) strlen(opts->cc) + 1)) goto error_close; -if (connect_fd_to_addr (fd, &addr, addrlen, opts->must_fail)) - goto error_close; +if (!opts->noconnect) +if (connect_fd_to_addr (fd, &addr, addrlen, opts->must_fail)) + goto error_close;return fd;
tools/testing/selftests/bpf/network_helpers.h +1 -0
@@ -21,6 +21,7 @@ struct network_helper_opts {const char *cc;int timeout_ms;bool must_fail; +bool noconnect; };/* ipv4 test vector */
Signature · 1 line
2.41.0
PATCH v3 5/6 bpf: selftests: Support custom type and proto for client sockets
In reply to Daniel XuExtend connect_to_fd_opts() to take optional type and protocol parameters for the client socket. These parameters are useful when opening a raw socket to send IP fragments.
--- tools/testing/selftests/bpf/network_helpers.c | 21 +++++++++++++------ tools/testing/selftests/bpf/network_helpers.h | 2 ++ 2 files changed, 17 insertions(+), 6 deletions(-)
tools/testing/selftests/bpf/network_helpers.c +15 -6
@@ -270,14 +270,23 @@ int connect_to_fd_opts(int server_fd, const struct network_helper_opts *opts) opts = &default_opts; optlen =sizeof (type); -if (getsockopt (server_fd,SOL_SOCKET ,SO_TYPE , &type, &optlen)) { -log_err ("getsockopt(SOL_TYPE)" ); -return -1 ; + +if (opts->type) { + type = opts->type; + }else { +if (getsockopt (server_fd,SOL_SOCKET ,SO_TYPE , &type, &optlen)) { +log_err ("getsockopt(SOL_TYPE)" ); +return -1 ; + } } -if (getsockopt (server_fd,SOL_SOCKET ,SO_PROTOCOL , &protocol, &optlen)) { -log_err ("getsockopt(SOL_PROTOCOL)" ); -return -1 ; +if (opts->proto) { + protocol = opts->proto; + }else { +if (getsockopt (server_fd,SOL_SOCKET ,SO_PROTOCOL , &protocol, &optlen)) { +log_err ("getsockopt(SOL_PROTOCOL)" ); +return -1 ; + } } addrlen =sizeof (addr);
tools/testing/selftests/bpf/network_helpers.h +2 -0
@@ -22,6 +22,8 @@ struct network_helper_opts {int timeout_ms;bool must_fail;bool noconnect; +int type; +int proto; };/* ipv4 test vector */
Signature · 1 line
2.41.0
PATCH v3 6/6 bpf: selftests: Add defrag selftests
In reply to Daniel XuThese selftests tests 2 major scenarios: the BPF based defragmentation can successfully be done and that packet pointers are invalidated after calls to the kfunc. The logic is similar for both ipv4 and ipv6.
In the first scenario, we create a UDP client and UDP echo server. The the server side is fairly straightforward: we attach the prog and simply echo back the message.
The on the client side, we send fragmented packets to and expect the reassembled message back from the server.
--- tools/testing/selftests/bpf/Makefile | 4 +- .../selftests/bpf/generate_udp_fragments.py | 90 ++++++ .../selftests/bpf/ip_check_defrag_frags.h | 57 ++++ .../bpf/prog_tests/ip_check_defrag.c | 282 ++++++++++++++++++ .../selftests/bpf/progs/ip_check_defrag.c | 104 +++++++ 5 files changed, 535 insertions(+), 2 deletions(-) create mode 100755 tools/testing/selftests/bpf/generate_udp_fragments.py create mode 100644 tools/testing/selftests/bpf/ip_check_defrag_frags.h create mode 100644 tools/testing/selftests/bpf/prog_tests/ip_check_defrag.c create mode 100644 tools/testing/selftests/bpf/progs/ip_check_defrag.c
tools/testing/selftests/bpf/Makefile +2 -2
@@ -565,8 +565,8 @@ TRUNNER_EXTRA_SOURCES := test_progs.c cgroup_helpers.c trace_helpers.c \ network_helpers.c testing_helpers.c \ btf_helpers.c flow_dissector_load.h \ cap_helpers.c test_loader.c xsk.c disasm.c \ - json_writer.c unpriv_helpers.c - + json_writer.c unpriv_helpers.c \ + ip_check_defrag_frags.h TRUNNER_EXTRA_FILES := $(OUTPUT)/urandom_read $(OUTPUT)/bpf_testmod.ko \ $(OUTPUT)/liburandom_read.so \ $(OUTPUT)/xdp_synproxy \
tools/testing/selftests/bpf/generate_udp_fragments.py new file +90 -0
90 lines not on this page · raw message
tools/testing/selftests/bpf/ip_check_defrag_frags.h new file +57 -0
57 lines not on this page · raw message
tools/testing/selftests/bpf/prog_tests/ip_check_defrag.c new file +282 -0
282 lines not on this page · raw message
tools/testing/selftests/bpf/progs/ip_check_defrag.c new file +104 -0
104 lines not on this page · raw message
Signature · 1 line
2.41.0
Re: [PATCH bpf-next v3 1/6] netfilter: defrag: Add glue hooks for enabling/disabling defrag
In reply to Daniel XuQuoted 6 lines · Daniel Xu
We want to be able to enable/disable IP packet defrag from core bpf/netfilter code. In other words, execute code from core that could possibly be built as a module.
To help avoid symbol resolution errors, use glue hooks that the modules will register callbacks with during module init.
Re: [PATCH bpf-next v3 2/6] netfilter: bpf: Support BPF_F_NETFILTER_IP_DEFRAG in netfilter link
In reply to Daniel XuQuoted 6 lines · Daniel Xu
This commit adds support for enabling IP defrag using pre-existing netfilter defrag support. Basically all the flag does is bump a refcnt while the link the active. Checks are also added to ensure the prog requesting defrag support is run _after_ netfilter defrag hooks.
Signed-off-by Daniel Xu <[email protected]>
Re: [PATCH bpf-next v3 3/6] netfilter: bpf: Prevent defrag module unload while link active
In reply to Daniel XuQuoted 5 lines · Daniel Xu
While in practice we could handle the module being unloaded while a netfilter link (that requested defrag) was active, it's a better user experience to prevent the defrag module from going away. It would violate user expectations if fragmented packets started showing up if some other part of the system tried to unload defrag module.
Re: [PATCH bpf-next v3 6/6] bpf: selftests: Add defrag selftests
In reply to Daniel XuQuoted 5 lines · Daniel Xu
These selftests tests 2 major scenarios: the BPF based defragmentation can successfully be done and that packet pointers are invalidated after calls to the kfunc. The logic is similar for both ipv4 and ipv6.
Quoted 8 lines
In the first scenario, we create a UDP client and UDP echo server. The the server side is fairly straightforward: we attach the prog and simply echo back the message.
The on the client side, we send fragmented packets to and expect the reassembled message back from the server.
Signed-off-by Daniel Xu <[email protected]>
The patches look good, but new tests are failing on arm64.
test_bpf_ip_check_defrag_ok:FAIL:server recvfrom unexpected server
recvfrom: actual -1 < expected 0
see BPF CI.
Feels like a timing issue, but pls take a look.