netfilter: nf_tables: add ebpf expression
Subsystems (4)
Florian Westphal
maintainerPATCH 1/1 netfilter: nf_tables: add ebpf expression
This expression is a native replacement for xtables 'bpf' match "pinned" mode. Userspace needs to pass a file descriptor referencing the program (of socket filter type).
Userspace should also pass the original pathname for that fd so userspace can print the original filename again.
Tag and program id are dumped to userspace on 'list' to allow to see which program is in use in case the filename isn't available/present.
cbpf bytecode isn't supported.
No new Kconfig option is added: Its included if BPF_SYSCALL is enabled.
Proposed nft userspace syntax is:
add rule ... ebpf pinned "/sys/fs/bpf/myprog"
--- include/net/netfilter/nf_tables_core.h | 3 + include/uapi/linux/netfilter/nf_tables.h | 18 ++++ net/netfilter/Makefile | 4 + net/netfilter/nf_tables_core.c | 7 ++ net/netfilter/nft_ebpf.c | 128 +++++++++++++++++++++++ 5 files changed, 160 insertions(+) create mode 100644 net/netfilter/nft_ebpf.c
include/net/netfilter/nf_tables_core.h +3 -0
@@ -18,6 +18,7 @@ extern struct nft_expr_type nft_meta_type;extern struct nft_expr_type nft_rt_type;extern struct nft_expr_type nft_exthdr_type;extern struct nft_expr_type nft_last_type; +extern struct nft_expr_type nft_ebpf_type;#ifdef CONFIG_NETWORK_SECMARK extern struct nft_object_type nft_secmark_obj_type;
@@ -148,4 +149,6 @@ void nft_rt_get_eval(const struct nft_expr *expr, struct nft_regs *regs, const struct nft_pktinfo *pkt);void nft_counter_eval (const struct nft_expr *expr,struct nft_regs *regs,const struct nft_pktinfo *pkt); +void nft_ebpf_eval (const struct nft_expr *expr,struct nft_regs *regs, +const struct nft_pktinfo *pkt);#endif /* _NET_NF_TABLES_CORE_H */
include/uapi/linux/netfilter/nf_tables.h +18 -0
@@ -805,6 +805,24 @@ enum nft_payload_attributes { };#define NFTA_PAYLOAD_MAX (__NFTA_PAYLOAD_MAX - 1) +/** +* enum nft_ebpf_attributes - nf_tables ebpf expression netlink attributes +* +* @NFTA_EBPF_FD: file descriptor holding ebpf program (NLA_U32) +* @NFTA_EBPF_FILENAME: file name, only for storage/printing (NLA_STRING) +* @NFTA_EBPF_ID: bpf program id (NLA_U32) +* @NFTA_EBPF_TAG: bpf tag (NLA_BINARY) +*/ +enum nft_ebpf_attributes { +NFTA_EBPF_UNSPEC , +NFTA_EBPF_FD , +NFTA_EBPF_FILENAME , +NFTA_EBPF_ID , +NFTA_EBPF_TAG , + __NFTA_EBPF_MAX, +}; +#define NFTA_EBPF_MAX (__NFTA_EBPF_MAX - 1) +enum nft_exthdr_flags {NFT_EXTHDR_F_PRESENT = (1 <<0 ), };
net/netfilter/Makefile +4 -0
@@ -90,6 +90,10 @@ nf_tables-objs += nft_set_pipapo_avx2.o endif endif +ifdef CONFIG_BPF_SYSCALL +nf_tables-objs += nft_ebpf.o +endif + obj-$(CONFIG_NF_TABLES) += nf_tables.o obj-$(CONFIG_NFT_COMPAT) += nft_compat.o obj-$(CONFIG_NFT_CONNLIMIT) += nft_connlimit.o
net/netfilter/nf_tables_core.c +7 -0
@@ -8,6 +8,7 @@#include <linux/kernel.h> #include <linux/module.h> #include <linux/init.h> +#include <linux/filter.h> #include <linux/list.h> #include <linux/rculist.h> #include <linux/skbuff.h>
@@ -209,6 +210,9 @@ static void expr_call_ops_eval(const struct nft_expr *expr,X (e, nft_dynset_eval);X (e, nft_rt_get_eval);X (e, nft_bitwise_eval); +#ifdef CONFIG_BPF_SYSCALL +X (e, nft_ebpf_eval); +#endif #undef X#endif /* CONFIG_RETPOLINE */ expr->ops->eval (expr, regs, pkt);
@@ -340,6 +344,9 @@ static struct nft_expr_type *nft_basic_types[] = { &nft_exthdr_type, &nft_last_type, &nft_counter_type, +#ifdef CONFIG_BPF_SYSCALL + &nft_ebpf_type, +#endif }; static struct nft_object_type *nft_basic_objects[] = {
net/netfilter/nft_ebpf.c new file +128 -0
128 lines not on this page · raw message
Signature · 1 line
2.35.1
Quoted 8 lines · Florian Westphal
This expression is a native replacement for xtables 'bpf' match "pinned" mode. Userspace needs to pass a file descriptor referencing the program (of socket filter type).
Userspace should also pass the original pathname for that fd so userspace can print the original filename again.Tag and program id are dumped to userspace on 'list' to allow to see which program is in use in case the filename isn't available/present.
It seems a bit odd to include the file path in the kernel as well. For one thing, the same object can be pinned multiple times in different paths (even in different mount namespaces), and there's also nothing preventing a different program to have been substituted by the pinned one by the time the value is echoed back.
Also, there's nothing checking that the path attribute actually contains a path, so it's really just an arbitrary label that the kernel promises to echo back. But doesn't NFT already have a per-rule comment feature, so why add another specifically for BPF? Instead we could just teach the userspace utility to extract metadata from the BPF program (based on the ID) like bpftool does. This would include the program name, BTW, so it does have a semantic identifier.
Quoted 7 lines
cbpf bytecode isn't supported.
No new
Kconfigoption is added: Its included ifBPF_SYSCALLis enabled.Proposed nft userspace syntax is:
add rule ... ebpf pinned "
/sys/fs/bpf/myprog"
Any plan to also teach the nft binary to load a BPF program from an ELF file (instead of relying on pinning)?
-Toke
Florian Westphal
maintainerQuoted 4 lines · Toke Høiland-Jørgensen
Quoted 2 lines
Tag and program id are dumped to userspace on 'list' to allow to see which program is in use in case the filename isn't available/present.
It seems a bit odd to include the file path in the kernel as well.
Its needed to be able to re-load the ruleset.
Quoted 3 lines
For
one thing, the same object can be pinned multiple times in different paths (even in different mount namespaces),
Sure.
Quoted 3 lines
and there's also nothing
preventing a different program to have been substituted by the pinned one by the time the value is echoed back.
Yes, but what would you expect it should do?
Quoted 3 lines
Also, there's nothing checking that the path attribute actually contains a path, so it's really just an arbitrary label that the kernel promises to echo back
Yes exactly.
Quoted 2 lines
But doesn't NFT already have a per-rule comment feature, so why add another specifically for BPF?
You can attach up to 256 bytes to a rule, yes.
Might not be enough for a longer path, and there could be multiple expressions in the same rule.
This way was the most simple solution.
Quoted 4 lines
Instead we could just teach the
userspace utility to extract metadata from the BPF program (based on the ID) like bpftool does. This would include the program name, BTW, so it does have a semantic identifier.
Sure, I could change the grammar so it expects a tag or ID, e.g. 'ebpf id 42'
If thats preferred, I can change this, it avoids the need for storing the name.
Quoted 5 lines
Quoted 2 lines
cbpf bytecode isn't supported.
add rule ... ebpf pinned "/sys/fs/bpf/myprog"Any plan to also teach the nft binary to load a BPF program from an ELF file (instead of relying on pinning)?
I used pinning because that is what '-m bpf' uses.
Quoted 7 lines · Florian Westphal
Quoted 4 lines · Toke Høiland-Jørgensen
Quoted 2 lines
Tag and program id are dumped to userspace on 'list' to allow to see which program is in use in case the filename isn't available/present.
It seems a bit odd to include the file path in the kernel as well.
Its needed to be able to re-load the ruleset.
How does that work, exactly? Is this so that the userspace binary can query the current ruleset, and feed it back to the kernel expecting it to stay the same? Because in that case, if the pinned object goes away in the meantime (or changes to a different program), this could lead to some really hard to debug errors, where a reload subtly changes the behaviour because the BPF program is not in fact the same.
Using IDs would avoid this ambiguity at least, so I think that's a better solution. We'd have to make sure the BPF program is not released completely until after the reload has finished, so that it doesn't suddenly disappear.
Quoted 8 lines
Quoted 2 lines
But doesn't NFT already have a per-rule comment feature, so why add another specifically for BPF?
You can attach up to 256 bytes to a rule, yes.
Might not be enough for a longer path, and there could be multiple expressions in the same rule.This way was the most simple solution.
My point here was more that if it's just a label for human consumption, the comment field should be fine, didn't realise it was needed for the tool operation (and see above re: that).
Quoted 10 lines
Quoted 4 lines
Instead we could just teach the
userspace utility to extract metadata from the BPF program (based on the ID) like bpftool does. This would include the program name, BTW, so it does have a semantic identifier.Sure, I could change the grammar so it expects a tag or ID, e.g. 'ebpf id 42'
If thats preferred, I can change this, it avoids the need for storing the name.
I think for echoing back, just relying on the ID is better as that is at least guaranteed to stay constant for the lifetime of the BPF program in the kernel. We could still support the 'pinned <path>' syntax on the command line so that the initial load could be done from a pinned file, just as a user interface improvement...
Quoted 7 lines
Quoted 5 lines
Quoted 2 lines
cbpf bytecode isn't supported.
add rule ... ebpf pinned "/sys/fs/bpf/myprog"Any plan to also teach the nft binary to load a BPF program from an ELF file (instead of relying on pinning)?
I used pinning because that is what '-m bpf' uses.
I'm not against supporting pinning, per se (except for the issues noted above), but we could do multiple things, including supporting loading the program from an object file. This is similar to how TC operates, for instance...
-Toke
Florian Westphal
maintainerQuoted 7 lines · Toke Høiland-Jørgensen
Quoted 3 lines
Quoted 1 line
It seems a bit odd to include the file path in the kernel as well.
Its needed to be able to re-load the ruleset.
How does that work, exactly? Is this so that the userspace binary can query the current ruleset, and feed it back to the kernel expecting it to stay the same?
Yes.
Quoted 4 lines
Because in that case, if the pinned object goes away
in the meantime (or changes to a different program), this could lead to some really hard to debug errors, where a reload subtly changes the behaviour because the BPF program is not in fact the same.
Correct, but thats kind of expected when the user changes programs logic.
Same with a 'nft list ruleset > /etc/nft.txt', reboot,
'nft -f /etc/nft.txt' fails because user forgot to load/pin the program first.
Quoted 4 lines
Using IDs would avoid this ambiguity at least, so I think that's a better solution. We'd have to make sure the BPF program is not released completely until after the reload has finished, so that it doesn't suddenly disappear.
This should be covered, the destructor runs after the ruleset has been detached from the data plan (and after a synchronize_rcu).
Quoted 5 lines
Quoted 1 line
This way was the most simple solution.
My point here was more that if it's just a label for human consumption, the comment field should be fine, didn't realise it was needed for the tool operation (and see above re: that).
Yes, this is unfortunate. I would like to avoid introducing an asymmetry between input and output (as in "... add rule ebpf pinned bla', but 'nft list ruleset' showing 'ebpf id 42') or similar, UNLESS we can somehow use that alternate output to reconstruct that was originally intended. And so far I can only see that happening with storing some label in the kernel for userspace to consume (elf filename, pinned name, program name ... ).
To give an example:
With 'ebpf id 42', we might be able to let this get echoed back as if user would have said 'ebpf progname myfilter' (I am making this up!), just to have a more 'stable' identifier.
This would make it necessary to also support load-by-program-name, of course.
Quoted 9 lines
Quoted 5 lines
Sure, I could change the grammar so it expects a tag or ID, e.g. 'ebpf id 42'
If thats preferred, I can change this, it avoids the need for storing the name.
I think for echoing back, just relying on the ID is better as that is at least guaranteed to stay constant for the lifetime of the BPF program in the kernel.
Yes, I realize that, this is why the id and tag are included in the netlink dump, but on the userspace side this information is currently hidden and only shown with --debug output.
Quoted 7 lines
Quoted 4 lines
Quoted 2 lines
Any plan to also teach the nft binary to load a BPF program from an ELF file (instead of relying on pinning)?
I used pinning because that is what '-m bpf' uses.
I'm not against supporting pinning, per se (except for the issues noted above),
Okay, thanks for clarifying. -m bpf is a bit older so I was not sure if pinning has been deprecated or something like that.
Quoted 3 lines
But we could do multiple things, including supporting loading the program from an object file. This is similar to how TC operates, for instance...
Right, there is no need to restrict this to one method.
Quoted 16 lines · Florian Westphal
Quoted 7 lines · Toke Høiland-Jørgensen
Quoted 3 lines
Quoted 1 line
It seems a bit odd to include the file path in the kernel as well.
Its needed to be able to re-load the ruleset.
How does that work, exactly? Is this so that the userspace binary can query the current ruleset, and feed it back to the kernel expecting it to stay the same?
Yes.
Quoted 4 lines
Because in that case, if the pinned object goes away
in the meantime (or changes to a different program), this could lead to some really hard to debug errors, where a reload subtly changes the behaviour because the BPF program is not in fact the same.
Quoted 6 lines
Correct, but thats kind of expected when the user changes programs logic.
Same with a 'nft list ruleset >
/etc/nft.txt', reboot,
'nft -f/etc/nft.txt' fails because user forgot to load/pin the program first.
Right, so under what conditions is the identifier expected to survive, exactly? It's okay if it fails after a reboot, but it should keep working while the system is up?
Quoted 14 lines
Quoted 5 lines
Quoted 1 line
This way was the most simple solution.
My point here was more that if it's just a label for human consumption, the comment field should be fine, didn't realise it was needed for the tool operation (and see above re: that).
Yes, this is unfortunate. I would like to avoid introducing an asymmetry between input and output (as in "... add rule ebpf pinned bla', but 'nft list ruleset' showing 'ebpf id 42') or similar, UNLESS we can somehow use that alternate output to reconstruct that was originally intended. And so far I can only see that happening with storing some label in the kernel for userspace to consume (elf filename, pinned name, program name ... ).
Quoted 8 lines
To give an example:
With 'ebpf id 42', we might be able to let this get echoed back as if user would have said 'ebpf progname myfilter' (I am making this up!), just to have a more 'stable' identifier.
This would make it necessary to also support load-by-program-name, of course.
Seems like this kind of mapping can be done in userspace without involving the kernel?
For example, the 'progname' thing could be implemented by defining an nft-specific pinning location so that 'ebpf progname myfilter' is equivalent to 'ebpf pinned /sys/bpf/nft/myfilter' and when nft receives an ID from the kernel it goes looking in /sys/bpf/nft to see if it can find the program with that ID and echoes it with the appropriate progname if it does exist?
This could also be extended, so that if a user does '... add rule ebpf file /usr/lib/bpf/myrule.o' the nft binary stashes the id -> .o file mapping somewhere (in /run for instance) so that it can echo back where it got it from later?
In either case I'm not really sure that there's much to be gained from asking the kernel to store an additional label with the program rule?
-Toke
Pablo Neira Ayuso
maintainerQuoted 62 lines · Toke Høiland-Jørgensen
Quoted 16 lines · Florian Westphal
Quoted 7 lines · Toke Høiland-Jørgensen
Quoted 3 lines
Quoted 1 line
It seems a bit odd to include the file path in the kernel as well.
Its needed to be able to re-load the ruleset.
How does that work, exactly? Is this so that the userspace binary can query the current ruleset, and feed it back to the kernel expecting it to stay the same?
Yes.
Quoted 4 lines
Because in that case, if the pinned object goes away
in the meantime (or changes to a different program), this could lead to some really hard to debug errors, where a reload subtly changes the behaviour because the BPF program is not in fact the same.Quoted 6 lines
Correct, but thats kind of expected when the user changes programs logic.
Same with a 'nft list ruleset >
/etc/nft.txt', reboot,
'nft -f/etc/nft.txt' fails because user forgot to load/pin the program first.Right, so under what conditions is the identifier expected to survive, exactly? It's okay if it fails after a reboot, but it should keep working while the system is up?
Quoted 14 lines
Quoted 5 lines
Quoted 1 line
This way was the most simple solution.
My point here was more that if it's just a label for human consumption, the comment field should be fine, didn't realise it was needed for the tool operation (and see above re: that).
Yes, this is unfortunate. I would like to avoid introducing an asymmetry between input and output (as in "... add rule ebpf pinned bla', but 'nft list ruleset' showing 'ebpf id 42') or similar, UNLESS we can somehow use that alternate output to reconstruct that was originally intended. And so far I can only see that happening with storing some label in the kernel for userspace to consume (elf filename, pinned name, program name ... ).
Quoted 8 lines
To give an example:
With 'ebpf id 42', we might be able to let this get echoed back as if user would have said 'ebpf progname myfilter' (I am making this up!), just to have a more 'stable' identifier.
This would make it necessary to also support load-by-program-name, of course.
Seems like this kind of mapping can be done in userspace without involving the kernel?
For example, the 'progname' thing could be implemented by defining an nft-specific pinning location so that 'ebpf progname myfilter' is equivalent to 'ebpf pinned
/sys/bpf/nft/myfilter' and when nft receives an ID from the kernel it goes looking in/sys/bpf/nftto see if it can find the program with that ID and echoes it with the appropriate progname if it does exist?
Quoted 7 lines
This could also be extended, so that if a user does '... add rule ebpf file
/usr/lib/bpf/myrule.o' the nft binary stashes the id -> .o file mapping somewhere (in /run for instance) so that it can echo back where it got it from later?In either case I'm not really sure that there's much to be gained from asking the kernel to store an additional label with the program rule?
@Florian, could you probably use the object infrastructure to refer to the program?
This might also allow you to refer to this new object type from nf_tables maps.
It would be good to avoid linear rule-based matching to select what program to run.
Maybe this also fits fine for your requirements?
Florian Westphal
maintainerQuoted 4 lines · Pablo Neira Ayuso
Quoted 1 line
asking the kernel to store an additional label with the program rule?
@Florian, could you probably use the object infrastructure to refer to the program?
Yes, I would like to extend objref infra later once this is accepted.
Quoted 2 lines
This might also allow you to refer to this new object type from
nf_tablesmaps.
Yes, but first nft needs to be able to construct some meaningful output again. If we don't attach a specific label (such as filename), we need to be able to reconstruct info based on what we can query via id/tag and bpf syscall.
objref infra doesn't help here unless we'll force something like 'nft-defined-objref-name-must-match-elf-binary-name', and I find that terrible.
Quoted 2 lines
It would be good to avoid linear rule-based matching to select what program to run.
Hmmm, I did not consider it a huge deal, its an ebpf program so users can dispatch to another program.
Objref is nice if the program to run should be selected from a criterion that isn't readily available to a sk_filter program though.
Pablo Neira Ayuso
maintainerQuoted 11 lines · Florian Westphal
Quoted 4 lines · Pablo Neira Ayuso
Quoted 1 line
asking the kernel to store an additional label with the program rule?
@Florian, could you probably use the object infrastructure to refer to the program?
Yes, I would like to extend objref infra later once this is accepted.
Quoted 2 lines
This might also allow you to refer to this new object type from
nf_tablesmaps.
Quoted 8 lines
Yes, but first nft needs to be able to construct some meaningful output again. If we don't attach a specific label (such as filename), we need to be able to reconstruct info based on what we can query via id/tag and bpf syscall.
objref infra doesn't help here unless we'll force something like 'nft-defined-objref-name-must-match-elf-binary-name', and I find that terrible.
OK, you don't have to select such an ugly long name ;)
But I get your point: users need to declare explicitly the object.
Quoted 8 lines
Quoted 2 lines
It would be good to avoid linear rule-based matching to select what program to run.
Hmmm, I did not consider it a huge deal, its an ebpf program so users can dispatch to another program.
Objref is nice if the program to run should be selected from a criterion that isn't readily available to a
sk_filterprogram though.
You can also perform updates on the object without the need for reloading your ruleset. And the declared object also allows for more attributes to be added on it moving forward.
I think this approach would also allow you to maintain symmetry between what you add and what it is shown in the listing?
Florian Westphal
maintainerQuoted 7 lines · Toke Høiland-Jørgensen
Quoted 3 lines
Same with a 'nft list ruleset >
/etc/nft.txt', reboot,
'nft -f/etc/nft.txt' fails because user forgot to load/pin the program first.Right, so under what conditions is the identifier expected to survive, exactly? It's okay if it fails after a reboot, but it should keep working while the system is up?
Right, thats the question. I think it boils down to 'least surprise', which to me would mean useable labels are:
1. pinned name
2. elf filename
3. filter name
3) has the advantage that afaiu I can extend nft to use the dumped id + program tag to query the name from the kernel, whereas 1+2 would need to store the label.
1 and 2 have the upside that its easy to handle a 'file not found' error.
Quoted 24 lines
Quoted 12 lines
Quoted 3 lines
My point here was more that if it's just a label for human consumption, the comment field should be fine, didn't realise it was needed for the tool operation (and see above re: that).
Yes, this is unfortunate. I would like to avoid introducing an asymmetry between input and output (as in "... add rule ebpf pinned bla', but 'nft list ruleset' showing 'ebpf id 42') or similar, UNLESS we can somehow use that alternate output to reconstruct that was originally intended. And so far I can only see that happening with storing some label in the kernel for userspace to consume (elf filename, pinned name, program name ... ).
Quoted 8 lines
To give an example:
With 'ebpf id 42', we might be able to let this get echoed back as if user would have said 'ebpf progname myfilter' (I am making this up!), just to have a more 'stable' identifier.
This would make it necessary to also support load-by-program-name, of course.
Seems like this kind of mapping can be done in userspace without involving the kernel?
Quoted 6 lines
For example, the 'progname' thing could be implemented by defining an nft-specific pinning location so that 'ebpf progname myfilter' is equivalent to 'ebpf pinned
/sys/bpf/nft/myfilter' and when nft receives an ID from the kernel it goes looking in/sys/bpf/nftto see if it can find the program with that ID and echoes it with the appropriate progname if it does exist?
Thats an interesting proposal.
I had not considered an nft specific namespace, just the raw pinned filename.
Quoted 4 lines
This could also be extended, so that if a user does '... add rule ebpf file
/usr/lib/bpf/myrule.o' the nft binary stashes the id -> .o file mapping somewhere (in /run for instance) so that it can echo back where it got it from later?
Oh, I would prefer to abuse the kernel as a "database" directly for that rather than keeping text files that have to be managed externally.
But, all things considered, what about this:
I'll respin, with the FILENAME attribute removed, so user says 'ebpf pinned bla', and on listing nft walks /sys/bpf/nft/ to see if it can find the name again.
If it can't find it, print the id instead.
This would mean nft would also have to understand
'ebpf id 12' on input, but I think thats fine. We can document that this is not the preferred method due to the difficulty of determining the correct id to use.
With this, no 'extra' userspace-sake info needs to be stored. We can revisit what do with 'ebpf file /bla/foo.o' once/if that gets added.
What do you think?
Will take a while because I'll need to extend the nft side first to cope with lack of 'FILENAME' attribute.
Quoted 17 lines · Florian Westphal
Quoted 7 lines · Toke Høiland-Jørgensen
Quoted 3 lines
Same with a 'nft list ruleset >
/etc/nft.txt', reboot,
'nft -f/etc/nft.txt' fails because user forgot to load/pin the program first.Right, so under what conditions is the identifier expected to survive, exactly? It's okay if it fails after a reboot, but it should keep working while the system is up?
Right, thats the question. I think it boils down to 'least surprise', which to me would mean useable labels are:
1. pinned name
2. elf filename
3. filter name
Quoted 6 lines
3) has the advantage that afaiu I can extend nft to use the dumped id + program tag to query the name from the kernel, whereas 1+2 would need to store the label.
1 and 2 have the upside that its easy to handle a 'file not found' error.
I'm strongly against calling into bpf from the inner guts of nft. Nack to all options discussed in this thread.
None of them make any sense.
Florian Westphal
maintainerQuoted 6 lines · Alexei Starovoitov
Quoted 2 lines
1 and 2 have the upside that its easy to handle a 'file not found' error.
I'm strongly against calling into bpf from the inner guts of nft. Nack to all options discussed in this thread.
None of them make any sense.
-v please. I can just rework userspace to allow going via xt_bpf but its brain damaged.
This helps gradually moving towards move epbf for those that still heavily rely on the classic forwarding path.
If you are open to BPF_PROG_TYPE_NETFILTER I can go that route as well, raw bpf program attachment via NF_HOOK and the bpf dispatcher, but it will take significantly longer to get there.
It involves reviving
https://lore.kernel.org/netfilter-devel/[email protected]/
as a first stage/merge goal.
Quoted 11 lines · Florian Westphal
Quoted 6 lines · Alexei Starovoitov
Quoted 2 lines
1 and 2 have the upside that its easy to handle a 'file not found' error.
I'm strongly against calling into bpf from the inner guts of nft. Nack to all options discussed in this thread.
None of them make any sense.-v please. I can just rework userspace to allow going via
xt_bpfbut its brain damaged.
Right. xt_bpf was a dead end from the start.
It's time to deprecate it and remove it.
Quoted 2 lines
This helps gradually moving towards move epbf for those that still heavily rely on the classic forwarding path.
No one is using it.
If it was, we would have seen at least one bug report over all these years. We've seen none.
tbh we had a fair share of wrong design decisions that look very reasonable early on and turned out to be useless with zero users.BPF_PROG_TYPE_SCHED_ACT and BPF_PROG_TYPE_LWT*
are in this category.
All this code does is bit rot.
As a minimum we shouldn't step on the same rakes. xt_ebpf would be the same dead code as xt_bpf.
Quoted 6 lines
If you are open to
BPF_PROG_TYPE_NETFILTERI can go that route as well, raw bpf program attachment viaNF_HOOKand the bpf dispatcher, but it will take significantly longer to get there.It involves reviving
https://lore.kernel.org/netfilter-devel/[email protected]/
I missed it earlier. What is the end goal ?
Optimize nft run-time with on the fly generation of bpf byte code ?
Daniel Borkmann
maintainerQuoted 22 lines · Alexei Starovoitov
Quoted 10 lines · Florian Westphal
Quoted 6 lines · Alexei Starovoitov
Quoted 2 lines
1 and 2 have the upside that its easy to handle a 'file not found' error.
I'm strongly against calling into bpf from the inner guts of nft. Nack to all options discussed in this thread.
None of them make any sense.-v please. I can just rework userspace to allow going via
xt_bpfbut its brain damaged.Right.
xt_bpfwas a dead end from the start.
It's time to deprecate it and remove it.Quoted 2 lines
This helps gradually moving towards move epbf for those that still heavily rely on the classic forwarding path.
No one is using it.
If it was, we would have seen at least one bug report over all these years. We've seen none.
Quoted 5 lines
tbh we had a fair share of wrong design decisions that look very reasonable early on and turned out to be useless with zero users.
BPF_PROG_TYPE_SCHED_ACTand BPF_PROG_TYPE_LWT*
are in this category. > All this code does is bit rot.
+1
Quoted 2 lines
As a minimum we shouldn't step on the same rakes.
xt_ebpfwould be the same dead code asxt_bpf.
+1, and on top, the user experience will just be horrible. :(
Quoted 9 lines
Quoted 6 lines
If you are open to
BPF_PROG_TYPE_NETFILTERI can go that route as well, raw bpf program attachment viaNF_HOOKand the bpf dispatcher, but it will take significantly longer to get there.It involves reviving
https://lore.kernel.org/netfilter-devel/[email protected]/I missed it earlier. What is the end goal ?
Optimize nft run-time with on the fly generation of bpf byte code ?
Or rather to provide a pendant to nft given existence of xt_bpf, and the latter will be removed at some point? (If so, can't we just deprecate the old xt_bpf?)
Thanks,
Daniel
Quoted 44 lines · Daniel Borkmann
Quoted 22 lines · Alexei Starovoitov
Quoted 10 lines · Florian Westphal
Quoted 6 lines · Alexei Starovoitov
Quoted 2 lines
1 and 2 have the upside that its easy to handle a 'file not found' error.
I'm strongly against calling into bpf from the inner guts of nft. Nack to all options discussed in this thread.
None of them make any sense.-v please. I can just rework userspace to allow going via
xt_bpfbut its brain damaged.Right.
xt_bpfwas a dead end from the start.
It's time to deprecate it and remove it.Quoted 2 lines
This helps gradually moving towards move epbf for those that still heavily rely on the classic forwarding path.
No one is using it.
If it was, we would have seen at least one bug report over all these years. We've seen none.Quoted 5 lines
tbh we had a fair share of wrong design decisions that look very reasonable early on and turned out to be useless with zero users.
BPF_PROG_TYPE_SCHED_ACTand BPF_PROG_TYPE_LWT*
are in this category. > All this code does is bit rot.+1
Quoted 2 lines
As a minimum we shouldn't step on the same rakes.
xt_ebpfwould be the same dead code asxt_bpf.+1, and on top, the user experience will just be horrible. :(
Quoted 7 lines
Quoted 6 lines
If you are open to
BPF_PROG_TYPE_NETFILTERI can go that route as well, raw bpf program attachment viaNF_HOOKand the bpf dispatcher, but it will take significantly longer to get there.It involves reviving
https://lore.kernel.org/netfilter-devel/[email protected]/
Quoted 6 lines
Quoted 2 lines
I missed it earlier. What is the end goal ?
Optimize nft run-time with on the fly generation of bpf byte code ?Or rather to provide a pendant to nft given existence of
xt_bpf, and the latter will be removed at some point? (If so, can't we just deprecate the oldxt_bpf?)
FWIW we've been using both lwt bpf and xt_bpf on our production workloads for a few years now.
xt_bpf allows us to apply custom sophisticated policy logic at connection establishment - which is not really possible (or efficient) using iptables/nft constructs - without needing to reinvent all the facilities that nf provides like connection tracking, ALGs, and simple filtering.
As for lwt bpf, We use it for load balancing towards collect md tunnels. While this can be done at tc egress for unfragmented packets, the lwt out hook - when used in tandem with nf fragment reassembly - provides a hooking point where a bpf program can see reassembled packets and load balance based on their internals.
Eyal.
Quoted 61 lines · Eyal Birger
Quoted 44 lines · Daniel Borkmann
Quoted 22 lines · Alexei Starovoitov
Quoted 10 lines · Florian Westphal
Quoted 6 lines · Alexei Starovoitov
Quoted 2 lines
1 and 2 have the upside that its easy to handle a 'file not found' error.
I'm strongly against calling into bpf from the inner guts of nft. Nack to all options discussed in this thread.
None of them make any sense.-v please. I can just rework userspace to allow going via
xt_bpfbut its brain damaged.Right.
xt_bpfwas a dead end from the start.
It's time to deprecate it and remove it.Quoted 2 lines
This helps gradually moving towards move epbf for those that still heavily rely on the classic forwarding path.
No one is using it.
If it was, we would have seen at least one bug report over all these years. We've seen none.Quoted 5 lines
tbh we had a fair share of wrong design decisions that look very reasonable early on and turned out to be useless with zero users.
BPF_PROG_TYPE_SCHED_ACTand BPF_PROG_TYPE_LWT*
are in this category. > All this code does is bit rot.+1
Quoted 2 lines
As a minimum we shouldn't step on the same rakes.
xt_ebpfwould be the same dead code asxt_bpf.+1, and on top, the user experience will just be horrible. :(
Quoted 7 lines
Quoted 6 lines
If you are open to
BPF_PROG_TYPE_NETFILTERI can go that route as well, raw bpf program attachment viaNF_HOOKand the bpf dispatcher, but it will take significantly longer to get there.It involves reviving
https://lore.kernel.org/netfilter-devel/[email protected]/Quoted 6 lines
Quoted 2 lines
I missed it earlier. What is the end goal ?
Optimize nft run-time with on the fly generation of bpf byte code ?Or rather to provide a pendant to nft given existence of
xt_bpf, and the latter will be removed at some point? (If so, can't we just deprecate the oldxt_bpf?)FWIW we've been using both lwt bpf and
xt_bpfon our production workloads for a few years now.
xt_bpfallows us to apply custom sophisticated policy logic at connection establishment - which is not really possible (or efficient) using iptables/nft constructs - without needing to reinvent all the facilities that nf provides like connection tracking, ALGs, and simple filtering.
Quoted 5 lines
As for lwt bpf, We use it for load balancing towards collect md tunnels. While this can be done at tc egress for unfragmented packets, the lwt out hook - when used in tandem with nf fragment reassembly - provides a hooking point where a bpf program can see reassembled packets and load balance based on their internals.
Sounds very interesting!
Any open source code to look at ?
Quoted 63 lines · Alexei Starovoitov
Quoted 61 lines · Eyal Birger
Quoted 44 lines · Daniel Borkmann
Quoted 22 lines · Alexei Starovoitov
Quoted 10 lines · Florian Westphal
Quoted 6 lines · Alexei Starovoitov
Quoted 2 lines
1 and 2 have the upside that its easy to handle a 'file not found' error.
I'm strongly against calling into bpf from the inner guts of nft. Nack to all options discussed in this thread.
None of them make any sense.-v please. I can just rework userspace to allow going via
xt_bpfbut its brain damaged.Right.
xt_bpfwas a dead end from the start.
It's time to deprecate it and remove it.Quoted 2 lines
This helps gradually moving towards move epbf for those that still heavily rely on the classic forwarding path.
No one is using it.
If it was, we would have seen at least one bug report over all these years. We've seen none.Quoted 5 lines
tbh we had a fair share of wrong design decisions that look very reasonable early on and turned out to be useless with zero users.
BPF_PROG_TYPE_SCHED_ACTand BPF_PROG_TYPE_LWT*
are in this category. > All this code does is bit rot.+1
Quoted 2 lines
As a minimum we shouldn't step on the same rakes.
xt_ebpfwould be the same dead code asxt_bpf.+1, and on top, the user experience will just be horrible. :(
Quoted 7 lines
Quoted 6 lines
If you are open to
BPF_PROG_TYPE_NETFILTERI can go that route as well, raw bpf program attachment viaNF_HOOKand the bpf dispatcher, but it will take significantly longer to get there.It involves reviving
https://lore.kernel.org/netfilter-devel/[email protected]/Quoted 6 lines
Quoted 2 lines
I missed it earlier. What is the end goal ?
Optimize nft run-time with on the fly generation of bpf byte code ?Or rather to provide a pendant to nft given existence of
xt_bpf, and the latter will be removed at some point? (If so, can't we just deprecate the oldxt_bpf?)FWIW we've been using both lwt bpf and
xt_bpfon our production workloads for a few years now.
xt_bpfallows us to apply custom sophisticated policy logic at connection establishment - which is not really possible (or efficient) using iptables/nft constructs - without needing to reinvent all the facilities that nf provides like connection tracking, ALGs, and simple filtering.
Quoted 8 lines
Quoted 5 lines
As for lwt bpf, We use it for load balancing towards collect md tunnels. While this can be done at tc egress for unfragmented packets, the lwt out hook - when used in tandem with nf fragment reassembly - provides a hooking point where a bpf program can see reassembled packets and load balance based on their internals.
Sounds very interesting!
Any open source code to look at ?
For these projects there isn't at this point. But some of the benefit in these specific hooking points is that our custom logic is very scoped and integrates well with the "classical" forwarding path.
In netfilter we have an identity based policy engine provisioning sets of bpf maps. These maps are use used by policy programs invoked by xt_bpf on connection establishment as part of a larger set of iptables rules.
In LWT this solved us a problem with fragmented traffic, as our load balancing solution supports - among other things - IPsec stickiness based on ESP-in-UDP SPI and as such needs to see unfragemented traffic.
Eyal.
Florian Westphal
maintainerQuoted 6 lines · Daniel Borkmann
Quoted 3 lines · Alexei Starovoitov
On Wed, Aug 31, 2022 at 8:53 AM Florian Westphal <[email protected]> wrote:
As a minimum we shouldn't step on the same rakes.xt_ebpfwould be the same dead code asxt_bpf.+1, and on top, the user experience will just be horrible. :(
Compared to what?
Quoted 7 lines
Quoted 7 lines
Quoted 6 lines
If you are open to
BPF_PROG_TYPE_NETFILTERI can go that route as well, raw bpf program attachment viaNF_HOOKand the bpf dispatcher, but it will take significantly longer to get there.It involves reviving
https://lore.kernel.org/netfilter-devel/[email protected]/
Quoted 6 lines
Quoted 2 lines
I missed it earlier. What is the end goal ?
Optimize nft run-time with on the fly generation of bpf byte code ?Or rather to provide a pendant to nft given existence of
xt_bpf, and the latter will be removed at some point? (If so, can't we just deprecate the oldxt_bpf?)
See my reply to Alexey, immediate goal was to get rid of the indirect calls by providing a tailored/jitted equivalent of nf_hook_slow().
The next step could be to allow implementation of netfilter hooks (i.e., kernel modules that call nf_register_net_hook()) in bpf but AFAIU it requires addition of BPF_PROG_TYPE_NETFILTER etc.
After that, yes, one could think about how to jit nft_do_chain() and all the rest of the nft machinery.
Quoted 25 lines · Florian Westphal
Quoted 6 lines · Daniel Borkmann
Quoted 3 lines · Alexei Starovoitov
On Wed, Aug 31, 2022 at 8:53 AM Florian Westphal <[email protected]> wrote:
As a minimum we shouldn't step on the same rakes.xt_ebpfwould be the same dead code asxt_bpf.+1, and on top, the user experience will just be horrible. :(
Compared to what?
Quoted 7 lines
Quoted 7 lines
Quoted 6 lines
If you are open to
BPF_PROG_TYPE_NETFILTERI can go that route as well, raw bpf program attachment viaNF_HOOKand the bpf dispatcher, but it will take significantly longer to get there.It involves reviving
https://lore.kernel.org/netfilter-devel/[email protected]/Quoted 6 lines
Quoted 2 lines
I missed it earlier. What is the end goal ?
Optimize nft run-time with on the fly generation of bpf byte code ?Or rather to provide a pendant to nft given existence of
xt_bpf, and the latter will be removed at some point? (If so, can't we just deprecate the oldxt_bpf?)
Quoted 6 lines
See my reply to Alexey, immediate goal was to get rid of the indirect calls by providing a tailored/jitted equivalent of
nf_hook_slow().The next step could be to allow implementation of netfilter hooks (i.e., kernel modules that call
nf_register_net_hook()) in bpf but AFAIU it requires addition ofBPF_PROG_TYPE_NETFILTERetc.
We were adding new prog and maps types in the past. Now new features are being added differently.
All of the networking either works with sk_buff-s or xdp frames. We try hard not to add any new uapi helpers. Everything is moving to kfuncs.
Other sub-systems should be able to use bpf without touching the bpf core. See hid-bpf as an example.
It needs several verifier improvements, but doesn't need new prog types, helpers, etc.
Quoted 2 lines
After that, yes, one could think about how to jit
nft_do_chain()and all the rest of the nft machinery.
Sounds like a ton of work. All that just to accelerate nft a bit? I think there are more impactful projects to work on.
For example, accelerating classic iptables with bpf would immediately help a bunch of users.
Florian Westphal
maintainerQuoted 7 lines · Alexei Starovoitov
Quoted 6 lines
See my reply to Alexey, immediate goal was to get rid of the indirect calls by providing a tailored/jitted equivalent of
nf_hook_slow().The next step could be to allow implementation of netfilter hooks (i.e., kernel modules that call
nf_register_net_hook()) in bpf but AFAIU it requires addition ofBPF_PROG_TYPE_NETFILTERetc.
Quoted 9 lines
We were adding new prog and maps types in the past. Now new features are being added differently.
All of the networking either works with sk_buff-s or xdp frames. We try hard not to add any new uapi helpers. Everything is moving to kfuncs.
Other sub-systems should be able to use bpf without touching the bpf core. See hid-bpf as an example.
It needs several verifier improvements, but doesn't need new prog types, helpers, etc.
I don't see how it can be done without a new prog type, the bpf progs would need access to "nf_hook_state" struct, passed as argument to nf_hook_slow() (and down to the individual xt_foo modules...).
We can't change the existing netfilter hook prototype to go by sk_buff * as that doesn't have all information, most prominent are the input and output net_device, but also okfn is needed for async reinject (nf_queue), the hook location and so on.
Quoted 7 lines
Quoted 2 lines
After that, yes, one could think about how to jit
nft_do_chain()and all the rest of the nft machinery.Sounds like a ton of work. All that just to accelerate nft a bit? I think there are more impactful projects to work on.
For example, accelerating classic iptables with bpf would immediately help a bunch of users.
Maybe, but from the problem points and the required effort it doesn't matter if the chosen target is iptables or nftables; as far as the time/effort needed I'd say they are identical.
The hard issues that need to be solved first are the same; they reside in the netfilter core and not in the specific interpreter (nft_do_chain vs. ipt_do_table and friends).
nf_tables might be *slightly* easier once that point would be reached because the core functionality is more integrated with nf_tables whereas in iptables there is more copypastry (ipt_do_table, ip6t_do_table, ebt_do_table, ...).
Florian Westphal
maintainerQuoted 6 lines · Alexei Starovoitov
Quoted 2 lines
This helps gradually moving towards move epbf for those that still heavily rely on the classic forwarding path.
No one is using it.
If it was, we would have seen at least one bug report over all these years. We've seen none.
Err, it IS used, else I would not have sent this patch.
Quoted 4 lines
very reasonable early on and turned out to be useless with zero users.
BPF_PROG_TYPE_SCHED_ACTand BPF_PROG_TYPE_LWT* are in this category.
I doubt it had 0 users. Those users probably moved to something better?
Quoted 2 lines
As a minimum we shouldn't step on the same rakes.
xt_ebpfwould be the same dead code asxt_bpf.
Its just 160 LOC or so, I don't see it has a huge technical debt.
Quoted 8 lines
Quoted 6 lines
If you are open to
BPF_PROG_TYPE_NETFILTERI can go that route as well, raw bpf program attachment viaNF_HOOKand the bpf dispatcher, but it will take significantly longer to get there.It involves reviving
https://lore.kernel.org/netfilter-devel/[email protected]/I missed it earlier. What is the end goal ?
Immediate goal: get rid of all indirect calls from NF_HOOK() invocations. Its about 2% speedup in my tests (with connection tracking+defrag enabled).
This series changes prototype of the callbacks to int foo(struct *), so I think it would be possible to build on this and allow attaching raw bpf progs/implement what is now a netfilter kernel module as a bpf program.
I have not spent time on this so far though, so I don't know yet how the "please attach prog id 12345 at FORWARD with prio 42" should be done.
Quoted 1 line
Optimize nft run-time with on the fly generation of bpf byte code ?
This could be done too, so far this JITs nf_hook_slow() only. The big question for nft run-time would be how and where to do the JIT translation.
I think that "nft run time jit" would be step 3, after allowing (re)implementation of netfilter modules via bpf programs.
Le 31/08/2022 à 23:57, Florian Westphal a écrit :
Quoted 10 lines
Quoted 6 lines · Alexei Starovoitov
Quoted 2 lines
This helps gradually moving towards move epbf for those that still heavily rely on the classic forwarding path.
No one is using it.
If it was, we would have seen at least one bug report over all these years. We've seen none.Err, it IS used, else I would not have sent this patch.
Quoted 7 lines
Quoted 4 lines
very reasonable early on and turned out to be useless with zero users.
BPF_PROG_TYPE_SCHED_ACTand BPF_PROG_TYPE_LWT* are in this category.I doubt it had 0 users. Those users probably moved to something better?
We are using BPF_PROG_TYPE_SCHED_ACT to perform custom encapsulations. What could we used to replace that?
Quoted 18 lines · Nicolas Dichtel
Quoted 15 lines · Florian Westphal
Quoted 6 lines · Alexei Starovoitov
Quoted 2 lines
This helps gradually moving towards move epbf for those that still heavily rely on the classic forwarding path.
No one is using it.
If it was, we would have seen at least one bug report over all these years. We've seen none.Err, it IS used, else I would not have sent this patch.
Quoted 4 lines
very reasonable early on and turned out to be useless with zero users.
BPF_PROG_TYPE_SCHED_ACTand BPF_PROG_TYPE_LWT* are in this category.
Quoted 4 lines
Quoted 2 lines
I doubt it had 0 users. Those users probably moved to something better?
We are using
BPF_PROG_TYPE_SCHED_ACTto perform custom encapsulations. What could we used to replace that?
SCHED_CLS. It has all of the features of cls and act combined.
Le 07/09/2022 à 05:04, Alexei Starovoitov a écrit :
Quoted 20 lines
Quoted 18 lines · Nicolas Dichtel
Quoted 15 lines · Florian Westphal
Quoted 6 lines · Alexei Starovoitov
Quoted 2 lines
This helps gradually moving towards move epbf for those that still heavily rely on the classic forwarding path.
No one is using it.
If it was, we would have seen at least one bug report over all these years. We've seen none.Err, it IS used, else I would not have sent this patch.
Quoted 4 lines
very reasonable early on and turned out to be useless with zero users.
BPF_PROG_TYPE_SCHED_ACTand BPF_PROG_TYPE_LWT* are in this category.
Quoted 6 lines
Quoted 4 lines
Quoted 2 lines
I doubt it had 0 users. Those users probably moved to something better?
We are using
BPF_PROG_TYPE_SCHED_ACTto perform custom encapsulations. What could we used to replace that?
SCHED_CLS. It has all of the features of cls and act combined.
Indeed, thanks.
Quoted 3 lines · Alexei Starovoitov
Right.
xt_bpfwas a dead end from the start.
It's time to deprecate it and remove it.
So does that extend to xt_u32, which is like a subset of BPF?
Quoted 13 lines
But, all things considered, what about this:
I'll respin, with the FILENAME attribute removed, so user says 'ebpf pinned bla', and on listing nft walks
/sys/bpf/nft/to see if it can find the name again.If it can't find it, print the id instead.
This would mean nft would also have to understand
'ebpf id 12' on input, but I think thats fine. We can document that this is not the preferred method due to the difficulty of determining the correct id to use.
Quoted 7 lines
With this, no 'extra' userspace-sake info needs to be stored. We can revisit what do with 'ebpf file /bla/foo.o' once/if that gets added.
What do you think?
Will take a while because I'll need to extend the nft side first to cope with lack of 'FILENAME' attribute.
To the extend it's still relevant, yeah, this seems like a reasonable plan to me :)
-Toke
Florian Westphal
maintainerQuoted 7 lines · Florian Westphal
Quoted 4 lines · Toke Høiland-Jørgensen
Quoted 2 lines
Tag and program id are dumped to userspace on 'list' to allow to see which program is in use in case the filename isn't available/present.
It seems a bit odd to include the file path in the kernel as well.
Its needed to be able to re-load the ruleset.
In particular, I can't find any better alternative.
load by id -> works, easy to echo back to userspace, but not stable identifier across reboots or add/del operations of the program.
load by tag -> similar, except that this time the tag needs to be adjusted whenever the program changes, so not ideal either.
load via ELF name -> same problems as the proposed 'pinned' mode, but perhaps a bit easier to use?
It has the slight advantage that users don't need to load/pin the program first, lifetime of the program would be tied to the nftables rule.
The downside is that nft needs to deal with possible rejection of the program instead of 'outsourcing' this problem to bpftool (or another program).