Add support for transmitting packets using XDP in bpf_prog_run()
Subsystems (6)
v7 cover Add support for transmitting packets using XDP in bpf_prog_run()
This series adds support for transmitting packets using XDP in bpf_prog_run(), by enabling a new mode "live packet" mode which will handle the XDP program return codes and redirect the packets to the stack or other devices.
The primary use case for this is testing the redirect map types and the ndo_xdp_xmit driver operation without an external traffic generator. But it turns out to also be useful for creating a programmable traffic generator in XDP, as well as injecting frames into the stack. A sample traffic generator, which was included in previous versions of the series, but now moved to xdp-tools, transmits up to 11.5 Mpps/core on my test machine.
To transmit the frames, the new mode instantiates a page_pool structure in bpf_prog_run() and initialises the pages to contain XDP frames with the data passed in by userspace. These frames can then be handled as though they came from the hardware XDP path, and the existing page_pool code takes care of returning and recycling them. The setup is optimised for high performance with a high number of repetitions to support stress testing and the traffic generator use case; see patch 1 for details.
v7:
- Extend the local_bh_disable() to cover the full test run loop, to prevent running concurrently with the softirq. Fixes a deadlock with veth xmit.
- Reinstate the forwarding sysctl setting in the selftest, and bump up the number of packets being transmitted to trigger the above bug.
- Update commit message to make it clear that user space can select the ingress interface.
v6:
- Fix meta vs data pointer setting and add a selftest for it
- Add local_bh_disable() around code passing packets up the stack
- Create a new netns for the selftest and use a TC program instead of the forwarding hack to count packets being XDP_PASS'ed from the test prog.
- Check for the correct ingress ifindex in the selftest
- Rebase and drop patches 1-5 that were already merged
v5:
- Rebase to current bpf-next
v4:
- Fix a few code style issues (Alexei)
- Also handle the other return codes: XDP_PASS builds skbs and injects them into the stack, and XDP_TX is turned into a redirect out the same interface (Alexei).
- Drop the last patch adding an xdp_trafficgen program to samples/bpf; this will live in xdp-tools instead (Alexei).
- Add a separate bpf_test_run_xdp_live() function to test_run.c instead of entangling the new mode in the existing bpf_test_run().
v3:
- Reorder patches to make sure they all build individually (Patchwork)
- Remove a couple of unused variables (Patchwork)
- Remove unlikely() annotation in slow path and add back John's ACK that I accidentally dropped for v2 (John)
v2:
- Split up up __xdp_do_redirect to avoid passing two pointers to it (John)
- Always reset context pointers before each test run (John)
- Use get_mac_addr() from xdp_sample_user.h instead of rolling our own (Kumar)
- Fix wrong offset for metadata pointer
Toke Høiland-Jørgensen (3):
bpf: Add "live packet" mode for XDP in bpf_prog_run() selftests/bpf: Move open_netns() and close_netns() into
network_helpers.c
selftests/bpf: Add selftest for XDP_REDIRECT in bpf_prog_run()
include/uapi/linux/bpf.h | 2 + kernel/bpf/Kconfig | 1 + net/bpf/test_run.c | 290 +++++++++++++++++- tools/include/uapi/linux/bpf.h | 2 + tools/testing/selftests/bpf/network_helpers.c | 86 ++++++ tools/testing/selftests/bpf/network_helpers.h | 9 + .../selftests/bpf/prog_tests/tc_redirect.c | 87 ------ .../bpf/prog_tests/xdp_do_redirect.c | 175 +++++++++++ .../bpf/progs/test_xdp_do_redirect.c | 85 +++++ 9 files changed, 642 insertions(+), 95 deletions(-) create mode 100644 tools/testing/selftests/bpf/prog_tests/xdp_do_redirect.c create mode 100644 tools/testing/selftests/bpf/progs/test_xdp_do_redirect.c
Signature · 1 line
2.34.1
PATCH v7 1/3 bpf: Add "live packet" mode for XDP in bpf_prog_run()
This adds support for running XDP programs through bpf_prog_run() in a mode that enables live packet processing of the resulting frames. Previous uses of bpf_prog_run() for XDP returned the XDP program return code and the modified packet data to userspace, which is useful for unit testing of XDP programs.
The existing bpf_prog_run() for XDP allows userspace to set the ingress ifindex and RXQ number as part of the context object being passed to the kernel. This patch reuses that code, but adds a new mode with different semantics, which can be selected with the new BPF_F_TEST_XDP_LIVE_FRAMES flag.
When running bpf_prog_run() in this mode, the XDP program return codes will be honoured: returning XDP_PASS will result in the frame being injected into the networking stack as if it came from the selected networking interface, while returning XDP_TX and XDP_REDIRECT will result in the frame being transmitted out that interface. XDP_TX is translated into an XDP_REDIRECT operation to the same interface, since the real XDP_TX action is only possible from within the network drivers themselves, not from the process context where bpf_prog_run() is executed.
Internally, this new mode of operation creates a page pool instance while setting up the test run, and feeds pages from that into the XDP program. The setup cost of this is amortised over the number of repetitions specified by userspace.
To support the performance testing use case, we further optimise the setup step so that all pages in the pool are pre-initialised with the packet data, and pre-computed context and xdp_frame objects stored at the start of each page. This makes it possible to entirely avoid touching the page content on each XDP program invocation, and enables sending up to 12 Mpps/core on my test box.
Because the data pages are recycled by the page pool, and the test runner doesn't re-initialise them for each run, subsequent invocations of the XDP program will see the packet data in the state it was after the last time it ran on that particular page. This means that an XDP program that modifies the packet before redirecting it has to be careful about which assumptions it makes about the packet content, but that is only an issue for the most naively written programs.
Enabling the new flag is only allowed when not setting ctx_out and data_out in the test specification, since using it means frames will be redirected somewhere else, so they can't be returned.
--- include/uapi/linux/bpf.h | 2 + kernel/bpf/Kconfig | 1 + net/bpf/test_run.c | 290 ++++++++++++++++++++++++++++++++- tools/include/uapi/linux/bpf.h | 2 + 4 files changed, 287 insertions(+), 8 deletions(-)
include/uapi/linux/bpf.h +2 -0
@@ -1225,6 +1225,8 @@ enum {/* If set, run the test on the cpu specified by bpf_attr.test.cpu */ #define BPF_F_TEST_RUN_ON_CPU (1U << 0) +/* If set, XDP frames will be transmitted after processing */ +#define BPF_F_TEST_XDP_LIVE_FRAMES (1U << 1)/* type for BPF_ENABLE_STATS */ enum bpf_stats_type {
kernel/bpf/Kconfig +1 -0
@@ -30,6 +30,7 @@ config BPF_SYSCALL select TASKS_TRACE_RCU select BINARY_PRINTF select NET_SOCK_MSG if NET + select PAGE_POOL if NET default n help Enable the bpf() system call that allows to manipulate BPF programs
net/bpf/test_run.c +282 -8
358 lines not on this page · raw message
tools/include/uapi/linux/bpf.h +2 -0
@@ -1225,6 +1225,8 @@ enum {/* If set, run the test on the cpu specified by bpf_attr.test.cpu */ #define BPF_F_TEST_RUN_ON_CPU (1U << 0) +/* If set, XDP frames will be transmitted after processing */ +#define BPF_F_TEST_XDP_LIVE_FRAMES (1U << 1)/* type for BPF_ENABLE_STATS */ enum bpf_stats_type {
Signature · 1 line
2.34.1
Re: [PATCH bpf-next v7 1/3] bpf: Add "live packet" mode for XDP in bpf_prog_run()
Quoted 8 lines · Toke Høiland-Jørgensen
Because the data pages are recycled by the page pool, and the test runner doesn't re-initialise them for each run, subsequent invocations of the XDP program will see the packet data in the state it was after the last time it ran on that particular page. This means that an XDP program that modifies the packet before redirecting it has to be careful about which assumptions it makes about the packet content, but that is only an issue for the most naively written programs.
This is too vague and partially incorrect.
The bpf program can do bpf_xdp_adjust_meta() and otherwise change packet boundaries. These effects will be seen by subsequent XDP_PASS/TX/REDIRECT, but on the next iteration the boundaries will get reset to the original values.
So the test runner actually re-initializes some parts of the data, but not the contents of the packet.
At least that's my understanding of the patch.
The users shouldn't need to dig into implementation to discover this. Please document it.
The more I think about it the more I believe that it warrants a little blurb in Documentation/bpf/ that describes what one can do with this "xdp live mode".
Another question comes to mind:
What happens when a program modifies the packet?
Does it mean that the 2nd frame will see the modified data? It will not, right?
It's the page pool size of packets that will be inited the same way at the beginning. Which is NAPI_POLL_WEIGHT * 2 == 128 packets. Why this number?
Should it be configurable?
Then the user can say: init N packets with this one pattern and the program will know that exactly N invocation will be with the same data, but N+1 it will see the 1st packet again that potentially was modified by the program.
Is it accurate?
Re: [PATCH bpf-next v7 1/3] bpf: Add "live packet" mode for XDP in bpf_prog_run()
Quoted 10 lines · Alexei Starovoitov
Quoted 8 lines · Toke Høiland-Jørgensen
Because the data pages are recycled by the page pool, and the test runner doesn't re-initialise them for each run, subsequent invocations of the XDP program will see the packet data in the state it was after the last time it ran on that particular page. This means that an XDP program that modifies the packet before redirecting it has to be careful about which assumptions it makes about the packet content, but that is only an issue for the most naively written programs.
Quoted 8 lines
This is too vague and partially incorrect.
The bpf program can dobpf_xdp_adjust_meta()and otherwise change packet boundaries. These effects will be seen by subsequent XDP_PASS/TX/REDIRECT, but on the next iteration the boundaries will get reset to the original values.
So the test runner actually re-initializes some parts of the data, but not the contents of the packet.
At least that's my understanding of the patch.
Yes, that's correct. Boundaries will be reset, data won't. The boundary reset was added later, though, so guess I neglected to update the commit message. Will fix.
Quoted 5 lines
The users shouldn't need to dig into implementation to discover this. Please document it.
The more I think about it the more I believe that it warrants a little blurb inDocumentation/bpf/that describes what one can do with this "xdp live mode".
Sure, can do. Doesn't look like BPF_PROG_RUN is documented in there at all, so guess I can start such a document :)
Quoted 7 lines
Another question comes to mind:
What happens when a program modifies the packet?
Does it mean that the 2nd frame will see the modified data? It will not, right?
It's the page pool size of packets that will be inited the same way at the beginning. Which isNAPI_POLL_WEIGHT* 2 == 128 packets. Why this number?
Yes, you're right: the next run won't see the modified packet data. The 128 pages is because we run the program loop in batches of 64 (like NAPI does, the fact that TEST_XDP_BATCH and NAPI_POLL_WEIGHT are the same is not a coincidence).
We need 2x because we want enough pages so we can keep running without allocating more, and the first batch can still be in flight on a different CPU while we're processing batch 2.
I experimented with different values, and 128 was the minimum size that didn't have a significant negative impact on performance, and above that saw diminishing returns.
Quoted 6 lines
Should it be configurable?
Then the user can say: init N packets with this one pattern and the program will know that exactly N invocation will be with the same data, but N+1 it will see the 1st packet again that potentially was modified by the program.
Is it accurate?
I thought about making it configurable, but the trouble is that it's not quite as straight-forward as the first N packets being "pristine": it depends on what happens to the packet afterwards:
On XDP_DROP, the page will be recycled immediately, whereas on XDP_{TX,REDIRECT} it will go through the egress driver after sitting in the bulk queue for a little while, so you can get reordering compared to the original execution order.
On XDP_PASS the kernel will release the page entirely from the pool when building an skb, so you'll never see that particular page again (and eventually page_pool will allocate a new batch that will be re-initialised to the original value).
If we do want to support a "pristine data" mode, I think the least cumbersome way would be to add a flag that would make the kernel re-initialise the packet data before every program invocation. The reason I didn't do this was because I didn't have a use case for it. The traffic generator use case only rewrites a tiny bit of the packet header, and it's just as easy to just keep rewriting it without assuming a particular previous value. And there's also the possibility of just calling bpf_prog_run() multiple times from userspace with a lower number of repetitions...
I'm not opposed to adding such a flag if you think it would be useful, though. WDYT?
-Toke
Re: [PATCH bpf-next v7 1/3] bpf: Add "live packet" mode for XDP in bpf_prog_run()
Quoted 3 lines · Toke Høiland-Jørgensen
Sure, can do. Doesn't look like
BPF_PROG_RUNis documented in there at all, so guess I can start such a document :)
prog_run was simple enough.
This live packet mode is a different level of complexity. Just look at the length of this thread.
We keep finding implementation details that will be relevant to anyone trying to use this interface.
They all will become part of uapi.
Quoted 13 lines
Quoted 7 lines
Another question comes to mind:
What happens when a program modifies the packet?
Does it mean that the 2nd frame will see the modified data? It will not, right?
It's the page pool size of packets that will be inited the same way at the beginning. Which isNAPI_POLL_WEIGHT* 2 == 128 packets. Why this number?Yes, you're right: the next run won't see the modified packet data. The 128 pages is because we run the program loop in batches of 64 (like NAPI does, the fact that
TEST_XDP_BATCHandNAPI_POLL_WEIGHTare the same is not a coincidence).
Quoted 7 lines
We need 2x because we want enough pages so we can keep running without allocating more, and the first batch can still be in flight on a different CPU while we're processing batch 2.
I experimented with different values, and 128 was the minimum size that didn't have a significant negative impact on performance, and above that saw diminishing returns.
I guess it's ok-ish to get stuck with 128.
It will be uapi that we cannot change though.
Are you comfortable with that?
Quoted 16 lines
Quoted 6 lines
Should it be configurable?
Then the user can say: init N packets with this one pattern and the program will know that exactly N invocation will be with the same data, but N+1 it will see the 1st packet again that potentially was modified by the program.
Is it accurate?I thought about making it configurable, but the trouble is that it's not quite as straight-forward as the first N packets being "pristine": it depends on what happens to the packet afterwards:
On
XDP_DROP, the page will be recycled immediately, whereas on XDP_{TX,REDIRECT} it will go through the egress driver after sitting in the bulk queue for a little while, so you can get reordering compared to the original execution order.
Quoted 4 lines
On
XDP_PASSthe kernel will release the page entirely from the pool when building an skb, so you'll never see that particular page again (and eventuallypage_poolwill allocate a new batch that will be re-initialised to the original value).
That all makes sense. Thanks for explaining.
Please document it and update the selftest.
Looks like XDP_DROP is not tested.
Single packet TX and REDIRECT is imo too weak to give confidence that the mechanism will not explode with millions of packets.
Quoted 10 lines
If we do want to support a "pristine data" mode, I think the least cumbersome way would be to add a flag that would make the kernel re-initialise the packet data before every program invocation. The reason I didn't do this was because I didn't have a use case for it. The traffic generator use case only rewrites a tiny bit of the packet header, and it's just as easy to just keep rewriting it without assuming a particular previous value. And there's also the possibility of just calling
bpf_prog_run()multiple times from userspace with a lower number of repetitions...
Quoted 2 lines
I'm not opposed to adding such a flag if you think it would be useful, though. WDYT?
reinit doesn't feel necessary.
How one would use this interface to send N different packets? The api provides an interface for only one.
It will be copied 128 times, but the prog_run call with repeat=1 will invoke bpf prog only once, right?
So technically doing N prog_run commands with different data and repeat=1 will achieve the result, right?
But it's not efficient, since 128 pages and 128 copies will be performed each time.
May be there is a use case for configurable page_pool size?
Re: [PATCH bpf-next v7 1/3] bpf: Add "live packet" mode for XDP in bpf_prog_run()
Quoted 11 lines · Alexei Starovoitov
Quoted 3 lines · Toke Høiland-Jørgensen
Sure, can do. Doesn't look like
BPF_PROG_RUNis documented in there at all, so guess I can start such a document :)
prog_runwas simple enough.
This live packet mode is a different level of complexity. Just look at the length of this thread.
We keep finding implementation details that will be relevant to anyone trying to use this interface.
They all will become part of uapi.
Sure, totally fine with documenting it. Just seems to me the most obvious place to put this is in a new Documentation/bpf/prog_test_run.rst file with a short introduction about the general BPF_PROG_RUN mechanism, and then a subsection dedicated to this facility.
Or would you rather I create something likeDocumentation/bpf/xdp_live_packets.rst ?
Quoted 17 lines
Quoted 17 lines
Quoted 7 lines
Another question comes to mind:
What happens when a program modifies the packet?
Does it mean that the 2nd frame will see the modified data? It will not, right?
It's the page pool size of packets that will be inited the same way at the beginning. Which isNAPI_POLL_WEIGHT* 2 == 128 packets. Why this number?Yes, you're right: the next run won't see the modified packet data. The 128 pages is because we run the program loop in batches of 64 (like NAPI does, the fact that
TEST_XDP_BATCHandNAPI_POLL_WEIGHTare the same is not a coincidence).We need 2x because we want enough pages so we can keep running without allocating more, and the first batch can still be in flight on a different CPU while we're processing batch 2.
Quoted 7 lines
Quoted 3 lines
I experimented with different values, and 128 was the minimum size that didn't have a significant negative impact on performance, and above that saw diminishing returns.
I guess it's ok-ish to get stuck with 128.
It will be uapi that we cannot change though.
Are you comfortable with that?
UAPI in what sense? I'm thinking of documenting it like:
"The packet data being supplied as data_in to BPF_PROG_RUN will be used
for the initial run of the XDP program. However, when running the program multiple times (with repeat > 1), only the packet *bounds* (i.e., the data, data_end and data_meta pointers) will be reset on each invocation, the packet data itself won't be rewritten. The pages backing the packets are recycled, but the order depends on the path the packet takes through the kernel, making it hard to predict when a particular modified page makes it back to the XDP program. In practice, this means that if the XDP program modifies the packet payload before sending out the packet, it has to be prepared to deal with subsequent invocations seeing either the initial data or the already-modified packet, in arbitrary order."
I don't think this makes any promises about any particular size of the page pool, so how does it constitute UAPI?
Quoted 21 lines
Quoted 16 lines
Quoted 6 lines
Should it be configurable?
Then the user can say: init N packets with this one pattern and the program will know that exactly N invocation will be with the same data, but N+1 it will see the 1st packet again that potentially was modified by the program.
Is it accurate?I thought about making it configurable, but the trouble is that it's not quite as straight-forward as the first N packets being "pristine": it depends on what happens to the packet afterwards:
On
XDP_DROP, the page will be recycled immediately, whereas on XDP_{TX,REDIRECT} it will go through the egress driver after sitting in the bulk queue for a little while, so you can get reordering compared to the original execution order.Quoted 4 lines
On
XDP_PASSthe kernel will release the page entirely from the pool when building an skb, so you'll never see that particular page again (and eventuallypage_poolwill allocate a new batch that will be re-initialised to the original value).
Quoted 6 lines
That all makes sense. Thanks for explaining.
Please document it and update the selftest.
Looks likeXDP_DROPis not tested.
Single packet TX and REDIRECT is imo too weak to give confidence that the mechanism will not explode with millions of packets.
OK, will do.
Quoted 10 lines
Quoted 10 lines
If we do want to support a "pristine data" mode, I think the least cumbersome way would be to add a flag that would make the kernel re-initialise the packet data before every program invocation. The reason I didn't do this was because I didn't have a use case for it. The traffic generator use case only rewrites a tiny bit of the packet header, and it's just as easy to just keep rewriting it without assuming a particular previous value. And there's also the possibility of just calling
bpf_prog_run()multiple times from userspace with a lower number of repetitions...
Quoted 6 lines
Quoted 2 lines
I'm not opposed to adding such a flag if you think it would be useful, though. WDYT?
reinit doesn't feel necessary.
How one would use this interface to send N different packets? The api provides an interface for only one.
By having the XDP program react appropriately. E.g., here is the XDP program used by the trafficgen tool to cycle through UDP ports when sending out the packets - it just reads the current value and updates based on that, so it doesn't matter if it sees the initial page or one it already modified:
const volatile __u16 port_start;
const volatile __u16 port_range;
volatile __u16 next_port = 0;
SEC("xdp")
int xdp_redirect_update_port(struct xdp_md *ctx)
{
void *data_end = (void *)(long)ctx->data_end;
void *data = (void *)(long)ctx->data;
__u16 cur_port, cksum_diff;
struct udphdr *hdr;
hdr = data + (sizeof(struct ethhdr) + sizeof(struct ipv6hdr));
if (hdr + 1 > data_end)
return XDP_ABORTED;
cur_port = bpf_ntohs(hdr->dest);
cksum_diff = next_port - cur_port;
if (cksum_diff) {
hdr->check = bpf_htons(~(~bpf_ntohs(hdr->check) + cksum_diff));
hdr->dest = bpf_htons(next_port);
}
if (next_port++ >= port_start + port_range - 1)
next_port = port_start;
return bpf_redirect(ifindex_out, 0);
}You could do something similar with a whole packet header or payload; or you could even populate a map with the full-size packets and copy that in based on a counter.
Quoted 7 lines
It will be copied 128 times, but the
prog_runcall with repeat=1 will invoke bpf prog only once, right?
So technically doing Nprog_runcommands with different data and repeat=1 will achieve the result, right?
But it's not efficient, since 128 pages and 128 copies will be performed each time.
May be there is a use case for configurablepage_poolsize?
Hmm, we could size the page_pool as min(repeat, 128) to avoid the extra copies when they won't be used?
Another question seeing as the merge window is imminent: How do you feel about merging this before the merge window? I can resubmit before it opens with the updated selftest and documentation, and we can deal with any tweaks during the -rcs; or would you rather postpone the whole thing until the next cycle?
-Toke
Re: [PATCH bpf-next v7 1/3] bpf: Add "live packet" mode for XDP in bpf_prog_run()
Quoted 6 lines · Toke Høiland-Jørgensen
Sure, totally fine with documenting it. Just seems to me the most obvious place to put this is in a new
Documentation/bpf/prog_test_run.rstfile with a short introduction about the generalBPF_PROG_RUNmechanism, and then a subsection dedicated to this facility.
sgtm
Quoted 19 lines
Quoted 3 lines
I guess it's ok-ish to get stuck with 128.
It will be uapi that we cannot change though.
Are you comfortable with that?UAPI in what sense? I'm thinking of documenting it like:
"The packet data being supplied as
data_intoBPF_PROG_RUNwill be used
for the initial run of the XDP program. However, when running the program multiple times (with repeat > 1), only the packet *bounds* (i.e., the data,data_endanddata_metapointers) will be reset on each invocation, the packet data itself won't be rewritten. The pages backing the packets are recycled, but the order depends on the path the packet takes through the kernel, making it hard to predict when a particular modified page makes it back to the XDP program. In practice, this means that if the XDP program modifies the packet payload before sending out the packet, it has to be prepared to deal with subsequent invocations seeing either the initial data or the already-modified packet, in arbitrary order."
Quoted 2 lines
I don't think this makes any promises about any particular size of the page pool, so how does it constitute UAPI?
Could you explain out-of-order scanario again?
It's possible only if xdp_redirect is done into different netdevs. Then they can xmit at different times and cycle pages back into the loop in different order. But TX or REDIRECT into the same netdev will keep the pages in the same order. So the program can rely on that.
Quoted 10 lines
Quoted 4 lines
reinit doesn't feel necessary.
How one would use this interface to send N different packets? The api provides an interface for only one.By having the XDP program react appropriately. E.g., here is the XDP program used by the trafficgen tool to cycle through UDP ports when sending out the packets - it just reads the current value and updates based on that, so it doesn't matter if it sees the initial page or one it already modified:
Sure. I think there is an untapped potential here.
With this live packet prog_run anyone can buy 10G or 100G nic equipped server and for free transform it into $300k+ IXIA beating machine. It could be a game changer. pktgen doesn't come close.
I'm thinking about generating and consuming test TCP traffic. TCP blaster would xmit 1M TCP connections through this live prog_run into eth0 and consume the traffic returning from "server under test" via a different XDP program attached to eth0.
The prog_run's xdp prog would need to send SYN, increment sequence number, and keep sane data in the packets. It could be HTTP request, for example.
To achive this IXIA beating setup the TCP blaster would need a full understanding of what page pool is doing with the packets. Just saying "in arbitrary order" is a non starter. It diminishes this live prog_run into pktgen equivalent which is still useful, but lots of potential is lost.
Quoted 5 lines
Another question seeing as the merge window is imminent: How do you feel about merging this before the merge window? I can resubmit before it opens with the updated selftest and documentation, and we can deal with any tweaks during the -rcs; or would you rather postpone the whole thing until the next cycle?
It's already too late for this merge window, but bpf-next is always open. Just like it was open for the last year. So please resubmit as soon as the tests are green and this discussion is over.
Re: [PATCH bpf-next v7 1/3] bpf: Add "live packet" mode for XDP in bpf_prog_run()
Quoted 9 lines · Alexei Starovoitov
Quoted 6 lines · Toke Høiland-Jørgensen
Sure, totally fine with documenting it. Just seems to me the most obvious place to put this is in a new
Documentation/bpf/prog_test_run.rstfile with a short introduction about the generalBPF_PROG_RUNmechanism, and then a subsection dedicated to this facility.sgtm
Great!
Quoted 22 lines
Quoted 19 lines
Quoted 3 lines
I guess it's ok-ish to get stuck with 128.
It will be uapi that we cannot change though.
Are you comfortable with that?UAPI in what sense? I'm thinking of documenting it like:
"The packet data being supplied as
data_intoBPF_PROG_RUNwill be used
for the initial run of the XDP program. However, when running the program multiple times (with repeat > 1), only the packet *bounds* (i.e., the data,data_endanddata_metapointers) will be reset on each invocation, the packet data itself won't be rewritten. The pages backing the packets are recycled, but the order depends on the path the packet takes through the kernel, making it hard to predict when a particular modified page makes it back to the XDP program. In practice, this means that if the XDP program modifies the packet payload before sending out the packet, it has to be prepared to deal with subsequent invocations seeing either the initial data or the already-modified packet, in arbitrary order."Quoted 2 lines
I don't think this makes any promises about any particular size of the page pool, so how does it constitute UAPI?
Quoted 6 lines
Could you explain out-of-order scanario again?
It's possible only ifxdp_redirectis done into different netdevs. Then they can xmit at different times and cycle pages back into the loop in different order. But TX or REDIRECT into the same netdev will keep the pages in the same order. So the program can rely on that.
I left that out on purpose: I feel it's exposing an internal implementation detail as UAPI (as you said). And I'm not convinced it really needed (or helpful) - see below.
Quoted 11 lines
Quoted 10 lines
Quoted 4 lines
reinit doesn't feel necessary.
How one would use this interface to send N different packets? The api provides an interface for only one.By having the XDP program react appropriately. E.g., here is the XDP program used by the trafficgen tool to cycle through UDP ports when sending out the packets - it just reads the current value and updates based on that, so it doesn't matter if it sees the initial page or one it already modified:
Quoted 10 lines
Sure. I think there is an untapped potential here.
With this live packetprog_runanyone can buy 10G or 100G nic equipped server and for free transform it into $300k+ IXIA beating machine. It could be a game changer. pktgen doesn't come close.
I'm thinking about generating and consuming test TCP traffic. TCP blaster would xmit 1M TCP connections through this liveprog_runinto eth0 and consume the traffic returning from "server under test" via a different XDP program attached to eth0.
The prog_run's xdp prog would need to send SYN, increment sequence number, and keep sane data in the packets. It could be HTTP request, for example.
I'm glad you see the potential :)
Quoted 5 lines
To achive this IXIA beating setup the TCP blaster would need a full understanding of what page pool is doing with the packets. Just saying "in arbitrary order" is a non starter. It diminishes this live
prog_runinto pktgen equivalent which is still useful, but lots of potential is lost.
I don't think a detailed knowledge of how the pages are recycled is needed to implement a TCP stream? Even if you just rely on the packets being recycled with a fixed period of 128 pages, how does that make your XDP program simpler? You'll still have to update the packet header for each packet, with state kept in a map; so why is it helpful to know when a particular page comes back?
I'll try implementing a TCP stream mode in xdp_trafficgen just to make sure I'm not missing something. But I believe that sending out a stream of packets that looks like a coherent TCP stream should be simple enough, at least. Dealing with the full handshake + CWND control loop will be harder, though, and right now I think it'll require multiple trips back to userspace.
Quoted 9 lines
Quoted 5 lines
Another question seeing as the merge window is imminent: How do you feel about merging this before the merge window? I can resubmit before it opens with the updated selftest and documentation, and we can deal with any tweaks during the -rcs; or would you rather postpone the whole thing until the next cycle?
It's already too late for this merge window, but bpf-next is always open. Just like it was open for the last year. So please resubmit as soon as the tests are green and this discussion is over.
Ah, OK. I was under the impression that the cutoff date was tomorrow; has that changed? But no worries, I'll spend my Sunday outside instead of coding, then, and come back to this tomorrow :)
-Toke
Re: [PATCH bpf-next v7 1/3] bpf: Add "live packet" mode for XDP in bpf_prog_run()
Quoted 4 lines · Toke Høiland-Jørgensen
I left that out on purpose: I feel it's exposing an internal implementation detail as UAPI (as you said). And I'm not convinced it really needed (or helpful) - see below.
It's irrelevant whether it's documented or not.
Once this implementation detail is being relied upon
by user space it becomes an undocumented uapi that we cannot change.
Quoted 6 lines
I'll try implementing a TCP stream mode in
xdp_trafficgenjust to make sure I'm not missing something. But I believe that sending out a stream of packets that looks like a coherent TCP stream should be simple enough, at least. Dealing with the full handshake + CWND control loop will be harder, though, and right now I think it'll require multiple trips back to userspace.
The patch set looks very close to being able to do such TCP streaming. Let's make sure nothing is missing from API before we land it.
Re: [PATCH bpf-next v7 1/3] bpf: Add "live packet" mode for XDP in bpf_prog_run()
Quoted 17 lines · Alexei Starovoitov
Quoted 4 lines · Toke Høiland-Jørgensen
I left that out on purpose: I feel it's exposing an internal implementation detail as UAPI (as you said). And I'm not convinced it really needed (or helpful) - see below.
It's irrelevant whether it's documented or not.
Once this implementation detail is being relied upon
by user space it becomes an undocumented uapi that we cannot change.Quoted 6 lines
I'll try implementing a TCP stream mode in
xdp_trafficgenjust to make sure I'm not missing something. But I believe that sending out a stream of packets that looks like a coherent TCP stream should be simple enough, at least. Dealing with the full handshake + CWND control loop will be harder, though, and right now I think it'll require multiple trips back to userspace.
Quoted 2 lines
The patch set looks very close to being able to do such TCP streaming. Let's make sure nothing is missing from API before we land it.
Hi Toke, I am also looking at ways to blast tcp packets by using bpf to overcome the pktgen udp-only limitation.
Are you planning to respin with a TCP stream mode in xdp_trafficgen ? Thanks !
Re: [PATCH bpf-next v7 1/3] bpf: Add "live packet" mode for XDP in bpf_prog_run()
Quoted 18 lines · Martin KaFai Lau
Quoted 17 lines · Alexei Starovoitov
Quoted 4 lines · Toke Høiland-Jørgensen
I left that out on purpose: I feel it's exposing an internal implementation detail as UAPI (as you said). And I'm not convinced it really needed (or helpful) - see below.
It's irrelevant whether it's documented or not.
Once this implementation detail is being relied upon
by user space it becomes an undocumented uapi that we cannot change.Quoted 6 lines
I'll try implementing a TCP stream mode in
xdp_trafficgenjust to make sure I'm not missing something. But I believe that sending out a stream of packets that looks like a coherent TCP stream should be simple enough, at least. Dealing with the full handshake + CWND control loop will be harder, though, and right now I think it'll require multiple trips back to userspace.
Quoted 6 lines
Quoted 2 lines
The patch set looks very close to being able to do such TCP streaming. Let's make sure nothing is missing from API before we land it.
Hi Toke, I am also looking at ways to blast tcp packets by using bpf to overcome the pktgen udp-only limitation.
Are you planning to respin with a TCP stream mode inxdp_trafficgen? Thanks !
Yes, working on it! Got sidetracked a bit, but hoping to have something to show for my efforts sometime next week :)
-Toke
PATCH v7 2/3 selftests/bpf: Move open_netns() and close_netns() into network_helpers.c
In reply to Toke Høiland-JørgensenThese will also be used by the xdp_do_redirect test being added in the next commit.
--- tools/testing/selftests/bpf/network_helpers.c | 86 ++++++++++++++++++ tools/testing/selftests/bpf/network_helpers.h | 9 ++ .../selftests/bpf/prog_tests/tc_redirect.c | 87 ------------------- 3 files changed, 95 insertions(+), 87 deletions(-)
tools/testing/selftests/bpf/network_helpers.c +86 -0
@@ -1,18 +1,25 @@// SPDX-License-Identifier: GPL-2.0-only +#define _GNU_SOURCE +#include <errno.h> #include <stdbool.h> #include <stdio.h> #include <string.h> #include <unistd.h> +#include <sched.h> #include <arpa/inet.h> +#include <sys/mount.h> +#include <sys/stat.h> #include <linux/err.h> #include <linux/in.h> #include <linux/in6.h> +#include <linux/limits.h> #include "bpf_util.h" #include "network_helpers.h" +#include "test_progs.h" #define clean_errno () (errno == 0 ? "None" : strerror(errno))#define log_err (MSG , ...) ({ \
@@ -356,3 +363,82 @@ char *ping_command(int family) }return "ping" ; } + +struct nstoken { +int orig_netns_fd; +}; + +static int setns_by_fd (int nsfd) +{ +int err; + + err =setns (nsfd,CLONE_NEWNET ); +close (nsfd); + +if (!ASSERT_OK (err,"setns" )) +return err; + +/* Switch /sys to the new namespace so that e.g. /sys/class/net +* reflects the devices in the new namespace. +*/ + err =unshare (CLONE_NEWNS ); +if (!ASSERT_OK (err,"unshare" )) +return err; + +/* Make our /sys mount private, so the following umount won't +* trigger the global umount in case it's shared. +*/ + err =mount ("none" ,"/sys" ,NULL ,MS_PRIVATE ,NULL ); +if (!ASSERT_OK (err,"remount private /sys" )) +return err; + + err =umount2 ("/sys" ,MNT_DETACH ); +if (!ASSERT_OK (err,"umount2 /sys" )) +return err; + + err =mount ("sysfs" ,"/sys" ,"sysfs" ,0 ,NULL ); +if (!ASSERT_OK (err,"mount /sys" )) +return err; + + err =mount ("bpffs" ,"/sys/fs/bpf" ,"bpf" ,0 ,NULL ); +if (!ASSERT_OK (err,"mount /sys/fs/bpf" )) +return err; + +return 0 ; +} + +struct nstoken *open_netns (const char *name) +{ +int nsfd; +char nspath[PATH_MAX ]; +int err; +struct nstoken *token; + + token =malloc (sizeof (struct nstoken )); +if (!ASSERT_OK_PTR (token,"malloc token" )) +return NULL ; + + token->orig_netns_fd =open ("/proc/self/ns/net" ,O_RDONLY ); +if (!ASSERT_GE (token->orig_netns_fd,0 ,"open /proc/self/ns/net" )) + goto fail; + +snprintf (nspath,sizeof (nspath),"%s/%s" ,"/var/run/netns" , name); + nsfd =open (nspath,O_RDONLY |O_CLOEXEC ); +if (!ASSERT_GE (nsfd,0 ,"open netns fd" )) + goto fail; + + err =setns_by_fd (nsfd); +if (!ASSERT_OK (err,"setns_by_fd" )) + goto fail; + +return token; +fail: +free (token); +return NULL ; +} + +void close_netns (struct nstoken *token) +{ +ASSERT_OK (setns_by_fd (token->orig_netns_fd),"setns_by_fd" ); +free (token); +}
tools/testing/selftests/bpf/network_helpers.h +9 -0
@@ -55,4 +55,13 @@ int make_sockaddr(int family, const char *addr_str, __u16 port, struct sockaddr_storage *addr, socklen_t *len);char *ping_command (int family); +struct nstoken ; +/** +* open_netns() - Switch to specified network namespace by name. +* +* Returns token with which to restore the original namespace +* using close_netns(). +*/ +struct nstoken *open_netns (const char *name); +void close_netns (struct nstoken *token);#endif
tools/testing/selftests/bpf/prog_tests/tc_redirect.c +0 -87
101 lines not on this page · raw message
Signature · 1 line
2.34.1
PATCH v7 3/3 selftests/bpf: Add selftest for XDP_REDIRECT in bpf_prog_run()
In reply to Toke Høiland-JørgensenThis adds a selftest for the XDP_REDIRECT facility in bpf_prog_run, that redirects packets into a veth and counts them using an XDP program on the other side of the veth pair and a TC program on the local side of the veth.
--- .../bpf/prog_tests/xdp_do_redirect.c | 175 ++++++++++++++++++ .../bpf/progs/test_xdp_do_redirect.c | 85 +++++++++ 2 files changed, 260 insertions(+) create mode 100644 tools/testing/selftests/bpf/prog_tests/xdp_do_redirect.c create mode 100644 tools/testing/selftests/bpf/progs/test_xdp_do_redirect.c
tools/testing/selftests/bpf/prog_tests/xdp_do_redirect.c new file +175 -0
@@ -0,0 +1,175 @@ +// SPDX-License-Identifier: GPL-2.0 +#include <test_progs.h> +#include <network_helpers.h> +#include <net/if.h> +#include <linux/if_ether.h> +#include <linux/if_packet.h> +#include <linux/ipv6.h> +#include <linux/in6.h> +#include <linux/udp.h> +#include <bpf/bpf_endian.h> +#include "test_xdp_do_redirect.skel.h" + +#define SYS (fmt, ...) \ + ({ \ + char cmd[1024]; \ + snprintf(cmd, sizeof(cmd), fmt, ##__VA_ARGS__); \ + if (!ASSERT_OK(system(cmd), cmd)) \ + goto out; \ + }) + +struct udp_packet { +struct ethhdr eth; +struct ipv6hdr iph; +struct udphdr udp; +__u8 payload[64 -sizeof (struct udphdr ) + -sizeof (struct ethhdr ) -sizeof (struct ipv6hdr )]; +} __packed; + +static struct udp_packet pkt_udp = { + .eth.h_proto =__bpf_constant_htons (ETH_P_IPV6 ), + .eth.h_dest = {0x00 ,0x11 ,0x22 ,0x33 ,0x44 ,0x55 }, + .eth.h_source = {0x66 ,0x77 ,0x88 ,0x99 ,0xaa ,0xbb }, + .iph.version =6 , + .iph.nexthdr =IPPROTO_UDP , + .iph.payload_len =bpf_htons (sizeof (struct udp_packet ) + - offsetof(struct udp_packet , udp)), + .iph.hop_limit =2 , + .iph.saddr.s6_addr16 = {bpf_htons (0xfc00 ),0 ,0 ,0 ,0 ,0 ,0 ,bpf_htons (1 )}, + .iph.daddr.s6_addr16 = {bpf_htons (0xfc00 ),0 ,0 ,0 ,0 ,0 ,0 ,bpf_htons (2 )}, + .udp.source =bpf_htons (1 ), + .udp.dest =bpf_htons (1 ), + .udp.len =bpf_htons (sizeof (struct udp_packet ) + - offsetof(struct udp_packet , udp)), + .payload = {0x42 },/* receiver XDP program matches on this */ +}; + +static int attach_tc_prog (struct bpf_tc_hook *hook,int fd) +{ + DECLARE_LIBBPF_OPTS(bpf_tc_opts, opts, .handle = 1, .priority = 1, .prog_fd = fd); +int ret; + + ret =bpf_tc_hook_create (hook); +if (!ASSERT_OK (ret,"create tc hook" )) +return ret; + + ret =bpf_tc_attach (hook, &opts); +if (!ASSERT_OK (ret,"bpf_tc_attach" )) { +bpf_tc_hook_destroy (hook); +return ret; + } + +return 0 ; +} + +#define NUM_PKTS 1000000 +void test_xdp_do_redirect (void ) +{ +int err, xdp_prog_fd, tc_prog_fd, ifindex_src, ifindex_dst; +char data[sizeof (pkt_udp) +sizeof (__u32)]; +struct test_xdp_do_redirect *skel =NULL ; +struct nstoken *nstoken =NULL ; +struct bpf_link *link; + +struct xdp_md ctx_in = { .data =sizeof (__u32), + .data_end =sizeof (data) }; +DECLARE_LIBBPF_OPTS (bpf_test_run_opts, opts, + .data_in = &data, + .data_size_in = sizeof(data), + .ctx_in = &ctx_in, + .ctx_size_in = sizeof(ctx_in), + .flags = BPF_F_TEST_XDP_LIVE_FRAMES, + .repeat = NUM_PKTS, + ); +DECLARE_LIBBPF_OPTS (bpf_tc_hook, tc_hook, + .attach_point = BPF_TC_INGRESS); + +memcpy (&data[sizeof (__u32)], &pkt_udp,sizeof (pkt_udp)); + *((__u32 *)data) =0x42 ;/* metadata test value */ + + skel =test_xdp_do_redirect__open (); +if (!ASSERT_OK_PTR (skel,"skel" )) +return ; + +/* The XDP program we run with bpf_prog_run() will cycle through all +* three xmit (PASS/TX/REDIRECT) return codes starting from above, and +* ending up with PASS, so we should end up with two packets on the dst +* iface and NUM_PKTS-2 in the TC hook. We match the packets on the UDP +* payload. +*/ +SYS ("ip netns add testns" ); + nstoken =open_netns ("testns" ); +if (!ASSERT_OK_PTR (nstoken,"setns" )) + goto out; + +SYS ("ip link add veth_src type veth peer name veth_dst" ); +SYS ("ip link set dev veth_src address 00:11:22:33:44:55" ); +SYS ("ip link set dev veth_dst address 66:77:88:99:aa:bb" ); +SYS ("ip link set dev veth_src up" ); +SYS ("ip link set dev veth_dst up" ); +SYS ("ip addr add dev veth_src fc00::1/64" ); +SYS ("ip addr add dev veth_dst fc00::2/64" ); +SYS ("ip neigh add fc00::2 dev veth_src lladdr 66:77:88:99:aa:bb" ); + +/* We enable forwarding in the test namespace because that will cause +* the packets that go through the kernel stack (with XDP_PASS) to be +* forwarded back out the same interface (because of the packet dst +* combined with the interface addresses). When this happens, the +* regular forwarding path will end up going through the same +* veth_xdp_xmit() call as the XDP_REDIRECT code, which can cause a +* deadlock if it happens on the same CPU. There's a local_bh_disable() +* in the test_run code to prevent this, but an earlier version of the +* code didn't have this, so we keep the test behaviour to make sure the +* bug doesn't resurface. +*/ +SYS ("sysctl -qw net.ipv6.conf.all.forwarding=1" ); + + ifindex_src =if_nametoindex ("veth_src" ); + ifindex_dst =if_nametoindex ("veth_dst" ); +if (!ASSERT_NEQ (ifindex_src,0 ,"ifindex_src" ) || + !ASSERT_NEQ (ifindex_dst,0 ,"ifindex_dst" )) + goto out; + +memcpy (skel->rodata->expect_dst, &pkt_udp.eth.h_dest,ETH_ALEN ); + skel->rodata->ifindex_out = ifindex_src;/* redirect back to the same iface */ + skel->rodata->ifindex_in = ifindex_src; + ctx_in.ingress_ifindex = ifindex_src; + tc_hook.ifindex = ifindex_src; + +if (!ASSERT_OK (test_xdp_do_redirect__load (skel),"load" )) + goto out; + + link =bpf_program__attach_xdp (skel->progs.xdp_count_pkts, ifindex_dst); +if (!ASSERT_OK_PTR (link,"prog_attach" )) + goto out; + skel->links.xdp_count_pkts = link; + + tc_prog_fd =bpf_program__fd (skel->progs.tc_count_pkts); +if (attach_tc_prog (&tc_hook, tc_prog_fd)) + goto out; + + xdp_prog_fd =bpf_program__fd (skel->progs.xdp_redirect); + err =bpf_prog_test_run_opts (xdp_prog_fd, &opts); +if (!ASSERT_OK (err,"prog_run" )) + goto out_tc; + +/* wait for the packets to be flushed */ +kern_sync_rcu (); + +/* There will be one packet sent through XDP_REDIRECT and one through +* XDP_TX; these will show up on the XDP counting program, while the +* rest will be counted at the TC ingress hook (and the counting program +* resets the packet payload so they don't get counted twice even though +* they are re-xmited out the veth device +*/ +ASSERT_EQ (skel->bss->pkts_seen_xdp,2 ,"pkt_count_xdp" ); +ASSERT_EQ (skel->bss->pkts_seen_tc,NUM_PKTS -2 ,"pkt_count_tc" ); + +out_tc: +bpf_tc_hook_destroy (&tc_hook); +out: +if (nstoken) +close_netns (nstoken); +system ("ip netns del testns" ); +test_xdp_do_redirect__destroy (skel); +}
tools/testing/selftests/bpf/progs/test_xdp_do_redirect.c new file +85 -0
85 lines not on this page · raw message
Signature · 1 line
2.34.1
Re: [PATCH bpf-next v7 3/3] selftests/bpf: Add selftest for XDP_REDIRECT in bpf_prog_run()
Review
Quoted 2 lines · Toke Høiland-Jørgensen
(quoted diff) +2
-0+ +#define NUM_PKTS 1000000
It takes 7 seconds on my kvm with kasan and lockdep and will take much longer in BPF CI.
So it needs to be lower otherwise CI will struggle.
tools/testing/selftests/bpf/prog_tests/xdp_do_redirect.c line 99
Quoted 6 lines
tools/testing/selftests/bpf/prog_tests/xdp_do_redirect.c new file +6
-0+/* The XDP program we run with bpf_prog_run() will cycle through all +* three xmit (PASS/TX/REDIRECT) return codes starting from above, and +* ending up with PASS, so we should end up with two packets on the dst +* iface and NUM_PKTS-2 in the TC hook. We match the packets on the UDP +* payload. +*/
could you keep cycling through all return codes? That should make the test stronger.
tools/testing/selftests/bpf/prog_tests/xdp_do_redirect.c line 125
Quoted 13 lines
tools/testing/selftests/bpf/prog_tests/xdp_do_redirect.c new file +13
-0+ +/* We enable forwarding in the test namespace because that will cause +* the packets that go through the kernel stack (with XDP_PASS) to be +* forwarded back out the same interface (because of the packet dst +* combined with the interface addresses). When this happens, the +* regular forwarding path will end up going through the same +* veth_xdp_xmit() call as the XDP_REDIRECT code, which can cause a +* deadlock if it happens on the same CPU. There's a local_bh_disable() +* in the test_run code to prevent this, but an earlier version of the +* code didn't have this, so we keep the test behaviour to make sure the +* bug doesn't resurface. +*/ +SYS ("sysctl -qw net.ipv6.conf.all.forwarding=1" );
Does it mean that without forwarding=1 the kernel will dead lock ?!
Re: [PATCH bpf-next v7 3/3] selftests/bpf: Add selftest for XDP_REDIRECT in bpf_prog_run()
Quoted 7 lines · Alexei Starovoitov
Review
Quoted 2 lines · Toke Høiland-Jørgensen
(quoted diff) +2
-0+ +#define NUM_PKTS 1000000It takes 7 seconds on my kvm with kasan and lockdep and will take much longer in BPF CI.
So it needs to be lower otherwise CI will struggle.
OK, I'll lower it.
Quoted 9 lines
tools/testing/selftests/bpf/prog_tests/xdp_do_redirect.c line 99
Quoted 6 lines
tools/testing/selftests/bpf/prog_tests/xdp_do_redirect.c new file +6
-0+/* The XDP program we run with bpf_prog_run() will cycle through all +* three xmit (PASS/TX/REDIRECT) return codes starting from above, and +* ending up with PASS, so we should end up with two packets on the dst +* iface and NUM_PKTS-2 in the TC hook. We match the packets on the UDP +* payload. +*/ could you keep cycling through all return codes? That should make the test stronger.
Can do.
Quoted 14 lines
tools/testing/selftests/bpf/prog_tests/xdp_do_redirect.c line 125
Quoted 13 lines
tools/testing/selftests/bpf/prog_tests/xdp_do_redirect.c new file +13
-0+ +/* We enable forwarding in the test namespace because that will cause +* the packets that go through the kernel stack (with XDP_PASS) to be +* forwarded back out the same interface (because of the packet dst +* combined with the interface addresses). When this happens, the +* regular forwarding path will end up going through the same +* veth_xdp_xmit() call as the XDP_REDIRECT code, which can cause a +* deadlock if it happens on the same CPU. There's a local_bh_disable() +* in the test_run code to prevent this, but an earlier version of the +* code didn't have this, so we keep the test behaviour to make sure the +* bug doesn't resurface. +*/ +SYS ("sysctl -qw net.ipv6.conf.all.forwarding=1" );
Quoted 1 line
Does it mean that without forwarding=1 the kernel will dead lock ?!
No, the deadlock is referring to the lockdep warning you posted. Which I fixed by moving around the local_bh_disable(); that comment is just meant to explain why the forwarding sysctl is set (so that the code path is still exercised even though it's no longer faulty). Reading it again now I can see that this was not entirely clear, will try to improve the wording :)
-Toke