Skip to contentSkip to conversation

vmlinux.h overlap/conflict with network protocol definitions

PATCH Stephen Hemminger 3 participants 8 messages Apr 4, 2024 – Apr 5, 2024

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

I am fixing the use of TC BPF in DPDK to use libbpf and bpftool. Luca recommended using vmlinux.h to address possible build and CO:RE issues. But it won't work.

There are missing pieces such as definitions of IPV6 next header fields (in linux/ipv6.h) and TC actions.

Without major hack surgery, not possible to use vmlinux.h instead. Since vmlinux.h defines may things that overlap with other headers.

Using:

$ /usr/sbin/bpftool -V
bpftool v7.3.0
using libbpf v1.3
features:
$ uname -r
6.6.15-amd64

The change to BPF program to use vmlinux is:

drivers/net/tap/bpf/tap_rss.c +1 -6
index 888b3bdc24..79f4ee31a1 100644
@@ -2,12 +2,7 @@
  * Copyright 2017 Mellanox Technologies, Ltd
  */
 
-#include <linux/in.h>
-#include <linux/if_ether.h>
-#include <linux/ip.h>
-#include <linux/ipv6.h>
-#include <linux/pkt_cls.h>
-#include <linux/bpf.h>
+#include "vmlinux.h"
 
 #include <bpf/bpf_helpers.h>
 #include <bpf/bpf_endian.h>

Resulting build failure is:

~/DPDK/tap $ ninja -C build
ninja: Entering directory `build'
[1/33] Generating drivers/net/tap/bpf/tap_rss.bpf.o with a custom command
FAILED: drivers/net/tap/bpf/tap_rss.o
/usr/bin/clang -O2 -Wall -Wextra -DTAP_MAX_QUEUES=16 -target bpf -g -c -idirafter /usr/include -idirafter /usr/include/x86_64-linux-gnu ../drivers/net/tap/bpf/tap_rss.c -o drivers/net/tap/bpf/tap_rss.o ../drivers/net/tap/bpf/tap_rss.c:116:8: error: use of undeclared identifier 'IPPROTO_HOPOPTS'

                case IPPROTO_HOPOPTS:
                     ^
../drivers/net/tap/bpf/tap_rss.c:117:8: error: use of undeclared identifier 'IPPROTO_ROUTING'
                case IPPROTO_ROUTING:
                     ^
../drivers/net/tap/bpf/tap_rss.c:118:8: error: use of undeclared identifier 'IPPROTO_DSTOPTS'
                case IPPROTO_DSTOPTS:
                     ^
../drivers/net/tap/bpf/tap_rss.c:126:8: error: use of undeclared identifier 'IPPROTO_FRAGMENT'
                case IPPROTO_FRAGMENT:
                     ^
../drivers/net/tap/bpf/tap_rss.c:210:33: error: use of undeclared identifier 'ETH_P_IP'
        if (skb->protocol == bpf_htons(ETH_P_IP))
                                       ^

../drivers/net/tap/bpf/tap_rss.c:210:33: error: use of undeclared identifier 'ETH_P_IP' ../drivers/net/tap/bpf/tap_rss.c:210:33: error: use of undeclared identifier 'ETH_P_IP' ../drivers/net/tap/bpf/tap_rss.c:210:33: error: use of undeclared identifier 'ETH_P_IP' ../drivers/net/tap/bpf/tap_rss.c:212:38: error: use of undeclared identifier 'ETH_P_IPV6'

        else if (skb->protocol == bpf_htons(ETH_P_IPV6))
                                            ^

../drivers/net/tap/bpf/tap_rss.c:212:38: error: use of undeclared identifier 'ETH_P_IPV6' ../drivers/net/tap/bpf/tap_rss.c:212:38: error: use of undeclared identifier 'ETH_P_IPV6' ../drivers/net/tap/bpf/tap_rss.c:212:38: error: use of undeclared identifier 'ETH_P_IPV6' ../drivers/net/tap/bpf/tap_rss.c:248:10: error: use of undeclared identifier 'TC_ACT_OK'

                return TC_ACT_OK;
                       ^
../drivers/net/tap/bpf/tap_rss.c:252:10: error: use of undeclared identifier 'TC_ACT_OK'
                return TC_ACT_OK;
                       ^
../drivers/net/tap/bpf/tap_rss.c:256:9: error: use of undeclared identifier 'TC_ACT_PIPE'
        return TC_ACT_PIPE;
               ^

15 errors generated.
ninja: build stopped: subcommand failed.

reviewer
#1 lore raw
Quoted 82 lines · Stephen Hemminger

I am fixing the use of TC BPF in DPDK to use libbpf and bpftool. Luca recommended using vmlinux.h to address possible build and CO:RE issues. But it won't work.

There are missing pieces such as definitions of IPV6 next header fields (in linux/ipv6.h) and TC actions.

Without major hack surgery, not possible to use vmlinux.h instead. Since vmlinux.h defines may things that overlap with other headers.

Using:

$ /usr/sbin/bpftool -V
bpftool v7.3.0
using libbpf v1.3
features:
$ uname -r
6.6.15-amd64

The change to BPF program to use vmlinux is:

drivers/net/tap/bpf/tap_rss.c +1 -6
index 888b3bdc24..79f4ee31a1 100644
@@ -2,12 +2,7 @@
   * Copyright 2017 Mellanox Technologies, Ltd
   */
 
-#include <linux/in.h>
-#include <linux/if_ether.h>
-#include <linux/ip.h>
-#include <linux/ipv6.h>
-#include <linux/pkt_cls.h>
-#include <linux/bpf.h>
+#include "vmlinux.h"
 
  #include <bpf/bpf_helpers.h>
  #include <bpf/bpf_endian.h>

Resulting build failure is:

~/DPDK/tap $ ninja -C build
ninja: Entering directory `build'
[1/33] Generating drivers/net/tap/bpf/tap_rss.bpf.o with a custom command
FAILED: drivers/net/tap/bpf/tap_rss.o
/usr/bin/clang -O2 -Wall -Wextra -DTAP_MAX_QUEUES=16 -target bpf -g -c -idirafter /usr/include -idirafter /usr/include/x86_64-linux-gnu ../drivers/net/tap/bpf/tap_rss.c -o drivers/net/tap/bpf/tap_rss.o ../drivers/net/tap/bpf/tap_rss.c:116:8: error: use of undeclared identifier 'IPPROTO_HOPOPTS'

                 case IPPROTO_HOPOPTS:
                      ^
../drivers/net/tap/bpf/tap_rss.c:117:8: error: use of undeclared identifier 'IPPROTO_ROUTING'
                 case IPPROTO_ROUTING:
                      ^
../drivers/net/tap/bpf/tap_rss.c:118:8: error: use of undeclared identifier 'IPPROTO_DSTOPTS'
                 case IPPROTO_DSTOPTS:
                      ^
../drivers/net/tap/bpf/tap_rss.c:126:8: error: use of undeclared identifier 'IPPROTO_FRAGMENT'
                 case IPPROTO_FRAGMENT:
                      ^
../drivers/net/tap/bpf/tap_rss.c:210:33: error: use of undeclared identifier 'ETH_P_IP'
         if (skb->protocol == bpf_htons(ETH_P_IP))
                                        ^

../drivers/net/tap/bpf/tap_rss.c:210:33: error: use of undeclared identifier 'ETH_P_IP' ../drivers/net/tap/bpf/tap_rss.c:210:33: error: use of undeclared identifier 'ETH_P_IP' ../drivers/net/tap/bpf/tap_rss.c:210:33: error: use of undeclared identifier 'ETH_P_IP' ../drivers/net/tap/bpf/tap_rss.c:212:38: error: use of undeclared identifier 'ETH_P_IPV6'

         else if (skb->protocol == bpf_htons(ETH_P_IPV6))
                                             ^

../drivers/net/tap/bpf/tap_rss.c:212:38: error: use of undeclared identifier 'ETH_P_IPV6' ../drivers/net/tap/bpf/tap_rss.c:212:38: error: use of undeclared identifier 'ETH_P_IPV6' ../drivers/net/tap/bpf/tap_rss.c:212:38: error: use of undeclared identifier 'ETH_P_IPV6' ../drivers/net/tap/bpf/tap_rss.c:248:10: error: use of undeclared identifier 'TC_ACT_OK'

                 return TC_ACT_OK;
                        ^
../drivers/net/tap/bpf/tap_rss.c:252:10: error: use of undeclared identifier 'TC_ACT_OK'
                 return TC_ACT_OK;
                        ^
../drivers/net/tap/bpf/tap_rss.c:256:9: error: use of undeclared identifier 'TC_ACT_PIPE'
         return TC_ACT_PIPE;
                ^
Quoted 2 lines

15 errors generated.
ninja: build stopped: subcommand failed.

This is a known issue as currently vmlinux.h does not support macros. There are some efforts by Edward Zingerman to support this but this has not done yet. At the same time, you could have a trivial header file like https://github.com/torvalds/linux/blob/master/tools/testing/selftests/bpf/progs/bpf_tracing_net.h to be used for bpf program and then your bpf program with vmlinux.h can have much easier CORE support.

#2 lore raw
Quoted 86 lines · On Thu, 4 Apr 2024 11:16:32 -0700 Yonghong Song <[email protected]> wrote:
Quoted 82 lines · Stephen Hemminger

I am fixing the use of TC BPF in DPDK to use libbpf and bpftool. Luca recommended using vmlinux.h to address possible build and CO:RE issues. But it won't work.

There are missing pieces such as definitions of IPV6 next header fields (in linux/ipv6.h) and TC actions.

Without major hack surgery, not possible to use vmlinux.h instead. Since vmlinux.h defines may things that overlap with other headers.

Using:

$ /usr/sbin/bpftool -V
bpftool v7.3.0
using libbpf v1.3
features:
$ uname -r
6.6.15-amd64

The change to BPF program to use vmlinux is:

drivers/net/tap/bpf/tap_rss.c +1 -6
index 888b3bdc24..79f4ee31a1 100644
@@ -2,12 +2,7 @@
   * Copyright 2017 Mellanox Technologies, Ltd
   */
 
-#include <linux/in.h>
-#include <linux/if_ether.h>
-#include <linux/ip.h>
-#include <linux/ipv6.h>
-#include <linux/pkt_cls.h>
-#include <linux/bpf.h>
+#include "vmlinux.h"
 
  #include <bpf/bpf_helpers.h>
  #include <bpf/bpf_endian.h>

Resulting build failure is:

~/DPDK/tap $ ninja -C build
ninja: Entering directory `build'
[1/33] Generating drivers/net/tap/bpf/tap_rss.bpf.o with a custom command
FAILED: drivers/net/tap/bpf/tap_rss.o
/usr/bin/clang -O2 -Wall -Wextra -DTAP_MAX_QUEUES=16 -target bpf -g -c -idirafter /usr/include -idirafter /usr/include/x86_64-linux-gnu ../drivers/net/tap/bpf/tap_rss.c -o drivers/net/tap/bpf/tap_rss.o ../drivers/net/tap/bpf/tap_rss.c:116:8: error: use of undeclared identifier 'IPPROTO_HOPOPTS'

                 case IPPROTO_HOPOPTS:
                      ^
../drivers/net/tap/bpf/tap_rss.c:117:8: error: use of undeclared identifier 'IPPROTO_ROUTING'
                 case IPPROTO_ROUTING:
                      ^
../drivers/net/tap/bpf/tap_rss.c:118:8: error: use of undeclared identifier 'IPPROTO_DSTOPTS'
                 case IPPROTO_DSTOPTS:
                      ^
../drivers/net/tap/bpf/tap_rss.c:126:8: error: use of undeclared identifier 'IPPROTO_FRAGMENT'
                 case IPPROTO_FRAGMENT:
                      ^
../drivers/net/tap/bpf/tap_rss.c:210:33: error: use of undeclared identifier 'ETH_P_IP'
         if (skb->protocol == bpf_htons(ETH_P_IP))
                                        ^

../drivers/net/tap/bpf/tap_rss.c:210:33: error: use of undeclared identifier 'ETH_P_IP' ../drivers/net/tap/bpf/tap_rss.c:210:33: error: use of undeclared identifier 'ETH_P_IP' ../drivers/net/tap/bpf/tap_rss.c:210:33: error: use of undeclared identifier 'ETH_P_IP' ../drivers/net/tap/bpf/tap_rss.c:212:38: error: use of undeclared identifier 'ETH_P_IPV6'

         else if (skb->protocol == bpf_htons(ETH_P_IPV6))
                                             ^

../drivers/net/tap/bpf/tap_rss.c:212:38: error: use of undeclared identifier 'ETH_P_IPV6' ../drivers/net/tap/bpf/tap_rss.c:212:38: error: use of undeclared identifier 'ETH_P_IPV6' ../drivers/net/tap/bpf/tap_rss.c:212:38: error: use of undeclared identifier 'ETH_P_IPV6' ../drivers/net/tap/bpf/tap_rss.c:248:10: error: use of undeclared identifier 'TC_ACT_OK'

                 return TC_ACT_OK;
                        ^
../drivers/net/tap/bpf/tap_rss.c:252:10: error: use of undeclared identifier 'TC_ACT_OK'
                 return TC_ACT_OK;
                        ^
../drivers/net/tap/bpf/tap_rss.c:256:9: error: use of undeclared identifier 'TC_ACT_PIPE'
         return TC_ACT_PIPE;
                ^
Quoted 2 lines

15 errors generated.
ninja: build stopped: subcommand failed.

Quoted 7 lines

This is a known issue as currently vmlinux.h does not support macros. There are some efforts by Edward Zingerman to support this but this has not done yet. At the same time, you could have a trivial header file like
https://github.com/torvalds/linux/blob/master/tools/testing/selftests/bpf/progs/bpf_tracing_net.h
to be used for bpf program and then your bpf program with vmlinux.h can have much easier CORE support.

That is an example of header surgery which I would rather avoid having to carry as long term technical debt baggage.

#3 lore raw
Quoted 16 lines · Stephen Hemminger
Quoted 12 lines · On Thu, 4 Apr 2024 11:16:32 -0700 Yonghong Song <[email protected]> wrote:
Quoted 11 lines · Stephen Hemminger

I am fixing the use of TC BPF in DPDK to use libbpf and bpftool. Luca recommended using vmlinux.h to address possible build and CO:RE issues. But it won't work.

There are missing pieces such as definitions of IPV6 next header fields (in linux/ipv6.h) and TC actions.

Without major hack surgery, not possible to use vmlinux.h instead. Since vmlinux.h defines may things that overlap with other headers.

Using:

[...]

Quoted 9 lines
Quoted 7 lines

This is a known issue as currently vmlinux.h does not support macros. There are some efforts by Edward Zingerman to support this but this has not done yet. At the same time, you could have a trivial header file like
https://github.com/torvalds/linux/blob/master/tools/testing/selftests/bpf/progs/bpf_tracing_net.h
to be used for bpf program and then your bpf program with vmlinux.h can have much easier CORE support.

Quoted 3 lines

That is an example of header surgery which I would rather avoid having to carry as long term technical debt baggage.

What's your ultimate goal? As Yonghong said, vmlinux.h is not compatible with other headers. So you have to pick either using vmlinux.h as a base + adding missing #define's (because those are not recorded in types, so can't be put into vmlinux.h), or not use vmlinux.h, use linux UAPI/internal headers and then use explicit CO-RE helpers/attributes to make your application CO-RE-relocatable.

It's not clear from your original email why exactly you wanted to switch to vmlinux.h in the first place.

#4 lore raw
Quoted 20 lines · On Thu, 4 Apr 2024 14:45:04 -0700 Andrii Nakryiko <[email protected]>…
Quoted 9 lines
Quoted 7 lines

This is a known issue as currently vmlinux.h does not support macros. There are some efforts by Edward Zingerman to support this but this has not done yet. At the same time, you could have a trivial header file like
https://github.com/torvalds/linux/blob/master/tools/testing/selftests/bpf/progs/bpf_tracing_net.h
to be used for bpf program and then your bpf program with vmlinux.h can have much easier CORE support.

Quoted 3 lines

That is an example of header surgery which I would rather avoid having to carry as long term technical debt baggage.

What's your ultimate goal? As Yonghong said, vmlinux.h is not compatible with other headers. So you have to pick either using vmlinux.h as a base + adding missing #define's (because those are not recorded in types, so can't be put into vmlinux.h), or not use vmlinux.h, use linux UAPI/internal headers and then use explicit CO-RE helpers/attributes to make your application CO-RE-relocatable.

Quoted 2 lines

It's not clear from your original email why exactly you wanted to switch to vmlinux.h in the first place.

Some backstory. There is not an existing TC filter for this, so the original developer had the idea of using BPF to do it.

The program is a small BPF program to implement a TC filter that looks at SKB and does mapping to queue based on L3 (or L3/L4) header. So not heavily dependent on kernel data structure, but sk_buff is not necessarily stable; actual layout depends on kernel config.

But the evolution of BPF has made the old code unusable. I am trying to get it working again, and cleanup to modern BPF by using BPF skeleton code.

Luca was one who had the suggestion about vmlinux.

Quoted 27 lines

Using bpftool to generate the header at build time is a bit icky, because it will look at sysfs on the build system, which is from the running kernel. But a build system's kernel might be some ancient LTS, and even be a completely different kconfig/build/distro from the actual runtime one.

We have ran in the same problem in systemd recently, and the solution is to have distros publish the vmlinux.h together with the kernel image/headers, that way we can rely on the fact that by build-depending on the right kernel package we get exactly the generated vmlinux.h that we want. This has already happened in Centos, Debian, Fedora and Arch, and I am trying to get Ubuntu onboard too.

The annoying thing is that every distro packages differently, so the path needs to be configurable with a meson option.

Feel free to pilfer the systemd meson glue:

https://github.com/systemd/systemd/pull/26826/commits/d917079e7e320aa281fc4ad6f8073b0814b9cb13

It's of course file to go to the runtime kernel if no vmlinux.h is specified, as a fallback, which is going to be useful for developers machines.

Signature · 2 lines
Kind regards,
Luca Boccassi
#5 lore raw
Quoted 27 lines · Stephen Hemminger
Quoted 20 lines · On Thu, 4 Apr 2024 14:45:04 -0700 Andrii Nakryiko <[email protected]>…
Quoted 9 lines
Quoted 7 lines

This is a known issue as currently vmlinux.h does not support macros. There are some efforts by Edward Zingerman to support this but this has not done yet. At the same time, you could have a trivial header file like
https://github.com/torvalds/linux/blob/master/tools/testing/selftests/bpf/progs/bpf_tracing_net.h
to be used for bpf program and then your bpf program with vmlinux.h can have much easier CORE support.

Quoted 3 lines

That is an example of header surgery which I would rather avoid having to carry as long term technical debt baggage.

What's your ultimate goal? As Yonghong said, vmlinux.h is not compatible with other headers. So you have to pick either using vmlinux.h as a base + adding missing #define's (because those are not recorded in types, so can't be put into vmlinux.h), or not use vmlinux.h, use linux UAPI/internal headers and then use explicit CO-RE helpers/attributes to make your application CO-RE-relocatable.

Quoted 2 lines

It's not clear from your original email why exactly you wanted to switch to vmlinux.h in the first place.

Quoted 8 lines

Some backstory. There is not an existing TC filter for this, so the original developer had the idea of using BPF to do it.

The program is a small BPF program to implement a TC filter that looks at SKB and does mapping to queue based on L3 (or L3/L4) header. So not heavily dependent on kernel data structure, but sk_buff is not necessarily stable; actual layout depends on kernel config.

If it's only a few fields from sk_buff that you need, you can define your own minimal sk_buff definition with __attribute__((preserve_access_index)) added to it. You don't have to declare fields in the right order, just make sure that field types match. E.g., something like:

struct sk_buff {
unsigned char *head;
unsigned char *data;
struct sock *sock;
} __attribute__((preserve_access_index));

You can call it struct sk_buff___mine if you already include sk_buff, to avoid the conflict. Libbpf will still understand that it should match it to struct sk_buff in the kernel.

Alternatively, you can just use BPF_CORE_READ() macro on types you get from kernel headers, even if they don't have that preserve_access_index attribute.

Or, you can just use __builtin_preserve_access_index(&skb->len) to access sk_buff's fields in CO-RE-relocatable way.

Or, you can have entire block within which all fields accesses will be CO-RE-relocatable:

int len;

__builtin_preserve_access_index(({
    len = skb->len;
    /* other skb accesses here as well */
}));

Many ways to have bolted on CO-RE even without vmlinux.h. vmlinux.h is convenient (apart from lack of #defines, but that's an orthogonal problem), but by no means required for BPF CO-RE.

Quoted 3 lines

But the evolution of BPF has made the old code unusable. I am trying to get it working again, and cleanup to modern BPF by using BPF skeleton code.

That's a good idea independently from vmlinux.h, yep.

Quoted 32 lines

Luca was one who had the suggestion about vmlinux.

Quoted 27 lines

Using bpftool to generate the header at build time is a bit icky, because it will look at sysfs on the build system, which is from the running kernel. But a build system's kernel might be some ancient LTS, and even be a completely different kconfig/build/distro from the actual runtime one.

We have ran in the same problem in systemd recently, and the solution is to have distros publish the vmlinux.h together with the kernel image/headers, that way we can rely on the fact that by build-depending on the right kernel package we get exactly the generated vmlinux.h that we want. This has already happened in Centos, Debian, Fedora and Arch, and I am trying to get Ubuntu onboard too.

The annoying thing is that every distro packages differently, so the path needs to be configurable with a meson option.

Feel free to pilfer the systemd meson glue:

https://github.com/systemd/systemd/pull/26826/commits/d917079e7e320aa281fc4ad6f8073b0814b9cb13

It's of course file to go to the runtime kernel if no vmlinux.h is specified, as a fallback, which is going to be useful for developers machines.

--
Kind regards,
Luca Boccassi

#6 lore raw
Quoted 66 lines · On Fri, 5 Apr 2024 10:31:46 -0700 Andrii Nakryiko <[email protected]>…
Quoted 27 lines · Stephen Hemminger
Quoted 20 lines · On Thu, 4 Apr 2024 14:45:04 -0700 Andrii Nakryiko <[email protected]>…
Quoted 9 lines
Quoted 7 lines

This is a known issue as currently vmlinux.h does not support macros. There are some efforts by Edward Zingerman to support this but this has not done yet. At the same time, you could have a trivial header file like
https://github.com/torvalds/linux/blob/master/tools/testing/selftests/bpf/progs/bpf_tracing_net.h
to be used for bpf program and then your bpf program with vmlinux.h can have much easier CORE support.

Quoted 3 lines

That is an example of header surgery which I would rather avoid having to carry as long term technical debt baggage.

What's your ultimate goal? As Yonghong said, vmlinux.h is not compatible with other headers. So you have to pick either using vmlinux.h as a base + adding missing #define's (because those are not recorded in types, so can't be put into vmlinux.h), or not use vmlinux.h, use linux UAPI/internal headers and then use explicit CO-RE helpers/attributes to make your application CO-RE-relocatable.

Quoted 2 lines

It's not clear from your original email why exactly you wanted to switch to vmlinux.h in the first place.

Quoted 8 lines

Some backstory. There is not an existing TC filter for this, so the original developer had the idea of using BPF to do it.

The program is a small BPF program to implement a TC filter that looks at SKB and does mapping to queue based on L3 (or L3/L4) header. So not heavily dependent on kernel data structure, but sk_buff is not necessarily stable; actual layout depends on kernel config.

If it's only a few fields from sk_buff that you need, you can define your own minimal sk_buff definition with __attribute__((preserve_access_index)) added to it. You don't have to declare fields in the right order, just make sure that field types match. E.g., something like:

struct sk_buff {
unsigned char *head;
unsigned char *data;
struct sock *sock;
} __attribute__((preserve_access_index));

You can call it struct sk_buff___mine if you already include sk_buff, to avoid the conflict. Libbpf will still understand that it should match it to struct sk_buff in the kernel.

Alternatively, you can just use BPF_CORE_READ() macro on types you get from kernel headers, even if they don't have that preserve_access_index attribute.

Or, you can just use __builtin_preserve_access_index(&skb->len) to access sk_buff's fields in CO-RE-relocatable way.

Or, you can have entire block within which all fields accesses will be CO-RE-relocatable:

int len;

Quoted 9 lines
__builtin_preserve_access_index(({
    len = skb->len;
    /* other skb accesses here as well */
}));

Many ways to have bolted on CO-RE even without vmlinux.h. vmlinux.h is convenient (apart from lack of #defines, but that's an orthogonal problem), but by no means required for BPF CO-RE.

The skbuff pointer is passed to the TC program. So BPF_CORE_READ doesn't make sense here. The biggest concern is that kernel config of the build machine may not match the kernel config of the eventual target, and some of the fields of the sk_buff are hidden based on #ifdef and may change.

#7 lore raw
Quoted 81 lines · Stephen Hemminger
Quoted 66 lines · On Fri, 5 Apr 2024 10:31:46 -0700 Andrii Nakryiko <[email protected]>…
Quoted 27 lines · Stephen Hemminger
Quoted 20 lines · On Thu, 4 Apr 2024 14:45:04 -0700 Andrii Nakryiko <[email protected]>…
Quoted 9 lines
Quoted 7 lines

This is a known issue as currently vmlinux.h does not support macros. There are some efforts by Edward Zingerman to support this but this has not done yet. At the same time, you could have a trivial header file like
https://github.com/torvalds/linux/blob/master/tools/testing/selftests/bpf/progs/bpf_tracing_net.h
to be used for bpf program and then your bpf program with vmlinux.h can have much easier CORE support.

Quoted 3 lines

That is an example of header surgery which I would rather avoid having to carry as long term technical debt baggage.

What's your ultimate goal? As Yonghong said, vmlinux.h is not compatible with other headers. So you have to pick either using vmlinux.h as a base + adding missing #define's (because those are not recorded in types, so can't be put into vmlinux.h), or not use vmlinux.h, use linux UAPI/internal headers and then use explicit CO-RE helpers/attributes to make your application CO-RE-relocatable.

Quoted 2 lines

It's not clear from your original email why exactly you wanted to switch to vmlinux.h in the first place.

Quoted 8 lines

Some backstory. There is not an existing TC filter for this, so the original developer had the idea of using BPF to do it.

The program is a small BPF program to implement a TC filter that looks at SKB and does mapping to queue based on L3 (or L3/L4) header. So not heavily dependent on kernel data structure, but sk_buff is not necessarily stable; actual layout depends on kernel config.

If it's only a few fields from sk_buff that you need, you can define your own minimal sk_buff definition with __attribute__((preserve_access_index)) added to it. You don't have to declare fields in the right order, just make sure that field types match. E.g., something like:

struct sk_buff {
unsigned char *head;
unsigned char *data;
struct sock *sock;
} __attribute__((preserve_access_index));

You can call it struct sk_buff___mine if you already include sk_buff, to avoid the conflict. Libbpf will still understand that it should match it to struct sk_buff in the kernel.

Alternatively, you can just use BPF_CORE_READ() macro on types you get from kernel headers, even if they don't have that preserve_access_index attribute.

Or, you can just use __builtin_preserve_access_index(&skb->len) to access sk_buff's fields in CO-RE-relocatable way.

Or, you can have entire block within which all fields accesses will be CO-RE-relocatable:

int len;

Quoted 9 lines
__builtin_preserve_access_index(({
    len = skb->len;
    /* other skb accesses here as well */
}));

Many ways to have bolted on CO-RE even without vmlinux.h. vmlinux.h is convenient (apart from lack of #defines, but that's an orthogonal problem), but by no means required for BPF CO-RE.

Quoted 4 lines

The skbuff pointer is passed to the TC program. So BPF_CORE_READ doesn't make sense here. The biggest concern is that kernel config of the build machine may not match the kernel config of the eventual target, and some of the fields of the sk_buff are hidden based on #ifdef and may change.

And everything else I wrote?..

Regardless, I think at this point if you want to get help, you probably should provide the source code.