Skip to contentSkip to conversation

BPF CI for Stable

DISCUSSION Shung-Hsi Yu 2 participants 4 messages Jun 5 – Jun 12 BPF

lore
Share & tools
lore
#0 lore raw

Hi,

Following up on the LSF/MM/BPF 2026 session "BPF in Stable Kernels, an Update"[1]. Since the event, 6.1 coverage has also been added[2] (thanks to Paul). With a collaborator on my side, it seem like a good time to ask:

Is it possible to get BPF CI for stable in its own repository under a GitHub organization (e.g. libbpf/stable-bpf-ci)?

Currently it lives as a personal repo that is fork of libbpf/libbpf[3]. As someone (I think it was Andrii) pointed out during the session, the vast files currently in my fork are not needed at all. The GitHub Action workflow only needs the .github/, plus the supplement files under ci/.

Technically BPF CI for stable can also simply be merged back to libbpf/libbpf, but that felt less ideal because:
- kbuilder-debian container image currently doesn't work when testing stable kernels (I haven't look at reason of failure, might not be hard to fix)
- actions that tests stable kernel will be mixed with actions targeting libbpf itself
- BPF CI for stable will have to contain version-specific tweaks (e.g. [4])

None of the above are fatal flaws that prevent libbf/libbpf to host files that run BPF selftests on stable kernel, so I'd say having a standalone repository is more of a personal preference.

OTOH we can also try kernel-patches/bpf-like approach and have a kernel-patches/stable, providing proper, fully-fledged BPF CI for stable that tests incoming patchset to stable (not sure if Daniel was asking about this during the session). But I find this to be much more work. The current approach of applying queued patches from stable-queue[5], while comes with quite some delay between the moment patch is proposed to stable mailing vs the moment said patch gets tested, seem like a workable compromise.

Thanks,
Shung-Hsi

1: https://speakerdeck.com/shunghsiyu/bpf-in-stable-kernels-an-update
2: https://github.com/shunghsiyu/libbpf/pull/5
3: https://github.com/shunghsiyu/libbpf/
4: https://github.com/shunghsiyu/libbpf/tree/master/ci/diffs/6.1
5: https://git.kernel.org/pub/scm/linux/kernel/git/stable/stable-queue.git/

reviewer
#1 lore raw
Quoted 9 lines · Shung-Hsi Yu

Hi,

Following up on the LSF/MM/BPF 2026 session "BPF in Stable Kernels, an Update"[1]. Since the event, 6.1 coverage has also been added[2] (thanks to Paul). With a collaborator on my side, it seem like a good time to ask:

Is it possible to get BPF CI for stable in its own repository under a GitHub organization (e.g. libbpf/stable-bpf-ci)?

Hi Shung-Hsi,

I think a better home for the BPF CI targeting stable kernels is kernel-patches org [1] for a couple of reasons. First, repos in the kernel-patches org have access to hardware - generic runners provided by Meta and s390x runners provided by IBM. Second reason is that this org hosts source of truth repos for most of BPF CI code, even though a big chunk of it lives in libbpf/ci.

Let me know if this makes sense, and I'll create a new repository and grant you write permissions there.

Quoted 8 lines

Currently it lives as a personal repo that is fork of libbpf/libbpf[3]. As someone (I think it was Andrii) pointed out during the session, the vast files currently in my fork are not needed at all. The GitHub Action workflow only needs the .github/, plus the supplement files under ci/.

Technically BPF CI for stable can also simply be merged back to libbpf/libbpf

No, that's a bad idea.

Quoted 4 lines

, but that felt less ideal because:
- kbuilder-debian container image currently doesn't work when testing stable kernels (I haven't look at reason of failure, might not be hard to fix)

Technically, it's not a hard requirement to use the container. It makes the build jobs a little faster and a little more reliable, but it was crafted for bpf-next testing, which may not be ideal for stable. If necessary you could develop a separate "kbuilder-stable" container, but it's up to you to decide whether it's needed.

Quoted 4 lines

- actions that tests stable kernel will be mixed with actions targeting libbpf itself
- BPF CI for stable will have to contain version-specific tweaks (e.g. [4])

This is fine and expected. It is of course great to be able to reuse code, but practically speaking, I think in this case it may be more work to keep the common parts compatible with both stable and bpf-next workflows, than maintaining two diverged CI trees.

The rule of thumb should be this: if you can use a libbpf/ci action, do it. If there is something very specific to the workflow, keep it in place. Don't be shy copy-pasting yaml fragments that work for you, it's fine.

Quoted 4 lines

None of the above are fatal flaws that prevent libbf/libbpf to host files that run BPF selftests on stable kernel, so I'd say having a standalone repository is more of a personal preference.

I agree that a standalone repository is the way to go.

Quoted 5 lines

OTOH we can also try kernel-patches/bpf-like approach and have a kernel-patches/stable, providing proper, fully-fledged BPF CI for stable that tests incoming patchset to stable (not sure if Daniel was asking about this during the session). But I find this to be much more work.

We don't have to do this work right away (or ever).

First let's set up the kernel-patches/stable (or whatever the name) with the implementation you already have.

We can enhance and improve it when there are cycles and willingness to do so in the future,

Quoted 4 lines

The current approach of applying queued patches from stable-queue[5], while comes with quite some delay between the moment patch is proposed to stable mailing vs the moment said patch gets tested, seem like a workable compromise.

This can be automated by the way.
Check out linux-next sync on current BPF CI, it's straightforward [2]. There is a job periodically syncing a branch, and the CI pipeline is triggered on push.

[1] https://github.com/kernel-patches/
[2] https://github.com/kernel-patches/vmtest/blob/master/.github/workflows/linux-next-sync.yml

Quoted 10 lines

Thanks,
Shung-Hsi

1: https://speakerdeck.com/shunghsiyu/bpf-in-stable-kernels-an-update
2: https://github.com/shunghsiyu/libbpf/pull/5
3: https://github.com/shunghsiyu/libbpf/
4: https://github.com/shunghsiyu/libbpf/tree/master/ci/diffs/6.1
5: https://git.kernel.org/pub/scm/linux/kernel/git/stable/stable-queue.git/

#2 lore raw
Quoted 1 line · Ihor Solodrai

On 6/5/26 12:45 AM, Shung-Hsi Yu wrote:

[...]

Quoted 12 lines
Quoted 2 lines

Is it possible to get BPF CI for stable in its own repository under a GitHub organization (e.g. libbpf/stable-bpf-ci)?

Hi Shung-Hsi,

I think a better home for the BPF CI targeting stable kernels is kernel-patches org [1] for a couple of reasons. First, repos in the kernel-patches org have access to hardware - generic runners provided by Meta and s390x runners provided by IBM. Second reason is that this org hosts source of truth repos for most of BPF CI code, even though a big chunk of it lives in libbpf/ci.

Quoted 2 lines

Let me know if this makes sense, and I'll create a new repository and grant you write permissions there.

Make sense, let's have it in kernel-patches org.

[...]

Quoted 10 lines
Quoted 2 lines

Technically BPF CI for stable can also simply be merged back to libbpf/libbpf

No, that's a bad idea.

Quoted 4 lines

, but that felt less ideal because:
- kbuilder-debian container image currently doesn't work when testing stable kernels (I haven't look at reason of failure, might not be hard to fix)

Quoted 5 lines

Technically, it's not a hard requirement to use the container. It makes the build jobs a little faster and a little more reliable, but it was crafted for bpf-next testing, which may not be ideal for stable. If necessary you could develop a separate "kbuilder-stable" container, but it's up to you to decide whether it's needed.

Understood, I'll stick with Ubuntu 24.04.

Quoted 13 lines
Quoted 4 lines

- actions that tests stable kernel will be mixed with actions targeting libbpf itself
- BPF CI for stable will have to contain version-specific tweaks (e.g. [4])

This is fine and expected. It is of course great to be able to reuse code, but practically speaking, I think in this case it may be more work to keep the common parts compatible with both stable and bpf-next workflows, than maintaining two diverged CI trees.

The rule of thumb should be this: if you can use a libbpf/ci action, do it. If there is something very specific to the workflow, keep it in place. Don't be shy copy-pasting yaml fragments that work for you, it's fine.

[...]

Quoted 5 lines
Quoted 4 lines

OTOH we can also try kernel-patches/bpf-like approach and have a kernel-patches/stable, providing proper, fully-fledged BPF CI for stable that tests incoming patchset to stable (not sure if Daniel was asking about this during the session). But I find this to be much more work.

Quoted 7 lines

We don't have to do this work right away (or ever).

First let's set up the kernel-patches/stable (or whatever the name) with the implementation you already have.

We can enhance and improve it when there are cycles and willingness to do so in the future,

Sounds good to me, let's go with kernel-patches/stable. Thanks!

Quoted 9 lines
Quoted 4 lines

The current approach of applying queued patches from stable-queue[5], while comes with quite some delay between the moment patch is proposed to stable mailing vs the moment said patch gets tested, seem like a workable compromise.

This can be automated by the way.
Check out linux-next sync on current BPF CI, it's straightforward [2]. There is a job periodically syncing a branch, and the CI pipeline is triggered on push.

I'll looking this, thanks for the pointer.

Shung-Hsi

Quoted 2 lines

[1] https://github.com/kernel-patches/
[2] https://github.com/kernel-patches/vmtest/blob/master/.github/workflows/linux-next-sync.yml

[...]

reviewer
#3 lore raw
Quoted 15 lines · Shung-Hsi Yu
Quoted 1 line · Ihor Solodrai

On 6/5/26 12:45 AM, Shung-Hsi Yu wrote:

[...]

Quoted 12 lines
Quoted 2 lines

Is it possible to get BPF CI for stable in its own repository under a GitHub organization (e.g. libbpf/stable-bpf-ci)?

Hi Shung-Hsi,

I think a better home for the BPF CI targeting stable kernels is kernel-patches org [1] for a couple of reasons. First, repos in the kernel-patches org have access to hardware - generic runners provided by Meta and s390x runners provided by IBM. Second reason is that this org hosts source of truth repos for most of BPF CI code, even though a big chunk of it lives in libbpf/ci.

Quoted 4 lines
Quoted 2 lines

Let me know if this makes sense, and I'll create a new repository and grant you write permissions there.

Make sense, let's have it in kernel-patches org.

Here you go: https://github.com/kernel-patches/linux-stable

I've sent you an invite.

I vibe-code basic sync with upstream. README describes how it works, but I'll reiterate here for visibility on the list.

The repository has a main branch with only the CI code, test branches for each tracked stable version, and Linux mainline mirror.

There is a sync job that runs daily. It:
* syncs torvalds/master
* then for each listed stable version
* pulls in the tree
* applies the patches from the queue repo
* overlays .github/ and ci/ from the main branch, adding a commit
* this is similar to what KPD does for bpf-next, but without patchwork and KPD
* pushes the result to linux-X.Y.y branch

The testing workflow (test.yml), which currently doesn't do anything interesting, is triggered on push to linux-* branches.

The scripts are straightforward, though a little verbose as is typical for vibe code. Feel free to edit it as you see fit, or even scrap it. I wanted to have something working before handing over the repo to you.

This way we have an automated sync + trigger running on schedule right out of the box. With that, you can add code to test.yml to actually build and run tests.

I think you can copy most of the stuff from your libbpf fork, but it'll have to be tested and debugged of course.

And the best part is you can make use of the hardware available to the kernel-patches org: beefy bare-metal machines for testing, s390x runners, codebuild. Oh, and AI reviews of course, if you want to experiment with that.

My understanding is you'll be maintaining the stable part of the CI, but feel free to reach out if you need help.

Have fun!

Quoted 66 lines

[...]

Quoted 10 lines
Quoted 2 lines

Technically BPF CI for stable can also simply be merged back to libbpf/libbpf

No, that's a bad idea.

Quoted 4 lines

, but that felt less ideal because:
- kbuilder-debian container image currently doesn't work when testing stable kernels (I haven't look at reason of failure, might not be hard to fix)

Quoted 5 lines

Technically, it's not a hard requirement to use the container. It makes the build jobs a little faster and a little more reliable, but it was crafted for bpf-next testing, which may not be ideal for stable. If necessary you could develop a separate "kbuilder-stable" container, but it's up to you to decide whether it's needed.

Understood, I'll stick with Ubuntu 24.04.

Quoted 13 lines
Quoted 4 lines

- actions that tests stable kernel will be mixed with actions targeting libbpf itself
- BPF CI for stable will have to contain version-specific tweaks (e.g. [4])

This is fine and expected. It is of course great to be able to reuse code, but practically speaking, I think in this case it may be more work to keep the common parts compatible with both stable and bpf-next workflows, than maintaining two diverged CI trees.

The rule of thumb should be this: if you can use a libbpf/ci action, do it. If there is something very specific to the workflow, keep it in place. Don't be shy copy-pasting yaml fragments that work for you, it's fine.

[...]

Quoted 5 lines
Quoted 4 lines

OTOH we can also try kernel-patches/bpf-like approach and have a kernel-patches/stable, providing proper, fully-fledged BPF CI for stable that tests incoming patchset to stable (not sure if Daniel was asking about this during the session). But I find this to be much more work.

Quoted 7 lines

We don't have to do this work right away (or ever).

First let's set up the kernel-patches/stable (or whatever the name) with the implementation you already have.

We can enhance and improve it when there are cycles and willingness to do so in the future,

Sounds good to me, let's go with kernel-patches/stable. Thanks!

Quoted 9 lines
Quoted 4 lines

The current approach of applying queued patches from stable-queue[5], while comes with quite some delay between the moment patch is proposed to stable mailing vs the moment said patch gets tested, seem like a workable compromise.

This can be automated by the way.
Check out linux-next sync on current BPF CI, it's straightforward [2]. There is a job periodically syncing a branch, and the CI pipeline is triggered on push.

I'll looking this, thanks for the pointer.

Shung-Hsi

Quoted 2 lines

[1] https://github.com/kernel-patches/
[2] https://github.com/kernel-patches/vmtest/blob/master/.github/workflows/linux-next-sync.yml

[...]