cgroup gets release after long time

hi,
Pavel reported an issue with bpf programs (attached to cgroup) not being released at the time when the cgroup is removed and are still visible in 'bpftool prog' list afterwards.

It seems like this is not bpf specific, because I was able to cut the bpf code from his example and still see delayed release of cgroup.

It happens only on cgroup2 fs (booted with systemd.unified_cgroup_hierarchy=1 kernel command line option), please check the attached program below and following scenario:

TERM 1

# gcc -o test test.c

TERM 2
# cd /sys/kernel/debug/tracing
# echo 1 > events/cgroup/cgroup_release/enable

TERM 1 -> create and remove cgroup1

# ./test group1
qemu-system-x86_64: terminating on signal 15 from pid 1775 (./test)

TERM 2
# cat trace_pipe
<nothing>

TERM 1 -> create and remove cgroup2

# ./test group2
qemu-system-x86_64: terminating on signal 15 from pid 1783 (./test)

TERM 2 - group1 being released
# cat trace_pipe
kworker/22:2-1135 [022] .... 2947.375526: cgroup_release: root=0 id=78 level=1 path=/group1

TERM 1 -> create and remove cgroup3

# ./test group3
qemu-system-x86_64: terminating on signal 15 from pid 1798 (./test)

TERM 2 - group2 being released
# cat trace_pipe
kworker/22:2-1135 [022] .... 2947.375526: cgroup_release: root=0 id=78 level=1 path=/group1 kworker/22:0-1787 [022] .... 2961.501261: cgroup_release: root=0 id=78 level=1 path=/group2

Looks like the previous cgroup release is triggered by creating another cgroup. If I don't do anything the cgroup is released (tracepoint shows) in about 90 seconds.

The cgroup_release tracepoint is triggered in css_release_work_fn, the same function where the cgroup_bpf_put is called, hence the delay in releasing of the bpf programs.

Is this expected or somehow configurable? It's confusing seeing all the bpf programs from removed cgroups being around. In Pavel's setup it's about 100 of them.

Note, I could reproduce this only with qemu-kvm being run in child process in the example below.

thoughts? thanks,
jirka

---
#include <fcntl.h>
#include <signal.h>
#include <stdio.h>
#include <string.h>
#include <sys/stat.h>
#include <sys/types.h>
#include <unistd.h>

#define CGROUP_PATH "/sys/fs/cgroup"

int
main(int argc, char **argv)

{
        pid_t pid = -1;
        char path[1024];
        int rc;

        pid = fork();

        if (pid == 0) {
                execl("/usr/bin/qemu-kvm",
                      "/usr/bin/qemu-kvm",
                      "-display", "none",
                      NULL);
                fprintf(stderr, "failed to start qemu process\n");
                _exit(-1);
        } else {
                int filefd = -1;
                char proc[1024];

                snprintf(path, 1024, "%s/%s", CGROUP_PATH, argv[1]);

                sleep(1);

                if (mkdir(path, 0755) < 0) {
                        fprintf(stderr, "failed to create cgroup '%s'\n", path);
                        return -1;
                }

                snprintf(proc, 1024, "%s/cgroup.procs", path);

                filefd = open(proc, O_WRONLY|O_TRUNC);
                if (filefd > 0) {
                        dprintf(filefd, "%u", pid);
                        close(filefd);
                }

                sleep(1);
        }

        if (pid > 0)
                kill(pid, SIGTERM);
        do {
                rc = rmdir(path);
        } while (rc != 0);

        return 0;
}

hi,
Pavel reported an issue with bpf programs (attached to cgroup) not being released at the time when the cgroup is removed and are still visible in 'bpftool prog' list afterwards.

It seems like this is not bpf specific, because I was able to cut the bpf code from his example and still see delayed release of cgroup.

It happens only on cgroup2 fs (booted with systemd.unified_cgroup_hierarchy=1 kernel command line option), please check the attached program below and following scenario:

TERM 1

# gcc -o test test.c

TERM 2
# cd /sys/kernel/debug/tracing
# echo 1 > events/cgroup/cgroup_release/enable

TERM 1 -> create and remove cgroup1

# ./test group1
qemu-system-x86_64: terminating on signal 15 from pid 1775 (./test)

TERM 2
# cat trace_pipe
<nothing>

TERM 1 -> create and remove cgroup2

# ./test group2
qemu-system-x86_64: terminating on signal 15 from pid 1783 (./test)

TERM 2 - group1 being released
# cat trace_pipe
kworker/22:2-1135 [022] .... 2947.375526: cgroup_release: root=0 id=78 level=1 path=/group1

TERM 1 -> create and remove cgroup3

# ./test group3
qemu-system-x86_64: terminating on signal 15 from pid 1798 (./test)

			TERM 2 - group2 being released
			# cat trace_pipe
			kworker/22:2-1135  [022] ....  2947.375526: cgroup_release: root=0 id=78 level=1 path=/group1
			kworker/22:0-1787  [022] ....  2961.501261: cgroup_release: root=0 id=78 level=1 path=/group2

Looks like the previous cgroup release is triggered by creating another cgroup. If I don't do anything the cgroup is released (tracepoint shows) in about 90 seconds.

The cgroup_release tracepoint is triggered in css_release_work_fn, the same function where the cgroup_bpf_put is called, hence the delay in releasing of the bpf programs.

Is this expected or somehow configurable? It's confusing seeing all the bpf programs from removed cgroups being around. In Pavel's setup it's about 100 of them.

Note, I could reproduce this only with qemu-kvm being run in child process in the example below.

thoughts? thanks,
jirka

---
#include <fcntl.h>
#include <signal.h>
#include <stdio.h>
#include <string.h>
#include <sys/stat.h>
#include <sys/types.h>
#include <unistd.h>

#define CGROUP_PATH "/sys/fs/cgroup"

int
main(int argc, char **argv)

{
	pid_t pid = -1;
	char path[1024];
	int rc;

	pid = fork();

	if (pid == 0) {
		execl("/usr/bin/qemu-kvm",
		      "/usr/bin/qemu-kvm",
		      "-display", "none",
		      NULL);
		fprintf(stderr, "failed to start qemu process\n");
		_exit(-1);
	} else {
		int filefd = -1;
		char proc[1024];

		snprintf(path, 1024, "%s/%s", CGROUP_PATH, argv[1]);

		sleep(1);

		if (mkdir(path, 0755) < 0) {
			fprintf(stderr, "failed to create cgroup '%s'\n", path);
			return -1;
		}

		snprintf(proc, 1024, "%s/cgroup.procs", path);

		filefd = open(proc, O_WRONLY|O_TRUNC);
		if (filefd > 0) {
			dprintf(filefd, "%u", pid);
			close(filefd);
		}

		sleep(1);
	}

	if (pid > 0)
		kill(pid, SIGTERM);
	do {
		rc = rmdir(path);
	} while (rc != 0);

	return 0;
}

Quoted 4 lines · Jiri Olsa

hi,
Pavel reported an issue with bpf programs (attached to cgroup) not being released at the time when the cgroup is removed and are still visible in 'bpftool prog' list afterwards.

Hi Jiri!

Can you, please, try the patch from
https://github.com/rgushchin/linux/commit/f77afa1952d81a1afa6c4872d342bf6721e148e2 ?

It should solve the problem, and I'm about to post it upstream.

Quoted 6 lines · Roman Gushchin

Quoted 4 lines · Jiri Olsa

hi,
Pavel reported an issue with bpf programs (attached to cgroup) not being released at the time when the cgroup is removed and are still visible in 'bpftool prog' list afterwards.

Quoted 6 lines

Hi Jiri!

Can you, please, try the patch from
https://github.com/rgushchin/linux/commit/f77afa1952d81a1afa6c4872d342bf6721e148e2 ?

It should solve the problem, and I'm about to post it upstream.

awesome, could you please cc me on the post?

Quoted 4 lines · Jiri Olsa

hi,
Pavel reported an issue with bpf programs (attached to cgroup) not being released at the time when the cgroup is removed and are still visible in 'bpftool prog' list afterwards.

Hi Jiri!

Can you, please, try the patch from
https://github.com/rgushchin/linux/commit/f77afa1952d81a1afa6c4872d342bf6721e148e2 ?

It should solve the problem, and I'm about to post it upstream.

Quoted 11 lines · Roman Gushchin

Quoted 4 lines · Jiri Olsa

hi,
Pavel reported an issue with bpf programs (attached to cgroup) not being released at the time when the cgroup is removed and are still visible in 'bpftool prog' list afterwards.

Hi Jiri!

Can you, please, try the patch from
https://github.com/rgushchin/linux/commit/f77afa1952d81a1afa6c4872d342bf6721e148e2 ?

Quoted 1 line

It should solve the problem, and I'm about to post it upstream.

Perfect, I'll give it a try with full libvirt setup as well.

Can we have this somehow detectable from user-space so libvirt can decide when to use BPF or not? I would like to avoid using BPF with libvirt if this issue is not fixed and we cannot simply workaround it as systemd automatically removes cgroups for us.

Quoted 12 lines · Pavel Hrdina

Quoted 11 lines · Roman Gushchin

Quoted 4 lines · Jiri Olsa

hi,
Pavel reported an issue with bpf programs (attached to cgroup) not being released at the time when the cgroup is removed and are still visible in 'bpftool prog' list afterwards.

Hi Jiri!

Can you, please, try the patch from
https://github.com/rgushchin/linux/commit/f77afa1952d81a1afa6c4872d342bf6721e148e2 ?

Quoted 8 lines

Quoted 1 line

It should solve the problem, and I'm about to post it upstream.

Perfect, I'll give it a try with full libvirt setup as well.

Can we have this somehow detectable from user-space so libvirt can decide when to use BPF or not? I would like to avoid using BPF with libvirt if this issue is not fixed and we cannot simply workaround it as systemd automatically removes cgroups for us.

Hm, I don't think there is a good way to detect it from userspace. At least I have no good ideas. Alexei? Daniel?

If you're interested in a particular stable version, we can probably treat it as a "fix", and backport.

Quoted 26 lines · Roman Gushchin

Quoted 12 lines · Pavel Hrdina

Quoted 11 lines · Roman Gushchin

Quoted 4 lines · Jiri Olsa

hi,
Pavel reported an issue with bpf programs (attached to cgroup) not being released at the time when the cgroup is removed and are still visible in 'bpftool prog' list afterwards.

Hi Jiri!

Can you, please, try the patch from
https://github.com/rgushchin/linux/commit/f77afa1952d81a1afa6c4872d342bf6721e148e2 ?

Quoted 8 lines

Quoted 1 line

It should solve the problem, and I'm about to post it upstream.

Perfect, I'll give it a try with full libvirt setup as well.

Can we have this somehow detectable from user-space so libvirt can decide when to use BPF or not? I would like to avoid using BPF with libvirt if this issue is not fixed and we cannot simply workaround it as systemd automatically removes cgroups for us.

Hm, I don't think there is a good way to detect it from userspace. At least I have no good ideas. Alexei? Daniel?

Quoted 2 lines

If you're interested in a particular stable version, we can probably treat it as a "fix", and backport.

right.
also user space workaround is trivial.
Just detach before rmdir.

Quoted 27 lines · Alexei Starovoitov

Quoted 26 lines · Roman Gushchin

Quoted 12 lines · Pavel Hrdina

Quoted 11 lines · Roman Gushchin

Quoted 4 lines · Jiri Olsa

hi,
Pavel reported an issue with bpf programs (attached to cgroup) not being released at the time when the cgroup is removed and are still visible in 'bpftool prog' list afterwards.

Hi Jiri!

Can you, please, try the patch from
https://github.com/rgushchin/linux/commit/f77afa1952d81a1afa6c4872d342bf6721e148e2 ?

Quoted 8 lines

Quoted 1 line

It should solve the problem, and I'm about to post it upstream.

Perfect, I'll give it a try with full libvirt setup as well.

Can we have this somehow detectable from user-space so libvirt can decide when to use BPF or not? I would like to avoid using BPF with libvirt if this issue is not fixed and we cannot simply workaround it as systemd automatically removes cgroups for us.

Hm, I don't think there is a good way to detect it from userspace. At least I have no good ideas. Alexei? Daniel?

Quoted 6 lines

Quoted 2 lines

If you're interested in a particular stable version, we can probably treat it as a "fix", and backport.

right.
also user space workaround is trivial.
Just detach before rmdir.

Well yes, it's trivial but not if you are using machined from systemd. Once libvirt kills QEMU process systemd automatically removes the cgroup so we don't have any chance to remove the BPF program.

Would it be too ugly to put something into
'/sys/kernel/cgroup/features'?

Quoted 39 lines · Pavel Hrdina

Quoted 27 lines · Alexei Starovoitov

Quoted 26 lines · Roman Gushchin

Quoted 12 lines · Pavel Hrdina

Quoted 11 lines · Roman Gushchin

Quoted 4 lines · Jiri Olsa

hi,
Pavel reported an issue with bpf programs (attached to cgroup) not being released at the time when the cgroup is removed and are still visible in 'bpftool prog' list afterwards.

Hi Jiri!

Can you, please, try the patch from
https://github.com/rgushchin/linux/commit/f77afa1952d81a1afa6c4872d342bf6721e148e2 ?

Quoted 8 lines

Quoted 1 line

It should solve the problem, and I'm about to post it upstream.

Perfect, I'll give it a try with full libvirt setup as well.

Can we have this somehow detectable from user-space so libvirt can decide when to use BPF or not? I would like to avoid using BPF with libvirt if this issue is not fixed and we cannot simply workaround it as systemd automatically removes cgroups for us.

Hm, I don't think there is a good way to detect it from userspace. At least I have no good ideas. Alexei? Daniel?

Quoted 6 lines

Quoted 2 lines

If you're interested in a particular stable version, we can probably treat it as a "fix", and backport.

right.
also user space workaround is trivial.
Just detach before rmdir.

Well yes, it's trivial but not if you are using machined from systemd. Once libvirt kills QEMU process systemd automatically removes the cgroup so we don't have any chance to remove the BPF program.

Quoted 2 lines

Would it be too ugly to put something into
'/sys/kernel/cgroup/features'?

I thought about it, but it seems that /sys/kernel/cgroup/features is also relatively new. So if we're not going to backport it (I mean auto-detaching), than we can simple look at the kernel version, right?

If we're going to backport it, the question is which stable version we're looking at.

In general, I don't see any reasons why cgroup/features can't be used.

Quoted 40 lines · Roman Gushchin

Quoted 39 lines · Pavel Hrdina

Quoted 27 lines · Alexei Starovoitov

Quoted 26 lines · Roman Gushchin

Quoted 12 lines · Pavel Hrdina

Quoted 11 lines · Roman Gushchin

Quoted 4 lines · Jiri Olsa

hi,
Pavel reported an issue with bpf programs (attached to cgroup) not being released at the time when the cgroup is removed and are still visible in 'bpftool prog' list afterwards.

Hi Jiri!

Can you, please, try the patch from
https://github.com/rgushchin/linux/commit/f77afa1952d81a1afa6c4872d342bf6721e148e2 ?

Quoted 8 lines

Quoted 1 line

It should solve the problem, and I'm about to post it upstream.

Perfect, I'll give it a try with full libvirt setup as well.

Can we have this somehow detectable from user-space so libvirt can decide when to use BPF or not? I would like to avoid using BPF with libvirt if this issue is not fixed and we cannot simply workaround it as systemd automatically removes cgroups for us.

Hm, I don't think there is a good way to detect it from userspace. At least I have no good ideas. Alexei? Daniel?

Quoted 6 lines

Quoted 2 lines

If you're interested in a particular stable version, we can probably treat it as a "fix", and backport.

right.
also user space workaround is trivial.
Just detach before rmdir.

Well yes, it's trivial but not if you are using machined from systemd. Once libvirt kills QEMU process systemd automatically removes the cgroup so we don't have any chance to remove the BPF program.

Quoted 6 lines

Quoted 2 lines

Would it be too ugly to put something into
'/sys/kernel/cgroup/features'?

I thought about it, but it seems that /sys/kernel/cgroup/features is also relatively new. So if we're not going to backport it (I mean auto-detaching), than we can simple look at the kernel version, right?

If you think only about upstream then the version check is in most cases good enough, but usually that's not the case and patches are backported to downstream distributions as well.

Yes, that file was introduced in kernel 4.15 so there are some limitations where the fix would be introspectable.

Quoted 3 lines

If we're going to backport it, the question is which stable version we're looking at.

Quoted 1 line

In general, I don't see any reasons why cgroup/features can't be used.

Perfect, in that case I would prefer if we could export it in cgroup/features as it will be easier for user-space to figure out whether it's safe to relay on proper cleanup behavior or not and it will make downstream distributions life easier.

Quoted 58 lines · Pavel Hrdina

Quoted 40 lines · Roman Gushchin

Quoted 39 lines · Pavel Hrdina

Quoted 27 lines · Alexei Starovoitov

Quoted 26 lines · Roman Gushchin

Quoted 12 lines · Pavel Hrdina

Quoted 11 lines · Roman Gushchin

Quoted 4 lines · Jiri Olsa

hi,
Pavel reported an issue with bpf programs (attached to cgroup) not being released at the time when the cgroup is removed and are still visible in 'bpftool prog' list afterwards.

Hi Jiri!

Can you, please, try the patch from
https://github.com/rgushchin/linux/commit/f77afa1952d81a1afa6c4872d342bf6721e148e2 ?

Quoted 8 lines

Quoted 1 line

It should solve the problem, and I'm about to post it upstream.

Perfect, I'll give it a try with full libvirt setup as well.

Can we have this somehow detectable from user-space so libvirt can decide when to use BPF or not? I would like to avoid using BPF with libvirt if this issue is not fixed and we cannot simply workaround it as systemd automatically removes cgroups for us.

Hm, I don't think there is a good way to detect it from userspace. At least I have no good ideas. Alexei? Daniel?

Quoted 6 lines

Quoted 2 lines

If you're interested in a particular stable version, we can probably treat it as a "fix", and backport.

right.
also user space workaround is trivial.
Just detach before rmdir.

Well yes, it's trivial but not if you are using machined from systemd. Once libvirt kills QEMU process systemd automatically removes the cgroup so we don't have any chance to remove the BPF program.

Quoted 6 lines

Quoted 2 lines

Would it be too ugly to put something into
'/sys/kernel/cgroup/features'?

I thought about it, but it seems that /sys/kernel/cgroup/features is also relatively new. So if we're not going to backport it (I mean auto-detaching), than we can simple look at the kernel version, right?

If you think only about upstream then the version check is in most cases good enough, but usually that's not the case and patches are backported to downstream distributions as well.

Yes, that file was introduced in kernel 4.15 so there are some limitations where the fix would be introspectable.

Quoted 3 lines

If we're going to backport it, the question is which stable version we're looking at.

Quoted 6 lines

Quoted 1 line

In general, I don't see any reasons why cgroup/features can't be used.

Perfect, in that case I would prefer if we could export it in cgroup/features as it will be easier for user-space to figure out whether it's safe to relay on proper cleanup behavior or not and it will make downstream distributions life easier.

Hello, Pavel!

Tejun noticed that cgroup features are supposed to match cgroupfs mount options, so it can't be used here. And this >= 4.15 limitation is also a significant constraint.