cgroups commit 61d1d219c4 breaks systemd

PATCH In review Jiri Kosina 4 participants 6 messages Mar 28, 2012 – Mar 30, 2012 cgroup

lore

Share & tools

lore

Jiri Kosina

author

2012-03-28 22:01 UTC #0 lore raw

Hi,

I am not able to boot current Linus' tree on system which is using systemd. In the early boot phase I get

Failed to create root cgroup hierarchy: No such process Failed to allocate manager object: No such process

and that's it, the system is completely unusable afterwards (not having any console spawned, no networking, etc).

Bisection pointed to

	commit 61d1d219c4c0761059236a46867bc49943c4d29d
	Author: Mandeep Singh Baines <[email protected]>
	Date:   Mon Jan 30 12:51:56 2012 -0800

	    cgroup: remove extra calls to find_existing_css_set

and I have verified that reverting this single commit on top of current Linus' tree makes the problem go away and the system boots properly.

strace output shows that with 61d1d219c applied, systemd is failing while trying to add itself to the systemd cgroup:

mkdir("/sys", 0755)                     = -1 EEXIST (File exists)
mkdir("/sys/fs", 0755)                  = -1 EEXIST (File exists)
mkdir("/sys/fs/cgroup", 0755)           = -1 EEXIST (File exists)
mkdir("/sys/fs/cgroup/systemd", 0755)   = -1 EEXIST (File exists)

mkdir("/sys/fs/cgroup/systemd/system", 0755) = 0 open("/sys/fs/cgroup/systemd/system/tasks", O_WRONLY|O_CREAT|O_TRUNC|O_CLOEXEC, 0666) = 6 fstat(6, {st_mode=S_IFREG|0644, st_size=0, ...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f60c4ecb000

write(6, "357\n", 4)                    = -1 ESRCH (No such process)
close(6)                                = 0
munmap(0x7f60c4ecb000, 4096)            = 0

writev(3, [{"<27>", 4}, {"systemd", 7}, {"[357]: ", 7}, {"Failed to create root cgroup hierarchy: No such process", 55}, {"\n", 1}], 5) = 74 open("/sys/fs/cgroup/systemd/system/tasks", O_RDONLY|O_CLOEXEC) = 6
fstat(6, {st_mode=S_IFREG|0644, st_size=0, ...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f60c4ecb000
read(6, "357\n", 4096) = 4
open("/sys/fs/cgroup/systemd/tasks", O_WRONLY|O_CREAT|O_TRUNC|O_CLOEXEC, 0666) = 7
fstat(7, {st_mode=S_IFREG|0644, st_size=0, ...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f60c4eca000

write(7, "357\n", 4)                    = -1 ESRCH (No such process)
close(7)                                = 0
munmap(0x7f60c4eca000, 4096)            = 0
read(6, "", 4096)                       = 0
close(6)                                = 0
munmap(0x7f60c4ecb000, 4096)            = 0
open("/sys/fs/cgroup/systemd/system/tasks", O_RDONLY|O_CLOEXEC) = 6
fstat(6, {st_mode=S_IFREG|0644, st_size=0, ...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f60c4ecb000
read(6, "", 4096)                       = 0
close(6)                                = 0
munmap(0x7f60c4ecb000, 4096)            = 0

open("/sys/fs/cgroup/systemd/system", O_RDONLY|O_NONBLOCK|O_DIRECTORY|O_CLOEXEC) = 6
getdents(6, /* 7 entries */, 32768) = 240
getdents(6, /* 0 entries */, 32768) = 0
lstat("/sys/fs/cgroup/systemd/system/tasks", {st_mode=S_IFREG|0644, st_size=0, ...}) = 0

rmdir("/sys/fs/cgroup/systemd/system")  = 0
close(6)                                = 0
close(4)                                = 0
close(5)                                = 0
close(0)                                = 0

writev(3, [{"<27>", 4}, {"systemd", 7}, {"[357]: ", 7}, {"Failed to allocate manager object: No such process", 50}, {"\n", 1}], 5) = 69

Signature · 2 lines

Jiri Kosina
SUSE Labs

Tejun Heo

maintainer

2012-03-29 16:04 UTC #1 lore raw

(using Li's new address and quoting whole message for him)

The change has been in linux-next for quite some time and I've been booting fedora 16 a lot of times on top of this. Weird. I'll try to find out what's wrong and if I can't spot it quick enough, will send a revert patch to Linus. We can try again later. Mandeep, any ideas?

Thanks.

Quoted 66 lines · Jiri Kosina

Hi,
I am not able to boot current Linus' tree on system which is using systemd. In the early boot phase I get
Failed to create root cgroup hierarchy: No such process Failed to allocate manager object: No such process
and that's it, the system is completely unusable afterwards (not having any console spawned, no networking, etc).
Bisection pointed to
	commit 61d1d219c4c0761059236a46867bc49943c4d29d
	Author: Mandeep Singh Baines <[email protected]>
	Date:   Mon Jan 30 12:51:56 2012 -0800

	    cgroup: remove extra calls to find_existing_css_set
and I have verified that reverting this single commit on top of current Linus' tree makes the problem go away and the system boots properly.
strace output shows that with 61d1d219c applied, systemd is failing while trying to add itself to the systemd cgroup:
mkdir("/sys", 0755)                     = -1 EEXIST (File exists)
mkdir("/sys/fs", 0755)                  = -1 EEXIST (File exists)
mkdir("/sys/fs/cgroup", 0755)           = -1 EEXIST (File exists)
mkdir("/sys/fs/cgroup/systemd", 0755)   = -1 EEXIST (File exists)
mkdir("/sys/fs/cgroup/systemd/system", 0755) = 0 open("/sys/fs/cgroup/systemd/system/tasks", O_WRONLY|O_CREAT|O_TRUNC|O_CLOEXEC, 0666) = 6 fstat(6, {st_mode=S_IFREG|0644, st_size=0, ...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f60c4ecb000
write(6, "357\n", 4)                    = -1 ESRCH (No such process)
close(6)                                = 0
munmap(0x7f60c4ecb000, 4096)            = 0
writev(3, [{"<27>", 4}, {"systemd", 7}, {"[357]: ", 7}, {"Failed to create root cgroup hierarchy: No such process", 55}, {"\n", 1}], 5) = 74 open("/sys/fs/cgroup/systemd/system/tasks", O_RDONLY|O_CLOEXEC) = 6
fstat(6, {st_mode=S_IFREG|0644, st_size=0, ...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f60c4ecb000
read(6, "357\n", 4096) = 4
open("/sys/fs/cgroup/systemd/tasks", O_WRONLY|O_CREAT|O_TRUNC|O_CLOEXEC, 0666) = 7
fstat(7, {st_mode=S_IFREG|0644, st_size=0, ...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f60c4eca000
write(7, "357\n", 4)                    = -1 ESRCH (No such process)
close(7)                                = 0
munmap(0x7f60c4eca000, 4096)            = 0
read(6, "", 4096)                       = 0
close(6)                                = 0
munmap(0x7f60c4ecb000, 4096)            = 0
open("/sys/fs/cgroup/systemd/system/tasks", O_RDONLY|O_CLOEXEC) = 6
fstat(6, {st_mode=S_IFREG|0644, st_size=0, ...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f60c4ecb000
read(6, "", 4096)                       = 0
close(6)                                = 0
munmap(0x7f60c4ecb000, 4096)            = 0
open("/sys/fs/cgroup/systemd/system", O_RDONLY|O_NONBLOCK|O_DIRECTORY|O_CLOEXEC) = 6
getdents(6, /* 7 entries */, 32768) = 240
getdents(6, /* 0 entries */, 32768) = 0
lstat("/sys/fs/cgroup/systemd/system/tasks", {st_mode=S_IFREG|0644, st_size=0, ...}) = 0
rmdir("/sys/fs/cgroup/systemd/system")  = 0
close(6)                                = 0
close(4)                                = 0
close(5)                                = 0
close(0)                                = 0
writev(3, [{"<27>", 4}, {"systemd", 7}, {"[357]: ", 7}, {"Failed to allocate manager object: No such process", 50}, {"\n", 1}], 5) = 69

Signature · 1 line

tejun

Tejun Heo

maintainer

2012-03-29 16:26 UTC #2 lore raw

PATCH 1/1 cgroup: cgroup_attach_task() could return -errno after success

61d1d219c4 "cgroup: remove extra calls to find_existing_css_set" made cgroup_task_migrate() return void. An unfortunate side effect was that cgroup_attach_task() was depending on that function's return value to clear its @retval on the success path. On cgroup mounts without any subsystem with ->can_attach() callback, cgroup_attach_task() ended up returning @retval without initializing it on success.

For some reason, gcc failed to warn about it and it didn't cause cgroup_attach_task() to return non-zero value in many cases, probably due to difference in register allocation. When the problem materializes, systemd fails to populate /systemd cgroup mount and fails to boot.

Fix it by initializing @retval to zero on declaration.

Signed-off-by Tejun Heo <[email protected]>

Reported-by Jiri Kosina <[email protected]>

LKML-Reference: <alpine.LNX.2.00.1203282354440.25526-ztGlSCb7Y1iN3ZZ/[email protected]>
---
Jiri, can you please confirm the fix?

Thanks.

 kernel/cgroup.c |    2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

kernel/cgroup.c +1 -1

index f4ea4b6..ed64cca 100644
@@ -1883,7 +1883,7 @@ static void cgroup_task_migrate(struct cgroup *cgrp, struct cgroup *oldcgrp,
  */
 int cgroup_attach_task(struct cgroup *cgrp, struct task_struct *tsk)
 {
-	int retval;
+	int retval = 0;
 	struct cgroup_subsys *ss, *failed_ss = NULL;
 	struct cgroup *oldcgrp;
 	struct cgroupfs_root *root = cgrp->root;

Mandeep Singh Baines

Reviewed-by on v1 1/1

2012-03-29 18:11 UTC #3 lore raw

Re: [PATCH for-3.4-fixes] cgroup: cgroup_attach_task() could return -errno after success

Quoted 14 lines · Tejun Heo

61d1d219c4 "cgroup: remove extra calls to find_existing_css_set" made cgroup_task_migrate() return void. An unfortunate side effect was that cgroup_attach_task() was depending on that function's return value to clear its @retval on the success path. On cgroup mounts without any subsystem with ->can_attach() callback, cgroup_attach_task() ended up returning @retval without initializing it on success.
For some reason, gcc failed to warn about it and it didn't cause cgroup_attach_task() to return non-zero value in many cases, probably due to difference in register allocation. When the problem materializes, systemd fails to populate /systemd cgroup mount and fails to boot.

Quoted 3 lines

Fix it by initializing @retval to zero on declaration.
Signed-off-by Tejun Heo <[email protected]>

Reviewed-by Mandeep Singh Baines <[email protected]>

Quoted 23 lines

Reported-by Jiri Kosina <[email protected]>

LKML-Reference: <alpine.LNX.2.00.1203282354440.25526-ztGlSCb7Y1iN3ZZ/[email protected]>
---
Jiri, can you please confirm the fix?

Thanks.

 kernel/cgroup.c |    2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

kernel/cgroup.c +1 -1

index f4ea4b6..ed64cca 100644
@@ -1883,7 +1883,7 @@ static void cgroup_task_migrate(struct cgroup *cgrp, struct cgroup *oldcgrp,
  */
 int cgroup_attach_task(struct cgroup *cgrp, struct task_struct *tsk)
 {
-	int retval;
+	int retval = 0;
 	struct cgroup_subsys *ss, *failed_ss = NULL;
 	struct cgroup *oldcgrp;
 	struct cgroupfs_root *root = cgrp->root;

Li Zefan

Acked-by on v1 1/1

2012-03-30 00:37 UTC #4 lore raw

Re: [PATCH for-3.4-fixes] cgroup: cgroup_attach_task() could return -errno after success

In reply to Tejun Heo

Quoted 14 lines · Tejun Heo

61d1d219c4 "cgroup: remove extra calls to find_existing_css_set" made cgroup_task_migrate() return void. An unfortunate side effect was that cgroup_attach_task() was depending on that function's return value to clear its @retval on the success path. On cgroup mounts without any subsystem with ->can_attach() callback, cgroup_attach_task() ended up returning @retval without initializing it on success.
For some reason, gcc failed to warn about it and it didn't cause cgroup_attach_task() to return non-zero value in many cases, probably due to difference in register allocation. When the problem materializes, systemd fails to populate /systemd cgroup mount and fails to boot.

Quoted 5 lines

Fix it by initializing @retval to zero on declaration.
Signed-off-by Tejun Heo <[email protected]>
Reported-by Jiri Kosina <[email protected]>
LKML-Reference: <alpine.LNX.2.00.1203282354440.25526-ztGlSCb7Y1iN3ZZ/[email protected]>

Acked-by Li Zefan <[email protected]>

Jiri Kosina

author Tested-by on v1 1/1

2012-03-30 08:38 UTC #5 lore raw

Re: [PATCH for-3.4-fixes] cgroup: cgroup_attach_task() could return -errno after success

In reply to Tejun Heo

Quoted 14 lines · Tejun Heo

61d1d219c4 "cgroup: remove extra calls to find_existing_css_set" made cgroup_task_migrate() return void. An unfortunate side effect was that cgroup_attach_task() was depending on that function's return value to clear its @retval on the success path. On cgroup mounts without any subsystem with ->can_attach() callback, cgroup_attach_task() ended up returning @retval without initializing it on success.
For some reason, gcc failed to warn about it and it didn't cause cgroup_attach_task() to return non-zero value in many cases, probably due to difference in register allocation. When the problem materializes, systemd fails to populate /systemd cgroup mount and fails to boot.

Quoted 7 lines

Fix it by initializing @retval to zero on declaration.
Signed-off-by Tejun Heo <[email protected]>
Reported-by Jiri Kosina <[email protected]>
LKML-Reference: <alpine.LNX.2.00.1203282354440.25526-ztGlSCb7Y1iN3ZZ/[email protected]>
---
Jiri, can you please confirm the fix?

Thanks Tejun, it works here.

Tested-by Jiri Kosina <[email protected]>

Signature · 2 lines

Jiri Kosina
SUSE Labs