Skip to contentSkip to conversation

cgroup panics with mmotm for 2.6.28-rc7

BUG Balbir Singh 3 participants 8 messages Dec 15, 2008 – Dec 19, 2008

lore
Share & tools
lore
#0 lore raw

Hi, Paul,

I see the following stack trace when I run my tests. I've not yet investigated the problem.

------------[ cut here ]------------
kernel BUG at kernel/cgroup.c:392!
invalid opcode: 0000 [#1] SMP 
last sysfs file: /sys/devices/pci0000:00/0000:00:1c.5/0000:03:00.0/irq
CPU 1 
Modules linked in: coretemp hwmon kvm_intel kvm rtc_cmos rtc_core

rtc_lib mptsas mptscsih mptbase scsi_transport_sas uhci_hcd ohci_hcd ehci_hcd
Pid: 3866, comm: libcgrouptest01 Tainted: G W 2.6.28-rc7-mm1
#3

RIP: 0010:[<ffffffff80272c5b>]  [<ffffffff80272c5b>]
link_css_set+0xf/0x5a
RSP: 0018:ffff8800388ebda8  EFLAGS: 00010246
RAX: ffffffff807a2790 RBX: ffff88003e9667e0 RCX: ffff88012f6ecff8
RDX: ffff88012f6ecff8 RSI: ffff88003e9667e0 RDI: ffff8800388ebdf8
RBP: ffff8800388ebda8 R08: ffff8800388ebdf8 R09: 0000000000000000
R10: ffff88003a55d000 R11: ffffe2000196d170 R12: ffff88003e9667e0
R13: 0000000000000001 R14: ffff880038838000 R15: ffffffff811d3c00
FS:  00007f482219e700(0000) GS:ffff88007ff064b0(0000)

knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 00007f48221cc000 CR3: 0000000038896000 CR4: 00000000000026e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Process libcgrouptest01 (pid: 3866, threadinfo ffff8800388ea000, task ffff880038838000)

Stack:
 ffff8800388ebe48 ffffffff802740dd ffffffff80272955 ffff88012f6ecff8
 ffff8800389b4028 ffff8800389b4000 ffffffff807aa720 ffff88012f6ecb68
 ffff88012fd852a8 0000000000000246 ffff8800388ebdf8 ffff8800388ebdf8
Call Trace:
 [<ffffffff802740dd>] cgroup_attach_task+0x234/0x3d0
 [<ffffffff80272955>] ? cgroup_lock_live_group+0x1a/0x36
 [<ffffffff80274380>] cgroup_tasks_write+0x107/0x12e
 [<ffffffff802742b8>] ? cgroup_tasks_write+0x3f/0x12e
 [<ffffffff802748d4>] cgroup_file_write+0xfb/0x22d
 [<ffffffff8039e30d>] ? __up_read+0x9b/0xa3
 [<ffffffff802ba77d>] vfs_write+0xae/0x157
 [<ffffffff802bac98>] sys_write+0x47/0x6f
 [<ffffffff8020bfdb>] system_call_fastpath+0x16/0x1b
Code: 8b 5b 30 48 39 c3 74 06 48 3b 5b 60 75 f1 48 39 c3 0f 94 c0 0f

b6 c0 5b 5a 5b c9 c3 4c 8b 07 55 48 89 d1 48 89 e5 49 39 f8 75 04 <0f> 0b eb fe 49 8b 40 08 49 8b 10 49 89 70 20 48 89 10 48 89 42 RIP [<ffffffff80272c5b>] link_css_set+0xf/0x5a
RSP <ffff8800388ebda8>

---[ end trace b73399e271602d45 ]---

I've setup my system using libcgroup to create default cgroups and to automatically classify all tasks to the "default group". I've made some changes to cgconfig and cgred scripts (they can be found under the source code of libcgroup (branches/balbir-tests).

Steps to reproduce

1. Start with the new scripts and move all tasks to a default cgroup
2. Ensure that cgred and cgconfig are running
3. Stop cgred and cgconfig and run libcgrouptest01 (patches posted by sudhir on the libcgroup mailing list).

The test log is

WARN: /dev/cgroup_controllers-1 already exist..overwriting C:DBG: fs_mounted as recieved from script=1
C:DBG: mountpoint1 as recieved from script=/dev/cgroup_controllers-1 sanity check pass. cgroup
TEST 1:PASS : cgroup_attach_task() Ret Value = 50014
Par: nullcgroup
TEST 2:PASS : cgroup_init() Ret Value = 0
TEST 3:PASS : cgroup_attach_task() Ret Value = 0 Task
found in group/s

TEST 4:PASS : cgroup_attach_task_pid()           Ret Value = 50016
TEST 1:PASS : cgroup_new_cgroup()                Ret Value = 0  
TEST 2:PASS : cgroup_create_cgroup()             Ret Value = 0   grp

found in fs

Signature · 1 line
	Balbir
#1 lore raw

That implies that we ran out of css_set objects when moving the task into the new cgroup.

Did you have all of the configured controllers mounted, or just a subset?

What date was this mmotm? Was it after Li's patch fixes went in on 8th Dec?

Paul

Quoted 90 lines · Balbir Singh

Hi, Paul,

I see the following stack trace when I run my tests. I've not yet investigated the problem.

------------[ cut here ]------------
kernel BUG at kernel/cgroup.c:392!
invalid opcode: 0000 [#1] SMP
last sysfs file: /sys/devices/pci0000:00/0000:00:1c.5/0000:03:00.0/irq
CPU 1
Modules linked in: coretemp hwmon kvm_intel kvm rtc_cmos rtc_core

rtc_lib mptsas mptscsih mptbase scsi_transport_sas uhci_hcd ohci_hcd ehci_hcd
Pid: 3866, comm: libcgrouptest01 Tainted: G W 2.6.28-rc7-mm1
#3

RIP: 0010:[<ffffffff80272c5b>]  [<ffffffff80272c5b>]
link_css_set+0xf/0x5a
RSP: 0018:ffff8800388ebda8  EFLAGS: 00010246
RAX: ffffffff807a2790 RBX: ffff88003e9667e0 RCX: ffff88012f6ecff8
RDX: ffff88012f6ecff8 RSI: ffff88003e9667e0 RDI: ffff8800388ebdf8
RBP: ffff8800388ebda8 R08: ffff8800388ebdf8 R09: 0000000000000000
R10: ffff88003a55d000 R11: ffffe2000196d170 R12: ffff88003e9667e0
R13: 0000000000000001 R14: ffff880038838000 R15: ffffffff811d3c00
FS:  00007f482219e700(0000) GS:ffff88007ff064b0(0000)

knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 00007f48221cc000 CR3: 0000000038896000 CR4: 00000000000026e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Process libcgrouptest01 (pid: 3866, threadinfo ffff8800388ea000, task ffff880038838000)

Stack:
 ffff8800388ebe48 ffffffff802740dd ffffffff80272955 ffff88012f6ecff8
 ffff8800389b4028 ffff8800389b4000 ffffffff807aa720 ffff88012f6ecb68
 ffff88012fd852a8 0000000000000246 ffff8800388ebdf8 ffff8800388ebdf8
Call Trace:
 [<ffffffff802740dd>] cgroup_attach_task+0x234/0x3d0
 [<ffffffff80272955>] ? cgroup_lock_live_group+0x1a/0x36
 [<ffffffff80274380>] cgroup_tasks_write+0x107/0x12e
 [<ffffffff802742b8>] ? cgroup_tasks_write+0x3f/0x12e
 [<ffffffff802748d4>] cgroup_file_write+0xfb/0x22d
 [<ffffffff8039e30d>] ? __up_read+0x9b/0xa3
 [<ffffffff802ba77d>] vfs_write+0xae/0x157
 [<ffffffff802bac98>] sys_write+0x47/0x6f
 [<ffffffff8020bfdb>] system_call_fastpath+0x16/0x1b
Code: 8b 5b 30 48 39 c3 74 06 48 3b 5b 60 75 f1 48 39 c3 0f 94 c0 0f

b6 c0 5b 5a 5b c9 c3 4c 8b 07 55 48 89 d1 48 89 e5 49 39 f8 75 04 <0f> 0b eb fe 49 8b 40 08 49 8b 10 49 89 70 20 48 89 10 48 89 42 RIP [<ffffffff80272c5b>] link_css_set+0xf/0x5a
RSP <ffff8800388ebda8>

---[ end trace b73399e271602d45 ]---

I've setup my system using libcgroup to create default cgroups and to automatically classify all tasks to the "default group". I've made some changes to cgconfig and cgred scripts (they can be found under the source code of libcgroup (branches/balbir-tests).

Steps to reproduce

1. Start with the new scripts and move all tasks to a default cgroup
2. Ensure that cgred and cgconfig are running
3. Stop cgred and cgconfig and run libcgrouptest01 (patches posted by sudhir on the libcgroup mailing list).

The test log is

WARN: /dev/cgroup_controllers-1 already exist..overwriting C:DBG: fs_mounted as recieved from script=1
C:DBG: mountpoint1 as recieved from script=/dev/cgroup_controllers-1 sanity check pass. cgroup
TEST 1:PASS : cgroup_attach_task() Ret Value = 50014
Par: nullcgroup
TEST 2:PASS : cgroup_init() Ret Value = 0
TEST 3:PASS : cgroup_attach_task() Ret Value = 0 Task
found in group/s

TEST 4:PASS : cgroup_attach_task_pid()           Ret Value = 50016
TEST 1:PASS : cgroup_new_cgroup()                Ret Value = 0
TEST 2:PASS : cgroup_create_cgroup()             Ret Value = 0   grp

found in fs

--
Balbir
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

#2 lore raw
Quoted 1 line · Paul Menage

That implies that we ran out of css_set objects

I mean, ran out of cg_cgroup_link objects.

Quoted 2 lines

when moving the task
into the new cgroup.

Paul

#3 lore raw
In reply to Paul Menage
Quoted 7 lines · Paul Menage

That implies that we ran out of css_set objects when moving the task into the new cgroup.

Did you have all of the configured controllers mounted, or just a subset?

What date was this mmotm? Was it after Li's patch fixes went in on 8th Dec?

It is probably my fault. :( I'm looking into this problem.

There are 2 related cleanup patches in -mm:

cgroups-add-inactive-subsystems-to-rootnodesubsys_list.patch (and -fix.patch) cgroups-introduce-link_css_set-to-remove-duplicate-code.patch (and -fix.patch)

If the bug is reproducable, could you revert the above patches and seee if the bug is still there.

Quoted 84 lines

Paul

Quoted 81 lines · Balbir Singh

Hi, Paul,

I see the following stack trace when I run my tests. I've not yet investigated the problem.

------------[ cut here ]------------
kernel BUG at kernel/cgroup.c:392!
invalid opcode: 0000 [#1] SMP
last sysfs file: /sys/devices/pci0000:00/0000:00:1c.5/0000:03:00.0/irq
CPU 1
Modules linked in: coretemp hwmon kvm_intel kvm rtc_cmos rtc_core

rtc_lib mptsas mptscsih mptbase scsi_transport_sas uhci_hcd ohci_hcd ehci_hcd
Pid: 3866, comm: libcgrouptest01 Tainted: G W 2.6.28-rc7-mm1
#3

RIP: 0010:[<ffffffff80272c5b>]  [<ffffffff80272c5b>]
link_css_set+0xf/0x5a
RSP: 0018:ffff8800388ebda8  EFLAGS: 00010246
RAX: ffffffff807a2790 RBX: ffff88003e9667e0 RCX: ffff88012f6ecff8
RDX: ffff88012f6ecff8 RSI: ffff88003e9667e0 RDI: ffff8800388ebdf8
RBP: ffff8800388ebda8 R08: ffff8800388ebdf8 R09: 0000000000000000
R10: ffff88003a55d000 R11: ffffe2000196d170 R12: ffff88003e9667e0
R13: 0000000000000001 R14: ffff880038838000 R15: ffffffff811d3c00
FS:  00007f482219e700(0000) GS:ffff88007ff064b0(0000)

knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 00007f48221cc000 CR3: 0000000038896000 CR4: 00000000000026e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Process libcgrouptest01 (pid: 3866, threadinfo ffff8800388ea000, task ffff880038838000)

Stack:
 ffff8800388ebe48 ffffffff802740dd ffffffff80272955 ffff88012f6ecff8
 ffff8800389b4028 ffff8800389b4000 ffffffff807aa720 ffff88012f6ecb68
 ffff88012fd852a8 0000000000000246 ffff8800388ebdf8 ffff8800388ebdf8
Call Trace:
 [<ffffffff802740dd>] cgroup_attach_task+0x234/0x3d0
 [<ffffffff80272955>] ? cgroup_lock_live_group+0x1a/0x36
 [<ffffffff80274380>] cgroup_tasks_write+0x107/0x12e
 [<ffffffff802742b8>] ? cgroup_tasks_write+0x3f/0x12e
 [<ffffffff802748d4>] cgroup_file_write+0xfb/0x22d
 [<ffffffff8039e30d>] ? __up_read+0x9b/0xa3
 [<ffffffff802ba77d>] vfs_write+0xae/0x157
 [<ffffffff802bac98>] sys_write+0x47/0x6f
 [<ffffffff8020bfdb>] system_call_fastpath+0x16/0x1b
Code: 8b 5b 30 48 39 c3 74 06 48 3b 5b 60 75 f1 48 39 c3 0f 94 c0 0f

b6 c0 5b 5a 5b c9 c3 4c 8b 07 55 48 89 d1 48 89 e5 49 39 f8 75 04 <0f> 0b eb fe 49 8b 40 08 49 8b 10 49 89 70 20 48 89 10 48 89 42 RIP [<ffffffff80272c5b>] link_css_set+0xf/0x5a
RSP <ffff8800388ebda8>

---[ end trace b73399e271602d45 ]---

I've setup my system using libcgroup to create default cgroups and to automatically classify all tasks to the "default group". I've made some changes to cgconfig and cgred scripts (they can be found under the source code of libcgroup (branches/balbir-tests).

Steps to reproduce

1. Start with the new scripts and move all tasks to a default cgroup
2. Ensure that cgred and cgconfig are running
3. Stop cgred and cgconfig and run libcgrouptest01 (patches posted by sudhir on the libcgroup mailing list).

The test log is

WARN: /dev/cgroup_controllers-1 already exist..overwriting C:DBG: fs_mounted as recieved from script=1
C:DBG: mountpoint1 as recieved from script=/dev/cgroup_controllers-1 sanity check pass. cgroup
TEST 1:PASS : cgroup_attach_task() Ret Value = 50014
Par: nullcgroup
TEST 2:PASS : cgroup_init() Ret Value = 0
TEST 3:PASS : cgroup_attach_task() Ret Value = 0 Task
found in group/s

TEST 4:PASS : cgroup_attach_task_pid()           Ret Value = 50016
TEST 1:PASS : cgroup_new_cgroup()                Ret Value = 0
TEST 2:PASS : cgroup_create_cgroup()             Ret Value = 0   grp

found in fs

#4 lore raw
Quoted 11 lines · Li Zefan
Quoted 7 lines · Paul Menage

That implies that we ran out of css_set objects when moving the task into the new cgroup.

Did you have all of the configured controllers mounted, or just a subset?

What date was this mmotm? Was it after Li's patch fixes went in on 8th Dec?

It is probably my fault. :( I'm looking into this problem.

Quoted 18 lines

There are 2 related cleanup patches in -mm:

cgroups-add-inactive-subsystems-to-rootnodesubsys_list.patch (and -fix.patch) cgroups-introduce-link_css_set-to-remove-duplicate-code.patch (and -fix.patch)

If the bug is reproducable, could you revert the above patches and seee if the bug is still there.

Quoted 10 lines

Paul

Quoted 7 lines · Balbir Singh

Hi, Paul,

I see the following stack trace when I run my tests. I've not yet investigated the problem.

------------[ cut here ]------------
kernel BUG at kernel/cgroup.c:392!

In latest -mm, this BUG_ON is line 398, and before the below 2 fixlet patches, the BUG_ON is line 392, so I guess you were using older -mm:

cgroups-add-inactive-subsystems-to-rootnodesubsys_list-fix.patch cgroups-introduce-link_css_set-to-remove-duplicate-code-fix.patch

Could you try the latest -mm kernel, or apply cgroups-add-inactive-subsystems-to-rootnodesubsys_list-fix.patch ?

kernel/cgroup.c renamed from kernel/cgroup.c~cgroups-add-inactive-subsystems-to-rootnodesubsys_list-fix +1 -1
diff -puN kernel/cgroup.c~cgroups-add-inactive-subsystems-to-rootnodesubsys_list-fix kernel/cgroup.c
@@ -2521,7 +2521,7 @@ static void __init cgroup_init_subsys(st
 	printk(KERN_INFO "Initializing cgroup subsys %s\n", ss->name);
 
 	/* Create the top cgroup state for this subsystem */
-	list_add(&ss->sibling, &rootnode.root_list);
+	list_add(&ss->sibling, &rootnode.subsys_list);
 	ss->root = &rootnode;
 	css = ss->create(ss, dummytop);
 	/* We don't handle early failures gracefully */
#5 lore raw
Quoted 45 lines · Li Zefan
Quoted 11 lines · Li Zefan
Quoted 7 lines · Paul Menage

That implies that we ran out of css_set objects when moving the task into the new cgroup.

Did you have all of the configured controllers mounted, or just a subset?

What date was this mmotm? Was it after Li's patch fixes went in on 8th Dec?

It is probably my fault. :( I'm looking into this problem.

Quoted 18 lines

There are 2 related cleanup patches in -mm:

cgroups-add-inactive-subsystems-to-rootnodesubsys_list.patch (and -fix.patch) cgroups-introduce-link_css_set-to-remove-duplicate-code.patch (and -fix.patch)

If the bug is reproducable, could you revert the above patches and seee if the bug is still there.

Quoted 10 lines

Paul

Quoted 7 lines · Balbir Singh

Hi, Paul,

I see the following stack trace when I run my tests. I've not yet investigated the problem.

------------[ cut here ]------------
kernel BUG at kernel/cgroup.c:392!

In latest -mm, this BUG_ON is line 398, and before the below 2 fixlet patches, the BUG_ON is line 392, so I guess you were using older -mm:

cgroups-add-inactive-subsystems-to-rootnodesubsys_list-fix.patch cgroups-introduce-link_css_set-to-remove-duplicate-code-fix.patch

Could you try the latest -mm kernel, or apply cgroups-add-inactive-subsystems-to-rootnodesubsys_list-fix.patch ?

diff -puN kernel/cgroup.c~cgroups-add-inactive-subsystems-to-rootnodesubsys_list-fix kernel/cgroup.c

kernel/cgroup.c static void __init cgroup_init_subsys(st · line 2527

Quoted 8 lines
kernel/cgroup.c renamed from kernel/cgroup.c~cgroups-add-inactive-subsystems-to-rootnodesubsys_list-fix +1 -1
@@ -2521,7 +2521,7 @@ static void __init cgroup_init_subsys(st
 	printk(KERN_INFO "Initializing cgroup subsys %s\n", ss->name);
 
 	/* Create the top cgroup state for this subsystem */
-	list_add(&ss->sibling, &rootnode.root_list);
+	list_add(&ss->sibling, &rootnode.subsys_list);
 	ss->root = &rootnode;
 	css = ss->create(ss, dummytop);
 	/* We don't handle early failures gracefully */

I'll move to the latest mm and make sure your fix is present.

Signature · 1 line
	Balbir
#6 lore raw
In reply to Li Zefan
Quoted 45 lines · Li Zefan
Quoted 11 lines · Li Zefan
Quoted 7 lines · Paul Menage

That implies that we ran out of css_set objects when moving the task into the new cgroup.

Did you have all of the configured controllers mounted, or just a subset?

What date was this mmotm? Was it after Li's patch fixes went in on 8th Dec?

It is probably my fault. :( I'm looking into this problem.

Quoted 18 lines

There are 2 related cleanup patches in -mm:

cgroups-add-inactive-subsystems-to-rootnodesubsys_list.patch (and -fix.patch) cgroups-introduce-link_css_set-to-remove-duplicate-code.patch (and -fix.patch)

If the bug is reproducable, could you revert the above patches and seee if the bug is still there.

Quoted 10 lines

Paul

Quoted 7 lines · Balbir Singh

Hi, Paul,

I see the following stack trace when I run my tests. I've not yet investigated the problem.

------------[ cut here ]------------
kernel BUG at kernel/cgroup.c:392!

In latest -mm, this BUG_ON is line 398, and before the below 2 fixlet patches, the BUG_ON is line 392, so I guess you were using older -mm:

cgroups-add-inactive-subsystems-to-rootnodesubsys_list-fix.patch cgroups-introduce-link_css_set-to-remove-duplicate-code-fix.patch

Could you try the latest -mm kernel, or apply cgroups-add-inactive-subsystems-to-rootnodesubsys_list-fix.patch ?

diff -puN kernel/cgroup.c~cgroups-add-inactive-subsystems-to-rootnodesubsys_list-fix kernel/cgroup.c

kernel/cgroup.c static void __init cgroup_init_subsys(st · line 2527

Quoted 8 lines
kernel/cgroup.c renamed from kernel/cgroup.c~cgroups-add-inactive-subsystems-to-rootnodesubsys_list-fix +1 -1
@@ -2521,7 +2521,7 @@ static void __init cgroup_init_subsys(st
 	printk(KERN_INFO "Initializing cgroup subsys %s\n", ss->name);
 
 	/* Create the top cgroup state for this subsystem */
-	list_add(&ss->sibling, &rootnode.root_list);
+	list_add(&ss->sibling, &rootnode.subsys_list);
 	ss->root = &rootnode;
 	css = ss->create(ss, dummytop);
 	/* We don't handle early failures gracefully */

Yes, this fix does work for me,

Signature · 2 lines
        Thanks!
	Balbir
#7 lore raw
Quoted 16 lines
Quoted 16 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line

kernel BUG at kernel/cgroup.c:392!

In latest -mm, this BUG_ON is line 398, and before the below 2 fixlet patches, the BUG_ON is line 392, so I guess you were using older -mm:

cgroups-add-inactive-subsystems-to-rootnodesubsys_list-fix.patch cgroups-introduce-link_css_set-to-remove-duplicate-code-fix.patch

Could you try the latest -mm kernel, or apply cgroups-add-inactive-subsystems-to-rootnodesubsys_list-fix.patch ?

diff -puN kernel/cgroup.c~cgroups-add-inactive-subsystems-to-rootnodesubsys_list-fix kernel/cgroup.c

kernel/cgroup.c renamed from kernel/cgroup.c~cgroups-add-inactive-subsystems-to-rootnodesubsys_list-fix +0 -0
@@ -2521,7 +2521,7 @@ static void __init cgroup_init_subsys(st
 	printk(KERN_INFO "Initializing cgroup subsys %s\n", ss->name);
 
Quoted 10 lines

kernel/cgroup.c static void __init cgroup_init_subsys(st · line 2527

Quoted 7 lines
kernel/cgroup.c renamed from kernel/cgroup.c~cgroups-add-inactive-subsystems-to-rootnodesubsys_list-fix +1 -1
 	/* Create the top cgroup state for this subsystem */
-	list_add(&ss->sibling, &rootnode.root_list);
+	list_add(&ss->sibling, &rootnode.subsys_list);
 	ss->root = &rootnode;
 	css = ss->create(ss, dummytop);
 	/* We don't handle early failures gracefully */

Yes, this fix does work for me,

Thanks! And sorry for the silly mistake I made.