Attaching a process to cgroups

Hi.

Is it possible to somehow fasten a process of pid attaching to cgroup? The problem is the pid attaches to a task-file with some strange delay:

22:28:00.788224 open("/sys/fs/cgroup/memory/virtwww/w_test-l24-apache1_4bdf3d13/apache/tasks", O_WRONLY|O_CREAT|O_APPEND, 0666) = 3 <0.000035> 22:28:00.788289 fstat(3, {st_mode=S_IFREG|0644, st_size=0, ...}) = 0 <0.000004>
22:28:00.788326 mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f5e78074000 <0.000005> 22:28:00.788355 fstat(3, {st_mode=S_IFREG|0644, st_size=0, ...}) = 0 <0.000004>

22:28:00.788389 lseek(3, 0, SEEK_SET)   = 0 <0.000004>
22:28:00.788426 write(3, "16317\n", 6)  = 6 <0.128094>
22:28:00.916578 close(3)                = 0 <0.000006>

For a comparison here's a test attaching pid-file in placed tmpfs:

22:24:41.892562 open("/tmp/w_test-l24-apache1_4bdf3d13/tasks", O_WRONLY|O_CREAT|O_APPEND, 0666) = 3 <0.000010> 22:24:41.892597 fstat(3, {st_mode=S_IFREG|0644, st_size=6, ...}) = 0 <0.000004>
22:24:41.892631 mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f5685b6f000 <0.000006> 22:24:41.892664 fstat(3, {st_mode=S_IFREG|0644, st_size=6, ...}) = 0 <0.000004>

22:24:41.892701 lseek(3, 6, SEEK_SET)   = 6 <0.000004>
22:24:41.892738 write(3, "25966\n", 6)  = 6 <0.000008>
22:24:41.892767 close(3)                = 0 <0.000005>

Here goes it immediately.

Signature · 6 lines

BRGDS. Alexey Vlasov.
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

What does "cat /sys/fs/cgroup/.../apache/memory.move_charge_at_immigrate" show ?

Hi.

Is it possible to somehow fasten a process of pid attaching to cgroup? The problem is the pid attaches to a task-file with some strange delay:

22:28:00.788224 open("/sys/fs/cgroup/memory/virtwww/w_test-l24-apache1_4bdf3d13/apache/tasks", O_WRONLY|O_CREAT|O_APPEND, 0666) = 3 <0.000035> 22:28:00.788289 fstat(3, {st_mode=S_IFREG|0644, st_size=0, ...}) = 0 <0.000004>
22:28:00.788326 mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f5e78074000 <0.000005> 22:28:00.788355 fstat(3, {st_mode=S_IFREG|0644, st_size=0, ...}) = 0 <0.000004>

22:28:00.788389 lseek(3, 0, SEEK_SET)   = 0 <0.000004>
22:28:00.788426 write(3, "16317\n", 6)  = 6 <0.128094>
22:28:00.916578 close(3)                = 0 <0.000006>

kernel/cgroup.c::cgroup_attach_task()
{
...

	synchronize_rcu();

...
}

Quoted 7 lines · Mike Galbraith

kernel/cgroup.c::cgroup_attach_task()
{
...

	synchronize_rcu();

...
}

So nothing can be done here? (I mean if only I knew how to fix it I wouldn't ask about it ;)

Quoted 9 lines · Alexey Vlasov

Quoted 7 lines · Mike Galbraith

kernel/cgroup.c::cgroup_attach_task()
{
...

	synchronize_rcu();

...
}

Quoted 2 lines

So nothing can be done here? (I mean if only I knew how to fix it I wouldn't ask about it ;)

Sure, kill the obnoxious thing, it's sitting right in the middle of the userspace interface.

Quoted 11 lines · Alexey Vlasov

Quoted 7 lines · Mike Galbraith

kernel/cgroup.c::cgroup_attach_task()
{
...

	synchronize_rcu();

...
}

So nothing can be done here? (I mean if only I knew how to fix it I wouldn't ask about it ;)

Sure, kill the obnoxious thing, it's sitting right in the middle of the userspace interface.

I banged on it a while back (wrt explosive android patches), extracted RCU from the userspace interface. It seemed to work great, much faster, couldn't make it explode. I wouldn't bet anything I wasn't willing to immediately part with that the result was really really safe though ;-)

Quoted 16 lines · Mike Galbraith

Quoted 11 lines · Alexey Vlasov

Quoted 7 lines · Mike Galbraith

kernel/cgroup.c::cgroup_attach_task()
{
...

	synchronize_rcu();

...
}

So nothing can be done here? (I mean if only I knew how to fix it I wouldn't ask about it ;)

Sure, kill the obnoxious thing, it's sitting right in the middle of the userspace interface.

Quoted 4 lines

I banged on it a while back (wrt explosive android patches), extracted RCU from the userspace interface. It seemed to work great, much faster, couldn't make it explode. I wouldn't bet anything I wasn't willing to immediately part with that the result was really really safe though ;-)

Or replace it with synchronize_rcu_expedited(). You can "get lucky" for quite some time removing synchronize_rcu() calls!

Quoted 22 lines · Paul E. McKenney

Quoted 20 lines · Mike Galbraith

Quoted 11 lines · Alexey Vlasov

Quoted 7 lines · Mike Galbraith

kernel/cgroup.c::cgroup_attach_task()
{
...

	synchronize_rcu();

...
}

So nothing can be done here? (I mean if only I knew how to fix it I wouldn't ask about it ;)

Sure, kill the obnoxious thing, it's sitting right in the middle of the userspace interface.

I banged on it a while back (wrt explosive android patches), extracted RCU from the userspace interface. It seemed to work great, much faster, couldn't make it explode. I wouldn't bet anything I wasn't willing to immediately part with that the result was really really safe though ;-)

Quoted 2 lines

Or replace it with synchronize_rcu_expedited(). You can "get lucky" for quite some time removing synchronize_rcu() calls!

s/remove/replace, but yup. A company that wanted to use the android patches plus my tinkering showed a fix they needed on top to close a race discovered in their testing. So yeah, even when all seems fine, extracting synchronize_rcu() may expose evils you couldn't encounter before, and didn't happen to encounter afterward.

I really did mean "remove". Removing a synchronize_rcu() does result in a race, but often an extremely low-probability race. So you can remove a synchronize_rcu() and get lucky for a long time, but sooner or later, something will explode.

Quoted 11 lines · Alexey Vlasov

Quoted 7 lines · Mike Galbraith

kernel/cgroup.c::cgroup_attach_task()
{
...

	synchronize_rcu();

...
}

So nothing can be done here? (I mean if only I knew how to fix it I wouldn't ask about it ;)

Sure, kill the obnoxious thing, it's sitting right in the middle of the userspace interface.

I banged on it a while back (wrt explosive android patches), extracted RCU from the userspace interface. It seemed to work great, much faster, couldn't make it explode. I wouldn't bet anything I wasn't willing to immediately part with that the result was really really safe though ;-)

-Mike

Quoted 16 lines · Mike Galbraith

Quoted 11 lines · Alexey Vlasov

Quoted 7 lines · Mike Galbraith

kernel/cgroup.c::cgroup_attach_task()
{
...

	synchronize_rcu();

...
}

So nothing can be done here? (I mean if only I knew how to fix it I wouldn't ask about it ;)

Sure, kill the obnoxious thing, it's sitting right in the middle of the userspace interface.

Quoted 6 lines

I banged on it a while back (wrt explosive android patches), extracted RCU from the userspace interface. It seemed to work great, much faster, couldn't make it explode. I wouldn't bet anything I wasn't willing to immediately part with that the result was really really safe though ;-)

-Mike

JFYI,

I'm testing the following patch in a bunch of hosts and I wasn't able to make any of them to explode, even running a multi-threaded cgroup-intensive workload, but probably I was just lucky (or unlucky, depending on the point of view).

It is basically the same Not-signed-off-by work posted by Mike a while
ago: https://lkml.org/lkml/2011/4/12/599.

In addition, I totally removed the synchronize_rcu() call from cgroup_attach_task() and added the call_rcu -> schedule_work removal also for css_set. The latter looks unnecessary to me from a logical point of view, or maybe I'm missing something, because I can't explain why with it I can't trigger any BUG / oops.

Mike, did you make any progress from your old patch?

Quoted 10 lines · Alexey Vlasov

Is it possible to somehow fasten a process of pid attaching to cgroup? The problem is the pid attaches to a task-file with some strange delay:

22:28:00.788224 open("/sys/fs/cgroup/memory/virtwww/w_test-l24-apache1_4bdf3d13/apache/tasks", O_WRONLY|O_CREAT|O_APPEND, 0666) = 3 <0.000035> 22:28:00.788289 fstat(3, {st_mode=S_IFREG|0644, st_size=0, ...}) = 0 <0.000004>
22:28:00.788326 mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f5e78074000 <0.000005> 22:28:00.788355 fstat(3, {st_mode=S_IFREG|0644, st_size=0, ...}) = 0 <0.000004>

22:28:00.788389 lseek(3, 0, SEEK_SET)   = 0 <0.000004>
22:28:00.788426 write(3, "16317\n", 6)  = 6 <0.128094>
22:28:00.916578 close(3)                = 0 <0.000006>

kernel/cgroup.c::cgroup_attach_task()
{
...

	synchronize_rcu();

...
}

Hi.

Now I've got almost 5k used groups and it got even worse. Now I've got almost 5k used groups and it got even worse.

If only write was working slower, now everything connected with cgroups is hardly working.

Could it be connected with synchronize_rcu()?

Hanging on read():

# strace -ttT cat /proc/cgroups 

17:30:43.825005 fstat(1, {st_mode=S_IFCHR|0620, st_rdev=makedev(136, 13), ...}) = 0 <0.000005> 17:30:43.825048 open("/proc/cgroups", O_RDONLY) = 3 <0.000014>
17:30:43.825085 fstat(3, {st_mode=S_IFREG|0444, st_size=0, ...}) = 0 <0.000004> 17:30:43.825125 fadvise64(3, 0, 0, POSIX_FADV_SEQUENTIAL) = 0 <0.000005> 17:30:43.825161 read(3, "#subsys_name\thierarchy\tnum_cgrou"..., 32768) = 112 <7.447084>

I'd profile it with perf, and expect to find a large pile of cycles.

Quoted 2 lines · Mike Galbraith

I'd profile it with perf, and expect to find a large pile of cycles.

I did it the as following:

# perf stat cat /proc/self/cgroup 

4:blkio:/
3:devices:/
2:memory:/
1:cpuacct:/

Performance counter stats for 'cat /proc/self/cgroup':

          0.472513 task-clock                #    0.000 CPUs utilized          
                 1 context-switches          #    0.002 M/sec                  
                 1 CPU-migrations            #    0.002 M/sec                  
               169 page-faults               #    0.358 M/sec                  
           1111521 cycles                    #    2.352 GHz                    
            784737 stalled-cycles-frontend   #   70.60% frontend cycles idle   
            445520 stalled-cycles-backend    #   40.08% backend  cycles idle   
            576622 instructions              #    0.52  insns per cycle        

# 1.36 stalled cycles per insn
120032 branches # 254.029 M/sec
6577 branch-misses # 5.48% of all branches

9.114631804 seconds time elapsed

# perf report --sort comm,dso

Kernel address maps (/proc/{kallsyms,modules}) were restricted.

Check /proc/sys/kernel/kptr_restrict before running 'perf record'.

If some relocation was applied (e.g. kexec) symbols may be misresolved.

Samples in kernel modules can't be resolved as well.

# ========

# captured on: Thu Jul 26 16:23:06 2012
# hostname : l24
# os release : 3.3.3-1gb-c-s-m
# perf version : 3.2
# arch : x86_64
# nrcpus online : 24
# nrcpus avail : 24
# cpudesc : Intel(R) Xeon(R) CPU E5645 @ 2.40GHz
# total memory : 74181032 kB
# cmdline : /usr/sbin/perf record cat /proc/self/cgroup

# event : name = cycles, type = 0, config = 0x0, config1 = 0x0, config2 = 0x0, excl_usr = 0, excl_kern = 0, id = { 1758, 1759, 1760, 1761, 1762, 1763, 1764, 1765, 1766, 1767, 1768, 1769, 1770, 1771, 1772, 1773, 1774, 1775, 1776, 1777, 1778, 1779, 1780, 1781 }
# HEADER_CPU_TOPOLOGY info available, use -I to display
# HEADER_NUMA_TOPOLOGY info available, use -I to display
# ========
#
# Events: 21  cycles
#
# Overhead  Command      Shared Object
# ........  .......  .................
#
   100.00%      cat  [kernel.kallsyms]

but I don't know what next unfortunately.

I also checked the same thing on the other server with the 2.6.37 kernel, there' some thousands cgroups too, and it somehow works there immediately.

Quoted 9 lines

Hanging on read():

# strace -ttT cat /proc/cgroups 

17:30:43.825005 fstat(1, {st_mode=S_IFCHR|0620, st_rdev=makedev(136, 13), ...}) = 0 <0.000005> 17:30:43.825048 open("/proc/cgroups", O_RDONLY) = 3 <0.000014>
17:30:43.825085 fstat(3, {st_mode=S_IFREG|0444, st_size=0, ...}) = 0 <0.000004> 17:30:43.825125 fadvise64(3, 0, 0, POSIX_FADV_SEQUENTIAL) = 0 <0.000005> 17:30:43.825161 read(3, "#subsys_name\thierarchy\tnum_cgrou"..., 32768) = 112 <7.447084>

Quoted 4 lines · Mike Galbraith

Quoted 4 lines

Hanging on read():

# strace -ttT cat /proc/cgroups 

Quoted 5 lines

Quoted 5 lines

17:30:43.825005 fstat(1, {st_mode=S_IFCHR|0620, st_rdev=makedev(136, 13), ...}) = 0 <0.000005> 17:30:43.825048 open("/proc/cgroups", O_RDONLY) = 3 <0.000014>
17:30:43.825085 fstat(3, {st_mode=S_IFREG|0444, st_size=0, ...}) = 0 <0.000004> 17:30:43.825125 fadvise64(3, 0, 0, POSIX_FADV_SEQUENTIAL) = 0 <0.000005> 17:30:43.825161 read(3, "#subsys_name\thierarchy\tnum_cgrou"..., 32768) = 112 <7.447084>

In general I've changed it to synchronize_rcu_expedited () and all the delays have gone both on writing and reading files from cgroups.

Quoted 3 lines · Alexey Vlasov

In general I've changed it to synchronize_rcu_expedited () and all the delays have gone both on writing and reading files from cgroups.

Is the writing and reading from cgroups something that your workload does all the time, or is it something that happens only on occasional updates to your cgroup configuration?