how to make memory.memsw.failcnt is nonzero
Subsystems (2)
memory.memsw.failcnt shows the number of memory+Swap hits limits.
So I think when memory+swap usage is equal to limit, memsw.failcnt should be nonzero.
I test as follows:
# uname -a Linux K-test 3.2.0-rc7-17-g371de6e #2 SMP Wed Dec 28 12:02:52 CST 2011 x86_64 x86_64 x86_64 GNU/Linux # mkdir /cgroup/memory/group # cd /cgroup/memory/group/ # echo 10M > memory.limit_in_bytes # echo 10M > memory.memsw.limit_in_bytes # echo $$ > tasks # dd if=/dev/zero of=/tmp/temp_file count=20 bs=1M Killed # cat memory.memsw.failcnt 0 # grep "failcnt" /var/log/messages | tail -2 Dec 28 17:05:52 K-test kernel: memory: usage 10240kB, limit 10240kB, failcnt 21 Dec 28 17:05:52 K-test kernel: memory+swap: usage 10240kB, limit 10240kB, failcnt 0
memory+swap usage is equal to limit, but memsw.failcnt is zero.
I change memory.memsw.limit_in_bytes to 15M.
# echo 15M > memory.memsw.limit_in_bytes # dd if=/dev/zero of=/tmp/temp_file count=20 bs=1M Killed # grep "failcnt" /var/log/messages | tail -2 Dec 28 17:08:45 K-test kernel: memory: usage 10240kB, limit 10240kB, failcnt 86 Dec 28 17:08:45 K-test kernel: memory+swap: usage 10240kB, limit 15360kB, failcnt 0 # cat memory.memsw.failcnt 0
The limit is 15M, but memory+swap usage also is 10M.
I think memory+swap usage should be 15M and memsw.failcnt should be nonzero.
This is a kernel bug or I misunderstand memory+swap?
I am looking forward to your feedback soon!
Thanks
Signature · 2 lines
Best Regards, Peng
[Let's add some people to the CC list]
Hi,
sorry for the late reply (some vacation and holiday)
Quoted 34 lines · Peng Haitao
memory.memsw.failcnt shows the number of memory+Swap hits limits.
So I think when memory+swap usage is equal to limit, memsw.failcnt should be nonzero.I test as follows:
# uname -a Linux K-test 3.2.0-rc7-17-g371de6e #2 SMP Wed Dec 28 12:02:52 CST 2011 x86_64 x86_64 x86_64 GNU/Linux # mkdir /cgroup/memory/group # cd /cgroup/memory/group/ # echo 10M > memory.limit_in_bytes # echo 10M > memory.memsw.limit_in_bytes # echo $$ > tasks # dd if=/dev/zero of=/tmp/temp_file count=20 bs=1M Killed # cat memory.memsw.failcnt 0 # grep "failcnt" /var/log/messages | tail -2 Dec 28 17:05:52 K-test kernel: memory: usage 10240kB, limit 10240kB, failcnt 21 Dec 28 17:05:52 K-test kernel: memory+swap: usage 10240kB, limit 10240kB, failcnt 0memory+swap usage is equal to limit, but memsw.failcnt is zero.
I change memory.memsw.limit_in_bytes to 15M.
# echo 15M > memory.memsw.limit_in_bytes # dd if=/dev/zero of=/tmp/temp_file count=20 bs=1M Killed # grep "failcnt" /var/log/messages | tail -2 Dec 28 17:08:45 K-test kernel: memory: usage 10240kB, limit 10240kB, failcnt 86 Dec 28 17:08:45 K-test kernel: memory+swap: usage 10240kB, limit 15360kB, failcnt 0 # cat memory.memsw.failcnt 0
Quoted 4 lines
The limit is 15M, but memory+swap usage also is 10M.
I think memory+swap usage should be 15M and memsw.failcnt should be nonzero.This is a kernel bug or I misunderstand memory+swap?
Well, we might end up with memory.failcnt > 0 while memory.memsw.failcnt == 0 quite easily and that would happen in the cases like you describe above when there is a big pagecache pressure on the hardlimit without much anonymous memory so there is not much that could be swapped out.
Please note that memsw.limit_in_bytes is triggered only if we have consumed some swap space already (and the feature is primarily intended to stop extensive swap usage in fact).
It goes like this: If we trigger hard limit (memory.limit_in_bytes) then we start the direct reclaim (with swap available). If we trigger memsw limit then we try to reclaim without swap available. We will OOM if we cannot reclaim enough to satisfy the respective limit.
The other part of the answer is, yes there is something wrong going on her because we definitely shouldn't OOM. The workload is a single threaded and we have a plenty of page cache that could be reclaimed easily. On the other hand we end up with:
# echo $$ > tasks /dev/memctl/a# echo 10M > memory.limit_in_bytes /dev/memctl/a# echo 10M > memory.memsw.limit_in_bytes /dev/memctl/a# dd if=/dev/zero of=/tmp/temp_file count=20 bs=1M Killed /dev/memctl/a# cat memory.stat cache 9265152 rss 143360 mapped_file 0 pgpgin 3352 pgpgout 1055 swap 0 pgfault 798 pgmajfault 1 inactive_anon 12288 active_anon 114688 inactive_file 9261056 active_file 4096 unevictable 0 [...]
So there is almost 10M of page cache that we can simply reclaim. If we use 40M limit then we are OK. So this looks like the small limit somehow tricks our math in the reclaim path and we think there is nothing to reclaim.
I will look into this.
Signature · 6 lines
Michal Hocko SUSE Labs SUSE LINUX s.r.o. Lihovarska 1060/12 190 00 Praha 9 Czech Republic
Michal Hocko said the following on 2012-1-4 0:04:
Quoted 9 lines
Quoted 9 lines
# echo 15M > memory.memsw.limit_in_bytes # dd if=/dev/zero of=/tmp/temp_file count=20 bs=1M Killed # grep "failcnt" /var/log/messages | tail -2 Dec 28 17:08:45 K-test kernel: memory: usage 10240kB, limit 10240kB, failcnt 86 Dec 28 17:08:45 K-test kernel: memory+swap: usage 10240kB, limit 15360kB, failcnt 0 # cat memory.memsw.failcnt 0
Quoted 8 lines
Quoted 3 lines
The limit is 15M, but memory+swap usage also is 10M.
I think memory+swap usage should be 15M and memsw.failcnt should be nonzero.So there is almost 10M of page cache that we can simply reclaim. If we use 40M limit then we are OK. So this looks like the small limit somehow tricks our math in the reclaim path and we think there is nothing to reclaim.
I will look into this.
Thanks for you reply.
If there is something wrong, I think the bug will be in mem_cgroup_do_charge() of mm/memcontrol.c
2210 ret = res_counter_charge(&memcg->res, csize, &fail_res);
2211
2212 if (likely(!ret)) {
2213 if (!do_swap_account)
2214 return CHARGE_OK;
2215 ret = res_counter_charge(&memcg->memsw, csize, &fail_res);
2216 if (likely(!ret))
2217 return CHARGE_OK;
2218
2219 res_counter_uncharge(&memcg->res, csize); 2220 mem_over_limit = mem_cgroup_from_res_counter(fail_res, memsw); 2221 flags |= MEM_CGROUP_RECLAIM_NOSWAP;
2222 } else
2223 mem_over_limit = mem_cgroup_from_res_counter(fail_res, res);
When hit memory.limit_in_bytes, res_counter_charge() will return -ENOMEM, this will execute line 2222: } else.
But I think when hit memory.limit_in_bytes, the function should determine further to memory.memsw.limit_in_bytes.
This think is OK?
Signature · 2 lines
Best Regards, Peng
Quoted 36 lines · Peng Haitao
Michal Hocko said the following on 2012-1-4 0:04:
Quoted 9 lines
Quoted 9 lines
# echo 15M > memory.memsw.limit_in_bytes # dd if=/dev/zero of=/tmp/temp_file count=20 bs=1M Killed # grep "failcnt" /var/log/messages | tail -2 Dec 28 17:08:45 K-test kernel: memory: usage 10240kB, limit 10240kB, failcnt 86 Dec 28 17:08:45 K-test kernel: memory+swap: usage 10240kB, limit 15360kB, failcnt 0 # cat memory.memsw.failcnt 0Quoted 8 lines
Quoted 3 lines
The limit is 15M, but memory+swap usage also is 10M.
I think memory+swap usage should be 15M and memsw.failcnt should be nonzero.So there is almost 10M of page cache that we can simply reclaim. If we use 40M limit then we are OK. So this looks like the small limit somehow tricks our math in the reclaim path and we think there is nothing to reclaim.
I will look into this.Thanks for you reply.
If there is something wrong, I think the bug will be inmem_cgroup_do_charge()ofmm/memcontrol.c2210 ret =
res_counter_charge(&memcg->res, csize, &fail_res);
2211
2212 if (likely(!ret)) {
2213 if (!do_swap_account)
2214 returnCHARGE_OK;
2215 ret =res_counter_charge(&memcg->memsw, csize, &fail_res);
2216 if (likely(!ret))
2217 returnCHARGE_OK;
22182219 res_counter_uncharge(&memcg->res, csize); 2220 mem_over_limit = mem_cgroup_from_res_counter(fail_res, memsw); 2221 flags |= MEM_CGROUP_RECLAIM_NOSWAP;
Quoted 8 lines
2222 } else
2223mem_over_limit=mem_cgroup_from_res_counter(fail_res, res);When hit memory.limit_in_bytes,
res_counter_charge()will return-ENOMEM, this will execute line 2222: } else.
But I think when hit memory.limit_in_bytes, the function should determine further to memory.memsw.limit_in_bytes.
This think is OK?
I don't think so. We have an invariant (hard limit is "stronger" than memsw limit) memory.limit_in_bytes <= memory.memsw.limit_in_bytes so when we hit the hard limit we do not have to consider memsw because resource counter:
a) we already have to do reclaim for hard limit
b) we check whether we might swap out later on in mem_cgroup_hierarchical_reclaim (root_memcg->memsw_is_minimum) so we will not end up swapping just to make hard limit ok and go over memsw limit.
Please also note that we will retry charging after reclaim if there is a chance to meet the limit.
Makes sense?
Signature · 13 lines
Michal Hocko SUSE Labs SUSE LINUX s.r.o. Lihovarska 1060/12 190 00 Praha 9 Czech Republic -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/ Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Michal Hocko said the following on 2012-1-6 18:12:
Quoted 6 lines
Quoted 6 lines
If there is something wrong, I think the bug will be in
mem_cgroup_do_charge()ofmm/memcontrol.c2210 ret =
res_counter_charge(&memcg->res, csize, &fail_res);
2211
2212 if (likely(!ret)) {
...
Quoted 20 lines
Quoted 9 lines
2221 flags |=
MEM_CGROUP_RECLAIM_NOSWAP;
2222 } else
2223mem_over_limit=mem_cgroup_from_res_counter(fail_res, res);When hit memory.limit_in_bytes,
res_counter_charge()will return-ENOMEM, this will execute line 2222: } else.
But I think when hit memory.limit_in_bytes, the function should determine further to memory.memsw.limit_in_bytes.
This think is OK?I don't think so. We have an invariant (hard limit is "stronger" than memsw limit) memory.limit_in_bytes <= memory.memsw.limit_in_bytes so when we hit the hard limit we do not have to consider memsw because resource counter:
a) we already have to do reclaim for hard limit
b) we check whether we might swap out later on inmem_cgroup_hierarchical_reclaim(root_memcg->memsw_is_minimum) so we will not end up swapping just to make hard limit ok and go over memsw limit.
Quoted 3 lines
Please also note that we will retry charging after reclaim if there is a chance to meet the limit.
Makes sense?
Yeah.
But I want to test memory.memsw.failcnt is nonzero, how steps? Thanks.
Signature · 2 lines
Best Regards, Peng
Quoted 29 lines · On Mon, 30 Jan 2012 10:47:33 +0800 Peng Haitao…
Michal Hocko said the following on 2012-1-6 18:12:
Quoted 6 lines
Quoted 6 lines
If there is something wrong, I think the bug will be in
mem_cgroup_do_charge()ofmm/memcontrol.c2210 ret =
res_counter_charge(&memcg->res, csize, &fail_res);
2211
2212 if (likely(!ret)) {...
Quoted 20 lines
Quoted 9 lines
2221 flags |=
MEM_CGROUP_RECLAIM_NOSWAP;
2222 } else
2223mem_over_limit=mem_cgroup_from_res_counter(fail_res, res);When hit memory.limit_in_bytes,
res_counter_charge()will return-ENOMEM, this will execute line 2222: } else.
But I think when hit memory.limit_in_bytes, the function should determine further to memory.memsw.limit_in_bytes.
This think is OK?I don't think so. We have an invariant (hard limit is "stronger" than memsw limit) memory.limit_in_bytes <= memory.memsw.limit_in_bytes so when we hit the hard limit we do not have to consider memsw because resource counter:
a) we already have to do reclaim for hard limit
b) we check whether we might swap out later on inmem_cgroup_hierarchical_reclaim(root_memcg->memsw_is_minimum) so we will not end up swapping just to make hard limit ok and go over memsw limit.
Quoted 9 lines
Quoted 3 lines
Please also note that we will retry charging after reclaim if there is a chance to meet the limit.
Makes sense?Yeah.
But I want to test memory.memsw.failcnt is nonzero, how steps? Thanks.
Here is a quick hacked test program. see below.
A rough test.
[root@bluextal memcg_test]# cgcreate -g memory:X
[root@bluextal memcg_test]# cgset -r memory.limit_in_bytes=200M X [root@bluextal memcg_test]# cgset -r memory.memsw.limit_in_bytes=300M X [root@bluextal memcg_test]# cgexec -g memory:X ./check 200 300 [root@bluextal memcg_test]# echo 0 > /cgroup/memory/X/memory.memsw.failcnt [root@bluextal memcg_test]# cat /cgroup/memory/X/memory.memsw.failcnt
0
[root@bluextal memcg_test]# cgexec -g memory:X ./check 200 300
Killed <-----------------------------------------------------------------------OOM Killed. [root@bluextal memcg_test]# cat /cgroup/memory/X/memory.memsw.failcnt
17 <-----------------------------------------------------------------------memsw failcnt up.
Easy way is
1. allocate memory in Anon.
2. kick out anon memory to swap as much as possible by file I/O.-------(*1)
3. delete file cache by some way (I used unlink() here.) --------------(*2)
4. allocate anon memory.
The important points are (*1) and (*2). see a program below.
You can prevent OOM (freeze-at-oom) by [root@bluextal memcg_test]# cgset -r memory.oom_control=1 X
Here is the memory.stat at OOM.
[root@bluextal test]# cat /cgroup/memory/X/memory.stat cache 0
rss 209666048mapped_file 0
pgpgin 30567
pgpgout 72381
swap 104906752
<snip>hierarchical_memory_limit 209715200 hierarchical_memsw_limit 314572800
rss+cache < memory.limit
rss+swap == memsw.limit.
==
#include <stdio.h>
#include <sys/types.h>
#include <sys/stat.h>
#include <fcntl.h>
#include <unistd.h>
#include <sys/mman.h>
#include <string.h>
int main(int argc, char *argv[])
{
char filename[] = "./tmpfile-for-test";
unsigned long mem_size = atoi(argv[1]);
unsigned long memsw_size = atoi(argv[2]);
unsigned long file_size;
int fd, len;
char *addr, *buf;
if (memsw_size < 100)
return 0;
mem_size *= 1024 * 1024;
memsw_size *= 1024 * 1024;
memsw_size = memsw_size - 10 * 1024 * 1024; /* 10M Bytes of margin */
addr = mmap(NULL, memsw_size, PROT_READ|PROT_WRITE,
MAP_PRIVATE | MAP_ANONYMOUS, 0, 0);
/* allocate pages and cause swap out */
memset(addr, 0, memsw_size);
/* create file, this will make more swaps. */
file_size = mem_size * 80 / 100;
fd = open(filename, O_RDWR| O_TRUNC, 0644);
buf = malloc(1024 *1024);
for (len = 0; len < file_size; len += 1024*1024) {
write(fd, buf, 1024*1024);
}
/* read the file again */
lseek(fd, SEEK_SET, 0);
for (len = 0; len < file_size; len += 1024 * 1024)
read(fd, buf, 1024 * 1024);
lseek(fd, SEEK_SET, 0);
for (len = 0; len < file_size; len += 1024 * 1024)
read(fd, buf, 1024 * 1024);
unlink(filename);
addr = malloc(9 * 1024 * 1024);
memset(addr, 0, 9 * 1024 * 1024);
printf("done\n");
sleep(100);
}Michal Hocko said the following on 2012-1-4 0:04:
Quoted 45 lines · Peng Haitao
Quoted 21 lines
memory.memsw.failcnt shows the number of memory+Swap hits limits.
So I think when memory+swap usage is equal to limit, memsw.failcnt should be nonzero.I test as follows:
# uname -a Linux K-test 3.2.0-rc7-17-g371de6e #2 SMP Wed Dec 28 12:02:52 CST 2011 x86_64 x86_64 x86_64 GNU/Linux # mkdir /cgroup/memory/group # cd /cgroup/memory/group/ # echo 10M > memory.limit_in_bytes # echo 10M > memory.memsw.limit_in_bytes # echo $$ > tasks # dd if=/dev/zero of=/tmp/temp_file count=20 bs=1M Killed # cat memory.memsw.failcnt 0 # grep "failcnt" /var/log/messages | tail -2 Dec 28 17:05:52 K-test kernel: memory: usage 10240kB, limit 10240kB, failcnt 21 Dec 28 17:05:52 K-test kernel: memory+swap: usage 10240kB, limit 10240kB, failcnt 0Quoted 2 lines
memory+swap usage is equal to limit, but memsw.failcnt is zero.
Please note that memsw.limit_in_bytes is triggered only if we have consumed some swap space already (and the feature is primarily intended to stop extensive swap usage in fact).
It goes like this: If we trigger hard limit (memory.limit_in_bytes) then we start the direct reclaim (with swap available). If we trigger memsw limit then we try to reclaim without swap available. We will OOM if we cannot reclaim enough to satisfy the respective limit.The other part of the answer is, yes there is something wrong going on her because we definitely shouldn't OOM. The workload is a single threaded and we have a plenty of page cache that could be reclaimed easily. On the other hand we end up with:
# echo $$ > tasks /dev/memctl/a# echo 10M > memory.limit_in_bytes /dev/memctl/a# echo 10M > memory.memsw.limit_in_bytes /dev/memctl/a# dd if=/dev/zero of=/tmp/temp_file count=20 bs=1M Killed /dev/memctl/a# cat memory.stat cache 9265152 [...]
Quoted 5 lines
So there is almost 10M of page cache that we can simply reclaim. If we use 40M limit then we are OK. So this looks like the small limit somehow tricks our math in the reclaim path and we think there is nothing to reclaim.
I will look into this.
Have any conclusion for this?
Thanks.
Signature · 2 lines
Best Regards, Peng
Quoted 47 lines · Peng Haitao
Michal Hocko said the following on 2012-1-4 0:04:
Quoted 45 lines · Peng Haitao
Quoted 21 lines
memory.memsw.failcnt shows the number of memory+Swap hits limits.
So I think when memory+swap usage is equal to limit, memsw.failcnt should be nonzero.I test as follows:
# uname -a Linux K-test 3.2.0-rc7-17-g371de6e #2 SMP Wed Dec 28 12:02:52 CST 2011 x86_64 x86_64 x86_64 GNU/Linux # mkdir /cgroup/memory/group # cd /cgroup/memory/group/ # echo 10M > memory.limit_in_bytes # echo 10M > memory.memsw.limit_in_bytes # echo $$ > tasks # dd if=/dev/zero of=/tmp/temp_file count=20 bs=1M Killed # cat memory.memsw.failcnt 0 # grep "failcnt" /var/log/messages | tail -2 Dec 28 17:05:52 K-test kernel: memory: usage 10240kB, limit 10240kB, failcnt 21 Dec 28 17:05:52 K-test kernel: memory+swap: usage 10240kB, limit 10240kB, failcnt 0Quoted 2 lines
memory+swap usage is equal to limit, but memsw.failcnt is zero.
Please note that memsw.limit_in_bytes is triggered only if we have consumed some swap space already (and the feature is primarily intended to stop extensive swap usage in fact).
It goes like this: If we trigger hard limit (memory.limit_in_bytes) then we start the direct reclaim (with swap available). If we trigger memsw limit then we try to reclaim without swap available. We will OOM if we cannot reclaim enough to satisfy the respective limit.The other part of the answer is, yes there is something wrong going on her because we definitely shouldn't OOM. The workload is a single threaded and we have a plenty of page cache that could be reclaimed easily. On the other hand we end up with:
# echo $$ > tasks /dev/memctl/a# echo 10M > memory.limit_in_bytes /dev/memctl/a# echo 10M > memory.memsw.limit_in_bytes /dev/memctl/a# dd if=/dev/zero of=/tmp/temp_file count=20 bs=1M Killed /dev/memctl/a# cat memory.stat cache 9265152 [...]
Quoted 7 lines
Quoted 5 lines
So there is almost 10M of page cache that we can simply reclaim. If we use 40M limit then we are OK. So this looks like the small limit somehow tricks our math in the reclaim path and we think there is nothing to reclaim.
I will look into this.Have any conclusion for this?
I am sorry, but I didn't get to this. The last two months were really busy and I am leaving for a long vacation next week. It's still on my todo list...
Quoted 1 line
Thanks.
Signature · 13 lines
Michal Hocko SUSE Labs SUSE LINUX s.r.o. Lihovarska 1060/12 190 00 Praha 9 Czech Republic -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/ Don't email: <a href=mailto:"[email protected]"> [email protected] </a>