Why memory.usage_in_bytes is always increasing after every mmap/dirty/unmap sequence

DISCUSSION bill4carson 3 participants 8 messages Mar 23, 2012 – Mar 27, 2012 Memory management

Share & tools

bill4carson

author

2012-03-23 08:05 UTC #0 lore raw

Hi, all

I'm playing with memory cgroup, I'm a bit confused why memory.usage in bytes is steadily increasing at 4K page pace after every mmap/dirty/unmap sequence.

On linux-3.6.34.10/linux-3.3.0-rc5
A simple test case does following:

a) mmap 128k memory in private anonymous way
b) dirty all 128k to demand physical page
c) print memory.usage_in_bytes <-- increased at 4K after every loop
d) unmap previous 128 memory
e) goto a) to repeat

And when the test case exit, memory.usage_in_bytes is not *ZERO*, but the previous increased value.

I'm puzzled about what I saw, can anyone please give me some tips to understand this?

Thanks in advance.

Signature · 10 lines

Love each day!

--bill

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

KAMEZAWA Hiroyuki

2012-03-23 08:20 UTC #1 lore raw

Quoted 6 lines · (2012/03/23 17:04), bill4carson wrote:

Hi, all
I'm playing with memory cgroup, I'm a bit confused why memory.usage in bytes is steadily increasing at 4K page pace after every mmap/dirty/unmap sequence.

Quoted 8 lines

On linux-3.6.34.10/linux-3.3.0-rc5
A simple test case does following:
a) mmap 128k memory in private anonymous way
b) dirty all 128k to demand physical page
c) print memory.usage_in_bytes <-- increased at 4K after every loop
d) unmap previous 128 memory
e) goto a) to repeat

In Documentation/cgroup/memory.txt
==
5.5 usage_in_bytes

For efficiency, as other kernel components, memory cgroup uses some optimization to avoid unnecessary cacheline false sharing. usage_in_bytes is affected by the method and doesn't show 'exact' value of memory(and swap) usage, it's an fuzz value for efficient access. (Of course, when necessary, it's synchronized.) If you want to know more exact memory usage, you should use RSS+CACHE(+SWAP) value in memory.stat(see 5.2).
==

In current implementation, memcg tries to charge resource in size of 32 pages. So, if you get 32 pages and free 32pages, usage_in_bytes may not change. This is affected by caches in other cpus and other flushing operations caused by some workload in other cgroups. memcg's usage_in_bytes is not precise in 128k degree.

- How memory.stat changes ?
- What happens when you do test with 4M alloc/free ?

Thanks,
-Kame

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

bill4carson

author

2012-03-23 08:44 UTC #2 lore raw

Quoted 29 lines · KAMEZAWA Hiroyuki

Quoted 6 lines · (2012/03/23 17:04), bill4carson wrote:
Hi, all
I'm playing with memory cgroup, I'm a bit confused why memory.usage in bytes is steadily increasing at 4K page pace after every mmap/dirty/unmap sequence.
Quoted 8 lines
On linux-3.6.34.10/linux-3.3.0-rc5
A simple test case does following:
a) mmap 128k memory in private anonymous way
b) dirty all 128k to demand physical page
c) print memory.usage_in_bytes<-- increased at 4K after every loop
d) unmap previous 128 memory
e) goto a) to repeat
In Documentation/cgroup/memory.txt
==
5.5 usage_in_bytes
For efficiency, as other kernel components, memory cgroup uses some optimization to avoid unnecessary cacheline false sharing. usage_in_bytes is affected by the method and doesn't show 'exact' value of memory(and swap) usage, it's an fuzz value for efficient access. (Of course, when necessary, it's synchronized.) If you want to know more exact memory usage, you should use RSS+CACHE(+SWAP) value in memory.stat(see 5.2).
==

Quoted 5 lines

In current implementation, memcg tries to charge resource in size of 32 pages. So, if you get 32 pages and free 32pages, usage_in_bytes may not change. This is affected by caches in other cpus and other flushing operations caused by some workload in other cgroups. memcg's usage_in_bytes is not precise in 128k degree.

Thanks for the internal design details.
I noticed on 2.6.34, it's checked on every 512 Kbytes

See http://lxr.linux.no/#linux+v2.6.34/mm/memcontrol.c#L571

And I haven't see the 3.3.0 changes.

Quoted 2 lines

- How memory.stat changes ?

root@localhost:/sys/fs/cgroup/memory/a> cat memory.stat;cat memory.usage_in_bytes cache 0
rss 131072 <------ when mmap/dirty/
mapped_file 0
pgpgin 1278
pgpgout 1246
inactive_anon 0
active_anon 131072
inactive_file 0
active_file 0
unevictable 0
hierarchical_memory_limit 9223372036854775807
total_cache 0
total_rss 131072
total_mapped_file 0
total_pgpgin 1278
total_pgpgout 1246
total_inactive_anon 0
total_active_anon 131072
total_inactive_file 0
total_active_file 0
total_unevictable 0

root@localhost:/sys/fs/cgroup/memory/a> cat memory.stat;cat memory.usage_in_bytes cache 0
rss 4096 <------ when mmap/dirty/unmap
mapped_file 0
pgpgin 1278
pgpgout 1277
inactive_anon 0
active_anon 4096
inactive_file 0
active_file 0
unevictable 0
hierarchical_memory_limit 9223372036854775807
total_cache 0
total_rss 4096
total_mapped_file 0
total_pgpgin 1278
total_pgpgout 1277
total_inactive_anon 0
total_active_anon 4096
total_inactive_file 0
total_active_file 0
total_unevictable 0

Quoted 2 lines

- What happens when you do test with 4M alloc/free ?

I tried on 2.6.34, it's the same behavior.

Quoted 11 lines

Thanks,
-Kame

Signature · 10 lines

Love each day!

--bill

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

bill4carson

author

2012-03-23 09:00 UTC #3 lore raw

In reply to KAMEZAWA Hiroyuki

Quoted 29 lines · KAMEZAWA Hiroyuki

Quoted 6 lines · (2012/03/23 17:04), bill4carson wrote:
Hi, all
I'm playing with memory cgroup, I'm a bit confused why memory.usage in bytes is steadily increasing at 4K page pace after every mmap/dirty/unmap sequence.
Quoted 8 lines
On linux-3.6.34.10/linux-3.3.0-rc5
A simple test case does following:
a) mmap 128k memory in private anonymous way
b) dirty all 128k to demand physical page
c) print memory.usage_in_bytes<-- increased at 4K after every loop
d) unmap previous 128 memory
e) goto a) to repeat
In Documentation/cgroup/memory.txt
==
5.5 usage_in_bytes
For efficiency, as other kernel components, memory cgroup uses some optimization to avoid unnecessary cacheline false sharing. usage_in_bytes is affected by the method and doesn't show 'exact' value of memory(and swap) usage, it's an fuzz value for efficient access. (Of course, when necessary, it's synchronized.) If you want to know more exact memory usage, you should use RSS+CACHE(+SWAP) value in memory.stat(see 5.2).
==

Quoted 6 lines

In current implementation, memcg tries to charge resource in size of 32 pages. So, if you get 32 pages and free 32pages, usage_in_bytes may not change. This is affected by caches in other cpus and other flushing operations caused by some workload in other cgroups. memcg's usage_in_bytes is not precise in 128k degree.

Yes, I tried to mmap/dirty/unmap in 32 times, when the usage_in_bytes reached 128k, it rolls back to 4k again. So it doesn't hurt any more.

I haven't found the code regarding to this behavior.

Quoted 14 lines

- How memory.stat changes ?
- What happens when you do test with 4M alloc/free ?
Thanks,
-Kame

Signature · 10 lines

Love each day!

--bill

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

KAMEZAWA Hiroyuki

2012-03-23 09:05 UTC #4 lore raw

Quoted 32 lines · (2012/03/23 17:59), bill4carson wrote:

Quoted 29 lines · KAMEZAWA Hiroyuki
Quoted 6 lines · (2012/03/23 17:04), bill4carson wrote:
Hi, all
I'm playing with memory cgroup, I'm a bit confused why memory.usage in bytes is steadily increasing at 4K page pace after every mmap/dirty/unmap sequence.
Quoted 8 lines
On linux-3.6.34.10/linux-3.3.0-rc5
A simple test case does following:
a) mmap 128k memory in private anonymous way
b) dirty all 128k to demand physical page
c) print memory.usage_in_bytes<-- increased at 4K after every loop
d) unmap previous 128 memory
e) goto a) to repeat
In Documentation/cgroup/memory.txt
==
5.5 usage_in_bytes
For efficiency, as other kernel components, memory cgroup uses some optimization to avoid unnecessary cacheline false sharing. usage_in_bytes is affected by the method and doesn't show 'exact' value of memory(and swap) usage, it's an fuzz value for efficient access. (Of course, when necessary, it's synchronized.) If you want to know more exact memory usage, you should use RSS+CACHE(+SWAP) value in memory.stat(see 5.2).
==

Quoted 8 lines

Quoted 6 lines
In current implementation, memcg tries to charge resource in size of 32 pages. So, if you get 32 pages and free 32pages, usage_in_bytes may not change. This is affected by caches in other cpus and other flushing operations caused by some workload in other cgroups. memcg's usage_in_bytes is not precise in 128k degree.
Yes, I tried to mmap/dirty/unmap in 32 times, when the usage_in_bytes reached 128k, it rolls back to 4k again. So it doesn't hurt any more.

rolls back before unmap() ?

Quoted 2 lines

I haven't found the code regarding to this behavior.

Could you post your test program ?

Thanks,
-Kame

bill4carson

author

2012-03-23 09:16 UTC #5 lore raw

Quoted 40 lines · KAMEZAWA Hiroyuki

Quoted 38 lines · (2012/03/23 17:59), bill4carson wrote:
Quoted 35 lines · KAMEZAWA Hiroyuki
Quoted 6 lines · (2012/03/23 17:04), bill4carson wrote:
Hi, all
I'm playing with memory cgroup, I'm a bit confused why memory.usage in bytes is steadily increasing at 4K page pace after every mmap/dirty/unmap sequence.
Quoted 8 lines
On linux-3.6.34.10/linux-3.3.0-rc5
A simple test case does following:
a) mmap 128k memory in private anonymous way
b) dirty all 128k to demand physical page
c) print memory.usage_in_bytes<-- increased at 4K after every loop
d) unmap previous 128 memory
e) goto a) to repeat
In Documentation/cgroup/memory.txt
==
5.5 usage_in_bytes
For efficiency, as other kernel components, memory cgroup uses some optimization to avoid unnecessary cacheline false sharing. usage_in_bytes is affected by the method and doesn't show 'exact' value of memory(and swap) usage, it's an fuzz value for efficient access. (Of course, when necessary, it's synchronized.) If you want to know more exact memory usage, you should use RSS+CACHE(+SWAP) value in memory.stat(see 5.2).
==
In current implementation, memcg tries to charge resource in size of 32 pages. So, if you get 32 pages and free 32pages, usage_in_bytes may not change. This is affected by caches in other cpus and other flushing operations caused by some workload in other cgroups. memcg's usage_in_bytes is not precise in 128k degree.

Quoted 6 lines

Quoted 2 lines
Yes, I tried to mmap/dirty/unmap in 32 times, when the usage_in_bytes reached 128k, it rolls back to 4k again. So it doesn't hurt any more.
rolls back before unmap() ?

After unmap

Quoted 6 lines

Quoted 2 lines
I haven't found the code regarding to this behavior.
Could you post your test program ?

Yes, it's a bit of messy, you can mock at me:)

-------------------------------------------------------------
#include <stdlib.h>
#include <stdio.h>
#include <string.h>
#include <errno.h>
#include <sys/mman.h>
#include <sys/types.h>
#include <unistd.h>
#include <sys/eventfd.h>
#include <fcntl.h>
#include <limits.h>

#define MMAP_SIZE (128*1024)

int * ptr_array[1024];
int depth=0;

char cmd;
int pagesize;
int mapsize;

void get_key(void)
{

	cmd = getchar();
	putchar('\n');
	getchar();
}

int is_this_key(char key)

{
	return (cmd == key) ? 1:0;
}

void getpage(int *ptr)

{
	int i;
	for (i = 0; i < mapsize/4098; i++){
		*ptr = 4;/* alloc a physical page */
		ptr += 1024; /*move to next page*/
	}
}
void show_stat(void)
{
	int i;

	for (i = depth; i--; i < 0){
		printf("[%2d]:%8p\n", i, ptr_array[i]);
	}

}
int main(int argc, char ** argv)
{

	int *ptr;
	int i;
    char usage_in_bytes_path[PATH_MAX];
	char          tasks_path[PATH_MAX];
    char                 tmp[PATH_MAX];
    int usage_in_bytes = -1;
	int fd_tasks = -1;

	char *root_path;
	int ret;

	if (argc > 2)
		printf("Usage: oomtst [map size]\n");

	mapsize = MMAP_SIZE;


	strcpy(tmp, argv[1]);
	root_path = dirname(tmp);
	printf("root_path:%s\n", root_path);

	ret = snprintf(usage_in_bytes_path, PATH_MAX, 
"%s/memory.usage_in_bytes", root_path);
	if (ret >= PATH_MAX) {
		fputs("Path to memory.usage_in_bytes is too long\n", stderr);
		goto out;
	}
	puts(usage_in_bytes_path);

	ret = snprintf(tasks_path, PATH_MAX, "%s/tasks", root_path);
	if (ret >= PATH_MAX) {
		fputs("Path to memory.usage_in_bytes is too long\n", stderr);
		goto out;
	}
	puts(tasks_path);
	
	fd_tasks = open(tasks_path, O_WRONLY);
	if (fd_tasks == -1) {
		fprintf(stderr, "Cannot open %s: %s\n", tasks_path,
				strerror(errno));
		goto out;
	}

	printf("Using PID:%u\n", getpid());
	{
		char tasks_str[32];

		ret = sprintf(tasks_str, "%d", getpid());
		ret = write(fd_tasks, tasks_str, strlen(tasks_str));

	}


	while(1){
		char used_bytes[64];
		uint64_t tmp;

		usage_in_bytes = open(usage_in_bytes_path, O_RDWR);
		if (usage_in_bytes == -1) {
			fprintf(stderr, "Cannot open %s: %s\n", usage_in_bytes_path,
				strerror(errno));
			goto out;
		}



		printf("Enter a command (m: malloc f: free ?: exit):");
		get_key();

		if (is_this_key('m')) {	
			ptr_array[depth] = mmap(NULL, mapsize, PROT_READ | PROT_WRITE, 
MAP_PRIVATE|MAP_ANON, 0, 0);
			if (ptr_array[depth] == NULL){
				perror("msg: cannot malloc\n");
				exit(2);
			}

			printf("malloc %d Kbytes at %p\n", mapsize/1024, ptr_array[depth]);
			getpage(ptr_array[depth]);
			depth++;

		}else if (is_this_key('f')) {
			if (depth == 0)
				break;

			printf("free %d Kbytes at %p\n", mapsize/1024, ptr_array[depth -1]);
			munmap(ptr_array[depth -1], mapsize);
			--depth;

		}else
			break;

		show_stat();


		memset(&used_bytes, 0, 33/*sizeof(used_bytes)*/);
		ret = read(usage_in_bytes, &used_bytes, 32);
		if (ret == -1) {
			perror("Cannot read from usage_in_bytes");
			break;
		}
		tmp = atoll(used_bytes);
        printf("used_bytes:%llu Kbytes\n", tmp/1024);

		if (usage_in_bytes >= 0)
			close(usage_in_bytes);

	}
	
	out:
		for (i = 0; i < depth; i++){
			printf("free %d Kbytes at %p\n", mapsize/1024, ptr_array[i]);
			munmap(ptr_array[i], mapsize);
		}


	if (usage_in_bytes >= 0)
		close(usage_in_bytes);

	return 0;
}

Quoted 4 lines

Thanks,
-Kame

Signature · 10 lines

Love each day!

--bill

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

KAMEZAWA Hiroyuki

2012-03-23 09:59 UTC #6 lore raw

Quoted 53 lines · (2012/03/23 18:15), bill4carson wrote:

Quoted 40 lines · KAMEZAWA Hiroyuki
Quoted 38 lines · (2012/03/23 17:59), bill4carson wrote:
Quoted 35 lines · KAMEZAWA Hiroyuki
Quoted 6 lines · (2012/03/23 17:04), bill4carson wrote:
Hi, all
I'm playing with memory cgroup, I'm a bit confused why memory.usage in bytes is steadily increasing at 4K page pace after every mmap/dirty/unmap sequence.
Quoted 8 lines
On linux-3.6.34.10/linux-3.3.0-rc5
A simple test case does following:
a) mmap 128k memory in private anonymous way
b) dirty all 128k to demand physical page
c) print memory.usage_in_bytes<-- increased at 4K after every loop
d) unmap previous 128 memory
e) goto a) to repeat
In Documentation/cgroup/memory.txt
==
5.5 usage_in_bytes
For efficiency, as other kernel components, memory cgroup uses some optimization to avoid unnecessary cacheline false sharing. usage_in_bytes is affected by the method and doesn't show 'exact' value of memory(and swap) usage, it's an fuzz value for efficient access. (Of course, when necessary, it's synchronized.) If you want to know more exact memory usage, you should use RSS+CACHE(+SWAP) value in memory.stat(see 5.2).
==
In current implementation, memcg tries to charge resource in size of 32 pages. So, if you get 32 pages and free 32pages, usage_in_bytes may not change. This is affected by caches in other cpus and other flushing operations caused by some workload in other cgroups. memcg's usage_in_bytes is not precise in 128k degree.
Quoted 6 lines
Quoted 2 lines
Yes, I tried to mmap/dirty/unmap in 32 times, when the usage_in_bytes reached 128k, it rolls back to 4k again. So it doesn't hurt any more.
rolls back before unmap() ?
After unmap
Quoted 1 line
Quoted 1 line

Quoted 6 lines

Quoted 5 lines
Quoted 1 line
I haven't found the code regarding to this behavior.
Could you post your test program ?
Yes, it's a bit of messy, you can mock at me:)

Thank you. I tried.

I checked /proc/<pid>/smaps and compares it before I press 'm' and after I press 'm'. So, anonymous should be increased 128k.

Then...
==
[root@bluextal kamezawa]# diff -Nru before.txt after.txt | grep Anonymous
Anonymous: 0 kB
-Anonymous: 12 kB
+Anonymous: 140 kB <============= 128kbytes you allocated
-Anonymous: 8 kB
+Anonymous: 12 kB <============= maybe some library's heap.

Then, 4k remains even if you free all allocated area by your program, I think.

Thanks,
-Kame

Glauber Costa

2012-03-27 08:26 UTC #7 lore raw

In reply to bill4carson

Quoted 5 lines · On 03/23/2012 09:59 AM, bill4carson wrote:

Quoted 1 line
Yes, I tried to mmap/dirty/unmap in 32 times, when the usage_in_bytes reached 128k, it rolls back to 4k again. So it doesn't hurt any more.
I haven't found the code regarding to this behavior.

That's actually quite annoying, IMHO.
I personally think that everytime one tries to read from usage, we should flush the caches and show the correct figures, or at least as correct as we can.

That's specially bad because under load, this is wrong by O(#cpus)...

For just reading the file, this might be okay because there is an alternative for it (although not that intuitive), but for the threshold code, we are probably hitting them a lot more than we should in big machines.