THP: AnonHugePages in /proc/[pid]/smaps is correct or not?

DISCUSSION Zhouping Liu 4 participants 10 messages Apr 1, 2013 – Apr 2, 2013 1 message not in the archive

Subsystems (3)

Memory management x86 x86 MM

lore

Share & tools

lore

A message that is not in the archive (sent off-list or not yet received): [email protected]

Zhouping Liu

author

2013-04-01 08:00 UTC #1 lore raw

Hi all,

I found THP can't correctly distinguish one anonymous hugepage map.

1. when /sys/kernel/mm/transparent_hugepage/enabled is 'always', the amount of THP always is one less.

Testing code:
---- snip --------
unsigned long hugepagesize = (1UL << 21);

int main()

{
	void *addr;
	int i;

	printf("pid is %d\n", getpid());

	for (i = 0; i < 5; i++) {
		addr = mmap(NULL, hugepagesize, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANON, -1, 0);

		if (addr == MAP_FAILED) {
			perror("mmap");
			return -1;
		}

		memset(addr, i, hugepagesize);
	}

	sleep(50);

	return 0;
}

------ snip ----------

the /proc/[pid]/smaps show that Anonymous is 10240kB but AnonHugePages is 8192Kb, one THP less:
----- snip --------
7f59ccc01000-7f59cd601000 rw-p 00000000 00:00 0

Size:              10240 kB
Rss:               10240 kB
Pss:               10240 kB
Shared_Clean:          0 kB
Shared_Dirty:          0 kB
Private_Clean:         0 kB
Private_Dirty:     10240 kB
Referenced:        10240 kB
Anonymous:         10240 kB
AnonHugePages:      8192 kB
Swap:                  0 kB
KernelPageSize:        4 kB
MMUPageSize:           4 kB
Locked:                0 kB
VmFlags: rd wr mr mw me ac hg

------- sinp ---------

2. when /sys/kernel/mm/transparent_hugepage/enabled is 'madvise', THP can't distinguish any one anonymous hugepage size:

Testing code:
-------- snip --------
unsigned long hugepagesize = (1UL << 21);

int main()

{
	void *addr;
	int i;

	printf("pid is %d\n", getpid());

	for (i = 0; i < 5; i++) {
		addr = mmap(NULL, hugepagesize, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANON, -1, 0);

		if (addr == MAP_FAILED) {
			perror("mmap");
			return -1;
		}

		if (madvise(addr, hugepagesize, MADV_HUGEPAGE) == -1) {
			perror("madvise");
			return -1;
		}

		memset(addr, i, hugepagesize);
	}

	sleep(50);

	return 0;
}

--------- snip ----------

The result is that it can't find any AnonHugePages from /proc/[pid]/smaps :
-------------- snip -------
7f0b38cd0000-7f0b396d0000 rw-p 00000000 00:00 0

Size:              10240 kB
Rss:               10240 kB
Pss:               10240 kB
Shared_Clean:          0 kB
Shared_Dirty:          0 kB
Private_Clean:         0 kB
Private_Dirty:     10240 kB
Referenced:        10240 kB
Anonymous:         10240 kB
AnonHugePages:         0 kB
Swap:                  0 kB
KernelPageSize:        4 kB
MMUPageSize:           4 kB
Locked:                0 kB
VmFlags: rd wr mr mw me ac

----------- snip ----------

3. when I made the address aligned with HUGEPAGESIZE using 'posix_memalign()' instead of mmap(), THP perform good, and can distinguish all anonymous huge pages.

my question is:
1. all the above behaviour is right?
2. why THP can't distinguish one naturally aligned huge page(generated by mmap())?

Signature · 8 lines

Thanks,
Zhouping

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

David Rientjes

2013-04-01 22:23 UTC #2 lore raw

Quoted 7 lines · Zhouping Liu

Hi all,
I found THP can't correctly distinguish one anonymous hugepage map.
1. when /sys/kernel/mm/transparent_hugepage/enabled is 'always', the amount of THP always is one less.

It's not a problem with identifying an anonymous mapping as a hugepage, setting thp enabled to "always" does not guarantee that they will always be allocatable or that your mmap() will be 2MB aligned. Your sample code is using mmap() instead of posix_memalign() so you'll probably only get 100% hugepages only 1/512th of the time.

Quoted 39 lines

2. when /sys/kernel/mm/transparent_hugepage/enabled is 'madvise', THP can't distinguish any one anonymous hugepage size:
Testing code:
-------- snip --------
unsigned long hugepagesize = (1UL << 21);
int main()
{
	void *addr;
	int i;

	printf("pid is %d\n", getpid());

	for (i = 0; i < 5; i++) {
		addr = mmap(NULL, hugepagesize, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANON, -1, 0);

		if (addr == MAP_FAILED) {
			perror("mmap");
			return -1;
		}

		if (madvise(addr, hugepagesize, MADV_HUGEPAGE) == -1) {
			perror("madvise");
			return -1;
		}

		memset(addr, i, hugepagesize);
	}

	sleep(50);

	return 0;
}
--------- snip ----------
The result is that it can't find any AnonHugePages from /proc/[pid]/smaps :
-------------- snip -------
7f0b38cd0000-7f0b396d0000 rw-p 00000000 00:00 0

Quoted 15 lines

Size:              10240 kB
Rss:               10240 kB
Pss:               10240 kB
Shared_Clean:          0 kB
Shared_Dirty:          0 kB
Private_Clean:         0 kB
Private_Dirty:     10240 kB
Referenced:        10240 kB
Anonymous:         10240 kB
AnonHugePages:         0 kB
Swap:                  0 kB
KernelPageSize:        4 kB
MMUPageSize:           4 kB
Locked:                0 kB
VmFlags: rd wr mr mw me ac

"hg" would be shown in VmFlags if your MADV_HUGEPAGE was successful, are you sure this is the right vma?

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Zhouping Liu

author

2013-04-02 03:11 UTC #3 lore raw

Quoted 6 lines · David Rientjes

Quoted 4 lines · Zhouping Liu
Hi all,
I found THP can't correctly distinguish one anonymous hugepage map.

Quoted 8 lines

Quoted 3 lines
1. when /sys/kernel/mm/transparent_hugepage/enabled is 'always', the amount of THP always is one less.
It's not a problem with identifying an anonymous mapping as a hugepage, setting thp enabled to "always" does not guarantee that they will always be allocatable or that your mmap() will be 2MB aligned. Your sample code is using mmap() instead of posix_memalign() so you'll probably only get 100% hugepages only 1/512th of the time.

I don't understand clearly the last sentence 'you'll probably only get 100% hugepages only 1/512th of the time.' could you please explain more details about 'only 1/512th of the time'?

Quoted 55 lines

Quoted 40 lines
2. when /sys/kernel/mm/transparent_hugepage/enabled is 'madvise', THP can't distinguish any one anonymous hugepage size:
Testing code:
-------- snip --------
unsigned long hugepagesize = (1UL << 21);
int main()
{
	void *addr;
	int i;

	printf("pid is %d\n", getpid());

	for (i = 0; i < 5; i++) {
		addr = mmap(NULL, hugepagesize, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANON, -1, 0);

		if (addr == MAP_FAILED) {
			perror("mmap");
			return -1;
		}

		if (madvise(addr, hugepagesize, MADV_HUGEPAGE) == -1) {
			perror("madvise");
			return -1;
		}

		memset(addr, i, hugepagesize);
	}

	sleep(50);

	return 0;
}
--------- snip ----------
The result is that it can't find any AnonHugePages from /proc/[pid]/smaps :
-------------- snip -------
7f0b38cd0000-7f0b396d0000 rw-p 00000000 00:00 0
Quoted 15 lines
Size:              10240 kB
Rss:               10240 kB
Pss:               10240 kB
Shared_Clean:          0 kB
Shared_Dirty:          0 kB
Private_Clean:         0 kB
Private_Dirty:     10240 kB
Referenced:        10240 kB
Anonymous:         10240 kB
AnonHugePages:         0 kB
Swap:                  0 kB
KernelPageSize:        4 kB
MMUPageSize:           4 kB
Locked:                0 kB
VmFlags: rd wr mr mw me ac

Quoted 2 lines

"hg" would be shown in VmFlags if your MADV_HUGEPAGE was successful, are you sure this is the right vma?

I think it's the same issue as the above, according to the sample code, it does mmap() five times in total, and each time it map 2MB anonymous maps, as all the maps maybe aren't 2MB aligned, so "AnonHugePages" show 0 kB, and no "hg" VmFalgs.

so, again, if I understand correctly, thp should tune the naturally aligned maps, such as generated by mmap()/malloc(), make such maps 'hugepagesize' aligned if the maps or vma is equal and greater than 'hugepagesize', doesn't it?

Thanks,
Zhouping

Lin Feng

2013-04-02 03:38 UTC #4 lore raw

Hi Zhouping,

Quoted 2 lines · Zhouping Liu

I don't understand clearly the last sentence 'you'll probably only get 100% hugepages only 1/512th of the time.' could you please explain more details about 'only 1/512th of the time'?

IIUC, thp size is 2M so it may be comprised of 512 normal page(size 4k). Since your test code is not 2M aligned(not using posix_memalign()) so the start address of the mapped vma will be random, such as 2M*i+4k*1, 2M*i+4k*2...2M*i+k4*511, there is 512 possibilities.

The only chance you get thp happens when the first map just starts at 2M*i, and the consequent maps also benefit from this.

-------- snip --------

Quoted 3 lines

so, again, if I understand correctly, thp should tune the naturally aligned maps, such as generated by mmap()/malloc(), make such maps 'hugepagesize' aligned if the maps or vma is equal and greater than 'hugepagesize', doesn't it?

We may gain performance improving from this.

thanks,
linfeng

David Rientjes

2013-04-02 03:44 UTC #5 lore raw

Quoted 5 lines · Lin Feng

Quoted 2 lines
so, again, if I understand correctly, thp should tune the naturally aligned maps, such as generated by mmap()/malloc(), make such maps 'hugepagesize' aligned if the maps or vma is equal and greater than 'hugepagesize', doesn't it?
We may gain performance improving from this.

To attain the maximum number of hugepages, you would naturally want to ensure that the mappings are done aligned to 2MB; for very large allocations, missing one or two hugepages typically won't hurt performance much. posix_memalign() is the best way of doing this which just wraps mmap() for the needed alignment. More interesting is creating your own custom malloc() that allocates in 2MB aligned chunks, if possible, and uses 2MB aligned arenas for its own metadata. If you do that for malloc(), then you'll only need to make code that does its own mmap()s to use posix_memalign().

Zhouping Liu

author

2013-04-02 03:50 UTC #6 lore raw

In reply to Lin Feng

Quoted 5 lines · Lin Feng

Hi Zhouping,
Quoted 2 lines · Zhouping Liu
I don't understand clearly the last sentence 'you'll probably only get 100% hugepages only 1/512th of the time.' could you please explain more details about 'only 1/512th of the time'?

Quoted 7 lines

IIUC, thp size is 2M so it may be comprised of 512 normal page(size 4k). Since your test code is not 2M aligned(not using posix_memalign()) so the start address of the mapped vma will be random, such as 2M*i+4k*1, 2M*i+4k*2...2M*i+k4*511, there is 512 possibilities.
The only chance you get thp happens when the first map just starts at 2M*i, and the consequent maps also benefit from this.

Feng, it's easy to understand now, thanks for your detailed explanation :)

Zhouping

Simon Jeons

2013-04-02 12:23 UTC #7 lore raw

In reply to David Rientjes

Hi David,

Quoted 4 lines · David Rientjes

Quoted 2 lines · Zhouping Liu
Hi all,

Quoted 8 lines

Quoted 5 lines
I found THP can't correctly distinguish one anonymous hugepage map.
1. when /sys/kernel/mm/transparent_hugepage/enabled is 'always', the amount of THP always is one less.
It's not a problem with identifying an anonymous mapping as a hugepage, setting thp enabled to "always" does not guarantee that they will always be allocatable or that your mmap() will be 2MB aligned. Your sample code

Both thp and hugetlb pages should be 2MB aligned, correct?

Quoted 57 lines

is using mmap() instead of posix_memalign() so you'll probably only get 100% hugepages only 1/512th of the time.
Quoted 39 lines
2. when /sys/kernel/mm/transparent_hugepage/enabled is 'madvise', THP can't distinguish any one anonymous hugepage size:
Testing code:
-------- snip --------
unsigned long hugepagesize = (1UL << 21);
int main()
{
	void *addr;
	int i;

	printf("pid is %d\n", getpid());

	for (i = 0; i < 5; i++) {
		addr = mmap(NULL, hugepagesize, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANON, -1, 0);

		if (addr == MAP_FAILED) {
			perror("mmap");
			return -1;
		}

		if (madvise(addr, hugepagesize, MADV_HUGEPAGE) == -1) {
			perror("madvise");
			return -1;
		}

		memset(addr, i, hugepagesize);
	}

	sleep(50);

	return 0;
}
--------- snip ----------
The result is that it can't find any AnonHugePages from /proc/[pid]/smaps :
-------------- snip -------
7f0b38cd0000-7f0b396d0000 rw-p 00000000 00:00 0
Quoted 15 lines
Size:              10240 kB
Rss:               10240 kB
Pss:               10240 kB
Shared_Clean:          0 kB
Shared_Dirty:          0 kB
Private_Clean:         0 kB
Private_Dirty:     10240 kB
Referenced:        10240 kB
Anonymous:         10240 kB
AnonHugePages:         0 kB
Swap:                  0 kB
KernelPageSize:        4 kB
MMUPageSize:           4 kB
Locked:                0 kB
VmFlags: rd wr mr mw me ac

Quoted 8 lines

"hg" would be shown in VmFlags if your MADV_HUGEPAGE was successful, are you sure this is the right vma?
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

David Rientjes

2013-04-02 18:09 UTC #8 lore raw

Quoted 2 lines · Simon Jeons

Both thp and hugetlb pages should be 2MB aligned, correct?

To answer this question and your followup reply at the same time: they come from one level higher in the page table so they will naturally need to be 2MB aligned.

Simon Jeons

2013-04-02 23:58 UTC #9 lore raw

Hi David,

Quoted 7 lines · David Rientjes

Quoted 2 lines · Simon Jeons
Both thp and hugetlb pages should be 2MB aligned, correct?
To answer this question and your followup reply at the same time: they come from one level higher in the page table so they will naturally need to be 2MB aligned.

When I hacking arch/x86/mm/hugetlbpage.c like this,

arch/x86/mm/hugetlbpage.c +0 -0

index ae1aa71..87f34ee 100644
@@ -354,14 +354,13 @@ hugetlb_get_unmapped_area(struct file *file,

unsigned long addr,

#endif /*HAVE_ARCH_HUGETLB_UNMAPPED_AREA*/

-#ifdef CONFIG_X86_64
static __init int setup_hugepagesz(char *opt)
{
unsigned long ps = memparse(opt, &opt);
if (ps == PMD_SIZE) {
hugetlb_add_hstate(PMD_SHIFT - PAGE_SHIFT);
- } else if (ps == PUD_SIZE && cpu_has_gbpages) {
- hugetlb_add_hstate(PUD_SHIFT - PAGE_SHIFT);
+ } else if (ps == PUD_SIZE) {
+ hugetlb_add_hstate(PMD_SHIFT - PAGE_SHIFT+4);
} else {
printk(KERN_ERR "hugepagesz: Unsupported page size %lu M\n", ps >> 20);

I set boot=hugepagesz=1G hugepages=10, then I got 10 32MB huge pages. What's the difference between these pages which I hacking and normal huge pages?

Simon Jeons

2013-04-02 12:26 UTC #10 lore raw

In reply to David Rientjes

Quoted 4 lines · David Rientjes

Quoted 2 lines · Zhouping Liu
Hi all,

Quoted 8 lines

Quoted 5 lines
I found THP can't correctly distinguish one anonymous hugepage map.
1. when /sys/kernel/mm/transparent_hugepage/enabled is 'always', the amount of THP always is one less.
It's not a problem with identifying an anonymous mapping as a hugepage, setting thp enabled to "always" does not guarantee that they will always be allocatable or that your mmap() will be 2MB aligned. Your sample code

Btw, why need 2MB aligned? Does it has relationship with tlb?

Quoted 58 lines

is using mmap() instead of posix_memalign() so you'll probably only get 100% hugepages only 1/512th of the time.
Quoted 39 lines
2. when /sys/kernel/mm/transparent_hugepage/enabled is 'madvise', THP can't distinguish any one anonymous hugepage size:
Testing code:
-------- snip --------
unsigned long hugepagesize = (1UL << 21);
int main()
{
	void *addr;
	int i;

	printf("pid is %d\n", getpid());

	for (i = 0; i < 5; i++) {
		addr = mmap(NULL, hugepagesize, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANON, -1, 0);

		if (addr == MAP_FAILED) {
			perror("mmap");
			return -1;
		}

		if (madvise(addr, hugepagesize, MADV_HUGEPAGE) == -1) {
			perror("madvise");
			return -1;
		}

		memset(addr, i, hugepagesize);
	}

	sleep(50);

	return 0;
}
--------- snip ----------
The result is that it can't find any AnonHugePages from /proc/[pid]/smaps :
-------------- snip -------
7f0b38cd0000-7f0b396d0000 rw-p 00000000 00:00 0
Quoted 15 lines
Size:              10240 kB
Rss:               10240 kB
Pss:               10240 kB
Shared_Clean:          0 kB
Shared_Dirty:          0 kB
Private_Clean:         0 kB
Private_Dirty:     10240 kB
Referenced:        10240 kB
Anonymous:         10240 kB
AnonHugePages:         0 kB
Swap:                  0 kB
KernelPageSize:        4 kB
MMUPageSize:           4 kB
Locked:                0 kB
VmFlags: rd wr mr mw me ac

Quoted 8 lines

"hg" would be shown in VmFlags if your MADV_HUGEPAGE was successful, are you sure this is the right vma?
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>