thp: too much anonymous hugepage caused 'khugepaged' thread stopped

BUG? Zhouping Liu 2 participants 6 messages Apr 7, 2013 1 message not in the archive Memory management

lore

Share & tools

lore

A message that is not in the archive (sent off-list or not yet received): [email protected]

Zhouping Liu

author

2013-04-07 06:29 UTC #1 lore raw

Hello All,

When I did some testing to check thp's performance, the following strange action occurred:

when a process try to allocate 500+(or other large value) anonymous hugepage, the 'khugepaged' thread will stop to scan vma. the testing system has 2Gb RAM, and the thp enabled value is 'always', set 0 to 'scan_sleep_millisecs'

you can use the following steps to confirm the issue:

---------------- example code ------------
/* file test_thp.c */

#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <sys/mman.h>

int main(int argc, char *argv[])

{
	int nr_thps = 1000, ret = 0;
	unsigned long hugepagesize, size;
	void *addr;

	hugepagesize = (1UL << 21);

	if (argc == 2)
		nr_thps = atoi(argv[1]);

	printf("try to allocate %d transparent hugepages\n", nr_thps);
	size = (unsigned long)nr_thps * hugepagesize;

	ret = posix_memalign(&addr, hugepagesize, size);
	if (ret != 0) {
		printf("posix_memalign failed\n");
		return ret;
	}

	memset (addr, 10, size);

	sleep(50);

	return ret;
}

-------- end example code -----------

executing './test_thp 500' in a system with 2GB RAM, the values in /sys/kernel/mm/transparent_hugepage/khugepaged/* will never change,
you can repeatedly do '# cat /sys/kernel/mm/transparent_hugepage/khugepaged/*' to check this.

as we know, when we set 0 to /sys/kernel/mm/transparent_hugepage/khugepaged/scan_sleep_millisecs, the /sys/kernel/mm/transparent_hugepage/khugepaged/full_to_scans will increasing at least, but the actual is opposite, the value is never change, so I checked 'khugepaged' thread, and found the 'khugepaged' is stopped:

# ps aux | grep -i hugepaged
root        67 10.9  0.0      0     0 ?        SN   Apr06 172:10 [khugepaged]
                                               ^^ 
also I did the same actions on some large machine, e.g on 16Gb RAM, 1000+ anonymous hugepages
will cause 'khugepaged' stopped, but there are 2Gb+ free memory, why is it? is that normal?
comments?

Signature · 8 lines

Thanks,
Zhouping

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Wanpeng Li

2013-04-07 07:10 UTC #2 lore raw

Quoted 56 lines · Zhouping Liu

Hello All,
When I did some testing to check thp's performance, the following strange action occurred:
when a process try to allocate 500+(or other large value) anonymous hugepage, the 'khugepaged' thread will stop to scan vma. the testing system has 2Gb RAM, and the thp enabled value is 'always', set 0 to 'scan_sleep_millisecs'
you can use the following steps to confirm the issue:
---------------- example code ------------
/* file test_thp.c */
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <sys/mman.h>
int main(int argc, char *argv[])
{
	int nr_thps = 1000, ret = 0;
	unsigned long hugepagesize, size;
	void *addr;

	hugepagesize = (1UL << 21);

	if (argc == 2)
		nr_thps = atoi(argv[1]);

	printf("try to allocate %d transparent hugepages\n", nr_thps);
	size = (unsigned long)nr_thps * hugepagesize;

	ret = posix_memalign(&addr, hugepagesize, size);
	if (ret != 0) {
		printf("posix_memalign failed\n");
		return ret;
	}

	memset (addr, 10, size);

	sleep(50);

	return ret;
}
-------- end example code -----------
executing './test_thp 500' in a system with 2GB RAM, the values in /sys/kernel/mm/transparent_hugepage/khugepaged/* will never change,
you can repeatedly do '# cat /sys/kernel/mm/transparent_hugepage/khugepaged/*' to check this.
as we know, when we set 0 to /sys/kernel/mm/transparent_hugepage/khugepaged/scan_sleep_millisecs, the /sys/kernel/mm/transparent_hugepage/khugepaged/full_to_scans will increasing at least, but the actual is opposite, the value is never change, so I checked 'khugepaged' thread, and found the 'khugepaged' is stopped:

Quoted 6 lines

# ps aux | grep -i hugepaged
root        67 10.9  0.0      0     0 ?        SN   Apr06 172:10 [khugepaged]
                                              ^^ 
also I did the same actions on some large machine, e.g on 16Gb RAM, 1000+ anonymous hugepages
will cause 'khugepaged' stopped, but there are 2Gb+ free memory, why is it? is that normal?
comments?

khugepaged will preallocate one hugepage in NUMA case or alloc one hugepage before collapse in UMA case. If the memory is serious fragmentation and can't successfully allocate hugepage, khugepaged will go to sleep one minute. scan_sleep_millisecs determines how many milliseconds to wait in khugepaged between each pass, however, alloc_sleep_millisecs(default value is one minute) determines how many milliseconds to wait in khugepaged if there's an hugepage allocation failure to throttle the next allocation attempt.

Regards,
Wanpeng Li

Quoted 10 lines

Signature · 9 lines

-- 
Thanks,
Zhouping

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Zhouping Liu

author

2013-04-07 07:47 UTC #3 lore raw

Quoted 63 lines · Wanpeng Li

Quoted 56 lines · Zhouping Liu
Hello All,
When I did some testing to check thp's performance, the following strange action occurred:
when a process try to allocate 500+(or other large value) anonymous hugepage, the 'khugepaged' thread will stop to scan vma. the testing system has 2Gb RAM, and the thp enabled value is 'always', set 0 to 'scan_sleep_millisecs'
you can use the following steps to confirm the issue:
---------------- example code ------------
/* file test_thp.c */
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <sys/mman.h>
int main(int argc, char *argv[])
{
	int nr_thps = 1000, ret = 0;
	unsigned long hugepagesize, size;
	void *addr;

	hugepagesize = (1UL << 21);

	if (argc == 2)
		nr_thps = atoi(argv[1]);

	printf("try to allocate %d transparent hugepages\n", nr_thps);
	size = (unsigned long)nr_thps * hugepagesize;

	ret = posix_memalign(&addr, hugepagesize, size);
	if (ret != 0) {
		printf("posix_memalign failed\n");
		return ret;
	}

	memset (addr, 10, size);

	sleep(50);

	return ret;
}
-------- end example code -----------
executing './test_thp 500' in a system with 2GB RAM, the values in /sys/kernel/mm/transparent_hugepage/khugepaged/* will never change,
you can repeatedly do '# cat /sys/kernel/mm/transparent_hugepage/khugepaged/*' to check this.
as we know, when we set 0 to /sys/kernel/mm/transparent_hugepage/khugepaged/scan_sleep_millisecs, the /sys/kernel/mm/transparent_hugepage/khugepaged/full_to_scans will increasing at least, but the actual is opposite, the value is never change, so I checked 'khugepaged' thread, and found the 'khugepaged' is stopped:
Quoted 6 lines
# ps aux | grep -i hugepaged
root        67 10.9  0.0      0     0 ?        SN   Apr06 172:10 [khugepaged]
                                               ^^
also I did the same actions on some large machine, e.g on 16Gb RAM, 1000+ anonymous hugepages
will cause 'khugepaged' stopped, but there are 2Gb+ free memory, why is it? is that normal?
comments?

Quoted 8 lines

khugepaged will preallocate one hugepage in NUMA case or alloc one hugepage before collapse in UMA case. If the memory is serious fragmentation and can't successfully allocate hugepage, khugepaged will go to sleep one minute. scan_sleep_millisecs determines how many milliseconds to wait in khugepaged between each pass, however, alloc_sleep_millisecs(default value is one minute) determines how many milliseconds to wait in khugepaged if there's an hugepage allocation failure to throttle the next allocation attempt.

I see, I only set scan_sleep_millisecs, didn't care alloc_sleep_millisecs value. yes, you are right, allocating memory failure made khugepaged stopped in my case.

Thanks very much :)

Zhouping

Wanpeng Li

2013-04-07 08:33 UTC #4 lore raw

In reply to Wanpeng Li

Quoted 64 lines · Wanpeng Li

Quoted 56 lines · Zhouping Liu
Hello All,
When I did some testing to check thp's performance, the following strange action occurred:
when a process try to allocate 500+(or other large value) anonymous hugepage, the 'khugepaged' thread will stop to scan vma. the testing system has 2Gb RAM, and the thp enabled value is 'always', set 0 to 'scan_sleep_millisecs'
you can use the following steps to confirm the issue:
---------------- example code ------------
/* file test_thp.c */
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <sys/mman.h>
int main(int argc, char *argv[])
{
	int nr_thps = 1000, ret = 0;
	unsigned long hugepagesize, size;
	void *addr;

	hugepagesize = (1UL << 21);

	if (argc == 2)
		nr_thps = atoi(argv[1]);

	printf("try to allocate %d transparent hugepages\n", nr_thps);
	size = (unsigned long)nr_thps * hugepagesize;

	ret = posix_memalign(&addr, hugepagesize, size);
	if (ret != 0) {
		printf("posix_memalign failed\n");
		return ret;
	}

	memset (addr, 10, size);

	sleep(50);

	return ret;
}
-------- end example code -----------
executing './test_thp 500' in a system with 2GB RAM, the values in /sys/kernel/mm/transparent_hugepage/khugepaged/* will never change,
you can repeatedly do '# cat /sys/kernel/mm/transparent_hugepage/khugepaged/*' to check this.
as we know, when we set 0 to /sys/kernel/mm/transparent_hugepage/khugepaged/scan_sleep_millisecs, the /sys/kernel/mm/transparent_hugepage/khugepaged/full_to_scans will increasing at least, but the actual is opposite, the value is never change, so I checked 'khugepaged' thread, and found the 'khugepaged' is stopped:
Quoted 6 lines
# ps aux | grep -i hugepaged
root        67 10.9  0.0      0     0 ?        SN   Apr06 172:10 [khugepaged]
                                              ^^ 
also I did the same actions on some large machine, e.g on 16Gb RAM, 1000+ anonymous hugepages
will cause 'khugepaged' stopped, but there are 2Gb+ free memory, why is it? is that normal?
comments?

Quoted 2 lines

khugepaged will preallocate one hugepage in NUMA case or alloc one hugepage before collapse in UMA case. If the memory is serious

Sorry, it should be reverse. khugepaged will preallocate one hugepage in UMA case and alloc one hugepage before collapse in NUMA case.

Regards,
Wanpeng Li

Quoted 21 lines

fragmentation and can't successfully allocate hugepage, khugepaged will go to sleep one minute. scan_sleep_millisecs determines how many milliseconds to wait in khugepaged between each pass, however, alloc_sleep_millisecs(default value is one minute) determines how many milliseconds to wait in khugepaged if there's an hugepage allocation failure to throttle the next allocation attempt.
Regards,
Wanpeng Li
Quoted 10 lines
Signature · 9 lines
-- 
Thanks,
Zhouping

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Quoted 5 lines

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Wanpeng Li

2013-04-07 08:33 UTC #5 lore raw

In reply to Wanpeng Li

Quoted 64 lines · Wanpeng Li

Quoted 56 lines · Zhouping Liu
Hello All,
When I did some testing to check thp's performance, the following strange action occurred:
when a process try to allocate 500+(or other large value) anonymous hugepage, the 'khugepaged' thread will stop to scan vma. the testing system has 2Gb RAM, and the thp enabled value is 'always', set 0 to 'scan_sleep_millisecs'
you can use the following steps to confirm the issue:
---------------- example code ------------
/* file test_thp.c */
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <sys/mman.h>
int main(int argc, char *argv[])
{
	int nr_thps = 1000, ret = 0;
	unsigned long hugepagesize, size;
	void *addr;

	hugepagesize = (1UL << 21);

	if (argc == 2)
		nr_thps = atoi(argv[1]);

	printf("try to allocate %d transparent hugepages\n", nr_thps);
	size = (unsigned long)nr_thps * hugepagesize;

	ret = posix_memalign(&addr, hugepagesize, size);
	if (ret != 0) {
		printf("posix_memalign failed\n");
		return ret;
	}

	memset (addr, 10, size);

	sleep(50);

	return ret;
}
-------- end example code -----------
executing './test_thp 500' in a system with 2GB RAM, the values in /sys/kernel/mm/transparent_hugepage/khugepaged/* will never change,
you can repeatedly do '# cat /sys/kernel/mm/transparent_hugepage/khugepaged/*' to check this.
as we know, when we set 0 to /sys/kernel/mm/transparent_hugepage/khugepaged/scan_sleep_millisecs, the /sys/kernel/mm/transparent_hugepage/khugepaged/full_to_scans will increasing at least, but the actual is opposite, the value is never change, so I checked 'khugepaged' thread, and found the 'khugepaged' is stopped:
Quoted 6 lines
# ps aux | grep -i hugepaged
root        67 10.9  0.0      0     0 ?        SN   Apr06 172:10 [khugepaged]
                                              ^^ 
also I did the same actions on some large machine, e.g on 16Gb RAM, 1000+ anonymous hugepages
will cause 'khugepaged' stopped, but there are 2Gb+ free memory, why is it? is that normal?
comments?

Quoted 2 lines

khugepaged will preallocate one hugepage in NUMA case or alloc one hugepage before collapse in UMA case. If the memory is serious

Sorry, it should be reverse. khugepaged will preallocate one hugepage in UMA case and alloc one hugepage before collapse in NUMA case.

Regards,
Wanpeng Li

Quoted 21 lines

fragmentation and can't successfully allocate hugepage, khugepaged will go to sleep one minute. scan_sleep_millisecs determines how many milliseconds to wait in khugepaged between each pass, however, alloc_sleep_millisecs(default value is one minute) determines how many milliseconds to wait in khugepaged if there's an hugepage allocation failure to throttle the next allocation attempt.
Regards,
Wanpeng Li
Quoted 10 lines
Signature · 9 lines
-- 
Thanks,
Zhouping

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Quoted 5 lines

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Wanpeng Li

2013-04-07 07:11 UTC #6 lore raw

In reply to Zhouping Liu

Quoted 56 lines · Zhouping Liu

Hello All,
When I did some testing to check thp's performance, the following strange action occurred:
when a process try to allocate 500+(or other large value) anonymous hugepage, the 'khugepaged' thread will stop to scan vma. the testing system has 2Gb RAM, and the thp enabled value is 'always', set 0 to 'scan_sleep_millisecs'
you can use the following steps to confirm the issue:
---------------- example code ------------
/* file test_thp.c */
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <sys/mman.h>
int main(int argc, char *argv[])
{
	int nr_thps = 1000, ret = 0;
	unsigned long hugepagesize, size;
	void *addr;

	hugepagesize = (1UL << 21);

	if (argc == 2)
		nr_thps = atoi(argv[1]);

	printf("try to allocate %d transparent hugepages\n", nr_thps);
	size = (unsigned long)nr_thps * hugepagesize;

	ret = posix_memalign(&addr, hugepagesize, size);
	if (ret != 0) {
		printf("posix_memalign failed\n");
		return ret;
	}

	memset (addr, 10, size);

	sleep(50);

	return ret;
}
-------- end example code -----------
executing './test_thp 500' in a system with 2GB RAM, the values in /sys/kernel/mm/transparent_hugepage/khugepaged/* will never change,
you can repeatedly do '# cat /sys/kernel/mm/transparent_hugepage/khugepaged/*' to check this.
as we know, when we set 0 to /sys/kernel/mm/transparent_hugepage/khugepaged/scan_sleep_millisecs, the /sys/kernel/mm/transparent_hugepage/khugepaged/full_to_scans will increasing at least, but the actual is opposite, the value is never change, so I checked 'khugepaged' thread, and found the 'khugepaged' is stopped:

Quoted 6 lines

# ps aux | grep -i hugepaged
root        67 10.9  0.0      0     0 ?        SN   Apr06 172:10 [khugepaged]
                                              ^^ 
also I did the same actions on some large machine, e.g on 16Gb RAM, 1000+ anonymous hugepages
will cause 'khugepaged' stopped, but there are 2Gb+ free memory, why is it? is that normal?
comments?

Regards,
Wanpeng Li

Quoted 10 lines

Signature · 9 lines

-- 
Thanks,
Zhouping

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>