Transparent Hugepage Support #17
Subsystems (30)
Quoted 10 lines · Theodore Tso
Quoted 3 lines · Mel Gorman
Does this clarify why
min_free_kbyteshelps and why the "recommended" value is what it is?Thanks, this is really helpful. I wonder if it might be a good idea to have a boot command-line option which automatically sets vm.min_free_kbytes to the right value?
I considered automatically adjusting it the first time huge pages are used, as a command-line option or even a magic value writting to proc. It's trivial to implement each option, just haven't gotten around to doing it. There was less pressure once the tool existed.
Quoted 3 lines
Most administrators who are used to using hugepages,
are most familiar with needing to set boot command-line options, and this way they won't need to try to find this new userspace utility.
The utility covers a host of other use cases as well e.g. creates mount points, sets quota, sizes pools (both static and dynamic), reports on the current state of the system, can auto tune shmem settings etc.
Quoted 2 lines
I was looking
for hugeadm on Ubuntu, for example, and I couldn't find it.
It's relatively recent and there isn't debian packaging for it (although an old one was sent to debian mentors once upon a time but never finished). It's on the TODO list of infinite woe to finish that packaging and go through Debian so it ends up in Ubuntu eventually.
Signature · 9 lines
Mel Gorman Part-time Phd Student Linux Technology Center University of Limerick IBM Dublin Software Lab -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Quoted 8 lines · Theodore Ts'o
Quoted 3 lines · Mel Gorman
Does this clarify why
min_free_kbyteshelps and why the "recommended" value is what it is?Thanks, this is really helpful. I wonder if it might be a good idea to have a boot command-line option which automatically sets vm.min_free_kbytes to the right value? Most administrators who are used to using hugepages, are most familiar with needing to set boot command-line options, and this way they won't need to try to find this new userspace utility. I was looking for hugeadm on Ubuntu, for example, and I couldn't find it.
It's part of libhugetlbfs. I also suggested in a earlier email this would better be "echo 1
Quoted 1 line
/sys/kernel/vm/set-recommended-min_free_kbytes" or
set-recommended-min_free_kbytes=1 at boot considering it's a 10 liner piece of code that does the math to set it. But it's no big deal on my side, the important thing is that we have that feature.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Quoted 10 lines · Linus Torvalds
Quoted 6 lines · Pekka Enberg
AFAIK, most modern GCs split memory in young and old generation "zones" and _copy_ surviving objects from the former to the latter if their lifetime exceeds some threshold. The JVM keeps scanning the smaller young generation very aggressively which causes TLB pressure and scans the larger old generation less often.
Quoted 4 lines
.. my only input to this is: numbers talk, bullsh*t walks.
I'm not interested in micro-benchmarks, either. I can show infinite TLB walk improvement in a microbenchmark.
Ok, I'll bite. I should be able to get some database workloads with hugepages, transparent hugepages, and without any hugepages at all.
-chris
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Quoted 11 lines · Chris Mason
Quoted 9 lines · Linus Torvalds
Quoted 6 lines · Pekka Enberg
AFAIK, most modern GCs split memory in young and old generation "zones" and _copy_ surviving objects from the former to the latter if their lifetime exceeds some threshold. The JVM keeps scanning the smaller young generation very aggressively which causes TLB pressure and scans the larger old generation less often.
Quoted 8 lines
Quoted 5 lines
.. my only input to this is: numbers talk, bullsh*t walks.
I'm not interested in micro-benchmarks, either. I can show infinite TLB walk improvement in a microbenchmark.
Ok, I'll bite. I should be able to get some database workloads with hugepages, transparent hugepages, and without any hugepages at all.
Please run them in conjunction with Mel Gorman's memory compaction, otherwise fragmentation may prevent huge pages from being instantiated.
Signature · 7 lines
Do not meddle in the internals of kernels, for they are subtle and quick to panic. -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Quoted 3 lines · Avi Kivity
Please run them in conjunction with Mel Gorman's memory compaction, otherwise fragmentation may prevent huge pages from being instantiated.
.. and then please run them in conjunction with somebody doing "make -j16" on the kernel at the same time, or just generally doing real work for a few days before hand.
The point is, there are benchmarks, and then there is real life. If we _know_ some feature only works for benchmarks, it should be discounted as such. It's like a compiler that is tuned for specint - at some point the numbers lose a lot of their meaning.
Linus
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Quoted 7 lines · Linus Torvalds
Quoted 3 lines · Avi Kivity
Please run them in conjunction with Mel Gorman's memory compaction, otherwise fragmentation may prevent huge pages from being instantiated.
Quoted 8 lines
.. and then please run them in conjunction with somebody doing "make -j16" on the kernel at the same time, or just generally doing real work for a few days before hand.
The point is, there are benchmarks, and then there is real life. If we _know_ some feature only works for benchmarks, it should be discounted as such. It's like a compiler that is tuned for specint - at some point the numbers lose a lot of their meaning.
Sure, I'll do my best to be brutal. Avi, Andrea please fire off to me a git tree or patch bomb for benchmarking. Please include all the patches you think it needs to go fast, including any config hints etc...
If you'd like numbers with and without a given set of patches, just let me know.
-chris
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Quoted 17 lines · Avi Kivity
Quoted 16 lines · Chris Mason
Quoted 14 lines · Linus Torvalds
Quoted 6 lines · Pekka Enberg
AFAIK, most modern GCs split memory in young and old generation "zones" and _copy_ surviving objects from the former to the latter if their lifetime exceeds some threshold. The JVM keeps scanning the smaller young generation very aggressively which causes TLB pressure and scans the larger old generation less often.
.. my only input to this is: numbers talk, bullsh*t walks.
I'm not interested in micro-benchmarks, either. I can show infinite TLB walk improvement in a microbenchmark.
Quoted 7 lines
Quoted 3 lines
Ok, I'll bite. I should be able to get some database workloads with hugepages, transparent hugepages, and without any hugepages at all.
Please run them in conjunction with Mel Gorman's memory compaction, otherwise fragmentation may prevent huge pages from being instantiated.
Strictly speaking, compaction is not necessary to allocate huge pages. What compaction gets you is
o Lower latency and cost of huge page allocation
o Works on swapless systems
What is important is that you run
hugeadm --set-recommended-min_free_kbytes
from the libhugetlbfs 2.8 package early in boot so that anti-fragmentation is doing as good as job as possible. If one is very curious, use the mm_page_alloc_extfrag to trace how often severe fragmentation-related events occur under default settings and with min_free_kbytes set properly.
Without the compaction patches, allocating huge pages will be occasionally *very* expensive as a large number of pages will need to be reclaimed. Most likely sympton is trashing while the database starts up. Allocation success rates will also be lower when under heavy load.
Running make -j16 at the same time is unlikely to make much of a difference from a hugepage allocation point of view. The performance figures will vary significantly of course as make competes with the database for CPU time and other resources.
Finally, benchmarking with databases is not new as such -
http://lwn.net/Articles/378641/ . This was on fairly simple hardware though as I didn't have access to hardware more suitable for database workloads. If you are running with transparent huge pages though, be sure to double check that huge pages are actually being used transparently.
Signature · 9 lines
Mel Gorman Part-time Phd Student Linux Technology Center University of Limerick IBM Dublin Software Lab -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Quoted 29 lines · Mel Gorman
Quoted 17 lines · Avi Kivity
Quoted 16 lines · Chris Mason
Quoted 14 lines · Linus Torvalds
Quoted 6 lines · Pekka Enberg
AFAIK, most modern GCs split memory in young and old generation "zones" and _copy_ surviving objects from the former to the latter if their lifetime exceeds some threshold. The JVM keeps scanning the smaller young generation very aggressively which causes TLB pressure and scans the larger old generation less often.
.. my only input to this is: numbers talk, bullsh*t walks.
I'm not interested in micro-benchmarks, either. I can show infinite TLB walk improvement in a microbenchmark.
Quoted 7 lines
Quoted 3 lines
Ok, I'll bite. I should be able to get some database workloads with hugepages, transparent hugepages, and without any hugepages at all.
Please run them in conjunction with Mel Gorman's memory compaction, otherwise fragmentation may prevent huge pages from being instantiated.
Strictly speaking, compaction is not necessary to allocate huge pages. What compaction gets you is
Quoted 7 lines
o Lower latency and cost of huge page allocation
o Works on swapless systemsWhat is important is that you run
hugeadm --set-recommended-min_free_kbytes
from the libhugetlbfs 2.8 package early in boot so that anti-fragmentation is doing as good as job as possible.
Great, I'll make sure to do this.
Quoted 10 lines
If one is very
curious, use themm_page_alloc_extfragto trace how often severe fragmentation-related events occur under default settings and withmin_free_kbytesset properly.Without the compaction patches, allocating huge pages will be occasionally *very* expensive as a large number of pages will need to be reclaimed. Most likely sympton is trashing while the database starts up. Allocation success rates will also be lower when under heavy load.
Quoted 4 lines
Running make -j16 at the same time is unlikely to make much of a difference from a hugepage allocation point of view. The performance figures will vary significantly of course as make competes with the database for CPU time and other resources.
Heh, Linus did actually say to run them concurrently with make -j16, but I read it as make -j16 before the database run. My goal will be to fragment the ram, then get a db in ram and see how fast it all goes.
Fragmenting memory during the run is only interesting to test compaction, I'd throw out the resulting db benchmark numbers and only count the number of transparent hugepages we were able to allocate.
Quoted 7 lines
Finally, benchmarking with databases is not new as such -
http://lwn.net/Articles/378641/ . This was on fairly simple hardware though as I didn't have access to hardware more suitable for database workloads. If you are running with transparent huge pages though, be sure to double check that huge pages are actually being used transparently.
Will do. It'll take me a few days to get the machines setup and a baseline measurement.
-chris
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>