hugetlb: introduce nr_overcommit_hugepages sysctl

Revert "hugetlb: Add hugetlb_dynamic_pool sysctl"
This reverts commit 54f9f80d6543fb7b157d3b11e2e7911dc1379790. Given the new sysctl nr_overcommit_hugepages, the boolean dynamic pool sysctl is not needed, as its semantics can be expressed by 0 in the overcommit sysctl (no dynamic pool) and non-0 in the overcommit sysctl (pool enabled).

Signed-off-by Nishanth Aravamudan <[email protected]>

Quoted 7 lines · Nishanth Aravamudan

Revert "hugetlb: Add hugetlb_dynamic_pool sysctl"
This reverts commit 54f9f80d6543fb7b157d3b11e2e7911dc1379790. Given the new sysctl nr_overcommit_hugepages, the boolean dynamic pool sysctl is not needed, as its semantics can be expressed by 0 in the overcommit sysctl (no dynamic pool) and non-0 in the overcommit sysctl (pool enabled).

Signed-off-by Nishanth Aravamudan <[email protected]>

This is recent enough that dependencies shouldn't have developed, but it'd be nice to stage user-visible API/ABI changes more consciously and carefully in the future. Or at least we should try to avoid the sorts of situations where we end up changing recently introduced user/kernel ABI's and API's shortly after merging. We'll run the risk of getting stuck with a user/kernel ABI we can't abandon for years on account of not fixing it up before dependencies develop if this happens too often.

Quoted 8 lines

This is recent enough that dependencies shouldn't have developed, but it'd be nice to stage user-visible API/ABI changes more consciously and carefully in the future. Or at least we should try to avoid the sorts of situations where we end up changing recently introduced user/kernel ABI's and API's shortly after merging. We'll run the risk of getting stuck with a user/kernel ABI we can't abandon for years on account of not fixing it up before dependencies develop if this happens too often.

I agree and I apologize if I'm making things hard for everyone. However, I hadn't fully considered the implications of the dynamic pool for my other patches. The patches moved from -mm to -linus rather quickly, iirc. No excuse, however, I should have been paying more attention. If folks really don't want things to change, I guess we could also just make the sysctl's per-node corresponding attribute be a boolean too. It just seems less flexible than this approach.

Revert "hugetlb: Add hugetlb_dynamic_pool sysctl"

This reverts commit 54f9f80d6543fb7b157d3b11e2e7911dc1379790.

Given the new sysctl nr_overcommit_hugepages, the boolean dynamic pool sysctl is not needed, as its semantics can be expressed by 0 in the overcommit sysctl (no dynamic pool) and non-0 in the overcommit sysctl (pool enabled).

Signed-off-by Nishanth Aravamudan <[email protected]>

While examining the code to support /proc/sys/vm/hugetlb_dynamic_pool, I became convinced that having a boolean sysctl was insufficient:

1) To support per-node control of hugepages, I have previously submitted patches to add a sysfs attribute related to nr_hugepages. However, with a boolean global value and per-mount quota enforcement constraining the dynamic pool, adding corresponding control of the dynamic pool on a per-node basis seems inconsistent to me.

Quoted 8 lines · Nishanth Aravamudan

While examining the code to support /proc/sys/vm/hugetlb_dynamic_pool, I became convinced that having a boolean sysctl was insufficient:

1) To support per-node control of hugepages, I have previously submitted patches to add a sysfs attribute related to nr_hugepages. However, with a boolean global value and per-mount quota enforcement constraining the dynamic pool, adding corresponding control of the dynamic pool on a per-node basis seems inconsistent to me.

Documentation/sysctl, please :)

Quoted 10 lines · Dave Hansen

Quoted 8 lines · Nishanth Aravamudan

While examining the code to support /proc/sys/vm/hugetlb_dynamic_pool, I became convinced that having a boolean sysctl was insufficient:

1) To support per-node control of hugepages, I have previously submitted patches to add a sysfs attribute related to nr_hugepages. However, with a boolean global value and per-mount quota enforcement constraining the dynamic pool, adding corresponding control of the dynamic pool on a per-node basis seems inconsistent to me.

Quoted 1 line

Documentation/sysctl, please :)

Err, yes, will need to updated that. I note that the old sysctl is not there...nor is nr_hugepages, for that matter. So maybe I'll just add a 3rd patch to fix the Documentation? I really just wanted to get the patches out there as soon as I got them tested...

Hrm, nr_hugepages is documented in vm/hugetlbpage.txt and not sysctl/vm.txt Should I document this sysctl there too?

Err, yes, will need to updated that. I note that the old sysctl is not there...nor is nr_hugepages, for that matter. So maybe I'll just add a 3rd patch to fix the Documentation? I really just wanted to get the patches out there as soon as I got them tested...

Quoted 4 lines · Nishanth Aravamudan

Err, yes, will need to updated that. I note that the old sysctl is not there...nor is nr_hugepages, for that matter. So maybe I'll just add a 3rd patch to fix the Documentation? I really just wanted to get the patches out there as soon as I got them tested...

Yeah, that should be fine. Adding nr_hugepages will probably get you bonus points. :)

Quoted 8 lines · Dave Hansen

Quoted 4 lines · Nishanth Aravamudan

Err, yes, will need to updated that. I note that the old sysctl is not there...nor is nr_hugepages, for that matter. So maybe I'll just add a 3rd patch to fix the Documentation? I really just wanted to get the patches out there as soon as I got them tested...

Yeah, that should be fine. Adding nr_hugepages will probably get you bonus points. :)

Documentation: updated hugetlb information

Documentation/vm/hugetlbpage.txt or failure of allocation depends on the amount of physically contiguous · line 97

Quoted 7 lines · Nishanth Aravamudan

Documentation/vm/hugetlbpage.txt +7 -0

+Caveat: Shrinking the pool via nr_hugepages while a surplus is in effect
+will allow the number of surplus huge pages to exceed the overcommit
+value, as the pool hugepages (which must have been in use for a surplus
+hugepages to be allocated) will become surplus hugepages.  As long as
+this condition holds, however, no more surplus huge pages will be
+allowed on the system until one of the two sysctls are increased
+sufficiently, or the surplus huge pages go out of use and are freed.

I guess you could, in theory, disallow the writes to the sysctl and return -EINVAL or -ENOSPC or something. But, I think documenting it like this is probably OK by itself and is pretty sane behavior given the circumstances.

Quoted 8 lines · Dave Hansen

Quoted 4 lines · Nishanth Aravamudan

Err, yes, will need to updated that. I note that the old sysctl is not there...nor is nr_hugepages, for that matter. So maybe I'll just add a 3rd patch to fix the Documentation? I really just wanted to get the patches out there as soon as I got them tested...

Yeah, that should be fine. Adding nr_hugepages will probably get you bonus points. :)

Documentation: updated hugetlb information

The hugetlb documentation has gotten a bit out of sync with the current code. Updated the sysctl file to refer to Documentation/vm/hugetlbpage.txt. Update that file to contain the current state of affairs (with the newer named sysctl in place).

Signed-off-by Nishanth Aravamudan <[email protected]>

---
This is 3/3 because it depends on 1/2 and 2/2 ... Not sure if this is complete enough, either. Adam, do you have any input?

Documentation/sysctl/vm.txt +19 -0

index b89570c..6f31f0a 100644
@@ -34,6 +34,8 @@ Currently, these files are in /proc/sys/vm:
 - oom_kill_allocating_task
 - mmap_min_address
 - numa_zonelist_order
+- nr_hugepages
+- nr_overcommit_hugepages
 
 ==============================================================
 

@@ -305,3 +307,20 @@ will select "node" order in following case.
 
 Otherwise, "zone" order will be selected. Default order is recommended unless
 this is causing problems for your system/application.
+
+==============================================================
+
+nr_hugepages
+
+Change the minimum size of the hugepage pool.
+
+See Documentation/vm/hugetlbpage.txt
+
+==============================================================
+
+nr_overcommit_hugepages
+
+Change the maximum size of the hugepage pool. The maximum is
+nr_hugepages + nr_overcommit_hugepages.
+
+See Documentation/vm/hugetlbpage.txt

Documentation/vm/hugetlbpage.txt +29 -6

index 51ccc48..f962d01 100644
@@ -30,9 +30,10 @@ alignment and size of the arguments to the above system calls.
 The output of "cat /proc/meminfo" will have lines like:
 
.....
-HugePages_Total: xxx
-HugePages_Free:  yyy
-HugePages_Rsvd:  www
+HugePages_Total: vvv
+HugePages_Free:  www
+HugePages_Rsvd:  xxx
+HugePages_Surp:  yyy
 Hugepagesize:    zzz kB
 
 where:

@@ -42,6 +43,10 @@ allocated.
 HugePages_Rsvd is short for "reserved," and is the number of hugepages
 for which a commitment to allocate from the pool has been made, but no
 allocation has yet been made. It's vaguely analogous to overcommit.
+HugePages_Surp is short for "surplus," and is the number of hugepages in
+the pool above the value in /proc/sys/vm/nr_hugepages. The maximum
+number of surplus hugepages is controlled by
+/proc/sys/vm/nr_overcommit_hugepages.
 
 /proc/filesystems should also show a filesystem of type "hugetlbfs" configured
 in the kernel.

@@ -71,7 +76,25 @@ or failure of allocation depends on the amount of physically contiguous
 memory that is preset in system at this time.  System administrators may want
 to put this command in one of the local rc init files.  This will enable the
 kernel to request huge pages early in the boot process (when the possibility
-of getting physical contiguous pages is still very high).
+of getting physical contiguous pages is still very high). In either
+case, adminstrators will want to verify the number of hugepages actually
+allocated by checking the sysctl or meminfo.
+
+/proc/sys/vm/nr_overcommit_hugepages indicates how large the pool of
+hugepages can grow, if more hugepages than /proc/sys/vm/nr_hugepages are
+requested by applications. echo'ing any non-zero value into this file
+indicates that the hugetlb subsystem is allowed to try to obtain
+hugepages from the buddy allocator, if the normal pool is exhausted. As
+these surplus hugepages go out of use, they are freed back to the buddy
+allocator.
+
+Caveat: Shrinking the pool via nr_hugepages while a surplus is in effect
+will allow the number of surplus huge pages to exceed the overcommit
+value, as the pool hugepages (which must have been in use for a surplus
+hugepages to be allocated) will become surplus hugepages.  As long as
+this condition holds, however, no more surplus huge pages will be
+allowed on the system until one of the two sysctls are increased
+sufficiently, or the surplus huge pages go out of use and are freed.
 
 If the user applications are going to request hugepages using mmap system
 call, then it is required that system administrator mount a file system of

@@ -94,8 +117,8 @@ provided on command line then no limits are set.  For size and nr_inodes
 options, you can use [G|g]/[M|m]/[K|k] to represent giga/mega/kilo. For
 example, size=2K has the same meaning as size=2048.
 
-read and write system calls are not supported on files that reside on hugetlb
-file systems.
+While read system calls are supported on files that reside on hugetlb
+file systems, write system calls are not.
 
 Regular chown, chgrp, and chmod commands (with right permissions) could be
 used to change the file attributes on hugetlbfs.

hugetlb: introduce nr_overcommit_hugepages sysctl

kernel/sysctl.c +0 -0

index 8ac5171..b85a128 100644
@@ -912,6 +912,14 @@ static struct ctl_table vm_table[] = {
 		.mode		= 0644,

Signed-off-by Nishanth Aravamudan <[email protected]>

---
Andrew, since 2.6.24 would be the first release with the dynamic_pool sysctl, I think these might deserve consideration for inclusion, even so late in the cycle? It all depends on how close to an (important?) user-space visible change removing the sysctl might be in 2.6.25? Obviously pending comments, acks, nacks from Adam et al.