libhugetlbfs testsuite failures and OOMs eventually kill my system

bug/regression Jan Stancek 3 participants 13 messages Oct 13, 2016 – Oct 18, 2016

Subsystems (2)

Memory management HugeTLB

lore

Share & tools

lore

Jan Stancek

author

2016-10-13 12:19 UTC #0 lore raw

Hi,

I'm running into ENOMEM failures with libhugetlbfs testsuite [1] on a power8 lpar system running 4.8 or latest git [2]. Repeated runs of this suite trigger multiple OOMs, that eventually kill entire system, it usually takes 3-5 runs:

* Total System Memory......: 18024 MB
* Shared Mem Max Mapping...: 320 MB
* System Huge Page Size....: 16 MB
* Available Huge Pages.....: 20
* Total size of Huge Pages.: 320 MB
* Remaining System Memory..: 17704 MB
* Huge Page User Group.....: hugepages (1001)

I see this only on ppc (BE/LE), x86_64 seems unaffected and successfully ran the tests for ~12 hours.

Bisect has identified following patch as culprit:
commit 67961f9db8c477026ea20ce05761bde6f8bf85b0
Author: Mike Kravetz <[email protected]>
Date: Wed Jun 8 15:33:42 2016 -0700
mm/hugetlb: fix huge page reserve accounting for private mappings

Following patch (made with my limited insight) applied to latest git [2] fixes the problem for me:

mm/hugetlb.c +1 -1

index ec49d9e..7261583 100644
@@ -1876,7 +1876,7 @@ static long __vma_reservation_common(struct hstate *h,
                 * return value of this routine is the opposite of the
                 * value returned from reserve map manipulation routines above.
                 */
-               if (ret)
+               if (ret >= 0)
                        return 0;
                else
                        return 1;

Regards,
Jan

[1] https://github.com/libhugetlbfs/libhugetlbfs
[2] v4.8-14230-gb67be92

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Mike Kravetz

2016-10-13 15:24 UTC #1 lore raw

Quoted 15 lines · Jan Stancek

Hi,
I'm running into ENOMEM failures with libhugetlbfs testsuite [1] on a power8 lpar system running 4.8 or latest git [2]. Repeated runs of this suite trigger multiple OOMs, that eventually kill entire system, it usually takes 3-5 runs:
* Total System Memory......: 18024 MB
* Shared Mem Max Mapping...: 320 MB
* System Huge Page Size....: 16 MB
* Available Huge Pages.....: 20
* Total size of Huge Pages.: 320 MB
* Remaining System Memory..: 17704 MB
* Huge Page User Group.....: hugepages (1001)

Quoted 9 lines

I see this only on ppc (BE/LE), x86_64 seems unaffected and successfully ran the tests for ~12 hours.
Bisect has identified following patch as culprit:
commit 67961f9db8c477026ea20ce05761bde6f8bf85b0
Author: Mike Kravetz <[email protected]>
Date: Wed Jun 8 15:33:42 2016 -0700
mm/hugetlb: fix huge page reserve accounting for private mappings

Thanks Jan, I'll take a look.

Quoted 10 lines

Following patch (made with my limited insight) applied to latest git [2] fixes the problem for me:
mm/hugetlb.c +0 -0
index ec49d9e..7261583 100644

mm/hugetlb.c static long __vma_reservation_common(struct hstate *h, · line 1882

Quoted 8 lines

mm/hugetlb.c +1 -1

@@ -1876,7 +1876,7 @@ static long __vma_reservation_common(struct hstate *h,
                 * return value of this routine is the opposite of the
                 * value returned from reserve map manipulation routines above.
                 */
-               if (ret)
+               if (ret >= 0)
                        return 0;
                else
                        return 1;

Do note that this code is only executed if this condition is true:

	else if (is_vma_resv_set(vma, HPAGE_RESV_OWNER) && ret >= 0) {

So, we would always return 0. This always tells the calling code that a reservation exists.

Signature · 1 line

Mike Kravetz

Quoted 6 lines

Regards,
Jan
[1] https://github.com/libhugetlbfs/libhugetlbfs
[2] v4.8-14230-gb67be92

Mike Kravetz

2016-10-13 23:26 UTC #2 lore raw

Quoted 8 lines · Mike Kravetz

Quoted 7 lines · Jan Stancek
Hi,
I'm running into ENOMEM failures with libhugetlbfs testsuite [1] on a power8 lpar system running 4.8 or latest git [2]. Repeated runs of this suite trigger multiple OOMs, that eventually kill entire system, it usually takes 3-5 runs:

Quoted 8 lines

Quoted 8 lines
* Total System Memory......: 18024 MB
* Shared Mem Max Mapping...: 320 MB
* System Huge Page Size....: 16 MB
* Available Huge Pages.....: 20
* Total size of Huge Pages.: 320 MB
* Remaining System Memory..: 17704 MB
* Huge Page User Group.....: hugepages (1001)

Hi Jan,

Any chance you can get the contents of /sys/kernel/mm/hugepages before and after the first run of libhugetlbfs testsuite on Power? Perhaps a script like:

cd /sys/kernel/mm/hugepages
for f in hugepages-*/*; do

	n=`cat $f`;
	echo -e "$n\t$f";

done

Just want to make sure the numbers look as they should.

Signature · 7 lines

Mike Kravetz

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Jan Stancek

author

2016-10-14 08:48 UTC #3 lore raw

Quoted 7 lines · Mike Kravetz

Hi Jan,
Any chance you can get the contents of /sys/kernel/mm/hugepages before and after the first run of libhugetlbfs testsuite on Power? Perhaps a script like:

Quoted 8 lines

cd /sys/kernel/mm/hugepages
for f in hugepages-*/*; do
	n=`cat $f`;
	echo -e "$n\t$f";
done
Just want to make sure the numbers look as they should.

Hi Mike,

Numbers are below. I have also isolated a single testcase from "func" group of tests: corrupt-by-cow-opt [1]. This test stops working if I run it 19 times (with 20 hugepages). And if I disable this test, "func" group tests can all pass repeatedly.

[1] https://github.com/libhugetlbfs/libhugetlbfs/blob/master/tests/corrupt-by-cow-opt.c

Regards,
Jan

Kernel is v4.8-14230-gb67be92, with reboot between each run.
1) Only func tests
System boot
After setup:

20      hugepages-16384kB/free_hugepages
20      hugepages-16384kB/nr_hugepages
20      hugepages-16384kB/nr_hugepages_mempolicy
0       hugepages-16384kB/nr_overcommit_hugepages
0       hugepages-16384kB/resv_hugepages
0       hugepages-16384kB/surplus_hugepages
0       hugepages-16777216kB/free_hugepages
0       hugepages-16777216kB/nr_hugepages
0       hugepages-16777216kB/nr_hugepages_mempolicy
0       hugepages-16777216kB/nr_overcommit_hugepages
0       hugepages-16777216kB/resv_hugepages
0       hugepages-16777216kB/surplus_hugepages

After func tests:
********** TEST SUMMARY
* 16M
* 32-bit 64-bit

*     Total testcases:     0     85
*             Skipped:     0      0
*                PASS:     0     81
*                FAIL:     0      4
*    Killed by signal:     0      0
*   Bad configuration:     0      0
*       Expected FAIL:     0      0
*     Unexpected PASS:     0      0
* Strange test result:     0      0

26      hugepages-16384kB/free_hugepages
26      hugepages-16384kB/nr_hugepages
26      hugepages-16384kB/nr_hugepages_mempolicy
0       hugepages-16384kB/nr_overcommit_hugepages
1       hugepages-16384kB/resv_hugepages
0       hugepages-16384kB/surplus_hugepages
0       hugepages-16777216kB/free_hugepages
0       hugepages-16777216kB/nr_hugepages
0       hugepages-16777216kB/nr_hugepages_mempolicy
0       hugepages-16777216kB/nr_overcommit_hugepages
0       hugepages-16777216kB/resv_hugepages
0       hugepages-16777216kB/surplus_hugepages

After test cleanup:
umount -a -t hugetlbfs
hugeadm --pool-pages-max ${HPSIZE}:0

1       hugepages-16384kB/free_hugepages
1       hugepages-16384kB/nr_hugepages
1       hugepages-16384kB/nr_hugepages_mempolicy
0       hugepages-16384kB/nr_overcommit_hugepages
1       hugepages-16384kB/resv_hugepages
1       hugepages-16384kB/surplus_hugepages
0       hugepages-16777216kB/free_hugepages
0       hugepages-16777216kB/nr_hugepages
0       hugepages-16777216kB/nr_hugepages_mempolicy
0       hugepages-16777216kB/nr_overcommit_hugepages
0       hugepages-16777216kB/resv_hugepages
0       hugepages-16777216kB/surplus_hugepages

---

2) Only stress tests
System boot
After setup:

20      hugepages-16384kB/free_hugepages
20      hugepages-16384kB/nr_hugepages
20      hugepages-16384kB/nr_hugepages_mempolicy
0       hugepages-16384kB/nr_overcommit_hugepages
0       hugepages-16384kB/resv_hugepages
0       hugepages-16384kB/surplus_hugepages
0       hugepages-16777216kB/free_hugepages
0       hugepages-16777216kB/nr_hugepages
0       hugepages-16777216kB/nr_hugepages_mempolicy
0       hugepages-16777216kB/nr_overcommit_hugepages
0       hugepages-16777216kB/resv_hugepages
0       hugepages-16777216kB/surplus_hugepages

After stress tests:

20      hugepages-16384kB/free_hugepages
20      hugepages-16384kB/nr_hugepages
20      hugepages-16384kB/nr_hugepages_mempolicy
0       hugepages-16384kB/nr_overcommit_hugepages
17      hugepages-16384kB/resv_hugepages
0       hugepages-16384kB/surplus_hugepages
0       hugepages-16777216kB/free_hugepages
0       hugepages-16777216kB/nr_hugepages
0       hugepages-16777216kB/nr_hugepages_mempolicy
0       hugepages-16777216kB/nr_overcommit_hugepages
0       hugepages-16777216kB/resv_hugepages
0       hugepages-16777216kB/surplus_hugepages

After cleanup:

17      hugepages-16384kB/free_hugepages
17      hugepages-16384kB/nr_hugepages
17      hugepages-16384kB/nr_hugepages_mempolicy
0       hugepages-16384kB/nr_overcommit_hugepages
17      hugepages-16384kB/resv_hugepages
17      hugepages-16384kB/surplus_hugepages
0       hugepages-16777216kB/free_hugepages
0       hugepages-16777216kB/nr_hugepages
0       hugepages-16777216kB/nr_hugepages_mempolicy
0       hugepages-16777216kB/nr_overcommit_hugepages
0       hugepages-16777216kB/resv_hugepages
0       hugepages-16777216kB/surplus_hugepages

---

3) only corrupt-by-cow-opt

System boot
After setup:

20      hugepages-16384kB/free_hugepages
20      hugepages-16384kB/nr_hugepages
20      hugepages-16384kB/nr_hugepages_mempolicy
0       hugepages-16384kB/nr_overcommit_hugepages
0       hugepages-16384kB/resv_hugepages
0       hugepages-16384kB/surplus_hugepages
0       hugepages-16777216kB/free_hugepages
0       hugepages-16777216kB/nr_hugepages
0       hugepages-16777216kB/nr_hugepages_mempolicy
0       hugepages-16777216kB/nr_overcommit_hugepages
0       hugepages-16777216kB/resv_hugepages
0       hugepages-16777216kB/surplus_hugepages

libhugetlbfs-2.18# env LD_LIBRARY_PATH=./obj64 ./tests/obj64/corrupt-by-cow-opt; /root/grab.sh Starting testcase "./tests/obj64/corrupt-by-cow-opt", pid 3298
Write s to 0x3effff000000 via shared mapping
Write p to 0x3effff000000 via private mapping
Read s from 0x3effff000000 via shared mapping
PASS

20      hugepages-16384kB/free_hugepages
20      hugepages-16384kB/nr_hugepages
20      hugepages-16384kB/nr_hugepages_mempolicy
0       hugepages-16384kB/nr_overcommit_hugepages
1       hugepages-16384kB/resv_hugepages
0       hugepages-16384kB/surplus_hugepages
0       hugepages-16777216kB/free_hugepages
0       hugepages-16777216kB/nr_hugepages
0       hugepages-16777216kB/nr_hugepages_mempolicy
0       hugepages-16777216kB/nr_overcommit_hugepages
0       hugepages-16777216kB/resv_hugepages
0       hugepages-16777216kB/surplus_hugepages

# env LD_LIBRARY_PATH=./obj64 ./tests/obj64/corrupt-by-cow-opt; /root/grab.sh
Starting testcase "./tests/obj64/corrupt-by-cow-opt", pid 3312
Write s to 0x3effff000000 via shared mapping
Write p to 0x3effff000000 via private mapping
Read s from 0x3effff000000 via shared mapping
PASS
20      hugepages-16384kB/free_hugepages
20      hugepages-16384kB/nr_hugepages
20      hugepages-16384kB/nr_hugepages_mempolicy
0       hugepages-16384kB/nr_overcommit_hugepages
2       hugepages-16384kB/resv_hugepages
0       hugepages-16384kB/surplus_hugepages
0       hugepages-16777216kB/free_hugepages
0       hugepages-16777216kB/nr_hugepages
0       hugepages-16777216kB/nr_hugepages_mempolicy
0       hugepages-16777216kB/nr_overcommit_hugepages
0       hugepages-16777216kB/resv_hugepages
0       hugepages-16777216kB/surplus_hugepages

(... output cut from ~17 iterations ...)

# env LD_LIBRARY_PATH=./obj64 ./tests/obj64/corrupt-by-cow-opt; /root/grab.sh
Starting testcase "./tests/obj64/corrupt-by-cow-opt", pid 3686
Write s to 0x3effff000000 via shared mapping
Bus error
20      hugepages-16384kB/free_hugepages
20      hugepages-16384kB/nr_hugepages
20      hugepages-16384kB/nr_hugepages_mempolicy
0       hugepages-16384kB/nr_overcommit_hugepages
19      hugepages-16384kB/resv_hugepages
0       hugepages-16384kB/surplus_hugepages
0       hugepages-16777216kB/free_hugepages
0       hugepages-16777216kB/nr_hugepages
0       hugepages-16777216kB/nr_hugepages_mempolicy
0       hugepages-16777216kB/nr_overcommit_hugepages
0       hugepages-16777216kB/resv_hugepages
0       hugepages-16777216kB/surplus_hugepages

# env LD_LIBRARY_PATH=./obj64 ./tests/obj64/corrupt-by-cow-opt; /root/grab.sh
Starting testcase "./tests/obj64/corrupt-by-cow-opt", pid 3700
Write s to 0x3effff000000 via shared mapping
FAIL    mmap() 2: Cannot allocate memory
20      hugepages-16384kB/free_hugepages
20      hugepages-16384kB/nr_hugepages
20      hugepages-16384kB/nr_hugepages_mempolicy
0       hugepages-16384kB/nr_overcommit_hugepages
19      hugepages-16384kB/resv_hugepages
0       hugepages-16384kB/surplus_hugepages
0       hugepages-16777216kB/free_hugepages
0       hugepages-16777216kB/nr_hugepages
0       hugepages-16777216kB/nr_hugepages_mempolicy
0       hugepages-16777216kB/nr_overcommit_hugepages
0       hugepages-16777216kB/resv_hugepages
0       hugepages-16777216kB/surplus_hugepages

Mike Kravetz

2016-10-14 23:57 UTC #4 lore raw

Quoted 17 lines · Jan Stancek

Quoted 7 lines · Mike Kravetz
Hi Jan,
Any chance you can get the contents of /sys/kernel/mm/hugepages before and after the first run of libhugetlbfs testsuite on Power? Perhaps a script like:
Quoted 8 lines
cd /sys/kernel/mm/hugepages
for f in hugepages-*/*; do
	n=`cat $f`;
	echo -e "$n\t$f";
done
Just want to make sure the numbers look as they should.

Quoted 6 lines

Hi Mike,
Numbers are below. I have also isolated a single testcase from "func" group of tests: corrupt-by-cow-opt [1]. This test stops working if I run it 19 times (with 20 hugepages). And if I disable this test, "func" group tests can all pass repeatedly.

Thanks Jan,

I appreciate your efforts.

Quoted 54 lines

[1] https://github.com/libhugetlbfs/libhugetlbfs/blob/master/tests/corrupt-by-cow-opt.c

Regards,
Jan

Kernel is v4.8-14230-gb67be92, with reboot between each run.
1) Only func tests
System boot
After setup:

20      hugepages-16384kB/free_hugepages
20      hugepages-16384kB/nr_hugepages
20      hugepages-16384kB/nr_hugepages_mempolicy
0       hugepages-16384kB/nr_overcommit_hugepages
0       hugepages-16384kB/resv_hugepages
0       hugepages-16384kB/surplus_hugepages
0       hugepages-16777216kB/free_hugepages
0       hugepages-16777216kB/nr_hugepages
0       hugepages-16777216kB/nr_hugepages_mempolicy
0       hugepages-16777216kB/nr_overcommit_hugepages
0       hugepages-16777216kB/resv_hugepages
0       hugepages-16777216kB/surplus_hugepages

After func tests:
********** TEST SUMMARY
* 16M
* 32-bit 64-bit

*     Total testcases:     0     85
*             Skipped:     0      0
*                PASS:     0     81
*                FAIL:     0      4
*    Killed by signal:     0      0
*   Bad configuration:     0      0
*       Expected FAIL:     0      0
*     Unexpected PASS:     0      0
* Strange test result:     0      0

26      hugepages-16384kB/free_hugepages
26      hugepages-16384kB/nr_hugepages
26      hugepages-16384kB/nr_hugepages_mempolicy
0       hugepages-16384kB/nr_overcommit_hugepages
1       hugepages-16384kB/resv_hugepages
0       hugepages-16384kB/surplus_hugepages
0       hugepages-16777216kB/free_hugepages
0       hugepages-16777216kB/nr_hugepages
0       hugepages-16777216kB/nr_hugepages_mempolicy
0       hugepages-16777216kB/nr_overcommit_hugepages
0       hugepages-16777216kB/resv_hugepages
0       hugepages-16777216kB/surplus_hugepages

After test cleanup:
umount -a -t hugetlbfs
hugeadm --pool-pages-max ${HPSIZE}:0

Quoted 13 lines

1       hugepages-16384kB/free_hugepages
1       hugepages-16384kB/nr_hugepages
1       hugepages-16384kB/nr_hugepages_mempolicy
0       hugepages-16384kB/nr_overcommit_hugepages
1       hugepages-16384kB/resv_hugepages
1       hugepages-16384kB/surplus_hugepages
0       hugepages-16777216kB/free_hugepages
0       hugepages-16777216kB/nr_hugepages
0       hugepages-16777216kB/nr_hugepages_mempolicy
0       hugepages-16777216kB/nr_overcommit_hugepages
0       hugepages-16777216kB/resv_hugepages
0       hugepages-16777216kB/surplus_hugepages

I am guessing the leaked reserve page is which is triggered by running the test you isolated corrupt-by-cow-opt.

Quoted 33 lines

---

2) Only stress tests
System boot
After setup:

20      hugepages-16384kB/free_hugepages
20      hugepages-16384kB/nr_hugepages
20      hugepages-16384kB/nr_hugepages_mempolicy
0       hugepages-16384kB/nr_overcommit_hugepages
0       hugepages-16384kB/resv_hugepages
0       hugepages-16384kB/surplus_hugepages
0       hugepages-16777216kB/free_hugepages
0       hugepages-16777216kB/nr_hugepages
0       hugepages-16777216kB/nr_hugepages_mempolicy
0       hugepages-16777216kB/nr_overcommit_hugepages
0       hugepages-16777216kB/resv_hugepages
0       hugepages-16777216kB/surplus_hugepages

After stress tests:

20      hugepages-16384kB/free_hugepages
20      hugepages-16384kB/nr_hugepages
20      hugepages-16384kB/nr_hugepages_mempolicy
0       hugepages-16384kB/nr_overcommit_hugepages
17      hugepages-16384kB/resv_hugepages
0       hugepages-16384kB/surplus_hugepages
0       hugepages-16777216kB/free_hugepages
0       hugepages-16777216kB/nr_hugepages
0       hugepages-16777216kB/nr_hugepages_mempolicy
0       hugepages-16777216kB/nr_overcommit_hugepages
0       hugepages-16777216kB/resv_hugepages
0       hugepages-16777216kB/surplus_hugepages

After cleanup:

Quoted 13 lines

17      hugepages-16384kB/free_hugepages
17      hugepages-16384kB/nr_hugepages
17      hugepages-16384kB/nr_hugepages_mempolicy
0       hugepages-16384kB/nr_overcommit_hugepages
17      hugepages-16384kB/resv_hugepages
17      hugepages-16384kB/surplus_hugepages
0       hugepages-16777216kB/free_hugepages
0       hugepages-16777216kB/nr_hugepages
0       hugepages-16777216kB/nr_hugepages_mempolicy
0       hugepages-16777216kB/nr_overcommit_hugepages
0       hugepages-16777216kB/resv_hugepages
0       hugepages-16777216kB/surplus_hugepages

This looks worse than the summary after running the functional tests.

Quoted 25 lines

---
3) only corrupt-by-cow-opt
System boot
After setup:
20      hugepages-16384kB/free_hugepages
20      hugepages-16384kB/nr_hugepages
20      hugepages-16384kB/nr_hugepages_mempolicy
0       hugepages-16384kB/nr_overcommit_hugepages
0       hugepages-16384kB/resv_hugepages
0       hugepages-16384kB/surplus_hugepages
0       hugepages-16777216kB/free_hugepages
0       hugepages-16777216kB/nr_hugepages
0       hugepages-16777216kB/nr_hugepages_mempolicy
0       hugepages-16777216kB/nr_overcommit_hugepages
0       hugepages-16777216kB/resv_hugepages
0       hugepages-16777216kB/surplus_hugepages
libhugetlbfs-2.18# env LD_LIBRARY_PATH=./obj64 ./tests/obj64/corrupt-by-cow-opt; /root/grab.sh Starting testcase "./tests/obj64/corrupt-by-cow-opt", pid 3298
Write s to 0x3effff000000 via shared mapping
Write p to 0x3effff000000 via private mapping
Read s from 0x3effff000000 via shared mapping
PASS

Quoted 12 lines

20      hugepages-16384kB/free_hugepages
20      hugepages-16384kB/nr_hugepages
20      hugepages-16384kB/nr_hugepages_mempolicy
0       hugepages-16384kB/nr_overcommit_hugepages
1       hugepages-16384kB/resv_hugepages
0       hugepages-16384kB/surplus_hugepages
0       hugepages-16777216kB/free_hugepages
0       hugepages-16777216kB/nr_hugepages
0       hugepages-16777216kB/nr_hugepages_mempolicy
0       hugepages-16777216kB/nr_overcommit_hugepages
0       hugepages-16777216kB/resv_hugepages
0       hugepages-16777216kB/surplus_hugepages

Leaked one reserve page

Quoted 11 lines

# env LD_LIBRARY_PATH=./obj64 ./tests/obj64/corrupt-by-cow-opt; /root/grab.sh
Starting testcase "./tests/obj64/corrupt-by-cow-opt", pid 3312
Write s to 0x3effff000000 via shared mapping
Write p to 0x3effff000000 via private mapping
Read s from 0x3effff000000 via shared mapping
PASS
20      hugepages-16384kB/free_hugepages
20      hugepages-16384kB/nr_hugepages
20      hugepages-16384kB/nr_hugepages_mempolicy
0       hugepages-16384kB/nr_overcommit_hugepages

Quoted 8 lines

2       hugepages-16384kB/resv_hugepages
0       hugepages-16384kB/surplus_hugepages
0       hugepages-16777216kB/free_hugepages
0       hugepages-16777216kB/nr_hugepages
0       hugepages-16777216kB/nr_hugepages_mempolicy
0       hugepages-16777216kB/nr_overcommit_hugepages
0       hugepages-16777216kB/resv_hugepages
0       hugepages-16777216kB/surplus_hugepages

It is pretty consistent that we leak a reserve page every time this test is run.

The interesting thing is that corrupt-by-cow-opt is a very simple test case. commit 67961f9db8c4 potentially changes the return value of the functions vma_has_reserves() and vma_needs/commit_reservation() for the owner (HPAGE_RESV_OWNER) of private mappings. running the test with and without the commit results in the same return values for these routines on x86. And, no leaked reserve pages.

Is it possible to revert this commit and run the libhugetlbs tests (func and stress) again while monitoring the counts in /sys? The counts should go to zero after cleanup as you describe above. I just want to make sure that this commit is causing all the problems you are seeing. If it is, then we can consider reverting and I can try to think of another way to address the original issue.

Thanks for your efforts on this. I can not reproduce on x86 or sparc and do not see any similar symptoms on these architectures.

Signature · 1 line

Mike Kravetz

Quoted 38 lines

(... output cut from ~17 iterations ...)

# env LD_LIBRARY_PATH=./obj64 ./tests/obj64/corrupt-by-cow-opt; /root/grab.sh
Starting testcase "./tests/obj64/corrupt-by-cow-opt", pid 3686
Write s to 0x3effff000000 via shared mapping
Bus error
20      hugepages-16384kB/free_hugepages
20      hugepages-16384kB/nr_hugepages
20      hugepages-16384kB/nr_hugepages_mempolicy
0       hugepages-16384kB/nr_overcommit_hugepages
19      hugepages-16384kB/resv_hugepages
0       hugepages-16384kB/surplus_hugepages
0       hugepages-16777216kB/free_hugepages
0       hugepages-16777216kB/nr_hugepages
0       hugepages-16777216kB/nr_hugepages_mempolicy
0       hugepages-16777216kB/nr_overcommit_hugepages
0       hugepages-16777216kB/resv_hugepages
0       hugepages-16777216kB/surplus_hugepages

# env LD_LIBRARY_PATH=./obj64 ./tests/obj64/corrupt-by-cow-opt; /root/grab.sh
Starting testcase "./tests/obj64/corrupt-by-cow-opt", pid 3700
Write s to 0x3effff000000 via shared mapping
FAIL    mmap() 2: Cannot allocate memory
20      hugepages-16384kB/free_hugepages
20      hugepages-16384kB/nr_hugepages
20      hugepages-16384kB/nr_hugepages_mempolicy
0       hugepages-16384kB/nr_overcommit_hugepages
19      hugepages-16384kB/resv_hugepages
0       hugepages-16384kB/surplus_hugepages
0       hugepages-16777216kB/free_hugepages
0       hugepages-16777216kB/nr_hugepages
0       hugepages-16777216kB/nr_hugepages_mempolicy
0       hugepages-16777216kB/nr_overcommit_hugepages
0       hugepages-16777216kB/resv_hugepages
0       hugepages-16777216kB/surplus_hugepages

Quoted 6 lines

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Aneesh Kumar K.V

2016-10-17 05:05 UTC #5 lore raw

Quoted 213 lines · Mike Kravetz

Quoted 17 lines · Jan Stancek

Quoted 7 lines · Mike Kravetz
Hi Jan,
Any chance you can get the contents of /sys/kernel/mm/hugepages before and after the first run of libhugetlbfs testsuite on Power? Perhaps a script like:
Quoted 8 lines
cd /sys/kernel/mm/hugepages
for f in hugepages-*/*; do
	n=`cat $f`;
	echo -e "$n\t$f";
done
Just want to make sure the numbers look as they should.

Quoted 6 lines

Hi Mike,
Numbers are below. I have also isolated a single testcase from "func" group of tests: corrupt-by-cow-opt [1]. This test stops working if I run it 19 times (with 20 hugepages). And if I disable this test, "func" group tests can all pass repeatedly.

Thanks Jan,

I appreciate your efforts.

Quoted 54 lines

[1] https://github.com/libhugetlbfs/libhugetlbfs/blob/master/tests/corrupt-by-cow-opt.c

Regards,
Jan

Kernel is v4.8-14230-gb67be92, with reboot between each run.
1) Only func tests
System boot
After setup:

20      hugepages-16384kB/free_hugepages
20      hugepages-16384kB/nr_hugepages
20      hugepages-16384kB/nr_hugepages_mempolicy
0       hugepages-16384kB/nr_overcommit_hugepages
0       hugepages-16384kB/resv_hugepages
0       hugepages-16384kB/surplus_hugepages
0       hugepages-16777216kB/free_hugepages
0       hugepages-16777216kB/nr_hugepages
0       hugepages-16777216kB/nr_hugepages_mempolicy
0       hugepages-16777216kB/nr_overcommit_hugepages
0       hugepages-16777216kB/resv_hugepages
0       hugepages-16777216kB/surplus_hugepages

After func tests:
********** TEST SUMMARY
* 16M
* 32-bit 64-bit

*     Total testcases:     0     85
*             Skipped:     0      0
*                PASS:     0     81
*                FAIL:     0      4
*    Killed by signal:     0      0
*   Bad configuration:     0      0
*       Expected FAIL:     0      0
*     Unexpected PASS:     0      0
* Strange test result:     0      0

26      hugepages-16384kB/free_hugepages
26      hugepages-16384kB/nr_hugepages
26      hugepages-16384kB/nr_hugepages_mempolicy
0       hugepages-16384kB/nr_overcommit_hugepages
1       hugepages-16384kB/resv_hugepages
0       hugepages-16384kB/surplus_hugepages
0       hugepages-16777216kB/free_hugepages
0       hugepages-16777216kB/nr_hugepages
0       hugepages-16777216kB/nr_hugepages_mempolicy
0       hugepages-16777216kB/nr_overcommit_hugepages
0       hugepages-16777216kB/resv_hugepages
0       hugepages-16777216kB/surplus_hugepages

After test cleanup:
umount -a -t hugetlbfs
hugeadm --pool-pages-max ${HPSIZE}:0

Quoted 13 lines

1       hugepages-16384kB/free_hugepages
1       hugepages-16384kB/nr_hugepages
1       hugepages-16384kB/nr_hugepages_mempolicy
0       hugepages-16384kB/nr_overcommit_hugepages
1       hugepages-16384kB/resv_hugepages
1       hugepages-16384kB/surplus_hugepages
0       hugepages-16777216kB/free_hugepages
0       hugepages-16777216kB/nr_hugepages
0       hugepages-16777216kB/nr_hugepages_mempolicy
0       hugepages-16777216kB/nr_overcommit_hugepages
0       hugepages-16777216kB/resv_hugepages
0       hugepages-16777216kB/surplus_hugepages

I am guessing the leaked reserve page is which is triggered by running the test you isolated corrupt-by-cow-opt.

Quoted 33 lines

---

2) Only stress tests
System boot
After setup:

20      hugepages-16384kB/free_hugepages
20      hugepages-16384kB/nr_hugepages
20      hugepages-16384kB/nr_hugepages_mempolicy
0       hugepages-16384kB/nr_overcommit_hugepages
0       hugepages-16384kB/resv_hugepages
0       hugepages-16384kB/surplus_hugepages
0       hugepages-16777216kB/free_hugepages
0       hugepages-16777216kB/nr_hugepages
0       hugepages-16777216kB/nr_hugepages_mempolicy
0       hugepages-16777216kB/nr_overcommit_hugepages
0       hugepages-16777216kB/resv_hugepages
0       hugepages-16777216kB/surplus_hugepages

After stress tests:

20      hugepages-16384kB/free_hugepages
20      hugepages-16384kB/nr_hugepages
20      hugepages-16384kB/nr_hugepages_mempolicy
0       hugepages-16384kB/nr_overcommit_hugepages
17      hugepages-16384kB/resv_hugepages
0       hugepages-16384kB/surplus_hugepages
0       hugepages-16777216kB/free_hugepages
0       hugepages-16777216kB/nr_hugepages
0       hugepages-16777216kB/nr_hugepages_mempolicy
0       hugepages-16777216kB/nr_overcommit_hugepages
0       hugepages-16777216kB/resv_hugepages
0       hugepages-16777216kB/surplus_hugepages

After cleanup:

Quoted 13 lines

17      hugepages-16384kB/free_hugepages
17      hugepages-16384kB/nr_hugepages
17      hugepages-16384kB/nr_hugepages_mempolicy
0       hugepages-16384kB/nr_overcommit_hugepages
17      hugepages-16384kB/resv_hugepages
17      hugepages-16384kB/surplus_hugepages
0       hugepages-16777216kB/free_hugepages
0       hugepages-16777216kB/nr_hugepages
0       hugepages-16777216kB/nr_hugepages_mempolicy
0       hugepages-16777216kB/nr_overcommit_hugepages
0       hugepages-16777216kB/resv_hugepages
0       hugepages-16777216kB/surplus_hugepages

This looks worse than the summary after running the functional tests.

Quoted 25 lines

---
3) only corrupt-by-cow-opt
System boot
After setup:
20      hugepages-16384kB/free_hugepages
20      hugepages-16384kB/nr_hugepages
20      hugepages-16384kB/nr_hugepages_mempolicy
0       hugepages-16384kB/nr_overcommit_hugepages
0       hugepages-16384kB/resv_hugepages
0       hugepages-16384kB/surplus_hugepages
0       hugepages-16777216kB/free_hugepages
0       hugepages-16777216kB/nr_hugepages
0       hugepages-16777216kB/nr_hugepages_mempolicy
0       hugepages-16777216kB/nr_overcommit_hugepages
0       hugepages-16777216kB/resv_hugepages
0       hugepages-16777216kB/surplus_hugepages
libhugetlbfs-2.18# env LD_LIBRARY_PATH=./obj64 ./tests/obj64/corrupt-by-cow-opt; /root/grab.sh Starting testcase "./tests/obj64/corrupt-by-cow-opt", pid 3298
Write s to 0x3effff000000 via shared mapping
Write p to 0x3effff000000 via private mapping
Read s from 0x3effff000000 via shared mapping
PASS

Quoted 12 lines

20      hugepages-16384kB/free_hugepages
20      hugepages-16384kB/nr_hugepages
20      hugepages-16384kB/nr_hugepages_mempolicy
0       hugepages-16384kB/nr_overcommit_hugepages
1       hugepages-16384kB/resv_hugepages
0       hugepages-16384kB/surplus_hugepages
0       hugepages-16777216kB/free_hugepages
0       hugepages-16777216kB/nr_hugepages
0       hugepages-16777216kB/nr_hugepages_mempolicy
0       hugepages-16777216kB/nr_overcommit_hugepages
0       hugepages-16777216kB/resv_hugepages
0       hugepages-16777216kB/surplus_hugepages

Leaked one reserve page

Quoted 11 lines

# env LD_LIBRARY_PATH=./obj64 ./tests/obj64/corrupt-by-cow-opt; /root/grab.sh
Starting testcase "./tests/obj64/corrupt-by-cow-opt", pid 3312
Write s to 0x3effff000000 via shared mapping
Write p to 0x3effff000000 via private mapping
Read s from 0x3effff000000 via shared mapping
PASS
20      hugepages-16384kB/free_hugepages
20      hugepages-16384kB/nr_hugepages
20      hugepages-16384kB/nr_hugepages_mempolicy
0       hugepages-16384kB/nr_overcommit_hugepages

Quoted 8 lines

2       hugepages-16384kB/resv_hugepages
0       hugepages-16384kB/surplus_hugepages
0       hugepages-16777216kB/free_hugepages
0       hugepages-16777216kB/nr_hugepages
0       hugepages-16777216kB/nr_hugepages_mempolicy
0       hugepages-16777216kB/nr_overcommit_hugepages
0       hugepages-16777216kB/resv_hugepages
0       hugepages-16777216kB/surplus_hugepages

It is pretty consistent that we leak a reserve page every time this test is run.

Quoted 6 lines

The interesting thing is that corrupt-by-cow-opt is a very simple test case. commit 67961f9db8c4 potentially changes the return value of the functions vma_has_reserves() and vma_needs/commit_reservation() for the owner (HPAGE_RESV_OWNER) of private mappings. running the test with and without the commit results in the same return values for these routines on x86. And, no leaked reserve pages.

looking at that commit, I am not sure region_chg output indicate a hole punched. ie, w.r.t private mapping when we mmap, we don't do a region_chg (hugetlb_reserve_page()). So with a fault later when we call vma_needs_reservation, we will find region_chg returning >= 0 right ?

Quoted 11 lines

Is it possible to revert this commit and run the libhugetlbs tests (func and stress) again while monitoring the counts in /sys? The counts should go to zero after cleanup as you describe above. I just want to make sure that this commit is causing all the problems you are seeing. If it is, then we can consider reverting and I can try to think of another way to address the original issue.
Thanks for your efforts on this. I can not reproduce on x86 or sparc and do not see any similar symptoms on these architectures.

Not sure how any of this is arch specific. So on both x86 and sparc we don't find the count going wrong as above ?

-aneesh

Mike Kravetz

2016-10-17 22:54 UTC #6 lore raw

Quoted 222 lines · Aneesh Kumar K.V

Quoted 213 lines · Mike Kravetz

Quoted 17 lines · Jan Stancek

Quoted 7 lines · Mike Kravetz
Hi Jan,
Any chance you can get the contents of /sys/kernel/mm/hugepages before and after the first run of libhugetlbfs testsuite on Power? Perhaps a script like:
Quoted 8 lines
cd /sys/kernel/mm/hugepages
for f in hugepages-*/*; do
	n=`cat $f`;
	echo -e "$n\t$f";
done
Just want to make sure the numbers look as they should.

Quoted 6 lines

Hi Mike,
Numbers are below. I have also isolated a single testcase from "func" group of tests: corrupt-by-cow-opt [1]. This test stops working if I run it 19 times (with 20 hugepages). And if I disable this test, "func" group tests can all pass repeatedly.

Thanks Jan,

I appreciate your efforts.

Quoted 54 lines

[1] https://github.com/libhugetlbfs/libhugetlbfs/blob/master/tests/corrupt-by-cow-opt.c

Regards,
Jan

Kernel is v4.8-14230-gb67be92, with reboot between each run.
1) Only func tests
System boot
After setup:

20      hugepages-16384kB/free_hugepages
20      hugepages-16384kB/nr_hugepages
20      hugepages-16384kB/nr_hugepages_mempolicy
0       hugepages-16384kB/nr_overcommit_hugepages
0       hugepages-16384kB/resv_hugepages
0       hugepages-16384kB/surplus_hugepages
0       hugepages-16777216kB/free_hugepages
0       hugepages-16777216kB/nr_hugepages
0       hugepages-16777216kB/nr_hugepages_mempolicy
0       hugepages-16777216kB/nr_overcommit_hugepages
0       hugepages-16777216kB/resv_hugepages
0       hugepages-16777216kB/surplus_hugepages

After func tests:
********** TEST SUMMARY
* 16M
* 32-bit 64-bit

*     Total testcases:     0     85
*             Skipped:     0      0
*                PASS:     0     81
*                FAIL:     0      4
*    Killed by signal:     0      0
*   Bad configuration:     0      0
*       Expected FAIL:     0      0
*     Unexpected PASS:     0      0
* Strange test result:     0      0

26      hugepages-16384kB/free_hugepages
26      hugepages-16384kB/nr_hugepages
26      hugepages-16384kB/nr_hugepages_mempolicy
0       hugepages-16384kB/nr_overcommit_hugepages
1       hugepages-16384kB/resv_hugepages
0       hugepages-16384kB/surplus_hugepages
0       hugepages-16777216kB/free_hugepages
0       hugepages-16777216kB/nr_hugepages
0       hugepages-16777216kB/nr_hugepages_mempolicy
0       hugepages-16777216kB/nr_overcommit_hugepages
0       hugepages-16777216kB/resv_hugepages
0       hugepages-16777216kB/surplus_hugepages

After test cleanup:
umount -a -t hugetlbfs
hugeadm --pool-pages-max ${HPSIZE}:0

Quoted 13 lines

1       hugepages-16384kB/free_hugepages
1       hugepages-16384kB/nr_hugepages
1       hugepages-16384kB/nr_hugepages_mempolicy
0       hugepages-16384kB/nr_overcommit_hugepages
1       hugepages-16384kB/resv_hugepages
1       hugepages-16384kB/surplus_hugepages
0       hugepages-16777216kB/free_hugepages
0       hugepages-16777216kB/nr_hugepages
0       hugepages-16777216kB/nr_hugepages_mempolicy
0       hugepages-16777216kB/nr_overcommit_hugepages
0       hugepages-16777216kB/resv_hugepages
0       hugepages-16777216kB/surplus_hugepages

I am guessing the leaked reserve page is which is triggered by running the test you isolated corrupt-by-cow-opt.

Quoted 33 lines

---

2) Only stress tests
System boot
After setup:

20      hugepages-16384kB/free_hugepages
20      hugepages-16384kB/nr_hugepages
20      hugepages-16384kB/nr_hugepages_mempolicy
0       hugepages-16384kB/nr_overcommit_hugepages
0       hugepages-16384kB/resv_hugepages
0       hugepages-16384kB/surplus_hugepages
0       hugepages-16777216kB/free_hugepages
0       hugepages-16777216kB/nr_hugepages
0       hugepages-16777216kB/nr_hugepages_mempolicy
0       hugepages-16777216kB/nr_overcommit_hugepages
0       hugepages-16777216kB/resv_hugepages
0       hugepages-16777216kB/surplus_hugepages

After stress tests:

20      hugepages-16384kB/free_hugepages
20      hugepages-16384kB/nr_hugepages
20      hugepages-16384kB/nr_hugepages_mempolicy
0       hugepages-16384kB/nr_overcommit_hugepages
17      hugepages-16384kB/resv_hugepages
0       hugepages-16384kB/surplus_hugepages
0       hugepages-16777216kB/free_hugepages
0       hugepages-16777216kB/nr_hugepages
0       hugepages-16777216kB/nr_hugepages_mempolicy
0       hugepages-16777216kB/nr_overcommit_hugepages
0       hugepages-16777216kB/resv_hugepages
0       hugepages-16777216kB/surplus_hugepages

After cleanup:

Quoted 13 lines

17      hugepages-16384kB/free_hugepages
17      hugepages-16384kB/nr_hugepages
17      hugepages-16384kB/nr_hugepages_mempolicy
0       hugepages-16384kB/nr_overcommit_hugepages
17      hugepages-16384kB/resv_hugepages
17      hugepages-16384kB/surplus_hugepages
0       hugepages-16777216kB/free_hugepages
0       hugepages-16777216kB/nr_hugepages
0       hugepages-16777216kB/nr_hugepages_mempolicy
0       hugepages-16777216kB/nr_overcommit_hugepages
0       hugepages-16777216kB/resv_hugepages
0       hugepages-16777216kB/surplus_hugepages

This looks worse than the summary after running the functional tests.

Quoted 25 lines

---
3) only corrupt-by-cow-opt
System boot
After setup:
20      hugepages-16384kB/free_hugepages
20      hugepages-16384kB/nr_hugepages
20      hugepages-16384kB/nr_hugepages_mempolicy
0       hugepages-16384kB/nr_overcommit_hugepages
0       hugepages-16384kB/resv_hugepages
0       hugepages-16384kB/surplus_hugepages
0       hugepages-16777216kB/free_hugepages
0       hugepages-16777216kB/nr_hugepages
0       hugepages-16777216kB/nr_hugepages_mempolicy
0       hugepages-16777216kB/nr_overcommit_hugepages
0       hugepages-16777216kB/resv_hugepages
0       hugepages-16777216kB/surplus_hugepages
libhugetlbfs-2.18# env LD_LIBRARY_PATH=./obj64 ./tests/obj64/corrupt-by-cow-opt; /root/grab.sh Starting testcase "./tests/obj64/corrupt-by-cow-opt", pid 3298
Write s to 0x3effff000000 via shared mapping
Write p to 0x3effff000000 via private mapping
Read s from 0x3effff000000 via shared mapping
PASS

Quoted 12 lines

20      hugepages-16384kB/free_hugepages
20      hugepages-16384kB/nr_hugepages
20      hugepages-16384kB/nr_hugepages_mempolicy
0       hugepages-16384kB/nr_overcommit_hugepages
1       hugepages-16384kB/resv_hugepages
0       hugepages-16384kB/surplus_hugepages
0       hugepages-16777216kB/free_hugepages
0       hugepages-16777216kB/nr_hugepages
0       hugepages-16777216kB/nr_hugepages_mempolicy
0       hugepages-16777216kB/nr_overcommit_hugepages
0       hugepages-16777216kB/resv_hugepages
0       hugepages-16777216kB/surplus_hugepages

Leaked one reserve page

Quoted 11 lines

# env LD_LIBRARY_PATH=./obj64 ./tests/obj64/corrupt-by-cow-opt; /root/grab.sh
Starting testcase "./tests/obj64/corrupt-by-cow-opt", pid 3312
Write s to 0x3effff000000 via shared mapping
Write p to 0x3effff000000 via private mapping
Read s from 0x3effff000000 via shared mapping
PASS
20      hugepages-16384kB/free_hugepages
20      hugepages-16384kB/nr_hugepages
20      hugepages-16384kB/nr_hugepages_mempolicy
0       hugepages-16384kB/nr_overcommit_hugepages

Quoted 8 lines

2       hugepages-16384kB/resv_hugepages
0       hugepages-16384kB/surplus_hugepages
0       hugepages-16777216kB/free_hugepages
0       hugepages-16777216kB/nr_hugepages
0       hugepages-16777216kB/nr_hugepages_mempolicy
0       hugepages-16777216kB/nr_overcommit_hugepages
0       hugepages-16777216kB/resv_hugepages
0       hugepages-16777216kB/surplus_hugepages

It is pretty consistent that we leak a reserve page every time this test is run.

Quoted 6 lines

The interesting thing is that corrupt-by-cow-opt is a very simple test case. commit 67961f9db8c4 potentially changes the return value of the functions vma_has_reserves() and vma_needs/commit_reservation() for the owner (HPAGE_RESV_OWNER) of private mappings. running the test with and without the commit results in the same return values for these routines on x86. And, no leaked reserve pages.

Quoted 5 lines

looking at that commit, I am not sure region_chg output indicate a hole punched. ie, w.r.t private mapping when we mmap, we don't do a region_chg (hugetlb_reserve_page()). So with a fault later when we call vma_needs_reservation, we will find region_chg returning >= 0 right ?

Let me try to explain.

When a private mapping is created, hugetlb_reserve_pages to reserve huge pages for the mapping. A reserve map is created and installed in the (vma_private) VMA. No reservation entries are actually created for the mapping. But, hugetlb_acct_memory() is called to reserve pages for the mapping in the global pool. This will adjust (increment) the global reserved huge page counter (resv_huge_pages).

As pages within the private mapping are faulted in, huge_page_alloc() is called to allocate the pages. Within alloc_huge_page, vma_needs_reservation is called to determine if there is a reservation for this allocation. If there is a reservation, the global count is adjusted (decremented). In any case where a page is returned to the caller, vma_commit_reservation is called and an entry for the page is created in the reserve map (VMA vma_private) of the mapping.

Once a page is instantiated within the private mapping, an entry exists in the reserve map and the reserve count has been adjusted to indicate that the reserve has been consumed. Subsequent faults will not instantiate a new page unless the original is somehow removed from the mapping. The only way a user can remove a page from the mapping is via a hole punch or truncate operation. Note that hole punch and truncate for huge pages only to apply to hugetlbfs backed mappings and not anonymous mappings.

hole punch and truncate will unmap huge pages from any private private mapping associated with the same offset in the hugetlbfs file. However, they will not remove entries from the VMA private_data reserve maps. Nor, will they adjust global reserve counts based on private mappings.

Now suppose a subsequent fault happened for a page private mapping removed via hole punch or truncate. Prior to commit 67961f9db8c4, vma_needs_reservation ALWAYS returned false to indicate that a reservation existed for the page. So, alloc_huge_page would consume a reserved page. The problem is that the reservation was consumed at the time of the first fault and no longer exist. This caused the global reserve count to be incorrect.

Commit 67961f9db8c4 looks at the VMA private reserve map to determine if the original reservation was consumed. If an entry exists in the map, it is assumed the reservation was consumed and no longer exists.

Signature · 7 lines

Mike Kravetz

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Mike Kravetz

2016-10-18 01:19 UTC #7 lore raw

Quoted 33 lines · Mike Kravetz

Quoted 6 lines · Aneesh Kumar K.V
looking at that commit, I am not sure region_chg output indicate a hole punched. ie, w.r.t private mapping when we mmap, we don't do a region_chg (hugetlb_reserve_page()). So with a fault later when we call vma_needs_reservation, we will find region_chg returning >= 0 right ?
Let me try to explain.
When a private mapping is created, hugetlb_reserve_pages to reserve huge pages for the mapping. A reserve map is created and installed in the (vma_private) VMA. No reservation entries are actually created for the mapping. But, hugetlb_acct_memory() is called to reserve pages for the mapping in the global pool. This will adjust (increment) the global reserved huge page counter (resv_huge_pages).
As pages within the private mapping are faulted in, huge_page_alloc() is called to allocate the pages. Within alloc_huge_page, vma_needs_reservation is called to determine if there is a reservation for this allocation. If there is a reservation, the global count is adjusted (decremented). In any case where a page is returned to the caller, vma_commit_reservation is called and an entry for the page is created in the reserve map (VMA vma_private) of the mapping.
Once a page is instantiated within the private mapping, an entry exists in the reserve map and the reserve count has been adjusted to indicate that the reserve has been consumed. Subsequent faults will not instantiate a new page unless the original is somehow removed from the mapping. The only way a user can remove a page from the mapping is via a hole punch or truncate operation. Note that hole punch and truncate for huge pages only to apply to hugetlbfs backed mappings and not anonymous mappings.

Quoted 4 lines

hole punch and truncate will unmap huge pages from any private private mapping associated with the same offset in the hugetlbfs file. However, they will not remove entries from the VMA private_data reserve maps. Nor, will they adjust global reserve counts based on private mappings.

Question. Should hole punch and truncate unmap private mappings? Commit 67961f9db8c4 is just trying to correctly handle that situation. If we do not unmap the private pages, then there is no need for this code.

Signature · 1 line

Mike Kravetz

Quoted 9 lines

Now suppose a subsequent fault happened for a page private mapping removed via hole punch or truncate. Prior to commit 67961f9db8c4, vma_needs_reservation ALWAYS returned false to indicate that a reservation existed for the page. So, alloc_huge_page would consume a reserved page. The problem is that the reservation was consumed at the time of the first fault and no longer exist. This caused the global reserve count to be incorrect.

Quoted 4 lines

Commit 67961f9db8c4 looks at the VMA private reserve map to determine if the original reservation was consumed. If an entry exists in the map, it is assumed the reservation was consumed and no longer exists.

Jan Stancek

author

2016-10-17 14:44 UTC #8 lore raw

In reply to Mike Kravetz

----- Original Message -----

Quoted 26 lines

From: "Mike Kravetz" <[email protected]>
To: "Jan Stancek" <[email protected]>, [email protected], [email protected]
Cc: "hillf zj" <[email protected]>, "dave hansen" <[email protected]>, "kirill shutemov" <[email protected]>, [email protected], [email protected], "aneesh kumar" <[email protected]>, "iamjoonsoo kim" <[email protected]>
Sent: Saturday, 15 October, 2016 1:57:31 AM
Subject: Re: [bug/regression] libhugetlbfs testsuite failures and OOMs eventually kill my system
It is pretty consistent that we leak a reserve page every time this test is run.
The interesting thing is that corrupt-by-cow-opt is a very simple test case. commit 67961f9db8c4 potentially changes the return value of the functions vma_has_reserves() and vma_needs/commit_reservation() for the owner (HPAGE_RESV_OWNER) of private mappings. running the test with and without the commit results in the same return values for these routines on x86. And, no leaked reserve pages.
Is it possible to revert this commit and run the libhugetlbs tests (func and stress) again while monitoring the counts in /sys? The counts should go to zero after cleanup as you describe above. I just want to make sure that this commit is causing all the problems you are seeing. If it is, then we can consider reverting and I can try to think of another way to address the original issue.

Quoted 6 lines

Thanks for your efforts on this. I can not reproduce on x86 or sparc and do not see any similar symptoms on these architectures.
--
Mike Kravetz

Hi Mike,

Revert of 67961f9db8c4 helps, I let whole suite run for 100 iterations, there were no issues.

I cut down reproducer and removed last mmap/write/munmap as that is enough to reproduce the problem. Then I started introducing some traces into kernel and noticed that on ppc I get 3 faults, while on x86 I get only 2.

Interesting is the 2nd fault, that is first write after mapping as PRIVATE. Following condition fails on ppc first time:

    if (likely(ptep && pte_same(huge_ptep_get(ptep), pte))) {

but it's immediately followed by fault that looks identical and in that one it evaluates as true.

Same with alloc_huge_page(), on x86_64 it's called twice, on ppc three times. In 2nd call vma_needs_reservation() returns 0, in 3rd it returns 1.

---- ppc -> 2nd and 3rd fault --- mmap(MAP_PRIVATE)
hugetlb_fault address: 3effff000000, flags: 55 hugetlb_cow old_page: f0000000010fc000 alloc_huge_page ret: f000000001100000 hugetlb_cow ptep: c000000455b27cf8, pte_same: 0 free_huge_page page: f000000001100000, restore_reserve: 1 hugetlb_fault address: 3effff000000, flags: 55 hugetlb_cow old_page: f0000000010fc000 alloc_huge_page ret: f000000001100000 hugetlb_cow ptep: c000000455b27cf8, pte_same: 1

--- x86_64 -> 2nd fault ---
mmap(MAP_PRIVATE)
hugetlb_fault address: 7f71a4200000, flags: 55 hugetlb_cow address 0x7f71a4200000, old_page: ffffea0008d20000 alloc_huge_page ret: ffffea0008d38000
hugetlb_cow ptep: ffff8802314c7908, pte_same: 1

Regards,
Jan

Aneesh Kumar K.V

2016-10-17 18:28 UTC #9 lore raw

Quoted 9 lines · Jan Stancek

Hi Mike,
Revert of 67961f9db8c4 helps, I let whole suite run for 100 iterations, there were no issues.
I cut down reproducer and removed last mmap/write/munmap as that is enough to reproduce the problem. Then I started introducing some traces into kernel and noticed that on ppc I get 3 faults, while on x86 I get only 2.

Quoted 5 lines

Interesting is the 2nd fault, that is first write after mapping as PRIVATE. Following condition fails on ppc first time:
    if (likely(ptep && pte_same(huge_ptep_get(ptep), pte))) {
but it's immediately followed by fault that looks identical and in that one it evaluates as true.

ok, we miss the _PAGE_PTE in new_pte there.

	new_pte = make_huge_pte(vma, page, ((vma->vm_flags & VM_WRITE)
				&& (vma->vm_flags & VM_SHARED)));
	set_huge_pte_at(mm, address, ptep, new_pte);

	hugetlb_count_add(pages_per_huge_page(h), mm);
	if ((flags & FAULT_FLAG_WRITE) && !(vma->vm_flags & VM_SHARED)) {
		/* Optimization, do the COW without a second fault */
		ret = hugetlb_cow(mm, vma, address, ptep, new_pte, page, ptl);
	}

IMHO that new_pte usage is wrong, because we don't consider flags that can possibly be added by set_huge_pte_at there. For pp64 we add _PAGE_PTE

Quoted 16 lines

Same with alloc_huge_page(), on x86_64 it's called twice, on ppc three times. In 2nd call vma_needs_reservation() returns 0, in 3rd it returns 1.
---- ppc -> 2nd and 3rd fault --- mmap(MAP_PRIVATE)
hugetlb_fault address: 3effff000000, flags: 55 hugetlb_cow old_page: f0000000010fc000 alloc_huge_page ret: f000000001100000 hugetlb_cow ptep: c000000455b27cf8, pte_same: 0 free_huge_page page: f000000001100000, restore_reserve: 1 hugetlb_fault address: 3effff000000, flags: 55 hugetlb_cow old_page: f0000000010fc000 alloc_huge_page ret: f000000001100000 hugetlb_cow ptep: c000000455b27cf8, pte_same: 1

Quoted 7 lines

--- x86_64 -> 2nd fault ---
mmap(MAP_PRIVATE)
hugetlb_fault address: 7f71a4200000, flags: 55 hugetlb_cow address 0x7f71a4200000, old_page: ffffea0008d20000 alloc_huge_page ret: ffffea0008d38000
hugetlb_cow ptep: ffff8802314c7908, pte_same: 1

But I guess we still have issue with respecting reservation here.

I will look at _PAGE_PTE and see what best we can do w.r.t hugetlb.

-aneesh

Mike Kravetz

2016-10-17 23:20 UTC #10 lore raw

Quoted 30 lines · Aneesh Kumar K.V

Quoted 9 lines · Jan Stancek
Hi Mike,
Revert of 67961f9db8c4 helps, I let whole suite run for 100 iterations, there were no issues.
I cut down reproducer and removed last mmap/write/munmap as that is enough to reproduce the problem. Then I started introducing some traces into kernel and noticed that on ppc I get 3 faults, while on x86 I get only 2.
Quoted 5 lines
Interesting is the 2nd fault, that is first write after mapping as PRIVATE. Following condition fails on ppc first time:
    if (likely(ptep && pte_same(huge_ptep_get(ptep), pte))) {
but it's immediately followed by fault that looks identical and in that one it evaluates as true.
ok, we miss the _PAGE_PTE in new_pte there.
	new_pte = make_huge_pte(vma, page, ((vma->vm_flags & VM_WRITE)
				&& (vma->vm_flags & VM_SHARED)));
	set_huge_pte_at(mm, address, ptep, new_pte);

	hugetlb_count_add(pages_per_huge_page(h), mm);
	if ((flags & FAULT_FLAG_WRITE) && !(vma->vm_flags & VM_SHARED)) {
		/* Optimization, do the COW without a second fault */
		ret = hugetlb_cow(mm, vma, address, ptep, new_pte, page, ptl);
	}

Quoted 3 lines

IMHO that new_pte usage is wrong, because we don't consider flags that can possibly be added by set_huge_pte_at there. For pp64 we add _PAGE_PTE

Thanks for looking at this Aneesh.

Quoted 11 lines

Quoted 11 lines
Same with alloc_huge_page(), on x86_64 it's called twice, on ppc three times. In 2nd call vma_needs_reservation() returns 0, in 3rd it returns 1.
---- ppc -> 2nd and 3rd fault --- mmap(MAP_PRIVATE)
hugetlb_fault address: 3effff000000, flags: 55 hugetlb_cow old_page: f0000000010fc000 alloc_huge_page ret: f000000001100000 hugetlb_cow ptep: c000000455b27cf8, pte_same: 0 free_huge_page page: f000000001100000, restore_reserve: 1

So, the interesting thing is that since we do not take the optimized path there is an additional fault. It looks like the additional fault results in the originally allocated page being free'ed and reserve count being incremented. As mentioned in the description of commit 67961f9db8c4, the VMA private reserve map will still contain an entry for the page. Therefore,
when a page allocation happens as the result of the next fault, it will think the reserved page has already been consumed and not use it. This is how we are 'leaking' reserved pages.

Quoted 13 lines

Quoted 5 lines
hugetlb_fault address: 3effff000000, flags: 55
hugetlb_cow old_page: f0000000010fc000
alloc_huge_page ret: f000000001100000
hugetlb_cow ptep: c000000455b27cf8, pte_same: 1
Quoted 7 lines
--- x86_64 -> 2nd fault ---
mmap(MAP_PRIVATE)
hugetlb_fault address: 7f71a4200000, flags: 55 hugetlb_cow address 0x7f71a4200000, old_page: ffffea0008d20000 alloc_huge_page ret: ffffea0008d38000
hugetlb_cow ptep: ffff8802314c7908, pte_same: 1

Quoted 5 lines

But I guess we still have issue with respecting reservation here.
I will look at _PAGE_PTE and see what best we can do w.r.t hugetlb.
-aneesh

If there was not the additional fault, we would not perform the additional free and alloc and not see this issue. However, the logic in 67961f9db8c4 also missed this error case (and I think any time we do not take the optimized
code path).

I suspect it is would be desirable to fix the code path for Power such that it does not do the additional fault (free/alloc). I'll take a look at the code for commit for 67961f9db8c4. It certainly misses the error case, and seems 'too fragile' to depend on the optimized code paths.

Signature · 7 lines

Mike Kravetz

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Aneesh Kumar K.V

2016-10-18 08:32 UTC #11 lore raw

In reply to Jan Stancek

Quoted 37 lines · Jan Stancek

Hi Mike,
Revert of 67961f9db8c4 helps, I let whole suite run for 100 iterations, there were no issues.
I cut down reproducer and removed last mmap/write/munmap as that is enough to reproduce the problem. Then I started introducing some traces into kernel and noticed that on ppc I get 3 faults, while on x86 I get only 2.
Interesting is the 2nd fault, that is first write after mapping as PRIVATE. Following condition fails on ppc first time:
    if (likely(ptep && pte_same(huge_ptep_get(ptep), pte))) {
but it's immediately followed by fault that looks identical and in that one it evaluates as true.
Same with alloc_huge_page(), on x86_64 it's called twice, on ppc three times. In 2nd call vma_needs_reservation() returns 0, in 3rd it returns 1.
---- ppc -> 2nd and 3rd fault --- mmap(MAP_PRIVATE)
hugetlb_fault address: 3effff000000, flags: 55 hugetlb_cow old_page: f0000000010fc000 alloc_huge_page ret: f000000001100000 hugetlb_cow ptep: c000000455b27cf8, pte_same: 0 free_huge_page page: f000000001100000, restore_reserve: 1 hugetlb_fault address: 3effff000000, flags: 55 hugetlb_cow old_page: f0000000010fc000 alloc_huge_page ret: f000000001100000 hugetlb_cow ptep: c000000455b27cf8, pte_same: 1
--- x86_64 -> 2nd fault ---
mmap(MAP_PRIVATE)
hugetlb_fault address: 7f71a4200000, flags: 55 hugetlb_cow address 0x7f71a4200000, old_page: ffffea0008d20000 alloc_huge_page ret: ffffea0008d38000
hugetlb_cow ptep: ffff8802314c7908, pte_same: 1

Quoted 3 lines

Regards,
Jan

Can you check with the below patch. I ran the corrupt-by-cow-opt test with this patch and resv count got correctly updated.

commit fb2e0c081d2922c8aaa49bbe166472aac68ef5e1

Author: Aneesh Kumar K.V <[email protected]>
Date:   Tue Oct 18 11:23:11 2016 +0530

mm/hugetlb: Use the right pte val for compare in hugetlb_cow

We cannot use the pte value used in set_pte_at for pte_same comparison, because archs like ppc64, filter/add new pte flag in set_pte_at. Instead fetch the pte value inside hugetlb_cow. We are comparing pte value to make sure the pte didn't change since we dropped the page table lock. hugetlb_cow get called with page table lock held, and we can take a copy of the pte value before we drop the page table lock.

Signed-off-by Aneesh Kumar K.V <[email protected]>

mm/hugetlb.c +7 -5

index ec49d9ef1eef..da8fbd02b92e 100644
@@ -3386,15 +3386,17 @@ static void unmap_ref_private(struct mm_struct *mm, struct vm_area_struct *vma,
  * Keep the pte_same checks anyway to make transition from the mutex easier.
  */
 static int hugetlb_cow(struct mm_struct *mm, struct vm_area_struct *vma,
-			unsigned long address, pte_t *ptep, pte_t pte,
-			struct page *pagecache_page, spinlock_t *ptl)
+		       unsigned long address, pte_t *ptep,
+		       struct page *pagecache_page, spinlock_t *ptl)
 {
+	pte_t pte;
 	struct hstate *h = hstate_vma(vma);
 	struct page *old_page, *new_page;
 	int ret = 0, outside_reserve = 0;
 	unsigned long mmun_start;	/* For mmu_notifiers */
 	unsigned long mmun_end;		/* For mmu_notifiers */
 
+	pte = huge_ptep_get(ptep);
 	old_page = pte_page(pte);
 
 retry_avoidcopy:

@@ -3668,7 +3670,7 @@ static int hugetlb_no_page(struct mm_struct *mm, struct vm_area_struct *vma,
 	hugetlb_count_add(pages_per_huge_page(h), mm);
 	if ((flags & FAULT_FLAG_WRITE) && !(vma->vm_flags & VM_SHARED)) {
 		/* Optimization, do the COW without a second fault */
-		ret = hugetlb_cow(mm, vma, address, ptep, new_pte, page, ptl);
+		ret = hugetlb_cow(mm, vma, address, ptep, page, ptl);
 	}
 
 	spin_unlock(ptl);

@@ -3822,8 +3824,8 @@ int hugetlb_fault(struct mm_struct *mm, struct vm_area_struct *vma,
 
 	if (flags & FAULT_FLAG_WRITE) {
 		if (!huge_pte_write(entry)) {
-			ret = hugetlb_cow(mm, vma, address, ptep, entry,
-					pagecache_page, ptl);
+			ret = hugetlb_cow(mm, vma, address, ptep,
+					  pagecache_page, ptl);
 			goto out_put_page;
 		}
 		entry = huge_pte_mkdirty(entry);

Jan Stancek

author

2016-10-18 11:28 UTC #12 lore raw

----- Original Message -----

Quoted 40 lines

Quoted 39 lines · Jan Stancek
Hi Mike,
Revert of 67961f9db8c4 helps, I let whole suite run for 100 iterations, there were no issues.
I cut down reproducer and removed last mmap/write/munmap as that is enough to reproduce the problem. Then I started introducing some traces into kernel
and noticed that on ppc I get 3 faults, while on x86 I get only 2.
Interesting is the 2nd fault, that is first write after mapping as PRIVATE. Following condition fails on ppc first time:
    if (likely(ptep && pte_same(huge_ptep_get(ptep), pte))) {
but it's immediately followed by fault that looks identical and in that one it evaluates as true.
Same with alloc_huge_page(), on x86_64 it's called twice, on ppc three times.
In 2nd call vma_needs_reservation() returns 0, in 3rd it returns 1.
---- ppc -> 2nd and 3rd fault --- mmap(MAP_PRIVATE)
hugetlb_fault address: 3effff000000, flags: 55 hugetlb_cow old_page: f0000000010fc000 alloc_huge_page ret: f000000001100000 hugetlb_cow ptep: c000000455b27cf8, pte_same: 0 free_huge_page page: f000000001100000, restore_reserve: 1 hugetlb_fault address: 3effff000000, flags: 55 hugetlb_cow old_page: f0000000010fc000 alloc_huge_page ret: f000000001100000 hugetlb_cow ptep: c000000455b27cf8, pte_same: 1
--- x86_64 -> 2nd fault ---
mmap(MAP_PRIVATE)
hugetlb_fault address: 7f71a4200000, flags: 55 hugetlb_cow address 0x7f71a4200000, old_page: ffffea0008d20000 alloc_huge_page ret: ffffea0008d38000
hugetlb_cow ptep: ffff8802314c7908, pte_same: 1

Quoted 7 lines

Quoted 3 lines
Regards,
Jan
Can you check with the below patch. I ran the corrupt-by-cow-opt test with this patch
and resv count got correctly updated.

I am running libhugetlbfs suite with patch below in loop for ~2 hours now and I don't see any problems/ENOMEMs/OOMs or leaked resv pages:

0       hugepages-16384kB/free_hugepages
0       hugepages-16384kB/nr_hugepages
0       hugepages-16384kB/nr_hugepages_mempolicy
0       hugepages-16384kB/nr_overcommit_hugepages
0       hugepages-16384kB/resv_hugepages
0       hugepages-16384kB/surplus_hugepages
0       hugepages-16777216kB/free_hugepages
0       hugepages-16777216kB/nr_hugepages
0       hugepages-16777216kB/nr_hugepages_mempolicy
0       hugepages-16777216kB/nr_overcommit_hugepages
0       hugepages-16777216kB/resv_hugepages
0       hugepages-16777216kB/surplus_hugepages

Regards,
Jan

Quoted 62 lines

commit fb2e0c081d2922c8aaa49bbe166472aac68ef5e1

Author: Aneesh Kumar K.V <[email protected]>
Date:   Tue Oct 18 11:23:11 2016 +0530

mm/hugetlb: Use the right pte val for compare in hugetlb_cow

Signed-off-by Aneesh Kumar K.V <[email protected]>

mm/hugetlb.c +4 -2

index ec49d9ef1eef..da8fbd02b92e 100644
@@ -3386,15 +3386,17 @@ static void unmap_ref_private(struct mm_struct *mm,
 struct vm_area_struct *vma,
  * Keep the pte_same checks anyway to make transition from the mutex easier.
  */
 static int hugetlb_cow(struct mm_struct *mm, struct vm_area_struct *vma,
-			unsigned long address, pte_t *ptep, pte_t pte,
-			struct page *pagecache_page, spinlock_t *ptl)
+		       unsigned long address, pte_t *ptep,
+		       struct page *pagecache_page, spinlock_t *ptl)
 {
+	pte_t pte;
 	struct hstate *h = hstate_vma(vma);
 	struct page *old_page, *new_page;
 	int ret = 0, outside_reserve = 0;
 	unsigned long mmun_start;	/* For mmu_notifiers */
 	unsigned long mmun_end;		/* For mmu_notifiers */
 
+	pte = huge_ptep_get(ptep);
 	old_page = pte_page(pte);

retry_avoidcopy:

mm/hugetlb.c +1 -1

@@ -3668,7 +3670,7 @@ static int hugetlb_no_page(struct mm_struct *mm, struct
 vm_area_struct *vma,
 	hugetlb_count_add(pages_per_huge_page(h), mm);
 	if ((flags & FAULT_FLAG_WRITE) && !(vma->vm_flags & VM_SHARED)) {
 		/* Optimization, do the COW without a second fault */
-		ret = hugetlb_cow(mm, vma, address, ptep, new_pte, page, ptl);
+		ret = hugetlb_cow(mm, vma, address, ptep, page, ptl);
 	}

mm/hugetlb.c +2 -2

 	spin_unlock(ptl);

@@ -3822,8 +3824,8 @@ int hugetlb_fault(struct mm_struct *mm, struct
 vm_area_struct *vma,
 
 	if (flags & FAULT_FLAG_WRITE) {
 		if (!huge_pte_write(entry)) {
-			ret = hugetlb_cow(mm, vma, address, ptep, entry,
-					pagecache_page, ptl);
+			ret = hugetlb_cow(mm, vma, address, ptep,
+					  pagecache_page, ptl);
 			goto out_put_page;
 		}

Quoted 3 lines

entry = huge_pte_mkdirty(entry);