Hugepages_Rsvd goes huge in 2.6.20-rc7

Hi all,

So, here's the current state of the hugepages portion of my /proc/meminfo (x86_64, 2.6.20-rc7, will test with 2.6.20 shortly, but AFAICS, there haven't been many changes to hugepage code between the two):

HugePages_Total:   100
HugePages_Free:    100
HugePages_Rsvd:  18446744073709551615
Hugepagesize:     2048 kB

That's not good :)

Context: I'm currently working on some patches for libhugetlbfs which should ultimately help us reduce our hugepage usage when remapping segments so they are backed by hugepages. The current algorithm maps in hugepage file as MAP_SHARED, copies over the segment data, then unmaps the file. It then unmaps the program's segments, and maps in the same hugepage file MAP_PRIVATE, so that we take COW faults. Now, the problem is, for writable segments (data) the COW fault instatiates a new hugepage, but the original MAP_SHARED hugepage stays resident in the page cache. So, for a program that could survive (after the initial remapping algorithm) with only 2 hugepages in use, uses 3 hugepages instead.

To work around this, I've modified the algorithm to prefault in the writable segment in the remapping code (via a one-byte read and write). Then, I issue a posix_fadvise(segment_fd, 0, 0, FADV_DONTNEED), to try and drop the shared hugepage from the page cache. With a small dummy relinked app (that just sleeps), this does reduce our run-time hugepage cost from 3 to 2. But, I'm noticing that libhugetlbfs' make func utility, which tests libhugetlbfs' functionality only, every so often leads to a lot of "VM killing process ...". This only appears to happen to a particular testcase (xBDT.linkshare, which remaps the BSS, data and text segments and tries to share the text segments between 2 processes), but when it does, it happens for a while (that is, if I try and run that particular test manually, it keeps getting killed) and /proc/meminfo reports a garbage value for HugePages_Rsvd like I listed above. If I rerun make func, sometimes the problem goes away (Rsvd returns to a sane value, as well...).

I've added Hugh & David to the Cc, because they discussed a similar problem a few months back. Maybe there is still a race somewhere?

I'm willing to test any possible fixes, and I'll work on making this more easily reproducible (although it seems to happen pretty regularly here) with a simpler test.

Thanks,
Nish

Signature · 2 lines

Nishanth Aravamudan <[email protected]>
IBM Linux Technology Center

Sorry, I botched Hugh's e-mail address, please make sure to reply to the correct one.

Thanks,
Nish

Quoted 6 lines · Nishanth Aravamudan

Hi all,

So, here's the current state of the hugepages portion of my /proc/meminfo (x86_64, 2.6.20-rc7, will test with 2.6.20 shortly, but AFAICS, there haven't been many changes to hugepage code between the two):

Quoted 13 lines · Nishanth Aravamudan

Sorry, I botched Hugh's e-mail address, please make sure to reply to the correct one.

Thanks,
Nish

Quoted 6 lines · Nishanth Aravamudan

Hi all,

So, here's the current state of the hugepages portion of my /proc/meminfo (x86_64, 2.6.20-rc7, will test with 2.6.20 shortly, but AFAICS, there haven't been many changes to hugepage code between the two):

Reproduced on 2.6.20, and I think I've got a means to make it more easily reproducible (at least on x86_64).

Please note, I found that when HugePages_Rsvd goes very large, I can make it return to 0 by running make func, but killing it before it gets to the sharing tests. Rsvd returns to 0 in this case.

So, here's my means of reproducing it (as root, from the libhugetlbfs root directory [1]):

# make sure everything is clean, hugepages wise
root@arkanoid# rm -rf /mnt/hugetlbfs/*
# if /proc/meminfo is already screwed up, run `make func` and kill it
# around when you see the mprotect testcase run, that seems to always
# work -- I'll try to be more scientific on this in a bit, to see which
# test causes the value to return to sanity

# run the linkshare testcase once, probably will die right away
root@arkanoid# HUGETLB_VERBOSE=99 HUGETLB_ELFMAP=y HUGETLB_SHARE=1 LD_LIBRARY_PATH=./obj64 ./tests/obj64/xBDT.linkshare
# you should see the testcase be killed, something like
# "FAIL    Child 1 killed by signal: Killed"
root@arkanoid# cat /proc/meminfo
# and a large value in meminfo now

Seems to happen every time I do this :) Note, part of this reproducibility stems from a small modification to the details I gave before. Before doing the posix_fadvise() call, I now do an fsync() on the file-descriptor. Without the fsync(), it may take one or two invocations before the test fails, but it still will in my experience so far.

Also note, that I'm not trying to defend the way I'm approaching this problem in libhugetlbfs (I'm very open to alternatives) -- but regardless of what I do there, I don't think Rsvd should be 18446744073709551615 ...

Quoted 17 lines · Nishanth Aravamudan

Quoted 13 lines · Nishanth Aravamudan

Sorry, I botched Hugh's e-mail address, please make sure to reply to the correct one.

Thanks,
Nish

Quoted 6 lines · Nishanth Aravamudan

Hi all,

So, here's the current state of the hugepages portion of my /proc/meminfo (x86_64, 2.6.20-rc7, will test with 2.6.20 shortly, but AFAICS, there haven't been many changes to hugepage code between the two):

Reproduced on 2.6.20, and I think I've got a means to make it more easily reproducible (at least on x86_64).

Quoted 4 lines

Also note, that I'm not trying to defend the way I'm approaching this problem in libhugetlbfs (I'm very open to alternatives) -- but regardless of what I do there, I don't think Rsvd should be 18446744073709551615 ...

Oh, certainly not. Clearly we're managing to decrement it more times than we're incrementing it somehow. I'd check the codepath for the madvise() thing, we may not be handling that properly.