Add MADV_COLLAPSE documentation
v3 cover Add MADV_COLLAPSE documentation
From: Zach O'Keefe <[email protected]>
v3 Forward
Changes from v2[1]:
- Patch 3 adds a NOTES section that records the security requirements when process_madvise(2) syscall was added in Linux 5.10 (Alex Colomar).
- Patch 3 also corrects documented PTRACE_MODE_READ_REALCREDS ptrace mode for process_madvise(2).
[1] https://lore.kernel.org/linux-man/[email protected]/
----------------------------------------------
v2 Forward
Changes from v1[2]:
- All patches have seen some reformatting and/or typo corrections (Alex Colomar).
- Patch 3 has had the commit description reworded to reflect that this commit is a "fix" rather than an "update" (Alex Colomar).
- Rebased on man-pages-6.01
[2] https://lore.kernel.org/linux-man/[email protected]/T/#m8e9e94ed52c99e7cf4969bd992492359c59a0faa
----------------------------------------------
Hey Alex / Michael,
This series adds MADV_COLLAPSE (expected for Linux 6.1) documentation to madvise(2) and process_madvise(2). A few prerequisite patches are included to fix up existing MADV_HUGEPAGE and process_madvise(2) documentation, as well as add some additional clarity for madvise(2) "advice" probing.
Series applies on man-pages-6.00, and I've attempted to use semantic newlines, though I can't claim I've made the right choices everywhere for long clauses.
Thanks,
Zach
Zach O'Keefe (4):
madvise.2: update THP file/shmem documentation for +5.4 madvise.2: document reliable probe for advice support process_madvise.2: fix capability and ptrace requirements madvise.2: add documentation for MADV_COLLAPSE
man2/madvise.2 | 133 +++++++++++++++++++++++++++++++++++++++-- man2/process_madvise.2 | 31 ++++++++-- 2 files changed, 155 insertions(+), 9 deletions(-)
Signature · 1 line
2.38.0.135.g90850a2211-goog
PATCH v3 1/4 madvise.2: update THP file/shmem documentation for +5.4
From: Zach O'Keefe <[email protected]>
Since Linux 5.4, Transparent Huge Pages now support both file-backed memory and shmem memory. Update MADV_HUGEPAGE advice description to reflect this.
Additionally, expand the description of requirements for memory to be considered eligible for THP: alignment / mapping requirements, VMA flags, prctl(2) settings, inode status, etc.
--- man2/madvise.2 | 38 +++++++++++++++++++++++++++++++++++--- 1 file changed, 35 insertions(+), 3 deletions(-)
man2/madvise.2 +35 -3
@@ -320,8 +320,6 @@ Enable Transparent Huge Pages (THP) for pages in the range specified by .I addr and .IR length . -Currently, Transparent Huge Pages work only with private anonymous pages (see -.BR mmap (2)). The kernel will regularly scan the areas marked as huge page candidates to replace them with huge pages. The kernel will also allocate huge pages directly when the region is
@@ -354,12 +352,46 @@ an access pattern that the developer knows in advance won't risk to increase the memory footprint of the application when transparent hugepages are enabled. .IP +.\" commit 99cb0dbd47a15d395bf3faa78dc122bc5efe3fc0 +Since Linux 5.4, +automatic scan of eligible areas and replacement by huge pages works with +private anonymous pages (see +.BR mmap (2)), +shmem pages, +and file-backed pages. +For all memory types, +memory may only be replaced by huge pages on hugepage-aligned boundaries. +For file-mapped memory \(em including tmpfs (see +.BR tmpfs (2)) +\(em the mapping must also be naturally hugepage-aligned within the file. +Additionally, +for file-backed, +non-tmpfs memory, +the file must not be open for write and the mapping must be executable. +.IP +The VMA must not be marked +.BR VM_NOHUGEPAGE , +.BR VM_HUGETLB , +.BR VM_IO , +.BR VM_DONTEXPAND , +.BR VM_MIXEDMAP , +or +.BR VM_PFNMAP , +nor can it be stack memory or backed by a DAX-enabled device +(unless the DAX device is hot-plugged as System RAM). +The process must also not have +.B PR_SET_THP_DISABLE +set (see +.BR prctl (2) ). +.IP The .B MADV_HUGEPAGE and .B MADV_NOHUGEPAGE operations are available only if the kernel was configured with -.BR CONFIG_TRANSPARENT_HUGEPAGE . +.B CONFIG_TRANSPARENT_HUGEPAGE +and file/shmem memory is only supported if the kernel was configured with +.BR CONFIG_READ_ONLY_THP_FOR_FS . .TP .BR MADV_NOHUGEPAGE " (since Linux 2.6.38)" Ensures that memory in the address range specified by
Signature · 1 line
2.38.0.135.g90850a2211-goog
Re: [PATCH man-pages v3 1/4] madvise.2: update THP file/shmem documentation for +5.4
Hi Zach!
Quoted 11 lines · Zach OKeefe
From: Zach O'Keefe <[email protected]>
Since Linux 5.4, Transparent Huge Pages now support both file-backed memory and shmem memory. Update
MADV_HUGEPAGEadvice description to reflect this.Additionally, expand the description of requirements for memory to be considered eligible for THP: alignment / mapping requirements, VMA flags, prctl(2) settings, inode status, etc.
Signed-off-by Zach O'Keefe <[email protected]>
Patch applied! Thanks.
Since you were interested, I amended it with the following diff:
man2/madvise.2 +5 -4
@@ -361,9 +361,10 @@ .SS Linux-specific advice values and file-backed pages. For all memory types, memory may only be replaced by huge pages on hugepage-aligned boundaries. -For file-mapped memory \(em including tmpfs (see -.BR tmpfs (2)) -\(em the mapping must also be naturally hugepage-aligned within the file. +For file-mapped memory +\(emincluding tmpfs (see +.BR tmpfs (2))\(em +the mapping must also be naturally hugepage-aligned within the file. Additionally, for file-backed, non-tmpfs memory,
@@ -382,7 +383,7 @@ .SS Linux-specific advice values The process must also not have .B PR_SET_THP_DISABLE set (see -.BR prctl (2) ). +.BR prctl (2)). .IP The .B MADV_HUGEPAGE
- The em dashes you used were correct with spaces in both sides; there are those who use them with spaces in both sides and those who use them with no spaces at all. And then there are those who put spaces as if they were parentheses (admittedly this is much less common). I prefer this latter case, since it makes technical texts more parseable.
- I used a semantic newline break for the em dashes, since they are similar to a coma.
- Removed a spurious space in BR that would have made the formatting weird.
Cheers,
Alex
Quoted 65 lines
--- man2/madvise.2 | 38 +++++++++++++++++++++++++++++++++++--- 1 file changed, 35 insertions(+), 3 deletions(-)man2/madvise.2 +35
-3@@ -320,8 +320,6 @@ Enable Transparent Huge Pages (THP) for pages in the range specified by .I addr and .IR length . -Currently, Transparent Huge Pages work only with private anonymous pages (see -.BR mmap (2)). The kernel will regularly scan the areas marked as huge page candidates to replace them with huge pages. The kernel will also allocate huge pages directly when the region is@@ -354,12 +352,46 @@ an access pattern that the developer knows in advance won't risk to increase the memory footprint of the application when transparent hugepages are enabled. .IP +.\" commit 99cb0dbd47a15d395bf3faa78dc122bc5efe3fc0 +Since Linux 5.4, +automatic scan of eligible areas and replacement by huge pages works with +private anonymous pages (see +.BR mmap (2)), +shmem pages, +and file-backed pages. +For all memory types, +memory may only be replaced by huge pages on hugepage-aligned boundaries. +For file-mapped memory \(em including tmpfs (see +.BR tmpfs (2)) +\(em the mapping must also be naturally hugepage-aligned within the file. +Additionally, +for file-backed, +non-tmpfs memory, +the file must not be open for write and the mapping must be executable. +.IP +The VMA must not be marked +.BR VM_NOHUGEPAGE , +.BR VM_HUGETLB , +.BR VM_IO , +.BR VM_DONTEXPAND , +.BR VM_MIXEDMAP , +or +.BR VM_PFNMAP , +nor can it be stack memory or backed by a DAX-enabled device +(unless the DAX device is hot-plugged as System RAM). +The process must also not have +.B PR_SET_THP_DISABLE +set (see +.BR prctl (2) ). +.IP The .B MADV_HUGEPAGE and .B MADV_NOHUGEPAGE operations are available only if the kernel was configured with -.BR CONFIG_TRANSPARENT_HUGEPAGE . +.B CONFIG_TRANSPARENT_HUGEPAGE +and file/shmem memory is only supported if the kernel was configured with +.BR CONFIG_READ_ONLY_THP_FOR_FS . .TP .BR MADV_NOHUGEPAGE " (since Linux 2.6.38)" Ensures that memory in the address range specified by
Signature · 1 line
<http://www.alejandro-colomar.es/>
Re: [PATCH man-pages v3 1/4] madvise.2: update THP file/shmem documentation for +5.4
Quoted 55 lines · Alejandro Colomar
Hi Zach!
Quoted 11 lines · Zach OKeefe
From: Zach O'Keefe <[email protected]>
Since Linux 5.4, Transparent Huge Pages now support both file-backed memory and shmem memory. Update
MADV_HUGEPAGEadvice description to reflect this.Additionally, expand the description of requirements for memory to be considered eligible for THP: alignment / mapping requirements, VMA flags, prctl(2) settings, inode status, etc.
Signed-off-by Zach O'Keefe <[email protected]>Patch applied! Thanks.
Since you were interested, I amended it with the following diff:man2/madvise.2 +5
-5@@ -361,9 +361,10 @@ .SS Linux-specific advice values and file-backed pages. For all memory types, memory may only be replaced by huge pages on hugepage-aligned boundaries. -For file-mapped memory \(em including tmpfs (see -.BR tmpfs (2)) -\(em the mapping must also be naturally hugepage-aligned within the file. +For file-mapped memory +\(emincluding tmpfs (see +.BR tmpfs (2))\(em +the mapping must also be naturally hugepage-aligned within the file. Additionally, for file-backed, non-tmpfs memory,@@ -382,7 +383,7 @@ .SS Linux-specific advice values The process must also not have .B PR_SET_THP_DISABLE set (see -.BR prctl (2) ). +.BR prctl (2)). .IP The .B MADV_HUGEPAGE - The em dashes you used were correct with spaces in both sides; there are thosewho use them with spaces in both sides and those who use them with no spaces at all. And then there are those who put spaces as if they were parentheses (admittedly this is much less common). I prefer this latter case, since it makes technical texts more parseable.
Quoted 7 lines
- I used a semantic newline break for the em dashes, since they are similar to a coma.
- Removed a spurious space in BR that would have made the formatting weird.
Cheers,
Thanks for taking the patch, and thanks for taking the time with the explanation! Glad I wasn't *too* far off :)
Changes look good to me - thank you!
Best,
Zach
Quoted 71 lines
Alex
Quoted 55 lines
--- man2/madvise.2 | 38 +++++++++++++++++++++++++++++++++++--- 1 file changed, 35 insertions(+), 3 deletions(-)man2/madvise.2 +32
-2@@ -320,8 +320,6 @@ Enable Transparent Huge Pages (THP) for pages in the range specified by .I addr and .IR length . -Currently, Transparent Huge Pages work only with private anonymous pages (see -.BR mmap (2)). The kernel will regularly scan the areas marked as huge page candidates to replace them with huge pages. The kernel will also allocate huge pages directly when the region is@@ -354,12 +352,46 @@ an access pattern that the developer knows in advance won't risk to increase the memory footprint of the application when transparent hugepages are enabled. .IP +.\" commit 99cb0dbd47a15d395bf3faa78dc122bc5efe3fc0 +Since Linux 5.4, +automatic scan of eligible areas and replacement by huge pages works with +private anonymous pages (see +.BR mmap (2)), +shmem pages, +and file-backed pages. +For all memory types, +memory may only be replaced by huge pages on hugepage-aligned boundaries. +For file-mapped memory \(em including tmpfs (see +.BR tmpfs (2)) +\(em the mapping must also be naturally hugepage-aligned within the file. +Additionally, +for file-backed, +non-tmpfs memory, +the file must not be open for write and the mapping must be executable. +.IP +The VMA must not be marked +.BR VM_NOHUGEPAGE , +.BR VM_HUGETLB , +.BR VM_IO , +.BR VM_DONTEXPAND , +.BR VM_MIXEDMAP , +or +.BR VM_PFNMAP , +nor can it be stack memory or backed by a DAX-enabled device +(unless the DAX device is hot-plugged as System RAM). +The process must also not have +.B PR_SET_THP_DISABLE +set (see +.BR prctl (2) ). +.IP The .B MADV_HUGEPAGEman2/madvise.2 an access pattern that the developer knows in advance won't risk · line 397
Quoted 10 lines
man2/madvise.2 +3
-1and .B MADV_NOHUGEPAGE operations are available only if the kernel was configured with -.BR CONFIG_TRANSPARENT_HUGEPAGE . +.B CONFIG_TRANSPARENT_HUGEPAGE +and file/shmem memory is only supported if the kernel was configured with +.BR CONFIG_READ_ONLY_THP_FOR_FS . .TP .BR MADV_NOHUGEPAGE " (since Linux 2.6.38)" Ensures that memory in the address range specified by
Re: [PATCH man-pages v3 1/4] madvise.2: update THP file/shmem documentation for +5.4
Hi Zach!
Quoted 3 lines · Zach O'Keefe
Thanks for taking the patch, and thanks for taking the time with the explanation! Glad I wasn't *too* far off :)
:)
BTW, I just documented the linters that are available in the build system.
It would have caught an issue an patch 2/4 that I didn't (although it requires a yet-unreleased groff version; so it won't be available to general public until it is released in a few months). You may want to run it in future patches ;)
I'll tell more about it in the thread for 2/4
Quoted 2 lines
Changes look good to me - thank you!
Cheers,
Alex
Signature · 1 line
<http://www.alejandro-colomar.es/>
PATCH v3 2/4 madvise.2: document reliable probe for advice support
In reply to Zach OKeefeFrom: Zach O'Keefe <[email protected]>
EINVAL is an overloaded error code for madvise(2) and it's not clear under what context it means "advice is not valid" vs another error.
Explicitly document that madvise(0, 0, advice) can reliably be used to probe for kernel support for "advice", returning zero iff "advice" is supported by the kernel.
--- man2/madvise.2 | 5 +++++ 1 file changed, 5 insertions(+)
man2/madvise.2 +5 -0
@@ -790,6 +790,11 @@ that are not mapped, the Linux version of ignores them and applies the call to the rest (but returns .B ENOMEM from the system call, as it should). +.PP +.IR madvise(0,\ 0,\ advice) +will return zero iff +.I advice +is supported by the kernel and can be relied on to probe for support. .\" .SH HISTORY .\" The .\" .BR madvise ()
Signature · 1 line
2.38.0.135.g90850a2211-goog
Re: [PATCH man-pages v3 2/4] madvise.2: document reliable probe for advice support
Hi Zach,
Quoted 10 lines · Zach OKeefe
From: Zach O'Keefe <[email protected]>
EINVAL is an overloaded error code for madvise(2) and it's not clear under what context it means "advice is not valid" vs another error.
Explicitly document that
madvise(0, 0, advice)can reliably be used to probe for kernel support for "advice", returning zero iff "advice" is supported by the kernel.Signed-off-by Zach O'Keefe <[email protected]>
Patch applied.
Thanks,
Alex
Quoted 20 lines
--- man2/madvise.2 | 5 +++++ 1 file changed, 5 insertions(+)man2/madvise.2 +5
-0@@ -790,6 +790,11 @@ that are not mapped, the Linux version of ignores them and applies the call to the rest (but returns .B ENOMEM from the system call, as it should). +.PP +.IR madvise(0,\ 0,\ advice) +will return zero iff +.I advice +is supported by the kernel and can be relied on to probe for support. .\" .SH HISTORY .\" The .\" .BR madvise ()
Signature · 1 line
<http://www.alejandro-colomar.es/>
Re: [PATCH man-pages v3 2/4] madvise.2: document reliable probe for advice support
Quoted 9 lines · Alejandro Colomar
Hi Zach,
Quoted 5 lines · Zach OKeefe
From: Zach O'Keefe <[email protected]>
EINVAL is an overloaded error code for madvise(2) and it's not clear under what context it means "advice is not valid" vs another error.
Quoted 7 lines
Quoted 5 lines
Explicitly document that
madvise(0, 0, advice)can reliably be used to probe for kernel support for "advice", returning zero iff "advice" is supported by the kernel.Signed-off-by Zach O'Keefe <[email protected]>Patch applied.
Thank you!
Best,
Zach
Quoted 27 lines
Thanks,
Alex
Quoted 10 lines
--- man2/madvise.2 | 5 +++++ 1 file changed, 5 insertions(+)man2/madvise.2 +0
-0@@ -790,6 +790,11 @@ that are not mapped, the Linux version of ignores them and applies the call to the rest (but returnsman2/madvise.2 that are not mapped, the Linux version of · line 800
Quoted 10 lines
man2/madvise.2 +5
-0.B ENOMEM from the system call, as it should). +.PP +.IR madvise(0,\ 0,\ advice) +will return zero iff +.I advice +is supported by the kernel and can be relied on to probe for support. .\" .SH HISTORY .\" The .\" .BR madvise ()
Re: [PATCH man-pages v3 2/4] madvise.2: document reliable probe for advice support
Hi Zach,
Quoted 11 lines · Zach O'Keefe
Quoted 1 line
Patch applied.
Thank you!
Best,
ZachQuoted 4 lines
Thanks,
Alex
I just caught a small formatting issue by running some linters through that file:
$ make -ij lint >/dev/null 2>&1 $ touch man2/madvise.2 $ make -k lint-man LINT (groff) tmp/lint/man2/madvise.2.lint-man.groff.touch an.tmac:man2/madvise.2:795: style: .IR expects at least 2 arguments, got 1 found style problems; aborting make: *** [lib/lint-man.mk:77: tmp/lint/man2/madvise.2.lint-man.groff.touch] Error 1 LINT (mandoc) tmp/lint/man2/madvise.2.lint-man.mandoc.touch mandoc: man2/madvise.2:15:5: STYLE: lower case character in document title: TH madvise mandoc: man2/madvise.2:15:15: WARNING: cannot parse date, using it verbatim: TH (date) make: *** [lib/lint-man.mk:87: tmp/lint/man2/madvise.2.lint-man.mandoc.touch] Error 2 make: Target 'lint-man' not remade because of errors.
The issues reported by mandoc(1) are expected; but the issue reported by groff(1) is legit. The offending line is:
$ sed -n 795p man2/madvise.2 .IR madvise(0,\ 0,\ advice)
Since all of the line is formatted in italics (no roman part), it should be .I and not .IR. After fixing that line, the linter now says:
$ make lint-man -k LINT (groff) tmp/lint/man2/madvise.2.lint-man.groff.touch LINT (mandoc) tmp/lint/man2/madvise.2.lint-man.mandoc.touch mandoc: man2/madvise.2:15:5: STYLE: lower case character in document title: TH madvise mandoc: man2/madvise.2:15:15: WARNING: cannot parse date, using it verbatim: TH (date) make: *** [lib/lint-man.mk:87: tmp/lint/man2/madvise.2.lint-man.mandoc.touch] Error 2 make: Target 'lint-man' not remade because of errors.
Ignoring the spurious mandoc(1) warnings, it's all good.
Documentation about how to use this feature is here (written today, so no way you could have run it; I should have, though :D):
<https://git.kernel.org/pub/scm/docs/man-pages/man-pages.git/tree/CONTRIBUTING#n133>
Cheers,
Alex
Quoted 23 lines
Quoted 23 lines
Quoted 10 lines
--- man2/madvise.2 | 5 +++++ 1 file changed, 5 insertions(+)man2/madvise.2 +0
-0@@ -790,6 +790,11 @@ that are not mapped, the Linux version of ignores them and applies the call to the rest (but returnsman2/madvise.2 that are not mapped, the Linux version of · line 800
Quoted 10 lines
man2/madvise.2 +5
-0.B ENOMEM from the system call, as it should). +.PP +.IR madvise(0,\ 0,\ advice) +will return zero iff +.I advice +is supported by the kernel and can be relied on to probe for support. .\" .SH HISTORY .\" The .\" .BR madvise ()
Signature · 1 line
<http://www.alejandro-colomar.es/>
Re: [PATCH man-pages v3 2/4] madvise.2: document reliable probe for advice support
Hey Alex,
Quoted 57 lines · Alejandro Colomar
Hi Zach,
Quoted 11 lines · Zach O'Keefe
Quoted 1 line
Patch applied.
Thank you!
Best,
ZachQuoted 4 lines
Thanks,
Alex
I just caught a small formatting issue by running some linters through that file:
$ make -ij lint >/dev/null 2>&1 $ touch man2/madvise.2 $ make -k lint-man LINT (groff) tmp/lint/man2/madvise.2.lint-man.groff.touch an.tmac:man2/madvise.2:795: style: .IR expects at least 2 arguments, got 1 found style problems; aborting make: *** [lib/lint-man.mk:77: tmp/lint/man2/madvise.2.lint-man.groff.touch] Error 1 LINT (mandoc) tmp/lint/man2/madvise.2.lint-man.mandoc.touch mandoc: man2/madvise.2:15:5: STYLE: lower case character in document title: TH madvise mandoc: man2/madvise.2:15:15: WARNING: cannot parse date, using it verbatim: TH (date) make: *** [lib/lint-man.mk:87: tmp/lint/man2/madvise.2.lint-man.mandoc.touch] Error 2 make: Target 'lint-man' not remade because of errors.The issues reported by mandoc(1) are expected; but the issue reported by groff(1) is legit. The offending line is:
$ sed -n 795p man2/madvise.2 .IR madvise(0,\ 0,\ advice)Since all of the line is formatted in italics (no roman part), it should be .I and not .IR.
After fixing that line, the linter now says:$ make lint-man -k LINT (groff) tmp/lint/man2/madvise.2.lint-man.groff.touch LINT (mandoc) tmp/lint/man2/madvise.2.lint-man.mandoc.touch mandoc: man2/madvise.2:15:5: STYLE: lower case character in document title: TH madvise mandoc: man2/madvise.2:15:15: WARNING: cannot parse date, using it verbatim: TH (date) make: *** [lib/lint-man.mk:87: tmp/lint/man2/madvise.2.lint-man.mandoc.touch] Error 2 make: Target 'lint-man' not remade because of errors.
Quoted 6 lines
Ignoring the spurious mandoc(1) warnings, it's all good.
Documentation about how to use this feature is here (written today, so no way you could have run it; I should have, though :D):
<https://git.kernel.org/pub/scm/docs/man-pages/man-pages.git/tree/CONTRIBUTING#n133>
Thanks for taking a look here and thanks for adding the documentation to CONTRIBUTING -- was easy to follow.
As you mentioned in 1/4, I don't have the groff(1) version (GNU groff version 1.22.4) to see the error, but it seems like a real fix (& I've taken a look at your fix patch - thank you :) )
Thanks again,
Best,
Zach
PATCH v3 3/4 process_madvise.2: fix capability and ptrace requirements
In reply to Zach OKeefeFrom: Zach O'Keefe <[email protected]>
The initial commit of process_madvise(2) to Linux, commit ecb8ac8b1f14 ("mm/madvise: introduce process_madvise() syscall: an external memory hinting API"), relied on PTRACE_MODE_ATTACH_FSCREDS (see ptrace(2)), but was amended by commit 96cfe2c0fd23 ("mm/madvise: replace ptrace attach requirement for process_madvise") which replaced this with a combination of PTRACE_MODE_READ and CAP_SYS_NICE (PTRACE_MODE_READ to prevent leaking ASLR metadata and CAP_SYS_NICE for influencing process performance).
The initial commit of process_madvise(2) to man-pages project, made after the second patch, included two errors:
1) CAP_SYS_ADMIN instead of CAP_SYS_NICE
2) PTRACE_MODE_READ_REALCREDS instead of PTRACE_MODE_READ_FSCREDS
Correct this in the man-page for process_madvise(2).
--- man2/process_madvise.2 | 21 +++++++++++++++++---- 1 file changed, 17 insertions(+), 4 deletions(-)
man2/process_madvise.2 +17 -4
@@ -105,16 +105,20 @@ remote process. No further elements will be processed beyond that point. (See the discussion regarding partial advice in RETURN VALUE.) .PP -Permission to apply advice to another process is governed by a +.\" commit 96cfe2c0fd23ea7c2368d14f769d287e7ae1082e +Starting in Linux 5.12, +permission to apply advice to another process is governed by ptrace access mode -.B PTRACE_MODE_READ_REALCREDS +.B PTRACE_MODE_READ_FSCREDS check (see .BR ptrace (2)); in addition, because of the performance implications of applying the advice, the caller must have the -.B CAP_SYS_ADMIN -capability. +.B CAP_SYS_NICE +capability +(see +.BR capabilities (7)). .SH RETURN VALUE On success, .BR process_madvise ()
@@ -180,6 +184,15 @@ configuration option. The .BR process_madvise () system call is Linux-specific. +.SH NOTES +When this system call first appeared in Linux 5.10, +permission to apply advice to another process was entirely governed by +ptrace access mode +.B PTRACE_MODE_ATTACH_FSCREDS +check (see +.BR ptrace (2)). +This requirement was relaxed in Linux 5.12 so that the caller didn't require +full control over the target process. .SH SEE ALSO .BR madvise (2), .BR pidfd_open (2),
Signature · 1 line
2.38.0.135.g90850a2211-goog
Re: [PATCH man-pages v3 3/4] process_madvise.2: fix capability and ptrace requirements
Hi Suren,
Quoted 17 lines · Zach OKeefe
From: Zach O'Keefe <[email protected]>
The initial commit of
process_madvise(2)to Linux, commitecb8ac8b1f14("mm/madvise: introduceprocess_madvise()syscall: an external memory hinting API"), relied onPTRACE_MODE_ATTACH_FSCREDS(see ptrace(2)), but was amended by commit96cfe2c0fd23("mm/madvise: replace ptrace attach requirement forprocess_madvise") which replaced this with a combination ofPTRACE_MODE_READandCAP_SYS_NICE(PTRACE_MODE_READto prevent leaking ASLR metadata andCAP_SYS_NICEfor influencing process performance).The initial commit of
process_madvise(2)to man-pages project, made after the second patch, included two errors:1)
CAP_SYS_ADMINinstead ofCAP_SYS_NICE
2)PTRACE_MODE_READ_REALCREDSinstead ofPTRACE_MODE_READ_FSCREDS
Quoted 6 lines
Correct this in the man-page for
process_madvise(2).Fixes a144f458b ("process_madvise.2: Document process_madvise(2)")Cc Suren Baghdasaryan <[email protected]>Cc Minchan Kim <[email protected]>Signed-off-by Zach O'Keefe <[email protected]>
You added your Reviewed-by tag to v2 of this patch. I guess you'd like to put it in this one too, but since it changed slightly, I'd like you to confirm.
Thanks,
Alex
Quoted 49 lines
--- man2/process_madvise.2 | 21 +++++++++++++++++---- 1 file changed, 17 insertions(+), 4 deletions(-)man2/process_madvise.2 +17
-4@@ -105,16 +105,20 @@ remote process. No further elements will be processed beyond that point. (See the discussion regarding partial advice in RETURN VALUE.) .PP -Permission to apply advice to another process is governed by a +.\" commit 96cfe2c0fd23ea7c2368d14f769d287e7ae1082e +Starting in Linux 5.12, +permission to apply advice to another process is governed by ptrace access mode -.B PTRACE_MODE_READ_REALCREDS +.B PTRACE_MODE_READ_FSCREDS check (see .BR ptrace (2)); in addition, because of the performance implications of applying the advice, the caller must have the -.B CAP_SYS_ADMIN -capability. +.B CAP_SYS_NICE +capability +(see +.BR capabilities (7)). .SH RETURN VALUE On success, .BR process_madvise ()@@ -180,6 +184,15 @@ configuration option. The .BR process_madvise () system call is Linux-specific. +.SH NOTES +When this system call first appeared in Linux 5.10, +permission to apply advice to another process was entirely governed by +ptrace access mode +.B PTRACE_MODE_ATTACH_FSCREDS +check (see +.BR ptrace (2)). +This requirement was relaxed in Linux 5.12 so that the caller didn't require +full control over the target process. .SH SEE ALSO .BR madvise (2), .BR pidfd_open (2),
Signature · 1 line
<http://www.alejandro-colomar.es/>
Suren Baghdasaryan
Reviewed-byRe: [PATCH man-pages v3 3/4] process_madvise.2: fix capability and ptrace requirements
Hi Alex,
Quoted 27 lines · Alejandro Colomar
Hi Suren,
Quoted 23 lines · Zach OKeefe
From: Zach O'Keefe <[email protected]>
The initial commit of
process_madvise(2)to Linux, commitecb8ac8b1f14("mm/madvise: introduceprocess_madvise()syscall: an external memory hinting API"), relied onPTRACE_MODE_ATTACH_FSCREDS(see ptrace(2)), but was amended by commit96cfe2c0fd23("mm/madvise: replace ptrace attach requirement forprocess_madvise") which replaced this with a combination ofPTRACE_MODE_READandCAP_SYS_NICE(PTRACE_MODE_READto prevent leaking ASLR metadata andCAP_SYS_NICEfor influencing process performance).The initial commit of
process_madvise(2)to man-pages project, made after the second patch, included two errors:1)
CAP_SYS_ADMINinstead ofCAP_SYS_NICE
2)PTRACE_MODE_READ_REALCREDSinstead ofPTRACE_MODE_READ_FSCREDSCorrect this in the man-page for
process_madvise(2).Fixes a144f458b ("process_madvise.2: Document process_madvise(2)")Cc Suren Baghdasaryan <[email protected]>Cc Minchan Kim <[email protected]>Signed-off-by Zach O'Keefe <[email protected]>
Quoted 3 lines
You added your Reviewed-by tag to v2 of this patch. I guess you'd like to put it in this one too, but since it changed slightly, I'd like you to confirm.
Thanks for the reminder!
Suren.
Quoted 57 lines
Thanks,
Alex
Quoted 39 lines
--- man2/process_madvise.2 | 21 +++++++++++++++++---- 1 file changed, 17 insertions(+), 4 deletions(-)man2/process_madvise.2 +10
-4@@ -105,16 +105,20 @@ remote process. No further elements will be processed beyond that point. (See the discussion regarding partial advice in RETURN VALUE.) .PP -Permission to apply advice to another process is governed by a +.\" commit 96cfe2c0fd23ea7c2368d14f769d287e7ae1082e +Starting in Linux 5.12, +permission to apply advice to another process is governed by ptrace access mode -.B PTRACE_MODE_READ_REALCREDS +.B PTRACE_MODE_READ_FSCREDS check (see .BR ptrace (2)); in addition, because of the performance implications of applying the advice, the caller must have the -.B CAP_SYS_ADMIN -capability. +.B CAP_SYS_NICE +capability +(see +.BR capabilities (7)). .SH RETURN VALUE On success, .BR process_madvise ()@@ -180,6 +184,15 @@ configuration option. The .BR process_madvise () system call is Linux-specific. +.SH NOTES +When this system call first appeared in Linux 5.10,man2/process_madvise.2 configuration option. · line 198
Quoted 10 lines
man2/process_madvise.2 +7
-0+permission to apply advice to another process was entirely governed by +ptrace access mode +.B PTRACE_MODE_ATTACH_FSCREDS +check (see +.BR ptrace (2)). +This requirement was relaxed in Linux 5.12 so that the caller didn't require +full control over the target process. .SH SEE ALSO .BR madvise (2), .BR pidfd_open (2),
Re: [PATCH man-pages v3 3/4] process_madvise.2: fix capability and ptrace requirements
Hi Suren, Zach,
Quoted 25 lines · Suren Baghdasaryan
Hi Alex,
Quoted 21 lines · Alejandro Colomar
Hi Suren,
Quoted 17 lines · Zach OKeefe
From: Zach O'Keefe <[email protected]>
The initial commit of
process_madvise(2)to Linux, commitecb8ac8b1f14("mm/madvise: introduceprocess_madvise()syscall: an external memory hinting API"), relied onPTRACE_MODE_ATTACH_FSCREDS(see ptrace(2)), but was amended by commit96cfe2c0fd23("mm/madvise: replace ptrace attach requirement forprocess_madvise") which replaced this with a combination ofPTRACE_MODE_READandCAP_SYS_NICE(PTRACE_MODE_READto prevent leaking ASLR metadata andCAP_SYS_NICEfor influencing process performance).The initial commit of
process_madvise(2)to man-pages project, made after the second patch, included two errors:1)
CAP_SYS_ADMINinstead ofCAP_SYS_NICE
2)PTRACE_MODE_READ_REALCREDSinstead ofPTRACE_MODE_READ_FSCREDS
Quoted 8 lines
Quoted 6 lines
Quoted 6 lines
Correct this in the man-page for
process_madvise(2).Fixes a144f458b ("process_madvise.2: Document process_madvise(2)")Cc Suren Baghdasaryan <[email protected]>Cc Minchan Kim <[email protected]>Signed-off-by Zach O'Keefe <[email protected]>Reviewed-by Suren Baghdasaryan <[email protected]>
Thanks! Patch applied.
Quoted 6 lines
Quoted 4 lines
You added your Reviewed-by tag to v2 of this patch. I guess you'd like to put it in this one too, but since it changed slightly, I'd like you to confirm.
Thanks for the reminder!
:)
Cheers,
Alex
Quoted 59 lines
Suren.
Quoted 57 lines
Thanks,
Alex
Quoted 39 lines
--- man2/process_madvise.2 | 21 +++++++++++++++++---- 1 file changed, 17 insertions(+), 4 deletions(-)man2/process_madvise.2 +10
-4@@ -105,16 +105,20 @@ remote process. No further elements will be processed beyond that point. (See the discussion regarding partial advice in RETURN VALUE.) .PP -Permission to apply advice to another process is governed by a +.\" commit 96cfe2c0fd23ea7c2368d14f769d287e7ae1082e +Starting in Linux 5.12, +permission to apply advice to another process is governed by ptrace access mode -.B PTRACE_MODE_READ_REALCREDS +.B PTRACE_MODE_READ_FSCREDS check (see .BR ptrace (2)); in addition, because of the performance implications of applying the advice, the caller must have the -.B CAP_SYS_ADMIN -capability. +.B CAP_SYS_NICE +capability +(see +.BR capabilities (7)). .SH RETURN VALUE On success, .BR process_madvise ()@@ -180,6 +184,15 @@ configuration option. The .BR process_madvise () system call is Linux-specific. +.SH NOTES +When this system call first appeared in Linux 5.10,man2/process_madvise.2 configuration option. · line 198
Quoted 10 lines
man2/process_madvise.2 +7
-0+permission to apply advice to another process was entirely governed by +ptrace access mode +.B PTRACE_MODE_ATTACH_FSCREDS +check (see +.BR ptrace (2)). +This requirement was relaxed in Linux 5.12 so that the caller didn't require +full control over the target process. .SH SEE ALSO .BR madvise (2), .BR pidfd_open (2),
Signature · 1 line
<http://www.alejandro-colomar.es/>
Re: [PATCH man-pages v3 3/4] process_madvise.2: fix capability and ptrace requirements
Quoted 31 lines · Alejandro Colomar
Hi Suren, Zach,
Quoted 27 lines · Suren Baghdasaryan
Hi Alex,
Quoted 23 lines · Alejandro Colomar
Hi Suren,
Quoted 19 lines · Zach OKeefe
From: Zach O'Keefe <[email protected]>
The initial commit of
process_madvise(2)to Linux, commitecb8ac8b1f14("mm/madvise: introduceprocess_madvise()syscall: an external memory hinting API"), relied onPTRACE_MODE_ATTACH_FSCREDS(see ptrace(2)), but was amended by commit96cfe2c0fd23("mm/madvise: replace ptrace attach requirement forprocess_madvise") which replaced this with a combination ofPTRACE_MODE_READandCAP_SYS_NICE(PTRACE_MODE_READto prevent leaking ASLR metadata andCAP_SYS_NICEfor influencing process performance).The initial commit of
process_madvise(2)to man-pages project, made after the second patch, included two errors:1)
CAP_SYS_ADMINinstead ofCAP_SYS_NICE
2)PTRACE_MODE_READ_REALCREDSinstead ofPTRACE_MODE_READ_FSCREDSCorrect this in the man-page for
process_madvise(2).
Quoted 8 lines
Quoted 6 lines
Quoted 4 lines
Quoted 4 lines
Fixes a144f458b ("process_madvise.2: Document process_madvise(2)")Cc Suren Baghdasaryan <[email protected]>Cc Minchan Kim <[email protected]>Signed-off-by Zach O'Keefe <[email protected]>Reviewed-by Suren Baghdasaryan <[email protected]>Thanks! Patch applied.
Thanks Suren & Alex!
Best,
Zach
Quoted 75 lines
Quoted 6 lines
Quoted 4 lines
You added your Reviewed-by tag to v2 of this patch. I guess you'd like to put it in this one too, but since it changed slightly, I'd like you to confirm.
Thanks for the reminder!
:)
Cheers,
Alex
Quoted 59 lines
Suren.
Quoted 57 lines
Thanks,
Alex
Quoted 39 lines
--- man2/process_madvise.2 | 21 +++++++++++++++++---- 1 file changed, 17 insertions(+), 4 deletions(-)man2/process_madvise.2 +10
-4@@ -105,16 +105,20 @@ remote process. No further elements will be processed beyond that point. (See the discussion regarding partial advice in RETURN VALUE.) .PP -Permission to apply advice to another process is governed by a +.\" commit 96cfe2c0fd23ea7c2368d14f769d287e7ae1082e +Starting in Linux 5.12, +permission to apply advice to another process is governed by ptrace access mode -.B PTRACE_MODE_READ_REALCREDS +.B PTRACE_MODE_READ_FSCREDS check (see .BR ptrace (2)); in addition, because of the performance implications of applying the advice, the caller must have the -.B CAP_SYS_ADMIN -capability. +.B CAP_SYS_NICE +capability +(see +.BR capabilities (7)). .SH RETURN VALUE On success, .BR process_madvise ()@@ -180,6 +184,15 @@ configuration option. The .BR process_madvise () system call is Linux-specific. +.SH NOTES +When this system call first appeared in Linux 5.10,man2/process_madvise.2 configuration option. · line 198
Quoted 10 lines
man2/process_madvise.2 +7
-0+permission to apply advice to another process was entirely governed by +ptrace access mode +.B PTRACE_MODE_ATTACH_FSCREDS +check (see +.BR ptrace (2)). +This requirement was relaxed in Linux 5.12 so that the caller didn't require +full control over the target process. .SH SEE ALSO .BR madvise (2), .BR pidfd_open (2),
PATCH v3 4/4 madvise.2: add documentation for MADV_COLLAPSE
In reply to Zach OKeefeFrom: Zach O'Keefe <[email protected]>
Linux 6.1 introduced MADV_COLLAPSE in upstream commit 7d8faaf15545 ("mm/madvise: introduce MADV_COLLAPSE sync hugepage collapse") and upstream commit 34488399fa08 ("mm/madvise: add file and shmem support to MADV_COLLAPSE"). Update the man-pages for madvise(2) and process_madvise(2).
--- man2/madvise.2 | 90 +++++++++++++++++++++++++++++++++++++++++- man2/process_madvise.2 | 10 +++++ 2 files changed, 98 insertions(+), 2 deletions(-)
man2/madvise.2 +88 -2
@@ -385,9 +385,10 @@ set (see .BR prctl (2) ). .IP The -.B MADV_HUGEPAGE +.BR MADV_HUGEPAGE , +.BR MADV_NOHUGEPAGE , and -.B MADV_NOHUGEPAGE +.B MADV_COLLAPSE operations are available only if the kernel was configured with .B CONFIG_TRANSPARENT_HUGEPAGE and file/shmem memory is only supported if the kernel was configured with
@@ -400,6 +401,81 @@ and .I length will not be backed by transparent hugepages. .TP +.BR MADV_COLLAPSE " (since Linux 6.1)" +.\" commit 7d8faaf155454f8798ec56404faca29a82689c77 +.\" commit 34488399fa08faaf664743fa54b271eb6f9e1321 +Perform a best-effort synchronous collapse of the native pages mapped by the +memory range into Transparent Huge Pages (THPs). +.B MADV_COLLAPSE +operates on the current state of memory of the calling process and makes no +persistent changes or guarantees on how pages will be mapped, +constructed, +or faulted in the future. +.IP +.B MADV_COLLAPSE +supports private anonymous pages (see +.BR mmap (2)), +shmem pages, +and file-backed pages. +See +.B MADV_HUGEPAGE +for general information on memory requirements for THP. +If the range provided spans multiple VMAs, +the semantics of the collapse over each VMA is independent from the others. +If collapse of a given huge page-aligned/sized region fails, +the operation may continue to attempt collapsing the remainder of the +specified memory. +.B MADV_COLLAPSE +will automatically clamp the provided range to be hugepage-aligned. +.IP +All non-resident pages covered by the range will first be +swapped/faulted-in, +before being copied onto a freshly allocated hugepage. +If the native pages compose the same PTE-mapped hugepage, +and are suitably aligned, +allocation of a new hugepage may be elided and collapse may happen +in-place. +Unmapped pages will have their data directly initialized to 0 in the new +hugepage. +However, +for every eligible hugepage-aligned/sized region to be collapsed, +at least one page must currently be backed by physical memory. +.IP +.BR MADV_COLLAPSE +is independent of any sysfs +(see +.BR sysfs (5)) +setting under +.IR /sys/kernel/mm/transparent_hugepage , +both in terms of determining THP eligibility, +and allocation semantics. +See Linux kernel source file +.I Documentation/admin\-guide/mm/transhuge.rst +for more information. +.BR MADV_COLLAPSE +also ignores +.B huge= +tmpfs mount when operating on tmpfs files. +Allocation for the new hugepage may enter direct reclaim and/or compaction, +regardless of VMA flags +(though +.BR VM_NOHUGEPAGE +is still respected). +.IP +When the system has multiple NUMA nodes, +the hugepage will be allocated from the node providing the most native +pages. +.IP +If all hugepage-sized/aligned regions covered by the provided range were +either successfully collapsed, +or were already PMD-mapped THPs, +this operation will be deemed successful. +Note that this doesn’t guarantee anything about other possible mappings of +the memory. +Also note that many failures might have occurred since the operation may +continue to collapse in the event collapse of a single hugepage-sized/aligned +region fails. +.TP .BR MADV_DONTDUMP " (since Linux 3.4)" .\" commit 909af768e88867016f427264ae39d27a57b6a8ed .\" commit accb61fe7bb0f5c2a4102239e4981650f9048519
@@ -619,6 +695,11 @@ A kernel resource was temporarily unavailable. .B EBADF The map exists, but the area maps something that isn't a file. .TP +.B EBUSY +(for +.BR MADV_COLLAPSE ) +Could not charge hugepage to cgroup: cgroup limit exceeded. +.TP .B EFAULT .I advice is
@@ -716,6 +797,11 @@ maximum resident set size. Not enough memory: paging in failed. .TP .B ENOMEM +(for +.BR MADV_COLLAPSE ) +Not enough memory: could not allocate hugepage. +.TP +.B ENOMEM Addresses in the specified range are not currently mapped, or are outside the address space of the process. .TP
man2/process_madvise.2 +10 -0
@@ -73,6 +73,10 @@ argument is one of the following values: See .BR madvise (2). .TP +.B MADV_COLLAPSE +See +.BR madvise (2). +.TP .B MADV_PAGEOUT See .BR madvise (2).
@@ -173,6 +177,12 @@ The caller does not have permission to access the address space of the process .TP .B ESRCH The target process does not exist (i.e., it has terminated and been waited on). +.PP +See +.BR madvise (2) +for +.IR advice -specific +errors. .SH VERSIONS This system call first appeared in Linux 5.10. .\" commit ecb8ac8b1f146915aa6b96449b66dd48984caacc
Signature · 1 line
2.38.0.135.g90850a2211-goog
Re: [PATCH man-pages v3 4/4] madvise.2: add documentation for MADV_COLLAPSE
Hi Zach!
Quoted 11 lines · Zach OKeefe
From: Zach O'Keefe <[email protected]>
Linux 6.1 introduced
MADV_COLLAPSEin upstream commit7d8faaf15545("mm/madvise: introduceMADV_COLLAPSEsync hugepage collapse") and upstream commit34488399fa08("mm/madvise: add file and shmem support toMADV_COLLAPSE"). Update the man-pages for madvise(2) andprocess_madvise(2).
There are a few issues with this patch:
alx@asus5775:~/src/linux/man-pages/man-pages$ make lint-man-groff LINT (groff) tmp/lint/man2/madvise.2.lint-man.groff.touch eqn:man2/madvise.2:473: error: invalid input character code '128' eqn:man2/madvise.2:473: error: invalid input character code '153' an.tmac:man2/madvise.2:445: style: .BR expects at least 2 arguments, got 1 an.tmac:man2/madvise.2:456: style: .BR expects at least 2 arguments, got 1 an.tmac:man2/madvise.2:463: style: .BR expects at least 2 arguments, got 1 found style problems; aborting make: *** [lib/lint-man.mk:77: tmp/lint/man2/madvise.2.lint-man.groff.touch] Error 1
Let's investigate them:
alx@asus5775:~/src/linux/man-pages/man-pages$ sed -n 473p man2/madvise.2 this operation will be deemed successful.
This one was a bit difficult to track, since the line count seems to be off by one:
alx@asus5775:~/src/linux/man-pages/man-pages$ tbl man2/madvise.2 | hd | grep -C1 ' 80 ' 00003d40 63 65 73 73 66 75 6c 2e 0a 4e 6f 74 65 20 74 68 |cessful..Note th| 00003d50 61 74 20 74 68 69 73 20 64 6f 65 73 6e e2 80 99 |at this doesn...| 00003d60 74 20 67 75 61 72 61 6e 74 65 65 20 61 6e 79 74 |t guarantee anyt| alx@asus5775:~/src/linux/man-pages/man-pages$ sed -n 474p man2/madvise.2 Note that this doesn’t guarantee anything about other possible mappings of
The issue was in line 474, and the issue is that it uses a weird single quote. Please use the foillowing ASCII character for the single quote (see ascii(7)):
047 39 27 '
The rest of issues seems trivial:
Use .B instead of .BR because there's no "roman" (i.e., non-bold) part.
alx@asus5775:~/src/linux/man-pages/man-pages$ sed -n 445p man2/madvise.2 .BR MADV_COLLAPSE alx@asus5775:~/src/linux/man-pages/man-pages$ sed -n 456p man2/madvise.2 .BR MADV_COLLAPSE alx@asus5775:~/src/linux/man-pages/man-pages$ sed -n 463p man2/madvise.2 .BR VM_NOHUGEPAGE
I'll report a bug to groff(1) about the issue with the line count.
Cheers,
Alex
Quoted 156 lines
--- man2/madvise.2 | 90 +++++++++++++++++++++++++++++++++++++++++- man2/process_madvise.2 | 10 +++++ 2 files changed, 98 insertions(+), 2 deletions(-)man2/madvise.2 +88
-2@@ -385,9 +385,10 @@ set (see .BR prctl (2) ). .IP The -.B MADV_HUGEPAGE +.BR MADV_HUGEPAGE , +.BR MADV_NOHUGEPAGE , and -.B MADV_NOHUGEPAGE +.B MADV_COLLAPSE operations are available only if the kernel was configured with .B CONFIG_TRANSPARENT_HUGEPAGE and file/shmem memory is only supported if the kernel was configured with@@ -400,6 +401,81 @@ and .I length will not be backed by transparent hugepages. .TP +.BR MADV_COLLAPSE " (since Linux 6.1)" +.\" commit 7d8faaf155454f8798ec56404faca29a82689c77 +.\" commit 34488399fa08faaf664743fa54b271eb6f9e1321 +Perform a best-effort synchronous collapse of the native pages mapped by the +memory range into Transparent Huge Pages (THPs). +.B MADV_COLLAPSE +operates on the current state of memory of the calling process and makes no +persistent changes or guarantees on how pages will be mapped, +constructed, +or faulted in the future. +.IP +.B MADV_COLLAPSE +supports private anonymous pages (see +.BR mmap (2)), +shmem pages, +and file-backed pages. +See +.B MADV_HUGEPAGE +for general information on memory requirements for THP. +If the range provided spans multiple VMAs, +the semantics of the collapse over each VMA is independent from the others. +If collapse of a given huge page-aligned/sized region fails, +the operation may continue to attempt collapsing the remainder of the +specified memory. +.B MADV_COLLAPSE +will automatically clamp the provided range to be hugepage-aligned. +.IP +All non-resident pages covered by the range will first be +swapped/faulted-in, +before being copied onto a freshly allocated hugepage. +If the native pages compose the same PTE-mapped hugepage, +and are suitably aligned, +allocation of a new hugepage may be elided and collapse may happen +in-place. +Unmapped pages will have their data directly initialized to 0 in the new +hugepage. +However, +for every eligible hugepage-aligned/sized region to be collapsed, +at least one page must currently be backed by physical memory. +.IP +.BR MADV_COLLAPSE +is independent of any sysfs +(see +.BR sysfs (5)) +setting under +.IR /sys/kernel/mm/transparent_hugepage , +both in terms of determining THP eligibility, +and allocation semantics. +See Linux kernel source file +.I Documentation/admin\-guide/mm/transhuge.rst +for more information. +.BR MADV_COLLAPSE +also ignores +.B huge= +tmpfs mount when operating on tmpfs files. +Allocation for the new hugepage may enter direct reclaim and/or compaction, +regardless of VMA flags +(though +.BR VM_NOHUGEPAGE +is still respected). +.IP +When the system has multiple NUMA nodes, +the hugepage will be allocated from the node providing the most native +pages. +.IP +If all hugepage-sized/aligned regions covered by the provided range were +either successfully collapsed, +or were already PMD-mapped THPs, +this operation will be deemed successful. +Note that this doesn’t guarantee anything about other possible mappings of +the memory. +Also note that many failures might have occurred since the operation may +continue to collapse in the event collapse of a single hugepage-sized/aligned +region fails. +.TP .BR MADV_DONTDUMP " (since Linux 3.4)" .\" commit 909af768e88867016f427264ae39d27a57b6a8ed .\" commit accb61fe7bb0f5c2a4102239e4981650f9048519@@ -619,6 +695,11 @@ A kernel resource was temporarily unavailable. .B EBADF The map exists, but the area maps something that isn't a file. .TP +.B EBUSY +(for +.BR MADV_COLLAPSE ) +Could not charge hugepage to cgroup: cgroup limit exceeded. +.TP .B EFAULT .I advice is@@ -716,6 +797,11 @@ maximum resident set size. Not enough memory: paging in failed. .TP .B ENOMEM +(for +.BR MADV_COLLAPSE ) +Not enough memory: could not allocate hugepage. +.TP +.B ENOMEM Addresses in the specified range are not currently mapped, or are outside the address space of the process. .TPman2/process_madvise.2 +10
-0@@ -73,6 +73,10 @@ argument is one of the following values: See .BR madvise (2). .TP +.B MADV_COLLAPSE +See +.BR madvise (2). +.TP .B MADV_PAGEOUT See .BR madvise (2).@@ -173,6 +177,12 @@ The caller does not have permission to access the address space of the process .TP .B ESRCH The target process does not exist (i.e., it has terminated and been waited on). +.PP +See +.BR madvise (2) +for +.IR advice -specific +errors. .SH VERSIONS This system call first appeared in Linux 5.10. .\" commit ecb8ac8b1f146915aa6b96449b66dd48984caacc
Signature · 1 line
<http://www.alejandro-colomar.es/>
Re: [PATCH man-pages v3 4/4] madvise.2: add documentation for MADV_COLLAPSE
Hey Alex!
Quoted 29 lines · Alejandro Colomar
Hi Zach!
Quoted 11 lines · Zach OKeefe
From: Zach O'Keefe <[email protected]>
Linux 6.1 introduced
MADV_COLLAPSEin upstream commit7d8faaf15545("mm/madvise: introduceMADV_COLLAPSEsync hugepage collapse") and upstream commit34488399fa08("mm/madvise: add file and shmem support toMADV_COLLAPSE"). Update the man-pages for madvise(2) andprocess_madvise(2).There are a few issues with this patch:
alx@asus5775:~/src/linux/man-pages/man-pages$ make lint-man-groff LINT (groff) tmp/lint/man2/madvise.2.lint-man.groff.touch eqn:man2/madvise.2:473: error: invalid input character code '128' eqn:man2/madvise.2:473: error: invalid input character code '153' an.tmac:man2/madvise.2:445: style: .BR expects at least 2 arguments, got 1 an.tmac:man2/madvise.2:456: style: .BR expects at least 2 arguments, got 1 an.tmac:man2/madvise.2:463: style: .BR expects at least 2 arguments, got 1 found style problems; aborting make: *** [lib/lint-man.mk:77: tmp/lint/man2/madvise.2.lint-man.groff.touch] Error 1
Quoted 2 lines
Let's investigate them:
Thank you :)
Quoted 13 lines
alx@asus5775:~/src/linux/man-pages/man-pages$ sed -n 473p man2/madvise.2 this operation will be deemed successful.This one was a bit difficult to track, since the line count seems to be off by one:
alx@asus5775:~/src/linux/man-pages/man-pages$ tbl man2/madvise.2 | hd | grep -C1 ' 80 ' 00003d40 63 65 73 73 66 75 6c 2e 0a 4e 6f 74 65 20 74 68 |cessful..Note th| 00003d50 61 74 20 74 68 69 73 20 64 6f 65 73 6e e2 80 99 |at this doesn...| 00003d60 74 20 67 75 61 72 61 6e 74 65 65 20 61 6e 79 74 |t guarantee anyt| alx@asus5775:~/src/linux/man-pages/man-pages$ sed -n 474p man2/madvise.2 Note that this doesn’t guarantee anything about other possible mappings of
Quoted 4 lines
The issue was in line 474, and the issue is that it uses a weird single quote. Please use the foillowing ASCII character for the single quote (see ascii(7)):
047 39 27 '
Very weird and good find! Honestly, I had prototyped this in Google Docs and copy-pasta'd this over as the basis. I tried testing this again - and same thing - Google Docs uses some other character. Anyways - glad you caught this.
Quoted 3 lines
The rest of issues seems trivial:
Use .B instead of .BR because there's no "roman" (i.e., non-bold) part.
This was the first time it clicked what ".BR" meant: "bold followed by roman".
Quoted 7 lines
alx@asus5775:~/src/linux/man-pages/man-pages$ sed -n 445p man2/madvise.2 .BR MADV_COLLAPSE alx@asus5775:~/src/linux/man-pages/man-pages$ sed -n 456p man2/madvise.2 .BR MADV_COLLAPSE alx@asus5775:~/src/linux/man-pages/man-pages$ sed -n 463p man2/madvise.2 .BR VM_NOHUGEPAGE
These didn't show up with my version of groff (as in 1/2), but I've applied the fixes and sent out a v4 for this patch. Again, thank you for all your help here!
Best,
Zach
Quoted 3 lines
I'll report a bug to groff(1) about the issue with the line count.
Ya that's an odd one. Sorry for having to encounter this - must have been quite confusing. Thank you!
Quoted 163 lines
Cheers,
Alex
Quoted 146 lines
--- man2/madvise.2 | 90 +++++++++++++++++++++++++++++++++++++++++- man2/process_madvise.2 | 10 +++++ 2 files changed, 98 insertions(+), 2 deletions(-)man2/madvise.2 +88
-2@@ -385,9 +385,10 @@ set (see .BR prctl (2) ). .IP The -.B MADV_HUGEPAGE +.BR MADV_HUGEPAGE , +.BR MADV_NOHUGEPAGE , and -.B MADV_NOHUGEPAGE +.B MADV_COLLAPSE operations are available only if the kernel was configured with .B CONFIG_TRANSPARENT_HUGEPAGE and file/shmem memory is only supported if the kernel was configured with@@ -400,6 +401,81 @@ and .I length will not be backed by transparent hugepages. .TP +.BR MADV_COLLAPSE " (since Linux 6.1)" +.\" commit 7d8faaf155454f8798ec56404faca29a82689c77 +.\" commit 34488399fa08faaf664743fa54b271eb6f9e1321 +Perform a best-effort synchronous collapse of the native pages mapped by the +memory range into Transparent Huge Pages (THPs). +.B MADV_COLLAPSE +operates on the current state of memory of the calling process and makes no +persistent changes or guarantees on how pages will be mapped, +constructed, +or faulted in the future. +.IP +.B MADV_COLLAPSE +supports private anonymous pages (see +.BR mmap (2)), +shmem pages, +and file-backed pages. +See +.B MADV_HUGEPAGE +for general information on memory requirements for THP. +If the range provided spans multiple VMAs, +the semantics of the collapse over each VMA is independent from the others. +If collapse of a given huge page-aligned/sized region fails, +the operation may continue to attempt collapsing the remainder of the +specified memory. +.B MADV_COLLAPSE +will automatically clamp the provided range to be hugepage-aligned. +.IP +All non-resident pages covered by the range will first be +swapped/faulted-in, +before being copied onto a freshly allocated hugepage. +If the native pages compose the same PTE-mapped hugepage, +and are suitably aligned, +allocation of a new hugepage may be elided and collapse may happen +in-place. +Unmapped pages will have their data directly initialized to 0 in the new +hugepage. +However, +for every eligible hugepage-aligned/sized region to be collapsed, +at least one page must currently be backed by physical memory. +.IP +.BR MADV_COLLAPSE +is independent of any sysfs +(see +.BR sysfs (5)) +setting under +.IR /sys/kernel/mm/transparent_hugepage , +both in terms of determining THP eligibility, +and allocation semantics. +See Linux kernel source file +.I Documentation/admin\-guide/mm/transhuge.rst +for more information. +.BR MADV_COLLAPSE +also ignores +.B huge= +tmpfs mount when operating on tmpfs files. +Allocation for the new hugepage may enter direct reclaim and/or compaction, +regardless of VMA flags +(though +.BR VM_NOHUGEPAGE +is still respected). +.IP +When the system has multiple NUMA nodes, +the hugepage will be allocated from the node providing the most native +pages. +.IP +If all hugepage-sized/aligned regions covered by the provided range were +either successfully collapsed, +or were already PMD-mapped THPs, +this operation will be deemed successful. +Note that this doesn’t guarantee anything about other possible mappings of +the memory. +Also note that many failures might have occurred since the operation may +continue to collapse in the event collapse of a single hugepage-sized/aligned +region fails. +.TP .BR MADV_DONTDUMP " (since Linux 3.4)" .\" commit 909af768e88867016f427264ae39d27a57b6a8ed .\" commit accb61fe7bb0f5c2a4102239e4981650f9048519@@ -619,6 +695,11 @@ A kernel resource was temporarily unavailable. .B EBADF The map exists, but the area maps something that isn't a file. .TP +.B EBUSY +(for +.BR MADV_COLLAPSE ) +Could not charge hugepage to cgroup: cgroup limit exceeded. +.TP .B EFAULT .I advice is@@ -716,6 +797,11 @@ maximum resident set size. Not enough memory: paging in failed. .TP .B ENOMEM +(for +.BR MADV_COLLAPSE ) +Not enough memory: could not allocate hugepage. +.TP +.B ENOMEM Addresses in the specified range are not currently mapped, or are outside the address space of the process. .TPman2/process_madvise.2 +4
-0@@ -73,6 +73,10 @@ argument is one of the following values: See .BR madvise (2). .TP +.B MADV_COLLAPSE +See +.BR madvise (2). +.TP .B MADV_PAGEOUT See .BR madvise (2).@@ -173,6 +177,12 @@ The caller does not have permission to access the address space of the process .TP .B ESRCHman2/process_madvise.2 The caller does not have permission to access the address space of the process · line 188
Quoted 10 lines
man2/process_madvise.2 +6
-0The target process does not exist (i.e., it has terminated and been waited on). +.PP +See +.BR madvise (2) +for +.IR advice -specific +errors. .SH VERSIONS This system call first appeared in Linux 5.10. .\" commit ecb8ac8b1f146915aa6b96449b66dd48984caacc
Re: [PATCH man-pages v3 4/4] madvise.2: add documentation for MADV_COLLAPSE
In reply to Alejandro ColomarQuoted 4 lines · Alejandro Colomar
The issue was in line 474, and the issue is that it uses a weird single quote. Please use the foillowing ASCII character for the single quote (see ascii(7)):
047 39 27 '
[...]
Quoted 1 line
I'll report a bug to groff(1) about the issue with the line count.
Thanks, Alex. There appear to be some very old bugs around input line number tracking in GNU eqn, possibly going back 30+ years.
I've committed a regression test[1] and fix.[2] The fix can be expected (along with literally hundreds of others) in groff 1.23.
And now I see I managed to sneak a cosmetic indentation error into the commit message for the second (but not the ChangeLog), where I'm stuck with it forever. Oh well.
Regards,
Branden
[1] https://git.savannah.gnu.org/cgit/groff.git/commit/?id=7e23e1342077a6d7c0b02c3d666f131d95f2b510
[2] https://git.savannah.gnu.org/cgit/groff.git/commit/?id=dc98a8b09e7f3dcfe968b978eb210f468db78cc9
Re: [PATCH man-pages v3 4/4] madvise.2: add documentation for MADV_COLLAPSE
Hey Branden!
Quoted 10 lines · G. Branden Robinson
Quoted 4 lines · Alejandro Colomar
The issue was in line 474, and the issue is that it uses a weird single quote. Please use the foillowing ASCII character for the single quote (see ascii(7)):
047 39 27 '[...]
Quoted 1 line
I'll report a bug to groff(1) about the issue with the line count.
Thanks, Alex. There appear to be some very old bugs around input line number tracking in GNU eqn, possibly going back 30+ years.
Heh! It feels good finding 30-yr-old bugs :)
Quoted 3 lines
I've committed a regression test[1] and fix.[2] The fix can be expected (along with literally hundreds of others) in groff 1.23.
Yeah, I'm waiting[1] for that release to have 'make lint' be usable by contributors. But it's been already proved in this patch set that it can be useful to catch things that I miss, even if I have to run it.
[1]: <https://git.kernel.org/pub/scm/docs/man-pages/man-pages.git/tree/INSTALL#n98>
Quoted 4 lines
And now I see I managed to sneak a cosmetic indentation error into the commit message for the second (but not the ChangeLog), where I'm stuck with it forever. Oh well.
:)
Cheers,
Alex
Quoted 6 lines
Regards,
Branden[1] https://git.savannah.gnu.org/cgit/groff.git/commit/?id=7e23e1342077a6d7c0b02c3d666f131d95f2b510
[2] https://git.savannah.gnu.org/cgit/groff.git/commit/?id=dc98a8b09e7f3dcfe968b978eb210f468db78cc9
Signature · 1 line
<http://www.alejandro-colomar.es/>
Re: [PATCH man-pages v3 4/4] madvise.2: add documentation for MADV_COLLAPSE
In reply to Zach OKeefeHi Zach,
Quoted 11 lines · Zach OKeefe
From: Zach O'Keefe <[email protected]>
Linux 6.1 introduced
MADV_COLLAPSEin upstream commit7d8faaf15545("mm/madvise: introduceMADV_COLLAPSEsync hugepage collapse") and upstream commit34488399fa08("mm/madvise: add file and shmem support toMADV_COLLAPSE"). Update the man-pages for madvise(2) andprocess_madvise(2).
Please see a few comments below.
Cheers,
Alex
Quoted 23 lines
--- man2/madvise.2 | 90 +++++++++++++++++++++++++++++++++++++++++- man2/process_madvise.2 | 10 +++++ 2 files changed, 98 insertions(+), 2 deletions(-)man2/madvise.2 +3
-2@@ -385,9 +385,10 @@ set (see .BR prctl (2) ). .IP The -.B MADV_HUGEPAGE +.BR MADV_HUGEPAGE , +.BR MADV_NOHUGEPAGE , and -.B MADV_NOHUGEPAGE +.B MADV_COLLAPSE operations are available only if the kernel was configured with .B CONFIG_TRANSPARENT_HUGEPAGE and file/shmem memory is only supported if the kernel was configured with
man2/madvise.2 and · line 407
Quoted 7 lines
man2/madvise.2 +4
-0@@ -400,6 +401,81 @@ and .I length will not be backed by transparent hugepages. .TP +.BR MADV_COLLAPSE " (since Linux 6.1)" +.\" commit 7d8faaf155454f8798ec56404faca29a82689c77 +.\" commit 34488399fa08faaf664743fa54b271eb6f9e1321 +Perform a best-effort synchronous collapse of the native pages mapped by the
Please use semantic line breaks. In this case, I'd break after "pages".
man-pages(7):
Use semantic newlines
In the source of a manual page, new sentences should be started on new lines, long sentences should be split into lines at clause breaks (com‐ mas, semicolons, colons, and so on), and long clauses should be split
at phrase boundaries. This convention, sometimes known as "semantic
newlines", makes it easier to see the effect of patches, which often operate at the level of individual sentences, clauses, or phrases.
man2/madvise.2 and · line 410
Quoted 3 lines
man2/madvise.2 +3
-0+memory range into Transparent Huge Pages (THPs). +.B MADV_COLLAPSE +operates on the current state of memory of the calling process and makes no
Here I'd break after "and".
Quoted 6 lines
man2/madvise.2 +6
-0+persistent changes or guarantees on how pages will be mapped, +constructed, +or faulted in the future. +.IP +.B MADV_COLLAPSE +supports private anonymous pages (see
man2/madvise.2 and · line 426
Quoted 10 lines
man2/madvise.2 +10
-0+.BR mmap (2)), +shmem pages, +and file-backed pages. +See +.B MADV_HUGEPAGE +for general information on memory requirements for THP. +If the range provided spans multiple VMAs, +the semantics of the collapse over each VMA is independent from the others. +If collapse of a given huge page-aligned/sized region fails, +the operation may continue to attempt collapsing the remainder of the
Break after "collapsing".
man2/madvise.2 and · line 431
Quoted 5 lines
man2/madvise.2 +5
-0+specified memory. +.B MADV_COLLAPSE +will automatically clamp the provided range to be hugepage-aligned. +.IP +All non-resident pages covered by the range will first be
Break after "range".
man2/madvise.2 and · line 436
Quoted 5 lines
man2/madvise.2 +5
-0+swapped/faulted-in, +before being copied onto a freshly allocated hugepage. +If the native pages compose the same PTE-mapped hugepage, +and are suitably aligned, +allocation of a new hugepage may be elided and collapse may happen
Break before or after "and".
man2/madvise.2 and · line 438
Quoted 2 lines
man2/madvise.2 +2
-0+in-place. +Unmapped pages will have their data directly initialized to 0 in the new
Break after "0".
man2/madvise.2 and · line 444
Quoted 6 lines
man2/madvise.2 +6
-0+hugepage. +However, +for every eligible hugepage-aligned/sized region to be collapsed, +at least one page must currently be backed by physical memory. +.IP +.BR MADV_COLLAPSE
s/BR/B/
man2/madvise.2 and · line 455
Quoted 11 lines
man2/madvise.2 +11
-0+is independent of any sysfs +(see +.BR sysfs (5)) +setting under +.IR /sys/kernel/mm/transparent_hugepage , +both in terms of determining THP eligibility, +and allocation semantics. +See Linux kernel source file +.I Documentation/admin\-guide/mm/transhuge.rst +for more information. +.BR MADV_COLLAPSE
s/BR/B/
man2/madvise.2 and · line 462
Quoted 7 lines
man2/madvise.2 +7
-0+also ignores +.B huge= +tmpfs mount when operating on tmpfs files. +Allocation for the new hugepage may enter direct reclaim and/or compaction, +regardless of VMA flags +(though +.BR VM_NOHUGEPAGE
s/BR/B/
man2/madvise.2 and · line 466
Quoted 4 lines
man2/madvise.2 +4
-0+is still respected). +.IP +When the system has multiple NUMA nodes, +the hugepage will be allocated from the node providing the most native
Break after "from".
man2/madvise.2 and · line 469
Quoted 3 lines
man2/madvise.2 +3
-0+pages. +.IP +If all hugepage-sized/aligned regions covered by the provided range were
Prefer English rather than "/".
man2/madvise.2 and · line 473
Quoted 4 lines
man2/madvise.2 +4
-0+either successfully collapsed, +or were already PMD-mapped THPs, +this operation will be deemed successful. +Note that this doesn’t guarantee anything about other possible mappings of
Break after "about".
man2/madvise.2 and · line 476
Quoted 3 lines
man2/madvise.2 +3
-0+the memory. +Also note that many failures might have occurred since the operation may +continue to collapse in the event collapse of a single hugepage-sized/aligned
Add some omitted "that" or something that will help readability to non-native-English readers.
And break at a better place.
Quoted 57 lines
man2/madvise.2 +12
-0+region fails. +.TP .BR MADV_DONTDUMP " (since Linux 3.4)" .\" commit 909af768e88867016f427264ae39d27a57b6a8ed .\" commit accb61fe7bb0f5c2a4102239e4981650f9048519@@ -619,6 +695,11 @@ A kernel resource was temporarily unavailable. .B EBADF The map exists, but the area maps something that isn't a file. .TP +.B EBUSY +(for +.BR MADV_COLLAPSE ) +Could not charge hugepage to cgroup: cgroup limit exceeded. +.TP .B EFAULT .I advice is@@ -716,6 +797,11 @@ maximum resident set size. Not enough memory: paging in failed. .TP .B ENOMEM +(for +.BR MADV_COLLAPSE ) +Not enough memory: could not allocate hugepage. +.TP +.B ENOMEM Addresses in the specified range are not currently mapped, or are outside the address space of the process. .TPman2/process_madvise.2 +10
-0@@ -73,6 +73,10 @@ argument is one of the following values: See .BR madvise (2). .TP +.B MADV_COLLAPSE +See +.BR madvise (2). +.TP .B MADV_PAGEOUT See .BR madvise (2).@@ -173,6 +177,12 @@ The caller does not have permission to access the address space of the process .TP .B ESRCH The target process does not exist (i.e., it has terminated and been waited on). +.PP +See +.BR madvise (2) +for +.IR advice -specific +errors. .SH VERSIONS This system call first appeared in Linux 5.10. .\" commit ecb8ac8b1f146915aa6b96449b66dd48984caacc
Signature · 1 line
<http://www.alejandro-colomar.es/>
Re: [PATCH man-pages v3 4/4] madvise.2: add documentation for MADV_COLLAPSE
Quoted 2 lines · Alejandro Colomar
Hi Zach,
Hey Alex,
Quoted 9 lines · Zach OKeefe
Quoted 8 lines
From: Zach O'Keefe <[email protected]>
Linux 6.1 introduced
MADV_COLLAPSEin upstream commit7d8faaf15545("mm/madvise: introduceMADV_COLLAPSEsync hugepage collapse") and upstream commit34488399fa08("mm/madvise: add file and shmem support toMADV_COLLAPSE"). Update the man-pages for madvise(2) andprocess_madvise(2).
Quoted 6 lines
Quoted 3 lines
Please see a few comments below.
Thanks for the mail. So, this patch was taken as commit b106cd5bf ("madvise.2: add documentation for MADV_COLLAPSE"). Some of your comments below were
applied (I think, by you) as fixes pre-commit. However, there are some new comments (or ones
that address the same lines, but in different ways). Is this mail to log ~ what changes were done,
or is there anything actionable here on my side?
Best,
Zach
Thanks for this.
Quoted 214 lines
Cheers,
Alex
Quoted 23 lines
--- man2/madvise.2 | 90 +++++++++++++++++++++++++++++++++++++++++- man2/process_madvise.2 | 10 +++++ 2 files changed, 98 insertions(+), 2 deletions(-)man2/madvise.2 +3
-2@@ -385,9 +385,10 @@ set (see .BR prctl (2) ). .IP The -.B MADV_HUGEPAGE +.BR MADV_HUGEPAGE , +.BR MADV_NOHUGEPAGE , and -.B MADV_NOHUGEPAGE +.B MADV_COLLAPSE operations are available only if the kernel was configured with .B CONFIG_TRANSPARENT_HUGEPAGE and file/shmem memory is only supported if the kernel was configured withman2/madvise.2 and · line 407
Quoted 7 lines
man2/madvise.2 +4
-0@@ -400,6 +401,81 @@ and .I length will not be backed by transparent hugepages. .TP +.BR MADV_COLLAPSE " (since Linux 6.1)" +.\" commit 7d8faaf155454f8798ec56404faca29a82689c77 +.\" commit 34488399fa08faaf664743fa54b271eb6f9e1321 +Perform a best-effort synchronous collapse of the native pages mapped by thePlease use semantic line breaks. In this case, I'd break after "pages".
man-pages(7):
Use semantic newlines
In the source of a manual page, new sentences should be started on new lines, long sentences should be split into lines at clause breaks (com‐ mas, semicolons, colons, and so on), and long clauses should be split
at phrase boundaries. This convention, sometimes known as "semantic
newlines", makes it easier to see the effect of patches, which often operate at the level of individual sentences, clauses, or phrases.man2/madvise.2 and · line 410
Quoted 3 lines
man2/madvise.2 +3
-0+memory range into Transparent Huge Pages (THPs). +.B MADV_COLLAPSE +operates on the current state of memory of the calling process and makes noHere I'd break after "and".
Quoted 6 lines
man2/madvise.2 +6
-0+persistent changes or guarantees on how pages will be mapped, +constructed, +or faulted in the future. +.IP +.B MADV_COLLAPSE +supports private anonymous pages (seeman2/madvise.2 and · line 426
Quoted 10 lines
man2/madvise.2 +10
-0+.BR mmap (2)), +shmem pages, +and file-backed pages. +See +.B MADV_HUGEPAGE +for general information on memory requirements for THP. +If the range provided spans multiple VMAs, +the semantics of the collapse over each VMA is independent from the others. +If collapse of a given huge page-aligned/sized region fails, +the operation may continue to attempt collapsing the remainder of theBreak after "collapsing".
man2/madvise.2 and · line 431
Quoted 5 lines
man2/madvise.2 +5
-0+specified memory. +.B MADV_COLLAPSE +will automatically clamp the provided range to be hugepage-aligned. +.IP +All non-resident pages covered by the range will first beBreak after "range".
man2/madvise.2 and · line 436
Quoted 5 lines
man2/madvise.2 +5
-0+swapped/faulted-in, +before being copied onto a freshly allocated hugepage. +If the native pages compose the same PTE-mapped hugepage, +and are suitably aligned, +allocation of a new hugepage may be elided and collapse may happenBreak before or after "and".
man2/madvise.2 and · line 438
Quoted 2 lines
man2/madvise.2 +2
-0+in-place. +Unmapped pages will have their data directly initialized to 0 in the newBreak after "0".
man2/madvise.2 and · line 444
Quoted 6 lines
man2/madvise.2 +6
-0+hugepage. +However, +for every eligible hugepage-aligned/sized region to be collapsed, +at least one page must currently be backed by physical memory. +.IP +.BR MADV_COLLAPSEs/BR/B/
man2/madvise.2 and · line 455
Quoted 11 lines
man2/madvise.2 +11
-0+is independent of any sysfs +(see +.BR sysfs (5)) +setting under +.IR /sys/kernel/mm/transparent_hugepage , +both in terms of determining THP eligibility, +and allocation semantics. +See Linux kernel source file +.I Documentation/admin\-guide/mm/transhuge.rst +for more information. +.BR MADV_COLLAPSEs/BR/B/
man2/madvise.2 and · line 462
Quoted 7 lines
man2/madvise.2 +7
-0+also ignores +.B huge= +tmpfs mount when operating on tmpfs files. +Allocation for the new hugepage may enter direct reclaim and/or compaction, +regardless of VMA flags +(though +.BR VM_NOHUGEPAGEs/BR/B/
man2/madvise.2 and · line 466
Quoted 4 lines
man2/madvise.2 +4
-0+is still respected). +.IP +When the system has multiple NUMA nodes, +the hugepage will be allocated from the node providing the most nativeBreak after "from".
man2/madvise.2 and · line 469
Quoted 3 lines
man2/madvise.2 +3
-0+pages. +.IP +If all hugepage-sized/aligned regions covered by the provided range werePrefer English rather than "/".
man2/madvise.2 and · line 473
Quoted 4 lines
man2/madvise.2 +4
-0+either successfully collapsed, +or were already PMD-mapped THPs, +this operation will be deemed successful. +Note that this doesn’t guarantee anything about other possible mappings ofBreak after "about".
man2/madvise.2 and · line 476
Quoted 3 lines
man2/madvise.2 +3
-0+the memory. +Also note that many failures might have occurred since the operation may +continue to collapse in the event collapse of a single hugepage-sized/alignedAdd some omitted "that" or something that will help readability to non-native-English readers.
And break at a better place.
Quoted 47 lines
man2/madvise.2 +12
-0+region fails. +.TP .BR MADV_DONTDUMP " (since Linux 3.4)" .\" commit 909af768e88867016f427264ae39d27a57b6a8ed .\" commit accb61fe7bb0f5c2a4102239e4981650f9048519@@ -619,6 +695,11 @@ A kernel resource was temporarily unavailable. .B EBADF The map exists, but the area maps something that isn't a file. .TP +.B EBUSY +(for +.BR MADV_COLLAPSE ) +Could not charge hugepage to cgroup: cgroup limit exceeded. +.TP .B EFAULT .I advice is@@ -716,6 +797,11 @@ maximum resident set size. Not enough memory: paging in failed. .TP .B ENOMEM +(for +.BR MADV_COLLAPSE ) +Not enough memory: could not allocate hugepage. +.TP +.B ENOMEM Addresses in the specified range are not currently mapped, or are outside the address space of the process. .TPman2/process_madvise.2 +4
-0@@ -73,6 +73,10 @@ argument is one of the following values: See .BR madvise (2). .TP +.B MADV_COLLAPSE +See +.BR madvise (2). +.TP .B MADV_PAGEOUT See .BR madvise (2).@@ -173,6 +177,12 @@ The caller does not have permission to access the address space of the process .TP .B ESRCHman2/process_madvise.2 The caller does not have permission to access the address space of the process · line 188
Quoted 10 lines
man2/process_madvise.2 +6
-0The target process does not exist (i.e., it has terminated and been waited on). +.PP +See +.BR madvise (2) +for +.IR advice -specific +errors. .SH VERSIONS This system call first appeared in Linux 5.10. .\" commit ecb8ac8b1f146915aa6b96449b66dd48984caacc
Re: [PATCH man-pages v3 4/4] madvise.2: add documentation for MADV_COLLAPSE
Hey Zach,
Quoted 23 lines · Zach O'Keefe
Quoted 2 lines · Alejandro Colomar
Hi Zach,
Hey Alex,
Quoted 9 lines · Zach OKeefe
Quoted 8 lines
From: Zach O'Keefe <[email protected]>
Linux 6.1 introduced
MADV_COLLAPSEin upstream commit7d8faaf15545("mm/madvise: introduceMADV_COLLAPSEsync hugepage collapse") and upstream commit34488399fa08("mm/madvise: add file and shmem support toMADV_COLLAPSE"). Update the man-pages for madvise(2) andprocess_madvise(2).Quoted 6 lines
Quoted 3 lines
Please see a few comments below.
Quoted 8 lines
Thanks for the mail. So, this patch was taken as commit
b106cd5bf("madvise.2: add documentation forMADV_COLLAPSE"). Some of your comments below were
applied (I think, by you) as fixes pre-commit. However, there are some new comments (or ones
that address the same lines, but in different ways). Is this mail to log ~ what changes were done,
or is there anything actionable here on my side?
Ah no, it's just that I had it marked as unread for some reason, so I thought I had forgotten to respond (and I forgot that I had applied it). :-)
So, no action required.
Regarding different suggestions, heh, it demonstrates that it's not exactly deterministic :P
Cheers,
Alex
P.S.: Do you know if I have anything missing from you or any of your collegues?
Quoted 219 lines
Best,
ZachThanks for this.
Quoted 214 lines
Cheers,
Alex
Quoted 23 lines
--- man2/madvise.2 | 90 +++++++++++++++++++++++++++++++++++++++++- man2/process_madvise.2 | 10 +++++ 2 files changed, 98 insertions(+), 2 deletions(-)man2/madvise.2 +3
-2@@ -385,9 +385,10 @@ set (see .BR prctl (2) ). .IP The -.B MADV_HUGEPAGE +.BR MADV_HUGEPAGE , +.BR MADV_NOHUGEPAGE , and -.B MADV_NOHUGEPAGE +.B MADV_COLLAPSE operations are available only if the kernel was configured with .B CONFIG_TRANSPARENT_HUGEPAGE and file/shmem memory is only supported if the kernel was configured withman2/madvise.2 and · line 407
Quoted 7 lines
man2/madvise.2 +4
-0@@ -400,6 +401,81 @@ and .I length will not be backed by transparent hugepages. .TP +.BR MADV_COLLAPSE " (since Linux 6.1)" +.\" commit 7d8faaf155454f8798ec56404faca29a82689c77 +.\" commit 34488399fa08faaf664743fa54b271eb6f9e1321 +Perform a best-effort synchronous collapse of the native pages mapped by thePlease use semantic line breaks. In this case, I'd break after "pages".
man-pages(7):
Use semantic newlines
In the source of a manual page, new sentences should be started on new lines, long sentences should be split into lines at clause breaks (com‐ mas, semicolons, colons, and so on), and long clauses should be split
at phrase boundaries. This convention, sometimes known as "semantic
newlines", makes it easier to see the effect of patches, which often operate at the level of individual sentences, clauses, or phrases.man2/madvise.2 and · line 410
Quoted 3 lines
man2/madvise.2 +3
-0+memory range into Transparent Huge Pages (THPs). +.B MADV_COLLAPSE +operates on the current state of memory of the calling process and makes noHere I'd break after "and".
Quoted 6 lines
man2/madvise.2 +6
-0+persistent changes or guarantees on how pages will be mapped, +constructed, +or faulted in the future. +.IP +.B MADV_COLLAPSE +supports private anonymous pages (seeman2/madvise.2 and · line 426
Quoted 10 lines
man2/madvise.2 +10
-0+.BR mmap (2)), +shmem pages, +and file-backed pages. +See +.B MADV_HUGEPAGE +for general information on memory requirements for THP. +If the range provided spans multiple VMAs, +the semantics of the collapse over each VMA is independent from the others. +If collapse of a given huge page-aligned/sized region fails, +the operation may continue to attempt collapsing the remainder of theBreak after "collapsing".
man2/madvise.2 and · line 431
Quoted 5 lines
man2/madvise.2 +5
-0+specified memory. +.B MADV_COLLAPSE +will automatically clamp the provided range to be hugepage-aligned. +.IP +All non-resident pages covered by the range will first beBreak after "range".
man2/madvise.2 and · line 436
Quoted 5 lines
man2/madvise.2 +5
-0+swapped/faulted-in, +before being copied onto a freshly allocated hugepage. +If the native pages compose the same PTE-mapped hugepage, +and are suitably aligned, +allocation of a new hugepage may be elided and collapse may happenBreak before or after "and".
man2/madvise.2 and · line 438
Quoted 2 lines
man2/madvise.2 +2
-0+in-place. +Unmapped pages will have their data directly initialized to 0 in the newBreak after "0".
man2/madvise.2 and · line 444
Quoted 6 lines
man2/madvise.2 +6
-0+hugepage. +However, +for every eligible hugepage-aligned/sized region to be collapsed, +at least one page must currently be backed by physical memory. +.IP +.BR MADV_COLLAPSEs/BR/B/
man2/madvise.2 and · line 455
Quoted 11 lines
man2/madvise.2 +11
-0+is independent of any sysfs +(see +.BR sysfs (5)) +setting under +.IR /sys/kernel/mm/transparent_hugepage , +both in terms of determining THP eligibility, +and allocation semantics. +See Linux kernel source file +.I Documentation/admin\-guide/mm/transhuge.rst +for more information. +.BR MADV_COLLAPSEs/BR/B/
man2/madvise.2 and · line 462
Quoted 7 lines
man2/madvise.2 +7
-0+also ignores +.B huge= +tmpfs mount when operating on tmpfs files. +Allocation for the new hugepage may enter direct reclaim and/or compaction, +regardless of VMA flags +(though +.BR VM_NOHUGEPAGEs/BR/B/
man2/madvise.2 and · line 466
Quoted 4 lines
man2/madvise.2 +4
-0+is still respected). +.IP +When the system has multiple NUMA nodes, +the hugepage will be allocated from the node providing the most nativeBreak after "from".
man2/madvise.2 and · line 469
Quoted 3 lines
man2/madvise.2 +3
-0+pages. +.IP +If all hugepage-sized/aligned regions covered by the provided range werePrefer English rather than "/".
man2/madvise.2 and · line 473
Quoted 4 lines
man2/madvise.2 +4
-0+either successfully collapsed, +or were already PMD-mapped THPs, +this operation will be deemed successful. +Note that this doesn’t guarantee anything about other possible mappings ofBreak after "about".
man2/madvise.2 and · line 476
Quoted 3 lines
man2/madvise.2 +3
-0+the memory. +Also note that many failures might have occurred since the operation may +continue to collapse in the event collapse of a single hugepage-sized/alignedAdd some omitted "that" or something that will help readability to non-native-English readers.
And break at a better place.
Quoted 47 lines
man2/madvise.2 +12
-0+region fails. +.TP .BR MADV_DONTDUMP " (since Linux 3.4)" .\" commit 909af768e88867016f427264ae39d27a57b6a8ed .\" commit accb61fe7bb0f5c2a4102239e4981650f9048519@@ -619,6 +695,11 @@ A kernel resource was temporarily unavailable. .B EBADF The map exists, but the area maps something that isn't a file. .TP +.B EBUSY +(for +.BR MADV_COLLAPSE ) +Could not charge hugepage to cgroup: cgroup limit exceeded. +.TP .B EFAULT .I advice is@@ -716,6 +797,11 @@ maximum resident set size. Not enough memory: paging in failed. .TP .B ENOMEM +(for +.BR MADV_COLLAPSE ) +Not enough memory: could not allocate hugepage. +.TP +.B ENOMEM Addresses in the specified range are not currently mapped, or are outside the address space of the process. .TPman2/process_madvise.2 +4
-0@@ -73,6 +73,10 @@ argument is one of the following values: See .BR madvise (2). .TP +.B MADV_COLLAPSE +See +.BR madvise (2). +.TP .B MADV_PAGEOUT See .BR madvise (2).@@ -173,6 +177,12 @@ The caller does not have permission to access the address space of the process .TP .B ESRCHman2/process_madvise.2 The caller does not have permission to access the address space of the process · line 188
Quoted 10 lines
man2/process_madvise.2 +6
-0The target process does not exist (i.e., it has terminated and been waited on). +.PP +See +.BR madvise (2) +for +.IR advice -specific +errors. .SH VERSIONS This system call first appeared in Linux 5.10. .\" commit ecb8ac8b1f146915aa6b96449b66dd48984caacc
Signature · 1 line
<http://www.alejandro-colomar.es/>
Re: [PATCH man-pages v3 4/4] madvise.2: add documentation for MADV_COLLAPSE
Quoted 36 lines · Alejandro Colomar
Hey Zach,
Quoted 23 lines · Zach O'Keefe
Quoted 2 lines · Alejandro Colomar
Hi Zach,
Hey Alex,
Quoted 9 lines · Zach OKeefe
Quoted 8 lines
From: Zach O'Keefe <[email protected]>
Linux 6.1 introduced
MADV_COLLAPSEin upstream commit7d8faaf15545("mm/madvise: introduceMADV_COLLAPSEsync hugepage collapse") and upstream commit34488399fa08("mm/madvise: add file and shmem support toMADV_COLLAPSE"). Update the man-pages for madvise(2) andprocess_madvise(2).Quoted 6 lines
Quoted 3 lines
Please see a few comments below.
Quoted 8 lines
Thanks for the mail. So, this patch was taken as commit
b106cd5bf("madvise.2: add documentation forMADV_COLLAPSE"). Some of your comments below were
applied (I think, by you) as fixes pre-commit. However, there are some new comments (or ones
that address the same lines, but in different ways). Is this mail to log ~ what changes were done,
or is there anything actionable here on my side?
Quoted 8 lines
Ah no, it's just that I had it marked as unread for some reason, so I thought I had forgotten to respond (and I forgot that I had applied it). :-)
So, no action required.
Regarding different suggestions, heh, it demonstrates that it's not exactly deterministic :P
Heh -- no worries :) Thanks for following up!
Quoted 5 lines
Cheers,
Alex
P.S.: Do you know if I have anything missing from you or any of your collegues?
At least on my part, I think you've taken all my patches (with help & edits -- thank you!). I can't speak for anyone else at Google, however (though, just a very hasty cross reference between git log and lore.kernel.org/linux-man seems to indicate patches sent from *@google.com since man-pages-6.00 have previously made it into man-pages-6.01, and nothing afterwards).
Have a great rest of your weekend,
Best,
Zach
Quoted 223 lines
Quoted 220 lines
Best,
ZachThanks for this.
Quoted 214 lines
Cheers,
Alex
Quoted 23 lines
--- man2/madvise.2 | 90 +++++++++++++++++++++++++++++++++++++++++- man2/process_madvise.2 | 10 +++++ 2 files changed, 98 insertions(+), 2 deletions(-)man2/madvise.2 +3
-2@@ -385,9 +385,10 @@ set (see .BR prctl (2) ). .IP The -.B MADV_HUGEPAGE +.BR MADV_HUGEPAGE , +.BR MADV_NOHUGEPAGE , and -.B MADV_NOHUGEPAGE +.B MADV_COLLAPSE operations are available only if the kernel was configured with .B CONFIG_TRANSPARENT_HUGEPAGE and file/shmem memory is only supported if the kernel was configured withman2/madvise.2 and · line 407
Quoted 7 lines
man2/madvise.2 +4
-0@@ -400,6 +401,81 @@ and .I length will not be backed by transparent hugepages. .TP +.BR MADV_COLLAPSE " (since Linux 6.1)" +.\" commit 7d8faaf155454f8798ec56404faca29a82689c77 +.\" commit 34488399fa08faaf664743fa54b271eb6f9e1321 +Perform a best-effort synchronous collapse of the native pages mapped by thePlease use semantic line breaks. In this case, I'd break after "pages".
man-pages(7):
Use semantic newlines
In the source of a manual page, new sentences should be started on new lines, long sentences should be split into lines at clause breaks (com‐ mas, semicolons, colons, and so on), and long clauses should be split
at phrase boundaries. This convention, sometimes known as "semantic
newlines", makes it easier to see the effect of patches, which often operate at the level of individual sentences, clauses, or phrases.man2/madvise.2 and · line 410
Quoted 3 lines
man2/madvise.2 +3
-0+memory range into Transparent Huge Pages (THPs). +.B MADV_COLLAPSE +operates on the current state of memory of the calling process and makes noHere I'd break after "and".
Quoted 6 lines
man2/madvise.2 +6
-0+persistent changes or guarantees on how pages will be mapped, +constructed, +or faulted in the future. +.IP +.B MADV_COLLAPSE +supports private anonymous pages (seeman2/madvise.2 and · line 426
Quoted 10 lines
man2/madvise.2 +10
-0+.BR mmap (2)), +shmem pages, +and file-backed pages. +See +.B MADV_HUGEPAGE +for general information on memory requirements for THP. +If the range provided spans multiple VMAs, +the semantics of the collapse over each VMA is independent from the others. +If collapse of a given huge page-aligned/sized region fails, +the operation may continue to attempt collapsing the remainder of theBreak after "collapsing".
man2/madvise.2 and · line 431
Quoted 5 lines
man2/madvise.2 +5
-0+specified memory. +.B MADV_COLLAPSE +will automatically clamp the provided range to be hugepage-aligned. +.IP +All non-resident pages covered by the range will first beBreak after "range".
man2/madvise.2 and · line 436
Quoted 5 lines
man2/madvise.2 +5
-0+swapped/faulted-in, +before being copied onto a freshly allocated hugepage. +If the native pages compose the same PTE-mapped hugepage, +and are suitably aligned, +allocation of a new hugepage may be elided and collapse may happenBreak before or after "and".
man2/madvise.2 and · line 438
Quoted 2 lines
man2/madvise.2 +2
-0+in-place. +Unmapped pages will have their data directly initialized to 0 in the newBreak after "0".
man2/madvise.2 and · line 444
Quoted 6 lines
man2/madvise.2 +6
-0+hugepage. +However, +for every eligible hugepage-aligned/sized region to be collapsed, +at least one page must currently be backed by physical memory. +.IP +.BR MADV_COLLAPSEs/BR/B/
man2/madvise.2 and · line 455
Quoted 11 lines
man2/madvise.2 +11
-0+is independent of any sysfs +(see +.BR sysfs (5)) +setting under +.IR /sys/kernel/mm/transparent_hugepage , +both in terms of determining THP eligibility, +and allocation semantics. +See Linux kernel source file +.I Documentation/admin\-guide/mm/transhuge.rst +for more information. +.BR MADV_COLLAPSEs/BR/B/
man2/madvise.2 and · line 462
Quoted 7 lines
man2/madvise.2 +7
-0+also ignores +.B huge= +tmpfs mount when operating on tmpfs files. +Allocation for the new hugepage may enter direct reclaim and/or compaction, +regardless of VMA flags +(though +.BR VM_NOHUGEPAGEs/BR/B/
man2/madvise.2 and · line 466
Quoted 4 lines
man2/madvise.2 +4
-0+is still respected). +.IP +When the system has multiple NUMA nodes, +the hugepage will be allocated from the node providing the most nativeBreak after "from".
man2/madvise.2 and · line 469
Quoted 3 lines
man2/madvise.2 +3
-0+pages. +.IP +If all hugepage-sized/aligned regions covered by the provided range werePrefer English rather than "/".
man2/madvise.2 and · line 473
Quoted 4 lines
man2/madvise.2 +4
-0+either successfully collapsed, +or were already PMD-mapped THPs, +this operation will be deemed successful. +Note that this doesn’t guarantee anything about other possible mappings ofBreak after "about".
man2/madvise.2 and · line 476
Quoted 3 lines
man2/madvise.2 +3
-0+the memory. +Also note that many failures might have occurred since the operation may +continue to collapse in the event collapse of a single hugepage-sized/alignedAdd some omitted "that" or something that will help readability to non-native-English readers.
And break at a better place.
Quoted 47 lines
man2/madvise.2 +12
-0+region fails. +.TP .BR MADV_DONTDUMP " (since Linux 3.4)" .\" commit 909af768e88867016f427264ae39d27a57b6a8ed .\" commit accb61fe7bb0f5c2a4102239e4981650f9048519@@ -619,6 +695,11 @@ A kernel resource was temporarily unavailable. .B EBADF The map exists, but the area maps something that isn't a file. .TP +.B EBUSY +(for +.BR MADV_COLLAPSE ) +Could not charge hugepage to cgroup: cgroup limit exceeded. +.TP .B EFAULT .I advice is@@ -716,6 +797,11 @@ maximum resident set size. Not enough memory: paging in failed. .TP .B ENOMEM +(for +.BR MADV_COLLAPSE ) +Not enough memory: could not allocate hugepage. +.TP +.B ENOMEM Addresses in the specified range are not currently mapped, or are outside the address space of the process. .TPman2/process_madvise.2 +4
-0@@ -73,6 +73,10 @@ argument is one of the following values: See .BR madvise (2). .TP +.B MADV_COLLAPSE +See +.BR madvise (2). +.TP .B MADV_PAGEOUT See .BR madvise (2).@@ -173,6 +177,12 @@ The caller does not have permission to access the address space of the process .TP .B ESRCHman2/process_madvise.2 The caller does not have permission to access the address space of the process · line 188
Quoted 10 lines
man2/process_madvise.2 +6
-0The target process does not exist (i.e., it has terminated and been waited on). +.PP +See +.BR madvise (2) +for +.IR advice -specific +errors. .SH VERSIONS This system call first appeared in Linux 5.10. .\" commit ecb8ac8b1f146915aa6b96449b66dd48984caacc
Re: [PATCH man-pages v3 4/4] madvise.2: add documentation for MADV_COLLAPSE
Hi Zach!
Quoted 1 line · Zach O'Keefe
Heh -- no worries :) Thanks for following up!
:)
Quoted 5 lines
Quoted 5 lines
Cheers,
Alex
Quoted 8 lines
Quoted 1 line
P.S.: Do you know if I have anything missing from you or any of your collegues?At least on my part, I think you've taken all my patches (with help & edits -- thank you!). I can't speak for anyone else at Google, however (though, just a very hasty cross reference between git log and lore.kernel.org/linux-man seems to indicate patches sent from *@google.com since man-pages-6.00 have previously made it into man-pages-6.01, and nothing afterwards).
Makes sense. 6.01 is very recent, and I don't remember any patches since then.
Quoted 2 lines
Have a great rest of your weekend,
Have a nice weekend!
Cheers,
Alex
Quoted 4 lines
Best,
Zach
Signature · 1 line
<http://www.alejandro-colomar.es/>