mm: allow a controlled amount of unfairness in the page lock
Subsystems (7)
PATCH 1/1 mm: allow a controlled amount of unfairness in the page lock
From: Linus Torvalds <[email protected]>
[ upstream commit 5ef64cc8987a9211d3f3667331ba3411a94ddc79 ]
Commit 2a9127fcf229 ("mm: rewrite wait_on_page_bit_common() logic") made the page locking entirely fair, in that if a waiter came in while the lock was held, the lock would be transferred to the lockers strictly in order.
That was intended to finally get rid of the long-reported watchdog failures that involved the page lock under extreme load, where a process could end up waiting essentially forever, as other page lockers stole the lock from under it.
It also improved some benchmarks, but it ended up causing huge performance regressions on others, simply because fair lock behavior doesn't end up giving out the lock as aggressively, causing better worst-case latency, but potentially much worse average latencies and throughput.
Instead of reverting that change entirely, this introduces a controlled amount of unfairness, with a sysctl knob to tune it if somebody needs to. But the default value should hopefully be good for any normal load, allowing a few rounds of lock stealing, but enforcing the strict ordering before the lock has been stolen too many times.
There is also a hint from Matthieu Baerts that the fair page coloring may end up exposing an ABBA deadlock that is hidden by the usual optimistic lock stealing, and while the unfairness doesn't fix the fundamental issue (and I'm still looking at that), it avoids it in practice.
The amount of unfairness can be modified by writing a new value to the 'sysctl_page_lock_unfairness' variable (default value of 5, exposed through /proc/sys/vm/page_lock_unfairness), but that is hopefully something we'd use mainly for debugging rather than being necessary for any deep system tuning.
This whole issue has exposed just how critical the page lock can be, and how contended it gets under certain locks. And the main contention doesn't really seem to be anything related to IO (which was the origin of this lock), but for things like just verifying that the page file mapping is stable while faulting in the page into a page table.
[ mheyne: fixed contextual conflict in mm/filemap.c due to missing
commit c7510ab2cf5c ("mm: abstract out wake_page_match() from wake_page_function()"). Added WQ_FLAG_CUSTOM due to missing commit 7f26482a872c ("locking/percpu-rwsem: Remove the embedded rwsem") ]
--- include/linux/mm.h | 2 + include/linux/wait.h | 2 + kernel/sysctl.c | 8 +++ mm/filemap.c | 160 ++++++++++++++++++++++++++++++++++--------- 4 files changed, 141 insertions(+), 31 deletions(-)
include/linux/mm.h +2 -0
@@ -37,6 +37,8 @@ struct user_struct;struct writeback_control ;struct bdi_writeback ; +extern int sysctl_page_lock_unfairness; +void init_mm_internals (void );#ifndef CONFIG_NEED_MULTIPLE_NODES /* Don't use mapnrs, do it properly */
include/linux/wait.h +2 -0
@@ -20,6 +20,8 @@ int default_wake_function(struct wait_queue_entry *wq_entry, unsigned mode, int#define WQ_FLAG_EXCLUSIVE 0x01#define WQ_FLAG_WOKEN 0x02#define WQ_FLAG_BOOKMARK 0x04 +#define WQ_FLAG_CUSTOM 0x08 +#define WQ_FLAG_DONE 0x10/* * A single wait-queue entry structure:
kernel/sysctl.c +8 -0
@@ -1563,6 +1563,14 @@ static struct ctl_table vm_table[] = { .proc_handler = percpu_pagelist_fraction_sysctl_handler, .extra1 = SYSCTL_ZERO, }, + { + .procname = "page_lock_unfairness", + .data = &sysctl_page_lock_unfairness, + .maxlen = sizeof(sysctl_page_lock_unfairness), + .mode = 0644, + .proc_handler = proc_dointvec_minmax, + .extra1 = SYSCTL_ZERO, + }, #ifdef CONFIG_MMU { .procname = "max_map_count",
mm/filemap.c +129 -31
244 lines not on this page · raw message
Signature · 12 lines
2.40.1 Amazon Development Center Germany GmbH Krausenstr. 38 10117 Berlin Geschaeftsfuehrung: Christian Schlaeger, Jonathan Weiss Eingetragen am Amtsgericht Charlottenburg unter HRB 149173 B Sitz: Berlin Ust-ID: DE 289 237 879
Greg KH
maintainerQuoted 60 lines · Maximilian Heyne
From: Linus Torvalds <[email protected]>
[ upstream commit
5ef64cc8987a9211d3f3667331ba3411a94ddc79]Commit
2a9127fcf229("mm: rewritewait_on_page_bit_common()logic") made the page locking entirely fair, in that if a waiter came in while the lock was held, the lock would be transferred to the lockers strictly in order.That was intended to finally get rid of the long-reported watchdog failures that involved the page lock under extreme load, where a process could end up waiting essentially forever, as other page lockers stole the lock from under it.
It also improved some benchmarks, but it ended up causing huge performance regressions on others, simply because fair lock behavior doesn't end up giving out the lock as aggressively, causing better worst-case latency, but potentially much worse average latencies and throughput.
Instead of reverting that change entirely, this introduces a controlled amount of unfairness, with a sysctl knob to tune it if somebody needs to. But the default value should hopefully be good for any normal load, allowing a few rounds of lock stealing, but enforcing the strict ordering before the lock has been stolen too many times.
There is also a hint from Matthieu Baerts that the fair page coloring may end up exposing an ABBA deadlock that is hidden by the usual optimistic lock stealing, and while the unfairness doesn't fix the fundamental issue (and I'm still looking at that), it avoids it in practice.
The amount of unfairness can be modified by writing a new value to the '
sysctl_page_lock_unfairness' variable (default value of 5, exposed through/proc/sys/vm/page_lock_unfairness), but that is hopefully something we'd use mainly for debugging rather than being necessary for any deep system tuning.This whole issue has exposed just how critical the page lock can be, and how contended it gets under certain locks. And the main contention doesn't really seem to be anything related to IO (which was the origin of this lock), but for things like just verifying that the page file mapping is stable while faulting in the page into a page table.
Reported-and-tested-by Michael Larabel <[email protected]>Tested-by Matthieu Baerts <[email protected]>Cc Dave Chinner <[email protected]>Cc Matthew Wilcox <[email protected]>Cc Chris Mason <[email protected]>Cc Jan Kara <[email protected]>Cc Amir Goldstein <[email protected]>Signed-off-by Linus Torvalds <[email protected]>CC <[email protected]> # 5.4[ mheyne: fixed contextual conflict in
mm/filemap.cdue to missing
commitc7510ab2cf5c("mm: abstract outwake_page_match()fromwake_page_function()"). AddedWQ_FLAG_CUSTOMdue to missing commit7f26482a872c("locking/percpu-rwsem: Remove the embedded rwsem") ]
Quoted 7 lines
Signed-off-by Maximilian Heyne <[email protected]>--- include/linux/mm.h | 2 + include/linux/wait.h | 2 + kernel/sysctl.c | 8 +++ mm/filemap.c | 160 ++++++++++++++++++++++++++++++++++--------- 4 files changed, 141 insertions(+), 31 deletions(-)
This was also backported here:
https://lore.kernel.org/r/[email protected]
before yours.
I took that one, can you verify that it is identical to yours and works properly as well?
thanks,
greg k-h
Maximilian Heyne
Tested-byQuoted 69 lines · Greg KH
Quoted 60 lines · Maximilian Heyne
From: Linus Torvalds <[email protected]>
[ upstream commit
5ef64cc8987a9211d3f3667331ba3411a94ddc79]Commit
2a9127fcf229("mm: rewritewait_on_page_bit_common()logic") made the page locking entirely fair, in that if a waiter came in while the lock was held, the lock would be transferred to the lockers strictly in order.That was intended to finally get rid of the long-reported watchdog failures that involved the page lock under extreme load, where a process could end up waiting essentially forever, as other page lockers stole the lock from under it.
It also improved some benchmarks, but it ended up causing huge performance regressions on others, simply because fair lock behavior doesn't end up giving out the lock as aggressively, causing better worst-case latency, but potentially much worse average latencies and throughput.
Instead of reverting that change entirely, this introduces a controlled amount of unfairness, with a sysctl knob to tune it if somebody needs to. But the default value should hopefully be good for any normal load, allowing a few rounds of lock stealing, but enforcing the strict ordering before the lock has been stolen too many times.
There is also a hint from Matthieu Baerts that the fair page coloring may end up exposing an ABBA deadlock that is hidden by the usual optimistic lock stealing, and while the unfairness doesn't fix the fundamental issue (and I'm still looking at that), it avoids it in practice.
The amount of unfairness can be modified by writing a new value to the '
sysctl_page_lock_unfairness' variable (default value of 5, exposed through/proc/sys/vm/page_lock_unfairness), but that is hopefully something we'd use mainly for debugging rather than being necessary for any deep system tuning.This whole issue has exposed just how critical the page lock can be, and how contended it gets under certain locks. And the main contention doesn't really seem to be anything related to IO (which was the origin of this lock), but for things like just verifying that the page file mapping is stable while faulting in the page into a page table.
Reported-and-tested-by Michael Larabel <[email protected]>Tested-by Matthieu Baerts <[email protected]>Cc Dave Chinner <[email protected]>Cc Matthew Wilcox <[email protected]>Cc Chris Mason <[email protected]>Cc Jan Kara <[email protected]>Cc Amir Goldstein <[email protected]>Signed-off-by Linus Torvalds <[email protected]>CC <[email protected]> # 5.4[ mheyne: fixed contextual conflict in
mm/filemap.cdue to missing
commitc7510ab2cf5c("mm: abstract outwake_page_match()fromwake_page_function()"). AddedWQ_FLAG_CUSTOMdue to missing commit7f26482a872c("locking/percpu-rwsem: Remove the embedded rwsem") ]Quoted 7 lines
Signed-off-by Maximilian Heyne <[email protected]>--- include/linux/mm.h | 2 + include/linux/wait.h | 2 + kernel/sysctl.c | 8 +++ mm/filemap.c | 160 ++++++++++++++++++++++++++++++++++--------- 4 files changed, 141 insertions(+), 31 deletions(-)
Quoted 6 lines
This was also backported here:
https://lore.kernel.org/r/[email protected]before yours.
I took that one, can you verify that it is identical to yours and works properly as well?
Yes it's identical and fixes the performance regression seen. Therefore,
for the other patch.
Thanks,
Max
Amazon Development Center Germany GmbH
Krausenstr. 38
10117 Berlin
Geschaeftsfuehrung: Christian Schlaeger, Jonathan Weiss Eingetragen am Amtsgericht Charlottenburg unter HRB 149173 B
Sitz: Berlin
Ust-ID: DE 289 237 879
Greg KH
maintainerQuoted 74 lines · Maximilian Heyne
Quoted 73 lines · Greg KH
Quoted 60 lines · Maximilian Heyne
From: Linus Torvalds <[email protected]>
[ upstream commit
5ef64cc8987a9211d3f3667331ba3411a94ddc79]Commit
2a9127fcf229("mm: rewritewait_on_page_bit_common()logic") made the page locking entirely fair, in that if a waiter came in while the lock was held, the lock would be transferred to the lockers strictly in order.That was intended to finally get rid of the long-reported watchdog failures that involved the page lock under extreme load, where a process could end up waiting essentially forever, as other page lockers stole the lock from under it.
It also improved some benchmarks, but it ended up causing huge performance regressions on others, simply because fair lock behavior doesn't end up giving out the lock as aggressively, causing better worst-case latency, but potentially much worse average latencies and throughput.
Instead of reverting that change entirely, this introduces a controlled amount of unfairness, with a sysctl knob to tune it if somebody needs to. But the default value should hopefully be good for any normal load, allowing a few rounds of lock stealing, but enforcing the strict ordering before the lock has been stolen too many times.
There is also a hint from Matthieu Baerts that the fair page coloring may end up exposing an ABBA deadlock that is hidden by the usual optimistic lock stealing, and while the unfairness doesn't fix the fundamental issue (and I'm still looking at that), it avoids it in practice.
The amount of unfairness can be modified by writing a new value to the '
sysctl_page_lock_unfairness' variable (default value of 5, exposed through/proc/sys/vm/page_lock_unfairness), but that is hopefully something we'd use mainly for debugging rather than being necessary for any deep system tuning.This whole issue has exposed just how critical the page lock can be, and how contended it gets under certain locks. And the main contention doesn't really seem to be anything related to IO (which was the origin of this lock), but for things like just verifying that the page file mapping is stable while faulting in the page into a page table.
Reported-and-tested-by Michael Larabel <[email protected]>Tested-by Matthieu Baerts <[email protected]>Cc Dave Chinner <[email protected]>Cc Matthew Wilcox <[email protected]>Cc Chris Mason <[email protected]>Cc Jan Kara <[email protected]>Cc Amir Goldstein <[email protected]>Signed-off-by Linus Torvalds <[email protected]>CC <[email protected]> # 5.4[ mheyne: fixed contextual conflict in
mm/filemap.cdue to missing
commitc7510ab2cf5c("mm: abstract outwake_page_match()fromwake_page_function()"). AddedWQ_FLAG_CUSTOMdue to missing commit7f26482a872c("locking/percpu-rwsem: Remove the embedded rwsem") ]Quoted 7 lines
Signed-off-by Maximilian Heyne <[email protected]>--- include/linux/mm.h | 2 + include/linux/wait.h | 2 + kernel/sysctl.c | 8 +++ mm/filemap.c | 160 ++++++++++++++++++++++++++++++++++--------- 4 files changed, 141 insertions(+), 31 deletions(-)This was also backported here:
https://lore.kernel.org/r/[email protected]before yours.
Quoted 8 lines
Quoted 2 lines
I took that one, can you verify that it is identical to yours and works properly as well?
Yes it's identical and fixes the performance regression seen. Therefore,
Tested-by Maximilian Heyne <[email protected]>for the other patch.
Thanks, I've added this to the patch now.
greg k-h