x86/split_lock: Add sysctl to control the misery mode
Subsystems (2)
PATCH v3 1/1 x86/split_lock: Add sysctl to control the misery mode
Commit b041b525dab9 ("x86/split_lock: Make life miserable for split lockers") changed the way the split lock detector works when in "warn" mode; basically, not only it shows the warn message, but also intentionally introduces a slowdown (through sleeping plus serialization mechanism) on such task. Based on discussions in [0], seems the warning alone wasn't enough motivation for userspace developers to fix their applications.
Happens that originally the proposal in [0] was to add a new mode which would warns + slowdown the "split locking" task, keeping the old warn mode untouched. In the end, that idea was discarded and the regular/default "warn" mode now slowdowns the applications. This is quite aggressive with regards proprietary/legacy programs that basically are unable to properly run in kernel with this change. While it is understandable that a malicious application could DoS by split locking, it seems unacceptable to regress old/proprietary userspace programs through a default configuration that previously worked. An example of such breakage was reported in [1].
So let's add a sysctl to allow controlling the "misery mode" behavior, as per Thomas suggestion on [2]. This way, users running legacy and/or proprietary software are allowed to still execute them with a decent performance while still observe the warning messages on kernel log.
[0] https://lore.kernel.org/lkml/[email protected]/
[1] https://github.com/doitsujin/dxvk/issues/2938
[2] https://lore.kernel.org/lkml/87pmf4bter.ffs@tglx/
---
V3:
- Reworked with the suggestions from Bagas and Dave - thanks!
- Rebased to v6.1-rc2.
Dave, in the end I kept the 2 delayed workers, seems easier than encompass the worker in a new struct and have a bool embedded there (we can't use data directly in the work struct, since it keeps the workqueue state). Lemme know if you (or others) really prefer the other approach and I can rework if required.
Thanks again for comments/reviews!
Cheers,
Guilherme
Documentation/admin-guide/sysctl/kernel.rst | 23 ++++++++ arch/x86/kernel/cpu/intel.c | 63 +++++++++++++++++---- 2 files changed, 76 insertions(+), 10 deletions(-)
Documentation/admin-guide/sysctl/kernel.rst +23 -0
@@ -1314,6 +1314,29 @@ watchdog work to be queued by the watchdog timer function, otherwise the NMI watchdog — if enabled — can detect a hard lockup condition. +split_lock_mitigate (x86 only) +============================== + +On x86, each "split lock" imposes a system-wide performance penalty. On larger +systems, large numbers of split locks from unprivileged users can result in +denials of service to well-behaved and potentially more important users. + +The kernel mitigates these bad users by detecting split locks and imposing +penalties: forcing them to wait and only allowing one core to execute split +locks at a time. + +These mitigations can make those bad applications unbearably slow. Setting +split_lock_mitigate=0 may restore some application performance, but will also +increase system exposure to denial of service attacks from split lock users. + += =================================================================== +0 Disable the mitigation mode - just warns the split lock on kernel log + and exposes the system to denials of service from the split lockers. +1 Enable the mitigation mode (this is the default) - penalizes the split + lockers with intentional performance degradation. += =================================================================== + + stack_erasing =============
arch/x86/kernel/cpu/intel.c +53 -10
97 lines not on this page · raw message
Signature · 1 line
2.38.0
Review
Quoted 7 lines · Guilherme G. Piccoli
(quoted diff) +7
-0+split_lock_mitigate (x86 only) +============================== + +On x86, each "split lock" imposes a system-wide performance penalty. On larger +systems, large numbers of split locks from unprivileged users can result in +denials of service to well-behaved and potentially more important users. +
DoS which harms legitimate users?
Quoted 6 lines
(quoted diff) +6
-0+The kernel mitigates these bad users by detecting split locks and imposing +penalties: forcing them to wait and only allowing one core to execute split +locks at a time. + +These mitigations can make those bad applications unbearably slow. Setting +split_lock_mitigate=0 may restore some application performance, but will also
Review
Quoted 10 lines
(quoted diff) +10
-0+increase system exposure to denial of service attacks from split lock users. + += =================================================================== +0 Disable the mitigation mode - just warns the split lock on kernel log + and exposes the system to denials of service from the split lockers. +1 Enable the mitigation mode (this is the default) - penalizes the split + lockers with intentional performance degradation. += =================================================================== + +
The prose can be improved:
---- >8 ----
Documentation/admin-guide/sysctl/kernel.rst +10 -11
@@ -1319,22 +1319,21 @@ split_lock_mitigate (x86 only) On x86, each "split lock" imposes a system-wide performance penalty. On larger systems, large numbers of split locks from unprivileged users can result in -denials of service to well-behaved and potentially more important users. +denials of service attack. -The kernel mitigates these bad users by detecting split locks and imposing +The kernel mitigates that condition by detecting split locks and imposing penalties: forcing them to wait and only allowing one core to execute split locks at a time. -These mitigations can make those bad applications unbearably slow. Setting -split_lock_mitigate=0 may restore some application performance, but will also -increase system exposure to denial of service attacks from split lock users. +These mitigations can make applications which rely on split locks unbearably +slow. Setting split_lock_mitigate=0 may restore performance, but will also +increase likelihood of DoS caused by split locks. -= =================================================================== -0 Disable the mitigation mode - just warns the split lock on kernel log - and exposes the system to denials of service from the split lockers. -1 Enable the mitigation mode (this is the default) - penalizes the split - lockers with intentional performance degradation. -= =================================================================== += ================================================================== +0 Disable the mitigation - just warns the split lock on kernel log. +1 Enable the mitigation (default) - penalizes the split lockers with + intentional performance degradation. += ================================================================== stack_erasing
Thanks.
Signature · 1 line
An old man doll... just what I always wanted! - Clara
Quoted 1 line · Bagas Sanjaya
The prose can be improved:
I'm not sure that's a universal improvement. It chops out some valuable context.
Quoted 5 lines · Dave Hansen
Quoted 1 line · Bagas Sanjaya
The prose can be improved:
I'm not sure that's a universal improvement. It chops out some valuable context.
Seems like you forgot to quote my suggested diff.
Signature · 1 line
An old man doll... just what I always wanted! - Clara
Other than that, is this version good enough in your opinion Dave?
Personally I think Bagas could submit a patch later and propose the changes to the docs as well.
Cheers,
Guilherme
Quoted 30 lines · Guilherme G. Piccoli
Commit
b041b525dab9("x86/split_lock: Make life miserable for split lockers") changed the way the split lock detector works when in "warn" mode; basically, not only it shows the warn message, but also intentionally introduces a slowdown (through sleeping plus serialization mechanism) on such task. Based on discussions in [0], seems the warning alone wasn't enough motivation for userspace developers to fix their applications.Happens that originally the proposal in [0] was to add a new mode which would warns + slowdown the "split locking" task, keeping the old warn mode untouched. In the end, that idea was discarded and the regular/default "warn" mode now slowdowns the applications. This is quite aggressive with regards proprietary/legacy programs that basically are unable to properly run in kernel with this change. While it is understandable that a malicious application could DoS by split locking, it seems unacceptable to regress old/proprietary userspace programs through a default configuration that previously worked. An example of such breakage was reported in [1].
So let's add a sysctl to allow controlling the "misery mode" behavior, as per Thomas suggestion on [2]. This way, users running legacy and/or proprietary software are allowed to still execute them with a decent performance while still observe the warning messages on kernel log.
[0] https://lore.kernel.org/lkml/[email protected]/
Quoted 12 lines
Fixes b041b525dab9 ("x86/split_lock: Make life miserable for split lockers")Cc Fenghua Yu <[email protected]>Cc Joshua Ashton <[email protected]>Cc Melissa Wen <[email protected]>Cc Paul Gofman <[email protected]>Cc Pavel Machek <[email protected]>Cc Pierre-Loup Griffais <[email protected]>Cc Tony Luck <[email protected]>Cc Zebediah Figura <[email protected]>Suggested-by Thomas Gleixner <[email protected]>Tested-by Andre Almeida <[email protected]>Signed-off-by Guilherme G. Piccoli <[email protected]>
Hi Thomas / Tony, is there anything else to improve here? Suggestions / reviews are greatly appreciated.
In case this version is good enough, I'd like to ask if it's possible to get it merged during the rc cycle for 6.1 (I'll also backport it to stable 6.0), so we can have this fix available for a bigger public.
Thanks,
Guilherme
Quoted 19 lines · Guilherme G. Piccoli
Commit
b041b525dab9("x86/split_lock: Make life miserable for split lockers") changed the way the split lock detector works when in "warn" mode; basically, not only it shows the warn message, but also intentionally introduces a slowdown (through sleeping plus serialization mechanism) on such task. Based on discussions in [0], seems the warning alone wasn't enough motivation for userspace developers to fix their applications.Happens that originally the proposal in [0] was to add a new mode which would warns + slowdown the "split locking" task, keeping the old warn mode untouched. In the end, that idea was discarded and the regular/default "warn" mode now slowdowns the applications. This is quite aggressive with regards proprietary/legacy programs that basically are unable to properly run in kernel with this change. While it is understandable that a malicious application could DoS by split locking, it seems unacceptable to regress old/proprietary userspace programs through a default configuration that previously worked. An example of such breakage was reported in [1].
Quoted 4 lines
So let's add a sysctl to allow controlling the "misery mode" behavior, as per Thomas suggestion on [2]. This way, users running legacy and/or proprietary software are allowed to still execute them with a decent performance while still observe the warning messages on kernel log.
-Tony