Skip to contentSkip to conversation

x86/split_lock: Add sysctl to control the misery mode

PATCH In review Guilherme G. Piccoli 4 participants 7 messages Oct 24, 2022 – Nov 10, 2022

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

PATCH v3 1/1 x86/split_lock: Add sysctl to control the misery mode

Commit b041b525dab9 ("x86/split_lock: Make life miserable for split lockers") changed the way the split lock detector works when in "warn" mode; basically, not only it shows the warn message, but also intentionally introduces a slowdown (through sleeping plus serialization mechanism) on such task. Based on discussions in [0], seems the warning alone wasn't enough motivation for userspace developers to fix their applications.

Happens that originally the proposal in [0] was to add a new mode which would warns + slowdown the "split locking" task, keeping the old warn mode untouched. In the end, that idea was discarded and the regular/default "warn" mode now slowdowns the applications. This is quite aggressive with regards proprietary/legacy programs that basically are unable to properly run in kernel with this change. While it is understandable that a malicious application could DoS by split locking, it seems unacceptable to regress old/proprietary userspace programs through a default configuration that previously worked. An example of such breakage was reported in [1].

So let's add a sysctl to allow controlling the "misery mode" behavior, as per Thomas suggestion on [2]. This way, users running legacy and/or proprietary software are allowed to still execute them with a decent performance while still observe the warning messages on kernel log.

[0] https://lore.kernel.org/lkml/[email protected]/

[1] https://github.com/doitsujin/dxvk/issues/2938

[2] https://lore.kernel.org/lkml/87pmf4bter.ffs@tglx/

Fixes b041b525dab9 ("x86/split_lock: Make life miserable for split lockers")
Cc Fenghua Yu <[email protected]>
Cc Joshua Ashton <[email protected]>
Cc Melissa Wen <[email protected]>
Cc Paul Gofman <[email protected]>
Cc Pavel Machek <[email protected]>
Cc Pierre-Loup Griffais <[email protected]>
Cc Tony Luck <[email protected]>
Cc Zebediah Figura <[email protected]>

---

V3:
- Reworked with the suggestions from Bagas and Dave - thanks!
- Rebased to v6.1-rc2.

Dave, in the end I kept the 2 delayed workers, seems easier than encompass the worker in a new struct and have a bool embedded there (we can't use data directly in the work struct, since it keeps the workqueue state). Lemme know if you (or others) really prefer the other approach and I can rework if required.

Thanks again for comments/reviews!
Cheers,

Guilherme

 Documentation/admin-guide/sysctl/kernel.rst | 23 ++++++++
 arch/x86/kernel/cpu/intel.c                 | 63 +++++++++++++++++----
 2 files changed, 76 insertions(+), 10 deletions(-)
Documentation/admin-guide/sysctl/kernel.rst +23 -0
index 98d1b198b2b4..c2c64c1b706f 100644
@@ -1314,6 +1314,29 @@ watchdog work to be queued by the watchdog timer function, otherwise the NMI
 watchdog — if enabled — can detect a hard lockup condition.
 
 
+split_lock_mitigate (x86 only)
+==============================
+
+On x86, each "split lock" imposes a system-wide performance penalty. On larger
+systems, large numbers of split locks from unprivileged users can result in
+denials of service to well-behaved and potentially more important users.
+
+The kernel mitigates these bad users by detecting split locks and imposing
+penalties: forcing them to wait and only allowing one core to execute split
+locks at a time.
+
+These mitigations can make those bad applications unbearably slow. Setting
+split_lock_mitigate=0 may restore some application performance, but will also
+increase system exposure to denial of service attacks from split lock users.
+
+= ===================================================================
+0 Disable the mitigation mode - just warns the split lock on kernel log
+  and exposes the system to denials of service from the split lockers.
+1 Enable the mitigation mode (this is the default) - penalizes the split
+  lockers with intentional performance degradation.
+= ===================================================================
+
+
 stack_erasing
 =============
 
arch/x86/kernel/cpu/intel.c +53 -10

97 lines not on this page · raw message

Signature · 1 line
2.38.0
#1 lore raw

Review

Quoted 7 lines · Guilherme G. Piccoli
(quoted diff) +7 -0
+split_lock_mitigate (x86 only)
+==============================
+
+On x86, each "split lock" imposes a system-wide performance penalty. On larger
+systems, large numbers of split locks from unprivileged users can result in
+denials of service to well-behaved and potentially more important users.
+

DoS which harms legitimate users?

Quoted 6 lines
(quoted diff) +6 -0
+The kernel mitigates these bad users by detecting split locks and imposing
+penalties: forcing them to wait and only allowing one core to execute split
+locks at a time.
+
+These mitigations can make those bad applications unbearably slow. Setting
+split_lock_mitigate=0 may restore some application performance, but will also

Review

Quoted 10 lines
(quoted diff) +10 -0
+increase system exposure to denial of service attacks from split lock users.
+
+= ===================================================================
+0 Disable the mitigation mode - just warns the split lock on kernel log
+  and exposes the system to denials of service from the split lockers.
+1 Enable the mitigation mode (this is the default) - penalizes the split
+  lockers with intentional performance degradation.
+= ===================================================================
+
+

The prose can be improved:

---- >8 ----

Documentation/admin-guide/sysctl/kernel.rst +10 -11
index c2c64c1b706ff6..cdc5b75adb4633 100644
@@ -1319,22 +1319,21 @@ split_lock_mitigate (x86 only)
 
 On x86, each "split lock" imposes a system-wide performance penalty. On larger
 systems, large numbers of split locks from unprivileged users can result in
-denials of service to well-behaved and potentially more important users.
+denials of service attack.
 
-The kernel mitigates these bad users by detecting split locks and imposing
+The kernel mitigates that condition by detecting split locks and imposing
 penalties: forcing them to wait and only allowing one core to execute split
 locks at a time.
 
-These mitigations can make those bad applications unbearably slow. Setting
-split_lock_mitigate=0 may restore some application performance, but will also
-increase system exposure to denial of service attacks from split lock users.
+These mitigations can make applications which rely on split locks unbearably
+slow. Setting split_lock_mitigate=0 may restore performance, but will also
+increase likelihood of DoS caused by split locks.
 
-= ===================================================================
-0 Disable the mitigation mode - just warns the split lock on kernel log
-  and exposes the system to denials of service from the split lockers.
-1 Enable the mitigation mode (this is the default) - penalizes the split
-  lockers with intentional performance degradation.
-= ===================================================================
+= ==================================================================
+0 Disable the mitigation - just warns the split lock on kernel log.
+1 Enable the mitigation (default) - penalizes the split lockers with
+  intentional performance degradation.
+= ==================================================================
 
 
 stack_erasing

Thanks.

Signature · 1 line
An old man doll... just what I always wanted! - Clara
#2 lore raw
Quoted 1 line · Bagas Sanjaya

The prose can be improved:

I'm not sure that's a universal improvement. It chops out some valuable context.

#3 lore raw
Quoted 5 lines · Dave Hansen
Quoted 1 line · Bagas Sanjaya

The prose can be improved:

I'm not sure that's a universal improvement. It chops out some valuable context.

Seems like you forgot to quote my suggested diff.

Signature · 1 line
An old man doll... just what I always wanted! - Clara
#4 lore raw
In reply to Dave Hansen

Other than that, is this version good enough in your opinion Dave?

Personally I think Bagas could submit a patch later and propose the changes to the docs as well.

Cheers,

Guilherme

#5 lore raw
In reply to Guilherme G. Piccoli
Quoted 30 lines · Guilherme G. Piccoli

Commit b041b525dab9 ("x86/split_lock: Make life miserable for split lockers") changed the way the split lock detector works when in "warn" mode; basically, not only it shows the warn message, but also intentionally introduces a slowdown (through sleeping plus serialization mechanism) on such task. Based on discussions in [0], seems the warning alone wasn't enough motivation for userspace developers to fix their applications.

Happens that originally the proposal in [0] was to add a new mode which would warns + slowdown the "split locking" task, keeping the old warn mode untouched. In the end, that idea was discarded and the regular/default "warn" mode now slowdowns the applications. This is quite aggressive with regards proprietary/legacy programs that basically are unable to properly run in kernel with this change. While it is understandable that a malicious application could DoS by split locking, it seems unacceptable to regress old/proprietary userspace programs through a default configuration that previously worked. An example of such breakage was reported in [1].

So let's add a sysctl to allow controlling the "misery mode" behavior, as per Thomas suggestion on [2]. This way, users running legacy and/or proprietary software are allowed to still execute them with a decent performance while still observe the warning messages on kernel log.

[0] https://lore.kernel.org/lkml/[email protected]/

[1] https://github.com/doitsujin/dxvk/issues/2938

[2] https://lore.kernel.org/lkml/87pmf4bter.ffs@tglx/

Quoted 12 lines
Fixes b041b525dab9 ("x86/split_lock: Make life miserable for split lockers")
Cc Fenghua Yu <[email protected]>
Cc Joshua Ashton <[email protected]>
Cc Melissa Wen <[email protected]>
Cc Paul Gofman <[email protected]>
Cc Pavel Machek <[email protected]>
Cc Pierre-Loup Griffais <[email protected]>
Cc Tony Luck <[email protected]>
Cc Zebediah Figura <[email protected]>

Hi Thomas / Tony, is there anything else to improve here? Suggestions / reviews are greatly appreciated.

In case this version is good enough, I'd like to ask if it's possible to get it merged during the rc cycle for 6.1 (I'll also backport it to stable 6.0), so we can have this fix available for a bigger public.

Thanks,

Guilherme

Reviewed-by on v3 1/1
#6 lore raw
In reply to Guilherme G. Piccoli
Quoted 19 lines · Guilherme G. Piccoli

Commit b041b525dab9 ("x86/split_lock: Make life miserable for split lockers") changed the way the split lock detector works when in "warn" mode; basically, not only it shows the warn message, but also intentionally introduces a slowdown (through sleeping plus serialization mechanism) on such task. Based on discussions in [0], seems the warning alone wasn't enough motivation for userspace developers to fix their applications.

Happens that originally the proposal in [0] was to add a new mode which would warns + slowdown the "split locking" task, keeping the old warn mode untouched. In the end, that idea was discarded and the regular/default "warn" mode now slowdowns the applications. This is quite aggressive with regards proprietary/legacy programs that basically are unable to properly run in kernel with this change. While it is understandable that a malicious application could DoS by split locking, it seems unacceptable to regress old/proprietary userspace programs through a default configuration that previously worked. An example of such breakage was reported in [1].

Quoted 4 lines

So let's add a sysctl to allow controlling the "misery mode" behavior, as per Thomas suggestion on [2]. This way, users running legacy and/or proprietary software are allowed to still execute them with a decent performance while still observe the warning messages on kernel log.

-Tony