Skip to contentSkip to conversation

x86/bus_lock: Enable bus lock detection

PATCH Superseded Fenghua Yu 2 participants 9 messages Nov 21, 2020

Subsystems (2)

v3 of 6 Superseded Latest is v6 →

lore
Share & tools
lore

Fenghua Yu

author
#0 lore raw

v3 cover x86/bus_lock: Enable bus lock detection

A bus lock [1] is acquired through either split locked access to writeback (WB) memory or any locked access to non-WB memory. This is typically >1000 cycles slower than an atomic operation within a cache line. It also disrupts performance on other cores.

Although split lock can be detected by #AC trap, the trap is triggered before the instruction acquires bus lock. This makes it difficult to mitigate bus lock (e.g. throttle the user application).

Some CPUs have ability to notify the kernel by an #DB trap after a user instruction acquires a bus lock and is executed. This allows the kernel to enforce user application throttling or mitigations.

#DB for bus lock detect fixes issues in #AC for split lock detect:
1) It's architectural ... just need to look at one CPUID bit to know it exists
2) The IA32_DEBUGCTL MSR, which reports bus lock in #DB, is per-thread. So each process or guest can have different behavior.
3) It has support for VMM/guests (new VMEXIT codes, etc).
4) It detects not only split locks but also bus locks from non-WB.

Hardware only generates #DB for bus lock detect when CPL>0 to avoid nested #DB from multiple bus locks while the first #DB is being handled.

Use the existing kernel command line option "split_lock_detect=" to handle #DB for bus lock:

split_lock_detect=
#AC for split lock #DB for bus lock

off Do nothing Do nothing

warn Kernel OOPs Warn once per task and
Warn once per task and and continues to run.
disable future checking When both features are
supported, warn in #DB

fatal Kernel OOPs Send SIGBUS to user.
Send SIGBUS to user When both features are
supported, split lock triggers #AC and bus lock from non-WB triggers #DB.

ratelimit:N Do nothing Limit bus lock rate to
N per second in the current non-root user.

Default split_lock_detect is "warn".

[1] Intel Instruction Set Extension Chapter 8: https://software.intel.com/sites/default/files/managed/c5/15/architecture-instruction-set-extensions-programming-reference.pdf

Change Log:
v3:
- Enable Bus Lock Detection when fatal to handle bus lock from non-WB (PeterZ).
- Add Acked-by: PeterZ in patch 2.

v2:
- Send SIGBUS in fatal case for bus lock #DB (PeterZ).

v1:
- Check bus lock bit by its positive polarity (Xiaoyao).
- Fix a few wording issues in the documentation (Randy).
[RFC v3 can be found at: https://lore.kernel.org/patchwork/cover/1329943/]

RFC v3:
- Remove DR6_RESERVED change (PeterZ).
- Simplify the documentation (Randy).

RFC v2:
- Architecture changed based on feedback from Thomas and PeterZ. #DB is
no longer generated for bus lock in ring0.
- Split the one single patch into four patches.
[RFC v1 can be found at: https://lore.kernel.org/lkml/[email protected]/]

Fenghua Yu (4):
x86/cpufeatures: Enumerate #DB for bus lock detection x86/bus_lock: Handle warn and fatal in #DB for bus lock x86/bus_lock: Set rate limit for bus lock Documentation/admin-guide: Change doc for split_lock_detect parameter

 .../admin-guide/kernel-parameters.txt         |  31 +++-
 arch/x86/include/asm/cpu.h                    |   9 +-
 arch/x86/include/asm/cpufeatures.h            |   1 +
 arch/x86/include/asm/msr-index.h              |   1 +
 arch/x86/include/uapi/asm/debugreg.h          |   1 +
 arch/x86/kernel/cpu/common.c                  |   2 +-
 arch/x86/kernel/cpu/intel.c                   | 155 +++++++++++++++---
 arch/x86/kernel/traps.c                       |   7 +
 include/linux/sched/user.h                    |   4 +-
 kernel/user.c                                 |   7 +
 10 files changed, 186 insertions(+), 32 deletions(-)
Signature · 1 line
2.29.2
author
#1 lore raw

PATCH v3 1/4 x86/cpufeatures: Enumerate #DB for bus lock detection

A bus lock is acquired though either split locked access to writeback (WB) memory or any locked access to non-WB memory. This is typically >1000 cycles slower than an atomic operation within a cache line. It also disrupts performance on other cores.

Some CPUs have ability to notify the kernel by an #DB trap after a user instruction acquires a bus lock and is executed. This allows the kernel to enforce user application throttling or mitigations.

The CPU feature flag to be shown in /proc/cpuinfo will be "bus_lock_detect".

---
 arch/x86/include/asm/cpufeatures.h | 1 +
 1 file changed, 1 insertion(+)
arch/x86/include/asm/cpufeatures.h +1 -0
index dad350d42ecf..f375d9cb8123 100644
@@ -352,6 +352,7 @@
 #define X86_FEATURE_AVX512_VPOPCNTDQ	(16*32+14) /* POPCNT for vectors of DW/QW */
 #define X86_FEATURE_LA57		(16*32+16) /* 5-level page tables */
 #define X86_FEATURE_RDPID		(16*32+22) /* RDPID instruction */
+#define X86_FEATURE_BUS_LOCK_DETECT	(16*32+24) /* Bus Lock detect */
 #define X86_FEATURE_CLDEMOTE		(16*32+25) /* CLDEMOTE instruction */
 #define X86_FEATURE_MOVDIRI		(16*32+27) /* MOVDIRI instruction */
 #define X86_FEATURE_MOVDIR64B		(16*32+28) /* MOVDIR64B instruction */
Signature · 1 line
2.29.2
author
#2 lore raw

PATCH v3 2/4 x86/bus_lock: Handle warn and fatal in #DB for bus lock

In reply to Fenghua Yu

#DB for bus lock is enabled by bus lock detection bit 2 in DEBUGCTL MSR while #AC for split lock is enabled by split lock detection bit 29 in TEST_CTRL MSR.

Delivery of #DB for bus lock in userspace clears DR6[11]. To avoid confusion in identifying #DB, #DB handler sets the bit to 1 before returning to the interrupted task.

Use the existing kernel command line option "split_lock_detect=" to handle #DB for bus lock:

split_lock_detect=
#AC for split lock #DB for bus lock

off Do nothing Do nothing

warn Kernel OOPs Warn once per task and
Warn once per task and and continues to run.
disable future checking When both features are
supported, warn in #DB

fatal Kernel OOPs Send SIGBUS to user
Send SIGBUS to user When both features are
supported, split lock triggers #AC and bus lock from non-WB triggers #DB.

Default option is "warn".

Hardware only generates #DB for bus lock detect when CPL>0 to avoid nested #DB from multiple bus locks while the first #DB is being handled. So no need to handle #DB for bus lock detected in the kernel.

---
Change Log:
v3:
- Enable Bus Lock Detection when fatal to handle bus lock from non-WB (PeterZ).

v2:
- Send SIGBUS in fatal case for bus lock #DB (PeterZ).

v1::
- Check bus lock bit by its positive polarity (Xiaoyao).

RFC v3:
- Remove DR6_RESERVED change (PeterZ).

 arch/x86/include/asm/cpu.h           |   9 +-
 arch/x86/include/asm/msr-index.h     |   1 +
 arch/x86/include/uapi/asm/debugreg.h |   1 +
 arch/x86/kernel/cpu/common.c         |   2 +-
 arch/x86/kernel/cpu/intel.c          | 126 ++++++++++++++++++++++-----
 arch/x86/kernel/traps.c              |   7 ++
 6 files changed, 123 insertions(+), 23 deletions(-)
arch/x86/include/asm/cpu.h +7 -2
index da78ccbd493b..c16b9bab502f 100644
@@ -41,12 +41,13 @@ unsigned int x86_family(unsigned int sig);
 unsigned int x86_model(unsigned int sig);
 unsigned int x86_stepping(unsigned int sig);
 #ifdef CONFIG_CPU_SUP_INTEL
-extern void __init cpu_set_core_cap_bits(struct cpuinfo_x86 *c);
+extern void __init sld_setup(struct cpuinfo_x86 *c);
 extern void switch_to_sld(unsigned long tifn);
 extern bool handle_user_split_lock(struct pt_regs *regs, long error_code);
 extern bool handle_guest_split_lock(unsigned long ip);
+extern void handle_bus_lock(struct pt_regs *regs);
 #else
-static inline void __init cpu_set_core_cap_bits(struct cpuinfo_x86 *c) {}
+static inline void __init sld_setup(struct cpuinfo_x86 *c) {}
 static inline void switch_to_sld(unsigned long tifn) {}
 static inline bool handle_user_split_lock(struct pt_regs *regs, long error_code)
 {
@@ -57,6 +58,10 @@ static inline bool handle_guest_split_lock(unsigned long ip)
 {
 	return false;
 }
+
+static inline void handle_bus_lock(struct pt_regs *regs)
+{
+}
 #endif
 #ifdef CONFIG_IA32_FEAT_CTL
 void init_ia32_feat_ctl(struct cpuinfo_x86 *c);
arch/x86/include/asm/msr-index.h +1 -0
index 972a34d93505..62f7534e0855 100644
@@ -264,6 +264,7 @@
 #define DEBUGCTLMSR_LBR			(1UL <<  0) /* last branch recording */
 #define DEBUGCTLMSR_BTF_SHIFT		1
 #define DEBUGCTLMSR_BTF			(1UL <<  1) /* single-step on branches */
+#define DEBUGCTLMSR_BUS_LOCK_DETECT	(1UL <<  2) /* Bus lock detect */
 #define DEBUGCTLMSR_TR			(1UL <<  6)
 #define DEBUGCTLMSR_BTS			(1UL <<  7)
 #define DEBUGCTLMSR_BTINT		(1UL <<  8)
arch/x86/include/uapi/asm/debugreg.h +1 -0
index d95d080b30e3..0007ba077c0c 100644
@@ -24,6 +24,7 @@
 #define DR_TRAP3	(0x8)		/* db3 */
 #define DR_TRAP_BITS	(DR_TRAP0|DR_TRAP1|DR_TRAP2|DR_TRAP3)
 
+#define DR_BUS_LOCK	(0x800)		/* bus_lock */
 #define DR_STEP		(0x4000)	/* single-step */
 #define DR_SWITCH	(0x8000)	/* task switch */
 
arch/x86/kernel/cpu/common.c +1 -1
index 35ad8480c464..92705ac301ec 100644
@@ -1327,7 +1327,7 @@ static void __init early_identify_cpu(struct cpuinfo_x86 *c)
 
 	cpu_set_bug_bits(c);
 
-	cpu_set_core_cap_bits(c);
+	sld_setup(c);
 
 	fpu__init_system(c);
 
arch/x86/kernel/cpu/intel.c +106 -20

203 lines not on this page · raw message

arch/x86/kernel/traps.c +7 -0
index e19df6cde35d..dea26135e10b 100644
@@ -970,6 +970,13 @@ static __always_inline void exc_debug_user(struct pt_regs *regs,
 		goto out_irq;
 	}
 
+	/*
+	 * Handle bus lock. #DB for bus lock can only be triggered from
+	 * userspace. The bus lock bit was flipped to positive polarity.
+	 */
+	if (dr6 & DR_BUS_LOCK)
+		handle_bus_lock(regs);
+
 	/* Add the virtual_dr6 bits for signals. */
 	dr6 |= current->thread.virtual_dr6;
 	if (dr6 & (DR_STEP | DR_TRAP_BITS) || icebp)
Signature · 1 line
2.29.2
author
#3 lore raw

PATCH v3 3/4 x86/bus_lock: Set rate limit for bus lock

In reply to Fenghua Yu

To enforce user application throttling or mitigations, extend the existing split lock detect kernel parameter:
split_lock_detect=ratelimit:N

It limits bus lock rate to N per second for non-root users.

---
 arch/x86/kernel/cpu/intel.c | 35 ++++++++++++++++++++++++++++++-----
 include/linux/sched/user.h  |  4 +++-
 kernel/user.c               |  7 +++++++
 3 files changed, 40 insertions(+), 6 deletions(-)
arch/x86/kernel/cpu/intel.c +30 -5
index 77c3e33f41c7..5eb5822a446d 100644
@@ -10,6 +10,9 @@
 #include <linux/thread_info.h>
 #include <linux/init.h>
 #include <linux/uaccess.h>
+#include <linux/cred.h>
+#include <linux/delay.h>
+#include <linux/sched/user.h>
 
 #include <asm/cpufeature.h>
 #include <asm/msr.h>
@@ -40,6 +43,7 @@ enum split_lock_detect_state {
 	sld_off = 0,
 	sld_warn,
 	sld_fatal,
+	sld_ratelimit,
 };
 
 /*
@@ -998,13 +1002,25 @@ static const struct {
 	{ "off",	sld_off   },
 	{ "warn",	sld_warn  },
 	{ "fatal",	sld_fatal },
+	{ "ratelimit:", sld_ratelimit },
 };
 
 static inline bool match_option(const char *arg, int arglen, const char *opt)
 {
-	int len = strlen(opt);
 
-	return len == arglen && !strncmp(arg, opt, len);
+	int len = strlen(opt), ratelimit;
+
+	if (strncmp(arg, opt, len))
+		return false;
+
+	if (sscanf(arg, "ratelimit:%d", &ratelimit) == 1 && ratelimit > 0 &&
+	    ratelimit_bl <= HZ / 2) {
+		ratelimit_bl = ratelimit;
+
+		return true;
+	}
+
+	return len == arglen;
 }
 
 static bool split_lock_verify_msr(bool on)
@@ -1085,10 +1101,10 @@ static void sld_update_msr(bool on)
 static void split_lock_init(void)
 {
 	/*
-	 * If supported, #DB for bus lock will handle warn
+	 * If supported, #DB for bus lock will handle warn or ratelimit
 	 * and #AC for split lock is disabled.
 	 */
-	if (bld && sld_state == sld_warn) {
+	if ((bld && sld_state == sld_warn) || sld_state == sld_ratelimit) {
 		split_lock_verify_msr(false);
 		return;
 	}
@@ -1147,7 +1163,8 @@ static void bus_lock_init(void)
 
 bool handle_user_split_lock(struct pt_regs *regs, long error_code)
 {
-	if ((regs->flags & X86_EFLAGS_AC) || !sld || sld_state == sld_fatal)
+	if ((regs->flags & X86_EFLAGS_AC) || !sld || sld_state == sld_fatal ||
+	    sld_state == sld_ratelimit)
 		return false;
 	split_lock_warn(regs->ip);
 	return true;
@@ -1168,6 +1185,10 @@ void handle_bus_lock(struct pt_regs *regs)
 	case sld_fatal:
 		force_sig_fault(SIGBUS, BUS_ADRALN, NULL);
 		break;
+	case sld_ratelimit:
+		while (!__ratelimit(&get_current_user()->ratelimit_bl))
+			msleep(1000 / ratelimit_bl);
+		break;
 	}
 }
 
@@ -1261,6 +1282,10 @@ static void sld_state_show(void)
 		if (bld)
 			pr_info("#DB: sending SIGBUS on user-space bus_locks%s\n", sld ? " from non-WB" : "");
 		break;
+	case sld_ratelimit:
+		if (bld)
+			pr_info("#DB: setting rate limit to %d/sec per user on non-root user-space bus_locks\n", ratelimit_bl);
+		break;
 	}
 }
 
include/linux/sched/user.h +3 -1
index a8ec3b6093fc..79f95002a123 100644
@@ -40,8 +40,9 @@ struct user_struct {
 	atomic_t nr_watches;	/* The number of watches this user currently has */
 #endif
 
-	/* Miscellaneous per-user rate limit */
+	/* Miscellaneous per-user rate limits */
 	struct ratelimit_state ratelimit;
+	struct ratelimit_state ratelimit_bl;
 };
 
 extern int uids_sysfs_init(void);
@@ -51,6 +52,7 @@ extern struct user_struct *find_user(kuid_t);
 extern struct user_struct root_user;
 #define INIT_USER (&root_user)
 
+extern int ratelimit_bl;
 
 /* per-UID process charging. */
 extern struct user_struct * alloc_uid(kuid_t);
kernel/user.c +7 -0
index b1635d94a1f2..8fc19706bc91 100644
@@ -103,6 +103,7 @@ struct user_struct root_user = {
 	.locked_shm     = 0,
 	.uid		= GLOBAL_ROOT_UID,
 	.ratelimit	= RATELIMIT_STATE_INIT(root_user.ratelimit, 0, 0),
+	.ratelimit_bl	= RATELIMIT_STATE_INIT(root_user.ratelimit_bl, 0, 0),
 };
 
 /*
@@ -172,6 +173,9 @@ void free_uid(struct user_struct *up)
 		free_user(up, flags);
 }
 
+/* Architectures (e.g. X86) may set this for rate-limited bus locks. */
+int ratelimit_bl;
+
 struct user_struct *alloc_uid(kuid_t uid)
 {
 	struct hlist_head *hashent = uidhashentry(uid);
@@ -190,6 +194,9 @@ struct user_struct *alloc_uid(kuid_t uid)
 		refcount_set(&new->__count, 1);
 		ratelimit_state_init(&new->ratelimit, HZ, 100);
 		ratelimit_set_flags(&new->ratelimit, RATELIMIT_MSG_ON_RELEASE);
+		ratelimit_state_init(&new->ratelimit_bl, HZ, ratelimit_bl);
+		ratelimit_set_flags(&new->ratelimit_bl,
+				    RATELIMIT_MSG_ON_RELEASE);
 
 		/*
 		 * Before adding this, check whether we raced
Signature · 1 line
2.29.2
author
#4 lore raw

PATCH v3 4/4 Documentation/admin-guide: Change doc for split_lock_detect parameter

In reply to Fenghua Yu

Since #DB for bus lock detect changes the split_lock_detect parameter, update the documentation for the changes.

---
Change Log:
v3:
- Enable Bus Lock Detection when fatal to handle bus lock from non-WB (PeterZ).

v1:
- Fix a few wording issues (Randy).

RFC v2:
- Simplify the documentation (Randy).

 .../admin-guide/kernel-parameters.txt         | 31 +++++++++++++++----
 1 file changed, 25 insertions(+), 6 deletions(-)
Documentation/admin-guide/kernel-parameters.txt +25 -6
index 44fde25bb221..dcdb4fd3597d 100644
@@ -5051,27 +5051,46 @@
 	spia_peddr=
 
 	split_lock_detect=
-			[X86] Enable split lock detection
+			[X86] Enable split lock detection or bus lock detection
 
 			When enabled (and if hardware support is present), atomic
 			instructions that access data across cache line
-			boundaries will result in an alignment check exception.
+			boundaries will result in an alignment check exception
+			for split lock detection or a debug exception for
+			bus lock detection.
 
 			off	- not enabled
 
-			warn	- the kernel will emit rate limited warnings
+			warn	- the kernel will emit rate-limited warnings
 				  about applications triggering the #AC
-				  exception. This mode is the default on CPUs
-				  that supports split lock detection.
+				  exception or the #DB exception. This mode is
+				  the default on CPUs that support split lock
+				  detection or bus lock detection. Default
+				  behavior is by #DB if both features are
+				  enabled in hardware.
 
 			fatal	- the kernel will send SIGBUS to applications
-				  that trigger the #AC exception.
+				  that trigger the #AC exception or the #DB
+				  exception. If both features are enabled in
+				  hardware, split lock triggers #AC and bus
+				  lock from non-WB triggers #DB.
+
+			ratelimit:N -
+				  Set rate limit to N bus locks per second
+				  for bus lock detection. 0 < N <= HZ/2 and
+				  N is approximate. Only applied to non-root
+				  users.
+
+				  N/A for split lock detection.
 
 			If an #AC exception is hit in the kernel or in
 			firmware (i.e. not while executing in user mode)
 			the kernel will oops in either "warn" or "fatal"
 			mode.
 
+			#DB exception for bus lock is triggered only when
+			CPL > 0.
+
 	srbds=		[X86,INTEL]
 			Control the Special Register Buffer Data Sampling
 			(SRBDS) mitigation.
Signature · 1 line
2.29.2
reviewer
#5 lore raw

Re: [PATCH v3 4/4] Documentation/admin-guide: Change doc for split_lock_detect parameter

Hi--

Documentation/admin-guide/kernel-parameters.txt line 5082

Quoted 5 lines · Fenghua Yu
Documentation/admin-guide/kernel-parameters.txt +5 -0
+			ratelimit:N -
+				  Set rate limit to N bus locks per second
+				  for bus lock detection. 0 < N <= HZ/2 and
+				  N is approximate. Only applied to non-root
+				  users.

Sorry, but I don't know what this means. I think it's the "and N is appropriate" that is confusing me.

0 < N <= HZ/2 and N is appropriate.

Signature · 1 line
~Randy
#6 lore raw

RE: [PATCH v3 4/4] Documentation/admin-guide: Change doc for split_lock_detect parameter

Hi, Randy,

Quoted 11 lines

Documentation/admin-guide/kernel-parameters.txt line 5082

Quoted 5 lines
Documentation/admin-guide/kernel-parameters.txt +5 -0
+			ratelimit:N -
+				  Set rate limit to N bus locks per second
+				  for bus lock detection. 0 < N <= HZ/2 and
+				  N is approximate. Only applied to non-root
+				  users.

Sorry, but I don't know what this means. I think it's the "and N is appropriate"
that is confusing me.

0 < N <= HZ/2 and N is appropriate.

You are right. I will remove "and N is appropriate" in the next version.

Could you please ack this patch? Can I add Acked-by from you in the updated patch?

Thank you very much for your review!

-Fenghua

reviewer Acked-by
#7 lore raw

Re: [PATCH v3 4/4] Documentation/admin-guide: Change doc for split_lock_detect parameter

Quoted 12 lines · Fenghua

Hi, Randy,

Quoted 10 lines

Documentation/admin-guide/kernel-parameters.txt line 5082

Quoted 5 lines
Documentation/admin-guide/kernel-parameters.txt +5 -0
+			ratelimit:N -
+				  Set rate limit to N bus locks per second
+				  for bus lock detection. 0 < N <= HZ/2 and
+				  N is approximate. Only applied to non-root
+				  users.

Sorry, but I don't know what this means. I think it's the "and N is appropriate"
that is confusing me.

Quoted 7 lines
Quoted 1 line

0 < N <= HZ/2 and N is appropriate.

You are right. I will remove "and N is appropriate" in the next version.

Could you please ack this patch? Can I add Acked-by from you in the updated patch?

Thank you very much for your review!

Sure, no problem.

thanks.

Signature · 1 line
~Randy
#8 lore raw

RE: [PATCH v3 4/4] Documentation/admin-guide: Change doc for split_lock_detect parameter

Hi, Randy,

Quoted 12 lines
Quoted 12 lines
Quoted 9 lines

Documentation/admin-guide/kernel-parameters.txt line 5082

Quoted 3 lines
Documentation/admin-guide/kernel-parameters.txt +3 -0
+				  for bus lock detection. 0 < N <= HZ/2 and
+				  N is approximate. Only applied to non-root
+				  users.

Sorry, but I don't know what this means. I think it's the "and N is appropriate"
that is confusing me.

0 < N <= HZ/2 and N is appropriate.

You are right. I will remove "and N is appropriate" in the next version.

Quoted 8 lines
Quoted 1 line

Could you please ack this patch? Can I add Acked-by from you in the

updated patch?

Quoted 2 lines

Thank you very much for your review!

Sure, no problem.

Really appreciate your review!

-Fenghua