Skip to contentSkip to conversation

smp: Use release stores for csd_lock_record() state

PATCH Applied Usama Arif 6 participants 6 messages Jun 29 – Jul 1

Subsystems (2)
lore
Share & tools
lore

Usama Arif

author
#0 lore raw

PATCH v3 1/1 smp: Use release stores for csd_lock_record() state

__csd_lock_record() publishes per-CPU diagnostic state (cur_csd, cur_csd_func, cur_csd_info) that is consumed from a remote CPU by csd_lock_wait_toolong() via smp_load_acquire(&cur_csd). To order the matching cur_csd_func/cur_csd_info stores before the cur_csd publication, the producer issues smp_wmb() before writing cur_csd; to order the publication before the subsequent callback execution or CSD unlock, it issues smp_mb() after the write. The clear path mirrors this with smp_mb() before storing NULL into cur_csd so the preceding callback/unlock is observed first.

The smp_mb() pair is heavier than what the consumer actually requires (on x86 each emits a locked full barrier). The consumer only needs to observe the matching cur_csd_func/cur_csd_info when it sees a non-NULL cur_csd, and to observe the preceding callback/unlock when it sees NULL -- both of which a release/acquire pair provides. The extra two-way ordering enforced by smp_mb() -- that cur_csd publication be observed before callback execution or unlock becomes visible -- would only matter if cur_csd were an exact live-state marker. csd_lock_wait_toolong() does not treat it that way: it snapshots cur_csd via smp_load_acquire() and then prints / dumps / re-IPIs without an RCU-style stall-ended recheck, so the diagnostic already tolerates the remote CPU completing its work between snapshot and report. cur_csd is best-effort context, not a precise stall boundary.

Replace the smp_wmb() + plain store + smp_mb() in the publish path, and the smp_mb() + plain store in the clear path, with smp_store_release(). This pairs with the smp_load_acquire() in csd_lock_wait_toolong(): preceding cur_csd_func/cur_csd_info stores become visible before a remote reader observes the non-NULL publication, and any preceding callback/unlock becomes visible before a reader observes the NULL clear.

---
v2 -> v3: https://lore.kernel.org/all/[email protected]/
- Restructure changelog into context/problem/solution form (Thomas Gleixner).
- Add reciprocal pairing comment on the smp_load_acquire() in csd_lock_wait_toolong() (Dmitry).

v1 -> v2: https://lore.kernel.org/all/[email protected]/
- Document where the smp_store_release() synchronizes with (Alan
Stern, Randy Dunlap and Paul McKenney).

---
 kernel/smp.c | 26 +++++++++++++++++++-------
 1 file changed, 19 insertions(+), 7 deletions(-)
kernel/smp.c +19 -7
index a0bb56bd8dda..8a847a34f132 100644
@@ -182,16 +182,22 @@ static atomic_t csd_bug_count = ATOMIC_INIT(0);
 static void __csd_lock_record(call_single_data_t *csd)
 {
 	if (!csd) {
-		smp_mb(); /* NULL cur_csd after unlock. */
-		__this_cpu_write(cur_csd, NULL);
+		/*
+		 * Pairs with smp_load_acquire() of cur_csd in
+		 * csd_lock_wait_toolong(): orders any preceding CSD
+		 * callback/unlock before a remote reader observes NULL.
+		 */
+		smp_store_release(this_cpu_ptr(&cur_csd), NULL);
 		return;
 	}
 	__this_cpu_write(cur_csd_func, csd->func);
 	__this_cpu_write(cur_csd_info, csd->info);
-	smp_wmb(); /* func and info before csd. */
-	__this_cpu_write(cur_csd, csd);
-	smp_mb(); /* Update cur_csd before function call. */
-		  /* Or before unlock, as the case may be. */
+	/*
+	 * Pairs with smp_load_acquire() of cur_csd in
+	 * csd_lock_wait_toolong(): publishes cur_csd_func and
+	 * cur_csd_info before the non-NULL pointer becomes visible.
+	 */
+	smp_store_release(this_cpu_ptr(&cur_csd), csd);
 }
 
 static __always_inline void csd_lock_record(call_single_data_t *csd)
@@ -272,7 +278,13 @@ static bool csd_lock_wait_toolong(call_single_data_t *csd, u64 ts0, u64 *ts1, in
 		cpux = 0;
 	else
 		cpux = cpu;
-	cpu_cur_csd = smp_load_acquire(&per_cpu(cur_csd, cpux)); /* Before func and info. */
+	/*
+	 * Pairs with smp_store_release() of cur_csd in __csd_lock_record():
+	 * a non-NULL cur_csd here implies cur_csd_func and cur_csd_info
+	 * are the matching publication; a NULL value is ordered after any
+	 * preceding CSD callback/unlock on the remote CPU.
+	 */
+	cpu_cur_csd = smp_load_acquire(&per_cpu(cur_csd, cpux));
 	/* How long since this CSD lock was stuck. */
 	ts_delta = ts2 - ts0;
 	pr_alert("csd: %s non-responsive CSD lock (#%d) on CPU#%d, waiting %lld ns for CPU#%02d %pS(%ps).\n",
Signature · 1 line
2.53.0-Meta
Reviewed-by on v3 1/1
#1 lore raw
Quoted 33 lines · Usama Arif

__csd_lock_record() publishes per-CPU diagnostic state (cur_csd, cur_csd_func, cur_csd_info) that is consumed from a remote CPU by csd_lock_wait_toolong() via smp_load_acquire(&cur_csd). To order the matching cur_csd_func/cur_csd_info stores before the cur_csd publication, the producer issues smp_wmb() before writing cur_csd; to order the publication before the subsequent callback execution or CSD unlock, it issues smp_mb() after the write. The clear path mirrors this with smp_mb() before storing NULL into cur_csd so the preceding callback/unlock is observed first.

The smp_mb() pair is heavier than what the consumer actually requires (on x86 each emits a locked full barrier). The consumer only needs to observe the matching cur_csd_func/cur_csd_info when it sees a non-NULL cur_csd, and to observe the preceding callback/unlock when it sees NULL -- both of which a release/acquire pair provides. The extra two-way ordering enforced by smp_mb() -- that cur_csd publication be observed before callback execution or unlock becomes visible -- would only matter if cur_csd were an exact live-state marker. csd_lock_wait_toolong() does not treat it that way: it snapshots cur_csd via smp_load_acquire() and then prints / dumps / re-IPIs without an RCU-style stall-ended recheck, so the diagnostic already tolerates the remote CPU completing its work between snapshot and report. cur_csd is best-effort context, not a precise stall boundary.

Replace the smp_wmb() + plain store + smp_mb() in the publish path, and the smp_mb() + plain store in the clear path, with smp_store_release(). This pairs with the smp_load_acquire() in csd_lock_wait_toolong(): preceding cur_csd_func/cur_csd_info stores become visible before a remote reader observes the non-NULL publication, and any preceding callback/unlock becomes visible before a reader observes the NULL clear.

Quoted 1 line
merge bot Applied
#2 lore raw
In reply to Usama Arif
The following commit has been merged into the smp/core branch of tip:

The following commit has been merged into the smp/core branch of tip:

Commit-ID:     ec9f57e6fefbc0497b9f047ebe60ef9adaae7480
Gitweb:        https://git.kernel.org/tip/ec9f57e6fefbc0497b9f047ebe60ef9adaae7480
Author:        Usama Arif <[email protected]>
AuthorDate:    Mon, 29 Jun 2026 03:57:45 -07:00
Committer:     Thomas Gleixner <[email protected]>
CommitterDate: Mon, 29 Jun 2026 15:53:14 +02:00

smp: Use release stores for csd_lock_record() state

__csd_lock_record() publishes per-CPU diagnostic state (cur_csd, cur_csd_func, cur_csd_info) that is consumed from a remote CPU by csd_lock_wait_toolong() via smp_load_acquire(&cur_csd). To order the matching cur_csd_func/cur_csd_info stores before the cur_csd publication, the producer issues smp_wmb() before writing cur_csd; to order the publication before the subsequent callback execution or CSD unlock, it issues smp_mb() after the write. The clear path mirrors this with smp_mb() before storing NULL into cur_csd so the preceding callback/unlock is observed first.

The smp_mb() pair is heavier than what the consumer actually requires (on x86 each emits a locked full barrier). The consumer only needs to observe the matching cur_csd_func/cur_csd_info when it sees a non-NULL cur_csd, and to observe the preceding callback/unlock when it sees NULL -- both of which a release/acquire pair provides. The extra two-way ordering enforced by smp_mb() -- that cur_csd publication be observed before callback execution or unlock becomes visible -- would only matter if cur_csd were an exact live-state marker. csd_lock_wait_toolong() does not treat it that way: it snapshots cur_csd via smp_load_acquire() and then prints / dumps / re-IPIs without an RCU-style stall-ended recheck, so the diagnostic already tolerates the remote CPU completing its work between snapshot and report. cur_csd is best-effort context, not a precise stall boundary.

Replace the smp_wmb() + plain store + smp_mb() in the publish path, and the smp_mb() + plain store in the clear path, with smp_store_release(). This pairs with the smp_load_acquire() in csd_lock_wait_toolong(): preceding cur_csd_func/cur_csd_info stores become visible before a remote reader observes the non-NULL publication, and any preceding callback/unlock becomes visible before a reader observes the NULL clear.

---
 kernel/smp.c | 26 +++++++++++++++++++-------
 1 file changed, 19 insertions(+), 7 deletions(-)
kernel/smp.c +19 -7
index a0bb56b..8a847a3 100644
@@ -182,16 +182,22 @@ static atomic_t csd_bug_count = ATOMIC_INIT(0);
 static void __csd_lock_record(call_single_data_t *csd)
 {
 	if (!csd) {
-		smp_mb(); /* NULL cur_csd after unlock. */
-		__this_cpu_write(cur_csd, NULL);
+		/*
+		 * Pairs with smp_load_acquire() of cur_csd in
+		 * csd_lock_wait_toolong(): orders any preceding CSD
+		 * callback/unlock before a remote reader observes NULL.
+		 */
+		smp_store_release(this_cpu_ptr(&cur_csd), NULL);
 		return;
 	}
 	__this_cpu_write(cur_csd_func, csd->func);
 	__this_cpu_write(cur_csd_info, csd->info);
-	smp_wmb(); /* func and info before csd. */
-	__this_cpu_write(cur_csd, csd);
-	smp_mb(); /* Update cur_csd before function call. */
-		  /* Or before unlock, as the case may be. */
+	/*
+	 * Pairs with smp_load_acquire() of cur_csd in
+	 * csd_lock_wait_toolong(): publishes cur_csd_func and
+	 * cur_csd_info before the non-NULL pointer becomes visible.
+	 */
+	smp_store_release(this_cpu_ptr(&cur_csd), csd);
 }
 
 static __always_inline void csd_lock_record(call_single_data_t *csd)
@@ -272,7 +278,13 @@ static bool csd_lock_wait_toolong(call_single_data_t *csd, u64 ts0, u64 *ts1, in
 		cpux = 0;
 	else
 		cpux = cpu;
-	cpu_cur_csd = smp_load_acquire(&per_cpu(cur_csd, cpux)); /* Before func and info. */
+	/*
+	 * Pairs with smp_store_release() of cur_csd in __csd_lock_record():
+	 * a non-NULL cur_csd here implies cur_csd_func and cur_csd_info
+	 * are the matching publication; a NULL value is ordered after any
+	 * preceding CSD callback/unlock on the remote CPU.
+	 */
+	cpu_cur_csd = smp_load_acquire(&per_cpu(cur_csd, cpux));
 	/* How long since this CSD lock was stuck. */
 	ts_delta = ts2 - ts0;
 	pr_alert("csd: %s non-responsive CSD lock (#%d) on CPU#%d, waiting %lld ns for CPU#%02d %pS(%ps).\n",
maintainer Reviewed-by on v3 1/1
#4 lore raw
In reply to Usama Arif
Quoted 33 lines · Usama Arif

__csd_lock_record() publishes per-CPU diagnostic state (cur_csd, cur_csd_func, cur_csd_info) that is consumed from a remote CPU by csd_lock_wait_toolong() via smp_load_acquire(&cur_csd). To order the matching cur_csd_func/cur_csd_info stores before the cur_csd publication, the producer issues smp_wmb() before writing cur_csd; to order the publication before the subsequent callback execution or CSD unlock, it issues smp_mb() after the write. The clear path mirrors this with smp_mb() before storing NULL into cur_csd so the preceding callback/unlock is observed first.

The smp_mb() pair is heavier than what the consumer actually requires (on x86 each emits a locked full barrier). The consumer only needs to observe the matching cur_csd_func/cur_csd_info when it sees a non-NULL cur_csd, and to observe the preceding callback/unlock when it sees NULL -- both of which a release/acquire pair provides. The extra two-way ordering enforced by smp_mb() -- that cur_csd publication be observed before callback execution or unlock becomes visible -- would only matter if cur_csd were an exact live-state marker. csd_lock_wait_toolong() does not treat it that way: it snapshots cur_csd via smp_load_acquire() and then prints / dumps / re-IPIs without an RCU-style stall-ended recheck, so the diagnostic already tolerates the remote CPU completing its work between snapshot and report. cur_csd is best-effort context, not a precise stall boundary.

Replace the smp_wmb() + plain store + smp_mb() in the publish path, and the smp_mb() + plain store in the clear path, with smp_store_release(). This pairs with the smp_load_acquire() in csd_lock_wait_toolong(): preceding cur_csd_func/cur_csd_info stores become visible before a remote reader observes the non-NULL publication, and any preceding callback/unlock becomes visible before a reader observes the NULL clear.

Quoted 1 line
Quoted 65 lines

---
v2 -> v3: https://lore.kernel.org/all/[email protected]/
- Restructure changelog into context/problem/solution form (Thomas Gleixner).
- Add reciprocal pairing comment on the smp_load_acquire() in csd_lock_wait_toolong() (Dmitry).

v1 -> v2: https://lore.kernel.org/all/[email protected]/
- Document where the smp_store_release() synchronizes with (Alan
Stern, Randy Dunlap and Paul McKenney).

---
 kernel/smp.c | 26 +++++++++++++++++++-------
 1 file changed, 19 insertions(+), 7 deletions(-)
kernel/smp.c +19 -7
index a0bb56bd8dda..8a847a34f132 100644
@@ -182,16 +182,22 @@ static atomic_t csd_bug_count = ATOMIC_INIT(0);
 static void __csd_lock_record(call_single_data_t *csd)
 {
 	if (!csd) {
-		smp_mb(); /* NULL cur_csd after unlock. */
-		__this_cpu_write(cur_csd, NULL);
+		/*
+		 * Pairs with smp_load_acquire() of cur_csd in
+		 * csd_lock_wait_toolong(): orders any preceding CSD
+		 * callback/unlock before a remote reader observes NULL.
+		 */
+		smp_store_release(this_cpu_ptr(&cur_csd), NULL);
 		return;
 	}
 	__this_cpu_write(cur_csd_func, csd->func);
 	__this_cpu_write(cur_csd_info, csd->info);
-	smp_wmb(); /* func and info before csd. */
-	__this_cpu_write(cur_csd, csd);
-	smp_mb(); /* Update cur_csd before function call. */
-		  /* Or before unlock, as the case may be. */
+	/*
+	 * Pairs with smp_load_acquire() of cur_csd in
+	 * csd_lock_wait_toolong(): publishes cur_csd_func and
+	 * cur_csd_info before the non-NULL pointer becomes visible.
+	 */
+	smp_store_release(this_cpu_ptr(&cur_csd), csd);
 }
 
 static __always_inline void csd_lock_record(call_single_data_t *csd)
@@ -272,7 +278,13 @@ static bool csd_lock_wait_toolong(call_single_data_t *csd, u64 ts0, u64 *ts1, in
 		cpux = 0;
 	else
 		cpux = cpu;
-	cpu_cur_csd = smp_load_acquire(&per_cpu(cur_csd, cpux)); /* Before func and info. */
+	/*
+	 * Pairs with smp_store_release() of cur_csd in __csd_lock_record():
+	 * a non-NULL cur_csd here implies cur_csd_func and cur_csd_info
+	 * are the matching publication; a NULL value is ordered after any
+	 * preceding CSD callback/unlock on the remote CPU.
+	 */
+	cpu_cur_csd = smp_load_acquire(&per_cpu(cur_csd, cpux));
 	/* How long since this CSD lock was stuck. */
 	ts_delta = ts2 - ts0;
 	pr_alert("csd: %s non-responsive CSD lock (#%d) on CPU#%d, waiting %lld ns for CPU#%02d %pS(%ps).\n",
Signature · 1 line
2.53.0-Meta
Reviewed-by on v3 1/1
#5 lore raw
In reply to Usama Arif
Quoted 86 lines · Usama Arif

__csd_lock_record() publishes per-CPU diagnostic state (cur_csd, cur_csd_func, cur_csd_info) that is consumed from a remote CPU by csd_lock_wait_toolong() via smp_load_acquire(&cur_csd). To order the matching cur_csd_func/cur_csd_info stores before the cur_csd publication, the producer issues smp_wmb() before writing cur_csd; to order the publication before the subsequent callback execution or CSD unlock, it issues smp_mb() after the write. The clear path mirrors this with smp_mb() before storing NULL into cur_csd so the preceding callback/unlock is observed first.

The smp_mb() pair is heavier than what the consumer actually requires (on x86 each emits a locked full barrier). The consumer only needs to observe the matching cur_csd_func/cur_csd_info when it sees a non-NULL cur_csd, and to observe the preceding callback/unlock when it sees NULL -- both of which a release/acquire pair provides. The extra two-way ordering enforced by smp_mb() -- that cur_csd publication be observed before callback execution or unlock becomes visible -- would only matter if cur_csd were an exact live-state marker. csd_lock_wait_toolong() does not treat it that way: it snapshots cur_csd via smp_load_acquire() and then prints / dumps / re-IPIs without an RCU-style stall-ended recheck, so the diagnostic already tolerates the remote CPU completing its work between snapshot and report. cur_csd is best-effort context, not a precise stall boundary.

Replace the smp_wmb() + plain store + smp_mb() in the publish path, and the smp_mb() + plain store in the clear path, with smp_store_release(). This pairs with the smp_load_acquire() in csd_lock_wait_toolong(): preceding cur_csd_func/cur_csd_info stores become visible before a remote reader observes the non-NULL publication, and any preceding callback/unlock becomes visible before a reader observes the NULL clear.

---
v2 -> v3: https://lore.kernel.org/all/[email protected]/
- Restructure changelog into context/problem/solution form (Thomas Gleixner).
- Add reciprocal pairing comment on the smp_load_acquire() in csd_lock_wait_toolong() (Dmitry).

v1 -> v2: https://lore.kernel.org/all/[email protected]/
- Document where the smp_store_release() synchronizes with (Alan
Stern, Randy Dunlap and Paul McKenney).

---
 kernel/smp.c | 26 +++++++++++++++++++-------
 1 file changed, 19 insertions(+), 7 deletions(-)
kernel/smp.c +12 -7
index a0bb56bd8dda..8a847a34f132 100644
@@ -182,16 +182,22 @@ static atomic_t csd_bug_count = ATOMIC_INIT(0);
 static void __csd_lock_record(call_single_data_t *csd)
 {
 	if (!csd) {
-		smp_mb(); /* NULL cur_csd after unlock. */
-		__this_cpu_write(cur_csd, NULL);
+		/*
+		 * Pairs with smp_load_acquire() of cur_csd in
+		 * csd_lock_wait_toolong(): orders any preceding CSD
+		 * callback/unlock before a remote reader observes NULL.
+		 */
+		smp_store_release(this_cpu_ptr(&cur_csd), NULL);
 		return;
 	}
 	__this_cpu_write(cur_csd_func, csd->func);
 	__this_cpu_write(cur_csd_info, csd->info);
-	smp_wmb(); /* func and info before csd. */
-	__this_cpu_write(cur_csd, csd);
-	smp_mb(); /* Update cur_csd before function call. */
-		  /* Or before unlock, as the case may be. */
+	/*
+	 * Pairs with smp_load_acquire() of cur_csd in
+	 * csd_lock_wait_toolong(): publishes cur_csd_func and
+	 * cur_csd_info before the non-NULL pointer becomes visible.
+	 */
+	smp_store_release(this_cpu_ptr(&cur_csd), csd);
 }
 
 static __always_inline void csd_lock_record(call_single_data_t *csd)
@@ -272,7 +278,13 @@ static bool csd_lock_wait_toolong(call_single_data_t *csd, u64 ts0, u64 *ts1, in
 		cpux = 0;
 	else
 		cpux = cpu;
-	cpu_cur_csd = smp_load_acquire(&per_cpu(cur_csd, cpux)); /* Before func and info. */

kernel/smp.c static bool csd_lock_wait_toolong(call_single_data_t *csd, u64 ts0, u64 *ts1, in · line 290

Quoted 10 lines
kernel/smp.c +7 -0
+	/*
+	 * Pairs with smp_store_release() of cur_csd in __csd_lock_record():
+	 * a non-NULL cur_csd here implies cur_csd_func and cur_csd_info
+	 * are the matching publication; a NULL value is ordered after any
+	 * preceding CSD callback/unlock on the remote CPU.
+	 */
+	cpu_cur_csd = smp_load_acquire(&per_cpu(cur_csd, cpux));
 	/* How long since this CSD lock was stuck. */
 	ts_delta = ts2 - ts0;
 	pr_alert("csd: %s non-responsive CSD lock (#%d) on CPU#%d, waiting %lld ns for CPU#%02d %pS(%ps).\n",
Signature · 1 line
Thanx, Kunwu
maintainer
#3 lore raw
In reply to tip-bot2 for Usama Arif
Quoted 58 lines · On Mon, Jun 29, 2026 at 01:57:42PM -0000, tip-bot2 for Usama Arif wrote:

The following commit has been merged into the smp/core branch of tip:

Commit-ID:     ec9f57e6fefbc0497b9f047ebe60ef9adaae7480
Gitweb:        https://git.kernel.org/tip/ec9f57e6fefbc0497b9f047ebe60ef9adaae7480
Author:        Usama Arif <[email protected]>
AuthorDate:    Mon, 29 Jun 2026 03:57:45 -07:00
Committer:     Thomas Gleixner <[email protected]>
CommitterDate: Mon, 29 Jun 2026 15:53:14 +02:00

smp: Use release stores for csd_lock_record() state

__csd_lock_record() publishes per-CPU diagnostic state (cur_csd, cur_csd_func, cur_csd_info) that is consumed from a remote CPU by csd_lock_wait_toolong() via smp_load_acquire(&cur_csd). To order the matching cur_csd_func/cur_csd_info stores before the cur_csd publication, the producer issues smp_wmb() before writing cur_csd; to order the publication before the subsequent callback execution or CSD unlock, it issues smp_mb() after the write. The clear path mirrors this with smp_mb() before storing NULL into cur_csd so the preceding callback/unlock is observed first.

The smp_mb() pair is heavier than what the consumer actually requires (on x86 each emits a locked full barrier). The consumer only needs to observe the matching cur_csd_func/cur_csd_info when it sees a non-NULL cur_csd, and to observe the preceding callback/unlock when it sees NULL -- both of which a release/acquire pair provides. The extra two-way ordering enforced by smp_mb() -- that cur_csd publication be observed before callback execution or unlock becomes visible -- would only matter if cur_csd were an exact live-state marker. csd_lock_wait_toolong() does not treat it that way: it snapshots cur_csd via smp_load_acquire() and then prints / dumps / re-IPIs without an RCU-style stall-ended recheck, so the diagnostic already tolerates the remote CPU completing its work between snapshot and report. cur_csd is best-effort context, not a precise stall boundary.

Replace the smp_wmb() + plain store + smp_mb() in the publish path, and the smp_mb() + plain store in the clear path, with smp_store_release(). This pairs with the smp_load_acquire() in csd_lock_wait_toolong(): preceding cur_csd_func/cur_csd_info stores become visible before a remote reader observes the non-NULL publication, and any preceding callback/unlock becomes visible before a reader observes the NULL clear.

---
 kernel/smp.c | 26 +++++++++++++++++++-------
 1 file changed, 19 insertions(+), 7 deletions(-)
kernel/smp.c +0 -0
index a0bb56b..8a847a3 100644
@@ -182,16 +182,22 @@ static atomic_t csd_bug_count = ATOMIC_INIT(0);
 static void __csd_lock_record(call_single_data_t *csd)

kernel/smp.c static atomic_t csd_bug_count = ATOMIC_INIT(0); · line 190

Quoted 10 lines
kernel/smp.c +6 -2
 {
 	if (!csd) {
-		smp_mb(); /* NULL cur_csd after unlock. */
-		__this_cpu_write(cur_csd, NULL);
+		/*
+		 * Pairs with smp_load_acquire() of cur_csd in
+		 * csd_lock_wait_toolong(): orders any preceding CSD
+		 * callback/unlock before a remote reader observes NULL.
+		 */
+		smp_store_release(this_cpu_ptr(&cur_csd), NULL);

So at some point I suppose we should have *this_cpu_store_release() I suppose. As is this generates suboptimal code on x86. But since I don't typically run with CSD_LOCK_WAIT_DEBUG=y, I'll pretend not to care at this exact moment.

Quoted 4 lines
kernel/smp.c +0 -0
 		return;
 	}
 	__this_cpu_write(cur_csd_func, csd->func);
 	__this_cpu_write(cur_csd_info, csd->info);

kernel/smp.c static atomic_t csd_bug_count = ATOMIC_INIT(0); · line 200

Quoted 10 lines
kernel/smp.c +6 -4
-	smp_wmb(); /* func and info before csd. */
-	__this_cpu_write(cur_csd, csd);
-	smp_mb(); /* Update cur_csd before function call. */
-		  /* Or before unlock, as the case may be. */
+	/*
+	 * Pairs with smp_load_acquire() of cur_csd in
+	 * csd_lock_wait_toolong(): publishes cur_csd_func and
+	 * cur_csd_info before the non-NULL pointer becomes visible.
+	 */
+	smp_store_release(this_cpu_ptr(&cur_csd), csd);

Let me say that RELEASE+ACQUIRE does not a full barrier make, specifically smp_store_release() and smp_load_acquire() do not have RCsc, they're RCtso or possibly weaker still (I don't remember). The thing that saves your bacon here is the address dependency.

Quoted 17 lines
kernel/smp.c +7 -1
 }
 static __always_inline void csd_lock_record(call_single_data_t *csd)
@@ -272,7 +278,13 @@ static bool csd_lock_wait_toolong(call_single_data_t *csd, u64 ts0, u64 *ts1, in
 		cpux = 0;
 	else
 		cpux = cpu;
-	cpu_cur_csd = smp_load_acquire(&per_cpu(cur_csd, cpux)); /* Before func and info. */
+	/*
+	 * Pairs with smp_store_release() of cur_csd in __csd_lock_record():
+	 * a non-NULL cur_csd here implies cur_csd_func and cur_csd_info
+	 * are the matching publication; a NULL value is ordered after any
+	 * preceding CSD callback/unlock on the remote CPU.
+	 */
+	cpu_cur_csd = smp_load_acquire(&per_cpu(cur_csd, cpux));
 	/* How long since this CSD lock was stuck. */
 	ts_delta = ts2 - ts0;
 	pr_alert("csd: %s non-responsive CSD lock (#%d) on CPU#%d, waiting %lld ns for CPU#%02d %pS(%ps).\n",