smp: Use release stores for csd_lock_record() state
Subsystems (2)
PATCH v3 1/1 smp: Use release stores for csd_lock_record() state
__csd_lock_record() publishes per-CPU diagnostic state (cur_csd, cur_csd_func, cur_csd_info) that is consumed from a remote CPU by csd_lock_wait_toolong() via smp_load_acquire(&cur_csd). To order the matching cur_csd_func/cur_csd_info stores before the cur_csd publication, the producer issues smp_wmb() before writing cur_csd; to order the publication before the subsequent callback execution or CSD unlock, it issues smp_mb() after the write. The clear path mirrors this with smp_mb() before storing NULL into cur_csd so the preceding callback/unlock is observed first.
The smp_mb() pair is heavier than what the consumer actually requires (on x86 each emits a locked full barrier). The consumer only needs to observe the matching cur_csd_func/cur_csd_info when it sees a non-NULL cur_csd, and to observe the preceding callback/unlock when it sees NULL -- both of which a release/acquire pair provides. The extra two-way ordering enforced by smp_mb() -- that cur_csd publication be observed before callback execution or unlock becomes visible -- would only matter if cur_csd were an exact live-state marker. csd_lock_wait_toolong() does not treat it that way: it snapshots cur_csd via smp_load_acquire() and then prints / dumps / re-IPIs without an RCU-style stall-ended recheck, so the diagnostic already tolerates the remote CPU completing its work between snapshot and report. cur_csd is best-effort context, not a precise stall boundary.
Replace the smp_wmb() + plain store + smp_mb() in the publish path, and the smp_mb() + plain store in the clear path, with smp_store_release(). This pairs with the smp_load_acquire() in csd_lock_wait_toolong(): preceding cur_csd_func/cur_csd_info stores become visible before a remote reader observes the non-NULL publication, and any preceding callback/unlock becomes visible before a reader observes the NULL clear.
---
v2 -> v3: https://lore.kernel.org/all/[email protected]/
- Restructure changelog into context/problem/solution form (Thomas Gleixner).
- Add reciprocal pairing comment on the smp_load_acquire() in csd_lock_wait_toolong() (Dmitry).
v1 -> v2: https://lore.kernel.org/all/[email protected]/
- Document where the smp_store_release() synchronizes with (Alan
Stern, Randy Dunlap and Paul McKenney).
--- kernel/smp.c | 26 +++++++++++++++++++------- 1 file changed, 19 insertions(+), 7 deletions(-)
kernel/smp.c +19 -7
@@ -182,16 +182,22 @@ static atomic_t csd_bug_count = ATOMIC_INIT(0);static void __csd_lock_record (call_single_data_t *csd) {if (!csd) { -smp_mb ();/* NULL cur_csd after unlock. */ -__this_cpu_write (cur_csd,NULL ); +/* +* Pairs with smp_load_acquire() of cur_csd in +* csd_lock_wait_toolong(): orders any preceding CSD +* callback/unlock before a remote reader observes NULL. +*/ +smp_store_release (this_cpu_ptr (&cur_csd),NULL );return ; }__this_cpu_write (cur_csd_func, csd->func);__this_cpu_write (cur_csd_info, csd->info); -smp_wmb ();/* func and info before csd. */ -__this_cpu_write (cur_csd, csd); -smp_mb ();/* Update cur_csd before function call. */ -/* Or before unlock, as the case may be. */ +/* +* Pairs with smp_load_acquire() of cur_csd in +* csd_lock_wait_toolong(): publishes cur_csd_func and +* cur_csd_info before the non-NULL pointer becomes visible. +*/ +smp_store_release (this_cpu_ptr (&cur_csd), csd); } static __always_inline void csd_lock_record(call_single_data_t *csd)
@@ -272,7 +278,13 @@ static bool csd_lock_wait_toolong(call_single_data_t *csd, u64 ts0, u64 *ts1, in cpux =0 ; else cpux = cpu; - cpu_cur_csd =smp_load_acquire (&per_cpu (cur_csd, cpux));/* Before func and info. */ +/* +* Pairs with smp_store_release() of cur_csd in __csd_lock_record(): +* a non-NULL cur_csd here implies cur_csd_func and cur_csd_info +* are the matching publication; a NULL value is ordered after any +* preceding CSD callback/unlock on the remote CPU. +*/ + cpu_cur_csd =smp_load_acquire (&per_cpu (cur_csd, cpux));/* How long since this CSD lock was stuck. */ ts_delta = ts2 - ts0; pr_alert("csd: %s non-responsive CSD lock (#%d) on CPU#%d, waiting %lld ns for CPU#%02d %pS(%ps).\n",
Signature · 1 line
2.53.0-Meta
Quoted 33 lines · Usama Arif
__csd_lock_record()publishes per-CPU diagnostic state (cur_csd,cur_csd_func,cur_csd_info) that is consumed from a remote CPU bycsd_lock_wait_toolong()viasmp_load_acquire(&cur_csd). To order the matching cur_csd_func/cur_csd_info stores before thecur_csdpublication, the producer issuessmp_wmb()before writingcur_csd; to order the publication before the subsequent callback execution or CSD unlock, it issuessmp_mb()after the write. The clear path mirrors this withsmp_mb()before storing NULL intocur_csdso the preceding callback/unlock is observed first.The
smp_mb()pair is heavier than what the consumer actually requires (on x86 each emits a locked full barrier). The consumer only needs to observe the matching cur_csd_func/cur_csd_info when it sees a non-NULLcur_csd, and to observe the preceding callback/unlock when it sees NULL -- both of which a release/acquire pair provides. The extra two-way ordering enforced bysmp_mb()-- thatcur_csdpublication be observed before callback execution or unlock becomes visible -- would only matter ifcur_csdwere an exact live-state marker.csd_lock_wait_toolong()does not treat it that way: it snapshotscur_csdviasmp_load_acquire()and then prints / dumps / re-IPIs without an RCU-style stall-ended recheck, so the diagnostic already tolerates the remote CPU completing its work between snapshot and report.cur_csdis best-effort context, not a precise stall boundary.Replace the
smp_wmb()+ plain store +smp_mb()in the publish path, and thesmp_mb()+ plain store in the clear path, withsmp_store_release(). This pairs with thesmp_load_acquire()incsd_lock_wait_toolong(): preceding cur_csd_func/cur_csd_info stores become visible before a remote reader observes the non-NULL publication, and any preceding callback/unlock becomes visible before a reader observes the NULL clear.
Quoted 1 line
Signed-off-by Usama Arif <[email protected]>
The following commit has been merged into the smp/core branch of tip:
The following commit has been merged into the smp/core branch of tip:
Commit-ID: ec9f57e6fefbc0497b9f047ebe60ef9adaae7480 Gitweb: https://git.kernel.org/tip/ec9f57e6fefbc0497b9f047ebe60ef9adaae7480 Author: Usama Arif <[email protected]> AuthorDate: Mon, 29 Jun 2026 03:57:45 -07:00 Committer: Thomas Gleixner <[email protected]> CommitterDate: Mon, 29 Jun 2026 15:53:14 +02:00
smp: Use release stores for csd_lock_record() state
__csd_lock_record() publishes per-CPU diagnostic state (cur_csd, cur_csd_func, cur_csd_info) that is consumed from a remote CPU by csd_lock_wait_toolong() via smp_load_acquire(&cur_csd). To order the matching cur_csd_func/cur_csd_info stores before the cur_csd publication, the producer issues smp_wmb() before writing cur_csd; to order the publication before the subsequent callback execution or CSD unlock, it issues smp_mb() after the write. The clear path mirrors this with smp_mb() before storing NULL into cur_csd so the preceding callback/unlock is observed first.
The smp_mb() pair is heavier than what the consumer actually requires (on x86 each emits a locked full barrier). The consumer only needs to observe the matching cur_csd_func/cur_csd_info when it sees a non-NULL cur_csd, and to observe the preceding callback/unlock when it sees NULL -- both of which a release/acquire pair provides. The extra two-way ordering enforced by smp_mb() -- that cur_csd publication be observed before callback execution or unlock becomes visible -- would only matter if cur_csd were an exact live-state marker. csd_lock_wait_toolong() does not treat it that way: it snapshots cur_csd via smp_load_acquire() and then prints / dumps / re-IPIs without an RCU-style stall-ended recheck, so the diagnostic already tolerates the remote CPU completing its work between snapshot and report. cur_csd is best-effort context, not a precise stall boundary.
Replace the smp_wmb() + plain store + smp_mb() in the publish path, and the smp_mb() + plain store in the clear path, with smp_store_release(). This pairs with the smp_load_acquire() in csd_lock_wait_toolong(): preceding cur_csd_func/cur_csd_info stores become visible before a remote reader observes the non-NULL publication, and any preceding callback/unlock becomes visible before a reader observes the NULL clear.
--- kernel/smp.c | 26 +++++++++++++++++++------- 1 file changed, 19 insertions(+), 7 deletions(-)
kernel/smp.c +19 -7
@@ -182,16 +182,22 @@ static atomic_t csd_bug_count = ATOMIC_INIT(0);static void __csd_lock_record (call_single_data_t *csd) {if (!csd) { -smp_mb ();/* NULL cur_csd after unlock. */ -__this_cpu_write (cur_csd,NULL ); +/* +* Pairs with smp_load_acquire() of cur_csd in +* csd_lock_wait_toolong(): orders any preceding CSD +* callback/unlock before a remote reader observes NULL. +*/ +smp_store_release (this_cpu_ptr (&cur_csd),NULL );return ; }__this_cpu_write (cur_csd_func, csd->func);__this_cpu_write (cur_csd_info, csd->info); -smp_wmb ();/* func and info before csd. */ -__this_cpu_write (cur_csd, csd); -smp_mb ();/* Update cur_csd before function call. */ -/* Or before unlock, as the case may be. */ +/* +* Pairs with smp_load_acquire() of cur_csd in +* csd_lock_wait_toolong(): publishes cur_csd_func and +* cur_csd_info before the non-NULL pointer becomes visible. +*/ +smp_store_release (this_cpu_ptr (&cur_csd), csd); } static __always_inline void csd_lock_record(call_single_data_t *csd)
@@ -272,7 +278,13 @@ static bool csd_lock_wait_toolong(call_single_data_t *csd, u64 ts0, u64 *ts1, in cpux =0 ; else cpux = cpu; - cpu_cur_csd =smp_load_acquire (&per_cpu (cur_csd, cpux));/* Before func and info. */ +/* +* Pairs with smp_store_release() of cur_csd in __csd_lock_record(): +* a non-NULL cur_csd here implies cur_csd_func and cur_csd_info +* are the matching publication; a NULL value is ordered after any +* preceding CSD callback/unlock on the remote CPU. +*/ + cpu_cur_csd =smp_load_acquire (&per_cpu (cur_csd, cpux));/* How long since this CSD lock was stuck. */ ts_delta = ts2 - ts0; pr_alert("csd: %s non-responsive CSD lock (#%d) on CPU#%d, waiting %lld ns for CPU#%02d %pS(%ps).\n",
Quoted 33 lines · Usama Arif
__csd_lock_record()publishes per-CPU diagnostic state (cur_csd,cur_csd_func,cur_csd_info) that is consumed from a remote CPU bycsd_lock_wait_toolong()viasmp_load_acquire(&cur_csd). To order the matching cur_csd_func/cur_csd_info stores before thecur_csdpublication, the producer issuessmp_wmb()before writingcur_csd; to order the publication before the subsequent callback execution or CSD unlock, it issuessmp_mb()after the write. The clear path mirrors this withsmp_mb()before storing NULL intocur_csdso the preceding callback/unlock is observed first.The
smp_mb()pair is heavier than what the consumer actually requires (on x86 each emits a locked full barrier). The consumer only needs to observe the matching cur_csd_func/cur_csd_info when it sees a non-NULLcur_csd, and to observe the preceding callback/unlock when it sees NULL -- both of which a release/acquire pair provides. The extra two-way ordering enforced bysmp_mb()-- thatcur_csdpublication be observed before callback execution or unlock becomes visible -- would only matter ifcur_csdwere an exact live-state marker.csd_lock_wait_toolong()does not treat it that way: it snapshotscur_csdviasmp_load_acquire()and then prints / dumps / re-IPIs without an RCU-style stall-ended recheck, so the diagnostic already tolerates the remote CPU completing its work between snapshot and report.cur_csdis best-effort context, not a precise stall boundary.Replace the
smp_wmb()+ plain store +smp_mb()in the publish path, and thesmp_mb()+ plain store in the clear path, withsmp_store_release(). This pairs with thesmp_load_acquire()incsd_lock_wait_toolong(): preceding cur_csd_func/cur_csd_info stores become visible before a remote reader observes the non-NULL publication, and any preceding callback/unlock becomes visible before a reader observes the NULL clear.
Quoted 1 line
Signed-off-by Usama Arif <[email protected]>
Quoted 65 lines
---
v2 -> v3: https://lore.kernel.org/all/[email protected]/
- Restructure changelog into context/problem/solution form (Thomas Gleixner).
- Add reciprocal pairing comment on thesmp_load_acquire()incsd_lock_wait_toolong()(Dmitry).v1 -> v2: https://lore.kernel.org/all/[email protected]/
- Document where thesmp_store_release()synchronizes with (Alan
Stern, Randy Dunlap and Paul McKenney).--- kernel/smp.c | 26 +++++++++++++++++++------- 1 file changed, 19 insertions(+), 7 deletions(-)kernel/smp.c +19
-7@@ -182,16 +182,22 @@ static atomic_t csd_bug_count = ATOMIC_INIT(0);static void __csd_lock_record (call_single_data_t *csd) {if (!csd) { -smp_mb ();/* NULL cur_csd after unlock. */ -__this_cpu_write (cur_csd,NULL ); +/* +* Pairs with smp_load_acquire() of cur_csd in +* csd_lock_wait_toolong(): orders any preceding CSD +* callback/unlock before a remote reader observes NULL. +*/ +smp_store_release (this_cpu_ptr (&cur_csd),NULL );return ; }__this_cpu_write (cur_csd_func, csd->func);__this_cpu_write (cur_csd_info, csd->info); -smp_wmb ();/* func and info before csd. */ -__this_cpu_write (cur_csd, csd); -smp_mb ();/* Update cur_csd before function call. */ -/* Or before unlock, as the case may be. */ +/* +* Pairs with smp_load_acquire() of cur_csd in +* csd_lock_wait_toolong(): publishes cur_csd_func and +* cur_csd_info before the non-NULL pointer becomes visible. +*/ +smp_store_release (this_cpu_ptr (&cur_csd), csd); } static __always_inline void csd_lock_record(call_single_data_t *csd)@@ -272,7 +278,13 @@ static bool csd_lock_wait_toolong(call_single_data_t *csd, u64 ts0, u64 *ts1, in cpux =0 ; else cpux = cpu; - cpu_cur_csd =smp_load_acquire (&per_cpu (cur_csd, cpux));/* Before func and info. */ +/* +* Pairs with smp_store_release() of cur_csd in __csd_lock_record(): +* a non-NULL cur_csd here implies cur_csd_func and cur_csd_info +* are the matching publication; a NULL value is ordered after any +* preceding CSD callback/unlock on the remote CPU. +*/ + cpu_cur_csd =smp_load_acquire (&per_cpu (cur_csd, cpux));/* How long since this CSD lock was stuck. */ ts_delta = ts2 - ts0; pr_alert("csd: %s non-responsive CSD lock (#%d) on CPU#%d, waiting %lld ns for CPU#%02d %pS(%ps).\n",Signature · 1 line
2.53.0-Meta
Quoted 86 lines · Usama Arif
__csd_lock_record()publishes per-CPU diagnostic state (cur_csd,cur_csd_func,cur_csd_info) that is consumed from a remote CPU bycsd_lock_wait_toolong()viasmp_load_acquire(&cur_csd). To order the matching cur_csd_func/cur_csd_info stores before thecur_csdpublication, the producer issuessmp_wmb()before writingcur_csd; to order the publication before the subsequent callback execution or CSD unlock, it issuessmp_mb()after the write. The clear path mirrors this withsmp_mb()before storing NULL intocur_csdso the preceding callback/unlock is observed first.The
smp_mb()pair is heavier than what the consumer actually requires (on x86 each emits a locked full barrier). The consumer only needs to observe the matching cur_csd_func/cur_csd_info when it sees a non-NULLcur_csd, and to observe the preceding callback/unlock when it sees NULL -- both of which a release/acquire pair provides. The extra two-way ordering enforced bysmp_mb()-- thatcur_csdpublication be observed before callback execution or unlock becomes visible -- would only matter ifcur_csdwere an exact live-state marker.csd_lock_wait_toolong()does not treat it that way: it snapshotscur_csdviasmp_load_acquire()and then prints / dumps / re-IPIs without an RCU-style stall-ended recheck, so the diagnostic already tolerates the remote CPU completing its work between snapshot and report.cur_csdis best-effort context, not a precise stall boundary.Replace the
smp_wmb()+ plain store +smp_mb()in the publish path, and thesmp_mb()+ plain store in the clear path, withsmp_store_release(). This pairs with thesmp_load_acquire()incsd_lock_wait_toolong(): preceding cur_csd_func/cur_csd_info stores become visible before a remote reader observes the non-NULL publication, and any preceding callback/unlock becomes visible before a reader observes the NULL clear.Signed-off-by Usama Arif <[email protected]>---
v2 -> v3: https://lore.kernel.org/all/[email protected]/
- Restructure changelog into context/problem/solution form (Thomas Gleixner).
- Add reciprocal pairing comment on thesmp_load_acquire()incsd_lock_wait_toolong()(Dmitry).v1 -> v2: https://lore.kernel.org/all/[email protected]/
- Document where thesmp_store_release()synchronizes with (Alan
Stern, Randy Dunlap and Paul McKenney).--- kernel/smp.c | 26 +++++++++++++++++++------- 1 file changed, 19 insertions(+), 7 deletions(-)kernel/smp.c +12
-7@@ -182,16 +182,22 @@ static atomic_t csd_bug_count = ATOMIC_INIT(0);static void __csd_lock_record (call_single_data_t *csd) {if (!csd) { -smp_mb ();/* NULL cur_csd after unlock. */ -__this_cpu_write (cur_csd,NULL ); +/* +* Pairs with smp_load_acquire() of cur_csd in +* csd_lock_wait_toolong(): orders any preceding CSD +* callback/unlock before a remote reader observes NULL. +*/ +smp_store_release (this_cpu_ptr (&cur_csd),NULL );return ; }__this_cpu_write (cur_csd_func, csd->func);__this_cpu_write (cur_csd_info, csd->info); -smp_wmb ();/* func and info before csd. */ -__this_cpu_write (cur_csd, csd); -smp_mb ();/* Update cur_csd before function call. */ -/* Or before unlock, as the case may be. */ +/* +* Pairs with smp_load_acquire() of cur_csd in +* csd_lock_wait_toolong(): publishes cur_csd_func and +* cur_csd_info before the non-NULL pointer becomes visible. +*/ +smp_store_release (this_cpu_ptr (&cur_csd), csd); } static __always_inline void csd_lock_record(call_single_data_t *csd)@@ -272,7 +278,13 @@ static bool csd_lock_wait_toolong(call_single_data_t *csd, u64 ts0, u64 *ts1, in cpux =0 ; else cpux = cpu; - cpu_cur_csd =smp_load_acquire (&per_cpu (cur_csd, cpux));/* Before func and info. */
kernel/smp.c static bool csd_lock_wait_toolong(call_single_data_t *csd, u64 ts0, u64 *ts1, in · line 290
Quoted 10 lines
kernel/smp.c +7
-0+/* +* Pairs with smp_store_release() of cur_csd in __csd_lock_record(): +* a non-NULL cur_csd here implies cur_csd_func and cur_csd_info +* are the matching publication; a NULL value is ordered after any +* preceding CSD callback/unlock on the remote CPU. +*/ + cpu_cur_csd =smp_load_acquire (&per_cpu (cur_csd, cpux));/* How long since this CSD lock was stuck. */ ts_delta = ts2 - ts0; pr_alert("csd: %s non-responsive CSD lock (#%d) on CPU#%d, waiting %lld ns for CPU#%02d %pS(%ps).\n",
Signature · 1 line
Thanx, Kunwu
Peter Zijlstra
maintainerQuoted 58 lines · On Mon, Jun 29, 2026 at 01:57:42PM -0000, tip-bot2 for Usama Arif wrote:
The following commit has been merged into the smp/core branch of tip:
Commit-ID: ec9f57e6fefbc0497b9f047ebe60ef9adaae7480 Gitweb: https://git.kernel.org/tip/ec9f57e6fefbc0497b9f047ebe60ef9adaae7480 Author: Usama Arif <[email protected]> AuthorDate: Mon, 29 Jun 2026 03:57:45 -07:00 Committer: Thomas Gleixner <[email protected]> CommitterDate: Mon, 29 Jun 2026 15:53:14 +02:00smp: Use release stores for
csd_lock_record()state
__csd_lock_record()publishes per-CPU diagnostic state (cur_csd,cur_csd_func,cur_csd_info) that is consumed from a remote CPU bycsd_lock_wait_toolong()viasmp_load_acquire(&cur_csd). To order the matching cur_csd_func/cur_csd_info stores before thecur_csdpublication, the producer issuessmp_wmb()before writingcur_csd; to order the publication before the subsequent callback execution or CSD unlock, it issuessmp_mb()after the write. The clear path mirrors this withsmp_mb()before storing NULL intocur_csdso the preceding callback/unlock is observed first.The
smp_mb()pair is heavier than what the consumer actually requires (on x86 each emits a locked full barrier). The consumer only needs to observe the matching cur_csd_func/cur_csd_info when it sees a non-NULLcur_csd, and to observe the preceding callback/unlock when it sees NULL -- both of which a release/acquire pair provides. The extra two-way ordering enforced bysmp_mb()-- thatcur_csdpublication be observed before callback execution or unlock becomes visible -- would only matter ifcur_csdwere an exact live-state marker.csd_lock_wait_toolong()does not treat it that way: it snapshotscur_csdviasmp_load_acquire()and then prints / dumps / re-IPIs without an RCU-style stall-ended recheck, so the diagnostic already tolerates the remote CPU completing its work between snapshot and report.cur_csdis best-effort context, not a precise stall boundary.Replace the
smp_wmb()+ plain store +smp_mb()in the publish path, and thesmp_mb()+ plain store in the clear path, withsmp_store_release(). This pairs with thesmp_load_acquire()incsd_lock_wait_toolong(): preceding cur_csd_func/cur_csd_info stores become visible before a remote reader observes the non-NULL publication, and any preceding callback/unlock becomes visible before a reader observes the NULL clear.Signed-off-by Usama Arif <[email protected]>Signed-off-by Thomas Gleixner <[email protected]>Reviewed-by Dmitry Ilvokhin <[email protected]>--- kernel/smp.c | 26 +++++++++++++++++++------- 1 file changed, 19 insertions(+), 7 deletions(-)kernel/smp.c +0
-0@@ -182,16 +182,22 @@ static atomic_t csd_bug_count = ATOMIC_INIT(0);static void __csd_lock_record (call_single_data_t *csd)
kernel/smp.c static atomic_t csd_bug_count = ATOMIC_INIT(0); · line 190
Quoted 10 lines
kernel/smp.c +6
-2{if (!csd) { -smp_mb ();/* NULL cur_csd after unlock. */ -__this_cpu_write (cur_csd,NULL ); +/* +* Pairs with smp_load_acquire() of cur_csd in +* csd_lock_wait_toolong(): orders any preceding CSD +* callback/unlock before a remote reader observes NULL. +*/ +smp_store_release (this_cpu_ptr (&cur_csd),NULL );
So at some point I suppose we should have *this_cpu_store_release() I suppose. As is this generates suboptimal code on x86. But since I don't typically run with CSD_LOCK_WAIT_DEBUG=y, I'll pretend not to care at this exact moment.
Quoted 4 lines
kernel/smp.c +0
-0return ; }__this_cpu_write (cur_csd_func, csd->func);__this_cpu_write (cur_csd_info, csd->info);
kernel/smp.c static atomic_t csd_bug_count = ATOMIC_INIT(0); · line 200
Quoted 10 lines
kernel/smp.c +6
-4-smp_wmb ();/* func and info before csd. */ -__this_cpu_write (cur_csd, csd); -smp_mb ();/* Update cur_csd before function call. */ -/* Or before unlock, as the case may be. */ +/* +* Pairs with smp_load_acquire() of cur_csd in +* csd_lock_wait_toolong(): publishes cur_csd_func and +* cur_csd_info before the non-NULL pointer becomes visible. +*/ +smp_store_release (this_cpu_ptr (&cur_csd), csd);
Let me say that RELEASE+ACQUIRE does not a full barrier make, specifically smp_store_release() and smp_load_acquire() do not have RCsc, they're RCtso or possibly weaker still (I don't remember). The thing that saves your bacon here is the address dependency.
Quoted 17 lines
kernel/smp.c +7
-1} static __always_inline void csd_lock_record(call_single_data_t *csd)@@ -272,7 +278,13 @@ static bool csd_lock_wait_toolong(call_single_data_t *csd, u64 ts0, u64 *ts1, in cpux =0 ; else cpux = cpu; - cpu_cur_csd =smp_load_acquire (&per_cpu (cur_csd, cpux));/* Before func and info. */ +/* +* Pairs with smp_store_release() of cur_csd in __csd_lock_record(): +* a non-NULL cur_csd here implies cur_csd_func and cur_csd_info +* are the matching publication; a NULL value is ordered after any +* preceding CSD callback/unlock on the remote CPU. +*/ + cpu_cur_csd =smp_load_acquire (&per_cpu (cur_csd, cpux));/* How long since this CSD lock was stuck. */ ts_delta = ts2 - ts0; pr_alert("csd: %s non-responsive CSD lock (#%d) on CPU#%d, waiting %lld ns for CPU#%02d %pS(%ps).\n",