x86/mm/pat: don't gate cpa_lock on debug_pagealloc_enabled()
PATCH 1/1 x86/mm/pat: don't gate cpa_lock on debug_pagealloc_enabled()
From: "Mike Rapoport (Microsoft)" <[email protected]>
Dave Hansen says:
My only question is *why*!?!? Why add extra locking complexity and rules to optimize debug_pagealloc, which is already horrendously slow.
Stop gating cpa_lock on debug_pagealloc_enabled() to simplify the code.
--- arch/x86/mm/pat/set_memory.c | 19 +++++++------------ 1 file changed, 7 insertions(+), 12 deletions(-)
arch/x86/mm/pat/set_memory.c +7 -12
@@ -62,10 +62,9 @@ enum cpa_warn {static const int cpa_warn_level =CPA_PROTECT ;/* -* Serialize cpa() (for !DEBUG_PAGEALLOC which uses large identity mappings) -* using cpa_lock. So that we don't allow any other cpu, with stale large tlb -* entries change the page attribute in parallel to some other cpu -* splitting a large page entry along with changing the attribute. +* Serialize cpa() using cpa_lock so that we don't allow any other cpu, with +* stale large tlb entries, to change the page attribute in parallel to some +* other cpu splitting a large page entry along with changing the attribute. */ static DEFINE_SPINLOCK (cpa_lock);
@@ -1235,11 +1234,9 @@ static int split_large_page(struct cpa_data *cpa, pte_t *kpte, {struct ptdesc *ptdesc; -if (!debug_pagealloc_enabled ()) -spin_unlock (&cpa_lock); +spin_unlock (&cpa_lock); ptdesc =pagetable_alloc (GFP_KERNEL ,0 ); -if (!debug_pagealloc_enabled ()) -spin_lock (&cpa_lock); +spin_lock (&cpa_lock);if (!ptdesc)return -ENOMEM ;
@@ -2023,11 +2020,9 @@ static int __change_page_attr_set_clr(struct cpa_data *cpa, int primary)if (cpa->flags & (CPA_ARRAY |CPA_PAGES_ARRAY )) cpa->numpages =1 ; -if (!debug_pagealloc_enabled ()) -spin_lock (&cpa_lock); +spin_lock (&cpa_lock); ret =__change_page_attr (cpa, primary); -if (!debug_pagealloc_enabled ()) -spin_unlock (&cpa_lock); +spin_unlock (&cpa_lock);if (ret) goto out;
base-commit: a13c140cc289c0b7b3770bce5b3ad42ab35074aa
Signature · 1 line
2.53.0
Quoted 7 lines · Mike Rapoport
From: "Mike Rapoport (Microsoft)" <[email protected]>
Dave Hansen says:
My only question is *why*!?!? Why add extra locking complexity and rules to optimizedebug_pagealloc, which is already horrendously slow.Stop gating
cpa_lockondebug_pagealloc_enabled()to simplify the code.
Thanks for sending this, Mike! I munged the changelog a bit and applied it.
[tip: x86/mm] x86/mm/pat: Don't gate cpa_lock on debug_pagealloc_enabled()
In reply to Mike RapoportThe following commit has been merged into the x86/mm branch of tip:
The following commit has been merged into the x86/mm branch of tip:
Commit-ID: 5fce67641a3ed9a0782eaa228ddece526461a367 Gitweb: https://git.kernel.org/tip/5fce67641a3ed9a0782eaa228ddece526461a367 Author: Mike Rapoport (Microsoft) <[email protected]> AuthorDate: Wed, 15 Jul 2026 17:45:19 +03:00 Committer: Dave Hansen <[email protected]> CommitterDate: Wed, 15 Jul 2026 08:00:49 -07:00
x86/mm/pat: Don't gate cpa_lock on debug_pagealloc_enabled()
The splitting and merging of kernel page table mappings between small and large is protected by cpa_lock. The merging is relatively new but the splitting is ancient.
The splitting has a locking optimization: since DEBUG_PAGEALLOC forces all mappings to 4k, there are no large pages to split. So the code that *might* cause a split can just skip the locking (and a few other things).
This is entertaining, but it adds complexity and makes for weird locking rules. Plus it's all for a debugging feature which makes the kernel super slow in the first place. Optimizing something which is already super slow and not used in production is not the best way to spend our complexity budget.
Stop gating cpa_lock on debug_pagealloc_enabled() to simplify the code and the locking rules.
[ dhansen: flesh out changelog ]
--- arch/x86/mm/pat/set_memory.c | 19 +++++++------------ 1 file changed, 7 insertions(+), 12 deletions(-)
arch/x86/mm/pat/set_memory.c +7 -12
@@ -62,10 +62,9 @@ enum cpa_warn {static const int cpa_warn_level =CPA_PROTECT ;/* -* Serialize cpa() (for !DEBUG_PAGEALLOC which uses large identity mappings) -* using cpa_lock. So that we don't allow any other cpu, with stale large tlb -* entries change the page attribute in parallel to some other cpu -* splitting a large page entry along with changing the attribute. +* Serialize cpa() using cpa_lock so that we don't allow any other cpu, with +* stale large tlb entries, to change the page attribute in parallel to some +* other cpu splitting a large page entry along with changing the attribute. */ static DEFINE_SPINLOCK (cpa_lock);
@@ -1234,11 +1233,9 @@ static int split_large_page(struct cpa_data *cpa, pte_t *kpte, {struct ptdesc *ptdesc; -if (!debug_pagealloc_enabled ()) -spin_unlock (&cpa_lock); +spin_unlock (&cpa_lock); ptdesc =pagetable_alloc (GFP_KERNEL ,0 ); -if (!debug_pagealloc_enabled ()) -spin_lock (&cpa_lock); +spin_lock (&cpa_lock);if (!ptdesc)return -ENOMEM ;
@@ -2022,11 +2019,9 @@ static int __change_page_attr_set_clr(struct cpa_data *cpa, int primary)if (cpa->flags & (CPA_ARRAY |CPA_PAGES_ARRAY )) cpa->numpages =1 ; -if (!debug_pagealloc_enabled ()) -spin_lock (&cpa_lock); +spin_lock (&cpa_lock); ret =__change_page_attr (cpa, primary); -if (!debug_pagealloc_enabled ()) -spin_unlock (&cpa_lock); +spin_unlock (&cpa_lock);if (ret) goto out;
Quoted 33 lines · Mike Rapoport
From: "Mike Rapoport (Microsoft)" <[email protected]>
Dave Hansen says:
My only question is *why*!?!? Why add extra locking complexity and rules to optimizedebug_pagealloc, which is already horrendously slow.Stop gating
cpa_lockondebug_pagealloc_enabled()to simplify the code.Suggested-by Dave Hansen <[email protected]>Signed-off-by Mike Rapoport (Microsoft) <[email protected]>--- arch/x86/mm/pat/set_memory.c | 19 +++++++------------ 1 file changed, 7 insertions(+), 12 deletions(-)arch/x86/mm/pat/set_memory.c +3
-4@@ -62,10 +62,9 @@ enum cpa_warn {static const int cpa_warn_level =CPA_PROTECT ;/* -* Serialize cpa() (for !DEBUG_PAGEALLOC which uses large identity mappings) -* using cpa_lock. So that we don't allow any other cpu, with stale large tlb -* entries change the page attribute in parallel to some other cpu -* splitting a large page entry along with changing the attribute. +* Serialize cpa() using cpa_lock so that we don't allow any other cpu, with +* stale large tlb entries, to change the page attribute in parallel to some +* other cpu splitting a large page entry along with changing the attribute. */ static DEFINE_SPINLOCK (cpa_lock);
arch/x86/mm/pat/set_memory.c static int split_large_page(struct cpa_data *cpa, pte_t *kpte, · line 1237
Quoted 6 lines
arch/x86/mm/pat/set_memory.c +1
-2@@ -1235,11 +1234,9 @@ static int split_large_page(struct cpa_data *cpa, pte_t *kpte, {struct ptdesc *ptdesc; -if (!debug_pagealloc_enabled ()) -spin_unlock (&cpa_lock); +spin_unlock (&cpa_lock);
-> _irqsave() is needed I think :) see below
arch/x86/mm/pat/set_memory.c static int split_large_page(struct cpa_data *cpa, pte_t *kpte, · line 1239
Quoted 4 lines
arch/x86/mm/pat/set_memory.c +1
-2ptdesc =pagetable_alloc (GFP_KERNEL ,0 ); -if (!debug_pagealloc_enabled ()) -spin_lock (&cpa_lock); +spin_lock (&cpa_lock);
-> _irqrestore() as below
Quoted 4 lines
arch/x86/mm/pat/set_memory.c +0
-0if (!ptdesc)return -ENOMEM ;
arch/x86/mm/pat/set_memory.c static int __change_page_attr_set_clr(struct cpa_data *cpa, int primary) · line 2025
Quoted 10 lines
arch/x86/mm/pat/set_memory.c +2
-4@@ -2023,11 +2020,9 @@ static int __change_page_attr_set_clr(struct cpa_data *cpa, int primary)if (cpa->flags & (CPA_ARRAY |CPA_PAGES_ARRAY )) cpa->numpages =1 ; -if (!debug_pagealloc_enabled ()) -spin_lock (&cpa_lock); +spin_lock (&cpa_lock); ret =__change_page_attr (cpa, primary); -if (!debug_pagealloc_enabled ()) -spin_unlock (&cpa_lock); +spin_unlock (&cpa_lock);
__kernel_map_pages() can be called from irq context:
< GFP_ATOMIC context >kfree() or whatever
-> ...
-> __free_pages_prepare()
-> debug_pagealloc_unmap_pages()
-> __kernel_map_pages()
-> __change_page_attr_set_clr()
-> cpa_lock spins [irqs off]
Sooo you're spin locking in irq context here, which is probably not a good idea.
All the cpa_lock spin locks have to be updated to reflect this.
So spin_lock_irqsave/restore I think?
But note that this turns Denis's cpa_lock patch ([0]) into a deadlock because it's held across an IPI on TLB flush. So that has to be changed too, or possibly dropped. I'll reply about that over there!
Quoted 9 lines
arch/x86/mm/pat/set_memory.c +0
-0if (ret) goto out;base-commit:
a13c140cc289c0b7b3770bce5b3ad42ab35074aa--
2.53.0
Thanks, Lorenzo
Quoted 37 lines · Lorenzo Stoakes (ARM)
Quoted 36 lines · Mike Rapoport
From: "Mike Rapoport (Microsoft)" <[email protected]>
Dave Hansen says:
My only question is *why*!?!? Why add extra locking complexity and rules to optimizedebug_pagealloc, which is already horrendously slow.Stop gating
cpa_lockondebug_pagealloc_enabled()to simplify the code.Suggested-by Dave Hansen <[email protected]>Signed-off-by Mike Rapoport (Microsoft) <[email protected]>--- arch/x86/mm/pat/set_memory.c | 19 +++++++------------ 1 file changed, 7 insertions(+), 12 deletions(-)arch/x86/mm/pat/set_memory.c +3
-4@@ -62,10 +62,9 @@ enum cpa_warn {static const int cpa_warn_level =CPA_PROTECT ;/* -* Serialize cpa() (for !DEBUG_PAGEALLOC which uses large identity mappings) -* using cpa_lock. So that we don't allow any other cpu, with stale large tlb -* entries change the page attribute in parallel to some other cpu -* splitting a large page entry along with changing the attribute. +* Serialize cpa() using cpa_lock so that we don't allow any other cpu, with +* stale large tlb entries, to change the page attribute in parallel to some +* other cpu splitting a large page entry along with changing the attribute. */ static DEFINE_SPINLOCK (cpa_lock);@@ -1235,11 +1234,9 @@ static int split_large_page(struct cpa_data *cpa, pte_t *kpte, {struct ptdesc *ptdesc;
Quoted 5 lines
arch/x86/mm/pat/set_memory.c static int split_large_page(struct cpa_data *cpa, pte_t *kpte, · line 1237
Quoted 3 lines
arch/x86/mm/pat/set_memory.c +1
-2-if (!debug_pagealloc_enabled ()) -spin_unlock (&cpa_lock); +spin_unlock (&cpa_lock);->
_irqsave()is needed I think :) see below
I'm rather thinking that it will be simpler overall to restore the if, but wrap it in cpa_lock()/cpa_unlock() helpers with a better comment than what we have now above DEFINE_SPINLOCK(cpa_lock).
Quoted 6 lines
arch/x86/mm/pat/set_memory.c static int split_large_page(struct cpa_data *cpa, pte_t *kpte, · line 1239
Quoted 4 lines
arch/x86/mm/pat/set_memory.c +1
-2ptdesc =pagetable_alloc (GFP_KERNEL ,0 ); -if (!debug_pagealloc_enabled ()) -spin_lock (&cpa_lock); +spin_lock (&cpa_lock);->
_irqrestore()as below
Signature · 2 lines
Sincerely yours, Mike.
Quoted 44 lines · Mike Rapoport
Quoted 37 lines · Lorenzo Stoakes (ARM)
Quoted 36 lines · Mike Rapoport
From: "Mike Rapoport (Microsoft)" <[email protected]>
Dave Hansen says:
My only question is *why*!?!? Why add extra locking complexity and rules to optimizedebug_pagealloc, which is already horrendously slow.Stop gating
cpa_lockondebug_pagealloc_enabled()to simplify the code.Suggested-by Dave Hansen <[email protected]>Signed-off-by Mike Rapoport (Microsoft) <[email protected]>--- arch/x86/mm/pat/set_memory.c | 19 +++++++------------ 1 file changed, 7 insertions(+), 12 deletions(-)arch/x86/mm/pat/set_memory.c +3
-4@@ -62,10 +62,9 @@ enum cpa_warn {static const int cpa_warn_level =CPA_PROTECT ;/* -* Serialize cpa() (for !DEBUG_PAGEALLOC which uses large identity mappings) -* using cpa_lock. So that we don't allow any other cpu, with stale large tlb -* entries change the page attribute in parallel to some other cpu -* splitting a large page entry along with changing the attribute. +* Serialize cpa() using cpa_lock so that we don't allow any other cpu, with +* stale large tlb entries, to change the page attribute in parallel to some +* other cpu splitting a large page entry along with changing the attribute. */ static DEFINE_SPINLOCK (cpa_lock);@@ -1235,11 +1234,9 @@ static int split_large_page(struct cpa_data *cpa, pte_t *kpte, {struct ptdesc *ptdesc;Quoted 5 lines
arch/x86/mm/pat/set_memory.c static int split_large_page(struct cpa_data *cpa, pte_t *kpte, · line 1237
Quoted 3 lines
arch/x86/mm/pat/set_memory.c +1
-2-if (!debug_pagealloc_enabled ()) -spin_unlock (&cpa_lock); +spin_unlock (&cpa_lock);->
_irqsave()is needed I think :) see below
Quoted 3 lines
I'm rather thinking that it will be simpler overall to restore the if, but wrap it in cpa_lock()/cpa_unlock() helpers with a better comment than what we have now above
DEFINE_SPINLOCK(cpa_lock).
Right yeah, like cpa_[un]lock() that has the conditional in it and explains why.
But that does leave us with unfortunate races too if you allow collapse I think :)
So if reinstated and we retain Denis's patch, his should gate the collapse on debug pagealloc again.
Oh what a web we have spun...
Quoted 11 lines
Quoted 7 lines
arch/x86/mm/pat/set_memory.c static int split_large_page(struct cpa_data *cpa, pte_t *kpte, · line 1239
Quoted 5 lines
arch/x86/mm/pat/set_memory.c +1
-2ptdesc =pagetable_alloc (GFP_KERNEL ,0 ); -if (!debug_pagealloc_enabled ()) -spin_lock (&cpa_lock); +spin_lock (&cpa_lock);->
_irqrestore()as below--
Sincerely yours,
Mike.
Cheers, Lorenzo
arch/x86/mm/pat/set_memory.c static int split_large_page(struct cpa_data *cpa, pte_t *kpte, · line 1239
Quoted 11 lines · Mike Rapoport
arch/x86/mm/pat/set_memory.c +2
-4@@ -1235,11 +1234,9 @@ static int split_large_page(struct cpa_data *cpa, pte_t *kpte, {struct ptdesc *ptdesc; -if (!debug_pagealloc_enabled ()) -spin_unlock (&cpa_lock); +spin_unlock (&cpa_lock); ptdesc =pagetable_alloc (GFP_KERNEL ,0 ); -if (!debug_pagealloc_enabled ()) -spin_lock (&cpa_lock); +spin_lock (&cpa_lock);
A DEBUG_PAGEALLOC + KASAN kernel from linux-next-20260727 fails to finish booting on a large VM (many CPUs + large memory). Boot stalls in early userspace with a non-responsive CSD lock; it never reaches a login.
I bisected it to this patch, which is the following on linux-next:
5fce67641a3e ("x86/mm/pat: Don't gate cpa_lock on debug_pagealloc_enabled()")
Reverting it (restoring the !debug_pagealloc_enabled() gating around cpa_lock) makes the same kernel boot reliably.
Symptom
-------
Boot hangs during udev coldplug (concurrent module loading, i.e. concurrent set_memory()). With csdlock_debug=1 the console shows:
smp: csd: Detected non-responsive CSD lock (#1) on CPU#55, waiting
5000000019 ns for CPU#57 do_flush_tlb_all+0x0/0x30(0x0).
smp: csd: CSD lock (#1) unresponsive.
Sending NMI from CPU 55 to CPUs 57:
NMI backtrace for cpu 57
CPU: 57 UID: 0 PID: 860 Comm: (udev-worker) Tainted: G E \
7.2.0-rc5-next-20260727 #1 PREEMPT(full)
RIP: 0010:queued_spin_lock_slowpath+0xae/0xd30
Call Trace:
<TASK>
do_raw_spin_lock+0x268/0x2f0
__change_page_attr_set_clr+0x32d/0x21f0
...CPU#55 is stuck in flush_tlb_all() waiting for CPU#57 to answer the IPI; CPU#57 is spinning on cpa_lock (RBX resolves to cpa_lock) inside __change_page_attr_set_clr(). The clocksource watchdog also reports remote CPUs timing out on the same IPIs just before this.
Is this a known issue?
--breno
Quoted 44 lines · Breno Leitao
arch/x86/mm/pat/set_memory.c static int split_large_page(struct cpa_data *cpa, pte_t *kpte, · line 1239
Quoted 11 lines · Mike Rapoport
arch/x86/mm/pat/set_memory.c +2
-4@@ -1235,11 +1234,9 @@ static int split_large_page(struct cpa_data *cpa, pte_t *kpte, {struct ptdesc *ptdesc; -if (!debug_pagealloc_enabled ()) -spin_unlock (&cpa_lock); +spin_unlock (&cpa_lock); ptdesc =pagetable_alloc (GFP_KERNEL ,0 ); -if (!debug_pagealloc_enabled ()) -spin_lock (&cpa_lock); +spin_lock (&cpa_lock);A
DEBUG_PAGEALLOC+ KASAN kernel from linux-next-20260727 fails to finish booting on a large VM (many CPUs + large memory). Boot stalls in early userspace with a non-responsive CSD lock; it never reaches a login.I bisected it to this patch, which is the following on linux-next:
5fce67641a3e("x86/mm/pat: Don't gatecpa_lockondebug_pagealloc_enabled()")Reverting it (restoring the !debug_pagealloc_enabled() gating around
cpa_lock) makes the same kernel boot reliably.Symptom
-------Boot hangs during udev coldplug (concurrent module loading, i.e. concurrent
set_memory()). With csdlock_debug=1 the console shows:smp: csd: Detected non-responsive CSD lock (#1) on CPU#55, waiting
5000000019 ns for CPU#57 do_flush_tlb_all+0x0/0x30(0x0). smp: csd: CSD lock (#1) unresponsive. Sending NMI from CPU 55 to CPUs 57: NMI backtrace for cpu 57 CPU: 57 UID: 0 PID: 860 Comm: (udev-worker) Tainted: G E \ 7.2.0-rc5-next-20260727 #1 PREEMPT(full) RIP: 0010:queued_spin_lock_slowpath+0xae/0xd30 Call Trace: <TASK> do_raw_spin_lock+0x268/0x2f0 __change_page_attr_set_clr+0x32d/0x21f0 ...
Quoted 6 lines
CPU#55 is stuck in
flush_tlb_all()waiting for CPU#57 to answer the IPI; CPU#57 is spinning oncpa_lock(RBX resolves tocpa_lock) inside__change_page_attr_set_clr(). The clocksource watchdog also reports remote CPUs timing out on the same IPIs just before this.Is this a known issue?
Yes:
https://lore.kernel.org/all/[email protected]
Quoted 1 line
--breno
Signature · 2 lines
Sincerely yours, Mike.