Skip to contentSkip to conversation

x86/mm/pat: don't gate cpa_lock on debug_pagealloc_enabled()

PATCH Applied Mike Rapoport 5 participants 8 messages Jul 15 – Jul 28

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

PATCH 1/1 x86/mm/pat: don't gate cpa_lock on debug_pagealloc_enabled()

From: "Mike Rapoport (Microsoft)" <[email protected]>

Dave Hansen says:
My only question is *why*!?!? Why add extra locking complexity and rules to optimize debug_pagealloc, which is already horrendously slow.

Stop gating cpa_lock on debug_pagealloc_enabled() to simplify the code.

---
 arch/x86/mm/pat/set_memory.c | 19 +++++++------------
 1 file changed, 7 insertions(+), 12 deletions(-)
arch/x86/mm/pat/set_memory.c +7 -12
index d023a40a1e03..e8316f5ffa8a 100644
@@ -62,10 +62,9 @@ enum cpa_warn {
 static const int cpa_warn_level = CPA_PROTECT;
 
 /*
- * Serialize cpa() (for !DEBUG_PAGEALLOC which uses large identity mappings)
- * using cpa_lock. So that we don't allow any other cpu, with stale large tlb
- * entries change the page attribute in parallel to some other cpu
- * splitting a large page entry along with changing the attribute.
+ * Serialize cpa() using cpa_lock so that we don't allow any other cpu, with
+ * stale large tlb entries, to change the page attribute in parallel to some
+ * other cpu splitting a large page entry along with changing the attribute.
  */
 static DEFINE_SPINLOCK(cpa_lock);
 
@@ -1235,11 +1234,9 @@ static int split_large_page(struct cpa_data *cpa, pte_t *kpte,
 {
 	struct ptdesc *ptdesc;
 
-	if (!debug_pagealloc_enabled())
-		spin_unlock(&cpa_lock);
+	spin_unlock(&cpa_lock);
 	ptdesc = pagetable_alloc(GFP_KERNEL, 0);
-	if (!debug_pagealloc_enabled())
-		spin_lock(&cpa_lock);
+	spin_lock(&cpa_lock);
 	if (!ptdesc)
 		return -ENOMEM;
 
@@ -2023,11 +2020,9 @@ static int __change_page_attr_set_clr(struct cpa_data *cpa, int primary)
 		if (cpa->flags & (CPA_ARRAY | CPA_PAGES_ARRAY))
 			cpa->numpages = 1;
 
-		if (!debug_pagealloc_enabled())
-			spin_lock(&cpa_lock);
+		spin_lock(&cpa_lock);
 		ret = __change_page_attr(cpa, primary);
-		if (!debug_pagealloc_enabled())
-			spin_unlock(&cpa_lock);
+		spin_unlock(&cpa_lock);
 		if (ret)
 			goto out;
 

base-commit: a13c140cc289c0b7b3770bce5b3ad42ab35074aa

Signature · 1 line
2.53.0
#1 lore raw
Quoted 7 lines · Mike Rapoport

From: "Mike Rapoport (Microsoft)" <[email protected]>

Dave Hansen says:
My only question is *why*!?!? Why add extra locking complexity and rules to optimize debug_pagealloc, which is already horrendously slow.

Stop gating cpa_lock on debug_pagealloc_enabled() to simplify the code.

Thanks for sending this, Mike! I munged the changelog a bit and applied it.

#2 lore raw

[tip: x86/mm] x86/mm/pat: Don't gate cpa_lock on debug_pagealloc_enabled()

In reply to Mike Rapoport
The following commit has been merged into the x86/mm branch of tip:

The following commit has been merged into the x86/mm branch of tip:

Commit-ID:     5fce67641a3ed9a0782eaa228ddece526461a367
Gitweb:        https://git.kernel.org/tip/5fce67641a3ed9a0782eaa228ddece526461a367
Author:        Mike Rapoport (Microsoft) <[email protected]>
AuthorDate:    Wed, 15 Jul 2026 17:45:19 +03:00
Committer:     Dave Hansen <[email protected]>
CommitterDate: Wed, 15 Jul 2026 08:00:49 -07:00

x86/mm/pat: Don't gate cpa_lock on debug_pagealloc_enabled()

The splitting and merging of kernel page table mappings between small and large is protected by cpa_lock. The merging is relatively new but the splitting is ancient.

The splitting has a locking optimization: since DEBUG_PAGEALLOC forces all mappings to 4k, there are no large pages to split. So the code that *might* cause a split can just skip the locking (and a few other things).

This is entertaining, but it adds complexity and makes for weird locking rules. Plus it's all for a debugging feature which makes the kernel super slow in the first place. Optimizing something which is already super slow and not used in production is not the best way to spend our complexity budget.

Stop gating cpa_lock on debug_pagealloc_enabled() to simplify the code and the locking rules.

[ dhansen: flesh out changelog ]

---
 arch/x86/mm/pat/set_memory.c | 19 +++++++------------
 1 file changed, 7 insertions(+), 12 deletions(-)
arch/x86/mm/pat/set_memory.c +7 -12
index 45623d4..e9b4083 100644
@@ -62,10 +62,9 @@ enum cpa_warn {
 static const int cpa_warn_level = CPA_PROTECT;
 
 /*
- * Serialize cpa() (for !DEBUG_PAGEALLOC which uses large identity mappings)
- * using cpa_lock. So that we don't allow any other cpu, with stale large tlb
- * entries change the page attribute in parallel to some other cpu
- * splitting a large page entry along with changing the attribute.
+ * Serialize cpa() using cpa_lock so that we don't allow any other cpu, with
+ * stale large tlb entries, to change the page attribute in parallel to some
+ * other cpu splitting a large page entry along with changing the attribute.
  */
 static DEFINE_SPINLOCK(cpa_lock);
 
@@ -1234,11 +1233,9 @@ static int split_large_page(struct cpa_data *cpa, pte_t *kpte,
 {
 	struct ptdesc *ptdesc;
 
-	if (!debug_pagealloc_enabled())
-		spin_unlock(&cpa_lock);
+	spin_unlock(&cpa_lock);
 	ptdesc = pagetable_alloc(GFP_KERNEL, 0);
-	if (!debug_pagealloc_enabled())
-		spin_lock(&cpa_lock);
+	spin_lock(&cpa_lock);
 	if (!ptdesc)
 		return -ENOMEM;
 
@@ -2022,11 +2019,9 @@ static int __change_page_attr_set_clr(struct cpa_data *cpa, int primary)
 		if (cpa->flags & (CPA_ARRAY | CPA_PAGES_ARRAY))
 			cpa->numpages = 1;
 
-		if (!debug_pagealloc_enabled())
-			spin_lock(&cpa_lock);
+		spin_lock(&cpa_lock);
 		ret = __change_page_attr(cpa, primary);
-		if (!debug_pagealloc_enabled())
-			spin_unlock(&cpa_lock);
+		spin_unlock(&cpa_lock);
 		if (ret)
 			goto out;
 
#3 lore raw
In reply to Mike Rapoport
Quoted 33 lines · Mike Rapoport

From: "Mike Rapoport (Microsoft)" <[email protected]>

Dave Hansen says:
My only question is *why*!?!? Why add extra locking complexity and rules to optimize debug_pagealloc, which is already horrendously slow.

Stop gating cpa_lock on debug_pagealloc_enabled() to simplify the code.

---
 arch/x86/mm/pat/set_memory.c | 19 +++++++------------
 1 file changed, 7 insertions(+), 12 deletions(-)
arch/x86/mm/pat/set_memory.c +3 -4
index d023a40a1e03..e8316f5ffa8a 100644
@@ -62,10 +62,9 @@ enum cpa_warn {
 static const int cpa_warn_level = CPA_PROTECT;
 
 /*
- * Serialize cpa() (for !DEBUG_PAGEALLOC which uses large identity mappings)
- * using cpa_lock. So that we don't allow any other cpu, with stale large tlb
- * entries change the page attribute in parallel to some other cpu
- * splitting a large page entry along with changing the attribute.
+ * Serialize cpa() using cpa_lock so that we don't allow any other cpu, with
+ * stale large tlb entries, to change the page attribute in parallel to some
+ * other cpu splitting a large page entry along with changing the attribute.
  */
 static DEFINE_SPINLOCK(cpa_lock);
 

arch/x86/mm/pat/set_memory.c static int split_large_page(struct cpa_data *cpa, pte_t *kpte, · line 1237

Quoted 6 lines
arch/x86/mm/pat/set_memory.c +1 -2
@@ -1235,11 +1234,9 @@ static int split_large_page(struct cpa_data *cpa, pte_t *kpte,
 {
 	struct ptdesc *ptdesc;
 
-	if (!debug_pagealloc_enabled())
-		spin_unlock(&cpa_lock);
+	spin_unlock(&cpa_lock);

-> _irqsave() is needed I think :) see below

arch/x86/mm/pat/set_memory.c static int split_large_page(struct cpa_data *cpa, pte_t *kpte, · line 1239

Quoted 4 lines
arch/x86/mm/pat/set_memory.c +1 -2
 	ptdesc = pagetable_alloc(GFP_KERNEL, 0);
-	if (!debug_pagealloc_enabled())
-		spin_lock(&cpa_lock);
+	spin_lock(&cpa_lock);

-> _irqrestore() as below

Quoted 4 lines
arch/x86/mm/pat/set_memory.c +0 -0
 	if (!ptdesc)
 		return -ENOMEM;
 

arch/x86/mm/pat/set_memory.c static int __change_page_attr_set_clr(struct cpa_data *cpa, int primary) · line 2025

Quoted 10 lines
arch/x86/mm/pat/set_memory.c +2 -4
@@ -2023,11 +2020,9 @@ static int __change_page_attr_set_clr(struct cpa_data *cpa, int primary)
 		if (cpa->flags & (CPA_ARRAY | CPA_PAGES_ARRAY))
 			cpa->numpages = 1;
 
-		if (!debug_pagealloc_enabled())
-			spin_lock(&cpa_lock);
+		spin_lock(&cpa_lock);
 		ret = __change_page_attr(cpa, primary);
-		if (!debug_pagealloc_enabled())
-			spin_unlock(&cpa_lock);
+		spin_unlock(&cpa_lock);

__kernel_map_pages() can be called from irq context:

< GFP_ATOMIC context >
kfree() or whatever
-> ...
-> __free_pages_prepare()
-> debug_pagealloc_unmap_pages()
-> __kernel_map_pages()
-> __change_page_attr_set_clr()
-> cpa_lock spins [irqs off]

Sooo you're spin locking in irq context here, which is probably not a good idea.

All the cpa_lock spin locks have to be updated to reflect this.

So spin_lock_irqsave/restore I think?

But note that this turns Denis's cpa_lock patch ([0]) into a deadlock because it's held across an IPI on TLB flush. So that has to be changed too, or possibly dropped. I'll reply about that over there!

Quoted 9 lines
arch/x86/mm/pat/set_memory.c +0 -0
 		if (ret)
 			goto out;
 

base-commit: a13c140cc289c0b7b3770bce5b3ad42ab35074aa --
2.53.0

Thanks, Lorenzo

[0]:https://lore.kernel.org/all/[email protected]/

#4 lore raw
Quoted 37 lines · Lorenzo Stoakes (ARM)
Quoted 36 lines · Mike Rapoport

From: "Mike Rapoport (Microsoft)" <[email protected]>

Dave Hansen says:
My only question is *why*!?!? Why add extra locking complexity and rules to optimize debug_pagealloc, which is already horrendously slow.

Stop gating cpa_lock on debug_pagealloc_enabled() to simplify the code.

---
 arch/x86/mm/pat/set_memory.c | 19 +++++++------------
 1 file changed, 7 insertions(+), 12 deletions(-)
arch/x86/mm/pat/set_memory.c +3 -4
index d023a40a1e03..e8316f5ffa8a 100644
@@ -62,10 +62,9 @@ enum cpa_warn {
 static const int cpa_warn_level = CPA_PROTECT;
 
 /*
- * Serialize cpa() (for !DEBUG_PAGEALLOC which uses large identity mappings)
- * using cpa_lock. So that we don't allow any other cpu, with stale large tlb
- * entries change the page attribute in parallel to some other cpu
- * splitting a large page entry along with changing the attribute.
+ * Serialize cpa() using cpa_lock so that we don't allow any other cpu, with
+ * stale large tlb entries, to change the page attribute in parallel to some
+ * other cpu splitting a large page entry along with changing the attribute.
  */
 static DEFINE_SPINLOCK(cpa_lock);
 
@@ -1235,11 +1234,9 @@ static int split_large_page(struct cpa_data *cpa, pte_t *kpte,
 {
 	struct ptdesc *ptdesc;
 
Quoted 5 lines

arch/x86/mm/pat/set_memory.c static int split_large_page(struct cpa_data *cpa, pte_t *kpte, · line 1237

Quoted 3 lines
arch/x86/mm/pat/set_memory.c +1 -2
-	if (!debug_pagealloc_enabled())
-		spin_unlock(&cpa_lock);
+	spin_unlock(&cpa_lock);

-> _irqsave() is needed I think :) see below

I'm rather thinking that it will be simpler overall to restore the if, but wrap it in cpa_lock()/cpa_unlock() helpers with a better comment than what we have now above DEFINE_SPINLOCK(cpa_lock).

Quoted 6 lines

arch/x86/mm/pat/set_memory.c static int split_large_page(struct cpa_data *cpa, pte_t *kpte, · line 1239

Quoted 4 lines
arch/x86/mm/pat/set_memory.c +1 -2
 	ptdesc = pagetable_alloc(GFP_KERNEL, 0);
-	if (!debug_pagealloc_enabled())
-		spin_lock(&cpa_lock);
+	spin_lock(&cpa_lock);

-> _irqrestore() as below

Signature · 2 lines
Sincerely yours,
Mike.
#5 lore raw
Quoted 44 lines · Mike Rapoport
Quoted 37 lines · Lorenzo Stoakes (ARM)
Quoted 36 lines · Mike Rapoport

From: "Mike Rapoport (Microsoft)" <[email protected]>

Dave Hansen says:
My only question is *why*!?!? Why add extra locking complexity and rules to optimize debug_pagealloc, which is already horrendously slow.

Stop gating cpa_lock on debug_pagealloc_enabled() to simplify the code.

---
 arch/x86/mm/pat/set_memory.c | 19 +++++++------------
 1 file changed, 7 insertions(+), 12 deletions(-)
arch/x86/mm/pat/set_memory.c +3 -4
index d023a40a1e03..e8316f5ffa8a 100644
@@ -62,10 +62,9 @@ enum cpa_warn {
 static const int cpa_warn_level = CPA_PROTECT;
 
 /*
- * Serialize cpa() (for !DEBUG_PAGEALLOC which uses large identity mappings)
- * using cpa_lock. So that we don't allow any other cpu, with stale large tlb
- * entries change the page attribute in parallel to some other cpu
- * splitting a large page entry along with changing the attribute.
+ * Serialize cpa() using cpa_lock so that we don't allow any other cpu, with
+ * stale large tlb entries, to change the page attribute in parallel to some
+ * other cpu splitting a large page entry along with changing the attribute.
  */
 static DEFINE_SPINLOCK(cpa_lock);
 
@@ -1235,11 +1234,9 @@ static int split_large_page(struct cpa_data *cpa, pte_t *kpte,
 {
 	struct ptdesc *ptdesc;
 
Quoted 5 lines

arch/x86/mm/pat/set_memory.c static int split_large_page(struct cpa_data *cpa, pte_t *kpte, · line 1237

Quoted 3 lines
arch/x86/mm/pat/set_memory.c +1 -2
-	if (!debug_pagealloc_enabled())
-		spin_unlock(&cpa_lock);
+	spin_unlock(&cpa_lock);

-> _irqsave() is needed I think :) see below

Quoted 3 lines

I'm rather thinking that it will be simpler overall to restore the if, but wrap it in cpa_lock()/cpa_unlock() helpers with a better comment than what we have now above DEFINE_SPINLOCK(cpa_lock).

Right yeah, like cpa_[un]lock() that has the conditional in it and explains why.

But that does leave us with unfortunate races too if you allow collapse I think :)

So if reinstated and we retain Denis's patch, his should gate the collapse on debug pagealloc again.

Oh what a web we have spun...

Quoted 11 lines
Quoted 7 lines

arch/x86/mm/pat/set_memory.c static int split_large_page(struct cpa_data *cpa, pte_t *kpte, · line 1239

Quoted 5 lines
arch/x86/mm/pat/set_memory.c +1 -2
 	ptdesc = pagetable_alloc(GFP_KERNEL, 0);
-	if (!debug_pagealloc_enabled())
-		spin_lock(&cpa_lock);
+	spin_lock(&cpa_lock);

-> _irqrestore() as below

--
Sincerely yours,
Mike.

Cheers, Lorenzo

#6 lore raw
In reply to Mike Rapoport

arch/x86/mm/pat/set_memory.c static int split_large_page(struct cpa_data *cpa, pte_t *kpte, · line 1239

Quoted 11 lines · Mike Rapoport
arch/x86/mm/pat/set_memory.c +2 -4
@@ -1235,11 +1234,9 @@ static int split_large_page(struct cpa_data *cpa, pte_t *kpte,
 {
 	struct ptdesc *ptdesc;
 
-	if (!debug_pagealloc_enabled())
-		spin_unlock(&cpa_lock);
+	spin_unlock(&cpa_lock);
 	ptdesc = pagetable_alloc(GFP_KERNEL, 0);
-	if (!debug_pagealloc_enabled())
-		spin_lock(&cpa_lock);
+	spin_lock(&cpa_lock);

A DEBUG_PAGEALLOC + KASAN kernel from linux-next-20260727 fails to finish booting on a large VM (many CPUs + large memory). Boot stalls in early userspace with a non-responsive CSD lock; it never reaches a login.

I bisected it to this patch, which is the following on linux-next:

5fce67641a3e ("x86/mm/pat: Don't gate cpa_lock on debug_pagealloc_enabled()")

Reverting it (restoring the !debug_pagealloc_enabled() gating around cpa_lock) makes the same kernel boot reliably.

Symptom
-------

Boot hangs during udev coldplug (concurrent module loading, i.e. concurrent set_memory()). With csdlock_debug=1 the console shows:

smp: csd: Detected non-responsive CSD lock (#1) on CPU#55, waiting

       5000000019 ns for CPU#57 do_flush_tlb_all+0x0/0x30(0x0).
  smp:  csd: CSD lock (#1) unresponsive.
  Sending NMI from CPU 55 to CPUs 57:
  NMI backtrace for cpu 57
  CPU: 57 UID: 0 PID: 860 Comm: (udev-worker) Tainted: G      E   \
       7.2.0-rc5-next-20260727 #1 PREEMPT(full)
  RIP: 0010:queued_spin_lock_slowpath+0xae/0xd30
  Call Trace:
   <TASK>
   do_raw_spin_lock+0x268/0x2f0
   __change_page_attr_set_clr+0x32d/0x21f0
   ...

CPU#55 is stuck in flush_tlb_all() waiting for CPU#57 to answer the IPI; CPU#57 is spinning on cpa_lock (RBX resolves to cpa_lock) inside __change_page_attr_set_clr(). The clocksource watchdog also reports remote CPUs timing out on the same IPIs just before this.

Is this a known issue?
--breno

#7 lore raw
Quoted 44 lines · Breno Leitao

arch/x86/mm/pat/set_memory.c static int split_large_page(struct cpa_data *cpa, pte_t *kpte, · line 1239

Quoted 11 lines · Mike Rapoport
arch/x86/mm/pat/set_memory.c +2 -4
@@ -1235,11 +1234,9 @@ static int split_large_page(struct cpa_data *cpa, pte_t *kpte,
 {
 	struct ptdesc *ptdesc;
 
-	if (!debug_pagealloc_enabled())
-		spin_unlock(&cpa_lock);
+	spin_unlock(&cpa_lock);
 	ptdesc = pagetable_alloc(GFP_KERNEL, 0);
-	if (!debug_pagealloc_enabled())
-		spin_lock(&cpa_lock);
+	spin_lock(&cpa_lock);

A DEBUG_PAGEALLOC + KASAN kernel from linux-next-20260727 fails to finish booting on a large VM (many CPUs + large memory). Boot stalls in early userspace with a non-responsive CSD lock; it never reaches a login.

I bisected it to this patch, which is the following on linux-next:

5fce67641a3e ("x86/mm/pat: Don't gate cpa_lock on debug_pagealloc_enabled()")

Reverting it (restoring the !debug_pagealloc_enabled() gating around cpa_lock) makes the same kernel boot reliably.

Symptom
-------

Boot hangs during udev coldplug (concurrent module loading, i.e. concurrent set_memory()). With csdlock_debug=1 the console shows:

smp: csd: Detected non-responsive CSD lock (#1) on CPU#55, waiting

       5000000019 ns for CPU#57 do_flush_tlb_all+0x0/0x30(0x0).
  smp:  csd: CSD lock (#1) unresponsive.
  Sending NMI from CPU 55 to CPUs 57:
  NMI backtrace for cpu 57
  CPU: 57 UID: 0 PID: 860 Comm: (udev-worker) Tainted: G      E   \
       7.2.0-rc5-next-20260727 #1 PREEMPT(full)
  RIP: 0010:queued_spin_lock_slowpath+0xae/0xd30
  Call Trace:
   <TASK>
   do_raw_spin_lock+0x268/0x2f0
   __change_page_attr_set_clr+0x32d/0x21f0
   ...
Quoted 6 lines

CPU#55 is stuck in flush_tlb_all() waiting for CPU#57 to answer the IPI; CPU#57 is spinning on cpa_lock (RBX resolves to cpa_lock) inside __change_page_attr_set_clr(). The clocksource watchdog also reports remote CPUs timing out on the same IPIs just before this.

Is this a known issue?

Yes:
https://lore.kernel.org/all/[email protected]

Quoted 1 line

--breno

Signature · 2 lines
Sincerely yours,
Mike.