sparc64: D-cache alias flushing fixes

Since commit f822a9a81a31 ("mm: optimize mremap() by PTE batching"), move_ptes() moves a batch of PTEs mapping consecutive pages of a folio at a time, but calls move_pte() once per batch with the first PTE only. sparc64, the only architecture that overrides move_pte(), flushes from the D-cache the page that PTE maps when the move changes its colour, so the other pages of a batch would keep lines of the old colour. This cannot happen today only because sparc64 has no pte_batch_hint(), and mremap_folio_pte_batch() then never batches.

Pass nr_ptes to move_pte(), as get_and_clear_ptes() takes it. The generic version ignores it, and sparc64 flushes every page of the batch.

Signed-off-by Imre Kaloz <[email protected]>

---
 arch/sparc/include/asm/pgtable_64.h | 16 +++++++++-------
 include/linux/pgtable.h             |  2 +-
 mm/mremap.c                         |  2 +-
 3 files changed, 11 insertions(+), 9 deletions(-)

arch/sparc/include/asm/pgtable_64.h +0 -0

index 1430489b9ac5..9ca787b3cbfe 100644
@@ -946,21 +946,23 @@ static inline void set_ptes(struct mm_struct *mm, unsigned long addr,
 	set_pte_at((mm), (addr), (ptep), __pte(0UL))
 

arch/sparc/include/asm/pgtable_64.h +0 -0

 ({									\
 	pte_t newpte = (pte);						\
 	if (tlb_type != hypervisor && pte_present(pte)) {		\
 		unsigned long this_pfn = pte_pfn(pte);			\

Since commit f822a9a81a31 ("mm: optimize mremap() by PTE batching"), move_ptes() moves a batch of PTEs mapping consecutive pages of a folio at a time, but calls move_pte() once per batch with the first PTE only. sparc64, the only architecture that overrides move_pte(), flushes from the D-cache the page that PTE maps when the move changes its colour, so the other pages of a batch would keep lines of the old colour. This cannot happen today only because sparc64 has no pte_batch_hint(), and mremap_folio_pte_batch() then never batches.

Pass nr_ptes to move_pte(), as get_and_clear_ptes() takes it. The generic version ignores it, and sparc64 flushes every page of the batch.

Signed-off-by Imre Kaloz <[email protected]>

---
 arch/sparc/include/asm/pgtable_64.h | 16 +++++++++-------
 include/linux/pgtable.h             |  2 +-
 mm/mremap.c                         |  2 +-
 3 files changed, 11 insertions(+), 9 deletions(-)

arch/sparc/include/asm/pgtable_64.h +9 -7

index 1430489b9ac5..9ca787b3cbfe 100644
@@ -946,21 +946,23 @@ static inline void set_ptes(struct mm_struct *mm, unsigned long addr,
 	set_pte_at((mm), (addr), (ptep), __pte(0UL))
 
 #ifdef DCACHE_ALIASING_POSSIBLE
-/* This flushes only the page the PTE maps, so it relies on move_ptes()
- * calling it for one PTE at a time, which the default pte_batch_hint()
- * guarantees.
+/* The nr PTEs map consecutive pages of one folio and all move by the
+ * same offset, so either every page of the batch aliases or none does.
  */
 #define __HAVE_ARCH_MOVE_PTE
-#define move_pte(pte, old_addr, new_addr)				\
+#define move_pte(pte, old_addr, new_addr, nr)				\
 ({									\
 	pte_t newpte = (pte);						\
 	if (tlb_type != hypervisor && pte_present(pte)) {		\
 		unsigned long this_pfn = pte_pfn(pte);			\
+		unsigned int i;						\
 									\
 		if (pfn_valid(this_pfn) &&				\
-		    (((old_addr) ^ (new_addr)) & (1 << 13)))		\
-			flush_dcache_page_all(current->mm,		\
-					      pfn_to_page(this_pfn));	\
+		    (((old_addr) ^ (new_addr)) & (1 << 13))) {		\
+			for (i = 0; i < (nr); i++)			\
+				flush_dcache_page_all(current->mm,	\
+					pfn_to_page(this_pfn + i));	\
+		}							\
 	}								\
 	newpte;								\
 })

include/linux/pgtable.h +1 -1

index 8c093c119e5a..b58c0e98994e 100644
@@ -1471,7 +1471,7 @@ static inline void arch_swap_restore(swp_entry_t entry, struct folio *folio)
 #endif
 
 #ifndef __HAVE_ARCH_MOVE_PTE
-#define move_pte(pte, old_addr, new_addr)	(pte)
+#define move_pte(pte, old_addr, new_addr, nr)	(pte)
 #endif
 
 #ifndef pte_accessible

mm/mremap.c +0 -0

index e8df5cdb0ac9..7fcae7f0082d 100644

Is there some way we can avoid this helper-that-shouldn't-exist-but-someone-hacked-it-in entirely? If it makes sparc64 slightly slower #dontcare.

Hi David,

Quoted 3 lines · David Hildenbrand (Arm)

Is there some way we can avoid this helper-that-shouldn't-exist-but-someone-hacked-it-in entirely? If it makes sparc64 slightly slower #dontcare.

I tried three ways of doing that on an Ultra 45 (UltraSPARC IIIi, two CPUs), all with move_pte() removed entirely:

A) sparc64's flush_cache_range() flushes every present page in the

    range from the D-cache;

B) sparc64's flush_cache_range() flushes the whole D-cache;
C) A for ranges of up to 10 pages, B above that.

All three pass a test that moves pages between cache colors with mremap() and checks for stale data. With no flush at all the same test finds about 15 million stale words per run, so it does catch the problem.

Medians of 20 runs over two boots per kernel, against this series' move_pte() as the baseline:

move_pte() A B C

  apt-cache gencaches           6.63 s     +0.2%      +0.3%      +0.1%
  fork+exec                                +0.7%      +3.1%      -0.6%
  mremap 8K, same color          14 us      20 us      43 us      20 us
  mremap 8K, other color         21 us      20 us      43 us      20 us
  mremap 1M, same color          30 us     524 us      60 us      59 us
  mremap 8M, same color         209 us    4152 us     229 us     227 us
  mremap 8M, other color       4176 us    4166 us     236 us     233 us
  mremap 8M THP, same color      14 us    3933 us      43 us      42 us

flush_cache_range() has more callers than mremap(), mostly in rmap, so I also timed reclaim, migration and writeback (medians of 5):

A B C

  MADV_PAGEOUT to swap          +0.4%     +12.9%      -1.3%
  reclaim under memory.max      +1.9%     +17.7%      +2.0%
  msync of shared file mappings -4.4%      -3.4%      -0.4%

compaction, move_pages() within 3% for all three

So A pays on large moves, B on reclaim, and C on neither: it costs the same as A on small ranges and the same as B on large ones. I'd go with C. It would only flush on Spitfire and Cheetah, which have the aliasing D-cache, so sun4v and the SPARC64 VII in Magnus's M3000 series skip it.

If that works for you, I'd suggest dropping 4/4 and using C as its replacement; it's below the scissors, on top of 1/4-3/4.

But that's too complicated, requiring page table walks ....
People running museum piece hardware and complaining about performance of mremap or memory reclaim should reconsider life choices.

So I'd say, make it as slow as you want while reducing core-MM complexity and not adding more arch complexity.

Hi David,

Quoted 3 lines · David Hildenbrand (Arm)

But that's too complicated, requiring page table walks ....
People running museum piece hardware and complaining about performance of mremap or memory reclaim should reconsider life choices.

I'm one of the museum folks myself, though more the kind who loves running a modern kernel on old hardware than the kind who complains about it :) A lot of these machines are still doing real work, and these days not everyone can, or wants to, just buy a new computer.

Some of us spend our time keeping these platforms working, and I'm not asking anyone else to carry that, which is why I tried to keep all of it in sparc.

Quoted 3 lines

So I'd say, make it as slow as you want while reducing core-MM complexity and not adding more arch complexity.

With C, move_pte() and __HAVE_ARCH_MOVE_PTE go away from include/linux/pgtable.h and mm/mremap.c just like with A and B. The walk lives in sparc's own flush_cache_range() and only runs on the CPUs that need it.

B would be less sparc code, but every UltraSPARC I-IV machine would pay 13-18% on reclaim for it, so I'd still lean towards C. If you feel strongly about B, though, I'm happy to send that instead.

Quoted 7 lines · Imre Kaloz

Hi David,

Quoted 3 lines · David Hildenbrand (Arm)

But that's too complicated, requiring page table walks ....
People running museum piece hardware and complaining about performance of mremap or memory reclaim should reconsider life choices.

Quoted 8 lines

I'm one of the museum folks myself, though more the kind who loves running a modern kernel on old hardware than the kind who complains about it :) A lot of these machines are still doing real work, and these days not everyone can, or wants to, just buy a new computer.

Some of us spend our time keeping these platforms working, and I'm not asking anyone else to carry that, which is why I tried to keep all of it in sparc.

Okay, so mremap() and reclaim memory performance concerns make you sleep bad at night? :)

Don't get me wrong, I know that running new kernels on old HW is fun.

Last week a got upstream Linux running on a 32bit x86 notebook from 2007 with the snap of a finger. Everything just works (using Alpine).

I couldn't care less about performance. Nobody should.

The biggest problem all of us with supporting old hardware is the maintanance cost. Not just core-mm, but also things like ad-hoc page table walkers you are adding we (from core-mm) might have to touch at some point.

If there is no good reason for the complexity, out it goes.

Because otherwise people will push more and more for removing legacy support completely, and that's not what you want, right?

Hi David,

Quoted 13 lines · David Hildenbrand (Arm)

Quoted 11 lines · Imre Kaloz

Hi David,

I'm one of the museum folks myself, though more the kind who loves running a modern kernel on old hardware than the kind who complains about it :) A lot of these machines are still doing real work, and these days not everyone can, or wants to, just buy a new computer.

Some of us spend our time keeping these platforms working, and I'm not asking anyone else to carry that, which is why I tried to keep all of it in sparc.

Quoted 7 lines

Okay, so mremap() and reclaim memory performance concerns make you sleep bad at night? :)

Don't get me wrong, I know that running new kernels on old HW is fun.

Last week a got upstream Linux running on a 32bit x86 notebook from 2007 with the snap of a finger. Everything just works (using Alpine).

That's cool!

Quoted 1 line

I couldn't care less about performance. Nobody should.

Well, try to put it into some perspective: For some people, old hardware is all what they have and given the current high prices, I assume many people will be stuck with old hardware for a while now. Think of developing countries as well.

Quoted 4 lines

The biggest problem all of us with supporting old hardware is the maintanance cost. Not just core-mm, but also things like ad-hoc page table walkers you are adding we (from core-mm) might have to touch at some point.

Well, but if people are using it in the end, I think it's justified to support it.

Quoted 2 lines

If there is no good reason for the complexity, out it goes.

The patch doesn't really look that complex though. It adds an additional parameter to move_pte() which adds a few more lines of code. Not really overly complex, I think.

Quoted 3 lines

Because otherwise people will push more and more for removing legacy support completely, and that's not what you want, right?

I don't think it's fair to make the kernel exclusive to very recent hardware only.

Quoted 1 line · John Paul Adrian Glaubitz

I don't think it's fair to make the kernel exclusive to very recent hardware only.

"Very recent" find the mistake.

(sorry, I'm unusually grumpy today)

Okay, so mremap() and reclaim memory performance concerns make you sleep bad at night? :)

Don't get me wrong, I know that running new kernels on old HW is fun. Last week a got upstream Linux running on a 32bit x86 notebook from 2007 with the snap of a finger. Everything just works (using Alpine).

The biggest problem all of us with supporting old hardware is the maintanance cost. Not just core-mm, but also things like ad-hoc page table walkers you are adding we (from core-mm) might have to touch at some point.

If there is no good reason for the complexity, out it goes.

Because otherwise people will push more and more for removing legacy support completely, and that's not what you want, right?

Quoted 8 lines

The biggest problem all of us with supporting old hardware is the maintanance cost. Not just core-mm, but also things like ad-hoc page table walkers you are adding we (from core-mm) might have to touch at some point.

If there is no good reason for the complexity, out it goes.

Because otherwise people will push more and more for removing legacy support completely, and that's not what you want, right?

Definitely not. Fair point, I was only looking at it from the sparc side, and keeping these platforms cheap to carry matters more than a few percent on reclaim. B it is, then; it's below the scissors, on top of 1/4-3/4. Thanks for taking the time to go through this!

Andreas, if David is happy with it, would you take it in place of 4/4?

Quoted 9 lines · Imre Kaloz

Quoted 8 lines

The biggest problem all of us with supporting old hardware is the maintanance cost. Not just core-mm, but also things like ad-hoc page table walkers you are adding we (from core-mm) might have to touch at some point.

If there is no good reason for the complexity, out it goes.

Because otherwise people will push more and more for removing legacy support completely, and that's not what you want, right?

Quoted 6 lines

Definitely not. Fair point, I was only looking at it from the sparc side, and keeping these platforms cheap to carry matters more than a few percent on reclaim. B it is, then; it's below the scissors, on top of 1/4-3/4. Thanks for taking the time to go through this!

Andreas, if David is happy with it, would you take it in place of 4/4?

I'm open to a solution like that, but it'll have to wait for the next round as we are far too far into the merge window.

Quoted 4 lines · Andreas Larsson

I'm open to a solution like that, but it'll have to wait for the next round as we are far too far into the merge window. Patch 3 stands on its own legs as a bugfix in the current context. So send it as a separate patch.

Fully agreed.

Quoted 4 lines

Because otherwise people will push more and more for removing legacy support completely, and that's not what you want, right?

Definitely not. Fair point, I was only looking at it from the sparc side, and keeping these platforms cheap to carry matters more than a few percent on reclaim. B it is, then; it's below the scissors, on top of 1/4-3/4. Thanks for taking the time to go through this!

Andreas, if David is happy with it, would you take it in place of 4/4?

These fix D-cache alias flushing on sparc64 CPUs with a virtually indexed L1 D-cache (Spitfire and Cheetah).

These fix D-cache alias flushing on sparc64 CPUs with a virtually indexed L1 D-cache (Spitfire and Cheetah).

1/4 makes xcall_deliver() bail out before it touches the trap block when there is no other CPU to call, so an early cross call, before init_IRQ() sets up the mondo blocks, can't end up writing to physical address 0.

2/4 flushes only the page a PTE maps instead of its whole folio, which removes about a million redundant page flushes from a colour-changing mremap() of a large folio (7.3 s on a Sun Fire V240).

3/4 implements flush_cache_vmap() and flush_cache_vunmap(), which have been no-ops. Stale vmalloc lines corrupt BPF programs and module data on UltraSPARC III.

4/4 passes the number of PTEs to move_pte(), as Andreas suggested. It touches mm/mremap.c and include/linux/pgtable.h, so it needs an ack from the mm folks.

1-3 don't depend on 4 and can go in on their own. 2 and 3 are for stable (3 needs 2), 1 and 4 aren't. The series is on top of the sparc for-next branch.

Changes in v3:
- new 1/4 and 4/4, following Andreas' review
- 2/4: reworded the message and the move_pte() comment, marked the UP #endif
- 3/4: reworded the message, dropped the Suggested-by (I have no email address for nsafran1217, the Link credits them), noted the stable dependency on 2/4
- picked up Stian's and Adrian's Tested-by tags

v2: https://lore.kernel.org/r/[email protected]

Imre Kaloz (4):
sparc64: skip the trap block in xcall_deliver() when there is no
target
sparc64: flush only the aliased page, not its whole folio
sparc64: flush vmalloc ranges from the D-cache on map and unmap
mm: pass the number of PTEs to move_pte()

 arch/sparc/include/asm/cacheflush_64.h |  9 ++--
 arch/sparc/include/asm/pgtable_64.h    | 14 ++++--
 arch/sparc/kernel/smp_64.c             | 51 +++++++++++++++-----
 arch/sparc/mm/init_64.c                | 65 ++++++++++++++++++++++++++
 arch/sparc/mm/tlb.c                    |  2 +-
 arch/sparc/mm/ultra.S                  | 25 ++++++++++
 include/linux/pgtable.h                |  2 +-
 mm/mremap.c                            |  2 +-
 8 files changed, 148 insertions(+), 22 deletions(-)

base-commit: 2bc58762a3d74e42de129d717ae3ba10ecf0ab96