Skip to contentSkip to conversation

mm: count zeromap read and set for swapout and swapin

RFC PATCH Superseded Barry Song 6 participants 29 messages Oct 27, 2024 – Oct 30, 2024

Subsystems (2)

v1 of 4 Superseded Latest is v4 →

lore
Share & tools
lore

Barry Song

author
#0 lore raw

RFC PATCH 1/1 mm: count zeromap read and set for swapout and swapin

From: Barry Song <[email protected]>

When the proportion of folios from the zero map is small, missing their accounting may not significantly impact profiling. However, it’s easy to construct a scenario where this becomes an issue—for example, allocating 1 GB of memory, writing zeros from userspace, followed by MADV_PAGEOUT, and then swapping it back in. In this case, the swap-out and swap-in counts seem to vanish into a black hole, potentially causing semantic ambiguity.

We have two ways to address this:

1. Add a separate counter specifically for the zero map.
2. Continue using the current accounting, treating the zero map like a normal backend. (This aligns with the current behavior of zRAM when supporting same-page fills at the device level.)

This patch adopts option 2. I'm curious if others have different opinions, so I'm marking it as RFC.

Fixes 0ca0c24e3211 ("mm: store zero pages to be swapped out in a bitmap")
Cc Usama Arif <[email protected]>
Cc Chengming Zhou <[email protected]>
Cc Yosry Ahmed <[email protected]>
Cc Nhat Pham <[email protected]>
Cc Johannes Weiner <[email protected]>
Cc David Hildenbrand <[email protected]>
Cc Hugh Dickins <[email protected]>
Cc Matthew Wilcox (Oracle) <[email protected]>
Cc Shakeel Butt <[email protected]>
Cc Andi Kleen <[email protected]>
Cc Baolin Wang <[email protected]>
Cc Chris Li <[email protected]>
Cc "Huang, Ying" <[email protected]>
Cc Kairui Song <[email protected]>
Cc Ryan Roberts <[email protected]>
---
 mm/page_io.c | 30 +++++++++++++++++-------------
 1 file changed, 17 insertions(+), 13 deletions(-)
mm/page_io.c +17 -13
index 5d9b6e6cf96c..90c5ea870038 100644
@@ -226,6 +226,19 @@ static void swap_zeromap_folio_clear(struct folio *folio)
 	}
 }
 
+static inline void count_swpout_vm_event(struct folio *folio)
+{
+#ifdef CONFIG_TRANSPARENT_HUGEPAGE
+	if (unlikely(folio_test_pmd_mappable(folio))) {
+		count_memcg_folio_events(folio, THP_SWPOUT, 1);
+		count_vm_event(THP_SWPOUT);
+	}
+#endif
+	count_mthp_stat(folio_order(folio), MTHP_STAT_SWPOUT);
+	count_memcg_folio_events(folio, PSWPOUT, folio_nr_pages(folio));
+	count_vm_events(PSWPOUT, folio_nr_pages(folio));
+}
+
 /*
  * We may have stale swap cache pages in memory: notice
  * them here and get rid of the unnecessary final write.
@@ -258,6 +271,7 @@ int swap_writepage(struct page *page, struct writeback_control *wbc)
 	 */
 	if (is_folio_zero_filled(folio)) {
 		swap_zeromap_folio_set(folio);
+		count_swpout_vm_event(folio);
 		folio_unlock(folio);
 		return 0;
 	} else {
@@ -282,19 +296,6 @@ int swap_writepage(struct page *page, struct writeback_control *wbc)
 	return 0;
 }
 
-static inline void count_swpout_vm_event(struct folio *folio)
-{
-#ifdef CONFIG_TRANSPARENT_HUGEPAGE
-	if (unlikely(folio_test_pmd_mappable(folio))) {
-		count_memcg_folio_events(folio, THP_SWPOUT, 1);
-		count_vm_event(THP_SWPOUT);
-	}
-#endif
-	count_mthp_stat(folio_order(folio), MTHP_STAT_SWPOUT);
-	count_memcg_folio_events(folio, PSWPOUT, folio_nr_pages(folio));
-	count_vm_events(PSWPOUT, folio_nr_pages(folio));
-}
-
 #if defined(CONFIG_MEMCG) && defined(CONFIG_BLK_CGROUP)
 static void bio_associate_blkg_from_page(struct bio *bio, struct folio *folio)
 {
@@ -621,6 +622,9 @@ void swap_read_folio(struct folio *folio, struct swap_iocb **plug)
 	delayacct_swapin_start();
 
 	if (swap_read_folio_zeromap(folio)) {
+		count_mthp_stat(folio_order(folio), MTHP_STAT_SWPIN);
+		count_memcg_folio_events(folio, PSWPIN, folio_nr_pages(folio));
+		count_vm_events(PSWPIN, folio_nr_pages(folio));
 		folio_unlock(folio);
 		goto finish;
 	} else if (zswap_load(folio)) {
Signature · 1 line
2.39.3 (Apple Git-146)
reviewer
#1 lore raw
Quoted 10 lines · Barry Song

From: Barry Song <[email protected]>

When the proportion of folios from the zero map is small, missing their accounting may not significantly impact profiling. However, it’s easy to construct a scenario where this becomes an issue—for example, allocating 1 GB of memory, writing zeros from userspace, followed by MADV_PAGEOUT, and then swapping it back in. In this case, the swap-out and swap-in counts seem to vanish into a black hole, potentially causing semantic ambiguity.

I agree. It also makes developing around this area more challenging. I'm working on the swap abstraction, and sometimes I can't tell if I screwed up somewhere, or if a proportion of these allocated entries go towards this optimization...

Thanks for taking a stab at fixing this, Barry!

Quoted 7 lines

We have two ways to address this:

1. Add a separate counter specifically for the zero map.
2. Continue using the current accounting, treating the zero map like a normal backend. (This aligns with the current behavior of zRAM when supporting same-page fills at the device level.)

Hmm, my understanding of the pswpout/pswpin counters is that they only apply to IO done directly to the backend device, no? That's why we have a separate set of counters for zswap, and do not count them towards pswp(in|out).

For users who have swap files on physical disks, the performance difference between reading directly from the swapfile and going through these optimizations could be really large. I think it makes sense to have a separate set of counters for zero-mapped pages (ideally, both at the host level and at the cgroup level?)

author
#2 lore raw
Quoted 33 lines · Nhat Pham
Quoted 10 lines · Barry Song

From: Barry Song <[email protected]>

When the proportion of folios from the zero map is small, missing their accounting may not significantly impact profiling. However, it’s easy to construct a scenario where this becomes an issue—for example, allocating 1 GB of memory, writing zeros from userspace, followed by MADV_PAGEOUT, and then swapping it back in. In this case, the swap-out and swap-in counts seem to vanish into a black hole, potentially causing semantic ambiguity.

I agree. It also makes developing around this area more challenging. I'm working on the swap abstraction, and sometimes I can't tell if I screwed up somewhere, or if a proportion of these allocated entries go towards this optimization...

Thanks for taking a stab at fixing this, Barry!

Quoted 7 lines

We have two ways to address this:

1. Add a separate counter specifically for the zero map.
2. Continue using the current accounting, treating the zero map like a normal backend. (This aligns with the current behavior of zRAM when supporting same-page fills at the device level.)

Hmm, my understanding of the pswpout/pswpin counters is that they only apply to IO done directly to the backend device, no? That's why we have a separate set of counters for zswap, and do not count them towards pswp(in|out).

Quoted 5 lines

For users who have swap files on physical disks, the performance difference between reading directly from the swapfile and going through these optimizations could be really large. I think it makes sense to have a separate set of counters for zero-mapped pages (ideally, both at the host level and at the cgroup level?)

agree it is better to have a separate counter for zeromap. then it raises a question: what is the proper name for it :-)

zeromap_swpin, zeromap_swpout seems too long? and zswpin and zswpout have been used by zswap

Thanks
barry

#3 lore raw
Quoted 40 lines · Barry Song
Quoted 33 lines · Nhat Pham
Quoted 10 lines · Barry Song

From: Barry Song <[email protected]>

When the proportion of folios from the zero map is small, missing their accounting may not significantly impact profiling. However, it’s easy to construct a scenario where this becomes an issue—for example, allocating 1 GB of memory, writing zeros from userspace, followed by MADV_PAGEOUT, and then swapping it back in. In this case, the swap-out and swap-in counts seem to vanish into a black hole, potentially causing semantic ambiguity.

I agree. It also makes developing around this area more challenging. I'm working on the swap abstraction, and sometimes I can't tell if I screwed up somewhere, or if a proportion of these allocated entries go towards this optimization...

Thanks for taking a stab at fixing this, Barry!

Quoted 7 lines

We have two ways to address this:

1. Add a separate counter specifically for the zero map.
2. Continue using the current accounting, treating the zero map like a normal backend. (This aligns with the current behavior of zRAM when supporting same-page fills at the device level.)

Hmm, my understanding of the pswpout/pswpin counters is that they only apply to IO done directly to the backend device, no? That's why we have a separate set of counters for zswap, and do not count them towards pswp(in|out).

Quoted 5 lines

For users who have swap files on physical disks, the performance difference between reading directly from the swapfile and going through these optimizations could be really large. I think it makes sense to have a separate set of counters for zero-mapped pages (ideally, both at the host level and at the cgroup level?)

Quoted 8 lines

agree it is better to have a separate counter for zeromap. then it raises a question: what is the proper name for it :-)

zeromap_swpin, zeromap_swpout seems too long? and zswpin and zswpout have been used by zswap

Thanks
barry

I wonder if instead of having counters, it might be better to keep track of the number of zeropages currently stored in zeromap, similar to how zswap_same_filled_pages did it. It will be more complicated then this patch, but would give more insight of the current state of the system.

Joshua (in CC) was going to have a look at that.

Thanks,
Usama

reviewer
#4 lore raw
Quoted 7 lines · Usama Arif

I wonder if instead of having counters, it might be better to keep track of the number of zeropages currently stored in zeromap, similar to how zswap_same_filled_pages did it. It will be more complicated then this patch, but would give more insight of the current state of the system.

Joshua (in CC) was going to have a look at that.

I don't think one can substitute for the other.

The "current zeromapped page" counter gives you a breakdown of where memory resides, whereas the in/out counters explain past performance based on events that have happened. That's why you have zswap, zswapped, zswpout, zswpin.

#5 lore raw
Quoted 10 lines · Nhat Pham
Quoted 7 lines · Usama Arif

I wonder if instead of having counters, it might be better to keep track of the number of zeropages currently stored in zeromap, similar to how zswap_same_filled_pages did it. It will be more complicated then this patch, but would give more insight of the current state of the system.

Joshua (in CC) was going to have a look at that.

I don't think one can substitute for the other.

Yes agreed, they have separate uses and provide different information, but maybe wasteful to have both types of counters? They are counters so maybe dont consume too much resources but I think we should still think about it..

If you think from a production context, I feel like the number of pages currently in zeromap could be more useful than just accumulation of all zeromap loads/stores, which after days is just going to be a very large number. You can compare the accumulations at different points in time, but they dont take into account freeing swap slots and swap_reclaim.

If we are open to having both types then its ok. But might be good to have that discussion here.

Quoted 5 lines

The "current zeromapped page" counter gives you a breakdown of where memory resides, whereas the in/out counters explain past performance based on events that have happened. That's why you have zswap, zswapped, zswpout, zswpin.

#6 lore raw
Quoted 11 lines · Usama Arif
Quoted 7 lines · Nhat Pham
Quoted 6 lines · Usama Arif

I wonder if instead of having counters, it might be better to keep track of the number of zeropages currently stored in zeromap, similar to how zswap_same_filled_pages did it. It will be more complicated then this patch, but would give more insight of the current state of the system.

Quoted 8 lines
Quoted 3 lines
Quoted 1 line

Joshua (in CC) was going to have a look at that.

I don't think one can substitute for the other.

Yes agreed, they have separate uses and provide different information, but maybe wasteful to have both types of counters? They are counters so maybe dont consume too much resources but I think we should still think about it..

Not for or against here, but I would say that statement is debatable at best for memcg stats :)

Each new counter consumes 2 longs per-memcg per-CPU (see memcg_vmstats_percpu), about 16 bytes, which is not a lot but it can quickly add up with a large number of CPUs/memcgs/stats.

Also, when flushing the stats we iterate all of them to propagate updates from per-CPU counters. This is already a slowpath so adding one stat is not a big deal, but again because we iterate all stats on multiple CPUs (and sometimes on each node as well), the overall flush latency becomes a concern sometimes.

All of that is not to say we shouldn't add more memcg stats, but we have to be mindful of the resources.

#7 lore raw
Quoted 28 lines · Yosry Ahmed
Quoted 11 lines · Usama Arif
Quoted 7 lines · Nhat Pham
Quoted 6 lines · Usama Arif

I wonder if instead of having counters, it might be better to keep track of the number of zeropages currently stored in zeromap, similar to how zswap_same_filled_pages did it. It will be more complicated then this patch, but would give more insight of the current state of the system.

Quoted 8 lines
Quoted 3 lines
Quoted 1 line

Joshua (in CC) was going to have a look at that.

I don't think one can substitute for the other.

Yes agreed, they have separate uses and provide different information, but maybe wasteful to have both types of counters? They are counters so maybe dont consume too much resources but I think we should still think about it..

Not for or against here, but I would say that statement is debatable at best for memcg stats :)

Each new counter consumes 2 longs per-memcg per-CPU (see memcg_vmstats_percpu), about 16 bytes, which is not a lot but it can quickly add up with a large number of CPUs/memcgs/stats.

Quoted 8 lines

Also, when flushing the stats we iterate all of them to propagate updates from per-CPU counters. This is already a slowpath so adding one stat is not a big deal, but again because we iterate all stats on multiple CPUs (and sometimes on each node as well), the overall flush latency becomes a concern sometimes.

All of that is not to say we shouldn't add more memcg stats, but we have to be mindful of the resources.

Yes agreed! Plus the cost of incrementing similar counters (which ofcourse is also not much).

Not trying to block this patch in anyway. Just think its a good point to discuss here if we are ok with both types of counters. If its too wasteful then which one we should have.

author
#8 lore raw
Quoted 41 lines · Usama Arif
Quoted 28 lines · Yosry Ahmed
Quoted 11 lines · Usama Arif
Quoted 7 lines · Nhat Pham
Quoted 6 lines · Usama Arif

I wonder if instead of having counters, it might be better to keep track of the number of zeropages currently stored in zeromap, similar to how zswap_same_filled_pages did it. It will be more complicated then this patch, but would give more insight of the current state of the system.

Quoted 8 lines
Quoted 3 lines
Quoted 1 line

Joshua (in CC) was going to have a look at that.

I don't think one can substitute for the other.

Yes agreed, they have separate uses and provide different information, but maybe wasteful to have both types of counters? They are counters so maybe dont consume too much resources but I think we should still think about it..

Not for or against here, but I would say that statement is debatable at best for memcg stats :)

Each new counter consumes 2 longs per-memcg per-CPU (see memcg_vmstats_percpu), about 16 bytes, which is not a lot but it can quickly add up with a large number of CPUs/memcgs/stats.

Quoted 8 lines

Also, when flushing the stats we iterate all of them to propagate updates from per-CPU counters. This is already a slowpath so adding one stat is not a big deal, but again because we iterate all stats on multiple CPUs (and sometimes on each node as well), the overall flush latency becomes a concern sometimes.

All of that is not to say we shouldn't add more memcg stats, but we have to be mindful of the resources.

Quoted 6 lines

Yes agreed! Plus the cost of incrementing similar counters (which ofcourse is also not much).

Not trying to block this patch in anyway. Just think its a good point to discuss here if we are ok with both types of counters. If its too wasteful then which one we should have.

Hi Usama,
my point is that with all the below three counters:
1. PSWPIN/PSWPOUT
2. ZSWPIN/ZSWPOUT
3. SWAPIN_SKIP/SWAPOUT_SKIP or (ZEROSWPIN, ZEROSWPOUT what ever)

Shouldn't we have been able to determine the portion of zeromap swap indirectly?

Thanks
Barry

#9 lore raw
Quoted 50 lines · Barry Song
Quoted 41 lines · Usama Arif
Quoted 28 lines · Yosry Ahmed
Quoted 11 lines · Usama Arif
Quoted 7 lines · Nhat Pham
Quoted 6 lines · Usama Arif

I wonder if instead of having counters, it might be better to keep track of the number of zeropages currently stored in zeromap, similar to how zswap_same_filled_pages did it. It will be more complicated then this patch, but would give more insight of the current state of the system.

Quoted 8 lines
Quoted 3 lines
Quoted 1 line

Joshua (in CC) was going to have a look at that.

I don't think one can substitute for the other.

Yes agreed, they have separate uses and provide different information, but maybe wasteful to have both types of counters? They are counters so maybe dont consume too much resources but I think we should still think about it..

Not for or against here, but I would say that statement is debatable at best for memcg stats :)

Each new counter consumes 2 longs per-memcg per-CPU (see memcg_vmstats_percpu), about 16 bytes, which is not a lot but it can quickly add up with a large number of CPUs/memcgs/stats.

Quoted 8 lines

Also, when flushing the stats we iterate all of them to propagate updates from per-CPU counters. This is already a slowpath so adding one stat is not a big deal, but again because we iterate all stats on multiple CPUs (and sometimes on each node as well), the overall flush latency becomes a concern sometimes.

All of that is not to say we shouldn't add more memcg stats, but we have to be mindful of the resources.

Quoted 6 lines

Yes agreed! Plus the cost of incrementing similar counters (which ofcourse is also not much).

Not trying to block this patch in anyway. Just think its a good point to discuss here if we are ok with both types of counters. If its too wasteful then which one we should have.

Quoted 8 lines

Hi Usama,
my point is that with all the below three counters:
1. PSWPIN/PSWPOUT
2. ZSWPIN/ZSWPOUT
3. SWAPIN_SKIP/SWAPOUT_SKIP or (ZEROSWPIN, ZEROSWPOUT what ever)

Shouldn't we have been able to determine the portion of zeromap swap indirectly?

What about swap entries that get freed without being swapped in (e.g. swapped out anon memory freed, MADV_FREE, shmem truncate, etc)?

Quoted 3 lines

Thanks
Barry

#10 lore raw
In reply to Barry Song
Quoted 49 lines · Barry Song
Quoted 41 lines · Usama Arif
Quoted 28 lines · Yosry Ahmed
Quoted 11 lines · Usama Arif
Quoted 7 lines · Nhat Pham
Quoted 6 lines · Usama Arif

I wonder if instead of having counters, it might be better to keep track of the number of zeropages currently stored in zeromap, similar to how zswap_same_filled_pages did it. It will be more complicated then this patch, but would give more insight of the current state of the system.

Quoted 8 lines
Quoted 3 lines
Quoted 1 line

Joshua (in CC) was going to have a look at that.

I don't think one can substitute for the other.

Yes agreed, they have separate uses and provide different information, but maybe wasteful to have both types of counters? They are counters so maybe dont consume too much resources but I think we should still think about it..

Not for or against here, but I would say that statement is debatable at best for memcg stats :)

Each new counter consumes 2 longs per-memcg per-CPU (see memcg_vmstats_percpu), about 16 bytes, which is not a lot but it can quickly add up with a large number of CPUs/memcgs/stats.

Quoted 8 lines

Also, when flushing the stats we iterate all of them to propagate updates from per-CPU counters. This is already a slowpath so adding one stat is not a big deal, but again because we iterate all stats on multiple CPUs (and sometimes on each node as well), the overall flush latency becomes a concern sometimes.

All of that is not to say we shouldn't add more memcg stats, but we have to be mindful of the resources.

Quoted 6 lines

Yes agreed! Plus the cost of incrementing similar counters (which ofcourse is also not much).

Not trying to block this patch in anyway. Just think its a good point to discuss here if we are ok with both types of counters. If its too wasteful then which one we should have.

Quoted 9 lines

Hi Usama,
my point is that with all the below three counters:
1. PSWPIN/PSWPOUT
2. ZSWPIN/ZSWPOUT
3. SWAPIN_SKIP/SWAPOUT_SKIP or (ZEROSWPIN, ZEROSWPOUT what ever)

Shouldn't we have been able to determine the portion of zeromap swap indirectly?

Hmm, I might be wrong, but I would have thought no?

What if you swapout a zero folio, but then discard it? zeromap_swpout would be incremented, but zeromap_swapin would not.

Quoted 2 lines

Thanks
Barry

author
#11 lore raw
Quoted 59 lines · Usama Arif
Quoted 55 lines · Barry Song
Quoted 41 lines · Usama Arif
Quoted 28 lines · Yosry Ahmed
Quoted 11 lines · Usama Arif
Quoted 7 lines · Nhat Pham
Quoted 6 lines · Usama Arif

I wonder if instead of having counters, it might be better to keep track of the number of zeropages currently stored in zeromap, similar to how zswap_same_filled_pages did it. It will be more complicated then this patch, but would give more insight of the current state of the system.

Quoted 8 lines
Quoted 3 lines
Quoted 1 line

Joshua (in CC) was going to have a look at that.

I don't think one can substitute for the other.

Yes agreed, they have separate uses and provide different information, but maybe wasteful to have both types of counters? They are counters so maybe dont consume too much resources but I think we should still think about it..

Not for or against here, but I would say that statement is debatable at best for memcg stats :)

Each new counter consumes 2 longs per-memcg per-CPU (see memcg_vmstats_percpu), about 16 bytes, which is not a lot but it can quickly add up with a large number of CPUs/memcgs/stats.

Quoted 8 lines

Also, when flushing the stats we iterate all of them to propagate updates from per-CPU counters. This is already a slowpath so adding one stat is not a big deal, but again because we iterate all stats on multiple CPUs (and sometimes on each node as well), the overall flush latency becomes a concern sometimes.

All of that is not to say we shouldn't add more memcg stats, but we have to be mindful of the resources.

Quoted 6 lines

Yes agreed! Plus the cost of incrementing similar counters (which ofcourse is also not much).

Not trying to block this patch in anyway. Just think its a good point to discuss here if we are ok with both types of counters. If its too wasteful then which one we should have.

Hi Usama,
my point is that with all the below three counters:
1. PSWPIN/PSWPOUT
2. ZSWPIN/ZSWPOUT
3. SWAPIN_SKIP/SWAPOUT_SKIP or (ZEROSWPIN, ZEROSWPOUT what ever)

Quoted 8 lines
Quoted 3 lines

Shouldn't we have been able to determine the portion of zeromap swap indirectly?

Hmm, I might be wrong, but I would have thought no?

What if you swapout a zero folio, but then discard it? zeromap_swpout would be incremented, but zeromap_swapin would not.

I understand. It looks like we have two issues to tackle:
1. We shouldn't let zeromap swap in or out anything that vanishes into a black hole
2. We want to find out how much I/O/memory has been saved due to zeromap so far

From my perspective, issue 1 requires a "fix", while issue 2 is more of an optimization.

I consider issue 1 to be more critical because, after observing a phone running for some time, I've been able to roughly estimate the portion zeromap can
help save using only PSWPOUT, ZSWPOUT, and SWAPOUT_SKIP, even without a SWPIN counter. However, I agree that issue 2 still holds significant value as a separate patch.

Thanks
Barry

#12 lore raw
Quoted 69 lines · Barry Song
Quoted 59 lines · Usama Arif
Quoted 55 lines · Barry Song
Quoted 41 lines · Usama Arif
Quoted 28 lines · Yosry Ahmed
Quoted 11 lines · Usama Arif
Quoted 7 lines · Nhat Pham
Quoted 6 lines · Usama Arif

I wonder if instead of having counters, it might be better to keep track of the number of zeropages currently stored in zeromap, similar to how zswap_same_filled_pages did it. It will be more complicated then this patch, but would give more insight of the current state of the system.

Quoted 8 lines
Quoted 3 lines
Quoted 1 line

Joshua (in CC) was going to have a look at that.

I don't think one can substitute for the other.

Yes agreed, they have separate uses and provide different information, but maybe wasteful to have both types of counters? They are counters so maybe dont consume too much resources but I think we should still think about it..

Not for or against here, but I would say that statement is debatable at best for memcg stats :)

Each new counter consumes 2 longs per-memcg per-CPU (see memcg_vmstats_percpu), about 16 bytes, which is not a lot but it can quickly add up with a large number of CPUs/memcgs/stats.

Quoted 8 lines

Also, when flushing the stats we iterate all of them to propagate updates from per-CPU counters. This is already a slowpath so adding one stat is not a big deal, but again because we iterate all stats on multiple CPUs (and sometimes on each node as well), the overall flush latency becomes a concern sometimes.

All of that is not to say we shouldn't add more memcg stats, but we have to be mindful of the resources.

Quoted 6 lines

Yes agreed! Plus the cost of incrementing similar counters (which ofcourse is also not much).

Not trying to block this patch in anyway. Just think its a good point to discuss here if we are ok with both types of counters. If its too wasteful then which one we should have.

Hi Usama,
my point is that with all the below three counters:
1. PSWPIN/PSWPOUT
2. ZSWPIN/ZSWPOUT
3. SWAPIN_SKIP/SWAPOUT_SKIP or (ZEROSWPIN, ZEROSWPOUT what ever)

Quoted 8 lines
Quoted 3 lines

Shouldn't we have been able to determine the portion of zeromap swap indirectly?

Hmm, I might be wrong, but I would have thought no?

What if you swapout a zero folio, but then discard it? zeromap_swpout would be incremented, but zeromap_swapin would not.

Quoted 7 lines

I understand. It looks like we have two issues to tackle:
1. We shouldn't let zeromap swap in or out anything that vanishes into a black hole
2. We want to find out how much I/O/memory has been saved due to zeromap so far

From my perspective, issue 1 requires a "fix", while issue 2 is more of an optimization.

Hmm I dont understand why point 1 would be an issue.

If its discarded thats fine as far as I can see.

As a reference, memory.stat.zswapped != memory.stat.zswapout - memory.stat.zswapin. Because zswapped would take into account swapped out anon memory freed, MADV_FREE, shmem truncate, etc as Yosry said about zeromap, But zswapout and zswapin dont.

Quoted 10 lines

I consider issue 1 to be more critical because, after observing a phone running for some time, I've been able to roughly estimate the portion zeromap can
help save using only PSWPOUT, ZSWPOUT, and SWAPOUT_SKIP, even without a SWPIN counter. However, I agree that issue 2 still holds significant value as a separate patch.

Thanks
Barry

author
#13 lore raw
Quoted 78 lines · Usama Arif
Quoted 74 lines · Barry Song
Quoted 59 lines · Usama Arif
Quoted 55 lines · Barry Song
Quoted 41 lines · Usama Arif
Quoted 28 lines · Yosry Ahmed
Quoted 11 lines · Usama Arif
Quoted 7 lines · Nhat Pham
Quoted 6 lines · Usama Arif

I wonder if instead of having counters, it might be better to keep track of the number of zeropages currently stored in zeromap, similar to how zswap_same_filled_pages did it. It will be more complicated then this patch, but would give more insight of the current state of the system.

Quoted 8 lines
Quoted 3 lines
Quoted 1 line

Joshua (in CC) was going to have a look at that.

I don't think one can substitute for the other.

Yes agreed, they have separate uses and provide different information, but maybe wasteful to have both types of counters? They are counters so maybe dont consume too much resources but I think we should still think about it..

Not for or against here, but I would say that statement is debatable at best for memcg stats :)

Each new counter consumes 2 longs per-memcg per-CPU (see memcg_vmstats_percpu), about 16 bytes, which is not a lot but it can quickly add up with a large number of CPUs/memcgs/stats.

Quoted 8 lines

Also, when flushing the stats we iterate all of them to propagate updates from per-CPU counters. This is already a slowpath so adding one stat is not a big deal, but again because we iterate all stats on multiple CPUs (and sometimes on each node as well), the overall flush latency becomes a concern sometimes.

All of that is not to say we shouldn't add more memcg stats, but we have to be mindful of the resources.

Quoted 6 lines

Yes agreed! Plus the cost of incrementing similar counters (which ofcourse is also not much).

Not trying to block this patch in anyway. Just think its a good point to discuss here if we are ok with both types of counters. If its too wasteful then which one we should have.

Hi Usama,
my point is that with all the below three counters:
1. PSWPIN/PSWPOUT
2. ZSWPIN/ZSWPOUT
3. SWAPIN_SKIP/SWAPOUT_SKIP or (ZEROSWPIN, ZEROSWPOUT what ever)

Quoted 8 lines
Quoted 3 lines

Shouldn't we have been able to determine the portion of zeromap swap indirectly?

Hmm, I might be wrong, but I would have thought no?

What if you swapout a zero folio, but then discard it? zeromap_swpout would be incremented, but zeromap_swapin would not.

I understand. It looks like we have two issues to tackle:
1. We shouldn't let zeromap swap in or out anything that vanishes into a black hole
2. We want to find out how much I/O/memory has been saved due to zeromap so far

Quoted 6 lines
Quoted 2 lines

From my perspective, issue 1 requires a "fix", while issue 2 is more of an optimization.

Hmm I dont understand why point 1 would be an issue.

If its discarded thats fine as far as I can see.

it is fine to you and probably me who knows zeromap as well :-) but any userspace code
as below might be entirely confused:

p = malloc(1G);
write p to 0; or write part of p to 0
madv_pageout(p, 1g)
read p to swapin.

The entire procedure used to involve 1GB of swap out and 1GB of swap in by any means. Now, it has recorded 0 swaps counted.

I don't expect userspace is as smart as you :-)

Quoted 4 lines

As a reference, memory.stat.zswapped != memory.stat.zswapout - memory.stat.zswapin. Because zswapped would take into account swapped out anon memory freed, MADV_FREE, shmem truncate, etc as Yosry said about zeromap, But zswapout and zswapin dont.

I understand. However, I believe what we really need to focus on is
this: if we’ve
swapped out, for instance, 100GB in the past hour, how much of that 100GB is zero? This information can help us assess the proportion of zero data in the workload, along with the potential benefits that zeromap can provide for memory, I/O space, or read/write operations. Additionally, having the second count can enhance accuracy when considering MADV_DONTNEED, FREE, TRUNCATE, and so on.

Quoted 10 lines
Quoted 10 lines

I consider issue 1 to be more critical because, after observing a phone running for some time, I've been able to roughly estimate the portion zeromap can
help save using only PSWPOUT, ZSWPOUT, and SWAPOUT_SKIP, even without a SWPIN counter. However, I agree that issue 2 still holds significant value as a separate patch.

Thanks
Barry

#14 lore raw
Quoted 95 lines · Barry Song
Quoted 78 lines · Usama Arif
Quoted 74 lines · Barry Song
Quoted 59 lines · Usama Arif
Quoted 55 lines · Barry Song
Quoted 41 lines · Usama Arif
Quoted 28 lines · Yosry Ahmed
Quoted 11 lines · Usama Arif
Quoted 7 lines · Nhat Pham
Quoted 6 lines · Usama Arif

I wonder if instead of having counters, it might be better to keep track of the number of zeropages currently stored in zeromap, similar to how zswap_same_filled_pages did it. It will be more complicated then this patch, but would give more insight of the current state of the system.

Quoted 8 lines
Quoted 3 lines
Quoted 1 line

Joshua (in CC) was going to have a look at that.

I don't think one can substitute for the other.

Yes agreed, they have separate uses and provide different information, but maybe wasteful to have both types of counters? They are counters so maybe dont consume too much resources but I think we should still think about it..

Not for or against here, but I would say that statement is debatable at best for memcg stats :)

Each new counter consumes 2 longs per-memcg per-CPU (see memcg_vmstats_percpu), about 16 bytes, which is not a lot but it can quickly add up with a large number of CPUs/memcgs/stats.

Quoted 8 lines

Also, when flushing the stats we iterate all of them to propagate updates from per-CPU counters. This is already a slowpath so adding one stat is not a big deal, but again because we iterate all stats on multiple CPUs (and sometimes on each node as well), the overall flush latency becomes a concern sometimes.

All of that is not to say we shouldn't add more memcg stats, but we have to be mindful of the resources.

Quoted 6 lines

Yes agreed! Plus the cost of incrementing similar counters (which ofcourse is also not much).

Not trying to block this patch in anyway. Just think its a good point to discuss here if we are ok with both types of counters. If its too wasteful then which one we should have.

Hi Usama,
my point is that with all the below three counters:
1. PSWPIN/PSWPOUT
2. ZSWPIN/ZSWPOUT
3. SWAPIN_SKIP/SWAPOUT_SKIP or (ZEROSWPIN, ZEROSWPOUT what ever)

Quoted 8 lines
Quoted 3 lines

Shouldn't we have been able to determine the portion of zeromap swap indirectly?

Hmm, I might be wrong, but I would have thought no?

What if you swapout a zero folio, but then discard it? zeromap_swpout would be incremented, but zeromap_swapin would not.

I understand. It looks like we have two issues to tackle:
1. We shouldn't let zeromap swap in or out anything that vanishes into a black hole
2. We want to find out how much I/O/memory has been saved due to zeromap so far

Quoted 6 lines
Quoted 2 lines

From my perspective, issue 1 requires a "fix", while issue 2 is more of an optimization.

Hmm I dont understand why point 1 would be an issue.

If its discarded thats fine as far as I can see.

it is fine to you and probably me who knows zeromap as well :-) but any userspace code
as below might be entirely confused:

p = malloc(1G);
write p to 0; or write part of p to 0
madv_pageout(p, 1g)
read p to swapin.

Quoted 5 lines

The entire procedure used to involve 1GB of swap out and 1GB of swap in by any means. Now, it has recorded 0 swaps counted.

I don't expect userspace is as smart as you :-)

Ah I completely agree, we need to account for it in some metric. I probably misunderstood when you said "We shouldn't let zeromap swap in or out anything that vanishes into a black hole", by we should not have the zeromap optimization for those cases. What I guess you meant is we need to account for it in some metric.

Quoted 5 lines
Quoted 4 lines

As a reference, memory.stat.zswapped != memory.stat.zswapout - memory.stat.zswapin. Because zswapped would take into account swapped out anon memory freed, MADV_FREE, shmem truncate, etc as Yosry said about zeromap, But zswapout and zswapin dont.

Quoted 9 lines

I understand. However, I believe what we really need to focus on is
this: if we’ve
swapped out, for instance, 100GB in the past hour, how much of that 100GB is zero? This information can help us assess the proportion of zero data in the workload, along with the potential benefits that zeromap can provide for memory, I/O space, or read/write operations. Additionally, having the second count can enhance accuracy when considering MADV_DONTNEED, FREE, TRUNCATE, and so on.

Yes completely agree!

I think we can look into adding all three metrics, zeromap_swapped, zeromap_swpout, zeromap_swpin (or whatever name works).

Quoted 13 lines
Quoted 10 lines
Quoted 10 lines

I consider issue 1 to be more critical because, after observing a phone running for some time, I've been able to roughly estimate the portion zeromap can
help save using only PSWPOUT, ZSWPOUT, and SWAPOUT_SKIP, even without a SWPIN counter. However, I agree that issue 2 still holds significant value as a separate patch.

Thanks
Barry

author
#15 lore raw
Quoted 123 lines · Usama Arif
Quoted 95 lines · Barry Song
Quoted 78 lines · Usama Arif
Quoted 74 lines · Barry Song
Quoted 59 lines · Usama Arif
Quoted 55 lines · Barry Song
Quoted 41 lines · Usama Arif
Quoted 28 lines · Yosry Ahmed
Quoted 11 lines · Usama Arif
Quoted 7 lines · Nhat Pham
Quoted 6 lines · Usama Arif

I wonder if instead of having counters, it might be better to keep track of the number of zeropages currently stored in zeromap, similar to how zswap_same_filled_pages did it. It will be more complicated then this patch, but would give more insight of the current state of the system.

Quoted 8 lines
Quoted 3 lines
Quoted 1 line

Joshua (in CC) was going to have a look at that.

I don't think one can substitute for the other.

Yes agreed, they have separate uses and provide different information, but maybe wasteful to have both types of counters? They are counters so maybe dont consume too much resources but I think we should still think about it..

Not for or against here, but I would say that statement is debatable at best for memcg stats :)

Each new counter consumes 2 longs per-memcg per-CPU (see memcg_vmstats_percpu), about 16 bytes, which is not a lot but it can quickly add up with a large number of CPUs/memcgs/stats.

Quoted 8 lines

Also, when flushing the stats we iterate all of them to propagate updates from per-CPU counters. This is already a slowpath so adding one stat is not a big deal, but again because we iterate all stats on multiple CPUs (and sometimes on each node as well), the overall flush latency becomes a concern sometimes.

All of that is not to say we shouldn't add more memcg stats, but we have to be mindful of the resources.

Quoted 6 lines

Yes agreed! Plus the cost of incrementing similar counters (which ofcourse is also not much).

Not trying to block this patch in anyway. Just think its a good point to discuss here if we are ok with both types of counters. If its too wasteful then which one we should have.

Hi Usama,
my point is that with all the below three counters:
1. PSWPIN/PSWPOUT
2. ZSWPIN/ZSWPOUT
3. SWAPIN_SKIP/SWAPOUT_SKIP or (ZEROSWPIN, ZEROSWPOUT what ever)

Quoted 8 lines
Quoted 3 lines

Shouldn't we have been able to determine the portion of zeromap swap indirectly?

Hmm, I might be wrong, but I would have thought no?

What if you swapout a zero folio, but then discard it? zeromap_swpout would be incremented, but zeromap_swapin would not.

I understand. It looks like we have two issues to tackle:
1. We shouldn't let zeromap swap in or out anything that vanishes into a black hole
2. We want to find out how much I/O/memory has been saved due to zeromap so far

Quoted 6 lines
Quoted 2 lines

From my perspective, issue 1 requires a "fix", while issue 2 is more of an optimization.

Hmm I dont understand why point 1 would be an issue.

If its discarded thats fine as far as I can see.

it is fine to you and probably me who knows zeromap as well :-) but any userspace code
as below might be entirely confused:

p = malloc(1G);
write p to 0; or write part of p to 0
madv_pageout(p, 1g)
read p to swapin.

Quoted 5 lines

The entire procedure used to involve 1GB of swap out and 1GB of swap in by any means. Now, it has recorded 0 swaps counted.

I don't expect userspace is as smart as you :-)

Ah I completely agree, we need to account for it in some metric. I probably misunderstood when you said "We shouldn't let zeromap swap in or out anything that vanishes into a black hole", by we should not have the zeromap optimization for those cases. What I guess you meant is we need to account for it in some metric.

Quoted 5 lines
Quoted 4 lines

As a reference, memory.stat.zswapped != memory.stat.zswapout - memory.stat.zswapin. Because zswapped would take into account swapped out anon memory freed, MADV_FREE, shmem truncate, etc as Yosry said about zeromap, But zswapout and zswapin dont.

Quoted 9 lines

I understand. However, I believe what we really need to focus on is
this: if we’ve
swapped out, for instance, 100GB in the past hour, how much of that 100GB is zero? This information can help us assess the proportion of zero data in the workload, along with the potential benefits that zeromap can provide for memory, I/O space, or read/write operations. Additionally, having the second count can enhance accuracy when considering MADV_DONTNEED, FREE, TRUNCATE, and so on.

Quoted 4 lines

Yes completely agree!

I think we can look into adding all three metrics, zeromap_swapped, zeromap_swpout, zeromap_swpin (or whatever name works).

It's great to reach an agreement. Let me work on some patches for it.

By the way, I recently had an idea: if we can conduct the zeromap check earlier - for example - before allocating swap slots and pageout(), could we completely eliminate swap slot occupation and allocation/release for zeromap data? For example, we could use a special swap entry value in the PTE to indicate zero content and directly fill it with zeros when swapping back. We've observed that swap slot allocation and freeing can consume a lot of CPU and slow down functions like zap_pte_range and swap-in. If we can entirely skip these steps, it could improve performance. However, I'm uncertain about the benefits we would gain if we only have 1-2% zeromap data.

I'm just putting this idea out there to see if you're interested in moving forward with it. :-)

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 8 lines
Quoted 8 lines
Quoted 7 lines
Quoted 7 lines

I consider issue 1 to be more critical because, after observing a phone running for some time, I've been able to roughly estimate the portion zeromap can
help save using only PSWPOUT, ZSWPOUT, and SWAPOUT_SKIP, even without a SWPIN counter. However, I agree that issue 2 still holds significant value as a separate patch.

Thanks
Barry

#16 lore raw
Quoted 124 lines · Barry Song
Quoted 123 lines · Usama Arif
Quoted 95 lines · Barry Song
Quoted 78 lines · Usama Arif
Quoted 74 lines · Barry Song
Quoted 59 lines · Usama Arif
Quoted 55 lines · Barry Song
Quoted 41 lines · Usama Arif
Quoted 28 lines · Yosry Ahmed
Quoted 11 lines · Usama Arif
Quoted 7 lines · Nhat Pham
Quoted 6 lines · Usama Arif

I wonder if instead of having counters, it might be better to keep track of the number of zeropages currently stored in zeromap, similar to how zswap_same_filled_pages did it. It will be more complicated then this patch, but would give more insight of the current state of the system.

Quoted 8 lines
Quoted 3 lines
Quoted 1 line

Joshua (in CC) was going to have a look at that.

I don't think one can substitute for the other.

Yes agreed, they have separate uses and provide different information, but maybe wasteful to have both types of counters? They are counters so maybe dont consume too much resources but I think we should still think about it..

Not for or against here, but I would say that statement is debatable at best for memcg stats :)

Each new counter consumes 2 longs per-memcg per-CPU (see memcg_vmstats_percpu), about 16 bytes, which is not a lot but it can quickly add up with a large number of CPUs/memcgs/stats.

Quoted 8 lines

Also, when flushing the stats we iterate all of them to propagate updates from per-CPU counters. This is already a slowpath so adding one stat is not a big deal, but again because we iterate all stats on multiple CPUs (and sometimes on each node as well), the overall flush latency becomes a concern sometimes.

All of that is not to say we shouldn't add more memcg stats, but we have to be mindful of the resources.

Quoted 6 lines

Yes agreed! Plus the cost of incrementing similar counters (which ofcourse is also not much).

Not trying to block this patch in anyway. Just think its a good point to discuss here if we are ok with both types of counters. If its too wasteful then which one we should have.

Hi Usama,
my point is that with all the below three counters:
1. PSWPIN/PSWPOUT
2. ZSWPIN/ZSWPOUT
3. SWAPIN_SKIP/SWAPOUT_SKIP or (ZEROSWPIN, ZEROSWPOUT what ever)

Quoted 8 lines
Quoted 3 lines

Shouldn't we have been able to determine the portion of zeromap swap indirectly?

Hmm, I might be wrong, but I would have thought no?

What if you swapout a zero folio, but then discard it? zeromap_swpout would be incremented, but zeromap_swapin would not.

I understand. It looks like we have two issues to tackle:
1. We shouldn't let zeromap swap in or out anything that vanishes into a black hole
2. We want to find out how much I/O/memory has been saved due to zeromap so far

Quoted 6 lines
Quoted 2 lines

From my perspective, issue 1 requires a "fix", while issue 2 is more of an optimization.

Hmm I dont understand why point 1 would be an issue.

If its discarded thats fine as far as I can see.

it is fine to you and probably me who knows zeromap as well :-) but any userspace code
as below might be entirely confused:

p = malloc(1G);
write p to 0; or write part of p to 0
madv_pageout(p, 1g)
read p to swapin.

Quoted 5 lines

The entire procedure used to involve 1GB of swap out and 1GB of swap in by any means. Now, it has recorded 0 swaps counted.

I don't expect userspace is as smart as you :-)

Ah I completely agree, we need to account for it in some metric. I probably misunderstood when you said "We shouldn't let zeromap swap in or out anything that vanishes into a black hole", by we should not have the zeromap optimization for those cases. What I guess you meant is we need to account for it in some metric.

Quoted 14 lines
Quoted 4 lines

As a reference, memory.stat.zswapped != memory.stat.zswapout - memory.stat.zswapin. Because zswapped would take into account swapped out anon memory freed, MADV_FREE, shmem truncate, etc as Yosry said about zeromap, But zswapout and zswapin dont.

I understand. However, I believe what we really need to focus on is
this: if we’ve
swapped out, for instance, 100GB in the past hour, how much of that 100GB is zero? This information can help us assess the proportion of zero data in the workload, along with the potential benefits that zeromap can provide for memory, I/O space, or read/write operations. Additionally, having the second count can enhance accuracy when considering MADV_DONTNEED, FREE, TRUNCATE, and so on.

Quoted 6 lines
Quoted 4 lines

Yes completely agree!

I think we can look into adding all three metrics, zeromap_swapped, zeromap_swpout, zeromap_swpin (or whatever name works).

It's great to reach an agreement. Let me work on some patches for it.

Thanks!

Quoted 11 lines

By the way, I recently had an idea: if we can conduct the zeromap check earlier - for example - before allocating swap slots and pageout(), could we completely eliminate swap slot occupation and allocation/release for zeromap data? For example, we could use a special swap entry value in the PTE to indicate zero content and directly fill it with zeros when swapping back. We've observed that swap slot allocation and freeing can consume a lot of CPU and slow down functions like zap_pte_range and swap-in. If we can entirely skip these steps, it could improve performance. However, I'm uncertain about the benefits we would gain if we only have 1-2% zeromap data.

If I remember correctly this was one of the ideas floated around in the initial version of the zeromap series, but it was evaluated as a lot more complicated to do than what the current zeromap code looks like. But I think its definitely worth looking into!

Quoted 19 lines

I'm just putting this idea out there to see if you're interested in moving forward with it. :-)

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 8 lines
Quoted 8 lines
Quoted 7 lines
Quoted 7 lines

I consider issue 1 to be more critical because, after observing a phone running for some time, I've been able to roughly estimate the portion zeromap can
help save using only PSWPOUT, ZSWPOUT, and SWAPOUT_SKIP, even without a SWPIN counter. However, I agree that issue 2 still holds significant value as a separate patch.

Thanks
Barry

author
#17 lore raw
Quoted 149 lines · Usama Arif
Quoted 124 lines · Barry Song
Quoted 123 lines · Usama Arif
Quoted 95 lines · Barry Song
Quoted 78 lines · Usama Arif
Quoted 74 lines · Barry Song
Quoted 59 lines · Usama Arif
Quoted 55 lines · Barry Song
Quoted 41 lines · Usama Arif
Quoted 28 lines · Yosry Ahmed
Quoted 11 lines · Usama Arif
Quoted 7 lines · Nhat Pham
Quoted 6 lines · Usama Arif

I wonder if instead of having counters, it might be better to keep track
of the number of zeropages currently stored in zeromap, similar to how
zswap_same_filled_pages did it. It will be more complicated then this
patch, but would give more insight of the current state of the system.

Quoted 8 lines
Quoted 3 lines
Quoted 1 line

Joshua (in CC) was going to have a look at that.

I don't think one can substitute for the other.

Yes agreed, they have separate uses and provide different information, but maybe wasteful to have both types of counters? They are counters so maybe dont consume too much resources but I think we should still think about it..

Not for or against here, but I would say that statement is debatable at best for memcg stats :)

Each new counter consumes 2 longs per-memcg per-CPU (see memcg_vmstats_percpu), about 16 bytes, which is not a lot but it can quickly add up with a large number of CPUs/memcgs/stats.

Quoted 8 lines

Also, when flushing the stats we iterate all of them to propagate updates from per-CPU counters. This is already a slowpath so adding one stat is not a big deal, but again because we iterate all stats on multiple CPUs (and sometimes on each node as well), the overall flush latency becomes a concern sometimes.

All of that is not to say we shouldn't add more memcg stats, but we have to be mindful of the resources.

Quoted 6 lines

Yes agreed! Plus the cost of incrementing similar counters (which ofcourse is also not much).

Not trying to block this patch in anyway. Just think its a good point to discuss here if we are ok with both types of counters. If its too wasteful then which one we should have.

Hi Usama,
my point is that with all the below three counters:
1. PSWPIN/PSWPOUT
2. ZSWPIN/ZSWPOUT
3. SWAPIN_SKIP/SWAPOUT_SKIP or (ZEROSWPIN, ZEROSWPOUT what ever)

Quoted 8 lines
Quoted 3 lines

Shouldn't we have been able to determine the portion of zeromap swap indirectly?

Hmm, I might be wrong, but I would have thought no?

What if you swapout a zero folio, but then discard it? zeromap_swpout would be incremented, but zeromap_swapin would not.

I understand. It looks like we have two issues to tackle:
1. We shouldn't let zeromap swap in or out anything that vanishes into a black hole
2. We want to find out how much I/O/memory has been saved due to zeromap so far

Quoted 6 lines
Quoted 2 lines

From my perspective, issue 1 requires a "fix", while issue 2 is more of an optimization.

Hmm I dont understand why point 1 would be an issue.

If its discarded thats fine as far as I can see.

it is fine to you and probably me who knows zeromap as well :-) but any userspace code
as below might be entirely confused:

p = malloc(1G);
write p to 0; or write part of p to 0
madv_pageout(p, 1g)
read p to swapin.

Quoted 5 lines

The entire procedure used to involve 1GB of swap out and 1GB of swap in by any means. Now, it has recorded 0 swaps counted.

I don't expect userspace is as smart as you :-)

Ah I completely agree, we need to account for it in some metric. I probably misunderstood when you said "We shouldn't let zeromap swap in or out anything that vanishes into a black hole", by we should not have the zeromap optimization for those cases. What I guess you meant is we need to account for it in some metric.

Quoted 14 lines
Quoted 4 lines

As a reference, memory.stat.zswapped != memory.stat.zswapout - memory.stat.zswapin. Because zswapped would take into account swapped out anon memory freed, MADV_FREE, shmem truncate, etc as Yosry said about zeromap, But zswapout and zswapin dont.

I understand. However, I believe what we really need to focus on is
this: if we’ve
swapped out, for instance, 100GB in the past hour, how much of that 100GB is zero? This information can help us assess the proportion of zero data in the workload, along with the potential benefits that zeromap can provide for memory, I/O space, or read/write operations. Additionally, having the second count can enhance accuracy when considering MADV_DONTNEED, FREE, TRUNCATE, and so on.

Quoted 6 lines
Quoted 4 lines

Yes completely agree!

I think we can look into adding all three metrics, zeromap_swapped, zeromap_swpout, zeromap_swpin (or whatever name works).

It's great to reach an agreement. Let me work on some patches for it.

Thanks!

Quoted 11 lines

By the way, I recently had an idea: if we can conduct the zeromap check earlier - for example - before allocating swap slots and pageout(), could we completely eliminate swap slot occupation and allocation/release for zeromap data? For example, we could use a special swap entry value in the PTE to indicate zero content and directly fill it with zeros when swapping back. We've observed that swap slot allocation and freeing can consume a lot of CPU and slow down functions like zap_pte_range and swap-in. If we can entirely skip these steps, it could improve performance. However, I'm uncertain about the benefits we would gain if we only have 1-2% zeromap data.

Quoted 4 lines

If I remember correctly this was one of the ideas floated around in the initial version of the zeromap series, but it was evaluated as a lot more complicated to do than what the current zeromap code looks like. But I think its definitely worth looking into!

Sorry for the noise. I didn't review the initial discussion. But my feeling is that it might be valuable considering the report from Zhiguo:

https://lore.kernel.org/linux-mm/[email protected]/

In fact, our recent benchmark also indicates that swap free could account for a significant portion in do_swap_page().

Quoted 9 lines
Quoted 9 lines

I'm just putting this idea out there to see if you're interested in moving forward with it. :-)

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 9 lines
Quoted 9 lines
Quoted 8 lines
Quoted 8 lines
Quoted 7 lines
Quoted 7 lines

I consider issue 1 to be more critical because, after observing a phone running for some time, I've been able to roughly estimate the portion zeromap can
help save using only PSWPOUT, ZSWPOUT, and SWAPOUT_SKIP, even without a SWPIN counter. However, I agree that issue 2 still holds significant value as a separate patch.

Thanks
Barry

#18 lore raw

[..]

Quoted 11 lines
Quoted 11 lines
Quoted 10 lines

By the way, I recently had an idea: if we can conduct the zeromap check earlier - for example - before allocating swap slots and pageout(), could we completely eliminate swap slot occupation and allocation/release for zeromap data? For example, we could use a special swap entry value in the PTE to indicate zero content and directly fill it with zeros when swapping back. We've observed that swap slot allocation and freeing can consume a lot of CPU and slow down functions like zap_pte_range and swap-in. If we can entirely skip these steps, it could improve performance. However, I'm uncertain about the benefits we would gain if we only have 1-2% zeromap data.

Quoted 4 lines
Quoted 4 lines

If I remember correctly this was one of the ideas floated around in the initial version of the zeromap series, but it was evaluated as a lot more complicated to do than what the current zeromap code looks like. But I think its definitely worth looking into!

Yup, I did suggest this on the first version:
https://lore.kernel.org/linux-mm/CAJD7tkYcTV_GOZV3qR6uxgFEvYXw1rP-h7WQjDnsdwM=g9cpAw@mail.gmail.com/

, and Usama took a stab at implementing it in the second version:
https://lore.kernel.org/linux-mm/[email protected]/

David and Shakeel pointed out a few problems. I think they are fixable, but the complexity/benefit tradeoff was getting unclear at that point.

If we can make it work without too much complexity, that would be great of course.

Quoted 8 lines

Sorry for the noise. I didn't review the initial discussion. But my feeling is that it might be valuable considering the report from Zhiguo:

https://lore.kernel.org/linux-mm/[email protected]/

In fact, our recent benchmark also indicates that swap free could account for a significant portion in do_swap_page().

As Shakeel mentioned in a reply to Usama's patch mentioned above, we would need to check the contents of the page after it's unmapped. So likely we need to allocate a swap slot, walk the rmap and unmap, check contents, walk the rmap again and update the PTEs, free the swap slot.

So the swap free will be essentially moved from the fault path to the reclaim path, not eliminated. It may still be worth it, not sure. We also need to make sure we keep the rmap intact after the first walk and unmap in case we need to go back and update the PTEs again.

Overall, I think the complexity is unlikely to be low.

author
#19 lore raw
Quoted 37 lines · Yosry Ahmed

[..]

Quoted 11 lines
Quoted 11 lines
Quoted 10 lines

By the way, I recently had an idea: if we can conduct the zeromap check earlier - for example - before allocating swap slots and pageout(), could we completely eliminate swap slot occupation and allocation/release for zeromap data? For example, we could use a special swap entry value in the PTE to indicate zero content and directly fill it with zeros when swapping back. We've observed that swap slot allocation and freeing can consume a lot of CPU and slow down functions like zap_pte_range and swap-in. If we can entirely skip these steps, it could improve performance. However, I'm uncertain about the benefits we would gain if we only have 1-2% zeromap data.

Quoted 4 lines
Quoted 4 lines

If I remember correctly this was one of the ideas floated around in the initial version of the zeromap series, but it was evaluated as a lot more complicated to do than what the current zeromap code looks like. But I think its definitely worth looking into!

Yup, I did suggest this on the first version:
https://lore.kernel.org/linux-mm/CAJD7tkYcTV_GOZV3qR6uxgFEvYXw1rP-h7WQjDnsdwM=g9cpAw@mail.gmail.com/

, and Usama took a stab at implementing it in the second version:
https://lore.kernel.org/linux-mm/[email protected]/

David and Shakeel pointed out a few problems. I think they are fixable, but the complexity/benefit tradeoff was getting unclear at that point.

If we can make it work without too much complexity, that would be great of course.

Quoted 6 lines

Sorry for the noise. I didn't review the initial discussion. But my feeling is that it might be valuable considering the report from Zhiguo:

https://lore.kernel.org/linux-mm/[email protected]/

Quoted 8 lines
Quoted 2 lines

In fact, our recent benchmark also indicates that swap free could account for a significant portion in do_swap_page().

As Shakeel mentioned in a reply to Usama's patch mentioned above, we would need to check the contents of the page after it's unmapped. So likely we need to allocate a swap slot, walk the rmap and unmap, check contents, walk the rmap again and update the PTEs, free the swap slot.

So the issue is that we can't check the content before allocating slots and unmapping during reclamation? If we find the content is zero, can we skip all slot operations and go directly to rmap/unmap by using a special PTE?

Quoted 6 lines

So the swap free will be essentially moved from the fault path to the reclaim path, not eliminated. It may still be worth it, not sure. We also need to make sure we keep the rmap intact after the first walk and unmap in case we need to go back and update the PTEs again.

Overall, I think the complexity is unlikely to be low.

#20 lore raw
Quoted 48 lines · Barry Song
Quoted 37 lines · Yosry Ahmed

[..]

Quoted 11 lines
Quoted 11 lines
Quoted 10 lines

By the way, I recently had an idea: if we can conduct the zeromap check earlier - for example - before allocating swap slots and pageout(), could we completely eliminate swap slot occupation and allocation/release for zeromap data? For example, we could use a special swap entry value in the PTE to indicate zero content and directly fill it with zeros when swapping back. We've observed that swap slot allocation and freeing can consume a lot of CPU and slow down functions like zap_pte_range and swap-in. If we can entirely skip these steps, it could improve performance. However, I'm uncertain about the benefits we would gain if we only have 1-2% zeromap data.

Quoted 4 lines
Quoted 4 lines

If I remember correctly this was one of the ideas floated around in the initial version of the zeromap series, but it was evaluated as a lot more complicated to do than what the current zeromap code looks like. But I think its definitely worth looking into!

Yup, I did suggest this on the first version:
https://lore.kernel.org/linux-mm/CAJD7tkYcTV_GOZV3qR6uxgFEvYXw1rP-h7WQjDnsdwM=g9cpAw@mail.gmail.com/

, and Usama took a stab at implementing it in the second version:
https://lore.kernel.org/linux-mm/[email protected]/

David and Shakeel pointed out a few problems. I think they are fixable, but the complexity/benefit tradeoff was getting unclear at that point.

If we can make it work without too much complexity, that would be great of course.

Quoted 6 lines

Sorry for the noise. I didn't review the initial discussion. But my feeling is that it might be valuable considering the report from Zhiguo:

https://lore.kernel.org/linux-mm/[email protected]/

Quoted 8 lines
Quoted 2 lines

In fact, our recent benchmark also indicates that swap free could account for a significant portion in do_swap_page().

As Shakeel mentioned in a reply to Usama's patch mentioned above, we would need to check the contents of the page after it's unmapped. So likely we need to allocate a swap slot, walk the rmap and unmap, check contents, walk the rmap again and update the PTEs, free the swap slot.

Quoted 3 lines

So the issue is that we can't check the content before allocating slots and unmapping during reclamation? If we find the content is zero, can we skip all slot operations and go directly to rmap/unmap by using a special PTE?

We need to unmap first before checking the content, otherwise the content can change right after we check it.

author
#21 lore raw
Quoted 50 lines · Yosry Ahmed
Quoted 48 lines · Barry Song
Quoted 45 lines · Yosry Ahmed

[..]

Quoted 11 lines
Quoted 11 lines
Quoted 10 lines

By the way, I recently had an idea: if we can conduct the zeromap check earlier - for example - before allocating swap slots and pageout(), could we completely eliminate swap slot occupation and allocation/release for zeromap data? For example, we could use a special swap entry value in the PTE to indicate zero content and directly fill it with zeros when swapping back. We've observed that swap slot allocation and freeing can consume a lot of CPU and slow down functions like zap_pte_range and swap-in. If we can entirely skip these steps, it could improve performance. However, I'm uncertain about the benefits we would gain if we only have 1-2% zeromap data.

Quoted 4 lines
Quoted 4 lines

If I remember correctly this was one of the ideas floated around in the initial version of the zeromap series, but it was evaluated as a lot more complicated to do than what the current zeromap code looks like. But I think its definitely worth looking into!

Yup, I did suggest this on the first version:
https://lore.kernel.org/linux-mm/CAJD7tkYcTV_GOZV3qR6uxgFEvYXw1rP-h7WQjDnsdwM=g9cpAw@mail.gmail.com/

, and Usama took a stab at implementing it in the second version:
https://lore.kernel.org/linux-mm/[email protected]/

David and Shakeel pointed out a few problems. I think they are fixable, but the complexity/benefit tradeoff was getting unclear at that point.

If we can make it work without too much complexity, that would be great of course.

Quoted 8 lines

Sorry for the noise. I didn't review the initial discussion. But my feeling is that it might be valuable considering the report from Zhiguo:

https://lore.kernel.org/linux-mm/[email protected]/

In fact, our recent benchmark also indicates that swap free could account for a significant portion in do_swap_page().

As Shakeel mentioned in a reply to Usama's patch mentioned above, we would need to check the contents of the page after it's unmapped. So likely we need to allocate a swap slot, walk the rmap and unmap, check contents, walk the rmap again and update the PTEs, free the swap slot.

Quoted 6 lines
Quoted 3 lines

So the issue is that we can't check the content before allocating slots and unmapping during reclamation? If we find the content is zero, can we skip all slot operations and go directly to rmap/unmap by using a special PTE?

We need to unmap first before checking the content, otherwise the content can change right after we check it.

Well, do we have a way to terminate the unmap if we find pte_dirty and ensure the folio is still mapped after try_to_unmap_one()? Then we could activate it again
after try_to_unmap.

It might just be noise. Let me take some more time to think about it. :-)

reviewer
#22 lore raw
Quoted 59 lines · Barry Song
Quoted 50 lines · Yosry Ahmed
Quoted 48 lines · Barry Song
Quoted 45 lines · Yosry Ahmed

[..]

Quoted 11 lines
Quoted 11 lines
Quoted 10 lines

By the way, I recently had an idea: if we can conduct the zeromap check earlier - for example - before allocating swap slots and pageout(), could we completely eliminate swap slot occupation and allocation/release for zeromap data? For example, we could use a special swap entry value in the PTE to indicate zero content and directly fill it with zeros when swapping back. We've observed that swap slot allocation and freeing can consume a lot of CPU and slow down functions like zap_pte_range and swap-in. If we can entirely skip these steps, it could improve performance. However, I'm uncertain about the benefits we would gain if we only have 1-2% zeromap data.

Quoted 4 lines
Quoted 4 lines

If I remember correctly this was one of the ideas floated around in the initial version of the zeromap series, but it was evaluated as a lot more complicated to do than what the current zeromap code looks like. But I think its definitely worth looking into!

Yup, I did suggest this on the first version:
https://lore.kernel.org/linux-mm/CAJD7tkYcTV_GOZV3qR6uxgFEvYXw1rP-h7WQjDnsdwM=g9cpAw@mail.gmail.com/

, and Usama took a stab at implementing it in the second version:
https://lore.kernel.org/linux-mm/[email protected]/

David and Shakeel pointed out a few problems. I think they are fixable, but the complexity/benefit tradeoff was getting unclear at that point.

If we can make it work without too much complexity, that would be great of course.

Quoted 8 lines

Sorry for the noise. I didn't review the initial discussion. But my feeling is that it might be valuable considering the report from Zhiguo:

https://lore.kernel.org/linux-mm/[email protected]/

In fact, our recent benchmark also indicates that swap free could account for a significant portion in do_swap_page().

As Shakeel mentioned in a reply to Usama's patch mentioned above, we would need to check the contents of the page after it's unmapped. So likely we need to allocate a swap slot, walk the rmap and unmap, check contents, walk the rmap again and update the PTEs, free the swap slot.

Quoted 6 lines
Quoted 3 lines

So the issue is that we can't check the content before allocating slots and unmapping during reclamation? If we find the content is zero, can we skip all slot operations and go directly to rmap/unmap by using a special PTE?

We need to unmap first before checking the content, otherwise the content can change right after we check it.

Quoted 6 lines

Well, do we have a way to terminate the unmap if we find pte_dirty and ensure the folio is still mapped after try_to_unmap_one()? Then we could activate it again
after try_to_unmap.

It might just be noise. Let me take some more time to think about it. :-)

FWIW, the swap abstraction layer Yosry proposed last year (and I'm working on right now) will allow you to store these zeromapped swap entries without requiring any swap slots allocated on the swapfile. It's basically the same thing as swap/zswap decoupling.

Not stopping you guys from optimizing it, since all I have right now is a (most certainly buggy) prototype + there might be benefits if we can get around the swap subsystem altogether for these zero mapped entries. Just letting you know there's a backup plan :)

#23 lore raw
Quoted 4 lines

FWIW, the swap abstraction layer Yosry proposed last year (and I'm working on right now) will allow you to store these zeromapped swap entries without requiring any swap slots allocated on the swapfile. It's basically the same thing as swap/zswap decoupling.

I don't know if I said this before, but thank you for keeping this alive. Much appreciated :)

Quoted 5 lines

Not stopping you guys from optimizing it, since all I have right now is a (most certainly buggy) prototype + there might be benefits if we can get around the swap subsystem altogether for these zero mapped entries. Just letting you know there's a backup plan :)

reviewer
#24 lore raw
Quoted 9 lines · Yosry Ahmed
Quoted 5 lines

FWIW, the swap abstraction layer Yosry proposed last year (and I'm working on right now) will allow you to store these zeromapped swap entries without requiring any swap slots allocated on the swapfile. It's basically the same thing as swap/zswap decoupling.

I don't know if I said this before, but thank you for keeping this alive. Much appreciated :)

Of course, it's my pleasure :) I'll be sure to cc you when I send the prototype out as an RFC - your input is much appreciated, as always :)

reviewer
#25 lore raw
In reply to Barry Song
Quoted 7 lines · Barry Song

agree it is better to have a separate counter for zeromap. then it raises a question: what is the proper name for it :-)

zeromap_swpin, zeromap_swpout seems too long? and zswpin and zswpout have been used by zswap

Hmmmmm. How about zeroswpin? zeroswpout?

zeromap_swpin and zeromap_swpout is not the end of the world, FWIW :)

#26 lore raw
Quoted 10 lines · Nhat Pham
Quoted 7 lines · Barry Song

agree it is better to have a separate counter for zeromap. then it raises a question: what is the proper name for it :-)

zeromap_swpin, zeromap_swpout seems too long? and zswpin and zswpout have been used by zswap

Hmmmmm. How about zeroswpin? zeroswpout?

Is this kind of a "swpin_skip" / "swpout_skip" ? Because we optimized it out?

Signature · 3 lines
Cheers,

David / dhildenb
#27 lore raw
In reply to Barry Song
kernel test robot noticed the following build errors:

Hi Barry,

[This is a private test report for your RFC patch.] kernel test robot noticed the following build errors:

[auto build test ERROR on akpm-mm/mm-everything]

url:    https://github.com/intel-lab-lkp/linux/commits/Barry-Song/mm-count-zeromap-read-and-set-for-swapout-and-swapin/20241027-092054
base:   https://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm.git mm-everything
patch link:    https://lore.kernel.org/r/20241027011959.9226-1-21cnbao%40gmail.com
patch subject: [PATCH RFC] mm: count zeromap read and set for swapout and swapin
config: x86_64-kexec (https://download.01.org/0day-ci/archive/20241027/[email protected]/config)
compiler: clang version 19.1.2 (https://github.com/llvm/llvm-project 7ba7d8e2f7b6445b60679da826210cdde29eaf8b)
reproduce (this is a W=1 build): (https://download.01.org/0day-ci/archive/20241027/[email protected]/reproduce)

If you fix the issue in a separate patch/commit (i.e. not just a new version of the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <[email protected]>
| Closes: https://lore.kernel.org/oe-kbuild-all/[email protected]/

All errors (new ones prefixed by >>):

In file included from mm/page_io.c:14:
In file included from include/linux/mm.h:2213:
include/linux/vmstat.h:504:43: warning: arithmetic between different enumeration types ('enum zone_stat_item' and 'enum numa_stat_item') [-Wenum-enum-conversion]

     504 |         return vmstat_text[NR_VM_ZONE_STAT_ITEMS +
         |                            ~~~~~~~~~~~~~~~~~~~~~ ^
     505 |                            item];
         |                            ~~~~
   include/linux/vmstat.h:511:43: warning: arithmetic between different enumeration types ('enum zone_stat_item' and 'enum numa_stat_item') [-Wenum-enum-conversion]
     511 |         return vmstat_text[NR_VM_ZONE_STAT_ITEMS +
         |                            ~~~~~~~~~~~~~~~~~~~~~ ^
     512 |                            NR_VM_NUMA_EVENT_ITEMS +
         |                            ~~~~~~~~~~~~~~~~~~~~~~
   include/linux/vmstat.h:518:36: warning: arithmetic between different enumeration types ('enum node_stat_item' and 'enum lru_list') [-Wenum-enum-conversion]
     518 |         return node_stat_name(NR_LRU_BASE + lru) + 3; // skip "nr_"
         |                               ~~~~~~~~~~~ ^ ~~~
   include/linux/vmstat.h:524:43: warning: arithmetic between different enumeration types ('enum zone_stat_item' and 'enum numa_stat_item') [-Wenum-enum-conversion]
     524 |         return vmstat_text[NR_VM_ZONE_STAT_ITEMS +
         |                            ~~~~~~~~~~~~~~~~~~~~~ ^
     525 |                            NR_VM_NUMA_EVENT_ITEMS +
         |                            ~~~~~~~~~~~~~~~~~~~~~~
>> mm/page_io.c:622:39: error: use of undeclared identifier 'MTHP_STAT_SWPIN'
     622 |                 count_mthp_stat(folio_order(folio), MTHP_STAT_SWPIN);
         |                                                     ^

4 warnings and 1 error generated.

vim +/MTHP_STAT_SWPIN +622 mm/page_io.c

597
598 void swap_read_folio(struct folio *folio, struct swap_iocb **plug)
599 {
600 struct swap_info_struct *sis = swp_swap_info(folio->swap);
601 bool synchronous = sis->flags & SWP_SYNCHRONOUS_IO;
602 bool workingset = folio_test_workingset(folio);
603 unsigned long pflags;
604 bool in_thrashing;
605
606 VM_BUG_ON_FOLIO(!folio_test_swapcache(folio) && !synchronous, folio);
607 VM_BUG_ON_FOLIO(!folio_test_locked(folio), folio);
608 VM_BUG_ON_FOLIO(folio_test_uptodate(folio), folio);
609
610 /*
611 * Count submission time as memory stall and delay. When the device
612 * is congested, or the submitting cgroup IO-throttled, submission
613 * can be a significant part of overall IO time.
614 */
615 if (workingset) {
616 delayacct_thrashing_start(&in_thrashing);
617 psi_memstall_enter(&pflags);
618 }
619 delayacct_swapin_start();
620
621 if (swap_read_folio_zeromap(folio)) {

 > 622			count_mthp_stat(folio_order(folio), MTHP_STAT_SWPIN);

623 count_memcg_folio_events(folio, PSWPIN, folio_nr_pages(folio));
624 count_vm_events(PSWPIN, folio_nr_pages(folio));
625 folio_unlock(folio);
626 goto finish;
627 } else if (zswap_load(folio)) {
628 folio_unlock(folio);
629 goto finish;
630 }
631
632 /* We have to read from slower devices. Increase zswap protection. */
633 zswap_folio_swapin(folio);
634
635 if (data_race(sis->flags & SWP_FS_OPS)) {
636 swap_read_folio_fs(folio, plug);
637 } else if (synchronous) {
638 swap_read_folio_bdev_sync(folio, sis);
639 } else {
640 swap_read_folio_bdev_async(folio, sis);
641 }
642
643 finish:
644 if (workingset) {
645 delayacct_thrashing_end(&in_thrashing);
646 psi_memstall_leave(&pflags);
647 }
648 delayacct_swapin_end();
649 }
650

Signature · 2 lines
0-DAY CI Kernel Test Service
https://github.com/intel/lkp-tests/wiki
#28 lore raw
In reply to Barry Song
kernel test robot noticed the following build errors:

Hi Barry,

[This is a private test report for your RFC patch.] kernel test robot noticed the following build errors:

[auto build test ERROR on akpm-mm/mm-everything]

url:    https://github.com/intel-lab-lkp/linux/commits/Barry-Song/mm-count-zeromap-read-and-set-for-swapout-and-swapin/20241027-092054
base:   https://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm.git mm-everything
patch link:    https://lore.kernel.org/r/20241027011959.9226-1-21cnbao%40gmail.com
patch subject: [PATCH RFC] mm: count zeromap read and set for swapout and swapin
config: x86_64-rhel-8.3 (https://download.01.org/0day-ci/archive/20241027/[email protected]/config)
compiler: gcc-12 (Debian 12.2.0-14) 12.2.0
reproduce (this is a W=1 build): (https://download.01.org/0day-ci/archive/20241027/[email protected]/reproduce)

If you fix the issue in a separate patch/commit (i.e. not just a new version of the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <[email protected]>
| Closes: https://lore.kernel.org/oe-kbuild-all/[email protected]/

All errors (new ones prefixed by >>):

mm/page_io.c: In function 'swap_read_folio':

>> mm/page_io.c:622:53: error: 'MTHP_STAT_SWPIN' undeclared (first use in this function); did you mean 'MTHP_STAT_SPLIT'?
     622 |                 count_mthp_stat(folio_order(folio), MTHP_STAT_SWPIN);
         |                                                     ^~~~~~~~~~~~~~~
         |                                                     MTHP_STAT_SPLIT

mm/page_io.c:622:53: note: each undeclared identifier is reported only once for each function it appears in

vim +622 mm/page_io.c

597
598 void swap_read_folio(struct folio *folio, struct swap_iocb **plug)
599 {
600 struct swap_info_struct *sis = swp_swap_info(folio->swap);
601 bool synchronous = sis->flags & SWP_SYNCHRONOUS_IO;
602 bool workingset = folio_test_workingset(folio);
603 unsigned long pflags;
604 bool in_thrashing;
605
606 VM_BUG_ON_FOLIO(!folio_test_swapcache(folio) && !synchronous, folio);
607 VM_BUG_ON_FOLIO(!folio_test_locked(folio), folio);
608 VM_BUG_ON_FOLIO(folio_test_uptodate(folio), folio);
609
610 /*
611 * Count submission time as memory stall and delay. When the device
612 * is congested, or the submitting cgroup IO-throttled, submission
613 * can be a significant part of overall IO time.
614 */
615 if (workingset) {
616 delayacct_thrashing_start(&in_thrashing);
617 psi_memstall_enter(&pflags);
618 }
619 delayacct_swapin_start();
620
621 if (swap_read_folio_zeromap(folio)) {

 > 622			count_mthp_stat(folio_order(folio), MTHP_STAT_SWPIN);

623 count_memcg_folio_events(folio, PSWPIN, folio_nr_pages(folio));
624 count_vm_events(PSWPIN, folio_nr_pages(folio));
625 folio_unlock(folio);
626 goto finish;
627 } else if (zswap_load(folio)) {
628 folio_unlock(folio);
629 goto finish;
630 }
631
632 /* We have to read from slower devices. Increase zswap protection. */
633 zswap_folio_swapin(folio);
634
635 if (data_race(sis->flags & SWP_FS_OPS)) {
636 swap_read_folio_fs(folio, plug);
637 } else if (synchronous) {
638 swap_read_folio_bdev_sync(folio, sis);
639 } else {
640 swap_read_folio_bdev_async(folio, sis);
641 }
642
643 finish:
644 if (workingset) {
645 delayacct_thrashing_end(&in_thrashing);
646 psi_memstall_leave(&pflags);
647 }
648 delayacct_swapin_end();
649 }
650

Signature · 2 lines
0-DAY CI Kernel Test Service
https://github.com/intel/lkp-tests/wiki