Skip to contentSkip to conversation

mm/vmscan: prevent MGLRU reclaim from pinning address space

PATCH Applied Suren Baghdasaryan 3 participants 12 messages Mar 22 – Mar 27

Subsystems (3)
lore
Share & tools
lore
#0 lore raw

PATCH 1/1 mm/vmscan: prevent MGLRU reclaim from pinning address space

When shrinking lruvec, MGLRU pins address space before walking it. This is excessive since all it needs for walking the page range is a stable mm_struct to be able to take and release mmap_read_lock and a stable mm->mm_mt tree to walk. This address space pinning results in delays when releasing the memory of a dying process. This also prevents mm reapers (both in-kernel oom-reaper and userspace process_mrelease()) from doing their job during MGLRU scan because they check task_will_free_mem() which will yield negative result due to the elevated mm->mm_users.

Replace unnecessary address space pinning with mm_struct pinning by replacing mmget/mmput with mmgrab/mmdrop calls. mm_mt is contained within mm_struct itself, therefore it won't be freed as long as mm_struct is stable and it won't change during the walk because mmap_read_lock is being held.

Fixes bd74fdaea146 ("mm: multi-gen LRU: support page table walks")
---
 mm/vmscan.c | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)
mm/vmscan.c +3 -2
index 33287ba4a500..68e8e90e38f5 100644
@@ -2863,8 +2863,9 @@ static struct mm_struct *get_next_mm(struct lru_gen_mm_walk *walk)
 		return NULL;
 
 	clear_bit(key, &mm->lru_gen.bitmap);
+	mmgrab(mm);
 
-	return mmget_not_zero(mm) ? mm : NULL;
+	return mm;
 }
 
 void lru_gen_add_mm(struct mm_struct *mm)
@@ -3064,7 +3065,7 @@ static bool iterate_mm_list(struct lru_gen_mm_walk *walk, struct mm_struct **ite
 		reset_bloom_filter(mm_state, walk->seq + 1);
 
 	if (*iter)
-		mmput_async(*iter);
+		mmdrop(*iter);
 
 	*iter = mm;
 

base-commit: 8c65073d94c8b7cc3170de31af38edc9f5d96f0e

Signature · 1 line
2.53.0.1018.g2bb0e51243-goog
#1 lore raw
Quoted 4 lines · Suren Baghdasaryan

When shrinking lruvec, MGLRU pins address space before walking it. This is excessive since all it needs for walking the page range is a stable mm_struct to be able to take and release mmap_read_lock and a stable mm->mm_mt tree to walk. This address space pinning results

Hmm, I guess exit_mmap() calls __mt_destroy(), but that'll just destroy allocated state and leave the tree empty right, so traversal of that tree at that point would just do nothing?

Quoted 23 lines

in delays when releasing the memory of a dying process. This also prevents mm reapers (both in-kernel oom-reaper and userspace process_mrelease()) from doing their job during MGLRU scan because they check task_will_free_mem() which will yield negative result due to the elevated mm->mm_users.

Replace unnecessary address space pinning with mm_struct pinning by replacing mmget/mmput with mmgrab/mmdrop calls. mm_mt is contained within mm_struct itself, therefore it won't be freed as long as mm_struct is stable and it won't change during the walk because mmap_read_lock is being held.

Fixes bd74fdaea146 ("mm: multi-gen LRU: support page table walks")
---
 mm/vmscan.c | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)
mm/vmscan.c +0 -0
index 33287ba4a500..68e8e90e38f5 100644

mm/vmscan.c static struct mm_struct *get_next_mm(struct lru_gen_mm_walk *walk) · line 2863

Quoted 1 line
mm/vmscan.c +0 -0
@@ -2863,8 +2863,9 @@ static struct mm_struct *get_next_mm(struct lru_gen_mm_walk *walk)
 		return NULL;

Not related to this series, but I really don't like how coupled MGLRU is to the rest of the 'classic' reclaim code.

Just in the middle of vmscan you walk into generic mm walker logic and the only hint it's MGLRU is you see lru_gen_xxx stuff (I'm also annoyed that we call it MGLRU but it's called lru_gen_xxx in the kernel :)

mm/vmscan.c static struct mm_struct *get_next_mm(struct lru_gen_mm_walk *walk) · line 2866

Quoted 3 lines
mm/vmscan.c +1 -0
 	clear_bit(key, &mm->lru_gen.bitmap);
+	mmgrab(mm);

Is the mm somehow pinned here or, on destruction, would move it from the mm list meaning that we can safely assume we have something sane in mm-> to grab? I guess this must have already been the case for mmget_not_zero() to have been used before though.

mm/vmscan.c static bool iterate_mm_list(struct lru_gen_mm_walk *walk, struct mm_struct **ite · line 3068

Quoted 12 lines
mm/vmscan.c +2 -2
-	return mmget_not_zero(mm) ? mm : NULL;
+	return mm;
 }
 
 void lru_gen_add_mm(struct mm_struct *mm)
@@ -3064,7 +3065,7 @@ static bool iterate_mm_list(struct lru_gen_mm_walk *walk, struct mm_struct **ite
 		reset_bloom_filter(mm_state, walk->seq + 1);
 
 	if (*iter)
-		mmput_async(*iter);
+		mmdrop(*iter);

This will now be a blocking call that could free the mm (via __mmdrop()), so could take a while, is that ok?

If before the code was intentionally deferring work here, doesn't that imply that being slow here might be an issue, somehow? Or was it just because they could? :)

Quoted 8 lines
mm/vmscan.c +0 -0
 	*iter = mm;
 
 

base-commit: 8c65073d94c8b7cc3170de31af38edc9f5d96f0e --
2.53.0.1018.g2bb0e51243-goog

Thanks, Lorenzo

#2 lore raw
Quoted 10 lines · Lorenzo Stoakes (Oracle)
Quoted 4 lines · Suren Baghdasaryan

When shrinking lruvec, MGLRU pins address space before walking it. This is excessive since all it needs for walking the page range is a stable mm_struct to be able to take and release mmap_read_lock and a stable mm->mm_mt tree to walk. This address space pinning results

Hmm, I guess exit_mmap() calls __mt_destroy(), but that'll just destroy allocated state and leave the tree empty right, so traversal of that tree at that point would just do nothing?

Correct. And __mt_destroy() happens under mmap_write_lock while traversal under mmap_read_lock, so they should not race.

Quoted 29 lines
Quoted 24 lines

in delays when releasing the memory of a dying process. This also prevents mm reapers (both in-kernel oom-reaper and userspace process_mrelease()) from doing their job during MGLRU scan because they check task_will_free_mem() which will yield negative result due to the elevated mm->mm_users.

Replace unnecessary address space pinning with mm_struct pinning by replacing mmget/mmput with mmgrab/mmdrop calls. mm_mt is contained within mm_struct itself, therefore it won't be freed as long as mm_struct is stable and it won't change during the walk because mmap_read_lock is being held.

Fixes bd74fdaea146 ("mm: multi-gen LRU: support page table walks")
---
 mm/vmscan.c | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)
mm/vmscan.c +0 -0
index 33287ba4a500..68e8e90e38f5 100644

mm/vmscan.c static struct mm_struct *get_next_mm(struct lru_gen_mm_walk *walk) · line 2863

Quoted 1 line
mm/vmscan.c +0 -0
@@ -2863,8 +2863,9 @@ static struct mm_struct *get_next_mm(struct lru_gen_mm_walk *walk)
              return NULL;

Not related to this series, but I really don't like how coupled MGLRU is to the rest of the 'classic' reclaim code.

Quoted 3 lines

Just in the middle of vmscan you walk into generic mm walker logic and the only hint it's MGLRU is you see lru_gen_xxx stuff (I'm also annoyed that we call it MGLRU but it's called lru_gen_xxx in the kernel :)

I don't have a strong opinion on this. Perhaps the naming can be changed outside of this series.

Quoted 9 lines

mm/vmscan.c static struct mm_struct *get_next_mm(struct lru_gen_mm_walk *walk) · line 2866

Quoted 4 lines
mm/vmscan.c +1 -0
      clear_bit(key, &mm->lru_gen.bitmap);
+     mmgrab(mm);

Is the mm somehow pinned here or, on destruction, would move it from the mm list meaning that we can safely assume we have something sane in mm-> to grab? I guess this must have already been the case for mmget_not_zero() to have been used before though.

Yes, mm is stable because it's fetched from mm_list. When mm is added to this list via lru_gen_add_mm(mm) it is referenced and that reference is dropped only after lru_gen_del_mm(mm) removes the mm from this list (see https://elixir.bootlin.com/linux/v7.0-rc4/source/kernel/fork.c#L1185
and https://elixir.bootlin.com/linux/v7.0-rc4/source/kernel/fork.c#L1187).
Addition, removal and retrieval from that list happen under mm_list->lock which prevents races.

Quoted 10 lines
Quoted 10 lines
mm/vmscan.c +1 -1
-     return mmget_not_zero(mm) ? mm : NULL;
+     return mm;
 }
 
 void lru_gen_add_mm(struct mm_struct *mm)
@@ -3064,7 +3065,7 @@ static bool iterate_mm_list(struct lru_gen_mm_walk *walk, struct mm_struct **ite
              reset_bloom_filter(mm_state, walk->seq + 1);
 
Quoted 6 lines

mm/vmscan.c static bool iterate_mm_list(struct lru_gen_mm_walk *walk, struct mm_struct **ite · line 3068

Quoted 3 lines
mm/vmscan.c +1 -1
      if (*iter)
-             mmput_async(*iter);
+             mmdrop(*iter);

This will now be a blocking call that could free the mm (via __mmdrop()), so could take a while, is that ok?

mmdrop() should not be a heavy-weight operation. It simply destroys the metadata associated with mm_struct. mmput() OTOH will call exit_mmap() if it drops the last reference and that can take a while because that's when we free the memory of the process. I believe that's why mmput_async() was used here.

Quoted 4 lines

If before the code was intentionally deferring work here, doesn't that imply that being slow here might be an issue, somehow? Or was it just because they could? :)

I think the reason was the possibility of calling mmput() -> __mmput() -> exit_mmap(mm) which could indeed block us for a while.

Quoted 11 lines
Quoted 9 lines
mm/vmscan.c +0 -0
      *iter = mm;
 
 

base-commit: 8c65073d94c8b7cc3170de31af38edc9f5d96f0e --
2.53.0.1018.g2bb0e51243-goog

Thanks, Lorenzo

reviewer Reviewed-by
#3 lore raw
Quoted 8 lines · Suren Baghdasaryan
Quoted 7 lines · Lorenzo Stoakes (Oracle)
Quoted 4 lines · Suren Baghdasaryan

When shrinking lruvec, MGLRU pins address space before walking it. This is excessive since all it needs for walking the page range is a stable mm_struct to be able to take and release mmap_read_lock and a stable mm->mm_mt tree to walk. This address space pinning results

Quoted 6 lines
Quoted 3 lines

Hmm, I guess exit_mmap() calls __mt_destroy(), but that'll just destroy allocated state and leave the tree empty right, so traversal of that tree at that point would just do nothing?

Correct. And __mt_destroy() happens under mmap_write_lock while traversal under mmap_read_lock, so they should not race.

Yeah that's fair.

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

in delays when releasing the memory of a dying process. This also prevents mm reapers (both in-kernel oom-reaper and userspace process_mrelease()) from doing their job during MGLRU scan because they check task_will_free_mem() which will yield negative result due to the elevated mm->mm_users.

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

Replace unnecessary address space pinning with mm_struct pinning by replacing mmget/mmput with mmgrab/mmdrop calls. mm_mt is contained within mm_struct itself, therefore it won't be freed as long as mm_struct is stable and it won't change during the walk because mmap_read_lock is being held.

Fixes bd74fdaea146 ("mm: multi-gen LRU: support page table walks")

Given you have cleared up my concerns, this LGTM, so:

Quoted 14 lines
Quoted 14 lines

mm/vmscan.c static struct mm_struct *get_next_mm(struct lru_gen_mm_walk *walk) · line 2863

Quoted 10 lines
---
 mm/vmscan.c | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)
mm/vmscan.c +0 -0
index 33287ba4a500..68e8e90e38f5 100644
@@ -2863,8 +2863,9 @@ static struct mm_struct *get_next_mm(struct lru_gen_mm_walk *walk)
              return NULL;

Not related to this series, but I really don't like how coupled MGLRU is to the rest of the 'classic' reclaim code.

Quoted 6 lines
Quoted 3 lines

Just in the middle of vmscan you walk into generic mm walker logic and the only hint it's MGLRU is you see lru_gen_xxx stuff (I'm also annoyed that we call it MGLRU but it's called lru_gen_xxx in the kernel :)

I don't have a strong opinion on this. Perhaps the naming can be changed outside of this series.

I was thinking more of a new file for mglru :>)

I believe we also need some more active maintainership also... but that's another issue ;)

Quoted 11 lines
Quoted 10 lines

mm/vmscan.c static struct mm_struct *get_next_mm(struct lru_gen_mm_walk *walk) · line 2866

Quoted 5 lines
mm/vmscan.c +1 -0
      clear_bit(key, &mm->lru_gen.bitmap);
+     mmgrab(mm);

Is the mm somehow pinned here or, on destruction, would move it from the mm list meaning that we can safely assume we have something sane in mm-> to grab? I guess this must have already been the case for mmget_not_zero() to have been used before though.

Quoted 7 lines

Yes, mm is stable because it's fetched from mm_list. When mm is added to this list via lru_gen_add_mm(mm) it is referenced and that reference is dropped only after lru_gen_del_mm(mm) removes the mm from this list (see https://elixir.bootlin.com/linux/v7.0-rc4/source/kernel/fork.c#L1185
and https://elixir.bootlin.com/linux/v7.0-rc4/source/kernel/fork.c#L1187).
Addition, removal and retrieval from that list happen under mm_list->lock which prevents races.

Ack, thanks!

Quoted 18 lines
Quoted 11 lines
Quoted 11 lines
mm/vmscan.c +1 -1
-     return mmget_not_zero(mm) ? mm : NULL;
+     return mm;
 }
 
 void lru_gen_add_mm(struct mm_struct *mm)
@@ -3064,7 +3065,7 @@ static bool iterate_mm_list(struct lru_gen_mm_walk *walk, struct mm_struct **ite
              reset_bloom_filter(mm_state, walk->seq + 1);
 
Quoted 6 lines

mm/vmscan.c static bool iterate_mm_list(struct lru_gen_mm_walk *walk, struct mm_struct **ite · line 3068

Quoted 3 lines
mm/vmscan.c +1 -1
      if (*iter)
-             mmput_async(*iter);
+             mmdrop(*iter);

This will now be a blocking call that could free the mm (via __mmdrop()), so could take a while, is that ok?

Quoted 5 lines

mmdrop() should not be a heavy-weight operation. It simply destroys the metadata associated with mm_struct. mmput() OTOH will call exit_mmap() if it drops the last reference and that can take a while because that's when we free the memory of the process. I believe that's why mmput_async() was used here.

Yeah that's fair enough! Thanks.

Quoted 8 lines
Quoted 5 lines

If before the code was intentionally deferring work here, doesn't that imply that being slow here might be an issue, somehow? Or was it just because they could? :)

I think the reason was the possibility of calling mmput() -> __mmput() -> exit_mmap(mm) which could indeed block us for a while.

Yeah fair :)

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
mm/vmscan.c +0 -0
      *iter = mm;
 
 
Quoted 6 lines
Quoted 6 lines
Quoted 4 lines

base-commit: 8c65073d94c8b7cc3170de31af38edc9f5d96f0e --
2.53.0.1018.g2bb0e51243-goog

Thanks, Lorenzo

Cheers, Lorenzo

#4 lore raw
Quoted 33 lines · Lorenzo Stoakes (Oracle)
Quoted 8 lines · Suren Baghdasaryan
Quoted 7 lines · Lorenzo Stoakes (Oracle)
Quoted 4 lines · Suren Baghdasaryan

When shrinking lruvec, MGLRU pins address space before walking it. This is excessive since all it needs for walking the page range is a stable mm_struct to be able to take and release mmap_read_lock and a stable mm->mm_mt tree to walk. This address space pinning results

Quoted 6 lines
Quoted 3 lines

Hmm, I guess exit_mmap() calls __mt_destroy(), but that'll just destroy allocated state and leave the tree empty right, so traversal of that tree at that point would just do nothing?

Correct. And __mt_destroy() happens under mmap_write_lock while traversal under mmap_read_lock, so they should not race.

Yeah that's fair.

Quoted 14 lines
Quoted 14 lines
Quoted 14 lines

in delays when releasing the memory of a dying process. This also prevents mm reapers (both in-kernel oom-reaper and userspace process_mrelease()) from doing their job during MGLRU scan because they check task_will_free_mem() which will yield negative result due to the elevated mm->mm_users.

Replace unnecessary address space pinning with mm_struct pinning by replacing mmget/mmput with mmgrab/mmdrop calls. mm_mt is contained within mm_struct itself, therefore it won't be freed as long as mm_struct is stable and it won't change during the walk because mmap_read_lock is being held.

Quoted 6 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Fixes bd74fdaea146 ("mm: multi-gen LRU: support page table walks")

Given you have cleared up my concerns, this LGTM, so:

Thanks!

Quoted 19 lines
Quoted 19 lines
Quoted 18 lines

mm/vmscan.c static struct mm_struct *get_next_mm(struct lru_gen_mm_walk *walk) · line 2863

Quoted 11 lines
---
 mm/vmscan.c | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)
mm/vmscan.c +0 -0
index 33287ba4a500..68e8e90e38f5 100644
@@ -2863,8 +2863,9 @@ static struct mm_struct *get_next_mm(struct lru_gen_mm_walk *walk)
              return NULL;

Not related to this series, but I really don't like how coupled MGLRU is to the rest of the 'classic' reclaim code.

Just in the middle of vmscan you walk into generic mm walker logic and the only hint it's MGLRU is you see lru_gen_xxx stuff (I'm also annoyed that we call it MGLRU but it's called lru_gen_xxx in the kernel :)

Quoted 7 lines
Quoted 2 lines

I don't have a strong opinion on this. Perhaps the naming can be changed outside of this series.

I was thinking more of a new file for mglru :>)

I believe we also need some more active maintainership also... but that's another issue ;)

Yes, in my team I volunteered one developer to actively review and support MGLRU. Over time I hope he will be in a position to help with maintenance.

Quoted 73 lines
Quoted 12 lines
Quoted 11 lines

mm/vmscan.c static struct mm_struct *get_next_mm(struct lru_gen_mm_walk *walk) · line 2866

Quoted 6 lines
mm/vmscan.c +1 -0
      clear_bit(key, &mm->lru_gen.bitmap);
+     mmgrab(mm);

Is the mm somehow pinned here or, on destruction, would move it from the mm list meaning that we can safely assume we have something sane in mm-> to grab? I guess this must have already been the case for mmget_not_zero() to have been used before though.

Quoted 7 lines

Yes, mm is stable because it's fetched from mm_list. When mm is added to this list via lru_gen_add_mm(mm) it is referenced and that reference is dropped only after lru_gen_del_mm(mm) removes the mm from this list (see https://elixir.bootlin.com/linux/v7.0-rc4/source/kernel/fork.c#L1185
and https://elixir.bootlin.com/linux/v7.0-rc4/source/kernel/fork.c#L1187).
Addition, removal and retrieval from that list happen under mm_list->lock which prevents races.

Ack, thanks!

Quoted 18 lines
Quoted 11 lines
Quoted 11 lines
mm/vmscan.c +1 -1
-     return mmget_not_zero(mm) ? mm : NULL;
+     return mm;
 }
 
 void lru_gen_add_mm(struct mm_struct *mm)
@@ -3064,7 +3065,7 @@ static bool iterate_mm_list(struct lru_gen_mm_walk *walk, struct mm_struct **ite
              reset_bloom_filter(mm_state, walk->seq + 1);
 
Quoted 6 lines

mm/vmscan.c static bool iterate_mm_list(struct lru_gen_mm_walk *walk, struct mm_struct **ite · line 3068

Quoted 3 lines
mm/vmscan.c +1 -1
      if (*iter)
-             mmput_async(*iter);
+             mmdrop(*iter);

This will now be a blocking call that could free the mm (via __mmdrop()), so could take a while, is that ok?

Quoted 5 lines

mmdrop() should not be a heavy-weight operation. It simply destroys the metadata associated with mm_struct. mmput() OTOH will call exit_mmap() if it drops the last reference and that can take a while because that's when we free the memory of the process. I believe that's why mmput_async() was used here.

Yeah that's fair enough! Thanks.

Quoted 8 lines
Quoted 5 lines

If before the code was intentionally deferring work here, doesn't that imply that being slow here might be an issue, somehow? Or was it just because they could? :)

I think the reason was the possibility of calling mmput() -> __mmput() -> exit_mmap(mm) which could indeed block us for a while.

Yeah fair :)

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
mm/vmscan.c +0 -0
      *iter = mm;
 
 
Quoted 6 lines
Quoted 6 lines
Quoted 4 lines

base-commit: 8c65073d94c8b7cc3170de31af38edc9f5d96f0e --
2.53.0.1018.g2bb0e51243-goog

Thanks, Lorenzo

Cheers, Lorenzo

#5 lore raw
In reply to Lorenzo Stoakes (Oracle)
Quoted 33 lines · Lorenzo Stoakes (Oracle)
Quoted 8 lines · Suren Baghdasaryan
Quoted 7 lines · Lorenzo Stoakes (Oracle)
Quoted 4 lines · Suren Baghdasaryan

When shrinking lruvec, MGLRU pins address space before walking it. This is excessive since all it needs for walking the page range is a stable mm_struct to be able to take and release mmap_read_lock and a stable mm->mm_mt tree to walk. This address space pinning results

Quoted 6 lines
Quoted 3 lines

Hmm, I guess exit_mmap() calls __mt_destroy(), but that'll just destroy allocated state and leave the tree empty right, so traversal of that tree at that point would just do nothing?

Correct. And __mt_destroy() happens under mmap_write_lock while traversal under mmap_read_lock, so they should not race.

Yeah that's fair.

Quoted 14 lines
Quoted 14 lines
Quoted 14 lines

in delays when releasing the memory of a dying process. This also prevents mm reapers (both in-kernel oom-reaper and userspace process_mrelease()) from doing their job during MGLRU scan because they check task_will_free_mem() which will yield negative result due to the elevated mm->mm_users.

Replace unnecessary address space pinning with mm_struct pinning by replacing mmget/mmput with mmgrab/mmdrop calls. mm_mt is contained within mm_struct itself, therefore it won't be freed as long as mm_struct is stable and it won't change during the walk because mmap_read_lock is being held.

Quoted 6 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Fixes bd74fdaea146 ("mm: multi-gen LRU: support page table walks")

Given you have cleared up my concerns, this LGTM, so:

Hi Andrew,
Any concerns about this patch or do you want someone maintaining MGLRU to Ack it before pulling into your tree?
The change is quite simple and I explain in my reply why it's safe, Sashiko seems to like it and I extensively tested it on Android. Please let me know if anything else is needed.
Thanks,
Suren.

Quoted 99 lines
Quoted 15 lines
Quoted 15 lines

mm/vmscan.c static struct mm_struct *get_next_mm(struct lru_gen_mm_walk *walk) · line 2863

Quoted 11 lines
---
 mm/vmscan.c | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)
mm/vmscan.c +0 -0
index 33287ba4a500..68e8e90e38f5 100644
@@ -2863,8 +2863,9 @@ static struct mm_struct *get_next_mm(struct lru_gen_mm_walk *walk)
              return NULL;

Not related to this series, but I really don't like how coupled MGLRU is to the rest of the 'classic' reclaim code.

Quoted 6 lines
Quoted 3 lines

Just in the middle of vmscan you walk into generic mm walker logic and the only hint it's MGLRU is you see lru_gen_xxx stuff (I'm also annoyed that we call it MGLRU but it's called lru_gen_xxx in the kernel :)

I don't have a strong opinion on this. Perhaps the naming can be changed outside of this series.

I was thinking more of a new file for mglru :>)

I believe we also need some more active maintainership also... but that's another issue ;)

Quoted 11 lines
Quoted 10 lines

mm/vmscan.c static struct mm_struct *get_next_mm(struct lru_gen_mm_walk *walk) · line 2866

Quoted 5 lines
mm/vmscan.c +1 -0
      clear_bit(key, &mm->lru_gen.bitmap);
+     mmgrab(mm);

Is the mm somehow pinned here or, on destruction, would move it from the mm list meaning that we can safely assume we have something sane in mm-> to grab? I guess this must have already been the case for mmget_not_zero() to have been used before though.

Quoted 7 lines

Yes, mm is stable because it's fetched from mm_list. When mm is added to this list via lru_gen_add_mm(mm) it is referenced and that reference is dropped only after lru_gen_del_mm(mm) removes the mm from this list (see https://elixir.bootlin.com/linux/v7.0-rc4/source/kernel/fork.c#L1185
and https://elixir.bootlin.com/linux/v7.0-rc4/source/kernel/fork.c#L1187).
Addition, removal and retrieval from that list happen under mm_list->lock which prevents races.

Ack, thanks!

Quoted 18 lines
Quoted 11 lines
Quoted 11 lines
mm/vmscan.c +1 -1
-     return mmget_not_zero(mm) ? mm : NULL;
+     return mm;
 }
 
 void lru_gen_add_mm(struct mm_struct *mm)
@@ -3064,7 +3065,7 @@ static bool iterate_mm_list(struct lru_gen_mm_walk *walk, struct mm_struct **ite
              reset_bloom_filter(mm_state, walk->seq + 1);
 
Quoted 6 lines

mm/vmscan.c static bool iterate_mm_list(struct lru_gen_mm_walk *walk, struct mm_struct **ite · line 3068

Quoted 3 lines
mm/vmscan.c +1 -1
      if (*iter)
-             mmput_async(*iter);
+             mmdrop(*iter);

This will now be a blocking call that could free the mm (via __mmdrop()), so could take a while, is that ok?

Quoted 5 lines

mmdrop() should not be a heavy-weight operation. It simply destroys the metadata associated with mm_struct. mmput() OTOH will call exit_mmap() if it drops the last reference and that can take a while because that's when we free the memory of the process. I believe that's why mmput_async() was used here.

Yeah that's fair enough! Thanks.

Quoted 8 lines
Quoted 5 lines

If before the code was intentionally deferring work here, doesn't that imply that being slow here might be an issue, somehow? Or was it just because they could? :)

I think the reason was the possibility of calling mmput() -> __mmput() -> exit_mmap(mm) which could indeed block us for a while.

Yeah fair :)

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
mm/vmscan.c +0 -0
      *iter = mm;
 
 
Quoted 6 lines
Quoted 6 lines
Quoted 4 lines

base-commit: 8c65073d94c8b7cc3170de31af38edc9f5d96f0e --
2.53.0.1018.g2bb0e51243-goog

Thanks, Lorenzo

Cheers, Lorenzo

maintainer
#6 lore raw
Quoted 7 lines · On Fri, 27 Mar 2026 08:20:06 -0700 Suren Baghdasaryan <[email protected]> wrote:
Quoted 3 lines

Given you have cleared up my concerns, this LGTM, so:

Hi Andrew,
Any concerns about this patch or do you want someone maintaining MGLRU to Ack it before pulling into your tree?

That would of course be welcome. But Suren+Lorenzo is more than enough for me.

Quoted 3 lines

The change is quite simple and I explain in my reply why it's safe, Sashiko seems to like it and I extensively tested it on Android. Please let me know if anything else is needed.

I think that if the fix is valuable enough for Android to backport then we at least should have the Fixes: there to help others who are using MGLRU on older kernels. (We don't have an "Improves:"!) And if it's a really big improvement then we could even pretend it's a bugfix and sneak it into -stable ;)

But the changelog doesn't say much at all about how this change affects users. Just "delays". So can you please expand on this (numbers are always nice) and I can paste that into the changelog.

Thanks.

#7 lore raw
Quoted 24 lines · Andrew Morton
Quoted 7 lines · On Fri, 27 Mar 2026 08:20:06 -0700 Suren Baghdasaryan <[email protected]> wrote:
Quoted 3 lines

Given you have cleared up my concerns, this LGTM, so:

Hi Andrew,
Any concerns about this patch or do you want someone maintaining MGLRU to Ack it before pulling into your tree?

That would of course be welcome. But Suren+Lorenzo is more than enough for me.

Quoted 3 lines

The change is quite simple and I explain in my reply why it's safe, Sashiko seems to like it and I extensively tested it on Android. Please let me know if anything else is needed.

I think that if the fix is valuable enough for Android to backport then we at least should have the Fixes: there to help others who are using MGLRU on older kernels. (We don't have an "Improves:"!) And if it's a really big improvement then we could even pretend it's a bugfix and sneak it into -stable ;)

Quoted 3 lines

But the changelog doesn't say much at all about how this change affects users. Just "delays". So can you please expand on this (numbers are always nice) and I can paste that into the changelog.

It affects the system in the sense that if the MM of the killed process is being reclaimed by kswapd then reapers won't be able to reap it. Even the process itself (which might have higher-priority than kswapd) will not free its memory until kswapd drops the last reference. IOW, we delay freeing the memory because kswapd is reclaiming it. In Android the visible result for us is that process_mrelease() (userspace reaper) skips MM in such cases and we see process memory not released for an unusually long time (secs).

Quoted 2 lines

Thanks.

#8 lore raw
In reply to Suren Baghdasaryan
Quoted 10 lines · Suren Baghdasaryan

When shrinking lruvec, MGLRU pins address space before walking it. This is excessive since all it needs for walking the page range is a stable mm_struct to be able to take and release mmap_read_lock and a stable mm->mm_mt tree to walk. This address space pinning results in delays when releasing the memory of a dying process. This also prevents mm reapers (both in-kernel oom-reaper and userspace process_mrelease()) from doing their job during MGLRU scan because they check task_will_free_mem() which will yield negative result due to the elevated mm->mm_users.

Quoted 8 lines

Replace unnecessary address space pinning with mm_struct pinning by replacing mmget/mmput with mmgrab/mmdrop calls. mm_mt is contained within mm_struct itself, therefore it won't be freed as long as mm_struct is stable and it won't change during the walk because mmap_read_lock is being held.

Fixes bd74fdaea146 ("mm: multi-gen LRU: support page table walks")

Oh and sorry, missed this - do we need a 'Fixed' for something that's just a perf improvement?

This doesn't seem buggy?

Thanks, Lorenzo

#9 lore raw
Quoted 18 lines · Lorenzo Stoakes (Oracle)
Quoted 16 lines · Suren Baghdasaryan

When shrinking lruvec, MGLRU pins address space before walking it. This is excessive since all it needs for walking the page range is a stable mm_struct to be able to take and release mmap_read_lock and a stable mm->mm_mt tree to walk. This address space pinning results in delays when releasing the memory of a dying process. This also prevents mm reapers (both in-kernel oom-reaper and userspace process_mrelease()) from doing their job during MGLRU scan because they check task_will_free_mem() which will yield negative result due to the elevated mm->mm_users.

Replace unnecessary address space pinning with mm_struct pinning by replacing mmget/mmput with mmgrab/mmdrop calls. mm_mt is contained within mm_struct itself, therefore it won't be freed as long as mm_struct is stable and it won't change during the walk because mmap_read_lock is being held.

Quoted 7 lines
Quoted 2 lines
Fixes bd74fdaea146 ("mm: multi-gen LRU: support page table walks")

Oh and sorry, missed this - do we need a 'Fixed' for something that's just a perf improvement?

This doesn't seem buggy?

Well, if we are pinning something we should not is that a bug? I didn't CC stable, so I don't expect it to be backported (though if it's accepted, we will backport it in Android as it causes real problems with reclaim), however if someone is looking for MGLRU fixes I would like this patch to be discoverable. If you feel strongly about it, I'll remove the tag.

Thanks for the review!

Quoted 2 lines

Thanks, Lorenzo

#10 lore raw
Quoted 25 lines · Suren Baghdasaryan
Quoted 24 lines · Lorenzo Stoakes (Oracle)
Quoted 10 lines · Suren Baghdasaryan

When shrinking lruvec, MGLRU pins address space before walking it. This is excessive since all it needs for walking the page range is a stable mm_struct to be able to take and release mmap_read_lock and a stable mm->mm_mt tree to walk. This address space pinning results in delays when releasing the memory of a dying process. This also prevents mm reapers (both in-kernel oom-reaper and userspace process_mrelease()) from doing their job during MGLRU scan because they check task_will_free_mem() which will yield negative result due to the elevated mm->mm_users.

Quoted 8 lines

Replace unnecessary address space pinning with mm_struct pinning by replacing mmget/mmput with mmgrab/mmdrop calls. mm_mt is contained within mm_struct itself, therefore it won't be freed as long as mm_struct is stable and it won't change during the walk because mmap_read_lock is being held.

Fixes bd74fdaea146 ("mm: multi-gen LRU: support page table walks")

Oh and sorry, missed this - do we need a 'Fixed' for something that's just a perf improvement?

Quoted 8 lines
Quoted 1 line

This doesn't seem buggy?

Well, if we are pinning something we should not is that a bug? I didn't CC stable, so I don't expect it to be backported (though if it's accepted, we will backport it in Android as it causes real problems with reclaim), however if someone is looking for MGLRU fixes I would like this patch to be discoverable. If you feel strongly about it, I'll remove the tag.

I don't feel that strongly about it :>)

Quoted 2 lines

Thanks for the review!

No problem! Working through your vma killable series atm also :)

Quoted 3 lines
Quoted 3 lines

Thanks, Lorenzo

Cheers, Lorenzo

#11 lore raw
Quoted 36 lines · Lorenzo Stoakes (Oracle)
Quoted 25 lines · Suren Baghdasaryan
Quoted 24 lines · Lorenzo Stoakes (Oracle)
Quoted 10 lines · Suren Baghdasaryan

When shrinking lruvec, MGLRU pins address space before walking it. This is excessive since all it needs for walking the page range is a stable mm_struct to be able to take and release mmap_read_lock and a stable mm->mm_mt tree to walk. This address space pinning results in delays when releasing the memory of a dying process. This also prevents mm reapers (both in-kernel oom-reaper and userspace process_mrelease()) from doing their job during MGLRU scan because they check task_will_free_mem() which will yield negative result due to the elevated mm->mm_users.

Quoted 8 lines

Replace unnecessary address space pinning with mm_struct pinning by replacing mmget/mmput with mmgrab/mmdrop calls. mm_mt is contained within mm_struct itself, therefore it won't be freed as long as mm_struct is stable and it won't change during the walk because mmap_read_lock is being held.

Fixes bd74fdaea146 ("mm: multi-gen LRU: support page table walks")

Oh and sorry, missed this - do we need a 'Fixed' for something that's just a perf improvement?

Quoted 8 lines
Quoted 1 line

This doesn't seem buggy?

Well, if we are pinning something we should not is that a bug? I didn't CC stable, so I don't expect it to be backported (though if it's accepted, we will backport it in Android as it causes real problems with reclaim), however if someone is looking for MGLRU fixes I would like this patch to be discoverable. If you feel strongly about it, I'll remove the tag.

Quoted 6 lines

I don't feel that strongly about it :>)

Quoted 2 lines

Thanks for the review!

No problem! Working through your vma killable series atm also :)

Thanks!
I will need to respin v5 after Sashiko found a couple of legit issues at [1] but I'll wait for you to post your feedback before sending it.

[1] https://sashiko.dev/#/patchset/[email protected]

Quoted 6 lines
Quoted 4 lines
Quoted 4 lines

Thanks, Lorenzo

Cheers, Lorenzo