Skip to contentSkip to conversation

A path forward to cleaning up dying cgroups?

DISCUSSION Hamza Mahfooz 9 participants 10 messages Feb 5, 2025 – Feb 6, 2025

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

I was just curious as to what the status of the issue described in [1] is. It appears that the last time someone took a stab at it was in [2].

Though it seems like there has been relative silence regarding it since then. So, has there been any discussion regarding the issue since then and does anyone know if there is consensus on how we should go about resolving this issue within the kernel?

BR,
Hamza

[1] https://lwn.net/Articles/895431/
[2] https://lore.kernel.org/r/[email protected]/

#1 lore raw
Cc Shakeel Butt <[email protected]>
Quoted 13 lines · Hamza Mahfooz

I was just curious as to what the status of the issue described in [1] is. It appears that the last time someone took a stab at it was in [2].

Though it seems like there has been relative silence regarding it since then. So, has there been any discussion regarding the issue since then and does anyone know if there is consensus on how we should go about resolving this issue within the kernel?

BR,
Hamza

[1] https://lwn.net/Articles/895431/
[2] https://lore.kernel.org/r/[email protected]/

maintainer
#2 lore raw
Quoted 5 lines · Hamza Mahfooz
Cc Shakeel Butt <[email protected]>
Quoted 2 lines · Hamza Mahfooz

I was just curious as to what the status of the issue described in [1] is. It appears that the last time someone took a stab at it was in [2].

If memory serves, the sticking point was whether pages should indeed be reparented on cgroup death, or whether they could be moved arbitrarily to other cgroups that are still using them.

It's a bit unfortunate, because the reparenting patches were tested and reviewed, and the arbitrary recharging was just an idea that ttbomk nobody seriously followed up on afterwards.

We also recently removed the charge moving code from cgroup1, along with the subtle page access/locking/accounting rules it imposed on the rest of the MM. I'm doubtful there is much appetite in either camp for bringing this back.

So I would still love to see Muchun's patches merged. They fix a seemingly universally experienced operational issue in memcg, and we shouldn't hold it up unless somebody actually posts alternative code.

Thoughts?

#3 lore raw
Quoted 7 lines · Johannes Weiner
Quoted 5 lines · Hamza Mahfooz
Cc Shakeel Butt <[email protected]>
Quoted 2 lines · Hamza Mahfooz

I was just curious as to what the status of the issue described in [1] is. It appears that the last time someone took a stab at it was in [2].

Quoted 7 lines

If memory serves, the sticking point was whether pages should indeed be reparented on cgroup death, or whether they could be moved arbitrarily to other cgroups that are still using them.

It's a bit unfortunate, because the reparenting patches were tested and reviewed, and the arbitrary recharging was just an idea that ttbomk nobody seriously followed up on afterwards.

There was an RFC series [1] for the recharging, but all memcg maintainers hated it :P

https://lore.kernel.org/lkml/[email protected]/

Quoted 5 lines

We also recently removed the charge moving code from cgroup1, along with the subtle page access/locking/accounting rules it imposed on the rest of the MM. I'm doubtful there is much appetite in either camp for bringing this back.

Yeah with the charge moving code gone the case for recharging grows weaker.

Quoted 6 lines

So I would still love to see Muchun's patches merged. They fix a seemingly universally experienced operational issue in memcg, and we shouldn't hold it up unless somebody actually posts alternative code.

Thoughts?

Adding Zach and Kinsey who were recently looking into this from the Google side.

#4 lore raw
Quoted 13 lines · Yosry Ahmed
Quoted 11 lines · Johannes Weiner
Quoted 5 lines · Hamza Mahfooz
Cc Shakeel Butt <[email protected]>
Quoted 2 lines · Hamza Mahfooz

I was just curious as to what the status of the issue described in [1] is. It appears that the last time someone took a stab at it was in [2].

If memory serves, the sticking point was whether pages should indeed be reparented on cgroup death, or whether they could be moved arbitrarily to other cgroups that are still using them.

Quoted 8 lines
Quoted 3 lines

It's a bit unfortunate, because the reparenting patches were tested and reviewed, and the arbitrary recharging was just an idea that ttbomk nobody seriously followed up on afterwards.

There was an RFC series [1] for the recharging, but all memcg maintainers hated it :P

https://lore.kernel.org/lkml/[email protected]/

We have been suffering from dying cgroup issues for years too, and I just saw this series. Will it be a good idea to combine this with reparenting instead (if we will go with the reparenting approach)? Using objcg API to charge the folios does help speed up the reparenting, but also adds some overhead and complexity. Just walking and reparenting the folios seems a more direct approach.

And another idea is, per our observation, dying cgroups have few pages that are mapped, as the process has all exited. Most folios are just cache. Shared mapped pages are minor especially for containers. So a deferred recharge on access seems good enough? Mapped folios may also be finally unmap someday and get recharged. And at least this makes accounting more accurate.

#5 lore raw
In reply to Johannes Weiner
Quoted 20 lines · Johannes Weiner
Quoted 5 lines · Hamza Mahfooz
Cc Shakeel Butt <[email protected]>
Quoted 2 lines · Hamza Mahfooz

I was just curious as to what the status of the issue described in [1] is. It appears that the last time someone took a stab at it was in [2].

If memory serves, the sticking point was whether pages should indeed be reparented on cgroup death, or whether they could be moved arbitrarily to other cgroups that are still using them.

It's a bit unfortunate, because the reparenting patches were tested and reviewed, and the arbitrary recharging was just an idea that ttbomk nobody seriously followed up on afterwards.

We also recently removed the charge moving code from cgroup1, along with the subtle page access/locking/accounting rules it imposed on the rest of the MM. I'm doubtful there is much appetite in either camp for bringing this back.

Quoted 5 lines

So I would still love to see Muchun's patches merged. They fix a seemingly universally experienced operational issue in memcg, and we shouldn't hold it up unless somebody actually posts alternative code.

Thoughts?

I don't have a strong opinion here. Reparenting is clearly not perfect, but I agree that we don't have any better solutions, only vague ideas. I believe Muchun's code would require some refresh, but generally is fine to merge.

This all comes up to the handling of memory shared between cgroups. Sharing can be spatial (2 or more simultaneously existing cgroups) or temporal (a cgroup is being deleted and recreated, the workload tries to reuse old pages). The reparenting turns temporal sharing into the spacial. It helps with dying cgroups, but comes at the cost of permanently wrong accounting and issues with the memory protection.

Thanks!

#6 lore raw
In reply to Johannes Weiner
Quoted 20 lines · Johannes Weiner
Quoted 5 lines · Hamza Mahfooz
Cc Shakeel Butt <[email protected]>
Quoted 2 lines · Hamza Mahfooz

I was just curious as to what the status of the issue described in [1] is. It appears that the last time someone took a stab at it was in [2].

If memory serves, the sticking point was whether pages should indeed be reparented on cgroup death, or whether they could be moved arbitrarily to other cgroups that are still using them.

It's a bit unfortunate, because the reparenting patches were tested and reviewed, and the arbitrary recharging was just an idea that ttbomk nobody seriously followed up on afterwards.

We also recently removed the charge moving code from cgroup1, along with the subtle page access/locking/accounting rules it imposed on the rest of the MM. I'm doubtful there is much appetite in either camp for bringing this back.

Quoted 5 lines

So I would still love to see Muchun's patches merged. They fix a seemingly universally experienced operational issue in memcg, and we shouldn't hold it up unless somebody actually posts alternative code.

Thoughts?

I think the recharging (or whatever the alternative) can be a followup to this. I agree this is a good change.

#7 lore raw
Quoted 25 lines · Shakeel Butt
Quoted 20 lines · Johannes Weiner
Quoted 5 lines · Hamza Mahfooz
Cc Shakeel Butt <[email protected]>
Quoted 2 lines · Hamza Mahfooz

I was just curious as to what the status of the issue described in [1] is. It appears that the last time someone took a stab at it was in [2].

If memory serves, the sticking point was whether pages should indeed be reparented on cgroup death, or whether they could be moved arbitrarily to other cgroups that are still using them.

It's a bit unfortunate, because the reparenting patches were tested and reviewed, and the arbitrary recharging was just an idea that ttbomk nobody seriously followed up on afterwards.

We also recently removed the charge moving code from cgroup1, along with the subtle page access/locking/accounting rules it imposed on the rest of the MM. I'm doubtful there is much appetite in either camp for bringing this back.

Quoted 8 lines
Quoted 5 lines

So I would still love to see Muchun's patches merged. They fix a seemingly universally experienced operational issue in memcg, and we shouldn't hold it up unless somebody actually posts alternative code.

Thoughts?

I think the recharging (or whatever the alternative) can be a followup to this. I agree this is a good change.

I agree with you. We've been encountering dying memory issues for years on our servers. As Roman said, I need to refresh my patches. So I need some time for refreshing.

Muchun,
Thanks.

#8 lore raw
Quoted 27 lines · Muchun Song
Quoted 27 lines · Shakeel Butt
Quoted 23 lines · Johannes Weiner
Quoted 5 lines · Hamza Mahfooz
Cc Shakeel Butt <[email protected]>
Quoted 2 lines · Hamza Mahfooz

I was just curious as to what the status of the issue described in [1] is. It appears that the last time someone took a stab at it was in [2].

If memory serves, the sticking point was whether pages should indeed be reparented on cgroup death, or whether they could be moved arbitrarily to other cgroups that are still using them.

It's a bit unfortunate, because the reparenting patches were tested and reviewed, and the arbitrary recharging was just an idea that ttbomk nobody seriously followed up on afterwards.

We also recently removed the charge moving code from cgroup1, along with the subtle page access/locking/accounting rules it imposed on the rest of the MM. I'm doubtful there is much appetite in either camp for bringing this back.

So I would still love to see Muchun's patches merged. They fix a seemingly universally experienced operational issue in memcg, and we shouldn't hold it up unless somebody actually posts alternative code.

Quoted 6 lines
Quoted 3 lines
Quoted 1 line

Thoughts?

I think the recharging (or whatever the alternative) can be a followup to this. I agree this is a good change.

I agree with you. We've been encountering dying memory issues for years on our servers. As Roman said, I need to refresh my patches. So I need some time for refreshing.

Glad to hear that. I have been waiting for a resolution of the dying memory cgroup problems for years :-)

Cheers,
Longman

#9 lore raw
In reply to Muchun Song
Quoted 29 lines · Muchun Song
Quoted 29 lines · Shakeel Butt
Quoted 24 lines · Johannes Weiner
Quoted 5 lines · Hamza Mahfooz
Cc Shakeel Butt <[email protected]>
Quoted 2 lines · Hamza Mahfooz

I was just curious as to what the status of the issue described in [1] is. It appears that the last time someone took a stab at it was in [2].

If memory serves, the sticking point was whether pages should indeed be reparented on cgroup death, or whether they could be moved arbitrarily to other cgroups that are still using them.

It's a bit unfortunate, because the reparenting patches were tested and reviewed, and the arbitrary recharging was just an idea that ttbomk nobody seriously followed up on afterwards.

We also recently removed the charge moving code from cgroup1, along with the subtle page access/locking/accounting rules it imposed on the rest of the MM. I'm doubtful there is much appetite in either camp for bringing this back.

So I would still love to see Muchun's patches merged. They fix a seemingly universally experienced operational issue in memcg, and we shouldn't hold it up unless somebody actually posts alternative code.

Quoted 9 lines
Quoted 4 lines
Quoted 1 line

Thoughts?

I think the recharging (or whatever the alternative) can be a followup to this. I agree this is a good change.

I agree with you. We've been encountering dying memory issues for years on our servers. As Roman said, I need to refresh my patches. So I need some time for refreshing.

We have seen the dying cgroups issue too and look forward to your patches. Happy to help with testing/reviewing.

Signature · 2 lines
Thanks,
Kamalesh