memcg: expose children memory usage for root

Linux kernel does not expose memory.current on the root memcg and there are applications which have to traverse all the top level memcgs to calculate the total memory charged in the system. This is more expensive (directory traversal and multiple open and reads) and is racy on a busy machine. As the kernel already have the needed information i.e. root's memory.current, why not expose that?

However root's memory.current will have a different semantics than the non-root's memory.current as the kernel skips the charging for root, so maybe it is better to have a different named interface for the root. Something like memory.children_usage only for root memcg.

Now there is still a question that why the kernel does not expose memory.current for the root. The historical reason was that the memcg charging was expensice and to provide the users to bypass the memcg charging by letting them run in the root. However do we still want to have this exception today? What is stopping us to start charging the root memcg as well. Of course the root will not have limits but the allocations will go through memcg charging and then the memory.current of root and non-root will have the same semantics.

This is an RFC to start a discussion on memcg charging for root.

Signed-off-by Shakeel Butt <[email protected]>

---
 Documentation/admin-guide/cgroup-v2.rst | 6 ++++++
 mm/memcontrol.c                         | 5 +++++
 2 files changed, 11 insertions(+)

Documentation/admin-guide/cgroup-v2.rst +6 -0

index 6c6075ed4aa5..e4afc05fd8ea 100644
@@ -1220,6 +1220,12 @@ PAGE_SIZE multiple when read back.
        The total amount of memory currently being used by the cgroup
        and its descendants.
 
+  memory.children_usage
+       A read-only single value file which exists only on root cgroup.
+
+       The total amount of memory currently being used by the
+        descendants of the root cgroup.
+
   memory.min
        A read-write single value file which exists on non-root
        cgroups.  The default is "0".

mm/memcontrol.c +5 -1

index 960371788687..eba8cf76d3d3 100644
@@ -4304,6 +4304,11 @@ static struct cftype memory_files[] = {
                .flags = CFTYPE_NOT_ON_ROOT,
                .read_u64 = memory_current_read,
        },
+       {
+               .name = "children_usage",
+               .flags = CFTYPE_ONLY_ON_ROOT,
+               .read_u64 = memory_current_read,
+       },
        {
                .name = "peak",
                .flags = CFTYPE_NOT_ON_ROOT,
--

2.43.0

Linux kernel does not expose memory.current on the root memcg and there are applications which have to traverse all the top level memcgs to calculate the total memory charged in the system. This is more expensive (directory traversal and multiple open and reads) and is racy on a busy machine. As the kernel already have the needed information i.e. root's memory.current, why not expose that?

However root's memory.current will have a different semantics than the non-root's memory.current as the kernel skips the charging for root, so maybe it is better to have a different named interface for the root. Something like memory.children_usage only for root memcg.

Now there is still a question that why the kernel does not expose memory.current for the root. The historical reason was that the memcg charging was expensice and to provide the users to bypass the memcg charging by letting them run in the root. However do we still want to have this exception today? What is stopping us to start charging the root memcg as well. Of course the root will not have limits but the allocations will go through memcg charging and then the memory.current of root and non-root will have the same semantics.

This is an RFC to start a discussion on memcg charging for root.

Signed-off-by Shakeel Butt <[email protected]>

---
 Documentation/admin-guide/cgroup-v2.rst | 6 ++++++
 mm/memcontrol.c                         | 5 +++++
 2 files changed, 11 insertions(+)

Documentation/admin-guide/cgroup-v2.rst +6 -0

index 6c6075ed4aa5..e4afc05fd8ea 100644
@@ -1220,6 +1220,12 @@ PAGE_SIZE multiple when read back.
        The total amount of memory currently being used by the cgroup
        and its descendants.
 
+  memory.children_usage
+       A read-only single value file which exists only on root cgroup.
+
+       The total amount of memory currently being used by the
+        descendants of the root cgroup.
+
   memory.min
        A read-write single value file which exists on non-root
        cgroups.  The default is "0".

mm/memcontrol.c +5 -1

index 960371788687..eba8cf76d3d3 100644
@@ -4304,6 +4304,11 @@ static struct cftype memory_files[] = {
                .flags = CFTYPE_NOT_ON_ROOT,
                .read_u64 = memory_current_read,
        },
+       {
+               .name = "children_usage",
+               .flags = CFTYPE_ONLY_ON_ROOT,
+               .read_u64 = memory_current_read,
+       },
        {
                .name = "peak",
                .flags = CFTYPE_NOT_ON_ROOT,
--

2.43.0

Quoted 22 lines · Shakeel Butt

Linux kernel does not expose memory.current on the root memcg and there are applications which have to traverse all the top level memcgs to calculate the total memory charged in the system. This is more expensive (directory traversal and multiple open and reads) and is racy on a busy machine. As the kernel already have the needed information i.e. root's memory.current, why not expose that?

However root's memory.current will have a different semantics than the non-root's memory.current as the kernel skips the charging for root, so maybe it is better to have a different named interface for the root. Something like memory.children_usage only for root memcg.

Now there is still a question that why the kernel does not expose memory.current for the root. The historical reason was that the memcg charging was expensice and to provide the users to bypass the memcg charging by letting them run in the root. However do we still want to have this exception today? What is stopping us to start charging the root memcg as well. Of course the root will not have limits but the allocations will go through memcg charging and then the memory.current of root and non-root will have the same semantics.

Quoted 1 line

This is an RFC to start a discussion on memcg charging for root.

Hi Shakeel,

Since the root already has a page_counter I'm not opposed to this new file as it doesn't increase the page_counter depth for children. However I don't currently have any use-cases for it that wouldn't be met by memory.stat in the root.

As far as charging, I've only ever seen kthreads and init in the root. You have workloads that run there?

Quoted 22 lines · Shakeel Butt

Linux kernel does not expose memory.current on the root memcg and there are applications which have to traverse all the top level memcgs to calculate the total memory charged in the system. This is more expensive (directory traversal and multiple open and reads) and is racy on a busy machine. As the kernel already have the needed information i.e. root's memory.current, why not expose that?

However root's memory.current will have a different semantics than the non-root's memory.current as the kernel skips the charging for root, so maybe it is better to have a different named interface for the root. Something like memory.children_usage only for root memcg.

Now there is still a question that why the kernel does not expose memory.current for the root. The historical reason was that the memcg charging was expensice and to provide the users to bypass the memcg charging by letting them run in the root. However do we still want to have this exception today? What is stopping us to start charging the root memcg as well. Of course the root will not have limits but the allocations will go through memcg charging and then the memory.current of root and non-root will have the same semantics.

Quoted 1 line

This is an RFC to start a discussion on memcg charging for root.

I vaguely remember when running some netperf tests (tcp_rr?) in a cgroup that the performance decreases considerably with every level down the hierarchy. I am assuming that charging was a part of the reason. If that's the case, charging the root will be similar to moving all workloads one level down the hierarchy in terms of charging overhead.

Quoted 23 lines · Yosry Ahmed

Quoted 22 lines · Shakeel Butt

Linux kernel does not expose memory.current on the root memcg and there are applications which have to traverse all the top level memcgs to calculate the total memory charged in the system. This is more expensive (directory traversal and multiple open and reads) and is racy on a busy machine. As the kernel already have the needed information i.e. root's memory.current, why not expose that?

However root's memory.current will have a different semantics than the non-root's memory.current as the kernel skips the charging for root, so maybe it is better to have a different named interface for the root. Something like memory.children_usage only for root memcg.

Now there is still a question that why the kernel does not expose memory.current for the root. The historical reason was that the memcg charging was expensice and to provide the users to bypass the memcg charging by letting them run in the root. However do we still want to have this exception today? What is stopping us to start charging the root memcg as well. Of course the root will not have limits but the allocations will go through memcg charging and then the memory.current of root and non-root will have the same semantics.

Quoted 8 lines

Quoted 1 line

This is an RFC to start a discussion on memcg charging for root.

I vaguely remember when running some netperf tests (tcp_rr?) in a cgroup that the performance decreases considerably with every level down the hierarchy. I am assuming that charging was a part of the reason. If that's the case, charging the root will be similar to moving all workloads one level down the hierarchy in terms of charging overhead.

No, the workloads running in non-root memcgs will not see any difference. Only the workloads running in root will see charging overhead.

Hi T.J.

Quoted 27 lines · T.J. Mercier

Quoted 22 lines · Shakeel Butt

Linux kernel does not expose memory.current on the root memcg and there are applications which have to traverse all the top level memcgs to calculate the total memory charged in the system. This is more expensive (directory traversal and multiple open and reads) and is racy on a busy machine. As the kernel already have the needed information i.e. root's memory.current, why not expose that?

However root's memory.current will have a different semantics than the non-root's memory.current as the kernel skips the charging for root, so maybe it is better to have a different named interface for the root. Something like memory.children_usage only for root memcg.

Now there is still a question that why the kernel does not expose memory.current for the root. The historical reason was that the memcg charging was expensice and to provide the users to bypass the memcg charging by letting them run in the root. However do we still want to have this exception today? What is stopping us to start charging the root memcg as well. Of course the root will not have limits but the allocations will go through memcg charging and then the memory.current of root and non-root will have the same semantics.

Quoted 1 line

This is an RFC to start a discussion on memcg charging for root.

Hi Shakeel,

Quoted 4 lines

Since the root already has a page_counter I'm not opposed to this new file as it doesn't increase the page_counter depth for children. However I don't currently have any use-cases for it that wouldn't be met by memory.stat in the root.

I think difference would be getting a single number versus accumulating different fields in memory.stat to get that number (memory used by root's children) which might be a bit error prone.

Quoted 2 lines

As far as charging, I've only ever seen kthreads and init in the root. You have workloads that run there?

No workloads in root. The charging is only to make the semanctics of root's memory.current same as its descendants.

Thanks,
Shakeel

Quoted 34 lines · Shakeel Butt

Quoted 23 lines · Yosry Ahmed

Quoted 22 lines · Shakeel Butt

Linux kernel does not expose memory.current on the root memcg and there are applications which have to traverse all the top level memcgs to calculate the total memory charged in the system. This is more expensive (directory traversal and multiple open and reads) and is racy on a busy machine. As the kernel already have the needed information i.e. root's memory.current, why not expose that?

However root's memory.current will have a different semantics than the non-root's memory.current as the kernel skips the charging for root, so maybe it is better to have a different named interface for the root. Something like memory.children_usage only for root memcg.

Now there is still a question that why the kernel does not expose memory.current for the root. The historical reason was that the memcg charging was expensice and to provide the users to bypass the memcg charging by letting them run in the root. However do we still want to have this exception today? What is stopping us to start charging the root memcg as well. Of course the root will not have limits but the allocations will go through memcg charging and then the memory.current of root and non-root will have the same semantics.

Quoted 8 lines

Quoted 1 line

This is an RFC to start a discussion on memcg charging for root.

I vaguely remember when running some netperf tests (tcp_rr?) in a cgroup that the performance decreases considerably with every level down the hierarchy. I am assuming that charging was a part of the reason. If that's the case, charging the root will be similar to moving all workloads one level down the hierarchy in terms of charging overhead.

Quoted 3 lines

No, the workloads running in non-root memcgs will not see any difference. Only the workloads running in root will see charging overhead.

Oh yeah we already charge the root's page counters hierarchically in the upstream kernel, we just do not charge them if the origin of the charge is the root itself.

We also have workloads that iterate top-level memcgs to calculate the total charged memory, so memory.children_usage for the root memcg would help.

As for memory.current, do you have any data about how much memory is charged to the root itself?

We think of the memory charged to the root
as system overhead, while the memory charged to top-level memcgs isn't.

So basically total_memory - root::memory.children_usage would be a fast way to get a rough estimation of system overhead. The same would not apply for total_memory - root::memory.current if I understand correctly.

Quoted 34 lines · Shakeel Butt

Quoted 23 lines · Yosry Ahmed

Quoted 22 lines · Shakeel Butt

Linux kernel does not expose memory.current on the root memcg and there are applications which have to traverse all the top level memcgs to calculate the total memory charged in the system. This is more expensive (directory traversal and multiple open and reads) and is racy on a busy machine. As the kernel already have the needed information i.e. root's memory.current, why not expose that?

However root's memory.current will have a different semantics than the non-root's memory.current as the kernel skips the charging for root, so maybe it is better to have a different named interface for the root. Something like memory.children_usage only for root memcg.

Now there is still a question that why the kernel does not expose memory.current for the root. The historical reason was that the memcg charging was expensice and to provide the users to bypass the memcg charging by letting them run in the root. However do we still want to have this exception today? What is stopping us to start charging the root memcg as well. Of course the root will not have limits but the allocations will go through memcg charging and then the memory.current of root and non-root will have the same semantics.

Quoted 8 lines

Quoted 1 line

This is an RFC to start a discussion on memcg charging for root.

I vaguely remember when running some netperf tests (tcp_rr?) in a cgroup that the performance decreases considerably with every level down the hierarchy. I am assuming that charging was a part of the reason. If that's the case, charging the root will be similar to moving all workloads one level down the hierarchy in terms of charging overhead.

Quoted 3 lines

No, the workloads running in non-root memcgs will not see any difference. Only the workloads running in root will see charging overhead.

Oh yeah we already charge the root's page counters hierarchically in the upstream kernel, we just do not charge them if the origin of the charge is the root itself.

We also have workloads that iterate top-level memcgs to calculate the total charged memory, so memory.children_usage for the root memcg would help.

As for memory.current, do you have any data about how much memory is charged to the root itself? We think of the memory charged to the root as system overhead, while the memory charged to top-level memcgs isn't.

So basically total_memory - root::memory.children_usage would be a fast way to get a rough estimation of system overhead. The same would not apply for total_memory - root::memory.current if I understand correctly.

Quoted 1 line · Yosry Ahmed

Quoted 9 lines

Oh yeah we already charge the root's page counters hierarchically in the upstream kernel, we just do not charge them if the origin of the charge is the root itself.

We also have workloads that iterate top-level memcgs to calculate the total charged memory, so memory.children_usage for the root memcg would help.

Quoted 2 lines

As for memory.current, do you have any data about how much memory is charged to the root itself?

Yeah I wonder if we'd be able to see any significant regressions for stuff that lives there today if we were to start charging it. I can try running a test with Android next week. I guess try_charge() is the main thing that would need to change to allow root charges?

Quoted 1 line · Shakeel Butt

Quoted 4 lines

No, the workloads running in non-root memcgs will not see any difference. Only the workloads running in root will see charging overhead.

Oh yeah we already charge the root's page counters hierarchically in the upstream kernel, we just do not charge them if the origin of the charge is the root itself.

We also have workloads that iterate top-level memcgs to calculate the total charged memory, so memory.children_usage for the root memcg would help.

As for memory.current, do you have any data about how much memory is charged to the root itself? We think of the memory charged to the root as system overhead, while the memory charged to top-level memcgs isn't.

So basically total_memory - root::memory.children_usage would be a fast way to get a rough estimation of system overhead. The same would not apply for total_memory - root::memory.current if I understand correctly.

Quoted 2 lines · Yosry Ahmed

Quoted 1 line · Shakeel Butt

[...]

Quoted 18 lines

Quoted 4 lines

No, the workloads running in non-root memcgs will not see any difference. Only the workloads running in root will see charging overhead.

Oh yeah we already charge the root's page counters hierarchically in the upstream kernel, we just do not charge them if the origin of the charge is the root itself.

We also have workloads that iterate top-level memcgs to calculate the total charged memory, so memory.children_usage for the root memcg would help.

As for memory.current, do you have any data about how much memory is charged to the root itself? We think of the memory charged to the root as system overhead, while the memory charged to top-level memcgs isn't.

Quoted 4 lines

So basically total_memory - root::memory.children_usage would be a fast way to get a rough estimation of system overhead. The same would not apply for total_memory - root::memory.current if I understand correctly.

Please note that root::memory.children_usage will have top level zombies included as well (at least until lru reparenting is not done). So, for your example it would provide good estimation of top level zombie memory through root::memory.children_usage - top_level(alive)::memory.current.

Quoted 28 lines · Shakeel Butt

Quoted 2 lines · Yosry Ahmed

Quoted 1 line · Shakeel Butt

[...]

Quoted 22 lines

Quoted 4 lines

No, the workloads running in non-root memcgs will not see any difference. Only the workloads running in root will see charging overhead.

Oh yeah we already charge the root's page counters hierarchically in the upstream kernel, we just do not charge them if the origin of the charge is the root itself.

We also have workloads that iterate top-level memcgs to calculate the total charged memory, so memory.children_usage for the root memcg would help.

As for memory.current, do you have any data about how much memory is charged to the root itself? We think of the memory charged to the root as system overhead, while the memory charged to top-level memcgs isn't.

So basically total_memory - root::memory.children_usage would be a fast way to get a rough estimation of system overhead. The same would not apply for total_memory - root::memory.current if I understand correctly.

Quoted 4 lines

Please note that root::memory.children_usage will have top level zombies included as well (at least until lru reparenting is not done). So, for your example it would provide good estimation of top level zombie memory through root::memory.children_usage - top_level(alive)::memory.current.

Good point. The fact that it includes the top-level zombies makes it less valuable for this use case, as zombie memory is considered system overhead as well. So we need to iterate the top level memcgs anyway.