mm: memcg/slab: optimize objcg stock draining
Subsystems (2)
PATCH 1/1 mm: memcg/slab: optimize objcg stock draining
Imran Khan reported a regression in hackbench results caused by the commit f2fe7b09a52b ("mm: memcg/slab: charge individual slab objects instead of pages"). The regression is noticeable in the case of a consequent allocation of several relatively large slab objects, e.g. skb's. As soon as the amount of stocked bytes exceeds PAGE_SIZE, drain_obj_stock() and __memcg_kmem_uncharge() are called, and it leads to a number of atomic operations in page_counter_uncharge().
The corresponding call graph is below (provided by Imran Khan):
|__alloc_skb | | | |__kmalloc_reserve.isra.61 | | | | | |__kmalloc_node_track_caller | | | | | | | |slab_pre_alloc_hook.constprop.88 | | | obj_cgroup_charge | | | | | | | | | |__memcg_kmem_charge | | | | | | | | | | | |page_counter_try_charge | | | | | | | | | |refill_obj_stock | | | | | | | | | | | |drain_obj_stock.isra.68 | | | | | | | | | | | | | |__memcg_kmem_uncharge | | | | | | | | | | | | | | | |page_counter_uncharge | | | | | | | | | | | | | | | | | |page_counter_cancel | | | | | | | | | | | |__slab_alloc | | | | | | | | | |___slab_alloc | | | | | | | | |slab_post_alloc_hook
Instead of directly uncharging the accounted kernel memory, it's possible to refill the generic page-sized per-cpu stock instead. It's a much faster operation, especially on a default hierarchy. As a bonus, __memcg_kmem_uncharge_page() will also get faster, so the freeing of page-sized kernel allocations (e.g. large kmallocs) will become faster.
A similar change has been done earlier for the socket memory by the commit 475d0487a2ad ("mm: memcontrol: use per-cpu stocks for socket memory uncharging").
--- mm/memcontrol.c | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-)
mm/memcontrol.c +1 -3
@@ -3122,9 +3122,7 @@ void __memcg_kmem_uncharge(struct mem_cgroup *memcg, unsigned int nr_pages)if (!cgroup_subsys_on_dfl (memory_cgrp_subsys))page_counter_uncharge (&memcg->kmem, nr_pages); -page_counter_uncharge (&memcg->memory, nr_pages); -if (do_memsw_account ()) -page_counter_uncharge (&memcg->memsw, nr_pages); +refill_stock (memcg, nr_pages); }/**
Signature · 1 line
2.26.2
Quoted 47 lines · Roman Gushchin
Imran Khan reported a regression in hackbench results caused by the commit
f2fe7b09a52b("mm: memcg/slab: charge individual slab objects instead of pages"). The regression is noticeable in the case of a consequent allocation of several relatively large slab objects, e.g. skb's. As soon as the amount of stocked bytes exceedsPAGE_SIZE,drain_obj_stock()and__memcg_kmem_uncharge()are called, and it leads to a number of atomic operations inpage_counter_uncharge().The corresponding call graph is below (provided by Imran Khan):
|__alloc_skb | | | |__kmalloc_reserve.isra.61 | | | | | |__kmalloc_node_track_caller | | | | | | | |slab_pre_alloc_hook.constprop.88 | | | obj_cgroup_charge | | | | | | | | | |__memcg_kmem_charge | | | | | | | | | | | |page_counter_try_charge | | | | | | | | | |refill_obj_stock | | | | | | | | | | | |drain_obj_stock.isra.68 | | | | | | | | | | | | | |__memcg_kmem_uncharge | | | | | | | | | | | | | | | |page_counter_uncharge | | | | | | | | | | | | | | | | | |page_counter_cancel | | | | | | | | | | | |__slab_alloc | | | | | | | | | |___slab_alloc | | | | | | | | |slab_post_alloc_hookInstead of directly uncharging the accounted kernel memory, it's possible to refill the generic page-sized per-cpu stock instead. It's a much faster operation, especially on a default hierarchy. As a bonus,
__memcg_kmem_uncharge_page()will also get faster, so the freeing of page-sized kernel allocations (e.g. large kmallocs) will become faster.
Quoted 6 lines
A similar change has been done earlier for the socket memory by the commit
475d0487a2ad("mm: memcontrol: use per-cpu stocks for socket memory uncharging").Signed-off-by Roman Gushchin <[email protected]>Reported-by Imran Khan <[email protected]>
I remember seeing this somewhere
https://lore.kernel.org/linux-mm/[email protected]/
Quoted 52 lines · Shakeel Butt
Quoted 51 lines · Roman Gushchin
Imran Khan reported a regression in hackbench results caused by the commit
f2fe7b09a52b("mm: memcg/slab: charge individual slab objects instead of pages"). The regression is noticeable in the case of a consequent allocation of several relatively large slab objects, e.g. skb's. As soon as the amount of stocked bytes exceedsPAGE_SIZE,drain_obj_stock()and__memcg_kmem_uncharge()are called, and it leads to a number of atomic operations inpage_counter_uncharge().The corresponding call graph is below (provided by Imran Khan):
|__alloc_skb | | | |__kmalloc_reserve.isra.61 | | | | | |__kmalloc_node_track_caller | | | | | | | |slab_pre_alloc_hook.constprop.88 | | | obj_cgroup_charge | | | | | | | | | |__memcg_kmem_charge | | | | | | | | | | | |page_counter_try_charge | | | | | | | | | |refill_obj_stock | | | | | | | | | | | |drain_obj_stock.isra.68 | | | | | | | | | | | | | |__memcg_kmem_uncharge | | | | | | | | | | | | | | | |page_counter_uncharge | | | | | | | | | | | | | | | | | |page_counter_cancel | | | | | | | | | | | |__slab_alloc | | | | | | | | | |___slab_alloc | | | | | | | | |slab_post_alloc_hookInstead of directly uncharging the accounted kernel memory, it's possible to refill the generic page-sized per-cpu stock instead. It's a much faster operation, especially on a default hierarchy. As a bonus,
__memcg_kmem_uncharge_page()will also get faster, so the freeing of page-sized kernel allocations (e.g. large kmallocs) will become faster.A similar change has been done earlier for the socket memory by the commit
475d0487a2ad("mm: memcontrol: use per-cpu stocks for socket memory uncharging").
Quoted 5 lines
Quoted 2 lines
Signed-off-by Roman Gushchin <[email protected]>Reported-by Imran Khan <[email protected]>I remember seeing this somewhere
https://lore.kernel.org/linux-mm/[email protected]/
Yes, we've discussed it a couple of times, as I remember. Looks like now we finally have a good reasoning/benchmark, thanks to Imran.
Quoted 2 lines
Reviewed-by Shakeel Butt <[email protected]>
Thank you for the review!
Quoted 46 lines · Roman Gushchin
Imran Khan reported a regression in hackbench results caused by the commit
f2fe7b09a52b("mm: memcg/slab: charge individual slab objects instead of pages"). The regression is noticeable in the case of a consequent allocation of several relatively large slab objects, e.g. skb's. As soon as the amount of stocked bytes exceedsPAGE_SIZE,drain_obj_stock()and__memcg_kmem_uncharge()are called, and it leads to a number of atomic operations inpage_counter_uncharge().The corresponding call graph is below (provided by Imran Khan):
|__alloc_skb | | | |__kmalloc_reserve.isra.61 | | | | | |__kmalloc_node_track_caller | | | | | | | |slab_pre_alloc_hook.constprop.88 | | | obj_cgroup_charge | | | | | | | | | |__memcg_kmem_charge | | | | | | | | | | | |page_counter_try_charge | | | | | | | | | |refill_obj_stock | | | | | | | | | | | |drain_obj_stock.isra.68 | | | | | | | | | | | | | |__memcg_kmem_uncharge | | | | | | | | | | | | | | | |page_counter_uncharge | | | | | | | | | | | | | | | | | |page_counter_cancel | | | | | | | | | | | |__slab_alloc | | | | | | | | | |___slab_alloc | | | | | | | | |slab_post_alloc_hookInstead of directly uncharging the accounted kernel memory, it's possible to refill the generic page-sized per-cpu stock instead. It's a much faster operation, especially on a default hierarchy. As a bonus,
__memcg_kmem_uncharge_page()will also get faster, so the freeing of page-sized kernel allocations (e.g. large kmallocs) will become faster.
Quoted 6 lines
A similar change has been done earlier for the socket memory by the commit
475d0487a2ad("mm: memcontrol: use per-cpu stocks for socket memory uncharging").Signed-off-by Roman Gushchin <[email protected]>Reported-by Imran Khan <[email protected]>
Quoted 20 lines
--- mm/memcontrol.c | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-)mm/memcontrol.c +1
-3@@ -3122,9 +3122,7 @@ void __memcg_kmem_uncharge(struct mem_cgroup *memcg, unsigned int nr_pages)if (!cgroup_subsys_on_dfl (memory_cgrp_subsys))page_counter_uncharge (&memcg->kmem, nr_pages); -page_counter_uncharge (&memcg->memory, nr_pages); -if (do_memsw_account ()) -page_counter_uncharge (&memcg->memsw, nr_pages); +refill_stock (memcg, nr_pages); }/**
Andrew Morton
maintainerQuoted 3 lines · On Tue, 5 Jan 2021 20:22:39 -0800 Roman Gushchin <[email protected]> wrote:
Imran Khan reported a regression in hackbench results caused by the commit
f2fe7b09a52b("mm: memcg/slab: charge individual slab objects instead of pages").
How large was the regression?
mm/memcontrol.c void __memcg_kmem_uncharge(struct mem_cgroup *memcg, unsigned int nr_pages) · line 3126
Quoted 11 lines
mm/memcontrol.c +1
-3@@ -3122,9 +3122,7 @@ void __memcg_kmem_uncharge(struct mem_cgroup *memcg, unsigned int nr_pages)if (!cgroup_subsys_on_dfl (memory_cgrp_subsys))page_counter_uncharge (&memcg->kmem, nr_pages); -page_counter_uncharge (&memcg->memory, nr_pages); -if (do_memsw_account ()) -page_counter_uncharge (&memcg->memsw, nr_pages); +refill_stock (memcg, nr_pages); }
IOW, which kernel version(s) should we be patching?
Quoted 7 lines · Andrew Morton
Quoted 3 lines · On Tue, 5 Jan 2021 20:22:39 -0800 Roman Gushchin <[email protected]> wrote:
Imran Khan reported a regression in hackbench results caused by the commit
f2fe7b09a52b("mm: memcg/slab: charge individual slab objects instead of pages").How large was the regression?
~16% according to Imran's data.
Quoted 7 lines
Quoted 7 lines
mm/memcontrol.c +0
-0@@ -3122,9 +3122,7 @@ void __memcg_kmem_uncharge(struct mem_cgroup *memcg, unsigned int nr_pages)if (!cgroup_subsys_on_dfl (memory_cgrp_subsys))page_counter_uncharge (&memcg->kmem, nr_pages);
Quoted 7 lines
mm/memcontrol.c void __memcg_kmem_uncharge(struct mem_cgroup *memcg, unsigned int nr_pages) · line 3126
Quoted 5 lines
mm/memcontrol.c +1
-3-page_counter_uncharge (&memcg->memory, nr_pages); -if (do_memsw_account ()) -page_counter_uncharge (&memcg->memsw, nr_pages); +refill_stock (memcg, nr_pages); }IOW, which kernel version(s) should we be patching?
5.9+
Thanks!
Quoted 32 lines · Roman Gushchin
Imran Khan reported a regression in hackbench results caused by the commit
f2fe7b09a52b("mm: memcg/slab: charge individual slab objects instead of pages"). The regression is noticeable in the case of a consequent allocation of several relatively large slab objects, e.g. skb's. As soon as the amount of stocked bytes exceedsPAGE_SIZE,drain_obj_stock()and__memcg_kmem_uncharge()are called, and it leads to a number of atomic operations inpage_counter_uncharge().The corresponding call graph is below (provided by Imran Khan):
|__alloc_skb | | | |__kmalloc_reserve.isra.61 | | | | | |__kmalloc_node_track_caller | | | | | | | |slab_pre_alloc_hook.constprop.88 | | | obj_cgroup_charge | | | | | | | | | |__memcg_kmem_charge | | | | | | | | | | | |page_counter_try_charge | | | | | | | | | |refill_obj_stock | | | | | | | | | | | |drain_obj_stock.isra.68 <--- draining old memcg | | | | | | | | | | | | | |__memcg_kmem_uncharge | | | | | | | | | | | | | | | |page_counter_uncharge | | | | | | | | | | | | | | | | | |page_counter_cancel[...]
mm/memcontrol.c void __memcg_kmem_uncharge(struct mem_cgroup *memcg, unsigned int nr_pages) · line 3125
Quoted 4 lines
mm/memcontrol.c +1
-3-page_counter_uncharge (&memcg->memory, nr_pages); -if (do_memsw_account ()) -page_counter_uncharge (&memcg->memsw, nr_pages); +refill_stock (memcg, nr_pages);
I noticed that refill_stock(memcg,...) switches the local stock to memcg.
In this particular call chain, the uncharged memcg is the old memcg of stock->cached_objcg. The refill_stock() then may switch stock->cached to the old memcg too. If the patch leads to better performance, then the switch probably doesn't happen at this moment (and I guess stock->cached_objcg and stock->cached can be independent to some extent, so the old memcg in one needn't be the old in the latter).
In conclusion
Michal