mm, memcg: inline mem_cgroup_{charge/uncharge} to improve disabled memcg config
Subsystems (3)
PATCH v2 1/1 mm, memcg: inline mem_cgroup_{charge/uncharge} to improve disabled memcg config
Inline mem_cgroup_{charge/uncharge} and mem_cgroup_uncharge_list functions functions to perform mem_cgroup_disabled static key check inline before calling the main body of the function. This minimizes the memcg overhead in the pagefault and exit_mmap paths when memcgs are disabled using cgroup_disable=memory command-line option.
This change results in ~0.4% overhead reduction when running PFT test comparing {CONFIG_MEMCG=n} against {CONFIG_MEMCG=y, cgroup_disable=memory} configurationon on an 8-core ARM64 Android device.
---
changes in v2:
- Changed mem_cgroup_charge to use the same inlining pattern as the rest of the functions per Johannes.
include/linux/memcontrol.h | 28 +++++++++++++++++++++++++--- mm/memcontrol.c | 29 ++++++++++------------------- 2 files changed, 35 insertions(+), 22 deletions(-)
include/linux/memcontrol.h +25 -3
@@ -693,13 +693,35 @@ static inline bool mem_cgroup_below_min(struct mem_cgroup *memcg)page_counter_read (&memcg->memory); } -int mem_cgroup_charge (struct page *page,struct mm_struct *mm,gfp_t gfp_mask); +int __mem_cgroup_charge (struct page *page,struct mm_struct *mm, +gfp_t gfp_mask); +static inline int mem_cgroup_charge (struct page *page,struct mm_struct *mm, +gfp_t gfp_mask) +{ +if (mem_cgroup_disabled ()) +return 0 ; +return __mem_cgroup_charge (page, mm, gfp_mask); +} +int mem_cgroup_swapin_charge_page (struct page *page,struct mm_struct *mm,gfp_t gfp,swp_entry_t entry);void mem_cgroup_swapin_uncharge_swap (swp_entry_t entry); -void mem_cgroup_uncharge (struct page *page); -void mem_cgroup_uncharge_list (struct list_head *page_list); +void __mem_cgroup_uncharge (struct page *page); +static inline void mem_cgroup_uncharge (struct page *page) +{ +if (mem_cgroup_disabled ()) +return ; +__mem_cgroup_uncharge (page); +} + +void __mem_cgroup_uncharge_list (struct list_head *page_list); +static inline void mem_cgroup_uncharge_list (struct list_head *page_list) +{ +if (mem_cgroup_disabled ()) +return ; +__mem_cgroup_uncharge_list (page_list); +}void mem_cgroup_migrate (struct page *oldpage,struct page *newpage);
mm/memcontrol.c +10 -19
@@ -6701,8 +6701,7 @@ void mem_cgroup_calculate_protection(struct mem_cgroup *root, atomic_long_read(&parent->memory.children_low_usage))); } -static int __mem_cgroup_charge (struct page *page,struct mem_cgroup *memcg, -gfp_t gfp) +static int charge_memcg (struct page *page,struct mem_cgroup *memcg,gfp_t gfp) {unsigned int nr_pages =thp_nr_pages (page);int ret;
@@ -6723,7 +6722,7 @@ static int __mem_cgroup_charge(struct page *page, struct mem_cgroup *memcg, }/** -* mem_cgroup_charge - charge a newly allocated page to a cgroup +* __mem_cgroup_charge - charge a newly allocated page to a cgroup * @page: page to charge * @mm: mm context of the victim * @gfp_mask: reclaim mode
@@ -6736,16 +6735,14 @@ static int __mem_cgroup_charge(struct page *page, struct mem_cgroup *memcg,* * Returns 0 on success. Otherwise, an error code is returned. */ -int mem_cgroup_charge (struct page *page,struct mm_struct *mm,gfp_t gfp_mask) +int __mem_cgroup_charge (struct page *page,struct mm_struct *mm, +gfp_t gfp_mask) {struct mem_cgroup *memcg;int ret; -if (mem_cgroup_disabled ()) -return 0 ; - memcg =get_mem_cgroup_from_mm (mm); - ret =__mem_cgroup_charge (page, memcg, gfp_mask); + ret =charge_memcg (page, memcg, gfp_mask);css_put (&memcg->css);return ret;
@@ -6780,7 +6777,7 @@ int mem_cgroup_swapin_charge_page(struct page *page, struct mm_struct *mm, memcg =get_mem_cgroup_from_mm (mm);rcu_read_unlock (); - ret =__mem_cgroup_charge (page, memcg, gfp); + ret =charge_memcg (page, memcg, gfp);css_put (&memcg->css);return ret;
@@ -6916,18 +6913,15 @@ static void uncharge_page(struct page *page, struct uncharge_gather *ug) }/** -* mem_cgroup_uncharge - uncharge a page +* __mem_cgroup_uncharge - uncharge a page * @page: page to uncharge * * Uncharge a page previously charged with mem_cgroup_charge(). */ -void mem_cgroup_uncharge (struct page *page) +void __mem_cgroup_uncharge (struct page *page) {struct uncharge_gather ug; -if (mem_cgroup_disabled ()) -return ; -/* Don't touch page->lru of any random page, pre-check: */ if (!page_memcg (page))return ;
@@ -6938,20 +6932,17 @@ void mem_cgroup_uncharge(struct page *page) }/** -* mem_cgroup_uncharge_list - uncharge a list of page +* __mem_cgroup_uncharge_list - uncharge a list of page * @page_list: list of pages to uncharge * * Uncharge a list of pages previously charged with * mem_cgroup_charge(). */ -void mem_cgroup_uncharge_list (struct list_head *page_list) +void __mem_cgroup_uncharge_list (struct list_head *page_list) {struct uncharge_gather ug;struct page *page; -if (mem_cgroup_disabled ()) -return ; -uncharge_gather_clear (&ug);list_for_each_entry (page, page_list, lru)uncharge_page (page, &ug);
Signature · 1 line
2.32.0.93.g670b81a890-goog
Quoted 11 lines · Suren Baghdasaryan
Inline mem_cgroup_{charge/uncharge} and
mem_cgroup_uncharge_listfunctions functions to performmem_cgroup_disabledstatic key check inline before calling the main body of the function. This minimizes the memcg overhead in the pagefault andexit_mmappaths when memcgs are disabled using cgroup_disable=memory command-line option.
This change results in ~0.4% overhead reduction when running PFT test comparing {CONFIG_MEMCG=n} against {CONFIG_MEMCG=y, cgroup_disable=memory} configurationon on an 8-core ARM64 Android device.Signed-off-by Suren Baghdasaryan <[email protected]>
Quoted 10 lines · Suren Baghdasaryan
Inline mem_cgroup_{charge/uncharge} and
mem_cgroup_uncharge_listfunctions functions to performmem_cgroup_disabledstatic key check inline before calling the main body of the function. This minimizes the memcg overhead in the pagefault andexit_mmappaths when memcgs are disabled using cgroup_disable=memory command-line option.
This change results in ~0.4% overhead reduction when running PFT test comparing {CONFIG_MEMCG=n} against {CONFIG_MEMCG=y, cgroup_disable=memory} configurationon on an 8-core ARM64 Android device.Signed-off-by Suren Baghdasaryan <[email protected]>
Thanks!