mm: memcg/slab: fix racy access to page->mem_cgroup in mem_cgroup_from_obj()
Subsystems (3)
PATCH 1/1 mm: memcg/slab: fix racy access to page->mem_cgroup in mem_cgroup_from_obj()
mem_cgroup_from_obj() checks the lowest bit of the page->mem_cgroup pointer to determine if the page has an attached obj_cgroup vector instead of a regular memcg pointer. If it's not set, it simple returns the page->mem_cgroup value as a struct mem_cgroup pointer.
The commit 10befea91b61 ("mm: memcg/slab: use a single set of kmem_caches for all allocations") changed the moment when this bit is set: if previously it was set on the allocation of the slab page, now it can be set well after, when the first accounted object is allocated on this page.
It opened a race: if page->mem_cgroup is set concurrently after the first page_has_obj_cgroups(page) check, a pointer to the obj_cgroups array can be returned as a memory cgroup pointer.
A simple check for page->mem_cgroup pointer for NULL before the page_has_obj_cgroups() check fixes the race. Indeed, if the pointer is not NULL, it's either a simple mem_cgroup pointer or a pointer to obj_cgroup vector. The pointer can be asynchronously changed from NULL to (obj_cgroup_vec | 0x1UL), but can't be changed from a valid memcg pointer to objcg vector or back.
If the object passed to mem_cgroup_from_obj() is a slab object and page->mem_cgroup is NULL, it means that the object is not accounted, so the function must return NULL.
I've discovered the race looking at the code, so far I haven't seen it in the wild.
--- mm/memcontrol.c | 11 +++++++++++ 1 file changed, 11 insertions(+)
mm/memcontrol.c +11 -0
@@ -2923,6 +2923,17 @@ struct mem_cgroup *mem_cgroup_from_obj(void *p) page =virt_to_head_page (p); +/* +* If page->mem_cgroup is set, it's either a simple mem_cgroup pointer +* or a pointer to obj_cgroup vector. In the latter case the lowest +* bit of the pointer is set. +* The page->mem_cgroup pointer can be asynchronously changed +* from NULL to (obj_cgroup_vec | 0x1UL), but can't be changed +* from a valid memcg pointer to objcg vector or back. +*/ +if (!page->mem_cgroup) +return NULL ; +/* * Slab objects are accounted individually, not per-page. * Memcg membership data for each individual object is saved in
Signature · 1 line
2.26.2
Forgot to cc stable@, an updated version is below.
Thanks!
--
From fe61af45ae570b143ca783ba4d013a0a2b923a15 Mon Sep 17 00:00:00 2001
From: Roman Gushchin <[email protected]>
Date: Wed, 9 Sep 2020 12:19:37 -0700
Subject: [PATCH] mm: memcg/slab: fix racy access to page->mem_cgroup in
mem_cgroup_from_obj()
mem_cgroup_from_obj() checks the lowest bit of the page->mem_cgroup pointer to determine if the page has an attached obj_cgroup vector instead of a regular memcg pointer. If it's not set, it simple returns the page->mem_cgroup value as a struct mem_cgroup pointer.
The commit 10befea91b61 ("mm: memcg/slab: use a single set of kmem_caches for all allocations") changed the moment when this bit is set: if previously it was set on the allocation of the slab page, now it can be set well after, when the first accounted object is allocated on this page.
It opened a race: if page->mem_cgroup is set concurrently after the first page_has_obj_cgroups(page) check, a pointer to the obj_cgroups array can be returned as a memory cgroup pointer.
A simple check for page->mem_cgroup pointer for NULL before the page_has_obj_cgroups() check fixes the race. Indeed, if the pointer is not NULL, it's either a simple mem_cgroup pointer or a pointer to obj_cgroup vector. The pointer can be asynchronously changed from NULL to (obj_cgroup_vec | 0x1UL), but can't be changed from a valid memcg pointer to objcg vector or back.
If the object passed to mem_cgroup_from_obj() is a slab object and page->mem_cgroup is NULL, it means that the object is not accounted, so the function must return NULL.
I've discovered the race looking at the code, so far I haven't seen it in the wild.
--- mm/memcontrol.c | 11 +++++++++++ 1 file changed, 11 insertions(+)
mm/memcontrol.c +11 -0
@@ -2923,6 +2923,17 @@ struct mem_cgroup *mem_cgroup_from_obj(void *p) page =virt_to_head_page (p); +/* +* If page->mem_cgroup is set, it's either a simple mem_cgroup pointer +* or a pointer to obj_cgroup vector. In the latter case the lowest +* bit of the pointer is set. +* The page->mem_cgroup pointer can be asynchronously changed +* from NULL to (obj_cgroup_vec | 0x1UL), but can't be changed +* from a valid memcg pointer to objcg vector or back. +*/ +if (!page->mem_cgroup) +return NULL ; +/* * Slab objects are accounted individually, not per-page. * Memcg membership data for each individual object is saved in
Signature · 1 line
2.26.2
Quoted 42 lines · Roman Gushchin
Forgot to cc stable@, an updated version is below.
Thanks!
--
From
fe61af45ae570b143ca783ba4d013a0a2b923a15Mon Sep 17 00:00:00 2001
From: Roman Gushchin <[email protected]>
Date: Wed, 9 Sep 2020 12:19:37 -0700
Subject: [PATCH] mm: memcg/slab: fix racy access topage->mem_cgroupin
mem_cgroup_from_obj()
mem_cgroup_from_obj()checks the lowest bit of thepage->mem_cgrouppointer to determine if the page has an attachedobj_cgroupvector instead of a regular memcg pointer. If it's not set, it simple returns thepage->mem_cgroupvalue as a structmem_cgrouppointer.The commit
10befea91b61("mm: memcg/slab: use a single set ofkmem_cachesfor all allocations") changed the moment when this bit is set: if previously it was set on the allocation of the slab page, now it can be set well after, when the first accounted object is allocated on this page.It opened a race: if
page->mem_cgroupis set concurrently after the firstpage_has_obj_cgroups(page)check, a pointer to theobj_cgroupsarray can be returned as a memory cgroup pointer.A simple check for
page->mem_cgrouppointer for NULL before thepage_has_obj_cgroups()check fixes the race. Indeed, if the pointer is not NULL, it's either a simplemem_cgrouppointer or a pointer toobj_cgroupvector. The pointer can be asynchronously changed from NULL to (obj_cgroup_vec|0x1UL), but can't be changed from a valid memcg pointer to objcg vector or back.If the object passed to
mem_cgroup_from_obj()is a slab object andpage->mem_cgroupis NULL, it means that the object is not accounted, so the function must return NULL.I've discovered the race looking at the code, so far I haven't seen it in the wild.
Quoted 7 lines
Fixes 10befea91b61 ("mm: memcg/slab: use a single set of kmem_caches for all allocations")Signed-off-by Roman Gushchin <[email protected]>Cc Johannes Weiner <[email protected]>Cc Vlastimil Babka <[email protected]>Cc Shakeel Butt <[email protected]>---
Is the caller list_lru_from_kmem() the concern or is this more about making mem_cgroup_from_obj() more future proof?
Also have you taken a look at [1]? I am still trying to figure out how that is possible.
[1] https://lore.kernel.org/lkml/20200901075321.GL4299@shao2-debian/
Quoted 51 lines · Shakeel Butt
Quoted 42 lines · Roman Gushchin
Forgot to cc stable@, an updated version is below.
Thanks!
--
From
fe61af45ae570b143ca783ba4d013a0a2b923a15Mon Sep 17 00:00:00 2001
From: Roman Gushchin <[email protected]>
Date: Wed, 9 Sep 2020 12:19:37 -0700
Subject: [PATCH] mm: memcg/slab: fix racy access topage->mem_cgroupin
mem_cgroup_from_obj()
mem_cgroup_from_obj()checks the lowest bit of thepage->mem_cgrouppointer to determine if the page has an attachedobj_cgroupvector instead of a regular memcg pointer. If it's not set, it simple returns thepage->mem_cgroupvalue as a structmem_cgrouppointer.The commit
10befea91b61("mm: memcg/slab: use a single set ofkmem_cachesfor all allocations") changed the moment when this bit is set: if previously it was set on the allocation of the slab page, now it can be set well after, when the first accounted object is allocated on this page.It opened a race: if
page->mem_cgroupis set concurrently after the firstpage_has_obj_cgroups(page)check, a pointer to theobj_cgroupsarray can be returned as a memory cgroup pointer.A simple check for
page->mem_cgrouppointer for NULL before thepage_has_obj_cgroups()check fixes the race. Indeed, if the pointer is not NULL, it's either a simplemem_cgrouppointer or a pointer toobj_cgroupvector. The pointer can be asynchronously changed from NULL to (obj_cgroup_vec|0x1UL), but can't be changed from a valid memcg pointer to objcg vector or back.If the object passed to
mem_cgroup_from_obj()is a slab object andpage->mem_cgroupis NULL, it means that the object is not accounted, so the function must return NULL.I've discovered the race looking at the code, so far I haven't seen it in the wild.
Quoted 7 lines
Fixes 10befea91b61 ("mm: memcg/slab: use a single set of kmem_caches for all allocations")Signed-off-by Roman Gushchin <[email protected]>Cc Johannes Weiner <[email protected]>Cc Vlastimil Babka <[email protected]>Cc Shakeel Butt <[email protected]>---
Quoted 2 lines
Is the caller
list_lru_from_kmem()the concern or is this more about makingmem_cgroup_from_obj()more future proof?
I was doing some refactorings around (see
https://lore.kernel.org/linux-mm/[email protected]/T/#t ), and just noticed it from looking at the code. I'm not aware of any real life consequences at the moment.
Quoted 5 lines
Also have you taken a look at [1]? I am still trying to figure out how that is possible.
[1] https://lore.kernel.org/lkml/20200901075321.GL4299@shao2-debian/
Hm, yeah, it's complicated. At the very first glance it looks like that the obj_cgroups vector is placed onto the very same page it describes, or at least it shares the kmem_cache with it, with some bad consequences. Could be something SLAB-specific, newer saw anything like that with SLUB.
Or maybe it's completely unrelated and has been attributed to this commit by mistake.
I've spent several hours running the provided test in a loop, but wasn't lucky enough to trigger it. Did you try?
Thanks!
Roman
Quoted 1 line · Roman Gushchin
[snip]
Quoted 13 lines
Quoted 5 lines
Also have you taken a look at [1]? I am still trying to figure out how that is possible.
[1] https://lore.kernel.org/lkml/20200901075321.GL4299@shao2-debian/
Hm, yeah, it's complicated. At the very first glance it looks like that the
obj_cgroupsvector is placed onto the very same page it describes, or at least it shares thekmem_cachewith it, with some bad consequences. Could be something SLAB-specific, newer saw anything like that with SLUB.
Or maybe it's completely unrelated and has been attributed to this commit by mistake.
Quoted 3 lines
I've spent several hours running the provided test in a loop, but wasn't lucky enough to trigger it. Did you try?
Yeah same, no success in reproducing it.
Shakeel Butt
Reviewed-byQuoted 42 lines · Roman Gushchin
Forgot to cc stable@, an updated version is below.
Thanks!
--
From
fe61af45ae570b143ca783ba4d013a0a2b923a15Mon Sep 17 00:00:00 2001
From: Roman Gushchin <[email protected]>
Date: Wed, 9 Sep 2020 12:19:37 -0700
Subject: [PATCH] mm: memcg/slab: fix racy access topage->mem_cgroupin
mem_cgroup_from_obj()
mem_cgroup_from_obj()checks the lowest bit of thepage->mem_cgrouppointer to determine if the page has an attachedobj_cgroupvector instead of a regular memcg pointer. If it's not set, it simple returns thepage->mem_cgroupvalue as a structmem_cgrouppointer.The commit
10befea91b61("mm: memcg/slab: use a single set ofkmem_cachesfor all allocations") changed the moment when this bit is set: if previously it was set on the allocation of the slab page, now it can be set well after, when the first accounted object is allocated on this page.It opened a race: if
page->mem_cgroupis set concurrently after the firstpage_has_obj_cgroups(page)check, a pointer to theobj_cgroupsarray can be returned as a memory cgroup pointer.A simple check for
page->mem_cgrouppointer for NULL before thepage_has_obj_cgroups()check fixes the race. Indeed, if the pointer is not NULL, it's either a simplemem_cgrouppointer or a pointer toobj_cgroupvector. The pointer can be asynchronously changed from NULL to (obj_cgroup_vec|0x1UL), but can't be changed from a valid memcg pointer to objcg vector or back.If the object passed to
mem_cgroup_from_obj()is a slab object andpage->mem_cgroupis NULL, it means that the object is not accounted, so the function must return NULL.I've discovered the race looking at the code, so far I haven't seen it in the wild.
Quoted 6 lines
Fixes 10befea91b61 ("mm: memcg/slab: use a single set of kmem_caches for all allocations")Signed-off-by Roman Gushchin <[email protected]>Cc Johannes Weiner <[email protected]>Cc Vlastimil Babka <[email protected]>Cc Shakeel Butt <[email protected]>
I think this patch is good to have as it will make mem_cgroup_from_obj() more future proof.