mm: memcontrol: fix NULL pointer deref and UAF from high_work_func() on dying memcg
Subsystems (4)
PATCH v3 1/1 mm: memcontrol: fix NULL pointer deref and UAF from high_work_func() on dying memcg
When memory.high is exceeded in non-task context, try_charge_memcg() schedules memcg->high_work without taking a reference on the memcg, and high_work_func() invokes reclaim_high() without pinning the memcg.
With obj_cgroup reparenting, introduced in commit bf4f059954dc ("mm:
memcg/slab: obj_cgroup API"), objects charged to a child memcg are reparented to its parent in mem_cgroup_css_offline() instead of holding a reference on the child until uncharged. Subsequent uncharges of those objects go to the parent, which can leave the child's memory use permanently above its high limit, so any later charge on the offlined memcg in non-task context schedules high_work. This was later widened from kernel memory to LRU folios when commit f1cf8d2f36dc ("mm:
memcontrol: eliminate the problem of dying memory cgroup for LRU folios") converted LRU folios to use obj_cgroup reparenting.
cancel_work_sync(&memcg->high_work) only runs at the end of teardown in mem_cgroup_css_free(), so high_work_func() can start or still be running reclaim_high() after mem_cgroup_css_released() or shrink_many() has called lru_gen_release_memcg() and unhashed lruvec->lrugen.list. If reclaim tries to rotate the memcg, lru_gen_rotate_memcg() dereferences a NULL pprev pointer. This issue is specific to MGLRU and was introduced with the memcg LRU in commit e4dde56cd208 ("mm: multi-gen LRU: per-node lru_gen_folio lists").
With either traditional LRU or MGLRU, css_free_rwork_fn() calls css_rstat_exit() which frees per-CPU rstat data before calling mem_cgroup_css_free(). If reclaim_high() is still running, rstat updates via mod_lruvec_state() -> __css_rstat_updated() cause a use-after-free on the per-CPU rstat data. This was introduced when commit 5da3bfa029d6 ("cgroup: use separate rstat trees for each subsystem") moved css_rstat_exit() ahead of ss->css_free().
Fix both problems by acquiring a reference with css_tryget_online() in high_work_func() before calling reclaim_high(), and by resetting memory.high and swap.high to PAGE_COUNTER_MAX in mem_cgroup_css_offline() so that charges via references on an offlined memcg do not trigger high_work on the dead child and skip checking live ancestors in try_charge_memcg().
Acquiring the reference in high_work_func() rather than before scheduling high_work is safe because mem_cgroup_css_free() calls irq_work_sync() and cancel_work_sync() which cancels pending work and waits for an already running high_work_func() to complete before mem_cgroup_free() frees the memcg. css_tryget_online() uses percpu_ref_tryget_live() and is safe to call after percpu_ref_exit() before the cgroup is freed unlike css_tryget().
Oops: general protection fault, probably for non-canonical address 0xdffffc0000000000: 0000 [#1] SMP KASAN NOPTI KASAN: null-ptr-deref in range [0x0000000000000000-0x0000000000000007] CPU: 1 UID: 0 PID: 27 Comm: kworker/1:0 Tainted: G B W 7.3.0-rc5-g2bba28e98578 #284 PREEMPT(full) Tainted: [B]=BAD_PAGE, [W]=WARN <-- Taint from lru_gen_rotate_memcg() VM_WARN_ON_ONCE Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-debian-1.17.0-1 04/01/2014 Workqueue: events high_work_func RIP: 0010:lru_gen_rotate_memcg+0x1a3/0xe90 Call Trace: <TASK> ? read_ctrl_pos+0x22f/0x3f0 shrink_lruvec+0x16cb/0x1b80 ? ret_from_fork_asm+0x1a/0x30 ? lapic_next_event+0x15/0x20 ? clockevents_program_event+0x2bb/0x6f0 ? lapic_next_event+0x15/0x20 ? __pfx_shrink_lruvec+0x10/0x10 ? __pfx_clockevents_program_event+0x10/0x10 ? _raw_spin_lock+0x84/0xe0 ? __pfx__raw_spin_lock+0x10/0x10 ? kasan_save_stack+0x30/0x50 ? psi_group_change+0x381/0x740 ? __pfx_sched_clock_cpu+0x10/0x10 ? save_fpregs_to_fpstate+0xc4/0x2c0 ? finish_task_switch.isra.0+0x1de/0xac0 ? __schedule+0xf62/0x42c0 ? __pfx___schedule+0x10/0x10 ? __pfx___irq_work_queue_local+0x10/0x10 ? shrink_node+0x626/0x31e0 shrink_node+0x626/0x31e0 ? ring_buffer_unlock_commit+0x11d/0x5c0 ? kswapd_clear_hopeless+0xa0/0x150 ? __pfx_shrink_node+0x10/0x10 ? 0xffffffffc0000095 do_try_to_free_pages+0x255/0x13b0 ? ring_buffer_unlock_commit+0x11d/0x5c0 ? reclaim_high+0x10a/0x1b0 try_to_free_mem_cgroup_pages+0x1f9/0x5a0 ? __pfx_try_to_free_mem_cgroup_pages+0x10/0x10 ? irqentry_exit+0xc6/0x670 reclaim_high+0x10a/0x1b0 ? __pfx_reclaim_high+0x10/0x10 ? process_one_work+0x5ff/0x1030 ? __pfx_high_work_func+0x10/0x10 process_one_work+0x6e2/0x1030 ? __pfx___schedule+0x10/0x10 ? __pfx_process_one_work+0x10/0x10 worker_thread+0x422/0xbb0 ? __pfx_worker_thread+0x10/0x10 kthread+0x2cd/0x3a0 ? recalc_sigpending+0x15c/0x200 ? __pfx_kthread+0x10/0x10 ret_from_fork+0x368/0x590 ? __pfx_ret_from_fork+0x10/0x10 ? __switch_to+0x588/0xf50 ? __pfx_kthread+0x10/0x10 ret_from_fork_asm+0x1a/0x30 </TASK>
---
Changes since V2 (https://lore.kernel.org/all/[email protected]/)
- Add RBs / Acks
- Explain why css_tryget_online() is safe in the commit message.
(Shakeel Butt)
- Update the commit message and subject to explain the cgroup UAF this
patch also fixes. Add Fixes: 5da3bfa029d6 ("cgroup: use separate rstat trees for each subsystem")
(Michal Hocko, Johannes Weiner)
Changes since V1 (https://lore.kernel.org/all/[email protected]/)
- Switch to Fixes: e4dde56cd208 ("mm: multi-gen LRU: per-node lru_gen_folio lists")
from Fixes: f1cf8d2f36dc ("mm: memcontrol: eliminate the problem of dying memory cgroup for LRU folios") per Muchun Song.
- Reorder and update commit message content to describe reparenting
separately from the commits that exposed and then widened the problem.
- Trim the backtrace a bit.
mm/memcontrol.c | 5 +++++ 1 file changed, 5 insertions(+)
mm/memcontrol.c +5 -0
@@ -2457,7 +2457,10 @@ static void high_work_func(struct work_struct *work)struct mem_cgroup *memcg; memcg = container_of(work, struct mem_cgroup, high_work); +if (!css_tryget_online (&memcg->css)) +return ;reclaim_high (memcg,MEMCG_CHARGE_BATCH ,GFP_KERNEL ); +css_put (&memcg->css); }static void high_irq_work_func (struct irq_work *work)
@@ -4463,6 +4466,8 @@ static void mem_cgroup_css_offline(struct cgroup_subsys_state *css)page_counter_set_min (&memcg->memory,0 );page_counter_set_low (&memcg->memory,0 ); +page_counter_set_high (&memcg->memory,PAGE_COUNTER_MAX ); +page_counter_set_high (&memcg->swap,PAGE_COUNTER_MAX );zswap_memcg_offline_cleanup (memcg);
base-commit: 2bba28e98578de622ef307d00875aa8f8bca3d10
Signature · 1 line
2.56.0.385.gac671c819a-goog
Quoted 105 lines · T.J. Mercier
When memory.high is exceeded in non-task context,
try_charge_memcg()schedulesmemcg->high_workwithout taking a reference on the memcg, andhigh_work_func()invokesreclaim_high()without pinning the memcg.With
obj_cgroupreparenting, introduced in commitbf4f059954dc("mm:
memcg/slab:obj_cgroupAPI"), objects charged to a child memcg are reparented to its parent inmem_cgroup_css_offline()instead of holding a reference on the child until uncharged. Subsequent uncharges of those objects go to the parent, which can leave the child's memory use permanently above its high limit, so any later charge on the offlined memcg in non-task context scheduleshigh_work. This was later widened from kernel memory to LRU folios when commitf1cf8d2f36dc("mm:
memcontrol: eliminate the problem of dying memory cgroup for LRU folios") converted LRU folios to useobj_cgroupreparenting.
cancel_work_sync(&memcg->high_work)only runs at the end of teardown inmem_cgroup_css_free(), sohigh_work_func()can start or still be runningreclaim_high()aftermem_cgroup_css_released()orshrink_many()has calledlru_gen_release_memcg()and unhashedlruvec->lrugen.list. If reclaim tries to rotate the memcg,lru_gen_rotate_memcg()dereferences a NULL pprev pointer. This issue is specific to MGLRU and was introduced with the memcg LRU in commite4dde56cd208("mm: multi-gen LRU: per-nodelru_gen_foliolists").With either traditional LRU or MGLRU,
css_free_rwork_fn()callscss_rstat_exit()which frees per-CPU rstat data before callingmem_cgroup_css_free(). Ifreclaim_high()is still running, rstat updates viamod_lruvec_state()->__css_rstat_updated()cause a use-after-free on the per-CPU rstat data. This was introduced when commit5da3bfa029d6("cgroup: use separate rstat trees for each subsystem") movedcss_rstat_exit()ahead ofss->css_free().Fix both problems by acquiring a reference with
css_tryget_online()inhigh_work_func()before callingreclaim_high(), and by resetting memory.high and swap.high toPAGE_COUNTER_MAXinmem_cgroup_css_offline()so that charges via references on an offlined memcg do not triggerhigh_workon the dead child and skip checking live ancestors intry_charge_memcg().Acquiring the reference in
high_work_func()rather than before schedulinghigh_workis safe becausemem_cgroup_css_free()callsirq_work_sync()andcancel_work_sync()which cancels pending work and waits for an already runninghigh_work_func()to complete beforemem_cgroup_free()frees the memcg.css_tryget_online()usespercpu_ref_tryget_live()and is safe to call afterpercpu_ref_exit()before the cgroup is freed unlikecss_tryget().Oops: general protection fault, probably for non-canonical address 0xdffffc0000000000: 0000 [#1] SMP KASAN NOPTI KASAN: null-ptr-deref in range [0x0000000000000000-0x0000000000000007] CPU: 1 UID: 0 PID: 27 Comm: kworker/1:0 Tainted: G B W 7.3.0-rc5-g2bba28e98578 #284 PREEMPT(full) Tainted: [B]=BAD_PAGE, [W]=WARN <-- Taint from lru_gen_rotate_memcg() VM_WARN_ON_ONCE Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-debian-1.17.0-1 04/01/2014 Workqueue: events high_work_func RIP: 0010:lru_gen_rotate_memcg+0x1a3/0xe90 Call Trace: <TASK> ? read_ctrl_pos+0x22f/0x3f0 shrink_lruvec+0x16cb/0x1b80 ? ret_from_fork_asm+0x1a/0x30 ? lapic_next_event+0x15/0x20 ? clockevents_program_event+0x2bb/0x6f0 ? lapic_next_event+0x15/0x20 ? __pfx_shrink_lruvec+0x10/0x10 ? __pfx_clockevents_program_event+0x10/0x10 ? _raw_spin_lock+0x84/0xe0 ? __pfx__raw_spin_lock+0x10/0x10 ? kasan_save_stack+0x30/0x50 ? psi_group_change+0x381/0x740 ? __pfx_sched_clock_cpu+0x10/0x10 ? save_fpregs_to_fpstate+0xc4/0x2c0 ? finish_task_switch.isra.0+0x1de/0xac0 ? __schedule+0xf62/0x42c0 ? __pfx___schedule+0x10/0x10 ? __pfx___irq_work_queue_local+0x10/0x10 ? shrink_node+0x626/0x31e0 shrink_node+0x626/0x31e0 ? ring_buffer_unlock_commit+0x11d/0x5c0 ? kswapd_clear_hopeless+0xa0/0x150 ? __pfx_shrink_node+0x10/0x10 ? 0xffffffffc0000095 do_try_to_free_pages+0x255/0x13b0 ? ring_buffer_unlock_commit+0x11d/0x5c0 ? reclaim_high+0x10a/0x1b0 try_to_free_mem_cgroup_pages+0x1f9/0x5a0 ? __pfx_try_to_free_mem_cgroup_pages+0x10/0x10 ? irqentry_exit+0xc6/0x670 reclaim_high+0x10a/0x1b0 ? __pfx_reclaim_high+0x10/0x10 ? process_one_work+0x5ff/0x1030 ? __pfx_high_work_func+0x10/0x10 process_one_work+0x6e2/0x1030 ? __pfx___schedule+0x10/0x10 ? __pfx_process_one_work+0x10/0x10 worker_thread+0x422/0xbb0 ? __pfx_worker_thread+0x10/0x10 kthread+0x2cd/0x3a0 ? recalc_sigpending+0x15c/0x200 ? __pfx_kthread+0x10/0x10 ret_from_fork+0x368/0x590 ? __pfx_ret_from_fork+0x10/0x10 ? __switch_to+0x588/0xf50 ? __pfx_kthread+0x10/0x10 ret_from_fork_asm+0x1a/0x30 </TASK>
Quoted 6 lines
Fixes e4dde56cd208 ("mm: multi-gen LRU: per-node lru_gen_folio lists")Fixes 5da3bfa029d6 ("cgroup: use separate rstat trees for each subsystem")Signed-off-by T.J. Mercier <[email protected]>Reviewed-by Johannes Weiner <[email protected]>Acked-by Muchun Song <[email protected]>
Quoted 105 lines · T.J. Mercier
When memory.high is exceeded in non-task context,
try_charge_memcg()schedulesmemcg->high_workwithout taking a reference on the memcg, andhigh_work_func()invokesreclaim_high()without pinning the memcg.With
obj_cgroupreparenting, introduced in commitbf4f059954dc("mm:
memcg/slab:obj_cgroupAPI"), objects charged to a child memcg are reparented to its parent inmem_cgroup_css_offline()instead of holding a reference on the child until uncharged. Subsequent uncharges of those objects go to the parent, which can leave the child's memory use permanently above its high limit, so any later charge on the offlined memcg in non-task context scheduleshigh_work. This was later widened from kernel memory to LRU folios when commitf1cf8d2f36dc("mm:
memcontrol: eliminate the problem of dying memory cgroup for LRU folios") converted LRU folios to useobj_cgroupreparenting.
cancel_work_sync(&memcg->high_work)only runs at the end of teardown inmem_cgroup_css_free(), sohigh_work_func()can start or still be runningreclaim_high()aftermem_cgroup_css_released()orshrink_many()has calledlru_gen_release_memcg()and unhashedlruvec->lrugen.list. If reclaim tries to rotate the memcg,lru_gen_rotate_memcg()dereferences a NULL pprev pointer. This issue is specific to MGLRU and was introduced with the memcg LRU in commite4dde56cd208("mm: multi-gen LRU: per-nodelru_gen_foliolists").With either traditional LRU or MGLRU,
css_free_rwork_fn()callscss_rstat_exit()which frees per-CPU rstat data before callingmem_cgroup_css_free(). Ifreclaim_high()is still running, rstat updates viamod_lruvec_state()->__css_rstat_updated()cause a use-after-free on the per-CPU rstat data. This was introduced when commit5da3bfa029d6("cgroup: use separate rstat trees for each subsystem") movedcss_rstat_exit()ahead ofss->css_free().Fix both problems by acquiring a reference with
css_tryget_online()inhigh_work_func()before callingreclaim_high(), and by resetting memory.high and swap.high toPAGE_COUNTER_MAXinmem_cgroup_css_offline()so that charges via references on an offlined memcg do not triggerhigh_workon the dead child and skip checking live ancestors intry_charge_memcg().Acquiring the reference in
high_work_func()rather than before schedulinghigh_workis safe becausemem_cgroup_css_free()callsirq_work_sync()andcancel_work_sync()which cancels pending work and waits for an already runninghigh_work_func()to complete beforemem_cgroup_free()frees the memcg.css_tryget_online()usespercpu_ref_tryget_live()and is safe to call afterpercpu_ref_exit()before the cgroup is freed unlikecss_tryget().Oops: general protection fault, probably for non-canonical address 0xdffffc0000000000: 0000 [#1] SMP KASAN NOPTI KASAN: null-ptr-deref in range [0x0000000000000000-0x0000000000000007] CPU: 1 UID: 0 PID: 27 Comm: kworker/1:0 Tainted: G B W 7.3.0-rc5-g2bba28e98578 #284 PREEMPT(full) Tainted: [B]=BAD_PAGE, [W]=WARN <-- Taint from lru_gen_rotate_memcg() VM_WARN_ON_ONCE Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-debian-1.17.0-1 04/01/2014 Workqueue: events high_work_func RIP: 0010:lru_gen_rotate_memcg+0x1a3/0xe90 Call Trace: <TASK> ? read_ctrl_pos+0x22f/0x3f0 shrink_lruvec+0x16cb/0x1b80 ? ret_from_fork_asm+0x1a/0x30 ? lapic_next_event+0x15/0x20 ? clockevents_program_event+0x2bb/0x6f0 ? lapic_next_event+0x15/0x20 ? __pfx_shrink_lruvec+0x10/0x10 ? __pfx_clockevents_program_event+0x10/0x10 ? _raw_spin_lock+0x84/0xe0 ? __pfx__raw_spin_lock+0x10/0x10 ? kasan_save_stack+0x30/0x50 ? psi_group_change+0x381/0x740 ? __pfx_sched_clock_cpu+0x10/0x10 ? save_fpregs_to_fpstate+0xc4/0x2c0 ? finish_task_switch.isra.0+0x1de/0xac0 ? __schedule+0xf62/0x42c0 ? __pfx___schedule+0x10/0x10 ? __pfx___irq_work_queue_local+0x10/0x10 ? shrink_node+0x626/0x31e0 shrink_node+0x626/0x31e0 ? ring_buffer_unlock_commit+0x11d/0x5c0 ? kswapd_clear_hopeless+0xa0/0x150 ? __pfx_shrink_node+0x10/0x10 ? 0xffffffffc0000095 do_try_to_free_pages+0x255/0x13b0 ? ring_buffer_unlock_commit+0x11d/0x5c0 ? reclaim_high+0x10a/0x1b0 try_to_free_mem_cgroup_pages+0x1f9/0x5a0 ? __pfx_try_to_free_mem_cgroup_pages+0x10/0x10 ? irqentry_exit+0xc6/0x670 reclaim_high+0x10a/0x1b0 ? __pfx_reclaim_high+0x10/0x10 ? process_one_work+0x5ff/0x1030 ? __pfx_high_work_func+0x10/0x10 process_one_work+0x6e2/0x1030 ? __pfx___schedule+0x10/0x10 ? __pfx_process_one_work+0x10/0x10 worker_thread+0x422/0xbb0 ? __pfx_worker_thread+0x10/0x10 kthread+0x2cd/0x3a0 ? recalc_sigpending+0x15c/0x200 ? __pfx_kthread+0x10/0x10 ret_from_fork+0x368/0x590 ? __pfx_ret_from_fork+0x10/0x10 ? __switch_to+0x588/0xf50 ? __pfx_kthread+0x10/0x10 ret_from_fork_asm+0x1a/0x30 </TASK>
Quoted 6 lines
Fixes e4dde56cd208 ("mm: multi-gen LRU: per-node lru_gen_folio lists")Fixes 5da3bfa029d6 ("cgroup: use separate rstat trees for each subsystem")Signed-off-by T.J. Mercier <[email protected]>Reviewed-by Johannes Weiner <[email protected]>Acked-by Muchun Song <[email protected]>
Thanks!
Signature · 2 lines
Michal Hocko SUSE Labs