memcg: effective high limit that avoids isolation issues
Subsystems (14)
Shakeel Butt
maintainercover memcg: effective high limit that avoids isolation issues
(My initial plan [1, 2] was to track kernel lock holders, let them bypass memcg limits and throttle them on unlock or on return to userspace. I was asked to look for alternative solutions, and this series is one.)
memcg limits can cause isolation problems. A task stuck in limit enforcement in the charge path keeps holding its locks, and tasks in other cgroups that need them wait for a limit that has nothing to do with them. btrfs, for example, charges its tree blocks to the root cgroup for this reason [3].
We see this in Meta's fleet. Over 28 days, on several tens of thousands of unique hosts, a bpftrace script [4] recorded sleeping locks held in memcg reclaim while another task waited for them. Most belong to the reclaiming workload, like its mmap_lock (680,579 times, up to 4.7s), but some are shared by the whole machine or a whole filesystem:
- cgroup_mutex, used by every cgroup change. mkdir charges the new memcg to the parent while holding it, and reclaims if the parent is at its limit: 350 times, up to 2.5s.
- kernfs_rwsem, used by cgroupfs lookups. The kernfs notification worker held it while queueing inotify events charged to the watcher's cgroup: 2,324 times, up to 3.3s.
- btrfs tree locks, used by every b-tree search and change. On kernels with [3], the XArray nodes that index tree blocks are still charged to the task, so tasks reclaimed with tree locks held 2,853 times, 850 of them with a root node locked, up to 2.6s.
Hung-task reports show worse cases. A container's systemd created a cgroup while its own cgroup was over memory.high, and mkdir ran memory.high reclaim with cgroup_mutex and the parent directory's lock held for over 30 minutes. The host's systemd, tasks forking into cgroups, oomd and a monitoring agent all waited, so the userspace OOM killer could not even read the cgroup tree.
memory.max is a hard limit, so this series does not change it. memory.high doesn't have to be enforced in the charge path: since commit b23afb93d317 ("memcg: punt high overage reclaim to return-to-userland path"), the charge only records the overage (usage above the limit), and the task reclaims and is throttled on return to userspace, where it holds no locks. Avoiding this priority inversion was one of the reasons for that commit.
But return to userspace is too late when one system call charges a lot, like mlock() of a large range. So commit c9afe31ec443 ("memcg:
synchronously enforce memory.high for large overcharges") handles the overage in the charge itself when the charge can block and more than a batch is pending. That brought the problem back: the task reclaims and sleeps, up to 2s per call, under whatever locks it holds. It also misses THP charges outside MADV_HUGEPAGE ranges, which can't block under the default defrag setting. RESULTS shows both problems.
This series is a first step in rethinking the memcg high limit around isolation. It makes memory.high enforcement asynchronous (to allocation and charging) again and keeps it effective for large calls. The task handles the overage on return to userspace, and also inside the operations that can charge a lot in one call, where it holds no locks or only the locks of the file it works on:
- Populating memory (mlock, mlockall, MAP_POPULATE, MAP_LOCKED, MADV_POPULATE_*): 2M at a time, with mmap_lock dropped in between.
- tmpfs fallocate, buffered writes (generic_perform_write(), iomap, btrfs) and reads into user memory (filemap_read(), shmem): once per folio or range.
- MADV_COLLAPSE: after each collapse, with mmap_lock dropped.
These points also throttle THP charges, which the synchronous step skips. A file's lock only blocks tasks using that file. Two cases hold
more: splice() from a pipe also holds the pipe's lock, and gfs2 holds its allocation glocks in writes that allocate. The synchronous step sleeps with these locks today too.
Why not keep the synchronous step, with a higher threshold, without its sleep, or only outside these operations? It would still reclaim or sleep under locks, like mmap_lock in THP faults and invalidate_lock in readahead, and it would not close the THP gap.
This also lets users drop memory.max, which still reclaims in the charge path and can stall lock holders as above. They can set only memory.high and let a userspace OOM killer like oomd [5], or a future BPF OOM policy [6], decide when to kill. Meta plans to do this.
The last two patches remove the synchronous step, update the memory.high documentation and fix test_memcg_high_sync, which missed the THP gap.
RESULTS
Tests run in a VM with 4 CPUs and THP on for anonymous memory and shmem (enabled=always, shmem_enabled=always, tmpfs huge=always, the default defrag=madvise), in a cgroup with memory.high=64M. Reads go into an untouched buffer. "No hooks" is the base plus only the patch that removes the synchronous step. Usage 18s into a 512M operation, without swap:
test base series no hooks
mlock, mlockall, MAP_POPULATE, 513-514M 84M 513-514M
MADV_POPULATE_WRITE
tmpfs fallocate and write 512-513M 84M 512-513M
reads of a tmpfs file 513M 84-85M 513M
MADV_COLLAPSE of a sparse range 96M 98M 514M
(one VMA, or one VMA per PMD)
reads of /dev/zero or /dev/urandom 513M 513M 513M
(known gaps)
Usage grows by one 2M folio per throttle, so it is 84-85M after 18s. On ext4, XFS and btrfs, 256M writes stay at 64-67M on both kernels (257-264M with no hooks), and reads with a cold page cache at 70-73M. Reads with a warm page cache, where only the buffer is charged, reach 257M on base and stay at 84-85M with the series.
With swap, the MAP_POPULATE, MADV_POPULATE_WRITE and tmpfs write tests finish in under 2s at 65-66M with the series (base: 157-169M, no hooks:
513M). tmpfs fallocate is throttled with or without swap. With swap, base fails it with -ENOMEM for small pages.
During mlock() of an 80M MADV_HUGEPAGE range, a task in another cgroup that reads the process's /proc/<pid>/cmdline every 10ms waits up to 15.3s on base and under 1ms with the series. Populating 2G with no limit takes the same time on both kernels.
All patches build with gcc and clang (W=1), with CONFIG_MEMCG=n and with btrfs as a module. A lockdep kernel showed no warnings. The memcontrol selftests pass, and the fixed test_memcg_high_sync fails on base.
KNOWN GAPS
Other paths that charge a lot in one call are now throttled only on return to userspace, and memory.max still limits them. Most can get a throttle point later, some after dropping a lock:
- /dev/zero and /dev/urandom reads, getrandom(), sendfile() to a socket, io_uring submissions, netfs and FUSE write-through writes
- MADV_WILLNEED, UFFDIO_COPY, KSM unmerge, mprotect() of mlocked mappings
- FIDEDUPERANGE, FS_IOC_ENABLE_VERITY, KVM_PRE_FAULT_MEMORY, guest_memfd fallocate, KVM page splitting
Rarer ones, like sendmmsg(), btrfs send and FITRIM, can wait. Memory pinning (VFIO, iommufd, RDMA, io_uring buffers, direct I/O and others), fork() and single large allocations (BPF maps and others) hold a lock for the whole call or can't be split, so they stay as they are.
[1] https://lore.kernel.org/[email protected]/
[2] https://lore.kernel.org/[email protected]/
[3] commit cf1dec76ba8a ("mm/filemap: add AS_KERNEL_FILE")
[4] https://github.com/bpftrace/user-tools/tree/master/reclaimlockmonitor
[5] https://github.com/facebookincubator/oomd
[6] https://lore.kernel.org/[email protected]
Based on mm-everything-2026-11-20-06-34.
Shakeel Butt (7):
memcg: add helpers to handle memory.high overage in long operations mm/gup: handle memory.high overage while populating
mm, fs: handle memory.high overage in buffered reads and writes mm/shmem: handle memory.high overage in fallocate and reads mm/khugepaged: handle memory.high overage in MADV_COLLAPSE
memcg: remove synchronous memory.high enforcement selftests/cgroup: make test_memcg_high_sync check that throttling
holds
Documentation/admin-guide/cgroup-v2.rst | 12 +++++- fs/btrfs/file.c | 9 +++++ fs/iomap/buffered-io.c | 9 +++++ include/linux/memcontrol.h | 34 ++++++++++++++++ mm/filemap.c | 31 +++++++++++++++ mm/gup.c | 9 ++++- mm/internal.h | 27 +++++++++++++ mm/khugepaged.c | 8 ++++ mm/madvise.c | 11 +++++- mm/memcontrol.c | 21 ++++------ mm/shmem.c | 37 ++++++++++++++++-- .../selftests/cgroup/test_memcontrol.c | 39 +++++++++++++++---- 12 files changed, 217 insertions(+), 30 deletions(-)
base-commit: b298839c2f77b60eea645a68d2245aa088463955
Signature · 1 line
2.53.1-Meta
Shakeel Butt
maintainerRFC PATCH 1/7 memcg: add helpers to handle memory.high overage in long operations
Since commit c9afe31ec443 ("memcg: synchronously enforce memory.high for large overcharges"), a charge that can block handles memory.high overage right away once more than a batch is pending. It reclaims and sleeps with whatever locks the caller holds, so tasks in other cgroups that need those locks wait too.
A later patch removes that step. Operations that can charge a lot in one call then need another place to handle the overage: a point in their loop where they hold no locks, or only the locks of their file.
Add two helpers for them. mem_cgroup_over_high_pending() says whether more than a batch is pending, the threshold the synchronous step uses. mem_cgroup_over_high_throttle() handles the overage if so; its comment says where to call it and which gfp mask to pass. Export __mem_cgroup_handle_over_high() for filesystems built as modules.
--- include/linux/memcontrol.h | 34 ++++++++++++++++++++++++++++++++++ mm/memcontrol.c | 1 + 2 files changed, 35 insertions(+)
include/linux/memcontrol.h +34 -0
@@ -942,6 +942,31 @@ static inline void mem_cgroup_handle_over_high(gfp_t gfp_mask)__mem_cgroup_handle_over_high (gfp_mask); } +/* +* True if the current task has more than a batch of memory.high or +* memory.swap.high overage pending. Each charge that finds overage +* counts at least a batch. +*/ +static inline bool mem_cgroup_over_high_pending (void ) +{ +return unlikely (current->memcg_nr_pages_over_high >MEMCG_CHARGE_BATCH ); +} + +/* +* Handle the current task's memory.high overage if more than a batch is +* pending, for operations that can charge a lot before they return to +* userspace. The task may reclaim and sleep here, so call it only where +* other tasks won't wait for it: with no locks held anywhere up the call +* chain, or only the locks of the file being worked on. Pass GFP_KERNEL +* if no filesystem locks are held, and mapping_gfp_constraint() in file +* read and write paths. +*/ +static inline void mem_cgroup_over_high_throttle (gfp_t gfp_mask) +{ +if (mem_cgroup_over_high_pending ()) +__mem_cgroup_handle_over_high (gfp_mask); +} +unsigned long mem_cgroup_get_max (const struct mem_cgroup *memcg); void mem_cgroup_print_oom_context(const struct mem_cgroup *memcg,
@@ -1376,6 +1401,15 @@ static inline void mem_cgroup_handle_over_high(gfp_t gfp_mask) { } +static inline bool mem_cgroup_over_high_pending (void ) +{ +return false; +} + +static inline void mem_cgroup_over_high_throttle (gfp_t gfp_mask) +{ +} +static inline struct mem_cgroup *mem_cgroup_get_oom_group (struct task_struct *victim,struct mem_cgroup *oom_domain) {
mm/memcontrol.c +1 -0
@@ -2707,6 +2707,7 @@ void __mem_cgroup_handle_over_high(gfp_t gfp_mask) out:css_put (&memcg->css); } +EXPORT_SYMBOL_GPL (__mem_cgroup_handle_over_high);/* * Called after @nr_pages were charged to @memcg. If @memcg or one of its
Signature · 1 line
2.53.1-Meta
Shakeel Butt
maintainerRFC PATCH 2/7 mm/gup: handle memory.high overage while populating
In reply to Shakeel Buttmlock(), mlockall(), mmap() with MAP_POPULATE or MAP_LOCKED, and MADV_POPULATE_READ/WRITE can fault in a lot of memory in one call, with mmap_lock held. When their charges can block, the charge path throttles them for memory.high with mmap_lock still held. In a VM with THP on, mlock() of an 80M MADV_HUGEPAGE range under memory.high=64M made another thread wait up to 17.2s to mmap(), and a task in another cgroup wait up to 15.3s to read the process's /proc/<pid>/cmdline. THP charges in VMAs without MADV_HUGEPAGE can't block under the default defrag setting, so they are not throttled at all: mlock(512M) reached 513M.
Throttle these calls with mmap_lock dropped instead. The populate loops, __mm_populate() and madvise_populate(), now fault in at most 2MiB per GUP call. Between calls, if more than a batch of overage is pending, they drop mmap_lock, handle it, and go on. GUP itself doesn't change. One fault can still charge a whole folio.
With this patch, mlock(512M) stays at 84M, and populating 2G with no limit takes as long as before. The mmap_lock stall goes away when a later patch removes the synchronous step.
--- mm/gup.c | 9 +++++++-- mm/internal.h | 27 +++++++++++++++++++++++++++ mm/madvise.c | 11 +++++++++-- 3 files changed, 43 insertions(+), 4 deletions(-)
mm/gup.c +7 -2
@@ -1951,6 +1951,7 @@ int __mm_populate(unsigned long start, unsigned long len, int ignore_errors) end = start + len;for (nstart = start; nstart < end; nstart = nend) { +populate_handle_over_high (mm, &locked);/* * We want to fault in pages for [nstart; end) address range. * Find first corresponding VMA.
@@ -1976,9 +1977,13 @@ int __mm_populate(unsigned long start, unsigned long len, int ignore_errors)/* * Now fault in a range of pages. populate_vma_page_range() * double checks the vma flags, so that it won't mlock pages -* if the vma was already munlocked. +* if the vma was already munlocked. Do one chunk at a time, +* see POPULATE_CHUNK_SIZE. On error, nend skips the rest of +* the VMA. */ - ret =populate_vma_page_range (vma, nstart, nend, &locked); + ret =populate_vma_page_range (vma, nstart, +populate_chunk_end (nstart, nend), + &locked);if (ret <0 ) {if (ignore_errors) { ret =0 ;
mm/internal.h +27 -0
@@ -948,6 +948,33 @@ extern long populate_vma_page_range(struct vm_area_struct *vma, unsigned long start, unsigned long end, int *locked);extern long faultin_page_range (struct mm_struct *mm,unsigned long start,unsigned long end,bool write,int *locked); + +/* +* __mm_populate() and madvise_populate() fault in at most +* POPULATE_CHUNK_SIZE at a time, and handle memory.high overage between +* chunks with mmap_lock dropped, so other tasks don't wait for them. +*/ +#define POPULATE_CHUNK_SIZE SZ_2M + +static inline unsigned long populate_chunk_end (unsigned long start, +unsigned long end) +{ +return min (end,round_down (start,POPULATE_CHUNK_SIZE ) + +POPULATE_CHUNK_SIZE ); +} + +static inline void populate_handle_over_high (struct mm_struct *mm, +int *locked) +{ +if (!mem_cgroup_over_high_pending ()) +return ; +if (*locked) { +mmap_read_unlock (mm); + *locked =0 ; + } +mem_cgroup_handle_over_high (GFP_KERNEL ); +} +bool mlock_future_ok (const struct mm_struct *mm,bool is_vma_locked,unsigned long bytes);
mm/madvise.c +9 -2
@@ -992,8 +992,15 @@ static long madvise_populate(struct madvise_behavior *madv_behavior)long pages;while (start < end) { -/* Populate (prefault) page tables readable/writable. */ - pages =faultin_page_range (mm, start, end, write, &locked); +/* +* Populate (prefault) page tables readable/writable, at most a +* chunk at a time, see POPULATE_CHUNK_SIZE. +*/ + pages =faultin_page_range (mm, start, +populate_chunk_end (start, end), + write, &locked); +if (pages >=0 ) +populate_handle_over_high (mm, &locked);if (!locked) {mmap_read_lock (mm); locked =1 ;
Signature · 1 line
2.53.1-Meta
[email protected]
AI reviewRe: [RFC PATCH 2/7] mm/gup: handle memory.high overage while populating
Thank you for your contribution! Sashiko AI review found 1 potential issue to consider: - [High] mm: unsigned long overflow to 0 in populate_chunk_end() near top of address space --
Thank you for your contribution! Sashiko AI review found 1 potential issue to consider:
- [High] mm: unsigned long overflow to 0 in populate_chunk_end() near top of address space
--
commit ec48ff5e44f8914e2b82f0cee190a74feb7645ec
Author: Shakeel Butt <[email protected]>
mm/gup: handle memory.high overage while populating
This commit chunks page fault-in loops in __mm_populate() and madvise_populate() into 2 MiB ranges and drops mmap_lock between chunks to handle pending memory.high overage.
Quoted 12 lines
mm/internal.h +4
-0@@ -948,6 +948,33 @@ extern long populate_vma_page_range(struct vm_area_struct *vma, unsigned long start, unsigned long end, int *locked);extern long faultin_page_range (struct mm_struct *mm,unsigned long start,unsigned long end,bool write,int *locked); + +/* +* __mm_populate() and madvise_populate() fault in at most +* POPULATE_CHUNK_SIZE at a time, and handle memory.high overage between
mm/internal.h extern long populate_vma_page_range(struct vm_area_struct *vma, · line 964
Quoted 10 lines
mm/internal.h +10
-0+* chunks with mmap_lock dropped, so other tasks don't wait for them. +*/ +#define POPULATE_CHUNK_SIZE SZ_2M + +static inline unsigned long populate_chunk_end (unsigned long start, +unsigned long end) +{ +return min (end,round_down (start,POPULATE_CHUNK_SIZE ) + +POPULATE_CHUNK_SIZE ); +}
[Severity: High]
Can populate_chunk_end() overflow to 0 when start is within the top POPULATE_CHUNK_SIZE (2 MiB) of the unsigned long address space?
When start >= ULONG_MAX - POPULATE_CHUNK_SIZE + 1, round_down(start, POPULATE_CHUNK_SIZE) + POPULATE_CHUNK_SIZE wraps to 0, so min(end, 0) returns 0. Passing 0 as end to populate_vma_page_range() or faultin_page_range() underflows nr_pages = (end - start) / PAGE_SIZE to (0 - start) / PAGE_SIZE, faulting pages past the requested end boundary. If a VMA in that range has VM_LOCKONFAULT set (for example, on 32-bit parisc or 64-bit s390 where TASK_SIZE reaches the top 2 MiB), populate_vma_page_range() returns that wrapped nr_pages and __mm_populate() computes
nend = nstart + ret * PAGE_SIZE == 0, looping forever from address 0 with mmap_read_lock(mm) held.
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=2
Shakeel Butt
maintainerRFC PATCH 3/7 mm, fs: handle memory.high overage in buffered reads and writes
In reply to Shakeel ButtA buffered read or write can copy almost 2GiB in one call. A write charges the page cache it fills. Dirty throttling limits only the dirty part, and does nothing on tmpfs and memfd, whose bdi has no BDI_CAP_WRITEBACK. A read charges the page cache it reads, and the anonymous pages it faults in when the user buffer is untouched, which can't be reclaimed without swap. Today the charge path throttles these calls for memory.high, except huge tmpfs pages and THP faults on the user buffer, whose charges can't block under the default defrag setting.
A later patch stops enforcing memory.high in the charge path, where the task may hold locks other tasks need. Handle the overage in the copy loops instead:
- generic_perform_write() and iomap_write_iter(): next to the dirty throttling call.
- btrfs_buffered_write(): between ranges. btrfs has its own loop, which the other two hooks don't cover.
- filemap_read(): after each folio is copied. Once per batch isn't enough, as a batch of large folios can hold tens of MiB.
Writers hold the inode's i_rwsem there, and dirty throttling already sleeps under it. Two cases hold more: splice() from a pipe also holds the pipe's lock, and gfs2 holds the resource group and quota glocks in writes that allocate, which other writers to that resource group wait for, on any node. The charge path sleeps with all of these locks today, plus those taken in write_begin(). In btrfs, copy_one_range() has already released its space reservation or handed it to delalloc, and dropped the snapshot lock of a NOCOW write.
Most readers hold no locks. XFS, NFS and netfs hold the file's lock shared, and gfs2 and ocfs2 a cluster lock, so a writer to that file waits while a reader is throttled, as it does today. filemap_read() keeps its folio batch referenced while the task is throttled. If it dropped the batch, reclaim could free the folios not copied yet, and they would be read again: in a VM, a cold 256M read under memory.high=64M then read five to nine times as much from disk for the same progress. After the sleep, it reads i_size and mapping_writably_mapped() again.
IOCB_NOWAIT writes and IOCB_NOWAIT or IOCB_WAITQ reads skip the hook:io_uring issues them with ctx->uring_lock held. Reads into kernel buffers skip it too: kernel_read() callers may hold other locks, like cred_guard_mutex in execve(). Reclaim uses the gfp mask of the allocation that follows, so filesystems that clear __GFP_FS get GFP_NOFS reclaim.
Without these hooks, once the synchronous step is gone, a 256M write under memory.high=64M in a VM reaches 257-264M on ext4, XFS and btrfs instead of 64-67M.
--- fs/btrfs/file.c | 9 +++++++++ fs/iomap/buffered-io.c | 9 +++++++++ mm/filemap.c | 31 +++++++++++++++++++++++++++++++ 3 files changed, 49 insertions(+)
fs/btrfs/file.c +9 -0
@@ -18,6 +18,7 @@#include <linux/uio.h> #include <linux/iversion.h> #include <linux/fsverity.h> +#include <linux/memcontrol.h> #include "ctree.h" #include "direct-io.h" #include "disk-io.h"
@@ -1320,6 +1321,7 @@ ssize_t btrfs_buffered_write(struct kiocb *iocb, struct iov_iter *iter)struct file *file = iocb->ki_filp;loff_t pos;struct inode *inode =file_inode (file); +const gfp_t high_gfp =btrfs_alloc_write_mask (inode->i_mapping);struct extent_changeset *data_reserved =NULL ;size_t num_written =0 ;ssize_t ret;
@@ -1356,6 +1358,13 @@ ssize_t btrfs_buffered_write(struct kiocb *iocb, struct iov_iter *iter)break ; pos += ret; num_written += ret; +/* +* A large write can charge a lot before the task returns to +* userspace. Handle memory.high overage between ranges, where +* btrfs holds only the inode lock. +*/ +if (!nowait) +mem_cgroup_over_high_throttle (high_gfp);cond_resched (); }
fs/iomap/buffered-io.c +9 -0
@@ -7,6 +7,7 @@#include <linux/buffer_head.h> #include <linux/writeback.h> #include <linux/swap.h> +#include <linux/memcontrol.h> #include <linux/migrate.h> #include <linux/fserror.h> #include <linux/fsverity.h>
@@ -1193,6 +1194,7 @@ static int iomap_write_iter(struct iomap_iter *iter, struct iov_iter *i,struct address_space *mapping = iter->inode->i_mapping;size_t chunk =mapping_max_folio_size (mapping);unsigned int bdp_flags = (iter->flags &IOMAP_NOWAIT ) ?BDP_ASYNC :0 ; +gfp_t high_gfp =mapping_gfp_constraint (mapping,GFP_KERNEL ); do { struct folio *folio;
@@ -1212,6 +1214,13 @@ static int iomap_write_iter(struct iomap_iter *iter, struct iov_iter *i,if (unlikely (status))break ; +/* +* A large write can charge a lot before the task returns to +* userspace. Handle memory.high overage here too. +*/ +if (!(iter->flags &IOMAP_NOWAIT )) +mem_cgroup_over_high_throttle (high_gfp); +if (bytes >iomap_length (iter)) bytes =iomap_length (iter);
mm/filemap.c +31 -0
@@ -2822,6 +2822,7 @@ ssize_t filemap_read(struct kiocb *iocb, struct iov_iter *iter,struct file_ra_state *ra = &filp->f_ra;struct address_space *mapping = filp->f_mapping;struct inode *inode = mapping->host; +gfp_t high_gfp =mapping_gfp_constraint (mapping,GFP_KERNEL );struct folio_batch fbatch;int i, error =0 ;bool writably_mapped;
@@ -2913,6 +2914,28 @@ ssize_t filemap_read(struct kiocb *iocb, struct iov_iter *iter, error = -EFAULT ;break ; } + +/* +* A large read can charge a lot, for the page cache and +* for faulting in the buffer. Handle memory.high +* overage after each folio, as a batch of large folios +* can be big. Keep the batch, or reclaim could free +* folios not copied yet and they would be read again. +* The file may change meanwhile, so check its size and +* mappings again. Reads into kernel buffers skip this: +* their callers may hold other locks. +*/ +if (user_backed_iter (iter) && + !(iocb->ki_flags & (IOCB_NOWAIT |IOCB_WAITQ )) && +mem_cgroup_over_high_pending ()) { +mem_cgroup_handle_over_high (high_gfp); + isize =i_size_read (inode); +if (iocb->ki_pos >= isize) +break ; + end_offset =min_t (loff_t, isize, + iocb->ki_pos + iter->count); + writably_mapped =mapping_writably_mapped (mapping); + } } put_folios:for (i =0 ; i <folio_batch_count (&fbatch); i++) {
@@ -4385,6 +4408,7 @@ ssize_t generic_perform_write(struct kiocb *iocb, struct iov_iter *i)struct address_space *mapping = file->f_mapping;const struct address_space_operations *a_ops = mapping->a_ops;size_t chunk =mapping_max_folio_size (mapping); +gfp_t high_gfp =mapping_gfp_constraint (mapping,GFP_KERNEL );long status =0 ;ssize_t written =0 ;
@@ -4401,6 +4425,13 @@ ssize_t generic_perform_write(struct kiocb *iocb, struct iov_iter *i) bytes =min (chunk - offset, bytes);balance_dirty_pages_ratelimited (mapping); +/* +* A large write can charge a lot before the task returns to +* userspace. Handle memory.high overage here too. +*/ +if (!(iocb->ki_flags &IOCB_NOWAIT )) +mem_cgroup_over_high_throttle (high_gfp); +if (fatal_signal_pending (current)) { status = -EINTR ;break ;
Signature · 1 line
2.53.1-Meta
[email protected]
AI reviewRe: [RFC PATCH 3/7] mm, fs: handle memory.high overage in buffered reads and writes
Thank you for your contribution! Sashiko AI review found 1 potential issue to consider: - [High] mm/filemap: read corruption in filemap_read() if i_size grows during throttling --
Thank you for your contribution! Sashiko AI review found 1 potential issue to consider:
- [High] mm/filemap: read corruption in filemap_read() if i_size grows during throttling
--
commit 0c9715c2d32c8a6aa3086d8505b1f86c7ebcf8cb
Author: Shakeel Butt <[email protected]>
mm, fs: handle memory.high overage in buffered reads and writes
This commit adds memory.high throttling hooks to buffered read and write copy loops so overage is handled outside the memcg charge path.
Quoted 4 lines
mm/filemap.c +0
-0
[ ... ]
Quoted 17 lines
mm/filemap.c +13
-0@@ -2913,6 +2914,28 @@ ssize_t filemap_read(struct kiocb *iocb, struct iov_iter *iter, error = -EFAULT ;break ; } + +/* +* A large read can charge a lot, for the page cache and +* for faulting in the buffer. Handle memory.high +* overage after each folio, as a batch of large folios +* can be big. Keep the batch, or reclaim could free +* folios not copied yet and they would be read again. +* The file may change meanwhile, so check its size and +* mappings again. Reads into kernel buffers skip this: +* their callers may hold other locks. +*/ + if (user_backed_iter(iter) && + !(iocb->ki_flags & (IOCB_NOWAIT | IOCB_WAITQ)) &&
mm/filemap.c ssize_t filemap_read(struct kiocb *iocb, struct iov_iter *iter, · line 2939
Quoted 10 lines
mm/filemap.c +9
-0+ mem_cgroup_over_high_pending()) { +mem_cgroup_handle_over_high (high_gfp); + isize =i_size_read (inode); +if (iocb->ki_pos >= isize) +break ; + end_offset =min_t (loff_t, isize, + iocb->ki_pos + iter->count); + writably_mapped =mapping_writably_mapped (mapping); + } }
[Severity: High]
Can refreshing isize and end_offset here corrupt the read buffer if i_size grows while the reader is throttled?
filemap_get_pages() does not clamp the batch to i_size, so fbatch can already hold uptodate folios past the initial end_offset (for example, from a concurrent append that marked fbatch.folios[i + 1] uptodate before calling i_size_write(), or after fallocate(FALLOC_FL_KEEP_SIZE)):
mm/filemap.c:filemap_read() {
...
for (i = 0; i < folio_batch_count(&fbatch); i++) {
struct folio *folio = fbatch.folios[i];
size_t fsize = folio_size(folio);
size_t offset = iocb->ki_pos & (fsize - 1);
size_t bytes = min_t(loff_t, end_offset - iocb->ki_pos,
fsize - offset);
size_t copied;
if (end_offset < folio_pos(folio))
break;
...
}When end_offset lies inside fbatch.folios[i], iteration i copies only up to end_offset (offset + copied < fsize), leaving iocb->ki_pos inside fbatch.folios[i]. Normally the next iteration immediately breaks out of the loop because end_offset < folio_pos(fbatch.folios[i + 1]).
If mem_cgroup_handle_over_high() runs at the end of iteration i and a concurrent writer extends i_size past folio_pos(fbatch.folios[i + 1]), end_offset is raised past folio_pos(fbatch.folios[i + 1]). The loop then advances to fbatch.folios[i + 1] while iocb->ki_pos still points into fbatch.folios[i], skipping the rest of fbatch.folios[i] and copying from fbatch.folios[i + 1] at the wrong file offset.
Breaking out of the fbatch loop when offset + copied < fsize (or before re-evaluating a larger end_offset) would keep iocb->ki_pos aligned with the current folio.
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=3
Shakeel Butt
maintainerRFC PATCH 4/7 mm/shmem: handle memory.high overage in fallocate and reads
In reply to Shakeel Buttfallocate() on tmpfs or memfd allocates the whole range in one call, and a read can copy almost 2GiB, faulting in the user buffer and swapping the file back in. Without swap, this memory can't be reclaimed, and the internal mount behind memfd_create(), MAP_SHARED|MAP_ANONYMOUS and SysV shm has no size limit. Today the charge path throttles both for memory.high, except huge page charges, which can't block under the default defrag setting. Writes go through generic_perform_write(), which the previous patch covers.
A later patch stops enforcing memory.high in the charge path, where the task may hold locks other tasks need. Handle the overage after each fallocate iteration, where only inode_lock is held, and after each folio a read copies, where no locks are held. This covers huge pages too. Reclaim doesn't need inode_lock: shmem_writeout() reads i_private under i_lock. As in filemap_read(), reads into kernel buffers and IOCB_NOWAIT or IOCB_WAITQ reads skip it. Reclaim uses the mapping's gfp constraint.
With swap, this reclaim finds the folios the fallocate added. shmem_writeout() won't swap them out and counts each try in nr_unswapped, and once there are more tries than new folios, the fallocate fails with -ENOMEM. memory.high should throttle the call, not fail it, so tries made by the fallocating task's own memory.high reclaim don't count. Tries by other reclaim still count. In a VM with swap, a 512M tmpfs fallocate under memory.high=64M fails with -ENOMEM 3 times out of 3 today with small pages, and would with huge pages too if this reclaim's tries counted. With this patch it is throttled instead.
--- mm/shmem.c | 37 ++++++++++++++++++++++++++++++++++--- 1 file changed, 34 insertions(+), 3 deletions(-)
mm/shmem.c +34 -3
@@ -110,6 +110,8 @@ struct shmem_falloc {pgoff_t next;/* the next page offset to be fallocated */ pgoff_t nr_falloced;/* how many new pages have been fallocated */ pgoff_t nr_unswapped;/* how often writeout refused to swap out */ +/* set while the fallocating task handles memory.high overage */ +struct task_struct *high_reclaimer; };struct shmem_options {
@@ -1918,10 +1920,13 @@ int shmem_writeout(struct swap_io_ctx *ctx, struct folio *folio,if (shmem_falloc && !shmem_falloc->waitq && index >= shmem_falloc->start && - index < shmem_falloc->next) - shmem_falloc->nr_unswapped += nr_pages; -else + index < shmem_falloc->next) { +/* See the memory.high comment in shmem_fallocate() */ +if (READ_ONCE (shmem_falloc->high_reclaimer) != current) + shmem_falloc->nr_unswapped += nr_pages; + }else { shmem_falloc =NULL ; + }spin_unlock (&inode->i_lock);if (shmem_falloc) goto redirty;
@@ -3567,6 +3572,7 @@ static ssize_t shmem_file_read_iter(struct kiocb *iocb, struct iov_iter *to)struct file *file = iocb->ki_filp;struct inode *inode =file_inode (file);struct address_space *mapping = inode->i_mapping; +gfp_t high_gfp =mapping_gfp_constraint (mapping,GFP_KERNEL );pgoff_t index;unsigned long offset;int error =0 ;
@@ -3675,6 +3681,15 @@ static ssize_t shmem_file_read_iter(struct kiocb *iocb, struct iov_iter *to) error = -EFAULT ;break ; } +/* +* A large read can charge a lot, for faulting in the buffer +* and swapping in the file. Handle memory.high overage here, +* where no locks are held. As in filemap_read(), reads into +* kernel buffers and reads that must not block skip this. +*/ +if (user_backed_iter (to) && + !(iocb->ki_flags & (IOCB_NOWAIT |IOCB_WAITQ ))) +mem_cgroup_over_high_throttle (high_gfp);cond_resched (); }
@@ -3883,6 +3898,7 @@ static long shmem_fallocate(struct file *file, int mode, loff_t offset,struct inode *inode =file_inode (file);struct shmem_sb_info *sbinfo =SHMEM_SB (inode->i_sb);struct shmem_inode_info *info =SHMEM_I (inode); +gfp_t high_gfp =mapping_gfp_constraint (inode->i_mapping,GFP_KERNEL );struct shmem_falloc shmem_falloc;pgoff_t start, index, end, undo_fallocend;loff_t aligned_end;
@@ -3962,6 +3978,7 @@ static long shmem_fallocate(struct file *file, int mode, loff_t offset, shmem_falloc.next = start; shmem_falloc.nr_falloced =0 ; shmem_falloc.nr_unswapped =0 ; + shmem_falloc.high_reclaimer =NULL ;spin_lock (&inode->i_lock);WRITE_ONCE (inode->i_private, &shmem_falloc);spin_unlock (&inode->i_lock);
@@ -4031,6 +4048,20 @@ static long shmem_fallocate(struct file *file, int mode, loff_t offset,folio_mark_dirty (folio);folio_unlock (folio);folio_put (folio); + +/* +* Handle memory.high overage here, where only inode_lock is +* held; reclaim doesn't take it. shmem_writeout() won't swap +* out the folios added so far, and counts each try toward the +* -ENOMEM check above. memory.high should only throttle the +* fallocate, so tell shmem_writeout() not to count tries made +* by this reclaim. +*/ +if (mem_cgroup_over_high_pending ()) { +WRITE_ONCE (shmem_falloc.high_reclaimer, current); +mem_cgroup_handle_over_high (high_gfp); +WRITE_ONCE (shmem_falloc.high_reclaimer,NULL ); + }cond_resched (); }
Signature · 1 line
2.53.1-Meta
Shakeel Butt
maintainerRFC PATCH 5/7 mm/khugepaged: handle memory.high overage in MADV_COLLAPSE
In reply to Shakeel ButtMADV_COLLAPSE doesn't limit empty PTEs, so it can turn a PMD range with one resident page into a 2MiB THP. Over a large sparse range, one madvise() call can charge far more than the range used before. Today the charge path throttles it for memory.high.
A later patch stops enforcing memory.high in the charge path, where the task may hold locks other tasks need. Handle the overage right after each collapse that dropped mmap_lock instead:collapse_single_pmd() drops it to allocate and charge the THP. This also covers the last PMD of each call, so madvise() over many small VMAs, or a process_madvise() vector of small ranges, is throttled too.
process_madvise() on another process charges that process's cgroup. If the caller is in that cgroup or below it, the caller handles the overage here. Otherwise high_work reclaims it and no task is throttled, as before.
--- mm/khugepaged.c | 8 ++++++++ 1 file changed, 8 insertions(+)
mm/khugepaged.c +8 -0
@@ -3306,6 +3306,14 @@ int madvise_collapse(struct vm_area_struct *vma, unsigned long start, result =collapse_single_pmd (addr, vma, &mmap_unlocked, cc); +/* +* collapse_single_pmd() drops mmap_lock to allocate and charge +* a THP. Handle memory.high overage before taking it again, +* including after the last PMD. +*/ +if (mmap_unlocked) +mem_cgroup_over_high_throttle (GFP_KERNEL ); +switch (result) {case SCAN_SUCCEED :case SCAN_PMD_MAPPED :
Signature · 1 line
2.53.1-Meta
Shakeel Butt
maintainerRFC PATCH 6/7 memcg: remove synchronous memory.high enforcement
In reply to Shakeel ButtA task stuck in memcg limit enforcement in the charge path keeps its locks, and tasks in other cgroups that need them wait for a limit that has nothing to do with them.
memory.max is a hard limit, so the charge path has to enforce it. memory.high doesn't: since commit b23afb93d317 ("memcg: punt high overage reclaim to return-to-userland path"), the charge only records the overage, and the task reclaims and is throttled on return to userspace, where it holds no locks. Avoiding this priority inversion was one of the reasons for that commit.
But return to userspace is too late when one system call charges a lot, like mlock() of a large range. So commit c9afe31ec443 ("memcg:
synchronously enforce memory.high for large overcharges") made a charge that can block handle the overage right away once more than a batch is pending. That step reclaims and sleeps, up to 2s per call, under whatever locks the caller holds: mmap_lock while mlock() faults in a MADV_HUGEPAGE range, or invalidate_lock and locked folios in readahead. It also misses THP charges in other ranges, which can't block under the default defrag setting.
The previous patches handle the overage where the task holds no locks, or only the locks of its file, in the operations that charge the most in one call: populating memory, shmem fallocate, buffered writes, reads into user memory and MADV_COLLAPSE. Remove the synchronous step.
Other paths now handle the overage only on return to userspace, and memory.max still limits them. Many can get a hook later. For example, reading /dev/urandom into an untouched buffer reaches 513M under memory.high=64M. With THP on, it did before too, but with small pages the synchronous step used to throttle it. Another is mprotect() making an mlocked private range writable, which copies its pages with mmap_lock held for writing.
Update the memory.high documentation to say when tasks are throttled.
--- Documentation/admin-guide/cgroup-v2.rst | 12 ++++++++++-- mm/memcontrol.c | 20 ++++++-------------- 2 files changed, 16 insertions(+), 16 deletions(-)
Documentation/admin-guide/cgroup-v2.rst +10 -2
@@ -1386,6 +1386,14 @@ write "max"; to set it to zero explicitly, write "0". over the high boundary, the processes of the cgroup are throttled and put under heavy reclaim pressure. + A task is throttled when it returns to userspace after + charging, and at some points in long operations, such as + populating memory or large buffered reads and writes. A + system call that allocates a lot without reaching such a + point can take the usage over the high boundary by up to + the amount it allocates. Kernel threads are not throttled: + a worker reclaims for them instead. + Going over the high limit never invokes the OOM killer and under extreme conditions the limit may be breached. The high limit should be used in scenarios where an external process
@@ -1396,8 +1404,8 @@ write "max"; to set it to zero explicitly, write "0". reclaim is bypassed. This is useful for admin processes that need to dynamically adjust the job's memory limits without expending their own CPU resources on memory reclamation. The - job will trigger the reclaim and/or get throttled on its - next charge request. + job will reclaim and/or get throttled after it charges + memory again. Please note that with O_NONBLOCK, there is a chance that the target memory cgroup may take indefinite amount of time to
mm/memcontrol.c +6 -14
@@ -2615,8 +2615,9 @@ static unsigned long calculate_high_delay(unsigned int nr_pages, }/* -* Handle pending high overage during a blocking charge or on return -* to userspace. +* Handle pending high overage on return to userspace, or at a point in a +* long-running operation where the task can sleep (see +* mem_cgroup_over_high_throttle()). */ void __mem_cgroup_handle_over_high (gfp_t gfp_mask) {
@@ -2725,9 +2726,9 @@ static void memcg_enforce_high(struct mem_cgroup *memcg, unsigned int nr_pages, !current->mm || !mm_match_cgroup (current->mm, memcg);/* -* Check the charged cgroup and its parents. Local charges normally defer -* reclaim until return to userspace so it can use GFP_KERNEL. Large -* pending overages may be handled synchronously below. The handler looks +* Check the charged cgroup and its parents. Local charges defer reclaim +* until return to userspace, or until a long-running operation reaches +* a point where it can sleep and calls the handler. The handler looks * up current->mm and rechecks the limits. If the task moved to another * cgroup in the meantime, the handler works on the new cgroup instead. * That is rare and costs little.
@@ -2765,15 +2766,6 @@ static void memcg_enforce_high(struct mem_cgroup *memcg, unsigned int nr_pages, current->memcg_nr_pages_over_high += nr_pages;set_notify_resume (current); } - -/* -* Earlier charges may have left high overage pending even if this charge -* found no new overage. Handle it now if this charge can block, to limit -* overrun while the task remains in the kernel. -*/ -if (current->memcg_nr_pages_over_high >MEMCG_CHARGE_BATCH && -gfpflags_allow_blocking (gfp_mask)) -__mem_cgroup_handle_over_high (gfp_mask); } static int try_charge_memcg(struct mem_cgroup *memcg, gfp_t gfp_mask,
Signature · 1 line
2.53.1-Meta
[email protected]
AI reviewRe: [RFC PATCH 6/7] memcg: remove synchronous memory.high enforcement
Thank you for your contribution! Sashiko AI review found 1 potential issue to consider: - [High] memcg: memcg_enforce_high() leaves vhost_task PF_USER_WORKER threads unthrottled --
Thank you for your contribution! Sashiko AI review found 1 potential issue to consider:
- [High] memcg: memcg_enforce_high() leaves vhost_task PF_USER_WORKER threads unthrottled
--
commit dfb4aaa1098cc7070f618d5995322b418d647d3c
Author: Shakeel Butt <[email protected]>
memcg: remove synchronous memory.high enforcement
This commit removes synchronous memory.high enforcement from memcg_enforce_high() so that pending high overages are handled on return to userspace or at explicit throttling points in long-running operations, and updates the cgroup v2 documentation accordingly.
Quoted 4 lines
mm/memcontrol.c +0
-0
[ ... ]
Quoted 4 lines
mm/memcontrol.c +0
-0@@ -2765,15 +2766,6 @@ static void memcg_enforce_high(struct mem_cgroup *memcg, unsigned int nr_pages, current->memcg_nr_pages_over_high += nr_pages;set_notify_resume (current); }
mm/memcontrol.c static void memcg_enforce_high(struct mem_cgroup *memcg, unsigned int nr_pages, · line 2769
Quoted 10 lines
mm/memcontrol.c +0
-9- -/* -* Earlier charges may have left high overage pending even if this charge -* found no new overage. Handle it now if this charge can block, to limit -* overrun while the task remains in the kernel. -*/ -if (current->memcg_nr_pages_over_high >MEMCG_CHARGE_BATCH && -gfpflags_allow_blocking (gfp_mask)) -__mem_cgroup_handle_over_high (gfp_mask); }
[Severity: High]
Does removing the synchronous __mem_cgroup_handle_over_high() call leave vhost_task worker threads completely unthrottled by memory.high and memory.swap.high?
vhost_task_create() clones workers with CLONE_VM and args.user_worker = 1, setting PF_USER_WORKER rather than PF_KTHREAD while sharing the owner's mm. In memcg_enforce_high(), use_high_work checks current->flags & PF_KTHREAD, so it evaluates to false for vhost workers and only increments current->memcg_nr_pages_over_high and calls set_notify_resume(current).
Unlike io_uring PF_USER_WORKER threads, which check TIF_NOTIFY_RESUME and call resume_user_mode_work(), vhost_task_fn() loops in kernel space until exit without ever returning to userspace or handling notify-resume work:
kernel/vhost_task.c:vhost_task_fn() {
for (;;) {
...
did_work = vtsk->fn(vtsk->data);
if (!did_work)
schedule();
}
do_exit(0);
}Previously, blocking charges from vhost workers (such as guest memory faults or skb allocations) were caught by the synchronous __mem_cgroup_handle_over_high() check here once memcg_nr_pages_over_high exceeded MEMCG_CHARGE_BATCH. Without it, vhost workers neither schedule high_work nor run return-to-userspace reclaim, bypassing memory.high and memory.swap.high for the lifetime of the worker while memcg_nr_pages_over_high overflows.
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=6
Shakeel Butt
maintainerRFC PATCH 7/7 selftests/cgroup: make test_memcg_high_sync check that throttling holds
In reply to Shakeel Butttest_memcg_high_sync mlock()s 200M with memory.high=30M and memory.max=140M, and expects memory.high to throttle the call before it reaches memory.max. But it checked memory.events right after the first high event, so it passed as soon as the handler ran once, even if the rest of the call was not throttled.
That hid a real gap: with THP=always and the default defrag=madvise, THP charges skipped the synchronous handler, and only a few page table charges ran it. In a VM, usage reached 85M within 115ms and memory.max about 2s later.
After the first event, check for 5 seconds that no max event shows up and the child is still running, then that memory.current went up. Counting high events isn't enough: one handler call can log many while it retries reclaim. Also kill and reap the child, which may still be throttled.
--- .../selftests/cgroup/test_memcontrol.c | 39 +++++++++++++++---- 1 file changed, 32 insertions(+), 7 deletions(-)
tools/testing/selftests/cgroup/test_memcontrol.c +32 -7
@@ -771,14 +771,16 @@ static int alloc_anon_mlock(const char *cgroup, void *arg)/* * This test checks that memory.high is able to throttle big single shot -* allocation i.e. large allocation within one kernel entry. +* allocation i.e. large allocation within one kernel entry. The name is +* from when this was done synchronously in the charge path. */ static int test_memcg_high_sync (const char *root) { -int ret =KSFT_FAIL , pid, fd =-1 ; +int ret =KSFT_FAIL , pid =-1 , fd =-1 , i;char *memcg;long pre_high, pre_max; -long post_high, post_max; +long first_high, first_current; +long post_max, post_current; memcg =cg_name (root,"memcg_test" );if (!memcg)
@@ -811,17 +813,40 @@ static int test_memcg_high_sync(const char *root)cg_wait_for (fd); - post_high =cg_read_key_long (memcg,"memory.events" ,"high " ); - post_max =cg_read_key_long (memcg,"memory.events" ,"max " ); -if (post_high <0 || post_max <0 ) +/* +* The first event only shows that throttling started. For 5 seconds, +* check that the child is still running and doesn't reach memory.max, +* then that memory.current went up, so it is throttled, not stuck. +* High events don't show progress: one handler call can log many. +*/ + first_high =cg_read_key_long (memcg,"memory.events" ,"high " ); + first_current =cg_read_long (memcg,"memory.current" ); +if (first_high <0 || first_high == pre_high || first_current <0 ) goto cleanup; -if (pre_high == post_high || pre_max != post_max) +for (i =0 ; i <50 ; i++) { +usleep (100000 ); + post_max =cg_read_key_long (memcg,"memory.events" ,"max " ); +if (post_max <0 || post_max != pre_max) + goto cleanup; +if (waitpid (pid,NULL ,WNOHANG ) == pid) { + pid =-1 ; + goto cleanup; + } + } + + post_current =cg_read_long (memcg,"memory.current" ); +if (post_current <= first_current) goto cleanup; ret =KSFT_PASS ; cleanup: +if (pid >0 ) { +kill (pid,SIGKILL ); +while (waitpid (pid,NULL ,0 ) <0 && errno ==EINTR ) + ; + }if (fd >=0 )close (fd);cg_destroy (memcg);
Signature · 1 line
2.53.1-Meta