mm/memory_hotplug: bound offline retry loops with a configurable limit
Subsystems (5)
cover mm/memory_hotplug: bound offline retry loops with a configurable limit
Memory offlining can loop forever when a memory block holds a page that can never be migrated or freed. This series adds an opt-in retry limit that lets offline_pages() bail out with -EBUSY instead of retrying forever. The default (0) preserves today's behaviour exactly.
Patch 1 is the fix; patch 2 is an in-tree selftest that reproduces the hang from an unsignalable kworker and verifies the limit recovers it.
1. The problem
==============
offline_pages() repeats two nested steps until the whole range is isolated:
1. Inner loop: find and migrate movable pages out of the range
(scan_movable_pages() + do_migrate_range()).
2. Outer loop: re-check isolation (test_pages_isolated()); if not fully isolated, go back to step 1.
Neither loop has a termination condition. When a page can never be migrated, scan_movable_pages() keeps returning the same pfn, do_migrate_range() keeps failing on it, and control never even reaches the outer isolation re-check - the inner loop spins forever. The admin guide acknowledges this:
"Further, memory offlining might retry for a long time (or even
forever), until aborted by the user."
The single escape in the loop is signal_pending(current):
offline_pages(pfn, end_pfn):
do { /* outer: isolation */
do { /* inner: migration */
if (signal_pending(current)) /* <-- ONLY escape */
goto failed_removal;
pfn = scan_movable_pages(pfn, end_pfn); do_migrate_range(pfn, end_pfn); /* may never succeed */
} while (pfn < end_pfn);
} while (test_pages_isolated(...));
2. Why the kernel itself should be able to bail
===============================================
We keep hitting this during memory hot-remove operations: a single stuck page can block the whole operation. This was also reported in
[4] earlier; one example, where offlining kept failing to migrate a busy block-device page-cache page (aops:def_blk_aops) in a normal zone:
[10880.889199] page dumped because: migration failure [10880.889232] migrating pfn 2a87b failed ret:1 [10880.889235] page: refcount:3 mapcount:0 mapping:00000000718ec5a6 index:0x857 pfn:0x2a87b [10880.889241] aops:def_blk_aops ino:800003 dentry name(?):"" [10880.889245] flags: 0x33ffffe00004104(referenced|active|private|node=3|zone=0|lastcpupid=0x1fffff) ... [10880.889291] migrating pfn 2a87b failed ret:1
A userspace-driven abort (timeout + signal) covers the common case where the offline is requested from process context, e.g. a tool writing to sysfs or an "echo offline > .../state". There may still be scenarios where handling this in the kernel itself is helpful
- for instance, hotplug requests that are processed entirely from kernel thread context. An ACPI DIMM eject (kacpi_hotplug_wq) runs offline_pages() on an ordered workqueue, where there is no task to receive the signal, so the loop's one escape (signal_pending()) can never fire and a stuck page also blocks every hotplug event queued behind it. For example, an ACPI DIMM eject stuck in the offline loop looks like:
task: kworker/u16:3 Workqueue: kacpi_hotplug acpi_hotplug_work_fn
Call Trace: offline_pages+0x... memory_subsys_offline+0x... device_offline+0x... acpi_bus_offline+0x... acpi_scan_hot_remove+0x... acpi_device_hotplug+0x... acpi_hotplug_work_fn+0x... process_one_work+0x... worker_thread+0x... kthread+0x...
3. Prior reports and proposals
==============================
Variants of this have surfaced repeatedly:
[1] 2020: zswap/z3fold pages made offlining loop forever; isolate_movable_page() permanently failed and the same pfn was retried indefinitely (same "isolation failed" signature as [5]):
https://lore.kernel.org/all/[email protected]/
[2] 2023: a proposal to bail after 3 consecutive migration failures on the same pfn. Nacked: any hard-coded retry limit leads to premature failures; pointer to userspace-driven termination:
https://lore.kernel.org/all/[email protected]/
[3] 2024: a proposal for a hard-coded limit of 5 migration retries (fuse temp-page series v4 4/6); dropped in v5 after review:
https://lore.kernel.org/linux-mm/[email protected]/
[4] 2025: ppc64 hot-unplug hang on aops:def_blk_aops migration failures:
https://lore.kernel.org/all/[email protected]/
[5] 2025: ppc64 hot-unplug hang 20+ hours on a single slab page failing isolation, also blocking pcp_batch_high_lock users:
https://lore.kernel.org/all/[email protected]/
The common failure mode is the same: when migration cannot succeed, offline_pages() has no way to give up on its own.
4. Why a configurable limit, not a hard-coded one
=================================================
The kernel used to have exactly this bound with a hard-coded limit of 5 migration retries plus a 120s timeout removed by commit 72b39cfc4d75 ("mm, memory_hotplug: do not fail offlining too early") because hard-coded policy caused premature failures on large machines. That commit explicitly left the door open:
"If we need some upper bound - e.g. timeout based - then we should
have a proper and user defined policy for that. In any case there should be a clear use case when introducing it."
In general, it would be good to have this handled in the kernel as well, by providing a configuration option: a user-defined policy, off by default, that lets the kernel bound the retries on its own. This helps the kworker paths above, which have no userspace to time out or signal them, but it is not limited to them: it also gives ordinary process-context offlines a way out when the userspace driver does not implement its own timeout/abort, so a single stuck page need not wedge the operation indefinitely.
5. The fix (patch 1)
====================
A single counter at the top of the inner migration loop bounds the number of migration passes:
max_passes = READ_ONCE(offline_migrate_max_passes);
if (max_passes && pass++ >= max_passes) {
dump_page(stuck page);
ret = -EBUSY;
goto failed_removal_isolated;
}Both unbounded cases pass through this point: a stuck page spins in the inner loop without ever reaching the outer test_pages_isolated() re-check (so an outer-loop counter would never fire), and every outer-loop restart re-enters the inner loop. Either way the counter only grows, so the limit eventually trips.
Because the value is re-read every pass, writing the parameter aborts an offline that is already stuck - an admin can rescue a hung kworker without rebooting. On bailout the existing failure path (shared with signal backoff and unmovable pages) un-isolates the range, sends MEM_CANCEL_OFFLINE and returns -EBUSY; the block stays online and usable, and dump_page() dumps the offending page.
It is a module parameter, default 0 (unlimited = today's behaviour); at 0 the check short-circuits and there is no functional change:
boot: memory_hotplug.offline_migrate_max_passes=
runtime: /sys/module/memory_hotplug/parameters/offline_migrate_max_passes
Alternative implementation: stall timeout
-----------------------------------------
Instead of counting passes, the bound could be a stall timeout: give up only after some time has elapsed with no forward progress. Any successfully migrated page resets the clock, so it fires only on an offline that is genuinely stuck, not one that is merely slow:
/* give up only after X ms with ZERO forward progress */ if (migrated_a_page)
last_progress = jiffies; /* any progress resets */
if (stall_ms && time_after(jiffies,
last_progress + msecs_to_jiffies(stall_ms)))
goto failed_removal_isolated; /* -EBUSY */
We are happy to send v2 in this shape if that or another approach is preferred.
6. Testing
==========
Patch 2 adds tools/testing/selftests/mm/vmtest_memory_hotplug.sh, a self-contained VM test (virtme-ng + qemu, boots the just-built kernel, no rootfs/QMP). It cold-plugs a pc-dimm, onlines it ZONE_MOVABLE, vmsplice-pins one page, and ejects it from inside the guest so the offline runs on the kacpi_hotplug_wq kworker; this reproduces the hang and then rescues it live. The vmsplice pin is a deliberate, targeted reproducer used only to make the hang path reproducible in a test VM.
Actual run (kernel log timestamps trimmed):
TAP version 13
1..3
# DIMM memory blocks: 32 33
# pinned one page in block memory33
# ejecting /sys/bus/acpi/devices/PNP0C80:00
# block memory33 going-offline, holding 10s
# do_migrate_range+0x1eb/0x260
# offline_pages+0x361/0x520
# memory_subsys_offline+0xdb/0x180
# device_offline+0xd0/0x130
# acpi_bus_offline+0x11f/0x190
# acpi_device_hotplug+0x1e8/0x3e0
# acpi_hotplug_work_fn+0x1e/0x30
# process_one_work+0x16a/0x310
# armed offline_migrate_max_passes=50
# memory offlining [mem 0x108000000-0x10fffffff]: giving up after 50 passes
ok 1 ACPI eject kworker hangs unbounded in offline_pages()
ok 2 stuck offline rescued by writing offline_migrate_max_passes
ok 3 memory block online and usable after the bailout
# Totals: pass:3 fail:0 xfail:0 xpass:0 skip:0 error:0The test SKIPs (never FAILs) without virtme-ng/qemu or on non-ACPI arches.
This RFC is to discuss the approach and find the best way to address the infinite retry loop in offline_pages(). Please let me know if you have any comments.
Thanks,
Aboorva
Aboorva Devarajan (2):
mm/memory_hotplug: bound offline retry loops with a configurable limit selftests/mm: add pc-dimm ACPI eject selftest for offline_migrate_max_passes
.../admin-guide/kernel-parameters.txt | 14 + .../admin-guide/mm/memory-hotplug.rst | 26 +- mm/memory_hotplug.c | 38 ++ tools/testing/selftests/mm/Makefile | 3 + tools/testing/selftests/mm/config | 3 + .../selftests/mm/vmtest_memory_hotplug.sh | 369 ++++++++++++++++++ 6 files changed, 452 insertions(+), 1 deletion(-) create mode 100755 tools/testing/selftests/mm/vmtest_memory_hotplug.sh
Signature · 1 line
2.54.0
RFC PATCH 1/2 mm/memory_hotplug: bound offline retry loops with a configurable limit
offline_pages() migrates and isolates pages in two nested loops with no upper bound. A page that can never be migrated or freed (a long-term pin, or a slab page that raced into the range) keeps the loop spinning and the offline never returns. This is a known issue, noted in the code ("TODO: fatal migration failures should bail out") and in the admin guide ("memory offlining might retry for a long time (or even forever), until aborted by the user").
The only escape is signal_pending(current), which works when userspace drives the offline but not for in-kernel callers. ACPI DIMM hot-unplug (kacpi_hotplug_wq) and similar in-kernel hotplug paths run offline_pages() on ordered workqueues where signal_pending() can never become true, so a stuck offline wedges the workqueue and blocks all later hotplug events with no way to abort it.
Add an opt-in backstop: a counter at the top of the inner migration loop bounds the number of passes over the range. The check sits in the inner loop because that is where a stuck page spins; since every outer pass runs the inner loop at least once, this bounds both loops. On the limit offline_pages() fails with -EBUSY and dump_page()s the stuck page.
The parameter (memory_hotplug.offline_migrate_max_passes) defaults to 0 (unlimited, today's behaviour) and is re-read every pass, so an offline that is already stuck can be rescued at runtime by writing a non-zero value. Such callers already handle -EBUSY, so no caller changes are needed.
--- .../admin-guide/kernel-parameters.txt | 14 +++++++ .../admin-guide/mm/memory-hotplug.rst | 26 ++++++++++++- mm/memory_hotplug.c | 38 +++++++++++++++++++ 3 files changed, 77 insertions(+), 1 deletion(-)
Documentation/admin-guide/kernel-parameters.txt +14 -0
@@ -3980,6 +3980,20 @@ Kernel parameters Note that even when enabled, there are a few cases where the feature is not effective. + memory_hotplug.offline_migrate_max_passes= + [KNL] Maximum number of migration passes + for memory offlining. + Format: <integer> + default: 0 (no limit, historical behaviour) + When non-zero, memory offlining gives up with + -EBUSY after this many migration passes over + the range, instead of retrying forever on a + stuck page. Each pass scans the range and + migrates what it can. + The parameter is re-read on every pass, so an + offline request that is already stuck can be + rescued at runtime by writing the parameter. + memtest= [KNL,X86,ARM,M68K,PPC,RISCV,EARLY] Enable memtest Format: <integer> default : 0 <disable>
Documentation/admin-guide/mm/memory-hotplug.rst +25 -1
@@ -224,7 +224,10 @@ increases memory offlining reliability; still, memory offlining can fail in some corner cases. Further, memory offlining might retry for a long time (or even forever), until -aborted by the user. +aborted by the user. The ``memory_hotplug.offline_migrate_max_passes`` +parameter can be used to bound the number of retry passes, making an offline +request that cannot make progress fail with ``-EBUSY`` instead of retrying +indefinitely (see `Module Parameters`_). Offlining of a memory block can be triggered via::
@@ -547,6 +550,27 @@ The following module parameters are currently defined: possible. Parameter availability depends on CONFIG_NUMA. +``offline_migrate_max_passes`` read-write: Maximum number of migration + passes for a single memory offline + request. Memory offlining retries to + migrate and isolate pages until it succeeds; + a page that can never be migrated or freed + makes it retry forever. When this parameter + is non-zero, an offline request gives up + with ``-EBUSY`` after the configured number + of migration passes and the memory block + stays online. + + The parameter is re-read on every pass, so + an offline request that is already stuck + retrying can be rescued at runtime by + writing a non-zero value, without a reboot. + + The default is "0", meaning no limit (the + historical behaviour). + + Parameter availability depends on + CONFIG_MEMORY_HOTREMOVE. ================================ =============================================== ZONE_MOVABLE
mm/memory_hotplug.c +38 -0
@@ -1785,6 +1785,12 @@ bool mhp_range_allowed(u64 start, u64 size, bool need_mapping) }#ifdef CONFIG_MEMORY_HOTREMOVE +/* Max offline_pages() migration passes before -EBUSY; 0 = retry forever. */ +static unsigned int offline_migrate_max_passes__read_mostly ; +module_param (offline_migrate_max_passes, uint,0644 ); +MODULE_PARM_DESC (offline_migrate_max_passes, +"Max migration passes before memory offline gives up (0 = no limit)" ); +/* * Scan pfn range [start,end) to find movable/migratable pages (LRU and * hugetlb folio, movable_ops pages). Will skip over most unmovable
@@ -1966,6 +1972,8 @@ int offline_pages(unsigned long start_pfn, unsigned long nr_pages,struct node_notify node_arg = { .nid =NUMA_NO_NODE , }; +unsigned int max_passes; +unsigned int pass =0 ;unsigned long flags;char *reason;int ret;
@@ -2062,6 +2070,36 @@ int offline_pages(unsigned long start_pfn, unsigned long nr_pages, goto failed_removal_isolated; } +/* +* A page that can never be migrated (e.g. a +* long-term pin) makes this loop retry forever. +* Give up after the configured number of passes. +* The limit is re-read on every pass so that an +* offline request that is already stuck here can +* be aborted at runtime by writing the parameter, +* which matters for in-kernel callers (e.g. ACPI +* DIMM hot-unplug) that run on kworkers and can +* never be aborted by a signal. +*/ + max_passes =READ_ONCE (offline_migrate_max_passes); +if (max_passes && pass++ >= max_passes) { +pr_warn ("memory offlining [mem %#010llx-%#010llx]: giving up after %u passes\n" , + (unsigned long long )start_pfn <<PAGE_SHIFT , + ((unsigned long long )end_pfn <<PAGE_SHIFT ) -1 , + pass -1 ); +/* +* Dump the page we last stopped at as a +* diagnostic hint: usually the stuck page, +* but just the range start after a restart. +*/ +if (pfn >= start_pfn && pfn < end_pfn) +dump_page (pfn_to_page (pfn), +"memory offline retry limit" ); + ret = -EBUSY ; + reason ="retry limit exceeded" ; + goto failed_removal_isolated; + } +cond_resched (); ret =scan_movable_pages (pfn, end_pfn, &pfn);
Signature · 1 line
2.54.0
RFC PATCH 2/2 selftests/mm: add pc-dimm ACPI eject selftest for offline_migrate_max_passes
In reply to Aboorva DevarajanAdd a self-contained VM-based selftest that reproduces the ACPI memory hot-unplug hang in kworker context and verifies that memory_hotplug.offline_migrate_max_passes recovers from it.
The test needs a real ACPI eject because the offline that hangs runs on the kacpi_hotplug_wq kworker (acpi_bus_offline -> device_offline -> offline_pages) and cannot be aborted by a signal.
It boots the just-built kernel under virtme-ng with a cold-plugged pc-dimm and runs everything inside the guest:
1. online the DIMM memory blocks as ZONE_MOVABLE;
2. long-term pin one page of the DIMM via vmsplice (a small pin
helper is embedded in the script and built at run time);
3. eject the DIMM via /sys/bus/acpi/devices/PNP0C80:*/eject, which runs the offline on the kacpi_hotplug_wq kworker, the same path as a hypervisor-initiated device_del;
4. observe the real hang: with the limit disabled (default), the block sits in "going-offline" while the kworker spins in offline_pages();
5. rescue it at runtime: write offline_migrate_max_passes and verify the kworker gives up with -EBUSY and the block returns to "online".
Sample output:
TAP version 13
1..3
# DIMM memory blocks: 32 33
# pinned one page in block memory33 # ejecting /sys/bus/acpi/devices/PNP0C80:00 # block memory33 going-offline, holding 10s # do_migrate_range+0x1eb/0x260 # offline_pages+0x361/0x520 # memory_subsys_offline+0xdb/0x180 # device_offline+0xd0/0x130 # acpi_bus_offline+0x11f/0x190 # acpi_device_hotplug+0x1e8/0x3e0 # acpi_hotplug_work_fn+0x1e/0x30 # process_one_work+0x16a/0x310 # armed offline_migrate_max_passes=50 # memory offlining [mem 0x108000000-0x10fffffff]: giving up after 50 passes ok 1 ACPI eject kworker hangs unbounded in offline_pages() ok 2 stuck offline rescued by writing offline_migrate_max_passes ok 3 memory block online and usable after the bailout # Totals: pass:3 fail:0 xfail:0 xpass:0 skip:0 error:0
The wrapper is added as TEST_PROGS_EXTENDED (like hwpoison-panic.sh):
installed and kept executable, but not run from a default kselftest invocation, because it spawns a VM itself. The config fragment gains the options the guest kernel needs (MEMORY_HOTPLUG, MEMORY_HOTREMOVE, ACPI_HOTPLUG_MEMORY). When vng, qemu, cc or the kernel knob are missing it skips cleanly (KSFT_SKIP), so it is safe to run anywhere.
The vmsplice-based pin in this test is a targeted technique to reliably reproduce the hang path for testing purposes; production scenarios typically involve slab pages or busy block-device page-cache racing into the offline range.
--- tools/testing/selftests/mm/Makefile | 3 + tools/testing/selftests/mm/config | 3 + .../selftests/mm/vmtest_memory_hotplug.sh | 369 ++++++++++++++++++ 3 files changed, 375 insertions(+) create mode 100755 tools/testing/selftests/mm/vmtest_memory_hotplug.sh
tools/testing/selftests/mm/Makefile +3 -0
@@ -177,6 +177,9 @@ TEST_PROGS += ksft_vmalloc.sh # Destructive: every successful run panics the kernel. Installed and # kept executable, but not run from a default kselftest invocation. TEST_PROGS_EXTENDED += hwpoison-panic.sh +# Spawns its own VM (virtme-ng); not run from a default kselftest +# invocation. +TEST_PROGS_EXTENDED += vmtest_memory_hotplug.sh TEST_FILES := test_vmalloc.sh TEST_FILES += test_hmm.sh
tools/testing/selftests/mm/config +3 -0
@@ -14,3 +14,6 @@ CONFIG_UPROBES=y CONFIG_MEMORY_FAILURE=y CONFIG_HWPOISON_INJECT=m CONFIG_PROC_MEM_ALWAYS_FORCE=y +CONFIG_MEMORY_HOTPLUG=y +CONFIG_MEMORY_HOTREMOVE=y +CONFIG_ACPI_HOTPLUG_MEMORY=y
tools/testing/selftests/mm/vmtest_memory_hotplug.sh new file +369 -0
369 lines not on this page · raw message
Signature · 1 line
2.54.0
David Hildenbrand (Arm)
maintainerQuoted 49 lines · Aboorva Devarajan
Memory offlining can loop forever when a memory block holds a page that can never be migrated or freed. This series adds an opt-in retry limit that lets
offline_pages()bail out with-EBUSYinstead of retrying forever. The default (0) preserves today's behaviour exactly.Patch 1 is the fix; patch 2 is an in-tree selftest that reproduces the hang from an unsignalable kworker and verifies the limit recovers it.
1. The problem
==============
offline_pages()repeats two nested steps until the whole range is isolated:1. Inner loop: find and migrate movable pages out of the range
(scan_movable_pages() + do_migrate_range()).2. Outer loop: re-check isolation (
test_pages_isolated()); if not fully isolated, go back to step 1.Neither loop has a termination condition. When a page can never be migrated,
scan_movable_pages()keeps returning the same pfn,do_migrate_range()keeps failing on it, and control never even reaches the outer isolation re-check - the inner loop spins forever. The admin guide acknowledges this:"Further, memory offlining might retry for a long time (or even
forever), until aborted by the user."The single escape in the loop is
signal_pending(current):
offline_pages(pfn, end_pfn):
do { /* outer: isolation */
do { /* inner: migration */
if (signal_pending(current)) /* <-- ONLY escape */
gotofailed_removal;
pfn =scan_movable_pages(pfn, end_pfn);do_migrate_range(pfn, end_pfn); /* may never succeed */
} while (pfn <end_pfn);
} while (test_pages_isolated(...));2. Why the kernel itself should be able to bail
===============================================We keep hitting this during memory hot-remove operations: a single stuck page can block the whole operation. This was also reported in
[4] earlier; one example, where offlining kept failing to migrate a busy block-device page-cache page (aops:def_blk_aops) in a normal zone:
Quoted 7 lines
[10880.889199] page dumped because: migration failure [10880.889232] migrating pfn 2a87b failed ret:1 [10880.889235] page: refcount:3 mapcount:0 mapping:00000000718ec5a6 index:0x857 pfn:0x2a87b [10880.889241] aops:def_blk_aops ino:800003 dentry name(?):"" [10880.889245] flags: 0x33ffffe00004104(referenced|active|private|node=3|zone=0|lastcpupid=0x1fffff) ... [10880.889291] migrating pfn 2a87b failed ret:1
Hi,
the text reads AI generated. If you did use AI, please disclose it properly.
Quick feedback:
1) Using passes is not really what we want in many cases (e.g., ZONE_MOVABLE where we might need a couple of seconds/minutes to complete offlining with a lot of concurrent activity). An actual timeout is also problematic (see below).
2) Having a toggle for all offline_pages() users is questionable. In particular, user-triggered offlining or offlinig triggered on ZONE_MOVABLE etc should not obey such timeouts.
I proposed a more restricted approach only for offline_and_remove_memory() previously [1]
[1] https://lore.kernel.org/all/[email protected]/
Michal back then commented "I really hate having timeouts back. They just proven to be hard to get right and it is essentially a policy implemented in the kernel. They simply do not belong to the kernel space IMHO."
And I agree.
I assume you run into such issues with DIMMs in VMs? virtio-mem handles that much nicer nowadays, by essentially doing the hard part that should fail easily through alloc_contig_range().
offline_pages() is just designed to retry forever.
Signature · 3 lines
Cheers, David
David Hildenbrand (Arm)
maintainerQuoted 81 lines · David Hildenbrand (Arm)
Quoted 49 lines · Aboorva Devarajan
Memory offlining can loop forever when a memory block holds a page that can never be migrated or freed. This series adds an opt-in retry limit that lets
offline_pages()bail out with-EBUSYinstead of retrying forever. The default (0) preserves today's behaviour exactly.Patch 1 is the fix; patch 2 is an in-tree selftest that reproduces the hang from an unsignalable kworker and verifies the limit recovers it.
1. The problem
==============
offline_pages()repeats two nested steps until the whole range is isolated:1. Inner loop: find and migrate movable pages out of the range
(scan_movable_pages() + do_migrate_range()).2. Outer loop: re-check isolation (
test_pages_isolated()); if not fully isolated, go back to step 1.Neither loop has a termination condition. When a page can never be migrated,
scan_movable_pages()keeps returning the same pfn,do_migrate_range()keeps failing on it, and control never even reaches the outer isolation re-check - the inner loop spins forever. The admin guide acknowledges this:"Further, memory offlining might retry for a long time (or even
forever), until aborted by the user."The single escape in the loop is
signal_pending(current):
offline_pages(pfn, end_pfn):
do { /* outer: isolation */
do { /* inner: migration */
if (signal_pending(current)) /* <-- ONLY escape */
gotofailed_removal;
pfn =scan_movable_pages(pfn, end_pfn);do_migrate_range(pfn, end_pfn); /* may never succeed */
} while (pfn <end_pfn);
} while (test_pages_isolated(...));2. Why the kernel itself should be able to bail
===============================================We keep hitting this during memory hot-remove operations: a single stuck page can block the whole operation. This was also reported in
[4] earlier; one example, where offlining kept failing to migrate a busy block-device page-cache page (aops:def_blk_aops) in a normal zone:Quoted 7 lines
[10880.889199] page dumped because: migration failure [10880.889232] migrating pfn 2a87b failed ret:1 [10880.889235] page: refcount:3 mapcount:0 mapping:00000000718ec5a6 index:0x857 pfn:0x2a87b [10880.889241] aops:def_blk_aops ino:800003 dentry name(?):"" [10880.889245] flags: 0x33ffffe00004104(referenced|active|private|node=3|zone=0|lastcpupid=0x1fffff) ... [10880.889291] migrating pfn 2a87b failed ret:1Hi,
the text reads AI generated. If you did use AI, please disclose it properly.
Quick feedback:
1) Using passes is not really what we want in many cases (e.g.,
ZONE_MOVABLEwhere we might need a couple of seconds/minutes to complete offlining with a lot of concurrent activity). An actual timeout is also problematic (see below).2) Having a toggle for all
offline_pages()users is questionable. In particular, user-triggered offlining or offlinig triggered onZONE_MOVABLEetc should not obey such timeouts.I proposed a more restricted approach only for
offline_and_remove_memory()previously [1][1] https://lore.kernel.org/all/[email protected]/
Michal back then commented "I really hate having timeouts back. They just proven to be hard to get right and it is essentially a policy implemented in the kernel. They simply do not belong to the kernel space IMHO."
Quoted 8 lines
And I agree.
I assume you run into such issues with DIMMs in VMs? virtio-mem handles that much nicer nowadays, by essentially doing the hard part that should fail easily through
alloc_contig_range().
offline_pages()is just designed to retry forever.
One thing we can definitely do is to just fail faster if we find an unmovable page in !ZONE_MOVABLE.
Usually that happens when we race offlining (has_unmovable_pages()) with page allocation, and failing on unmovable pages is actually perfectly fine.
But for ZONE_MOVABLE we should keep retrying, because some pages might only look temporarily unmovable.
So that would be the low hanging fruit: on !ZONE_MOVABLE, fail faster.
Signature · 3 lines
Cheers, David
Quoted 59 lines · David Hildenbrand (Arm)
Quoted 58 lines · David Hildenbrand (Arm)
Quoted 56 lines · Aboorva Devarajan
Memory offlining can loop forever when a memory block holds a page that can never be migrated or freed. This series adds an opt-in retry limit that lets
offline_pages()bail out with-EBUSYinstead of retrying forever. The default (0) preserves today's behaviour exactly.Patch 1 is the fix; patch 2 is an in-tree selftest that reproduces the hang from an unsignalable kworker and verifies the limit recovers it.
1. The problem
==============
offline_pages()repeats two nested steps until the whole range is isolated:1. Inner loop: find and migrate movable pages out of the range (
scan_movable_pages()+do_migrate_range()).
2. Outer loop: re-check isolation (test_pages_isolated()); if not fully isolated, go back to step 1.Neither loop has a termination condition. When a page can never be migrated,
scan_movable_pages()keeps returning the same pfn,do_migrate_range()keeps failing on it, and control never even reaches the outer isolation re-check - the inner loop spins forever. The admin guide acknowledges this:"Further, memory offlining might retry for a long time (or even forever), until aborted by the user."
The single escape in the loop is
signal_pending(current):
offline_pages(pfn, end_pfn):
do { /* outer: isolation */ do { /* inner: migration */ if (signal_pending(current)) /* <-- ONLY escape */ gotofailed_removal;
pfn =scan_movable_pages(pfn, end_pfn);do_migrate_range(pfn, end_pfn); /* may never succeed */ } while (pfn <end_pfn);
} while (test_pages_isolated(...));2. Why the kernel itself should be able to bail
===============================================We keep hitting this during memory hot-remove operations: a single stuck page can block the whole operation. This was also reported in
[4] earlier; one example, where offlining kept failing to migrate a busy block-device page-cache page (aops:def_blk_aops) in a normal zone:[10880.889199] page dumped because: migration failure [10880.889232] migrating pfn 2a87b failed ret:1 [10880.889235] page: refcount:3 mapcount:0 mapping:00000000718ec5a6 index:0x857 pfn:0x2a87b [10880.889241] aops:def_blk_aops ino:800003 dentry name(?):"" [10880.889245] flags: 0x33ffffe00004104(referenced|active|private|node=3|zone=0|lastcpupid=0x1fffff) ... [10880.889291] migrating pfn 2a87b failed ret:1
Quoted 2 lines
Quoted 2 lines
Hi,
Hi David,
Thanks a lot for the review.
Quoted 1 line
Quoted 1 line
the text reads AI generated. If you did use AI, please disclose it properly.
yes, I did use LLM to help re-phrase this. I'll make sure to disclose the usage going forward.
Quoted 14 lines
Quoted 14 lines
Quick feedback:
1) Using passes is not really what we want in many cases (e.g.,
ZONE_MOVABLEwhere we might need a couple of seconds/minutes to complete offlining with a lot of concurrent activity). An actual timeout is also problematic (see below).2) Having a toggle for all
offline_pages()users is questionable. In particular, user-triggered offlining or offlinig triggered onZONE_MOVABLEetc should not obey such timeouts.I proposed a more restricted approach only for
offline_and_remove_memory()previously [1]
Quoted 7 lines
Quoted 7 lines
[1] https://lore.kernel.org/all/[email protected]/
Michal back then commented "I really hate having timeouts back. They just proven to be hard to get right and it is essentially a policy implemented in the kernel. They simply do not belong to the kernel space IMHO."
And I agree.
Thanks for pointing to [1]. On our end, on powerpc (pseries),
drmgr (the userspace tool that triggers memory hotplug) already wraps memory hotplug operations with a timeout for exactly this reason. However, since we've now seen this issue occur a couple of times under system load in ZONE_NORMAL, and also as per my understanding offline_pages() can be triggered from other kworker path where it cannot be interrupted and could end up retrying indefinitely, So I wanted to get a consensus on how this should be handled.
Quoted 8 lines
Quoted 7 lines
I assume you run into such issues with DIMMs in VMs? virtio-mem handles that much nicer nowadays, by essentially doing the hard part that should fail easily through
alloc_contig_range().
offline_pages()is just designed to retry forever.
Yes, I do not hit this issue with virtio-mem path, looks like the unsafe unplug mode that relied on offline_pages() for migration was removed in f504e15b94eb ("virtio-mem:
remove unsafe unplug in Big Block Mode (BBM)"), so now it rather goes through the alloc_contig_range() path, which gives up with -EBUSY after a few migration attempts on the same page, so offline_pages() never sees the stuck page.
We originally hit this on a pseries LPAR, during memory hot-remove. offline_pages() gets stuck on a single page that won't move, with two specific cases as below, both in ZONE_NORMAL:
Case 1. a slab page that ends up in the isolated range:
[83310.373767] page_type: f5(slab) [83310.373770] raw: 023ffffe00000000 c0000028e001fa00 [83310.373774] raw: 0000000000000000 0000000001e101e1 [83310.373778] page dumped because: isolation failed [83310.373788] failed to isolate pfn 4dc68
Case 2. a busy block-device page-cache page that keeps failing to migrate:
[10880.889058] page: refcount:3 mapcount:0 mapping:00000000718ec5a6 index:0x857 pfn:0x2a87b [10880.889062] memcg:c00000006a67e000 [10880.889064] aops:def_blk_aops ino:800003 dentry name(?):"" [10880.889069] flags: 0x33ffffe00004104(referenced|active|private|node=3|zone=0|lastcpupid=0x1fffff) [10880.889073] raw: 033ffffe00004104 c0000000b8667960 c0000000b8667960 c0000000130f8dd8 [10880.889077] raw: 0000000000000857 c000000172abe1e0 00000003ffffffff c00000006a67e000 [10880.889081] page dumped because: migration failure [10880.889114] migrating pfn 2a87b failed ret:1
I can also trigger it in a VM by hot-adding a DIMM, pinning a page and removing the DIMM. This is the part I wanted to highlight, the ACPI DIMM remove runs offline_pages() from the ACPI kworker, and as far as I understand, nothing can abort it, there is no signal to deliver, so there is no way to cancel the work, so it spins until reboot in this case as well?
Quoted 10 lines
One thing we can definitely do is to just fail faster if we find an unmovable page in !ZONE_MOVABLE.
Usually that happens when we race offlining (
has_unmovable_pages()) with page allocation, and failing on unmovable pages is actually perfectly fine.But for
ZONE_MOVABLEwe should keep retrying, because some pages might only look temporarily unmovable.So that would be the low hanging fruit: on !ZONE_MOVABLE, fail faster.
This handles scenarios like Case 1. A page can still be allocated from the range after isolation and become slab. When test_pages_isolated() fails, we can re-check that page with page_is_unmovable() and return -EBUSY if it's unmovable. I'm currently testing this approach will send a patch for review.
Case 2 is different. The page is on the LRU, so it is considered movable and the fast-fail condition abovewon't be triggered. Instead page migration keeps failing, causing offline_pages() to retry indefinitely
and can hang forever. This could be handled with a timeout or a user
policy in the application that
initiated the hotplug operation (signals). However, for the kworker
paths that trigger memory
offlining (ACPI remove, etc.), should this instead be handled by the
driver rather than relying onoffline_pages(), similar to virtio-mem?
Please let me know your comments.
Thanks,
Aboorva
David Hildenbrand (Arm)
maintainerQuoted 7 lines
Quoted 7 lines
Usually that happens when we race offlining (
has_unmovable_pages()) with page allocation, and failing on unmovable pages is actually perfectly fine.But for
ZONE_MOVABLEwe should keep retrying, because some pages might only look temporarily unmovable.
Quoted 6 lines
Quoted 1 line
So that would be the low hanging fruit: on !ZONE_MOVABLE, fail faster.
This handles scenarios like Case 1. A page can still be allocated from the range after isolation and become slab. When
test_pages_isolated()fails, we can re-check that page withpage_is_unmovable()and return-EBUSYif it's unmovable. I'm currently testing this approach will send a patch for review.
Yes, and that we should handle. It's a known race and we should just fail faster in that case. !ZONE_MOVABLE.
Quoted 10 lines
Case 2 is different. The page is on the LRU, so it is considered movable and the fast-fail condition abovewon't be triggered. Instead page migration keeps failing, causing
offline_pages()to retry indefinitely
and can hang forever. This could be handled with a timeout or a user
policy in the application that
initiated the hotplug operation (signals). However, for the kworker
paths that trigger memory
offlining (ACPI remove, etc.), should this instead be handled by the
driver rather than relying onoffline_pages(), similar to virtio-mem?
If it's on ZONE_MOVABLE, then we should figure out why it keeps failing forever and try fixing that instead.
Above you write: "I can also trigger it in a VM by hot-adding a DIMM, pinning a page and removing the DIMM."
Anybody who intends to take a longterm pin must be using FOLL_LONGTERM, which will take care of exactly this condition: migrating memory out of ZONE_MOVABLE first.
So which interface does you reproducer use to trigger pinning?
Signature · 3 lines
Cheers, David