Skip to content

mm/memory_hotplug: allow to specify a default online_type

David Hildenbrand linux-hyperv linux-mm linuxppc-dev lkml 3 versions – In review

Get series with b4
b4 am [email protected]

Versions newest first · linked by author and title

  1. v3 Latest8 patches 10 repliesIn review
  2. v28 patches 20 repliesApplied 8/8
  3. v15 patches 18 repliesSuperseded

Changes in v3 from the cover letter

- "hv_balloon: don't check for memhp_auto_online manually" -- init_completion() before register_memory_notifier()
- Minor typo fix

Patches in v3 No applies recorded yet

#Subject and filesSizeRepliesApplied
1/8drivers/base/memory: rename MMOP_ONLINE_KEEP to MMOP_ONLINEdrivers/base/memory.c include/linux/memory_hotplug.h+10 -51 reply
2/8drivers/base/memory: map MMOP_OFFLINE to 0drivers/base/memory.c include/linux/memory_hotplug.h+5 -81 reply
3/8drivers/base/memory: store mapping between MMOP_* and string in an arraydrivers/base/memory.c+23 -154 replies
4/8powernv/memtrace: always online added memory blocksarch/powerpc/platforms/powernv/memtrace.c+4 -100 replies
5/8hv_balloon: don't check for memhp_auto_online manuallydrivers/hv/hv_balloon.c+10 -150 replies
6/8mm/memory_hotplug: unexport memhp_auto_onlinemm/memory_hotplug.c+0 -11 reply
7/8mm/memory_hotplug: convert memhp_auto_online to store an online_typedrivers/base/memory.c include/linux/memory_hotplug.h mm/memory_hotplug.c+12 -121 reply
8/8mm/memory_hotplug: allow to specify a default online_typedrivers/base/memory.c include/linux/memory_hotplug.h mm/memory_hotplug.c+11 -101 reply

Reviews trailers per patch

All 8 patches have a Reviewed-by or Acked-by of their own; the series has one for all of them.

PatchWei YangMichal HockoMichael EllermanVitaly KuznetsovPankaj GuptaBaoquan He
SeriesR-b
1/8R-b A-b
2/8R-b A-b A-b
3/8R-b A-b A-b
4/8R-b A-b A-b
5/8R-b
6/8R-b A-b A-b
7/8R-b A-b A-b
8/8R-b A-b A-b

Trailers given in replies are filled; those carried in the patches from earlier versions are outlined with dashes. Scroll the table sideways for all 6 reviewers.

Read the v3 thread

Cover letter [PATCH v3 0/8] mm/memory_hotplug: allow to specify a default online_type

Distributions nowadays use udev rules ([1] [2]) to specify if and how to online hotplugged memory. The rules seem to get more complex with many special cases. Due to the various special cases, CONFIG_MEMORY_HOTPLUG_DEFAULT_ONLINE cannot be used. All memory hotplug is handled via udev rules.

Everytime we hotplug memory, the udev rule will come to the same conclusion. Especially Hyper-V (but also soon virtio-mem) add a lot of memory in separate memory blocks and wait for memory to get onlined by user space before continuing to add more memory blocks (to not add memory faster than it is getting onlined). This of course slows down the whole memory hotplug process.

To make the job of distributions easier and to avoid udev rules that get more and more complicated, let's extend the mechanism provided by
- /sys/devices/system/memory/auto_online_blocks
- "memhp_default_state=" on the kernel cmdline
to be able to specify also "online_movable" as well as "online_kernel"

v2 -> v3:
- "hv_balloon: don't check for memhp_auto_online manually" -- init_completion() before register_memory_notifier()
- Minor typo fix

v1 -> v2:
- Tweaked some patch descriptions
- Added
-- "powernv/memtrace: always online added memory blocks"
-- "hv_balloon: don't check for memhp_auto_online manually" -- "mm/memory_hotplug: unexport memhp_auto_online"
- "mm/memory_hotplug: convert memhp_auto_online to store an online_type" -- No longer touches hv/memtrace code

=== Example /usr/libexec/config-memhotplug ===

#!/bin/bash

VIRT=systemd-detect-virt --vm
ARCH=uname -p

sense_virtio_mem() {
if [ -d "/sys/bus/virtio/drivers/virtio_mem/" ]; then
DEVICES=find /sys/bus/virtio/drivers/virtio_mem/ -maxdepth 1 -type l | wc -l if [ $DEVICES != "0" ]; then
return 0
fi
fi
return 1
}

if [ ! -e "/sys/devices/system/memory/auto_online_blocks" ]; then
echo "Memory hotplug configuration support missing in the kernel" exit 1
fi

if grep "memhp_default_state=" /proc/cmdline > /dev/null; then
echo "Memory hotplug configuration overridden in kernel cmdline (memhp_default_state=)" exit 1
fi

if [ $VIRT == "microsoft" ]; then
echo "Detected Hyper-V on $ARCH"
# Hyper-V wants all memory in ZONE_NORMAL
ONLINE_TYPE="online_kernel"
elif sense_virtio_mem; then
echo "Detected virtio-mem on $ARCH"

  # virtio-mem wants all memory in ZONE_NORMAL
  ONLINE_TYPE="online_kernel"
elif [ $ARCH == "s390x" ] || [ $ARCH == "s390" ]; then
  echo "Detected $ARCH"
  # standby memory should not be onlined automatically
  ONLINE_TYPE="offline"
elif [ $ARCH == "ppc64" ] || [ $ARCH == "ppc64le" ]; then
  echo "Detected" $ARCH
  # PPC64 onlines all hotplugged memory right from the kernel
  ONLINE_TYPE="offline"
elif [ $VIRT == "none" ]; then
  echo "Detected bare-metal on $ARCH"
  # Bare metal users expect hotplugged memory to be unpluggable. We assume
  # that ZONE imbalances on such enterpise servers cannot happen and is
  # properly documented
  ONLINE_TYPE="online_movable"
else
  # TODO: Hypervisors that want to unplug DIMMs and can guarantee that ZONE
  # imbalances won't happen
  echo "Detected $VIRT on $ARCH"
  # Usually, ballooning is used in virtual environments, so memory should go to
  # ZONE_NORMAL. However, sometimes "movable_node" is relevant.
  ONLINE_TYPE="online"
fi

echo "Selected online_type:" $ONLINE_TYPE

# Configure what to do with memory that will be hotplugged in the future echo $ONLINE_TYPE 2>/dev/null > /sys/devices/system/memory/auto_online_blocks if [ $? != "0" ]; then
echo "Memory hotplug cannot be configured (e.g., old kernel or missing permissions)" # A backup udev rule should handle old kernels if necessary
exit 1
fi

# Process all already pluggedd blocks (e.g., DIMMs, but also Hyper-V or virtio-mem) if [ $ONLINE_TYPE != "offline" ]; then
for MEMORY in /sys/devices/system/memory/memory*; do
STATE=cat $MEMORY/state
if [ $STATE == "offline" ]; then
echo $ONLINE_TYPE > $MEMORY/state
fi
done
fi

=== Example /usr/lib/systemd/system/config-memhotplug.service ===

[Unit]
Description=Configure memory hotplug behavior DefaultDependencies=no
Conflicts=shutdown.target
Before=sysinit.target shutdown.target After=systemd-modules-load.service ConditionPathExists=|/sys/devices/system/memory/auto_online_blocks

[Service]
ExecStart=/usr/libexec/config-memhotplug
Type=oneshot
TimeoutSec=0
RemainAfterExit=yes

[Install]
WantedBy=sysinit.target

=== Example modification to the 40-redhat.rules [2] ===

40-redhat.rules-new renamed from 40-redhat.rules +3 -0
index 2c690e5..168fd03 100644
@@ -6,6 +6,9 @@ SUBSYSTEM=="cpu", ACTION=="add", TEST=="online", ATTR{online}=="0", ATTR{online}
 # Memory hotadd request
 SUBSYSTEM!="memory", GOTO="memory_hotplug_end"
 ACTION!="add", GOTO="memory_hotplug_end"
+# memory hotplug behavior configured
+PROGRAM=="grep online /sys/devices/system/memory/auto_online_blocks", GOTO="memory_hotplug_end"
+
 PROGRAM="/bin/uname -p", RESULT=="s390*", GOTO="memory_hotplug_end"
 
 ENV{.state}="online"

===

[1] https://github.com/lnykryn/systemd-rhel/pull/281
[2] https://github.com/lnykryn/systemd-rhel/blob/staging/rules/40-redhat.rules

Cc Vitaly Kuznetsov <[email protected]>
Cc Yumei Huang <[email protected]>
Cc Igor Mammedov <[email protected]>
Cc Baoquan He <[email protected]>
Cc Eduardo Habkost <[email protected]>
Cc Milan Zamazal <[email protected]>

David Hildenbrand (8):
drivers/base/memory: rename MMOP_ONLINE_KEEP to MMOP_ONLINE drivers/base/memory: map MMOP_OFFLINE to 0 drivers/base/memory: store mapping between MMOP_* and string in an
array
powernv/memtrace: always online added memory blocks
hv_balloon: don't check for memhp_auto_online manually mm/memory_hotplug: unexport memhp_auto_online mm/memory_hotplug: convert memhp_auto_online to store an online_type mm/memory_hotplug: allow to specify a default online_type

 arch/powerpc/platforms/powernv/memtrace.c | 14 ++---
 drivers/base/memory.c                     | 71 ++++++++++++-----------
 drivers/hv/hv_balloon.c                   | 25 ++++----
 include/linux/memory_hotplug.h            | 13 ++++-
 mm/memory_hotplug.c                       | 16 ++---
 5 files changed, 69 insertions(+), 70 deletions(-)
Signature · 1 line
2.24.1