systemd stateless configuration
v6 cover systemd stateless configuration
This patch set is largely Jonas Bonn's to move towards a "stateless" configuration:
These patches make some modifications to systemd with the long-term goal of being able to run OE in systemd's "stateless" configuration. "Stateless" boils down to building an image with empty /etc and /var directories so that volatile (tmpfs) filesystems can be mounted there; this requires that the system subsequently be able to populate these directories dynamically, which systemd mostly takes care of if things are done right.
In these patches:
i) Don't include machine-id in writable images so that systemd can run
its first-boot machinery
ii) Move systemd configuration files out of /etc iii) Allow systemd to dynamically enable services and populate
/etc/systemd/system via the presets mechanism
There's a long way to go to get to a working "stateless" configuration. Getting to a "volatile" system (just empty /var) should be easier and I'll post patches moving things in that direction shortly.
However as a result of the systemd 242 upgrade, which includes 01d2041e41f4 ("meson: stop creating enablement symlinks in /etc during installation"), services such as systemd-networkd are no longer enabled in images.
This patch set fixes this problem in addition to satisfying the goal of moving towards "stateless" configurations.
The issue with respect to image testing during CI was caused by systemd-time-wait-sync.service being enabled due to the lack of a default preset policy:
https://www.freedesktop.org/wiki/Software/systemd/Preset/#howto
Changes in v6:
- switch configuration to simple overrides in /usr/lib/systemd/*.conf.d
- make systemd RRECOMMENDS rather than RDEPENDS on systemd-conf
- don't exit in postinst as when that executes we're actually a concatenation of all fragments
- validate SYSTEMD_AUTO_ENABLE is enable or disable
- rewrite systemctl-native in Python
- moved systemctl preset-all to IMAGE_PREPROCESS so it runs after ROOTFS, run for all images, not just read-only
Changes in v5:
- rebased for systemd 242
- install default preset distribution policy of "enable nothing"
Alex Kiernan (3):
systemd-conf: simplify creation of machine-specific configuration
systemctl-native: Rewrite in Python supporting preset-all and mask
image: call systemctl preset-all for images
Jonas Bonn (3):
systemd: don't build firstboot by default
systemd: do not create machine-id
systemd: create preset files instead of installing in image
meta/classes/image.bbclass | 9 +- meta/classes/rootfs-postcommands.bbclass | 6 + meta/classes/systemd.bbclass | 41 +- .../systemd/systemd-conf/journald.conf | 3 + .../systemd/systemd-conf/logind.conf | 2 + .../systemd/systemd-conf/system.conf | 2 + .../systemd/systemd-conf/system.conf-qemuall | 3 + meta/recipes-core/systemd/systemd-conf_242.bb | 61 +-- .../systemd/systemd-systemctl/systemctl | 476 ++++++++++-------- .../systemd/systemd/99-default.preset | 1 + meta/recipes-core/systemd/systemd_242.bb | 26 +- 11 files changed, 360 insertions(+), 270 deletions(-) create mode 100644 meta/recipes-core/systemd/systemd-conf/journald.conf create mode 100644 meta/recipes-core/systemd/systemd-conf/logind.conf create mode 100644 meta/recipes-core/systemd/systemd-conf/system.conf create mode 100644 meta/recipes-core/systemd/systemd-conf/system.conf-qemuall create mode 100644 meta/recipes-core/systemd/systemd/99-default.preset
Signature · 1 line
2.17.1
PATCH v6 1/6 systemd: don't build firstboot by default
From: Jonas Bonn <[email protected]>
The firstboot service prompts the user for information about the host at first boot. Systemd determines whether or not a boot is a "first boot" by the existence of the file /etc/machine-id. Since oe-core always includes this file (it is part of the systemd package), the firstboot service never runs so this service is being built but never run.
A follow-up patch to this one will remove the machine-id from the systemd build and allow it to be created automatically by systemd at "first boot". With that patch, we don't want the firstboot service to suddenly start being invoked and presenting a prompt to the user.
With this patch, the firstboot service becomes a PACKAGECONFIG option that the user must actively select.
---
Changes in v6: None
Changes in v5:
- rebased for systemd 242
meta/recipes-core/systemd/systemd_242.bb | 1 - 1 file changed, 1 deletion(-)
meta/recipes-core/systemd/systemd_242.bb +0 -1
@@ -66,7 +66,6 @@ PACKAGECONFIG ??= " \ acl \ backlight \ binfmt \ - firstboot \ gshadow \ hibernate \ hostnamed \
Signature · 1 line
2.17.1
PATCH v6 2/6 systemd: do not create machine-id
In reply to Alex KiernanFrom: Jonas Bonn <[email protected]>
There is no reason to have an emtpy machine-id as part of the systemd package. Either:
i) the filesystem is writable and the file will be created automatically; or
ii) the filesystem is read-only, in which case the empty machine-id file should be created as part of the read-only-rootfs tweaks.
---
Changes in v6: None
Changes in v5:
- install default preset distribution policy of "enable nothing"
meta/classes/rootfs-postcommands.bbclass | 6 ++++++ meta/recipes-core/systemd/systemd-conf_242.bb | 9 ++------- meta/recipes-core/systemd/systemd/99-default.preset | 1 + meta/recipes-core/systemd/systemd_242.bb | 6 +++++- 4 files changed, 14 insertions(+), 8 deletions(-) create mode 100644 meta/recipes-core/systemd/systemd/99-default.preset
meta/classes/rootfs-postcommands.bbclass +6 -0
@@ -126,6 +126,12 @@ read_only_rootfs_hook () { ${IMAGE_ROOTFS}/etc/init.d/populate-volatile.sh fi fi + + if ${@bb.utils.contains("DISTRO_FEATURES", "systemd", "true", "false", d)}; then + # Create machine-id + # 20:12 < mezcalero> koen: you have three options: a) run systemd-machine-id-setup at install time, b) have / read-only and an empty file there (for stateless) and c) boot with / writable + touch ${IMAGE_ROOTFS}${sysconfdir}/machine-id + fi } #
meta/recipes-core/systemd/systemd-conf_242.bb +2 -7
@@ -7,14 +7,13 @@ DefaultTimeoutStartSec setting." PACKAGE_ARCH = "${MACHINE_ARCH}" -CONFFILES_${PN} = "${sysconfdir}/machine-id \ -${sysconfdir}/systemd/coredump.conf \ +CONFFILES_${PN} = "${sysconfdir}/systemd/coredump.conf \ ${sysconfdir}/systemd/journald.conf \ ${sysconfdir}/systemd/logind.conf \ ${sysconfdir}/systemd/system.conf \ ${sysconfdir}/systemd/user.conf" -FILES_${PN} = "${sysconfdir}/machine-id ${sysconfdir}/systemd" +FILES_${PN} = "${sysconfdir}/systemd" do_configure[noexec] = '1' do_compile[noexec] = '1'
@@ -23,10 +22,6 @@ do_install() { rm -rf ${D}/${sysconfdir}/systemd install -d ${D}/${sysconfdir}/systemd - # Create machine-id - # 20:12 < mezcalero> koen: you have three options: a) run systemd-machine-id-setup at install time, b) have / read-only and an empty file there (for stateless) and c) boot with / writable - touch ${D}${sysconfdir}/machine-id - install -m 0644 ${S}/src/coredump/coredump.conf ${D}${sysconfdir}/systemd/coredump.conf install -m 0644 ${S}/src/journal/journald.conf ${D}${sysconfdir}/systemd/journald.conf
meta/recipes-core/systemd/systemd/99-default.preset new file +1 -0
@@ -0,0 +1 @@ +disable *
meta/recipes-core/systemd/systemd_242.bb +5 -1
@@ -22,6 +22,7 @@ SRC_URI += "file://touchscreen.rules \ file://0003-implment-systemd-sysv-install-for-OE.patch \ file://0004-rules-whitelist-hd-devices.patch \ file://0005-rules-watch-metadata-changes-in-ide-devices.patch \ + file://99-default.preset \ " # patches needed by musl
@@ -277,7 +278,6 @@ do_install() { fi # conf files are handled by systemd-conf - rm -f ${D}${sysconfdir}/machine-id rm -f ${D}${sysconfdir}/systemd/coredump.conf rm -f ${D}${sysconfdir}/systemd/journald.conf rm -f ${D}${sysconfdir}/systemd/logind.conf
@@ -287,6 +287,10 @@ do_install() { # duplicate udevadm for postinst script install -d ${D}${libexecdir} ln ${D}${base_bindir}/udevadm ${D}${libexecdir}/${MLPREFIX}udevadm + + # install default policy for presets + # https://www.freedesktop.org/wiki/Software/systemd/Preset/#howto + install -Dm 0644 ${WORKDIR}/99-default.preset ${D}${systemd_unitdir}/system-preset/99-default.preset }
Signature · 1 line
2.17.1
Re: [PATCH v6 2/6] systemd: do not create machine-id
Quoted 6 lines · Alex Kiernan
From: Jonas Bonn <[email protected]>
There is no reason to have an emtpy machine-id as part of the systemd package. Either:
Quoted 7 lines
i) the filesystem is writable and the file will be created automatically; or
ii) the filesystem is read-only, in which case the empty machine-id file should be created as part of the read-only-rootfs tweaks.Signed-off-by Jonas Bonn <[email protected]>Signed-off-by Alex Kiernan <[email protected]>
This patch now results in no "/etc/machine-id" being created which breaks systemd-networkd WiFi connections.
I now always see this error when trying to connect to WiFi withwpa_supplicant: "DHCP4 CLIENT: Failed to set IAID+DUID"
Alistair
Quoted 108 lines
---
Changes in v6: None
Changes in v5:
- install default preset distribution policy of "enable nothing"meta/classes/rootfs-postcommands.bbclass | 6 ++++++ meta/recipes-core/systemd/systemd-conf_242.bb | 9 ++------- meta/recipes-core/systemd/systemd/99-default.preset | 1 + meta/recipes-core/systemd/systemd_242.bb | 6 +++++- 4 files changed, 14 insertions(+), 8 deletions(-) create mode 100644 meta/recipes-core/systemd/systemd/99-default.presetmeta/classes/rootfs-postcommands.bbclass +6
-0@@ -126,6 +126,12 @@ read_only_rootfs_hook () { ${IMAGE_ROOTFS}/etc/init.d/populate-volatile.sh fi fi + + if ${@bb.utils.contains("DISTRO_FEATURES", "systemd", "true", "false", d)}; then + # Create machine-id + # 20:12 < mezcalero> koen: you have three options: a) run systemd-machine-id-setup at install time, b) have / read-only and an empty file there (for stateless) and c) boot with / writable + touch ${IMAGE_ROOTFS}${sysconfdir}/machine-id + fi } #meta/recipes-core/systemd/systemd-conf_242.bb +2
-7@@ -7,14 +7,13 @@ DefaultTimeoutStartSec setting." PACKAGE_ARCH = "${MACHINE_ARCH}" -CONFFILES_${PN} = "${sysconfdir}/machine-id \ -${sysconfdir}/systemd/coredump.conf \ +CONFFILES_${PN} = "${sysconfdir}/systemd/coredump.conf \ ${sysconfdir}/systemd/journald.conf \ ${sysconfdir}/systemd/logind.conf \ ${sysconfdir}/systemd/system.conf \ ${sysconfdir}/systemd/user.conf" -FILES_${PN} = "${sysconfdir}/machine-id ${sysconfdir}/systemd" +FILES_${PN} = "${sysconfdir}/systemd" do_configure[noexec] = '1' do_compile[noexec] = '1'@@ -23,10 +22,6 @@ do_install() { rm -rf ${D}/${sysconfdir}/systemd install -d ${D}/${sysconfdir}/systemd - # Create machine-id - # 20:12 < mezcalero> koen: you have three options: a) run systemd-machine-id-setup at install time, b) have / read-only and an empty file there (for stateless) and c) boot with / writable - touch ${D}${sysconfdir}/machine-id - install -m 0644 ${S}/src/coredump/coredump.conf ${D}${sysconfdir}/systemd/coredump.conf install -m 0644 ${S}/src/journal/journald.conf ${D}${sysconfdir}/systemd/journald.confmeta/recipes-core/systemd/systemd/99-default.preset new file +1
-0@@ -0,0 +1 @@ +disable *meta/recipes-core/systemd/systemd_242.bb +5
-2@@ -22,6 +22,7 @@ SRC_URI += "file://touchscreen.rules \ file://0003-implment-systemd-sysv-install-for-OE.patch \ file://0004-rules-whitelist-hd-devices.patch \ file://0005-rules-watch-metadata-changes-in-ide-devices.patch \ + file://99-default.preset \ " # patches needed by musl@@ -277,7 +278,6 @@ do_install() { fi # conf files are handled by systemd-conf - rm -f ${D}${sysconfdir}/machine-id rm -f ${D}${sysconfdir}/systemd/coredump.conf rm -f ${D}${sysconfdir}/systemd/journald.conf rm -f ${D}${sysconfdir}/systemd/logind.conf@@ -287,6 +287,10 @@ do_install() { # duplicate udevadm for postinst script install -d ${D}${libexecdir} ln ${D}${base_bindir}/udevadm ${D}${libexecdir}/${MLPREFIX}udevadm + + # install default policy for presets + # https://www.freedesktop.org/wiki/Software/systemd/Preset/#howto + install -Dm 0644 ${WORKDIR}/99-default.preset ${D}${systemd_unitdir}/system-preset/99-default.preset } --2.17.1
--
_______________________________________________
Openembedded-core mailing list
[email protected]
http://lists.openembedded.org/mailman/listinfo/openembedded-core
Re: [PATCH v6 2/6] systemd: do not create machine-id
Quoted 13 lines · Alistair Francis
Quoted 11 lines · Alex Kiernan
From: Jonas Bonn <[email protected]>
There is no reason to have an emtpy machine-id as part of the systemd package. Either:
i) the filesystem is writable and the file will be created automatically; or
ii) the filesystem is read-only, in which case the empty machine-id file should be created as part of the read-only-rootfs tweaks.
Quoted 9 lines
Quoted 2 lines
Signed-off-by Jonas Bonn <[email protected]>Signed-off-by Alex Kiernan <[email protected]>This patch now results in no "
/etc/machine-id" being created which breaks systemd-networkd WiFi connections.I now always see this error when trying to connect to WiFi with
wpa_supplicant: "DHCP4 CLIENT: Failed to set IAID+DUID"
I posted this series earlier that fixes everything I know about (including the lack of /etc/machine-id in the stateful case):
https://patchwork.openembedded.org/series/17495/
If you can test and I'll have a look if there's anything else that needs addressing?
Signature · 1 line
Alex Kiernan
Re: [PATCH v6 2/6] systemd: do not create machine-id
Quoted 25 lines · Alex Kiernan
Quoted 13 lines · Alistair Francis
Quoted 11 lines · Alex Kiernan
From: Jonas Bonn <[email protected]>
There is no reason to have an emtpy machine-id as part of the systemd package. Either:
i) the filesystem is writable and the file will be created automatically; or
ii) the filesystem is read-only, in which case the empty machine-id file should be created as part of the read-only-rootfs tweaks.Quoted 9 lines
Quoted 2 lines
Signed-off-by Jonas Bonn <[email protected]>Signed-off-by Alex Kiernan <[email protected]>This patch now results in no "
/etc/machine-id" being created which breaks systemd-networkd WiFi connections.I now always see this error when trying to connect to WiFi with
wpa_supplicant: "DHCP4 CLIENT: Failed to set IAID+DUID"
Quoted 7 lines
I posted this series earlier that fixes everything I know about (including the lack of
/etc/machine-idin the stateful case):https://patchwork.openembedded.org/series/17495/
If you can test and I'll have a look if there's anything else that needs addressing?
Sorry for the long delay, I didn't get a chance to test it until now. It's all fixed on master now with your patches.
Alistair
Quoted 3 lines
--
Alex Kiernan
PATCH v6 3/6 systemd-conf: simplify creation of machine-specific configuration
In reply to Alex KiernanFrom: Alex Kiernan <[email protected]>
The configuration files that systemd installs are just skeletons detailing the available options and their default values. The recommended means of changing the configuration is to provide snippets in configuration directories. For example, journald.conf settings are best set in /usr/lib/system.d/journald.conf.d/ and can be overridden by the user by providing overriding snippets in /etc/systemd/journald.conf.d/.
The systemd-conf package is just providing machine-specific overrides for some systemd defaults.
This patch restores the installation of config files by systemd and reduces systemd-conf to just providing the config snippets in /usr/lib/systemd/*.conf.d. This simpilfies the systemd-conf recipe considerably since it now just sets up a couple of text files and doesn't even need access to the systemd source anymore.
License-Update: configuration snippets licensing is independent of systemd licensing
---
Changes in v6:
- switch configuration to simple overrides in /usr/lib/systemd/*.conf.d
- make systemd RRECOMMENDS rather than RDEPENDS on systemd-conf
Changes in v5: None
.../systemd/systemd-conf/journald.conf | 3 + .../systemd/systemd-conf/logind.conf | 2 + .../systemd/systemd-conf/system.conf | 2 + .../systemd/systemd-conf/system.conf-qemuall | 3 + meta/recipes-core/systemd/systemd-conf_242.bb | 56 +++++++------------ meta/recipes-core/systemd/systemd_242.bb | 19 ++++--- 6 files changed, 41 insertions(+), 44 deletions(-) create mode 100644 meta/recipes-core/systemd/systemd-conf/journald.conf create mode 100644 meta/recipes-core/systemd/systemd-conf/logind.conf create mode 100644 meta/recipes-core/systemd/systemd-conf/system.conf create mode 100644 meta/recipes-core/systemd/systemd-conf/system.conf-qemuall
meta/recipes-core/systemd/systemd-conf/journald.conf new file +3 -0
@@ -0,0 +1,3 @@ +[Journal] +ForwardToSyslog=yes +RuntimeMaxUse=64M
meta/recipes-core/systemd/systemd-conf/logind.conf new file +2 -0
@@ -0,0 +1,2 @@ +[Login] +KillUserProcesses=yes
meta/recipes-core/systemd/systemd-conf/system.conf new file +2 -0
@@ -0,0 +1,2 @@ +[Manager] +DefaultMemoryAccounting=yes
meta/recipes-core/systemd/systemd-conf/system.conf-qemuall new file +3 -0
@@ -0,0 +1,3 @@ +[Manager] +# Change DefaultTimeoutStartSec from 90s to 240s +DefaultTimeoutStartSec = 240s
meta/recipes-core/systemd/systemd-conf_242.bb +20 -36
68 lines not on this page · raw message
meta/recipes-core/systemd/systemd_242.bb +11 -8
@@ -277,13 +277,6 @@ do_install() { fi fi - # conf files are handled by systemd-conf - rm -f ${D}${sysconfdir}/systemd/coredump.conf - rm -f ${D}${sysconfdir}/systemd/journald.conf - rm -f ${D}${sysconfdir}/systemd/logind.conf - rm -f ${D}${sysconfdir}/systemd/system.conf - rm -f ${D}${sysconfdir}/systemd/user.conf - # duplicate udevadm for postinst script install -d ${D}${libexecdir} ln ${D}${base_bindir}/udevadm ${D}${libexecdir}/${MLPREFIX}udevadm
@@ -486,6 +479,15 @@ FILES_${PN}-extra-utils = "\ ${rootlibexecdir}/systemd/systemd-cgroups-agent \ " +CONFFILES_${PN} = "${sysconfdir}/systemd/coredump.conf \ + ${sysconfdir}/systemd/journald.conf \ + ${sysconfdir}/systemd/logind.conf \ + ${sysconfdir}/systemd/system.conf \ + ${sysconfdir}/systemd/user.conf \ + ${sysconfdir}/systemd/resolved.conf \ + ${sysconfdir}/systemd/timesyncd.conf \ +" + FILES_${PN} = " ${base_bindir}/* \ ${base_sbindir}/shutdown \ ${base_sbindir}/halt \
@@ -549,7 +551,7 @@ FILES_${PN}-dev += "${base_libdir}/security/*.la ${datadir}/dbus-1/interfaces/ $ RDEPENDS_${PN} += "kmod dbus util-linux-mount util-linux-umount udev (= ${EXTENDPKGV}) util-linux-agetty util-linux-fsck" RDEPENDS_${PN} += "${@bb.utils.contains('PACKAGECONFIG', 'serial-getty-generator', '', 'systemd-serialgetty', d)}" -RDEPENDS_${PN} += "volatile-binds update-rc.d systemd-conf" +RDEPENDS_${PN} += "volatile-binds update-rc.d" RDEPENDS_${PN} += "${@bb.utils.contains('PACKAGECONFIG', 'myhostname', 'libnss-myhostname', '', d)}" RRECOMMENDS_${PN} += "systemd-extra-utils \
@@ -557,6 +559,7 @@ RRECOMMENDS_${PN} += "systemd-extra-utils \ e2fsprogs-e2fsck \ kernel-module-autofs4 kernel-module-unix kernel-module-ipv6 \ os-release \ + systemd-conf \ " INSANE_SKIP_${PN} += "dev-so libdir"
Signature · 1 line
2.17.1
PATCH v6 4/6 systemd: create preset files instead of installing in image
In reply to Alex KiernanFrom: Jonas Bonn <[email protected]>
At first boot, systemd will create the /etc/systemd/system directory from service preset files. As such, for a normal, writable /etc (writable rootfs), there is no need to set up this directory at image creation time.
This patch changes the systemd machinery to create preset files and to rely on systemd to do the service enablement.
This breaks the read-only-rootfs case; there's a fix for this in a follow-up patch.
---
Changes in v6:
- don't exit in postinst as when that executes we're actually a concatenation of all fragments
- validate SYSTEMD_AUTO_ENABLE is enable or disable
Changes in v5: None
meta/classes/systemd.bbclass | 41 +++++++++++++++++------------------- 1 file changed, 19 insertions(+), 22 deletions(-)
meta/classes/systemd.bbclass +19 -22
@@ -16,45 +16,30 @@ python __anonymous() { # from doing any work so that pure-systemd images don't have redundant init # files. if bb.utils.contains('DISTRO_FEATURES', 'systemd', True, False, d): - d.appendVar("DEPENDS", " systemd-systemctl-native") - d.appendVar("PACKAGE_WRITE_DEPS", " systemd-systemctl-native") if not bb.utils.contains('DISTRO_FEATURES', 'sysvinit', True, False, d): d.setVar("INHIBIT_UPDATERCD_BBCLASS", "1") } systemd_postinst() { -OPTS="" - -if [ -n "$D" ]; then - OPTS="--root=$D" -fi - if type systemctl >/dev/null 2>/dev/null; then if [ -z "$D" ]; then systemctl daemon-reload - fi - - systemctl $OPTS ${SYSTEMD_AUTO_ENABLE} ${SYSTEMD_SERVICE_ESCAPED} + systemctl preset "${SYSTEMD_SERVICE_ESCAPED}" - if [ -z "$D" -a "${SYSTEMD_AUTO_ENABLE}" = "enable" ]; then - systemctl --no-block restart ${SYSTEMD_SERVICE_ESCAPED} + if [ "${SYSTEMD_AUTO_ENABLE}" = "enable" ]; then + systemctl --no-block restart "${SYSTEMD_SERVICE_ESCAPED}" + fi fi fi } systemd_prerm() { -OPTS="" - -if [ -n "$D" ]; then - OPTS="--root=$D" -fi - if type systemctl >/dev/null 2>/dev/null; then if [ -z "$D" ]; then - systemctl stop ${SYSTEMD_SERVICE_ESCAPED} - fi + systemctl stop "${SYSTEMD_SERVICE_ESCAPED}" - systemctl $OPTS disable ${SYSTEMD_SERVICE_ESCAPED} + systemctl disable "${SYSTEMD_SERVICE_ESCAPED}" + fi fi }
@@ -177,12 +162,24 @@ python systemd_populate_packages() { else: bb.fatal("SYSTEMD_SERVICE_%s value %s does not exist" % (pkg_systemd, service)) + def systemd_create_presets(pkg): + action = get_package_var(d, 'SYSTEMD_AUTO_ENABLE', pkg) + if action not in ("enable", "disable"): + bb.fatal("SYSTEMD_AUTO_ENABLE_%s '%s' is not 'enable' or 'disable'" % (pkg, action)) + presetf = oe.path.join(d.getVar("PKGD"), d.getVar("systemd_unitdir"), "system-preset/98-%s.preset" % pkg) + bb.utils.mkdirhier(os.path.dirname(presetf)) + with open(presetf, 'a') as fd: + for service in d.getVar('SYSTEMD_SERVICE_%s' % pkg).split(): + fd.write("%s %s\n" % (action,service)) + d.appendVar("FILES_%s" % pkg, ' ' + oe.path.join(d.getVar("systemd_unitdir"), "system-preset/98-%s.preset" % pkg)) + # Run all modifications once when creating package if os.path.exists(d.getVar("D")): for pkg in d.getVar('SYSTEMD_PACKAGES').split(): systemd_check_package(pkg) if d.getVar('SYSTEMD_SERVICE_' + pkg): systemd_generate_package_scripts(pkg) + systemd_create_presets(pkg) systemd_check_services() }
Signature · 1 line
2.17.1
PATCH v6 5/6 systemctl-native: Rewrite in Python supporting preset-all and mask
In reply to Alex KiernanRewrite systemctl-native in Python so that extending/testing it is easier.
Now that the systemd class sets up service presets instead of actively enabling services, the 'enable' and 'disable' subcommands for systemctl are not actually used anywhere. As such, we can remove these to make sure that nobody inadvertently introduces new uses of them.
This implementation covers preset-all and mask which are the only options used in the current code, but should be readily extensible to other commands.
We use preset-all at image construction time to populate the symlinks used by systemd.
---
Changes in v6:
- rewrite systemctl-native in Python
Changes in v5: None
.../systemd/systemd-systemctl/systemctl | 476 ++++++++++-------- 1 file changed, 280 insertions(+), 196 deletions(-)
meta/recipes-core/systemd/systemd-systemctl/systemctl +280 -196
@@ -1,196 +1,280 @@ -#!/bin/sh -echo "Started $0 $*" - -ROOT= - -# parse command line params -action= -while [ $# != 0 ]; do - opt="$1" - - case "$opt" in - enable) - shift - - action="$opt" - services="$1" - cmd_args="1" - shift - ;; - disable) - shift - - action="$opt" - services="$1" - cmd_args="1" - shift - ;; - mask) - shift - - action="$opt" - services="$1" - cmd_args="1" - shift - ;; - preset) - shift - - action="$opt" - services="$1" - cmd_args="1" - shift - ;; - --root=*) - ROOT=${opt##--root=} - cmd_args="0" - shift - ;; - *) - if [ "$cmd_args" = "1" ]; then - services="$services $opt" - shift - else - echo "'$opt' is an unkown option; exiting with error" - exit 1 - fi - ;; - esac -done -if [ "$action" = "preset" -a "$service_file" = "" ]; then - services=$(for f in `find $ROOT/etc/systemd/system $ROOT/lib/systemd/system $ROOT/usr/lib/systemd/system -type f 2>1`; do basename $f; done) - services="$services $opt" - presetall=1 -fi - -for service in $services; do - if [ "$presetall" = "1" ]; then - action="preset" - fi - if [ "$action" = "mask" ]; then - if [ ! -d $ROOT/etc/systemd/system/ ]; then - mkdir -p $ROOT/etc/systemd/system/ - fi - cmd="ln -s /dev/null $ROOT/etc/systemd/system/$service" - echo "$cmd" - $cmd - exit 0 - fi - - service_base_file=`echo $service | sed 's/\(@\).*\(\.[^.]\+\)/\1\2/'` - if [ -z `echo $service | sed '/@/p;d'` ]; then - echo "Try to find location of $service..." - service_template=false - else - echo "Try to find location of template $service_base_file of instance $service..." - service_template=true - instance_specified=`echo $service | sed 's/^.\+@\(.*\)\.[^.]\+/\1/'` - fi - - # find service file - for p in $ROOT/etc/systemd/system \ - $ROOT/lib/systemd/system \ - $ROOT/usr/lib/systemd/system; do - if [ -e $p/$service_base_file ]; then - service_file=$p/$service_base_file - service_file=${service_file##$ROOT} - fi - done - if [ -z "$service_file" ]; then - echo "'$service_base_file' couldn't be found; exiting with error" - exit 1 - fi - echo "Found $service in $service_file" - - # If any new unit types are added to systemd they should be added - # to this regular expression. - unit_types_re='\.\(service\|socket\|device\|mount\|automount\|swap\|target\|target\.wants\|path\|timer\|snapshot\)\s*$' - if [ "$action" = "preset" ]; then - action=`egrep -sh $service $ROOT/etc/systemd/user-preset/*.preset | cut -f1 -d' '` - if [ -z "$action" ]; then - globalpreset=`egrep -sh '\*' $ROOT/etc/systemd/user-preset/*.preset | cut -f1 -d' '` - if [ -n "$globalpreset" ]; then - action="$globalpreset" - else - action="enable" - fi - fi - fi - # create the required symbolic links - wanted_by=$(sed '/^WantedBy[[:space:]]*=/s,[^=]*=,,p;d' "$ROOT/$service_file" \ - | tr ',' '\n' \ - | grep "$unit_types_re") - - required_by=$(sed '/^RequiredBy[[:space:]]*=/s,[^=]*=,,p;d' "$ROOT/$service_file" \ - | tr ',' '\n' \ - | grep "$unit_types_re") - - for dependency in WantedBy RequiredBy; do - if [ "$dependency" = "WantedBy" ]; then - suffix="wants" - dependency_list="$wanted_by" - elif [ "$dependency" = "RequiredBy" ]; then - suffix="requires" - dependency_list="$required_by" - fi - for r in $dependency_list; do - echo "$dependency=$r found in $service" - if [ -n "$instance_specified" ]; then - # substitute wildcards in the dependency - r=`echo $r | sed "s/%i/$instance_specified/g"` - fi - - if [ "$action" = "enable" ]; then - enable_service=$service - if [ "$service_template" = true -a -z "$instance_specified" ]; then - default_instance=$(sed '/^DefaultInstance[[:space:]]*=/s,[^=]*=,,p;d' "$ROOT/$service_file") - if [ -z $default_instance ]; then - echo "Template unit without instance or DefaultInstance directive, nothing to enable" - continue - else - echo "Found DefaultInstance $default_instance, enabling it" - enable_service=$(echo $service | sed "s/@/@$(echo $default_instance | sed 's/\\/\\\\/g')/") - fi - fi - mkdir -p $ROOT/etc/systemd/system/$r.$suffix - ln -s $service_file $ROOT/etc/systemd/system/$r.$suffix/$enable_service - echo "Enabled $enable_service for $r." - else - if [ "$service_template" = true -a -z "$instance_specified" ]; then - disable_service="$ROOT/etc/systemd/system/$r.$suffix/`echo $service | sed 's/@/@*/'`" - else - disable_service="$ROOT/etc/systemd/system/$r.$suffix/$service" - fi - rm -f $disable_service - [ -d $ROOT/etc/systemd/system/$r.$suffix ] && rmdir --ignore-fail-on-non-empty -p $ROOT/etc/systemd/system/$r.$suffix - echo "Disabled ${disable_service##$ROOT/etc/systemd/system/$r.$suffix/} for $r." - fi - done - done - - # create the required symbolic 'Alias' links - alias=$(sed '/^Alias[[:space:]]*=/s,[^=]*=,,p;d' "$ROOT/$service_file" \ - | tr ',' '\n' \ - | grep "$unit_types_re") - - for r in $alias; do - if [ "$action" = "enable" ]; then - mkdir -p $ROOT/etc/systemd/system - ln -s $service_file $ROOT/etc/systemd/system/$r - echo "Enabled $service for $alias." - else - rm -f $ROOT/etc/systemd/system/$r - echo "Disabled $service for $alias." - fi - done - - # call us for the other required scripts - also=$(sed '/^Also[[:space:]]*=/s,[^=]*=,,p;d' "$ROOT/$service_file" \ - | tr ',' '\n') - for a in $also; do - echo "Also=$a found in $service" - if [ "$action" = "enable" ]; then - $0 --root=$ROOT enable $a - fi - done -done +#!/usr/bin/env python3 +"""systemctl: subset of systemctl used for image construction + +Mask/preset systemd units +""" + +import argparse +import fnmatch +import os +import re +import sys + +from collections import namedtuple +from pathlib import Path + +version = 1.0 + +ROOT = Path("/") +SYSCONFDIR = Path("etc") +BASE_LIBDIR = Path("lib") +LIBDIR = Path("usr", "lib") + + +class SystemdFile(): + """Class representing a single systemd configuration file""" + def __init__(self, root, path): + self.sections = dict() + self._parse(root, path) + + def _parse(self, root, path): + """Parse a systemd syntax configuration file + + Args: + path: A pathlib.Path object pointing to the file + + """ + skip_re = re.compile(r"^\s*([#;]|$)") + section_re = re.compile(r"^\s*\[(?P<section>.*)\]") + kv_re = re.compile(r"^\s*(?P<key>[^\s]+)\s*=\s*(?P<value>.*)") + section = None + + if path.is_symlink(): + try: + path.resolve() + except FileNotFoundError: + # broken symlink, try relative to root + path = root / Path(os.readlink(str(path))).relative_to(ROOT) + + with path.open() as f: + for line in f: + if skip_re.match(line): + continue + + line = line.rstrip("\n") + m = section_re.match(line) + if m: + section = dict() + self.sections[m.group('section')] = section + continue + + while line.endswith("\\"): + line += f.readline().rstrip("\n") + + m = kv_re.match(line) + k = m.group('key') + v = m.group('value') + if k not in section: + section[k] = list() + section[k].extend(v.split()) + + def get(self, section, prop): + """Get a property from section + + Args: + section: Section to retrieve property from + prop: Property to retrieve + + Returns: + List representing all properties of type prop in section. + + Raises: + KeyError: if ``section`` or ``prop`` not found + """ + return self.sections[section][prop] + + +class Presets(): + """Class representing all systemd presets""" + def __init__(self, scope, root): + self.directives = list() + self._collect_presets(scope, root) + + def _parse_presets(self, presets): + """Parse presets out of a set of preset files""" + skip_re = re.compile(r"^\s*([#;]|$)") + directive_re = re.compile(r"^\s*(?P<action>enable|disable)\s+(?P<unit_name>(.+))") + + Directive = namedtuple("Directive", "action unit_name") + for preset in presets: + with preset.open() as f: + for line in f: + m = directive_re.match(line) + if m: + directive = Directive(action=m.group('action'), + unit_name=m.group('unit_name')) + self.directives.append(directive) + elif skip_re.match(line): + pass + else: + sys.exit("Unparsed preset line in {}".format(preset)) + + def _collect_presets(self, scope, root): + """Collect list of preset files""" + locations = [SYSCONFDIR / "systemd"] + # Handle the usrmerge case by ignoring /lib when it's a symlink + if not BASE_LIBDIR.is_symlink(): + locations.append(BASE_LIBDIR / "systemd") + locations.append(LIBDIR / "systemd") + + presets = dict() + for location in locations: + paths = (root / location / scope).glob("*.preset") + for path in paths: + # earlier names override later ones + if path.name not in presets: + presets[path.name] = path + + self._parse_presets([v for k, v in sorted(presets.items())]) + + def state(self, unit_name): + """Return state of preset for unit_name + + Args: + presets: set of presets + unit_name: name of the unit + + Returns: + None: no matching preset + `enable`: unit_name is enabled + `disable`: unit_name is disabled + """ + for directive in self.directives: + if fnmatch.fnmatch(unit_name, directive.unit_name): + return directive.action + + return None + + +def collect_services(root): + """Collect list of service files""" + locations = [SYSCONFDIR / "systemd"] + # Handle the usrmerge case by ignoring /lib when it's a symlink + if not BASE_LIBDIR.is_symlink(): + locations.append(BASE_LIBDIR / "systemd") + locations.append(LIBDIR / "systemd") + + services = dict() + for location in locations: + paths = (root / location / "system").glob("*") + for path in paths: + if path.is_dir(): + continue + # implement earlier names override later ones + if path.name not in services: + services[path.name] = path + + return services + + +def add_link(path, target): + try: + path.parent.mkdir(parents=True) + except FileExistsError: + pass + if not path.is_symlink(): + print("ln -s {} {}".format(target, path)) + path.symlink_to(target) + + +def process_deps(root, config, service, location, prop, dirstem): + systemdir = SYSCONFDIR / "systemd" / "system" + + target = ROOT / location.relative_to(root) + try: + for dependent in config.get('Install', prop): + wants = root / systemdir / "{}.{}".format(dependent, dirstem) / service + add_link(wants, target) + + except KeyError: + pass + + +def enable(root, service, location, services): + if location.is_symlink(): + # ignore aliases + return + + config = SystemdFile(root, location) + template = re.match(r"[^@]+@(?P<instance>[^\.]*)\.", service) + if template: + instance = template.group('instance') + if not instance: + try: + instance = config.get('Install', 'DefaultInstance')[0] + service = service.replace("@.", "@{}.".format(instance)) + except KeyError: + pass + if instance is None: + return + else: + instance = None + + process_deps(root, config, service, location, 'WantedBy', 'wants') + process_deps(root, config, service, location, 'RequiredBy', 'requires') + + try: + for also in config.get('Install', 'Also'): + enable(root, also, services[also], services) + + except KeyError: + pass + + systemdir = root / SYSCONFDIR / "systemd" / "system" + target = ROOT / location.relative_to(root) + try: + for dest in config.get('Install', 'Alias'): + alias = systemdir / dest + add_link(alias, target) + + except KeyError: + pass + + +def preset_all(root): + presets = Presets('system-preset', root) + services = collect_services(root) + + for service, location in services.items(): + state = presets.state(service) + + if state == "enable" or state is None: + enable(root, service, location, services) + + +def mask(root, *services): + systemdir = root / SYSCONFDIR / "systemd" / "system" + for service in services: + add_link(systemdir / service, "/dev/null") + + +def main(): + if sys.version_info < (3, 4, 0): + sys.exit("Python 3.4 or greater is required") + + parser = argparse.ArgumentParser() + parser.add_argument('command', nargs=1, choices=['mask', 'preset-all']) + parser.add_argument('service', nargs=argparse.REMAINDER) + parser.add_argument('--root') + parser.add_argument('--preset-mode', + choices=['full', 'enable-only', 'disable-only'], + default='full') + + args = parser.parse_args() + + root = Path(args.root) if args.root else ROOT + command = args.command[0] + if command == "mask": + mask(root, *args.service) + elif command == "preset-all": + if len(args.service) != 0: + sys.exit("Too many arguments.") + if args.preset_mode != "enable-only": + sys.exit("Only enable-only is supported as preset-mode.") + preset_all(root) + else: + raise RuntimeError() + + +if __name__ == '__main__': + main()
Signature · 1 line
2.17.1
PATCH v6 6/6 image: call systemctl preset-all for images
In reply to Alex KiernanFrom: Alex Kiernan <[email protected]>
Rather than rely on systemd's default invocation of preset-all at runtime, we pre-populate the symlink tree as part of of the image. This is done late so any overrides of presets during rootfs construction should already have happened.
Whilst we don't strictly need this for the read-write root case, it avoids boot time churn; for read-only root we have to do it here.
---
Changes in v6:
- moved systemctl preset-all to IMAGE_PREPROCESS so it runs after ROOTFS, run for all images, not just read-only
Changes in v5: None
meta/classes/image.bbclass | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-)
meta/classes/image.bbclass +8 -1
@@ -664,6 +664,13 @@ reproducible_final_image_task () { find ${IMAGE_ROOTFS} -exec touch -h --date=@$REPRODUCIBLE_TIMESTAMP_ROOTFS {} \; fi } -IMAGE_PREPROCESS_COMMAND_append = " reproducible_final_image_task; " + +IMAGE_EXTRADEPENDS += "${@bb.utils.contains('DISTRO_FEATURES', 'systemd', 'systemd-systemctl-native', '', d)}" + +systemd_preset_all () { + systemctl --root="${IMAGE_ROOTFS}" --preset-mode=enable-only preset-all +} + +IMAGE_PREPROCESS_COMMAND_append = " ${@bb.utils.contains('DISTRO_FEATURES', 'systemd', 'systemd_preset_all;', '', d)} reproducible_final_image_task; " CVE_PRODUCT = ""
Signature · 1 line
2.17.1
Re: [PATCH v6 6/6] image: call systemctl preset-all for images
Hi Alex,
Quoted 32 lines · Alex Kiernan
From: Alex Kiernan <[email protected]>
Rather than rely on systemd's default invocation of preset-all at runtime, we pre-populate the symlink tree as part of of the image. This is done late so any overrides of presets during rootfs construction should already have happened.
Whilst we don't strictly need this for the read-write root case, it avoids boot time churn; for read-only root we have to do it here.
Signed-off-by Alex Kiernan <[email protected]>---
Changes in v6:
- moved systemctl preset-all toIMAGE_PREPROCESSso it runs after ROOTFS, run for all images, not just read-onlyChanges in v5: None
meta/classes/image.bbclass | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-)meta/classes/image.bbclass +0
-1@@ -664,6 +664,13 @@ reproducible_final_image_task () { find ${IMAGE_ROOTFS} -exec touch -h --date=@$REPRODUCIBLE_TIMESTAMP_ROOTFS {} \; fi } -IMAGE_PREPROCESS_COMMAND_append = " reproducible_final_image_task; "
meta/classes/image.bbclass reproducible_final_image_task () { · line 676
Quoted 10 lines
meta/classes/image.bbclass +8
-0+ +IMAGE_EXTRADEPENDS += "${@bb.utils.contains('DISTRO_FEATURES', 'systemd', 'systemd-systemctl-native', '', d)}" + +systemd_preset_all () { + systemctl --root="${IMAGE_ROOTFS}" --preset-mode=enable-only preset-all +} + +IMAGE_PREPROCESS_COMMAND_append = " ${@bb.utils.contains('DISTRO_FEATURES', 'systemd', 'systemd_preset_all;', '', d)} reproducible_final_image_task; " CVE_PRODUCT = ""
In the interest of being able to produce a "stateless" system, I'd prefer to see us trying to empty out /etc at buildtime. That will allows us to error out when things are being installed directly to /etc such that they won't be set up properly at runtime when the system is "stateless". Allow things that _can_ be set up at runtime to be so; the runtime cost is small and paid only at first boot for stateful systems.
Calling preset-all at buildtime should only be necessary for read-only rootfs; even there, long term, /etc should be writable as that's what systemd (and others) expect.
/Jonas
Quoted 70 lines · Alex Kiernan
This patch set is largely Jonas Bonn's to move towards a "stateless" configuration:
These patches make some modifications to systemd with the long-term goal of being able to run OE in systemd's "stateless" configuration. "Stateless" boils down to building an image with empty /etc and /var directories so that volatile (tmpfs) filesystems can be mounted there; this requires that the system subsequently be able to populate these directories dynamically, which systemd mostly takes care of if things are done right.
In these patches:
i) Don't include machine-id in writable images so that systemd can run
its first-boot machinery
ii) Move systemd configuration files out of /etc iii) Allow systemd to dynamically enable services and populate
/etc/systemd/systemvia the presets mechanismThere's a long way to go to get to a working "stateless" configuration. Getting to a "volatile" system (just empty /var) should be easier and I'll post patches moving things in that direction shortly.
However as a result of the systemd 242 upgrade, which includes
01d2041e41f4("meson: stop creating enablement symlinks in /etc during installation"), services such as systemd-networkd are no longer enabled in images.This patch set fixes this problem in addition to satisfying the goal of moving towards "stateless" configurations.
The issue with respect to image testing during CI was caused by systemd-time-wait-sync.service being enabled due to the lack of a default preset policy:
https://www.freedesktop.org/wiki/Software/systemd/Preset/#howto
Changes in v6:
- switch configuration to simple overrides in/usr/lib/systemd/*.conf.d
- make systemd RRECOMMENDS rather than RDEPENDS on systemd-conf
- don't exit in postinst as when that executes we're actually a concatenation of all fragments
- validateSYSTEMD_AUTO_ENABLEisenableordisable
- rewrite systemctl-native in Python
- moved systemctl preset-all toIMAGE_PREPROCESSso it runs after ROOTFS, run for all images, not just read-onlyChanges in v5:
- rebased for systemd 242
- install default preset distribution policy of "enable nothing"Alex Kiernan (3):
systemd-conf: simplify creation of machine-specific configuration
systemctl-native: Rewrite in Python supporting preset-all and mask
image: call systemctl preset-all for imagesJonas Bonn (3):
systemd: don't build firstboot by default
systemd: do not create machine-id
systemd: create preset files instead of installing in imagemeta/classes/image.bbclass | 9 +- meta/classes/rootfs-postcommands.bbclass | 6 + meta/classes/systemd.bbclass | 41 +- .../systemd/systemd-conf/journald.conf | 3 + .../systemd/systemd-conf/logind.conf | 2 + .../systemd/systemd-conf/system.conf | 2 + .../systemd/systemd-conf/system.conf-qemuall | 3 + meta/recipes-core/systemd/systemd-conf_242.bb | 61 +-- .../systemd/systemd-systemctl/systemctl | 476 ++++++++++--------
Quoted 9 lines
.../systemd/systemd/99-default.preset | 1 + meta/recipes-core/systemd/systemd_242.bb | 26 +- 11 files changed, 360 insertions(+), 270 deletions(-) create mode 100644 meta/recipes-core/systemd/systemd-conf/journald.conf create mode 100644 meta/recipes-core/systemd/systemd-conf/logind.conf create mode 100644 meta/recipes-core/systemd/systemd-conf/system.conf create mode 100644 meta/recipes-core/systemd/systemd-conf/system.conf-qemuall create mode 100644 meta/recipes-core/systemd/systemd/99-default.preset
Sigh...
this still has issues - if you boot with ro on the kernel command line and without an initramfs, then / is read-only when systemd starts and it basically refuses to do anything:
[ 7.222134] systemd[1]: No hostname configured. [ 7.227266] systemd[1]: Set hostname to <localhost>. [ 7.232622] systemd[1]: System cannot boot: Missing /etc/machine-id and /etc is mounted read-only. [ 7.241750] systemd[1]: Booting up is supported only when: [ 7.247362] systemd[1]: 1) /etc/machine-id exists and is populated. [ 7.253752] systemd[1]: 2) /etc/machine-id exists and is empty. [ 7.259757] systemd[1]: 3) /etc/machine-id is missing and /etc is writable.
Note this has nothing to do with read-only-rootfs, this is just a regular boot... that said a bunch of things that were broken now work, so it's progress!
I'm leaning towards having systemctl-native touch /etc/machine-id when it runs, unless you explicitly ask for stateless in DISTRO_FEATURES... patches to follow.
--
Alex Kiernan
Hi Alex,
Quoted 96 lines · Alex Kiernan
Quoted 73 lines · Alex Kiernan
This patch set is largely Jonas Bonn's to move towards a "stateless" configuration:
These patches make some modifications to systemd with the long-term goal of being able to run OE in systemd's "stateless" configuration. "Stateless" boils down to building an image with empty /etc and /var directories so that volatile (tmpfs) filesystems can be mounted there; this requires that the system subsequently be able to populate these directories dynamically, which systemd mostly takes care of if things are done right.
In these patches:
i) Don't include machine-id in writable images so that systemd can run
its first-boot machinery
ii) Move systemd configuration files out of /etc iii) Allow systemd to dynamically enable services and populate
/etc/systemd/systemvia the presets mechanismThere's a long way to go to get to a working "stateless" configuration. Getting to a "volatile" system (just empty /var) should be easier and I'll post patches moving things in that direction shortly.
However as a result of the systemd 242 upgrade, which includes
01d2041e41f4("meson: stop creating enablement symlinks in /etc during installation"), services such as systemd-networkd are no longer enabled in images.This patch set fixes this problem in addition to satisfying the goal of moving towards "stateless" configurations.
The issue with respect to image testing during CI was caused by systemd-time-wait-sync.service being enabled due to the lack of a default preset policy:
https://www.freedesktop.org/wiki/Software/systemd/Preset/#howto
Changes in v6:
- switch configuration to simple overrides in/usr/lib/systemd/*.conf.d
- make systemd RRECOMMENDS rather than RDEPENDS on systemd-conf
- don't exit in postinst as when that executes we're actually a concatenation of all fragments
- validateSYSTEMD_AUTO_ENABLEisenableordisable
- rewrite systemctl-native in Python
- moved systemctl preset-all toIMAGE_PREPROCESSso it runs after ROOTFS, run for all images, not just read-onlyChanges in v5:
- rebased for systemd 242
- install default preset distribution policy of "enable nothing"Alex Kiernan (3):
systemd-conf: simplify creation of machine-specific configuration
systemctl-native: Rewrite in Python supporting preset-all and mask
image: call systemctl preset-all for imagesJonas Bonn (3):
systemd: don't build firstboot by default
systemd: do not create machine-id
systemd: create preset files instead of installing in imagemeta/classes/image.bbclass | 9 +- meta/classes/rootfs-postcommands.bbclass | 6 + meta/classes/systemd.bbclass | 41 +- .../systemd/systemd-conf/journald.conf | 3 + .../systemd/systemd-conf/logind.conf | 2 + .../systemd/systemd-conf/system.conf | 2 + .../systemd/systemd-conf/system.conf-qemuall | 3 + meta/recipes-core/systemd/systemd-conf_242.bb | 61 +-- .../systemd/systemd-systemctl/systemctl | 476 ++++++++++-------- .../systemd/systemd/99-default.preset | 1 + meta/recipes-core/systemd/systemd_242.bb | 26 +- 11 files changed, 360 insertions(+), 270 deletions(-)Quoted 6 lines
create mode 100644
meta/recipes-core/systemd/systemd-conf/journald.confcreate mode 100644meta/recipes-core/systemd/systemd-conf/logind.confcreate mode 100644meta/recipes-core/systemd/systemd-conf/system.confcreate mode 100644 meta/recipes-core/systemd/systemd-conf/system.conf-qemuall create mode 100644 meta/recipes-core/systemd/systemd/99-default.presetSigh...
this still has issues - if you boot with
roon the kernel command line and without an initramfs, then / is read-only when systemd starts and it basically refuses to do anything:[ 7.222134] systemd[1]: No hostname configured. [ 7.227266] systemd[1]: Set hostname to <localhost>. [ 7.232622] systemd[1]: System cannot boot: Missing /etc/machine-id and /etc is mounted read-only. [ 7.241750] systemd[1]: Booting up is supported only when: [ 7.247362] systemd[1]: 1) /etc/machine-id exists and is populated. [ 7.253752] systemd[1]: 2) /etc/machine-id exists and is empty. [ 7.259757] systemd[1]: 3) /etc/machine-id is missing and /etc is writable.
Quoted 7 lines
Note this has nothing to do with read-only-rootfs, this is just a regular boot... that said a bunch of things that were broken now work, so it's progress!
I'm leaning towards having systemctl-native touch
/etc/machine-idwhen it runs, unless you explicitly ask for stateless inDISTRO_FEATURES... patches to follow.
The paradigm that systemd follows is that /etc is _always_ writable. The read-only rootfs that OE produces with a read-only /etc is pathological from systemd's point of view (as is your 'ro' kernel parameter example). The way to handle this is to:
i) mount a tmpfs over /etc
ii) move the contents of /etc to /usr/share/factory/etc at buildtime and have tmpfiles.d snippets that take care of populating /etc at runtime
I have a set of patches that does this, but it's all pretty fragile at this point in time. It scans /etc at buildtime, moves files and links to factory/{var,etc}, sets up tmpfiles.d snippets for files and directories. Doing this, however, somewhat requires for things that _can_ be set up at runtime to be so; what remains in /etc should only be stuff that doesn't set itself up at runtime _yet_. For this reason, I'm inclined to say that you _don't_ want to leave the /etc/systemd/system nor /etc/machine-id files in the image.
/Jonas
Quoted 4 lines
--
Alex Kiernan
Quoted 122 lines
-----Original Message-----
From: [email protected] <openembedded- [email protected]> On Behalf Of Jonas Bonn
Sent: den 3 maj 2019 14:55
To: Alex Kiernan <[email protected]>; OE-core <openembedded- [email protected]>
Subject: Re: [OE-core] [OE-Core][PATCH v6 0/6] systemd stateless configurationHi Alex,
Quoted 97 lines · Alex Kiernan
Quoted 75 lines · Alex Kiernan
This patch set is largely Jonas Bonn's to move towards a "stateless" configuration:
These patches make some modifications to systemd with the long- term goal of being able to run OE in systemd's "stateless" configuration. "Stateless" boils down to building an image with empty /etc and /var directories so that volatile (tmpfs) filesystems can be mounted there; this requires that the system subsequently be able to populate these directories dynamically, which systemd mostly takes care of if things are done right.
In these patches:
i) Don't include machine-id in writable images so that
systemd can run its first-boot machinery
ii) Move systemd configuration files out of /etc iii) Allow systemd to dynamically enable services and
populate/etc/systemd/systemvia the presets mechanismThere's a long way to go to get to a working "stateless" configuration. Getting to a "volatile" system (just empty /var) should be easier and I'll post patches moving things in that direction shortly.
However as a result of the systemd 242 upgrade, which includes
01d2041e41f4("meson: stop creating enablement symlinks in /etc during installation"), services such as systemd-networkd are no longer enabled in images.This patch set fixes this problem in addition to satisfying the goal of moving towards "stateless" configurations.
The issue with respect to image testing during CI was caused by systemd-time-wait-sync.service being enabled due to the lack of a default preset policy:
https://www.freedesktop.org/wiki/Software/systemd/Preset/#howto
Changes in v6:
- switch configuration to simple overrides in/usr/lib/systemd/*.conf.d
- make systemd RRECOMMENDS rather than RDEPENDS on systemd-conf
- don't exit in postinst as when that executes we're actually a concatenation of all fragments
- validateSYSTEMD_AUTO_ENABLEisenableordisable
- rewrite systemctl-native in Python
- moved systemctl preset-all toIMAGE_PREPROCESSso it runs after ROOTFS, run for all images, not just read-onlyChanges in v5:
- rebased for systemd 242
- install default preset distribution policy of "enable nothing"Alex Kiernan (3):
systemd-conf: simplify creation of machine-specific configuration
systemctl-native: Rewrite in Python supporting preset-all and mask
image: call systemctl preset-all for imagesJonas Bonn (3):
systemd: don't build firstboot by default
systemd: do not create machine-id
systemd: create preset files instead of installing in imagemeta/classes/image.bbclass | 9 +- meta/classes/rootfs-postcommands.bbclass | 6 + meta/classes/systemd.bbclass | 41 +- .../systemd/systemd-conf/journald.conf | 3 + .../systemd/systemd-conf/logind.conf | 2 + .../systemd/systemd-conf/system.conf | 2 + .../systemd/systemd-conf/system.conf-qemuall | 3 + meta/recipes-core/systemd/systemd-conf_242.bb | 61 +-- .../systemd/systemd-systemctl/systemctl | 476 ++++++++++-------- .../systemd/systemd/99-default.preset | 1 + meta/recipes-core/systemd/systemd_242.bb | 26 +- 11 files changed, 360 insertions(+), 270 deletions(-)Quoted 5 lines
create mode 100644
meta/recipes-core/systemd/systemd-conf/journald.confcreate mode 100644meta/recipes-core/systemd/systemd-conf/logind.confcreate mode 100644meta/recipes-core/systemd/systemd-conf/system.confcreate mode 100644 meta/recipes-core/systemd/systemd-conf/system.conf-qemuall create mode 100644 meta/recipes-core/systemd/systemd/99-default.presetSigh...
this still has issues - if you boot with
roon the kernel command line and without an initramfs, then / is read-only when systemd starts and it basically refuses to do anything:[ 7.222134] systemd[1]: No hostname configured. [ 7.227266] systemd[1]: Set hostname to <localhost>. [ 7.232622] systemd[1]: System cannot boot: Missing /etc/machine-id and /etc is mounted read-only. [ 7.241750] systemd[1]: Booting up is supported only when: [ 7.247362] systemd[1]: 1) /etc/machine-id exists and is populated. [ 7.253752] systemd[1]: 2) /etc/machine-id exists and is empty. [ 7.259757] systemd[1]: 3) /etc/machine-id is missing and /etc is writable.Quoted 7 lines
Note this has nothing to do with read-only-rootfs, this is just a regular boot... that said a bunch of things that were broken now work, so it's progress!
I'm leaning towards having systemctl-native touch
/etc/machine-idwhen it runs, unless you explicitly ask for stateless inDISTRO_FEATURES... patches to follow.The paradigm that systemd follows is that /etc is _always_ writable. The read-only rootfs that OE produces with a read-only /etc is pathological from systemd's point of view (as is your 'ro' kernel parameter example). The way to handle this is to:
Quoted 4 lines
i) mount a tmpfs over /etc
ii) move the contents of /etc to/usr/share/factory/etcat buildtime and
have tmpfiles.d snippets that take care of populating /etc at runtime
That is probably the way to handle it for a stateless system. However, even without stateless, having /etc being writable is almost a necessity when using systemd. In our case, we mount an overlayfs on top of /etc to make it writable (before systemd is started). As I do not see us switching to a stateless system any time soon, I believe it would be good to be able to differentiate between the tree variants for how /etc is used:
read-only, writable and stateless. Today the only DISTRO_FEATURE is read-only-rootfs, which is not really relevant for /etc.
Quoted 11 lines
I have a set of patches that does this, but it's all pretty fragile at this point in time. It scans /etc at buildtime, moves files and links to factory/{var,etc}, sets up tmpfiles.d snippets for files and directories. Doing this, however, somewhat requires for things that _can_ be set up at runtime to be so; what remains in /etc should only be
stuff that doesn't set itself up at runtime _yet_. For this reason, I'm
inclined to say that you _don't_ want to leave the/etc/systemd/systemnor/etc/machine-idfiles in the image.
Quoted 3 lines
/Jonas
Quoted 1 line
Alex Kiernan
//Peter
Quoted 117 lines · Jonas Bonn
Hi Alex,
Quoted 96 lines · Alex Kiernan
Quoted 73 lines · Alex Kiernan
This patch set is largely Jonas Bonn's to move towards a "stateless" configuration:
These patches make some modifications to systemd with the long-term goal of being able to run OE in systemd's "stateless" configuration. "Stateless" boils down to building an image with empty /etc and /var directories so that volatile (tmpfs) filesystems can be mounted there; this requires that the system subsequently be able to populate these directories dynamically, which systemd mostly takes care of if things are done right.
In these patches:
i) Don't include machine-id in writable images so that systemd can run
its first-boot machinery
ii) Move systemd configuration files out of /etc iii) Allow systemd to dynamically enable services and populate
/etc/systemd/systemvia the presets mechanismThere's a long way to go to get to a working "stateless" configuration. Getting to a "volatile" system (just empty /var) should be easier and I'll post patches moving things in that direction shortly.
However as a result of the systemd 242 upgrade, which includes
01d2041e41f4("meson: stop creating enablement symlinks in /etc during installation"), services such as systemd-networkd are no longer enabled in images.This patch set fixes this problem in addition to satisfying the goal of moving towards "stateless" configurations.
The issue with respect to image testing during CI was caused by systemd-time-wait-sync.service being enabled due to the lack of a default preset policy:
https://www.freedesktop.org/wiki/Software/systemd/Preset/#howto
Changes in v6:
- switch configuration to simple overrides in/usr/lib/systemd/*.conf.d
- make systemd RRECOMMENDS rather than RDEPENDS on systemd-conf
- don't exit in postinst as when that executes we're actually a concatenation of all fragments
- validateSYSTEMD_AUTO_ENABLEisenableordisable
- rewrite systemctl-native in Python
- moved systemctl preset-all toIMAGE_PREPROCESSso it runs after ROOTFS, run for all images, not just read-onlyChanges in v5:
- rebased for systemd 242
- install default preset distribution policy of "enable nothing"Alex Kiernan (3):
systemd-conf: simplify creation of machine-specific configuration
systemctl-native: Rewrite in Python supporting preset-all and mask
image: call systemctl preset-all for imagesJonas Bonn (3):
systemd: don't build firstboot by default
systemd: do not create machine-id
systemd: create preset files instead of installing in imagemeta/classes/image.bbclass | 9 +- meta/classes/rootfs-postcommands.bbclass | 6 + meta/classes/systemd.bbclass | 41 +- .../systemd/systemd-conf/journald.conf | 3 + .../systemd/systemd-conf/logind.conf | 2 + .../systemd/systemd-conf/system.conf | 2 + .../systemd/systemd-conf/system.conf-qemuall | 3 + meta/recipes-core/systemd/systemd-conf_242.bb | 61 +-- .../systemd/systemd-systemctl/systemctl | 476 ++++++++++-------- .../systemd/systemd/99-default.preset | 1 + meta/recipes-core/systemd/systemd_242.bb | 26 +- 11 files changed, 360 insertions(+), 270 deletions(-)Quoted 6 lines
create mode 100644
meta/recipes-core/systemd/systemd-conf/journald.confcreate mode 100644meta/recipes-core/systemd/systemd-conf/logind.confcreate mode 100644meta/recipes-core/systemd/systemd-conf/system.confcreate mode 100644 meta/recipes-core/systemd/systemd-conf/system.conf-qemuall create mode 100644 meta/recipes-core/systemd/systemd/99-default.presetSigh...
this still has issues - if you boot with
roon the kernel command line and without an initramfs, then / is read-only when systemd starts and it basically refuses to do anything:[ 7.222134] systemd[1]: No hostname configured. [ 7.227266] systemd[1]: Set hostname to <localhost>. [ 7.232622] systemd[1]: System cannot boot: Missing /etc/machine-id and /etc is mounted read-only. [ 7.241750] systemd[1]: Booting up is supported only when: [ 7.247362] systemd[1]: 1) /etc/machine-id exists and is populated. [ 7.253752] systemd[1]: 2) /etc/machine-id exists and is empty. [ 7.259757] systemd[1]: 3) /etc/machine-id is missing and /etc is writable.Quoted 7 lines
Note this has nothing to do with read-only-rootfs, this is just a regular boot... that said a bunch of things that were broken now work, so it's progress!
I'm leaning towards having systemctl-native touch
/etc/machine-idwhen it runs, unless you explicitly ask for stateless inDISTRO_FEATURES... patches to follow.The paradigm that systemd follows is that /etc is _always_ writable. The read-only rootfs that OE produces with a read-only /etc is pathological from systemd's point of view (as is your 'ro' kernel parameter example). The way to handle this is to:
i) mount a tmpfs over /etc
ii) move the contents of /etc to/usr/share/factory/etcat buildtime and have tmpfiles.d snippets that take care of populating /etc at runtime
Quoted 9 lines
I have a set of patches that does this, but it's all pretty fragile at this point in time. It scans /etc at buildtime, moves files and links to factory/{var,etc}, sets up tmpfiles.d snippets for files and directories. Doing this, however, somewhat requires for things that _can_ be set up at runtime to be so; what remains in /etc should only be stuff that doesn't set itself up at runtime _yet_. For this reason, I'm inclined to say that you _don't_ want to leave the
/etc/systemd/systemnor/etc/machine-idfiles in the image.
I think the problem we're going to bump into is that ro getting passed in on the kernel command line I get from U-Boot and whilst OSTree gives me enough levers to make that per-boot, I know in our legacy world (and so lots of other peoples worlds), doing that kind of change isn't a goer... so whatever we do I think we're going to need lots of options. Obviously I'm relying on systemd doing a rw remount of / early in the boot.
I think setting /etc/machine-id blank and populating the wants/requires (and all the aliases/also stuff...) or doing none of it are valid options, but what I did with half of it is wrong :|
About to send out a patch set which follow's Chen's suggestion way back of adding a 'stateless' DISTRO_FEATURE which controls this and gets you all or nothing, which I think should fix anyone who's currently broken, but at the same time give us a lever which we can play with to get an actual stateless build. Persistent mutable /etc (in fact mutable anything) is the bane of my life, so I'm interested in making it work... unfortunately there's still quite a lot of gaps!
--
Alex Kiernan