Skip to contentSkip to conversation

systemd stateless configuration

PATCH In review Alex Kiernan 4 participants 15 messages May 2, 2019 – May 10, 2019

lore
Share & tools
lore
#0 lore raw

v6 cover systemd stateless configuration

This patch set is largely Jonas Bonn's to move towards a "stateless" configuration:

These patches make some modifications to systemd with the long-term goal of being able to run OE in systemd's "stateless" configuration. "Stateless" boils down to building an image with empty /etc and /var directories so that volatile (tmpfs) filesystems can be mounted there; this requires that the system subsequently be able to populate these directories dynamically, which systemd mostly takes care of if things are done right.

In these patches:
i) Don't include machine-id in writable images so that systemd can run
its first-boot machinery
ii) Move systemd configuration files out of /etc iii) Allow systemd to dynamically enable services and populate
/etc/systemd/system via the presets mechanism

There's a long way to go to get to a working "stateless" configuration. Getting to a "volatile" system (just empty /var) should be easier and I'll post patches moving things in that direction shortly.

However as a result of the systemd 242 upgrade, which includes 01d2041e41f4 ("meson: stop creating enablement symlinks in /etc during installation"), services such as systemd-networkd are no longer enabled in images.

This patch set fixes this problem in addition to satisfying the goal of moving towards "stateless" configurations.

The issue with respect to image testing during CI was caused by systemd-time-wait-sync.service being enabled due to the lack of a default preset policy:

https://www.freedesktop.org/wiki/Software/systemd/Preset/#howto

Changes in v6:
- switch configuration to simple overrides in /usr/lib/systemd/*.conf.d
- make systemd RRECOMMENDS rather than RDEPENDS on systemd-conf
- don't exit in postinst as when that executes we're actually a concatenation of all fragments
- validate SYSTEMD_AUTO_ENABLE is enable or disable
- rewrite systemctl-native in Python
- moved systemctl preset-all to IMAGE_PREPROCESS so it runs after ROOTFS, run for all images, not just read-only

Changes in v5:
- rebased for systemd 242
- install default preset distribution policy of "enable nothing"

Alex Kiernan (3):
systemd-conf: simplify creation of machine-specific configuration
systemctl-native: Rewrite in Python supporting preset-all and mask
image: call systemctl preset-all for images

Jonas Bonn (3):
systemd: don't build firstboot by default
systemd: do not create machine-id
systemd: create preset files instead of installing in image

 meta/classes/image.bbclass                    |   9 +-
 meta/classes/rootfs-postcommands.bbclass      |   6 +
 meta/classes/systemd.bbclass                  |  41 +-
 .../systemd/systemd-conf/journald.conf        |   3 +
 .../systemd/systemd-conf/logind.conf          |   2 +
 .../systemd/systemd-conf/system.conf          |   2 +
 .../systemd/systemd-conf/system.conf-qemuall  |   3 +
 meta/recipes-core/systemd/systemd-conf_242.bb |  61 +--
 .../systemd/systemd-systemctl/systemctl       | 476 ++++++++++--------
 .../systemd/systemd/99-default.preset         |   1 +
 meta/recipes-core/systemd/systemd_242.bb      |  26 +-
 11 files changed, 360 insertions(+), 270 deletions(-)
 create mode 100644 meta/recipes-core/systemd/systemd-conf/journald.conf
 create mode 100644 meta/recipes-core/systemd/systemd-conf/logind.conf
 create mode 100644 meta/recipes-core/systemd/systemd-conf/system.conf
 create mode 100644 meta/recipes-core/systemd/systemd-conf/system.conf-qemuall
 create mode 100644 meta/recipes-core/systemd/systemd/99-default.preset
Signature · 1 line
2.17.1
#1 lore raw

PATCH v6 1/6 systemd: don't build firstboot by default

From: Jonas Bonn <[email protected]>

The firstboot service prompts the user for information about the host at first boot. Systemd determines whether or not a boot is a "first boot" by the existence of the file /etc/machine-id. Since oe-core always includes this file (it is part of the systemd package), the firstboot service never runs so this service is being built but never run.

A follow-up patch to this one will remove the machine-id from the systemd build and allow it to be created automatically by systemd at "first boot". With that patch, we don't want the firstboot service to suddenly start being invoked and presenting a prompt to the user.

With this patch, the firstboot service becomes a PACKAGECONFIG option that the user must actively select.

---

Changes in v6: None
Changes in v5:
- rebased for systemd 242

 meta/recipes-core/systemd/systemd_242.bb | 1 -
 1 file changed, 1 deletion(-)
meta/recipes-core/systemd/systemd_242.bb +0 -1
index 33d3e0b000d9..9f194f2b6ca1 100644
@@ -66,7 +66,6 @@ PACKAGECONFIG ??= " \
     acl \
     backlight \
     binfmt \
-    firstboot \
     gshadow \
     hibernate \
     hostnamed \
Signature · 1 line
2.17.1
#2 lore raw

PATCH v6 2/6 systemd: do not create machine-id

In reply to Alex Kiernan

From: Jonas Bonn <[email protected]>

There is no reason to have an emtpy machine-id as part of the systemd package. Either:

i) the filesystem is writable and the file will be created automatically; or
ii) the filesystem is read-only, in which case the empty machine-id file should be created as part of the read-only-rootfs tweaks.

---

Changes in v6: None
Changes in v5:
- install default preset distribution policy of "enable nothing"

 meta/classes/rootfs-postcommands.bbclass            | 6 ++++++
 meta/recipes-core/systemd/systemd-conf_242.bb       | 9 ++-------
 meta/recipes-core/systemd/systemd/99-default.preset | 1 +
 meta/recipes-core/systemd/systemd_242.bb            | 6 +++++-
 4 files changed, 14 insertions(+), 8 deletions(-)
 create mode 100644 meta/recipes-core/systemd/systemd/99-default.preset
meta/classes/rootfs-postcommands.bbclass +6 -0
index bde58ad6cd36..89f8efd32363 100644
@@ -126,6 +126,12 @@ read_only_rootfs_hook () {
 			${IMAGE_ROOTFS}/etc/init.d/populate-volatile.sh
 		fi
 	fi
+
+	if ${@bb.utils.contains("DISTRO_FEATURES", "systemd", "true", "false", d)}; then
+	# Create machine-id
+	# 20:12 < mezcalero> koen: you have three options: a) run systemd-machine-id-setup at install time, b) have / read-only and an empty file there (for stateless) and c) boot with / writable
+		touch ${IMAGE_ROOTFS}${sysconfdir}/machine-id
+	fi
 }
 
 #
meta/recipes-core/systemd/systemd-conf_242.bb +2 -7
index 9bb27fd96d9c..7fe2e1105bef 100644
@@ -7,14 +7,13 @@ DefaultTimeoutStartSec setting."
 
 PACKAGE_ARCH = "${MACHINE_ARCH}"
 
-CONFFILES_${PN} = "${sysconfdir}/machine-id \
-${sysconfdir}/systemd/coredump.conf \
+CONFFILES_${PN} = "${sysconfdir}/systemd/coredump.conf \
 ${sysconfdir}/systemd/journald.conf \
 ${sysconfdir}/systemd/logind.conf \
 ${sysconfdir}/systemd/system.conf \
 ${sysconfdir}/systemd/user.conf"
 
-FILES_${PN} = "${sysconfdir}/machine-id ${sysconfdir}/systemd"
+FILES_${PN} = "${sysconfdir}/systemd"
 
 do_configure[noexec] = '1'
 do_compile[noexec] = '1'
@@ -23,10 +22,6 @@ do_install() {
 	rm -rf ${D}/${sysconfdir}/systemd
 	install -d ${D}/${sysconfdir}/systemd
 
-	# Create machine-id
-	# 20:12 < mezcalero> koen: you have three options: a) run systemd-machine-id-setup at install time, b) have / read-only and an empty file there (for stateless) and c) boot with / writable
-	touch ${D}${sysconfdir}/machine-id
-
 	install -m 0644 ${S}/src/coredump/coredump.conf ${D}${sysconfdir}/systemd/coredump.conf
 
 	install -m 0644 ${S}/src/journal/journald.conf ${D}${sysconfdir}/systemd/journald.conf
meta/recipes-core/systemd/systemd/99-default.preset new file +1 -0
new file mode 100644
index 000000000000..1f29b50597f0
@@ -0,0 +1 @@
+disable *
meta/recipes-core/systemd/systemd_242.bb +5 -1
index 9f194f2b6ca1..7d1b0ec13a44 100644
@@ -22,6 +22,7 @@ SRC_URI += "file://touchscreen.rules \
            file://0003-implment-systemd-sysv-install-for-OE.patch \
            file://0004-rules-whitelist-hd-devices.patch \
            file://0005-rules-watch-metadata-changes-in-ide-devices.patch \
+           file://99-default.preset \
            "
 
 # patches needed by musl
@@ -277,7 +278,6 @@ do_install() {
 	fi
 
 	# conf files are handled by systemd-conf
-	rm -f ${D}${sysconfdir}/machine-id
 	rm -f ${D}${sysconfdir}/systemd/coredump.conf
 	rm -f ${D}${sysconfdir}/systemd/journald.conf
 	rm -f ${D}${sysconfdir}/systemd/logind.conf
@@ -287,6 +287,10 @@ do_install() {
 	# duplicate udevadm for postinst script
 	install -d ${D}${libexecdir}
 	ln ${D}${base_bindir}/udevadm ${D}${libexecdir}/${MLPREFIX}udevadm
+
+	# install default policy for presets
+	# https://www.freedesktop.org/wiki/Software/systemd/Preset/#howto
+	install -Dm 0644 ${WORKDIR}/99-default.preset ${D}${systemd_unitdir}/system-preset/99-default.preset
 }
 
 
Signature · 1 line
2.17.1
#3 lore raw

Re: [PATCH v6 2/6] systemd: do not create machine-id

Quoted 6 lines · Alex Kiernan

From: Jonas Bonn <[email protected]>

There is no reason to have an emtpy machine-id as part of the systemd package. Either:

Quoted 7 lines

i) the filesystem is writable and the file will be created automatically; or
ii) the filesystem is read-only, in which case the empty machine-id file should be created as part of the read-only-rootfs tweaks.

This patch now results in no "/etc/machine-id" being created which breaks systemd-networkd WiFi connections.

I now always see this error when trying to connect to WiFi with
wpa_supplicant: "DHCP4 CLIENT: Failed to set IAID+DUID"

Alistair

Quoted 108 lines

---

Changes in v6: None
Changes in v5:
- install default preset distribution policy of "enable nothing"

 meta/classes/rootfs-postcommands.bbclass            | 6 ++++++
 meta/recipes-core/systemd/systemd-conf_242.bb       | 9 ++-------
 meta/recipes-core/systemd/systemd/99-default.preset | 1 +
 meta/recipes-core/systemd/systemd_242.bb            | 6 +++++-
 4 files changed, 14 insertions(+), 8 deletions(-)
 create mode 100644 meta/recipes-core/systemd/systemd/99-default.preset
meta/classes/rootfs-postcommands.bbclass +6 -0
index bde58ad6cd36..89f8efd32363 100644
@@ -126,6 +126,12 @@ read_only_rootfs_hook () {
                        ${IMAGE_ROOTFS}/etc/init.d/populate-volatile.sh
                fi
        fi
+
+       if ${@bb.utils.contains("DISTRO_FEATURES", "systemd", "true", "false", d)}; then
+       # Create machine-id
+       # 20:12 < mezcalero> koen: you have three options: a) run systemd-machine-id-setup at install time, b) have / read-only and an empty file there (for stateless) and c) boot with / writable
+               touch ${IMAGE_ROOTFS}${sysconfdir}/machine-id
+       fi
 }
 
 #
meta/recipes-core/systemd/systemd-conf_242.bb +2 -7
index 9bb27fd96d9c..7fe2e1105bef 100644
@@ -7,14 +7,13 @@ DefaultTimeoutStartSec setting."
 
 PACKAGE_ARCH = "${MACHINE_ARCH}"
 
-CONFFILES_${PN} = "${sysconfdir}/machine-id \
-${sysconfdir}/systemd/coredump.conf \
+CONFFILES_${PN} = "${sysconfdir}/systemd/coredump.conf \
 ${sysconfdir}/systemd/journald.conf \
 ${sysconfdir}/systemd/logind.conf \
 ${sysconfdir}/systemd/system.conf \
 ${sysconfdir}/systemd/user.conf"
 
-FILES_${PN} = "${sysconfdir}/machine-id ${sysconfdir}/systemd"
+FILES_${PN} = "${sysconfdir}/systemd"
 
 do_configure[noexec] = '1'
 do_compile[noexec] = '1'
@@ -23,10 +22,6 @@ do_install() {
        rm -rf ${D}/${sysconfdir}/systemd
        install -d ${D}/${sysconfdir}/systemd
 
-       # Create machine-id
-       # 20:12 < mezcalero> koen: you have three options: a) run systemd-machine-id-setup at install time, b) have / read-only and an empty file there (for stateless) and c) boot with / writable
-       touch ${D}${sysconfdir}/machine-id
-
        install -m 0644 ${S}/src/coredump/coredump.conf ${D}${sysconfdir}/systemd/coredump.conf
 
        install -m 0644 ${S}/src/journal/journald.conf ${D}${sysconfdir}/systemd/journald.conf
meta/recipes-core/systemd/systemd/99-default.preset new file +1 -0
new file mode 100644
index 000000000000..1f29b50597f0
@@ -0,0 +1 @@
+disable *
meta/recipes-core/systemd/systemd_242.bb +5 -2
index 9f194f2b6ca1..7d1b0ec13a44 100644
@@ -22,6 +22,7 @@ SRC_URI += "file://touchscreen.rules \
            file://0003-implment-systemd-sysv-install-for-OE.patch \
            file://0004-rules-whitelist-hd-devices.patch \
            file://0005-rules-watch-metadata-changes-in-ide-devices.patch \
+           file://99-default.preset \
            "
 
 # patches needed by musl
@@ -277,7 +278,6 @@ do_install() {
        fi
 
        # conf files are handled by systemd-conf
-       rm -f ${D}${sysconfdir}/machine-id
        rm -f ${D}${sysconfdir}/systemd/coredump.conf
        rm -f ${D}${sysconfdir}/systemd/journald.conf
        rm -f ${D}${sysconfdir}/systemd/logind.conf
@@ -287,6 +287,10 @@ do_install() {
        # duplicate udevadm for postinst script
        install -d ${D}${libexecdir}
        ln ${D}${base_bindir}/udevadm ${D}${libexecdir}/${MLPREFIX}udevadm
+
+       # install default policy for presets
+       # https://www.freedesktop.org/wiki/Software/systemd/Preset/#howto
+       install -Dm 0644 ${WORKDIR}/99-default.preset ${D}${systemd_unitdir}/system-preset/99-default.preset
 }
 
 
--

2.17.1

--
_______________________________________________
Openembedded-core mailing list
[email protected]
http://lists.openembedded.org/mailman/listinfo/openembedded-core

#4 lore raw

Re: [PATCH v6 2/6] systemd: do not create machine-id

Quoted 13 lines · Alistair Francis
Quoted 11 lines · Alex Kiernan

From: Jonas Bonn <[email protected]>

There is no reason to have an emtpy machine-id as part of the systemd package. Either:

i) the filesystem is writable and the file will be created automatically; or
ii) the filesystem is read-only, in which case the empty machine-id file should be created as part of the read-only-rootfs tweaks.

Quoted 9 lines
Quoted 2 lines

This patch now results in no "/etc/machine-id" being created which breaks systemd-networkd WiFi connections.

I now always see this error when trying to connect to WiFi with
wpa_supplicant: "DHCP4 CLIENT: Failed to set IAID+DUID"

I posted this series earlier that fixes everything I know about (including the lack of /etc/machine-id in the stateful case):

https://patchwork.openembedded.org/series/17495/

If you can test and I'll have a look if there's anything else that needs addressing?

Signature · 1 line
Alex Kiernan
#5 lore raw

Re: [PATCH v6 2/6] systemd: do not create machine-id

Quoted 25 lines · Alex Kiernan
Quoted 13 lines · Alistair Francis
Quoted 11 lines · Alex Kiernan

From: Jonas Bonn <[email protected]>

There is no reason to have an emtpy machine-id as part of the systemd package. Either:

i) the filesystem is writable and the file will be created automatically; or
ii) the filesystem is read-only, in which case the empty machine-id file should be created as part of the read-only-rootfs tweaks.

Quoted 9 lines
Quoted 2 lines

This patch now results in no "/etc/machine-id" being created which breaks systemd-networkd WiFi connections.

I now always see this error when trying to connect to WiFi with
wpa_supplicant: "DHCP4 CLIENT: Failed to set IAID+DUID"

Quoted 7 lines

I posted this series earlier that fixes everything I know about (including the lack of /etc/machine-id in the stateful case):

https://patchwork.openembedded.org/series/17495/

If you can test and I'll have a look if there's anything else that needs addressing?

Sorry for the long delay, I didn't get a chance to test it until now. It's all fixed on master now with your patches.

Alistair

Quoted 3 lines

--
Alex Kiernan

#6 lore raw

PATCH v6 3/6 systemd-conf: simplify creation of machine-specific configuration

In reply to Alex Kiernan

From: Alex Kiernan <[email protected]>

The configuration files that systemd installs are just skeletons detailing the available options and their default values. The recommended means of changing the configuration is to provide snippets in configuration directories. For example, journald.conf settings are best set in /usr/lib/system.d/journald.conf.d/ and can be overridden by the user by providing overriding snippets in /etc/systemd/journald.conf.d/.

The systemd-conf package is just providing machine-specific overrides for some systemd defaults.

This patch restores the installation of config files by systemd and reduces systemd-conf to just providing the config snippets in /usr/lib/systemd/*.conf.d. This simpilfies the systemd-conf recipe considerably since it now just sets up a couple of text files and doesn't even need access to the systemd source anymore.

License-Update: configuration snippets licensing is independent of systemd licensing

---

Changes in v6:
- switch configuration to simple overrides in /usr/lib/systemd/*.conf.d
- make systemd RRECOMMENDS rather than RDEPENDS on systemd-conf

Changes in v5: None

 .../systemd/systemd-conf/journald.conf        |  3 +
 .../systemd/systemd-conf/logind.conf          |  2 +
 .../systemd/systemd-conf/system.conf          |  2 +
 .../systemd/systemd-conf/system.conf-qemuall  |  3 +
 meta/recipes-core/systemd/systemd-conf_242.bb | 56 +++++++------------
 meta/recipes-core/systemd/systemd_242.bb      | 19 ++++---
 6 files changed, 41 insertions(+), 44 deletions(-)
 create mode 100644 meta/recipes-core/systemd/systemd-conf/journald.conf
 create mode 100644 meta/recipes-core/systemd/systemd-conf/logind.conf
 create mode 100644 meta/recipes-core/systemd/systemd-conf/system.conf
 create mode 100644 meta/recipes-core/systemd/systemd-conf/system.conf-qemuall
meta/recipes-core/systemd/systemd-conf/journald.conf new file +3 -0
new file mode 100644
index 000000000000..fd4d969fa7c3
@@ -0,0 +1,3 @@
+[Journal]
+ForwardToSyslog=yes
+RuntimeMaxUse=64M
meta/recipes-core/systemd/systemd-conf/logind.conf new file +2 -0
new file mode 100644
index 000000000000..bf7f6920fa70
@@ -0,0 +1,2 @@
+[Login]
+KillUserProcesses=yes
meta/recipes-core/systemd/systemd-conf/system.conf new file +2 -0
new file mode 100644
index 000000000000..554cab26c244
@@ -0,0 +1,2 @@
+[Manager]
+DefaultMemoryAccounting=yes
meta/recipes-core/systemd/systemd-conf/system.conf-qemuall new file +3 -0
new file mode 100644
index 000000000000..741be27cb44c
@@ -0,0 +1,3 @@
+[Manager]
+# Change DefaultTimeoutStartSec from 90s to 240s
+DefaultTimeoutStartSec = 240s
meta/recipes-core/systemd/systemd-conf_242.bb +20 -36

68 lines not on this page · raw message

meta/recipes-core/systemd/systemd_242.bb +11 -8
index 7d1b0ec13a44..a6b39d57e8f9 100644
@@ -277,13 +277,6 @@ do_install() {
 		fi
 	fi
 
-	# conf files are handled by systemd-conf
-	rm -f ${D}${sysconfdir}/systemd/coredump.conf
-	rm -f ${D}${sysconfdir}/systemd/journald.conf
-	rm -f ${D}${sysconfdir}/systemd/logind.conf
-	rm -f ${D}${sysconfdir}/systemd/system.conf
-	rm -f ${D}${sysconfdir}/systemd/user.conf
-
 	# duplicate udevadm for postinst script
 	install -d ${D}${libexecdir}
 	ln ${D}${base_bindir}/udevadm ${D}${libexecdir}/${MLPREFIX}udevadm
@@ -486,6 +479,15 @@ FILES_${PN}-extra-utils = "\
                         ${rootlibexecdir}/systemd/systemd-cgroups-agent \
 "
 
+CONFFILES_${PN} = "${sysconfdir}/systemd/coredump.conf \
+	${sysconfdir}/systemd/journald.conf \
+	${sysconfdir}/systemd/logind.conf \
+	${sysconfdir}/systemd/system.conf \
+	${sysconfdir}/systemd/user.conf \
+	${sysconfdir}/systemd/resolved.conf \
+	${sysconfdir}/systemd/timesyncd.conf \
+"
+
 FILES_${PN} = " ${base_bindir}/* \
                 ${base_sbindir}/shutdown \
                 ${base_sbindir}/halt \
@@ -549,7 +551,7 @@ FILES_${PN}-dev += "${base_libdir}/security/*.la ${datadir}/dbus-1/interfaces/ $
 
 RDEPENDS_${PN} += "kmod dbus util-linux-mount util-linux-umount udev (= ${EXTENDPKGV}) util-linux-agetty util-linux-fsck"
 RDEPENDS_${PN} += "${@bb.utils.contains('PACKAGECONFIG', 'serial-getty-generator', '', 'systemd-serialgetty', d)}"
-RDEPENDS_${PN} += "volatile-binds update-rc.d systemd-conf"
+RDEPENDS_${PN} += "volatile-binds update-rc.d"
 RDEPENDS_${PN} += "${@bb.utils.contains('PACKAGECONFIG', 'myhostname', 'libnss-myhostname', '', d)}"
 
 RRECOMMENDS_${PN} += "systemd-extra-utils \
@@ -557,6 +559,7 @@ RRECOMMENDS_${PN} += "systemd-extra-utils \
                       e2fsprogs-e2fsck \
                       kernel-module-autofs4 kernel-module-unix kernel-module-ipv6 \
                       os-release \
+                      systemd-conf \
 "
 
 INSANE_SKIP_${PN} += "dev-so libdir"
Signature · 1 line
2.17.1
#7 lore raw

PATCH v6 4/6 systemd: create preset files instead of installing in image

In reply to Alex Kiernan

From: Jonas Bonn <[email protected]>

At first boot, systemd will create the /etc/systemd/system directory from service preset files. As such, for a normal, writable /etc (writable rootfs), there is no need to set up this directory at image creation time.

This patch changes the systemd machinery to create preset files and to rely on systemd to do the service enablement.

This breaks the read-only-rootfs case; there's a fix for this in a follow-up patch.

---

Changes in v6:
- don't exit in postinst as when that executes we're actually a concatenation of all fragments
- validate SYSTEMD_AUTO_ENABLE is enable or disable

Changes in v5: None

 meta/classes/systemd.bbclass | 41 +++++++++++++++++-------------------
 1 file changed, 19 insertions(+), 22 deletions(-)
meta/classes/systemd.bbclass +19 -22
index c8f4fdec88dc..3935685c18cb 100644
@@ -16,45 +16,30 @@ python __anonymous() {
     # from doing any work so that pure-systemd images don't have redundant init
     # files.
     if bb.utils.contains('DISTRO_FEATURES', 'systemd', True, False, d):
-        d.appendVar("DEPENDS", " systemd-systemctl-native")
-        d.appendVar("PACKAGE_WRITE_DEPS", " systemd-systemctl-native")
         if not bb.utils.contains('DISTRO_FEATURES', 'sysvinit', True, False, d):
             d.setVar("INHIBIT_UPDATERCD_BBCLASS", "1")
 }
 
 systemd_postinst() {
-OPTS=""
-
-if [ -n "$D" ]; then
-    OPTS="--root=$D"
-fi
-
 if type systemctl >/dev/null 2>/dev/null; then
 	if [ -z "$D" ]; then
 		systemctl daemon-reload
-	fi
-
-	systemctl $OPTS ${SYSTEMD_AUTO_ENABLE} ${SYSTEMD_SERVICE_ESCAPED}
+		systemctl preset "${SYSTEMD_SERVICE_ESCAPED}"
 
-	if [ -z "$D" -a "${SYSTEMD_AUTO_ENABLE}" = "enable" ]; then
-		systemctl --no-block restart ${SYSTEMD_SERVICE_ESCAPED}
+		if [ "${SYSTEMD_AUTO_ENABLE}" = "enable" ]; then
+			systemctl --no-block restart "${SYSTEMD_SERVICE_ESCAPED}"
+		fi
 	fi
 fi
 }
 
 systemd_prerm() {
-OPTS=""
-
-if [ -n "$D" ]; then
-    OPTS="--root=$D"
-fi
-
 if type systemctl >/dev/null 2>/dev/null; then
 	if [ -z "$D" ]; then
-		systemctl stop ${SYSTEMD_SERVICE_ESCAPED}
-	fi
+		systemctl stop "${SYSTEMD_SERVICE_ESCAPED}"
 
-	systemctl $OPTS disable ${SYSTEMD_SERVICE_ESCAPED}
+		systemctl disable "${SYSTEMD_SERVICE_ESCAPED}"
+	fi
 fi
 }
 
@@ -177,12 +162,24 @@ python systemd_populate_packages() {
                 else:
                     bb.fatal("SYSTEMD_SERVICE_%s value %s does not exist" % (pkg_systemd, service))
 
+    def systemd_create_presets(pkg):
+        action = get_package_var(d, 'SYSTEMD_AUTO_ENABLE', pkg)
+        if action not in ("enable", "disable"):
+            bb.fatal("SYSTEMD_AUTO_ENABLE_%s '%s' is not 'enable' or 'disable'" % (pkg, action))
+        presetf = oe.path.join(d.getVar("PKGD"), d.getVar("systemd_unitdir"), "system-preset/98-%s.preset" % pkg)
+        bb.utils.mkdirhier(os.path.dirname(presetf))
+        with open(presetf, 'a') as fd:
+            for service in d.getVar('SYSTEMD_SERVICE_%s' % pkg).split():
+                fd.write("%s %s\n" % (action,service))
+        d.appendVar("FILES_%s" % pkg, ' ' + oe.path.join(d.getVar("systemd_unitdir"), "system-preset/98-%s.preset" % pkg))
+
     # Run all modifications once when creating package
     if os.path.exists(d.getVar("D")):
         for pkg in d.getVar('SYSTEMD_PACKAGES').split():
             systemd_check_package(pkg)
             if d.getVar('SYSTEMD_SERVICE_' + pkg):
                 systemd_generate_package_scripts(pkg)
+                systemd_create_presets(pkg)
         systemd_check_services()
 }
 
Signature · 1 line
2.17.1
#8 lore raw

PATCH v6 5/6 systemctl-native: Rewrite in Python supporting preset-all and mask

In reply to Alex Kiernan

Rewrite systemctl-native in Python so that extending/testing it is easier.

Now that the systemd class sets up service presets instead of actively enabling services, the 'enable' and 'disable' subcommands for systemctl are not actually used anywhere. As such, we can remove these to make sure that nobody inadvertently introduces new uses of them.

This implementation covers preset-all and mask which are the only options used in the current code, but should be readily extensible to other commands.

We use preset-all at image construction time to populate the symlinks used by systemd.

---

Changes in v6:
- rewrite systemctl-native in Python

Changes in v5: None

 .../systemd/systemd-systemctl/systemctl       | 476 ++++++++++--------
 1 file changed, 280 insertions(+), 196 deletions(-)
meta/recipes-core/systemd/systemd-systemctl/systemctl +280 -196
index 2bc648961761..d7d4e0d29a09 100755
@@ -1,196 +1,280 @@
-#!/bin/sh
-echo "Started $0 $*"
-
-ROOT=
-
-# parse command line params
-action=
-while [ $# != 0 ]; do
-	opt="$1"
-
-	case "$opt" in
-		enable)
-			shift
-
-			action="$opt"
-			services="$1"
-			cmd_args="1"
-			shift
-			;;
-		disable)
-			shift
-
-			action="$opt"
-			services="$1"
-			cmd_args="1"
-			shift
-			;;
-		mask)
-			shift
-
-			action="$opt"
-			services="$1"
-			cmd_args="1"
-			shift
-			;;
-		preset)
-			shift
-
-			action="$opt"
-			services="$1"
-			cmd_args="1"
-			shift
-			;;
-		--root=*)
-			ROOT=${opt##--root=}
-			cmd_args="0"
-			shift
-			;;
-		*)
-			if [ "$cmd_args" = "1" ]; then
-				services="$services $opt" 
-				shift
-			else
-				echo "'$opt' is an unkown option; exiting with error"
-				exit 1
-			fi
-			;;
-	esac
-done
-if [ "$action" = "preset" -a "$service_file" = "" ]; then
-	services=$(for f in `find $ROOT/etc/systemd/system $ROOT/lib/systemd/system $ROOT/usr/lib/systemd/system -type f 2>1`; do basename $f; done)
-	services="$services $opt"
-	presetall=1
-fi
-
-for service in $services; do
-	if [ "$presetall" = "1" ]; then
-		action="preset"
-	fi
-	if [ "$action" = "mask" ]; then
-		if [ ! -d $ROOT/etc/systemd/system/ ]; then
-			mkdir -p $ROOT/etc/systemd/system/
-		fi
-		cmd="ln -s /dev/null $ROOT/etc/systemd/system/$service"
-		echo "$cmd"
-		$cmd
-		exit 0
-	fi
-
-	service_base_file=`echo $service | sed 's/\(@\).*\(\.[^.]\+\)/\1\2/'`
-	if [ -z `echo $service | sed '/@/p;d'` ]; then
-		echo "Try to find location of $service..."
-		service_template=false
-	else
-		echo "Try to find location of template $service_base_file of instance $service..."
-		service_template=true
-		instance_specified=`echo $service | sed 's/^.\+@\(.*\)\.[^.]\+/\1/'`
-	fi
-
-	# find service file
-	for p in $ROOT/etc/systemd/system \
-		 $ROOT/lib/systemd/system \
-		 $ROOT/usr/lib/systemd/system; do
-		if [ -e $p/$service_base_file ]; then
-			service_file=$p/$service_base_file
-			service_file=${service_file##$ROOT}
-		fi
-	done
-	if [ -z "$service_file" ]; then
-		echo "'$service_base_file' couldn't be found; exiting with error"
-		exit 1
-	fi
-	echo "Found $service in $service_file"
-
-	# If any new unit types are added to systemd they should be added
-	# to this regular expression.
-	unit_types_re='\.\(service\|socket\|device\|mount\|automount\|swap\|target\|target\.wants\|path\|timer\|snapshot\)\s*$'
-	if [ "$action" = "preset" ]; then
-		action=`egrep -sh  $service $ROOT/etc/systemd/user-preset/*.preset | cut -f1 -d' '`
-		if [ -z "$action" ]; then
-			globalpreset=`egrep -sh  '\*'  $ROOT/etc/systemd/user-preset/*.preset | cut -f1 -d' '`
-			if [ -n "$globalpreset" ]; then
-				action="$globalpreset"
-			else
-				action="enable"
-			fi
-		fi
-	fi
-	# create the required symbolic links
-	wanted_by=$(sed '/^WantedBy[[:space:]]*=/s,[^=]*=,,p;d' "$ROOT/$service_file" \
-		        | tr ',' '\n' \
-		        | grep "$unit_types_re")
-
-	required_by=$(sed '/^RequiredBy[[:space:]]*=/s,[^=]*=,,p;d' "$ROOT/$service_file" \
-		        | tr ',' '\n' \
-		        | grep "$unit_types_re")
-
-	for dependency in WantedBy RequiredBy; do
-		if [ "$dependency" = "WantedBy" ]; then
-			suffix="wants"
-			dependency_list="$wanted_by"
-		elif [ "$dependency" = "RequiredBy" ]; then
-			suffix="requires"
-			dependency_list="$required_by"
-		fi
-		for r in $dependency_list; do
-			echo "$dependency=$r found in $service"
-			if [ -n "$instance_specified" ]; then
-				# substitute wildcards in the dependency
-				r=`echo $r | sed "s/%i/$instance_specified/g"`
-			fi
-
-			if [ "$action" = "enable" ]; then
-				enable_service=$service
-				if [ "$service_template" = true -a -z "$instance_specified" ]; then
-					default_instance=$(sed '/^DefaultInstance[[:space:]]*=/s,[^=]*=,,p;d' "$ROOT/$service_file")
-					if [ -z $default_instance ]; then
-						echo "Template unit without instance or DefaultInstance directive, nothing to enable"
-						continue
-					else
-						echo "Found DefaultInstance $default_instance, enabling it"
-						enable_service=$(echo $service | sed "s/@/@$(echo $default_instance | sed 's/\\/\\\\/g')/")
-					fi
-				fi
-				mkdir -p $ROOT/etc/systemd/system/$r.$suffix
-				ln -s $service_file $ROOT/etc/systemd/system/$r.$suffix/$enable_service
-				echo "Enabled $enable_service for $r."
-			else
-				if [ "$service_template" = true -a -z "$instance_specified" ]; then
-					disable_service="$ROOT/etc/systemd/system/$r.$suffix/`echo $service | sed 's/@/@*/'`"
-				else
-					disable_service="$ROOT/etc/systemd/system/$r.$suffix/$service"
-				fi
-				rm -f $disable_service
-				[ -d $ROOT/etc/systemd/system/$r.$suffix ] && rmdir --ignore-fail-on-non-empty -p $ROOT/etc/systemd/system/$r.$suffix
-				echo "Disabled ${disable_service##$ROOT/etc/systemd/system/$r.$suffix/} for $r."
-			fi
-		done
-	done
-
-	# create the required symbolic 'Alias' links
-	alias=$(sed '/^Alias[[:space:]]*=/s,[^=]*=,,p;d' "$ROOT/$service_file" \
-		        | tr ',' '\n' \
-		        | grep "$unit_types_re")
-
-	for r in $alias; do
-		if [ "$action" = "enable" ]; then
-			mkdir -p $ROOT/etc/systemd/system
-			ln -s $service_file $ROOT/etc/systemd/system/$r
-			echo "Enabled $service for $alias."
-		else
-			rm -f $ROOT/etc/systemd/system/$r
-			echo "Disabled $service for $alias."
-		fi
-	done
-
-	# call us for the other required scripts
-	also=$(sed '/^Also[[:space:]]*=/s,[^=]*=,,p;d' "$ROOT/$service_file" \
-		   | tr ',' '\n')
-	for a in $also; do
-		echo "Also=$a found in $service"
-		if [ "$action" = "enable" ]; then
-			$0 --root=$ROOT enable $a
-		fi
-	done
-done
+#!/usr/bin/env python3
+"""systemctl: subset of systemctl used for image construction
+
+Mask/preset systemd units
+"""
+
+import argparse
+import fnmatch
+import os
+import re
+import sys
+
+from collections import namedtuple
+from pathlib import Path
+
+version = 1.0
+
+ROOT = Path("/")
+SYSCONFDIR = Path("etc")
+BASE_LIBDIR = Path("lib")
+LIBDIR = Path("usr", "lib")
+
+
+class SystemdFile():
+    """Class representing a single systemd configuration file"""
+    def __init__(self, root, path):
+        self.sections = dict()
+        self._parse(root, path)
+
+    def _parse(self, root, path):
+        """Parse a systemd syntax configuration file
+
+        Args:
+            path: A pathlib.Path object pointing to the file
+
+        """
+        skip_re = re.compile(r"^\s*([#;]|$)")
+        section_re = re.compile(r"^\s*\[(?P<section>.*)\]")
+        kv_re = re.compile(r"^\s*(?P<key>[^\s]+)\s*=\s*(?P<value>.*)")
+        section = None
+
+        if path.is_symlink():
+            try:
+                path.resolve()
+            except FileNotFoundError:
+                # broken symlink, try relative to root
+                path = root / Path(os.readlink(str(path))).relative_to(ROOT)
+
+        with path.open() as f:
+            for line in f:
+                if skip_re.match(line):
+                    continue
+
+                line = line.rstrip("\n")
+                m = section_re.match(line)
+                if m:
+                    section = dict()
+                    self.sections[m.group('section')] = section
+                    continue
+
+                while line.endswith("\\"):
+                    line += f.readline().rstrip("\n")
+
+                m = kv_re.match(line)
+                k = m.group('key')
+                v = m.group('value')
+                if k not in section:
+                    section[k] = list()
+                section[k].extend(v.split())
+
+    def get(self, section, prop):
+        """Get a property from section
+
+        Args:
+            section: Section to retrieve property from
+            prop: Property to retrieve
+
+        Returns:
+            List representing all properties of type prop in section.
+
+        Raises:
+            KeyError: if ``section`` or ``prop`` not found
+        """
+        return self.sections[section][prop]
+
+
+class Presets():
+    """Class representing all systemd presets"""
+    def __init__(self, scope, root):
+        self.directives = list()
+        self._collect_presets(scope, root)
+
+    def _parse_presets(self, presets):
+        """Parse presets out of a set of preset files"""
+        skip_re = re.compile(r"^\s*([#;]|$)")
+        directive_re = re.compile(r"^\s*(?P<action>enable|disable)\s+(?P<unit_name>(.+))")
+
+        Directive = namedtuple("Directive", "action unit_name")
+        for preset in presets:
+            with preset.open() as f:
+                for line in f:
+                    m = directive_re.match(line)
+                    if m:
+                        directive = Directive(action=m.group('action'),
+                                              unit_name=m.group('unit_name'))
+                        self.directives.append(directive)
+                    elif skip_re.match(line):
+                        pass
+                    else:
+                        sys.exit("Unparsed preset line in {}".format(preset))
+
+    def _collect_presets(self, scope, root):
+        """Collect list of preset files"""
+        locations = [SYSCONFDIR / "systemd"]
+        # Handle the usrmerge case by ignoring /lib when it's a symlink
+        if not BASE_LIBDIR.is_symlink():
+            locations.append(BASE_LIBDIR / "systemd")
+        locations.append(LIBDIR / "systemd")
+
+        presets = dict()
+        for location in locations:
+            paths = (root / location / scope).glob("*.preset")
+            for path in paths:
+                # earlier names override later ones
+                if path.name not in presets:
+                    presets[path.name] = path
+
+        self._parse_presets([v for k, v in sorted(presets.items())])
+
+    def state(self, unit_name):
+        """Return state of preset for unit_name
+
+        Args:
+            presets: set of presets
+            unit_name: name of the unit
+
+        Returns:
+            None: no matching preset
+            `enable`: unit_name is enabled
+            `disable`: unit_name is disabled
+        """
+        for directive in self.directives:
+            if fnmatch.fnmatch(unit_name, directive.unit_name):
+                return directive.action
+
+        return None
+
+
+def collect_services(root):
+    """Collect list of service files"""
+    locations = [SYSCONFDIR / "systemd"]
+    # Handle the usrmerge case by ignoring /lib when it's a symlink
+    if not BASE_LIBDIR.is_symlink():
+        locations.append(BASE_LIBDIR / "systemd")
+    locations.append(LIBDIR / "systemd")
+
+    services = dict()
+    for location in locations:
+        paths = (root / location / "system").glob("*")
+        for path in paths:
+            if path.is_dir():
+                continue
+            # implement earlier names override later ones
+            if path.name not in services:
+                services[path.name] = path
+
+    return services
+
+
+def add_link(path, target):
+    try:
+        path.parent.mkdir(parents=True)
+    except FileExistsError:
+        pass
+    if not path.is_symlink():
+        print("ln -s {} {}".format(target, path))
+        path.symlink_to(target)
+
+
+def process_deps(root, config, service, location, prop, dirstem):
+    systemdir = SYSCONFDIR / "systemd" / "system"
+
+    target = ROOT / location.relative_to(root)
+    try:
+        for dependent in config.get('Install', prop):
+            wants = root / systemdir / "{}.{}".format(dependent, dirstem) / service
+            add_link(wants, target)
+
+    except KeyError:
+        pass
+
+
+def enable(root, service, location, services):
+    if location.is_symlink():
+        # ignore aliases
+        return
+
+    config = SystemdFile(root, location)
+    template = re.match(r"[^@]+@(?P<instance>[^\.]*)\.", service)
+    if template:
+        instance = template.group('instance')
+        if not instance:
+            try:
+                instance = config.get('Install', 'DefaultInstance')[0]
+                service = service.replace("@.", "@{}.".format(instance))
+            except KeyError:
+                pass
+        if instance is None:
+            return
+    else:
+        instance = None
+
+    process_deps(root, config, service, location, 'WantedBy', 'wants')
+    process_deps(root, config, service, location, 'RequiredBy', 'requires')
+
+    try:
+        for also in config.get('Install', 'Also'):
+            enable(root, also, services[also], services)
+
+    except KeyError:
+        pass
+
+    systemdir = root / SYSCONFDIR / "systemd" / "system"
+    target = ROOT / location.relative_to(root)
+    try:
+        for dest in config.get('Install', 'Alias'):
+            alias = systemdir / dest
+            add_link(alias, target)
+
+    except KeyError:
+        pass
+
+
+def preset_all(root):
+    presets = Presets('system-preset', root)
+    services = collect_services(root)
+
+    for service, location in services.items():
+        state = presets.state(service)
+
+        if state == "enable" or state is None:
+            enable(root, service, location, services)
+
+
+def mask(root, *services):
+    systemdir = root / SYSCONFDIR / "systemd" / "system"
+    for service in services:
+        add_link(systemdir / service, "/dev/null")
+
+
+def main():
+    if sys.version_info < (3, 4, 0):
+        sys.exit("Python 3.4 or greater is required")
+
+    parser = argparse.ArgumentParser()
+    parser.add_argument('command', nargs=1, choices=['mask', 'preset-all'])
+    parser.add_argument('service', nargs=argparse.REMAINDER)
+    parser.add_argument('--root')
+    parser.add_argument('--preset-mode',
+                        choices=['full', 'enable-only', 'disable-only'],
+                        default='full')
+
+    args = parser.parse_args()
+
+    root = Path(args.root) if args.root else ROOT
+    command = args.command[0]
+    if command == "mask":
+        mask(root, *args.service)
+    elif command == "preset-all":
+        if len(args.service) != 0:
+            sys.exit("Too many arguments.")
+        if args.preset_mode != "enable-only":
+            sys.exit("Only enable-only is supported as preset-mode.")
+        preset_all(root)
+    else:
+        raise RuntimeError()
+
+
+if __name__ == '__main__':
+    main()
Signature · 1 line
2.17.1
#9 lore raw

PATCH v6 6/6 image: call systemctl preset-all for images

In reply to Alex Kiernan

From: Alex Kiernan <[email protected]>

Rather than rely on systemd's default invocation of preset-all at runtime, we pre-populate the symlink tree as part of of the image. This is done late so any overrides of presets during rootfs construction should already have happened.

Whilst we don't strictly need this for the read-write root case, it avoids boot time churn; for read-only root we have to do it here.

---

Changes in v6:
- moved systemctl preset-all to IMAGE_PREPROCESS so it runs after ROOTFS, run for all images, not just read-only

Changes in v5: None

 meta/classes/image.bbclass | 9 ++++++++-
 1 file changed, 8 insertions(+), 1 deletion(-)
meta/classes/image.bbclass +8 -1
index 276d0d31f4a5..a23403c0827a 100644
@@ -664,6 +664,13 @@ reproducible_final_image_task () {
         find  ${IMAGE_ROOTFS} -exec touch -h  --date=@$REPRODUCIBLE_TIMESTAMP_ROOTFS {} \;
     fi
 }
-IMAGE_PREPROCESS_COMMAND_append = " reproducible_final_image_task; "
+
+IMAGE_EXTRADEPENDS += "${@bb.utils.contains('DISTRO_FEATURES', 'systemd', 'systemd-systemctl-native', '', d)}"
+
+systemd_preset_all () {
+	systemctl --root="${IMAGE_ROOTFS}" --preset-mode=enable-only preset-all
+}
+
+IMAGE_PREPROCESS_COMMAND_append = " ${@bb.utils.contains('DISTRO_FEATURES', 'systemd', 'systemd_preset_all;', '', d)} reproducible_final_image_task; "
 
 CVE_PRODUCT = ""
Signature · 1 line
2.17.1
#10 lore raw

Re: [PATCH v6 6/6] image: call systemctl preset-all for images

Hi Alex,

Quoted 32 lines · Alex Kiernan

From: Alex Kiernan <[email protected]>

Rather than rely on systemd's default invocation of preset-all at runtime, we pre-populate the symlink tree as part of of the image. This is done late so any overrides of presets during rootfs construction should already have happened.

Whilst we don't strictly need this for the read-write root case, it avoids boot time churn; for read-only root we have to do it here.

---

Changes in v6:
- moved systemctl preset-all to IMAGE_PREPROCESS so it runs after ROOTFS, run for all images, not just read-only

Changes in v5: None

  meta/classes/image.bbclass | 9 ++++++++-
  1 file changed, 8 insertions(+), 1 deletion(-)
meta/classes/image.bbclass +0 -1
index 276d0d31f4a5..a23403c0827a 100644
@@ -664,6 +664,13 @@ reproducible_final_image_task () {
          find  ${IMAGE_ROOTFS} -exec touch -h  --date=@$REPRODUCIBLE_TIMESTAMP_ROOTFS {} \;
      fi
  }
-IMAGE_PREPROCESS_COMMAND_append = " reproducible_final_image_task; "

meta/classes/image.bbclass reproducible_final_image_task () { · line 676

Quoted 10 lines
meta/classes/image.bbclass +8 -0
+
+IMAGE_EXTRADEPENDS += "${@bb.utils.contains('DISTRO_FEATURES', 'systemd', 'systemd-systemctl-native', '', d)}"
+
+systemd_preset_all () {
+	systemctl --root="${IMAGE_ROOTFS}" --preset-mode=enable-only preset-all
+}
+
+IMAGE_PREPROCESS_COMMAND_append = " ${@bb.utils.contains('DISTRO_FEATURES', 'systemd', 'systemd_preset_all;', '', d)} reproducible_final_image_task; "
 
  CVE_PRODUCT = ""

In the interest of being able to produce a "stateless" system, I'd prefer to see us trying to empty out /etc at buildtime. That will allows us to error out when things are being installed directly to /etc such that they won't be set up properly at runtime when the system is "stateless". Allow things that _can_ be set up at runtime to be so; the runtime cost is small and paid only at first boot for stateful systems.

Calling preset-all at buildtime should only be necessary for read-only rootfs; even there, long term, /etc should be writable as that's what systemd (and others) expect.

/Jonas

#11 lore raw
In reply to Alex Kiernan
Quoted 70 lines · Alex Kiernan

This patch set is largely Jonas Bonn's to move towards a "stateless" configuration:

These patches make some modifications to systemd with the long-term goal of being able to run OE in systemd's "stateless" configuration. "Stateless" boils down to building an image with empty /etc and /var directories so that volatile (tmpfs) filesystems can be mounted there; this requires that the system subsequently be able to populate these directories dynamically, which systemd mostly takes care of if things are done right.

In these patches:
i) Don't include machine-id in writable images so that systemd can run
its first-boot machinery
ii) Move systemd configuration files out of /etc iii) Allow systemd to dynamically enable services and populate
/etc/systemd/system via the presets mechanism

There's a long way to go to get to a working "stateless" configuration. Getting to a "volatile" system (just empty /var) should be easier and I'll post patches moving things in that direction shortly.

However as a result of the systemd 242 upgrade, which includes 01d2041e41f4 ("meson: stop creating enablement symlinks in /etc during installation"), services such as systemd-networkd are no longer enabled in images.

This patch set fixes this problem in addition to satisfying the goal of moving towards "stateless" configurations.

The issue with respect to image testing during CI was caused by systemd-time-wait-sync.service being enabled due to the lack of a default preset policy:

https://www.freedesktop.org/wiki/Software/systemd/Preset/#howto

Changes in v6:
- switch configuration to simple overrides in /usr/lib/systemd/*.conf.d
- make systemd RRECOMMENDS rather than RDEPENDS on systemd-conf
- don't exit in postinst as when that executes we're actually a concatenation of all fragments
- validate SYSTEMD_AUTO_ENABLE is enable or disable
- rewrite systemctl-native in Python
- moved systemctl preset-all to IMAGE_PREPROCESS so it runs after ROOTFS, run for all images, not just read-only

Changes in v5:
- rebased for systemd 242
- install default preset distribution policy of "enable nothing"

Alex Kiernan (3):
systemd-conf: simplify creation of machine-specific configuration
systemctl-native: Rewrite in Python supporting preset-all and mask
image: call systemctl preset-all for images

Jonas Bonn (3):
systemd: don't build firstboot by default
systemd: do not create machine-id
systemd: create preset files instead of installing in image

 meta/classes/image.bbclass                    |   9 +-
 meta/classes/rootfs-postcommands.bbclass      |   6 +
 meta/classes/systemd.bbclass                  |  41 +-
 .../systemd/systemd-conf/journald.conf        |   3 +
 .../systemd/systemd-conf/logind.conf          |   2 +
 .../systemd/systemd-conf/system.conf          |   2 +
 .../systemd/systemd-conf/system.conf-qemuall  |   3 +
 meta/recipes-core/systemd/systemd-conf_242.bb |  61 +--
 .../systemd/systemd-systemctl/systemctl       | 476 ++++++++++--------
Quoted 9 lines
 .../systemd/systemd/99-default.preset         |   1 +
 meta/recipes-core/systemd/systemd_242.bb      |  26 +-
 11 files changed, 360 insertions(+), 270 deletions(-)
 create mode 100644 meta/recipes-core/systemd/systemd-conf/journald.conf
 create mode 100644 meta/recipes-core/systemd/systemd-conf/logind.conf
 create mode 100644 meta/recipes-core/systemd/systemd-conf/system.conf
 create mode 100644 meta/recipes-core/systemd/systemd-conf/system.conf-qemuall
 create mode 100644 meta/recipes-core/systemd/systemd/99-default.preset

Sigh...

this still has issues - if you boot with ro on the kernel command line and without an initramfs, then / is read-only when systemd starts and it basically refuses to do anything:

[    7.222134] systemd[1]: No hostname configured.
[    7.227266] systemd[1]: Set hostname to <localhost>.
[    7.232622] systemd[1]: System cannot boot: Missing /etc/machine-id
and /etc is mounted read-only.
[    7.241750] systemd[1]: Booting up is supported only when:
[    7.247362] systemd[1]: 1) /etc/machine-id exists and is populated.
[    7.253752] systemd[1]: 2) /etc/machine-id exists and is empty.
[    7.259757] systemd[1]: 3) /etc/machine-id is missing and /etc is writable.

Note this has nothing to do with read-only-rootfs, this is just a regular boot... that said a bunch of things that were broken now work, so it's progress!

I'm leaning towards having systemctl-native touch /etc/machine-id when it runs, unless you explicitly ask for stateless in DISTRO_FEATURES... patches to follow.

--
Alex Kiernan

#12 lore raw

Hi Alex,

Quoted 96 lines · Alex Kiernan
Quoted 73 lines · Alex Kiernan

This patch set is largely Jonas Bonn's to move towards a "stateless" configuration:

These patches make some modifications to systemd with the long-term goal of being able to run OE in systemd's "stateless" configuration. "Stateless" boils down to building an image with empty /etc and /var directories so that volatile (tmpfs) filesystems can be mounted there; this requires that the system subsequently be able to populate these directories dynamically, which systemd mostly takes care of if things are done right.

In these patches:
i) Don't include machine-id in writable images so that systemd can run
its first-boot machinery
ii) Move systemd configuration files out of /etc iii) Allow systemd to dynamically enable services and populate
/etc/systemd/system via the presets mechanism

There's a long way to go to get to a working "stateless" configuration. Getting to a "volatile" system (just empty /var) should be easier and I'll post patches moving things in that direction shortly.

However as a result of the systemd 242 upgrade, which includes 01d2041e41f4 ("meson: stop creating enablement symlinks in /etc during installation"), services such as systemd-networkd are no longer enabled in images.

This patch set fixes this problem in addition to satisfying the goal of moving towards "stateless" configurations.

The issue with respect to image testing during CI was caused by systemd-time-wait-sync.service being enabled due to the lack of a default preset policy:

https://www.freedesktop.org/wiki/Software/systemd/Preset/#howto

Changes in v6:
- switch configuration to simple overrides in /usr/lib/systemd/*.conf.d
- make systemd RRECOMMENDS rather than RDEPENDS on systemd-conf
- don't exit in postinst as when that executes we're actually a concatenation of all fragments
- validate SYSTEMD_AUTO_ENABLE is enable or disable
- rewrite systemctl-native in Python
- moved systemctl preset-all to IMAGE_PREPROCESS so it runs after ROOTFS, run for all images, not just read-only

Changes in v5:
- rebased for systemd 242
- install default preset distribution policy of "enable nothing"

Alex Kiernan (3):
systemd-conf: simplify creation of machine-specific configuration
systemctl-native: Rewrite in Python supporting preset-all and mask
image: call systemctl preset-all for images

Jonas Bonn (3):
systemd: don't build firstboot by default
systemd: do not create machine-id
systemd: create preset files instead of installing in image

  meta/classes/image.bbclass                    |   9 +-
  meta/classes/rootfs-postcommands.bbclass      |   6 +
  meta/classes/systemd.bbclass                  |  41 +-
  .../systemd/systemd-conf/journald.conf        |   3 +
  .../systemd/systemd-conf/logind.conf          |   2 +
  .../systemd/systemd-conf/system.conf          |   2 +
  .../systemd/systemd-conf/system.conf-qemuall  |   3 +
  meta/recipes-core/systemd/systemd-conf_242.bb |  61 +--
  .../systemd/systemd-systemctl/systemctl       | 476 ++++++++++--------
  .../systemd/systemd/99-default.preset         |   1 +
  meta/recipes-core/systemd/systemd_242.bb      |  26 +-
  11 files changed, 360 insertions(+), 270 deletions(-)
Quoted 6 lines

create mode 100644 meta/recipes-core/systemd/systemd-conf/journald.conf create mode 100644 meta/recipes-core/systemd/systemd-conf/logind.conf create mode 100644 meta/recipes-core/systemd/systemd-conf/system.conf create mode 100644 meta/recipes-core/systemd/systemd-conf/system.conf-qemuall create mode 100644 meta/recipes-core/systemd/systemd/99-default.preset

Sigh...

this still has issues - if you boot with ro on the kernel command line and without an initramfs, then / is read-only when systemd starts and it basically refuses to do anything:

[    7.222134] systemd[1]: No hostname configured.
[    7.227266] systemd[1]: Set hostname to <localhost>.
[    7.232622] systemd[1]: System cannot boot: Missing /etc/machine-id
and /etc is mounted read-only.
[    7.241750] systemd[1]: Booting up is supported only when:
[    7.247362] systemd[1]: 1) /etc/machine-id exists and is populated.
[    7.253752] systemd[1]: 2) /etc/machine-id exists and is empty.
[    7.259757] systemd[1]: 3) /etc/machine-id is missing and /etc is writable.
Quoted 7 lines

Note this has nothing to do with read-only-rootfs, this is just a regular boot... that said a bunch of things that were broken now work, so it's progress!

I'm leaning towards having systemctl-native touch /etc/machine-id when it runs, unless you explicitly ask for stateless in DISTRO_FEATURES... patches to follow.

The paradigm that systemd follows is that /etc is _always_ writable. The read-only rootfs that OE produces with a read-only /etc is pathological from systemd's point of view (as is your 'ro' kernel parameter example). The way to handle this is to:

i) mount a tmpfs over /etc
ii) move the contents of /etc to /usr/share/factory/etc at buildtime and have tmpfiles.d snippets that take care of populating /etc at runtime

I have a set of patches that does this, but it's all pretty fragile at this point in time. It scans /etc at buildtime, moves files and links to factory/{var,etc}, sets up tmpfiles.d snippets for files and directories. Doing this, however, somewhat requires for things that _can_ be set up at runtime to be so; what remains in /etc should only be stuff that doesn't set itself up at runtime _yet_. For this reason, I'm inclined to say that you _don't_ want to leave the /etc/systemd/system nor /etc/machine-id files in the image.

/Jonas

Quoted 4 lines

--
Alex Kiernan

#13 lore raw
Quoted 122 lines

-----Original Message-----
From: [email protected] <openembedded- [email protected]> On Behalf Of Jonas Bonn
Sent: den 3 maj 2019 14:55
To: Alex Kiernan <[email protected]>; OE-core <openembedded- [email protected]>
Subject: Re: [OE-core] [OE-Core][PATCH v6 0/6] systemd stateless configuration

Hi Alex,

Quoted 97 lines · Alex Kiernan
Quoted 75 lines · Alex Kiernan

This patch set is largely Jonas Bonn's to move towards a "stateless" configuration:

These patches make some modifications to systemd with the long- term goal of being able to run OE in systemd's "stateless" configuration. "Stateless" boils down to building an image with empty /etc and /var directories so that volatile (tmpfs) filesystems can be mounted there; this requires that the system subsequently be able to populate these directories dynamically, which systemd mostly takes care of if things are done right.

In these patches:
i) Don't include machine-id in writable images so that
systemd can run its first-boot machinery
ii) Move systemd configuration files out of /etc iii) Allow systemd to dynamically enable services and
populate /etc/systemd/system via the presets mechanism

There's a long way to go to get to a working "stateless" configuration. Getting to a "volatile" system (just empty /var) should be easier and I'll post patches moving things in that direction shortly.

However as a result of the systemd 242 upgrade, which includes 01d2041e41f4 ("meson: stop creating enablement symlinks in /etc during installation"), services such as systemd-networkd are no longer enabled in images.

This patch set fixes this problem in addition to satisfying the goal of moving towards "stateless" configurations.

The issue with respect to image testing during CI was caused by systemd-time-wait-sync.service being enabled due to the lack of a default preset policy:

https://www.freedesktop.org/wiki/Software/systemd/Preset/#howto

Changes in v6:
- switch configuration to simple overrides in /usr/lib/systemd/*.conf.d
- make systemd RRECOMMENDS rather than RDEPENDS on systemd-conf
- don't exit in postinst as when that executes we're actually a concatenation of all fragments
- validate SYSTEMD_AUTO_ENABLE is enable or disable
- rewrite systemctl-native in Python
- moved systemctl preset-all to IMAGE_PREPROCESS so it runs after ROOTFS, run for all images, not just read-only

Changes in v5:
- rebased for systemd 242
- install default preset distribution policy of "enable nothing"

Alex Kiernan (3):
systemd-conf: simplify creation of machine-specific configuration
systemctl-native: Rewrite in Python supporting preset-all and mask
image: call systemctl preset-all for images

Jonas Bonn (3):
systemd: don't build firstboot by default
systemd: do not create machine-id
systemd: create preset files instead of installing in image

  meta/classes/image.bbclass                    |   9 +-
  meta/classes/rootfs-postcommands.bbclass      |   6 +
  meta/classes/systemd.bbclass                  |  41 +-
  .../systemd/systemd-conf/journald.conf        |   3 +
  .../systemd/systemd-conf/logind.conf          |   2 +
  .../systemd/systemd-conf/system.conf          |   2 +
  .../systemd/systemd-conf/system.conf-qemuall  |   3 +
  meta/recipes-core/systemd/systemd-conf_242.bb |  61 +--
  .../systemd/systemd-systemctl/systemctl       | 476 ++++++++++--------
  .../systemd/systemd/99-default.preset         |   1 +
  meta/recipes-core/systemd/systemd_242.bb      |  26 +-
  11 files changed, 360 insertions(+), 270 deletions(-)
Quoted 5 lines

create mode 100644 meta/recipes-core/systemd/systemd-conf/journald.conf create mode 100644 meta/recipes-core/systemd/systemd-conf/logind.conf create mode 100644 meta/recipes-core/systemd/systemd-conf/system.conf create mode 100644 meta/recipes-core/systemd/systemd-conf/system.conf-qemuall create mode 100644 meta/recipes-core/systemd/systemd/99-default.preset

Sigh...

this still has issues - if you boot with ro on the kernel command line and without an initramfs, then / is read-only when systemd starts and it basically refuses to do anything:

[    7.222134] systemd[1]: No hostname configured.
[    7.227266] systemd[1]: Set hostname to <localhost>.
[    7.232622] systemd[1]: System cannot boot: Missing /etc/machine-id and /etc is mounted read-only.
[    7.241750] systemd[1]: Booting up is supported only when:
[    7.247362] systemd[1]: 1) /etc/machine-id exists and is populated.
[    7.253752] systemd[1]: 2) /etc/machine-id exists and is empty.
[    7.259757] systemd[1]: 3) /etc/machine-id is missing and /etc is writable.
Quoted 7 lines

Note this has nothing to do with read-only-rootfs, this is just a regular boot... that said a bunch of things that were broken now work, so it's progress!

I'm leaning towards having systemctl-native touch /etc/machine-id when it runs, unless you explicitly ask for stateless in DISTRO_FEATURES... patches to follow.

The paradigm that systemd follows is that /etc is _always_ writable. The read-only rootfs that OE produces with a read-only /etc is pathological from systemd's point of view (as is your 'ro' kernel parameter example). The way to handle this is to:

Quoted 4 lines

i) mount a tmpfs over /etc
ii) move the contents of /etc to /usr/share/factory/etc at buildtime and
have tmpfiles.d snippets that take care of populating /etc at runtime

That is probably the way to handle it for a stateless system. However, even without stateless, having /etc being writable is almost a necessity when using systemd. In our case, we mount an overlayfs on top of /etc to make it writable (before systemd is started). As I do not see us switching to a stateless system any time soon, I believe it would be good to be able to differentiate between the tree variants for how /etc is used:
read-only, writable and stateless. Today the only DISTRO_FEATURE is read-only-rootfs, which is not really relevant for /etc.

Quoted 11 lines

I have a set of patches that does this, but it's all pretty fragile at this point in time. It scans /etc at buildtime, moves files and links to factory/{var,etc}, sets up tmpfiles.d snippets for files and directories. Doing this, however, somewhat requires for things that _can_ be set up at runtime to be so; what remains in /etc should only be
stuff that doesn't set itself up at runtime _yet_. For this reason, I'm
inclined to say that you _don't_ want to leave the /etc/systemd/system nor /etc/machine-id files in the image.

Quoted 3 lines

/Jonas

Quoted 1 line

Alex Kiernan

//Peter

#14 lore raw
In reply to Jonas Bonn
Quoted 117 lines · Jonas Bonn

Hi Alex,

Quoted 96 lines · Alex Kiernan
Quoted 73 lines · Alex Kiernan

This patch set is largely Jonas Bonn's to move towards a "stateless" configuration:

These patches make some modifications to systemd with the long-term goal of being able to run OE in systemd's "stateless" configuration. "Stateless" boils down to building an image with empty /etc and /var directories so that volatile (tmpfs) filesystems can be mounted there; this requires that the system subsequently be able to populate these directories dynamically, which systemd mostly takes care of if things are done right.

In these patches:
i) Don't include machine-id in writable images so that systemd can run
its first-boot machinery
ii) Move systemd configuration files out of /etc iii) Allow systemd to dynamically enable services and populate
/etc/systemd/system via the presets mechanism

There's a long way to go to get to a working "stateless" configuration. Getting to a "volatile" system (just empty /var) should be easier and I'll post patches moving things in that direction shortly.

However as a result of the systemd 242 upgrade, which includes 01d2041e41f4 ("meson: stop creating enablement symlinks in /etc during installation"), services such as systemd-networkd are no longer enabled in images.

This patch set fixes this problem in addition to satisfying the goal of moving towards "stateless" configurations.

The issue with respect to image testing during CI was caused by systemd-time-wait-sync.service being enabled due to the lack of a default preset policy:

https://www.freedesktop.org/wiki/Software/systemd/Preset/#howto

Changes in v6:
- switch configuration to simple overrides in /usr/lib/systemd/*.conf.d
- make systemd RRECOMMENDS rather than RDEPENDS on systemd-conf
- don't exit in postinst as when that executes we're actually a concatenation of all fragments
- validate SYSTEMD_AUTO_ENABLE is enable or disable
- rewrite systemctl-native in Python
- moved systemctl preset-all to IMAGE_PREPROCESS so it runs after ROOTFS, run for all images, not just read-only

Changes in v5:
- rebased for systemd 242
- install default preset distribution policy of "enable nothing"

Alex Kiernan (3):
systemd-conf: simplify creation of machine-specific configuration
systemctl-native: Rewrite in Python supporting preset-all and mask
image: call systemctl preset-all for images

Jonas Bonn (3):
systemd: don't build firstboot by default
systemd: do not create machine-id
systemd: create preset files instead of installing in image

  meta/classes/image.bbclass                    |   9 +-
  meta/classes/rootfs-postcommands.bbclass      |   6 +
  meta/classes/systemd.bbclass                  |  41 +-
  .../systemd/systemd-conf/journald.conf        |   3 +
  .../systemd/systemd-conf/logind.conf          |   2 +
  .../systemd/systemd-conf/system.conf          |   2 +
  .../systemd/systemd-conf/system.conf-qemuall  |   3 +
  meta/recipes-core/systemd/systemd-conf_242.bb |  61 +--
  .../systemd/systemd-systemctl/systemctl       | 476 ++++++++++--------
  .../systemd/systemd/99-default.preset         |   1 +
  meta/recipes-core/systemd/systemd_242.bb      |  26 +-
  11 files changed, 360 insertions(+), 270 deletions(-)
Quoted 6 lines

create mode 100644 meta/recipes-core/systemd/systemd-conf/journald.conf create mode 100644 meta/recipes-core/systemd/systemd-conf/logind.conf create mode 100644 meta/recipes-core/systemd/systemd-conf/system.conf create mode 100644 meta/recipes-core/systemd/systemd-conf/system.conf-qemuall create mode 100644 meta/recipes-core/systemd/systemd/99-default.preset

Sigh...

this still has issues - if you boot with ro on the kernel command line and without an initramfs, then / is read-only when systemd starts and it basically refuses to do anything:

[    7.222134] systemd[1]: No hostname configured.
[    7.227266] systemd[1]: Set hostname to <localhost>.
[    7.232622] systemd[1]: System cannot boot: Missing /etc/machine-id
and /etc is mounted read-only.
[    7.241750] systemd[1]: Booting up is supported only when:
[    7.247362] systemd[1]: 1) /etc/machine-id exists and is populated.
[    7.253752] systemd[1]: 2) /etc/machine-id exists and is empty.
[    7.259757] systemd[1]: 3) /etc/machine-id is missing and /etc is writable.
Quoted 7 lines

Note this has nothing to do with read-only-rootfs, this is just a regular boot... that said a bunch of things that were broken now work, so it's progress!

I'm leaning towards having systemctl-native touch /etc/machine-id when it runs, unless you explicitly ask for stateless in DISTRO_FEATURES... patches to follow.

The paradigm that systemd follows is that /etc is _always_ writable. The read-only rootfs that OE produces with a read-only /etc is pathological from systemd's point of view (as is your 'ro' kernel parameter example). The way to handle this is to:

i) mount a tmpfs over /etc
ii) move the contents of /etc to /usr/share/factory/etc at buildtime and have tmpfiles.d snippets that take care of populating /etc at runtime

Quoted 9 lines

I have a set of patches that does this, but it's all pretty fragile at this point in time. It scans /etc at buildtime, moves files and links to factory/{var,etc}, sets up tmpfiles.d snippets for files and directories. Doing this, however, somewhat requires for things that _can_ be set up at runtime to be so; what remains in /etc should only be stuff that doesn't set itself up at runtime _yet_. For this reason, I'm inclined to say that you _don't_ want to leave the /etc/systemd/system nor /etc/machine-id files in the image.

I think the problem we're going to bump into is that ro getting passed in on the kernel command line I get from U-Boot and whilst OSTree gives me enough levers to make that per-boot, I know in our legacy world (and so lots of other peoples worlds), doing that kind of change isn't a goer... so whatever we do I think we're going to need lots of options. Obviously I'm relying on systemd doing a rw remount of / early in the boot.

I think setting /etc/machine-id blank and populating the wants/requires (and all the aliases/also stuff...) or doing none of it are valid options, but what I did with half of it is wrong :|

About to send out a patch set which follow's Chen's suggestion way back of adding a 'stateless' DISTRO_FEATURE which controls this and gets you all or nothing, which I think should fix anyone who's currently broken, but at the same time give us a lever which we can play with to get an actual stateless build. Persistent mutable /etc (in fact mutable anything) is the bane of my life, so I'm interested in making it work... unfortunately there's still quite a lot of gaps!

--
Alex Kiernan