Skip to contentSkip to conversation

package/systemd: hybrid cgroupfs hierarchy for docker compatibility

PATCH New Christian Stewart 4 participants 8 messages Nov 11, 2019

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 package/systemd: hybrid cgroupfs hierarchy for docker compatibility

Docker fails to start with "Devices cgroup isn't mounted" as of systemd 243. According to the systemd documentation:

systemd now defaults to the "unified" cgroup hierarchy setup during build-time, i.e. -Ddefault-hierarchy=unified is now the build-time default. Previously, -Ddefault-hierarchy=hybrid was the default. [...] Downstream production distributions might want to continue to use -Ddefault-hierarchy=hybrid (or even =legacy) for their builds as unfortunately the popular container managers have not caught up with the kernel API changes.

Changing this option to "hybrid" or "legacy" fixes the Docker startup.

Reference: https://github.com/opencontainers/runc/issues/654

---
 package/systemd/systemd.mk | 1 +
 1 file changed, 1 insertion(+)
package/systemd/systemd.mk +1 -0
index 94d5f703cd..92490eb86b 100644
@@ -26,6 +26,7 @@ SYSTEMD_CONF_OPTS += \
 	-Dima=false \
 	-Dldconfig=false \
 	-Ddefault-dnssec=no \
+	-Ddefault-hierarchy=hybrid \
 	-Dtests=false \
 	-Dsplit-bin=true \
 	-Dsplit-usr=false \
Signature · 1 line
2.24.0
#1 lore raw

PATCH 1/1 package/systemd: hybrid cgroupfs hierarchy for docker compatibility

Quoted 25 lines · Christian Stewart

Docker fails to start with "Devices cgroup isn't mounted" as of systemd 243. According to the systemd documentation:

systemd now defaults to the "unified" cgroup hierarchy setup during build-time, i.e. -Ddefault-hierarchy=unified is now the build-time default. Previously, -Ddefault-hierarchy=hybrid was the default. [...] Downstream production distributions might want to continue to use -Ddefault-hierarchy=hybrid (or even =legacy) for their builds as unfortunately the popular container managers have not caught up with the kernel API changes.

Changing this option to "hybrid" or "legacy" fixes the Docker startup.

Reference: https://github.com/opencontainers/runc/issues/654

---
 package/systemd/systemd.mk | 1 +
 1 file changed, 1 insertion(+)
package/systemd/systemd.mk +0 -0
index 94d5f703cd..92490eb86b 100644

package/systemd/systemd.mk SYSTEMD_CONF_OPTS += \ · line 29

Quoted 4 lines
package/systemd/systemd.mk +1 -0
@@ -26,6 +26,7 @@ SYSTEMD_CONF_OPTS += \
        -Dima=false \
        -Dldconfig=false \
        -Ddefault-dnssec=no \
+       -Ddefault-hierarchy=hybrid \

This should probably be conditionally enabled only when packages that don't support the unified cgroup hierarchy are enabled.

Quoted 6 lines
package/systemd/systemd.mk +0 -1
        -Dtests=false \
        -Dsplit-bin=true \
        -Dsplit-usr=false \
--

2.24.0

#2 lore raw

PATCH 1/1 package/systemd: hybrid cgroupfs hierarchy for docker compatibility

Keeping a list like that is going to be tricky...
Let's just hope the pressure on docker is strong enough and in a version or two we can move to unified
also:

Le lun. 11 nov. 2019 ? 10:05, James Hilliard <[email protected]> a ?crit :

Quoted 45 lines
Quoted 2 lines · Christian Stewart

Docker fails to start with "Devices cgroup isn't mounted" as of systemd

243.

Quoted 4 lines

According to the systemd documentation:

systemd now defaults to the "unified" cgroup hierarchy setup during build-time, i.e. -Ddefault-hierarchy=unified is now the build-time

default.

Quoted 1 line

Previously, -Ddefault-hierarchy=hybrid was the default. [...]

Downstream

Quoted 2 lines

production distributions might want to continue to use -Ddefault-hierarchy=hybrid (or even =legacy) for their builds as

unfortunately

Quoted 1 line

the popular container managers have not caught up with the kernel API

changes.

Quoted 15 lines

Changing this option to "hybrid" or "legacy" fixes the Docker startup.

Reference: https://github.com/opencontainers/runc/issues/654

---
 package/systemd/systemd.mk | 1 +
 1 file changed, 1 insertion(+)
package/systemd/systemd.mk +0 -0
index 94d5f703cd..92490eb86b 100644

package/systemd/systemd.mk SYSTEMD_CONF_OPTS += \ · line 29

Quoted 4 lines
package/systemd/systemd.mk +1 -0
@@ -26,6 +26,7 @@ SYSTEMD_CONF_OPTS += \
        -Dima=false \
        -Dldconfig=false \
        -Ddefault-dnssec=no \
+       -Ddefault-hierarchy=hybrid \

This should probably be conditionally enabled only when packages that don't support the unified cgroup hierarchy are enabled.

Quoted 6 lines
package/systemd/systemd.mk +0 -1
        -Dtests=false \
        -Dsplit-bin=true \
        -Dsplit-usr=false \
--

2.24.0

Signature · 21 lines
[image: SMILE]  <http://www.smile.eu/>

20 rue des Jardins
92600 Asni?res-sur-Seine
*J?r?my ROSEN*
Architecte technique

[image: email] jeremy.rosen at smile.fr
[image: phone]  +33 6 88 25 87 42
[image: url] http://www.smile.eu

[image: Twitter] <https://twitter.com/GroupeSmile> [image: Facebook]
<https://www.facebook.com/smileopensource> [image: LinkedIn]
<https://www.linkedin.com/company/smile> [image: Github]
<https://github.com/Smile-SA>

[image: D?couvrez l?univers Smile, rendez-vous sur smile.eu]
<https://www.smile.eu/fr/publications/livres-blancs/yocto?utm_source=signature&utm_medium=email&utm_campaign=signature>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.busybox.net/pipermail/buildroot/attachments/20191111/5ce8548b/attachment.html>
#3 lore raw

PATCH 1/1 package/systemd: hybrid cgroupfs hierarchy for docker compatibility

Hello,

Quoted 1 line · On Mon, 11 Nov 2019 11:09:27 +0100 J?r?my ROSEN <[email protected]> wrote:

Keeping a list like that is going to be tricky...

You don't really have to keep a list. You can have a blind option:

config BR2_PACKAGE_SYSTEMD_NEEDS_HYBRID_HIERARCHY
bool

systemd.mk would observe this boolean to enable (or not) the hybrid hierarchy, and packages that need it would select this boolean.

That being said, is there any drawback in unconditionally using the hybrid hierarchy? James, since you suggested to make it conditional, do you have some arguments/feedback?

Thanks,

Thomas

Signature · 3 lines
Thomas Petazzoni, CTO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com
#4 lore raw

PATCH 1/1 package/systemd: hybrid cgroupfs hierarchy for docker compatibility

It's about what implementation of cgroup is used by systemd to maintain the cgroup-tree of processes

* legacy is v1
* hybrid is both v1 and v2
* unified is v2 only
(note that only the default is changed, this can be changed via the kernel command-line at boot time)

v1 is deprecated on the kernel side because it was... a mess. The systemd people are migrating stuff from v1 to v2 as new controllers are added to v2, but at this point there is little left that is v1 only thus the change of the default value

Very few user-space projects care about cgroups : mainly systemd and container-managers. Container-manager need to interact with systemd as described in
https://systemd.io/CGROUP_DELEGATION.html which allows to drop v1 and to correctly spread the responsibility between systemd and the container managers without
stepping on each other's toes.
Also note that Fedora31 will use the unified cgroup hierarchy. A good summary of adoption status :
https://medium.com/nttlabs/cgroup-v2-596d035be4d7

At that point, I think staying in hybrid mode is the safest bet :
everything in user-space hasn't migrated yet.
Let Fedora take the fire and force everybody to migrate, and let's revisit the subject once (at least) Docker has migrated...

Le lun. 11 nov. 2019 ? 15:39, Thomas Petazzoni <[email protected]> a ?crit :

Quoted 27 lines

Hello,

Quoted 1 line · On Mon, 11 Nov 2019 11:09:27 +0100 J?r?my ROSEN <[email protected]> wrote:

Keeping a list like that is going to be tricky...

You don't really have to keep a list. You can have a blind option:

config BR2_PACKAGE_SYSTEMD_NEEDS_HYBRID_HIERARCHY
bool

systemd.mk would observe this boolean to enable (or not) the hybrid hierarchy, and packages that need it would select this boolean.

That being said, is there any drawback in unconditionally using the hybrid hierarchy? James, since you suggested to make it conditional, do you have some arguments/feedback?

Thanks,

Thomas
--
Thomas Petazzoni, CTO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com

Signature · 21 lines
[image: SMILE]  <http://www.smile.eu/>

20 rue des Jardins
92600 Asni?res-sur-Seine
*J?r?my ROSEN*
Architecte technique

[image: email] jeremy.rosen at smile.fr
[image: phone]  +33 6 88 25 87 42
[image: url] http://www.smile.eu

[image: Twitter] <https://twitter.com/GroupeSmile> [image: Facebook]
<https://www.facebook.com/smileopensource> [image: LinkedIn]
<https://www.linkedin.com/company/smile> [image: Github]
<https://github.com/Smile-SA>

[image: D?couvrez l?univers Smile, rendez-vous sur smile.eu]
<https://www.smile.eu/fr/publications/livres-blancs/yocto?utm_source=signature&utm_medium=email&utm_campaign=signature>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.busybox.net/pipermail/buildroot/attachments/20191111/9acb432c/attachment.html>
#5 lore raw

PATCH 1/1 package/systemd: hybrid cgroupfs hierarchy for docker compatibility

Hello,

(If possible, you should bottom post instead of top post. Thanks!)

Quoted 25 lines · On Mon, 11 Nov 2019 15:56:04 +0100 J?r?my ROSEN <[email protected]> wrote:

It's about what implementation of cgroup is used by systemd to maintain the cgroup-tree of processes

* legacy is v1
* hybrid is both v1 and v2
* unified is v2 only
(note that only the default is changed, this can be changed via the kernel command-line at boot time)

v1 is deprecated on the kernel side because it was... a mess. The systemd people are migrating stuff from v1 to v2 as new controllers are added to v2, but at this point there is little left that is v1 only thus the change of the default value

Very few user-space projects care about cgroups : mainly systemd and container-managers. Container-manager need to interact with systemd as described in
https://systemd.io/CGROUP_DELEGATION.html which allows to drop v1 and to correctly spread the responsibility between systemd and the container managers without
stepping on each other's toes.
Also note that Fedora31 will use the unified cgroup hierarchy. A good summary of adoption status :
https://medium.com/nttlabs/cgroup-v2-596d035be4d7

Quoted 4 lines

At that point, I think staying in hybrid mode is the safest bet :
everything in user-space hasn't migrated yet.
Let Fedora take the fire and force everybody to migrate, and let's revisit the subject once (at least) Docker has migrated...

OK, thanks a lot for this explanation, much clearer!

Thomas

Signature · 3 lines
Thomas Petazzoni, CTO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com
#6 lore raw

PATCH 1/1 package/systemd: hybrid cgroupfs hierarchy for docker compatibility

In reply to Christian Stewart
Quoted 10 lines · On Mon, 11 Nov 2019 00:33:48 -0800 Christian Stewart <[email protected]> wrote:

Docker fails to start with "Devices cgroup isn't mounted" as of systemd 243. According to the systemd documentation:

systemd now defaults to the "unified" cgroup hierarchy setup during build-time, i.e. -Ddefault-hierarchy=unified is now the build-time default. Previously, -Ddefault-hierarchy=hybrid was the default. [...] Downstream production distributions might want to continue to use -Ddefault-hierarchy=hybrid (or even =legacy) for their builds as unfortunately the popular container managers have not caught up with the kernel API changes.

Quoted 8 lines

Changing this option to "hybrid" or "legacy" fixes the Docker startup.

Reference: https://github.com/opencontainers/runc/issues/654

---
 package/systemd/systemd.mk | 1 +
 1 file changed, 1 insertion(+)

Applied to master, thanks. It would be nice to have some runtime tests around Docker, to verify that it is still working properly at runtime, so that we can catch this kind of issues.

Thanks!

Thomas

Signature · 3 lines
Thomas Petazzoni, CTO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com
#7 lore raw

PATCH 1/1 package/systemd: hybrid cgroupfs hierarchy for docker compatibility

Hi all,

(re-sending this from the correct email address)

Quoted 2 lines · Jeremy ROSEN

v1 is deprecated on the kernel side because it was... a mess. The systemd people are migrating stuff from v1 to v2 as new controllers a=

re

Quoted 2 lines

added to v2, but at this point there is little left that is v1 only thus the change of the default value

Quoted 6 lines · Thomas Petazzoni
Quoted 6 lines

systemd.mk would observe this boolean to enable (or not) the hybrid hierarchy, and packages that need it would select this boolean.

That being said, is there any drawback in unconditionally using the hybrid hierarchy? James, since you suggested to make it conditional, do you have some arguments/feedback?

I originally planned to submit this patch with a 3-state option for which mode is desired (legacy, hybrid, merged) and then having Docker and other packages depend on hybrid or legacy.

However, I realized that this would be quite a lot of work to then trace down all of the packages that potentially need the hybrid configuration, and that in the name of "not breaking userspace" it makes sense for Buildroot to follow the direction in the systemd NEWS and just default to "hybrid" for now.

If someone wants to do the above work to make the 3-state option, we could merge that first, with "hybrid" as the default, and later add dependencies from Docker -> systemd for that option.

Quoted 4 lines

At that point, I think staying in hybrid mode is the safest bet :
everything in user-space hasn't migrated yet.
Let Fedora take the fire and force everybody to migrate, and let's revisit the subject once (at least) Docker has migrated...

Agreed, we can probably wait until Docker has finished their migration to begin seriously considering moving off of "hybrid."

Best regards,
Christian

Quoted 31 lines · Thomas Petazzoni
Quoted 1 line · On Mon, 11 Nov 2019 00:33:48 -0800 Christian Stewart <[email protected]> wrote:

Docker fails to start with "Devices cgroup isn't mounted" as of systemd

243.

Quoted 4 lines

According to the systemd documentation:

systemd now defaults to the "unified" cgroup hierarchy setup during build-time, i.e. -Ddefault-hierarchy=unified is now the build-time

default.

Quoted 1 line

Previously, -Ddefault-hierarchy=hybrid was the default. [...]

Downstream

Quoted 2 lines

production distributions might want to continue to use -Ddefault-hierarchy=hybrid (or even =legacy) for their builds as

unfortunately

Quoted 1 line

the popular container managers have not caught up with the kernel API

changes.

Quoted 9 lines

Changing this option to "hybrid" or "legacy" fixes the Docker startup.

Reference: https://github.com/opencontainers/runc/issues/654

---
 package/systemd/systemd.mk | 1 +
 1 file changed, 1 insertion(+)

Applied to master, thanks. It would be nice to have some runtime tests around Docker, to verify that it is still working properly at runtime, so that we can catch this kind of issues.

Quoted 8 lines

Thanks!

Thomas
--
Thomas Petazzoni, CTO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.busybox.net/pipermail/buildroot/attachments/20191111/4c017e02/attachment.html>