Skip to contentSkip to conversation

Should systemd be marked as incompatible with musl?

DISCUSSION Adrian Bunk 5 participants 21 messages May 23, 2019 – May 29, 2019

lore
Share & tools
lore
#0 lore raw

systemd is fundamentally Linux-only and not portable to other kernels.

systemd upstream is using glibc extensions not present in other C libraries.

systemd upstream is accepting technically correct patches that help building with musl, but there is no interest upstream in keeping systemd working with non-glibc C libraries.

The way things currently go, systemd/musl will require an ever-growing amount of not upstreamable patches - and this is not sustainable.

It is also not clear where the systemd/musl combination makes sense:
If an embedded system is so size limited that the size of glibc matters, then systemd is unlikely to be an option.

Other musl-using distribution also seem to favour more lightweight init systems.

If someone has important usecases for using systemd with musl it might be possible to solve find solutions in an upstreamable way, but this would require a significant longterm commitment.

The current state is rather bad, some examples:

0001-Use-getenv-when-secure-versions-are-not-available.patch looks like waiting for a CVE number.

0008-add-missing-FTW_-macros-for-musl.patch seems to break code in areas (rng, smack, selinux) where breakages can have security implications.

0002-don-t-use-glibc-specific-qsort_r.patch might introduce race conditions that cause data corruption.

0007-don-t-fail-if-GLOB_BRACE-and-GLOB_ALTDIRFUNC-is-not.patch makes some functions behave different from what is expected by callers.

0017-Do-not-disable-buffering-when-writing-to-oom_score_a.patch and 0001-do-not-disable-buffer-in-writing-files.patch seems like an issue that would require debugging the root cause and then addressing it.

All this looks bad, and is expected to become worse.

In this situation it would be better to not claim that using systemd with musl would be a supported option.

cu
Adrian

Signature · 4 lines
       "Is there not promise of rain?" Ling Tan asked suddenly out
        of the darkness. There had been need of rain for many days.
       "Only a promise," Lao Er said.
                                       Pearl S. Buck - Dragon Seed
#1 lore raw
Quoted 11 lines · Adrian Bunk

systemd is fundamentally Linux-only and not portable to other kernels.

systemd upstream is using glibc extensions not present in other C libraries.

systemd upstream is accepting technically correct patches that help building with musl, but there is no interest upstream in keeping systemd working with non-glibc C libraries.

The way things currently go, systemd/musl will require an ever-growing amount of not upstreamable patches - and this is not sustainable.

I think I have to agree with you: several of the extensions are for security purposes, so we're potentially actively introducing issues.

Ross

#2 lore raw
Quoted 10 lines · Ross
Quoted 9 lines · Adrian Bunk

systemd is fundamentally Linux-only and not portable to other kernels.

systemd upstream is using glibc extensions not present in other C libraries.

systemd upstream is accepting technically correct patches that help building with musl, but there is no interest upstream in keeping systemd working with non-glibc C libraries.

Quoted 6 lines
Quoted 2 lines

The way things currently go, systemd/musl will require an ever-growing amount of not upstreamable patches - and this is not sustainable.

I think I have to agree with you: several of the extensions are for security purposes, so we're potentially actively introducing issues.

Ross

I suggest we only apply these musl patches when the libc is musl.

SRC_URI += "${@d.getVar('SRC_URI_MUSL') if d.getVar('TCLIBC') == 'musl' else ''}"

In this way, these patches do not affect glibc users. People who actually use musl will test these patches and maybe improve them if they really care about it.

For example, the following patches are written by me.

0017-Do-not-disable-buffering-when-writing-to-oom_score_a.patch and 0001-do-not-disable-buffer-in-writing-files.patch

These two patches are to solve runtime problems in musl based system. I knew there's something different in buffering mechanism between glibc and musl that causes this problem, but I really did not want to spend time digging into it. And as systemd was using unbuffered way for writing files for a long time, I just disabled them.

As Adrian and Ross pointed out, these patches are introducing potential security problems. I agree that we should do something about it. At a minimum, the glibc based system should not be affected.

I'll send out patch and please help review it.

Best Regards,
Chen Qi

#3 lore raw
Quoted 16 lines · ChenQi
Quoted 15 lines · Ross
Quoted 11 lines · Adrian Bunk

systemd is fundamentally Linux-only and not portable to other kernels.

systemd upstream is using glibc extensions not present in other C libraries.

systemd upstream is accepting technically correct patches that help building with musl, but there is no interest upstream in keeping systemd working with non-glibc C libraries.

The way things currently go, systemd/musl will require an ever-growing amount of not upstreamable patches - and this is not sustainable.

I think I have to agree with you: several of the extensions are for security purposes, so we're potentially actively introducing issues.

Quoted 8 lines
Quoted 1 line

Ross

I suggest we only apply these musl patches when the libc is musl.

SRC_URI += "${@d.getVar('SRC_URI_MUSL') if d.getVar('TCLIBC') == 'musl' else ''}"

In this way, these patches do not affect glibc users. People who actually use musl will test these patches and maybe improve them if they really care about it.

thats acceptable approach. So far we have helped upstreams of apps to cleanup or add support for musl and it has worked reasonably well, systemd is a bit of odd man out since they explicitly seems to not care for non glibc/linux systems, still they do accept cleanups which are resulting from such ports so its not completely wasted excercise, but I think dropping systemd support completely from musl is not an option I would like to go with, there are cases where this makes sense. Especially when you have to cater to different set of devices from small to big, userspace remaining same is big advantage atleast in the world I am in.

Quoted 15 lines

For example, the following patches are written by me.

0017-Do-not-disable-buffering-when-writing-to-oom_score_a.patch and 0001-do-not-disable-buffer-in-writing-files.patch

These two patches are to solve runtime problems in musl based system. I knew there's something different in buffering mechanism between glibc and musl that causes this problem, but I really did not want to spend time digging into it. And as systemd was using unbuffered way for writing files for a long time, I just disabled them.

As Adrian and Ross pointed out, these patches are introducing potential security problems. I agree that we should do something about it. At a minimum, the glibc based system should not be affected.

I'll send out patch and please help review it.

Best Regards,
Chen Qi

#4 lore raw
Quoted 7 lines · Khem Raj

...
but I think dropping
systemd support completely from musl is not an option I would like to go with, there are cases where this makes sense. Especially when you have to cater to different set of devices from small to big, userspace remaining same is big advantage atleast in the world I am in.
...

That's a good point - when arguing against systemd as default init system.

systemd is bigger than glibc, therefore on very small systems where the size of the C library matters using systemd is usually not an option.

cu
Adrian

Signature · 4 lines
       "Is there not promise of rain?" Ling Tan asked suddenly out
        of the darkness. There had been need of rain for many days.
       "Only a promise," Lao Er said.
                                       Pearl S. Buck - Dragon Seed
#5 lore raw
Quoted 9 lines · Adrian Bunk
Quoted 7 lines · Khem Raj

...
but I think dropping
systemd support completely from musl is not an option I would like to go with, there are cases where this makes sense. Especially when you have to cater to different set of devices from small to big, userspace remaining same is big advantage atleast in the world I am in.
...

Quoted 5 lines

That's a good point - when arguing against systemd as default init system.

systemd is bigger than glibc, therefore on very small systems where the size of the C library matters using systemd is usually not an option.

Yes, design-wise I concur, in practice, desktop distros rule the linux world and systemd is quite prevalent there, so sometimes you have to wear the shoes of same color, atleast thats what I see. apps want to run uniformly on both OE and non-OE systems

Quoted 3 lines

cu
Adrian

#6 lore raw
Quoted 12 lines · Khem Raj
Quoted 9 lines · Adrian Bunk
Quoted 7 lines · Khem Raj

...
but I think dropping
systemd support completely from musl is not an option I would like to go with, there are cases where this makes sense. Especially when you have to cater to different set of devices from small to big, userspace remaining same is big advantage atleast in the world I am in.
...

Quoted 7 lines
Quoted 4 lines

That's a good point - when arguing against systemd as default init system.

systemd is bigger than glibc, therefore on very small systems where the size of the C library matters using systemd is usually not an option.

Yes, design-wise I concur, in practice, desktop distros rule the linux world and systemd is quite prevalent there,

Desktop distros don't use musl - it wouldn't make sense.

Quoted 2 lines

so sometimes you have to wear the
shoes of same color, atleast thats what I see.

Is any musl-using distribution supporting it with systemd?

After a cursory look around it doesn't seem that trying to support the weird combination of systemd and musl brings compatibility with anyone else.

Quoted 1 line

apps want to run uniformly on both OE and non-OE systems

This implies using glibc, which is the correct choice in most cases.

cu
Adrian

Signature · 4 lines
       "Is there not promise of rain?" Ling Tan asked suddenly out
        of the darkness. There had been need of rain for many days.
       "Only a promise," Lao Er said.
                                       Pearl S. Buck - Dragon Seed
#7 lore raw
Quoted 16 lines · Adrian Bunk
Quoted 14 lines · Khem Raj
Quoted 11 lines · Adrian Bunk
Quoted 7 lines · Khem Raj

...
but I think dropping
systemd support completely from musl is not an option I would like to go with, there are cases where this makes sense. Especially when you have to cater to different set of devices from small to big, userspace remaining same is big advantage atleast in the world I am in.
...

That's a good point - when arguing against systemd as default init system.

Quoted 8 lines
Quoted 5 lines
Quoted 2 lines

systemd is bigger than glibc, therefore on very small systems where the size of the C library matters using systemd is usually not an option.

Yes, design-wise I concur, in practice, desktop distros rule the linux world and systemd is quite prevalent there,

Desktop distros don't use musl - it wouldn't make sense.

Yes, what I was saying is that decisions on init systems and like that are influenced by desktops too. Since the apps are being written for across the board portfolio where some platforms are desktop/server driven some are more embedded

Quoted 23 lines
Quoted 2 lines

so sometimes you have to wear the
shoes of same color, atleast thats what I see.

Is any musl-using distribution supporting it with systemd?

After a cursory look around it doesn't seem that trying to support the weird combination of systemd and musl brings compatibility with anyone else.

Quoted 1 line

apps want to run uniformly on both OE and non-OE systems

This implies using glibc, which is the correct choice in most cases.

cu
Adrian

--

"Is there not promise of rain?" Ling Tan asked suddenly out
of the darkness. There had been need of rain for many days.
"Only a promise," Lao Er said.
Pearl S. Buck - Dragon Seed

#8 lore raw
Quoted 20 lines · Khem Raj
Quoted 19 lines · Adrian Bunk
Quoted 17 lines · Khem Raj
Quoted 9 lines · Adrian Bunk
Quoted 7 lines · Khem Raj

...
but I think dropping
systemd support completely from musl is not an option I would like to go with, there are cases where this makes sense. Especially when you have to cater to different set of devices from small to big, userspace remaining same is big advantage atleast in the world I am in.
...

Quoted 4 lines

That's a good point - when arguing against systemd as default init system.

systemd is bigger than glibc, therefore on very small systems where the size of the C library matters using systemd is usually not an option.

Yes, design-wise I concur, in practice, desktop distros rule the linux world and systemd is quite prevalent there,

Quoted 6 lines
Quoted 1 line

Desktop distros don't use musl - it wouldn't make sense.

Yes, what I was saying is that decisions on init systems and like that are influenced by desktops too. Since the apps are being written for across the board portfolio where some platforms are desktop/server driven some are more embedded

The point is that most of the time using musl doesn't make sense.

Definitely not on a desktop, and also not with systemd as init system.

I haven't done exact measurements and it would also depend on the architecture, but the only real-world benefit of using musl instead of glibc for an OE user is something like "3 MB smaller in an -Os build".

On many of todays embedded systems such a small size difference is irrelevant. In these cases the correct solution that stays compatible with everything else is to use glibc.

And on really tiny systems where every single MB counts, all other design choices also have a high emphasis on size.

Using systemd instead of busybox init (or some other small init system) would cost you much more space than what the C library choice gave.

And the current sad state of the systemd musl patching that makes it compile but creates misbehaving functions and security vulnerabilities makes it an even worse idea to use systemd with musl.

cu
Adrian

Signature · 4 lines
       "Is there not promise of rain?" Ling Tan asked suddenly out
        of the darkness. There had been need of rain for many days.
       "Only a promise," Lao Er said.
                                       Pearl S. Buck - Dragon Seed
#9 lore raw
Quoted 33 lines · Adrian Bunk
Quoted 20 lines · Khem Raj
Quoted 19 lines · Adrian Bunk
Quoted 17 lines · Khem Raj
Quoted 9 lines · Adrian Bunk
Quoted 7 lines · Khem Raj

...
but I think dropping
systemd support completely from musl is not an option I would like to go with, there are cases where this makes sense. Especially when you have to cater to different set of devices from small to big, userspace remaining same is big advantage atleast in the world I am in.
...

Quoted 4 lines

That's a good point - when arguing against systemd as default init system.

systemd is bigger than glibc, therefore on very small systems where the size of the C library matters using systemd is usually not an option.

Yes, design-wise I concur, in practice, desktop distros rule the linux world and systemd is quite prevalent there,

Quoted 6 lines
Quoted 1 line

Desktop distros don't use musl - it wouldn't make sense.

Yes, what I was saying is that decisions on init systems and like that are influenced by desktops too. Since the apps are being written for across the board portfolio where some platforms are desktop/server driven some are more embedded

The point is that most of the time using musl doesn't make sense.

Definitely not on a desktop, and also not with systemd as init system.

Quoted 7 lines

I haven't done exact measurements and it would also depend on the architecture, but the only real-world benefit of using musl instead of glibc for an OE user is something like "3 MB smaller in an -Os build".

On many of todays embedded systems such a small size difference is irrelevant. In these cases the correct solution that stays compatible with everything else is to use glibc.

No, there is DRAM use difference too.

Quoted 11 lines

And on really tiny systems where every single MB counts, all other design choices also have a high emphasis on size.

Using systemd instead of busybox init (or some other small init system) would cost you much more space than what the C library choice gave.

And the current sad state of the systemd musl patching that makes it compile but creates misbehaving functions and security vulnerabilities makes it an even worse idea to use systemd with musl.

I think we should put in plan for 2.8 and define the scope, since we are switching
poky defaults to systemd, this might have testing implications for musl especially from
CI perpective

Quoted 10 lines

cu
Adrian

--

"Is there not promise of rain?" Ling Tan asked suddenly out
of the darkness. There had been need of rain for many days.
"Only a promise," Lao Er said.
Pearl S. Buck - Dragon Seed

#10 lore raw
Quoted 38 lines · Khem Raj
Quoted 37 lines · Adrian Bunk
Quoted 20 lines · Khem Raj
Quoted 19 lines · Adrian Bunk
Quoted 17 lines · Khem Raj
Quoted 9 lines · Adrian Bunk
Quoted 7 lines · Khem Raj

...
but I think dropping
systemd support completely from musl is not an option I would like to go with, there are cases where this makes sense. Especially when you have to cater to different set of devices from small to big, userspace remaining same is big advantage atleast in the world I am in.
...

Quoted 4 lines

That's a good point - when arguing against systemd as default init system.

systemd is bigger than glibc, therefore on very small systems where the size of the C library matters using systemd is usually not an option.

Yes, design-wise I concur, in practice, desktop distros rule the linux world and systemd is quite prevalent there,

Quoted 6 lines
Quoted 1 line

Desktop distros don't use musl - it wouldn't make sense.

Yes, what I was saying is that decisions on init systems and like that are influenced by desktops too. Since the apps are being written for across the board portfolio where some platforms are desktop/server driven some are more embedded

The point is that most of the time using musl doesn't make sense.

Definitely not on a desktop, and also not with systemd as init system.

I haven't done exact measurements and it would also depend on the architecture, but the only real-world benefit of using musl instead of glibc for an OE user is something like "3 MB smaller in an -Os build".

Quoted 5 lines
Quoted 3 lines

On many of todays embedded systems such a small size difference is irrelevant. In these cases the correct solution that stays compatible with everything else is to use glibc.

No, there is DRAM use difference too.

I didn't limit the size difference to the filesystem.

(But in practice RAM is usually a smaller problem than the filesystem.)

Quoted 6 lines
Quoted 6 lines

And on really tiny systems where every single MB counts, all other design choices also have a high emphasis on size.

Using systemd instead of busybox init (or some other small init system) would cost you much more space than what the C library choice gave.

Quoted 8 lines
Quoted 4 lines

And the current sad state of the systemd musl patching that makes it compile but creates misbehaving functions and security vulnerabilities makes it an even worse idea to use systemd with musl.

I think we should put in plan for 2.8 and define the scope, since we are switching
poky defaults to systemd,

Switching glibc builds to systemd as default is a reasonable decision.

Switching musl builds to systemd as default would be very bad.

Combining a tiny C library with a huge init system completely misses the configurations where the tiny C library actually makes sense for users.

Quoted 3 lines

this might have testing implications for
musl especially from
CI perpective

The CI should aim at testing what makes sense for users.

Offering musl becomes pointless if there is no small init system that is properly supported and tested with musl builds.

cu
Adrian

Signature · 4 lines
       "Is there not promise of rain?" Ling Tan asked suddenly out
        of the darkness. There had been need of rain for many days.
       "Only a promise," Lao Er said.
                                       Pearl S. Buck - Dragon Seed
#11 lore raw
Quoted 65 lines · Adrian Bunk
Quoted 38 lines · Khem Raj
Quoted 37 lines · Adrian Bunk
Quoted 20 lines · Khem Raj
Quoted 19 lines · Adrian Bunk
Quoted 17 lines · Khem Raj
Quoted 9 lines · Adrian Bunk
Quoted 7 lines · Khem Raj

...
but I think dropping
systemd support completely from musl is not an option I would like to go with, there are cases where this makes sense. Especially when you have to cater to different set of devices from small to big, userspace remaining same is big advantage atleast in the world I am in.
...

Quoted 4 lines

That's a good point - when arguing against systemd as default init system.

systemd is bigger than glibc, therefore on very small systems where the size of the C library matters using systemd is usually not an option.

Yes, design-wise I concur, in practice, desktop distros rule the linux world and systemd is quite prevalent there,

Quoted 6 lines
Quoted 1 line

Desktop distros don't use musl - it wouldn't make sense.

Yes, what I was saying is that decisions on init systems and like that are influenced by desktops too. Since the apps are being written for across the board portfolio where some platforms are desktop/server driven some are more embedded

The point is that most of the time using musl doesn't make sense.

Definitely not on a desktop, and also not with systemd as init system.

I haven't done exact measurements and it would also depend on the architecture, but the only real-world benefit of using musl instead of glibc for an OE user is something like "3 MB smaller in an -Os build".

Quoted 5 lines
Quoted 3 lines

On many of todays embedded systems such a small size difference is irrelevant. In these cases the correct solution that stays compatible with everything else is to use glibc.

No, there is DRAM use difference too.

I didn't limit the size difference to the filesystem.

(But in practice RAM is usually a smaller problem than the filesystem.)

Quoted 6 lines
Quoted 6 lines

And on really tiny systems where every single MB counts, all other design choices also have a high emphasis on size.

Using systemd instead of busybox init (or some other small init system) would cost you much more space than what the C library choice gave.

Quoted 8 lines
Quoted 4 lines

And the current sad state of the systemd musl patching that makes it compile but creates misbehaving functions and security vulnerabilities makes it an even worse idea to use systemd with musl.

I think we should put in plan for 2.8 and define the scope, since we are switching
poky defaults to systemd,

Quoted 7 lines

Switching glibc builds to systemd as default is a reasonable decision.

Switching musl builds to systemd as default would be very bad.

Combining a tiny C library with a huge init system completely misses the configurations where the tiny C library actually makes sense for users.

For new projects yes. However I know of a project (a big project, shipping millions of devices) which picked systemd and glibc long ago and is now running out of space. They already have various solutions to free up Flash (some apps switched to being runtime downloadable, etc) but if/when more savings need to be found then switching from glibc to musl might be a less invasive change than switching from systemd to some other init system. We obviously shouldn't make decisions for OE today based on the historical decisions of one project, but I just want to make the point that real world projects have a lifetime and may end up with a combination of systemd + musl due on past decisions that may not make sense for a new project starting today.

#12 lore raw
Quoted 75 lines · Andre McCurdy
Quoted 65 lines · Adrian Bunk
Quoted 38 lines · Khem Raj
Quoted 37 lines · Adrian Bunk
Quoted 20 lines · Khem Raj
Quoted 19 lines · Adrian Bunk
Quoted 17 lines · Khem Raj
Quoted 9 lines · Adrian Bunk
Quoted 7 lines · Khem Raj

...
but I think dropping
systemd support completely from musl is not an option I would like to go with, there are cases where this makes sense. Especially when you have to cater to different set of devices from small to big, userspace remaining same is big advantage atleast in the world I am in.
...

Quoted 4 lines

That's a good point - when arguing against systemd as default init system.

systemd is bigger than glibc, therefore on very small systems where the size of the C library matters using systemd is usually not an option.

Yes, design-wise I concur, in practice, desktop distros rule the linux world and systemd is quite prevalent there,

Quoted 6 lines
Quoted 1 line

Desktop distros don't use musl - it wouldn't make sense.

Yes, what I was saying is that decisions on init systems and like that are influenced by desktops too. Since the apps are being written for across the board portfolio where some platforms are desktop/server driven some are more embedded

The point is that most of the time using musl doesn't make sense.

Definitely not on a desktop, and also not with systemd as init system.

I haven't done exact measurements and it would also depend on the architecture, but the only real-world benefit of using musl instead of glibc for an OE user is something like "3 MB smaller in an -Os build".

Quoted 5 lines
Quoted 3 lines

On many of todays embedded systems such a small size difference is irrelevant. In these cases the correct solution that stays compatible with everything else is to use glibc.

No, there is DRAM use difference too.

I didn't limit the size difference to the filesystem.

(But in practice RAM is usually a smaller problem than the filesystem.)

Quoted 6 lines
Quoted 6 lines

And on really tiny systems where every single MB counts, all other design choices also have a high emphasis on size.

Using systemd instead of busybox init (or some other small init system) would cost you much more space than what the C library choice gave.

Quoted 8 lines
Quoted 4 lines

And the current sad state of the systemd musl patching that makes it compile but creates misbehaving functions and security vulnerabilities makes it an even worse idea to use systemd with musl.

I think we should put in plan for 2.8 and define the scope, since we are switching
poky defaults to systemd,

Quoted 7 lines

Switching glibc builds to systemd as default is a reasonable decision.

Switching musl builds to systemd as default would be very bad.

Combining a tiny C library with a huge init system completely misses the configurations where the tiny C library actually makes sense for users.

Quoted 12 lines

For new projects yes. However I know of a project (a big project, shipping millions of devices) which picked systemd and glibc long ago and is now running out of space. They already have various solutions to free up Flash (some apps switched to being runtime downloadable, etc) but if/when more savings need to be found then switching from glibc to musl might be a less invasive change than switching from systemd to some other init system. We obviously shouldn't make decisions for OE today based on the historical decisions of one project, but I just want to make the point that real world projects have a lifetime and may end up with a combination of systemd + musl due on past decisions that may not make sense for a new project starting today.

rightly so, and it can also mean that project moves away from OE if headwind is too strong.

#13 lore raw
In reply to Andre McCurdy
Quoted 15 lines · Andre McCurdy
Quoted 1 line · Adrian Bunk

On Fri, May 24, 2019 at 11:04:50AM -0700, Khem Raj wrote:

...

Quoted 11 lines
Quoted 3 lines

I think we should put in plan for 2.8 and define the scope, since we are switching
poky defaults to systemd,

Switching glibc builds to systemd as default is a reasonable decision.

Switching musl builds to systemd as default would be very bad.

Combining a tiny C library with a huge init system completely misses the configurations where the tiny C library actually makes sense for users.

Quoted 12 lines

For new projects yes. However I know of a project (a big project, shipping millions of devices) which picked systemd and glibc long ago and is now running out of space. They already have various solutions to free up Flash (some apps switched to being runtime downloadable, etc) but if/when more savings need to be found then switching from glibc to musl might be a less invasive change than switching from systemd to some other init system. We obviously shouldn't make decisions for OE today based on the historical decisions of one project, but I just want to make the point that real world projects have a lifetime and may end up with a combination of systemd + musl due on past decisions that may not make sense for a new project starting today.

I am feeling guilty that there are two only partially related topics mixed in this discussion.

In this part of the discussion the topic was what the default (and CI-tested) init system for musl should be - it seems obvious to me that systemd is not what users will usually want to use with musl.

But there is also the topic whether systemd on musl is in a state that it should be made available at all.

Does any of these millions of devices have untrusted users or an internet connection?

At that point my email that started this thread becomes relevant, the fact that the systemd/musl patches in OE add new security vulnerabilities and other bugs - and none of the systemd-on-musl proponents seems to consider this a problem they have to fix ASAP.

cu
Adrian

Signature · 4 lines
       "Is there not promise of rain?" Ling Tan asked suddenly out
        of the darkness. There had been need of rain for many days.
       "Only a promise," Lao Er said.
                                       Pearl S. Buck - Dragon Seed
#14 lore raw
Quoted 29 lines · Adrian Bunk
Quoted 15 lines · Andre McCurdy
Quoted 1 line · Adrian Bunk

On Fri, May 24, 2019 at 11:04:50AM -0700, Khem Raj wrote:

...

Quoted 11 lines
Quoted 3 lines

I think we should put in plan for 2.8 and define the scope, since we are switching
poky defaults to systemd,

Switching glibc builds to systemd as default is a reasonable decision.

Switching musl builds to systemd as default would be very bad.

Combining a tiny C library with a huge init system completely misses the configurations where the tiny C library actually makes sense for users.

Quoted 12 lines

For new projects yes. However I know of a project (a big project, shipping millions of devices) which picked systemd and glibc long ago and is now running out of space. They already have various solutions to free up Flash (some apps switched to being runtime downloadable, etc) but if/when more savings need to be found then switching from glibc to musl might be a less invasive change than switching from systemd to some other init system. We obviously shouldn't make decisions for OE today based on the historical decisions of one project, but I just want to make the point that real world projects have a lifetime and may end up with a combination of systemd + musl due on past decisions that may not make sense for a new project starting today.

Quoted 6 lines

I am feeling guilty that there are two only partially related topics mixed in this discussion.

In this part of the discussion the topic was what the default (and CI-tested) init system for musl should be - it seems obvious to me that systemd is not what users will usually want to use with musl.

It would be great to have an alternative init system option for smaller devices in oe-core. I don't think collectively we have the development capacity to support it though (it's probably far more work than properly fixing all the currently known issues with systemd on musl...).

Quoted 2 lines

But there is also the topic whether systemd on musl is in a state that it should be made available at all.

I think it's wrong to remove stuff from oe-core just because it isn't perfect or isn't CI tested. Having something in oe-core means there's a common version to collaborate on. If it gets kicked out then development efforts inevitably fragment. Users are generally smart enough to understand the concept of an experimental feature - although we could perhaps do a better job of identifying them. Maybe one day when users can create a custom distro config by running "make menuconfig" there will be an option to enable experimental features and the combination of musl and systemd would be hidden behind that. Until then, perhaps we could add a sanity check warning if musl and systemd are enabled together?

Quoted 2 lines

Does any of these millions of devices have untrusted users or an internet connection?

No local users (if I remember right, everything runs as root) but they do all have broadband internet connections.

Quoted 4 lines

At that point my email that started this thread becomes relevant, the fact that the systemd/musl patches in OE add new security vulnerabilities and other bugs - and none of the systemd-on-musl proponents seems to consider this a problem they have to fix ASAP.

It's certainly a problem and we should try to fix it. It's not at all uncommon that patches to fix build issues with musl, clang, a new version of gcc, etc have a life cycle... a first pass just to fix the build and then updates as issues are found or better solutions get merged upstream. It's the normal process. You could argue that we are sometimes too quick to merge the first pass hacks and too slow to review and update them, but unfortunately that's just a consequence of limited developer resources (and it's always more fun to try to get the latest version of something working than review and cleanup old patches...).

#15 lore raw
Quoted 37 lines · Andre McCurdy
Quoted 29 lines · Adrian Bunk
Quoted 15 lines · Andre McCurdy
Quoted 1 line · Adrian Bunk

On Fri, May 24, 2019 at 11:04:50AM -0700, Khem Raj wrote:

...

Quoted 11 lines
Quoted 3 lines

I think we should put in plan for 2.8 and define the scope, since we are switching
poky defaults to systemd,

Switching glibc builds to systemd as default is a reasonable decision.

Switching musl builds to systemd as default would be very bad.

Combining a tiny C library with a huge init system completely misses the configurations where the tiny C library actually makes sense for users.

Quoted 12 lines

For new projects yes. However I know of a project (a big project, shipping millions of devices) which picked systemd and glibc long ago and is now running out of space. They already have various solutions to free up Flash (some apps switched to being runtime downloadable, etc) but if/when more savings need to be found then switching from glibc to musl might be a less invasive change than switching from systemd to some other init system. We obviously shouldn't make decisions for OE today based on the historical decisions of one project, but I just want to make the point that real world projects have a lifetime and may end up with a combination of systemd + musl due on past decisions that may not make sense for a new project starting today.

Quoted 6 lines

I am feeling guilty that there are two only partially related topics mixed in this discussion.

In this part of the discussion the topic was what the default (and CI-tested) init system for musl should be - it seems obvious to me that systemd is not what users will usually want to use with musl.

Quoted 5 lines

It would be great to have an alternative init system option for smaller devices in oe-core. I don't think collectively we have the development capacity to support it though (it's probably far more work than properly fixing all the currently known issues with systemd on musl...).

Is there development capacity to support musl?

Supporting musl is a real pain across the board,
with new issues all the time.

For really tiny systems you need both a tiny C library and a tiny init system, so not properly supporting the combination of both forces users to use alternative options instead of OE.

Which minimizes the benefits gained by the pains of supporting musl.

Quoted 14 lines
Quoted 2 lines

But there is also the topic whether systemd on musl is in a state that it should be made available at all.

I think it's wrong to remove stuff from oe-core just because it isn't perfect or isn't CI tested. Having something in oe-core means there's a common version to collaborate on. If it gets kicked out then development efforts inevitably fragment. Users are generally smart enough to understand the concept of an experimental feature - although we could perhaps do a better job of identifying them. Maybe one day when users can create a custom distro config by running "make menuconfig" there will be an option to enable experimental features and the combination of musl and systemd would be hidden behind that. Until then, perhaps we could add a sanity check warning if musl and systemd are enabled together?

That would be an improvement.

But it conflicts with the intention to make systemd the default and CI-tested init system for musl.

Quoted 6 lines

...

Quoted 4 lines

At that point my email that started this thread becomes relevant, the fact that the systemd/musl patches in OE add new security vulnerabilities and other bugs - and none of the systemd-on-musl proponents seems to consider this a problem they have to fix ASAP.

Quoted 10 lines

It's certainly a problem and we should try to fix it. It's not at all uncommon that patches to fix build issues with musl, clang, a new version of gcc, etc have a life cycle... a first pass just to fix the build and then updates as issues are found or better solutions get merged upstream. It's the normal process. You could argue that we are sometimes too quick to merge the first pass hacks and too slow to review and update them, but unfortunately that's just a consequence of limited developer resources (and it's always more fun to try to get the latest version of something working than review and cleanup old patches...).

If upstreaming is possible at all.

With systemd on musl there is also the fundamental problem that neither of the upstreams is interested in compromising their software for the other.

Not to blame either of them, there is simply a fundamental conflict between the systemd "use all functionality available" and the musl "be a tiny C library" and "follow standards and provide only some GNU extensions".

One example:

systemd uses qsort_r.
musl upstream doesn't want to add qsort_r since it is nonstandard and BSD and glibc ended up providing incompatible versions. Most C libraries for Linux just follow glibc on that, but musl upstream is not doing this with a reasonable technical justification.

For 0002-don-t-use-glibc-specific-qsort_r.patch (which looks as if it does cause data corruption) upstream is therefore in practice POSIX, unless you manage to convince systemd upstream to use something like gnulib to provide all the functionality they use that is not provided by musl. Which would also be a constant effort you have to makes since systemd upstream would not care about musl when making changes to their code.

Even harder are cases like on_exit(3) (not currently hit by systemd but by other Linux-only software), which might not be implementable as an external function outside the C library.

cu
Adrian

Signature · 4 lines
       "Is there not promise of rain?" Ling Tan asked suddenly out
        of the darkness. There had been need of rain for many days.
       "Only a promise," Lao Er said.
                                       Pearl S. Buck - Dragon Seed
#16 lore raw
Quoted 2 lines · Adrian Bunk

Is there development capacity to support musl?

It would be shame if there wasn't enough developer capacity to support musl but in the end it comes down to what contributors to OE want to work on.

There's a danger that the areas which OE contributors want to work on don't exactly match what the overall Embedded Linux community wants and we alienate potential users (for example we ditched uClibc support not because no one wants to use uClibc any more but because no contributors to OE wanted to spend time on uClibc) but overall the model seems to work OK. I think right now we have enough interest in musl to keep it alive.

Quoted 2 lines

Supporting musl is a real pain across the board,
with new issues all the time.

There are always new issues and bugs to be solved in OE as a consequence of trying to keep all packages up to date. Whether the issues arising from musl are a real pain or a fun new set of problems to solve is mostly a matter of perspective.

Quoted 4 lines

For really tiny systems you need both a tiny C library and a tiny init> system, so not properly supporting the combination of both forces users to use alternative options instead of OE.

Which minimizes the benefits gained by the pains of supporting musl.

A modern tiny init system would be nice to have, but it's not essential or fair to say that musl is useless without one. Many projects, especially tiny ones, manage fine with init scripts and custom process management.

Quoted 39 lines
Quoted 10 lines

It's certainly a problem and we should try to fix it. It's not at all uncommon that patches to fix build issues with musl, clang, a new version of gcc, etc have a life cycle... a first pass just to fix the build and then updates as issues are found or better solutions get merged upstream. It's the normal process. You could argue that we are sometimes too quick to merge the first pass hacks and too slow to review and update them, but unfortunately that's just a consequence of limited developer resources (and it's always more fun to try to get the latest version of something working than review and cleanup old patches...).

If upstreaming is possible at all.

With systemd on musl there is also the fundamental problem that neither of the upstreams is interested in compromising their software for the other.

Not to blame either of them, there is simply a fundamental conflict between the systemd "use all functionality available" and the musl "be a tiny C library" and "follow standards and provide only some GNU extensions".

One example:

systemd uses qsort_r.
musl upstream doesn't want to add qsort_r since it is nonstandard and BSD and glibc ended up providing incompatible versions. Most C libraries for Linux just follow glibc on that, but musl upstream is not doing this with a reasonable technical justification.

For 0002-don-t-use-glibc-specific-qsort_r.patch (which looks as if it does cause data corruption) upstream is therefore in practice POSIX, unless you manage to convince systemd upstream to use something like gnulib to provide all the functionality they use that is not provided by musl. Which would also be a constant effort you have to makes since systemd upstream would not care about musl when making changes to their code.

Quoted 3 lines

Even harder are cases like on_exit(3) (not currently hit by systemd but by other Linux-only software), which might not be implementable as an external function outside the C library.

Upstreamable solutions should always be preferred but if you can accept that OE may need to carry patches which will never be upstreamed (which isn't too much of a stretch given the age of some of the patches we carry for gcc, busybox, etc) then none of the currently known issues with systemd and musl seem like fundamental problems. A few pragmatic patches applied by OE would go a long way to bridging the conflicting goals of the two upstream projects. It's basically the approach we've taken already - the question is just one of improving the patches we already have (and maybe patching musl to add missing functionality instead of only trying to patch systemd to not depend on it).

#17 lore raw
Quoted 9 lines · Andre McCurdy

On Sat, May 25, 2019 at 12:25 AM Adrian Bunk <[email protected]> wrote:
...

Quoted 2 lines

Supporting musl is a real pain across the board,
with new issues all the time.

There are always new issues and bugs to be solved in OE as a consequence of trying to keep all packages up to date. Whether the issues arising from musl are a real pain or a fun new set of problems to solve is mostly a matter of perspective.

Usually someone submits a change, and later gets notified that the change was dropped from master-next due to a musl issue.

That's not fun.

And all these compile errors with musl due to header order are a real WTF, every other library (not limited to C libraries) is now doing headers properly so that any order works. No fun in supporting a broken design.

Quoted 9 lines
Quoted 4 lines

For really tiny systems you need both a tiny C library and a tiny init> system, so not properly supporting the combination of both forces users to use alternative options instead of OE.

Which minimizes the benefits gained by the pains of supporting musl.

A modern tiny init system would be nice to have, but it's not essential or fair to say that musl is useless without one. Many projects, especially tiny ones, manage fine with init scripts and custom process management.

I was not asking for "modern".

If init scripts are not default and CI tested with musl, then init scripts will soon become a broken part of OE.

Quoted 7 lines

...
A few pragmatic patches applied by OE would go a long way to bridging the conflicting goals of the two upstream projects. It's basically the approach we've taken already - the question is just one of improving the patches we already have (and maybe patching musl to add missing functionality instead of only trying to patch systemd to not depend on it).

I already tried patching musl in OE.
The change got reverted.

There are people who think that OE-specific patches to musl are not acceptable, and that it is better to force everyone in OE to patch individual packages all the time instead of adding a not upstreamable patch to musl.

cu
Adrian

Signature · 4 lines
       "Is there not promise of rain?" Ling Tan asked suddenly out
        of the darkness. There had been need of rain for many days.
       "Only a promise," Lao Er said.
                                       Pearl S. Buck - Dragon Seed
#18 lore raw
Quoted 12 lines · Adrian Bunk
Quoted 9 lines · Andre McCurdy

On Sat, May 25, 2019 at 12:25 AM Adrian Bunk <[email protected]> wrote:
...

Quoted 2 lines

Supporting musl is a real pain across the board,
with new issues all the time.

There are always new issues and bugs to be solved in OE as a consequence of trying to keep all packages up to date. Whether the issues arising from musl are a real pain or a fun new set of problems to solve is mostly a matter of perspective.

Quoted 4 lines

Usually someone submits a change, and later gets notified that the change was dropped from master-next due to a musl issue.

That's not fun.

I have re-iterated many times that if your patch regresses musl and you dont want to fix it
when I provide feedback on the patch, say it there. master-next is merely an integration branch
it does not mean that if a patch shows up on master-next then its any good or bad for that matter
its for CI purpose only. master-next queue gets re-aligned all the time. My point to drop the patch
is that submitter would be interested in fixing it which most of the times submitters are kind enough
but I will respect reluctance to fix musl of for that matter larger OE interests that a developer might
not care. Many developers have chipped in to fix regressions in past and I am hopeful they will
chime in future too.

OE is not a distro, its an infrastructure to build custom distros and thats its USP, if we were to stick with
one distro then there are plenty out there which are far well supported.

Quoted 5 lines

And all these compile errors with musl due to header order are a real WTF, every other library (not limited to C libraries) is now doing headers properly so that any order works. No fun in supporting a broken design.

I think these are concerns you must raise in musl community, OE is downstream where we can experiment but not fix these problems, thy should
be fixed in repsective upstreams as much as possible upsttream might have answers or might benefit from this feedback

Quoted 10 lines
Quoted 9 lines
Quoted 4 lines

For really tiny systems you need both a tiny C library and a tiny init> system, so not properly supporting the combination of both forces users to use alternative options instead of OE.

Which minimizes the benefits gained by the pains of supporting musl.

A modern tiny init system would be nice to have, but it's not essential or fair to say that musl is useless without one. Many projects, especially tiny ones, manage fine with init scripts and custom process management.

Quoted 4 lines

I was not asking for "modern".

If init scripts are not default and CI tested with musl, then init scripts will soon become a broken part of OE.

Infact, there are CI system not exposed to community where its testing sysvinit with musl

Quoted 9 lines
Quoted 8 lines

...
A few pragmatic patches applied by OE would go a long way to bridging the conflicting goals of the two upstream projects. It's basically the approach we've taken already - the question is just one of improving the patches we already have (and maybe patching musl to add missing functionality instead of only trying to patch systemd to not depend on it).

Quoted 3 lines

I already tried patching musl in OE.
The change got reverted.

Its costly to change fundamental APIs in libc which are not accepted upstream, especially in libc which will
go into SDKs and will become default API set solely provided by OE thats a huge cost in time. I suggested
you to submit the patch upstream musl, I still encourage you to do so.

Quoted 5 lines

There are people who think that OE-specific patches to musl are not acceptable, and that it is better to force everyone in OE to patch individual packages all the time instead of adding a not upstreamable patch to musl.

I think you understanding is a bit different here, more than one package using a specific non-standard API
does not legitimize it to become part of C library, Ask glibc developer community they would tell you how much libraries and functions they want to drop but can not.

Quoted 10 lines

cu
Adrian

--

"Is there not promise of rain?" Ling Tan asked suddenly out
of the darkness. There had been need of rain for many days.
"Only a promise," Lao Er said.
Pearl S. Buck - Dragon Seed

#19 lore raw
Quoted 11 lines · Khem Raj

On Wed, May 29, 2019 at 9:31 AM Adrian Bunk <[email protected]> wrote:
...

Quoted 3 lines

And all these compile errors with musl due to header order are a real WTF, every other library (not limited to C libraries) is now doing headers properly so that any order works. No fun in supporting a broken design.

I think these are concerns you must raise in musl community, OE is downstream where we can experiment but not fix these problems, thy should
be fixed in repsective upstreams as much as possible upsttream might have answers or might benefit from this feedback

This doesn't work when the root cause is an intentional design decision by upstream, and everyone bringing up such a topic again will just be considered annoying.

Just like asking musl upstream for a __MUSL__ define would not be successful, but would be required e.g. for making the musl patch in webkitgtk upstreamable.

Quoted 8 lines

...

Quoted 7 lines
Quoted 6 lines

A few pragmatic patches applied by OE would go a long way to bridging the conflicting goals of the two upstream projects. It's basically the approach we've taken already - the question is just one of improving the patches we already have (and maybe patching musl to add missing functionality instead of only trying to patch systemd to not depend on it).

Quoted 7 lines
Quoted 2 lines

I already tried patching musl in OE.
The change got reverted.

Its costly to change fundamental APIs in libc which are not accepted upstream, especially in libc which will
go into SDKs and will become default API set solely provided by OE thats a huge cost in time.

A macro in a header does not change the ABI or fundamental APIs.

Quoted 3 lines

I suggested
you to submit the patch upstream musl, I still encourage you to do so.
...

This was a dead end - musl upstream thinks that software shouldn't be doing "loop on EINTR".

At that point the only realistic options are to either patch TEMP_FAILURE_RETRY once into musl, or to patch it into all current and future users.

cu
Adrian

Signature · 4 lines
       "Is there not promise of rain?" Ling Tan asked suddenly out
        of the darkness. There had been need of rain for many days.
       "Only a promise," Lao Er said.
                                       Pearl S. Buck - Dragon Seed
#20 lore raw
In reply to Adrian Bunk
Quoted 11 lines · Adrian Bunk
Quoted 9 lines · Andre McCurdy

On Sat, May 25, 2019 at 12:25 AM Adrian Bunk <[email protected]> wrote:
...

Quoted 2 lines

Supporting musl is a real pain across the board,
with new issues all the time.

There are always new issues and bugs to be solved in OE as a consequence of trying to keep all packages up to date. Whether the issues arising from musl are a real pain or a fun new set of problems to solve is mostly a matter of perspective.

Quoted 4 lines

Usually someone submits a change, and later gets notified that the change was dropped from master-next due to a musl issue.

That's not fun.

Maybe that happens sometimes but I'm not sure that's the "usual" case. Patches to OE get dropped or ignored for many reasons. If your patch gets as far as master-next and then gets dropped with some feedback it's already done better than many others :-) If you're going to contribute to OE you need to listen to the feedback... and be a little persistent!

Quoted 3 lines

And all these compile errors with musl due to header order are a real WTF, every other library (not limited to C libraries) is now doing headers properly so that any order works. No fun in supporting a broken design.

The musl header ordering issues (the potential for duplication of definitions between low level kernel headers and libc headers) is specific to C libraries. It's actually a perfect example of a problem created by a hard line stance from upstream musl which could be completely solved by a small pragmatic patch applied by OE.

Quoted 10 lines
Quoted 9 lines
Quoted 4 lines

For really tiny systems you need both a tiny C library and a tiny init> system, so not properly supporting the combination of both forces users to use alternative options instead of OE.

Which minimizes the benefits gained by the pains of supporting musl.

A modern tiny init system would be nice to have, but it's not essential or fair to say that musl is useless without one. Many projects, especially tiny ones, manage fine with init scripts and custom process management.

Quoted 4 lines

I was not asking for "modern".

If init scripts are not default and CI tested with musl, then init scripts will soon become a broken part of OE.

By that definition (ie anything which is not CI tested is broken) then most of OE is broken. CI testing covers a very narrow set of configurations and if you configure for a real world project you're likely to stray away from them. A few releases back, openssh didn't work for Thumb2 targets because of a conflict between binutils and openssl but wasn't caught by CI testing because (at the time) CI testing was ARMv5 only. In OE 2.7, sqlite3 is broken for big-endian ARM targets (which aren't part of the CI testing). There are many other examples. If you plan to use OE in a real product you'd better plan to do your own CI testing and bug fixing. For many (most?) users that's perfectly acceptable - it's far better to have functionality in OE which is basically sound but may need some tweaks than to strip out everything which hasn't been CI tested.

Quoted 8 lines
Quoted 7 lines

...
A few pragmatic patches applied by OE would go a long way to bridging the conflicting goals of the two upstream projects. It's basically the approach we've taken already - the question is just one of improving the patches we already have (and maybe patching musl to add missing functionality instead of only trying to patch systemd to not depend on it).

Quoted 7 lines

I already tried patching musl in OE.
The change got reverted.

There are people who think that OE-specific patches to musl are not acceptable, and that it is better to force everyone in OE to patch individual packages all the time instead of adding a not upstreamable patch to musl.

Yes, I saw that. The rules on what is and what is not acceptable in OE are not particularly well defined (and then not consistently applied). It's just the way it is...