Problems with Ubuntu 16.04 kernels and wireguard 1.0.20200611

DISCUSSION Gregory ORIOL 2 participants 3 messages Jun 17, 2020 – Jun 18, 2020

Share & tools

Gregory ORIOL

author

2020-06-17 07:51 UTC #0 lore raw

Hi,

Since commit e24c9a9265af40781fa27b5de11dd5b78925c5be to wireguard-linux-compat with a fix for some Ubuntu versions, we are experiencing a problem with some older, but still LTS, versions of Ubuntu 16.04: older kernels 4.4.0-148, 4.4.0-166 fail to build with wireguard 1.0.20200611

So, any system running an (or still having an installed) "older" kernel and doing an apt upgrade to install wireguard 1.0.20200611 would fail during the wireguard dkms step, while trying to build wireguard for all the kernels available.

The problem gets more problematic when a newer kernel 4.4.0-184 gets installed with the same apt upgrade: then, trying to downgrade wireguard also fails; none of the 1.0.20200611 or 1.0.20200520 versions work anymore with this combination of old/new kernels...

To recap :

# wireguard 1.0.20200520
 - ok with kernels 4.4.0-148, 4.4.0-166
 - fails with kernel 4.4.0-184
# wireguard 1.0.20200611
 - fails with kernels 4.4.0-148, 4.4.0-166
 - ok with kernel 4.4.0-184

(nb: we see it now with -184 but it could have started with an earlier version)

We could partially fix this by manually getting each deb/src and doing dkms install:
dkms install wireguard/1.0.20200520 -k 4.4.0-148-generic dkms install wireguard/1.0.20200611 -k 4.4.0-184-generic
...
But apt upgrade is still broken.

While we could boot onto the newer kernel and remove the older ones to get rid of the problem, this situation would prevent from having a "previous working" kernel on the system, which is not very safe.

Could there be a fix for this made to wireguard-linux-compat for those versions?

Best regards,
Greg

Jason A. Donenfeld

2020-06-17 08:13 UTC #1 lore raw

Hi Gregory,

Quoted 34 lines · Gregory ORIOL

Since commit e24c9a9265af40781fa27b5de11dd5b78925c5be to wireguard-linux-compat with a fix for some Ubuntu versions, we are experiencing a problem with some older, but still LTS, versions of Ubuntu 16.04: older kernels 4.4.0-148, 4.4.0-166 fail to build with wireguard 1.0.20200611
So, any system running an (or still having an installed) "older" kernel and doing an apt upgrade to install wireguard 1.0.20200611 would fail during the wireguard dkms step, while trying to build wireguard for all the kernels available.
The problem gets more problematic when a newer kernel 4.4.0-184 gets installed with the same apt upgrade: then, trying to downgrade wireguard also fails; none of the 1.0.20200611 or 1.0.20200520 versions work anymore with this combination of old/new kernels...
To recap :
# wireguard 1.0.20200520
 - ok with kernels 4.4.0-148, 4.4.0-166
 - fails with kernel 4.4.0-184
# wireguard 1.0.20200611
 - fails with kernels 4.4.0-148, 4.4.0-166
 - ok with kernel 4.4.0-184
(nb: we see it now with -184 but it could have started with an earlier version)
We could partially fix this by manually getting each deb/src and doing dkms install:
dkms install wireguard/1.0.20200520 -k 4.4.0-148-generic dkms install wireguard/1.0.20200611 -k 4.4.0-184-generic
...
But apt upgrade is still broken.

Quoted 6 lines

While we could boot onto the newer kernel and remove the older ones to get rid of the problem, this situation would prevent from having a "previous working" kernel on the system, which is not very safe.
Could there be a fix for this made to wireguard-linux-compat for those versions?

Unfortunately, I don't have a super good solution for you right now. The wireguard-linux-compat repo is developed against the latest Ubuntu kernels that they put out once every three weeks. You can see them being tested at the bottom of
<https://www.wireguard.com/build-status/>. The backport against upstream mainline kernels is z-granular (for an x.y.z versioning scheme), but Ubuntu's release cycle and versioning scheme is a bit too chaotic to make it reasonable to try to manage all the differences between their kernels every three weeks. So for distro kernels -- Ubuntu, RHEL, Debian, etc -- we typically just develop against the latest one, and try to make sure that we release it at the right time so users aren't caught with no working version. This means, unfortunately, that when there are badly breaking changes, like in this last cycle, you have to uninstall the old kernels or mask them from dkms, in order to get dkms to avoid building for them and only building for the new kernel. There might be other more complicated solutions that closely track version dependencies or do compile time feature probing, but that comes with a maintenance burden far too arduous for a distro frankenkernel.

But there is hope!

Canonical is adding WireGuard to 18.04 and 16.04, and this is coming in two steps:

Step 1) The wireguard-dkms and wireguard-tools packages will be added to the package archives, so that you won't have to use the PPA. This means that Canonical's kernel team will include wireguard-dkms in their development tests, so that they won't accidentally ship kernels with build breakage, like what you experienced last week.

Step 2) The wireguard-dkms package will get built by Canonical, signed, and shipped alongside the other modules, so that you won't have to install wireguard-dkms, and it will just come out of the box with the normal kernel updates. This is already the case with 20.04 and 19.10. They're working on it now for 18.04, and I really really hope to see that happen by the next cycle. And maybe if we ask apw (CC'd) nicely, he'll even do it for 16.04 too.

Regards,
Jason

Gregory ORIOL

author

2020-06-18 00:46 UTC #2 lore raw

Hi,

Thanks for your feedback!

Couldn't that issue be fixed by adapting the IF condition that has been changed in commit e24c9a9265af40781fa27b5de11dd5b78925c5be?
I know it's not very practical to support/check this, plus it's not wireguard's fault, but that will likely cause trouble to other users too.

We could obviously wait a few weeks with a broken apt upgrade state until a new new kernel version is released and then switch to it, and remove the old ones, so then we will have an n-1 kernel -184 working and the new-new one as current. But that feels like a very unstable situation for our systems until then...

Regards,
Gregory

Jason A. Donenfeld a écrit le 17/06/2020 à 10:13 :

Quoted 84 lines

Hi Gregory,
Quoted 34 lines · Gregory ORIOL
Since commit e24c9a9265af40781fa27b5de11dd5b78925c5be to wireguard-linux-compat with a fix for some Ubuntu versions, we are experiencing a problem with some older, but still LTS, versions of Ubuntu 16.04: older kernels 4.4.0-148, 4.4.0-166 fail to build with wireguard 1.0.20200611
So, any system running an (or still having an installed) "older" kernel and doing an apt upgrade to install wireguard 1.0.20200611 would fail during the wireguard dkms step, while trying to build wireguard for all the kernels available.
The problem gets more problematic when a newer kernel 4.4.0-184 gets installed with the same apt upgrade: then, trying to downgrade wireguard also fails; none of the 1.0.20200611 or 1.0.20200520 versions work anymore with this combination of old/new kernels...
To recap :
# wireguard 1.0.20200520
 - ok with kernels 4.4.0-148, 4.4.0-166
 - fails with kernel 4.4.0-184
# wireguard 1.0.20200611
 - fails with kernels 4.4.0-148, 4.4.0-166
 - ok with kernel 4.4.0-184
(nb: we see it now with -184 but it could have started with an earlier version)
We could partially fix this by manually getting each deb/src and doing dkms install:
dkms install wireguard/1.0.20200520 -k 4.4.0-148-generic dkms install wireguard/1.0.20200611 -k 4.4.0-184-generic
...
But apt upgrade is still broken.
Quoted 6 lines
While we could boot onto the newer kernel and remove the older ones to get rid of the problem, this situation would prevent from having a "previous working" kernel on the system, which is not very safe.
Could there be a fix for this made to wireguard-linux-compat for those versions?
Unfortunately, I don't have a super good solution for you right now. The wireguard-linux-compat repo is developed against the latest Ubuntu kernels that they put out once every three weeks. You can see them being tested at the bottom of
<https://www.wireguard.com/build-status/>. The backport against upstream mainline kernels is z-granular (for an x.y.z versioning scheme), but Ubuntu's release cycle and versioning scheme is a bit too chaotic to make it reasonable to try to manage all the differences between their kernels every three weeks. So for distro kernels -- Ubuntu, RHEL, Debian, etc -- we typically just develop against the latest one, and try to make sure that we release it at the right time so users aren't caught with no working version. This means, unfortunately, that when there are badly breaking changes, like in this last cycle, you have to uninstall the old kernels or mask them from dkms, in order to get dkms to avoid building for them and only building for the new kernel. There might be other more complicated solutions that closely track version dependencies or do compile time feature probing, but that comes with a maintenance burden far too arduous for a distro frankenkernel.
But there is hope!
Canonical is adding WireGuard to 18.04 and 16.04, and this is coming in two steps:
Step 1) The wireguard-dkms and wireguard-tools packages will be added to the package archives, so that you won't have to use the PPA. This means that Canonical's kernel team will include wireguard-dkms in their development tests, so that they won't accidentally ship kernels with build breakage, like what you experienced last week.
Step 2) The wireguard-dkms package will get built by Canonical, signed, and shipped alongside the other modules, so that you won't have to install wireguard-dkms, and it will just come out of the box with the normal kernel updates. This is already the case with 20.04 and 19.10. They're working on it now for 18.04, and I really really hope to see that happen by the next cycle. And maybe if we ask apw (CC'd) nicely, he'll even do it for 16.04 too.
Regards,
Jason