Skip to contentSkip to conversation

Problem with IPv6 privacy addresses in 7.0

DISCUSSION Chris Adams 9 participants 32 messages May 21 – May 29

Subsystems (3)
lore
Share & tools
lore
#0 lore raw

My normal use of my desktop system (Fedora 43) is to suspend at night and wake in the morning, and then immediately SSH to my remote server, where I stay logged in. I use NetworkManager and have ipv6.ip6-privacy set to prefer-temp-addr, and SSH configured to use SSH keepalives.

When I upgraded to kernel 7.0, I started having an issue with this - my SSH session gets dropped, usually several times, in the first hour or so, then I usually don't have any problem the rest of the day. If I run an IPv4 session at the same time, that seems to be fine, so I don't think it's a network issue (I know that dual-stack doesn't always take the same path).

What seems to be happening is that privacy addresses are removed while the SSH session is still using them, before the timeout even. I wrote a script to watch public v6 addresses being added and removed, and this is what I've seen so far today (the number at the end is the valid_lft seconds), with the public prefix masked:

2026-05-21 07:39:17 removed xx::f4f/128 4673
2026-05-21 07:39:20 added xx::f4f/128 5400
2026-05-21 07:41:20 removed xx:3e8c:f8ff:fe60:1d5a/64 4922
2026-05-21 07:41:20 removed xx:7cb1:c518:1be0:d81d/64 4922
2026-05-21 07:41:23 added xx:3e8c:f8ff:fe60:1d5a/64 5398
2026-05-21 07:41:23 added xx:596a:f6f5:67b2:1d8f/64 5398
2026-05-21 08:14:43 added xx:fac3:61f6:ad18:d712/64 4987
2026-05-21 08:14:43 removed xx:596a:f6f5:67b2:1d8f/64 4991
2026-05-21 08:30:26 added xx:84b4:244e:bb14:94fd/64 5398
2026-05-21 08:30:26 removed xx:fac3:61f6:ad18:d712/64 5120

I woke the system at 07:39:08 and SSHed at 07:39:39, which used the d81d source address. That dropped in 2 minutes and I reconnected, which used the 1d8f address. That dropped at 08:14:43, I didn't notice right away, I reconnected at 08:31:28 which used the 94fd address.

Signature · 1 line
Chris Adams <[email protected]>
maintainer
#1 lore raw
Quoted 29 lines · On Thu, 21 May 2026 08:53:10 -0500 Chris Adams wrote:

My normal use of my desktop system (Fedora 43) is to suspend at night and wake in the morning, and then immediately SSH to my remote server, where I stay logged in. I use NetworkManager and have ipv6.ip6-privacy set to prefer-temp-addr, and SSH configured to use SSH keepalives.

When I upgraded to kernel 7.0, I started having an issue with this - my SSH session gets dropped, usually several times, in the first hour or so, then I usually don't have any problem the rest of the day. If I run an IPv4 session at the same time, that seems to be fine, so I don't think it's a network issue (I know that dual-stack doesn't always take the same path).

What seems to be happening is that privacy addresses are removed while the SSH session is still using them, before the timeout even. I wrote a script to watch public v6 addresses being added and removed, and this is what I've seen so far today (the number at the end is the valid_lft seconds), with the public prefix masked:

2026-05-21 07:39:17 removed xx::f4f/128 4673
2026-05-21 07:39:20 added xx::f4f/128 5400
2026-05-21 07:41:20 removed xx:3e8c:f8ff:fe60:1d5a/64 4922
2026-05-21 07:41:20 removed xx:7cb1:c518:1be0:d81d/64 4922
2026-05-21 07:41:23 added xx:3e8c:f8ff:fe60:1d5a/64 5398
2026-05-21 07:41:23 added xx:596a:f6f5:67b2:1d8f/64 5398
2026-05-21 08:14:43 added xx:fac3:61f6:ad18:d712/64 4987
2026-05-21 08:14:43 removed xx:596a:f6f5:67b2:1d8f/64 4991
2026-05-21 08:30:26 added xx:84b4:244e:bb14:94fd/64 5398
2026-05-21 08:30:26 removed xx:fac3:61f6:ad18:d712/64 5120
Quoted 4 lines

I woke the system at 07:39:08 and SSHed at 07:39:39, which used the d81d source address. That dropped in 2 minutes and I reconnected, which used the 1d8f address. That dropped at 08:14:43, I didn't notice right away, I reconnected at 08:31:28 which used the 94fd address.

Hi! Adding more people to CC. Do you know if you upgraded from 6.18 or 6.19?

Would you be able to try testing with some commits reverted? On a quick look the candidates would be:

cb3de96eea66 ("ipv6: preserve insertion order for same-scope addresses") c7dc5b522882 ("ipv6: clean up routes when manually removing address with a lifetime")

Less likely:
5023479627e3 ("ipv6: Switch to higher-level SHA-1 functions")
9e371b0ba7f5 ("ipv6: addrconf: reduce default temp_valid_lft to 2 days") 6af51e9f3133 ("ipv6: Remove permanent routes from tb6_gc_hlist when all exceptions expire.")

#2 lore raw

Heh, I opened a Fedora BZ on this and JUST updated it... RHBZ 2480928

Quoted 30 lines · Jakub Kicinski
Quoted 29 lines · On Thu, 21 May 2026 08:53:10 -0500 Chris Adams wrote:

My normal use of my desktop system (Fedora 43) is to suspend at night and wake in the morning, and then immediately SSH to my remote server, where I stay logged in. I use NetworkManager and have ipv6.ip6-privacy set to prefer-temp-addr, and SSH configured to use SSH keepalives.

When I upgraded to kernel 7.0, I started having an issue with this - my SSH session gets dropped, usually several times, in the first hour or so, then I usually don't have any problem the rest of the day. If I run an IPv4 session at the same time, that seems to be fine, so I don't think it's a network issue (I know that dual-stack doesn't always take the same path).

What seems to be happening is that privacy addresses are removed while the SSH session is still using them, before the timeout even. I wrote a script to watch public v6 addresses being added and removed, and this is what I've seen so far today (the number at the end is the valid_lft seconds), with the public prefix masked:

2026-05-21 07:39:17 removed xx::f4f/128 4673
2026-05-21 07:39:20 added xx::f4f/128 5400
2026-05-21 07:41:20 removed xx:3e8c:f8ff:fe60:1d5a/64 4922
2026-05-21 07:41:20 removed xx:7cb1:c518:1be0:d81d/64 4922
2026-05-21 07:41:23 added xx:3e8c:f8ff:fe60:1d5a/64 5398
2026-05-21 07:41:23 added xx:596a:f6f5:67b2:1d8f/64 5398
2026-05-21 08:14:43 added xx:fac3:61f6:ad18:d712/64 4987
2026-05-21 08:14:43 removed xx:596a:f6f5:67b2:1d8f/64 4991
2026-05-21 08:30:26 added xx:84b4:244e:bb14:94fd/64 5398
2026-05-21 08:30:26 removed xx:fac3:61f6:ad18:d712/64 5120
Quoted 7 lines
Quoted 4 lines

I woke the system at 07:39:08 and SSHed at 07:39:39, which used the d81d source address. That dropped in 2 minutes and I reconnected, which used the 1d8f address. That dropped at 08:14:43, I didn't notice right away, I reconnected at 08:31:28 which used the 94fd address.

Hi! Adding more people to CC. Do you know if you upgraded from 6.18 or 6.19?

It was 6.19 to 7.0.

Quoted 4 lines

Would you be able to try testing with some commits reverted? On a quick look the candidates would be:

cb3de96eea66 ("ipv6: preserve insertion order for same-scope addresses")

It's this one.

I figured out that it happens after stopping a VM (and I usually start/stop a VM for a bit in the morning, which is why it happened more than once). So I set up a VM with a nested VM, running up-to-date Fedora 44, and then was able to bisect pretty easily, and it landed on this commit.

Fedora is using NetworkManager, and IIRC NM does some part of privacy address management (right?). NM didn't change, so maybe this commit is confusing something in NM?

Signature · 1 line
Chris Adams <[email protected]>
maintainer
#3 lore raw
Quoted 11 lines · On Tue, 26 May 2026 20:06:41 -0500 Chris Adams wrote:
Quoted 2 lines

Hi! Adding more people to CC. Do you know if you upgraded from 6.18 or 6.19?

It was 6.19 to 7.0.

Quoted 4 lines

Would you be able to try testing with some commits reverted? On a quick look the candidates would be:

cb3de96eea66 ("ipv6: preserve insertion order for same-scope addresses")

It's this one.

Phew, the second one was mine :)

Quoted 9 lines

I figured out that it happens after stopping a VM (and I usually start/stop a VM for a bit in the morning, which is why it happened more than once). So I set up a VM with a nested VM, running up-to-date Fedora 44, and then was able to bisect pretty easily, and it landed on this commit.

Fedora is using NetworkManager, and IIRC NM does some part of privacy address management (right?). NM didn't change, so maybe this commit is confusing something in NM?

Sounds plausible, pretty sure we knew this commit was risky to begin with, but we had no direct proof that it'd break real life users.

Revert is the right course of action here. Would you be willing/able to send the revert with your problem description and a Fixes tag pointing to the reverted commit?

#4 lore raw
Quoted 25 lines · Jakub Kicinski
Quoted 11 lines · On Tue, 26 May 2026 20:06:41 -0500 Chris Adams wrote:
Quoted 2 lines

Hi! Adding more people to CC. Do you know if you upgraded from 6.18 or 6.19?

It was 6.19 to 7.0.

Quoted 4 lines

Would you be able to try testing with some commits reverted? On a quick look the candidates would be:

cb3de96eea66 ("ipv6: preserve insertion order for same-scope addresses")

It's this one.

Phew, the second one was mine :)

Quoted 9 lines

I figured out that it happens after stopping a VM (and I usually start/stop a VM for a bit in the morning, which is why it happened more than once). So I set up a VM with a nested VM, running up-to-date Fedora 44, and then was able to bisect pretty easily, and it landed on this commit.

Fedora is using NetworkManager, and IIRC NM does some part of privacy address management (right?). NM didn't change, so maybe this commit is confusing something in NM?

Quoted 6 lines

Sounds plausible, pretty sure we knew this commit was risky to begin with, but we had no direct proof that it'd break real life users.

Revert is the right course of action here. Would you be willing/able to send the revert with your problem description and a Fixes tag pointing to the reverted commit?

So... I'm not very git-fluent and haven't submitted a kernel patch in many years, so really am not sure of the necessary/proper steps.

Signature · 1 line
Chris Adams <[email protected]>
#5 lore raw
Quoted 29 lines · Chris Adams
Quoted 28 lines · Jakub Kicinski
Quoted 11 lines · On Tue, 26 May 2026 20:06:41 -0500 Chris Adams wrote:
Quoted 2 lines

Hi! Adding more people to CC. Do you know if you upgraded from 6.18 or 6.19?

It was 6.19 to 7.0.

Quoted 4 lines

Would you be able to try testing with some commits reverted? On a quick look the candidates would be:

cb3de96eea66 ("ipv6: preserve insertion order for same-scope addresses")

It's this one.

Phew, the second one was mine :)

Quoted 9 lines

I figured out that it happens after stopping a VM (and I usually start/stop a VM for a bit in the morning, which is why it happened more than once). So I set up a VM with a nested VM, running up-to-date Fedora 44, and then was able to bisect pretty easily, and it landed on this commit.

Fedora is using NetworkManager, and IIRC NM does some part of privacy address management (right?). NM didn't change, so maybe this commit is confusing something in NM?

Sounds plausible, pretty sure we knew this commit was risky to begin with, but we had no direct proof that it'd break real life users.

Quoted 6 lines
Quoted 3 lines

Revert is the right course of action here. Would you be willing/able to send the revert with your problem description and a Fixes tag pointing to the reverted commit?

So... I'm not very git-fluent and haven't submitted a kernel patch in many years, so really am not sure of the necessary/proper steps.

I don't mind to handle it. I will CC you, is that fine for you?

Thanks for the report btw!

maintainer
#6 lore raw
In reply to Chris Adams
Quoted 13 lines · On Wed, 27 May 2026 16:13:56 -0500 Chris Adams wrote:
Quoted 13 lines
Quoted 9 lines

I figured out that it happens after stopping a VM (and I usually start/stop a VM for a bit in the morning, which is why it happened more than once). So I set up a VM with a nested VM, running up-to-date Fedora 44, and then was able to bisect pretty easily, and it landed on this commit.

Fedora is using NetworkManager, and IIRC NM does some part of privacy address management (right?). NM didn't change, so maybe this commit is confusing something in NM?

Sounds plausible, pretty sure we knew this commit was risky to begin with, but we had no direct proof that it'd break real life users.

Quoted 6 lines
Quoted 3 lines

Revert is the right course of action here. Would you be willing/able to send the revert with your problem description and a Fixes tag pointing to the reverted commit?

So... I'm not very git-fluent and haven't submitted a kernel patch in many years, so really am not sure of the necessary/proper steps.

I wonder how you managed to bisect it then :)

git clone ..
git revert cb3de96eea66

you can use db5dadb562cabb6 for inspiration on how to format the commit message. You can use an LLM to help, they are plenty capable enough to handle a revert. For sending either git send-email or you can use b4 relay. https://b4.docs.kernel.org/en/latest/

If you prefer us to handle this - that's perfectly fine, just let me know. I'm asking if you _want_ to handle it, as some folks like the sense of accomplishment of having upstream commits.

#7 lore raw
In reply to Jakub Kicinski
Quoted 25 lines · Jakub Kicinski
Quoted 11 lines · On Tue, 26 May 2026 20:06:41 -0500 Chris Adams wrote:
Quoted 2 lines

Hi! Adding more people to CC. Do you know if you upgraded from 6.18 or 6.19?

It was 6.19 to 7.0.

Quoted 4 lines

Would you be able to try testing with some commits reverted? On a quick look the candidates would be:

cb3de96eea66 ("ipv6: preserve insertion order for same-scope addresses")

It's this one.

Phew, the second one was mine :)

Quoted 9 lines

I figured out that it happens after stopping a VM (and I usually start/stop a VM for a bit in the morning, which is why it happened more than once). So I set up a VM with a nested VM, running up-to-date Fedora 44, and then was able to bisect pretty easily, and it landed on this commit.

Fedora is using NetworkManager, and IIRC NM does some part of privacy address management (right?). NM didn't change, so maybe this commit is confusing something in NM?

Quoted 6 lines

Sounds plausible, pretty sure we knew this commit was risky to begin with, but we had no direct proof that it'd break real life users.

Revert is the right course of action here. Would you be willing/able to send the revert with your problem description and a Fixes tag pointing to the reverted commit?

I did want to add a little more test note:

It's definitely an interaction with NetworkManager. If I stop NM and run my VM start/stop test, nothing unexpected happens after. If NM is running and I do my VM test, when the next router advertisement is received, NM replaces the privacy addresses.

Signature · 1 line
Chris Adams <[email protected]>
#8 lore raw
Quoted 33 lines · Chris Adams
Quoted 25 lines · Jakub Kicinski
Quoted 11 lines · On Tue, 26 May 2026 20:06:41 -0500 Chris Adams wrote:
Quoted 2 lines

Hi! Adding more people to CC. Do you know if you upgraded from 6.18 or 6.19?

It was 6.19 to 7.0.

Quoted 4 lines

Would you be able to try testing with some commits reverted? On a quick look the candidates would be:

cb3de96eea66 ("ipv6: preserve insertion order for same-scope addresses")

It's this one.

Phew, the second one was mine :)

Quoted 9 lines

I figured out that it happens after stopping a VM (and I usually start/stop a VM for a bit in the morning, which is why it happened more than once). So I set up a VM with a nested VM, running up-to-date Fedora 44, and then was able to bisect pretty easily, and it landed on this commit.

Fedora is using NetworkManager, and IIRC NM does some part of privacy address management (right?). NM didn't change, so maybe this commit is confusing something in NM?

Quoted 6 lines

Sounds plausible, pretty sure we knew this commit was risky to begin with, but we had no direct proof that it'd break real life users.

Revert is the right course of action here. Would you be willing/able to send the revert with your problem description and a Fixes tag pointing to the reverted commit?

Quoted 7 lines

I did want to add a little more test note:

It's definitely an interaction with NetworkManager. If I stop NM and run my VM start/stop test, nothing unexpected happens after. If NM is running and I do my VM test, when the next router advertisement is received, NM replaces the privacy addresses.

As someone that is experienced in NetworkManager, I can confirm it is related. NetworkManager is querying the IPv6 address and when the connection is configured with ipv6.ip6-privacy=2 (prefer-temp-addr), NetworkManager creates a route to make the system use the temporary address for outgoing connections by default.

If the order is messed up, the address picked will likely be too. One could argue that this is partially fault of NetworkManager and that it should check the timestamps or preferred times rather than order.. but well, the rule is "do not break userspace".

I hope this clarifies things.

Thanks,
Fernando.

maintainer
#9 lore raw
Quoted 23 lines · On Wed, 27 May 2026 23:59:47 +0200 Fernando Fernandez Mancera wrote:
Quoted 8 lines · Chris Adams
Quoted 6 lines · Jakub Kicinski

Sounds plausible, pretty sure we knew this commit was risky to begin with, but we had no direct proof that it'd break real life users.

Revert is the right course of action here. Would you be willing/able to send the revert with your problem description and a Fixes tag pointing to the reverted commit?

Quoted 7 lines

I did want to add a little more test note:

It's definitely an interaction with NetworkManager. If I stop NM and run my VM start/stop test, nothing unexpected happens after. If NM is running and I do my VM test, when the next router advertisement is received, NM replaces the privacy addresses.

As someone that is experienced in NetworkManager, I can confirm it is related. NetworkManager is querying the IPv6 address and when the connection is configured with ipv6.ip6-privacy=2 (prefer-temp-addr), NetworkManager creates a route to make the system use the temporary address for outgoing connections by default.

Quoted 4 lines

If the order is messed up, the address picked will likely be too. One could argue that this is partially fault of NetworkManager and that it should check the timestamps or preferred times rather than order.. but well, the rule is "do not break userspace".

Right, plus the justification for the change wasn't very strong in the first place. My internal compass is still pointing towards a revert.

Quoted 1 line

I hope this clarifies things.

#10 lore raw
In reply to Fernando Fernandez Mancera

Hi Fernando, Jakub,

Quoted 48 lines · On Wed, 27 May 2026 23:59:47 +0200 Fernando Fernandez Mancera…
Quoted 33 lines · Chris Adams
Quoted 25 lines · Jakub Kicinski
Quoted 11 lines · On Tue, 26 May 2026 20:06:41 -0500 Chris Adams wrote:
Quoted 2 lines

Hi! Adding more people to CC. Do you know if you upgraded from 6.18 or 6.19?

It was 6.19 to 7.0.

Quoted 4 lines

Would you be able to try testing with some commits reverted? On a quick look the candidates would be:

cb3de96eea66 ("ipv6: preserve insertion order for same-scope addresses")

It's this one.

Phew, the second one was mine :)

Quoted 9 lines

I figured out that it happens after stopping a VM (and I usually start/stop a VM for a bit in the morning, which is why it happened more than once). So I set up a VM with a nested VM, running up-to-date Fedora 44, and then was able to bisect pretty easily, and it landed on this commit.

Fedora is using NetworkManager, and IIRC NM does some part of privacy address management (right?). NM didn't change, so maybe this commit is confusing something in NM?

Quoted 6 lines

Sounds plausible, pretty sure we knew this commit was risky to begin with, but we had no direct proof that it'd break real life users.

Revert is the right course of action here. Would you be willing/able to send the revert with your problem description and a Fixes tag pointing to the reverted commit?

Quoted 7 lines

I did want to add a little more test note:

It's definitely an interaction with NetworkManager. If I stop NM and run my VM start/stop test, nothing unexpected happens after. If NM is running and I do my VM test, when the next router advertisement is received, NM replaces the privacy addresses.

As someone that is experienced in NetworkManager, I can confirm it is related. NetworkManager is querying the IPv6 address and when the connection is configured with ipv6.ip6-privacy=2 (prefer-temp-addr), NetworkManager creates a route to make the system use the temporary address for outgoing connections by default.

Quoted 6 lines

If the order is messed up, the address picked will likely be too. One could argue that this is partially fault of NetworkManager and that it should check the timestamps or preferred times rather than order.. but well, the rule is "do not break userspace".

I hope this clarifies things.

Not entirely. I'm looking into this right now, but note that the purpose of that commit is to *preserve* the order of addresses as they were inserted, not to mess it up.

Before that, addresses were stored and returned via netlink *reversed* compared to the insertion, which is rather surprising, also because it's the opposite of what we do for IPv4 addresses, and caused issues with pasta(1) as it copies addresses into containers in the same order as reported via netlink.

Do you happen to know exactly in which way the order happens to be wrong now?

Also note that userspace was broken and fixed a couple of times:

https://bugs.passt.top/show_bug.cgi?id=175#c10

so I would look into fixing that properly if doable. If it takes too long and this is causing issues for a lot of people meanwhile of course it might make sense to revert, but I'd give it a quick try at least.

Signature · 1 line
Stefano
#11 lore raw
Quoted 64 lines · Stefano Brivio

Hi Fernando, Jakub,

Quoted 48 lines · On Wed, 27 May 2026 23:59:47 +0200 Fernando Fernandez Mancera…
Quoted 33 lines · Chris Adams
Quoted 25 lines · Jakub Kicinski
Quoted 11 lines · On Tue, 26 May 2026 20:06:41 -0500 Chris Adams wrote:
Quoted 2 lines

Hi! Adding more people to CC. Do you know if you upgraded from 6.18 or 6.19?

It was 6.19 to 7.0.

Quoted 4 lines

Would you be able to try testing with some commits reverted? On a quick look the candidates would be:

cb3de96eea66 ("ipv6: preserve insertion order for same-scope addresses")

It's this one.

Phew, the second one was mine :)

Quoted 9 lines

I figured out that it happens after stopping a VM (and I usually start/stop a VM for a bit in the morning, which is why it happened more than once). So I set up a VM with a nested VM, running up-to-date Fedora 44, and then was able to bisect pretty easily, and it landed on this commit.

Fedora is using NetworkManager, and IIRC NM does some part of privacy address management (right?). NM didn't change, so maybe this commit is confusing something in NM?

Quoted 6 lines

Sounds plausible, pretty sure we knew this commit was risky to begin with, but we had no direct proof that it'd break real life users.

Revert is the right course of action here. Would you be willing/able to send the revert with your problem description and a Fixes tag pointing to the reverted commit?

Quoted 7 lines

I did want to add a little more test note:

It's definitely an interaction with NetworkManager. If I stop NM and run my VM start/stop test, nothing unexpected happens after. If NM is running and I do my VM test, when the next router advertisement is received, NM replaces the privacy addresses.

As someone that is experienced in NetworkManager, I can confirm it is related. NetworkManager is querying the IPv6 address and when the connection is configured with ipv6.ip6-privacy=2 (prefer-temp-addr), NetworkManager creates a route to make the system use the temporary address for outgoing connections by default.

Quoted 6 lines

If the order is messed up, the address picked will likely be too. One could argue that this is partially fault of NetworkManager and that it should check the timestamps or preferred times rather than order.. but well, the rule is "do not break userspace".

I hope this clarifies things.

Not entirely. I'm looking into this right now, but note that the purpose of that commit is to *preserve* the order of addresses as they were inserted, not to mess it up.

Quoted 6 lines

Before that, addresses were stored and returned via netlink *reversed* compared to the insertion, which is rather surprising, also because it's the opposite of what we do for IPv4 addresses, and caused issues with pasta(1) as it copies addresses into containers in the same order as reported via netlink.

Right, to make it clear. I agree with you here. The current order is the *right* or rather intuitive order.

Quoted 3 lines

Do you happen to know exactly in which way the order happens to be wrong now?

It is not wrong, but userspace applications were parsing addresses and expecting them in the previous order whether it was correct or not. Now, that parsing is broken because they are coming in an unexpected (maybe more intuitive) order.

FWIW; Iñigo Huguet (CC'ed) pointed out that this is breaking a NetworkManager test and they were going to report it.

Quoted 8 lines

Also note that userspace was broken and fixed a couple of times:

   https://bugs.passt.top/show_bug.cgi?id=175#c10

so I would look into fixing that properly if doable. If it takes too long and this is causing issues for a lot of people meanwhile of course it might make sense to revert, but I'd give it a quick try at least.

I think the solution shouldn't modify the order of the list by default. Maybe we can introduce a flag for the dump operation?

#12 lore raw
Quoted 74 lines · On Thu, 28 May 2026 12:46:05 +0200 Fernando Fernandez Mancera…
Quoted 64 lines · Stefano Brivio

Hi Fernando, Jakub,

Quoted 48 lines · On Wed, 27 May 2026 23:59:47 +0200 Fernando Fernandez Mancera…
Quoted 33 lines · Chris Adams
Quoted 25 lines · Jakub Kicinski
Quoted 11 lines · On Tue, 26 May 2026 20:06:41 -0500 Chris Adams wrote:
Quoted 2 lines

Hi! Adding more people to CC. Do you know if you upgraded from 6.18 or 6.19?

It was 6.19 to 7.0.

Quoted 4 lines

Would you be able to try testing with some commits reverted? On a quick look the candidates would be:

cb3de96eea66 ("ipv6: preserve insertion order for same-scope addresses")

It's this one.

Phew, the second one was mine :)

Quoted 9 lines

I figured out that it happens after stopping a VM (and I usually start/stop a VM for a bit in the morning, which is why it happened more than once). So I set up a VM with a nested VM, running up-to-date Fedora 44, and then was able to bisect pretty easily, and it landed on this commit.

Fedora is using NetworkManager, and IIRC NM does some part of privacy address management (right?). NM didn't change, so maybe this commit is confusing something in NM?

Quoted 6 lines

Sounds plausible, pretty sure we knew this commit was risky to begin with, but we had no direct proof that it'd break real life users.

Revert is the right course of action here. Would you be willing/able to send the revert with your problem description and a Fixes tag pointing to the reverted commit?

Quoted 7 lines

I did want to add a little more test note:

It's definitely an interaction with NetworkManager. If I stop NM and run my VM start/stop test, nothing unexpected happens after. If NM is running and I do my VM test, when the next router advertisement is received, NM replaces the privacy addresses.

As someone that is experienced in NetworkManager, I can confirm it is related. NetworkManager is querying the IPv6 address and when the connection is configured with ipv6.ip6-privacy=2 (prefer-temp-addr), NetworkManager creates a route to make the system use the temporary address for outgoing connections by default.

Quoted 6 lines

If the order is messed up, the address picked will likely be too. One could argue that this is partially fault of NetworkManager and that it should check the timestamps or preferred times rather than order.. but well, the rule is "do not break userspace".

I hope this clarifies things.

Not entirely. I'm looking into this right now, but note that the purpose of that commit is to *preserve* the order of addresses as they were inserted, not to mess it up.

Quoted 5 lines

Before that, addresses were stored and returned via netlink *reversed* compared to the insertion, which is rather surprising, also because it's the opposite of what we do for IPv4 addresses, and caused issues with pasta(1) as it copies addresses into containers in the same order as reported via netlink.

Right, to make it clear. I agree with you here. The current order is the *right* or rather intuitive order.

Quoted 7 lines
Quoted 2 lines

Do you happen to know exactly in which way the order happens to be wrong now?

It is not wrong, but userspace applications were parsing addresses and expecting them in the previous order whether it was correct or not. Now, that parsing is broken because they are coming in an unexpected (maybe more intuitive) order.

Ouch, I see.

Two questions:

- could there be other issues coming from the fact that NetworkManager relies on a particular order (whether right or wrong) of addresses? If yes, I guess NetworkManager would benefit from a fix in any case (that is, we broke it a bit harder, but maybe it was already broken?)

- regardless of this, would there be a way to make NetworkManager robust to the order?

I'm asking this because yes, strictly speaking, you might say this change breaks userspace (NetworkManager), but at the same time it fixes another part of it (pasta), and changing it back would break pasta again (in a way we can't really fix).

Now, I suppose NetworkManager is much more universally used and it's definitely a more mature project so it's a bigger userspace breakage, but at the same time it's a ton of containers (Podman uses pasta by default, Docker/moby optionally via rootlesskit) we might introduce a regression for.

So I'm wondering if in practice it wouldn't be better to fix NetworkManager instead.

Quoted 5 lines

FWIW; Iñigo Huguet (CC'ed) pointed out that this is breaking a NetworkManager test and they were going to report it.

Quoted 2 lines

Also note that userspace was broken and fixed a couple of times:

Quoted 8 lines
Quoted 5 lines
   https://bugs.passt.top/show_bug.cgi?id=175#c10

so I would look into fixing that properly if doable. If it takes too long and this is causing issues for a lot of people meanwhile of course it might make sense to revert, but I'd give it a quick try at least.

I think the solution shouldn't modify the order of the list by default. Maybe we can introduce a flag for the dump operation?

I'm a bit struggling with the name and how complicated a description would be. NLM_F_RIGHT_ORDER? I guess you're suggesting to get the "wrong" / reversed order by default... even though it's the opposite compared to IPv4? Or NLM_F_REVERSED, otherwise? Is it really worth it?

Signature · 1 line
Stefano
#13 lore raw
Quoted 85 lines · Stefano Brivio
Quoted 74 lines · On Thu, 28 May 2026 12:46:05 +0200 Fernando Fernandez Mancera…
Quoted 64 lines · Stefano Brivio

Hi Fernando, Jakub,

Quoted 48 lines · On Wed, 27 May 2026 23:59:47 +0200 Fernando Fernandez Mancera…
Quoted 33 lines · Chris Adams
Quoted 25 lines · Jakub Kicinski
Quoted 11 lines · On Tue, 26 May 2026 20:06:41 -0500 Chris Adams wrote:
Quoted 2 lines

Hi! Adding more people to CC. Do you know if you upgraded from 6.18 or 6.19?

It was 6.19 to 7.0.

Quoted 4 lines

Would you be able to try testing with some commits reverted? On a quick look the candidates would be:

cb3de96eea66 ("ipv6: preserve insertion order for same-scope addresses")

It's this one.

Phew, the second one was mine :)

Quoted 9 lines

I figured out that it happens after stopping a VM (and I usually start/stop a VM for a bit in the morning, which is why it happened more than once). So I set up a VM with a nested VM, running up-to-date Fedora 44, and then was able to bisect pretty easily, and it landed on this commit.

Fedora is using NetworkManager, and IIRC NM does some part of privacy address management (right?). NM didn't change, so maybe this commit is confusing something in NM?

Quoted 6 lines

Sounds plausible, pretty sure we knew this commit was risky to begin with, but we had no direct proof that it'd break real life users.

Revert is the right course of action here. Would you be willing/able to send the revert with your problem description and a Fixes tag pointing to the reverted commit?

Quoted 7 lines

I did want to add a little more test note:

It's definitely an interaction with NetworkManager. If I stop NM and run my VM start/stop test, nothing unexpected happens after. If NM is running and I do my VM test, when the next router advertisement is received, NM replaces the privacy addresses.

As someone that is experienced in NetworkManager, I can confirm it is related. NetworkManager is querying the IPv6 address and when the connection is configured with ipv6.ip6-privacy=2 (prefer-temp-addr), NetworkManager creates a route to make the system use the temporary address for outgoing connections by default.

Quoted 6 lines

If the order is messed up, the address picked will likely be too. One could argue that this is partially fault of NetworkManager and that it should check the timestamps or preferred times rather than order.. but well, the rule is "do not break userspace".

I hope this clarifies things.

Not entirely. I'm looking into this right now, but note that the purpose of that commit is to *preserve* the order of addresses as they were inserted, not to mess it up.

Quoted 5 lines

Before that, addresses were stored and returned via netlink *reversed* compared to the insertion, which is rather surprising, also because it's the opposite of what we do for IPv4 addresses, and caused issues with pasta(1) as it copies addresses into containers in the same order as reported via netlink.

Right, to make it clear. I agree with you here. The current order is the *right* or rather intuitive order.

Quoted 7 lines
Quoted 2 lines

Do you happen to know exactly in which way the order happens to be wrong now?

It is not wrong, but userspace applications were parsing addresses and expecting them in the previous order whether it was correct or not. Now, that parsing is broken because they are coming in an unexpected (maybe more intuitive) order.

Quoted 9 lines

Ouch, I see.

Two questions:

- could there be other issues coming from the fact that NetworkManager relies on a particular order (whether right or wrong) of addresses? If yes, I guess NetworkManager would benefit from a fix in any case (that is, we broke it a bit harder, but maybe it was already broken?)

Yes, I believe the right way to pick a temporary address is to use the most recent one and the right way to see that is by looking to timestamps not looking at the order.

Quoted 8 lines

- regardless of this, would there be a way to make NetworkManager robust to the order?

I'm asking this because yes, strictly speaking, you might say this change breaks userspace (NetworkManager), but at the same time it fixes another part of it (pasta), and changing it back would break pasta again (in a way we can't really fix).

Ugh, that is a mess.

Quoted 6 lines

Now, I suppose NetworkManager is much more universally used and it's definitely a more mature project so it's a bigger userspace breakage, but at the same time it's a ton of containers (Podman uses pasta by default, Docker/moby optionally via rootlesskit) we might introduce a regression for.

I think the main argument here is: while we know it is affecting NetworkManager, we cannot know if it is breaking something else. The kernel rule is usually "do not break userspace" and this is likely considered a regression from my understanding. Cc'ing Thorsten as he is an expert on regressions.

Quoted 3 lines

So I'm wondering if in practice it wouldn't be better to fix NetworkManager instead.

We can fix future versions of NetworkManager but we cannot fix released versions. That is, if someone upgrade kernel but not NetworkManager, it would break. I guess the same applies to pasta.. so we must understand what has been the most prevalent behavior.

Quoted 11 lines
Quoted 11 lines

FWIW; Iñigo Huguet (CC'ed) pointed out that this is breaking a NetworkManager test and they were going to report it.

Quoted 7 lines

Also note that userspace was broken and fixed a couple of times:

    https://bugs.passt.top/show_bug.cgi?id=175#c10

so I would look into fixing that properly if doable. If it takes too long and this is causing issues for a lot of people meanwhile of course it might make sense to revert, but I'd give it a quick try at least.

Quoted 8 lines
Quoted 2 lines

I think the solution shouldn't modify the order of the list by default. Maybe we can introduce a flag for the dump operation?

I'm a bit struggling with the name and how complicated a description would be. NLM_F_RIGHT_ORDER? I guess you're suggesting to get the "wrong" / reversed order by default... even though it's the opposite compared to IPv4? Or NLM_F_REVERSED, otherwise? Is it really worth it?

I don't have a clear idea on how to do it so it is acceptable, right now. FWIW; I think consistent behavior between IPv4 and IPv6 isn't a goal. They are designed in different ways and implemented in different ways too. There are many situations where they behave completely different e.g ECMP routing.

I have sent the revert patch but more likely for AI review + CI. I do not want it to be merged until we reach a consensus, we can discard it if we decide to do so.

Thanks,
Fernando.

#14 lore raw
Quoted 8 lines · Fernando Fernandez Mancera
Quoted 7 lines · Stefano Brivio
Quoted 1 line · On Thu, 28 May 2026 12:46:05 +0200 Fernando Fernandez Mancera…

On 5/28/26 7:38 AM, Stefano Brivio wrote:

- regardless of this, would there be a way to make NetworkManager robust to the order?

Quoted 6 lines
Quoted 4 lines

   I'm asking this because yes, strictly speaking, you might say this change breaks userspace (NetworkManager), but at the same time it fixes another part of it (pasta), and changing it back would break pasta again (in a way we can't really fix).

Ugh, that is a mess.

Well, yes, but one that happens regularly. :-D

The problem here is that it was reported only some months after the culprit landed, as fixing the regression through a revert or so could now cause a bigger regression. Do we assume this to be the case here? It sounds like pasta would "only" be broken about as much as it was before -- then a revert or something like that is the right solution to get back to the status quo. Or does pasta already depend on the new behavior somehow and would now break even more if we reverted the culprit? Then it's really a mess. :-/

But there are ways to fix even this, it's just that most of them are ugly. Like adding some bit somewhere to /proc/ or so that a fixed NetworkManager (if it's the only affected app) could flip by default to change the things from the old behavior to the new one; and one pasta could check that bit and warn. Config options are also an option, but that's even uglier.

Quoted 11 lines
Quoted 5 lines

   Now, I suppose NetworkManager is much more universally used and it's definitely a more mature project so it's a bigger userspace breakage, but at the same time it's a ton of containers (Podman uses pasta by default, Docker/moby optionally via rootlesskit) we might introduce a regression for.

I think the main argument here is: while we know it is affecting NetworkManager, we cannot know if it is breaking something else. The kernel rule is usually "do not break userspace" and this is likely considered a regression from my understanding. Cc'ing Thorsten as he is an expert on regressions.

From the mail at the top of the thread it definitely sounds like one.

Ciao, Thorsten

#15 lore raw
Quoted 14 lines · On Thu, 28 May 2026 14:29:50 +0200 Thorsten Leemhuis…
Quoted 13 lines · Fernando Fernandez Mancera
Quoted 11 lines · Stefano Brivio
Quoted 1 line · On Thu, 28 May 2026 12:46:05 +0200 Fernando Fernandez Mancera…

On 5/28/26 7:38 AM, Stefano Brivio wrote:

- regardless of this, would there be a way to make NetworkManager robust to the order?

   I'm asking this because yes, strictly speaking, you might say this change breaks userspace (NetworkManager), but at the same time it fixes another part of it (pasta), and changing it back would break pasta again (in a way we can't really fix).

Quoted 7 lines
Quoted 1 line

Ugh, that is a mess.

Well, yes, but one that happens regularly. :-D

The problem here is that it was reported only some months after the culprit landed, as fixing the regression through a revert or so could now cause a bigger regression. Do we assume this to be the case here?

Not a bigger regression, but a regression definitely, even just for 'ip address' or 'ip address showdump' which would now go back to display / save addresses in the reversed order for IPv6 (only).

Quoted 5 lines

It
sounds like pasta would "only" be broken about as much as it was before -- then a revert or something like that is the right solution to get back to the status quo. Or does pasta already depend on the new behavior somehow and would now break even more if we reverted the culprit?

It doesn't specifically depend on it. There might be users depending on it because, if they choose to copy container addresses from the host, they will be reversed, and also picked by the kernel in the reversed order (if all other criteria based on scope and prefix length are the same).

The likelihood of that breaking setups should be relatively small. It's just not zero so if we had another sane option to keep this "fixed" I would probably prefer that.

Quoted 1 line

Then it's really a mess. :-/

I guess not so much, it shouldn't be a drama for pasta and containers.

But then again if we revert this, how do we fix the issue later, without resorting to any of the ugly things below?

Quoted 6 lines

But there are ways to fix even this, it's just that most of them are ugly. Like adding some bit somewhere to /proc/ or so that a fixed NetworkManager (if it's the only affected app) could flip by default to change the things from the old behavior to the new one; and one pasta could check that bit and warn. Config options are also an option, but that's even uglier.

Actually, an eventually fixed version of NetworkManager doesn't need to know the behaviour of the kernel: it can just order addresses by timestamps instead, as Fernando mentioned.

And I'm not sure how relevant this is, but if we revert the fix, current combinations of NetworkManager / kernel versions would be anyway affected.

So at this point the only robust / complete fix would be changing NetworkManager to sort addresses as needed. Are you suggesting that we should anyway try to minimise the temporal impact of this with a revert?

Quoted 13 lines
Quoted 11 lines
Quoted 5 lines

   Now, I suppose NetworkManager is much more universally used and it's definitely a more mature project so it's a bigger userspace breakage, but at the same time it's a ton of containers (Podman uses pasta by default, Docker/moby optionally via rootlesskit) we might introduce a regression for.

I think the main argument here is: while we know it is affecting NetworkManager, we cannot know if it is breaking something else. The kernel rule is usually "do not break userspace" and this is likely considered a regression from my understanding. Cc'ing Thorsten as he is an expert on regressions.

From the mail at the top of the thread it definitely sounds like one.

Signature · 1 line
Stefano
#16 lore raw
Quoted 15 lines · Stefano Brivio
Quoted 4 lines · On Thu, 28 May 2026 14:29:50 +0200 Thorsten Leemhuis…
Quoted 3 lines · Fernando Fernandez Mancera
Quoted 2 lines · Stefano Brivio

On Thu, 28 May 2026 12:46:05 +0200
Fernando Fernandez Mancera <[email protected]> wrote:

Quoted 8 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line · Stefano Brivio

But there are ways to fix even this, it's just that most of them are ugly. Like adding some bit somewhere to /proc/ or so that a fixed NetworkManager (if it's the only affected app) could flip by default to change the things from the old behavior to the new one; and one pasta could check that bit and warn. Config options are also an option, but that's even uglier.

Quoted 3 lines

Actually, an eventually fixed version of NetworkManager doesn't need to know the behaviour of the kernel: it can just order addresses by timestamps instead, as Fernando mentioned.

Sure, but the trick I tried to describe is: NetworkManager knows it can handle the new way of things (and thus avoid the regression the reporter described at the start of the thread) and thus could at boot time tell the kernel that by changing some bit in /proc/ -- and make it switch from the old to the new order, so that pasta will then do the right thing, too.

But as I said: ugly, even if that uglyness can be removed after a few years.

Quoted 2 lines

Are you suggesting that we
should anyway try to minimise the temporal impact of this with a revert?

I'd like to leave judgement here to the network maintainers, they are better equipped to make judgment calls like this.

Ciao, Thorsten

#17 lore raw
In reply to Stefano Brivio

CCing Beniamino, NetworkManager maintainer.

Quoted 3 lines · Stefano Brivio

Not a bigger regression, but a regression definitely, even just for 'ip address' or 'ip address showdump' which would now go back to display / save addresses in the reversed order for IPv6 (only).

One could argue that the bug had to be fixed in iproute2 and not in the kernel, to avoid breaking UAPI.

Quoted 3 lines

Actually, an eventually fixed version of NetworkManager doesn't need to know the behaviour of the kernel: it can just order addresses by timestamps instead, as Fernando mentioned.

If the fix must be in NetworkManager, we only need to parse them in non-reverse order like IPv4, I guess.

Quoted 3 lines

And I'm not sure how relevant this is, but if we revert the fix, current combinations of NetworkManager / kernel versions would be anyway affected.

Probably many, for who knows how much time, unless all distros backport the NetworkManager patch. Many don't backport much except a small bunch of security fixes.

Quoted 3 lines

So at this point the only robust / complete fix would be changing NetworkManager to sort addresses as needed. Are you suggesting that we should anyway try to minimise the temporal impact of this with a revert?

I don't think it's the "only" robust fix. Normally one would think first about not breaking UAPI. Kernel's UAPI is not great or intuitive in many places, but normally it's never improved to avoid breaking backward compatibility. Adding a GIVE_ME_THE_RIGHT_ORDER flag could help too, but I agree that it sounds a bit silly to have an opt-in option to force the right behaviour.

Then, the only robust fix would be to fix pasta and iproute2.

Here we know of 2 affected userspace programs, but we don't know if more could be discovered in the future.

That said, it is true that this case is very unintuitive. Insertion is direct-order for both IPv4 and 6. Dump is reversed, but only for IPv6. Userspace programs won't expect this at all. Programs that do it right it's probably because they encountered a bug because they were initially reading in direct-order, and changed to reverse-order. I haven't dug into NetworkManager's history but I bet this was the case.

In this case I would be in favor of not reverting the kernel change, and fix NetworkManager, but I'd like to hear Beniamino's opinion, as I'm not being much involved in NetworkManager any more.

Signature · 1 line
Íñigo Huguet
#18 lore raw
Quoted 9 lines · On Thu, 28 May 2026 16:15:38 +0200 Íñigo Huguet <[email protected]> wrote:

CCing Beniamino, NetworkManager maintainer.

Quoted 3 lines · Stefano Brivio

Not a bigger regression, but a regression definitely, even just for 'ip address' or 'ip address showdump' which would now go back to display / save addresses in the reversed order for IPv6 (only).

One could argue that the bug had to be fixed in iproute2 and not in the kernel, to avoid breaking UAPI.

I think what's considered UAPI in this case isn't so clear-cut. I guess it would have been pretty hard for anybody not familiar with NetworkManager's codebase to imagine that an application would rely on IPv6 addresses (and only IPv6) to be returned in the reverse order.

Pushing this to the extreme, one could argue for example that if we accidentally byteswapped the last 16 bits of some IPv6 addresses, and NetworkManager started to work around that, that would start being considered established UAPI...

Quoted 6 lines
Quoted 3 lines

Actually, an eventually fixed version of NetworkManager doesn't need to know the behaviour of the kernel: it can just order addresses by timestamps instead, as Fernando mentioned.

If the fix must be in NetworkManager, we only need to parse them in non-reverse order like IPv4, I guess.

But that would then require some form of detection, and, at least according to Fernando, isn't the most robust option anyway, as ideally NetworkManager shouldn't rely on the order at all.

Quoted 8 lines
Quoted 3 lines

And I'm not sure how relevant this is, but if we revert the fix, current combinations of NetworkManager / kernel versions would be anyway affected.

Probably many, for who knows how much time, unless all distros backport the NetworkManager patch. Many don't backport much except a small bunch of security fixes.

Quoted 6 lines
Quoted 3 lines

So at this point the only robust / complete fix would be changing NetworkManager to sort addresses as needed. Are you suggesting that we should anyway try to minimise the temporal impact of this with a revert?

I don't think it's the "only" robust fix. Normally one would think first about not breaking UAPI.

Sure... I would just argue that in this case it was pretty hard to imagine that as an established UAPI trait.

Quoted 7 lines

Kernel's UAPI is not great or intuitive
in many places, but normally it's never improved to avoid breaking backward compatibility. Adding a GIVE_ME_THE_RIGHT_ORDER flag could help too, but I agree that it sounds a bit silly to have an opt-in option to force the right behaviour.

Then, the only robust fix would be to fix pasta and iproute2.

I was mentioning that we can't really fix this in pasta (unless, again, we implement some form of detection, or something on the lines of what Thorsten suggested) because we're just copying addresses to a container and we don't know much about them.

We could add detection and swap them, which is rather complicated for us because we currently don't store them as we copy them (we don't use dynamic memory allocation), but of course that would be on us to figure out, somehow.

Quoted 5 lines

Here we know of 2 affected userspace programs, but we don't know if more could be discovered in the future.

That said, it is true that this case is very unintuitive. Insertion is direct-order for both IPv4 and 6. Dump is reversed, but only for IPv6.

Wait... that's how confusing it is. That's also what I thought at the beginning of https://bugs.passt.top/show_bug.cgi?id=175, but it turned out that insertion is actually reversed for IPv6. Dump isn't.

So this is also inconsistent in the kernel itself: if everything else is equal, address selection depends on the order of addresses as inserted, which is the opposite compared to IPv4. Maybe this is the most surprising fact in all this.

Quoted 4 lines

Userspace programs won't expect this at all. Programs that do it right it's probably because they encountered a bug because they were initially reading in direct-order, and changed to reverse-order. I haven't dug into NetworkManager's history but I bet this was the case.

I guess so too, yes.

Quoted 3 lines

In this case I would be in favor of not reverting the kernel change, and fix NetworkManager, but I'd like to hear Beniamino's opinion, as I'm not being much involved in NetworkManager any more.

Signature · 1 line
Stefano
#19 lore raw
Quoted 4 lines · Stefano Brivio

I think what's considered UAPI in this case isn't so clear-cut. I guess it would have been pretty hard for anybody not familiar with NetworkManager's codebase to imagine that an application would rely on IPv6 addresses (and only IPv6) to be returned in the reverse order.

IMHO it's not because an application would rely on the order. It's because the order matters, as the first one is considered the primary. This is not NetworkManager specific. Or I am mistaken? I'm speaking by memory

Quoted 4 lines

Pushing this to the extreme, one could argue for example that if we accidentally byteswapped the last 16 bits of some IPv6 addresses, and NetworkManager started to work around that, that would start being considered established UAPI...

Fair enough. To some extent, it could be considered a long-standing bug. You can fix bugs even if userspace programs started to workaround them.

But, on the other side, it can be easier considered just a bad design that, after a long time, should not be changed anymore because it's a established UAPI.

Quoted 6 lines
Quoted 2 lines

If the fix must be in NetworkManager, we only need to parse them in non-reverse order like IPv4, I guess.

But that would then require some form of detection, and, at least according to Fernando, isn't the most robust option anyway, as ideally NetworkManager shouldn't rely on the order at all.

True

Quoted 3 lines

Wait... that's how confusing it is. That's also what I thought at the beginning of https://bugs.passt.top/show_bug.cgi?id=175, but it turned out that insertion is actually reversed for IPv6. Dump isn't.

Uh, OK. That's even worse because even scripts using ip addr add ... may think the first inserted address is the primary.

As I said, I leave for Beniamino the considerations from the NetworkManager side.

Signature · 1 line
Íñigo Huguet
#20 lore raw
Quoted 10 lines · Íñigo Huguet
Quoted 4 lines · Stefano Brivio

I think what's considered UAPI in this case isn't so clear-cut. I guess it would have been pretty hard for anybody not familiar with NetworkManager's codebase to imagine that an application would rely on IPv6 addresses (and only IPv6) to be returned in the reverse order.

IMHO it's not because an application would rely on the order. It's because the order matters, as the first one is considered the primary. This is not NetworkManager specific. Or I am mistaken? I'm speaking by memory

Yes, exactly. If you do:

ip addr add dev X 3fff::1/64
ip addr dev dev X 3fff::2/64

then the old kernel chooses the address that was added last as source address for outgoing communications. With the new kernel it chooses the address that was added first.

I think that any script or network management tool that cares about the source address selection is impacted. Indeed, the commit had effects on one of the selftests, which had to be modified to change the order of iproute2 invocations.

Quoted 8 lines
Quoted 6 lines
Quoted 2 lines

If the fix must be in NetworkManager, we only need to parse them in non-reverse order like IPv4, I guess.

But that would then require some form of detection, and, at least according to Fernando, isn't the most robust option anyway, as ideally NetworkManager shouldn't rely on the order at all.

True

Correct, if the new behavior is considered better, there should be a way to detect which order must be used. Otherwise userspace tools won't be able to maintain the same behavior with different kernels.

Beniamino

#21 lore raw
Quoted 9 lines · On Thu, 28 May 2026 18:01:27 +0200 Beniamino Galvani <[email protected]>…
Quoted 8 lines · Íñigo Huguet
Quoted 4 lines · Stefano Brivio

I think what's considered UAPI in this case isn't so clear-cut. I guess it would have been pretty hard for anybody not familiar with NetworkManager's codebase to imagine that an application would rely on IPv6 addresses (and only IPv6) to be returned in the reverse order.

IMHO it's not because an application would rely on the order. It's because the order matters, as the first one is considered the primary.

Yes, that's also the case, but:

Quoted 2 lines
Quoted 2 lines

This is not NetworkManager specific. Or I am mistaken? I'm speaking by memory

...from what I understood of the issue at hand there's a part that's specific to NetworkManager here, because NetworkManager replaces / deletes the Privacy Extensions addresses, instead of different addresses, because it relies on that specific (reversed) order.

Anyway, yes, I see your point about UAPI now.

Quoted 21 lines

Yes, exactly. If you do:

ip addr add dev X 3fff::1/64
ip addr dev dev X 3fff::2/64

then the old kernel chooses the address that was added last as source address for outgoing communications. With the new kernel it chooses the address that was added first.

I think that any script or network management tool that cares about the source address selection is impacted. Indeed, the commit had effects on one of the selftests, which had to be modified to change the order of iproute2 invocations.

Quoted 7 lines
Quoted 6 lines
Quoted 2 lines

If the fix must be in NetworkManager, we only need to parse them in non-reverse order like IPv4, I guess.

But that would then require some form of detection, and, at least according to Fernando, isn't the most robust option anyway, as ideally NetworkManager shouldn't rely on the order at all.

Quoted 5 lines
Quoted 1 line

True

Correct, if the new behavior is considered better, there should be a way to detect which order must be used. Otherwise userspace tools won't be able to maintain the same behavior with different kernels.

My remark here is about whether NetworkManager needs to detect this at all. If it used timestamps to detect recent / older addresses, as Fernando mentioned, then you wouldn't need any detection at all, right? Or is there something else we're missing?

By the way, if really needed, one could detect that with the equivalent of:

  ip a a ::2 dev lo && ip a a ::3 dev lo && [ $(ip -6 -j a | jq -rM '.[] .addr_info[0].local') = ::3 ] && echo wrong || echo right

...depending on the application, that's not necessarily practical, but so far we have no evidence of any application needing to do that at all.

Signature · 1 line
Stefano
#22 lore raw
Quoted 48 lines · Stefano Brivio
Quoted 9 lines · On Thu, 28 May 2026 18:01:27 +0200 Beniamino Galvani <[email protected]>…
Quoted 8 lines · Íñigo Huguet
Quoted 5 lines · Stefano Brivio

I think what's considered UAPI in this case isn't so clear- cut. I guess it would have been pretty hard for anybody not familiar with NetworkManager's codebase to imagine that an application would rely on IPv6 addresses (and only IPv6) to be returned in the reverse order.

Quoted 3 lines
Quoted 3 lines

IMHO it's not because an application would rely on the order. It's because the order matters, as the first one is considered the primary.

Yes, that's also the case, but:

Quoted 2 lines
Quoted 2 lines

This is not NetworkManager specific. Or I am mistaken? I'm speaking by memory

...from what I understood of the issue at hand there's a part that's specific to NetworkManager here, because NetworkManager replaces / deletes the Privacy Extensions addresses, instead of different addresses, because it relies on that specific (reversed) order.

Anyway, yes, I see your point about UAPI now.

Quoted 23 lines

Yes, exactly. If you do:

ip addr add dev X 3fff::1/64 ip addr dev dev X 3fff::2/64

then the old kernel chooses the address that was added last as source address for outgoing communications. With the new kernel it chooses the address that was added first.

I think that any script or network management tool that cares about the source address selection is impacted. Indeed, the commit had effects on one of the selftests, which had to be modified to change the order of iproute2 invocations.

Quoted 9 lines
Quoted 7 lines
Quoted 2 lines

If the fix must be in NetworkManager, we only need to parse them in non-reverse order like IPv4, I guess.

But that would then require some form of detection, and, at least according to Fernando, isn't the most robust option anyway, as ideally NetworkManager shouldn't rely on the order at all.

True

Quoted 9 lines
Quoted 4 lines

Correct, if the new behavior is considered better, there should be a way to detect which order must be used. Otherwise userspace tools won't be able to maintain the same behavior with different kernels.

My remark here is about whether NetworkManager needs to detect this
at all. If it used timestamps to detect recent / older addresses, as Fernando mentioned, then you wouldn't need any detection at all, right? Or is there something else we're missing?

Ohno. Now that Beniamino and Iñigo mentioned it, this will likely break
many other environments. In essence, many tools relies on the previous ordering to identify which address is the primary one.

E.g cloud tooling communicating with the metadata server via IMDS(v2) to configure IPv6 primary and secondary addresses. They are likely relying on the ordering for that.

So maybe we could fix whatever is wrong with NetworkManager and temporary addresses order but I don't think this will scale for the cloud tooling with the primary/secondary issue.

Quoted 10 lines

By the way, if really needed, one could detect that with the equivalent of:

ip a a ::2 dev lo && ip a a ::3 dev lo && [ $(ip -6 -j a | jq -rM '. [] .addr_info[0].local') = ::3 ] && echo wrong || echo right

...depending on the application, that's not necessarily practical, but so far we have no evidence of any application needing to do that at all.

Plenty of tools (NetworkManager included) uses netlink so this isn't useful for them.

For now I am more in favor of the revert and to look for an alternative for pasta situation (like using a flag or something). But I am not a maintainer so it isn't my call :)

#23 lore raw
Quoted 72 lines · Fernando Fernandez Mancera
Quoted 48 lines · Stefano Brivio
Quoted 9 lines · On Thu, 28 May 2026 18:01:27 +0200 Beniamino Galvani <[email protected]>…
Quoted 8 lines · Íñigo Huguet
Quoted 5 lines · Stefano Brivio

I think what's considered UAPI in this case isn't so clear- cut. I guess it would have been pretty hard for anybody not familiar with NetworkManager's codebase to imagine that an application would rely on IPv6 addresses (and only IPv6) to be returned in the reverse order.

Quoted 3 lines
Quoted 3 lines

IMHO it's not because an application would rely on the order. It's because the order matters, as the first one is considered the primary.

Yes, that's also the case, but:

Quoted 2 lines
Quoted 2 lines

This is not NetworkManager specific. Or I am mistaken? I'm speaking by memory

...from what I understood of the issue at hand there's a part that's specific to NetworkManager here, because NetworkManager replaces / deletes the Privacy Extensions addresses, instead of different addresses, because it relies on that specific (reversed) order.

Anyway, yes, I see your point about UAPI now.

Quoted 23 lines

Yes, exactly. If you do:

ip addr add dev X 3fff::1/64 ip addr dev dev X 3fff::2/64

then the old kernel chooses the address that was added last as source address for outgoing communications. With the new kernel it chooses the address that was added first.

I think that any script or network management tool that cares about the source address selection is impacted. Indeed, the commit had effects on one of the selftests, which had to be modified to change the order of iproute2 invocations.

Quoted 9 lines
Quoted 7 lines
Quoted 2 lines

If the fix must be in NetworkManager, we only need to parse them in non-reverse order like IPv4, I guess.

But that would then require some form of detection, and, at least according to Fernando, isn't the most robust option anyway, as ideally NetworkManager shouldn't rely on the order at all.

True

Quoted 9 lines
Quoted 4 lines

Correct, if the new behavior is considered better, there should be a way to detect which order must be used. Otherwise userspace tools won't be able to maintain the same behavior with different kernels.

My remark here is about whether NetworkManager needs to detect this
at all. If it used timestamps to detect recent / older addresses, as Fernando mentioned, then you wouldn't need any detection at all, right? Or is there something else we're missing?

Ohno. Now that Beniamino and Iñigo mentioned it, this will likely break
many other environments. In essence, many tools relies on the previous ordering to identify which address is the primary one.

E.g cloud tooling communicating with the metadata server via IMDS(v2) to configure IPv6 primary and secondary addresses. They are likely relying on the ordering for that.

So maybe we could fix whatever is wrong with NetworkManager and temporary addresses order but I don't think this will scale for the cloud tooling with the primary/secondary issue.

Quoted 6 lines

By the way, if really needed, one could detect that with the equivalent of:

ip a a ::2 dev lo && ip a a ::3 dev lo && [ $(ip -6 -j a | jq -rM '. [] .addr_info[0].local') = ::3 ] && echo wrong || echo right

Quoted 8 lines
Quoted 4 lines

...depending on the application, that's not necessarily practical, but so far we have no evidence of any application needing to do that at all.

Plenty of tools (NetworkManager included) uses netlink so this isn't useful for them.

For now I am more in favor of the revert and to look for an alternative for pasta situation (like using a flag or something). But I am not a maintainer so it isn't my call :)

*facepalm* I just noticed the "with the equivalent of", disregard this last part tho.

#24 lore raw
Quoted 64 lines · On Thu, 28 May 2026 20:50:48 +0200 Fernando Fernandez Mancera…
Quoted 63 lines · Fernando Fernandez Mancera
Quoted 61 lines · Stefano Brivio
Quoted 9 lines · On Thu, 28 May 2026 18:01:27 +0200 Beniamino Galvani <[email protected]>…
Quoted 8 lines · Íñigo Huguet
Quoted 5 lines · Stefano Brivio

I think what's considered UAPI in this case isn't so clear- cut. I guess it would have been pretty hard for anybody not familiar with NetworkManager's codebase to imagine that an application would rely on IPv6 addresses (and only IPv6) to be returned in the reverse order.

Quoted 3 lines
Quoted 3 lines

IMHO it's not because an application would rely on the order. It's because the order matters, as the first one is considered the primary.

Yes, that's also the case, but:

Quoted 2 lines
Quoted 2 lines

This is not NetworkManager specific. Or I am mistaken? I'm speaking by memory

...from what I understood of the issue at hand there's a part that's specific to NetworkManager here, because NetworkManager replaces / deletes the Privacy Extensions addresses, instead of different addresses, because it relies on that specific (reversed) order.

Anyway, yes, I see your point about UAPI now.

Quoted 21 lines

Yes, exactly. If you do:

ip addr add dev X 3fff::1/64 ip addr dev dev X 3fff::2/64

then the old kernel chooses the address that was added last as source address for outgoing communications. With the new kernel it chooses the address that was added first.

I think that any script or network management tool that cares about the source address selection is impacted. Indeed, the commit had effects on one of the selftests, which had to be modified to change the order of iproute2 invocations.

Quoted 8 lines
Quoted 7 lines
Quoted 2 lines

If the fix must be in NetworkManager, we only need to parse them in non-reverse order like IPv4, I guess.

But that would then require some form of detection, and, at least according to Fernando, isn't the most robust option anyway, as ideally NetworkManager shouldn't rely on the order at all.

Quoted 6 lines
Quoted 1 line

True

Correct, if the new behavior is considered better, there should be a way to detect which order must be used. Otherwise userspace tools won't be able to maintain the same behavior with different kernels.

My remark here is about whether NetworkManager needs to detect this at all. If it used timestamps to detect recent / older addresses, as Fernando mentioned, then you wouldn't need any detection at all, right? Or is there something else we're missing?

Quoted 7 lines
Quoted 7 lines

Ohno. Now that Beniamino and Iñigo mentioned it, this will likely break many other environments. In essence, many tools relies on the previous ordering to identify which address is the primary one.

E.g cloud tooling communicating with the metadata server via IMDS(v2) to configure IPv6 primary and secondary addresses. They are likely relying on the ordering for that.

I haven't seen any tool specifically relying on insertion order for this so far and I'm having a hard time believing this kind of tooling wouldn't rely explicitly on home / care-of addresses or different labels -- see RFC 5014 and RFC 6724 Section 5. (or, perhaps clearer, the examples in section 10.1, in particular rule 4. and rule 6.

But I'll look for more convincing examples in a bit (maybe you have some at hand?)

This is all implemented by __ipv6_dev_get_saddr() and friends, by the way.

Quoted 14 lines
Quoted 14 lines

So maybe we could fix whatever is wrong with NetworkManager and temporary addresses order but I don't think this will scale for the cloud tooling with the primary/secondary issue.

Quoted 9 lines

By the way, if really needed, one could detect that with the equivalent of:

ip a a ::2 dev lo && ip a a ::3 dev lo && [ $(ip -6 -j a | jq -rM '. [] .addr_info[0].local') = ::3 ] && echo wrong || echo right

...depending on the application, that's not necessarily practical, but so far we have no evidence of any application needing to do that at all.

Quoted 6 lines
Quoted 6 lines

Plenty of tools (NetworkManager included) uses netlink so this isn't useful for them.

For now I am more in favor of the revert and to look for an alternative for pasta situation (like using a flag or something). But I am not a maintainer so it isn't my call :)

One question I started wondering about later is: should the flag affect insertion or reporting, in case? I think we would need one for insertion, otherwise it's not really generic / functional.

Quoted 2 lines

*facepalm* I just noticed the "with the equivalent of", disregard this last part tho.

Right, yeah, having a netlink implementation makes that a bit more convenient. :)

Signature · 1 line
Stefano
#25 lore raw
Quoted 103 lines · Stefano Brivio
Quoted 64 lines · On Thu, 28 May 2026 20:50:48 +0200 Fernando Fernandez Mancera…
Quoted 63 lines · Fernando Fernandez Mancera
Quoted 61 lines · Stefano Brivio
Quoted 9 lines · On Thu, 28 May 2026 18:01:27 +0200 Beniamino Galvani <[email protected]>…
Quoted 8 lines · Íñigo Huguet
Quoted 5 lines · Stefano Brivio

I think what's considered UAPI in this case isn't so clear- cut. I guess it would have been pretty hard for anybody not familiar with NetworkManager's codebase to imagine that an application would rely on IPv6 addresses (and only IPv6) to be returned in the reverse order.

Quoted 3 lines
Quoted 3 lines

IMHO it's not because an application would rely on the order. It's because the order matters, as the first one is considered the primary.

Yes, that's also the case, but:

Quoted 2 lines
Quoted 2 lines

This is not NetworkManager specific. Or I am mistaken? I'm speaking by memory

...from what I understood of the issue at hand there's a part that's specific to NetworkManager here, because NetworkManager replaces / deletes the Privacy Extensions addresses, instead of different addresses, because it relies on that specific (reversed) order.

Anyway, yes, I see your point about UAPI now.

Quoted 21 lines

Yes, exactly. If you do:

ip addr add dev X 3fff::1/64 ip addr dev dev X 3fff::2/64

then the old kernel chooses the address that was added last as source address for outgoing communications. With the new kernel it chooses the address that was added first.

I think that any script or network management tool that cares about the source address selection is impacted. Indeed, the commit had effects on one of the selftests, which had to be modified to change the order of iproute2 invocations.

Quoted 8 lines
Quoted 7 lines
Quoted 2 lines

If the fix must be in NetworkManager, we only need to parse them in non-reverse order like IPv4, I guess.

But that would then require some form of detection, and, at least according to Fernando, isn't the most robust option anyway, as ideally NetworkManager shouldn't rely on the order at all.

Quoted 6 lines
Quoted 1 line

True

Correct, if the new behavior is considered better, there should be a way to detect which order must be used. Otherwise userspace tools won't be able to maintain the same behavior with different kernels.

My remark here is about whether NetworkManager needs to detect this at all. If it used timestamps to detect recent / older addresses, as Fernando mentioned, then you wouldn't need any detection at all, right? Or is there something else we're missing?

Quoted 7 lines
Quoted 7 lines

Ohno. Now that Beniamino and Iñigo mentioned it, this will likely break many other environments. In essence, many tools relies on the previous ordering to identify which address is the primary one.

E.g cloud tooling communicating with the metadata server via IMDS(v2) to configure IPv6 primary and secondary addresses. They are likely relying on the ordering for that.

I haven't seen any tool specifically relying on insertion order for this so far and I'm having a hard time believing this kind of tooling wouldn't rely explicitly on home / care-of addresses or different labels -- see RFC 5014 and RFC 6724 Section 5. (or, perhaps clearer, the examples in section 10.1, in particular rule 4. and rule 6.

But I'll look for more convincing examples in a bit (maybe you have some at hand?)

This is all implemented by __ipv6_dev_get_saddr() and friends, by the way.

Quoted 17 lines
Quoted 17 lines

So maybe we could fix whatever is wrong with NetworkManager and temporary addresses order but I don't think this will scale for the cloud tooling with the primary/secondary issue.

Quoted 9 lines

By the way, if really needed, one could detect that with the equivalent of:

ip a a ::2 dev lo && ip a a ::3 dev lo && [ $(ip -6 -j a | jq -rM '. [] .addr_info[0].local') = ::3 ] && echo wrong || echo right

...depending on the application, that's not necessarily practical, but so far we have no evidence of any application needing to do that at all.

Plenty of tools (NetworkManager included) uses netlink so this isn't useful for them.

Quoted 7 lines
Quoted 3 lines
Quoted 3 lines

For now I am more in favor of the revert and to look for an alternative for pasta situation (like using a flag or something). But I am not a maintainer so it isn't my call :)

One question I started wondering about later is: should the flag affect insertion or reporting, in case? I think we would need one for insertion, otherwise it's not really generic / functional.

I'd certainly agree. The bug? oddity? inconsistency? is on the insertion side so the fix should be on that side (whether or not it's conditional on a new flag). If we "fixed" it on the reporting side we'd instead be creating another bug that cancels its effect for some, but not all, purposes.

The idea of NLM_INSERT_LAST seems marginally less ghastly to me than NLM_F_DUMP_REVERSED, though I'm not sure I could articulate why. Perhaps because we could also allow the former for IPv4 too (as a no-op).

Quoted 9 lines
Quoted 2 lines

*facepalm* I just noticed the "with the equivalent of", disregard this last part tho.

Right, yeah, having a netlink implementation makes that a bit more convenient. :)

Signature · 1 line
Stefano
Signature · 4 lines
David Gibson (he or they)	| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you, not the other way
				| around.
http://www.ozlabs.org/~dgibson
#26 lore raw
In reply to Stefano Brivio
Quoted 19 lines · Stefano Brivio
Quoted 19 lines
Quoted 19 lines
Quoted 19 lines
Quoted 18 lines

about the source address selection is impacted. Indeed, the commit had effects on one of the selftests, which had to be modified to change the order of iproute2 invocations.

Quoted 9 lines
Quoted 7 lines
Quoted 2 lines

If the fix must be in NetworkManager, we only need to parse them in non-reverse order like IPv4, I guess.

But that would then require some form of detection, and, at least according to Fernando, isn't the most robust option anyway, as ideally NetworkManager shouldn't rely on the order at all.

True

Correct, if the new behavior is considered better, there should be a way to detect which order must be used. Otherwise userspace tools won't be able to maintain the same behavior with different kernels.

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

My remark here is about whether NetworkManager needs to detect this at all. If it used timestamps to detect recent / older addresses, as Fernando mentioned, then you wouldn't need any detection at all, right? Or is there something else we're missing?

The problem arises from how NetworkManager handles updates (e.g. after receiving a Router Advertisement). At each update NM determines the list of addresses to configure and checks if the addresses are already in the right order in the kernel. If they aren't, NM removes and re-adds them in reverse to achieve the desired order. Since kernel 7.0+, the order changed and the addresses always appear in the reverse order.

This creates 2 negative effects. First, it breaks source preference:
if users configured a profile with addr1=A, addr2=B because they wanted A to be preferred, now B is preferred. This is not NetworkManager-specific, it affects also simple scripts that add two addresses (like the selftest that had to be changed in the commit).

But most importantly, at each commit NM detects that the order is wrong and constantly removes and re-adds the addresses. This continuous cycle is what causes the bug that Chris reported.

BTW, NM doesn't touch the temporary addresses directly; they are automatically removed when the corresponding SLAAC address is removed. Since the problem is not only about temporary addresses we can't rely on timestamps.

Quoted 8 lines
Quoted 7 lines
Quoted 7 lines

Ohno. Now that Beniamino and Iñigo mentioned it, this will likely break many other environments. In essence, many tools relies on the previous ordering to identify which address is the primary one.

E.g cloud tooling communicating with the metadata server via IMDS(v2) to configure IPv6 primary and secondary addresses. They are likely relying on the ordering for that.

Quoted 5 lines

I haven't seen any tool specifically relying on insertion order for this so far and I'm having a hard time believing this kind of tooling wouldn't rely explicitly on home / care-of addresses or different labels -- see RFC 5014 and RFC 6724 Section 5. (or, perhaps clearer, the examples in section 10.1, in particular rule 4. and rule 6.

I'm not familiar with home addresses, reading the RFC it seems that setting the flag might have effect not only on source address selection but also on other aspects?

Quoted 2 lines

But I'll look for more convincing examples in a bit (maybe you have some at hand?)

I remember we had users reporting issues about the ordering of IPv6 addresses, I'll try to find links to them.

Beniamino

#27 lore raw
Quoted 25 lines · On Fri, 29 May 2026 10:40:29 +0200 Beniamino Galvani <[email protected]>…
Quoted 19 lines · Stefano Brivio
Quoted 19 lines
Quoted 19 lines
Quoted 19 lines
Quoted 18 lines

about the source address selection is impacted. Indeed, the commit had effects on one of the selftests, which had to be modified to change the order of iproute2 invocations.

Quoted 9 lines
Quoted 7 lines
Quoted 2 lines

If the fix must be in NetworkManager, we only need to parse them in non-reverse order like IPv4, I guess.

But that would then require some form of detection, and, at least according to Fernando, isn't the most robust option anyway, as ideally NetworkManager shouldn't rely on the order at all.

True

Correct, if the new behavior is considered better, there should be a way to detect which order must be used. Otherwise userspace tools won't be able to maintain the same behavior with different kernels.

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

My remark here is about whether NetworkManager needs to detect this at all. If it used timestamps to detect recent / older addresses, as Fernando mentioned, then you wouldn't need any detection at all, right? Or is there something else we're missing?

Quoted 7 lines

The problem arises from how NetworkManager handles updates (e.g. after receiving a Router Advertisement). At each update NM determines the list of addresses to configure and checks if the addresses are already in the right order in the kernel. If they aren't, NM removes and re-adds them in reverse to achieve the desired order. Since kernel 7.0+, the order changed and the addresses always appear in the reverse order.

Oh, I see now, thanks for explaining. That's a bit more than just relying on a given order. On the other hand, it sounds like you have a possible detection mechanism already implemented. :)

Quoted 5 lines

This creates 2 negative effects. First, it breaks source preference:
if users configured a profile with addr1=A, addr2=B because they wanted A to be preferred, now B is preferred. This is not NetworkManager-specific, it affects also simple scripts that add two addresses (like the selftest that had to be changed in the commit).

At the same time, it fixes the kernel behaviour for anything that might expect the same outcome as IPv4, or relying on iproute2's save / restore functionality, as I'm showing here:

https://lore.kernel.org/all/20260529114216.2e42c4dd@elisabeth/

...one might argue that it's more likely to break things than fixing them at this point. I'm not sure.

Quoted 8 lines

But most importantly, at each commit NM detects that the order is wrong and constantly removes and re-adds the addresses. This continuous cycle is what causes the bug that Chris reported.

BTW, NM doesn't touch the temporary addresses directly; they are automatically removed when the corresponding SLAAC address is removed. Since the problem is not only about temporary addresses we can't rely on timestamps.

So if the kernel change is not reverted you would need to have a detection mechanism and change NetworkManager's behaviour according to the detected kernel behaviour, correct? I guess it's nasty / ugly? But doable?

Quoted 14 lines
Quoted 8 lines
Quoted 7 lines
Quoted 7 lines

Ohno. Now that Beniamino and Iñigo mentioned it, this will likely break many other environments. In essence, many tools relies on the previous ordering to identify which address is the primary one.

E.g cloud tooling communicating with the metadata server via IMDS(v2) to configure IPv6 primary and secondary addresses. They are likely relying on the ordering for that.

Quoted 5 lines

I haven't seen any tool specifically relying on insertion order for this so far and I'm having a hard time believing this kind of tooling wouldn't rely explicitly on home / care-of addresses or different labels -- see RFC 5014 and RFC 6724 Section 5. (or, perhaps clearer, the examples in section 10.1, in particular rule 4. and rule 6.

Quoted 3 lines

I'm not familiar with home addresses, reading the RFC it seems that setting the flag might have effect not only on source address selection but also on other aspects?

In theory yes, that should affect the destination address as well in the sense that getaddrinfo() should prefer the same type (home or care-of) of address as the candidate source address, but I'm not aware of any C library for Linux supporting ai_eflags / AI_EXTFLAGS that would be needed for this (RFC 5014 section 11. discusses the requirement).

As far as I know those flags are commonly used just to prefer a given source address. I'm not sure if it's used by IMDSv2 tooling, I couldn't find any evidence so far.

Quoted 5 lines
Quoted 2 lines

But I'll look for more convincing examples in a bit (maybe you have some at hand?)

I remember we had users reporting issues about the ordering of IPv6 addresses, I'll try to find links to them.

I'm currently asking around about IMDSv2 cases. I think it would be really interesting to see what users might have reported around NetworkManager over the years, in one sense or another.

As to an hypothetical NLM_F_INSERT_LAST flag, I think that would be fine for pasta(1) and containers, because the same "buggy" behaviour that one might have on a host / parent namespace would be replicated to other namespaces, and we're mostly interested in consistency.

The effect of keeping the "wrong" behaviour as default looks rather problematic to me (see the iproute2 example) but we could also "fix" iproute2 by setting the flag and maybe propose that as default after a number of years.

Signature · 1 line
Stefano
#28 lore raw
In reply to Stefano Brivio
Quoted 3 lines

Actually, an eventually fixed version of NetworkManager doesn't need to know the behaviour of the kernel: it can just order addresses by timestamps instead, as Fernando mentioned.

Can pasta also use this scheme to order the addresses? Is there a way pasta can be independent of the order?

Maybe we should just make the order random, so equally breaking everybody, and pushing user space to not assume any order :-)

Andrew

#29 lore raw
Quoted 6 lines · On Thu, 28 May 2026 16:34:02 +0200 Andrew Lunn <[email protected]> wrote:
Quoted 3 lines

Actually, an eventually fixed version of NetworkManager doesn't need to know the behaviour of the kernel: it can just order addresses by timestamps instead, as Fernando mentioned.

Can pasta also use this scheme to order the addresses? Is there a way pasta can be independent of the order?

pasta itself doesn't even care, it just inserts (copies) addresses one by one as returned. The kernel cares, because it preferentially picks the first one, as stored, within a given scope.

The problem is that they are inserted in the opposite order than one would expect (that is, opposite to what's used by the kernel to select them, and opposite to what's done for IPv4).

This (with an older kernel version) is quite "funny" (pasta by default copies everything it finds to the inner namespace):

---

$ pasta -6 -Ix --no-ra
# ip a a fd00::1 dev x
# ip a a fd43::1 dev x
# ip a a 2620::1 dev x
# ip link set dev x up
# ip a s x
2: x: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether 86:de:6b:53:5c:05 brd ff:ff:ff:ff:ff:ff
    inet6 2620::1/128 scope global tentative 
       valid_lft forever preferred_lft forever
    inet6 fd43::1/128 scope global tentative 
       valid_lft forever preferred_lft forever
    inet6 fd00::1/128 scope global tentative 
       valid_lft forever preferred_lft forever
# pasta -6 --config-net --no-ra
# ip a s x scope global
2: x: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 65520 qdisc fq_codel state UNKNOWN group default qlen 1000
    link/ether 32:d4:5b:f0:fd:50 brd ff:ff:ff:ff:ff:ff
    inet6 fd00::1/128 scope global nodad 
       valid_lft forever preferred_lft forever
    inet6 fd43::1/128 scope global nodad 
       valid_lft forever preferred_lft forever
    inet6 2620::1/128 scope global nodad 
       valid_lft forever preferred_lft forever
# pasta -6 --config-net --no-ra
# ip a s x scope global
2: x: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 65520 qdisc fq_codel state UNKNOWN group default qlen 1000
    link/ether 1a:ed:4a:27:f6:9f brd ff:ff:ff:ff:ff:ff
    inet6 2620::1/128 scope global nodad 
       valid_lft forever preferred_lft forever
    inet6 fd43::1/128 scope global nodad 
       valid_lft forever preferred_lft forever
    inet6 fd00::1/128 scope global nodad 
       valid_lft forever preferred_lft forever
---

...at every level of namespacing, the order is swapped. Note that iproute2 also reports them in the order they're sent over netlink.

Quoted 2 lines

Maybe we should just make the order random, so equally breaking everybody, and pushing user space to not assume any order :-)

The thing is... the kernel assumes the order. :) But it also has to.

Signature · 1 line
Stefano
#30 lore raw
Quoted 14 lines · Stefano Brivio
Quoted 6 lines · On Thu, 28 May 2026 16:34:02 +0200 Andrew Lunn <[email protected]> wrote:
Quoted 3 lines

Actually, an eventually fixed version of NetworkManager doesn't need to know the behaviour of the kernel: it can just order addresses by timestamps instead, as Fernando mentioned.

Can pasta also use this scheme to order the addresses? Is there a way pasta can be independent of the order?

pasta itself doesn't even care, it just inserts (copies) addresses one by one as returned. The kernel cares, because it preferentially picks the first one, as stored, within a given scope.

Quoted 6 lines

The problem is that they are inserted in the opposite order than one would expect (that is, opposite to what's used by the kernel to select them, and opposite to what's done for IPv4).

This (with an older kernel version) is quite "funny" (pasta by default copies everything it finds to the inner namespace):

Fwiw, a hypothetical tool which copied network configuration from one namespace to an independent one - or which saved network configuration to restore it later - would likely hit the same issue.

Quoted 48 lines

---

$ pasta -6 -Ix --no-ra
# ip a a fd00::1 dev x
# ip a a fd43::1 dev x
# ip a a 2620::1 dev x
# ip link set dev x up
# ip a s x
2: x: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether 86:de:6b:53:5c:05 brd ff:ff:ff:ff:ff:ff
    inet6 2620::1/128 scope global tentative 
       valid_lft forever preferred_lft forever
    inet6 fd43::1/128 scope global tentative 
       valid_lft forever preferred_lft forever
    inet6 fd00::1/128 scope global tentative 
       valid_lft forever preferred_lft forever
# pasta -6 --config-net --no-ra
# ip a s x scope global
2: x: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 65520 qdisc fq_codel state UNKNOWN group default qlen 1000
    link/ether 32:d4:5b:f0:fd:50 brd ff:ff:ff:ff:ff:ff
    inet6 fd00::1/128 scope global nodad 
       valid_lft forever preferred_lft forever
    inet6 fd43::1/128 scope global nodad 
       valid_lft forever preferred_lft forever
    inet6 2620::1/128 scope global nodad 
       valid_lft forever preferred_lft forever
# pasta -6 --config-net --no-ra
# ip a s x scope global
2: x: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 65520 qdisc fq_codel state UNKNOWN group default qlen 1000
    link/ether 1a:ed:4a:27:f6:9f brd ff:ff:ff:ff:ff:ff
    inet6 2620::1/128 scope global nodad 
       valid_lft forever preferred_lft forever
    inet6 fd43::1/128 scope global nodad 
       valid_lft forever preferred_lft forever
    inet6 fd00::1/128 scope global nodad 
       valid_lft forever preferred_lft forever
---

...at every level of namespacing, the order is swapped. Note that iproute2 also reports them in the order they're sent over netlink.

Quoted 2 lines

Maybe we should just make the order random, so equally breaking everybody, and pushing user space to not assume any order :-)

The thing is... the kernel assumes the order. :) But it also has to.

Signature · 1 line
Stefano
Signature · 4 lines
David Gibson (he or they)	| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you, not the other way
				| around.
http://www.ozlabs.org/~dgibson
#31 lore raw
Quoted 19 lines · On Fri, 29 May 2026 14:48:47 +1000 David Gibson <[email protected]>…
Quoted 18 lines · Stefano Brivio
Quoted 6 lines · On Thu, 28 May 2026 16:34:02 +0200 Andrew Lunn <[email protected]> wrote:
Quoted 3 lines

Actually, an eventually fixed version of NetworkManager doesn't need to know the behaviour of the kernel: it can just order addresses by timestamps instead, as Fernando mentioned.

Can pasta also use this scheme to order the addresses? Is there a way pasta can be independent of the order?

pasta itself doesn't even care, it just inserts (copies) addresses one by one as returned. The kernel cares, because it preferentially picks the first one, as stored, within a given scope.

The problem is that they are inserted in the opposite order than one would expect (that is, opposite to what's used by the kernel to select them, and opposite to what's done for IPv4).

Quoted 6 lines
Quoted 2 lines

This (with an older kernel version) is quite "funny" (pasta by default copies everything it finds to the inner namespace):

Fwiw, a hypothetical tool which copied network configuration from one namespace to an independent one - or which saved network configuration to restore it later - would likely hit the same issue.

It's not even hypothetical: iproute2 has that functionality (ip address save / restore) and addresses are restored in the wrong order on kernel versions without this change:

# ip a a ::2 dev x && ip a a ::3 dev x
# ip a s x
2: x: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether c2:72:d1:7b:ac:26 brd ff:ff:ff:ff:ff:ff
    inet6 ::3/128 scope global tentative 
       valid_lft forever preferred_lft forever
    inet6 ::2/128 scope global tentative 
       valid_lft forever preferred_lft forever
# ip -6 a save dev x > dump
# ip a f x
# ip a restore < dump
# ip a s x
2: x: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether c2:72:d1:7b:ac:26 brd ff:ff:ff:ff:ff:ff
    inet6 ::2/128 scope global tentative 
       valid_lft forever preferred_lft forever
    inet6 ::3/128 scope global tentative 
       valid_lft forever preferred_lft forever

So scripts using this are anyway broken (well, they would get the order right on every other save/restore, which makes the breakage even more subtle).

Signature · 1 line
Stefano