STP on 80211s is broken in 6.4-rc4

regression Nicolas Escande 4 participants 11 messages Jun 6, 2023 – Jul 11, 2023

Subsystems (2)

802.11 (including cfg80211/NL80211) Regressions

lore

Share & tools

lore

Nicolas Escande

author

2023-06-06 10:57 UTC #0 lore raw

Hello Felix,

As user of the mesh part of mac80211 on multiple products at work let me say thank you for all the work you do on wifi, especially on 80211s, and especially the recent improvements you made for mesh fast RX/TX & cross vendor AMSDU compat

We upgraded our kernel from an older (5.15) to a newer 6.4. The problem is STP doesn't work anymore and alas we use it for now (for the better or worse).

What I gathered so far from my setup:
- we use ath9k & ath10k
- in my case STP frames are received as regular packet and not as amsdu
- the received packets have a wrong length of 44 in tcpdump (instead of 38 with our previous kernel)
- llc_fixup_skb() tries to pull some 41 bytes out of a 35 bytes packet this makes llc_rcv() discard the frames & breaks STP

From bisecting the culprit seems to be 986e43b19ae9176093da35e0a844e65c8bf9ede7 (wifi: mac80211: fix receiving A-MSDU frames on mesh interfaces)

I guess that your changes to handle both ampdu subframes & normal frames in the same datapath ends up putting a wrong skb->len for STP (multicast) frames ? Honestly I don't understand enough of the 80211 internals & spec to pinpoint the exact problem.

It seems this change was already in the 6.3 kernel so I guess someone should have seen it before (but I didn't find anything..) ? Maybe I missed something...

Anyway I'm happy to provide more info or try anything you throw at me.

Thanks,

---
Nicolas E.

Bagas Sanjaya

2023-06-10 06:44 UTC #1 lore raw

Quoted 25 lines · Nicolas Escande

Hello Felix,
As user of the mesh part of mac80211 on multiple products at work let me say thank you for all the work you do on wifi, especially on 80211s, and especially the recent improvements you made for mesh fast RX/TX & cross vendor AMSDU compat
We upgraded our kernel from an older (5.15) to a newer 6.4. The problem is STP doesn't work anymore and alas we use it for now (for the better or worse).
What I gathered so far from my setup:
- we use ath9k & ath10k
- in my case STP frames are received as regular packet and not as amsdu
- the received packets have a wrong length of 44 in tcpdump (instead of 38 with our previous kernel)
- llc_fixup_skb() tries to pull some 41 bytes out of a 35 bytes packet this makes llc_rcv() discard the frames & breaks STP
Quoted 1 line
From bisecting the culprit seems to be 986e43b19ae9176093da35e0a844e65c8bf9ede7
(wifi: mac80211: fix receiving A-MSDU frames on mesh interfaces)
I guess that your changes to handle both ampdu subframes & normal frames in the same datapath ends up putting a wrong skb->len for STP (multicast) frames ? Honestly I don't understand enough of the 80211 internals & spec to pinpoint the exact problem.

Quoted 5 lines

It seems this change was already in the 6.3 kernel so I guess someone should have seen it before (but I didn't find anything..) ? Maybe I missed something...
Anyway I'm happy to provide more info or try anything you throw at me.

Thanks for the regression report. I'm adding it to regzbot:

(Felix: it looks like this regression is introcued by a commit authored by you. Would you like to take a look on it?)

#regzbot ^introduced: 986e43b19ae917

Signature · 1 line

An old man doll... just what I always wanted! - Clara

Linux regression tracking (Thorsten Leemhuis)

2023-06-15 12:54 UTC #2 lore raw

Quoted 32 lines · Bagas Sanjaya

Quoted 25 lines · Nicolas Escande
Hello Felix,
As user of the mesh part of mac80211 on multiple products at work let me say thank you for all the work you do on wifi, especially on 80211s, and especially the recent improvements you made for mesh fast RX/TX & cross vendor AMSDU compat
We upgraded our kernel from an older (5.15) to a newer 6.4. The problem is STP doesn't work anymore and alas we use it for now (for the better or worse).
What I gathered so far from my setup:
- we use ath9k & ath10k
- in my case STP frames are received as regular packet and not as amsdu
- the received packets have a wrong length of 44 in tcpdump (instead of 38 with our previous kernel)
- llc_fixup_skb() tries to pull some 41 bytes out of a 35 bytes packet this makes llc_rcv() discard the frames & breaks STP
Quoted 1 line
From bisecting the culprit seems to be 986e43b19ae9176093da35e0a844e65c8bf9ede7
(wifi: mac80211: fix receiving A-MSDU frames on mesh interfaces)
I guess that your changes to handle both ampdu subframes & normal frames in the same datapath ends up putting a wrong skb->len for STP (multicast) frames ? Honestly I don't understand enough of the 80211 internals & spec to pinpoint the exact problem.
Quoted 5 lines
It seems this change was already in the 6.3 kernel so I guess someone should have seen it before (but I didn't find anything..) ? Maybe I missed something...
Anyway I'm happy to provide more info or try anything you throw at me.

Quoted 6 lines

Thanks for the regression report. I'm adding it to regzbot:
(Felix: it looks like this regression is introcued by a commit authored by you. Would you like to take a look on it?)
#regzbot ^introduced: 986e43b19ae917

Hmmm, Felix did not reply. But let's ignore that for now.

Nicolas, I noticed there are a few patches in next that refer to the culprit. Might be worth giving this series a try:

https://lore.kernel.org/all/[email protected]/

Ciao, Thorsten (wearing his 'the Linux kernel's regression tracker' hat) --
Everything you wanna know about Linux kernel regression tracking:
https://linux-regtracking.leemhuis.info/about/#tldr
If I did something stupid, please tell me, as explained on that page.

Nicolas Escande

author

2023-06-16 07:46 UTC #3 lore raw

Quoted 33 lines · Linux regression tracking (Thorsten Leemhuis)

Quoted 32 lines · Bagas Sanjaya
Quoted 30 lines · Nicolas Escande
Hello Felix,
As user of the mesh part of mac80211 on multiple products at work let me say thank you for all the work you do on wifi, especially on 80211s, and especially the recent improvements you made for mesh fast RX/TX & cross vendor AMSDU compat
We upgraded our kernel from an older (5.15) to a newer 6.4. The problem is STP doesn't work anymore and alas we use it for now (for the better or worse).
What I gathered so far from my setup:
- we use ath9k & ath10k
- in my case STP frames are received as regular packet and not as amsdu
- the received packets have a wrong length of 44 in tcpdump (instead of 38 with our previous kernel)
- llc_fixup_skb() tries to pull some 41 bytes out of a 35 bytes packet this makes llc_rcv() discard the frames & breaks STP
Quoted 1 line
From bisecting the culprit seems to be 986e43b19ae9176093da35e0a844e65c8bf9ede7
(wifi: mac80211: fix receiving A-MSDU frames on mesh interfaces)
I guess that your changes to handle both ampdu subframes & normal frames in the same datapath ends up putting a wrong skb->len for STP (multicast) frames ? Honestly I don't understand enough of the 80211 internals & spec to pinpoint the exact problem.
It seems this change was already in the 6.3 kernel so I guess someone should have seen it before (but I didn't find anything..) ? Maybe I missed something...
Anyway I'm happy to provide more info or try anything you throw at me.

Quoted 8 lines

Quoted 6 lines
Thanks for the regression report. I'm adding it to regzbot:
(Felix: it looks like this regression is introcued by a commit authored by you. Would you like to take a look on it?)
#regzbot ^introduced: 986e43b19ae917
Hmmm, Felix did not reply. But let's ignore that for now.

I haven't seen mails from felix on the list for a few days, I'm guessing he's unavailable for now but I'll hapilly wait.

Quoted 5 lines

Nicolas, I noticed there are a few patches in next that refer to the culprit. Might be worth giving this series a try:
https://lore.kernel.org/all/[email protected]/

Well this series already landed in 6.4 and that is the version I did my initial testing with. So no luck there.

Quoted 6 lines

Ciao, Thorsten (wearing his 'the Linux kernel's regression tracker' hat) --
Everything you wanna know about Linux kernel regression tracking:
https://linux-regtracking.leemhuis.info/about/#tldr
If I did something stupid, please tell me, as explained on that page.

Linux regression tracking (Thorsten Leemhuis)

2023-06-16 09:25 UTC #4 lore raw

Quoted 7 lines · Nicolas Escande

Quoted 4 lines · Linux regression tracking (Thorsten Leemhuis)
Quoted 1 line · Bagas Sanjaya
On Tue, Jun 06, 2023 at 12:55:57PM +0200, Nicolas Escande wrote:
Hmmm, Felix did not reply. But let's ignore that for now.
I haven't seen mails from felix on the list for a few days, I'm guessing he's unavailable for now but I'll hapilly wait.

Okay.

Quoted 5 lines

Quoted 3 lines
Nicolas, I noticed there are a few patches in next that refer to the culprit. Might be worth giving this series a try:
https://lore.kernel.org/all/[email protected]/
Well this series already landed in 6.4 and that is the version I did my initial testing with. So no luck there.

What? Ohh, sorry for the noise, I had missed that they were in mainline already.

Bagas Sanjaya

2023-06-16 12:17 UTC #5 lore raw

Quoted 11 lines · Linux regression tracking (Thorsten Leemhuis)

Quoted 7 lines · Nicolas Escande
Quoted 4 lines · Linux regression tracking (Thorsten Leemhuis)
Quoted 1 line · Bagas Sanjaya
On Tue, Jun 06, 2023 at 12:55:57PM +0200, Nicolas Escande wrote:
Hmmm, Felix did not reply. But let's ignore that for now.
I haven't seen mails from felix on the list for a few days, I'm guessing he's unavailable for now but I'll hapilly wait.
Okay.

Quoted 9 lines

Quoted 5 lines
Quoted 3 lines
Nicolas, I noticed there are a few patches in next that refer to the culprit. Might be worth giving this series a try:
https://lore.kernel.org/all/[email protected]/
Well this series already landed in 6.4 and that is the version I did my initial testing with. So no luck there.
What? Ohh, sorry for the noise, I had missed that they were in mainline already.

Hi Thorsten,

Should this be removed from tracking as inconclusive?

Signature · 1 line

An old man doll... just what I always wanted! - Clara

Linux regression tracking (Thorsten Leemhuis)

2023-06-16 12:33 UTC #6 lore raw

Quoted 15 lines · Bagas Sanjaya

Quoted 14 lines · Linux regression tracking (Thorsten Leemhuis)
Quoted 7 lines · Nicolas Escande
Quoted 4 lines · Linux regression tracking (Thorsten Leemhuis)
Quoted 1 line · Bagas Sanjaya
On Tue, Jun 06, 2023 at 12:55:57PM +0200, Nicolas Escande wrote:
Hmmm, Felix did not reply. But let's ignore that for now.
I haven't seen mails from felix on the list for a few days, I'm guessing he's unavailable for now but I'll hapilly wait.
Okay.
Quoted 3 lines
Quoted 3 lines
Nicolas, I noticed there are a few patches in next that refer to the culprit. Might be worth giving this series a try:
https://lore.kernel.org/all/[email protected]/

Quoted 7 lines

Quoted 5 lines
Quoted 2 lines
Well this series already landed in 6.4 and that is the version I did my initial testing with. So no luck there.
What? Ohh, sorry for the noise, I had missed that they were in mainline already.
Should this be removed from tracking as inconclusive?

Ehh, why? Afaics this is still a regression, just not one the reporter considers urgent; that is fine for me, unless more people start to report the problem.

Linux regression tracking (Thorsten Leemhuis)

2023-07-10 11:32 UTC #7 lore raw

In reply to Nicolas Escande

Quoted 30 lines · Nicolas Escande

Quoted 29 lines · Linux regression tracking (Thorsten Leemhuis)
Quoted 28 lines · Bagas Sanjaya
Quoted 27 lines · Nicolas Escande
As user of the mesh part of mac80211 on multiple products at work let me say thank you for all the work you do on wifi, especially on 80211s, and especially the recent improvements you made for mesh fast RX/TX & cross vendor AMSDU compat
We upgraded our kernel from an older (5.15) to a newer 6.4. The problem is STP doesn't work anymore and alas we use it for now (for the better or worse).
What I gathered so far from my setup:
- we use ath9k & ath10k
- in my case STP frames are received as regular packet and not as amsdu
- the received packets have a wrong length of 44 in tcpdump (instead of 38 with our previous kernel)
- llc_fixup_skb() tries to pull some 41 bytes out of a 35 bytes packet this makes llc_rcv() discard the frames & breaks STP
Quoted 1 line
From bisecting the culprit seems to be 986e43b19ae9176093da35e0a844e65c8bf9ede7
(wifi: mac80211: fix receiving A-MSDU frames on mesh interfaces)
I guess that your changes to handle both ampdu subframes & normal frames in the same datapath ends up putting a wrong skb->len for STP (multicast) frames ? Honestly I don't understand enough of the 80211 internals & spec to pinpoint the exact problem.
It seems this change was already in the 6.3 kernel so I guess someone should have seen it before (but I didn't find anything..) ? Maybe I missed something...

Quoted 6 lines

Quoted 3 lines
Quoted 1 line
Quoted 1 line
Anyway I'm happy to provide more info or try anything you throw at me.
[...]
Hmmm, Felix did not reply. But let's ignore that for now.
I haven't seen mails from felix on the list for a few days, I'm guessing he's unavailable for now but I'll hapilly wait.

Still no progress. Hmmm. Are you still okay with that? I've seen no other reports about this, so waiting is somewhat (albeit not completely) fine for me if it is for you.

But in any case it might be good if you could recheck 6.5-rc1.

Nicolas Escande

author

2023-07-10 16:50 UTC #8 lore raw

Quoted 35 lines · Linux regression tracking (Thorsten Leemhuis)

Quoted 34 lines · Nicolas Escande
Quoted 32 lines · Linux regression tracking (Thorsten Leemhuis)
Quoted 25 lines · Bagas Sanjaya
Quoted 24 lines · Nicolas Escande
As user of the mesh part of mac80211 on multiple products at work let me say thank you for all the work you do on wifi, especially on 80211s, and especially the recent improvements you made for mesh fast RX/TX & cross vendor AMSDU compat
We upgraded our kernel from an older (5.15) to a newer 6.4. The problem is STP doesn't work anymore and alas we use it for now (for the better or worse).
What I gathered so far from my setup:
- we use ath9k & ath10k
- in my case STP frames are received as regular packet and not as amsdu
- the received packets have a wrong length of 44 in tcpdump (instead of 38 with our previous kernel)
- llc_fixup_skb() tries to pull some 41 bytes out of a 35 bytes packet this makes llc_rcv() discard the frames & breaks STP
Quoted 1 line
From bisecting the culprit seems to be 986e43b19ae9176093da35e0a844e65c8bf9ede7
(wifi: mac80211: fix receiving A-MSDU frames on mesh interfaces)
I guess that your changes to handle both ampdu subframes & normal frames in the same datapath ends up putting a wrong skb->len for STP (multicast) frames ? Honestly I don't understand enough of the 80211 internals & spec to pinpoint the exact problem.
Quoted 4 lines
Quoted 4 lines
It seems this change was already in the 6.3 kernel so I guess someone should have seen it before (but I didn't find anything..) ? Maybe I missed something...
Anyway I'm happy to provide more info or try anything you throw at me.
[...]
Hmmm, Felix did not reply. But let's ignore that for now.

Quoted 6 lines

Quoted 2 lines
I haven't seen mails from felix on the list for a few days, I'm guessing he's unavailable for now but I'll hapilly wait.
Still no progress. Hmmm. Are you still okay with that? I've seen no other reports about this, so waiting is somewhat (albeit not completely) fine for me if it is for you.

I'm not so surprised no one else reported it, using STP on wifi (and 802.11s) is not a really common thing to do, to be honest (and STP on wifi is unreliable). Even though some openwrt guys do it for sure, I'm guessing their kernel version is lagging behind...

Quoted 2 lines

But in any case it might be good if you could recheck 6.5-rc1.

Testing on 6.5 as a whole won't be as easy for me as testing a single patch on top of 6.4. I'll do my best to try but from what I saw nothing got merged that would even remotely help me on this issue.

I am not loosing hope that Felix or someone that understands this stuff better finds the time to look into this. I'm guessing it's the summer vacation effet.

Quoted 6 lines

Ciao, Thorsten (wearing his 'the Linux kernel's regression tracker' hat) --
Everything you wanna know about Linux kernel regression tracking:
https://linux-regtracking.leemhuis.info/about/#tldr
If I did something stupid, please tell me, as explained on that page.

Felix Fietkau

2023-07-11 11:12 UTC #9 lore raw

Quoted 46 lines · Nicolas Escande

Quoted 34 lines · Linux regression tracking (Thorsten Leemhuis)
Quoted 33 lines · Nicolas Escande
Quoted 31 lines · Linux regression tracking (Thorsten Leemhuis)
Quoted 24 lines · Bagas Sanjaya
Quoted 23 lines · Nicolas Escande
As user of the mesh part of mac80211 on multiple products at work let me say thank you for all the work you do on wifi, especially on 80211s, and especially the recent improvements you made for mesh fast RX/TX & cross vendor AMSDU compat
We upgraded our kernel from an older (5.15) to a newer 6.4. The problem is STP doesn't work anymore and alas we use it for now (for the better or worse).
What I gathered so far from my setup:
- we use ath9k & ath10k
- in my case STP frames are received as regular packet and not as amsdu
- the received packets have a wrong length of 44 in tcpdump (instead of 38 with our previous kernel)
- llc_fixup_skb() tries to pull some 41 bytes out of a 35 bytes packet this makes llc_rcv() discard the frames & breaks STP
Quoted 1 line
From bisecting the culprit seems to be 986e43b19ae9176093da35e0a844e65c8bf9ede7
(wifi: mac80211: fix receiving A-MSDU frames on mesh interfaces)
I guess that your changes to handle both ampdu subframes & normal frames in the same datapath ends up putting a wrong skb->len for STP (multicast) frames ? Honestly I don't understand enough of the 80211 internals & spec to pinpoint the exact problem.
Quoted 4 lines
Quoted 4 lines
It seems this change was already in the 6.3 kernel so I guess someone should have seen it before (but I didn't find anything..) ? Maybe I missed something...
Anyway I'm happy to provide more info or try anything you throw at me.
[...]
Hmmm, Felix did not reply. But let's ignore that for now.
Quoted 6 lines
Quoted 2 lines
I haven't seen mails from felix on the list for a few days, I'm guessing he's unavailable for now but I'll hapilly wait.
Still no progress. Hmmm. Are you still okay with that? I've seen no other reports about this, so waiting is somewhat (albeit not completely) fine for me if it is for you.
I'm not so surprised no one else reported it, using STP on wifi (and 802.11s) is not a really common thing to do, to be honest (and STP on wifi is unreliable). Even though some openwrt guys do it for sure, I'm guessing their kernel version is lagging behind...
Quoted 1 line

Quoted 7 lines

Quoted 1 line
But in any case it might be good if you could recheck 6.5-rc1.
Testing on 6.5 as a whole won't be as easy for me as testing a single patch on top of 6.4. I'll do my best to try but from what I saw nothing got merged that would even remotely help me on this issue.
I am not loosing hope that Felix or someone that understands this stuff better finds the time to look into this. I'm guessing it's the summer vacation effet.

Sorry for the delay. This should fix the regression, please test. I will submit it for 6.5 soon.
---

net/wireless/util.c +2 -0

@@ -580,6 +580,8 @@ int ieee80211_strip_8023_mesh_hdr(struct
 		hdrlen += ETH_ALEN + 2;
 	else if (!pskb_may_pull(skb, hdrlen))
 		return -EINVAL;
+	else
+		payload.eth.h_proto = htons(skb->len - hdrlen);
 
 	mesh_addr = skb->data + sizeof(payload.eth) + ETH_ALEN;
 	switch (payload.flags & MESH_FLAGS_AE) {

Nicolas Escande

author

2023-07-11 12:15 UTC #10 lore raw

Quoted 62 lines · Felix Fietkau

Quoted 47 lines · Nicolas Escande
Quoted 35 lines · Linux regression tracking (Thorsten Leemhuis)
Quoted 34 lines · Nicolas Escande
Quoted 32 lines · Linux regression tracking (Thorsten Leemhuis)
Quoted 25 lines · Bagas Sanjaya
Quoted 24 lines · Nicolas Escande
As user of the mesh part of mac80211 on multiple products at work let me say thank you for all the work you do on wifi, especially on 80211s, and especially the recent improvements you made for mesh fast RX/TX & cross vendor AMSDU compat
We upgraded our kernel from an older (5.15) to a newer 6.4. The problem is STP doesn't work anymore and alas we use it for now (for the better or worse).
What I gathered so far from my setup:
- we use ath9k & ath10k
- in my case STP frames are received as regular packet and not as amsdu
- the received packets have a wrong length of 44 in tcpdump (instead of 38 with our previous kernel)
- llc_fixup_skb() tries to pull some 41 bytes out of a 35 bytes packet this makes llc_rcv() discard the frames & breaks STP
Quoted 1 line
From bisecting the culprit seems to be 986e43b19ae9176093da35e0a844e65c8bf9ede7
(wifi: mac80211: fix receiving A-MSDU frames on mesh interfaces)
I guess that your changes to handle both ampdu subframes & normal frames in the same datapath ends up putting a wrong skb->len for STP (multicast) frames ? Honestly I don't understand enough of the 80211 internals & spec to pinpoint the exact problem.
Quoted 4 lines
Quoted 4 lines
It seems this change was already in the 6.3 kernel so I guess someone should have seen it before (but I didn't find anything..) ? Maybe I missed something...
Anyway I'm happy to provide more info or try anything you throw at me.
[...]
Hmmm, Felix did not reply. But let's ignore that for now.
Quoted 6 lines
Quoted 2 lines
I haven't seen mails from felix on the list for a few days, I'm guessing he's unavailable for now but I'll hapilly wait.
Still no progress. Hmmm. Are you still okay with that? I've seen no other reports about this, so waiting is somewhat (albeit not completely) fine for me if it is for you.
I'm not so surprised no one else reported it, using STP on wifi (and 802.11s) is not a really common thing to do, to be honest (and STP on wifi is unreliable). Even though some openwrt guys do it for sure, I'm guessing their kernel version is lagging behind...
Quoted 1 line
Quoted 7 lines
Quoted 1 line
But in any case it might be good if you could recheck 6.5-rc1.
Testing on 6.5 as a whole won't be as easy for me as testing a single patch on top of 6.4. I'll do my best to try but from what I saw nothing got merged that would even remotely help me on this issue.
I am not loosing hope that Felix or someone that understands this stuff better finds the time to look into this. I'm guessing it's the summer vacation effet.
Sorry for the delay. This should fix the regression, please test. I will submit it for 6.5 soon.
---

net/wireless/util.c int ieee80211_strip_8023_mesh_hdr(struct · line 587

Quoted 8 lines

net/wireless/util.c +2 -0

@@ -580,6 +580,8 @@ int ieee80211_strip_8023_mesh_hdr(struct
  		hdrlen += ETH_ALEN + 2;
  	else if (!pskb_may_pull(skb, hdrlen))
  		return -EINVAL;
+	else
+		payload.eth.h_proto = htons(skb->len - hdrlen);
 
  	mesh_addr = skb->data + sizeof(payload.eth) + ETH_ALEN;
  	switch (payload.flags & MESH_FLAGS_AE) {

Great, that does the trick for me
Thanks Felix