Skip to contentSkip to conversation

aquantia: Sunshine/Moonlight UDP video streaming broken since 5b4015ad833c ("net: aquantia: Remove redundant UDP length adjustment with GSO_PARTIAL")

REGRESSION Matthew Schwartz 2 participants 8 messages Apr 26 – May 12

Subsystems (3)
lore
Share & tools
lore
#0 lore raw

Hello,

When using a previously working setup of remote streaming from my workstation to another device via Sunshine (the host server) and Moonlight (the client app) on my home network, I no longer receive any video output on the client app after upgrading my host workstation to kernel 7.0. Reverting back to kernel 6.19 on the host restored my setup to a working state.

After bisecting, I landed on 5b4015ad833c ("net: aquantia: Remove redundant UDP length adjustment with GSO_PARTIAL") as the first bad commit. I confirmed this by moving the cable to my second on-board NIC (Intel) on the same workstation, which restored video output without any other kernel changes. My affected on-board NIC is Aquantia AQC113 [1d6a:04c0] (rev 03), atlantic driver, firmware 1.3.34, MTU 1500.

Looking into it a bit further, ethtool -K enp97s0 tx-udp-segmentation off also serves as a workaround on my Aquantia port without changing to my other ethernet port. The working Intel NIC reports tx-udp-segmentation as "off [fixed]", so traffic falls back to software UDP segmentation on there.

Please let me know if there's any additional info I can provide.

Thanks,
Matt

#regzbot introduced: 5b4015ad833c

#1 lore raw

Hello Matthew,

Quoted 6 lines · Matthew Schwartz

Hello,

When using a previously working setup of remote streaming from my workstation to another device via Sunshine (the host server) and Moonlight (the client app) on my home network, I no longer receive any video output on the client app after upgrading my host workstation to kernel 7.0. Reverting back to kernel 6.19 on the host restored my setup to a working state.

After bisecting, I landed on 5b4015ad833c ("net: aquantia: Remove redundant UDP length adjustment with GSO_PARTIAL") as the first bad commit. I confirmed this by moving the cable to my second on-board NIC (Intel) on the same workstation, which restored video output without any other kernel changes. My affected on-board NIC is Aquantia AQC113 [1d6a:04c0] (rev 03), atlantic driver, firmware 1.3.34, MTU 1500.

Quoted 8 lines

Looking into it a bit further, ethtool -K enp97s0 tx-udp-segmentation off also serves as a workaround on my Aquantia port without changing to my other ethernet port. The working Intel NIC reports tx-udp-segmentation as "off [fixed]", so traffic falls back to software UDP segmentation on there.

Please let me know if there's any additional info I can provide.

Thanks,
Matt

#regzbot introduced: 5b4015ad833c

Thank you for the report and the bisect!

I will take a look and try to figure out what's wrong (though I don't have real hardware to test on).
Is the userspace app open source? can I see its code and try to run it myself?

I will be OOO for the rest of the week, hope to have some meaningful reply by the end of next week.

#2 lore raw
Quoted 16 lines · Gal Pressman

Hello Matthew,

Quoted 13 lines · Matthew Schwartz

Hello,

When using a previously working setup of remote streaming from my workstation to another device via Sunshine (the host server) and Moonlight (the client app) on my home network, I no longer receive any video output on the client app after upgrading my host workstation to kernel 7.0. Reverting back to kernel 6.19 on the host restored my setup to a working state.

After bisecting, I landed on 5b4015ad833c ("net: aquantia: Remove redundant UDP length adjustment with GSO_PARTIAL") as the first bad commit. I confirmed this by moving the cable to my second on-board NIC (Intel) on the same workstation, which restored video output without any other kernel changes. My affected on-board NIC is Aquantia AQC113 [1d6a:04c0] (rev 03), atlantic driver, firmware 1.3.34, MTU 1500.

Looking into it a bit further, ethtool -K enp97s0 tx-udp-segmentation off also serves as a workaround on my Aquantia port without changing to my other ethernet port. The working Intel NIC reports tx-udp-segmentation as "off [fixed]", so traffic falls back to software UDP segmentation on there.

Please let me know if there's any additional info I can provide.

Thanks,
Matt

Quoted 8 lines
Quoted 1 line

#regzbot introduced: 5b4015ad833c

Thank you for the report and the bisect!

I will take a look and try to figure out what's wrong (though I don't have real hardware to test on).
Is the userspace app open source? can I see its code and try to run it myself?

Thanks for the reply. The code for Sunshine is available here: https://github.com/LizardByte/Sunshine and the code for Moonlight is here: https://github.com/moonlight-stream/moonlight-qt.

I have been using the Arch Linux Sunshine package which I installed by following the Linux instructions here: https://docs.lizardbyte.dev/projects/sunshine/latest/md_docs_2getting__started.html, but there are also binaries for other distros or it's buildable from source. For Moonlight, I have been using the Flatpak distributed on Flathub because the client device runs an atomic rootfs, but you can also use any other device that Moonlight supports.

Quoted 3 lines

I will be OOO for the rest of the week, hope to have some meaningful reply by the end of next week.

#3 lore raw
Quoted 21 lines · Matthew Schwartz
Quoted 20 lines · Gal Pressman

Hello Matthew,

Quoted 6 lines · Matthew Schwartz

Hello,

When using a previously working setup of remote streaming from my workstation to another device via Sunshine (the host server) and Moonlight (the client app) on my home network, I no longer receive any video output on the client app after upgrading my host workstation to kernel 7.0. Reverting back to kernel 6.19 on the host restored my setup to a working state.

After bisecting, I landed on 5b4015ad833c ("net: aquantia: Remove redundant UDP length adjustment with GSO_PARTIAL") as the first bad commit. I confirmed this by moving the cable to my second on-board NIC (Intel) on the same workstation, which restored video output without any other kernel changes. My affected on-board NIC is Aquantia AQC113 [1d6a:04c0] (rev 03), atlantic driver, firmware 1.3.34, MTU 1500.

Quoted 8 lines

Looking into it a bit further, ethtool -K enp97s0 tx-udp-segmentation off also serves as a workaround on my Aquantia port without changing to my other ethernet port. The working Intel NIC reports tx-udp-segmentation as "off [fixed]", so traffic falls back to software UDP segmentation on there.

Please let me know if there's any additional info I can provide.

Thanks,
Matt

#regzbot introduced: 5b4015ad833c

Thank you for the report and the bisect!

Quoted 13 lines
Quoted 4 lines

I will take a look and try to figure out what's wrong (though I don't have real hardware to test on).
Is the userspace app open source? can I see its code and try to run it myself?

Thanks for the reply. The code for Sunshine is available here: https://github.com/LizardByte/Sunshine and the code for Moonlight is here: https://github.com/moonlight-stream/moonlight-qt.

I have been using the Arch Linux Sunshine package which I installed by following the Linux instructions here: https://docs.lizardbyte.dev/projects/sunshine/latest/md_docs_2getting__started.html, but there are also binaries for other distros or it's buildable from source. For Moonlight, I have been using the Flatpak distributed on Flathub because the client device runs an atomic rootfs, but you can also use any other device that Moonlight supports.

Quoted 3 lines

I will be OOO for the rest of the week, hope to have some meaningful reply by the end of next week.

I think I see the issue, do you mind testing the following diff?

index a0813d425b71..5bd1706b11b0 100644

net/ipv4/udp_offload.c +10 -4
@@ -599,10 +599,22 @@ struct sk_buff *__udp_gso_segment(struct sk_buff *gso_skb,
 		uh = udp_hdr(seg);
 	}
 
-	/* last packet can be partial gso_size, account for that in checksum */
-	newlen = htons(skb_tail_pointer(seg) - skb_transport_header(seg) +
-		       seg->data_len);
-	check = csum16_add(csum16_sub(uh->check, uh->len), newlen);
+	if (skb_is_gso(seg)) {
+		newlen = msslen;
+	} else {
+		/* last packet can be partial gso_size, account for that in
+		 * checksum.
+		 */
+		newlen = htons(skb_tail_pointer(seg) -
+			       skb_transport_header(seg) + seg->data_len);
+		check = csum16_add(csum16_sub(uh->check, uh->len), newlen);
+	}
 
 	uh->len = newlen;
 	uh->check = check;
#4 lore raw
Quoted 33 lines · Gal Pressman
Quoted 28 lines · Matthew Schwartz
Quoted 17 lines · Gal Pressman

Hello Matthew,

Quoted 13 lines · Matthew Schwartz

Hello,

When using a previously working setup of remote streaming from my workstation to another device via Sunshine (the host server) and Moonlight (the client app) on my home network, I no longer receive any video output on the client app after upgrading my host workstation to kernel 7.0. Reverting back to kernel 6.19 on the host restored my setup to a working state.

After bisecting, I landed on 5b4015ad833c ("net: aquantia: Remove redundant UDP length adjustment with GSO_PARTIAL") as the first bad commit. I confirmed this by moving the cable to my second on-board NIC (Intel) on the same workstation, which restored video output without any other kernel changes. My affected on-board NIC is Aquantia AQC113 [1d6a:04c0] (rev 03), atlantic driver, firmware 1.3.34, MTU 1500.

Looking into it a bit further, ethtool -K enp97s0 tx-udp-segmentation off also serves as a workaround on my Aquantia port without changing to my other ethernet port. The working Intel NIC reports tx-udp-segmentation as "off [fixed]", so traffic falls back to software UDP segmentation on there.

Please let me know if there's any additional info I can provide.

Thanks,
Matt

Quoted 8 lines
Quoted 1 line

#regzbot introduced: 5b4015ad833c

Thank you for the report and the bisect!

I will take a look and try to figure out what's wrong (though I don't have real hardware to test on).
Is the userspace app open source? can I see its code and try to run it myself?

Thanks for the reply. The code for Sunshine is available here: https://github.com/LizardByte/Sunshine and the code for Moonlight is here: https://github.com/moonlight-stream/moonlight-qt.

Quoted 8 lines
Quoted 6 lines

I have been using the Arch Linux Sunshine package which I installed by following the Linux instructions here: https://docs.lizardbyte.dev/projects/sunshine/latest/md_docs_2getting__started.html, but there are also binaries for other distros or it's buildable from source. For Moonlight, I have been using the Flatpak distributed on Flathub because the client device runs an atomic rootfs, but you can also use any other device that Moonlight supports.

Quoted 3 lines

I will be OOO for the rest of the week, hope to have some meaningful reply by the end of next week.

I think I see the issue, do you mind testing the following diff?

Sorry for the delay. I applied this diff and I’m able to stream from my host to clients again without any issues.

If you send this version upstream, feel free to add my Tested-by to it.

Thanks,
Matt

Quoted 25 lines

index a0813d425b71..5bd1706b11b0 100644

net/ipv4/udp_offload.c +10 -4
@@ -599,10 +599,22 @@ struct sk_buff *__udp_gso_segment(struct sk_buff *gso_skb,
       uh = udp_hdr(seg);
   }
 
-    /* last packet can be partial gso_size, account for that in checksum */
-    newlen = htons(skb_tail_pointer(seg) - skb_transport_header(seg) +
-               seg->data_len);
-    check = csum16_add(csum16_sub(uh->check, uh->len), newlen);
+    if (skb_is_gso(seg)) {
+        newlen = msslen;
+    } else {
+        /* last packet can be partial gso_size, account for that in
+         * checksum.
+         */
+        newlen = htons(skb_tail_pointer(seg) -
+                   skb_transport_header(seg) + seg->data_len);
+        check = csum16_add(csum16_sub(uh->check, uh->len), newlen);
+    }
 
   uh->len = newlen;
   uh->check = check;
#5 lore raw
Quoted 40 lines · Matthew Schwartz
Quoted 40 lines · Gal Pressman
Quoted 34 lines · Matthew Schwartz
Quoted 17 lines · Gal Pressman

Hello Matthew,

Quoted 13 lines · Matthew Schwartz

Hello,

When using a previously working setup of remote streaming from my workstation to another device via Sunshine (the host server) and Moonlight (the client app) on my home network, I no longer receive any video output on the client app after upgrading my host workstation to kernel 7.0. Reverting back to kernel 6.19 on the host restored my setup to a working state.

After bisecting, I landed on 5b4015ad833c ("net: aquantia: Remove redundant UDP length adjustment with GSO_PARTIAL") as the first bad commit. I confirmed this by moving the cable to my second on-board NIC (Intel) on the same workstation, which restored video output without any other kernel changes. My affected on-board NIC is Aquantia AQC113 [1d6a:04c0] (rev 03), atlantic driver, firmware 1.3.34, MTU 1500.

Looking into it a bit further, ethtool -K enp97s0 tx-udp-segmentation off also serves as a workaround on my Aquantia port without changing to my other ethernet port. The working Intel NIC reports tx-udp-segmentation as "off [fixed]", so traffic falls back to software UDP segmentation on there.

Please let me know if there's any additional info I can provide.

Thanks,
Matt

Quoted 8 lines
Quoted 1 line

#regzbot introduced: 5b4015ad833c

Thank you for the report and the bisect!

I will take a look and try to figure out what's wrong (though I don't have real hardware to test on).
Is the userspace app open source? can I see its code and try to run it myself?

Thanks for the reply. The code for Sunshine is available here: https://github.com/LizardByte/Sunshine and the code for Moonlight is here: https://github.com/moonlight-stream/moonlight-qt.

I have been using the Arch Linux Sunshine package which I installed by following the Linux instructions here: https://docs.lizardbyte.dev/projects/sunshine/latest/md_docs_2getting__started.html, but there are also binaries for other distros or it's buildable from source. For Moonlight, I have been using the Flatpak distributed on Flathub because the client device runs an atomic rootfs, but you can also use any other device that Moonlight supports.

Quoted 3 lines

I will be OOO for the rest of the week, hope to have some meaningful reply by the end of next week.

Quoted 5 lines
Quoted 1 line

I think I see the issue, do you mind testing the following diff?

Sorry for the delay. I applied this diff and I’m able to stream from my host to clients again without any issues.

If you send this version upstream, feel free to add my Tested-by to it.

Thanks for verifying!

I will finalize this and submit a proper patch soon.

#6 lore raw
Quoted 43 lines · Gal Pressman
Quoted 42 lines · Matthew Schwartz
Quoted 41 lines · Gal Pressman
Quoted 21 lines · Matthew Schwartz
Quoted 21 lines · Gal Pressman

Hello Matthew,

Quoted 6 lines · Matthew Schwartz

Hello,

When using a previously working setup of remote streaming from my workstation to another device via Sunshine (the host server) and Moonlight (the client app) on my home network, I no longer receive any video output on the client app after upgrading my host workstation to kernel 7.0. Reverting back to kernel 6.19 on the host restored my setup to a working state.

After bisecting, I landed on 5b4015ad833c ("net: aquantia: Remove redundant UDP length adjustment with GSO_PARTIAL") as the first bad commit. I confirmed this by moving the cable to my second on-board NIC (Intel) on the same workstation, which restored video output without any other kernel changes. My affected on-board NIC is Aquantia AQC113 [1d6a:04c0] (rev 03), atlantic driver, firmware 1.3.34, MTU 1500.

Quoted 8 lines

Looking into it a bit further, ethtool -K enp97s0 tx-udp-segmentation off also serves as a workaround on my Aquantia port without changing to my other ethernet port. The working Intel NIC reports tx-udp-segmentation as "off [fixed]", so traffic falls back to software UDP segmentation on there.

Please let me know if there's any additional info I can provide.

Thanks,
Matt

#regzbot introduced: 5b4015ad833c

Thank you for the report and the bisect!

Quoted 13 lines
Quoted 4 lines

I will take a look and try to figure out what's wrong (though I don't have real hardware to test on).
Is the userspace app open source? can I see its code and try to run it myself?

Thanks for the reply. The code for Sunshine is available here: https://github.com/LizardByte/Sunshine and the code for Moonlight is here: https://github.com/moonlight-stream/moonlight-qt.

I have been using the Arch Linux Sunshine package which I installed by following the Linux instructions here: https://docs.lizardbyte.dev/projects/sunshine/latest/md_docs_2getting__started.html, but there are also binaries for other distros or it's buildable from source. For Moonlight, I have been using the Flatpak distributed on Flathub because the client device runs an atomic rootfs, but you can also use any other device that Moonlight supports.

Quoted 3 lines

I will be OOO for the rest of the week, hope to have some meaningful reply by the end of next week.

I think I see the issue, do you mind testing the following diff?

Quoted 7 lines
Quoted 3 lines

Sorry for the delay. I applied this diff and I’m able to stream from my host to clients again without any issues.

If you send this version upstream, feel free to add my Tested-by to it.

Thanks for verifying!

I will finalize this and submit a proper patch soon.

Hey Matthew,

I made a small change to the patch, can you please verify it still fixes the issue?

net/ipv4/udp_offload.c +1 -1
index a0813d425b71..71df45f9488a 100644
@@ -604,7 +604,7 @@ struct sk_buff *__udp_gso_segment(struct sk_buff *gso_skb,
 		       seg->data_len);
 	check = csum16_add(csum16_sub(uh->check, uh->len), newlen);
 
-	uh->len = newlen;
+	uh->len = skb_is_gso(seg) ? msslen : newlen;
 	uh->check = check;
 
 	if (seg->ip_summed == CHECKSUM_PARTIAL)
#7 lore raw
Quoted 48 lines · Gal Pressman
Quoted 47 lines · Gal Pressman
Quoted 45 lines · Matthew Schwartz
Quoted 33 lines · Gal Pressman
Quoted 28 lines · Matthew Schwartz
Quoted 17 lines · Gal Pressman

Hello Matthew,

Quoted 13 lines · Matthew Schwartz

Hello,

When using a previously working setup of remote streaming from my workstation to another device via Sunshine (the host server) and Moonlight (the client app) on my home network, I no longer receive any video output on the client app after upgrading my host workstation to kernel 7.0. Reverting back to kernel 6.19 on the host restored my setup to a working state.

After bisecting, I landed on 5b4015ad833c ("net: aquantia: Remove redundant UDP length adjustment with GSO_PARTIAL") as the first bad commit. I confirmed this by moving the cable to my second on-board NIC (Intel) on the same workstation, which restored video output without any other kernel changes. My affected on-board NIC is Aquantia AQC113 [1d6a:04c0] (rev 03), atlantic driver, firmware 1.3.34, MTU 1500.

Looking into it a bit further, ethtool -K enp97s0 tx-udp-segmentation off also serves as a workaround on my Aquantia port without changing to my other ethernet port. The working Intel NIC reports tx-udp-segmentation as "off [fixed]", so traffic falls back to software UDP segmentation on there.

Please let me know if there's any additional info I can provide.

Thanks,
Matt

Quoted 8 lines
Quoted 1 line

#regzbot introduced: 5b4015ad833c

Thank you for the report and the bisect!

I will take a look and try to figure out what's wrong (though I don't have real hardware to test on).
Is the userspace app open source? can I see its code and try to run it myself?

Thanks for the reply. The code for Sunshine is available here: https://github.com/LizardByte/Sunshine and the code for Moonlight is here: https://github.com/moonlight-stream/moonlight-qt.

Quoted 8 lines
Quoted 6 lines

I have been using the Arch Linux Sunshine package which I installed by following the Linux instructions here: https://docs.lizardbyte.dev/projects/sunshine/latest/md_docs_2getting__started.html, but there are also binaries for other distros or it's buildable from source. For Moonlight, I have been using the Flatpak distributed on Flathub because the client device runs an atomic rootfs, but you can also use any other device that Moonlight supports.

Quoted 3 lines

I will be OOO for the rest of the week, hope to have some meaningful reply by the end of next week.

I think I see the issue, do you mind testing the following diff?

Sorry for the delay. I applied this diff and I’m able to stream from my host to clients again without any issues.

If you send this version upstream, feel free to add my Tested-by to it.

Quoted 8 lines
Quoted 3 lines

Thanks for verifying!

I will finalize this and submit a proper patch soon.

Hey Matthew,

I made a small change to the patch, can you please verify it still fixes the issue?

Yes, this diff also fixes the issue.

Quoted 14 lines
net/ipv4/udp_offload.c +1 -1
index a0813d425b71..71df45f9488a 100644
@@ -604,7 +604,7 @@ struct sk_buff *__udp_gso_segment(struct sk_buff *gso_skb,
 		       seg->data_len);
 	check = csum16_add(csum16_sub(uh->check, uh->len), newlen);
 
-	uh->len = newlen;
+	uh->len = skb_is_gso(seg) ? msslen : newlen;
 	uh->check = check;
 
 	if (seg->ip_summed == CHECKSUM_PARTIAL)