WARN_ON_ONCE() at rtl8169_tso_csum_v2()

Bug 209423 Heiner Kallweit 4 participants 19 messages Oct 1, 2020 – Jan 20, 2021 1 message not in the archive

Subsystems (3)

Networking Network drivers TCP

lore

Share & tools

lore

A message that is not in the archive (sent off-list or not yet received): [email protected]/

Heiner Kallweit

author

2020-10-01 20:34 UTC #1 lore raw

I have a problem with the following code in ndo_start_xmit() of the r8169 driver. A user reported the WARN being triggered due to gso_size > 0 and gso_type = 0. The chip supports TSO(6). The driver is widely used, therefore I'd expect much more such reports if it should be a common problem. Not sure what's special. My primary question: Is it a valid use case that gso_size is greater than 0, and no SKB_GSO_ flag is set?
Any hint would be appreciated.

u32 mss = shinfo->gso_size;

	if (mss) {
		if (shinfo->gso_type & SKB_GSO_TCPV4) {
			opts[0] |= TD1_GTSENV4;
		} else if (shinfo->gso_type & SKB_GSO_TCPV6) {
			if (skb_cow_head(skb, 0))
				return false;

			tcp_v6_gso_csum_prep(skb);
			opts[0] |= TD1_GTSENV6;
		} else {
			WARN_ON_ONCE(1);
		}

-------- Forwarded Message --------
Subject: [Bug 209423] WARN_ON_ONCE() at rtl8169_tso_csum_v2()
Date: Thu, 01 Oct 2020 19:19:24 +0000
From: [email protected]
To: [email protected]

https://bugzilla.kernel.org/show_bug.cgi?id=209423

--- Comment #7 from Damian Wrobel ([email protected]) --- Here it comes:

[86678.377120] ------------[ cut here ]------------
[86678.377155] gso_size = 1448, gso_type = 0x00000000
[86678.377381] WARNING: CPU: 0 PID: 0 at
drivers/net/ethernet/realtek/r8169_main.c:4095 rtl8169_start_xmit+0x489/0x800
[r8169]
[86678.377393] Modules linked in: tun nft_nat nft_masq nft_objref

nf_conntrack_netbios_ns nf_conntrack_broadcast nft_fib_inet nft_fib_ipv4 nft_fib_ipv6 nft_fib nft_reject_inet nf_reject_ipv4 nf_reject_ipv6 nft_reject nft_ct nft_chain_nat ip_set_hash_net ip6table_nat ip6table_mangle ip6table_raw ip6table_security iptable_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 iptable_mangle iptable_raw iptable_security ip_set nf_tables nfnetlink ip6table_filter ip6_tables iptable_filter sunrpc vfat fat snd_hda_codec_realtek edac_mce_amd snd_hda_codec_generic snd_hda_codec_hdmi ledtrig_audio kvm_amd snd_hda_intel snd_intel_dspcfg ccp snd_hda_codec kvm snd_hda_core snd_hwdep snd_pcm hp_wmi snd_timer wmi_bmof sparse_keymap irqbypass snd sp5100_tco i2c_piix4 soundcore k10temp fam15h_power rfkill_gpio rfkill acpi_cpufreq ip_tables xfs amdgpu iommu_v2 gpu_sched i2c_algo_bit ttm drm_kms_helper cec drm crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel ax88179_178a usbnet serio_raw r8169 mii

[86678.377442]  wmi video
[86678.377486] CPU: 0 PID: 0 Comm: swapper/0 Not tainted 5.8.12-201.fc32.x86_64
#1
[86678.377495] Hardware name: HP HP t630 Thin Client/8158, BIOS M40 v01.12
02/04/2020
[86678.377511] RIP: 0010:rtl8169_start_xmit+0x489/0x800 [r8169]
[86678.377521] Code: 10 0f 85 43 01 00 00 80 3d bb 20 01 00 00 0f 85 16 fe ff
ff 44 89 ee 48 c7 c7 b0 72 36 c0 c6 05 a4 20 01 00 01 e8 0d 33 d8 e1 <0f> 0b 44
8b 44 24 28 8b 74 24 2c 48 8b 8d c8 00 00 00 e9 e9 fd ff
[86678.377533] RSP: 0018:ffffa8f280003c80 EFLAGS: 00010282
[86678.377542] RAX: 0000000000000026 RBX: ffff8d331abc6000 RCX:
0000000000000000
[86678.377551] RDX: ffff8d331b427060 RSI: ffff8d331b418d00 RDI:
0000000000000300
[86678.377559] RBP: ffff8d32b5bb8200 R08: 00000000000003d0 R09:
000000000000000d
[86678.377576] R10: 0000000000000730 R11: ffffa8f280003b15 R12:
00000000000001c0
[86678.377596] R13: 00000000000005a8 R14: 0000000000000022 R15:
000000000000001c
[86678.377606] FS:  0000000000000000(0000) GS:ffff8d331b400000(0000)
knlGS:0000000000000000
[86678.377617] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[86678.377624] CR2: 00007fa516f64520 CR3: 00000000b6de6000 CR4:
00000000001406f0
[86678.377632] Call Trace:
[86678.377641]  <IRQ>
[86678.377657]  dev_hard_start_xmit+0x8d/0x1d0
[86678.377676]  sch_direct_xmit+0xeb/0x2f0
[86678.377687]  __dev_queue_xmit+0x710/0x8a0
[86678.377713]  ? nf_confirm+0xcb/0xf0 [nf_conntrack]
[86678.377725]  ? nf_hook_slow+0x3f/0xb0
[86678.377735]  ip_finish_output2+0x2ad/0x560
[86678.377746]  __netif_receive_skb_core+0x4f0/0xf40
[86678.377758]  ? packet_rcv+0x44/0x490
[86678.377770]  __netif_receive_skb_one_core+0x2d/0x70
[86678.377779]  process_backlog+0x96/0x160
[86678.377789]  net_rx_action+0x13c/0x3e0
[86678.377804]  ? usbnet_bh+0x24/0x2b0 [usbnet]
[86678.377815]  __do_softirq+0xd9/0x2c4
[86678.377825]  asm_call_on_stack+0x12/0x20
[86678.377835]  </IRQ>
[86678.377845]  do_softirq_own_stack+0x39/0x50
[86678.377855]  irq_exit_rcu+0xc2/0x100
[86678.377865]  common_interrupt+0x75/0x140
[86678.377875]  asm_common_interrupt+0x1e/0x40
[86678.377885] RIP: 0010:cpuidle_enter_state+0xb6/0x3f0
[86678.377894] Code: e0 ab 6b 5d e8 ab c4 7b ff 49 89 c7 0f 1f 44 00 00 31 ff
e8 7c dd 7b ff 80 7c 24 0f 00 0f 85 d4 01 00 00 fb 66 0f 1f 44 00 00 <45> 85 e4
0f 88 e0 01 00 00 49 63 d4 4c 2b 7c 24 10 48 8d 04 52 48
[86678.377907] RSP: 0018:ffffffffa3a03e58 EFLAGS: 00000246
[86678.377915] RAX: ffff8d331b42a2c0 RBX: ffff8d3312f3e400 RCX:
000000000000001f
[86678.377923] RDX: 0000000000000000 RSI: 00000000401ec2e2 RDI:
0000000000000000
[86678.377931] RBP: ffffffffa3b78960 R08: 00004ed561df8e36 R09:
0000000000000006
[86678.377939] R10: 000000000000001d R11: 000000000000000e R12:
0000000000000002
[86678.377956] R13: ffff8d3312f3e400 R14: 0000000000000002 R15:
00004ed561df8e36
[86678.377970]  ? cpuidle_enter_state+0xa4/0x3f0
[86678.377980]  cpuidle_enter+0x29/0x40
[86678.377990]  do_idle+0x1d5/0x2a0
[86678.377999]  cpu_startup_entry+0x19/0x20
[86678.378009]  start_kernel+0x7f4/0x804
[86678.378022]  secondary_startup_64+0xb6/0xc0
[86678.378032] ---[ end trace 263bcddb7119c953 ]---

Signature · 2 lines

You are receiving this mail because:
You are on the CC list for the bug.

Eric Dumazet

2020-10-02 08:26 UTC #2 lore raw

Quoted 11 lines · Heiner Kallweit

I have a problem with the following code in ndo_start_xmit() of the r8169 driver. A user reported the WARN being triggered due to gso_size > 0 and gso_type = 0. The chip supports TSO(6). The driver is widely used, therefore I'd expect much more such reports if it should be a common problem. Not sure what's special. My primary question: Is it a valid use case that gso_size is greater than 0, and no SKB_GSO_ flag is set?
Any hint would be appreciated.

Maybe this is not a TCP packet ? But in this case GSO should have taken place.

You might add a
pr_err_once("gso_type=%x\n", shinfo->gso_type);

Quoted 124 lines

u32 mss = shinfo->gso_size;

        if (mss) {



                if (shinfo->gso_type & SKB_GSO_TCPV4) {
                        opts[0] |= TD1_GTSENV4;
                } else if (shinfo->gso_type & SKB_GSO_TCPV6) {
                        if (skb_cow_head(skb, 0))
                                return false;

                        tcp_v6_gso_csum_prep(skb);
                        opts[0] |= TD1_GTSENV6;
                } else {
                        WARN_ON_ONCE(1);
                }

-------- Forwarded Message --------
Subject: [Bug 209423] WARN_ON_ONCE() at rtl8169_tso_csum_v2()
Date: Thu, 01 Oct 2020 19:19:24 +0000
From: [email protected]
To: [email protected]

https://bugzilla.kernel.org/show_bug.cgi?id=209423

--- Comment #7 from Damian Wrobel ([email protected]) --- Here it comes:

[86678.377120] ------------[ cut here ]------------
[86678.377155] gso_size = 1448, gso_type = 0x00000000
[86678.377381] WARNING: CPU: 0 PID: 0 at
drivers/net/ethernet/realtek/r8169_main.c:4095 rtl8169_start_xmit+0x489/0x800
[r8169]
[86678.377393] Modules linked in: tun nft_nat nft_masq nft_objref

[86678.377442]  wmi video
[86678.377486] CPU: 0 PID: 0 Comm: swapper/0 Not tainted 5.8.12-201.fc32.x86_64
#1
[86678.377495] Hardware name: HP HP t630 Thin Client/8158, BIOS M40 v01.12
02/04/2020
[86678.377511] RIP: 0010:rtl8169_start_xmit+0x489/0x800 [r8169]
[86678.377521] Code: 10 0f 85 43 01 00 00 80 3d bb 20 01 00 00 0f 85 16 fe ff
ff 44 89 ee 48 c7 c7 b0 72 36 c0 c6 05 a4 20 01 00 01 e8 0d 33 d8 e1 <0f> 0b 44
8b 44 24 28 8b 74 24 2c 48 8b 8d c8 00 00 00 e9 e9 fd ff
[86678.377533] RSP: 0018:ffffa8f280003c80 EFLAGS: 00010282
[86678.377542] RAX: 0000000000000026 RBX: ffff8d331abc6000 RCX:
0000000000000000
[86678.377551] RDX: ffff8d331b427060 RSI: ffff8d331b418d00 RDI:
0000000000000300
[86678.377559] RBP: ffff8d32b5bb8200 R08: 00000000000003d0 R09:
000000000000000d
[86678.377576] R10: 0000000000000730 R11: ffffa8f280003b15 R12:
00000000000001c0
[86678.377596] R13: 00000000000005a8 R14: 0000000000000022 R15:
000000000000001c
[86678.377606] FS:  0000000000000000(0000) GS:ffff8d331b400000(0000)
knlGS:0000000000000000
[86678.377617] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[86678.377624] CR2: 00007fa516f64520 CR3: 00000000b6de6000 CR4:
00000000001406f0
[86678.377632] Call Trace:
[86678.377641]  <IRQ>
[86678.377657]  dev_hard_start_xmit+0x8d/0x1d0
[86678.377676]  sch_direct_xmit+0xeb/0x2f0
[86678.377687]  __dev_queue_xmit+0x710/0x8a0
[86678.377713]  ? nf_confirm+0xcb/0xf0 [nf_conntrack]
[86678.377725]  ? nf_hook_slow+0x3f/0xb0
[86678.377735]  ip_finish_output2+0x2ad/0x560
[86678.377746]  __netif_receive_skb_core+0x4f0/0xf40
[86678.377758]  ? packet_rcv+0x44/0x490
[86678.377770]  __netif_receive_skb_one_core+0x2d/0x70
[86678.377779]  process_backlog+0x96/0x160
[86678.377789]  net_rx_action+0x13c/0x3e0
[86678.377804]  ? usbnet_bh+0x24/0x2b0 [usbnet]
[86678.377815]  __do_softirq+0xd9/0x2c4
[86678.377825]  asm_call_on_stack+0x12/0x20
[86678.377835]  </IRQ>
[86678.377845]  do_softirq_own_stack+0x39/0x50
[86678.377855]  irq_exit_rcu+0xc2/0x100
[86678.377865]  common_interrupt+0x75/0x140
[86678.377875]  asm_common_interrupt+0x1e/0x40
[86678.377885] RIP: 0010:cpuidle_enter_state+0xb6/0x3f0
[86678.377894] Code: e0 ab 6b 5d e8 ab c4 7b ff 49 89 c7 0f 1f 44 00 00 31 ff
e8 7c dd 7b ff 80 7c 24 0f 00 0f 85 d4 01 00 00 fb 66 0f 1f 44 00 00 <45> 85 e4
0f 88 e0 01 00 00 49 63 d4 4c 2b 7c 24 10 48 8d 04 52 48
[86678.377907] RSP: 0018:ffffffffa3a03e58 EFLAGS: 00000246
[86678.377915] RAX: ffff8d331b42a2c0 RBX: ffff8d3312f3e400 RCX:
000000000000001f
[86678.377923] RDX: 0000000000000000 RSI: 00000000401ec2e2 RDI:
0000000000000000
[86678.377931] RBP: ffffffffa3b78960 R08: 00004ed561df8e36 R09:
0000000000000006
[86678.377939] R10: 000000000000001d R11: 000000000000000e R12:
0000000000000002
[86678.377956] R13: ffff8d3312f3e400 R14: 0000000000000002 R15:
00004ed561df8e36
[86678.377970]  ? cpuidle_enter_state+0xa4/0x3f0
[86678.377980]  cpuidle_enter+0x29/0x40
[86678.377990]  do_idle+0x1d5/0x2a0
[86678.377999]  cpu_startup_entry+0x19/0x20
[86678.378009]  start_kernel+0x7f4/0x804
[86678.378022]  secondary_startup_64+0xb6/0xc0
[86678.378032] ---[ end trace 263bcddb7119c953 ]---

--
You are receiving this mail because:
You are on the CC list for the bug.

Eric Dumazet

2020-10-02 08:32 UTC #3 lore raw

Quoted 13 lines · Eric Dumazet

Quoted 11 lines · Heiner Kallweit
I have a problem with the following code in ndo_start_xmit() of the r8169 driver. A user reported the WARN being triggered due to gso_size > 0 and gso_type = 0. The chip supports TSO(6). The driver is widely used, therefore I'd expect much more such reports if it should be a common problem. Not sure what's special. My primary question: Is it a valid use case that gso_size is greater than 0, and no SKB_GSO_ flag is set?
Any hint would be appreciated.

Quoted 40 lines

Maybe this is not a TCP packet ? But in this case GSO should have taken place.
You might add a
pr_err_once("gso_type=%x\n", shinfo->gso_type);
Quoted 35 lines
u32 mss = shinfo->gso_size;
        if (mss) {



                if (shinfo->gso_type & SKB_GSO_TCPV4) {
                        opts[0] |= TD1_GTSENV4;
                } else if (shinfo->gso_type & SKB_GSO_TCPV6) {
                        if (skb_cow_head(skb, 0))
                                return false;

                        tcp_v6_gso_csum_prep(skb);
                        opts[0] |= TD1_GTSENV6;
                } else {
                        WARN_ON_ONCE(1);
                }
-------- Forwarded Message --------
Subject: [Bug 209423] WARN_ON_ONCE() at rtl8169_tso_csum_v2()
Date: Thu, 01 Oct 2020 19:19:24 +0000
From: [email protected]
To: [email protected]
https://bugzilla.kernel.org/show_bug.cgi?id=209423
--- Comment #7 from Damian Wrobel ([email protected]) --- Here it comes:
[86678.377120] ------------[ cut here ]------------
[86678.377155] gso_size = 1448, gso_type = 0x00000000

Ah, sorry I see you already printed gso_type

Must then be a bug somewhere :/

Eric Dumazet

2020-10-02 08:46 UTC #4 lore raw

Quoted 19 lines · Eric Dumazet

Quoted 15 lines · Eric Dumazet
Quoted 11 lines · Heiner Kallweit
I have a problem with the following code in ndo_start_xmit() of the r8169 driver. A user reported the WARN being triggered due to gso_size > 0 and gso_type = 0. The chip supports TSO(6). The driver is widely used, therefore I'd expect much more such reports if it should be a common problem. Not sure what's special. My primary question: Is it a valid use case that gso_size is greater than 0, and no SKB_GSO_ flag is set?
Any hint would be appreciated.
Maybe this is not a TCP packet ? But in this case GSO should have taken place.

Quoted 8 lines

Quoted 3 lines
You might add a
pr_err_once("gso_type=%x\n", shinfo->gso_type);
Ah, sorry I see you already printed gso_type
Must then be a bug somewhere :/

napi_reuse_skb() does :

skb_shinfo(skb)->gso_type = 0;

It does _not_ clear gso_size.

I wonder if in some cases we could reuse an skb while gso_size is not zero.

Normally, we set it only from dev_gro_receive() when the skb is queued into GRO engine (status being GRO_HELD)

Heiner Kallweit

author

2020-10-02 11:10 UTC #5 lore raw

Quoted 32 lines · Eric Dumazet

Quoted 19 lines · Eric Dumazet
Quoted 15 lines · Eric Dumazet
Quoted 11 lines · Heiner Kallweit
I have a problem with the following code in ndo_start_xmit() of the r8169 driver. A user reported the WARN being triggered due to gso_size > 0 and gso_type = 0. The chip supports TSO(6). The driver is widely used, therefore I'd expect much more such reports if it should be a common problem. Not sure what's special. My primary question: Is it a valid use case that gso_size is greater than 0, and no SKB_GSO_ flag is set?
Any hint would be appreciated.
Maybe this is not a TCP packet ? But in this case GSO should have taken place.
Quoted 8 lines
Quoted 3 lines
You might add a
pr_err_once("gso_type=%x\n", shinfo->gso_type);
Ah, sorry I see you already printed gso_type
Must then be a bug somewhere :/
napi_reuse_skb() does :

Quoted 9 lines

skb_shinfo(skb)->gso_type = 0;
It does _not_ clear gso_size.
I wonder if in some cases we could reuse an skb while gso_size is not zero.
Normally, we set it only from dev_gro_receive() when the skb is queued into GRO engine (status being GRO_HELD)

Thanks Eric. I'm no expert that deep in the network stack and just wonder why napi_reuse_skb() re-initializes less fields in shinfo than __alloc_skb(). The latter one does a
memset(shinfo, 0, offsetof(struct skb_shared_info, dataref));

What I can do is letting the affected user test the following.

net/core/dev.c +1 -0

index 62b06523b..8e75399cc 100644
@@ -6088,6 +6088,7 @@ static void napi_reuse_skb(struct napi_struct *napi, struct sk_buff *skb)
 
 	skb->encapsulation = 0;
 	skb_shinfo(skb)->gso_type = 0;
+	skb_shinfo(skb)->gso_size = 0;
 	skb->truesize = SKB_TRUESIZE(skb_end_offset(skb));
 	skb_ext_reset(skb);

Signature · 1 line

2.28.0

Eric Dumazet

2020-10-02 11:49 UTC #6 lore raw

Quoted 40 lines · Heiner Kallweit

Quoted 38 lines · Eric Dumazet
Quoted 19 lines · Eric Dumazet
Quoted 15 lines · Eric Dumazet
Quoted 11 lines · Heiner Kallweit
I have a problem with the following code in ndo_start_xmit() of the r8169 driver. A user reported the WARN being triggered due to gso_size > 0 and gso_type = 0. The chip supports TSO(6). The driver is widely used, therefore I'd expect much more such reports if it should be a common problem. Not sure what's special. My primary question: Is it a valid use case that gso_size is greater than 0, and no SKB_GSO_ flag is set?
Any hint would be appreciated.
Maybe this is not a TCP packet ? But in this case GSO should have taken place.
Quoted 8 lines
Quoted 3 lines
You might add a
pr_err_once("gso_type=%x\n", shinfo->gso_type);
Ah, sorry I see you already printed gso_type
Must then be a bug somewhere :/
napi_reuse_skb() does :
skb_shinfo(skb)->gso_type = 0;
It does _not_ clear gso_size.
I wonder if in some cases we could reuse an skb while gso_size is not zero.

Quoted 8 lines

Quoted 3 lines
Normally, we set it only from dev_gro_receive() when the skb is queued into GRO engine (status being GRO_HELD)
Thanks Eric. I'm no expert that deep in the network stack and just wonder why napi_reuse_skb() re-initializes less fields in shinfo than __alloc_skb(). The latter one does a
memset(shinfo, 0, offsetof(struct skb_shared_info, dataref));

memset() over the whole thing is more expensive.

Here we know the prior state of some fields, while __alloc_skb() just got a piece of memory with random content.

Quoted 7 lines

What I can do is letting the affected user test the following.
net/core/dev.c +0 -0
index 62b06523b..8e75399cc 100644

net/core/dev.c static void napi_reuse_skb(struct napi_struct *napi, struct sk_buff *skb) · line 6094

Quoted 7 lines

net/core/dev.c +1 -0

@@ -6088,6 +6088,7 @@ static void napi_reuse_skb(struct napi_struct *napi, struct sk_buff *skb)
 
        skb->encapsulation = 0;
        skb_shinfo(skb)->gso_type = 0;
+       skb_shinfo(skb)->gso_size = 0;
        skb->truesize = SKB_TRUESIZE(skb_end_offset(skb));
        skb_ext_reset(skb);

As I hinted, this should not be needed.

For debugging purposes, I would rather do :

BUG_ON(skb_shinfo(skb)->gso_size);

Nothing in GRO stack will change gso_size, unless the packet is queued by GRO layer (after this, napi_reuse_skb() wont be called)

napi_reuse_skb() is only used when a packet has been aggregated to another, and at this point gso_size should be still 0.

Heiner Kallweit

author

2020-10-08 16:37 UTC #7 lore raw

Quoted 72 lines · Eric Dumazet

Quoted 40 lines · Heiner Kallweit
Quoted 38 lines · Eric Dumazet
Quoted 19 lines · Eric Dumazet
Quoted 15 lines · Eric Dumazet
Quoted 11 lines · Heiner Kallweit
I have a problem with the following code in ndo_start_xmit() of the r8169 driver. A user reported the WARN being triggered due to gso_size > 0 and gso_type = 0. The chip supports TSO(6). The driver is widely used, therefore I'd expect much more such reports if it should be a common problem. Not sure what's special. My primary question: Is it a valid use case that gso_size is greater than 0, and no SKB_GSO_ flag is set?
Any hint would be appreciated.
Maybe this is not a TCP packet ? But in this case GSO should have taken place.
Quoted 8 lines
Quoted 3 lines
You might add a
pr_err_once("gso_type=%x\n", shinfo->gso_type);
Ah, sorry I see you already printed gso_type
Must then be a bug somewhere :/
napi_reuse_skb() does :
skb_shinfo(skb)->gso_type = 0;
It does _not_ clear gso_size.
I wonder if in some cases we could reuse an skb while gso_size is not zero.
Quoted 8 lines
Quoted 3 lines
Normally, we set it only from dev_gro_receive() when the skb is queued into GRO engine (status being GRO_HELD)
Thanks Eric. I'm no expert that deep in the network stack and just wonder why napi_reuse_skb() re-initializes less fields in shinfo than __alloc_skb(). The latter one does a
memset(shinfo, 0, offsetof(struct skb_shared_info, dataref));
memset() over the whole thing is more expensive.
Here we know the prior state of some fields, while __alloc_skb() just got a piece of memory with random content.
Quoted 7 lines
What I can do is letting the affected user test the following.
net/core/dev.c +0 -0
index 62b06523b..8e75399cc 100644
net/core/dev.c static void napi_reuse_skb(struct napi_struct *napi, struct sk_buff *skb) · line 6094
Quoted 7 lines
net/core/dev.c +1 -0
@@ -6088,6 +6088,7 @@ static void napi_reuse_skb(struct napi_struct *napi, struct sk_buff *skb)
 
        skb->encapsulation = 0;
        skb_shinfo(skb)->gso_type = 0;
+       skb_shinfo(skb)->gso_size = 0;
        skb->truesize = SKB_TRUESIZE(skb_end_offset(skb));
        skb_ext_reset(skb);
 
As I hinted, this should not be needed.

Quoted 4 lines

For debugging purposes, I would rather do :
BUG_ON(skb_shinfo(skb)->gso_size);

We did the following for debugging:

net/core/dev.c +3 -0

index 62b06523b..4c943b774 100644
@@ -3491,6 +3491,9 @@ static netdev_features_t gso_features_check(const struct sk_buff *skb,
 {
 	u16 gso_segs = skb_shinfo(skb)->gso_segs;
 
+	if (!skb_shinfo(skb)->gso_type)
+		skb_warn_bad_offload(skb);
+
 	if (gso_segs > dev->gso_max_segs)
 		return features & ~NETIF_F_GSO_MASK;

Following skb then triggered the skb_warn_bad_offload. Not sure whether this helps to find out where in the network stack something goes wrong.

[236222.967236] skb len=134 headroom=778 headlen=134 tailroom=31536
                mac=(778,14) net=(792,20) trans=812
                shinfo(txflags=0 nr_frags=0 gso(size=568 type=0 segs=1))
                csum(0x0 ip_summed=1 complete_sw=0 valid=0 level=0)
                hash(0x0 sw=0 l4=0) proto=0x0800 pkttype=0 iif=4
[236222.967297] dev name=enp1s0 feat=0x0x00000100000041b2
[236222.967392] skb linear:   00000000: 00 13 3b a0 01 e8 7c d3 0a 2d 1b 3b 08 00 45 00
[236222.967404] skb linear:   00000010: 00 78 e2 e6 00 00 7b 06 52 e1 d8 3a d0 ce c0 a8
[236222.967415] skb linear:   00000020: a0 06 01 bb 8b c6 53 91 be 5e 6e 60 bd e2 80 18
[236222.967426] skb linear:   00000030: 01 13 5c f6 00 00 01 01 08 0a 3d d6 6a a3 63 ea
[236222.967437] skb linear:   00000040: 5c d9 17 03 03 00 3f af 00 01 84 45 e2 36 e4 6a
[236222.967454] skb linear:   00000050: 3d 76 a8 7f d7 12 fa 72 4b d1 d0 74 0d c1 49 77
[236222.967466] skb linear:   00000060: 8b a4 bb 04 e5 aa 03 61 d3 e6 1f c9 0d 3e 46 c8
[236222.967477] skb linear:   00000070: cd 1f 7d ce e8 a7 84 84 01 5d 1f b4 ee 4f 27 63
[236222.967488] skb linear:   00000080: d2 a1 ab 1f 26 1d

Quoted 7 lines

Nothing in GRO stack will change gso_size, unless the packet is queued by GRO layer (after this, napi_reuse_skb() wont be called)
napi_reuse_skb() is only used when a packet has been aggregated to another, and at this point gso_size should be still 0.

Eric Dumazet

2020-10-08 17:16 UTC #8 lore raw

Quoted 86 lines · Heiner Kallweit

Quoted 72 lines · Eric Dumazet
Quoted 40 lines · Heiner Kallweit
Quoted 38 lines · Eric Dumazet
Quoted 19 lines · Eric Dumazet
Quoted 15 lines · Eric Dumazet
Quoted 11 lines · Heiner Kallweit
I have a problem with the following code in ndo_start_xmit() of the r8169 driver. A user reported the WARN being triggered due to gso_size > 0 and gso_type = 0. The chip supports TSO(6). The driver is widely used, therefore I'd expect much more such reports if it should be a common problem. Not sure what's special. My primary question: Is it a valid use case that gso_size is greater than 0, and no SKB_GSO_ flag is set?
Any hint would be appreciated.
Maybe this is not a TCP packet ? But in this case GSO should have taken place.
Quoted 8 lines
Quoted 3 lines
You might add a
pr_err_once("gso_type=%x\n", shinfo->gso_type);
Ah, sorry I see you already printed gso_type
Must then be a bug somewhere :/
napi_reuse_skb() does :
skb_shinfo(skb)->gso_type = 0;
It does _not_ clear gso_size.
I wonder if in some cases we could reuse an skb while gso_size is not zero.
Quoted 8 lines
Quoted 3 lines
Normally, we set it only from dev_gro_receive() when the skb is queued into GRO engine (status being GRO_HELD)
Thanks Eric. I'm no expert that deep in the network stack and just wonder why napi_reuse_skb() re-initializes less fields in shinfo than __alloc_skb(). The latter one does a
memset(shinfo, 0, offsetof(struct skb_shared_info, dataref));
memset() over the whole thing is more expensive.
Here we know the prior state of some fields, while __alloc_skb() just got a piece of memory with random content.
Quoted 7 lines
What I can do is letting the affected user test the following.
net/core/dev.c +0 -0
index 62b06523b..8e75399cc 100644
net/core/dev.c static void napi_reuse_skb(struct napi_struct *napi, struct sk_buff *skb) · line 6094
Quoted 7 lines
net/core/dev.c +1 -0
@@ -6088,6 +6088,7 @@ static void napi_reuse_skb(struct napi_struct *napi, struct sk_buff *skb)
 
        skb->encapsulation = 0;
        skb_shinfo(skb)->gso_type = 0;
+       skb_shinfo(skb)->gso_size = 0;
        skb->truesize = SKB_TRUESIZE(skb_end_offset(skb));
        skb_ext_reset(skb);
 
As I hinted, this should not be needed.
Quoted 4 lines
For debugging purposes, I would rather do :
BUG_ON(skb_shinfo(skb)->gso_size);
We did the following for debugging:
net/core/dev.c +0 -0
index 62b06523b..4c943b774 100644

net/core/dev.c static netdev_features_t gso_features_check(const struct sk_buff *skb, · line 3495

Quoted 5 lines

net/core/dev.c +2 -0

@@ -3491,6 +3491,9 @@ static netdev_features_t gso_features_check(const struct sk_buff *skb,
 {
        u16 gso_segs = skb_shinfo(skb)->gso_segs;
 
+       if (!skb_shinfo(skb)->gso_type)
+               skb_warn_bad_offload(skb);

You also want to get a stack trace here, to give us the call graph.

Quoted 34 lines

net/core/dev.c +1 -0

+
        if (gso_segs > dev->gso_max_segs)
                return features & ~NETIF_F_GSO_MASK;

Following skb then triggered the skb_warn_bad_offload. Not sure whether this helps to find out where in the network stack something goes wrong.

[236222.967236] skb len=134 headroom=778 headlen=134 tailroom=31536
                mac=(778,14) net=(792,20) trans=812
                shinfo(txflags=0 nr_frags=0 gso(size=568 type=0 segs=1))
                csum(0x0 ip_summed=1 complete_sw=0 valid=0 level=0)
                hash(0x0 sw=0 l4=0) proto=0x0800 pkttype=0 iif=4
[236222.967297] dev name=enp1s0 feat=0x0x00000100000041b2
[236222.967392] skb linear:   00000000: 00 13 3b a0 01 e8 7c d3 0a 2d 1b 3b 08 00 45 00
[236222.967404] skb linear:   00000010: 00 78 e2 e6 00 00 7b 06 52 e1 d8 3a d0 ce c0 a8
[236222.967415] skb linear:   00000020: a0 06 01 bb 8b c6 53 91 be 5e 6e 60 bd e2 80 18
[236222.967426] skb linear:   00000030: 01 13 5c f6 00 00 01 01 08 0a 3d d6 6a a3 63 ea
[236222.967437] skb linear:   00000040: 5c d9 17 03 03 00 3f af 00 01 84 45 e2 36 e4 6a
[236222.967454] skb linear:   00000050: 3d 76 a8 7f d7 12 fa 72 4b d1 d0 74 0d c1 49 77
[236222.967466] skb linear:   00000060: 8b a4 bb 04 e5 aa 03 61 d3 e6 1f c9 0d 3e 46 c8
[236222.967477] skb linear:   00000070: cd 1f 7d ce e8 a7 84 84 01 5d 1f b4 ee 4f 27 63
[236222.967488] skb linear:   00000080: d2 a1 ab 1f 26 1d

Quoted 7 lines

Nothing in GRO stack will change gso_size, unless the packet is queued by GRO layer (after this, napi_reuse_skb() wont be called)
napi_reuse_skb() is only used when a packet has been aggregated to another, and at this point gso_size should be still 0.

Heiner Kallweit

author

2020-10-08 18:42 UTC #9 lore raw

Quoted 90 lines · Eric Dumazet

Quoted 89 lines · Heiner Kallweit
Quoted 72 lines · Eric Dumazet
Quoted 40 lines · Heiner Kallweit
Quoted 38 lines · Eric Dumazet
Quoted 19 lines · Eric Dumazet
Quoted 15 lines · Eric Dumazet
Quoted 11 lines · Heiner Kallweit
I have a problem with the following code in ndo_start_xmit() of the r8169 driver. A user reported the WARN being triggered due to gso_size > 0 and gso_type = 0. The chip supports TSO(6). The driver is widely used, therefore I'd expect much more such reports if it should be a common problem. Not sure what's special. My primary question: Is it a valid use case that gso_size is greater than 0, and no SKB_GSO_ flag is set?
Any hint would be appreciated.
Maybe this is not a TCP packet ? But in this case GSO should have taken place.
Quoted 8 lines
Quoted 3 lines
You might add a
pr_err_once("gso_type=%x\n", shinfo->gso_type);
Ah, sorry I see you already printed gso_type
Must then be a bug somewhere :/
napi_reuse_skb() does :
skb_shinfo(skb)->gso_type = 0;
It does _not_ clear gso_size.
I wonder if in some cases we could reuse an skb while gso_size is not zero.
Quoted 8 lines
Quoted 3 lines
Normally, we set it only from dev_gro_receive() when the skb is queued into GRO engine (status being GRO_HELD)
Thanks Eric. I'm no expert that deep in the network stack and just wonder why napi_reuse_skb() re-initializes less fields in shinfo than __alloc_skb(). The latter one does a
memset(shinfo, 0, offsetof(struct skb_shared_info, dataref));
memset() over the whole thing is more expensive.
Here we know the prior state of some fields, while __alloc_skb() just got a piece of memory with random content.
Quoted 7 lines
What I can do is letting the affected user test the following.
net/core/dev.c +0 -0
index 62b06523b..8e75399cc 100644
net/core/dev.c static void napi_reuse_skb(struct napi_struct *napi, struct sk_buff *skb) · line 6094
Quoted 7 lines
net/core/dev.c +1 -0
@@ -6088,6 +6088,7 @@ static void napi_reuse_skb(struct napi_struct *napi, struct sk_buff *skb)
 
        skb->encapsulation = 0;
        skb_shinfo(skb)->gso_type = 0;
+       skb_shinfo(skb)->gso_size = 0;
        skb->truesize = SKB_TRUESIZE(skb_end_offset(skb));
        skb_ext_reset(skb);
 
As I hinted, this should not be needed.
Quoted 4 lines
For debugging purposes, I would rather do :
BUG_ON(skb_shinfo(skb)->gso_size);
We did the following for debugging:
net/core/dev.c +0 -0
index 62b06523b..4c943b774 100644
@@ -3491,6 +3491,9 @@ static netdev_features_t gso_features_check(const struct sk_buff *skb,
 {
        u16 gso_segs = skb_shinfo(skb)->gso_segs;
 

Quoted 5 lines

net/core/dev.c static netdev_features_t gso_features_check(const struct sk_buff *skb, · line 3495
Quoted 2 lines
net/core/dev.c +2 -0
+       if (!skb_shinfo(skb)->gso_type)
+               skb_warn_bad_offload(skb);
You also want to get a stack trace here, to give us the call graph.

Here it comes, full story is in https://bugzilla.kernel.org/show_bug.cgi?id=209423

[236222.967498] ------------[ cut here ]------------
[236222.967508] r8169: caps=(0x00000100000041b2, 0x0000000000000000)
[236222.967668] WARNING: CPU: 0 PID: 0 at net/core/dev.c:3184 skb_warn_bad_offload+0x72/0xe0
[236222.967691] Modules linked in: tcp_diag udp_diag raw_diag inet_diag unix_diag tun nft_nat nft_masq nft_objref nf_conntrack_netbios_ns nf_conntrack_broadcast nft_fib_inet nft_fib_ipv4 nft_fib_ipv6 nft_fib nft_reject_inet nf_reject_ipv4 nf_reject_ipv6 nft_reject nft_ct nft_chain_nat ip_set_hash_net ip6table_nat ip6table_mangle ip6table_raw ip6table_security iptable_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 iptable_mangle iptable_raw iptable_security ip_set nf_tables nfnetlink ip6table_filter ip6_tables iptable_filter sunrpc vfat fat snd_hda_codec_realtek snd_hda_codec_generic edac_mce_amd ledtrig_audio kvm_amd snd_hda_codec_hdmi ccp snd_hda_intel snd_intel_dspcfg kvm snd_hda_codec snd_hda_core snd_hwdep irqbypass snd_pcm snd_timer snd hp_wmi sp5100_tco sparse_keymap wmi_bmof fam15h_power k10temp i2c_piix4 soundcore rfkill_gpio rfkill acpi_cpufreq ip_tables xfs amdgpu iommu_v2 gpu_sched i2c_algo_bit ttm drm_kms_helper cec crct10dif_pclmul crc32_pclmul crc32c_intel drm
[236222.967776]  ghash_clmulni_intel ax88179_178a serio_raw usbnet mii r8169 wmi video
[236222.967858] CPU: 0 PID: 0 Comm: swapper/0 Not tainted 5.8.12-203.fc32.x86_64 #1
[236222.967870] Hardware name: HP HP t630 Thin Client/8158, BIOS M40 v01.12 02/04/2020
[236222.967895] RIP: 0010:skb_warn_bad_offload+0x72/0xe0
[236222.967908] Code: 8d 95 c8 00 00 00 48 8d 88 e8 01 00 00 48 85 c0 48 c7 c0 d8 d7 15 a4 48 0f 44 c8 4c 89 e6 48 c7 c7 90 7b 47 a4 e8 04 85 72 ff <0f> 0b 5b 5d 41 5c c3 80 7d 00 00 49 c7 c4 3b 28 40 a4 74 ac be 25
[236222.967926] RSP: 0018:ffffa8f9c0003c80 EFLAGS: 00010282
[236222.967938] RAX: 0000000000000034 RBX: ffff8d7090f2cd00 RCX: 0000000000000000
[236222.967951] RDX: ffff8d709b427060 RSI: ffff8d709b418d00 RDI: 0000000000000300
[236222.967962] RBP: ffff8d709a9fc000 R08: 0000000000000406 R09: 0720072007200720
[236222.967974] R10: 0720072007200720 R11: 0729073007300730 R12: ffffffffc012e729
[236222.967986] R13: ffffa8f9c0003d3b R14: 0000000000000000 R15: ffff8d70367652ac
[236222.968000] FS:  0000000000000000(0000) GS:ffff8d709b400000(0000) knlGS:0000000000000000
[236222.968013] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[236222.968023] CR2: 00007f3cf5ebf010 CR3: 0000000113cc6000 CR4: 00000000001406f0
[236222.968035] Call Trace:
[236222.968047]  <IRQ>
[236222.968064]  netif_skb_features+0x25e/0x2c0
[236222.968084]  ? ipt_do_table+0x333/0x600 [ip_tables]
[236222.968098]  validate_xmit_skb+0x1d/0x300
[236222.968111]  validate_xmit_skb_list+0x48/0x70
[236222.968126]  sch_direct_xmit+0x129/0x2f0
[236222.968140]  __dev_queue_xmit+0x710/0x8a0
[236222.968184]  ? nf_confirm+0xcb/0xf0 [nf_conntrack]
[236222.968200]  ? nf_hook_slow+0x3f/0xb0
[236222.968214]  ip_finish_output2+0x2ad/0x560
[236222.968229]  __netif_receive_skb_core+0x4f0/0xf40
[236222.968244]  ? packet_rcv+0x44/0x490
[236222.968257]  __netif_receive_skb_one_core+0x2d/0x70
[236222.968277]  process_backlog+0x96/0x160
[236222.968290]  net_rx_action+0x13c/0x3e0
[236222.968312]  ? usbnet_bh+0x24/0x2b0 [usbnet]
[236222.968327]  __do_softirq+0xd9/0x2c4
[236222.968340]  asm_call_on_stack+0x12/0x20
[236222.968350]  </IRQ>
[236222.968362]  do_softirq_own_stack+0x39/0x50
[236222.968376]  irq_exit_rcu+0xc2/0x100
[236222.968389]  common_interrupt+0x75/0x140
[236222.968405]  asm_common_interrupt+0x1e/0x40
[236222.968427] RIP: 0010:native_safe_halt+0xe/0x10
[236222.968438] Code: 02 20 48 8b 00 a8 08 75 c4 e9 7b ff ff ff cc cc cc cc cc cc cc cc cc cc cc cc cc cc e9 07 00 00 00 0f 00 2d f6 69 49 00 fb f4 <c3> 90 e9 07 00 00 00 0f 00 2d e6 69 49 00 f4 c3 cc cc 0f 1f 44 00
[236222.968456] RSP: 0018:ffffffffa4a03e08 EFLAGS: 00000246
[236222.968467] RAX: 0000000000004000 RBX: 0000000000000001 RCX: 000000000000001f
[236222.968480] RDX: 4ec4ec4ec4ec4ec5 RSI: ffffffffa4b78960 RDI: ffff8d7092f45c00
[236222.968492] RBP: ffff8d709a288000 R08: 0000d6d7f20a4084 R09: 0000000000000006
[236222.968504] R10: 0000000000000022 R11: 000000000000000f R12: ffff8d709a288064
[236222.968515] R13: 0000000000000001 R14: 0000000000000001 R15: 0000000000000000
[236222.968535]  acpi_safe_halt+0x1b/0x30
[236222.968549]  acpi_idle_enter+0x27e/0x2e0
[236222.968566]  cpuidle_enter_state+0x81/0x3f0
[236222.968589]  cpuidle_enter+0x29/0x40
[236222.968602]  do_idle+0x1d5/0x2a0
[236222.968615]  cpu_startup_entry+0x19/0x20
[236222.968628]  start_kernel+0x7f4/0x804
[236222.968645]  secondary_startup_64+0xb6/0xc0
[236222.968659] ---[ end trace 8a4d7f639ad88505 ]---

Quoted 35 lines

net/core/dev.c +1 -0

+
        if (gso_segs > dev->gso_max_segs)
                return features & ~NETIF_F_GSO_MASK;

Following skb then triggered the skb_warn_bad_offload. Not sure whether this helps to find out where in the network stack something goes wrong.

[236222.967236] skb len=134 headroom=778 headlen=134 tailroom=31536
                mac=(778,14) net=(792,20) trans=812
                shinfo(txflags=0 nr_frags=0 gso(size=568 type=0 segs=1))
                csum(0x0 ip_summed=1 complete_sw=0 valid=0 level=0)
                hash(0x0 sw=0 l4=0) proto=0x0800 pkttype=0 iif=4
[236222.967297] dev name=enp1s0 feat=0x0x00000100000041b2
[236222.967392] skb linear:   00000000: 00 13 3b a0 01 e8 7c d3 0a 2d 1b 3b 08 00 45 00
[236222.967404] skb linear:   00000010: 00 78 e2 e6 00 00 7b 06 52 e1 d8 3a d0 ce c0 a8
[236222.967415] skb linear:   00000020: a0 06 01 bb 8b c6 53 91 be 5e 6e 60 bd e2 80 18
[236222.967426] skb linear:   00000030: 01 13 5c f6 00 00 01 01 08 0a 3d d6 6a a3 63 ea
[236222.967437] skb linear:   00000040: 5c d9 17 03 03 00 3f af 00 01 84 45 e2 36 e4 6a
[236222.967454] skb linear:   00000050: 3d 76 a8 7f d7 12 fa 72 4b d1 d0 74 0d c1 49 77
[236222.967466] skb linear:   00000060: 8b a4 bb 04 e5 aa 03 61 d3 e6 1f c9 0d 3e 46 c8
[236222.967477] skb linear:   00000070: cd 1f 7d ce e8 a7 84 84 01 5d 1f b4 ee 4f 27 63
[236222.967488] skb linear:   00000080: d2 a1 ab 1f 26 1d

Quoted 7 lines

Nothing in GRO stack will change gso_size, unless the packet is queued by GRO layer (after this, napi_reuse_skb() wont be called)
napi_reuse_skb() is only used when a packet has been aggregated to another, and at this point gso_size should be still 0.

Eric Dumazet

2020-10-08 18:50 UTC #10 lore raw

Quoted 152 lines · Heiner Kallweit

Quoted 90 lines · Eric Dumazet
Quoted 89 lines · Heiner Kallweit
Quoted 72 lines · Eric Dumazet
Quoted 40 lines · Heiner Kallweit
Quoted 38 lines · Eric Dumazet
Quoted 19 lines · Eric Dumazet
Quoted 15 lines · Eric Dumazet
Quoted 11 lines · Heiner Kallweit
I have a problem with the following code in ndo_start_xmit() of the r8169 driver. A user reported the WARN being triggered due to gso_size > 0 and gso_type = 0. The chip supports TSO(6). The driver is widely used, therefore I'd expect much more such reports if it should be a common problem. Not sure what's special. My primary question: Is it a valid use case that gso_size is greater than 0, and no SKB_GSO_ flag is set?
Any hint would be appreciated.
Maybe this is not a TCP packet ? But in this case GSO should have taken place.
Quoted 8 lines
Quoted 3 lines
You might add a
pr_err_once("gso_type=%x\n", shinfo->gso_type);
Ah, sorry I see you already printed gso_type
Must then be a bug somewhere :/
napi_reuse_skb() does :
skb_shinfo(skb)->gso_type = 0;
It does _not_ clear gso_size.
I wonder if in some cases we could reuse an skb while gso_size is not zero.
Quoted 8 lines
Quoted 3 lines
Normally, we set it only from dev_gro_receive() when the skb is queued into GRO engine (status being GRO_HELD)
Thanks Eric. I'm no expert that deep in the network stack and just wonder why napi_reuse_skb() re-initializes less fields in shinfo than __alloc_skb(). The latter one does a
memset(shinfo, 0, offsetof(struct skb_shared_info, dataref));
memset() over the whole thing is more expensive.
Here we know the prior state of some fields, while __alloc_skb() just got a piece of memory with random content.
Quoted 7 lines
What I can do is letting the affected user test the following.
net/core/dev.c +0 -0
index 62b06523b..8e75399cc 100644
net/core/dev.c static void napi_reuse_skb(struct napi_struct *napi, struct sk_buff *skb) · line 6094
Quoted 7 lines
net/core/dev.c +1 -0
@@ -6088,6 +6088,7 @@ static void napi_reuse_skb(struct napi_struct *napi, struct sk_buff *skb)
 
        skb->encapsulation = 0;
        skb_shinfo(skb)->gso_type = 0;
+       skb_shinfo(skb)->gso_size = 0;
        skb->truesize = SKB_TRUESIZE(skb_end_offset(skb));
        skb_ext_reset(skb);
 
As I hinted, this should not be needed.
Quoted 4 lines
For debugging purposes, I would rather do :
BUG_ON(skb_shinfo(skb)->gso_size);
We did the following for debugging:
net/core/dev.c +0 -0
index 62b06523b..4c943b774 100644
@@ -3491,6 +3491,9 @@ static netdev_features_t gso_features_check(const struct sk_buff *skb,
 {
        u16 gso_segs = skb_shinfo(skb)->gso_segs;
 
Quoted 5 lines
net/core/dev.c static netdev_features_t gso_features_check(const struct sk_buff *skb, · line 3495
Quoted 2 lines
net/core/dev.c +2 -0
+       if (!skb_shinfo(skb)->gso_type)
+               skb_warn_bad_offload(skb);
You also want to get a stack trace here, to give us the call graph.
Here it comes, full story is in https://bugzilla.kernel.org/show_bug.cgi?id=209423
[236222.967498] ------------[ cut here ]------------
[236222.967508] r8169: caps=(0x00000100000041b2, 0x0000000000000000)
[236222.967668] WARNING: CPU: 0 PID: 0 at net/core/dev.c:3184 skb_warn_bad_offload+0x72/0xe0
[236222.967691] Modules linked in: tcp_diag udp_diag raw_diag inet_diag unix_diag tun nft_nat nft_masq nft_objref nf_conntrack_netbios_ns nf_conntrack_broadcast nft_fib_inet nft_fib_ipv4 nft_fib_ipv6 nft_fib nft_reject_inet nf_reject_ipv4 nf_reject_ipv6 nft_reject nft_ct nft_chain_nat ip_set_hash_net ip6table_nat ip6table_mangle ip6table_raw ip6table_security iptable_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 iptable_mangle iptable_raw iptable_security ip_set nf_tables nfnetlink ip6table_filter ip6_tables iptable_filter sunrpc vfat fat snd_hda_codec_realtek snd_hda_codec_generic edac_mce_amd ledtrig_audio kvm_amd snd_hda_codec_hdmi ccp snd_hda_intel snd_intel_dspcfg kvm snd_hda_codec snd_hda_core snd_hwdep irqbypass snd_pcm snd_timer snd hp_wmi sp5100_tco sparse_keymap wmi_bmof fam15h_power k10temp i2c_piix4 soundcore rfkill_gpio rfkill acpi_cpufreq ip_tables xfs amdgpu iommu_v2 gpu_sched i2c_algo_bit ttm drm_kms_helper cec crct10dif_pclmul crc32_pclmul crc32c_intel drm
[236222.967776]  ghash_clmulni_intel ax88179_178a serio_raw usbnet mii r8169 wmi video
[236222.967858] CPU: 0 PID: 0 Comm: swapper/0 Not tainted 5.8.12-203.fc32.x86_64 #1
[236222.967870] Hardware name: HP HP t630 Thin Client/8158, BIOS M40 v01.12 02/04/2020
[236222.967895] RIP: 0010:skb_warn_bad_offload+0x72/0xe0
[236222.967908] Code: 8d 95 c8 00 00 00 48 8d 88 e8 01 00 00 48 85 c0 48 c7 c0 d8 d7 15 a4 48 0f 44 c8 4c 89 e6 48 c7 c7 90 7b 47 a4 e8 04 85 72 ff <0f> 0b 5b 5d 41 5c c3 80 7d 00 00 49 c7 c4 3b 28 40 a4 74 ac be 25
[236222.967926] RSP: 0018:ffffa8f9c0003c80 EFLAGS: 00010282
[236222.967938] RAX: 0000000000000034 RBX: ffff8d7090f2cd00 RCX: 0000000000000000
[236222.967951] RDX: ffff8d709b427060 RSI: ffff8d709b418d00 RDI: 0000000000000300
[236222.967962] RBP: ffff8d709a9fc000 R08: 0000000000000406 R09: 0720072007200720
[236222.967974] R10: 0720072007200720 R11: 0729073007300730 R12: ffffffffc012e729
[236222.967986] R13: ffffa8f9c0003d3b R14: 0000000000000000 R15: ffff8d70367652ac
[236222.968000] FS:  0000000000000000(0000) GS:ffff8d709b400000(0000) knlGS:0000000000000000
[236222.968013] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[236222.968023] CR2: 00007f3cf5ebf010 CR3: 0000000113cc6000 CR4: 00000000001406f0
[236222.968035] Call Trace:
[236222.968047]  <IRQ>
[236222.968064]  netif_skb_features+0x25e/0x2c0
[236222.968084]  ? ipt_do_table+0x333/0x600 [ip_tables]
[236222.968098]  validate_xmit_skb+0x1d/0x300
[236222.968111]  validate_xmit_skb_list+0x48/0x70
[236222.968126]  sch_direct_xmit+0x129/0x2f0
[236222.968140]  __dev_queue_xmit+0x710/0x8a0
[236222.968184]  ? nf_confirm+0xcb/0xf0 [nf_conntrack]
[236222.968200]  ? nf_hook_slow+0x3f/0xb0
[236222.968214]  ip_finish_output2+0x2ad/0x560
[236222.968229]  __netif_receive_skb_core+0x4f0/0xf40
[236222.968244]  ? packet_rcv+0x44/0x490
[236222.968257]  __netif_receive_skb_one_core+0x2d/0x70
[236222.968277]  process_backlog+0x96/0x160
[236222.968290]  net_rx_action+0x13c/0x3e0
[236222.968312]  ? usbnet_bh+0x24/0x2b0 [usbnet]
[236222.968327]  __do_softirq+0xd9/0x2c4
[236222.968340]  asm_call_on_stack+0x12/0x20
[236222.968350]  </IRQ>
[236222.968362]  do_softirq_own_stack+0x39/0x50
[236222.968376]  irq_exit_rcu+0xc2/0x100
[236222.968389]  common_interrupt+0x75/0x140
[236222.968405]  asm_common_interrupt+0x1e/0x40
[236222.968427] RIP: 0010:native_safe_halt+0xe/0x10
[236222.968438] Code: 02 20 48 8b 00 a8 08 75 c4 e9 7b ff ff ff cc cc cc cc cc cc cc cc cc cc cc cc cc cc e9 07 00 00 00 0f 00 2d f6 69 49 00 fb f4 <c3> 90 e9 07 00 00 00 0f 00 2d e6 69 49 00 f4 c3 cc cc 0f 1f 44 00
[236222.968456] RSP: 0018:ffffffffa4a03e08 EFLAGS: 00000246
[236222.968467] RAX: 0000000000004000 RBX: 0000000000000001 RCX: 000000000000001f
[236222.968480] RDX: 4ec4ec4ec4ec4ec5 RSI: ffffffffa4b78960 RDI: ffff8d7092f45c00
[236222.968492] RBP: ffff8d709a288000 R08: 0000d6d7f20a4084 R09: 0000000000000006
[236222.968504] R10: 0000000000000022 R11: 000000000000000f R12: ffff8d709a288064
[236222.968515] R13: 0000000000000001 R14: 0000000000000001 R15: 0000000000000000
[236222.968535]  acpi_safe_halt+0x1b/0x30

Quoted 10 lines

[236222.968549]  acpi_idle_enter+0x27e/0x2e0
[236222.968566]  cpuidle_enter_state+0x81/0x3f0
[236222.968589]  cpuidle_enter+0x29/0x40
[236222.968602]  do_idle+0x1d5/0x2a0
[236222.968615]  cpu_startup_entry+0x19/0x20
[236222.968628]  start_kernel+0x7f4/0x804
[236222.968645]  secondary_startup_64+0xb6/0xc0
[236222.968659] ---[ end trace 8a4d7f639ad88505 ]---

OK, it would be nice to know what is the input interface

if4 -> look at "ip link | grep 4:"

Then identifying the driver that built such a strange packet (32000 bytes allocated in skb->head)

ethtool -i ifname

Quoted 36 lines

Quoted 35 lines

net/core/dev.c +1 -0

+
        if (gso_segs > dev->gso_max_segs)
                return features & ~NETIF_F_GSO_MASK;

Following skb then triggered the skb_warn_bad_offload. Not sure whether this helps to find out where in the network stack something goes wrong.

[236222.967236] skb len=134 headroom=778 headlen=134 tailroom=31536
                mac=(778,14) net=(792,20) trans=812
                shinfo(txflags=0 nr_frags=0 gso(size=568 type=0 segs=1))
                csum(0x0 ip_summed=1 complete_sw=0 valid=0 level=0)
                hash(0x0 sw=0 l4=0) proto=0x0800 pkttype=0 iif=4
[236222.967297] dev name=enp1s0 feat=0x0x00000100000041b2
[236222.967392] skb linear:   00000000: 00 13 3b a0 01 e8 7c d3 0a 2d 1b 3b 08 00 45 00
[236222.967404] skb linear:   00000010: 00 78 e2 e6 00 00 7b 06 52 e1 d8 3a d0 ce c0 a8
[236222.967415] skb linear:   00000020: a0 06 01 bb 8b c6 53 91 be 5e 6e 60 bd e2 80 18
[236222.967426] skb linear:   00000030: 01 13 5c f6 00 00 01 01 08 0a 3d d6 6a a3 63 ea
[236222.967437] skb linear:   00000040: 5c d9 17 03 03 00 3f af 00 01 84 45 e2 36 e4 6a
[236222.967454] skb linear:   00000050: 3d 76 a8 7f d7 12 fa 72 4b d1 d0 74 0d c1 49 77
[236222.967466] skb linear:   00000060: 8b a4 bb 04 e5 aa 03 61 d3 e6 1f c9 0d 3e 46 c8
[236222.967477] skb linear:   00000070: cd 1f 7d ce e8 a7 84 84 01 5d 1f b4 ee 4f 27 63
[236222.967488] skb linear:   00000080: d2 a1 ab 1f 26 1d

Quoted 7 lines

Nothing in GRO stack will change gso_size, unless the packet is queued by GRO layer (after this, napi_reuse_skb() wont be called)
napi_reuse_skb() is only used when a packet has been aggregated to another, and at this point gso_size should be still 0.

Eric Dumazet

2020-10-08 19:07 UTC #11 lore raw

Quoted 11 lines · Eric Dumazet

OK, it would be nice to know what is the input interface
if4 -> look at "ip link | grep 4:"
Then identifying the driver that built such a strange packet (32000 bytes allocated in skb->head)
ethtool -i ifname

According to https://bugzilla.kernel.org/show_bug.cgi?id=209423

iif4 is the tun200 interface used by openvpn.

So this might be a tun bug, or lack of proper SKB_GSO_DODGY validation in our stack for buggy/malicious packets.

Heiner Kallweit

author

2020-10-08 20:54 UTC #12 lore raw

Quoted 15 lines · Eric Dumazet

Quoted 11 lines · Eric Dumazet
OK, it would be nice to know what is the input interface
if4 -> look at "ip link | grep 4:"
Then identifying the driver that built such a strange packet (32000 bytes allocated in skb->head)
ethtool -i ifname

Quoted 8 lines

According to https://bugzilla.kernel.org/show_bug.cgi?id=209423
iif4 is the tun200 interface used by openvpn.
So this might be a tun bug, or lack of proper SKB_GSO_DODGY validation in our stack for buggy/malicious packets.

Following old commit sounds like it might be related:
622e0ca1cd4d ("gro: Fix bogus gso_size on the first fraglist entry")

This code however was removed later in 58025e46ea2d ("net: gro: remove obsolete code from skb_gro_receive()")

Eric Dumazet

2020-10-09 08:29 UTC #13 lore raw

Quoted 25 lines · Heiner Kallweit

Quoted 15 lines · Eric Dumazet
Quoted 11 lines · Eric Dumazet
OK, it would be nice to know what is the input interface
if4 -> look at "ip link | grep 4:"
Then identifying the driver that built such a strange packet (32000 bytes allocated in skb->head)
ethtool -i ifname
Quoted 8 lines
According to https://bugzilla.kernel.org/show_bug.cgi?id=209423
iif4 is the tun200 interface used by openvpn.
So this might be a tun bug, or lack of proper SKB_GSO_DODGY validation in our stack for buggy/malicious packets.

Quoted 6 lines

Following old commit sounds like it might be related:
622e0ca1cd4d ("gro: Fix bogus gso_size on the first fraglist entry")
This code however was removed later in 58025e46ea2d ("net: gro: remove obsolete code from skb_gro_receive()")

GRO wont keep in its queues a GSO packet dev_gro_receive()
...
NAPI_GRO_CB(skb)->flush = skb_is_gso(skb) || skb_has_frag_list(skb);
...

Also note that tun no longer can inject a packet with a length of 134 bytes pretending to have gso_size == 538

Look at virtio_net_hdr_to_skb() and commits
6dd912f82680 ("net: check untrusted gso_size at kernel entry") 7c6d2ecbda83 ("net: be more gentle about silly gso requests coming from user")

Really looking at the skb layout I suspect some usbnet bug and a use-after-free.

ASAN build might help.

Juerg Haefliger

Acked-by Neal Cardwell

2021-01-19 23:36 UTC #14 lore raw

In reply to Eric Dumazet

Quoted 164 lines · On Thu, 8 Oct 2020 20:50:28 +0200 Eric Dumazet <[email protected]> wrote:

Quoted 152 lines · Heiner Kallweit
Quoted 90 lines · Eric Dumazet
Quoted 89 lines · Heiner Kallweit
Quoted 72 lines · Eric Dumazet
Quoted 40 lines · Heiner Kallweit
Quoted 38 lines · Eric Dumazet
Quoted 19 lines · Eric Dumazet
Quoted 15 lines · Eric Dumazet
Quoted 11 lines · Heiner Kallweit
I have a problem with the following code in ndo_start_xmit() of the r8169 driver. A user reported the WARN being triggered due to gso_size > 0 and gso_type = 0. The chip supports TSO(6). The driver is widely used, therefore I'd expect much more such reports if it should be a common problem. Not sure what's special. My primary question: Is it a valid use case that gso_size is greater than 0, and no SKB_GSO_ flag is set?
Any hint would be appreciated.
Maybe this is not a TCP packet ? But in this case GSO should have taken place.
Quoted 8 lines
Quoted 3 lines
You might add a
pr_err_once("gso_type=%x\n", shinfo->gso_type);
Ah, sorry I see you already printed gso_type
Must then be a bug somewhere :/
napi_reuse_skb() does :
skb_shinfo(skb)->gso_type = 0;
It does _not_ clear gso_size.
I wonder if in some cases we could reuse an skb while gso_size is not zero.
Quoted 8 lines
Quoted 3 lines
Normally, we set it only from dev_gro_receive() when the skb is queued into GRO engine (status being GRO_HELD)
Thanks Eric. I'm no expert that deep in the network stack and just wonder why napi_reuse_skb() re-initializes less fields in shinfo than __alloc_skb(). The latter one does a
memset(shinfo, 0, offsetof(struct skb_shared_info, dataref));
memset() over the whole thing is more expensive.
Here we know the prior state of some fields, while __alloc_skb() just got a piece of memory with random content.
Quoted 7 lines
What I can do is letting the affected user test the following.
net/core/dev.c +0 -0
index 62b06523b..8e75399cc 100644
net/core/dev.c static void napi_reuse_skb(struct napi_struct *napi, struct sk_buff *skb) · line 6094
Quoted 7 lines
net/core/dev.c +1 -0
@@ -6088,6 +6088,7 @@ static void napi_reuse_skb(struct napi_struct *napi, struct sk_buff *skb)
 
        skb->encapsulation = 0;
        skb_shinfo(skb)->gso_type = 0;
+       skb_shinfo(skb)->gso_size = 0;
        skb->truesize = SKB_TRUESIZE(skb_end_offset(skb));
        skb_ext_reset(skb);
 
As I hinted, this should not be needed.
Quoted 4 lines
For debugging purposes, I would rather do :
BUG_ON(skb_shinfo(skb)->gso_size);
We did the following for debugging:
net/core/dev.c +0 -0
index 62b06523b..4c943b774 100644
@@ -3491,6 +3491,9 @@ static netdev_features_t gso_features_check(const struct sk_buff *skb,
 {
        u16 gso_segs = skb_shinfo(skb)->gso_segs;
 
Quoted 5 lines
net/core/dev.c static netdev_features_t gso_features_check(const struct sk_buff *skb, · line 3495
Quoted 2 lines
net/core/dev.c +2 -0
+       if (!skb_shinfo(skb)->gso_type)
+               skb_warn_bad_offload(skb);  
You also want to get a stack trace here, to give us the call graph.
Here it comes, full story is in https://bugzilla.kernel.org/show_bug.cgi?id=209423
[236222.967498] ------------[ cut here ]------------
[236222.967508] r8169: caps=(0x00000100000041b2, 0x0000000000000000)
[236222.967668] WARNING: CPU: 0 PID: 0 at net/core/dev.c:3184 skb_warn_bad_offload+0x72/0xe0
[236222.967691] Modules linked in: tcp_diag udp_diag raw_diag inet_diag unix_diag tun nft_nat nft_masq nft_objref nf_conntrack_netbios_ns nf_conntrack_broadcast nft_fib_inet nft_fib_ipv4 nft_fib_ipv6 nft_fib nft_reject_inet nf_reject_ipv4 nf_reject_ipv6 nft_reject nft_ct nft_chain_nat ip_set_hash_net ip6table_nat ip6table_mangle ip6table_raw ip6table_security iptable_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 iptable_mangle iptable_raw iptable_security ip_set nf_tables nfnetlink ip6table_filter ip6_tables iptable_filter sunrpc vfat fat snd_hda_codec_realtek snd_hda_codec_generic edac_mce_amd ledtrig_audio kvm_amd snd_hda_codec_hdmi ccp snd_hda_intel snd_intel_dspcfg kvm snd_hda_codec snd_hda_core snd_hwdep irqbypass snd_pcm snd_timer snd hp_wmi sp5100_tco sparse_keymap wmi_bmof fam15h_power k10temp i2c_piix4 soundcore rfkill_gpio rfkill acpi_cpufreq ip_tables xfs amdgpu iommu_v2 gpu_sched i2c_algo_bit ttm drm_kms_helper cec crct10dif_pclmul crc32_pclmul crc32c_intel drm
[236222.967776]  ghash_clmulni_intel ax88179_178a serio_raw usbnet mii r8169 wmi video
[236222.967858] CPU: 0 PID: 0 Comm: swapper/0 Not tainted 5.8.12-203.fc32.x86_64 #1
[236222.967870] Hardware name: HP HP t630 Thin Client/8158, BIOS M40 v01.12 02/04/2020
[236222.967895] RIP: 0010:skb_warn_bad_offload+0x72/0xe0
[236222.967908] Code: 8d 95 c8 00 00 00 48 8d 88 e8 01 00 00 48 85 c0 48 c7 c0 d8 d7 15 a4 48 0f 44 c8 4c 89 e6 48 c7 c7 90 7b 47 a4 e8 04 85 72 ff <0f> 0b 5b 5d 41 5c c3 80 7d 00 00 49 c7 c4 3b 28 40 a4 74 ac be 25
[236222.967926] RSP: 0018:ffffa8f9c0003c80 EFLAGS: 00010282
[236222.967938] RAX: 0000000000000034 RBX: ffff8d7090f2cd00 RCX: 0000000000000000
[236222.967951] RDX: ffff8d709b427060 RSI: ffff8d709b418d00 RDI: 0000000000000300
[236222.967962] RBP: ffff8d709a9fc000 R08: 0000000000000406 R09: 0720072007200720
[236222.967974] R10: 0720072007200720 R11: 0729073007300730 R12: ffffffffc012e729
[236222.967986] R13: ffffa8f9c0003d3b R14: 0000000000000000 R15: ffff8d70367652ac
[236222.968000] FS:  0000000000000000(0000) GS:ffff8d709b400000(0000) knlGS:0000000000000000
[236222.968013] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[236222.968023] CR2: 00007f3cf5ebf010 CR3: 0000000113cc6000 CR4: 00000000001406f0
[236222.968035] Call Trace:
[236222.968047]  <IRQ>
[236222.968064]  netif_skb_features+0x25e/0x2c0
[236222.968084]  ? ipt_do_table+0x333/0x600 [ip_tables]
[236222.968098]  validate_xmit_skb+0x1d/0x300
[236222.968111]  validate_xmit_skb_list+0x48/0x70
[236222.968126]  sch_direct_xmit+0x129/0x2f0
[236222.968140]  __dev_queue_xmit+0x710/0x8a0
[236222.968184]  ? nf_confirm+0xcb/0xf0 [nf_conntrack]
[236222.968200]  ? nf_hook_slow+0x3f/0xb0
[236222.968214]  ip_finish_output2+0x2ad/0x560
[236222.968229]  __netif_receive_skb_core+0x4f0/0xf40
[236222.968244]  ? packet_rcv+0x44/0x490
[236222.968257]  __netif_receive_skb_one_core+0x2d/0x70
[236222.968277]  process_backlog+0x96/0x160
[236222.968290]  net_rx_action+0x13c/0x3e0
[236222.968312]  ? usbnet_bh+0x24/0x2b0 [usbnet]
[236222.968327]  __do_softirq+0xd9/0x2c4
[236222.968340]  asm_call_on_stack+0x12/0x20
[236222.968350]  </IRQ>
[236222.968362]  do_softirq_own_stack+0x39/0x50
[236222.968376]  irq_exit_rcu+0xc2/0x100
[236222.968389]  common_interrupt+0x75/0x140
[236222.968405]  asm_common_interrupt+0x1e/0x40
[236222.968427] RIP: 0010:native_safe_halt+0xe/0x10
[236222.968438] Code: 02 20 48 8b 00 a8 08 75 c4 e9 7b ff ff ff cc cc cc cc cc cc cc cc cc cc cc cc cc cc e9 07 00 00 00 0f 00 2d f6 69 49 00 fb f4 <c3> 90 e9 07 00 00 00 0f 00 2d e6 69 49 00 f4 c3 cc cc 0f 1f 44 00
[236222.968456] RSP: 0018:ffffffffa4a03e08 EFLAGS: 00000246
[236222.968467] RAX: 0000000000004000 RBX: 0000000000000001 RCX: 000000000000001f
[236222.968480] RDX: 4ec4ec4ec4ec4ec5 RSI: ffffffffa4b78960 RDI: ffff8d7092f45c00
[236222.968492] RBP: ffff8d709a288000 R08: 0000d6d7f20a4084 R09: 0000000000000006
[236222.968504] R10: 0000000000000022 R11: 000000000000000f R12: ffff8d709a288064
[236222.968515] R13: 0000000000000001 R14: 0000000000000001 R15: 0000000000000000
[236222.968535]  acpi_safe_halt+0x1b/0x30
Quoted 10 lines
[236222.968549]  acpi_idle_enter+0x27e/0x2e0
[236222.968566]  cpuidle_enter_state+0x81/0x3f0
[236222.968589]  cpuidle_enter+0x29/0x40
[236222.968602]  do_idle+0x1d5/0x2a0
[236222.968615]  cpu_startup_entry+0x19/0x20
[236222.968628]  start_kernel+0x7f4/0x804
[236222.968645]  secondary_startup_64+0xb6/0xc0
[236222.968659] ---[ end trace 8a4d7f639ad88505 ]---

Quoted 46 lines

OK, it would be nice to know what is the input interface
if4 -> look at "ip link | grep 4:"
Then identifying the driver that built such a strange packet (32000 bytes allocated in skb->head)
ethtool -i ifname
Quoted 35 lines
Quoted 35 lines
Quoted 35 lines
net/core/dev.c +1 -0
+
        if (gso_segs > dev->gso_max_segs)
                return features & ~NETIF_F_GSO_MASK;
 
Following skb then triggered the skb_warn_bad_offload. Not sure whether this helps to find out where in the network stack something goes wrong.
[236222.967236] skb len=134 headroom=778 headlen=134 tailroom=31536
                mac=(778,14) net=(792,20) trans=812
                shinfo(txflags=0 nr_frags=0 gso(size=568 type=0 segs=1))
                csum(0x0 ip_summed=1 complete_sw=0 valid=0 level=0)
                hash(0x0 sw=0 l4=0) proto=0x0800 pkttype=0 iif=4
[236222.967297] dev name=enp1s0 feat=0x0x00000100000041b2
[236222.967392] skb linear:   00000000: 00 13 3b a0 01 e8 7c d3 0a 2d 1b 3b 08 00 45 00
[236222.967404] skb linear:   00000010: 00 78 e2 e6 00 00 7b 06 52 e1 d8 3a d0 ce c0 a8
[236222.967415] skb linear:   00000020: a0 06 01 bb 8b c6 53 91 be 5e 6e 60 bd e2 80 18
[236222.967426] skb linear:   00000030: 01 13 5c f6 00 00 01 01 08 0a 3d d6 6a a3 63 ea
[236222.967437] skb linear:   00000040: 5c d9 17 03 03 00 3f af 00 01 84 45 e2 36 e4 6a
[236222.967454] skb linear:   00000050: 3d 76 a8 7f d7 12 fa 72 4b d1 d0 74 0d c1 49 77
[236222.967466] skb linear:   00000060: 8b a4 bb 04 e5 aa 03 61 d3 e6 1f c9 0d 3e 46 c8
[236222.967477] skb linear:   00000070: cd 1f 7d ce e8 a7 84 84 01 5d 1f b4 ee 4f 27 63
[236222.967488] skb linear:   00000080: d2 a1 ab 1f 26 1d
Quoted 7 lines
Nothing in GRO stack will change gso_size, unless the packet is queued by GRO layer (after this, napi_reuse_skb() wont be called)
napi_reuse_skb() is only used when a packet has been aggregated to another, and at this point gso_size should be still 0.

I seem to have stumbled over the same or a similar issue with a Raspberry Pi 3B+ running 5.11-rc4 and using the on-board lan78xx USB NIC. The Pi is used as a gateway. If I enable IP forwarding on the Pi and pound on eth0 [1], I get tons of the below warnings after a couple of seconds:

Jan 19 07:55:22 rpi-3b-plus-rev1d3-abfb kernel: [ 1914.744157] skb len=54 headroom=5194 headlen=54 tailroom=10816
Jan 19 07:55:22 rpi-3b-plus-rev1d3-abfb kernel: [ 1914.744157] mac=(5194,14) net=(5208,20) trans=5228
Jan 19 07:55:22 rpi-3b-plus-rev1d3-abfb kernel: [ 1914.744157] shinfo(txflags=0 nr_frags=0 gso(size=1448 type=0 segs=1))
Jan 19 07:55:22 rpi-3b-plus-rev1d3-abfb kernel: [ 1914.744157] csum(0xe505 ip_summed=0 complete_sw=0 valid=0 level=0)
Jan 19 07:55:22 rpi-3b-plus-rev1d3-abfb kernel: [ 1914.744157] hash(0x0 sw=0 l4=0) proto=0x0800 pkttype=0 iif=2
Jan 19 07:55:22 rpi-3b-plus-rev1d3-abfb kernel: [ 1914.774147] dev name=eth0 feat=0x0x0000010000114b09
Jan 19 07:55:22 rpi-3b-plus-rev1d3-abfb kernel: [ 1914.779355] skb linear:   00000000: e0 28 6d 9e b9 22 b8 27 eb 3e ab fb 08 00 45 00
Jan 19 07:55:22 rpi-3b-plus-rev1d3-abfb kernel: [ 1914.787365] skb linear:   00000010: 00 28 00 00 40 00 3f 06 41 d0 c0 a8 63 84 02 14
Jan 19 07:55:22 rpi-3b-plus-rev1d3-abfb kernel: [ 1914.795266] skb linear:   00000020: d3 bf ed 3e 01 bb d4 0f 88 7e 00 00 00 00 50 04
Jan 19 07:55:22 rpi-3b-plus-rev1d3-abfb kernel: [ 1914.803168] skb linear:   00000030: 00 00 6a 58 00 00
Jan 19 07:55:22 rpi-3b-plus-rev1d3-abfb kernel: [ 1914.808384] ------------[ cut here ]------------
Jan 19 07:55:22 rpi-3b-plus-rev1d3-abfb kernel: [ 1914.813200] lan78xx: caps=(0x0000010000114b09, 0x0000000000000000)
Jan 19 07:55:22 rpi-3b-plus-rev1d3-abfb kernel: [ 1914.819717] WARNING: CPU: 0 PID: 0 at net/core/dev.c:3197 skb_warn_bad_offload+0x84/0x100
Jan 19 07:55:22 rpi-3b-plus-rev1d3-abfb kernel: [ 1914.828190] Modules linked in:
Jan 19 07:55:22 rpi-3b-plus-rev1d3-abfb kernel: [ 1914.831354] CPU: 0 PID: 0 Comm: swapper/0 Not tainted 5.11.0-rc4 #103
Jan 19 07:55:22 rpi-3b-plus-rev1d3-abfb kernel: [ 1914.838009] Hardware name: Raspberry Pi 3 Model B Plus Rev 1.3 (DT)
Jan 19 07:55:22 rpi-3b-plus-rev1d3-abfb kernel: [ 1914.844478] pstate: 60000005 (nZCv daif -PAN -UAO -TCO BTYPE=--)
Jan 19 07:55:22 rpi-3b-plus-rev1d3-abfb kernel: [ 1914.850685] pc : skb_warn_bad_offload+0x84/0x100
Jan 19 07:55:22 rpi-3b-plus-rev1d3-abfb kernel: [ 1914.855464] lr : skb_warn_bad_offload+0x84/0x100
Jan 19 07:55:22 rpi-3b-plus-rev1d3-abfb kernel: [ 1914.860242] sp : ffff800010003850
Jan 19 07:55:22 rpi-3b-plus-rev1d3-abfb kernel: [ 1914.863665] x29: ffff800010003850 x28: ffff7a96fb196290 
Jan 19 07:55:22 rpi-3b-plus-rev1d3-abfb kernel: [ 1914.869160] x27: ffff7a96c5958300 x26: 0000000000000001 
Jan 19 07:55:22 rpi-3b-plus-rev1d3-abfb kernel: [ 1914.874654] x25: ffffa73eee323000 x24: ffff7a96ee84b000 
Jan 19 07:55:22 rpi-3b-plus-rev1d3-abfb kernel: [ 1914.880148] x23: ffffa73eee7f4f00 x22: 0000000000000000 
Jan 19 07:55:22 rpi-3b-plus-rev1d3-abfb kernel: [ 1914.885642] x21: ffffa73eee0327e0 x20: ffff7a96ee84b000 
Jan 19 07:55:22 rpi-3b-plus-rev1d3-abfb kernel: [ 1914.891136] x19: ffff7a96c5958300 x18: 0000000000000010 
Jan 19 07:55:22 rpi-3b-plus-rev1d3-abfb kernel: [ 1914.896630] x17: 0000000000000000 x16: 0000000000000000 
Jan 19 07:55:22 rpi-3b-plus-rev1d3-abfb kernel: [ 1914.902123] x15: 000000000000ad55 x14: 0000000000000010 
Jan 19 07:55:22 rpi-3b-plus-rev1d3-abfb kernel: [ 1914.907617] x13: 00000000ffffffff x12: ffffa73eedd9d950 
Jan 19 07:55:22 rpi-3b-plus-rev1d3-abfb kernel: [ 1914.913109] x11: ffffa73eee885de0 x10: ffffa73eee86dda0 
Jan 19 07:55:22 rpi-3b-plus-rev1d3-abfb kernel: [ 1914.918603] x9 : ffffa73eecf2f45c x8 : 0000000000017fe8 
Jan 19 07:55:22 rpi-3b-plus-rev1d3-abfb kernel: [ 1914.924097] x7 : c0000000ffffefff x6 : 0000000000000003 
Jan 19 07:55:22 rpi-3b-plus-rev1d3-abfb kernel: [ 1914.929590] x5 : 0000000000000000 x4 : 0000000000000000 
Jan 19 07:55:22 rpi-3b-plus-rev1d3-abfb kernel: [ 1914.935081] x3 : 0000000000000100 x2 : 0000000000001000 
Jan 19 07:55:22 rpi-3b-plus-rev1d3-abfb kernel: [ 1914.940575] x1 : 0000000000000000 x0 : 0000000000000000 
Jan 19 07:55:22 rpi-3b-plus-rev1d3-abfb kernel: [ 1914.946070] Call trace:
Jan 19 07:55:22 rpi-3b-plus-rev1d3-abfb kernel: [ 1914.948599]  skb_warn_bad_offload+0x84/0x100
Jan 19 07:55:22 rpi-3b-plus-rev1d3-abfb kernel: [ 1914.953020]  netif_skb_features+0x218/0x2a0
Jan 19 07:55:22 rpi-3b-plus-rev1d3-abfb kernel: [ 1914.957350]  validate_xmit_skb.isra.0+0x28/0x2c8
Jan 19 07:55:22 rpi-3b-plus-rev1d3-abfb kernel: [ 1914.962125]  validate_xmit_skb_list+0x44/0x98
Jan 19 07:55:22 rpi-3b-plus-rev1d3-abfb kernel: [ 1914.966631]  sch_direct_xmit+0xf0/0x3a8
Jan 19 07:55:22 rpi-3b-plus-rev1d3-abfb kernel: [ 1914.970599]  __qdisc_run+0x140/0x668
Jan 19 07:55:22 rpi-3b-plus-rev1d3-abfb kernel: [ 1914.974297]  __dev_queue_xmit+0x59c/0x980
Jan 19 07:55:22 rpi-3b-plus-rev1d3-abfb kernel: [ 1914.978446]  dev_queue_xmit+0x1c/0x28
Jan 19 07:55:22 rpi-3b-plus-rev1d3-abfb kernel: [ 1914.982237]  ip_finish_output2+0x30c/0x558
Jan 19 07:55:22 rpi-3b-plus-rev1d3-abfb kernel: [ 1914.986476]  __ip_finish_output+0xe4/0x260
Jan 19 07:55:22 rpi-3b-plus-rev1d3-abfb kernel: [ 1914.990715]  ip_finish_output+0x3c/0xd8
Jan 19 07:55:22 rpi-3b-plus-rev1d3-abfb kernel: [ 1914.994683]  ip_output+0xb4/0x148
Jan 19 07:55:22 rpi-3b-plus-rev1d3-abfb kernel: [ 1914.998116]  ip_forward_finish+0x7c/0xc0
Jan 19 07:55:22 rpi-3b-plus-rev1d3-abfb kernel: [ 1915.002174]  ip_forward+0x42c/0x4f0
Jan 19 07:55:22 rpi-3b-plus-rev1d3-abfb kernel: [ 1915.005783]  ip_rcv_finish+0x98/0xb8
Jan 19 07:55:22 rpi-3b-plus-rev1d3-abfb kernel: [ 1915.009481]  ip_rcv+0xe0/0xf0
Jan 19 07:55:22 rpi-3b-plus-rev1d3-abfb kernel: [ 1915.012552]  __netif_receive_skb_one_core+0x5c/0x88
Jan 19 07:55:22 rpi-3b-plus-rev1d3-abfb kernel: [ 1915.017597]  __netif_receive_skb+0x20/0x70
Jan 19 07:55:22 rpi-3b-plus-rev1d3-abfb kernel: [ 1915.021834]  process_backlog+0xc0/0x1d0
Jan 19 07:55:22 rpi-3b-plus-rev1d3-abfb kernel: [ 1915.025802]  net_rx_action+0x134/0x478
Jan 19 07:55:22 rpi-3b-plus-rev1d3-abfb kernel: [ 1915.029682]  __do_softirq+0x130/0x378
Jan 19 07:55:22 rpi-3b-plus-rev1d3-abfb kernel: [ 1915.033472]  irq_exit+0xc0/0xe8
Jan 19 07:55:22 rpi-3b-plus-rev1d3-abfb kernel: [ 1915.036725]  __handle_domain_irq+0x70/0xc8
Jan 19 07:55:22 rpi-3b-plus-rev1d3-abfb kernel: [ 1915.040963]  bcm2836_arm_irqchip_handle_irq+0x6c/0x80
Jan 19 07:55:22 rpi-3b-plus-rev1d3-abfb kernel: [ 1915.046185]  el1_irq+0xb4/0x140
Jan 19 07:55:22 rpi-3b-plus-rev1d3-abfb kernel: [ 1915.053377]  arch_cpu_idle+0x18/0x28
Jan 19 07:55:22 rpi-3b-plus-rev1d3-abfb kernel: [ 1915.060981]  default_idle_call+0x44/0x178
Jan 19 07:55:22 rpi-3b-plus-rev1d3-abfb kernel: [ 1915.069009]  do_idle+0x224/0x270
Jan 19 07:55:22 rpi-3b-plus-rev1d3-abfb kernel: [ 1915.076147]  cpu_startup_entry+0x30/0x98
Jan 19 07:55:22 rpi-3b-plus-rev1d3-abfb kernel: [ 1915.083916]  rest_init+0xc8/0xd8
Jan 19 07:55:22 rpi-3b-plus-rev1d3-abfb kernel: [ 1915.090937]  arch_call_rest_init+0x18/0x24
Jan 19 07:55:22 rpi-3b-plus-rev1d3-abfb kernel: [ 1915.098829]  start_kernel+0x57c/0x5b8
Jan 19 07:55:22 rpi-3b-plus-rev1d3-abfb kernel: [ 1915.106251] ---[ end trace c3d8dd12ce1805e0 ]---

If I also add the following rule:

  $ iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE
I get a single warning followed by a TX timeout:

Jan 19 08:15:47 rpi-3b-plus-rev1d3-abfb kernel: [   81.516888] skb len=66 headroom=5194 headlen=66 tailroom=10804
Jan 19 08:15:47 rpi-3b-plus-rev1d3-abfb kernel: [   81.516888] mac=(5194,14) net=(5208,20) trans=5228
Jan 19 08:15:47 rpi-3b-plus-rev1d3-abfb kernel: [   81.516888] shinfo(txflags=0 nr_frags=0 gso(size=1448 type=0 segs=1))
Jan 19 08:15:47 rpi-3b-plus-rev1d3-abfb kernel: [   81.516888] csum(0xeedb ip_summed=1 complete_sw=0 valid=0 level=0)
Jan 19 08:15:47 rpi-3b-plus-rev1d3-abfb kernel: [   81.516888] hash(0x0 sw=0 l4=0) proto=0x0800 pkttype=0 iif=2
Jan 19 08:15:47 rpi-3b-plus-rev1d3-abfb kernel: [   81.546872] dev name=eth0 feat=0x0x0000010000114b09
Jan 19 08:15:47 rpi-3b-plus-rev1d3-abfb kernel: [   81.552060] skb linear:   00000000: e0 28 6d 9e b9 22 b8 27 eb 3e ab fb 08 00 45 00
Jan 19 08:15:47 rpi-3b-plus-rev1d3-abfb kernel: [   81.560090] skb linear:   00000010: 00 34 90 99 40 00 3f 06 87 40 c0 a8 63 84 22 6b
Jan 19 08:15:47 rpi-3b-plus-rev1d3-abfb kernel: [   81.568019] skb linear:   00000020: dd 52 d0 ac 00 50 35 e0 1e 2c 78 02 47 fa 80 10
Jan 19 08:15:47 rpi-3b-plus-rev1d3-abfb kernel: [   81.575921] skb linear:   00000030: 01 f6 d6 96 00 00 01 01 08 0a 50 c9 d7 4b cd 2e
Jan 19 08:15:47 rpi-3b-plus-rev1d3-abfb kernel: [   81.583918] skb linear:   00000040: 9f fc
Jan 19 08:15:47 rpi-3b-plus-rev1d3-abfb kernel: [   81.588105] ------------[ cut here ]------------
Jan 19 08:15:47 rpi-3b-plus-rev1d3-abfb kernel: [   81.592920] lan78xx: caps=(0x0000010000114b09, 0x0000000000000000)
Jan 19 08:15:47 rpi-3b-plus-rev1d3-abfb kernel: [   81.599429] WARNING: CPU: 0 PID: 0 at net/core/dev.c:3197 skb_warn_bad_offload+0x84/0x100
Jan 19 08:15:47 rpi-3b-plus-rev1d3-abfb kernel: [   81.607900] Modules linked in:
Jan 19 08:15:47 rpi-3b-plus-rev1d3-abfb kernel: [   81.611064] CPU: 0 PID: 0 Comm: swapper/0 Not tainted 5.11.0-rc4 #103
Jan 19 08:15:47 rpi-3b-plus-rev1d3-abfb kernel: [   81.617720] Hardware name: Raspberry Pi 3 Model B Plus Rev 1.3 (DT)
Jan 19 08:15:47 rpi-3b-plus-rev1d3-abfb kernel: [   81.624189] pstate: 60000005 (nZCv daif -PAN -UAO -TCO BTYPE=--)
Jan 19 08:15:47 rpi-3b-plus-rev1d3-abfb kernel: [   81.630396] pc : skb_warn_bad_offload+0x84/0x100
Jan 19 08:15:47 rpi-3b-plus-rev1d3-abfb kernel: [   81.635175] lr : skb_warn_bad_offload+0x84/0x100
Jan 19 08:15:47 rpi-3b-plus-rev1d3-abfb kernel: [   81.639953] sp : ffff800010003810
Jan 19 08:15:47 rpi-3b-plus-rev1d3-abfb kernel: [   81.643374] x29: ffff800010003810 x28: ffff50043b196290 
Jan 19 08:15:47 rpi-3b-plus-rev1d3-abfb kernel: [   81.648870] x27: ffff500407371600 x26: 0000000000000001 
Jan 19 08:15:47 rpi-3b-plus-rev1d3-abfb kernel: [   81.654365] x25: ffffa1fa11b23000 x24: ffff50042e96b000 
Jan 19 08:15:47 rpi-3b-plus-rev1d3-abfb kernel: [   81.659859] x23: ffffa1fa11ff4f00 x22: 0000000000000000 
Jan 19 08:15:47 rpi-3b-plus-rev1d3-abfb kernel: [   81.665353] x21: ffffa1fa118327e0 x20: ffff50042e96b000 
Jan 19 08:15:47 rpi-3b-plus-rev1d3-abfb kernel: [   81.670847] x19: ffff500407371600 x18: 0000000000000010 
Jan 19 08:15:47 rpi-3b-plus-rev1d3-abfb kernel: [   81.676340] x17: 0000000000000000 x16: 0000000000000000 
Jan 19 08:15:47 rpi-3b-plus-rev1d3-abfb kernel: [   81.681833] x15: 000000000000ad55 x14: 0000000000000010 
Jan 19 08:15:47 rpi-3b-plus-rev1d3-abfb kernel: [   81.687326] x13: 00000000ffffffff x12: ffffa1fa1159d950 
Jan 19 08:15:47 rpi-3b-plus-rev1d3-abfb kernel: [   81.692819] x11: ffffa1fa12085de0 x10: ffffa1fa1206dda0 
Jan 19 08:15:47 rpi-3b-plus-rev1d3-abfb kernel: [   81.698313] x9 : ffffa1fa1072f45c x8 : 0000000000017fe8 
Jan 19 08:15:47 rpi-3b-plus-rev1d3-abfb kernel: [   81.703806] x7 : c0000000ffffefff x6 : 0000000000000003 
Jan 19 08:15:47 rpi-3b-plus-rev1d3-abfb kernel: [   81.709300] x5 : 0000000000000000 x4 : 0000000000000000 
Jan 19 08:15:47 rpi-3b-plus-rev1d3-abfb kernel: [   81.714791] x3 : 0000000000000100 x2 : 0000000000001000 
Jan 19 08:15:47 rpi-3b-plus-rev1d3-abfb kernel: [   81.720283] x1 : 0000000000000000 x0 : 0000000000000000 
Jan 19 08:15:47 rpi-3b-plus-rev1d3-abfb kernel: [   81.725778] Call trace:
Jan 19 08:15:47 rpi-3b-plus-rev1d3-abfb kernel: [   81.728306]  skb_warn_bad_offload+0x84/0x100
Jan 19 08:15:47 rpi-3b-plus-rev1d3-abfb kernel: [   81.732728]  netif_skb_features+0x218/0x2a0
Jan 19 08:15:47 rpi-3b-plus-rev1d3-abfb kernel: [   81.737057]  validate_xmit_skb.isra.0+0x28/0x2c8
Jan 19 08:15:47 rpi-3b-plus-rev1d3-abfb kernel: [   81.741833]  validate_xmit_skb_list+0x44/0x98
Jan 19 08:15:47 rpi-3b-plus-rev1d3-abfb kernel: [   81.746339]  sch_direct_xmit+0xf0/0x3a8
Jan 19 08:15:47 rpi-3b-plus-rev1d3-abfb kernel: [   81.750309]  __qdisc_run+0x140/0x668
Jan 19 08:15:47 rpi-3b-plus-rev1d3-abfb kernel: [   81.754008]  __dev_queue_xmit+0x59c/0x980
Jan 19 08:15:47 rpi-3b-plus-rev1d3-abfb kernel: [   81.758156]  dev_queue_xmit+0x1c/0x28
Jan 19 08:15:47 rpi-3b-plus-rev1d3-abfb kernel: [   81.761945]  neigh_resolve_output+0x108/0x230
Jan 19 08:15:47 rpi-3b-plus-rev1d3-abfb kernel: [   81.766450]  ip_finish_output2+0x180/0x558
Jan 19 08:15:47 rpi-3b-plus-rev1d3-abfb kernel: [   81.770690]  __ip_finish_output+0xe4/0x260
Jan 19 08:15:47 rpi-3b-plus-rev1d3-abfb kernel: [   81.774928]  ip_finish_output+0x3c/0xd8
Jan 19 08:15:47 rpi-3b-plus-rev1d3-abfb kernel: [   81.778896]  ip_output+0xb4/0x148
Jan 19 08:15:47 rpi-3b-plus-rev1d3-abfb kernel: [   81.782328]  ip_forward_finish+0x7c/0xc0
Jan 19 08:15:47 rpi-3b-plus-rev1d3-abfb kernel: [   81.786385]  ip_forward+0x42c/0x4f0
Jan 19 08:15:47 rpi-3b-plus-rev1d3-abfb kernel: [   81.789995]  ip_rcv_finish+0x98/0xb8
Jan 19 08:15:47 rpi-3b-plus-rev1d3-abfb kernel: [   81.793694]  ip_rcv+0xe0/0xf0
Jan 19 08:15:47 rpi-3b-plus-rev1d3-abfb kernel: [   81.796765]  __netif_receive_skb_one_core+0x5c/0x88
Jan 19 08:15:47 rpi-3b-plus-rev1d3-abfb kernel: [   81.801810]  __netif_receive_skb+0x20/0x70
Jan 19 08:15:47 rpi-3b-plus-rev1d3-abfb kernel: [   81.806047]  process_backlog+0xc0/0x1d0
Jan 19 08:15:47 rpi-3b-plus-rev1d3-abfb kernel: [   81.810016]  net_rx_action+0x134/0x478
Jan 19 08:15:47 rpi-3b-plus-rev1d3-abfb kernel: [   81.813897]  __do_softirq+0x130/0x378
Jan 19 08:15:47 rpi-3b-plus-rev1d3-abfb kernel: [   81.817686]  irq_exit+0xc0/0xe8
Jan 19 08:15:47 rpi-3b-plus-rev1d3-abfb kernel: [   81.820940]  __handle_domain_irq+0x70/0xc8
Jan 19 08:15:47 rpi-3b-plus-rev1d3-abfb kernel: [   81.829099]  bcm2836_arm_irqchip_handle_irq+0x6c/0x80
Jan 19 08:15:47 rpi-3b-plus-rev1d3-abfb kernel: [   81.838223]  el1_irq+0xb4/0x140
Jan 19 08:15:47 rpi-3b-plus-rev1d3-abfb kernel: [   81.845371]  arch_cpu_idle+0x18/0x28
Jan 19 08:15:47 rpi-3b-plus-rev1d3-abfb kernel: [   81.852882]  default_idle_call+0x44/0x178
Jan 19 08:15:47 rpi-3b-plus-rev1d3-abfb kernel: [   81.860756]  do_idle+0x224/0x270
Jan 19 08:15:47 rpi-3b-plus-rev1d3-abfb kernel: [   81.867794]  cpu_startup_entry+0x30/0x98
Jan 19 08:15:47 rpi-3b-plus-rev1d3-abfb kernel: [   81.875516]  rest_init+0xc8/0xd8
Jan 19 08:15:47 rpi-3b-plus-rev1d3-abfb kernel: [   81.882496]  arch_call_rest_init+0x18/0x24
Jan 19 08:15:47 rpi-3b-plus-rev1d3-abfb kernel: [   81.890352]  start_kernel+0x57c/0x5b8
Jan 19 08:15:47 rpi-3b-plus-rev1d3-abfb kernel: [   81.897706] ---[ end trace a5789410f231a10b ]---
Jan 19 08:16:15 rpi-3b-plus-rev1d3-abfb kernel: [  110.046337] ------------[ cut here ]------------
Jan 19 08:16:15 rpi-3b-plus-rev1d3-abfb kernel: [  110.054787] NETDEV WATCHDOG: eth0 (lan78xx): transmit queue 0 timed out
Jan 19 08:16:15 rpi-3b-plus-rev1d3-abfb kernel: [  110.065356] WARNING: CPU: 2 PID: 0 at net/sched/sch_generic.c:442 dev_watchdog+0x384/0x390
Jan 19 08:16:15 rpi-3b-plus-rev1d3-abfb kernel: [  110.077534] Modules linked in:
Jan 19 08:16:15 rpi-3b-plus-rev1d3-abfb kernel: [  110.084361] CPU: 2 PID: 0 Comm: swapper/2 Tainted: G        W         5.11.0-rc4 #103
Jan 19 08:16:15 rpi-3b-plus-rev1d3-abfb kernel: [  110.096114] Hardware name: Raspberry Pi 3 Model B Plus Rev 1.3 (DT)
Jan 19 08:16:15 rpi-3b-plus-rev1d3-abfb kernel: [  110.106246] pstate: 60000005 (nZCv daif -PAN -UAO -TCO BTYPE=--)
Jan 19 08:16:15 rpi-3b-plus-rev1d3-abfb kernel: [  110.116085] pc : dev_watchdog+0x384/0x390
Jan 19 08:16:15 rpi-3b-plus-rev1d3-abfb kernel: [  110.123857] lr : dev_watchdog+0x384/0x390
Jan 19 08:16:15 rpi-3b-plus-rev1d3-abfb kernel: [  110.131558] sp : ffff800010013d90
Jan 19 08:16:15 rpi-3b-plus-rev1d3-abfb kernel: [  110.138497] x29: ffff800010013d90 x28: 0000000000000140 
Jan 19 08:16:15 rpi-3b-plus-rev1d3-abfb kernel: [  110.147472] x27: 00000000ffffffff x26: ffffa1fa11b23000 
Jan 19 08:16:15 rpi-3b-plus-rev1d3-abfb kernel: [  110.156489] x25: 0000000000000002 x24: 0000000000000000 
Jan 19 08:16:15 rpi-3b-plus-rev1d3-abfb kernel: [  110.165496] x23: 0000000000000001 x22: ffff50042e96b000 
Jan 19 08:16:15 rpi-3b-plus-rev1d3-abfb kernel: [  110.174494] x21: ffff50042e96b440 x20: ffffa1fa11fe7000 
Jan 19 08:16:15 rpi-3b-plus-rev1d3-abfb kernel: [  110.183490] x19: 0000000000000000 x18: 0000000000000010 
Jan 19 08:16:15 rpi-3b-plus-rev1d3-abfb kernel: [  110.192493] x17: 0000000000000000 x16: 0000000000000000 
Jan 19 08:16:15 rpi-3b-plus-rev1d3-abfb kernel: [  110.201473] x15: 000000000000ad55 x14: 0000000000000010 
Jan 19 08:16:15 rpi-3b-plus-rev1d3-abfb kernel: [  110.210439] x13: 00000000ffffffff x12: ffffa1fa1159d950 
Jan 19 08:16:15 rpi-3b-plus-rev1d3-abfb kernel: [  110.219397] x11: ffffa1fa12085de0 x10: ffffa1fa1206dda0 
Jan 19 08:16:15 rpi-3b-plus-rev1d3-abfb kernel: [  110.228367] x9 : ffffa1fa1072f45c x8 : 0000000000017fe8 
Jan 19 08:16:15 rpi-3b-plus-rev1d3-abfb kernel: [  110.237362] x7 : c0000000ffffefff x6 : 0000000000000003 
Jan 19 08:16:15 rpi-3b-plus-rev1d3-abfb kernel: [  110.246353] x5 : 0000000000000000 x4 : 0000000000000000 
Jan 19 08:16:15 rpi-3b-plus-rev1d3-abfb kernel: [  110.255328] x3 : 0000000000000100 x2 : 0000000000001000 
Jan 19 08:16:15 rpi-3b-plus-rev1d3-abfb kernel: [  110.264273] x1 : 0000000000000000 x0 : 0000000000000000 
Jan 19 08:16:15 rpi-3b-plus-rev1d3-abfb kernel: [  110.273192] Call trace:
Jan 19 08:16:15 rpi-3b-plus-rev1d3-abfb kernel: [  110.279183]  dev_watchdog+0x384/0x390
Jan 19 08:16:15 rpi-3b-plus-rev1d3-abfb kernel: [  110.286461]  call_timer_fn+0x38/0x188
Jan 19 08:16:15 rpi-3b-plus-rev1d3-abfb kernel: [  110.293762]  run_timer_softirq+0x494/0x688
Jan 19 08:16:15 rpi-3b-plus-rev1d3-abfb kernel: [  110.301489]  __do_softirq+0x130/0x378
Jan 19 08:16:15 rpi-3b-plus-rev1d3-abfb kernel: [  110.308767]  irq_exit+0xc0/0xe8
Jan 19 08:16:15 rpi-3b-plus-rev1d3-abfb kernel: [  110.315500]  __handle_domain_irq+0x70/0xc8
Jan 19 08:16:15 rpi-3b-plus-rev1d3-abfb kernel: [  110.323214]  bcm2836_arm_irqchip_handle_irq+0x6c/0x80
Jan 19 08:16:15 rpi-3b-plus-rev1d3-abfb kernel: [  110.331940]  el1_irq+0xb4/0x140
Jan 19 08:16:15 rpi-3b-plus-rev1d3-abfb kernel: [  110.338706]  arch_cpu_idle+0x18/0x28
Jan 19 08:16:15 rpi-3b-plus-rev1d3-abfb kernel: [  110.345916]  default_idle_call+0x44/0x178
Jan 19 08:16:15 rpi-3b-plus-rev1d3-abfb kernel: [  110.353577]  do_idle+0x224/0x270
Jan 19 08:16:15 rpi-3b-plus-rev1d3-abfb kernel: [  110.360433]  cpu_startup_entry+0x2c/0x98
Jan 19 08:16:15 rpi-3b-plus-rev1d3-abfb kernel: [  110.368000]  secondary_start_kernel+0x148/0x180
Jan 19 08:16:15 rpi-3b-plus-rev1d3-abfb kernel: [  110.376199] ---[ end trace a5789410f231a10c ]---

I did some bisecting and found commit [2] to be problematic. Reverting that commit plus the two follow-on fixes [3] and [4] prevents the warnings and timeout. I'm no networking expert so can't determine if [2] is broken or merely exposes a different underlying issue. I failed to reproduce the problem using a dedicated Realtek-based USB NIC plugged into the Pi, which points towards the lan78xx driver/HW being the culprit.

Enabling KASAN didn't trigger any error reports.

Let me know if there's anything else I can try to narrow this down.

...Juerg

[1]
On the Pi, I run:

  $ nc -l 1234 | dd status=progress >/dev/null

And on another machine, that is configured to use the Pi as the gateway:

  $ nc 192.168.99.115 1234 < /dev/urandom
and a couple of firefox instances that keep opening public URls.

[2]

Author: Eric Dumazet <[email protected]>
Date:   Tue Nov 27 14:42:03 2018 -0800

tcp: implement coalescing on backlog queue

In case GRO is not as efficient as it should be or disabled, we might have a user thread trapped in __release_sock() while softirq handler flood packets up to the point we have to drop.

This patch balances work done from user thread and softirq, to give more chances to __release_sock() to complete its work before new packets are added the the backlog.

This also helps if we receive many ACK packets, since GRO does not aggregate them.

This patch brings ~60% throughput increase on a receiver without GRO, but the spectacular gain is really on 1000x release_sock() latency reduction I have measured.

Signed-off-by Eric Dumazet <[email protected]>

Cc Neal Cardwell <[email protected]>

Cc Yuchung Cheng <[email protected]>

Acked-by Neal Cardwell <[email protected]>

Signed-off-by David S. Miller <[email protected]>

[3] 86bccd036713 tcp: fix receive window update in tcp_add_backlog()
[4] ca2fe2956ace tcp: add sanity tests in tcp_add_backlog()

Eric Dumazet

2021-01-19 23:59 UTC #15 lore raw

Quoted 234 lines · Juerg Haefliger

Jan 19 07:55:22 rpi-3b-plus-rev1d3-abfb kernel: [ 1914.744157] skb len=54 headroom=5194 headlen=54 tailroom=10816
Jan 19 07:55:22 rpi-3b-plus-rev1d3-abfb kernel: [ 1914.744157] mac=(5194,14) net=(5208,20) trans=5228
Jan 19 07:55:22 rpi-3b-plus-rev1d3-abfb kernel: [ 1914.744157] shinfo(txflags=0 nr_frags=0 gso(size=1448 type=0 segs=1))
Jan 19 07:55:22 rpi-3b-plus-rev1d3-abfb kernel: [ 1914.744157] csum(0xe505 ip_summed=0 complete_sw=0 valid=0 level=0)
Jan 19 07:55:22 rpi-3b-plus-rev1d3-abfb kernel: [ 1914.744157] hash(0x0 sw=0 l4=0) proto=0x0800 pkttype=0 iif=2
Jan 19 07:55:22 rpi-3b-plus-rev1d3-abfb kernel: [ 1914.774147] dev name=eth0 feat=0x0x0000010000114b09
Jan 19 07:55:22 rpi-3b-plus-rev1d3-abfb kernel: [ 1914.779355] skb linear:   00000000: e0 28 6d 9e b9 22 b8 27 eb 3e ab fb 08 00 45 00
Jan 19 07:55:22 rpi-3b-plus-rev1d3-abfb kernel: [ 1914.787365] skb linear:   00000010: 00 28 00 00 40 00 3f 06 41 d0 c0 a8 63 84 02 14
Jan 19 07:55:22 rpi-3b-plus-rev1d3-abfb kernel: [ 1914.795266] skb linear:   00000020: d3 bf ed 3e 01 bb d4 0f 88 7e 00 00 00 00 50 04
Jan 19 07:55:22 rpi-3b-plus-rev1d3-abfb kernel: [ 1914.803168] skb linear:   00000030: 00 00 6a 58 00 00
Jan 19 07:55:22 rpi-3b-plus-rev1d3-abfb kernel: [ 1914.808384] ------------[ cut here ]------------
Jan 19 07:55:22 rpi-3b-plus-rev1d3-abfb kernel: [ 1914.813200] lan78xx: caps=(0x0000010000114b09, 0x0000000000000000)
Jan 19 07:55:22 rpi-3b-plus-rev1d3-abfb kernel: [ 1914.819717] WARNING: CPU: 0 PID: 0 at net/core/dev.c:3197 skb_warn_bad_offload+0x84/0x100
Jan 19 07:55:22 rpi-3b-plus-rev1d3-abfb kernel: [ 1914.828190] Modules linked in:
Jan 19 07:55:22 rpi-3b-plus-rev1d3-abfb kernel: [ 1914.831354] CPU: 0 PID: 0 Comm: swapper/0 Not tainted 5.11.0-rc4 #103
Jan 19 07:55:22 rpi-3b-plus-rev1d3-abfb kernel: [ 1914.838009] Hardware name: Raspberry Pi 3 Model B Plus Rev 1.3 (DT)
Jan 19 07:55:22 rpi-3b-plus-rev1d3-abfb kernel: [ 1914.844478] pstate: 60000005 (nZCv daif -PAN -UAO -TCO BTYPE=--)
Jan 19 07:55:22 rpi-3b-plus-rev1d3-abfb kernel: [ 1914.850685] pc : skb_warn_bad_offload+0x84/0x100
Jan 19 07:55:22 rpi-3b-plus-rev1d3-abfb kernel: [ 1914.855464] lr : skb_warn_bad_offload+0x84/0x100
Jan 19 07:55:22 rpi-3b-plus-rev1d3-abfb kernel: [ 1914.860242] sp : ffff800010003850
Jan 19 07:55:22 rpi-3b-plus-rev1d3-abfb kernel: [ 1914.863665] x29: ffff800010003850 x28: ffff7a96fb196290
Jan 19 07:55:22 rpi-3b-plus-rev1d3-abfb kernel: [ 1914.869160] x27: ffff7a96c5958300 x26: 0000000000000001
Jan 19 07:55:22 rpi-3b-plus-rev1d3-abfb kernel: [ 1914.874654] x25: ffffa73eee323000 x24: ffff7a96ee84b000
Jan 19 07:55:22 rpi-3b-plus-rev1d3-abfb kernel: [ 1914.880148] x23: ffffa73eee7f4f00 x22: 0000000000000000
Jan 19 07:55:22 rpi-3b-plus-rev1d3-abfb kernel: [ 1914.885642] x21: ffffa73eee0327e0 x20: ffff7a96ee84b000
Jan 19 07:55:22 rpi-3b-plus-rev1d3-abfb kernel: [ 1914.891136] x19: ffff7a96c5958300 x18: 0000000000000010
Jan 19 07:55:22 rpi-3b-plus-rev1d3-abfb kernel: [ 1914.896630] x17: 0000000000000000 x16: 0000000000000000
Jan 19 07:55:22 rpi-3b-plus-rev1d3-abfb kernel: [ 1914.902123] x15: 000000000000ad55 x14: 0000000000000010
Jan 19 07:55:22 rpi-3b-plus-rev1d3-abfb kernel: [ 1914.907617] x13: 00000000ffffffff x12: ffffa73eedd9d950
Jan 19 07:55:22 rpi-3b-plus-rev1d3-abfb kernel: [ 1914.913109] x11: ffffa73eee885de0 x10: ffffa73eee86dda0
Jan 19 07:55:22 rpi-3b-plus-rev1d3-abfb kernel: [ 1914.918603] x9 : ffffa73eecf2f45c x8 : 0000000000017fe8
Jan 19 07:55:22 rpi-3b-plus-rev1d3-abfb kernel: [ 1914.924097] x7 : c0000000ffffefff x6 : 0000000000000003
Jan 19 07:55:22 rpi-3b-plus-rev1d3-abfb kernel: [ 1914.929590] x5 : 0000000000000000 x4 : 0000000000000000
Jan 19 07:55:22 rpi-3b-plus-rev1d3-abfb kernel: [ 1914.935081] x3 : 0000000000000100 x2 : 0000000000001000
Jan 19 07:55:22 rpi-3b-plus-rev1d3-abfb kernel: [ 1914.940575] x1 : 0000000000000000 x0 : 0000000000000000
Jan 19 07:55:22 rpi-3b-plus-rev1d3-abfb kernel: [ 1914.946070] Call trace:
Jan 19 07:55:22 rpi-3b-plus-rev1d3-abfb kernel: [ 1914.948599]  skb_warn_bad_offload+0x84/0x100
Jan 19 07:55:22 rpi-3b-plus-rev1d3-abfb kernel: [ 1914.953020]  netif_skb_features+0x218/0x2a0
Jan 19 07:55:22 rpi-3b-plus-rev1d3-abfb kernel: [ 1914.957350]  validate_xmit_skb.isra.0+0x28/0x2c8
Jan 19 07:55:22 rpi-3b-plus-rev1d3-abfb kernel: [ 1914.962125]  validate_xmit_skb_list+0x44/0x98
Jan 19 07:55:22 rpi-3b-plus-rev1d3-abfb kernel: [ 1914.966631]  sch_direct_xmit+0xf0/0x3a8
Jan 19 07:55:22 rpi-3b-plus-rev1d3-abfb kernel: [ 1914.970599]  __qdisc_run+0x140/0x668
Jan 19 07:55:22 rpi-3b-plus-rev1d3-abfb kernel: [ 1914.974297]  __dev_queue_xmit+0x59c/0x980
Jan 19 07:55:22 rpi-3b-plus-rev1d3-abfb kernel: [ 1914.978446]  dev_queue_xmit+0x1c/0x28
Jan 19 07:55:22 rpi-3b-plus-rev1d3-abfb kernel: [ 1914.982237]  ip_finish_output2+0x30c/0x558
Jan 19 07:55:22 rpi-3b-plus-rev1d3-abfb kernel: [ 1914.986476]  __ip_finish_output+0xe4/0x260
Jan 19 07:55:22 rpi-3b-plus-rev1d3-abfb kernel: [ 1914.990715]  ip_finish_output+0x3c/0xd8
Jan 19 07:55:22 rpi-3b-plus-rev1d3-abfb kernel: [ 1914.994683]  ip_output+0xb4/0x148
Jan 19 07:55:22 rpi-3b-plus-rev1d3-abfb kernel: [ 1914.998116]  ip_forward_finish+0x7c/0xc0
Jan 19 07:55:22 rpi-3b-plus-rev1d3-abfb kernel: [ 1915.002174]  ip_forward+0x42c/0x4f0
Jan 19 07:55:22 rpi-3b-plus-rev1d3-abfb kernel: [ 1915.005783]  ip_rcv_finish+0x98/0xb8
Jan 19 07:55:22 rpi-3b-plus-rev1d3-abfb kernel: [ 1915.009481]  ip_rcv+0xe0/0xf0
Jan 19 07:55:22 rpi-3b-plus-rev1d3-abfb kernel: [ 1915.012552]  __netif_receive_skb_one_core+0x5c/0x88
Jan 19 07:55:22 rpi-3b-plus-rev1d3-abfb kernel: [ 1915.017597]  __netif_receive_skb+0x20/0x70
Jan 19 07:55:22 rpi-3b-plus-rev1d3-abfb kernel: [ 1915.021834]  process_backlog+0xc0/0x1d0
Jan 19 07:55:22 rpi-3b-plus-rev1d3-abfb kernel: [ 1915.025802]  net_rx_action+0x134/0x478
Jan 19 07:55:22 rpi-3b-plus-rev1d3-abfb kernel: [ 1915.029682]  __do_softirq+0x130/0x378
Jan 19 07:55:22 rpi-3b-plus-rev1d3-abfb kernel: [ 1915.033472]  irq_exit+0xc0/0xe8
Jan 19 07:55:22 rpi-3b-plus-rev1d3-abfb kernel: [ 1915.036725]  __handle_domain_irq+0x70/0xc8
Jan 19 07:55:22 rpi-3b-plus-rev1d3-abfb kernel: [ 1915.040963]  bcm2836_arm_irqchip_handle_irq+0x6c/0x80
Jan 19 07:55:22 rpi-3b-plus-rev1d3-abfb kernel: [ 1915.046185]  el1_irq+0xb4/0x140
Jan 19 07:55:22 rpi-3b-plus-rev1d3-abfb kernel: [ 1915.053377]  arch_cpu_idle+0x18/0x28
Jan 19 07:55:22 rpi-3b-plus-rev1d3-abfb kernel: [ 1915.060981]  default_idle_call+0x44/0x178
Jan 19 07:55:22 rpi-3b-plus-rev1d3-abfb kernel: [ 1915.069009]  do_idle+0x224/0x270
Jan 19 07:55:22 rpi-3b-plus-rev1d3-abfb kernel: [ 1915.076147]  cpu_startup_entry+0x30/0x98
Jan 19 07:55:22 rpi-3b-plus-rev1d3-abfb kernel: [ 1915.083916]  rest_init+0xc8/0xd8
Jan 19 07:55:22 rpi-3b-plus-rev1d3-abfb kernel: [ 1915.090937]  arch_call_rest_init+0x18/0x24
Jan 19 07:55:22 rpi-3b-plus-rev1d3-abfb kernel: [ 1915.098829]  start_kernel+0x57c/0x5b8
Jan 19 07:55:22 rpi-3b-plus-rev1d3-abfb kernel: [ 1915.106251] ---[ end trace c3d8dd12ce1805e0 ]---

If I also add the following rule:

  $ iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE
I get a single warning followed by a TX timeout:

Jan 19 08:15:47 rpi-3b-plus-rev1d3-abfb kernel: [   81.516888] skb len=66 headroom=5194 headlen=66 tailroom=10804
Jan 19 08:15:47 rpi-3b-plus-rev1d3-abfb kernel: [   81.516888] mac=(5194,14) net=(5208,20) trans=5228
Jan 19 08:15:47 rpi-3b-plus-rev1d3-abfb kernel: [   81.516888] shinfo(txflags=0 nr_frags=0 gso(size=1448 type=0 segs=1))
Jan 19 08:15:47 rpi-3b-plus-rev1d3-abfb kernel: [   81.516888] csum(0xeedb ip_summed=1 complete_sw=0 valid=0 level=0)
Jan 19 08:15:47 rpi-3b-plus-rev1d3-abfb kernel: [   81.516888] hash(0x0 sw=0 l4=0) proto=0x0800 pkttype=0 iif=2
Jan 19 08:15:47 rpi-3b-plus-rev1d3-abfb kernel: [   81.546872] dev name=eth0 feat=0x0x0000010000114b09
Jan 19 08:15:47 rpi-3b-plus-rev1d3-abfb kernel: [   81.552060] skb linear:   00000000: e0 28 6d 9e b9 22 b8 27 eb 3e ab fb 08 00 45 00
Jan 19 08:15:47 rpi-3b-plus-rev1d3-abfb kernel: [   81.560090] skb linear:   00000010: 00 34 90 99 40 00 3f 06 87 40 c0 a8 63 84 22 6b
Jan 19 08:15:47 rpi-3b-plus-rev1d3-abfb kernel: [   81.568019] skb linear:   00000020: dd 52 d0 ac 00 50 35 e0 1e 2c 78 02 47 fa 80 10
Jan 19 08:15:47 rpi-3b-plus-rev1d3-abfb kernel: [   81.575921] skb linear:   00000030: 01 f6 d6 96 00 00 01 01 08 0a 50 c9 d7 4b cd 2e
Jan 19 08:15:47 rpi-3b-plus-rev1d3-abfb kernel: [   81.583918] skb linear:   00000040: 9f fc
Jan 19 08:15:47 rpi-3b-plus-rev1d3-abfb kernel: [   81.588105] ------------[ cut here ]------------
Jan 19 08:15:47 rpi-3b-plus-rev1d3-abfb kernel: [   81.592920] lan78xx: caps=(0x0000010000114b09, 0x0000000000000000)
Jan 19 08:15:47 rpi-3b-plus-rev1d3-abfb kernel: [   81.599429] WARNING: CPU: 0 PID: 0 at net/core/dev.c:3197 skb_warn_bad_offload+0x84/0x100
Jan 19 08:15:47 rpi-3b-plus-rev1d3-abfb kernel: [   81.607900] Modules linked in:
Jan 19 08:15:47 rpi-3b-plus-rev1d3-abfb kernel: [   81.611064] CPU: 0 PID: 0 Comm: swapper/0 Not tainted 5.11.0-rc4 #103
Jan 19 08:15:47 rpi-3b-plus-rev1d3-abfb kernel: [   81.617720] Hardware name: Raspberry Pi 3 Model B Plus Rev 1.3 (DT)
Jan 19 08:15:47 rpi-3b-plus-rev1d3-abfb kernel: [   81.624189] pstate: 60000005 (nZCv daif -PAN -UAO -TCO BTYPE=--)
Jan 19 08:15:47 rpi-3b-plus-rev1d3-abfb kernel: [   81.630396] pc : skb_warn_bad_offload+0x84/0x100
Jan 19 08:15:47 rpi-3b-plus-rev1d3-abfb kernel: [   81.635175] lr : skb_warn_bad_offload+0x84/0x100
Jan 19 08:15:47 rpi-3b-plus-rev1d3-abfb kernel: [   81.639953] sp : ffff800010003810
Jan 19 08:15:47 rpi-3b-plus-rev1d3-abfb kernel: [   81.643374] x29: ffff800010003810 x28: ffff50043b196290
Jan 19 08:15:47 rpi-3b-plus-rev1d3-abfb kernel: [   81.648870] x27: ffff500407371600 x26: 0000000000000001
Jan 19 08:15:47 rpi-3b-plus-rev1d3-abfb kernel: [   81.654365] x25: ffffa1fa11b23000 x24: ffff50042e96b000
Jan 19 08:15:47 rpi-3b-plus-rev1d3-abfb kernel: [   81.659859] x23: ffffa1fa11ff4f00 x22: 0000000000000000
Jan 19 08:15:47 rpi-3b-plus-rev1d3-abfb kernel: [   81.665353] x21: ffffa1fa118327e0 x20: ffff50042e96b000
Jan 19 08:15:47 rpi-3b-plus-rev1d3-abfb kernel: [   81.670847] x19: ffff500407371600 x18: 0000000000000010
Jan 19 08:15:47 rpi-3b-plus-rev1d3-abfb kernel: [   81.676340] x17: 0000000000000000 x16: 0000000000000000
Jan 19 08:15:47 rpi-3b-plus-rev1d3-abfb kernel: [   81.681833] x15: 000000000000ad55 x14: 0000000000000010
Jan 19 08:15:47 rpi-3b-plus-rev1d3-abfb kernel: [   81.687326] x13: 00000000ffffffff x12: ffffa1fa1159d950
Jan 19 08:15:47 rpi-3b-plus-rev1d3-abfb kernel: [   81.692819] x11: ffffa1fa12085de0 x10: ffffa1fa1206dda0
Jan 19 08:15:47 rpi-3b-plus-rev1d3-abfb kernel: [   81.698313] x9 : ffffa1fa1072f45c x8 : 0000000000017fe8
Jan 19 08:15:47 rpi-3b-plus-rev1d3-abfb kernel: [   81.703806] x7 : c0000000ffffefff x6 : 0000000000000003
Jan 19 08:15:47 rpi-3b-plus-rev1d3-abfb kernel: [   81.709300] x5 : 0000000000000000 x4 : 0000000000000000
Jan 19 08:15:47 rpi-3b-plus-rev1d3-abfb kernel: [   81.714791] x3 : 0000000000000100 x2 : 0000000000001000
Jan 19 08:15:47 rpi-3b-plus-rev1d3-abfb kernel: [   81.720283] x1 : 0000000000000000 x0 : 0000000000000000
Jan 19 08:15:47 rpi-3b-plus-rev1d3-abfb kernel: [   81.725778] Call trace:
Jan 19 08:15:47 rpi-3b-plus-rev1d3-abfb kernel: [   81.728306]  skb_warn_bad_offload+0x84/0x100
Jan 19 08:15:47 rpi-3b-plus-rev1d3-abfb kernel: [   81.732728]  netif_skb_features+0x218/0x2a0
Jan 19 08:15:47 rpi-3b-plus-rev1d3-abfb kernel: [   81.737057]  validate_xmit_skb.isra.0+0x28/0x2c8
Jan 19 08:15:47 rpi-3b-plus-rev1d3-abfb kernel: [   81.741833]  validate_xmit_skb_list+0x44/0x98
Jan 19 08:15:47 rpi-3b-plus-rev1d3-abfb kernel: [   81.746339]  sch_direct_xmit+0xf0/0x3a8
Jan 19 08:15:47 rpi-3b-plus-rev1d3-abfb kernel: [   81.750309]  __qdisc_run+0x140/0x668
Jan 19 08:15:47 rpi-3b-plus-rev1d3-abfb kernel: [   81.754008]  __dev_queue_xmit+0x59c/0x980
Jan 19 08:15:47 rpi-3b-plus-rev1d3-abfb kernel: [   81.758156]  dev_queue_xmit+0x1c/0x28
Jan 19 08:15:47 rpi-3b-plus-rev1d3-abfb kernel: [   81.761945]  neigh_resolve_output+0x108/0x230
Jan 19 08:15:47 rpi-3b-plus-rev1d3-abfb kernel: [   81.766450]  ip_finish_output2+0x180/0x558
Jan 19 08:15:47 rpi-3b-plus-rev1d3-abfb kernel: [   81.770690]  __ip_finish_output+0xe4/0x260
Jan 19 08:15:47 rpi-3b-plus-rev1d3-abfb kernel: [   81.774928]  ip_finish_output+0x3c/0xd8
Jan 19 08:15:47 rpi-3b-plus-rev1d3-abfb kernel: [   81.778896]  ip_output+0xb4/0x148
Jan 19 08:15:47 rpi-3b-plus-rev1d3-abfb kernel: [   81.782328]  ip_forward_finish+0x7c/0xc0
Jan 19 08:15:47 rpi-3b-plus-rev1d3-abfb kernel: [   81.786385]  ip_forward+0x42c/0x4f0
Jan 19 08:15:47 rpi-3b-plus-rev1d3-abfb kernel: [   81.789995]  ip_rcv_finish+0x98/0xb8
Jan 19 08:15:47 rpi-3b-plus-rev1d3-abfb kernel: [   81.793694]  ip_rcv+0xe0/0xf0
Jan 19 08:15:47 rpi-3b-plus-rev1d3-abfb kernel: [   81.796765]  __netif_receive_skb_one_core+0x5c/0x88
Jan 19 08:15:47 rpi-3b-plus-rev1d3-abfb kernel: [   81.801810]  __netif_receive_skb+0x20/0x70
Jan 19 08:15:47 rpi-3b-plus-rev1d3-abfb kernel: [   81.806047]  process_backlog+0xc0/0x1d0
Jan 19 08:15:47 rpi-3b-plus-rev1d3-abfb kernel: [   81.810016]  net_rx_action+0x134/0x478
Jan 19 08:15:47 rpi-3b-plus-rev1d3-abfb kernel: [   81.813897]  __do_softirq+0x130/0x378
Jan 19 08:15:47 rpi-3b-plus-rev1d3-abfb kernel: [   81.817686]  irq_exit+0xc0/0xe8
Jan 19 08:15:47 rpi-3b-plus-rev1d3-abfb kernel: [   81.820940]  __handle_domain_irq+0x70/0xc8
Jan 19 08:15:47 rpi-3b-plus-rev1d3-abfb kernel: [   81.829099]  bcm2836_arm_irqchip_handle_irq+0x6c/0x80
Jan 19 08:15:47 rpi-3b-plus-rev1d3-abfb kernel: [   81.838223]  el1_irq+0xb4/0x140
Jan 19 08:15:47 rpi-3b-plus-rev1d3-abfb kernel: [   81.845371]  arch_cpu_idle+0x18/0x28
Jan 19 08:15:47 rpi-3b-plus-rev1d3-abfb kernel: [   81.852882]  default_idle_call+0x44/0x178
Jan 19 08:15:47 rpi-3b-plus-rev1d3-abfb kernel: [   81.860756]  do_idle+0x224/0x270
Jan 19 08:15:47 rpi-3b-plus-rev1d3-abfb kernel: [   81.867794]  cpu_startup_entry+0x30/0x98
Jan 19 08:15:47 rpi-3b-plus-rev1d3-abfb kernel: [   81.875516]  rest_init+0xc8/0xd8
Jan 19 08:15:47 rpi-3b-plus-rev1d3-abfb kernel: [   81.882496]  arch_call_rest_init+0x18/0x24
Jan 19 08:15:47 rpi-3b-plus-rev1d3-abfb kernel: [   81.890352]  start_kernel+0x57c/0x5b8
Jan 19 08:15:47 rpi-3b-plus-rev1d3-abfb kernel: [   81.897706] ---[ end trace a5789410f231a10b ]---
Jan 19 08:16:15 rpi-3b-plus-rev1d3-abfb kernel: [  110.046337] ------------[ cut here ]------------
Jan 19 08:16:15 rpi-3b-plus-rev1d3-abfb kernel: [  110.054787] NETDEV WATCHDOG: eth0 (lan78xx): transmit queue 0 timed out
Jan 19 08:16:15 rpi-3b-plus-rev1d3-abfb kernel: [  110.065356] WARNING: CPU: 2 PID: 0 at net/sched/sch_generic.c:442 dev_watchdog+0x384/0x390
Jan 19 08:16:15 rpi-3b-plus-rev1d3-abfb kernel: [  110.077534] Modules linked in:
Jan 19 08:16:15 rpi-3b-plus-rev1d3-abfb kernel: [  110.084361] CPU: 2 PID: 0 Comm: swapper/2 Tainted: G        W         5.11.0-rc4 #103
Jan 19 08:16:15 rpi-3b-plus-rev1d3-abfb kernel: [  110.096114] Hardware name: Raspberry Pi 3 Model B Plus Rev 1.3 (DT)
Jan 19 08:16:15 rpi-3b-plus-rev1d3-abfb kernel: [  110.106246] pstate: 60000005 (nZCv daif -PAN -UAO -TCO BTYPE=--)
Jan 19 08:16:15 rpi-3b-plus-rev1d3-abfb kernel: [  110.116085] pc : dev_watchdog+0x384/0x390
Jan 19 08:16:15 rpi-3b-plus-rev1d3-abfb kernel: [  110.123857] lr : dev_watchdog+0x384/0x390
Jan 19 08:16:15 rpi-3b-plus-rev1d3-abfb kernel: [  110.131558] sp : ffff800010013d90
Jan 19 08:16:15 rpi-3b-plus-rev1d3-abfb kernel: [  110.138497] x29: ffff800010013d90 x28: 0000000000000140
Jan 19 08:16:15 rpi-3b-plus-rev1d3-abfb kernel: [  110.147472] x27: 00000000ffffffff x26: ffffa1fa11b23000
Jan 19 08:16:15 rpi-3b-plus-rev1d3-abfb kernel: [  110.156489] x25: 0000000000000002 x24: 0000000000000000
Jan 19 08:16:15 rpi-3b-plus-rev1d3-abfb kernel: [  110.165496] x23: 0000000000000001 x22: ffff50042e96b000
Jan 19 08:16:15 rpi-3b-plus-rev1d3-abfb kernel: [  110.174494] x21: ffff50042e96b440 x20: ffffa1fa11fe7000
Jan 19 08:16:15 rpi-3b-plus-rev1d3-abfb kernel: [  110.183490] x19: 0000000000000000 x18: 0000000000000010
Jan 19 08:16:15 rpi-3b-plus-rev1d3-abfb kernel: [  110.192493] x17: 0000000000000000 x16: 0000000000000000
Jan 19 08:16:15 rpi-3b-plus-rev1d3-abfb kernel: [  110.201473] x15: 000000000000ad55 x14: 0000000000000010
Jan 19 08:16:15 rpi-3b-plus-rev1d3-abfb kernel: [  110.210439] x13: 00000000ffffffff x12: ffffa1fa1159d950
Jan 19 08:16:15 rpi-3b-plus-rev1d3-abfb kernel: [  110.219397] x11: ffffa1fa12085de0 x10: ffffa1fa1206dda0
Jan 19 08:16:15 rpi-3b-plus-rev1d3-abfb kernel: [  110.228367] x9 : ffffa1fa1072f45c x8 : 0000000000017fe8
Jan 19 08:16:15 rpi-3b-plus-rev1d3-abfb kernel: [  110.237362] x7 : c0000000ffffefff x6 : 0000000000000003
Jan 19 08:16:15 rpi-3b-plus-rev1d3-abfb kernel: [  110.246353] x5 : 0000000000000000 x4 : 0000000000000000
Jan 19 08:16:15 rpi-3b-plus-rev1d3-abfb kernel: [  110.255328] x3 : 0000000000000100 x2 : 0000000000001000
Jan 19 08:16:15 rpi-3b-plus-rev1d3-abfb kernel: [  110.264273] x1 : 0000000000000000 x0 : 0000000000000000
Jan 19 08:16:15 rpi-3b-plus-rev1d3-abfb kernel: [  110.273192] Call trace:
Jan 19 08:16:15 rpi-3b-plus-rev1d3-abfb kernel: [  110.279183]  dev_watchdog+0x384/0x390
Jan 19 08:16:15 rpi-3b-plus-rev1d3-abfb kernel: [  110.286461]  call_timer_fn+0x38/0x188
Jan 19 08:16:15 rpi-3b-plus-rev1d3-abfb kernel: [  110.293762]  run_timer_softirq+0x494/0x688
Jan 19 08:16:15 rpi-3b-plus-rev1d3-abfb kernel: [  110.301489]  __do_softirq+0x130/0x378
Jan 19 08:16:15 rpi-3b-plus-rev1d3-abfb kernel: [  110.308767]  irq_exit+0xc0/0xe8
Jan 19 08:16:15 rpi-3b-plus-rev1d3-abfb kernel: [  110.315500]  __handle_domain_irq+0x70/0xc8
Jan 19 08:16:15 rpi-3b-plus-rev1d3-abfb kernel: [  110.323214]  bcm2836_arm_irqchip_handle_irq+0x6c/0x80
Jan 19 08:16:15 rpi-3b-plus-rev1d3-abfb kernel: [  110.331940]  el1_irq+0xb4/0x140
Jan 19 08:16:15 rpi-3b-plus-rev1d3-abfb kernel: [  110.338706]  arch_cpu_idle+0x18/0x28
Jan 19 08:16:15 rpi-3b-plus-rev1d3-abfb kernel: [  110.345916]  default_idle_call+0x44/0x178
Jan 19 08:16:15 rpi-3b-plus-rev1d3-abfb kernel: [  110.353577]  do_idle+0x224/0x270
Jan 19 08:16:15 rpi-3b-plus-rev1d3-abfb kernel: [  110.360433]  cpu_startup_entry+0x2c/0x98
Jan 19 08:16:15 rpi-3b-plus-rev1d3-abfb kernel: [  110.368000]  secondary_start_kernel+0x148/0x180
Jan 19 08:16:15 rpi-3b-plus-rev1d3-abfb kernel: [  110.376199] ---[ end trace a5789410f231a10c ]---

Enabling KASAN didn't trigger any error reports.

Let me know if there's anything else I can try to narrow this down.

...Juerg

[1]
On the Pi, I run:

  $ nc -l 1234 | dd status=progress >/dev/null

And on another machine, that is configured to use the Pi as the gateway:

  $ nc 192.168.99.115 1234 < /dev/urandom
and a couple of firefox instances that keep opening public URls.

[2]

Author: Eric Dumazet <[email protected]>
Date:   Tue Nov 27 14:42:03 2018 -0800

tcp: implement coalescing on backlog queue

In case GRO is not as efficient as it should be or disabled, we might have a user thread trapped in __release_sock() while softirq handler flood packets up to the point we have to drop.

This patch balances work done from user thread and softirq, to give more chances to __release_sock() to complete its work before new packets are added the the backlog.

This also helps if we receive many ACK packets, since GRO does not aggregate them.

This patch brings ~60% throughput increase on a receiver without GRO, but the spectacular gain is really on 1000x release_sock() latency reduction I have measured.

Quoted 8 lines

Signed-off-by Eric Dumazet <[email protected]>
Cc Neal Cardwell <[email protected]>
Cc Yuchung Cheng <[email protected]>
Acked-by Neal Cardwell <[email protected]>
Signed-off-by David S. Miller <[email protected]>
[3] 86bccd036713 tcp: fix receive window update in tcp_add_backlog()
[4] ca2fe2956ace tcp: add sanity tests in tcp_add_backlog()

Oops. Very nice detective work :)

It is true that the skb_clone() done in lan78xx (and some other usb drivers) is probably triggering this issue.
(lan78xx is also lying about skb->truesize)

skb_try_coalesce() bails if the target skb is cloned, but not if the source is.

Can you try the following patch ?

net/ipv4/tcp_ipv4.c +0 -0

index 58207c7769d05693b650e3c93e4ef405a5d4b23a..4e82745d336fc3fb0d9ce8c92aaeb39702f64b8a

100644

net/ipv4/tcp_ipv4.c +2 -0

@@ -1760,6 +1760,7 @@ int tcp_v4_early_demux(struct sk_buff *skb)
 bool tcp_add_backlog(struct sock *sk, struct sk_buff *skb)
 {
        u32 limit = READ_ONCE(sk->sk_rcvbuf) + READ_ONCE(sk->sk_sndbuf);
+       u32 tail_gso_size, tail_gso_segs;
        struct skb_shared_info *shinfo;
        const struct tcphdr *th;
        struct tcphdr *thtail;

@@ -1767,6 +1768,7 @@ bool tcp_add_backlog(struct sock *sk, struct sk_buff *skb)
        unsigned int hdrlen;
        bool fragstolen;
        u32 gso_segs;
+       u32 gso_size;
        int delta;
 
        /* In case all data was pulled from skb frags (in __pskb_pull_tail()),

@@ -1792,13 +1794,6 @@ bool tcp_add_backlog(struct sock *sk, struct

sk_buff *skb)

         */
        th = (const struct tcphdr *)skb->data;
        hdrlen = th->doff * 4;

- shinfo = skb_shinfo(skb);
-
- if (!shinfo->gso_size)
- shinfo->gso_size = skb->len - hdrlen;
-
- if (!shinfo->gso_segs)
- shinfo->gso_segs = 1;

        tail = sk->sk_backlog.tail;
        if (!tail)
@@ -1821,6 +1816,15 @@ bool tcp_add_backlog(struct sock *sk, struct
sk_buff *skb)
                goto no_coalesce;

        __skb_pull(skb, hdrlen);
+
+       shinfo = skb_shinfo(skb);
+       gso_size = shinfo->gso_size ?: skb->len;
+       gso_segs = shinfo->gso_segs ?: 1;
+
+       shinfo = skb_shinfo(tail);
+       tail_gso_size = shinfo->gso_size ?: (tail->len - hdrlen);
+       tail_gso_segs = shinfo->gso_segs ?: 1;
+
        if (skb_try_coalesce(tail, skb, &fragstolen, &delta)) {
                TCP_SKB_CB(tail)->end_seq = TCP_SKB_CB(skb)->end_seq;

@@ -1847,11 +1851,8 @@ bool tcp_add_backlog(struct sock *sk, struct sk_buff *skb)

                }

                /* Not as strict as GRO. We only need to carry mss max value */
-               skb_shinfo(tail)->gso_size = max(shinfo->gso_size,
-                                                skb_shinfo(tail)->gso_size);
-
-               gso_segs = skb_shinfo(tail)->gso_segs + shinfo->gso_segs;
-               skb_shinfo(tail)->gso_segs = min_t(u32, gso_segs, 0xFFFF);
+               shinfo->gso_size = max(gso_size, tail_gso_size);
+               shinfo->gso_segs = min_t(u32, gso_segs + tail_gso_segs, 0xFFFF);

                sk->sk_backlog.len += delta;
                __NET_INC_STATS(sock_net(sk),

Juerg Haefliger

Tested-by

2021-01-19 15:40 UTC #16 lore raw

Quoted 249 lines · On Tue, 19 Jan 2021 14:54:31 +0100 Eric Dumazet <[email protected]> wrote:

Quoted 234 lines · Juerg Haefliger

Jan 19 07:55:22 rpi-3b-plus-rev1d3-abfb kernel: [ 1914.744157] skb len=54 headroom=5194 headlen=54 tailroom=10816
Jan 19 07:55:22 rpi-3b-plus-rev1d3-abfb kernel: [ 1914.744157] mac=(5194,14) net=(5208,20) trans=5228
Jan 19 07:55:22 rpi-3b-plus-rev1d3-abfb kernel: [ 1914.744157] shinfo(txflags=0 nr_frags=0 gso(size=1448 type=0 segs=1))
Jan 19 07:55:22 rpi-3b-plus-rev1d3-abfb kernel: [ 1914.744157] csum(0xe505 ip_summed=0 complete_sw=0 valid=0 level=0)
Jan 19 07:55:22 rpi-3b-plus-rev1d3-abfb kernel: [ 1914.744157] hash(0x0 sw=0 l4=0) proto=0x0800 pkttype=0 iif=2
Jan 19 07:55:22 rpi-3b-plus-rev1d3-abfb kernel: [ 1914.774147] dev name=eth0 feat=0x0x0000010000114b09
Jan 19 07:55:22 rpi-3b-plus-rev1d3-abfb kernel: [ 1914.779355] skb linear:   00000000: e0 28 6d 9e b9 22 b8 27 eb 3e ab fb 08 00 45 00
Jan 19 07:55:22 rpi-3b-plus-rev1d3-abfb kernel: [ 1914.787365] skb linear:   00000010: 00 28 00 00 40 00 3f 06 41 d0 c0 a8 63 84 02 14
Jan 19 07:55:22 rpi-3b-plus-rev1d3-abfb kernel: [ 1914.795266] skb linear:   00000020: d3 bf ed 3e 01 bb d4 0f 88 7e 00 00 00 00 50 04
Jan 19 07:55:22 rpi-3b-plus-rev1d3-abfb kernel: [ 1914.803168] skb linear:   00000030: 00 00 6a 58 00 00
Jan 19 07:55:22 rpi-3b-plus-rev1d3-abfb kernel: [ 1914.808384] ------------[ cut here ]------------
Jan 19 07:55:22 rpi-3b-plus-rev1d3-abfb kernel: [ 1914.813200] lan78xx: caps=(0x0000010000114b09, 0x0000000000000000)
Jan 19 07:55:22 rpi-3b-plus-rev1d3-abfb kernel: [ 1914.819717] WARNING: CPU: 0 PID: 0 at net/core/dev.c:3197 skb_warn_bad_offload+0x84/0x100
Jan 19 07:55:22 rpi-3b-plus-rev1d3-abfb kernel: [ 1914.828190] Modules linked in:
Jan 19 07:55:22 rpi-3b-plus-rev1d3-abfb kernel: [ 1914.831354] CPU: 0 PID: 0 Comm: swapper/0 Not tainted 5.11.0-rc4 #103
Jan 19 07:55:22 rpi-3b-plus-rev1d3-abfb kernel: [ 1914.838009] Hardware name: Raspberry Pi 3 Model B Plus Rev 1.3 (DT)
Jan 19 07:55:22 rpi-3b-plus-rev1d3-abfb kernel: [ 1914.844478] pstate: 60000005 (nZCv daif -PAN -UAO -TCO BTYPE=--)
Jan 19 07:55:22 rpi-3b-plus-rev1d3-abfb kernel: [ 1914.850685] pc : skb_warn_bad_offload+0x84/0x100
Jan 19 07:55:22 rpi-3b-plus-rev1d3-abfb kernel: [ 1914.855464] lr : skb_warn_bad_offload+0x84/0x100
Jan 19 07:55:22 rpi-3b-plus-rev1d3-abfb kernel: [ 1914.860242] sp : ffff800010003850
Jan 19 07:55:22 rpi-3b-plus-rev1d3-abfb kernel: [ 1914.863665] x29: ffff800010003850 x28: ffff7a96fb196290
Jan 19 07:55:22 rpi-3b-plus-rev1d3-abfb kernel: [ 1914.869160] x27: ffff7a96c5958300 x26: 0000000000000001
Jan 19 07:55:22 rpi-3b-plus-rev1d3-abfb kernel: [ 1914.874654] x25: ffffa73eee323000 x24: ffff7a96ee84b000
Jan 19 07:55:22 rpi-3b-plus-rev1d3-abfb kernel: [ 1914.880148] x23: ffffa73eee7f4f00 x22: 0000000000000000
Jan 19 07:55:22 rpi-3b-plus-rev1d3-abfb kernel: [ 1914.885642] x21: ffffa73eee0327e0 x20: ffff7a96ee84b000
Jan 19 07:55:22 rpi-3b-plus-rev1d3-abfb kernel: [ 1914.891136] x19: ffff7a96c5958300 x18: 0000000000000010
Jan 19 07:55:22 rpi-3b-plus-rev1d3-abfb kernel: [ 1914.896630] x17: 0000000000000000 x16: 0000000000000000
Jan 19 07:55:22 rpi-3b-plus-rev1d3-abfb kernel: [ 1914.902123] x15: 000000000000ad55 x14: 0000000000000010
Jan 19 07:55:22 rpi-3b-plus-rev1d3-abfb kernel: [ 1914.907617] x13: 00000000ffffffff x12: ffffa73eedd9d950
Jan 19 07:55:22 rpi-3b-plus-rev1d3-abfb kernel: [ 1914.913109] x11: ffffa73eee885de0 x10: ffffa73eee86dda0
Jan 19 07:55:22 rpi-3b-plus-rev1d3-abfb kernel: [ 1914.918603] x9 : ffffa73eecf2f45c x8 : 0000000000017fe8
Jan 19 07:55:22 rpi-3b-plus-rev1d3-abfb kernel: [ 1914.924097] x7 : c0000000ffffefff x6 : 0000000000000003
Jan 19 07:55:22 rpi-3b-plus-rev1d3-abfb kernel: [ 1914.929590] x5 : 0000000000000000 x4 : 0000000000000000
Jan 19 07:55:22 rpi-3b-plus-rev1d3-abfb kernel: [ 1914.935081] x3 : 0000000000000100 x2 : 0000000000001000
Jan 19 07:55:22 rpi-3b-plus-rev1d3-abfb kernel: [ 1914.940575] x1 : 0000000000000000 x0 : 0000000000000000
Jan 19 07:55:22 rpi-3b-plus-rev1d3-abfb kernel: [ 1914.946070] Call trace:
Jan 19 07:55:22 rpi-3b-plus-rev1d3-abfb kernel: [ 1914.948599]  skb_warn_bad_offload+0x84/0x100
Jan 19 07:55:22 rpi-3b-plus-rev1d3-abfb kernel: [ 1914.953020]  netif_skb_features+0x218/0x2a0
Jan 19 07:55:22 rpi-3b-plus-rev1d3-abfb kernel: [ 1914.957350]  validate_xmit_skb.isra.0+0x28/0x2c8
Jan 19 07:55:22 rpi-3b-plus-rev1d3-abfb kernel: [ 1914.962125]  validate_xmit_skb_list+0x44/0x98
Jan 19 07:55:22 rpi-3b-plus-rev1d3-abfb kernel: [ 1914.966631]  sch_direct_xmit+0xf0/0x3a8
Jan 19 07:55:22 rpi-3b-plus-rev1d3-abfb kernel: [ 1914.970599]  __qdisc_run+0x140/0x668
Jan 19 07:55:22 rpi-3b-plus-rev1d3-abfb kernel: [ 1914.974297]  __dev_queue_xmit+0x59c/0x980
Jan 19 07:55:22 rpi-3b-plus-rev1d3-abfb kernel: [ 1914.978446]  dev_queue_xmit+0x1c/0x28
Jan 19 07:55:22 rpi-3b-plus-rev1d3-abfb kernel: [ 1914.982237]  ip_finish_output2+0x30c/0x558
Jan 19 07:55:22 rpi-3b-plus-rev1d3-abfb kernel: [ 1914.986476]  __ip_finish_output+0xe4/0x260
Jan 19 07:55:22 rpi-3b-plus-rev1d3-abfb kernel: [ 1914.990715]  ip_finish_output+0x3c/0xd8
Jan 19 07:55:22 rpi-3b-plus-rev1d3-abfb kernel: [ 1914.994683]  ip_output+0xb4/0x148
Jan 19 07:55:22 rpi-3b-plus-rev1d3-abfb kernel: [ 1914.998116]  ip_forward_finish+0x7c/0xc0
Jan 19 07:55:22 rpi-3b-plus-rev1d3-abfb kernel: [ 1915.002174]  ip_forward+0x42c/0x4f0
Jan 19 07:55:22 rpi-3b-plus-rev1d3-abfb kernel: [ 1915.005783]  ip_rcv_finish+0x98/0xb8
Jan 19 07:55:22 rpi-3b-plus-rev1d3-abfb kernel: [ 1915.009481]  ip_rcv+0xe0/0xf0
Jan 19 07:55:22 rpi-3b-plus-rev1d3-abfb kernel: [ 1915.012552]  __netif_receive_skb_one_core+0x5c/0x88
Jan 19 07:55:22 rpi-3b-plus-rev1d3-abfb kernel: [ 1915.017597]  __netif_receive_skb+0x20/0x70
Jan 19 07:55:22 rpi-3b-plus-rev1d3-abfb kernel: [ 1915.021834]  process_backlog+0xc0/0x1d0
Jan 19 07:55:22 rpi-3b-plus-rev1d3-abfb kernel: [ 1915.025802]  net_rx_action+0x134/0x478
Jan 19 07:55:22 rpi-3b-plus-rev1d3-abfb kernel: [ 1915.029682]  __do_softirq+0x130/0x378
Jan 19 07:55:22 rpi-3b-plus-rev1d3-abfb kernel: [ 1915.033472]  irq_exit+0xc0/0xe8
Jan 19 07:55:22 rpi-3b-plus-rev1d3-abfb kernel: [ 1915.036725]  __handle_domain_irq+0x70/0xc8
Jan 19 07:55:22 rpi-3b-plus-rev1d3-abfb kernel: [ 1915.040963]  bcm2836_arm_irqchip_handle_irq+0x6c/0x80
Jan 19 07:55:22 rpi-3b-plus-rev1d3-abfb kernel: [ 1915.046185]  el1_irq+0xb4/0x140
Jan 19 07:55:22 rpi-3b-plus-rev1d3-abfb kernel: [ 1915.053377]  arch_cpu_idle+0x18/0x28
Jan 19 07:55:22 rpi-3b-plus-rev1d3-abfb kernel: [ 1915.060981]  default_idle_call+0x44/0x178
Jan 19 07:55:22 rpi-3b-plus-rev1d3-abfb kernel: [ 1915.069009]  do_idle+0x224/0x270
Jan 19 07:55:22 rpi-3b-plus-rev1d3-abfb kernel: [ 1915.076147]  cpu_startup_entry+0x30/0x98
Jan 19 07:55:22 rpi-3b-plus-rev1d3-abfb kernel: [ 1915.083916]  rest_init+0xc8/0xd8
Jan 19 07:55:22 rpi-3b-plus-rev1d3-abfb kernel: [ 1915.090937]  arch_call_rest_init+0x18/0x24
Jan 19 07:55:22 rpi-3b-plus-rev1d3-abfb kernel: [ 1915.098829]  start_kernel+0x57c/0x5b8
Jan 19 07:55:22 rpi-3b-plus-rev1d3-abfb kernel: [ 1915.106251] ---[ end trace c3d8dd12ce1805e0 ]---

If I also add the following rule:

  $ iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE
I get a single warning followed by a TX timeout:

Jan 19 08:15:47 rpi-3b-plus-rev1d3-abfb kernel: [   81.516888] skb len=66 headroom=5194 headlen=66 tailroom=10804
Jan 19 08:15:47 rpi-3b-plus-rev1d3-abfb kernel: [   81.516888] mac=(5194,14) net=(5208,20) trans=5228
Jan 19 08:15:47 rpi-3b-plus-rev1d3-abfb kernel: [   81.516888] shinfo(txflags=0 nr_frags=0 gso(size=1448 type=0 segs=1))
Jan 19 08:15:47 rpi-3b-plus-rev1d3-abfb kernel: [   81.516888] csum(0xeedb ip_summed=1 complete_sw=0 valid=0 level=0)
Jan 19 08:15:47 rpi-3b-plus-rev1d3-abfb kernel: [   81.516888] hash(0x0 sw=0 l4=0) proto=0x0800 pkttype=0 iif=2
Jan 19 08:15:47 rpi-3b-plus-rev1d3-abfb kernel: [   81.546872] dev name=eth0 feat=0x0x0000010000114b09
Jan 19 08:15:47 rpi-3b-plus-rev1d3-abfb kernel: [   81.552060] skb linear:   00000000: e0 28 6d 9e b9 22 b8 27 eb 3e ab fb 08 00 45 00
Jan 19 08:15:47 rpi-3b-plus-rev1d3-abfb kernel: [   81.560090] skb linear:   00000010: 00 34 90 99 40 00 3f 06 87 40 c0 a8 63 84 22 6b
Jan 19 08:15:47 rpi-3b-plus-rev1d3-abfb kernel: [   81.568019] skb linear:   00000020: dd 52 d0 ac 00 50 35 e0 1e 2c 78 02 47 fa 80 10
Jan 19 08:15:47 rpi-3b-plus-rev1d3-abfb kernel: [   81.575921] skb linear:   00000030: 01 f6 d6 96 00 00 01 01 08 0a 50 c9 d7 4b cd 2e
Jan 19 08:15:47 rpi-3b-plus-rev1d3-abfb kernel: [   81.583918] skb linear:   00000040: 9f fc
Jan 19 08:15:47 rpi-3b-plus-rev1d3-abfb kernel: [   81.588105] ------------[ cut here ]------------
Jan 19 08:15:47 rpi-3b-plus-rev1d3-abfb kernel: [   81.592920] lan78xx: caps=(0x0000010000114b09, 0x0000000000000000)
Jan 19 08:15:47 rpi-3b-plus-rev1d3-abfb kernel: [   81.599429] WARNING: CPU: 0 PID: 0 at net/core/dev.c:3197 skb_warn_bad_offload+0x84/0x100
Jan 19 08:15:47 rpi-3b-plus-rev1d3-abfb kernel: [   81.607900] Modules linked in:
Jan 19 08:15:47 rpi-3b-plus-rev1d3-abfb kernel: [   81.611064] CPU: 0 PID: 0 Comm: swapper/0 Not tainted 5.11.0-rc4 #103
Jan 19 08:15:47 rpi-3b-plus-rev1d3-abfb kernel: [   81.617720] Hardware name: Raspberry Pi 3 Model B Plus Rev 1.3 (DT)
Jan 19 08:15:47 rpi-3b-plus-rev1d3-abfb kernel: [   81.624189] pstate: 60000005 (nZCv daif -PAN -UAO -TCO BTYPE=--)
Jan 19 08:15:47 rpi-3b-plus-rev1d3-abfb kernel: [   81.630396] pc : skb_warn_bad_offload+0x84/0x100
Jan 19 08:15:47 rpi-3b-plus-rev1d3-abfb kernel: [   81.635175] lr : skb_warn_bad_offload+0x84/0x100
Jan 19 08:15:47 rpi-3b-plus-rev1d3-abfb kernel: [   81.639953] sp : ffff800010003810
Jan 19 08:15:47 rpi-3b-plus-rev1d3-abfb kernel: [   81.643374] x29: ffff800010003810 x28: ffff50043b196290
Jan 19 08:15:47 rpi-3b-plus-rev1d3-abfb kernel: [   81.648870] x27: ffff500407371600 x26: 0000000000000001
Jan 19 08:15:47 rpi-3b-plus-rev1d3-abfb kernel: [   81.654365] x25: ffffa1fa11b23000 x24: ffff50042e96b000
Jan 19 08:15:47 rpi-3b-plus-rev1d3-abfb kernel: [   81.659859] x23: ffffa1fa11ff4f00 x22: 0000000000000000
Jan 19 08:15:47 rpi-3b-plus-rev1d3-abfb kernel: [   81.665353] x21: ffffa1fa118327e0 x20: ffff50042e96b000
Jan 19 08:15:47 rpi-3b-plus-rev1d3-abfb kernel: [   81.670847] x19: ffff500407371600 x18: 0000000000000010
Jan 19 08:15:47 rpi-3b-plus-rev1d3-abfb kernel: [   81.676340] x17: 0000000000000000 x16: 0000000000000000
Jan 19 08:15:47 rpi-3b-plus-rev1d3-abfb kernel: [   81.681833] x15: 000000000000ad55 x14: 0000000000000010
Jan 19 08:15:47 rpi-3b-plus-rev1d3-abfb kernel: [   81.687326] x13: 00000000ffffffff x12: ffffa1fa1159d950
Jan 19 08:15:47 rpi-3b-plus-rev1d3-abfb kernel: [   81.692819] x11: ffffa1fa12085de0 x10: ffffa1fa1206dda0
Jan 19 08:15:47 rpi-3b-plus-rev1d3-abfb kernel: [   81.698313] x9 : ffffa1fa1072f45c x8 : 0000000000017fe8
Jan 19 08:15:47 rpi-3b-plus-rev1d3-abfb kernel: [   81.703806] x7 : c0000000ffffefff x6 : 0000000000000003
Jan 19 08:15:47 rpi-3b-plus-rev1d3-abfb kernel: [   81.709300] x5 : 0000000000000000 x4 : 0000000000000000
Jan 19 08:15:47 rpi-3b-plus-rev1d3-abfb kernel: [   81.714791] x3 : 0000000000000100 x2 : 0000000000001000
Jan 19 08:15:47 rpi-3b-plus-rev1d3-abfb kernel: [   81.720283] x1 : 0000000000000000 x0 : 0000000000000000
Jan 19 08:15:47 rpi-3b-plus-rev1d3-abfb kernel: [   81.725778] Call trace:
Jan 19 08:15:47 rpi-3b-plus-rev1d3-abfb kernel: [   81.728306]  skb_warn_bad_offload+0x84/0x100
Jan 19 08:15:47 rpi-3b-plus-rev1d3-abfb kernel: [   81.732728]  netif_skb_features+0x218/0x2a0
Jan 19 08:15:47 rpi-3b-plus-rev1d3-abfb kernel: [   81.737057]  validate_xmit_skb.isra.0+0x28/0x2c8
Jan 19 08:15:47 rpi-3b-plus-rev1d3-abfb kernel: [   81.741833]  validate_xmit_skb_list+0x44/0x98
Jan 19 08:15:47 rpi-3b-plus-rev1d3-abfb kernel: [   81.746339]  sch_direct_xmit+0xf0/0x3a8
Jan 19 08:15:47 rpi-3b-plus-rev1d3-abfb kernel: [   81.750309]  __qdisc_run+0x140/0x668
Jan 19 08:15:47 rpi-3b-plus-rev1d3-abfb kernel: [   81.754008]  __dev_queue_xmit+0x59c/0x980
Jan 19 08:15:47 rpi-3b-plus-rev1d3-abfb kernel: [   81.758156]  dev_queue_xmit+0x1c/0x28
Jan 19 08:15:47 rpi-3b-plus-rev1d3-abfb kernel: [   81.761945]  neigh_resolve_output+0x108/0x230
Jan 19 08:15:47 rpi-3b-plus-rev1d3-abfb kernel: [   81.766450]  ip_finish_output2+0x180/0x558
Jan 19 08:15:47 rpi-3b-plus-rev1d3-abfb kernel: [   81.770690]  __ip_finish_output+0xe4/0x260
Jan 19 08:15:47 rpi-3b-plus-rev1d3-abfb kernel: [   81.774928]  ip_finish_output+0x3c/0xd8
Jan 19 08:15:47 rpi-3b-plus-rev1d3-abfb kernel: [   81.778896]  ip_output+0xb4/0x148
Jan 19 08:15:47 rpi-3b-plus-rev1d3-abfb kernel: [   81.782328]  ip_forward_finish+0x7c/0xc0
Jan 19 08:15:47 rpi-3b-plus-rev1d3-abfb kernel: [   81.786385]  ip_forward+0x42c/0x4f0
Jan 19 08:15:47 rpi-3b-plus-rev1d3-abfb kernel: [   81.789995]  ip_rcv_finish+0x98/0xb8
Jan 19 08:15:47 rpi-3b-plus-rev1d3-abfb kernel: [   81.793694]  ip_rcv+0xe0/0xf0
Jan 19 08:15:47 rpi-3b-plus-rev1d3-abfb kernel: [   81.796765]  __netif_receive_skb_one_core+0x5c/0x88
Jan 19 08:15:47 rpi-3b-plus-rev1d3-abfb kernel: [   81.801810]  __netif_receive_skb+0x20/0x70
Jan 19 08:15:47 rpi-3b-plus-rev1d3-abfb kernel: [   81.806047]  process_backlog+0xc0/0x1d0
Jan 19 08:15:47 rpi-3b-plus-rev1d3-abfb kernel: [   81.810016]  net_rx_action+0x134/0x478
Jan 19 08:15:47 rpi-3b-plus-rev1d3-abfb kernel: [   81.813897]  __do_softirq+0x130/0x378
Jan 19 08:15:47 rpi-3b-plus-rev1d3-abfb kernel: [   81.817686]  irq_exit+0xc0/0xe8
Jan 19 08:15:47 rpi-3b-plus-rev1d3-abfb kernel: [   81.820940]  __handle_domain_irq+0x70/0xc8
Jan 19 08:15:47 rpi-3b-plus-rev1d3-abfb kernel: [   81.829099]  bcm2836_arm_irqchip_handle_irq+0x6c/0x80
Jan 19 08:15:47 rpi-3b-plus-rev1d3-abfb kernel: [   81.838223]  el1_irq+0xb4/0x140
Jan 19 08:15:47 rpi-3b-plus-rev1d3-abfb kernel: [   81.845371]  arch_cpu_idle+0x18/0x28
Jan 19 08:15:47 rpi-3b-plus-rev1d3-abfb kernel: [   81.852882]  default_idle_call+0x44/0x178
Jan 19 08:15:47 rpi-3b-plus-rev1d3-abfb kernel: [   81.860756]  do_idle+0x224/0x270
Jan 19 08:15:47 rpi-3b-plus-rev1d3-abfb kernel: [   81.867794]  cpu_startup_entry+0x30/0x98
Jan 19 08:15:47 rpi-3b-plus-rev1d3-abfb kernel: [   81.875516]  rest_init+0xc8/0xd8
Jan 19 08:15:47 rpi-3b-plus-rev1d3-abfb kernel: [   81.882496]  arch_call_rest_init+0x18/0x24
Jan 19 08:15:47 rpi-3b-plus-rev1d3-abfb kernel: [   81.890352]  start_kernel+0x57c/0x5b8
Jan 19 08:15:47 rpi-3b-plus-rev1d3-abfb kernel: [   81.897706] ---[ end trace a5789410f231a10b ]---
Jan 19 08:16:15 rpi-3b-plus-rev1d3-abfb kernel: [  110.046337] ------------[ cut here ]------------
Jan 19 08:16:15 rpi-3b-plus-rev1d3-abfb kernel: [  110.054787] NETDEV WATCHDOG: eth0 (lan78xx): transmit queue 0 timed out
Jan 19 08:16:15 rpi-3b-plus-rev1d3-abfb kernel: [  110.065356] WARNING: CPU: 2 PID: 0 at net/sched/sch_generic.c:442 dev_watchdog+0x384/0x390
Jan 19 08:16:15 rpi-3b-plus-rev1d3-abfb kernel: [  110.077534] Modules linked in:
Jan 19 08:16:15 rpi-3b-plus-rev1d3-abfb kernel: [  110.084361] CPU: 2 PID: 0 Comm: swapper/2 Tainted: G        W         5.11.0-rc4 #103
Jan 19 08:16:15 rpi-3b-plus-rev1d3-abfb kernel: [  110.096114] Hardware name: Raspberry Pi 3 Model B Plus Rev 1.3 (DT)
Jan 19 08:16:15 rpi-3b-plus-rev1d3-abfb kernel: [  110.106246] pstate: 60000005 (nZCv daif -PAN -UAO -TCO BTYPE=--)
Jan 19 08:16:15 rpi-3b-plus-rev1d3-abfb kernel: [  110.116085] pc : dev_watchdog+0x384/0x390
Jan 19 08:16:15 rpi-3b-plus-rev1d3-abfb kernel: [  110.123857] lr : dev_watchdog+0x384/0x390
Jan 19 08:16:15 rpi-3b-plus-rev1d3-abfb kernel: [  110.131558] sp : ffff800010013d90
Jan 19 08:16:15 rpi-3b-plus-rev1d3-abfb kernel: [  110.138497] x29: ffff800010013d90 x28: 0000000000000140
Jan 19 08:16:15 rpi-3b-plus-rev1d3-abfb kernel: [  110.147472] x27: 00000000ffffffff x26: ffffa1fa11b23000
Jan 19 08:16:15 rpi-3b-plus-rev1d3-abfb kernel: [  110.156489] x25: 0000000000000002 x24: 0000000000000000
Jan 19 08:16:15 rpi-3b-plus-rev1d3-abfb kernel: [  110.165496] x23: 0000000000000001 x22: ffff50042e96b000
Jan 19 08:16:15 rpi-3b-plus-rev1d3-abfb kernel: [  110.174494] x21: ffff50042e96b440 x20: ffffa1fa11fe7000
Jan 19 08:16:15 rpi-3b-plus-rev1d3-abfb kernel: [  110.183490] x19: 0000000000000000 x18: 0000000000000010
Jan 19 08:16:15 rpi-3b-plus-rev1d3-abfb kernel: [  110.192493] x17: 0000000000000000 x16: 0000000000000000
Jan 19 08:16:15 rpi-3b-plus-rev1d3-abfb kernel: [  110.201473] x15: 000000000000ad55 x14: 0000000000000010
Jan 19 08:16:15 rpi-3b-plus-rev1d3-abfb kernel: [  110.210439] x13: 00000000ffffffff x12: ffffa1fa1159d950
Jan 19 08:16:15 rpi-3b-plus-rev1d3-abfb kernel: [  110.219397] x11: ffffa1fa12085de0 x10: ffffa1fa1206dda0
Jan 19 08:16:15 rpi-3b-plus-rev1d3-abfb kernel: [  110.228367] x9 : ffffa1fa1072f45c x8 : 0000000000017fe8
Jan 19 08:16:15 rpi-3b-plus-rev1d3-abfb kernel: [  110.237362] x7 : c0000000ffffefff x6 : 0000000000000003
Jan 19 08:16:15 rpi-3b-plus-rev1d3-abfb kernel: [  110.246353] x5 : 0000000000000000 x4 : 0000000000000000
Jan 19 08:16:15 rpi-3b-plus-rev1d3-abfb kernel: [  110.255328] x3 : 0000000000000100 x2 : 0000000000001000
Jan 19 08:16:15 rpi-3b-plus-rev1d3-abfb kernel: [  110.264273] x1 : 0000000000000000 x0 : 0000000000000000
Jan 19 08:16:15 rpi-3b-plus-rev1d3-abfb kernel: [  110.273192] Call trace:
Jan 19 08:16:15 rpi-3b-plus-rev1d3-abfb kernel: [  110.279183]  dev_watchdog+0x384/0x390
Jan 19 08:16:15 rpi-3b-plus-rev1d3-abfb kernel: [  110.286461]  call_timer_fn+0x38/0x188
Jan 19 08:16:15 rpi-3b-plus-rev1d3-abfb kernel: [  110.293762]  run_timer_softirq+0x494/0x688
Jan 19 08:16:15 rpi-3b-plus-rev1d3-abfb kernel: [  110.301489]  __do_softirq+0x130/0x378
Jan 19 08:16:15 rpi-3b-plus-rev1d3-abfb kernel: [  110.308767]  irq_exit+0xc0/0xe8
Jan 19 08:16:15 rpi-3b-plus-rev1d3-abfb kernel: [  110.315500]  __handle_domain_irq+0x70/0xc8
Jan 19 08:16:15 rpi-3b-plus-rev1d3-abfb kernel: [  110.323214]  bcm2836_arm_irqchip_handle_irq+0x6c/0x80
Jan 19 08:16:15 rpi-3b-plus-rev1d3-abfb kernel: [  110.331940]  el1_irq+0xb4/0x140
Jan 19 08:16:15 rpi-3b-plus-rev1d3-abfb kernel: [  110.338706]  arch_cpu_idle+0x18/0x28
Jan 19 08:16:15 rpi-3b-plus-rev1d3-abfb kernel: [  110.345916]  default_idle_call+0x44/0x178
Jan 19 08:16:15 rpi-3b-plus-rev1d3-abfb kernel: [  110.353577]  do_idle+0x224/0x270
Jan 19 08:16:15 rpi-3b-plus-rev1d3-abfb kernel: [  110.360433]  cpu_startup_entry+0x2c/0x98
Jan 19 08:16:15 rpi-3b-plus-rev1d3-abfb kernel: [  110.368000]  secondary_start_kernel+0x148/0x180
Jan 19 08:16:15 rpi-3b-plus-rev1d3-abfb kernel: [  110.376199] ---[ end trace a5789410f231a10c ]---

Enabling KASAN didn't trigger any error reports.

Let me know if there's anything else I can try to narrow this down.

...Juerg

[1]
On the Pi, I run:

  $ nc -l 1234 | dd status=progress >/dev/null

And on another machine, that is configured to use the Pi as the gateway:

  $ nc 192.168.99.115 1234 < /dev/urandom
and a couple of firefox instances that keep opening public URls.

[2]

Author: Eric Dumazet <[email protected]>
Date:   Tue Nov 27 14:42:03 2018 -0800

tcp: implement coalescing on backlog queue

In case GRO is not as efficient as it should be or disabled, we might have a user thread trapped in __release_sock() while softirq handler flood packets up to the point we have to drop.

This patch balances work done from user thread and softirq, to give more chances to __release_sock() to complete its work before new packets are added the the backlog.

This also helps if we receive many ACK packets, since GRO does not aggregate them.

This patch brings ~60% throughput increase on a receiver without GRO, but the spectacular gain is really on 1000x release_sock() latency reduction I have measured.

Quoted 8 lines

Signed-off-by Eric Dumazet <[email protected]>
Cc Neal Cardwell <[email protected]>
Cc Yuchung Cheng <[email protected]>
Acked-by Neal Cardwell <[email protected]>
Signed-off-by David S. Miller <[email protected]>
[3] 86bccd036713 tcp: fix receive window update in tcp_add_backlog()
[4] ca2fe2956ace tcp: add sanity tests in tcp_add_backlog()

Oops. Very nice detective work :)

Quoted 8 lines

It is true that the skb_clone() done in lan78xx (and some other usb drivers) is probably triggering this issue.
(lan78xx is also lying about skb->truesize)
skb_try_coalesce() bails if the target skb is cloned, but not if the source is.
Can you try the following patch ?

Works. Nice :-)

If you submit this and care you can add:

Tested-by Juerg Haefliger <[email protected]>

Thanks a lot for the quick turnaround!

...Juerg

Quoted 68 lines

net/ipv4/tcp_ipv4.c +0 -0

index 58207c7769d05693b650e3c93e4ef405a5d4b23a..4e82745d336fc3fb0d9ce8c92aaeb39702f64b8a

100644

net/ipv4/tcp_ipv4.c +2 -7

@@ -1760,6 +1760,7 @@ int tcp_v4_early_demux(struct sk_buff *skb)
 bool tcp_add_backlog(struct sock *sk, struct sk_buff *skb)
 {
        u32 limit = READ_ONCE(sk->sk_rcvbuf) + READ_ONCE(sk->sk_sndbuf);
+       u32 tail_gso_size, tail_gso_segs;
        struct skb_shared_info *shinfo;
        const struct tcphdr *th;
        struct tcphdr *thtail;

@@ -1767,6 +1768,7 @@ bool tcp_add_backlog(struct sock *sk, struct sk_buff *skb)
        unsigned int hdrlen;
        bool fragstolen;
        u32 gso_segs;
+       u32 gso_size;
        int delta;
 
        /* In case all data was pulled from skb frags (in __pskb_pull_tail()),

@@ -1792,13 +1794,6 @@ bool tcp_add_backlog(struct sock *sk, struct
 sk_buff *skb)
         */
        th = (const struct tcphdr *)skb->data;
        hdrlen = th->doff * 4;
-       shinfo = skb_shinfo(skb);
-
-       if (!shinfo->gso_size)
-               shinfo->gso_size = skb->len - hdrlen;
-
-       if (!shinfo->gso_segs)
-               shinfo->gso_segs = 1;
 
        tail = sk->sk_backlog.tail;

        if (!tail)

net/ipv4/tcp_ipv4.c +11 -5

@@ -1821,6 +1816,15 @@ bool tcp_add_backlog(struct sock *sk, struct
 sk_buff *skb)
                goto no_coalesce;
 
        __skb_pull(skb, hdrlen);
+
+       shinfo = skb_shinfo(skb);
+       gso_size = shinfo->gso_size ?: skb->len;
+       gso_segs = shinfo->gso_segs ?: 1;
+
+       shinfo = skb_shinfo(tail);
+       tail_gso_size = shinfo->gso_size ?: (tail->len - hdrlen);
+       tail_gso_segs = shinfo->gso_segs ?: 1;
+
        if (skb_try_coalesce(tail, skb, &fragstolen, &delta)) {
                TCP_SKB_CB(tail)->end_seq = TCP_SKB_CB(skb)->end_seq;

@@ -1847,11 +1851,8 @@ bool tcp_add_backlog(struct sock *sk, struct
 sk_buff *skb)
                }
 
                /* Not as strict as GRO. We only need to carry mss max value */
-               skb_shinfo(tail)->gso_size = max(shinfo->gso_size,
-                                                skb_shinfo(tail)->gso_size);
-
-               gso_segs = skb_shinfo(tail)->gso_segs + shinfo->gso_segs;
-               skb_shinfo(tail)->gso_segs = min_t(u32, gso_segs, 0xFFFF);
+               shinfo->gso_size = max(gso_size, tail_gso_size);
+               shinfo->gso_segs = min_t(u32, gso_segs + tail_gso_segs, 0xFFFF);
 
                sk->sk_backlog.len += delta;

                __NET_INC_STATS(sock_net(sk),

Eric Dumazet

2021-01-19 15:52 UTC #17 lore raw

Quoted 12 lines · Juerg Haefliger

Quoted 7 lines · On Tue, 19 Jan 2021 14:54:31 +0100 Eric Dumazet <[email protected]> wrote:
Oops. Very nice detective work :)
It is true that the skb_clone() done in lan78xx (and some other usb drivers) is probably triggering this issue.
(lan78xx is also lying about skb->truesize)

Quoted 7 lines

Quoted 4 lines
skb_try_coalesce() bails if the target skb is cloned, but not if the source is.
Can you try the following patch ?
Works. Nice :-)

Excellent !

Quoted 3 lines

If you submit this and care you can add:
Tested-by Juerg Haefliger <[email protected]>

Sure, I will also add a :

Bisected-by Juerg Haefliger <[email protected]>

Because you did quite a lot of work narrowing the problem !

Quoted 6 lines

Thanks a lot for the quick turnaround!
...Juerg

Heiner Kallweit

author

2021-01-20 00:16 UTC #18 lore raw

In reply to Juerg Haefliger

Quoted 447 lines · Juerg Haefliger

Quoted 164 lines · On Thu, 8 Oct 2020 20:50:28 +0200 Eric Dumazet <[email protected]> wrote:

Quoted 152 lines · Heiner Kallweit
Quoted 90 lines · Eric Dumazet
Quoted 89 lines · Heiner Kallweit
Quoted 72 lines · Eric Dumazet
Quoted 40 lines · Heiner Kallweit
Quoted 38 lines · Eric Dumazet
Quoted 19 lines · Eric Dumazet
Quoted 15 lines · Eric Dumazet
Quoted 11 lines · Heiner Kallweit
I have a problem with the following code in ndo_start_xmit() of the r8169 driver. A user reported the WARN being triggered due to gso_size > 0 and gso_type = 0. The chip supports TSO(6). The driver is widely used, therefore I'd expect much more such reports if it should be a common problem. Not sure what's special. My primary question: Is it a valid use case that gso_size is greater than 0, and no SKB_GSO_ flag is set?
Any hint would be appreciated.
Maybe this is not a TCP packet ? But in this case GSO should have taken place.
Quoted 8 lines
Quoted 3 lines
You might add a
pr_err_once("gso_type=%x\n", shinfo->gso_type);
Ah, sorry I see you already printed gso_type
Must then be a bug somewhere :/
napi_reuse_skb() does :
skb_shinfo(skb)->gso_type = 0;
It does _not_ clear gso_size.
I wonder if in some cases we could reuse an skb while gso_size is not zero.
Quoted 8 lines
Quoted 3 lines
Normally, we set it only from dev_gro_receive() when the skb is queued into GRO engine (status being GRO_HELD)
Thanks Eric. I'm no expert that deep in the network stack and just wonder why napi_reuse_skb() re-initializes less fields in shinfo than __alloc_skb(). The latter one does a
memset(shinfo, 0, offsetof(struct skb_shared_info, dataref));
memset() over the whole thing is more expensive.
Here we know the prior state of some fields, while __alloc_skb() just got a piece of memory with random content.
Quoted 7 lines
What I can do is letting the affected user test the following.
net/core/dev.c +0 -0
index 62b06523b..8e75399cc 100644
net/core/dev.c static void napi_reuse_skb(struct napi_struct *napi, struct sk_buff *skb) · line 6094
Quoted 7 lines
net/core/dev.c +1 -0
@@ -6088,6 +6088,7 @@ static void napi_reuse_skb(struct napi_struct *napi, struct sk_buff *skb)
 
        skb->encapsulation = 0;
        skb_shinfo(skb)->gso_type = 0;
+       skb_shinfo(skb)->gso_size = 0;
        skb->truesize = SKB_TRUESIZE(skb_end_offset(skb));
        skb_ext_reset(skb);
 
As I hinted, this should not be needed.
Quoted 4 lines
For debugging purposes, I would rather do :
BUG_ON(skb_shinfo(skb)->gso_size);
We did the following for debugging:
net/core/dev.c +0 -0
index 62b06523b..4c943b774 100644
@@ -3491,6 +3491,9 @@ static netdev_features_t gso_features_check(const struct sk_buff *skb,
 {
        u16 gso_segs = skb_shinfo(skb)->gso_segs;
 
Quoted 5 lines
net/core/dev.c static netdev_features_t gso_features_check(const struct sk_buff *skb, · line 3495
Quoted 2 lines
net/core/dev.c +2 -0
+       if (!skb_shinfo(skb)->gso_type)
+               skb_warn_bad_offload(skb);  
You also want to get a stack trace here, to give us the call graph.
Here it comes, full story is in https://bugzilla.kernel.org/show_bug.cgi?id=209423
[236222.967498] ------------[ cut here ]------------
[236222.967508] r8169: caps=(0x00000100000041b2, 0x0000000000000000)
[236222.967668] WARNING: CPU: 0 PID: 0 at net/core/dev.c:3184 skb_warn_bad_offload+0x72/0xe0
[236222.967691] Modules linked in: tcp_diag udp_diag raw_diag inet_diag unix_diag tun nft_nat nft_masq nft_objref nf_conntrack_netbios_ns nf_conntrack_broadcast nft_fib_inet nft_fib_ipv4 nft_fib_ipv6 nft_fib nft_reject_inet nf_reject_ipv4 nf_reject_ipv6 nft_reject nft_ct nft_chain_nat ip_set_hash_net ip6table_nat ip6table_mangle ip6table_raw ip6table_security iptable_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 iptable_mangle iptable_raw iptable_security ip_set nf_tables nfnetlink ip6table_filter ip6_tables iptable_filter sunrpc vfat fat snd_hda_codec_realtek snd_hda_codec_generic edac_mce_amd ledtrig_audio kvm_amd snd_hda_codec_hdmi ccp snd_hda_intel snd_intel_dspcfg kvm snd_hda_codec snd_hda_core snd_hwdep irqbypass snd_pcm snd_timer snd hp_wmi sp5100_tco sparse_keymap wmi_bmof fam15h_power k10temp i2c_piix4 soundcore rfkill_gpio rfkill acpi_cpufreq ip_tables xfs amdgpu iommu_v2 gpu_sched i2c_algo_bit ttm drm_kms_helper cec crct10dif_pclmul crc32_pclmul crc32c_intel drm
[236222.967776]  ghash_clmulni_intel ax88179_178a serio_raw usbnet mii r8169 wmi video
[236222.967858] CPU: 0 PID: 0 Comm: swapper/0 Not tainted 5.8.12-203.fc32.x86_64 #1
[236222.967870] Hardware name: HP HP t630 Thin Client/8158, BIOS M40 v01.12 02/04/2020
[236222.967895] RIP: 0010:skb_warn_bad_offload+0x72/0xe0
[236222.967908] Code: 8d 95 c8 00 00 00 48 8d 88 e8 01 00 00 48 85 c0 48 c7 c0 d8 d7 15 a4 48 0f 44 c8 4c 89 e6 48 c7 c7 90 7b 47 a4 e8 04 85 72 ff <0f> 0b 5b 5d 41 5c c3 80 7d 00 00 49 c7 c4 3b 28 40 a4 74 ac be 25
[236222.967926] RSP: 0018:ffffa8f9c0003c80 EFLAGS: 00010282
[236222.967938] RAX: 0000000000000034 RBX: ffff8d7090f2cd00 RCX: 0000000000000000
[236222.967951] RDX: ffff8d709b427060 RSI: ffff8d709b418d00 RDI: 0000000000000300
[236222.967962] RBP: ffff8d709a9fc000 R08: 0000000000000406 R09: 0720072007200720
[236222.967974] R10: 0720072007200720 R11: 0729073007300730 R12: ffffffffc012e729
[236222.967986] R13: ffffa8f9c0003d3b R14: 0000000000000000 R15: ffff8d70367652ac
[236222.968000] FS:  0000000000000000(0000) GS:ffff8d709b400000(0000) knlGS:0000000000000000
[236222.968013] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[236222.968023] CR2: 00007f3cf5ebf010 CR3: 0000000113cc6000 CR4: 00000000001406f0
[236222.968035] Call Trace:
[236222.968047]  <IRQ>
[236222.968064]  netif_skb_features+0x25e/0x2c0
[236222.968084]  ? ipt_do_table+0x333/0x600 [ip_tables]
[236222.968098]  validate_xmit_skb+0x1d/0x300
[236222.968111]  validate_xmit_skb_list+0x48/0x70
[236222.968126]  sch_direct_xmit+0x129/0x2f0
[236222.968140]  __dev_queue_xmit+0x710/0x8a0
[236222.968184]  ? nf_confirm+0xcb/0xf0 [nf_conntrack]
[236222.968200]  ? nf_hook_slow+0x3f/0xb0
[236222.968214]  ip_finish_output2+0x2ad/0x560
[236222.968229]  __netif_receive_skb_core+0x4f0/0xf40
[236222.968244]  ? packet_rcv+0x44/0x490
[236222.968257]  __netif_receive_skb_one_core+0x2d/0x70
[236222.968277]  process_backlog+0x96/0x160
[236222.968290]  net_rx_action+0x13c/0x3e0
[236222.968312]  ? usbnet_bh+0x24/0x2b0 [usbnet]
[236222.968327]  __do_softirq+0xd9/0x2c4
[236222.968340]  asm_call_on_stack+0x12/0x20
[236222.968350]  </IRQ>
[236222.968362]  do_softirq_own_stack+0x39/0x50
[236222.968376]  irq_exit_rcu+0xc2/0x100
[236222.968389]  common_interrupt+0x75/0x140
[236222.968405]  asm_common_interrupt+0x1e/0x40
[236222.968427] RIP: 0010:native_safe_halt+0xe/0x10
[236222.968438] Code: 02 20 48 8b 00 a8 08 75 c4 e9 7b ff ff ff cc cc cc cc cc cc cc cc cc cc cc cc cc cc e9 07 00 00 00 0f 00 2d f6 69 49 00 fb f4 <c3> 90 e9 07 00 00 00 0f 00 2d e6 69 49 00 f4 c3 cc cc 0f 1f 44 00
[236222.968456] RSP: 0018:ffffffffa4a03e08 EFLAGS: 00000246
[236222.968467] RAX: 0000000000004000 RBX: 0000000000000001 RCX: 000000000000001f
[236222.968480] RDX: 4ec4ec4ec4ec4ec5 RSI: ffffffffa4b78960 RDI: ffff8d7092f45c00
[236222.968492] RBP: ffff8d709a288000 R08: 0000d6d7f20a4084 R09: 0000000000000006
[236222.968504] R10: 0000000000000022 R11: 000000000000000f R12: ffff8d709a288064
[236222.968515] R13: 0000000000000001 R14: 0000000000000001 R15: 0000000000000000
[236222.968535]  acpi_safe_halt+0x1b/0x30
Quoted 10 lines
[236222.968549]  acpi_idle_enter+0x27e/0x2e0
[236222.968566]  cpuidle_enter_state+0x81/0x3f0
[236222.968589]  cpuidle_enter+0x29/0x40
[236222.968602]  do_idle+0x1d5/0x2a0
[236222.968615]  cpu_startup_entry+0x19/0x20
[236222.968628]  start_kernel+0x7f4/0x804
[236222.968645]  secondary_startup_64+0xb6/0xc0
[236222.968659] ---[ end trace 8a4d7f639ad88505 ]---

Quoted 46 lines

OK, it would be nice to know what is the input interface
if4 -> look at "ip link | grep 4:"
Then identifying the driver that built such a strange packet (32000 bytes allocated in skb->head)
ethtool -i ifname
Quoted 35 lines
Quoted 35 lines
Quoted 35 lines
net/core/dev.c +1 -0
+
        if (gso_segs > dev->gso_max_segs)
                return features & ~NETIF_F_GSO_MASK;
 
Following skb then triggered the skb_warn_bad_offload. Not sure whether this helps to find out where in the network stack something goes wrong.
[236222.967236] skb len=134 headroom=778 headlen=134 tailroom=31536
                mac=(778,14) net=(792,20) trans=812
                shinfo(txflags=0 nr_frags=0 gso(size=568 type=0 segs=1))
                csum(0x0 ip_summed=1 complete_sw=0 valid=0 level=0)
                hash(0x0 sw=0 l4=0) proto=0x0800 pkttype=0 iif=4
[236222.967297] dev name=enp1s0 feat=0x0x00000100000041b2
[236222.967392] skb linear:   00000000: 00 13 3b a0 01 e8 7c d3 0a 2d 1b 3b 08 00 45 00
[236222.967404] skb linear:   00000010: 00 78 e2 e6 00 00 7b 06 52 e1 d8 3a d0 ce c0 a8
[236222.967415] skb linear:   00000020: a0 06 01 bb 8b c6 53 91 be 5e 6e 60 bd e2 80 18
[236222.967426] skb linear:   00000030: 01 13 5c f6 00 00 01 01 08 0a 3d d6 6a a3 63 ea
[236222.967437] skb linear:   00000040: 5c d9 17 03 03 00 3f af 00 01 84 45 e2 36 e4 6a
[236222.967454] skb linear:   00000050: 3d 76 a8 7f d7 12 fa 72 4b d1 d0 74 0d c1 49 77
[236222.967466] skb linear:   00000060: 8b a4 bb 04 e5 aa 03 61 d3 e6 1f c9 0d 3e 46 c8
[236222.967477] skb linear:   00000070: cd 1f 7d ce e8 a7 84 84 01 5d 1f b4 ee 4f 27 63
[236222.967488] skb linear:   00000080: d2 a1 ab 1f 26 1d
Quoted 7 lines
Nothing in GRO stack will change gso_size, unless the packet is queued by GRO layer (after this, napi_reuse_skb() wont be called)
napi_reuse_skb() is only used when a packet has been aggregated to another, and at this point gso_size should be still 0.

Jan 19 07:55:22 rpi-3b-plus-rev1d3-abfb kernel: [ 1914.744157] skb len=54 headroom=5194 headlen=54 tailroom=10816
Jan 19 07:55:22 rpi-3b-plus-rev1d3-abfb kernel: [ 1914.744157] mac=(5194,14) net=(5208,20) trans=5228
Jan 19 07:55:22 rpi-3b-plus-rev1d3-abfb kernel: [ 1914.744157] shinfo(txflags=0 nr_frags=0 gso(size=1448 type=0 segs=1))
Jan 19 07:55:22 rpi-3b-plus-rev1d3-abfb kernel: [ 1914.744157] csum(0xe505 ip_summed=0 complete_sw=0 valid=0 level=0)
Jan 19 07:55:22 rpi-3b-plus-rev1d3-abfb kernel: [ 1914.744157] hash(0x0 sw=0 l4=0) proto=0x0800 pkttype=0 iif=2
Jan 19 07:55:22 rpi-3b-plus-rev1d3-abfb kernel: [ 1914.774147] dev name=eth0 feat=0x0x0000010000114b09
Jan 19 07:55:22 rpi-3b-plus-rev1d3-abfb kernel: [ 1914.779355] skb linear:   00000000: e0 28 6d 9e b9 22 b8 27 eb 3e ab fb 08 00 45 00
Jan 19 07:55:22 rpi-3b-plus-rev1d3-abfb kernel: [ 1914.787365] skb linear:   00000010: 00 28 00 00 40 00 3f 06 41 d0 c0 a8 63 84 02 14
Jan 19 07:55:22 rpi-3b-plus-rev1d3-abfb kernel: [ 1914.795266] skb linear:   00000020: d3 bf ed 3e 01 bb d4 0f 88 7e 00 00 00 00 50 04
Jan 19 07:55:22 rpi-3b-plus-rev1d3-abfb kernel: [ 1914.803168] skb linear:   00000030: 00 00 6a 58 00 00
Jan 19 07:55:22 rpi-3b-plus-rev1d3-abfb kernel: [ 1914.808384] ------------[ cut here ]------------
Jan 19 07:55:22 rpi-3b-plus-rev1d3-abfb kernel: [ 1914.813200] lan78xx: caps=(0x0000010000114b09, 0x0000000000000000)
Jan 19 07:55:22 rpi-3b-plus-rev1d3-abfb kernel: [ 1914.819717] WARNING: CPU: 0 PID: 0 at net/core/dev.c:3197 skb_warn_bad_offload+0x84/0x100
Jan 19 07:55:22 rpi-3b-plus-rev1d3-abfb kernel: [ 1914.828190] Modules linked in:
Jan 19 07:55:22 rpi-3b-plus-rev1d3-abfb kernel: [ 1914.831354] CPU: 0 PID: 0 Comm: swapper/0 Not tainted 5.11.0-rc4 #103
Jan 19 07:55:22 rpi-3b-plus-rev1d3-abfb kernel: [ 1914.838009] Hardware name: Raspberry Pi 3 Model B Plus Rev 1.3 (DT)
Jan 19 07:55:22 rpi-3b-plus-rev1d3-abfb kernel: [ 1914.844478] pstate: 60000005 (nZCv daif -PAN -UAO -TCO BTYPE=--)
Jan 19 07:55:22 rpi-3b-plus-rev1d3-abfb kernel: [ 1914.850685] pc : skb_warn_bad_offload+0x84/0x100
Jan 19 07:55:22 rpi-3b-plus-rev1d3-abfb kernel: [ 1914.855464] lr : skb_warn_bad_offload+0x84/0x100
Jan 19 07:55:22 rpi-3b-plus-rev1d3-abfb kernel: [ 1914.860242] sp : ffff800010003850
Jan 19 07:55:22 rpi-3b-plus-rev1d3-abfb kernel: [ 1914.863665] x29: ffff800010003850 x28: ffff7a96fb196290 
Jan 19 07:55:22 rpi-3b-plus-rev1d3-abfb kernel: [ 1914.869160] x27: ffff7a96c5958300 x26: 0000000000000001 
Jan 19 07:55:22 rpi-3b-plus-rev1d3-abfb kernel: [ 1914.874654] x25: ffffa73eee323000 x24: ffff7a96ee84b000 
Jan 19 07:55:22 rpi-3b-plus-rev1d3-abfb kernel: [ 1914.880148] x23: ffffa73eee7f4f00 x22: 0000000000000000 
Jan 19 07:55:22 rpi-3b-plus-rev1d3-abfb kernel: [ 1914.885642] x21: ffffa73eee0327e0 x20: ffff7a96ee84b000 
Jan 19 07:55:22 rpi-3b-plus-rev1d3-abfb kernel: [ 1914.891136] x19: ffff7a96c5958300 x18: 0000000000000010 
Jan 19 07:55:22 rpi-3b-plus-rev1d3-abfb kernel: [ 1914.896630] x17: 0000000000000000 x16: 0000000000000000 
Jan 19 07:55:22 rpi-3b-plus-rev1d3-abfb kernel: [ 1914.902123] x15: 000000000000ad55 x14: 0000000000000010 
Jan 19 07:55:22 rpi-3b-plus-rev1d3-abfb kernel: [ 1914.907617] x13: 00000000ffffffff x12: ffffa73eedd9d950 
Jan 19 07:55:22 rpi-3b-plus-rev1d3-abfb kernel: [ 1914.913109] x11: ffffa73eee885de0 x10: ffffa73eee86dda0 
Jan 19 07:55:22 rpi-3b-plus-rev1d3-abfb kernel: [ 1914.918603] x9 : ffffa73eecf2f45c x8 : 0000000000017fe8 
Jan 19 07:55:22 rpi-3b-plus-rev1d3-abfb kernel: [ 1914.924097] x7 : c0000000ffffefff x6 : 0000000000000003 
Jan 19 07:55:22 rpi-3b-plus-rev1d3-abfb kernel: [ 1914.929590] x5 : 0000000000000000 x4 : 0000000000000000 
Jan 19 07:55:22 rpi-3b-plus-rev1d3-abfb kernel: [ 1914.935081] x3 : 0000000000000100 x2 : 0000000000001000 
Jan 19 07:55:22 rpi-3b-plus-rev1d3-abfb kernel: [ 1914.940575] x1 : 0000000000000000 x0 : 0000000000000000 
Jan 19 07:55:22 rpi-3b-plus-rev1d3-abfb kernel: [ 1914.946070] Call trace:
Jan 19 07:55:22 rpi-3b-plus-rev1d3-abfb kernel: [ 1914.948599]  skb_warn_bad_offload+0x84/0x100
Jan 19 07:55:22 rpi-3b-plus-rev1d3-abfb kernel: [ 1914.953020]  netif_skb_features+0x218/0x2a0
Jan 19 07:55:22 rpi-3b-plus-rev1d3-abfb kernel: [ 1914.957350]  validate_xmit_skb.isra.0+0x28/0x2c8
Jan 19 07:55:22 rpi-3b-plus-rev1d3-abfb kernel: [ 1914.962125]  validate_xmit_skb_list+0x44/0x98
Jan 19 07:55:22 rpi-3b-plus-rev1d3-abfb kernel: [ 1914.966631]  sch_direct_xmit+0xf0/0x3a8
Jan 19 07:55:22 rpi-3b-plus-rev1d3-abfb kernel: [ 1914.970599]  __qdisc_run+0x140/0x668
Jan 19 07:55:22 rpi-3b-plus-rev1d3-abfb kernel: [ 1914.974297]  __dev_queue_xmit+0x59c/0x980
Jan 19 07:55:22 rpi-3b-plus-rev1d3-abfb kernel: [ 1914.978446]  dev_queue_xmit+0x1c/0x28
Jan 19 07:55:22 rpi-3b-plus-rev1d3-abfb kernel: [ 1914.982237]  ip_finish_output2+0x30c/0x558
Jan 19 07:55:22 rpi-3b-plus-rev1d3-abfb kernel: [ 1914.986476]  __ip_finish_output+0xe4/0x260
Jan 19 07:55:22 rpi-3b-plus-rev1d3-abfb kernel: [ 1914.990715]  ip_finish_output+0x3c/0xd8
Jan 19 07:55:22 rpi-3b-plus-rev1d3-abfb kernel: [ 1914.994683]  ip_output+0xb4/0x148
Jan 19 07:55:22 rpi-3b-plus-rev1d3-abfb kernel: [ 1914.998116]  ip_forward_finish+0x7c/0xc0
Jan 19 07:55:22 rpi-3b-plus-rev1d3-abfb kernel: [ 1915.002174]  ip_forward+0x42c/0x4f0
Jan 19 07:55:22 rpi-3b-plus-rev1d3-abfb kernel: [ 1915.005783]  ip_rcv_finish+0x98/0xb8
Jan 19 07:55:22 rpi-3b-plus-rev1d3-abfb kernel: [ 1915.009481]  ip_rcv+0xe0/0xf0
Jan 19 07:55:22 rpi-3b-plus-rev1d3-abfb kernel: [ 1915.012552]  __netif_receive_skb_one_core+0x5c/0x88
Jan 19 07:55:22 rpi-3b-plus-rev1d3-abfb kernel: [ 1915.017597]  __netif_receive_skb+0x20/0x70
Jan 19 07:55:22 rpi-3b-plus-rev1d3-abfb kernel: [ 1915.021834]  process_backlog+0xc0/0x1d0
Jan 19 07:55:22 rpi-3b-plus-rev1d3-abfb kernel: [ 1915.025802]  net_rx_action+0x134/0x478
Jan 19 07:55:22 rpi-3b-plus-rev1d3-abfb kernel: [ 1915.029682]  __do_softirq+0x130/0x378
Jan 19 07:55:22 rpi-3b-plus-rev1d3-abfb kernel: [ 1915.033472]  irq_exit+0xc0/0xe8
Jan 19 07:55:22 rpi-3b-plus-rev1d3-abfb kernel: [ 1915.036725]  __handle_domain_irq+0x70/0xc8
Jan 19 07:55:22 rpi-3b-plus-rev1d3-abfb kernel: [ 1915.040963]  bcm2836_arm_irqchip_handle_irq+0x6c/0x80
Jan 19 07:55:22 rpi-3b-plus-rev1d3-abfb kernel: [ 1915.046185]  el1_irq+0xb4/0x140
Jan 19 07:55:22 rpi-3b-plus-rev1d3-abfb kernel: [ 1915.053377]  arch_cpu_idle+0x18/0x28
Jan 19 07:55:22 rpi-3b-plus-rev1d3-abfb kernel: [ 1915.060981]  default_idle_call+0x44/0x178
Jan 19 07:55:22 rpi-3b-plus-rev1d3-abfb kernel: [ 1915.069009]  do_idle+0x224/0x270
Jan 19 07:55:22 rpi-3b-plus-rev1d3-abfb kernel: [ 1915.076147]  cpu_startup_entry+0x30/0x98
Jan 19 07:55:22 rpi-3b-plus-rev1d3-abfb kernel: [ 1915.083916]  rest_init+0xc8/0xd8
Jan 19 07:55:22 rpi-3b-plus-rev1d3-abfb kernel: [ 1915.090937]  arch_call_rest_init+0x18/0x24
Jan 19 07:55:22 rpi-3b-plus-rev1d3-abfb kernel: [ 1915.098829]  start_kernel+0x57c/0x5b8
Jan 19 07:55:22 rpi-3b-plus-rev1d3-abfb kernel: [ 1915.106251] ---[ end trace c3d8dd12ce1805e0 ]---

If I also add the following rule:

  $ iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE
I get a single warning followed by a TX timeout:

Jan 19 08:15:47 rpi-3b-plus-rev1d3-abfb kernel: [   81.516888] skb len=66 headroom=5194 headlen=66 tailroom=10804
Jan 19 08:15:47 rpi-3b-plus-rev1d3-abfb kernel: [   81.516888] mac=(5194,14) net=(5208,20) trans=5228
Jan 19 08:15:47 rpi-3b-plus-rev1d3-abfb kernel: [   81.516888] shinfo(txflags=0 nr_frags=0 gso(size=1448 type=0 segs=1))
Jan 19 08:15:47 rpi-3b-plus-rev1d3-abfb kernel: [   81.516888] csum(0xeedb ip_summed=1 complete_sw=0 valid=0 level=0)
Jan 19 08:15:47 rpi-3b-plus-rev1d3-abfb kernel: [   81.516888] hash(0x0 sw=0 l4=0) proto=0x0800 pkttype=0 iif=2
Jan 19 08:15:47 rpi-3b-plus-rev1d3-abfb kernel: [   81.546872] dev name=eth0 feat=0x0x0000010000114b09
Jan 19 08:15:47 rpi-3b-plus-rev1d3-abfb kernel: [   81.552060] skb linear:   00000000: e0 28 6d 9e b9 22 b8 27 eb 3e ab fb 08 00 45 00
Jan 19 08:15:47 rpi-3b-plus-rev1d3-abfb kernel: [   81.560090] skb linear:   00000010: 00 34 90 99 40 00 3f 06 87 40 c0 a8 63 84 22 6b
Jan 19 08:15:47 rpi-3b-plus-rev1d3-abfb kernel: [   81.568019] skb linear:   00000020: dd 52 d0 ac 00 50 35 e0 1e 2c 78 02 47 fa 80 10
Jan 19 08:15:47 rpi-3b-plus-rev1d3-abfb kernel: [   81.575921] skb linear:   00000030: 01 f6 d6 96 00 00 01 01 08 0a 50 c9 d7 4b cd 2e
Jan 19 08:15:47 rpi-3b-plus-rev1d3-abfb kernel: [   81.583918] skb linear:   00000040: 9f fc
Jan 19 08:15:47 rpi-3b-plus-rev1d3-abfb kernel: [   81.588105] ------------[ cut here ]------------
Jan 19 08:15:47 rpi-3b-plus-rev1d3-abfb kernel: [   81.592920] lan78xx: caps=(0x0000010000114b09, 0x0000000000000000)
Jan 19 08:15:47 rpi-3b-plus-rev1d3-abfb kernel: [   81.599429] WARNING: CPU: 0 PID: 0 at net/core/dev.c:3197 skb_warn_bad_offload+0x84/0x100
Jan 19 08:15:47 rpi-3b-plus-rev1d3-abfb kernel: [   81.607900] Modules linked in:
Jan 19 08:15:47 rpi-3b-plus-rev1d3-abfb kernel: [   81.611064] CPU: 0 PID: 0 Comm: swapper/0 Not tainted 5.11.0-rc4 #103
Jan 19 08:15:47 rpi-3b-plus-rev1d3-abfb kernel: [   81.617720] Hardware name: Raspberry Pi 3 Model B Plus Rev 1.3 (DT)
Jan 19 08:15:47 rpi-3b-plus-rev1d3-abfb kernel: [   81.624189] pstate: 60000005 (nZCv daif -PAN -UAO -TCO BTYPE=--)
Jan 19 08:15:47 rpi-3b-plus-rev1d3-abfb kernel: [   81.630396] pc : skb_warn_bad_offload+0x84/0x100
Jan 19 08:15:47 rpi-3b-plus-rev1d3-abfb kernel: [   81.635175] lr : skb_warn_bad_offload+0x84/0x100
Jan 19 08:15:47 rpi-3b-plus-rev1d3-abfb kernel: [   81.639953] sp : ffff800010003810
Jan 19 08:15:47 rpi-3b-plus-rev1d3-abfb kernel: [   81.643374] x29: ffff800010003810 x28: ffff50043b196290 
Jan 19 08:15:47 rpi-3b-plus-rev1d3-abfb kernel: [   81.648870] x27: ffff500407371600 x26: 0000000000000001 
Jan 19 08:15:47 rpi-3b-plus-rev1d3-abfb kernel: [   81.654365] x25: ffffa1fa11b23000 x24: ffff50042e96b000 
Jan 19 08:15:47 rpi-3b-plus-rev1d3-abfb kernel: [   81.659859] x23: ffffa1fa11ff4f00 x22: 0000000000000000 
Jan 19 08:15:47 rpi-3b-plus-rev1d3-abfb kernel: [   81.665353] x21: ffffa1fa118327e0 x20: ffff50042e96b000 
Jan 19 08:15:47 rpi-3b-plus-rev1d3-abfb kernel: [   81.670847] x19: ffff500407371600 x18: 0000000000000010 
Jan 19 08:15:47 rpi-3b-plus-rev1d3-abfb kernel: [   81.676340] x17: 0000000000000000 x16: 0000000000000000 
Jan 19 08:15:47 rpi-3b-plus-rev1d3-abfb kernel: [   81.681833] x15: 000000000000ad55 x14: 0000000000000010 
Jan 19 08:15:47 rpi-3b-plus-rev1d3-abfb kernel: [   81.687326] x13: 00000000ffffffff x12: ffffa1fa1159d950 
Jan 19 08:15:47 rpi-3b-plus-rev1d3-abfb kernel: [   81.692819] x11: ffffa1fa12085de0 x10: ffffa1fa1206dda0 
Jan 19 08:15:47 rpi-3b-plus-rev1d3-abfb kernel: [   81.698313] x9 : ffffa1fa1072f45c x8 : 0000000000017fe8 
Jan 19 08:15:47 rpi-3b-plus-rev1d3-abfb kernel: [   81.703806] x7 : c0000000ffffefff x6 : 0000000000000003 
Jan 19 08:15:47 rpi-3b-plus-rev1d3-abfb kernel: [   81.709300] x5 : 0000000000000000 x4 : 0000000000000000 
Jan 19 08:15:47 rpi-3b-plus-rev1d3-abfb kernel: [   81.714791] x3 : 0000000000000100 x2 : 0000000000001000 
Jan 19 08:15:47 rpi-3b-plus-rev1d3-abfb kernel: [   81.720283] x1 : 0000000000000000 x0 : 0000000000000000 
Jan 19 08:15:47 rpi-3b-plus-rev1d3-abfb kernel: [   81.725778] Call trace:
Jan 19 08:15:47 rpi-3b-plus-rev1d3-abfb kernel: [   81.728306]  skb_warn_bad_offload+0x84/0x100
Jan 19 08:15:47 rpi-3b-plus-rev1d3-abfb kernel: [   81.732728]  netif_skb_features+0x218/0x2a0
Jan 19 08:15:47 rpi-3b-plus-rev1d3-abfb kernel: [   81.737057]  validate_xmit_skb.isra.0+0x28/0x2c8
Jan 19 08:15:47 rpi-3b-plus-rev1d3-abfb kernel: [   81.741833]  validate_xmit_skb_list+0x44/0x98
Jan 19 08:15:47 rpi-3b-plus-rev1d3-abfb kernel: [   81.746339]  sch_direct_xmit+0xf0/0x3a8
Jan 19 08:15:47 rpi-3b-plus-rev1d3-abfb kernel: [   81.750309]  __qdisc_run+0x140/0x668
Jan 19 08:15:47 rpi-3b-plus-rev1d3-abfb kernel: [   81.754008]  __dev_queue_xmit+0x59c/0x980
Jan 19 08:15:47 rpi-3b-plus-rev1d3-abfb kernel: [   81.758156]  dev_queue_xmit+0x1c/0x28
Jan 19 08:15:47 rpi-3b-plus-rev1d3-abfb kernel: [   81.761945]  neigh_resolve_output+0x108/0x230
Jan 19 08:15:47 rpi-3b-plus-rev1d3-abfb kernel: [   81.766450]  ip_finish_output2+0x180/0x558
Jan 19 08:15:47 rpi-3b-plus-rev1d3-abfb kernel: [   81.770690]  __ip_finish_output+0xe4/0x260
Jan 19 08:15:47 rpi-3b-plus-rev1d3-abfb kernel: [   81.774928]  ip_finish_output+0x3c/0xd8
Jan 19 08:15:47 rpi-3b-plus-rev1d3-abfb kernel: [   81.778896]  ip_output+0xb4/0x148
Jan 19 08:15:47 rpi-3b-plus-rev1d3-abfb kernel: [   81.782328]  ip_forward_finish+0x7c/0xc0
Jan 19 08:15:47 rpi-3b-plus-rev1d3-abfb kernel: [   81.786385]  ip_forward+0x42c/0x4f0
Jan 19 08:15:47 rpi-3b-plus-rev1d3-abfb kernel: [   81.789995]  ip_rcv_finish+0x98/0xb8
Jan 19 08:15:47 rpi-3b-plus-rev1d3-abfb kernel: [   81.793694]  ip_rcv+0xe0/0xf0
Jan 19 08:15:47 rpi-3b-plus-rev1d3-abfb kernel: [   81.796765]  __netif_receive_skb_one_core+0x5c/0x88
Jan 19 08:15:47 rpi-3b-plus-rev1d3-abfb kernel: [   81.801810]  __netif_receive_skb+0x20/0x70
Jan 19 08:15:47 rpi-3b-plus-rev1d3-abfb kernel: [   81.806047]  process_backlog+0xc0/0x1d0
Jan 19 08:15:47 rpi-3b-plus-rev1d3-abfb kernel: [   81.810016]  net_rx_action+0x134/0x478
Jan 19 08:15:47 rpi-3b-plus-rev1d3-abfb kernel: [   81.813897]  __do_softirq+0x130/0x378
Jan 19 08:15:47 rpi-3b-plus-rev1d3-abfb kernel: [   81.817686]  irq_exit+0xc0/0xe8
Jan 19 08:15:47 rpi-3b-plus-rev1d3-abfb kernel: [   81.820940]  __handle_domain_irq+0x70/0xc8
Jan 19 08:15:47 rpi-3b-plus-rev1d3-abfb kernel: [   81.829099]  bcm2836_arm_irqchip_handle_irq+0x6c/0x80
Jan 19 08:15:47 rpi-3b-plus-rev1d3-abfb kernel: [   81.838223]  el1_irq+0xb4/0x140
Jan 19 08:15:47 rpi-3b-plus-rev1d3-abfb kernel: [   81.845371]  arch_cpu_idle+0x18/0x28
Jan 19 08:15:47 rpi-3b-plus-rev1d3-abfb kernel: [   81.852882]  default_idle_call+0x44/0x178
Jan 19 08:15:47 rpi-3b-plus-rev1d3-abfb kernel: [   81.860756]  do_idle+0x224/0x270
Jan 19 08:15:47 rpi-3b-plus-rev1d3-abfb kernel: [   81.867794]  cpu_startup_entry+0x30/0x98
Jan 19 08:15:47 rpi-3b-plus-rev1d3-abfb kernel: [   81.875516]  rest_init+0xc8/0xd8
Jan 19 08:15:47 rpi-3b-plus-rev1d3-abfb kernel: [   81.882496]  arch_call_rest_init+0x18/0x24
Jan 19 08:15:47 rpi-3b-plus-rev1d3-abfb kernel: [   81.890352]  start_kernel+0x57c/0x5b8
Jan 19 08:15:47 rpi-3b-plus-rev1d3-abfb kernel: [   81.897706] ---[ end trace a5789410f231a10b ]---
Jan 19 08:16:15 rpi-3b-plus-rev1d3-abfb kernel: [  110.046337] ------------[ cut here ]------------
Jan 19 08:16:15 rpi-3b-plus-rev1d3-abfb kernel: [  110.054787] NETDEV WATCHDOG: eth0 (lan78xx): transmit queue 0 timed out
Jan 19 08:16:15 rpi-3b-plus-rev1d3-abfb kernel: [  110.065356] WARNING: CPU: 2 PID: 0 at net/sched/sch_generic.c:442 dev_watchdog+0x384/0x390
Jan 19 08:16:15 rpi-3b-plus-rev1d3-abfb kernel: [  110.077534] Modules linked in:
Jan 19 08:16:15 rpi-3b-plus-rev1d3-abfb kernel: [  110.084361] CPU: 2 PID: 0 Comm: swapper/2 Tainted: G        W         5.11.0-rc4 #103
Jan 19 08:16:15 rpi-3b-plus-rev1d3-abfb kernel: [  110.096114] Hardware name: Raspberry Pi 3 Model B Plus Rev 1.3 (DT)
Jan 19 08:16:15 rpi-3b-plus-rev1d3-abfb kernel: [  110.106246] pstate: 60000005 (nZCv daif -PAN -UAO -TCO BTYPE=--)
Jan 19 08:16:15 rpi-3b-plus-rev1d3-abfb kernel: [  110.116085] pc : dev_watchdog+0x384/0x390
Jan 19 08:16:15 rpi-3b-plus-rev1d3-abfb kernel: [  110.123857] lr : dev_watchdog+0x384/0x390
Jan 19 08:16:15 rpi-3b-plus-rev1d3-abfb kernel: [  110.131558] sp : ffff800010013d90
Jan 19 08:16:15 rpi-3b-plus-rev1d3-abfb kernel: [  110.138497] x29: ffff800010013d90 x28: 0000000000000140 
Jan 19 08:16:15 rpi-3b-plus-rev1d3-abfb kernel: [  110.147472] x27: 00000000ffffffff x26: ffffa1fa11b23000 
Jan 19 08:16:15 rpi-3b-plus-rev1d3-abfb kernel: [  110.156489] x25: 0000000000000002 x24: 0000000000000000 
Jan 19 08:16:15 rpi-3b-plus-rev1d3-abfb kernel: [  110.165496] x23: 0000000000000001 x22: ffff50042e96b000 
Jan 19 08:16:15 rpi-3b-plus-rev1d3-abfb kernel: [  110.174494] x21: ffff50042e96b440 x20: ffffa1fa11fe7000 
Jan 19 08:16:15 rpi-3b-plus-rev1d3-abfb kernel: [  110.183490] x19: 0000000000000000 x18: 0000000000000010 
Jan 19 08:16:15 rpi-3b-plus-rev1d3-abfb kernel: [  110.192493] x17: 0000000000000000 x16: 0000000000000000 
Jan 19 08:16:15 rpi-3b-plus-rev1d3-abfb kernel: [  110.201473] x15: 000000000000ad55 x14: 0000000000000010 
Jan 19 08:16:15 rpi-3b-plus-rev1d3-abfb kernel: [  110.210439] x13: 00000000ffffffff x12: ffffa1fa1159d950 
Jan 19 08:16:15 rpi-3b-plus-rev1d3-abfb kernel: [  110.219397] x11: ffffa1fa12085de0 x10: ffffa1fa1206dda0 
Jan 19 08:16:15 rpi-3b-plus-rev1d3-abfb kernel: [  110.228367] x9 : ffffa1fa1072f45c x8 : 0000000000017fe8 
Jan 19 08:16:15 rpi-3b-plus-rev1d3-abfb kernel: [  110.237362] x7 : c0000000ffffefff x6 : 0000000000000003 
Jan 19 08:16:15 rpi-3b-plus-rev1d3-abfb kernel: [  110.246353] x5 : 0000000000000000 x4 : 0000000000000000 
Jan 19 08:16:15 rpi-3b-plus-rev1d3-abfb kernel: [  110.255328] x3 : 0000000000000100 x2 : 0000000000001000 
Jan 19 08:16:15 rpi-3b-plus-rev1d3-abfb kernel: [  110.264273] x1 : 0000000000000000 x0 : 0000000000000000 
Jan 19 08:16:15 rpi-3b-plus-rev1d3-abfb kernel: [  110.273192] Call trace:
Jan 19 08:16:15 rpi-3b-plus-rev1d3-abfb kernel: [  110.279183]  dev_watchdog+0x384/0x390
Jan 19 08:16:15 rpi-3b-plus-rev1d3-abfb kernel: [  110.286461]  call_timer_fn+0x38/0x188
Jan 19 08:16:15 rpi-3b-plus-rev1d3-abfb kernel: [  110.293762]  run_timer_softirq+0x494/0x688
Jan 19 08:16:15 rpi-3b-plus-rev1d3-abfb kernel: [  110.301489]  __do_softirq+0x130/0x378
Jan 19 08:16:15 rpi-3b-plus-rev1d3-abfb kernel: [  110.308767]  irq_exit+0xc0/0xe8
Jan 19 08:16:15 rpi-3b-plus-rev1d3-abfb kernel: [  110.315500]  __handle_domain_irq+0x70/0xc8
Jan 19 08:16:15 rpi-3b-plus-rev1d3-abfb kernel: [  110.323214]  bcm2836_arm_irqchip_handle_irq+0x6c/0x80
Jan 19 08:16:15 rpi-3b-plus-rev1d3-abfb kernel: [  110.331940]  el1_irq+0xb4/0x140
Jan 19 08:16:15 rpi-3b-plus-rev1d3-abfb kernel: [  110.338706]  arch_cpu_idle+0x18/0x28
Jan 19 08:16:15 rpi-3b-plus-rev1d3-abfb kernel: [  110.345916]  default_idle_call+0x44/0x178
Jan 19 08:16:15 rpi-3b-plus-rev1d3-abfb kernel: [  110.353577]  do_idle+0x224/0x270
Jan 19 08:16:15 rpi-3b-plus-rev1d3-abfb kernel: [  110.360433]  cpu_startup_entry+0x2c/0x98
Jan 19 08:16:15 rpi-3b-plus-rev1d3-abfb kernel: [  110.368000]  secondary_start_kernel+0x148/0x180
Jan 19 08:16:15 rpi-3b-plus-rev1d3-abfb kernel: [  110.376199] ---[ end trace a5789410f231a10c ]---

Enabling KASAN didn't trigger any error reports.

Let me know if there's anything else I can try to narrow this down.

...Juerg

[1]
On the Pi, I run:

  $ nc -l 1234 | dd status=progress >/dev/null

And on another machine, that is configured to use the Pi as the gateway:

  $ nc 192.168.99.115 1234 < /dev/urandom
and a couple of firefox instances that keep opening public URls.

[2]

Author: Eric Dumazet <[email protected]>
Date:   Tue Nov 27 14:42:03 2018 -0800

tcp: implement coalescing on backlog queue

In case GRO is not as efficient as it should be or disabled, we might have a user thread trapped in __release_sock() while softirq handler flood packets up to the point we have to drop.

This patch balances work done from user thread and softirq, to give more chances to __release_sock() to complete its work before new packets are added the the backlog.

This also helps if we receive many ACK packets, since GRO does not aggregate them.

This patch brings ~60% throughput increase on a receiver without GRO, but the spectacular gain is really on 1000x release_sock() latency reduction I have measured.

Quoted 9 lines

Signed-off-by Eric Dumazet <[email protected]>
Cc Neal Cardwell <[email protected]>
Cc Yuchung Cheng <[email protected]>
Acked-by Neal Cardwell <[email protected]>
Signed-off-by David S. Miller <[email protected]>
[3] 86bccd036713 tcp: fix receive window update in tcp_add_backlog()
[4] ca2fe2956ace tcp: add sanity tests in tcp_add_backlog()

In tcp_add_backlog() we have the following that looks like it could be related to the problem. gso_type doesn't get set, not sure however whether this is a bug or intentional (because we expect gso_type to be set already or because it's supposed to be set somewhere else). Meybe Eric can comment on this.

	if (!shinfo->gso_size)
		shinfo->gso_size = skb->len - hdrlen;

	if (!shinfo->gso_segs)
		shinfo->gso_segs = 1;

Eric Dumazet

2021-01-19 23:52 UTC #19 lore raw

Quoted 9 lines · Heiner Kallweit

In tcp_add_backlog() we have the following that looks like it could be related to the problem. gso_type doesn't get set, not sure however whether this is a bug or intentional (because we expect gso_type to be set already or because it's supposed to be set somewhere else). Meybe Eric can comment on this.

Quoted 6 lines

        if (!shinfo->gso_size)
                shinfo->gso_size = skb->len - hdrlen;

        if (!shinfo->gso_segs)
                shinfo->gso_segs = 1;

Yes, at this point TCP is supposed to own the skb, which is partially true.

Check for skb_cloned() in places like skb_try_coalesce()

I think that calling skb_unclone() would be terribly expensive for all these USB drivers having fake skb
(all clones from a giant one), and thus very big headroom that would be copied from generic expand head.