Skip to contentSkip to conversation

tainted warnings with tcp splicing in 3.7.1

PATCH Applied Christian Becker 6 participants 17 messages Jan 9, 2013 – Jan 12, 2013

Subsystems (3)
Networking Network drivers TCP
lore
Share & tools
lore

Christian Becker

author
#0 lore raw

Hi,

we´ve installed 3.7.1 yesterday on one of our loadbalancer nodes in order to get TFO.

Unfortunately the kernel started to print warnings every couple of minutes when using tcp splicing in haproxy.

We´ve built the kernel yesterday from the 3.7.1 sources without any modifications.

The System contains two Intel Xeon X6550, 64 GB RAM and there are two kinds of NICs:

Broadcom Corporation NetXtreme II BCM5709 (Driver bnx2) Emulex Corporation OneConnect 10Gb NIC (Driver be2net)

The bnx2 adapters are not in use and disconnected and the be2net handle about 1 GBit/s of traffic.

We´ve downgraded again to our old kernel version, but i guess you should take a look at this.

There are two kinds of messages:

an 9 11:34:28 srv11 kernel: [ 1081.334970] ------------[ cut here ]------------

Jan  9 11:34:28 srv11 kernel: [ 1081.353685] WARNING: at net/ipv4/tcp.c:1330 tcp_cleanup_rbuf+0x4d/0xfc()
Jan  9 11:34:28 srv11 kernel: [ 1081.371956] Hardware name: System x3690 X5 -[7148Z68]-
Jan  9 11:34:28 srv11 kernel: [ 1081.391820] cleanup rbuf bug: copied AD3BCF1 seq AD370AF rcvnxt AD3CF13
Jan  9 11:34:28 srv11 kernel: [ 1081.408971] Modules linked in: 8021q garp stp llc nls_utf8 nls_cp437 vfat fat kvm_intel coretemp kvm joydev acpi_cpufreq mperf crc32c_intel hid_generic shpchp snd_pcm snd_timer snd lpc_ich mfd_core cdc_ether usb
net mii evdev serio_raw ioatdma processor i2c_i801 tpm_tis dca soundcore snd_page_alloc pcspkr tpm microcode i2c_core tpm_bios thermal_sys button pci_hotplug ext4 mbcache jbd2 crc16 dm_mod sg sr_mod cdrom sd_mod crc_t10dif usbhid ata_generic hi
d uhci_hcd ata_piix libata megaraid_sas bnx2 ehci_hcd usbcore scsi_mod usb_common be2net
Jan  9 11:34:28 srv11 kernel: [ 1081.532314] Pid: 13763, comm: haproxy Not tainted 3.7.1 #1
Jan  9 11:34:28 srv11 kernel: [ 1081.554349] Call Trace:
Jan  9 11:34:28 srv11 kernel: [ 1081.573762]  [<ffffffff8103ef70>] ? warn_slowpath_common+0x78/0x8c
Jan  9 11:34:28 srv11 kernel: [ 1081.596750]  [<ffffffff8103f023>] ? warn_slowpath_fmt+0x45/0x4a
Jan  9 11:34:28 srv11 kernel: [ 1081.617853]  [<ffffffff81297d06>] ? sock_pipe_buf_release+0xe/0xe
Jan  9 11:34:28 srv11 kernel: [ 1081.639111]  [<ffffffff812d37c2>] ? tcp_cleanup_rbuf+0x4d/0xfc
Jan  9 11:34:28 srv11 kernel: [ 1081.661541]  [<ffffffff812d39f4>] ? tcp_read_sock+0x183/0x194
Jan  9 11:34:28 srv11 kernel: [ 1081.681164]  [<ffffffff812d423d>] ? tcp_sendpage+0x45b/0x45b
Jan  9 11:34:28 srv11 kernel: [ 1081.703355]  [<ffffffff812d3ad8>] ? tcp_splice_read+0xd3/0x223
Jan  9 11:34:28 srv11 kernel: [ 1081.724912]  [<ffffffff8112d9b9>] ? sys_splice+0x345/0x3c0
Jan  9 11:34:28 srv11 kernel: [ 1081.744525]  [<ffffffff81364b69>] ? system_call_fastpath+0x16/0x1b
Jan  9 11:34:28 srv11 kernel: [ 1081.766683] ---[ end trace fb4ffd749d51e56f ]---
Jan  9 11:52:42 srv11 kernel: [ 2174.882971] WARNING: at net/ipv4/tcp.c:1639 tcp_recvmsg+0x2ca/0x9de()
Jan  9 11:52:42 srv11 kernel: [ 2174.901182] Hardware name: System x3690 X5 -[7148Z68]-
Jan  9 11:52:42 srv11 kernel: [ 2174.921229] recvmsg bug 2: copied 4B9C4CE6 seq 4B9BE950 rcvnxt 4B9C4CE6 fl 0
Jan  9 11:52:42 srv11 kernel: [ 2174.941681] Modules linked in: 8021q garp stp llc nls_utf8 nls_cp437 vfat fat kvm_intel coretemp kvm joydev acpi_cpufreq mperf crc32c_intel hid_generic shpchp snd_pcm snd_timer snd lpc_ich mfd_core cdc_ether usb
net mii evdev serio_raw ioatdma processor i2c_i801 tpm_tis dca soundcore snd_page_alloc pcspkr tpm microcode i2c_core tpm_bios thermal_sys button pci_hotplug ext4 mbcache jbd2 crc16 dm_mod sg sr_mod cdrom sd_mod crc_t10dif usbhid ata_generic hi
d uhci_hcd ata_piix libata megaraid_sas bnx2 ehci_hcd usbcore scsi_mod usb_common be2net
Jan  9 11:52:42 srv11 kernel: [ 2175.075389] Pid: 13250, comm: haproxy Tainted: G        W    3.7.1 #1
Jan  9 11:52:42 srv11 kernel: [ 2175.099391] Call Trace:
Jan  9 11:52:42 srv11 kernel: [ 2175.122727]  [<ffffffff8103ef70>] ? warn_slowpath_common+0x78/0x8c
Jan  9 11:52:42 srv11 kernel: [ 2175.147250]  [<ffffffff8103f023>] ? warn_slowpath_fmt+0x45/0x4a
Jan  9 11:52:42 srv11 kernel: [ 2175.170289]  [<ffffffff81295f64>] ? release_sock+0xe6/0x11c
Jan  9 11:52:43 srv11 kernel: [ 2175.192639]  [<ffffffff812d45d2>] ? tcp_recvmsg+0x2ca/0x9de
Jan  9 11:52:43 srv11 kernel: [ 2175.215020]  [<ffffffff812e0b71>] ? tcp_write_xmit+0x849/0x946
Jan  9 11:52:43 srv11 kernel: [ 2175.237682]  [<ffffffff812f2720>] ? inet_recvmsg+0x64/0x75
Jan  9 11:52:43 srv11 kernel: [ 2175.259727]  [<ffffffff8129052d>] ? sock_recvmsg+0x56/0x6e
Jan  9 11:52:43 srv11 kernel: [ 2175.281111]  [<ffffffff8128fedf>] ? sockfd_lookup_light+0x1a/0x50
Jan  9 11:52:43 srv11 kernel: [ 2175.300573]  [<ffffffff812923f4>] ? sys_recvfrom+0xbf/0x120
Jan  9 11:52:43 srv11 kernel: [ 2175.320073]  [<ffffffff8135d9f7>] ? __schedule+0x4c9/0x4f6
Jan  9 11:52:43 srv11 kernel: [ 2175.341151]  [<ffffffff81364b69>] ? system_call_fastpath+0x16/0x1b
Jan  9 11:52:43 srv11 kernel: [ 2175.360859] ---[ end trace fb4ffd749d51e5a7 ]---

As you can see here, the messages are appearing every couple of minutes:

Jan  9 11:34:28 srv11 kernel: [ 1081.353685] WARNING: at net/ipv4/tcp.c:1330 tcp_cleanup_rbuf+0x4d/0xfc()
Jan  9 11:34:29 srv11 kernel: [ 1081.809446] WARNING: at net/ipv4/tcp.c:1330 tcp_cleanup_rbuf+0x4d/0xfc()
Jan  9 11:34:29 srv11 kernel: [ 1082.235052] WARNING: at net/ipv4/tcp.c:1639 tcp_recvmsg+0x2ca/0x9de()
Jan  9 11:34:29 srv11 kernel: [ 1082.692732] WARNING: at net/ipv4/tcp.c:1639 tcp_recvmsg+0x2ca/0x9de()
Jan  9 11:34:30 srv11 kernel: [ 1083.177807] WARNING: at net/ipv4/tcp.c:1639 tcp_recvmsg+0x2ca/0x9de()
Jan  9 11:34:30 srv11 kernel: [ 1083.698475] WARNING: at net/ipv4/tcp.c:1639 tcp_recvmsg+0x2ca/0x9de()
Jan  9 11:34:31 srv11 kernel: [ 1084.221899] WARNING: at net/ipv4/tcp.c:1639 tcp_recvmsg+0x2ca/0x9de()
Jan  9 11:34:31 srv11 kernel: [ 1084.746992] WARNING: at net/ipv4/tcp.c:1639 tcp_recvmsg+0x2ca/0x9de()
Jan  9 11:34:32 srv11 kernel: [ 1085.267199] WARNING: at net/ipv4/tcp.c:1330 tcp_cleanup_rbuf+0x4d/0xfc()
Jan  9 11:37:16 srv11 kernel: [ 1249.252200] WARNING: at net/ipv4/tcp.c:1330 tcp_cleanup_rbuf+0x4d/0xfc()
Jan  9 11:37:17 srv11 kernel: [ 1249.782915] WARNING: at net/ipv4/tcp.c:1330 tcp_cleanup_rbuf+0x4d/0xfc()
Jan  9 11:37:17 srv11 kernel: [ 1250.315653] WARNING: at net/ipv4/tcp.c:1330 tcp_cleanup_rbuf+0x4d/0xfc()
Jan  9 11:37:18 srv11 kernel: [ 1250.867306] WARNING: at net/ipv4/tcp.c:1330 tcp_cleanup_rbuf+0x4d/0xfc()
Jan  9 11:37:18 srv11 kernel: [ 1251.383968] WARNING: at net/ipv4/tcp.c:1330 tcp_cleanup_rbuf+0x4d/0xfc()
Jan  9 11:37:19 srv11 kernel: [ 1251.897631] WARNING: at net/ipv4/tcp.c:1330 tcp_cleanup_rbuf+0x4d/0xfc()
Jan  9 11:37:19 srv11 kernel: [ 1252.412535] WARNING: at net/ipv4/tcp.c:1330 tcp_cleanup_rbuf+0x4d/0xfc()
Jan  9 11:37:20 srv11 kernel: [ 1252.921313] WARNING: at net/ipv4/tcp.c:1330 tcp_cleanup_rbuf+0x4d/0xfc()
Jan  9 11:40:13 srv11 kernel: [ 1425.644620] WARNING: at net/ipv4/tcp.c:1330 tcp_cleanup_rbuf+0x4d/0xfc()
Jan  9 11:40:13 srv11 kernel: [ 1426.314292] WARNING: at net/ipv4/tcp.c:1330 tcp_cleanup_rbuf+0x4d/0xfc()
Jan  9 11:40:14 srv11 kernel: [ 1426.749374] WARNING: at net/ipv4/tcp.c:1330 tcp_cleanup_rbuf+0x4d/0xfc()
Jan  9 11:40:14 srv11 kernel: [ 1427.198672] WARNING: at net/ipv4/tcp.c:1330 tcp_cleanup_rbuf+0x4d/0xfc()
Jan  9 11:40:15 srv11 kernel: [ 1427.711731] WARNING: at net/ipv4/tcp.c:1330 tcp_cleanup_rbuf+0x4d/0xfc()
Jan  9 11:40:15 srv11 kernel: [ 1428.189284] WARNING: at net/ipv4/tcp.c:1639 tcp_recvmsg+0x2ca/0x9de()
Jan  9 11:40:16 srv11 kernel: [ 1428.701230] WARNING: at net/ipv4/tcp.c:1330 tcp_cleanup_rbuf+0x4d/0xfc()
Jan  9 11:46:09 srv11 kernel: [ 1781.780019] WARNING: at net/ipv4/tcp.c:1330 tcp_cleanup_rbuf+0x4d/0xfc()
Jan  9 11:46:09 srv11 kernel: [ 1782.350094] WARNING: at net/ipv4/tcp.c:1330 tcp_cleanup_rbuf+0x4d/0xfc()
Jan  9 11:46:10 srv11 kernel: [ 1782.853933] WARNING: at net/ipv4/tcp.c:1330 tcp_cleanup_rbuf+0x4d/0xfc()
Jan  9 11:46:10 srv11 kernel: [ 1783.303487] WARNING: at net/ipv4/tcp.c:1330 tcp_cleanup_rbuf+0x4d/0xfc()
Jan  9 11:46:11 srv11 kernel: [ 1783.757022] WARNING: at net/ipv4/tcp.c:1330 tcp_cleanup_rbuf+0x4d/0xfc()
Jan  9 11:46:11 srv11 kernel: [ 1784.260932] WARNING: at net/ipv4/tcp.c:1639 tcp_recvmsg+0x2ca/0x9de()
Jan  9 11:46:12 srv11 kernel: [ 1784.874611] WARNING: at net/ipv4/tcp.c:1639 tcp_recvmsg+0x2ca/0x9de()
Jan  9 11:46:13 srv11 kernel: [ 1785.457209] WARNING: at net/ipv4/tcp.c:1639 tcp_recvmsg+0x2ca/0x9de()
Jan  9 11:46:13 srv11 kernel: [ 1785.933210] WARNING: at net/ipv4/tcp.c:1639 tcp_recvmsg+0x2ca/0x9de()
Jan  9 11:46:14 srv11 kernel: [ 1786.400550] WARNING: at net/ipv4/tcp.c:1330 tcp_cleanup_rbuf+0x4d/0xfc()
Jan  9 11:46:14 srv11 kernel: [ 1786.941620] WARNING: at net/ipv4/tcp.c:1639 tcp_recvmsg+0x2ca/0x9de()
Jan  9 11:46:15 srv11 kernel: [ 1787.510305] WARNING: at net/ipv4/tcp.c:1639 tcp_recvmsg+0x2ca/0x9de()
Jan  9 11:46:15 srv11 kernel: [ 1788.077590] WARNING: at net/ipv4/tcp.c:1330 tcp_cleanup_rbuf+0x4d/0xfc()
Jan  9 11:46:54 srv11 kernel: [ 1827.316572] WARNING: at net/ipv4/tcp.c:1330 tcp_cleanup_rbuf+0x4d/0xfc()
Jan  9 11:46:55 srv11 kernel: [ 1827.721460] WARNING: at net/ipv4/tcp.c:1330 tcp_cleanup_rbuf+0x4d/0xfc()
Jan  9 11:46:55 srv11 kernel: [ 1828.150593] WARNING: at net/ipv4/tcp.c:1330 tcp_cleanup_rbuf+0x4d/0xfc()
Jan  9 11:46:56 srv11 kernel: [ 1828.620180] WARNING: at net/ipv4/tcp.c:1330 tcp_cleanup_rbuf+0x4d/0xfc()
Jan  9 11:46:56 srv11 kernel: [ 1829.120693] WARNING: at net/ipv4/tcp.c:1330 tcp_cleanup_rbuf+0x4d/0xfc()
Jan  9 11:46:57 srv11 kernel: [ 1829.597494] WARNING: at net/ipv4/tcp.c:1639 tcp_recvmsg+0x2ca/0x9de()
Jan  9 11:46:57 srv11 kernel: [ 1830.189551] WARNING: at net/ipv4/tcp.c:1639 tcp_recvmsg+0x2ca/0x9de()
Jan  9 11:46:58 srv11 kernel: [ 1830.745157] WARNING: at net/ipv4/tcp.c:1639 tcp_recvmsg+0x2ca/0x9de()
Jan  9 11:46:58 srv11 kernel: [ 1831.299635] WARNING: at net/ipv4/tcp.c:1639 tcp_recvmsg+0x2ca/0x9de()
Jan  9 11:46:59 srv11 kernel: [ 1831.852929] WARNING: at net/ipv4/tcp.c:1639 tcp_recvmsg+0x2ca/0x9de()
Jan  9 11:47:00 srv11 kernel: [ 1832.408950] WARNING: at net/ipv4/tcp.c:1330 tcp_cleanup_rbuf+0x4d/0xfc()
Jan  9 11:52:38 srv11 kernel: [ 2170.882305] WARNING: at net/ipv4/tcp.c:1330 tcp_cleanup_rbuf+0x4d/0xfc()
Jan  9 11:52:39 srv11 kernel: [ 2171.342827] WARNING: at net/ipv4/tcp.c:1639 tcp_recvmsg+0x2ca/0x9de()
Jan  9 11:52:39 srv11 kernel: [ 2171.853599] WARNING: at net/ipv4/tcp.c:1639 tcp_recvmsg+0x2ca/0x9de()
Jan  9 11:52:40 srv11 kernel: [ 2172.355757] WARNING: at net/ipv4/tcp.c:1639 tcp_recvmsg+0x2ca/0x9de()
Jan  9 11:52:40 srv11 kernel: [ 2172.856194] WARNING: at net/ipv4/tcp.c:1639 tcp_recvmsg+0x2ca/0x9de()
Jan  9 11:52:41 srv11 kernel: [ 2173.357738] WARNING: at net/ipv4/tcp.c:1639 tcp_recvmsg+0x2ca/0x9de()
Jan  9 11:52:41 srv11 kernel: [ 2173.857774] WARNING: at net/ipv4/tcp.c:1639 tcp_recvmsg+0x2ca/0x9de()
Jan  9 11:52:42 srv11 kernel: [ 2174.361131] WARNING: at net/ipv4/tcp.c:1330 tcp_cleanup_rbuf+0x4d/0xfc()
Jan  9 11:52:42 srv11 kernel: [ 2174.882971] WARNING: at net/ipv4/tcp.c:1639 tcp_recvmsg+0x2ca/0x9de()
Jan  9 11:52:43 srv11 kernel: [ 2175.397627] WARNING: at net/ipv4/tcp.c:1639 tcp_recvmsg+0x2ca/0x9de()
Jan  9 11:52:43 srv11 kernel: [ 2175.911679] WARNING: at net/ipv4/tcp.c:1639 tcp_recvmsg+0x2ca/0x9de()
Jan  9 11:52:44 srv11 kernel: [ 2176.424397] WARNING: at net/ipv4/tcp.c:1639 tcp_recvmsg+0x2ca/0x9de()
Jan  9 11:52:44 srv11 kernel: [ 2176.940545] WARNING: at net/ipv4/tcp.c:1639 tcp_recvmsg+0x2ca/0x9de()
Jan  9 11:52:45 srv11 kernel: [ 2177.456173] WARNING: at net/ipv4/tcp.c:1639 tcp_recvmsg+0x2ca/0x9de()
Jan  9 11:52:45 srv11 kernel: [ 2177.971222] WARNING: at net/ipv4/tcp.c:1330 tcp_cleanup_rbuf+0x4d/0xfc()
#1 lore raw

For the record:
Another user reported similar warnings in a 3.5.0 kernel back in September [1] on the haproxy mailing list with the following kernel warnings:

[142654.793193] ------------[ cut here ]------------
[142654.793395] WARNING: at net/ipv4/tcp.c:1301 tcp_cleanup_rbuf+0x54/0x150()
[142654.793972] Hardware name: System Product Name
[142654.794573] cleanup rbuf bug: copied 68D1EF11 seq 68CFF65F rcvnxt 68D3565D
[142654.795165] Modules linked in: ixgbe(O) binfmt_misc 8021q fcoe

garp stp llc libfcoe libfc scsi_transport_fc scsi_tgt ip6t_REJECT nf _conntrack_ipv6 nf_defrag_ipv6 xt_state nf_conntrack ip6table_filter ip6_tables snd_hda_codec_hdmi snd_hda_codec_realtek snd_hda_intel snd_hda_codec nouveau snd_hwdep igb snd_seq snd_seq_device snd_pcm eeepc_wmi asus_wmi ttm drm_kms_helper sparse_keymap snd_timer snd drm coretemp rfkill i2c_algo_bit mxm_wmi wmi video lpc_ich mfd_core crc32c_intel r8169 i2c_i801 i2c_core soundcore snd_page_alloc mii mdio pcspkr serio_raw ghash_clmulni_intel microcode uinput [last unloaded:
ixgbe]

[142654.798215] Pid: 18374, comm: haproxy Tainted: G        W  O 3.5.0 #1
[142654.798838] Call Trace:
[142654.799440]  [<ffffffff810422cf>] warn_slowpath_common+0x7f/0xc0
[142654.800031]  [<ffffffff810423c6>] warn_slowpath_fmt+0x46/0x50
[142654.800653]  [<ffffffff8147c270>] ? sock_pipe_buf_release+0x20/0x20
[142654.801237]  [<ffffffff814cf294>] tcp_cleanup_rbuf+0x54/0x150
[142654.801847]  [<ffffffff814d0ae1>] tcp_read_sock+0x1b1/0x200
[142654.802440]  [<ffffffff81472777>] ? sock_sendpage+0x27/0x30
[142654.803037]  [<ffffffff814ccd60>] ? tcp_done+0x90/0x90
[142654.803644]  [<ffffffff814d0bf0>] tcp_splice_read+0xc0/0x250
[142654.804239]  [<ffffffff814726b2>] sock_splice_read+0x62/0x80
[142654.804843]  [<ffffffff8118c73b>] do_splice_to+0x7b/0xa0
[142654.805457]  [<ffffffff8118e850>] sys_splice+0x540/0x560
[142654.806040]  [<ffffffff8159aed2>] system_call_fastpath+0x16/0x1b
[142654.806646] ---[ end trace 46d7fb693af33fde ]---

[1] http://permalink.gmane.org/gmane.comp.web.haproxy/9559 -

#2 lore raw
In reply to Christian Becker
Quoted 117 lines · Christian Becker

Hi,

we´ve installed 3.7.1 yesterday on one of our loadbalancer nodes in order to get TFO.

Unfortunately the kernel started to print warnings every couple of minutes when using tcp splicing in haproxy.

We´ve built the kernel yesterday from the 3.7.1 sources without any modifications.

The System contains two Intel Xeon X6550, 64 GB RAM and there are two kinds of NICs:

Broadcom Corporation NetXtreme II BCM5709 (Driver bnx2) Emulex Corporation OneConnect 10Gb NIC (Driver be2net)

The bnx2 adapters are not in use and disconnected and the be2net handle about 1 GBit/s of traffic.

We´ve downgraded again to our old kernel version, but i guess you should take a look at this.

There are two kinds of messages:

an 9 11:34:28 srv11 kernel: [ 1081.334970] ------------[ cut here ]------------

Jan  9 11:34:28 srv11 kernel: [ 1081.353685] WARNING: at net/ipv4/tcp.c:1330 tcp_cleanup_rbuf+0x4d/0xfc()
Jan  9 11:34:28 srv11 kernel: [ 1081.371956] Hardware name: System x3690 X5 -[7148Z68]-
Jan  9 11:34:28 srv11 kernel: [ 1081.391820] cleanup rbuf bug: copied AD3BCF1 seq AD370AF rcvnxt AD3CF13
Jan  9 11:34:28 srv11 kernel: [ 1081.408971] Modules linked in: 8021q garp stp llc nls_utf8 nls_cp437 vfat fat kvm_intel coretemp kvm joydev acpi_cpufreq mperf crc32c_intel hid_generic shpchp snd_pcm snd_timer snd lpc_ich mfd_core cdc_ether usb
net mii evdev serio_raw ioatdma processor i2c_i801 tpm_tis dca soundcore snd_page_alloc pcspkr tpm microcode i2c_core tpm_bios thermal_sys button pci_hotplug ext4 mbcache jbd2 crc16 dm_mod sg sr_mod cdrom sd_mod crc_t10dif usbhid ata_generic hi
d uhci_hcd ata_piix libata megaraid_sas bnx2 ehci_hcd usbcore scsi_mod usb_common be2net
Jan  9 11:34:28 srv11 kernel: [ 1081.532314] Pid: 13763, comm: haproxy Not tainted 3.7.1 #1
Jan  9 11:34:28 srv11 kernel: [ 1081.554349] Call Trace:
Jan  9 11:34:28 srv11 kernel: [ 1081.573762]  [<ffffffff8103ef70>] ? warn_slowpath_common+0x78/0x8c
Jan  9 11:34:28 srv11 kernel: [ 1081.596750]  [<ffffffff8103f023>] ? warn_slowpath_fmt+0x45/0x4a
Jan  9 11:34:28 srv11 kernel: [ 1081.617853]  [<ffffffff81297d06>] ? sock_pipe_buf_release+0xe/0xe
Jan  9 11:34:28 srv11 kernel: [ 1081.639111]  [<ffffffff812d37c2>] ? tcp_cleanup_rbuf+0x4d/0xfc
Jan  9 11:34:28 srv11 kernel: [ 1081.661541]  [<ffffffff812d39f4>] ? tcp_read_sock+0x183/0x194
Jan  9 11:34:28 srv11 kernel: [ 1081.681164]  [<ffffffff812d423d>] ? tcp_sendpage+0x45b/0x45b
Jan  9 11:34:28 srv11 kernel: [ 1081.703355]  [<ffffffff812d3ad8>] ? tcp_splice_read+0xd3/0x223
Jan  9 11:34:28 srv11 kernel: [ 1081.724912]  [<ffffffff8112d9b9>] ? sys_splice+0x345/0x3c0
Jan  9 11:34:28 srv11 kernel: [ 1081.744525]  [<ffffffff81364b69>] ? system_call_fastpath+0x16/0x1b
Jan  9 11:34:28 srv11 kernel: [ 1081.766683] ---[ end trace fb4ffd749d51e56f ]---
Jan  9 11:52:42 srv11 kernel: [ 2174.882971] WARNING: at net/ipv4/tcp.c:1639 tcp_recvmsg+0x2ca/0x9de()
Jan  9 11:52:42 srv11 kernel: [ 2174.901182] Hardware name: System x3690 X5 -[7148Z68]-
Jan  9 11:52:42 srv11 kernel: [ 2174.921229] recvmsg bug 2: copied 4B9C4CE6 seq 4B9BE950 rcvnxt 4B9C4CE6 fl 0
Jan  9 11:52:42 srv11 kernel: [ 2174.941681] Modules linked in: 8021q garp stp llc nls_utf8 nls_cp437 vfat fat kvm_intel coretemp kvm joydev acpi_cpufreq mperf crc32c_intel hid_generic shpchp snd_pcm snd_timer snd lpc_ich mfd_core cdc_ether usb
net mii evdev serio_raw ioatdma processor i2c_i801 tpm_tis dca soundcore snd_page_alloc pcspkr tpm microcode i2c_core tpm_bios thermal_sys button pci_hotplug ext4 mbcache jbd2 crc16 dm_mod sg sr_mod cdrom sd_mod crc_t10dif usbhid ata_generic hi
d uhci_hcd ata_piix libata megaraid_sas bnx2 ehci_hcd usbcore scsi_mod usb_common be2net
Jan  9 11:52:42 srv11 kernel: [ 2175.075389] Pid: 13250, comm: haproxy Tainted: G        W    3.7.1 #1
Jan  9 11:52:42 srv11 kernel: [ 2175.099391] Call Trace:
Jan  9 11:52:42 srv11 kernel: [ 2175.122727]  [<ffffffff8103ef70>] ? warn_slowpath_common+0x78/0x8c
Jan  9 11:52:42 srv11 kernel: [ 2175.147250]  [<ffffffff8103f023>] ? warn_slowpath_fmt+0x45/0x4a
Jan  9 11:52:42 srv11 kernel: [ 2175.170289]  [<ffffffff81295f64>] ? release_sock+0xe6/0x11c
Jan  9 11:52:43 srv11 kernel: [ 2175.192639]  [<ffffffff812d45d2>] ? tcp_recvmsg+0x2ca/0x9de
Jan  9 11:52:43 srv11 kernel: [ 2175.215020]  [<ffffffff812e0b71>] ? tcp_write_xmit+0x849/0x946
Jan  9 11:52:43 srv11 kernel: [ 2175.237682]  [<ffffffff812f2720>] ? inet_recvmsg+0x64/0x75
Jan  9 11:52:43 srv11 kernel: [ 2175.259727]  [<ffffffff8129052d>] ? sock_recvmsg+0x56/0x6e
Jan  9 11:52:43 srv11 kernel: [ 2175.281111]  [<ffffffff8128fedf>] ? sockfd_lookup_light+0x1a/0x50
Jan  9 11:52:43 srv11 kernel: [ 2175.300573]  [<ffffffff812923f4>] ? sys_recvfrom+0xbf/0x120
Jan  9 11:52:43 srv11 kernel: [ 2175.320073]  [<ffffffff8135d9f7>] ? __schedule+0x4c9/0x4f6
Jan  9 11:52:43 srv11 kernel: [ 2175.341151]  [<ffffffff81364b69>] ? system_call_fastpath+0x16/0x1b
Jan  9 11:52:43 srv11 kernel: [ 2175.360859] ---[ end trace fb4ffd749d51e5a7 ]---

As you can see here, the messages are appearing every couple of minutes:

Jan  9 11:34:28 srv11 kernel: [ 1081.353685] WARNING: at net/ipv4/tcp.c:1330 tcp_cleanup_rbuf+0x4d/0xfc()
Jan  9 11:34:29 srv11 kernel: [ 1081.809446] WARNING: at net/ipv4/tcp.c:1330 tcp_cleanup_rbuf+0x4d/0xfc()
Jan  9 11:34:29 srv11 kernel: [ 1082.235052] WARNING: at net/ipv4/tcp.c:1639 tcp_recvmsg+0x2ca/0x9de()
Jan  9 11:34:29 srv11 kernel: [ 1082.692732] WARNING: at net/ipv4/tcp.c:1639 tcp_recvmsg+0x2ca/0x9de()
Jan  9 11:34:30 srv11 kernel: [ 1083.177807] WARNING: at net/ipv4/tcp.c:1639 tcp_recvmsg+0x2ca/0x9de()
Jan  9 11:34:30 srv11 kernel: [ 1083.698475] WARNING: at net/ipv4/tcp.c:1639 tcp_recvmsg+0x2ca/0x9de()
Jan  9 11:34:31 srv11 kernel: [ 1084.221899] WARNING: at net/ipv4/tcp.c:1639 tcp_recvmsg+0x2ca/0x9de()
Jan  9 11:34:31 srv11 kernel: [ 1084.746992] WARNING: at net/ipv4/tcp.c:1639 tcp_recvmsg+0x2ca/0x9de()
Jan  9 11:34:32 srv11 kernel: [ 1085.267199] WARNING: at net/ipv4/tcp.c:1330 tcp_cleanup_rbuf+0x4d/0xfc()
Jan  9 11:37:16 srv11 kernel: [ 1249.252200] WARNING: at net/ipv4/tcp.c:1330 tcp_cleanup_rbuf+0x4d/0xfc()
Jan  9 11:37:17 srv11 kernel: [ 1249.782915] WARNING: at net/ipv4/tcp.c:1330 tcp_cleanup_rbuf+0x4d/0xfc()
Jan  9 11:37:17 srv11 kernel: [ 1250.315653] WARNING: at net/ipv4/tcp.c:1330 tcp_cleanup_rbuf+0x4d/0xfc()
Jan  9 11:37:18 srv11 kernel: [ 1250.867306] WARNING: at net/ipv4/tcp.c:1330 tcp_cleanup_rbuf+0x4d/0xfc()
Jan  9 11:37:18 srv11 kernel: [ 1251.383968] WARNING: at net/ipv4/tcp.c:1330 tcp_cleanup_rbuf+0x4d/0xfc()
Jan  9 11:37:19 srv11 kernel: [ 1251.897631] WARNING: at net/ipv4/tcp.c:1330 tcp_cleanup_rbuf+0x4d/0xfc()
Jan  9 11:37:19 srv11 kernel: [ 1252.412535] WARNING: at net/ipv4/tcp.c:1330 tcp_cleanup_rbuf+0x4d/0xfc()
Jan  9 11:37:20 srv11 kernel: [ 1252.921313] WARNING: at net/ipv4/tcp.c:1330 tcp_cleanup_rbuf+0x4d/0xfc()
Jan  9 11:40:13 srv11 kernel: [ 1425.644620] WARNING: at net/ipv4/tcp.c:1330 tcp_cleanup_rbuf+0x4d/0xfc()
Jan  9 11:40:13 srv11 kernel: [ 1426.314292] WARNING: at net/ipv4/tcp.c:1330 tcp_cleanup_rbuf+0x4d/0xfc()
Jan  9 11:40:14 srv11 kernel: [ 1426.749374] WARNING: at net/ipv4/tcp.c:1330 tcp_cleanup_rbuf+0x4d/0xfc()
Jan  9 11:40:14 srv11 kernel: [ 1427.198672] WARNING: at net/ipv4/tcp.c:1330 tcp_cleanup_rbuf+0x4d/0xfc()
Jan  9 11:40:15 srv11 kernel: [ 1427.711731] WARNING: at net/ipv4/tcp.c:1330 tcp_cleanup_rbuf+0x4d/0xfc()
Jan  9 11:40:15 srv11 kernel: [ 1428.189284] WARNING: at net/ipv4/tcp.c:1639 tcp_recvmsg+0x2ca/0x9de()
Jan  9 11:40:16 srv11 kernel: [ 1428.701230] WARNING: at net/ipv4/tcp.c:1330 tcp_cleanup_rbuf+0x4d/0xfc()
Jan  9 11:46:09 srv11 kernel: [ 1781.780019] WARNING: at net/ipv4/tcp.c:1330 tcp_cleanup_rbuf+0x4d/0xfc()
Jan  9 11:46:09 srv11 kernel: [ 1782.350094] WARNING: at net/ipv4/tcp.c:1330 tcp_cleanup_rbuf+0x4d/0xfc()
Jan  9 11:46:10 srv11 kernel: [ 1782.853933] WARNING: at net/ipv4/tcp.c:1330 tcp_cleanup_rbuf+0x4d/0xfc()
Jan  9 11:46:10 srv11 kernel: [ 1783.303487] WARNING: at net/ipv4/tcp.c:1330 tcp_cleanup_rbuf+0x4d/0xfc()
Jan  9 11:46:11 srv11 kernel: [ 1783.757022] WARNING: at net/ipv4/tcp.c:1330 tcp_cleanup_rbuf+0x4d/0xfc()
Jan  9 11:46:11 srv11 kernel: [ 1784.260932] WARNING: at net/ipv4/tcp.c:1639 tcp_recvmsg+0x2ca/0x9de()
Jan  9 11:46:12 srv11 kernel: [ 1784.874611] WARNING: at net/ipv4/tcp.c:1639 tcp_recvmsg+0x2ca/0x9de()
Jan  9 11:46:13 srv11 kernel: [ 1785.457209] WARNING: at net/ipv4/tcp.c:1639 tcp_recvmsg+0x2ca/0x9de()
Jan  9 11:46:13 srv11 kernel: [ 1785.933210] WARNING: at net/ipv4/tcp.c:1639 tcp_recvmsg+0x2ca/0x9de()
Jan  9 11:46:14 srv11 kernel: [ 1786.400550] WARNING: at net/ipv4/tcp.c:1330 tcp_cleanup_rbuf+0x4d/0xfc()
Jan  9 11:46:14 srv11 kernel: [ 1786.941620] WARNING: at net/ipv4/tcp.c:1639 tcp_recvmsg+0x2ca/0x9de()
Jan  9 11:46:15 srv11 kernel: [ 1787.510305] WARNING: at net/ipv4/tcp.c:1639 tcp_recvmsg+0x2ca/0x9de()
Jan  9 11:46:15 srv11 kernel: [ 1788.077590] WARNING: at net/ipv4/tcp.c:1330 tcp_cleanup_rbuf+0x4d/0xfc()
Jan  9 11:46:54 srv11 kernel: [ 1827.316572] WARNING: at net/ipv4/tcp.c:1330 tcp_cleanup_rbuf+0x4d/0xfc()
Jan  9 11:46:55 srv11 kernel: [ 1827.721460] WARNING: at net/ipv4/tcp.c:1330 tcp_cleanup_rbuf+0x4d/0xfc()
Jan  9 11:46:55 srv11 kernel: [ 1828.150593] WARNING: at net/ipv4/tcp.c:1330 tcp_cleanup_rbuf+0x4d/0xfc()
Jan  9 11:46:56 srv11 kernel: [ 1828.620180] WARNING: at net/ipv4/tcp.c:1330 tcp_cleanup_rbuf+0x4d/0xfc()
Jan  9 11:46:56 srv11 kernel: [ 1829.120693] WARNING: at net/ipv4/tcp.c:1330 tcp_cleanup_rbuf+0x4d/0xfc()
Jan  9 11:46:57 srv11 kernel: [ 1829.597494] WARNING: at net/ipv4/tcp.c:1639 tcp_recvmsg+0x2ca/0x9de()
Jan  9 11:46:57 srv11 kernel: [ 1830.189551] WARNING: at net/ipv4/tcp.c:1639 tcp_recvmsg+0x2ca/0x9de()
Jan  9 11:46:58 srv11 kernel: [ 1830.745157] WARNING: at net/ipv4/tcp.c:1639 tcp_recvmsg+0x2ca/0x9de()
Jan  9 11:46:58 srv11 kernel: [ 1831.299635] WARNING: at net/ipv4/tcp.c:1639 tcp_recvmsg+0x2ca/0x9de()
Jan  9 11:46:59 srv11 kernel: [ 1831.852929] WARNING: at net/ipv4/tcp.c:1639 tcp_recvmsg+0x2ca/0x9de()
Jan  9 11:47:00 srv11 kernel: [ 1832.408950] WARNING: at net/ipv4/tcp.c:1330 tcp_cleanup_rbuf+0x4d/0xfc()
Jan  9 11:52:38 srv11 kernel: [ 2170.882305] WARNING: at net/ipv4/tcp.c:1330 tcp_cleanup_rbuf+0x4d/0xfc()
Jan  9 11:52:39 srv11 kernel: [ 2171.342827] WARNING: at net/ipv4/tcp.c:1639 tcp_recvmsg+0x2ca/0x9de()
Jan  9 11:52:39 srv11 kernel: [ 2171.853599] WARNING: at net/ipv4/tcp.c:1639 tcp_recvmsg+0x2ca/0x9de()
Jan  9 11:52:40 srv11 kernel: [ 2172.355757] WARNING: at net/ipv4/tcp.c:1639 tcp_recvmsg+0x2ca/0x9de()
Jan  9 11:52:40 srv11 kernel: [ 2172.856194] WARNING: at net/ipv4/tcp.c:1639 tcp_recvmsg+0x2ca/0x9de()
Jan  9 11:52:41 srv11 kernel: [ 2173.357738] WARNING: at net/ipv4/tcp.c:1639 tcp_recvmsg+0x2ca/0x9de()
Jan  9 11:52:41 srv11 kernel: [ 2173.857774] WARNING: at net/ipv4/tcp.c:1639 tcp_recvmsg+0x2ca/0x9de()
Quoted 8 lines
Jan  9 11:52:42 srv11 kernel: [ 2174.361131] WARNING: at net/ipv4/tcp.c:1330 tcp_cleanup_rbuf+0x4d/0xfc()
Jan  9 11:52:42 srv11 kernel: [ 2174.882971] WARNING: at net/ipv4/tcp.c:1639 tcp_recvmsg+0x2ca/0x9de()
Jan  9 11:52:43 srv11 kernel: [ 2175.397627] WARNING: at net/ipv4/tcp.c:1639 tcp_recvmsg+0x2ca/0x9de()
Jan  9 11:52:43 srv11 kernel: [ 2175.911679] WARNING: at net/ipv4/tcp.c:1639 tcp_recvmsg+0x2ca/0x9de()
Jan  9 11:52:44 srv11 kernel: [ 2176.424397] WARNING: at net/ipv4/tcp.c:1639 tcp_recvmsg+0x2ca/0x9de()
Jan  9 11:52:44 srv11 kernel: [ 2176.940545] WARNING: at net/ipv4/tcp.c:1639 tcp_recvmsg+0x2ca/0x9de()
Jan  9 11:52:45 srv11 kernel: [ 2177.456173] WARNING: at net/ipv4/tcp.c:1639 tcp_recvmsg+0x2ca/0x9de()
Jan  9 11:52:45 srv11 kernel: [ 2177.971222] WARNING: at net/ipv4/tcp.c:1330 tcp_cleanup_rbuf+0x4d/0xfc()--

Thanks for this report

This might be a bug because of TCP collapsing

A "netstat -s" would have been a very useful information.

#3 lore raw
Quoted 127 lines · Eric Dumazet
Quoted 117 lines · Christian Becker

Hi,

we´ve installed 3.7.1 yesterday on one of our loadbalancer nodes in order to get TFO.

Unfortunately the kernel started to print warnings every couple of minutes when using tcp splicing in haproxy.

We´ve built the kernel yesterday from the 3.7.1 sources without any modifications.

The System contains two Intel Xeon X6550, 64 GB RAM and there are two kinds of NICs:

Broadcom Corporation NetXtreme II BCM5709 (Driver bnx2) Emulex Corporation OneConnect 10Gb NIC (Driver be2net)

The bnx2 adapters are not in use and disconnected and the be2net handle about 1 GBit/s of traffic.

We´ve downgraded again to our old kernel version, but i guess you should take a look at this.

There are two kinds of messages:

an 9 11:34:28 srv11 kernel: [ 1081.334970] ------------[ cut here ]------------

Jan  9 11:34:28 srv11 kernel: [ 1081.353685] WARNING: at net/ipv4/tcp.c:1330 tcp_cleanup_rbuf+0x4d/0xfc()
Jan  9 11:34:28 srv11 kernel: [ 1081.371956] Hardware name: System x3690 X5 -[7148Z68]-
Jan  9 11:34:28 srv11 kernel: [ 1081.391820] cleanup rbuf bug: copied AD3BCF1 seq AD370AF rcvnxt AD3CF13
Jan  9 11:34:28 srv11 kernel: [ 1081.408971] Modules linked in: 8021q garp stp llc nls_utf8 nls_cp437 vfat fat kvm_intel coretemp kvm joydev acpi_cpufreq mperf crc32c_intel hid_generic shpchp snd_pcm snd_timer snd lpc_ich mfd_core cdc_ether usb
net mii evdev serio_raw ioatdma processor i2c_i801 tpm_tis dca soundcore snd_page_alloc pcspkr tpm microcode i2c_core tpm_bios thermal_sys button pci_hotplug ext4 mbcache jbd2 crc16 dm_mod sg sr_mod cdrom sd_mod crc_t10dif usbhid ata_generic hi
d uhci_hcd ata_piix libata megaraid_sas bnx2 ehci_hcd usbcore scsi_mod usb_common be2net
Jan  9 11:34:28 srv11 kernel: [ 1081.532314] Pid: 13763, comm: haproxy Not tainted 3.7.1 #1
Jan  9 11:34:28 srv11 kernel: [ 1081.554349] Call Trace:
Jan  9 11:34:28 srv11 kernel: [ 1081.573762]  [<ffffffff8103ef70>] ? warn_slowpath_common+0x78/0x8c
Jan  9 11:34:28 srv11 kernel: [ 1081.596750]  [<ffffffff8103f023>] ? warn_slowpath_fmt+0x45/0x4a
Jan  9 11:34:28 srv11 kernel: [ 1081.617853]  [<ffffffff81297d06>] ? sock_pipe_buf_release+0xe/0xe
Jan  9 11:34:28 srv11 kernel: [ 1081.639111]  [<ffffffff812d37c2>] ? tcp_cleanup_rbuf+0x4d/0xfc
Jan  9 11:34:28 srv11 kernel: [ 1081.661541]  [<ffffffff812d39f4>] ? tcp_read_sock+0x183/0x194
Jan  9 11:34:28 srv11 kernel: [ 1081.681164]  [<ffffffff812d423d>] ? tcp_sendpage+0x45b/0x45b
Jan  9 11:34:28 srv11 kernel: [ 1081.703355]  [<ffffffff812d3ad8>] ? tcp_splice_read+0xd3/0x223
Jan  9 11:34:28 srv11 kernel: [ 1081.724912]  [<ffffffff8112d9b9>] ? sys_splice+0x345/0x3c0
Jan  9 11:34:28 srv11 kernel: [ 1081.744525]  [<ffffffff81364b69>] ? system_call_fastpath+0x16/0x1b
Jan  9 11:34:28 srv11 kernel: [ 1081.766683] ---[ end trace fb4ffd749d51e56f ]---
Jan  9 11:52:42 srv11 kernel: [ 2174.882971] WARNING: at net/ipv4/tcp.c:1639 tcp_recvmsg+0x2ca/0x9de()
Jan  9 11:52:42 srv11 kernel: [ 2174.901182] Hardware name: System x3690 X5 -[7148Z68]-
Jan  9 11:52:42 srv11 kernel: [ 2174.921229] recvmsg bug 2: copied 4B9C4CE6 seq 4B9BE950 rcvnxt 4B9C4CE6 fl 0
Jan  9 11:52:42 srv11 kernel: [ 2174.941681] Modules linked in: 8021q garp stp llc nls_utf8 nls_cp437 vfat fat kvm_intel coretemp kvm joydev acpi_cpufreq mperf crc32c_intel hid_generic shpchp snd_pcm snd_timer snd lpc_ich mfd_core cdc_ether usb
net mii evdev serio_raw ioatdma processor i2c_i801 tpm_tis dca soundcore snd_page_alloc pcspkr tpm microcode i2c_core tpm_bios thermal_sys button pci_hotplug ext4 mbcache jbd2 crc16 dm_mod sg sr_mod cdrom sd_mod crc_t10dif usbhid ata_generic hi
d uhci_hcd ata_piix libata megaraid_sas bnx2 ehci_hcd usbcore scsi_mod usb_common be2net
Jan  9 11:52:42 srv11 kernel: [ 2175.075389] Pid: 13250, comm: haproxy Tainted: G        W    3.7.1 #1
Jan  9 11:52:42 srv11 kernel: [ 2175.099391] Call Trace:
Jan  9 11:52:42 srv11 kernel: [ 2175.122727]  [<ffffffff8103ef70>] ? warn_slowpath_common+0x78/0x8c
Jan  9 11:52:42 srv11 kernel: [ 2175.147250]  [<ffffffff8103f023>] ? warn_slowpath_fmt+0x45/0x4a
Jan  9 11:52:42 srv11 kernel: [ 2175.170289]  [<ffffffff81295f64>] ? release_sock+0xe6/0x11c
Jan  9 11:52:43 srv11 kernel: [ 2175.192639]  [<ffffffff812d45d2>] ? tcp_recvmsg+0x2ca/0x9de
Jan  9 11:52:43 srv11 kernel: [ 2175.215020]  [<ffffffff812e0b71>] ? tcp_write_xmit+0x849/0x946
Jan  9 11:52:43 srv11 kernel: [ 2175.237682]  [<ffffffff812f2720>] ? inet_recvmsg+0x64/0x75
Jan  9 11:52:43 srv11 kernel: [ 2175.259727]  [<ffffffff8129052d>] ? sock_recvmsg+0x56/0x6e
Jan  9 11:52:43 srv11 kernel: [ 2175.281111]  [<ffffffff8128fedf>] ? sockfd_lookup_light+0x1a/0x50
Jan  9 11:52:43 srv11 kernel: [ 2175.300573]  [<ffffffff812923f4>] ? sys_recvfrom+0xbf/0x120
Jan  9 11:52:43 srv11 kernel: [ 2175.320073]  [<ffffffff8135d9f7>] ? __schedule+0x4c9/0x4f6
Jan  9 11:52:43 srv11 kernel: [ 2175.341151]  [<ffffffff81364b69>] ? system_call_fastpath+0x16/0x1b
Jan  9 11:52:43 srv11 kernel: [ 2175.360859] ---[ end trace fb4ffd749d51e5a7 ]---

As you can see here, the messages are appearing every couple of minutes:

Jan  9 11:34:28 srv11 kernel: [ 1081.353685] WARNING: at net/ipv4/tcp.c:1330 tcp_cleanup_rbuf+0x4d/0xfc()
Jan  9 11:34:29 srv11 kernel: [ 1081.809446] WARNING: at net/ipv4/tcp.c:1330 tcp_cleanup_rbuf+0x4d/0xfc()
Jan  9 11:34:29 srv11 kernel: [ 1082.235052] WARNING: at net/ipv4/tcp.c:1639 tcp_recvmsg+0x2ca/0x9de()
Jan  9 11:34:29 srv11 kernel: [ 1082.692732] WARNING: at net/ipv4/tcp.c:1639 tcp_recvmsg+0x2ca/0x9de()
Jan  9 11:34:30 srv11 kernel: [ 1083.177807] WARNING: at net/ipv4/tcp.c:1639 tcp_recvmsg+0x2ca/0x9de()
Jan  9 11:34:30 srv11 kernel: [ 1083.698475] WARNING: at net/ipv4/tcp.c:1639 tcp_recvmsg+0x2ca/0x9de()
Jan  9 11:34:31 srv11 kernel: [ 1084.221899] WARNING: at net/ipv4/tcp.c:1639 tcp_recvmsg+0x2ca/0x9de()
Jan  9 11:34:31 srv11 kernel: [ 1084.746992] WARNING: at net/ipv4/tcp.c:1639 tcp_recvmsg+0x2ca/0x9de()
Jan  9 11:34:32 srv11 kernel: [ 1085.267199] WARNING: at net/ipv4/tcp.c:1330 tcp_cleanup_rbuf+0x4d/0xfc()
Jan  9 11:37:16 srv11 kernel: [ 1249.252200] WARNING: at net/ipv4/tcp.c:1330 tcp_cleanup_rbuf+0x4d/0xfc()
Jan  9 11:37:17 srv11 kernel: [ 1249.782915] WARNING: at net/ipv4/tcp.c:1330 tcp_cleanup_rbuf+0x4d/0xfc()
Jan  9 11:37:17 srv11 kernel: [ 1250.315653] WARNING: at net/ipv4/tcp.c:1330 tcp_cleanup_rbuf+0x4d/0xfc()
Jan  9 11:37:18 srv11 kernel: [ 1250.867306] WARNING: at net/ipv4/tcp.c:1330 tcp_cleanup_rbuf+0x4d/0xfc()
Jan  9 11:37:18 srv11 kernel: [ 1251.383968] WARNING: at net/ipv4/tcp.c:1330 tcp_cleanup_rbuf+0x4d/0xfc()
Jan  9 11:37:19 srv11 kernel: [ 1251.897631] WARNING: at net/ipv4/tcp.c:1330 tcp_cleanup_rbuf+0x4d/0xfc()
Jan  9 11:37:19 srv11 kernel: [ 1252.412535] WARNING: at net/ipv4/tcp.c:1330 tcp_cleanup_rbuf+0x4d/0xfc()
Jan  9 11:37:20 srv11 kernel: [ 1252.921313] WARNING: at net/ipv4/tcp.c:1330 tcp_cleanup_rbuf+0x4d/0xfc()
Jan  9 11:40:13 srv11 kernel: [ 1425.644620] WARNING: at net/ipv4/tcp.c:1330 tcp_cleanup_rbuf+0x4d/0xfc()
Jan  9 11:40:13 srv11 kernel: [ 1426.314292] WARNING: at net/ipv4/tcp.c:1330 tcp_cleanup_rbuf+0x4d/0xfc()
Jan  9 11:40:14 srv11 kernel: [ 1426.749374] WARNING: at net/ipv4/tcp.c:1330 tcp_cleanup_rbuf+0x4d/0xfc()
Jan  9 11:40:14 srv11 kernel: [ 1427.198672] WARNING: at net/ipv4/tcp.c:1330 tcp_cleanup_rbuf+0x4d/0xfc()
Jan  9 11:40:15 srv11 kernel: [ 1427.711731] WARNING: at net/ipv4/tcp.c:1330 tcp_cleanup_rbuf+0x4d/0xfc()
Jan  9 11:40:15 srv11 kernel: [ 1428.189284] WARNING: at net/ipv4/tcp.c:1639 tcp_recvmsg+0x2ca/0x9de()
Jan  9 11:40:16 srv11 kernel: [ 1428.701230] WARNING: at net/ipv4/tcp.c:1330 tcp_cleanup_rbuf+0x4d/0xfc()
Jan  9 11:46:09 srv11 kernel: [ 1781.780019] WARNING: at net/ipv4/tcp.c:1330 tcp_cleanup_rbuf+0x4d/0xfc()
Jan  9 11:46:09 srv11 kernel: [ 1782.350094] WARNING: at net/ipv4/tcp.c:1330 tcp_cleanup_rbuf+0x4d/0xfc()
Jan  9 11:46:10 srv11 kernel: [ 1782.853933] WARNING: at net/ipv4/tcp.c:1330 tcp_cleanup_rbuf+0x4d/0xfc()
Jan  9 11:46:10 srv11 kernel: [ 1783.303487] WARNING: at net/ipv4/tcp.c:1330 tcp_cleanup_rbuf+0x4d/0xfc()
Jan  9 11:46:11 srv11 kernel: [ 1783.757022] WARNING: at net/ipv4/tcp.c:1330 tcp_cleanup_rbuf+0x4d/0xfc()
Jan  9 11:46:11 srv11 kernel: [ 1784.260932] WARNING: at net/ipv4/tcp.c:1639 tcp_recvmsg+0x2ca/0x9de()
Jan  9 11:46:12 srv11 kernel: [ 1784.874611] WARNING: at net/ipv4/tcp.c:1639 tcp_recvmsg+0x2ca/0x9de()
Jan  9 11:46:13 srv11 kernel: [ 1785.457209] WARNING: at net/ipv4/tcp.c:1639 tcp_recvmsg+0x2ca/0x9de()
Jan  9 11:46:13 srv11 kernel: [ 1785.933210] WARNING: at net/ipv4/tcp.c:1639 tcp_recvmsg+0x2ca/0x9de()
Jan  9 11:46:14 srv11 kernel: [ 1786.400550] WARNING: at net/ipv4/tcp.c:1330 tcp_cleanup_rbuf+0x4d/0xfc()
Jan  9 11:46:14 srv11 kernel: [ 1786.941620] WARNING: at net/ipv4/tcp.c:1639 tcp_recvmsg+0x2ca/0x9de()
Jan  9 11:46:15 srv11 kernel: [ 1787.510305] WARNING: at net/ipv4/tcp.c:1639 tcp_recvmsg+0x2ca/0x9de()
Jan  9 11:46:15 srv11 kernel: [ 1788.077590] WARNING: at net/ipv4/tcp.c:1330 tcp_cleanup_rbuf+0x4d/0xfc()
Jan  9 11:46:54 srv11 kernel: [ 1827.316572] WARNING: at net/ipv4/tcp.c:1330 tcp_cleanup_rbuf+0x4d/0xfc()
Jan  9 11:46:55 srv11 kernel: [ 1827.721460] WARNING: at net/ipv4/tcp.c:1330 tcp_cleanup_rbuf+0x4d/0xfc()
Jan  9 11:46:55 srv11 kernel: [ 1828.150593] WARNING: at net/ipv4/tcp.c:1330 tcp_cleanup_rbuf+0x4d/0xfc()
Jan  9 11:46:56 srv11 kernel: [ 1828.620180] WARNING: at net/ipv4/tcp.c:1330 tcp_cleanup_rbuf+0x4d/0xfc()
Jan  9 11:46:56 srv11 kernel: [ 1829.120693] WARNING: at net/ipv4/tcp.c:1330 tcp_cleanup_rbuf+0x4d/0xfc()
Jan  9 11:46:57 srv11 kernel: [ 1829.597494] WARNING: at net/ipv4/tcp.c:1639 tcp_recvmsg+0x2ca/0x9de()
Jan  9 11:46:57 srv11 kernel: [ 1830.189551] WARNING: at net/ipv4/tcp.c:1639 tcp_recvmsg+0x2ca/0x9de()
Jan  9 11:46:58 srv11 kernel: [ 1830.745157] WARNING: at net/ipv4/tcp.c:1639 tcp_recvmsg+0x2ca/0x9de()
Jan  9 11:46:58 srv11 kernel: [ 1831.299635] WARNING: at net/ipv4/tcp.c:1639 tcp_recvmsg+0x2ca/0x9de()
Jan  9 11:46:59 srv11 kernel: [ 1831.852929] WARNING: at net/ipv4/tcp.c:1639 tcp_recvmsg+0x2ca/0x9de()
Jan  9 11:47:00 srv11 kernel: [ 1832.408950] WARNING: at net/ipv4/tcp.c:1330 tcp_cleanup_rbuf+0x4d/0xfc()
Jan  9 11:52:38 srv11 kernel: [ 2170.882305] WARNING: at net/ipv4/tcp.c:1330 tcp_cleanup_rbuf+0x4d/0xfc()
Jan  9 11:52:39 srv11 kernel: [ 2171.342827] WARNING: at net/ipv4/tcp.c:1639 tcp_recvmsg+0x2ca/0x9de()
Jan  9 11:52:39 srv11 kernel: [ 2171.853599] WARNING: at net/ipv4/tcp.c:1639 tcp_recvmsg+0x2ca/0x9de()
Jan  9 11:52:40 srv11 kernel: [ 2172.355757] WARNING: at net/ipv4/tcp.c:1639 tcp_recvmsg+0x2ca/0x9de()
Jan  9 11:52:40 srv11 kernel: [ 2172.856194] WARNING: at net/ipv4/tcp.c:1639 tcp_recvmsg+0x2ca/0x9de()
Jan  9 11:52:41 srv11 kernel: [ 2173.357738] WARNING: at net/ipv4/tcp.c:1639 tcp_recvmsg+0x2ca/0x9de()
Jan  9 11:52:41 srv11 kernel: [ 2173.857774] WARNING: at net/ipv4/tcp.c:1639 tcp_recvmsg+0x2ca/0x9de()
Quoted 8 lines
Jan  9 11:52:42 srv11 kernel: [ 2174.361131] WARNING: at net/ipv4/tcp.c:1330 tcp_cleanup_rbuf+0x4d/0xfc()
Jan  9 11:52:42 srv11 kernel: [ 2174.882971] WARNING: at net/ipv4/tcp.c:1639 tcp_recvmsg+0x2ca/0x9de()
Jan  9 11:52:43 srv11 kernel: [ 2175.397627] WARNING: at net/ipv4/tcp.c:1639 tcp_recvmsg+0x2ca/0x9de()
Jan  9 11:52:43 srv11 kernel: [ 2175.911679] WARNING: at net/ipv4/tcp.c:1639 tcp_recvmsg+0x2ca/0x9de()
Jan  9 11:52:44 srv11 kernel: [ 2176.424397] WARNING: at net/ipv4/tcp.c:1639 tcp_recvmsg+0x2ca/0x9de()
Jan  9 11:52:44 srv11 kernel: [ 2176.940545] WARNING: at net/ipv4/tcp.c:1639 tcp_recvmsg+0x2ca/0x9de()
Jan  9 11:52:45 srv11 kernel: [ 2177.456173] WARNING: at net/ipv4/tcp.c:1639 tcp_recvmsg+0x2ca/0x9de()
Jan  9 11:52:45 srv11 kernel: [ 2177.971222] WARNING: at net/ipv4/tcp.c:1330 tcp_cleanup_rbuf+0x4d/0xfc()--
Quoted 7 lines

Thanks for this report

This might be a bug because of TCP collapsing

A "netstat -s" would have been a very useful information.

My feeling is that tcp_recv_skb() should eat skbs instead of only finding the right one

Thats because skb_splice_bits() releases the socket lock before calling splice_to_pipe()

Once socket is released, other incoming TCP frames can be processed, and the skb we are actually processing might be 'collapsed' into smaller units.

Christian, if I send you patches, are you OK to test them ?

#4 lore raw

From: Eric Dumazet <[email protected]>

Quoted 3 lines · Eric Dumazet

My feeling is that tcp_recv_skb() should eat skbs instead of only finding the right one

Thats indeed the case.

Quoted 10 lines

Thats because skb_splice_bits() releases the socket lock before calling splice_to_pipe()

Once socket is released, other incoming TCP frames can be processed, and the skb we are actually processing might be 'collapsed' into smaller units.

Christian, if I send you patches, are you OK to test them ?

Here is the patch fixing this issue.

Thanks a lot for your report, this is a very very old bug.

GRO being more deployed, and with TCP coalescing as well, chances to trigger this bug increased a lot.

To reproduce it, I had to force MSS=400 and stress the receiver, adding extra delays in skb_splice_bits() with socket lock being not held.

[PATCH] tcp: fix splice() and tcp collapsing interaction

Under unusual circumstances, TCP collapse can split a big GRO TCP packet while its being used in a splice(socket->pipe) operation.

skb_splice_bits() releases the socket lock before calling splice_to_pipe().

[ 1081.353685] WARNING: at net/ipv4/tcp.c:1330 tcp_cleanup_rbuf+0x4d/0xfc()
[ 1081.371956] Hardware name: System x3690 X5 -[7148Z68]-
[ 1081.391820] cleanup rbuf bug: copied AD3BCF1 seq AD370AF rcvnxt AD3CF13

To fix this problem, we must eat skbs in tcp_recv_skb().

Remove the inline keyword from tcp_recv_skb() definition since it has three call sites.

Reported-by Christian Becker <[email protected]>
Cc Willy Tarreau <[email protected]>
Signed-off-by Eric Dumazet <[email protected]>
---
 net/ipv4/tcp.c |   13 ++++++++++---
 1 file changed, 10 insertions(+), 3 deletions(-)
net/ipv4/tcp.c +10 -3
index 1ca2536..1f901be 100644
@@ -1428,12 +1428,12 @@ static void tcp_service_net_dma(struct sock *sk, bool wait)
 }
 #endif
 
-static inline struct sk_buff *tcp_recv_skb(struct sock *sk, u32 seq, u32 *off)
+static struct sk_buff *tcp_recv_skb(struct sock *sk, u32 seq, u32 *off)
 {
 	struct sk_buff *skb;
 	u32 offset;
 
-	skb_queue_walk(&sk->sk_receive_queue, skb) {
+	while ((skb = skb_peek(&sk->sk_receive_queue)) != NULL) {
 		offset = seq - TCP_SKB_CB(skb)->seq;
 		if (tcp_hdr(skb)->syn)
 			offset--;

@@ -1441,6 +1441,11 @@ static inline struct sk_buff *tcp_recv_skb(struct sock *sk, u32 seq, u32 *off)
 			*off = offset;
 			return skb;
 		}
+		/* This looks weird, but this can happen if TCP collapsing
+		 * splitted a fat GRO packet, while we released socket lock
+		 * in skb_splice_bits()
+		 */
+		sk_eat_skb(sk, skb, false);
 	}
 	return NULL;
 }

@@ -1520,8 +1525,10 @@ int tcp_read_sock(struct sock *sk, read_descriptor_t *desc,
 	tcp_rcv_space_adjust(sk);
 
 	/* Clean up data we have read: This will do ACK frames. */
-	if (copied > 0)
+	if (copied > 0) {
+		tcp_recv_skb(sk, seq, &offset);
 		tcp_cleanup_rbuf(sk, copied);
+	}
 	return copied;
 }
 EXPORT_SYMBOL(tcp_read_sock);
#5 lore raw
Quoted 23 lines · Eric Dumazet

From: Eric Dumazet <[email protected]>

Quoted 3 lines · Eric Dumazet

My feeling is that tcp_recv_skb() should eat skbs instead of only finding the right one

Thats indeed the case.

Quoted 10 lines

Thats because skb_splice_bits() releases the socket lock before calling splice_to_pipe()

Once socket is released, other incoming TCP frames can be processed, and the skb we are actually processing might be 'collapsed' into smaller units.

Christian, if I send you patches, are you OK to test them ?

Here is the patch fixing this issue.

Quoted 8 lines

Thanks a lot for your report, this is a very very old bug.

GRO being more deployed, and with TCP coalescing as well, chances to trigger this bug increased a lot.

To reproduce it, I had to force MSS=400 and stress the receiver, adding extra delays in skb_splice_bits() with socket lock being not held.

FWIW, I tested your patch here and did not notice any regression compared to last week-end tests, at various MTU size combinations.

Cheers,
Willy

Tested-by Willy Tarreau
#6 lore raw
Quoted 3 lines · Willy Tarreau

FWIW, I tested your patch here and did not notice any regression compared to last week-end tests, at various MTU size combinations.

Thanks Willy

Tested-by Willy Tarreau <[email protected]>

I believe I need to fix net-next,

commit 9ca1b22d6d228177e6f929f6818a1cd3d5e30c4a
(net: splice: avoid high order page splitting)

missed the loopback case : the skb->head might need several linear_to_page() calls.

I'll send a patch after full testing.

#7 lore raw
Quoted 7 lines · Eric Dumazet
Quoted 3 lines · Willy Tarreau

FWIW, I tested your patch here and did not notice any regression compared to last week-end tests, at various MTU size combinations.

Thanks Willy

I also want to thanks Rick, as the latest netperf has splice() support.

Thanks Rick !

#8 lore raw
Quoted 3 lines · Eric Dumazet

I also want to thanks Rick, as the latest netperf has splice() support.

Thanks Rick !

You are quite welcome - and thank you for helping me get it to actually work :)

Those wishing to try it themselves should grab the top-of-trunk netperf bits from http://www.netperf.org/svn/netperf2/trunk . The use of splice() is gated by a test-specific -V option:

raj@tardy:~/netperf2_trunk/src$ ./netperf -t omni -- -d recv -V
OMNI Receive TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to localhost.localdomain () port 0 AF_INET : copy avoidance : demo
Remote      Local       Remote Elapsed Throughput Throughput
Send Socket Recv Socket Send   Time               Units
Size        Size        Size   (sec)
Final       Final
1661688     4194304     16384  10.00   26103.14   10^6bits/s

You should see that "copy avoidance" appearing in the test banner. It will also "take" for things like a migrated TCP_mumble test. For those cases where you don't see a throughput change, enabling CPU utilization measurement and looking at that and service demand should show a difference.

happy benchmarking,

rick jones

#9 lore raw
Quoted 21 lines · Rick Jones
Quoted 3 lines · Eric Dumazet

I also want to thanks Rick, as the latest netperf has splice() support.

Thanks Rick !

You are quite welcome - and thank you for helping me get it to actually work :)

Those wishing to try it themselves should grab the top-of-trunk netperf bits from http://www.netperf.org/svn/netperf2/trunk . The use of splice() is gated by a test-specific -V option:

raj@tardy:~/netperf2_trunk/src$ ./netperf -t omni -- -d recv -V
OMNI Receive TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 
localhost.localdomain () port 0 AF_INET : copy avoidance : demo
Remote      Local       Remote Elapsed Throughput Throughput
Send Socket Recv Socket Send   Time               Units
Size        Size        Size   (sec)
Final       Final
1661688     4194304     16384  10.00   26103.14   10^6bits/s
Quoted 4 lines

You should see that "copy avoidance" appearing in the test banner. It will also "take" for things like a migrated TCP_mumble test. For those cases where you don't see a throughput change, enabling CPU utilization measurement and looking at that and service demand should show a difference.

Thanks Rick !

Can we use zero copy for the sender as well (sendfile() or vmsplice()) ?

Here are some results :

Reference time (no splice())

# ./netperf -H 127.0.0.1 -t omni -- -d recv     
OMNI Receive TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 127.0.0.1 ()
port 0 AF_INET
catcher: timer popped with times_up != 0
Remote      Local       Remote Elapsed Throughput Throughput  
Send Socket Recv Socket Send   Time               Units       
Size        Size        Size   (sec)                          
Final       Final                                             
2097152     2097152     16384  10.00   33237.74   10^6bits/s

zero copy at receiver (splice(socket ->pipe), splice(pipe -> /dev/null))

# ./netperf -H 127.0.0.1 -t omni -- -d recv -V
OMNI Receive TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 127.0.0.1 ()
port 0 AF_INET : copy avoidance
catcher: timer popped with times_up != 0
Remote      Local       Remote Elapsed Throughput Throughput  
Send Socket Recv Socket Send   Time               Units       
Size        Size        Size   (sec)                          
Final       Final                                             
1325580     2097152     16384  10.00   51980.60   10^6bits/s
#10 lore raw
Quoted 1 line · Eric Dumazet

Can we use zero copy for the sender as well (sendfile() or vmsplice()) ?

Not at present. The TCP_SENDFILE test has not been "migrated" and there is no corresponding sendfile() test in the omni code, so the attempt to enable copy-avoidance won't "take."

I'll take that as a feature request for netperf.

rick

#11 lore raw
Quoted 6 lines · Rick Jones
Quoted 1 line · Eric Dumazet

Can we use zero copy for the sender as well (sendfile() or vmsplice()) ?

Not at present. The TCP_SENDFILE test has not been "migrated" and there is no corresponding sendfile() test in the omni code, so the attempt to enable copy-avoidance won't "take."

Note that in my httpterm server, I'm using tee+splice(). I had a problem with vmsplice() in the past, I don't remember which one.

For the "inject" http client, I'm using recv(MSG_TRUNC) which is always faster than splice() and does the job well, considering that the goal is to count and destroy data very fast.

By combining them both on the same machine over the loopback, I see 70 Gbps on a core-2 duo 2.66 GHz. These are obviously not real gigs, but at least it shows what the network stack can do when we remove memory copies !

Regards,
Willy

#12 lore raw

PATCH 1/1 net: splice: fix __splice_segment()

In reply to Eric Dumazet

From: Eric Dumazet <[email protected]>

commit 9ca1b22d6d2 (net: splice: avoid high order page splitting) forgot that skb->head could need a copy into several page frags.

This could be the case for loopback traffic mostly.

Also remove now useless skb argument from linear_to_page() and __splice_segment() prototypes.

Signed-off-by Eric Dumazet <[email protected]>
Cc Willy Tarreau <[email protected]>
---
 net/core/skbuff.c |   28 +++++++++++++++-------------
 1 file changed, 15 insertions(+), 13 deletions(-)
net/core/skbuff.c +15 -13
index 1e1b9ea..2568c44 100644
@@ -1652,7 +1652,7 @@ static void sock_spd_release(struct splice_pipe_desc *spd, unsigned int i)
 
 static struct page *linear_to_page(struct page *page, unsigned int *len,
 				   unsigned int *offset,
-				   struct sk_buff *skb, struct sock *sk)
+				   struct sock *sk)
 {
 	struct page_frag *pfrag = sk_page_frag(sk);
 

@@ -1685,14 +1685,14 @@ static bool spd_can_coalesce(const struct splice_pipe_desc *spd,
 static bool spd_fill_page(struct splice_pipe_desc *spd,
 			  struct pipe_inode_info *pipe, struct page *page,
 			  unsigned int *len, unsigned int offset,
-			  struct sk_buff *skb, bool linear,
+			  bool linear,
 			  struct sock *sk)
 {
 	if (unlikely(spd->nr_pages == MAX_SKB_FRAGS))
 		return true;
 
 	if (linear) {
-		page = linear_to_page(page, len, &offset, skb, sk);
+		page = linear_to_page(page, len, &offset, sk);
 		if (!page)
 			return true;
 	}

@@ -1711,13 +1711,11 @@ static bool spd_fill_page(struct splice_pipe_desc *spd,
 
 static bool __splice_segment(struct page *page, unsigned int poff,
 			     unsigned int plen, unsigned int *off,
-			     unsigned int *len, struct sk_buff *skb,
+			     unsigned int *len,
 			     struct splice_pipe_desc *spd, bool linear,
 			     struct sock *sk,
 			     struct pipe_inode_info *pipe)
 {
-	unsigned int flen;
-
 	if (!*len)
 		return true;
 

@@ -1732,12 +1730,16 @@ static bool __splice_segment(struct page *page, unsigned int poff,
 	plen -= *off;
 	*off = 0;
 
-	flen = min(*len, plen);
-
-	if (spd_fill_page(spd, pipe, page, &flen, poff, skb, linear, sk))
-		return true;
+	do {
+		unsigned int flen = min(*len, plen);
 
-	*len -= flen;
+		if (spd_fill_page(spd, pipe, page, &flen, poff,
+				  linear, sk))
+			return true;
+		poff += flen;
+		plen -= flen;
+		*len -= flen;
+	} while (*len && plen);
 
 	return false;
 }

@@ -1760,7 +1762,7 @@ static bool __skb_splice_bits(struct sk_buff *skb, struct pipe_inode_info *pipe,
 	if (__splice_segment(virt_to_page(skb->data),
 			     (unsigned long) skb->data & (PAGE_SIZE - 1),
 			     skb_headlen(skb),
-			     offset, len, skb, spd,
+			     offset, len, spd,
 			     skb_head_is_locked(skb),
 			     sk, pipe))
 		return true;

@@ -1773,7 +1775,7 @@ static bool __skb_splice_bits(struct sk_buff *skb, struct pipe_inode_info *pipe,
 
 		if (__splice_segment(skb_frag_page(f),
 				     f->page_offset, skb_frag_size(f),
-				     offset, len, skb, spd, false, sk, pipe))
+				     offset, len, spd, false, sk, pipe))
 			return true;
 	}
maintainer Applied
#13 lore raw

Re: [PATCH net-next] net: splice: fix __splice_segment()

From: Eric Dumazet <[email protected]>
Date: Fri, 11 Jan 2013 16:46:37 -0800

Quoted 5 lines

From: Eric Dumazet <[email protected]>

commit 9ca1b22d6d2 (net: splice: avoid high order page splitting) forgot that skb->head could need a copy into several page frags.

Quoted 7 lines

This could be the case for loopback traffic mostly.

Also remove now useless skb argument from linear_to_page() and __splice_segment() prototypes.

Signed-off-by Eric Dumazet <[email protected]>
Cc Willy Tarreau <[email protected]>

Applied, thanks Eric.

#14 lore raw
In reply to Eric Dumazet

Hi Eric,

this is probably a dumb question ... but since the fix is in net/ipv4/tcp.c I was asking myself if this can affect IPv6 as well?

Thanks,

Lukas

Quoted 68 lines

[PATCH] tcp: fix splice() and tcp collapsing interaction

Under unusual circumstances, TCP collapse can split a big GRO TCP packet while its being used in a splice(socket->pipe) operation.

skb_splice_bits() releases the socket lock before calling splice_to_pipe().

[ 1081.353685] WARNING: at net/ipv4/tcp.c:1330 tcp_cleanup_rbuf+0x4d/0xfc()
[ 1081.371956] Hardware name: System x3690 X5 -[7148Z68]-
[ 1081.391820] cleanup rbuf bug: copied AD3BCF1 seq AD370AF rcvnxt AD3CF13

To fix this problem, we must eat skbs in tcp_recv_skb().

Remove the inline keyword from tcp_recv_skb() definition since it has three call sites.

Reported-by Christian Becker <[email protected]>
Cc Willy Tarreau <[email protected]>
Signed-off-by Eric Dumazet <[email protected]>
---
 net/ipv4/tcp.c |   13 ++++++++++---
 1 file changed, 10 insertions(+), 3 deletions(-)
net/ipv4/tcp.c +10 -3
index 1ca2536..1f901be 100644
@@ -1428,12 +1428,12 @@ static void tcp_service_net_dma(struct sock *sk, bool wait)
 }
 #endif
 
-static inline struct sk_buff *tcp_recv_skb(struct sock *sk, u32 seq, u32 *off)
+static struct sk_buff *tcp_recv_skb(struct sock *sk, u32 seq, u32 *off)
 {
 	struct sk_buff *skb;
 	u32 offset;
 
-	skb_queue_walk(&sk->sk_receive_queue, skb) {
+	while ((skb = skb_peek(&sk->sk_receive_queue)) != NULL) {
 		offset = seq - TCP_SKB_CB(skb)->seq;
 		if (tcp_hdr(skb)->syn)
 			offset--;

@@ -1441,6 +1441,11 @@ static inline struct sk_buff *tcp_recv_skb(struct sock *sk, u32 seq, u32 *off)
 			*off = offset;
 			return skb;
 		}
+		/* This looks weird, but this can happen if TCP collapsing
+		 * splitted a fat GRO packet, while we released socket lock
+		 * in skb_splice_bits()
+		 */
+		sk_eat_skb(sk, skb, false);
 	}
 	return NULL;
 }

@@ -1520,8 +1525,10 @@ int tcp_read_sock(struct sock *sk, read_descriptor_t *desc,
 	tcp_rcv_space_adjust(sk);
 
 	/* Clean up data we have read: This will do ACK frames. */
-	if (copied > 0)
+	if (copied > 0) {
+		tcp_recv_skb(sk, seq, &offset);
 		tcp_cleanup_rbuf(sk, copied);
+	}
 	return copied;
 }
 EXPORT_SYMBOL(tcp_read_sock);
#15 lore raw
Quoted 4 lines · Lukas Tribus

Hi Eric,

this is probably a dumb question ... but since the fix is in net/ipv4/tcp.c I was asking myself if this can affect IPv6 as well?

Yes it can, net/ipv4 contains generic TCP code.

maintainer Applied
#16 lore raw
In reply to Eric Dumazet

From: Eric Dumazet <[email protected]>
Date: Wed, 09 Jan 2013 22:59:09 -0800

Quoted 12 lines

[PATCH] tcp: fix splice() and tcp collapsing interaction

Under unusual circumstances, TCP collapse can split a big GRO TCP packet while its being used in a splice(socket->pipe) operation.

skb_splice_bits() releases the socket lock before calling splice_to_pipe().

[ 1081.353685] WARNING: at net/ipv4/tcp.c:1330 tcp_cleanup_rbuf+0x4d/0xfc()
[ 1081.371956] Hardware name: System x3690 X5 -[7148Z68]-
[ 1081.391820] cleanup rbuf bug: copied AD3BCF1 seq AD370AF rcvnxt AD3CF13
Quoted 8 lines

To fix this problem, we must eat skbs in tcp_recv_skb().

Remove the inline keyword from tcp_recv_skb() definition since it has three call sites.

Reported-by Christian Becker <[email protected]>
Cc Willy Tarreau <[email protected]>
Signed-off-by Eric Dumazet <[email protected]>

Applied.