Fragmented UDP packets trigger rx_missed_errors on 82599EB
Hi
While investigating a upper level network issue, I found out the root cause may be triggered by packet loss at NIC level, showed by rx_missed_errors.
kernel: linux-2.6.32-358.el6.x86_64
server: iperf -s -B 192.168.5.1 -u
client: iperf -c 192.168.5.1 -u -b 10G -i 1 -t 1000 -P 12 -l 3k Use -l to specify buffers large than MTU to create fragmented IP packets.
1. Tune rx ring from 512 to max 4096 does help for single flow, but still got great rx_missed_errors from multiple flows.
2. Using latest net-next 4.0.0-rc4 shows the same effect.
3. Got 9.4Gbits/sec even though rx_missed_errors shows NIC level packets drop.
rx_missed_errors value comes from RXMPC, where 82599 data sheet 8.2.3.5.1 says:
"Missed packet interrupt is activated for each received packet that overflows the Rx packet buffer (overrun).
he packet is dropped and also increments the associated RXMPC[n] counter."
I'm not sure it means my env is mis-configured or anything I'm missing obviously. Any hints?
Attached several logs as below.
# ethtool -S eth4 NIC statistics: rx_packets: 1047869017 tx_packets: 206275776 rx_bytes: 1103333268576 tx_bytes: 289198212456 rx_pkts_nic: 1047200292 tx_pkts_nic: 206275773 rx_bytes_nic: 1907927064202 tx_bytes_nic: 290023317512 lsc_int: 17 tx_busy: 0 non_eop_descs: 0 rx_errors: 0 tx_errors: 0 rx_dropped: 0 tx_dropped: 0 multicast: 0 broadcast: 4310 rx_no_buffer_count: 0 collisions: 0 rx_over_errors: 0 rx_crc_errors: 0 rx_frame_errors: 0 hw_rsc_aggregated: 0 hw_rsc_flushed: 0 fdir_match: 0 fdir_miss: 6545204 fdir_overflow: 0 rx_fifo_errors: 0 rx_missed_errors: 638609576 <-------- tx_aborted_errors: 0 tx_carrier_errors: 0 tx_fifo_errors: 0 tx_heartbeat_errors: 0 tx_timeout_count: 0 tx_restart_queue: 0 rx_long_length_errors: 0 rx_short_length_errors: 0 tx_flow_control_xon: 174182 rx_flow_control_xon: 0 tx_flow_control_xoff: 946044
# numactl --hardware available: 4 nodes (0-3) node 0 cpus: 0 1 2 3 4 20 21 22 23 24 node 0 size: 24466 MB node 0 free: 22444 MB node 1 cpus: 5 6 7 8 9 25 26 27 28 29 node 1 size: 16384 MB node 1 free: 15831 MB node 2 cpus: 10 11 12 13 14 30 31 32 33 34 node 2 size: 16384 MB node 2 free: 15791 MB node 3 cpus: 15 16 17 18 19 35 36 37 38 39 node 3 size: 24576 MB node 3 free: 22508 MB node distances: node 0 1 2 3 0: 10 21 31 31 1: 21 10 31 31 2: 31 31 10 21 3: 31 31 21 10
# ethtool -g eth4 Ring parameters for eth4: Pre-set maximums: RX: 4096 RX Mini: 0 RX Jumbo: 0 TX: 4096 Current hardware settings: RX: 4096 <---- I tweak it from 512 to max 4096, it helps for single flow, but still not good for multiple flows. RX Mini: 0 RX Jumbo: 0 TX: 512
# ethtool -a eth4 Pause parameters for eth4: Autonegotiate: on RX: on TX: on
# ethtool -c eth4 Coalesce parameters for eth4: Adaptive RX: off TX: off stats-block-usecs: 0 sample-interval: 0 pkt-rate-low: 0 pkt-rate-high: 0
rx-usecs: 1
rx-frames: 0
rx-usecs-irq: 0
rx-frames-irq: 0
tx-usecs: 0
tx-frames: 0
tx-usecs-irq: 0
tx-frames-irq: 0
rx-usecs-low: 0
rx-frame-low: 0
tx-usecs-low: 0
tx-frame-low: 0
rx-usecs-high: 0
rx-frame-high: 0
tx-usecs-high: 0
tx-frame-high: 0
# lspci -vv (Assuming I'm using 84:00.0) 84:00.0 Ethernet controller: Intel Corporation 82599EB 10-Gigabit SFI/SFP+ Network Connection (rev 01) Subsystem: Intel Corporation Ethernet Server Adapter X520-2 Physical Slot: 803 Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisINTx+ Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- Latency: 0, Cache Line Size: 64 bytes Interrupt: pin A routed to IRQ 66 Region 0: Memory at 387fffb80000 (64-bit, prefetchable) [size=512K] Region 2: I/O ports at 8020 [size=32] Region 4: Memory at 387fffc04000 (64-bit, prefetchable) [size=16K] Capabilities: [40] Power Management version 3 Flags: PMEClk- DSI+ D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold-) Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=1 PME- Capabilities: [50] MSI: Enable- Count=1/1 Maskable+ 64bit+ Address: 0000000000000000 Data: 0000 Masking: 00000000 Pending: 00000000 Capabilities: [70] MSI-X: Enable+ Count=64 Masked- Vector table: BAR=4 offset=00000000 PBA: BAR=4 offset=00002000 Capabilities: [a0] Express (v2) Endpoint, MSI 00 DevCap: MaxPayload 512 bytes, PhantFunc 0, Latency L0s <512ns, L1 <64us ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset+ DevCtl: Report errors: Correctable+ Non-Fatal+ Fatal+ Unsupported+ RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+ FLReset- MaxPayload 256 bytes, MaxReadReq 512 bytes DevSta: CorrErr- UncorrErr- FatalErr+ UnsuppReq+ AuxPwr- TransPend- LnkCap: Port #2, Speed 5GT/s, Width x8, ASPM L0s, Latency L0 <1us, L1 <8us ClockPM- Surprise- LLActRep- BwNot- LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- Retrain- CommClk+ ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt- LnkSta: Speed 5GT/s, Width x8, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt- DevCap2: Completion Timeout: Range ABCD, TimeoutDis+, LTR-, OBFF Not Supported DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis- Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS- Compliance De-emphasis: -6dB LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1- EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest- Capabilities: [100 v1] Advanced Error Reporting UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq+ ACSViol- UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt+ UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq+ ACSViol- UESvrt: DLP+ SDES- TLP+ FCP+ CmpltTO+ CmpltAbrt+ UnxCmplt+ RxOF+ MalfTLP+ ECRC+ UnsupReq+ ACSViol- CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr- CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+ AERCap: First Error Pointer: 00, GenCap+ CGenEn- ChkCap+ ChkEn- Capabilities: [140 v1] Device Serial Number 90-e2-ba-ff-ff-50-8d-f0 Capabilities: [150 v1] Alternative Routing-ID Interpretation (ARI) ARICap: MFVC- ACS-, Next Function: 1 ARICtl: MFVC- ACS-, Function Group: 0 Capabilities: [160 v1] Single Root I/O Virtualization (SR-IOV) IOVCap: Migration-, Interrupt Message Number: 000 IOVCtl: Enable- Migration- Interrupt- MSE- ARIHierarchy+ IOVSta: Migration- Initial VFs: 64, Total VFs: 64, Number of VFs: 64, Function Dependency Link: 00 VF offset: 128, stride: 2, Device ID: 10ed Supported Page Size: 00000553, System Page Size: 00000001 Region 0: Memory at 00000000c8000000 (64-bit, non-prefetchable) Region 3: Memory@00000000c8100000 (64-bit, non-prefetchable) VF Migration: offset: 00000000, BIR: 0 Kernel driver in use: ixgbe Kernel modules: ixgbe
84:00.1 Ethernet controller: Intel Corporation 82599EB 10-Gigabit SFI/SFP+ Network Connection (rev 01)
Subsystem: Intel Corporation Ethernet Server Adapter X520-2
Physical Slot: 803
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisINTx+
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0, Cache Line Size: 64 bytes
Interrupt: pin B routed to IRQ 69
Region 0: Memory at 387fffb00000 (64-bit, prefetchable) [size=512K]
Region 2: I/O ports at 8000 [size=32]
Region 4: Memory at 387fffc00000 (64-bit, prefetchable) [size=16K]
Capabilities: [40] Power Management version 3
Flags: PMEClk- DSI+ D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold-)
Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=1 PME-
Capabilities: [50] MSI: Enable- Count=1/1 Maskable+ 64bit+
Address: 0000000000000000 Data: 0000
Masking: 00000000 Pending: 00000000
Capabilities: [70] MSI-X: Enable+ Count=64 Masked-
Vector table: BAR=4 offset=00000000
PBA: BAR=4 offset=00002000
Capabilities: [a0] Express (v2) Endpoint, MSI 00
DevCap: MaxPayload 512 bytes, PhantFunc 0, Latency L0s <512ns, L1 <64us
ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset+
DevCtl: Report errors: Correctable+ Non-Fatal+ Fatal+ Unsupported+
RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+ FLReset-
MaxPayload 256 bytes, MaxReadReq 512 bytes
DevSta: CorrErr- UncorrErr- FatalErr+ UnsuppReq+ AuxPwr- TransPend-
LnkCap: Port #2, Speed 5GT/s, Width x8, ASPM L0s, Latency L0 <1us, L1 <8us
ClockPM- Surprise- LLActRep- BwNot-
LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- Retrain- CommClk+
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 5GT/s, Width x8, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
DevCap2: Completion Timeout: Range ABCD, TimeoutDis+, LTR-, OBFF Not Supported
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1- EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
Capabilities: [100 v1] Advanced Error Reporting
UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq+ ACSViol-
UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt+ UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq+ ACSViol-
UESvrt: DLP+ SDES- TLP+ FCP+ CmpltTO+ CmpltAbrt+ UnxCmplt+ RxOF+ MalfTLP+ ECRC+ UnsupReq+ ACSViol-
CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
AERCap: First Error Pointer: 00, GenCap+ CGenEn- ChkCap+ ChkEn-
Capabilities: [140 v1] Device Serial Number 90-e2-ba-ff-ff-50-8d-f0
Capabilities: [150 v1] Alternative Routing-ID Interpretation (ARI)
ARICap: MFVC- ACS-, Next Function: 0
ARICtl: MFVC- ACS-, Function Group: 0
Capabilities: [160 v1] Single Root I/O Virtualization (SR-IOV)
IOVCap: Migration-, Interrupt Message Number: 000
IOVCtl: Enable- Migration- Interrupt- MSE- ARIHierarchy-
IOVSta: Migration-
Initial VFs: 64, Total VFs: 64, Number of VFs: 64, Function Dependency Link: 01
VF offset: 128, stride: 2, Device ID: 10ed
Supported Page Size: 00000553, System Page Size: 00000001
Region 0: Memory at 00000000c8200000 (64-bit, non-prefetchable)
Region 3: Memory@00000000c8300000 (64-bit, non-prefetchable)
VF Migration: offset: 00000000, BIR: 0
Kernel driver in use: ixgbe
Kernel modules: ixgbe
# lspci -t -+-[0000:ff]-+-08.0 | +-08.2 | +-08.3 | +-09.0 | +-09.2 | +-09.3 | +-0b.0 | +-0b.1 | +-0b.2 | +-0c.0 | +-0c.1 | +-0c.2 | +-0c.3 | +-0c.4 | +-0c.5 | +-0c.6 | +-0c.7 | +-0d.0 | +-0d.1 | +-0f.0 | +-0f.1 | +-0f.2 | +-0f.3 | +-0f.4 | +-0f.5 | +-0f.6 | +-10.0 | +-10.1 | +-10.5 | +-10.6 | +-10.7 | +-12.0 | +-12.1 | +-12.4 | +-12.5 | +-13.0 | +-13.1 | +-13.2 | +-13.3 | +-13.6 | +-13.7 | +-14.0 | +-14.1 | +-14.2 | +-14.3 | +-14.4 | +-14.5 | +-14.6 | +-14.7 | +-16.0 | +-16.1 | +-16.2 | +-16.3 | +-16.6 | +-16.7 | +-17.0 | +-17.1 | +-17.2 | +-17.3 | +-17.4 | +-17.5 | +-17.6 | +-17.7 | +-1e.0 | +-1e.1 | +-1e.2 | +-1e.3 | +-1e.4 | +-1f.0 | \-1f.2 +-[0000:80]-+-01.0-[81-82]--+-00.0 | | \-00.1 | +-03.0-[83]-- | +-03.2-[84-85]--+-00.0 | | \-00.1 | +-04.0 | +-04.1 | +-04.2 | +-04.3 | +-04.4 | +-04.5 | +-04.6 | +-04.7 | +-05.0 | +-05.1 | +-05.2 | \-05.4 +-[0000:7f]-+-08.0 | +-08.2 | +-08.3 | +-09.0 | +-09.2 | +-09.3 | +-0b.0 | +-0b.1 | +-0b.2 | +-0c.0 | +-0c.1 | +-0c.2 | +-0c.3 | +-0c.4 | +-0c.5 | +-0c.6 | +-0c.7 | +-0d.0 | +-0d.1 | +-0f.0 | +-0f.1 | +-0f.2 | +-0f.3 | +-0f.4 | +-0f.5 | +-0f.6 | +-10.0 | +-10.1 | +-10.5 | +-10.6 | +-10.7 | +-12.0 | +-12.1 | +-12.4 | +-12.5 | +-13.0 | +-13.1 | +-13.2 | +-13.3 | +-13.6 | +-13.7 | +-14.0 | +-14.1 | +-14.2 | +-14.3 | +-14.4 | +-14.5 | +-14.6 | +-14.7 | +-16.0 | +-16.1 | +-16.2 | +-16.3 | +-16.6 | +-16.7 | +-17.0 | +-17.1 | +-17.2 | +-17.3 | +-17.4 | +-17.5 | +-17.6 | +-17.7 | +-1e.0 | +-1e.1 | +-1e.2 | +-1e.3 | +-1e.4 | +-1f.0 | \-1f.2 \-[0000:00]-+-00.0 +-01.0-[01]-- +-02.0-[02]-- +-02.2-[03-04]--+-00.0 | \-00.1 +-03.0-[05]-- +-03.2-[06]-- +-04.0 +-04.1 +-04.2 +-04.3 +-04.4 +-04.5 +-04.6 +-04.7 +-05.0 +-05.1 +-05.2 +-05.4 +-11.0 +-11.4 +-14.0 +-16.0 +-16.1 +-1a.0 +-1c.0-[07]----00.0 +-1d.0 +-1f.0 +-1f.2 \-1f.3
Signature · 2 lines
???????????????? ????????????????
Tantilov
Quoted 3 lines
-----Original Message-----
From: Fan Du [mailto:fengyuleidian0615 at gmail.com]
Sent: Wednesday, April 01, 2015 8:56 PM
Quoted 17 lines
Cc: Du, Fan; intel-wired-lan at lists.osuosl.org
Subject: [E1000-devel] Fragmented UDP packets triggerrx_missed_errorson 82599EBHi
While investigating a upper level network issue, I found out the root cause may be triggered by packet loss at NIC level, showed by
rx_missed_errors.kernel: linux-2.6.32-358.el6.x86_64
server: iperf -s -B 192.168.5.1 -u
client: iperf -c 192.168.5.1 -u -b 10G -i 1 -t 1000 -P 12 -l 3k Use -l to specify buffers large than MTU to create fragmented IP packets.1. Tune rx ring from 512 to max 4096 does help for single flow, but still got great
rx_missed_errorsfrom multiple flows.
2. Using latest net-next 4.0.0-rc4 shows the same effect.
3. Got 9.4Gbits/sec even thoughrx_missed_errorsshows NIC level packets drop.
Quoted 6 lines
rx_missed_errorsvalue comes from RXMPC, where 82599 data sheet 8.2.3.5.1 says:
"Missed packet interrupt is activated for each received packet that overflows the Rx packet buffer > (overrun). he packet is dropped and also increments the associated RXMPC[n] counter."I'm not sure it means my env is mis-configured or anything I'm missing obviously. Any hints?
In simple terms packets are coming in faster than the interface can receive them. See below.
Quoted 35 lines
Attached several logs as below.
# ethtool -S eth4 NIC statistics: rx_packets: 1047869017 tx_packets: 206275776 rx_bytes: 1103333268576 tx_bytes: 289198212456 rx_pkts_nic: 1047200292 tx_pkts_nic: 206275773 rx_bytes_nic: 1907927064202 tx_bytes_nic: 290023317512 lsc_int: 17 tx_busy: 0 non_eop_descs: 0 rx_errors: 0 tx_errors: 0 rx_dropped: 0 tx_dropped: 0 multicast: 0 broadcast: 4310 rx_no_buffer_count: 0 collisions: 0 rx_over_errors: 0 rx_crc_errors: 0 rx_frame_errors: 0 hw_rsc_aggregated: 0 hw_rsc_flushed: 0 fdir_match: 0 fdir_miss: 6545204 fdir_overflow: 0 rx_fifo_errors: 0 rx_missed_errors: 638609576 <-------- tx_aborted_errors: 0 tx_carrier_errors: 0 tx_fifo_errors: 0
Quoted 8 lines
tx_heartbeat_errors: 0 tx_timeout_count: 0 tx_restart_queue: 0 rx_long_length_errors: 0 rx_short_length_errors: 0 tx_flow_control_xon: 174182 rx_flow_control_xon: 0 tx_flow_control_xoff: 946044
Your interface is generating XOFF packets, which means that it cannot keep up with the
upcoming traffic.
You can disable flow control and look at the stats again - usually it will spill over
to other counters like rx_no_buffer, or rx_no_dma
Quoted 13 lines
# numactl --hardware available: 4 nodes (0-3) node 0 cpus: 0 1 2 3 4 20 21 22 23 24 node 0 size: 24466 MB node 0 free: 22444 MB node 1 cpus: 5 6 7 8 9 25 26 27 28 29 node 1 size: 16384 MB node 1 free: 15831 MB node 2 cpus: 10 11 12 13 14 30 31 32 33 34 node 2 size: 16384 MB node 2 free: 15791 MB node 3 cpus: 15 16 17 18 19 35 36 37 38 39
Quoted 8 lines
node 3 size: 24576 MB node 3 free: 22508 MB node distances: node 0 1 2 3 0: 10 21 31 31 1: 21 10 31 31 2: 31 31 10 21 3: 31 31 21 10
Since you have 4 nodes you may want to check your board layout and try to pin the queues and iperf to the same
node as the network interface. See if that helps.
If you want to debug your numa allocations in more detail, check out this tool:
http://www.intel.com/software/pcm
Thanks,
Emil
? 2015?04?03? 01:52, Tantilov, Emil S ??:
Quoted 21 lines
Quoted 13 lines
Quoted 13 lines
# numactl --hardware available: 4 nodes (0-3) node 0 cpus: 0 1 2 3 4 20 21 22 23 24 node 0 size: 24466 MB node 0 free: 22444 MB node 1 cpus: 5 6 7 8 9 25 26 27 28 29 node 1 size: 16384 MB node 1 free: 15831 MB node 2 cpus: 10 11 12 13 14 30 31 32 33 34 node 2 size: 16384 MB node 2 free: 15791 MB node 3 cpus: 15 16 17 18 19 35 36 37 38 39Quoted 8 lines
Quoted 8 lines
node 3 size: 24576 MB
node 3 free: 22508 MB
node distances:
node 0 1 2 3
0: 10 21 31 31
1: 21 10 31 31
2: 31 31 10 21
3: 31 31 21 10
Quoted 2 lines
Since you have 4 nodes you may want to check your board layout and try to pin the queues and iperf to the same
node as the network interface. See if that helps.
Thanks for the hints.
UDP is used here, so it actually doesn't matter to tie iperf server on the same core for receiving the flow. I didn't launch iperf server, only running iperf client to send *fragmented* UDP packets. On the server side, still ifconfig shows dropped packets increasing, and ethtool -S confirmed with rx_missed_errors also climbing.
Client: iperf -c SEVER_IP -u -b 10G -i 1 -t 100000 -P 12 -l 30k
Server:
kernel: 4.0.0-rc4 , when buffer size(-l) >= 30k, no rx_missed_errors
when buffer size(-l) < 30k, confirmed rx_missed_errors
Server:
kernel: 2.6.32-358 , when buffer size(-l) >= 10k, no rx_missed_errors
when buffer size(-l) < 10k, confirmed rx_missed_errors
Any suggestions?
Quoted 2 lines
If you want to debug your numa allocations in more detail, check out this tool:
http://www.intel.com/software/pcm
Signature · 2 lines
???????????????? ????????????????
Tantilov
Quoted 28 lines
-----Original Message-----
From: Fan Du [mailto:fengyuleidian0615 at gmail.com]
Sent: Monday, April 06, 2015 6:52 PM
Subject: Re: [E1000-devel] Fragmented UDP packets triggerrx_missed_errorson 82599EB? 2015?04?03? 01:52, Tantilov, Emil S ??:
Quoted 22 lines
Quoted 22 lines
Quoted 13 lines
# numactl --hardware available: 4 nodes (0-3) node 0 cpus: 0 1 2 3 4 20 21 22 23 24 node 0 size: 24466 MB node 0 free: 22444 MB node 1 cpus: 5 6 7 8 9 25 26 27 28 29 node 1 size: 16384 MB node 1 free: 15831 MB node 2 cpus: 10 11 12 13 14 30 31 32 33 34 node 2 size: 16384 MB node 2 free: 15791 MB node 3 cpus: 15 16 17 18 19 35 36 37 38 39Quoted 8 lines
node 3 size: 24576 MB node 3 free: 22508 MB node distances: node 0 1 2 3 0: 10 21 31 31 1: 21 10 31 31 2: 31 31 10 21 3: 31 31 21 10Since you have 4 nodes you may want to check your board layout and try to pin the queues and iperf to the same
Quoted 8 lines
Quoted 1 line
node as the network interface. See if that helps.
Thanks for the hints.
UDP is used here, so it actually doesn't matter to tie iperf server on the same core for receiving the flow. I didn't launch iperf server, only running iperf client to send *fragmented* UDP packets. On the server side, still ifconfig shows dropped packets increasing, and ethtool -S confirmed with
rx_missed_errorsalso climbing.
One difference is that fragmented UDP packets would go to a single queue, provided you have UDP RSS enabled.
You can force missed packets by overwhelming the receiver with small UDP packets, so I'm not sure that the UDP packets being fragmented have much to do with it.
Quoted 11 lines
Client: iperf -c
SEVER_IP-u -b 10G -i 1 -t 100000 -P 12 -l 30kServer:
kernel: 4.0.0-rc4 , when buffer size(-l) >= 30k, norx_missed_errors
when buffer size(-l) < 30k, confirmedrx_missed_errorsServer:
kernel: 2.6.32-358 , when buffer size(-l) >= 10k, norx_missed_errors
when buffer size(-l) < 10k, confirmedrx_missed_errorsAny suggestions?
Monitor the stats while running the test and look for patterns (e.g. rx_missed_errors incrementing along with other stats like tx_xoff, queues etc), you may also want to monitor interrupts/sec and see if it changes with the different flows.
Thanks,
Emil