ICMP packets - ll_temac with Microblaze
Subsystems (4)
Hi Eric and David,
I have found one problem with ll_temac driver and
this commit: net: more accurate skb truesize
sha1: 87fb4b7b533073eeeaed0b6bf7c2328995f6c075
The problem is only with icmp packets from the target. It is sent and driver receive it but it is not proceed to the application.
The problem I see is that kmalloc_node_track_caller allocate specific size and then this size is changed by SKB_WITH_OVERHEAD(ksize(data)). The problem is with netdev_alloc_skb_ip_align which calls __alloc_skb function.
Currently driver uses setting for jumbo frames (9k). When I change it to use mtu (1,5k) then everything is ok.
Can you give me some hints what can be wrong?
Thanks,
Michal
Signature · 4 lines
Michal Simek, Ing. (M.Eng) w: www.monstr.eu p: +42-0-721842854 Maintainer of Linux kernel 2.6 Microblaze Linux - http://www.monstr.eu/fdt/ Microblaze U-BOOT custodian
Quoted 9 lines · Michal Simek
Hi Eric and David,
I have found one problem with
ll_temacdriver and
this commit: net: more accurate skb truesize
sha1:87fb4b7b533073eeeaed0b6bf7c2328995f6c075The problem is only with icmp packets from the target. It is sent and driver receive it but it is not proceed to the application.
Quoted 9 lines
The problem I see is that
kmalloc_node_track_callerallocate specific size and then this size is changed by SKB_WITH_OVERHEAD(ksize(data)). The problem is withnetdev_alloc_skb_ip_alignwhich calls__alloc_skbfunction.Currently driver uses setting for jumbo frames (9k). When I change it to use mtu (1,5k) then everything is ok.
Can you give me some hints what can be wrong?
Is it with SLUB, SLAB or SLOB allocator ?
Quoted 14 lines · Eric Dumazet
Quoted 13 lines · Michal Simek
Hi Eric and David,
I have found one problem with
ll_temacdriver and
this commit: net: more accurate skb truesize
sha1:87fb4b7b533073eeeaed0b6bf7c2328995f6c075The problem is only with icmp packets from the target. It is sent and driver receive it but it is not proceed to the application.
The problem I see is that
kmalloc_node_track_callerallocate specific size and then this size is changed by SKB_WITH_OVERHEAD(ksize(data)). The problem is withnetdev_alloc_skb_ip_alignwhich calls__alloc_skbfunction.
Quoted 7 lines
Quoted 5 lines
Currently driver uses setting for jumbo frames (9k). When I change it to use mtu (1,5k) then everything is ok.
Can you give me some hints what can be wrong?
Is it with SLUB, SLAB or SLOB allocator ?
SLAB.
CONFIG_SLAB=y
# CONFIG_SLUB is not set
# CONFIG_SLOB is not set
Michal
Signature · 4 lines
Michal Simek, Ing. (M.Eng) w: www.monstr.eu p: +42-0-721842854 Maintainer of Linux kernel 2.6 Microblaze Linux - http://www.monstr.eu/fdt/ Microblaze U-BOOT custodian
Quoted 14 lines · Eric Dumazet
Quoted 13 lines · Michal Simek
Hi Eric and David,
I have found one problem with
ll_temacdriver and
this commit: net: more accurate skb truesize
sha1:87fb4b7b533073eeeaed0b6bf7c2328995f6c075The problem is only with icmp packets from the target. It is sent and driver receive it but it is not proceed to the application.
The problem I see is that
kmalloc_node_track_callerallocate specific size and then this size is changed by SKB_WITH_OVERHEAD(ksize(data)). The problem is withnetdev_alloc_skb_ip_alignwhich calls__alloc_skbfunction.
Quoted 9 lines
Quoted 5 lines
Currently driver uses setting for jumbo frames (9k). When I change it to use mtu (1,5k) then everything is ok.
Can you give me some hints what can be wrong?
Is it with SLUB, SLAB or SLOB allocator ?
(I am referring to commit bc417e30f8d (net: Add back alignment for size for __alloc_skb)
Quoted 18 lines · Eric Dumazet
Quoted 17 lines · Eric Dumazet
Quoted 16 lines · Michal Simek
Hi Eric and David,
I have found one problem with
ll_temacdriver and
this commit: net: more accurate skb truesize
sha1:87fb4b7b533073eeeaed0b6bf7c2328995f6c075The problem is only with icmp packets from the target. It is sent and driver receive it but it is not proceed to the application.
The problem I see is that
kmalloc_node_track_callerallocate specific size and then this size is changed by SKB_WITH_OVERHEAD(ksize(data)). The problem is withnetdev_alloc_skb_ip_alignwhich calls__alloc_skbfunction.Currently driver uses setting for jumbo frames (9k). When I change it to use mtu (1,5k) then everything is ok.
Quoted 8 lines
Quoted 5 lines
Quoted 2 lines
Can you give me some hints what can be wrong?
Is it with SLUB, SLAB or SLOB allocator ?
(I am referring to commit
bc417e30f8d(net: Add back alignment for size for__alloc_skb)
I have seen that commit. Using the latest&greatest version with this patch.
Michal
Signature · 4 lines
Michal Simek, Ing. (M.Eng) w: www.monstr.eu p: +42-0-721842854 Maintainer of Linux kernel 2.6 Microblaze Linux - http://www.monstr.eu/fdt/ Microblaze U-BOOT custodian
Quoted 21 lines · Michal Simek
Quoted 20 lines · Eric Dumazet
Quoted 19 lines · Eric Dumazet
Quoted 18 lines · Michal Simek
Hi Eric and David,
I have found one problem with
ll_temacdriver and
this commit: net: more accurate skb truesize
sha1:87fb4b7b533073eeeaed0b6bf7c2328995f6c075The problem is only with icmp packets from the target. It is sent and driver receive it but it is not proceed to the application.
The problem I see is that
kmalloc_node_track_callerallocate specific size and then this size is changed by SKB_WITH_OVERHEAD(ksize(data)). The problem is withnetdev_alloc_skb_ip_alignwhich calls__alloc_skbfunction.Currently driver uses setting for jumbo frames (9k). When I change it to use mtu (1,5k) then everything is ok.
Can you give me some hints what can be wrong?
Quoted 9 lines
Quoted 6 lines
Quoted 3 lines
Is it with SLUB, SLAB or SLOB allocator ?
(I am referring to commit
bc417e30f8d(net: Add back alignment for size for__alloc_skb)I have seen that commit. Using the latest&greatest version with this patch.
So after these changes, struct skb_shared_info is located to the very end of allocated memory.
Maybe there is a problem with MTU=9000, since assuming PAGE_SIZE=4096 on your machine (is it ?), we have a 16384 bytes block of memory, and ~16000 available in skb head, instead of ~9000
Quoted 30 lines · Eric Dumazet
Quoted 19 lines · Michal Simek
Quoted 18 lines · Eric Dumazet
Quoted 17 lines · Eric Dumazet
Quoted 16 lines · Michal Simek
Hi Eric and David,
I have found one problem with
ll_temacdriver and
this commit: net: more accurate skb truesize
sha1:87fb4b7b533073eeeaed0b6bf7c2328995f6c075The problem is only with icmp packets from the target. It is sent and driver receive it but it is not proceed to the application.
The problem I see is that
kmalloc_node_track_callerallocate specific size and then this size is changed by SKB_WITH_OVERHEAD(ksize(data)). The problem is withnetdev_alloc_skb_ip_alignwhich calls__alloc_skbfunction.Currently driver uses setting for jumbo frames (9k). When I change it to use mtu (1,5k) then everything is ok.
Quoted 9 lines
Quoted 7 lines
Quoted 5 lines
Quoted 2 lines
Can you give me some hints what can be wrong?
Is it with SLUB, SLAB or SLOB allocator ?
(I am referring to commit
bc417e30f8d(net: Add back alignment for size for__alloc_skb)I have seen that commit. Using the latest&greatest version with this patch.
Quoted 6 lines
So after these changes, struct
skb_shared_infois located to the very end of allocated memory.Maybe there is a problem with MTU=9000, since assuming PAGE_SIZE=4096 on your machine (is it ?), we have a 16384 bytes block of memory, and ~16000 available in skb head, instead of ~9000
yes, page size is 4k.
Here is the log for allocation - origin size for MTU 9000 is 0x2420(9248) and size after overhead is 0x3f40(16192)
kmalloc_node_track_caller size 00002420, SKB_WITH_OVERHEAD(ksize(data)) 00003f40
skb->head c7870000 skb->data c7870000 skb->tail c7870000
skb->truesize 000040c0skb->end c7873f40
Michal
Signature · 4 lines
Michal Simek, Ing. (M.Eng) w: www.monstr.eu p: +42-0-721842854 Maintainer of Linux kernel 2.6 Microblaze Linux - http://www.monstr.eu/fdt/ Microblaze U-BOOT custodian
Quoted 5 lines · Michal Simek
yes, page size is 4k.
Here is the log for allocation - origin size for MTU 9000 is 0x2420(9248) and size after overhead is 0x3f40(16192)
Quoted 7 lines
kmalloc_node_track_callersize 00002420, SKB_WITH_OVERHEAD(ksize(data))00003f40skb->head c7870000 skb->data c7870000 skb->tail c7870000
skb->truesize000040c0skb->endc7873f40
So basically truesize is bigger than previous kernel, and you hit a socket limit. (RCVBUF ?)
truesize is more correct, so we uncover prior bugs.
We should allow at least one packet, regardless of its size :(
Quoted 15 lines · Eric Dumazet
Quoted 5 lines · Michal Simek
yes, page size is 4k.
Here is the log for allocation - origin size for MTU 9000 is 0x2420(9248) and size after overhead is 0x3f40(16192)
Quoted 7 lines
kmalloc_node_track_callersize 00002420, SKB_WITH_OVERHEAD(ksize(data))00003f40skb->head c7870000 skb->data c7870000 skb->tail c7870000
skb->truesize000040c0skb->endc7873f40
Quoted 2 lines
So basically truesize is bigger than previous kernel, and you hit a socket limit. (RCVBUF ?)
Where is that limit setup?
Michal
Signature · 4 lines
Michal Simek, Ing. (M.Eng) w: www.monstr.eu p: +42-0-721842854 Maintainer of Linux kernel 2.6 Microblaze Linux - http://www.monstr.eu/fdt/ Microblaze U-BOOT custodian
Quoted 16 lines · Michal Simek
Quoted 15 lines · Eric Dumazet
Quoted 12 lines · Michal Simek
yes, page size is 4k.
Here is the log for allocation - origin size for MTU 9000 is 0x2420(9248) and size after overhead is 0x3f40(16192)
kmalloc_node_track_callersize 00002420, SKB_WITH_OVERHEAD(ksize(data))00003f40skb->head c7870000 skb->data c7870000 skb->tail c7870000
skb->truesize000040c0skb->endc7873f40
Quoted 5 lines
Quoted 2 lines
So basically truesize is bigger than previous kernel, and you hit a socket limit. (RCVBUF ?)
Where is that limit setup?
Might be in the application itself.
setsockopt(socket, SOL_SOCKET, SO_RCVBUF, &limit, sizeof(limit));
kernel currently apply a factor of two to take into account skb overhead. This might be not enough with jumbo frames.
Quoted 15 lines · Eric Dumazet
Quoted 5 lines · Michal Simek
yes, page size is 4k.
Here is the log for allocation - origin size for MTU 9000 is 0x2420(9248) and size after overhead is 0x3f40(16192)
Quoted 7 lines
kmalloc_node_track_callersize 00002420, SKB_WITH_OVERHEAD(ksize(data))00003f40skb->head c7870000 skb->data c7870000 skb->tail c7870000
skb->truesize000040c0skb->endc7873f40
Quoted 8 lines
So basically truesize is bigger than previous kernel, and you hit a socket limit. (RCVBUF ?)
truesize is more correct, so we uncover prior bugs.
We should allow at least one packet, regardless of its size :(
I would try the following patch :
include/net/sock.h +3 -1
@@ -662,12 +662,14 @@ static inline void __sk_add_backlog(struct sock *sk, struct sk_buff *skb)/* * Take into account size of receive queue and backlog queue +* Do not take into account this skb truesize, +* to allow even a single big packet to come. */ static inline bool sk_rcvqueues_full (const struct sock *sk,const struct sk_buff *skb) {unsigned int qsize = sk->sk_backlog.len +atomic_read (&sk->sk_rmem_alloc); -return qsize + skb->truesize > sk->sk_rcvbuf; +return qsize > sk->sk_rcvbuf; }/* The per-socket spinlock must be held here. */
Quoted 37 lines · Eric Dumazet
Quoted 14 lines · Eric Dumazet
Quoted 5 lines · Michal Simek
yes, page size is 4k.
Here is the log for allocation - origin size for MTU 9000 is 0x2420(9248) and size after overhead is 0x3f40(16192)
Quoted 7 lines
kmalloc_node_track_callersize 00002420, SKB_WITH_OVERHEAD(ksize(data))00003f40skb->head c7870000 skb->data c7870000 skb->tail c7870000
skb->truesize000040c0skb->endc7873f40Quoted 8 lines
So basically truesize is bigger than previous kernel, and you hit a socket limit. (RCVBUF ?)
truesize is more correct, so we uncover prior bugs.
We should allow at least one packet, regardless of its size :(
I would try the following patch :
include/net/sock.h +2
-0@@ -662,12 +662,14 @@ static inline void __sk_add_backlog(struct sock *sk, struct sk_buff *skb)/* * Take into account size of receive queue and backlog queue +* Do not take into account this skb truesize, +* to allow even a single big packet to come.
include/net/sock.h static inline void __sk_add_backlog(struct sock *sk, struct sk_buff *skb) · line 675
Quoted 10 lines
include/net/sock.h +1
-1*/ static inline bool sk_rcvqueues_full (const struct sock *sk,const struct sk_buff *skb) {unsigned int qsize = sk->sk_backlog.len +atomic_read (&sk->sk_rmem_alloc); -return qsize + skb->truesize > sk->sk_rcvbuf; +return qsize > sk->sk_rcvbuf; }/* The per-socket spinlock must be held here. */
It doesn't fix the problem. Do you want to add some printk to code?
Michal
Signature · 4 lines
Michal Simek, Ing. (M.Eng) w: www.monstr.eu p: +42-0-721842854 Maintainer of Linux kernel 2.6 Microblaze Linux - http://www.monstr.eu/fdt/ Microblaze U-BOOT custodian
Quoted 46 lines · Michal Simek
Quoted 45 lines · Eric Dumazet
Quoted 14 lines · Eric Dumazet
Quoted 5 lines · Michal Simek
yes, page size is 4k.
Here is the log for allocation - origin size for MTU 9000 is 0x2420(9248) and size after overhead is 0x3f40(16192)
Quoted 7 lines
kmalloc_node_track_callersize 00002420, SKB_WITH_OVERHEAD(ksize(data))00003f40skb->head c7870000 skb->data c7870000 skb->tail c7870000
skb->truesize000040c0skb->endc7873f40Quoted 8 lines
So basically truesize is bigger than previous kernel, and you hit a socket limit. (RCVBUF ?)
truesize is more correct, so we uncover prior bugs.
We should allow at least one packet, regardless of its size :(
I would try the following patch :
include/net/sock.h +3
-1@@ -662,12 +662,14 @@ static inline void __sk_add_backlog(struct sock *sk, struct sk_buff *skb)/* * Take into account size of receive queue and backlog queue +* Do not take into account this skb truesize, +* to allow even a single big packet to come. */ static inline bool sk_rcvqueues_full (const struct sock *sk,const struct sk_buff *skb) {unsigned int qsize = sk->sk_backlog.len +atomic_read (&sk->sk_rmem_alloc); -return qsize + skb->truesize > sk->sk_rcvbuf; +return qsize > sk->sk_rcvbuf; }
Quoted 9 lines
include/net/sock.h static inline void __sk_add_backlog(struct sock *sk, struct sk_buff *skb) · line 675
Quoted 2 lines
include/net/sock.h +0
-0/* The per-socket spinlock must be held here. */ It doesn't fix the problem. Do you want to add some printk to code?
Michal
try to change SOCK_MIN_RCVBUF
include/net/sock.h
#define SOCK_MIN_RCVBUF (2048 + sizeof(struct sk_buff))
->
#define SOCK_MIN_RCVBUF (16384 + sizeof(struct sk_buff))
Quoted 57 lines · Eric Dumazet
Quoted 45 lines · Michal Simek
Quoted 44 lines · Eric Dumazet
Quoted 14 lines · Eric Dumazet
Quoted 5 lines · Michal Simek
yes, page size is 4k.
Here is the log for allocation - origin size for MTU 9000 is 0x2420(9248) and size after overhead is 0x3f40(16192)
Quoted 7 lines
kmalloc_node_track_callersize 00002420, SKB_WITH_OVERHEAD(ksize(data))00003f40skb->head c7870000 skb->data c7870000 skb->tail c7870000
skb->truesize000040c0skb->endc7873f40Quoted 8 lines
So basically truesize is bigger than previous kernel, and you hit a socket limit. (RCVBUF ?)
truesize is more correct, so we uncover prior bugs.
We should allow at least one packet, regardless of its size :(
I would try the following patch :
include/net/sock.h +3
-1@@ -662,12 +662,14 @@ static inline void __sk_add_backlog(struct sock *sk, struct sk_buff *skb)/* * Take into account size of receive queue and backlog queue +* Do not take into account this skb truesize, +* to allow even a single big packet to come. */ static inline bool sk_rcvqueues_full (const struct sock *sk,const struct sk_buff *skb) {unsigned int qsize = sk->sk_backlog.len +atomic_read (&sk->sk_rmem_alloc); -return qsize + skb->truesize > sk->sk_rcvbuf; +return qsize > sk->sk_rcvbuf; }Quoted 8 lines
include/net/sock.h static inline void __sk_add_backlog(struct sock *sk, struct sk_buff *skb) · line 675
Quoted 2 lines
include/net/sock.h +0
-0/* The per-socket spinlock must be held here. */ It doesn't fix the problem. Do you want to add some printk to code?
Michal
try to change
SOCK_MIN_RCVBUF
Quoted 8 lines
include/net/sock.h#define
SOCK_MIN_RCVBUF(2048 +sizeof(struct sk_buff))->
#define
SOCK_MIN_RCVBUF(16384 +sizeof(struct sk_buff))
Doesn't work. :-(
Michal
Signature · 4 lines
Michal Simek, Ing. (M.Eng) w: www.monstr.eu p: +42-0-721842854 Maintainer of Linux kernel 2.6 Microblaze Linux - http://www.monstr.eu/fdt/ Microblaze U-BOOT custodian
Quoted 5 lines · Michal Simek
Quoted 2 lines
#define
SOCK_MIN_RCVBUF(16384 +sizeof(struct sk_buff))Doesn't work. :-(
Sorry, I have no idea where packet is dropped, I dont know your application.
You might send "netstat -s" output if you cant find it ...
You spoke of ICMP message, I cant find a limit in this area ?
Quoted 9 lines · Eric Dumazet
Quoted 4 lines · Michal Simek
Quoted 2 lines
#define
SOCK_MIN_RCVBUF(16384 +sizeof(struct sk_buff))Doesn't work. :-(
Sorry, I have no idea where packet is dropped, I dont know your application.
Application is simple ping from Busybox - nothing special.
Quoted 1 line
You might send "netstat -s" output if you cant find it ...
~ # ./netstat -s
Ip:
36 total packets received
2 with invalid addresses
0 forwarded
0 incoming packets discarded
34 incoming packets delivered
34 requests sent out
Icmp:
34 ICMP messages received
0 input ICMP message failed.
ICMP input histogram:
echo replies: 34
34 ICMP messages sent
0 ICMP messages failed
ICMP output histogram:
echo request: 34
IcmpMsg:
InType0: 34
OutType8: 34Tcp:
0 active connections openings
0 passive connection openings
0 failed connection attempts
0 connection resets received
0 connections established
0 segments received
0 segments send out
0 segments retransmited
0 bad segments received.
0 resets sent
Udp:
0 packets received
0 packets to unknown port received.
0 packet receive errors
0 packets sent
RcvbufErrors: 0
SndbufErrors: 0
UdpLite:
InDatagrams: 0
NoPorts: 0
InErrors: 0
OutDatagrams: 0
RcvbufErrors: 0
SndbufErrors: 0
TcpExt:
ArpFilter: 0
0 packets header predicted
TCPPureAcks: 0
TCPHPAcks: 0
TCPRenoRecovery: 0
TCPSackRecovery: 0
TCPSACKReneging: 0
TCPFACKReorder: 0
TCPSACKReorder: 0
TCPRenoReorder: 0
TCPTSReorder: 0
TCPFullUndo: 0
TCPPartialUndo: 0
TCPDSACKUndo: 0
TCPLossUndo: 0
TCPLoss: 0
TCPLostRetransmit: 0
TCPRenoFailures: 0
TCPSackFailures: 0
TCPLossFailures: 0
TCPFastRetrans: 0
TCPForwardRetrans: 0
TCPSlowStartRetrans: 0
TCPTimeouts: 0
TCPRenoRecoveryFail: 0
TCPSackRecoveryFail: 0
TCPSchedulerFailed: 0
TCPRcvCollapsed: 0
TCPDSACKOldSent: 0
TCPDSACKOfoSent: 0
TCPDSACKRecv: 0
TCPDSACKOfoRecv: 0
TCPAbortOnSyn: 0
TCPAbortOnData: 0
TCPAbortOnClose: 0
TCPAbortOnMemory: 0
TCPAbortOnTimeout: 0
TCPAbortOnLinger: 0
TCPAbortFailed: 0
TCPMemoryPressures: 0
TCPSACKDiscard: 0
TCPDSACKIgnoredOld: 0
TCPDSACKIgnoredNoUndo: 0
TCPSpuriousRTOs: 0
TCPMD5NotFound: 0
TCPMD5Unexpected: 0
TCPSackShifted: 0
TCPSackMerged: 0
TCPSackShiftFallback: 0
TCPBacklogDrop: 0
TCPMinTTLDrop: 0
TCPDeferAcceptDrop: 0
IPReversePathFilter: 0
TCPTimeWaitOverflow: 0
TCPReqQFullDoCookies: 0
TCPReqQFullDrop: 0
IpExt:
InNoRoutes: 0
InTruncatedPkts: 0
InMcastPkts: 0
OutMcastPkts: 0
InBcastPkts: 0
OutBcastPkts: 0
InOctets: 4008
OutOctets: 2856
InMcastOctets: 0
OutMcastOctets: 0
InBcastOctets: 0
OutBcastOctets: 0Quoted 1 line
You spoke of ICMP message, I cant find a limit in this area ?
ok. Can you provide me any background why size should be setup by size = SKB_WITH_OVERHEAD(ksize(data));
and not to use size which is passed to kmalloc in __alloc_skb.
In past I have seen some issues because of caches but this is not that case because on QEMU (where cache is not modeled) everything works the same as on HW.
Below is the log what I see.
Thanks,
Michal
~ # ping -c 5 192.168.0.101
PING 192.168.0.101 (192.168.0.101): 56 data bytes
--- 192.168.0.101 ping statistics ---
5 packets transmitted, 0 packets received, 100% packet loss ~ # mount -t nfs -o nolock -o tcp 192.168.0.101:/tftpboot/nfs /mnt ~ # ls /mnt/ | head
0001-sd.patch 0001-syscalls-getdtablesize01-fix-for-missing-etc-hosts.patch 1
1-xils
2
arp
arp-mb
busybox
core
exec
~ # umount /mnt/
~ # ./netstat -s
Ip:
36 total packets received
2 with invalid addresses
0 forwarded
0 incoming packets discarded
34 incoming packets delivered
43 requests sent out
Icmp:
5 ICMP messages received
0 input ICMP message failed.
ICMP input histogram:
echo replies: 5
5 ICMP messages sent
0 ICMP messages failed
ICMP output histogram:
echo request: 5
IcmpMsg:
InType0: 5
OutType8: 5Tcp:
4 active connections openings
0 passive connection openings
0 failed connection attempts
0 connection resets received
0 connections established
29 segments received
38 segments send out
0 segments retransmited
0 bad segments received.
0 resets sent
Udp:
0 packets received
0 packets to unknown port received.
0 packet receive errors
0 packets sent
RcvbufErrors: 0
SndbufErrors: 0
UdpLite:
InDatagrams: 0
NoPorts: 0
InErrors: 0
OutDatagrams: 0
RcvbufErrors: 0
SndbufErrors: 0
TcpExt:
ArpFilter: 0
1 delayed acks sent
10 packets header predicted
TCPPureAcks: 4
TCPHPAcks: 6
TCPRenoRecovery: 0
TCPSackRecovery: 0
TCPSACKReneging: 0
TCPFACKReorder: 0
TCPSACKReorder: 0
TCPRenoReorder: 0
TCPTSReorder: 0
TCPFullUndo: 0
TCPPartialUndo: 0
TCPDSACKUndo: 0
TCPLossUndo: 0
TCPLoss: 0
TCPLostRetransmit: 0
TCPRenoFailures: 0
TCPSackFailures: 0
TCPLossFailures: 0
TCPFastRetrans: 0
TCPForwardRetrans: 0
TCPSlowStartRetrans: 0
TCPTimeouts: 0
TCPRenoRecoveryFail: 0
TCPSackRecoveryFail: 0
TCPSchedulerFailed: 0
TCPRcvCollapsed: 0
TCPDSACKOldSent: 0
TCPDSACKOfoSent: 0
TCPDSACKRecv: 0
TCPDSACKOfoRecv: 0
TCPAbortOnSyn: 0
TCPAbortOnData: 0
TCPAbortOnClose: 0
TCPAbortOnMemory: 0
TCPAbortOnTimeout: 0
TCPAbortOnLinger: 0
TCPAbortFailed: 0
TCPMemoryPressures: 0
TCPSACKDiscard: 0
TCPDSACKIgnoredOld: 0
TCPDSACKIgnoredNoUndo: 0
TCPSpuriousRTOs: 0
TCPMD5NotFound: 0
TCPMD5Unexpected: 0
TCPSackShifted: 0
TCPSackMerged: 0
TCPSackShiftFallback: 0
TCPBacklogDrop: 0
TCPMinTTLDrop: 0
TCPDeferAcceptDrop: 0
IPReversePathFilter: 0
TCPTimeWaitOverflow: 0
TCPReqQFullDoCookies: 0
TCPReqQFullDrop: 0
IpExt:
InNoRoutes: 0
InTruncatedPkts: 0
InMcastPkts: 0
OutMcastPkts: 0
InBcastPkts: 0
OutBcastPkts: 0
InOctets: 11348
OutOctets: 3752
InMcastOctets: 0
OutMcastOctets: 0
InBcastOctets: 0
OutBcastOctets: 0~ #
~ #
~ # tcpdump
device eth0 entered promiscuous mode
tcpdump: listening on eth0
00:01:31.421191 192.168.0.101 > 192.168.0.100: icmp: echo request (DF) 00:01:31.479686 192.168.0.100 > 192.168.0.101: icmp: echo reply 00:01:31.442325 arp who-has 192.168.0.1 tell 192.168.0.100 00:01:31.446052 arp reply 192.168.0.1 is-at f4:ec:38:b2:6a:9e 00:01:31.446256 192.168.0.100.47044 > 192.168.0.1.53: 45191+ (44) (DF) 00:01:31.461859 192.168.0.1.53 > 192.168.0.100.47044: 45191 NXDomain 0/1/0 (121) 00:01:31.464125 192.168.0.100.41833 > 192.168.0.1.53: 37485+ (44) (DF) 00:01:31.478026 192.168.0.1.53 > 192.168.0.100.41833: 37485 NXDomain 0/1/0 (121) 00:01:31.481794 192.168.0.100.35138 > 192.168.0.1.53: 56594+ (42) (DF) 00:01:31.497202 192.168.0.1.53 > 192.168.0.100.35138: 56594 NXDomain 0/1/0 (119) 00:01:32.419130 192.168.0.101 > 192.168.0.100: icmp: echo request (DF) 00:01:32.419482 192.168.0.100 > 192.168.0.101: icmp: echo reply 00:01:33.418342 192.168.0.101 > 192.168.0.100: icmp: echo request (DF) 00:01:33.418693 192.168.0.100 > 192.168.0.101: icmp: echo reply 00:01:34.419327 192.168.0.101 > 192.168.0.100: icmp: echo request (DF) 00:01:34.419676 192.168.0.100 > 192.168.0.101: icmp: echo reply 00:01:35.418042 192.168.0.101 > 192.168.0.100: icmp: echo request (DF) 00:01:35.418406 192.168.0.100 > 192.168.0.101: icmp: echo reply 00:01:36.418923 192.168.0.101 > 192.168.0.100: icmp: echo request (DF) 00:01:36.419275 192.168.0.100 > 192.168.0.101: icmp: echo reply 00:01:36.459936 arp who-has 192.168.0.100 tell 192.168.0.1 00:01:36.460216 arp reply 192.168.0.100 is-at 0:a:35:0:19:15 00:01:37.418800 192.168.0.101 > 192.168.0.100: icmp: echo request (DF) 00:01:37.419149 192.168.0.100 > 192.168.0.101: icmp: echo reply 00:01:38.419247 192.168.0.101 > 192.168.0.100: icmp: echo request (DF) 00:01:38.419617 192.168.0.100 > 192.168.0.101: icmp: echo reply 00:01:39.418820 192.168.0.101 > 192.168.0.100: icmp: echo request (DF) 00:01:39.419169 192.168.0.100 > 192.168.0.101: icmp: echo reply
56 packets received by filter
0 packets droppeddevice eth0 left promiscuous mode
by kernel
~ #
~ # ping 192.168.0.101
PING 192.168.0.101 (192.168.0.101): 56 data bytes
--- 192.168.0.101 ping statistics ---
2 packets transmitted, 0 packets received, 100% packet loss
~ # ping 192.168.0.1
PING 192.168.0.1 (192.168.0.1): 56 data bytes
--- 192.168.0.1 ping statistics ---
1 packets transmitted, 0 packets received, 100% packet loss
~ # ping 192.168.0.101 -c 10 &
~ # PING 192.168.0.101 (192.168.0.101): 56 data bytes
~ # tcpdump
device eth0 entered promiscuous mode
tcpdump: listening on eth0
00:01:59.615918 192.168.0.100 > 192.168.0.101: icmp: echo request (DF) 00:01:59.619166 192.168.0.101 > 192.168.0.100: icmp: echo reply 00:01:59.636473 192.168.0.100.57212 > 192.168.0.1.53: 43026+ (44) (DF) 00:01:59.650838 192.168.0.1.53 > 192.168.0.100.57212: 43026 NXDomain 0/1/0 (121) 00:01:59.652915 192.168.0.100.41475 > 192.168.0.1.53: 28371+ (44) (DF) 00:01:59.666907 192.168.0.1.53 > 192.168.0.100.41475: 28371 NXDomain 0/1/0 (121) 00:01:59.670832 192.168.0.100.47770 > 192.168.0.1.53: 61908+ (42) (DF) 00:01:59.684921 192.168.0.1.53 > 192.168.0.100.47770: 61908 NXDomain 0/1/0 (119) 00:02:00.616493 192.168.0.100 > 192.168.0.101: icmp: echo request (DF) 00:02:00.620237 192.168.0.101 > 192.168.0.100: icmp: echo reply 00:02:00.624275 arp who-has 192.168.0.101 tell 192.168.0.100 00:02:00.627936 arp reply 192.168.0.101 is-at 0:22:15:fb:34:a1 00:02:01.617232 192.168.0.100 > 192.168.0.101: icmp: echo request (DF) 00:02:01.620980 192.168.0.101 > 192.168.0.100: icmp: echo reply 00:02:02.617962 192.168.0.100 > 192.168.0.101: icmp: echo request (DF) 00:02:02.621687 192.168.0.101 > 192.168.0.100: icmp: echo reply 00:02:03.618687 192.168.0.100 > 192.168.0.101: icmp: echo request (DF) 00:02:03.622389 192.168.0.101 > 192.168.0.100: icmp: echo reply 00:02:04.619414 192.168.0.100 > 192.168.0.101: icmp: echo request (DF) 00:02:04.623129 192.168.0.101 > 192.168.0.100: icmp: echo reply
42 packets received by filter
0 packets droppeddevice eth0 left promiscuous mode
by kernel
~ #
--- 192.168.0.101 ping statistics ---
10 packets transmitted, 0 packets received, 100% packet loss
[1] + Done(1) ping 192.168.0.101 -c 10
~ #
Signature · 4 lines
Michal Simek, Ing. (M.Eng) w: www.monstr.eu p: +42-0-721842854 Maintainer of Linux kernel 2.6 Microblaze Linux - http://www.monstr.eu/fdt/ Microblaze U-BOOT custodian
Quoted 3 lines · Michal Simek
ok. Can you provide me any background why size should be setup by size = SKB_WITH_OVERHEAD(ksize(data));
and not to use size which is passed to kmalloc in__alloc_skb.
Its all about memory accounting (based on skb->truesize)
Prior to the patch, we could fool memory accounting because skbs claimed to use less memory than what they really used.
And crash machines eventually.
Now memory accouting is fixed, we probably need to change some points in the kernel, where we previously accepted a small skb, but not a very large one.
Since "ping" probably uses SOCK_RAW sockets, I'll try this one :
(We dont care of _this_ skb truesize, only on the count of previously queued packets)
net/packet/af_packet.c +2 -3
@@ -1631,8 +1631,7 @@ static int packet_rcv(struct sk_buff *skb, struct net_device *dev,if (snaplen > res) snaplen = res; -if (atomic_read (&sk->sk_rmem_alloc) + skb->truesize >= - (unsigned )sk->sk_rcvbuf) +if (atomic_read (&sk->sk_rmem_alloc) >= (unsigned )sk->sk_rcvbuf) goto drop_n_acct;if (skb_shared (skb)) {
@@ -1763,7 +1762,7 @@ static int tpacket_rcv(struct sk_buff *skb, struct net_device *dev,if (po->tp_version <=TPACKET_V2 ) {if (macoff + snaplen > po->rx_ring.frame_size) {if (po->copy_thresh && -atomic_read (&sk->sk_rmem_alloc) + skb->truesize +atomic_read (&sk->sk_rmem_alloc) < (unsigned )sk->sk_rcvbuf) {if (skb_shared (skb)) { copy_skb =skb_clone (skb,GFP_ATOMIC );
Quoted 42 lines · Eric Dumazet
Quoted 3 lines · Michal Simek
ok. Can you provide me any background why size should be setup by size = SKB_WITH_OVERHEAD(ksize(data));
and not to use size which is passed to kmalloc in__alloc_skb.Its all about memory accounting (based on
skb->truesize)Prior to the patch, we could fool memory accounting because skbs claimed to use less memory than what they really used.
And crash machines eventually.
Now memory accouting is fixed, we probably need to change some points in the kernel, where we previously accepted a small skb, but not a very large one.
Since "ping" probably uses
SOCK_RAWsockets, I'll try this one :(We dont care of _this_ skb truesize, only on the count of previously queued packets)
net/packet/af_packet.c +1
-2@@ -1631,8 +1631,7 @@ static int packet_rcv(struct sk_buff *skb, struct net_device *dev,if (snaplen > res) snaplen = res; -if (atomic_read (&sk->sk_rmem_alloc) + skb->truesize >= - (unsigned )sk->sk_rcvbuf) +if (atomic_read (&sk->sk_rmem_alloc) >= (unsigned )sk->sk_rcvbuf) goto drop_n_acct;if (skb_shared (skb)) {
net/packet/af_packet.c static int tpacket_rcv(struct sk_buff *skb, struct net_device *dev, · line 1768
Quoted 8 lines
net/packet/af_packet.c +1
-1@@ -1763,7 +1762,7 @@ static int tpacket_rcv(struct sk_buff *skb, struct net_device *dev,if (po->tp_version <=TPACKET_V2 ) {if (macoff + snaplen > po->rx_ring.frame_size) {if (po->copy_thresh && -atomic_read (&sk->sk_rmem_alloc) + skb->truesize +atomic_read (&sk->sk_rmem_alloc) < (unsigned )sk->sk_rcvbuf) {if (skb_shared (skb)) { copy_skb =skb_clone (skb,GFP_ATOMIC );
It doesn't work too.
It is possible to see this behavior on qemu. What about if I prepare you package with cross toolchain, rootfs and you can add debug message where you want?
I have also tried ll_temac driver with ppc440 and behavior is the same.
Max FRAME_SIZE pass to netdev_alloc_skb_ip_align is 7966. For this value ping works. (For ll_temac driver it is #define XTE_JUMBO_MTU 7948 from ll_temac.h)
Thanks,
Michal
Signature · 4 lines
Michal Simek, Ing. (M.Eng) w: www.monstr.eu p: +42-0-721842854 Maintainer of Linux kernel 2.6 Microblaze Linux - http://www.monstr.eu/fdt/ Microblaze U-BOOT custodian
Quoted 56 lines · Michal Simek
Quoted 42 lines · Eric Dumazet
Quoted 3 lines · Michal Simek
ok. Can you provide me any background why size should be setup by size = SKB_WITH_OVERHEAD(ksize(data));
and not to use size which is passed to kmalloc in__alloc_skb.Its all about memory accounting (based on
skb->truesize)Prior to the patch, we could fool memory accounting because skbs claimed to use less memory than what they really used.
And crash machines eventually.
Now memory accouting is fixed, we probably need to change some points in the kernel, where we previously accepted a small skb, but not a very large one.
Since "ping" probably uses
SOCK_RAWsockets, I'll try this one :(We dont care of _this_ skb truesize, only on the count of previously queued packets)
net/packet/af_packet.c +1
-2@@ -1631,8 +1631,7 @@ static int packet_rcv(struct sk_buff *skb, struct net_device *dev,if (snaplen > res) snaplen = res; -if (atomic_read (&sk->sk_rmem_alloc) + skb->truesize >= - (unsigned )sk->sk_rcvbuf) +if (atomic_read (&sk->sk_rmem_alloc) >= (unsigned )sk->sk_rcvbuf) goto drop_n_acct;if (skb_shared (skb)) {net/packet/af_packet.c static int tpacket_rcv(struct sk_buff *skb, struct net_device *dev, · line 1768
Quoted 8 lines
net/packet/af_packet.c +1
-1@@ -1763,7 +1762,7 @@ static int tpacket_rcv(struct sk_buff *skb, struct net_device *dev,if (po->tp_version <=TPACKET_V2 ) {if (macoff + snaplen > po->rx_ring.frame_size) {if (po->copy_thresh && -atomic_read (&sk->sk_rmem_alloc) + skb->truesize +atomic_read (&sk->sk_rmem_alloc) < (unsigned )sk->sk_rcvbuf) {if (skb_shared (skb)) { copy_skb =skb_clone (skb,GFP_ATOMIC );It doesn't work too.
It is possible to see this behavior on qemu. What about if I prepare you package with cross toolchain, rootfs and you can add debug message where you want?
Quoted 4 lines
I have also tried
ll_temacdriver with ppc440 and behavior is the same.Max
FRAME_SIZEpass tonetdev_alloc_skb_ip_alignis 7966. For this value ping works. (Forll_temacdriver it is #defineXTE_JUMBO_MTU7948 fromll_temac.h)
I did several tests with MTU 9000 on my machines and it works well.
It seems my pings (iputils-sss20071127 or iputils-sss20101006) uses a big enough RCVBUF
setsockopt(3, SOL_SOCKET, SO_RCVBUF, [65536], 4) = 0 getsockopt(3, SOL_SOCKET, SO_RCVBUF, [131072], [4]) = 0
Could you check with "strace ping..." what is doing busybox ?
Quoted 64 lines · Eric Dumazet
Quoted 56 lines · Michal Simek
Quoted 42 lines · Eric Dumazet
Quoted 3 lines · Michal Simek
ok. Can you provide me any background why size should be setup by size = SKB_WITH_OVERHEAD(ksize(data));
and not to use size which is passed to kmalloc in__alloc_skb.Its all about memory accounting (based on
skb->truesize)Prior to the patch, we could fool memory accounting because skbs claimed to use less memory than what they really used.
And crash machines eventually.
Now memory accouting is fixed, we probably need to change some points in the kernel, where we previously accepted a small skb, but not a very large one.
Since "ping" probably uses
SOCK_RAWsockets, I'll try this one :(We dont care of _this_ skb truesize, only on the count of previously queued packets)
net/packet/af_packet.c +1
-2@@ -1631,8 +1631,7 @@ static int packet_rcv(struct sk_buff *skb, struct net_device *dev,if (snaplen > res) snaplen = res; -if (atomic_read (&sk->sk_rmem_alloc) + skb->truesize >= - (unsigned )sk->sk_rcvbuf) +if (atomic_read (&sk->sk_rmem_alloc) >= (unsigned )sk->sk_rcvbuf) goto drop_n_acct;if (skb_shared (skb)) {net/packet/af_packet.c static int tpacket_rcv(struct sk_buff *skb, struct net_device *dev, · line 1768
Quoted 8 lines
net/packet/af_packet.c +1
-1@@ -1763,7 +1762,7 @@ static int tpacket_rcv(struct sk_buff *skb, struct net_device *dev,if (po->tp_version <=TPACKET_V2 ) {if (macoff + snaplen > po->rx_ring.frame_size) {if (po->copy_thresh && -atomic_read (&sk->sk_rmem_alloc) + skb->truesize +atomic_read (&sk->sk_rmem_alloc) < (unsigned )sk->sk_rcvbuf) {if (skb_shared (skb)) { copy_skb =skb_clone (skb,GFP_ATOMIC );It doesn't work too.
It is possible to see this behavior on qemu. What about if I prepare you package with cross toolchain, rootfs and you can add debug message where you want?Quoted 4 lines
I have also tried
ll_temacdriver with ppc440 and behavior is the same.Max
FRAME_SIZEpass tonetdev_alloc_skb_ip_alignis 7966. For this value ping works. (Forll_temacdriver it is #defineXTE_JUMBO_MTU7948 fromll_temac.h)I did several tests with MTU 9000 on my machines and it works well.
Quoted 7 lines
It seems my pings (iputils-sss20071127 or iputils-sss20101006) uses a big enough RCVBUF
setsockopt(3, SOL_SOCKET, SO_RCVBUF, [65536], 4)= 0getsockopt(3, SOL_SOCKET, SO_RCVBUF, [131072], [4])= 0Could you check with "strace ping..." what is doing busybox ?
I found it : Its too small for jumbo frames.
setsockopt(3, SOL_SOCKET, SO_RCVBUF, [7280], 4) = 0
networking/ping.c
/* set recv buf (needed if we can get lots of responses: flood ping,
* broadcast ping etc) */
sockopt = (datalen * 2) + 7 * 1024; /* giving it a bit of extra room */ setsockopt(pingsock, SOL_SOCKET, SO_RCVBUF, &sockopt, sizeof(sockopt));
Quoted 80 lines · Eric Dumazet
Quoted 64 lines · Eric Dumazet
Quoted 56 lines · Michal Simek
Quoted 42 lines · Eric Dumazet
Quoted 3 lines · Michal Simek
ok. Can you provide me any background why size should be setup by size = SKB_WITH_OVERHEAD(ksize(data));
and not to use size which is passed to kmalloc in__alloc_skb.Its all about memory accounting (based on
skb->truesize)Prior to the patch, we could fool memory accounting because skbs claimed to use less memory than what they really used.
And crash machines eventually.
Now memory accouting is fixed, we probably need to change some points in the kernel, where we previously accepted a small skb, but not a very large one.
Since "ping" probably uses
SOCK_RAWsockets, I'll try this one :(We dont care of _this_ skb truesize, only on the count of previously queued packets)
net/packet/af_packet.c +1
-2@@ -1631,8 +1631,7 @@ static int packet_rcv(struct sk_buff *skb, struct net_device *dev,if (snaplen > res) snaplen = res; -if (atomic_read (&sk->sk_rmem_alloc) + skb->truesize >= - (unsigned )sk->sk_rcvbuf) +if (atomic_read (&sk->sk_rmem_alloc) >= (unsigned )sk->sk_rcvbuf) goto drop_n_acct;if (skb_shared (skb)) {net/packet/af_packet.c static int tpacket_rcv(struct sk_buff *skb, struct net_device *dev, · line 1768
Quoted 8 lines
net/packet/af_packet.c +1
-1@@ -1763,7 +1762,7 @@ static int tpacket_rcv(struct sk_buff *skb, struct net_device *dev,if (po->tp_version <=TPACKET_V2 ) {if (macoff + snaplen > po->rx_ring.frame_size) {if (po->copy_thresh && -atomic_read (&sk->sk_rmem_alloc) + skb->truesize +atomic_read (&sk->sk_rmem_alloc) < (unsigned )sk->sk_rcvbuf) {if (skb_shared (skb)) { copy_skb =skb_clone (skb,GFP_ATOMIC );It doesn't work too.
It is possible to see this behavior on qemu. What about if I prepare you package with cross toolchain, rootfs and you can add debug message where you want?Quoted 4 lines
I have also tried
ll_temacdriver with ppc440 and behavior is the same.Max
FRAME_SIZEpass tonetdev_alloc_skb_ip_alignis 7966. For this value ping works. (Forll_temacdriver it is #defineXTE_JUMBO_MTU7948 fromll_temac.h)I did several tests with MTU 9000 on my machines and it works well.
Quoted 7 lines
It seems my pings (iputils-sss20071127 or iputils-sss20101006) uses a big enough RCVBUF
setsockopt(3, SOL_SOCKET, SO_RCVBUF, [65536], 4)= 0getsockopt(3, SOL_SOCKET, SO_RCVBUF, [131072], [4])= 0Could you check with "strace ping..." what is doing busybox ?
I found it : Its too small for jumbo frames.
setsockopt(3, SOL_SOCKET, SO_RCVBUF, [7280], 4)= 0
networking/ping.c
Quoted 8 lines
/* set recv buf (needed if we can get lots of responses: flood ping,
* broadcast ping etc) */
sockopt = (datalen * 2) + 7 * 1024; /* giving it a bit of extra room */setsockopt(pingsock, SOL_SOCKET, SO_RCVBUF, &sockopt, sizeof(sockopt));--
Why receive buffer size MUST bigger than the jumbo frames size even if the packet size is small?
Quoted 3 lines
To unsubscribe from this list: send the line "unsubscribe netdev" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 2 lines · Jun Zhao
Why receive buffer size MUST bigger than the jumbo frames size even if the packet size is small?
Thats the way it is in linux.
We count the memory size used by the packet.
If not, a malicious attacker could send 1-byte frames and exhaust your kernel memory.
If you want to reduce it, you might use copybreak : Some drivers copy the data into a small skb instead of providing a jumbo frame to upper stack.
tg3 for example.
Quoted 9 lines · Eric Dumazet
Quoted 2 lines · Jun Zhao
Why receive buffer size MUST bigger than the jumbo frames size even if the packet size is small?
Thats the way it is in linux.
We count the memory size used by the packet.
Quoted 11 lines
If not, a malicious attacker could send 1-byte frames and exhaust your kernel memory.
If you want to reduce it, you might use copybreak : Some drivers copy the data into a small skb instead of providing a jumbo frame to upper stack.
tg3 for example.
Eric Dumazet:
I got it, thanks for you explanation.
Quoted 3 lines
If not, a malicious attacker could send 1-byte frames and exhaust your kernel memory.
That used to happen in SVR4 - it was possible for single byte TCP (etc) data to be queued at a STREAM head (which counted actual data bytes) in a 2k message block.
Actually we also managed to use all kernel memory queueing zero sized STREAMS messages.
Quoted 3 lines
If you want to reduce it, you might use copybreak : Some drivers copy the data into a small skb instead of providing a jumbo frame to upper stack.
Many, many moons ago I wrote an ethernet driver that received into an array of 128 (mostly) 512byte buffers. Full sized frames would have multiple rx ring entries, but could almost always be copied into a correctly
sized buffer with a single aligned copy (cache-line aligned if useful). This was significantly faster than other schemes - especially on systems where the iommu needed setting to allow the ethernet hardware to access memory.
I don't know if the linux skb buffers would allow the ethernet driver to use (say) 1600 byte rx buffers, and link them together when a long frame arrives. Most ethernet HW I've seen will fragment long receives.
The painful hardware is that which enforces a 4n byte alignment on the rx buffer! - on systens that can't do misaligned accesses. 2 bytes of junk would be fine!
David
Quoted 9 lines · David Laight
Many, many moons ago I wrote an ethernet driver that received into an array of 128 (mostly) 512byte buffers. Full sized frames would have multiple rx ring entries, but could almost always be copied into a correctly
sized buffer with a single aligned copy (cache-line aligned if useful). This was significantly faster than other schemes - especially on systems where the iommu needed setting to allow the ethernet hardware to access memory.
Quoted 8 lines
I don't know if the linux skb buffers would allow the ethernet driver to use (say) 1600 byte rx buffers, and link them together when a long frame arrives. Most ethernet HW I've seen will fragment long receives.
The painful hardware is that which enforces a 4n byte alignment on the rx buffer! - on systens that can't do misaligned accesses. 2 bytes of junk would be fine!
It all depends on hardware capabilities.
Old hardware required a single area per frame. So driver had to talk to the hardware the same way.
We now have hardware able to split data into several frags to reduce the overhead.
(Even using different pools to get a low number of frags, and good filling ratio)
Take a look at NIU for example.
Quoted 56 lines · Michal Simek
Quoted 42 lines · Eric Dumazet
Quoted 3 lines · Michal Simek
ok. Can you provide me any background why size should be setup by size = SKB_WITH_OVERHEAD(ksize(data));
and not to use size which is passed to kmalloc in__alloc_skb.Its all about memory accounting (based on
skb->truesize)Prior to the patch, we could fool memory accounting because skbs claimed to use less memory than what they really used.
And crash machines eventually.
Now memory accouting is fixed, we probably need to change some points in the kernel, where we previously accepted a small skb, but not a very large one.
Since "ping" probably uses
SOCK_RAWsockets, I'll try this one :(We dont care of _this_ skb truesize, only on the count of previously queued packets)
net/packet/af_packet.c +1
-2@@ -1631,8 +1631,7 @@ static int packet_rcv(struct sk_buff *skb, struct net_device *dev,if (snaplen > res) snaplen = res; -if (atomic_read (&sk->sk_rmem_alloc) + skb->truesize >= - (unsigned )sk->sk_rcvbuf) +if (atomic_read (&sk->sk_rmem_alloc) >= (unsigned )sk->sk_rcvbuf) goto drop_n_acct;if (skb_shared (skb)) {net/packet/af_packet.c static int tpacket_rcv(struct sk_buff *skb, struct net_device *dev, · line 1768
Quoted 8 lines
net/packet/af_packet.c +1
-1@@ -1763,7 +1762,7 @@ static int tpacket_rcv(struct sk_buff *skb, struct net_device *dev,if (po->tp_version <=TPACKET_V2 ) {if (macoff + snaplen > po->rx_ring.frame_size) {if (po->copy_thresh && -atomic_read (&sk->sk_rmem_alloc) + skb->truesize +atomic_read (&sk->sk_rmem_alloc) < (unsigned )sk->sk_rcvbuf) {if (skb_shared (skb)) { copy_skb =skb_clone (skb,GFP_ATOMIC );It doesn't work too.
It is possible to see this behavior on qemu. What about if I prepare you package with cross toolchain, rootfs and you can add debug message where you want?
Quoted 5 lines
I have also tried
ll_temacdriver with ppc440 and behavior is the same.Max
FRAME_SIZEpass tonetdev_alloc_skb_ip_alignis 7966. For this value ping works. (Forll_temacdriver it is #defineXTE_JUMBO_MTU7948 fromll_temac.h)
I wonder if you applied/tested my patch correctly, since it really should had help in your case (allowing first received packet to be queued, even if very big)
Given the way NIC drivers work (pre-allocating big buffers before hardware fill frames in them), SO_RCVBUF limits are difficult to respect.
We should allow at least one frame, even with a very small SO_RCVBUF setting, or silently cap a minimum sane value.
Quoted 3 lines · Eric Dumazet
I wonder if you applied/tested my patch correctly, since it really should had help in your case (allowing first received packet to be queued, even if very big)
Hmm, I missed another spot in sock_queue_rcv_skb(). (RAW sockets are not PACKET :) )
Here is the patch again, I tested it with 'busybox ping' and MTU=9000, it solved the problem for me.
Thanks
[PATCH net-next] net: relax rcvbuf limits
skb->truesize might be big even for a small packet.
Its even bigger after commit 87fb4b7b533 (net: more accurate skb truesize) and big MTU.
We should allow queueing at least one packet per receiver, even with a low RCVBUF setting.
--- include/net/sock.h | 4 +++- net/core/sock.c | 6 +----- net/packet/af_packet.c | 6 ++---- 3 files changed, 6 insertions(+), 10 deletions(-)
include/net/sock.h +3 -1
@@ -662,12 +662,14 @@ static inline void __sk_add_backlog(struct sock *sk, struct sk_buff *skb)/* * Take into account size of receive queue and backlog queue +* Do not take into account this skb truesize, +* to allow even a single big packet to come. */ static inline bool sk_rcvqueues_full (const struct sock *sk,const struct sk_buff *skb) {unsigned int qsize = sk->sk_backlog.len +atomic_read (&sk->sk_rmem_alloc); -return qsize + skb->truesize > sk->sk_rcvbuf; +return qsize > sk->sk_rcvbuf; }/* The per-socket spinlock must be held here. */
net/core/sock.c +1 -5
@@ -339,11 +339,7 @@ int sock_queue_rcv_skb(struct sock *sk, struct sk_buff *skb)unsigned long flags;struct sk_buff_head *list = &sk->sk_receive_queue; -/* Cast sk->rcvbuf to unsigned... It's pointless, but reduces -number of warnings when compiling with -W --ANK -*/ -if (atomic_read (&sk->sk_rmem_alloc) + skb->truesize >= - (unsigned )sk->sk_rcvbuf) { +if (atomic_read (&sk->sk_rmem_alloc) >= sk->sk_rcvbuf) {atomic_inc (&sk->sk_drops);trace_sock_rcvqueue_full (sk, skb);return -ENOMEM ;
net/packet/af_packet.c +2 -4
@@ -1631,8 +1631,7 @@ static int packet_rcv(struct sk_buff *skb, struct net_device *dev,if (snaplen > res) snaplen = res; -if (atomic_read (&sk->sk_rmem_alloc) + skb->truesize >= - (unsigned )sk->sk_rcvbuf) +if (atomic_read (&sk->sk_rmem_alloc) >= sk->sk_rcvbuf) goto drop_n_acct;if (skb_shared (skb)) {
@@ -1763,8 +1762,7 @@ static int tpacket_rcv(struct sk_buff *skb, struct net_device *dev,if (po->tp_version <=TPACKET_V2 ) {if (macoff + snaplen > po->rx_ring.frame_size) {if (po->copy_thresh && -atomic_read (&sk->sk_rmem_alloc) + skb->truesize - < (unsigned )sk->sk_rcvbuf) { +atomic_read (&sk->sk_rmem_alloc) < sk->sk_rcvbuf) {if (skb_shared (skb)) { copy_skb =skb_clone (skb,GFP_ATOMIC ); }else {
From: Eric Dumazet <[email protected]>
Date: Wed, 21 Dec 2011 18:11:44 +0100
Quoted 4 lines
[PATCH net-next] net: relax rcvbuf limits
skb->truesizemight be big even for a small packet.
Quoted 8 lines
Its even bigger after commit
87fb4b7b533(net: more accurate skb truesize) and big MTU.We should allow queueing at least one packet per receiver, even with a low RCVBUF setting.
Reported-by Michal Simek <[email protected]>Signed-off-by Eric Dumazet <[email protected]>
Applied to net-next, although I was tempted to put it into net.
We may end up backporting this into -stable at some point, we'll see.
Quoted 13 lines · David Miller
From: Eric Dumazet <[email protected]>
Date: Wed, 21 Dec 2011 18:11:44 +0100Quoted 10 lines
[PATCH net-next] net: relax rcvbuf limits
skb->truesizemight be big even for a small packet.Its even bigger after commit
87fb4b7b533(net: more accurate skb truesize) and big MTU.We should allow queueing at least one packet per receiver, even with a low RCVBUF setting.
Quoted 7 lines
Quoted 2 lines
Reported-by Michal Simek <[email protected]>Signed-off-by Eric Dumazet <[email protected]>Applied to net-next, although I was tempted to put it into net.
We may end up backporting this into -stable at some point, we'll see.
Yes, it works. Thanks Eric.
I hope that this patch will be in v3.2.
Thanks,
Michal
Signature · 4 lines
Michal Simek, Ing. (M.Eng) w: www.monstr.eu p: +42-0-721842854 Maintainer of Linux kernel 2.6 Microblaze Linux - http://www.monstr.eu/fdt/ Microblaze U-BOOT custodian
Quoted 17 lines · Michal Simek
Quoted 16 lines · David Miller
From: Eric Dumazet <[email protected]>
Date: Wed, 21 Dec 2011 18:11:44 +0100Quoted 12 lines
[PATCH net-next] net: relax rcvbuf limits
skb->truesizemight be big even for a small packet.Its even bigger after commit
87fb4b7b533(net: more accurate skb truesize) and big MTU.We should allow queueing at least one packet per receiver, even with a low RCVBUF setting.
Reported-by Michal Simek <[email protected]>Signed-off-by Eric Dumazet <[email protected]>
Quoted 9 lines
Quoted 4 lines
Applied to net-next, although I was tempted to put it into net.
We may end up backporting this into -stable at some point, we'll see.
Yes, it works. Thanks Eric.
I hope that this patch will be in v3.2.
Thanks for testing !
I overlooked fact that commit 87fb4b7b533 was already in 3.2, so yes, we probably can push this to 3.2
(By the way, busybox ping probably doesnt work on 3.1 kernel with a MTU=9000 non copybreak driver, so its not a clear 3.2 regression)
Quoted 23 lines · Eric Dumazet
Quoted 22 lines · Michal Simek
Quoted 13 lines · David Miller
From: Eric Dumazet <[email protected]>
Date: Wed, 21 Dec 2011 18:11:44 +0100Quoted 10 lines
[PATCH net-next] net: relax rcvbuf limits
skb->truesizemight be big even for a small packet.Its even bigger after commit
87fb4b7b533(net: more accurate skb truesize) and big MTU.We should allow queueing at least one packet per receiver, even with a low RCVBUF setting.
Quoted 6 lines
Quoted 2 lines
Reported-by Michal Simek <[email protected]>Signed-off-by Eric Dumazet <[email protected]>Applied to net-next, although I was tempted to put it into net.
We may end up backporting this into -stable at some point, we'll see.
Yes, it works. Thanks Eric.
Quoted 7 lines
Quoted 2 lines
I hope that this patch will be in v3.2.
Thanks for testing !
I overlooked fact that commit
87fb4b7b533was already in 3.2, so yes, we probably can push this to 3.2
great.
Quoted 2 lines
(By the way, busybox ping probably doesnt work on 3.1 kernel with a MTU=9000 non copybreak driver, so its not a clear 3.2 regression)
good to know.
Thanks,
Michal
Signature · 4 lines
Michal Simek, Ing. (M.Eng) w: www.monstr.eu p: +42-0-721842854 Maintainer of Linux kernel 2.6 Microblaze Linux - http://www.monstr.eu/fdt/ Microblaze U-BOOT custodian
Quoted 10 lines · Michal Simek
Hi Eric and David,
I have found one problem with
ll_temacdriver and
this commit: net: more accurate skb truesize
sha1:87fb4b7b533073eeeaed0b6bf7c2328995f6c075The problem is only with icmp packets from the target. It is sent and driver receive it
but it is not proceed to the application.
Hi, Michal
What's the type of icmp you are using? such as "EchoReps", we can find the actually handler routine by this type.
And you said the packet already received by driver, can you tell me the way you used to confirm this?
And have you checked when was the icmp packet being dropped, is it in icmp_rcv or before or later?
Thanks,
Michael Wang
Quoted 16 lines
The problem I see is that
kmalloc_node_track_callerallocate specific size and then this size is changed by SKB_WITH_OVERHEAD(ksize(data)).
The problem is withnetdev_alloc_skb_ip_alignwhich calls__alloc_skbfunction.Currently driver uses setting for jumbo frames (9k). When I change it to use mtu (1,5k) then
everything is ok.Can you give me some hints what can be wrong?
Thanks,
Michal
Quoted 15 lines · Michael Wang
Quoted 10 lines · Michal Simek
Hi Eric and David,
I have found one problem with
ll_temacdriver and
this commit: net: more accurate skb truesize
sha1:87fb4b7b533073eeeaed0b6bf7c2328995f6c075The problem is only with icmp packets from the target. It is sent and driver receive it
but it is not proceed to the application.Hi, Michal
Quoted 8 lines
What's the type of icmp you are using? such as "EchoReps", we can find the actually handler routine by this type.
And you said the packet already received by driver, can you tell me the way you used to confirm this?
And have you checked when was the icmp packet being dropped, is it in
icmp_rcvor before or later?
Packet was dropped right before being queued in RAW socket receive_queue, because of low sk_rcvbuf setting (against skb->truesize)
Quoted 19 lines · Eric Dumazet
Quoted 18 lines · Michael Wang
Quoted 10 lines · Michal Simek
Hi Eric and David,
I have found one problem with
ll_temacdriver and
this commit: net: more accurate skb truesize
sha1:87fb4b7b533073eeeaed0b6bf7c2328995f6c075The problem is only with icmp packets from the target. It is sent and driver receive it
but it is not proceed to the application.Hi, Michal
What's the type of icmp you are using? such as "EchoReps", we can find the actually handler routine by this type.
Quoted 9 lines
Quoted 5 lines
And you said the packet already received by driver, can you tell me the way you used to confirm this?
And have you checked when was the icmp packet being dropped, is it in
icmp_rcvor before or later?Packet was dropped right before being queued in RAW socket
receive_queue, because of lowsk_rcvbufsetting (againstskb->truesize)
Hi, Eric
So do you mean the way is:
ip_local_deliver_finish --> raw_local_deliver --> raw_v4_input --> raw_rcv --> raw_rcv_skb --> sock_queue_rcv_skb
And in sock_queue_rcv_skb:
if (atomic_read(&sk->sk_rmem_alloc) + skb->truesize >=
(unsigned)sk->sk_rcvbuf) {
atomic_inc(&sk->sk_drops);
trace_sock_rcvqueue_full(sk, skb);
return -ENOMEM; //drop?
}Is this the way the packet being dropped? Is this confirmed by some printk?
Sorry if the question is naive, I just want to make sure I am checking at the right place.
Regards,
Michael Wang
Quoted 2 lines
Quoted 16 lines · Michael Wang
Hi, Eric
So do you mean the way is:
ip_local_deliver_finish-->raw_local_deliver-->raw_v4_input-->raw_rcv-->raw_rcv_skb-->sock_queue_rcv_skbAnd in
sock_queue_rcv_skb:if (atomic_read(&sk->sk_rmem_alloc) + skb->truesize >= (unsigned)sk->sk_rcvbuf) { atomic_inc(&sk->sk_drops); trace_sock_rcvqueue_full(sk, skb); return -ENOMEM; //drop? }
Quoted 5 lines
Is this the way the packet being dropped? Is this confirmed by some printk?
Sorry if the question is naive, I just want to make sure I am checking at the right place.
Yep
Are you aware I already posted a patch to solve the problem ?
Quoted 24 lines · Eric Dumazet
Quoted 16 lines · Michael Wang
Hi, Eric
So do you mean the way is:
ip_local_deliver_finish-->raw_local_deliver-->raw_v4_input-->raw_rcv-->raw_rcv_skb-->sock_queue_rcv_skbAnd in
sock_queue_rcv_skb:if (atomic_read(&sk->sk_rmem_alloc) + skb->truesize >= (unsigned)sk->sk_rcvbuf) { atomic_inc(&sk->sk_drops); trace_sock_rcvqueue_full(sk, skb); return -ENOMEM; //drop? }Quoted 5 lines
Is this the way the packet being dropped? Is this confirmed by some printk?
Sorry if the question is naive, I just want to make sure I am checking at the right place.
Quoted 6 lines
Yep
Are you aware I already posted a patch to solve the problem ?
Oh, sorry, I don't know about that, congratulation :)
Regards,
Michael Wang