Skip to contentSkip to conversation

ICMP packets - ll_temac with Microblaze

DISCUSSION Applied Michal Simek 6 participants 36 messages Dec 21, 2011 – Dec 22, 2011

Subsystems (4)
lore
Share & tools
lore
#0 lore raw

Hi Eric and David,

I have found one problem with ll_temac driver and
this commit: net: more accurate skb truesize
sha1: 87fb4b7b533073eeeaed0b6bf7c2328995f6c075

The problem is only with icmp packets from the target. It is sent and driver receive it but it is not proceed to the application.

The problem I see is that kmalloc_node_track_caller allocate specific size and then this size is changed by SKB_WITH_OVERHEAD(ksize(data)). The problem is with netdev_alloc_skb_ip_align which calls __alloc_skb function.

Currently driver uses setting for jumbo frames (9k). When I change it to use mtu (1,5k) then everything is ok.

Can you give me some hints what can be wrong?

Thanks,
Michal

Signature · 4 lines
Michal Simek, Ing. (M.Eng)
w: www.monstr.eu p: +42-0-721842854
Maintainer of Linux kernel 2.6 Microblaze Linux - http://www.monstr.eu/fdt/
Microblaze U-BOOT custodian
#1 lore raw
Quoted 9 lines · Michal Simek

Hi Eric and David,

I have found one problem with ll_temac driver and
this commit: net: more accurate skb truesize
sha1: 87fb4b7b533073eeeaed0b6bf7c2328995f6c075

The problem is only with icmp packets from the target. It is sent and driver receive it but it is not proceed to the application.

Quoted 9 lines

The problem I see is that kmalloc_node_track_caller allocate specific size and then this size is changed by SKB_WITH_OVERHEAD(ksize(data)). The problem is with netdev_alloc_skb_ip_align which calls __alloc_skb function.

Currently driver uses setting for jumbo frames (9k). When I change it to use mtu (1,5k) then everything is ok.

Can you give me some hints what can be wrong?

Is it with SLUB, SLAB or SLOB allocator ?

#2 lore raw
Quoted 14 lines · Eric Dumazet
Quoted 13 lines · Michal Simek

Hi Eric and David,

I have found one problem with ll_temac driver and
this commit: net: more accurate skb truesize
sha1: 87fb4b7b533073eeeaed0b6bf7c2328995f6c075

The problem is only with icmp packets from the target. It is sent and driver receive it but it is not proceed to the application.

The problem I see is that kmalloc_node_track_caller allocate specific size and then this size is changed by SKB_WITH_OVERHEAD(ksize(data)). The problem is with netdev_alloc_skb_ip_align which calls __alloc_skb function.

Quoted 7 lines
Quoted 5 lines

Currently driver uses setting for jumbo frames (9k). When I change it to use mtu (1,5k) then everything is ok.

Can you give me some hints what can be wrong?

Is it with SLUB, SLAB or SLOB allocator ?

SLAB.

CONFIG_SLAB=y
# CONFIG_SLUB is not set
# CONFIG_SLOB is not set

Michal

Signature · 4 lines
Michal Simek, Ing. (M.Eng)
w: www.monstr.eu p: +42-0-721842854
Maintainer of Linux kernel 2.6 Microblaze Linux - http://www.monstr.eu/fdt/
Microblaze U-BOOT custodian
#3 lore raw
In reply to Eric Dumazet
Quoted 14 lines · Eric Dumazet
Quoted 13 lines · Michal Simek

Hi Eric and David,

I have found one problem with ll_temac driver and
this commit: net: more accurate skb truesize
sha1: 87fb4b7b533073eeeaed0b6bf7c2328995f6c075

The problem is only with icmp packets from the target. It is sent and driver receive it but it is not proceed to the application.

The problem I see is that kmalloc_node_track_caller allocate specific size and then this size is changed by SKB_WITH_OVERHEAD(ksize(data)). The problem is with netdev_alloc_skb_ip_align which calls __alloc_skb function.

Quoted 9 lines
Quoted 5 lines

Currently driver uses setting for jumbo frames (9k). When I change it to use mtu (1,5k) then everything is ok.

Can you give me some hints what can be wrong?

Is it with SLUB, SLAB or SLOB allocator ?

(I am referring to commit bc417e30f8d (net: Add back alignment for size for __alloc_skb)

#4 lore raw
Quoted 18 lines · Eric Dumazet
Quoted 17 lines · Eric Dumazet
Quoted 16 lines · Michal Simek

Hi Eric and David,

I have found one problem with ll_temac driver and
this commit: net: more accurate skb truesize
sha1: 87fb4b7b533073eeeaed0b6bf7c2328995f6c075

The problem is only with icmp packets from the target. It is sent and driver receive it but it is not proceed to the application.

The problem I see is that kmalloc_node_track_caller allocate specific size and then this size is changed by SKB_WITH_OVERHEAD(ksize(data)). The problem is with netdev_alloc_skb_ip_align which calls __alloc_skb function.

Currently driver uses setting for jumbo frames (9k). When I change it to use mtu (1,5k) then everything is ok.

Quoted 8 lines
Quoted 5 lines
Quoted 2 lines

Can you give me some hints what can be wrong?

Is it with SLUB, SLAB or SLOB allocator ?

(I am referring to commit bc417e30f8d (net: Add back alignment for size for __alloc_skb)

I have seen that commit. Using the latest&greatest version with this patch.

Michal

Signature · 4 lines
Michal Simek, Ing. (M.Eng)
w: www.monstr.eu p: +42-0-721842854
Maintainer of Linux kernel 2.6 Microblaze Linux - http://www.monstr.eu/fdt/
Microblaze U-BOOT custodian
#5 lore raw
Quoted 21 lines · Michal Simek
Quoted 20 lines · Eric Dumazet
Quoted 19 lines · Eric Dumazet
Quoted 18 lines · Michal Simek

Hi Eric and David,

I have found one problem with ll_temac driver and
this commit: net: more accurate skb truesize
sha1: 87fb4b7b533073eeeaed0b6bf7c2328995f6c075

The problem is only with icmp packets from the target. It is sent and driver receive it but it is not proceed to the application.

The problem I see is that kmalloc_node_track_caller allocate specific size and then this size is changed by SKB_WITH_OVERHEAD(ksize(data)). The problem is with netdev_alloc_skb_ip_align which calls __alloc_skb function.

Currently driver uses setting for jumbo frames (9k). When I change it to use mtu (1,5k) then everything is ok.

Can you give me some hints what can be wrong?

Quoted 9 lines
Quoted 6 lines
Quoted 3 lines

Is it with SLUB, SLAB or SLOB allocator ?

(I am referring to commit bc417e30f8d (net: Add back alignment for size for __alloc_skb)

I have seen that commit. Using the latest&greatest version with this patch.

So after these changes, struct skb_shared_info is located to the very end of allocated memory.

Maybe there is a problem with MTU=9000, since assuming PAGE_SIZE=4096 on your machine (is it ?), we have a 16384 bytes block of memory, and ~16000 available in skb head, instead of ~9000

#6 lore raw
Quoted 30 lines · Eric Dumazet
Quoted 19 lines · Michal Simek
Quoted 18 lines · Eric Dumazet
Quoted 17 lines · Eric Dumazet
Quoted 16 lines · Michal Simek

Hi Eric and David,

I have found one problem with ll_temac driver and
this commit: net: more accurate skb truesize
sha1: 87fb4b7b533073eeeaed0b6bf7c2328995f6c075

The problem is only with icmp packets from the target. It is sent and driver receive it but it is not proceed to the application.

The problem I see is that kmalloc_node_track_caller allocate specific size and then this size is changed by SKB_WITH_OVERHEAD(ksize(data)). The problem is with netdev_alloc_skb_ip_align which calls __alloc_skb function.

Currently driver uses setting for jumbo frames (9k). When I change it to use mtu (1,5k) then everything is ok.

Quoted 9 lines
Quoted 7 lines
Quoted 5 lines
Quoted 2 lines

Can you give me some hints what can be wrong?

Is it with SLUB, SLAB or SLOB allocator ?

(I am referring to commit bc417e30f8d (net: Add back alignment for size for __alloc_skb)

I have seen that commit. Using the latest&greatest version with this patch.

Quoted 6 lines

So after these changes, struct skb_shared_info is located to the very end of allocated memory.

Maybe there is a problem with MTU=9000, since assuming PAGE_SIZE=4096 on your machine (is it ?), we have a 16384 bytes block of memory, and ~16000 available in skb head, instead of ~9000

yes, page size is 4k.

Here is the log for allocation - origin size for MTU 9000 is 0x2420(9248) and size after overhead is 0x3f40(16192)

kmalloc_node_track_caller size 00002420, SKB_WITH_OVERHEAD(ksize(data)) 00003f40

skb->head     c7870000
skb->data     c7870000
skb->tail     c7870000

skb->truesize 000040c0
skb->end c7873f40

Michal

Signature · 4 lines
Michal Simek, Ing. (M.Eng)
w: www.monstr.eu p: +42-0-721842854
Maintainer of Linux kernel 2.6 Microblaze Linux - http://www.monstr.eu/fdt/
Microblaze U-BOOT custodian
#7 lore raw
Quoted 5 lines · Michal Simek

yes, page size is 4k.

Here is the log for allocation - origin size for MTU 9000 is 0x2420(9248) and size after overhead is 0x3f40(16192)

Quoted 7 lines

kmalloc_node_track_caller size 00002420, SKB_WITH_OVERHEAD(ksize(data)) 00003f40

skb->head     c7870000
skb->data     c7870000
skb->tail     c7870000

skb->truesize 000040c0
skb->end c7873f40

So basically truesize is bigger than previous kernel, and you hit a socket limit. (RCVBUF ?)

truesize is more correct, so we uncover prior bugs.

We should allow at least one packet, regardless of its size :(

#8 lore raw
Quoted 15 lines · Eric Dumazet
Quoted 5 lines · Michal Simek

yes, page size is 4k.

Here is the log for allocation - origin size for MTU 9000 is 0x2420(9248) and size after overhead is 0x3f40(16192)

Quoted 7 lines

kmalloc_node_track_caller size 00002420, SKB_WITH_OVERHEAD(ksize(data)) 00003f40

skb->head     c7870000
skb->data     c7870000
skb->tail     c7870000

skb->truesize 000040c0
skb->end c7873f40

Quoted 2 lines

So basically truesize is bigger than previous kernel, and you hit a socket limit. (RCVBUF ?)

Where is that limit setup?

Michal

Signature · 4 lines
Michal Simek, Ing. (M.Eng)
w: www.monstr.eu p: +42-0-721842854
Maintainer of Linux kernel 2.6 Microblaze Linux - http://www.monstr.eu/fdt/
Microblaze U-BOOT custodian
#10 lore raw
In reply to Eric Dumazet
Quoted 15 lines · Eric Dumazet
Quoted 5 lines · Michal Simek

yes, page size is 4k.

Here is the log for allocation - origin size for MTU 9000 is 0x2420(9248) and size after overhead is 0x3f40(16192)

Quoted 7 lines

kmalloc_node_track_caller size 00002420, SKB_WITH_OVERHEAD(ksize(data)) 00003f40

skb->head     c7870000
skb->data     c7870000
skb->tail     c7870000

skb->truesize 000040c0
skb->end c7873f40

Quoted 8 lines

So basically truesize is bigger than previous kernel, and you hit a socket limit. (RCVBUF ?)

truesize is more correct, so we uncover prior bugs.

We should allow at least one packet, regardless of its size :(

I would try the following patch :

include/net/sock.h +3 -1
index bf6b9fd..21bb3b5 100644
@@ -662,12 +662,14 @@ static inline void __sk_add_backlog(struct sock *sk, struct sk_buff *skb)
 
 /*
  * Take into account size of receive queue and backlog queue
+ * Do not take into account this skb truesize,
+ * to allow even a single big packet to come.
  */
 static inline bool sk_rcvqueues_full(const struct sock *sk, const struct sk_buff *skb)
 {
 	unsigned int qsize = sk->sk_backlog.len + atomic_read(&sk->sk_rmem_alloc);
 
-	return qsize + skb->truesize > sk->sk_rcvbuf;
+	return qsize > sk->sk_rcvbuf;
 }
 
 /* The per-socket spinlock must be held here. */
#9 lore raw
In reply to Michal Simek
Quoted 16 lines · Michal Simek
Quoted 15 lines · Eric Dumazet
Quoted 12 lines · Michal Simek

yes, page size is 4k.

Here is the log for allocation - origin size for MTU 9000 is 0x2420(9248) and size after overhead is 0x3f40(16192)

kmalloc_node_track_caller size 00002420, SKB_WITH_OVERHEAD(ksize(data)) 00003f40

skb->head     c7870000
skb->data     c7870000
skb->tail     c7870000

skb->truesize 000040c0
skb->end c7873f40

Quoted 5 lines
Quoted 2 lines

So basically truesize is bigger than previous kernel, and you hit a socket limit. (RCVBUF ?)

Where is that limit setup?

Might be in the application itself.

setsockopt(socket, SOL_SOCKET, SO_RCVBUF, &limit, sizeof(limit));

kernel currently apply a factor of two to take into account skb overhead. This might be not enough with jumbo frames.

#11 lore raw
In reply to Eric Dumazet
Quoted 37 lines · Eric Dumazet
Quoted 14 lines · Eric Dumazet
Quoted 5 lines · Michal Simek

yes, page size is 4k.

Here is the log for allocation - origin size for MTU 9000 is 0x2420(9248) and size after overhead is 0x3f40(16192)

Quoted 7 lines

kmalloc_node_track_caller size 00002420, SKB_WITH_OVERHEAD(ksize(data)) 00003f40

skb->head     c7870000
skb->data     c7870000
skb->tail     c7870000

skb->truesize 000040c0
skb->end c7873f40

Quoted 8 lines

So basically truesize is bigger than previous kernel, and you hit a socket limit. (RCVBUF ?)

truesize is more correct, so we uncover prior bugs.

We should allow at least one packet, regardless of its size :(

I would try the following patch :

include/net/sock.h +2 -0
index bf6b9fd..21bb3b5 100644
@@ -662,12 +662,14 @@ static inline void __sk_add_backlog(struct sock *sk, struct sk_buff *skb)
 
 /*
  * Take into account size of receive queue and backlog queue
+ * Do not take into account this skb truesize,
+ * to allow even a single big packet to come.

include/net/sock.h static inline void __sk_add_backlog(struct sock *sk, struct sk_buff *skb) · line 675

Quoted 10 lines
include/net/sock.h +1 -1
  */
 static inline bool sk_rcvqueues_full(const struct sock *sk, const struct sk_buff *skb)
 {
 	unsigned int qsize = sk->sk_backlog.len + atomic_read(&sk->sk_rmem_alloc);
 
-	return qsize + skb->truesize > sk->sk_rcvbuf;
+	return qsize > sk->sk_rcvbuf;
 }
 
 /* The per-socket spinlock must be held here. */

It doesn't fix the problem. Do you want to add some printk to code?

Michal

Signature · 4 lines
Michal Simek, Ing. (M.Eng)
w: www.monstr.eu p: +42-0-721842854
Maintainer of Linux kernel 2.6 Microblaze Linux - http://www.monstr.eu/fdt/
Microblaze U-BOOT custodian
#12 lore raw
Quoted 46 lines · Michal Simek
Quoted 45 lines · Eric Dumazet
Quoted 14 lines · Eric Dumazet
Quoted 5 lines · Michal Simek

yes, page size is 4k.

Here is the log for allocation - origin size for MTU 9000 is 0x2420(9248) and size after overhead is 0x3f40(16192)

Quoted 7 lines

kmalloc_node_track_caller size 00002420, SKB_WITH_OVERHEAD(ksize(data)) 00003f40

skb->head     c7870000
skb->data     c7870000
skb->tail     c7870000

skb->truesize 000040c0
skb->end c7873f40

Quoted 8 lines

So basically truesize is bigger than previous kernel, and you hit a socket limit. (RCVBUF ?)

truesize is more correct, so we uncover prior bugs.

We should allow at least one packet, regardless of its size :(

I would try the following patch :

include/net/sock.h +3 -1
index bf6b9fd..21bb3b5 100644
@@ -662,12 +662,14 @@ static inline void __sk_add_backlog(struct sock *sk, struct sk_buff *skb)
 
 /*
  * Take into account size of receive queue and backlog queue
+ * Do not take into account this skb truesize,
+ * to allow even a single big packet to come.
  */
 static inline bool sk_rcvqueues_full(const struct sock *sk, const struct sk_buff *skb)
 {
 	unsigned int qsize = sk->sk_backlog.len + atomic_read(&sk->sk_rmem_alloc);
 
-	return qsize + skb->truesize > sk->sk_rcvbuf;
+	return qsize > sk->sk_rcvbuf;
 }
 
Quoted 9 lines

include/net/sock.h static inline void __sk_add_backlog(struct sock *sk, struct sk_buff *skb) · line 675

Quoted 2 lines
include/net/sock.h +0 -0
 /* The per-socket spinlock must be held here. */

It doesn't fix the problem. Do you want to add some printk to code?

Michal

try to change SOCK_MIN_RCVBUF

include/net/sock.h

#define SOCK_MIN_RCVBUF (2048 + sizeof(struct sk_buff))

->

#define SOCK_MIN_RCVBUF (16384 + sizeof(struct sk_buff))

#13 lore raw
Quoted 57 lines · Eric Dumazet
Quoted 45 lines · Michal Simek
Quoted 44 lines · Eric Dumazet
Quoted 14 lines · Eric Dumazet
Quoted 5 lines · Michal Simek

yes, page size is 4k.

Here is the log for allocation - origin size for MTU 9000 is 0x2420(9248) and size after overhead is 0x3f40(16192)

Quoted 7 lines

kmalloc_node_track_caller size 00002420, SKB_WITH_OVERHEAD(ksize(data)) 00003f40

skb->head     c7870000
skb->data     c7870000
skb->tail     c7870000

skb->truesize 000040c0
skb->end c7873f40

Quoted 8 lines

So basically truesize is bigger than previous kernel, and you hit a socket limit. (RCVBUF ?)

truesize is more correct, so we uncover prior bugs.

We should allow at least one packet, regardless of its size :(

I would try the following patch :

include/net/sock.h +3 -1
index bf6b9fd..21bb3b5 100644
@@ -662,12 +662,14 @@ static inline void __sk_add_backlog(struct sock *sk, struct sk_buff *skb)
 
 /*
  * Take into account size of receive queue and backlog queue
+ * Do not take into account this skb truesize,
+ * to allow even a single big packet to come.
  */
 static inline bool sk_rcvqueues_full(const struct sock *sk, const struct sk_buff *skb)
 {
 	unsigned int qsize = sk->sk_backlog.len + atomic_read(&sk->sk_rmem_alloc);
 
-	return qsize + skb->truesize > sk->sk_rcvbuf;
+	return qsize > sk->sk_rcvbuf;
 }
 
Quoted 8 lines

include/net/sock.h static inline void __sk_add_backlog(struct sock *sk, struct sk_buff *skb) · line 675

Quoted 2 lines
include/net/sock.h +0 -0
 /* The per-socket spinlock must be held here. */

It doesn't fix the problem. Do you want to add some printk to code?

Michal

try to change SOCK_MIN_RCVBUF

Quoted 8 lines

include/net/sock.h

#define SOCK_MIN_RCVBUF (2048 + sizeof(struct sk_buff))

->

#define SOCK_MIN_RCVBUF (16384 + sizeof(struct sk_buff))

Doesn't work. :-(

Michal

Signature · 4 lines
Michal Simek, Ing. (M.Eng)
w: www.monstr.eu p: +42-0-721842854
Maintainer of Linux kernel 2.6 Microblaze Linux - http://www.monstr.eu/fdt/
Microblaze U-BOOT custodian
#14 lore raw
Quoted 5 lines · Michal Simek
Quoted 2 lines

#define SOCK_MIN_RCVBUF (16384 + sizeof(struct sk_buff))

Doesn't work. :-(

Sorry, I have no idea where packet is dropped, I dont know your application.

You might send "netstat -s" output if you cant find it ...

You spoke of ICMP message, I cant find a limit in this area ?

#15 lore raw
Quoted 9 lines · Eric Dumazet
Quoted 4 lines · Michal Simek
Quoted 2 lines

#define SOCK_MIN_RCVBUF (16384 + sizeof(struct sk_buff))

Doesn't work. :-(

Sorry, I have no idea where packet is dropped, I dont know your application.

Application is simple ping from Busybox - nothing special.

Quoted 1 line

You might send "netstat -s" output if you cant find it ...

~ # ./netstat -s
Ip:
36 total packets received
2 with invalid addresses
0 forwarded
0 incoming packets discarded
34 incoming packets delivered
34 requests sent out
Icmp:
34 ICMP messages received
0 input ICMP message failed.
ICMP input histogram:
echo replies: 34
34 ICMP messages sent
0 ICMP messages failed
ICMP output histogram:
echo request: 34
IcmpMsg:

        InType0: 34
        OutType8: 34

Tcp:
0 active connections openings
0 passive connection openings
0 failed connection attempts
0 connection resets received
0 connections established
0 segments received
0 segments send out
0 segments retransmited
0 bad segments received.
0 resets sent
Udp:
0 packets received
0 packets to unknown port received.
0 packet receive errors
0 packets sent
RcvbufErrors: 0
SndbufErrors: 0
UdpLite:
InDatagrams: 0
NoPorts: 0
InErrors: 0
OutDatagrams: 0
RcvbufErrors: 0
SndbufErrors: 0
TcpExt:
ArpFilter: 0
0 packets header predicted
TCPPureAcks: 0
TCPHPAcks: 0
TCPRenoRecovery: 0
TCPSackRecovery: 0
TCPSACKReneging: 0
TCPFACKReorder: 0
TCPSACKReorder: 0
TCPRenoReorder: 0
TCPTSReorder: 0
TCPFullUndo: 0
TCPPartialUndo: 0
TCPDSACKUndo: 0
TCPLossUndo: 0
TCPLoss: 0
TCPLostRetransmit: 0
TCPRenoFailures: 0
TCPSackFailures: 0
TCPLossFailures: 0
TCPFastRetrans: 0
TCPForwardRetrans: 0
TCPSlowStartRetrans: 0
TCPTimeouts: 0
TCPRenoRecoveryFail: 0
TCPSackRecoveryFail: 0
TCPSchedulerFailed: 0
TCPRcvCollapsed: 0
TCPDSACKOldSent: 0
TCPDSACKOfoSent: 0
TCPDSACKRecv: 0
TCPDSACKOfoRecv: 0
TCPAbortOnSyn: 0
TCPAbortOnData: 0
TCPAbortOnClose: 0
TCPAbortOnMemory: 0
TCPAbortOnTimeout: 0
TCPAbortOnLinger: 0
TCPAbortFailed: 0
TCPMemoryPressures: 0
TCPSACKDiscard: 0
TCPDSACKIgnoredOld: 0
TCPDSACKIgnoredNoUndo: 0
TCPSpuriousRTOs: 0
TCPMD5NotFound: 0
TCPMD5Unexpected: 0
TCPSackShifted: 0
TCPSackMerged: 0
TCPSackShiftFallback: 0
TCPBacklogDrop: 0
TCPMinTTLDrop: 0
TCPDeferAcceptDrop: 0
IPReversePathFilter: 0
TCPTimeWaitOverflow: 0
TCPReqQFullDoCookies: 0
TCPReqQFullDrop: 0
IpExt:

    InNoRoutes: 0
    InTruncatedPkts: 0
    InMcastPkts: 0
    OutMcastPkts: 0
    InBcastPkts: 0
    OutBcastPkts: 0
    InOctets: 4008
    OutOctets: 2856
    InMcastOctets: 0
    OutMcastOctets: 0
    InBcastOctets: 0
    OutBcastOctets: 0
Quoted 1 line

You spoke of ICMP message, I cant find a limit in this area ?

ok. Can you provide me any background why size should be setup by size = SKB_WITH_OVERHEAD(ksize(data));
and not to use size which is passed to kmalloc in __alloc_skb.

In past I have seen some issues because of caches but this is not that case because on QEMU (where cache is not modeled) everything works the same as on HW.

Below is the log what I see.

Thanks,
Michal

~ # ping -c 5 192.168.0.101
PING 192.168.0.101 (192.168.0.101): 56 data bytes

--- 192.168.0.101 ping statistics ---
5 packets transmitted, 0 packets received, 100% packet loss ~ # mount -t nfs -o nolock -o tcp 192.168.0.101:/tftpboot/nfs /mnt ~ # ls /mnt/ | head
0001-sd.patch 0001-syscalls-getdtablesize01-fix-for-missing-etc-hosts.patch 1
1-xils
2
arp
arp-mb
busybox
core
exec
~ # umount /mnt/
~ # ./netstat -s
Ip:
36 total packets received
2 with invalid addresses
0 forwarded
0 incoming packets discarded
34 incoming packets delivered
43 requests sent out
Icmp:
5 ICMP messages received
0 input ICMP message failed.
ICMP input histogram:
echo replies: 5
5 ICMP messages sent
0 ICMP messages failed
ICMP output histogram:
echo request: 5
IcmpMsg:

        InType0: 5
        OutType8: 5

Tcp:
4 active connections openings
0 passive connection openings
0 failed connection attempts
0 connection resets received
0 connections established
29 segments received
38 segments send out
0 segments retransmited
0 bad segments received.
0 resets sent
Udp:
0 packets received
0 packets to unknown port received.
0 packet receive errors
0 packets sent
RcvbufErrors: 0
SndbufErrors: 0
UdpLite:
InDatagrams: 0
NoPorts: 0
InErrors: 0
OutDatagrams: 0
RcvbufErrors: 0
SndbufErrors: 0
TcpExt:
ArpFilter: 0
1 delayed acks sent
10 packets header predicted
TCPPureAcks: 4
TCPHPAcks: 6
TCPRenoRecovery: 0
TCPSackRecovery: 0
TCPSACKReneging: 0
TCPFACKReorder: 0
TCPSACKReorder: 0
TCPRenoReorder: 0
TCPTSReorder: 0
TCPFullUndo: 0
TCPPartialUndo: 0
TCPDSACKUndo: 0
TCPLossUndo: 0
TCPLoss: 0
TCPLostRetransmit: 0
TCPRenoFailures: 0
TCPSackFailures: 0
TCPLossFailures: 0
TCPFastRetrans: 0
TCPForwardRetrans: 0
TCPSlowStartRetrans: 0
TCPTimeouts: 0
TCPRenoRecoveryFail: 0
TCPSackRecoveryFail: 0
TCPSchedulerFailed: 0
TCPRcvCollapsed: 0
TCPDSACKOldSent: 0
TCPDSACKOfoSent: 0
TCPDSACKRecv: 0
TCPDSACKOfoRecv: 0
TCPAbortOnSyn: 0
TCPAbortOnData: 0
TCPAbortOnClose: 0
TCPAbortOnMemory: 0
TCPAbortOnTimeout: 0
TCPAbortOnLinger: 0
TCPAbortFailed: 0
TCPMemoryPressures: 0
TCPSACKDiscard: 0
TCPDSACKIgnoredOld: 0
TCPDSACKIgnoredNoUndo: 0
TCPSpuriousRTOs: 0
TCPMD5NotFound: 0
TCPMD5Unexpected: 0
TCPSackShifted: 0
TCPSackMerged: 0
TCPSackShiftFallback: 0
TCPBacklogDrop: 0
TCPMinTTLDrop: 0
TCPDeferAcceptDrop: 0
IPReversePathFilter: 0
TCPTimeWaitOverflow: 0
TCPReqQFullDoCookies: 0
TCPReqQFullDrop: 0
IpExt:

    InNoRoutes: 0
    InTruncatedPkts: 0
    InMcastPkts: 0
    OutMcastPkts: 0
    InBcastPkts: 0
    OutBcastPkts: 0
    InOctets: 11348
    OutOctets: 3752
    InMcastOctets: 0
    OutMcastOctets: 0
    InBcastOctets: 0
    OutBcastOctets: 0

~ #
~ #
~ # tcpdump
device eth0 entered promiscuous mode
tcpdump: listening on eth0
00:01:31.421191 192.168.0.101 > 192.168.0.100: icmp: echo request (DF) 00:01:31.479686 192.168.0.100 > 192.168.0.101: icmp: echo reply 00:01:31.442325 arp who-has 192.168.0.1 tell 192.168.0.100 00:01:31.446052 arp reply 192.168.0.1 is-at f4:ec:38:b2:6a:9e 00:01:31.446256 192.168.0.100.47044 > 192.168.0.1.53: 45191+ (44) (DF) 00:01:31.461859 192.168.0.1.53 > 192.168.0.100.47044: 45191 NXDomain 0/1/0 (121) 00:01:31.464125 192.168.0.100.41833 > 192.168.0.1.53: 37485+ (44) (DF) 00:01:31.478026 192.168.0.1.53 > 192.168.0.100.41833: 37485 NXDomain 0/1/0 (121) 00:01:31.481794 192.168.0.100.35138 > 192.168.0.1.53: 56594+ (42) (DF) 00:01:31.497202 192.168.0.1.53 > 192.168.0.100.35138: 56594 NXDomain 0/1/0 (119) 00:01:32.419130 192.168.0.101 > 192.168.0.100: icmp: echo request (DF) 00:01:32.419482 192.168.0.100 > 192.168.0.101: icmp: echo reply 00:01:33.418342 192.168.0.101 > 192.168.0.100: icmp: echo request (DF) 00:01:33.418693 192.168.0.100 > 192.168.0.101: icmp: echo reply 00:01:34.419327 192.168.0.101 > 192.168.0.100: icmp: echo request (DF) 00:01:34.419676 192.168.0.100 > 192.168.0.101: icmp: echo reply 00:01:35.418042 192.168.0.101 > 192.168.0.100: icmp: echo request (DF) 00:01:35.418406 192.168.0.100 > 192.168.0.101: icmp: echo reply 00:01:36.418923 192.168.0.101 > 192.168.0.100: icmp: echo request (DF) 00:01:36.419275 192.168.0.100 > 192.168.0.101: icmp: echo reply 00:01:36.459936 arp who-has 192.168.0.100 tell 192.168.0.1 00:01:36.460216 arp reply 192.168.0.100 is-at 0:a:35:0:19:15 00:01:37.418800 192.168.0.101 > 192.168.0.100: icmp: echo request (DF) 00:01:37.419149 192.168.0.100 > 192.168.0.101: icmp: echo reply 00:01:38.419247 192.168.0.101 > 192.168.0.100: icmp: echo request (DF) 00:01:38.419617 192.168.0.100 > 192.168.0.101: icmp: echo reply 00:01:39.418820 192.168.0.101 > 192.168.0.100: icmp: echo request (DF) 00:01:39.419169 192.168.0.100 > 192.168.0.101: icmp: echo reply

56 packets received by filter
0 packets droppeddevice eth0 left promiscuous mode
by kernel
~ #
~ # ping 192.168.0.101
PING 192.168.0.101 (192.168.0.101): 56 data bytes

--- 192.168.0.101 ping statistics ---
2 packets transmitted, 0 packets received, 100% packet loss
~ # ping 192.168.0.1
PING 192.168.0.1 (192.168.0.1): 56 data bytes

--- 192.168.0.1 ping statistics ---
1 packets transmitted, 0 packets received, 100% packet loss
~ # ping 192.168.0.101 -c 10 &
~ # PING 192.168.0.101 (192.168.0.101): 56 data bytes
~ # tcpdump
device eth0 entered promiscuous mode
tcpdump: listening on eth0
00:01:59.615918 192.168.0.100 > 192.168.0.101: icmp: echo request (DF) 00:01:59.619166 192.168.0.101 > 192.168.0.100: icmp: echo reply 00:01:59.636473 192.168.0.100.57212 > 192.168.0.1.53: 43026+ (44) (DF) 00:01:59.650838 192.168.0.1.53 > 192.168.0.100.57212: 43026 NXDomain 0/1/0 (121) 00:01:59.652915 192.168.0.100.41475 > 192.168.0.1.53: 28371+ (44) (DF) 00:01:59.666907 192.168.0.1.53 > 192.168.0.100.41475: 28371 NXDomain 0/1/0 (121) 00:01:59.670832 192.168.0.100.47770 > 192.168.0.1.53: 61908+ (42) (DF) 00:01:59.684921 192.168.0.1.53 > 192.168.0.100.47770: 61908 NXDomain 0/1/0 (119) 00:02:00.616493 192.168.0.100 > 192.168.0.101: icmp: echo request (DF) 00:02:00.620237 192.168.0.101 > 192.168.0.100: icmp: echo reply 00:02:00.624275 arp who-has 192.168.0.101 tell 192.168.0.100 00:02:00.627936 arp reply 192.168.0.101 is-at 0:22:15:fb:34:a1 00:02:01.617232 192.168.0.100 > 192.168.0.101: icmp: echo request (DF) 00:02:01.620980 192.168.0.101 > 192.168.0.100: icmp: echo reply 00:02:02.617962 192.168.0.100 > 192.168.0.101: icmp: echo request (DF) 00:02:02.621687 192.168.0.101 > 192.168.0.100: icmp: echo reply 00:02:03.618687 192.168.0.100 > 192.168.0.101: icmp: echo request (DF) 00:02:03.622389 192.168.0.101 > 192.168.0.100: icmp: echo reply 00:02:04.619414 192.168.0.100 > 192.168.0.101: icmp: echo request (DF) 00:02:04.623129 192.168.0.101 > 192.168.0.100: icmp: echo reply

42 packets received by filter
0 packets droppeddevice eth0 left promiscuous mode
by kernel
~ #
--- 192.168.0.101 ping statistics ---
10 packets transmitted, 0 packets received, 100% packet loss

[1] + Done(1) ping 192.168.0.101 -c 10
~ #

Signature · 4 lines
Michal Simek, Ing. (M.Eng)
w: www.monstr.eu p: +42-0-721842854
Maintainer of Linux kernel 2.6 Microblaze Linux - http://www.monstr.eu/fdt/
Microblaze U-BOOT custodian
#16 lore raw
Quoted 3 lines · Michal Simek

ok. Can you provide me any background why size should be setup by size = SKB_WITH_OVERHEAD(ksize(data));
and not to use size which is passed to kmalloc in __alloc_skb.

Its all about memory accounting (based on skb->truesize)

Prior to the patch, we could fool memory accounting because skbs claimed to use less memory than what they really used.

And crash machines eventually.

Now memory accouting is fixed, we probably need to change some points in the kernel, where we previously accepted a small skb, but not a very large one.

Since "ping" probably uses SOCK_RAW sockets, I'll try this one :

(We dont care of _this_ skb truesize, only on the count of previously queued packets)

net/packet/af_packet.c +2 -3
index 0da505c..a809a48 100644
@@ -1631,8 +1631,7 @@ static int packet_rcv(struct sk_buff *skb, struct net_device *dev,
 	if (snaplen > res)
 		snaplen = res;
 
-	if (atomic_read(&sk->sk_rmem_alloc) + skb->truesize >=
-	    (unsigned)sk->sk_rcvbuf)
+	if (atomic_read(&sk->sk_rmem_alloc) >= (unsigned)sk->sk_rcvbuf)
 		goto drop_n_acct;
 
 	if (skb_shared(skb)) {
@@ -1763,7 +1762,7 @@ static int tpacket_rcv(struct sk_buff *skb, struct net_device *dev,
 	if (po->tp_version <= TPACKET_V2) {
 		if (macoff + snaplen > po->rx_ring.frame_size) {
 			if (po->copy_thresh &&
-				atomic_read(&sk->sk_rmem_alloc) + skb->truesize
+				atomic_read(&sk->sk_rmem_alloc)
 				< (unsigned)sk->sk_rcvbuf) {
 				if (skb_shared(skb)) {
 					copy_skb = skb_clone(skb, GFP_ATOMIC);
#17 lore raw
Quoted 42 lines · Eric Dumazet
Quoted 3 lines · Michal Simek

ok. Can you provide me any background why size should be setup by size = SKB_WITH_OVERHEAD(ksize(data));
and not to use size which is passed to kmalloc in __alloc_skb.

Its all about memory accounting (based on skb->truesize)

Prior to the patch, we could fool memory accounting because skbs claimed to use less memory than what they really used.

And crash machines eventually.

Now memory accouting is fixed, we probably need to change some points in the kernel, where we previously accepted a small skb, but not a very large one.

Since "ping" probably uses SOCK_RAW sockets, I'll try this one :

(We dont care of _this_ skb truesize, only on the count of previously queued packets)

net/packet/af_packet.c +1 -2
index 0da505c..a809a48 100644
@@ -1631,8 +1631,7 @@ static int packet_rcv(struct sk_buff *skb, struct net_device *dev,
 	if (snaplen > res)
 		snaplen = res;
 
-	if (atomic_read(&sk->sk_rmem_alloc) + skb->truesize >=
-	    (unsigned)sk->sk_rcvbuf)
+	if (atomic_read(&sk->sk_rmem_alloc) >= (unsigned)sk->sk_rcvbuf)
 		goto drop_n_acct;
 
 	if (skb_shared(skb)) {

net/packet/af_packet.c static int tpacket_rcv(struct sk_buff *skb, struct net_device *dev, · line 1768

Quoted 8 lines
net/packet/af_packet.c +1 -1
@@ -1763,7 +1762,7 @@ static int tpacket_rcv(struct sk_buff *skb, struct net_device *dev,
 	if (po->tp_version <= TPACKET_V2) {
 		if (macoff + snaplen > po->rx_ring.frame_size) {
 			if (po->copy_thresh &&
-				atomic_read(&sk->sk_rmem_alloc) + skb->truesize
+				atomic_read(&sk->sk_rmem_alloc)
 				< (unsigned)sk->sk_rcvbuf) {
 				if (skb_shared(skb)) {
 					copy_skb = skb_clone(skb, GFP_ATOMIC);

It doesn't work too.
It is possible to see this behavior on qemu. What about if I prepare you package with cross toolchain, rootfs and you can add debug message where you want?

I have also tried ll_temac driver with ppc440 and behavior is the same.

Max FRAME_SIZE pass to netdev_alloc_skb_ip_align is 7966. For this value ping works. (For ll_temac driver it is #define XTE_JUMBO_MTU 7948 from ll_temac.h)

Thanks,
Michal

Signature · 4 lines
Michal Simek, Ing. (M.Eng)
w: www.monstr.eu p: +42-0-721842854
Maintainer of Linux kernel 2.6 Microblaze Linux - http://www.monstr.eu/fdt/
Microblaze U-BOOT custodian
#18 lore raw
Quoted 56 lines · Michal Simek
Quoted 42 lines · Eric Dumazet
Quoted 3 lines · Michal Simek

ok. Can you provide me any background why size should be setup by size = SKB_WITH_OVERHEAD(ksize(data));
and not to use size which is passed to kmalloc in __alloc_skb.

Its all about memory accounting (based on skb->truesize)

Prior to the patch, we could fool memory accounting because skbs claimed to use less memory than what they really used.

And crash machines eventually.

Now memory accouting is fixed, we probably need to change some points in the kernel, where we previously accepted a small skb, but not a very large one.

Since "ping" probably uses SOCK_RAW sockets, I'll try this one :

(We dont care of _this_ skb truesize, only on the count of previously queued packets)

net/packet/af_packet.c +1 -2
index 0da505c..a809a48 100644
@@ -1631,8 +1631,7 @@ static int packet_rcv(struct sk_buff *skb, struct net_device *dev,
 	if (snaplen > res)
 		snaplen = res;
 
-	if (atomic_read(&sk->sk_rmem_alloc) + skb->truesize >=
-	    (unsigned)sk->sk_rcvbuf)
+	if (atomic_read(&sk->sk_rmem_alloc) >= (unsigned)sk->sk_rcvbuf)
 		goto drop_n_acct;
 
 	if (skb_shared(skb)) {

net/packet/af_packet.c static int tpacket_rcv(struct sk_buff *skb, struct net_device *dev, · line 1768

Quoted 8 lines
net/packet/af_packet.c +1 -1
@@ -1763,7 +1762,7 @@ static int tpacket_rcv(struct sk_buff *skb, struct net_device *dev,
 	if (po->tp_version <= TPACKET_V2) {
 		if (macoff + snaplen > po->rx_ring.frame_size) {
 			if (po->copy_thresh &&
-				atomic_read(&sk->sk_rmem_alloc) + skb->truesize
+				atomic_read(&sk->sk_rmem_alloc)
 				< (unsigned)sk->sk_rcvbuf) {
 				if (skb_shared(skb)) {
 					copy_skb = skb_clone(skb, GFP_ATOMIC);

It doesn't work too.
It is possible to see this behavior on qemu. What about if I prepare you package with cross toolchain, rootfs and you can add debug message where you want?

Quoted 4 lines

I have also tried ll_temac driver with ppc440 and behavior is the same.

Max FRAME_SIZE pass to netdev_alloc_skb_ip_align is 7966. For this value ping works. (For ll_temac driver it is #define XTE_JUMBO_MTU 7948 from ll_temac.h)

I did several tests with MTU 9000 on my machines and it works well.

It seems my pings (iputils-sss20071127 or iputils-sss20101006) uses a big enough RCVBUF

setsockopt(3, SOL_SOCKET, SO_RCVBUF, [65536], 4) = 0 getsockopt(3, SOL_SOCKET, SO_RCVBUF, [131072], [4]) = 0

Could you check with "strace ping..." what is doing busybox ?

#19 lore raw
Quoted 64 lines · Eric Dumazet
Quoted 56 lines · Michal Simek
Quoted 42 lines · Eric Dumazet
Quoted 3 lines · Michal Simek

ok. Can you provide me any background why size should be setup by size = SKB_WITH_OVERHEAD(ksize(data));
and not to use size which is passed to kmalloc in __alloc_skb.

Its all about memory accounting (based on skb->truesize)

Prior to the patch, we could fool memory accounting because skbs claimed to use less memory than what they really used.

And crash machines eventually.

Now memory accouting is fixed, we probably need to change some points in the kernel, where we previously accepted a small skb, but not a very large one.

Since "ping" probably uses SOCK_RAW sockets, I'll try this one :

(We dont care of _this_ skb truesize, only on the count of previously queued packets)

net/packet/af_packet.c +1 -2
index 0da505c..a809a48 100644
@@ -1631,8 +1631,7 @@ static int packet_rcv(struct sk_buff *skb, struct net_device *dev,
 	if (snaplen > res)
 		snaplen = res;
 
-	if (atomic_read(&sk->sk_rmem_alloc) + skb->truesize >=
-	    (unsigned)sk->sk_rcvbuf)
+	if (atomic_read(&sk->sk_rmem_alloc) >= (unsigned)sk->sk_rcvbuf)
 		goto drop_n_acct;
 
 	if (skb_shared(skb)) {

net/packet/af_packet.c static int tpacket_rcv(struct sk_buff *skb, struct net_device *dev, · line 1768

Quoted 8 lines
net/packet/af_packet.c +1 -1
@@ -1763,7 +1762,7 @@ static int tpacket_rcv(struct sk_buff *skb, struct net_device *dev,
 	if (po->tp_version <= TPACKET_V2) {
 		if (macoff + snaplen > po->rx_ring.frame_size) {
 			if (po->copy_thresh &&
-				atomic_read(&sk->sk_rmem_alloc) + skb->truesize
+				atomic_read(&sk->sk_rmem_alloc)
 				< (unsigned)sk->sk_rcvbuf) {
 				if (skb_shared(skb)) {
 					copy_skb = skb_clone(skb, GFP_ATOMIC);

It doesn't work too.
It is possible to see this behavior on qemu. What about if I prepare you package with cross toolchain, rootfs and you can add debug message where you want?

Quoted 4 lines

I have also tried ll_temac driver with ppc440 and behavior is the same.

Max FRAME_SIZE pass to netdev_alloc_skb_ip_align is 7966. For this value ping works. (For ll_temac driver it is #define XTE_JUMBO_MTU 7948 from ll_temac.h)

I did several tests with MTU 9000 on my machines and it works well.

Quoted 7 lines

It seems my pings (iputils-sss20071127 or iputils-sss20101006) uses a big enough RCVBUF

setsockopt(3, SOL_SOCKET, SO_RCVBUF, [65536], 4) = 0 getsockopt(3, SOL_SOCKET, SO_RCVBUF, [131072], [4]) = 0

Could you check with "strace ping..." what is doing busybox ?

I found it : Its too small for jumbo frames.

setsockopt(3, SOL_SOCKET, SO_RCVBUF, [7280], 4) = 0

networking/ping.c

/* set recv buf (needed if we can get lots of responses: flood ping,
* broadcast ping etc) */
sockopt = (datalen * 2) + 7 * 1024; /* giving it a bit of extra room */ setsockopt(pingsock, SOL_SOCKET, SO_RCVBUF, &sockopt, sizeof(sockopt));

#25 lore raw
In reply to Michal Simek
Quoted 56 lines · Michal Simek
Quoted 42 lines · Eric Dumazet
Quoted 3 lines · Michal Simek

ok. Can you provide me any background why size should be setup by size = SKB_WITH_OVERHEAD(ksize(data));
and not to use size which is passed to kmalloc in __alloc_skb.

Its all about memory accounting (based on skb->truesize)

Prior to the patch, we could fool memory accounting because skbs claimed to use less memory than what they really used.

And crash machines eventually.

Now memory accouting is fixed, we probably need to change some points in the kernel, where we previously accepted a small skb, but not a very large one.

Since "ping" probably uses SOCK_RAW sockets, I'll try this one :

(We dont care of _this_ skb truesize, only on the count of previously queued packets)

net/packet/af_packet.c +1 -2
index 0da505c..a809a48 100644
@@ -1631,8 +1631,7 @@ static int packet_rcv(struct sk_buff *skb, struct net_device *dev,
 	if (snaplen > res)
 		snaplen = res;
 
-	if (atomic_read(&sk->sk_rmem_alloc) + skb->truesize >=
-	    (unsigned)sk->sk_rcvbuf)
+	if (atomic_read(&sk->sk_rmem_alloc) >= (unsigned)sk->sk_rcvbuf)
 		goto drop_n_acct;
 
 	if (skb_shared(skb)) {

net/packet/af_packet.c static int tpacket_rcv(struct sk_buff *skb, struct net_device *dev, · line 1768

Quoted 8 lines
net/packet/af_packet.c +1 -1
@@ -1763,7 +1762,7 @@ static int tpacket_rcv(struct sk_buff *skb, struct net_device *dev,
 	if (po->tp_version <= TPACKET_V2) {
 		if (macoff + snaplen > po->rx_ring.frame_size) {
 			if (po->copy_thresh &&
-				atomic_read(&sk->sk_rmem_alloc) + skb->truesize
+				atomic_read(&sk->sk_rmem_alloc)
 				< (unsigned)sk->sk_rcvbuf) {
 				if (skb_shared(skb)) {
 					copy_skb = skb_clone(skb, GFP_ATOMIC);

It doesn't work too.
It is possible to see this behavior on qemu. What about if I prepare you package with cross toolchain, rootfs and you can add debug message where you want?

Quoted 5 lines

I have also tried ll_temac driver with ppc440 and behavior is the same.

Max FRAME_SIZE pass to netdev_alloc_skb_ip_align is 7966. For this value ping works. (For ll_temac driver it is #define XTE_JUMBO_MTU 7948 from ll_temac.h)

I wonder if you applied/tested my patch correctly, since it really should had help in your case (allowing first received packet to be queued, even if very big)

Given the way NIC drivers work (pre-allocating big buffers before hardware fill frames in them), SO_RCVBUF limits are difficult to respect.

We should allow at least one frame, even with a very small SO_RCVBUF setting, or silently cap a minimum sane value.

#20 lore raw
In reply to Eric Dumazet
Quoted 80 lines · Eric Dumazet
Quoted 64 lines · Eric Dumazet
Quoted 56 lines · Michal Simek
Quoted 42 lines · Eric Dumazet
Quoted 3 lines · Michal Simek

ok. Can you provide me any background why size should be setup by size = SKB_WITH_OVERHEAD(ksize(data));
and not to use size which is passed to kmalloc in __alloc_skb.

Its all about memory accounting (based on skb->truesize)

Prior to the patch, we could fool memory accounting because skbs claimed to use less memory than what they really used.

And crash machines eventually.

Now memory accouting is fixed, we probably need to change some points in the kernel, where we previously accepted a small skb, but not a very large one.

Since "ping" probably uses SOCK_RAW sockets, I'll try this one :

(We dont care of _this_ skb truesize, only on the count of previously queued packets)

net/packet/af_packet.c +1 -2
index 0da505c..a809a48 100644
@@ -1631,8 +1631,7 @@ static int packet_rcv(struct sk_buff *skb, struct net_device *dev,
 	if (snaplen > res)
 		snaplen = res;
 
-	if (atomic_read(&sk->sk_rmem_alloc) + skb->truesize >=
-	    (unsigned)sk->sk_rcvbuf)
+	if (atomic_read(&sk->sk_rmem_alloc) >= (unsigned)sk->sk_rcvbuf)
 		goto drop_n_acct;
 
 	if (skb_shared(skb)) {

net/packet/af_packet.c static int tpacket_rcv(struct sk_buff *skb, struct net_device *dev, · line 1768

Quoted 8 lines
net/packet/af_packet.c +1 -1
@@ -1763,7 +1762,7 @@ static int tpacket_rcv(struct sk_buff *skb, struct net_device *dev,
 	if (po->tp_version <= TPACKET_V2) {
 		if (macoff + snaplen > po->rx_ring.frame_size) {
 			if (po->copy_thresh &&
-				atomic_read(&sk->sk_rmem_alloc) + skb->truesize
+				atomic_read(&sk->sk_rmem_alloc)
 				< (unsigned)sk->sk_rcvbuf) {
 				if (skb_shared(skb)) {
 					copy_skb = skb_clone(skb, GFP_ATOMIC);

It doesn't work too.
It is possible to see this behavior on qemu. What about if I prepare you package with cross toolchain, rootfs and you can add debug message where you want?

Quoted 4 lines

I have also tried ll_temac driver with ppc440 and behavior is the same.

Max FRAME_SIZE pass to netdev_alloc_skb_ip_align is 7966. For this value ping works. (For ll_temac driver it is #define XTE_JUMBO_MTU 7948 from ll_temac.h)

I did several tests with MTU 9000 on my machines and it works well.

Quoted 7 lines

It seems my pings (iputils-sss20071127 or iputils-sss20101006) uses a big enough RCVBUF

setsockopt(3, SOL_SOCKET, SO_RCVBUF, [65536], 4) = 0 getsockopt(3, SOL_SOCKET, SO_RCVBUF, [131072], [4]) = 0

Could you check with "strace ping..." what is doing busybox ?

I found it : Its too small for jumbo frames.

setsockopt(3, SOL_SOCKET, SO_RCVBUF, [7280], 4) = 0

networking/ping.c

Quoted 8 lines

/* set recv buf (needed if we can get lots of responses: flood ping,
* broadcast ping etc) */
sockopt = (datalen * 2) + 7 * 1024; /* giving it a bit of extra room */ setsockopt(pingsock, SOL_SOCKET, SO_RCVBUF, &sockopt, sizeof(sockopt));

--

Why receive buffer size MUST bigger than the jumbo frames size even if the packet size is small?

Quoted 3 lines

To unsubscribe from this list: send the line "unsubscribe netdev" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

#21 lore raw
Quoted 2 lines · Jun Zhao

Why receive buffer size MUST bigger than the jumbo frames size even if the packet size is small?

Thats the way it is in linux.

We count the memory size used by the packet.

If not, a malicious attacker could send 1-byte frames and exhaust your kernel memory.

If you want to reduce it, you might use copybreak : Some drivers copy the data into a small skb instead of providing a jumbo frame to upper stack.

tg3 for example.

#22 lore raw
Quoted 9 lines · Eric Dumazet
Quoted 2 lines · Jun Zhao

Why receive buffer size MUST bigger than the jumbo frames size even if the packet size is small?

Thats the way it is in linux.

We count the memory size used by the packet.

Quoted 11 lines

If not, a malicious attacker could send 1-byte frames and exhaust your kernel memory.

If you want to reduce it, you might use copybreak : Some drivers copy the data into a small skb instead of providing a jumbo frame to upper stack.

tg3 for example.

Eric Dumazet:

I got it, thanks for you explanation.

#23 lore raw
In reply to Eric Dumazet
Quoted 3 lines

If not, a malicious attacker could send 1-byte frames and exhaust your kernel memory.

That used to happen in SVR4 - it was possible for single byte TCP (etc) data to be queued at a STREAM head (which counted actual data bytes) in a 2k message block.
Actually we also managed to use all kernel memory queueing zero sized STREAMS messages.

Quoted 3 lines

If you want to reduce it, you might use copybreak : Some drivers copy the data into a small skb instead of providing a jumbo frame to upper stack.

Many, many moons ago I wrote an ethernet driver that received into an array of 128 (mostly) 512byte buffers. Full sized frames would have multiple rx ring entries, but could almost always be copied into a correctly
sized buffer with a single aligned copy (cache-line aligned if useful). This was significantly faster than other schemes - especially on systems where the iommu needed setting to allow the ethernet hardware to access memory.

I don't know if the linux skb buffers would allow the ethernet driver to use (say) 1600 byte rx buffers, and link them together when a long frame arrives. Most ethernet HW I've seen will fragment long receives.

The painful hardware is that which enforces a 4n byte alignment on the rx buffer! - on systens that can't do misaligned accesses. 2 bytes of junk would be fine!

David

#24 lore raw
Quoted 9 lines · David Laight

Many, many moons ago I wrote an ethernet driver that received into an array of 128 (mostly) 512byte buffers. Full sized frames would have multiple rx ring entries, but could almost always be copied into a correctly
sized buffer with a single aligned copy (cache-line aligned if useful). This was significantly faster than other schemes - especially on systems where the iommu needed setting to allow the ethernet hardware to access memory.

Quoted 8 lines

I don't know if the linux skb buffers would allow the ethernet driver to use (say) 1600 byte rx buffers, and link them together when a long frame arrives. Most ethernet HW I've seen will fragment long receives.

The painful hardware is that which enforces a 4n byte alignment on the rx buffer! - on systens that can't do misaligned accesses. 2 bytes of junk would be fine!

It all depends on hardware capabilities.

Old hardware required a single area per frame. So driver had to talk to the hardware the same way.

We now have hardware able to split data into several frags to reduce the overhead.

(Even using different pools to get a low number of frags, and good filling ratio)

Take a look at NIU for example.

#26 lore raw
In reply to Eric Dumazet
Quoted 3 lines · Eric Dumazet

I wonder if you applied/tested my patch correctly, since it really should had help in your case (allowing first received packet to be queued, even if very big)

Hmm, I missed another spot in sock_queue_rcv_skb(). (RAW sockets are not PACKET :) )

Here is the patch again, I tested it with 'busybox ping' and MTU=9000, it solved the problem for me.

Thanks

[PATCH net-next] net: relax rcvbuf limits

skb->truesize might be big even for a small packet.

Its even bigger after commit 87fb4b7b533 (net: more accurate skb truesize) and big MTU.

We should allow queueing at least one packet per receiver, even with a low RCVBUF setting.

---
 include/net/sock.h     |    4 +++-
 net/core/sock.c        |    6 +-----
 net/packet/af_packet.c |    6 ++----
 3 files changed, 6 insertions(+), 10 deletions(-)
include/net/sock.h +3 -1
index bf6b9fd..21bb3b5 100644
@@ -662,12 +662,14 @@ static inline void __sk_add_backlog(struct sock *sk, struct sk_buff *skb)
 
 /*
  * Take into account size of receive queue and backlog queue
+ * Do not take into account this skb truesize,
+ * to allow even a single big packet to come.
  */
 static inline bool sk_rcvqueues_full(const struct sock *sk, const struct sk_buff *skb)
 {
 	unsigned int qsize = sk->sk_backlog.len + atomic_read(&sk->sk_rmem_alloc);
 
-	return qsize + skb->truesize > sk->sk_rcvbuf;
+	return qsize > sk->sk_rcvbuf;
 }
 
 /* The per-socket spinlock must be held here. */
net/core/sock.c +1 -5
index a343286..347b6d9 100644
@@ -339,11 +339,7 @@ int sock_queue_rcv_skb(struct sock *sk, struct sk_buff *skb)
 	unsigned long flags;
 	struct sk_buff_head *list = &sk->sk_receive_queue;
 
-	/* Cast sk->rcvbuf to unsigned... It's pointless, but reduces
-	   number of warnings when compiling with -W --ANK
-	 */
-	if (atomic_read(&sk->sk_rmem_alloc) + skb->truesize >=
-	    (unsigned)sk->sk_rcvbuf) {
+	if (atomic_read(&sk->sk_rmem_alloc) >= sk->sk_rcvbuf) {
 		atomic_inc(&sk->sk_drops);
 		trace_sock_rcvqueue_full(sk, skb);
 		return -ENOMEM;
net/packet/af_packet.c +2 -4
index 0da505c..e56ca75 100644
@@ -1631,8 +1631,7 @@ static int packet_rcv(struct sk_buff *skb, struct net_device *dev,
 	if (snaplen > res)
 		snaplen = res;
 
-	if (atomic_read(&sk->sk_rmem_alloc) + skb->truesize >=
-	    (unsigned)sk->sk_rcvbuf)
+	if (atomic_read(&sk->sk_rmem_alloc) >= sk->sk_rcvbuf)
 		goto drop_n_acct;
 
 	if (skb_shared(skb)) {
@@ -1763,8 +1762,7 @@ static int tpacket_rcv(struct sk_buff *skb, struct net_device *dev,
 	if (po->tp_version <= TPACKET_V2) {
 		if (macoff + snaplen > po->rx_ring.frame_size) {
 			if (po->copy_thresh &&
-				atomic_read(&sk->sk_rmem_alloc) + skb->truesize
-				< (unsigned)sk->sk_rcvbuf) {
+			    atomic_read(&sk->sk_rmem_alloc) < sk->sk_rcvbuf) {
 				if (skb_shared(skb)) {
 					copy_skb = skb_clone(skb, GFP_ATOMIC);
 				} else {
maintainer Applied
#27 lore raw

From: Eric Dumazet <[email protected]>
Date: Wed, 21 Dec 2011 18:11:44 +0100

Quoted 4 lines

[PATCH net-next] net: relax rcvbuf limits

skb->truesize might be big even for a small packet.

Quoted 8 lines

Its even bigger after commit 87fb4b7b533 (net: more accurate skb truesize) and big MTU.

We should allow queueing at least one packet per receiver, even with a low RCVBUF setting.

Applied to net-next, although I was tempted to put it into net.

We may end up backporting this into -stable at some point, we'll see.

#28 lore raw
Quoted 13 lines · David Miller

From: Eric Dumazet <[email protected]>
Date: Wed, 21 Dec 2011 18:11:44 +0100

Quoted 10 lines

[PATCH net-next] net: relax rcvbuf limits

skb->truesize might be big even for a small packet.

Its even bigger after commit 87fb4b7b533 (net: more accurate skb truesize) and big MTU.

We should allow queueing at least one packet per receiver, even with a low RCVBUF setting.

Quoted 7 lines
Quoted 2 lines

Applied to net-next, although I was tempted to put it into net.

We may end up backporting this into -stable at some point, we'll see.

Yes, it works. Thanks Eric.

I hope that this patch will be in v3.2.

Thanks,
Michal

Signature · 4 lines
Michal Simek, Ing. (M.Eng)
w: www.monstr.eu p: +42-0-721842854
Maintainer of Linux kernel 2.6 Microblaze Linux - http://www.monstr.eu/fdt/
Microblaze U-BOOT custodian
#29 lore raw
Quoted 17 lines · Michal Simek
Quoted 16 lines · David Miller

From: Eric Dumazet <[email protected]>
Date: Wed, 21 Dec 2011 18:11:44 +0100

Quoted 12 lines

[PATCH net-next] net: relax rcvbuf limits

skb->truesize might be big even for a small packet.

Its even bigger after commit 87fb4b7b533 (net: more accurate skb truesize) and big MTU.

We should allow queueing at least one packet per receiver, even with a low RCVBUF setting.

Quoted 9 lines
Quoted 4 lines

Applied to net-next, although I was tempted to put it into net.

We may end up backporting this into -stable at some point, we'll see.

Yes, it works. Thanks Eric.

I hope that this patch will be in v3.2.

Thanks for testing !

I overlooked fact that commit 87fb4b7b533 was already in 3.2, so yes, we probably can push this to 3.2

(By the way, busybox ping probably doesnt work on 3.1 kernel with a MTU=9000 non copybreak driver, so its not a clear 3.2 regression)

#30 lore raw
Quoted 23 lines · Eric Dumazet
Quoted 22 lines · Michal Simek
Quoted 13 lines · David Miller

From: Eric Dumazet <[email protected]>
Date: Wed, 21 Dec 2011 18:11:44 +0100

Quoted 10 lines

[PATCH net-next] net: relax rcvbuf limits

skb->truesize might be big even for a small packet.

Its even bigger after commit 87fb4b7b533 (net: more accurate skb truesize) and big MTU.

We should allow queueing at least one packet per receiver, even with a low RCVBUF setting.

Quoted 6 lines
Quoted 2 lines

Applied to net-next, although I was tempted to put it into net.

We may end up backporting this into -stable at some point, we'll see.

Yes, it works. Thanks Eric.

Quoted 7 lines
Quoted 2 lines

I hope that this patch will be in v3.2.

Thanks for testing !

I overlooked fact that commit 87fb4b7b533 was already in 3.2, so yes, we probably can push this to 3.2

great.

Quoted 2 lines

(By the way, busybox ping probably doesnt work on 3.1 kernel with a MTU=9000 non copybreak driver, so its not a clear 3.2 regression)

good to know.

Thanks,
Michal

Signature · 4 lines
Michal Simek, Ing. (M.Eng)
w: www.monstr.eu p: +42-0-721842854
Maintainer of Linux kernel 2.6 Microblaze Linux - http://www.monstr.eu/fdt/
Microblaze U-BOOT custodian
#31 lore raw
In reply to Michal Simek
Quoted 10 lines · Michal Simek

Hi Eric and David,

I have found one problem with ll_temac driver and
this commit: net: more accurate skb truesize
sha1: 87fb4b7b533073eeeaed0b6bf7c2328995f6c075

The problem is only with icmp packets from the target. It is sent and driver receive it
but it is not proceed to the application.

Hi, Michal

What's the type of icmp you are using? such as "EchoReps", we can find the actually handler routine by this type.

And you said the packet already received by driver, can you tell me the way you used to confirm this?

And have you checked when was the icmp packet being dropped, is it in icmp_rcv or before or later?

Thanks,
Michael Wang

Quoted 16 lines

The problem I see is that kmalloc_node_track_caller allocate specific size and then this size is changed by SKB_WITH_OVERHEAD(ksize(data)).
The problem is with netdev_alloc_skb_ip_align which calls __alloc_skb function.

Currently driver uses setting for jumbo frames (9k). When I change it to use mtu (1,5k) then
everything is ok.

Can you give me some hints what can be wrong?

Thanks,
Michal

#32 lore raw
Quoted 15 lines · Michael Wang
Quoted 10 lines · Michal Simek

Hi Eric and David,

I have found one problem with ll_temac driver and
this commit: net: more accurate skb truesize
sha1: 87fb4b7b533073eeeaed0b6bf7c2328995f6c075

The problem is only with icmp packets from the target. It is sent and driver receive it
but it is not proceed to the application.

Hi, Michal

Quoted 8 lines

What's the type of icmp you are using? such as "EchoReps", we can find the actually handler routine by this type.

And you said the packet already received by driver, can you tell me the way you used to confirm this?

And have you checked when was the icmp packet being dropped, is it in icmp_rcv or before or later?

Packet was dropped right before being queued in RAW socket receive_queue, because of low sk_rcvbuf setting (against skb->truesize)

#33 lore raw
Quoted 19 lines · Eric Dumazet
Quoted 18 lines · Michael Wang
Quoted 10 lines · Michal Simek

Hi Eric and David,

I have found one problem with ll_temac driver and
this commit: net: more accurate skb truesize
sha1: 87fb4b7b533073eeeaed0b6bf7c2328995f6c075

The problem is only with icmp packets from the target. It is sent and driver receive it
but it is not proceed to the application.

Hi, Michal

What's the type of icmp you are using? such as "EchoReps", we can find the actually handler routine by this type.

Quoted 9 lines
Quoted 5 lines

And you said the packet already received by driver, can you tell me the way you used to confirm this?

And have you checked when was the icmp packet being dropped, is it in icmp_rcv or before or later?

Packet was dropped right before being queued in RAW socket receive_queue, because of low sk_rcvbuf setting (against skb->truesize)

Hi, Eric

So do you mean the way is:

ip_local_deliver_finish --> raw_local_deliver --> raw_v4_input --> raw_rcv --> raw_rcv_skb --> sock_queue_rcv_skb

And in sock_queue_rcv_skb:

        if (atomic_read(&sk->sk_rmem_alloc) + skb->truesize >=
            (unsigned)sk->sk_rcvbuf) {
                atomic_inc(&sk->sk_drops);
                trace_sock_rcvqueue_full(sk, skb);
                return -ENOMEM;				//drop?
        }

Is this the way the packet being dropped? Is this confirmed by some printk?

Sorry if the question is naive, I just want to make sure I am checking at the right place.

Regards,
Michael Wang

Quoted 2 lines
#34 lore raw
Quoted 16 lines · Michael Wang

Hi, Eric

So do you mean the way is:

ip_local_deliver_finish --> raw_local_deliver --> raw_v4_input --> raw_rcv --> raw_rcv_skb --> sock_queue_rcv_skb

And in sock_queue_rcv_skb:

        if (atomic_read(&sk->sk_rmem_alloc) + skb->truesize >=
            (unsigned)sk->sk_rcvbuf) {
                atomic_inc(&sk->sk_drops);
                trace_sock_rcvqueue_full(sk, skb);
                return -ENOMEM;				//drop?
        }
Quoted 5 lines

Is this the way the packet being dropped? Is this confirmed by some printk?

Sorry if the question is naive, I just want to make sure I am checking at the right place.

Yep

Are you aware I already posted a patch to solve the problem ?

http://patchwork.ozlabs.org/patch/132698/

#35 lore raw
Quoted 24 lines · Eric Dumazet
Quoted 16 lines · Michael Wang

Hi, Eric

So do you mean the way is:

ip_local_deliver_finish --> raw_local_deliver --> raw_v4_input --> raw_rcv --> raw_rcv_skb --> sock_queue_rcv_skb

And in sock_queue_rcv_skb:

        if (atomic_read(&sk->sk_rmem_alloc) + skb->truesize >=
            (unsigned)sk->sk_rcvbuf) {
                atomic_inc(&sk->sk_drops);
                trace_sock_rcvqueue_full(sk, skb);
                return -ENOMEM;				//drop?
        }
Quoted 5 lines

Is this the way the packet being dropped? Is this confirmed by some printk?

Sorry if the question is naive, I just want to make sure I am checking at the right place.

Quoted 6 lines

Yep

Are you aware I already posted a patch to solve the problem ?

http://patchwork.ozlabs.org/patch/132698/

Oh, sorry, I don't know about that, congratulation :)

Regards,
Michael Wang

Quoted 2 lines