BUG: NIU driver: strange issues with multicast "UDP: short packet"
Subsystems (3)
I have this strange issues with multicast and the NIU driver (Sun neptune Quad NIC). When receiving multicast traffic the kernel log spams with something like this:
__ratelimit: 5071 callbacks suppressed
UDP: short packet: From 81.161.2.106:0 44063/1324 to 233.123.173.7:0
UDP: short packet: From 81.161.2.106:0 44063/1324 to 233.123.173.7:0
UDP: short packet: From 81.161.2.106:8304 27493/1324 to 233.123.173.7:24931
UDP: short packet: From 81.161.2.106:8304 27493/1324 to 233.123.173.7:24931
UDP: short packet: From 81.161.2.106:8304 27493/1324 to 233.123.173.7:24931
UDP: short packet: From 81.161.2.106:8304 27493/1324 to 233.123.173.7:24931
UDP: short packet: From 81.161.2.106:8304 27493/1324 to 233.123.173.7:24931
UDP: short packet: From 81.161.2.106:8304 27493/1324 to 233.123.173.7:24931
UDP: short packet: From 81.161.2.106:8304 27493/1324 to 233.123.173.7:24931
UDP: short packet: From 81.161.2.106:8304 27493/1324 to 233.123.173.7:24931 __ratelimit: 3555 callbacks suppressed
UDP: short packet: From 81.161.2.106:0 49320/1324 to 233.123.173.7:0
UDP: short packet: From 81.161.2.106:0 49320/1324 to 233.123.173.7:0
UDP: short packet: From 81.161.2.106:8304 27493/1324 to 233.123.173.7:24931
UDP: short packet: From 81.161.2.106:8304 27493/1324 to 233.123.173.7:24931
UDP: short packet: From 81.161.2.106:8304 27493/1324 to 233.123.173.7:24931
Tcpdumping the packets, the contents of the packets are correct.
The next strange thing is that I can make the log messages go away, if I setup a static multicast route out another interface.
smcroute -a eth52 81.161.2.106 233.123.173.7 eth21
Any hints whats going wrong?
Signature · 7 lines
Med venlig hilsen / Best regards Jesper Brouer ComX Networks A/S Linux Network developer Cand. Scient Datalog / MSc. Author of http://adsl-optimizer.dk LinkedIn: http://www.linkedin.com/in/brouer
David Miller
maintainerFrom: Jesper Dangaard Brouer <[email protected]>
Date: Tue, 03 Feb 2009 14:38:20 +0100
Quoted 6 lines
UDP: short packet: From 81.161.2.106:0 44063/1324 to 233.123.173.7:0
UDP: short packet: From 81.161.2.106:0 44063/1324 to 233.123.173.7:0
UDP: short packet: From 81.161.2.106:8304 27493/1324 to 233.123.173.7:24931
UDP: short packet: From 81.161.2.106:8304 27493/1324 to 233.123.173.7:24931
UDP: short packet: From 81.161.2.106:8304 27493/1324 to 233.123.173.7:24931
UDP: short packet: From 81.161.2.106:8304 27493/1324 to 233.123.173.7:24931
The UDP header length field is garbage in all of these packets.
In the first two packets here, the source and dest ports are both zero. This is also garbage.,
Quoted 1 line
Tcpdumping the packets, the contents of the packets are correct.
So something corrupts the packet on the way to UDP input.
Quoted 4 lines
The next strange thing is that I can make the log messages go away, if I setup a static multicast route out another interface.
smcroute -a eth52 81.161.2.106 233.123.173.7 eth21
That could be a good clue.
Do you happen to have multicast routing enabled on this machine? If the multicast destination is non-local and IN_DEV_FORWARD is set on the interface, that puts the packet through ip_mr_input.
ip_mr_input() will clone the SKB if it should be delivered locally as well as be forwarded.
If the packet does get multicast forwarded, there is all kinds of funny code that mangles the packet, for example ipmr_cache_report().
Any of this could be corrupting what UDP ends up seeing.
To be honest all of the SKB handling in the ipmr.c file is very scary.
Quoted 11 lines · David Miller
From: Jesper Dangaard Brouer <[email protected]>
Date: Tue, 03 Feb 2009 14:38:20 +0100Quoted 6 lines
UDP: short packet: From 81.161.2.106:0 44063/1324 to 233.123.173.7:0
UDP: short packet: From 81.161.2.106:0 44063/1324 to 233.123.173.7:0
UDP: short packet: From 81.161.2.106:8304 27493/1324 to 233.123.173.7:24931
UDP: short packet: From 81.161.2.106:8304 27493/1324 to 233.123.173.7:24931
UDP: short packet: From 81.161.2.106:8304 27493/1324 to 233.123.173.7:24931
UDP: short packet: From 81.161.2.106:8304 27493/1324 to 233.123.173.7:24931The UDP header length field is garbage in all of these packets.
Yes, but this only happens on the NIU/netpune NIC. I works with the igb driver, I have both a 82575 and a 82576 NIC.
Quoted 7 lines
In the first two packets here, the source and dest ports are both zero. This is also garbage.,
Quoted 1 line
Tcpdumping the packets, the contents of the packets are correct.
So something corrupts the packet on the way to UDP input.
Quoted 8 lines
Quoted 4 lines
The next strange thing is that I can make the log messages go away, if I setup a static multicast route out another interface.
smcroute -a eth52 81.161.2.106 233.123.173.7 eth21
That could be a good clue.
Do you happen to have multicast routing enabled on this machine?
YesCONFIG_IP_MULTICAST=y
Looking through ipmr.c, I should also tell you that I have enabled:
CONFIG_IP_PIMSM_V1=yCONFIG_IP_PIMSM_V2=y
Quoted 2 lines
If the multicast destination is non-local and
IN_DEV_FORWARDis set on the interface, that puts the packet throughip_mr_input.
Don't you mean IN_DEV_MFORWARD ?
Quoted 2 lines
ip_mr_input()will clone the SKB if it should be delivered locally as well as be forwarded.
Interesting another path is choosen for mc forwarding.
If I setup a bridge for L2 forward the packets, what path is choosen then?
Quoted 6 lines
If the packet does get multicast forwarded, there is all kinds of funny code that mangles the packet, for example
ipmr_cache_report().Any of this could be corrupting what UDP ends up seeing.
To be honest all of the SKB handling in the
ipmr.cfile is very scary.
Hmmm, that doesn't sound very reassuring...
See you around...
Signature · 7 lines
Med venlig hilsen / Best regards Jesper Brouer ComX Networks A/S Linux Network developer Cand. Scient Datalog / MSc. Author of http://adsl-optimizer.dk LinkedIn: http://www.linkedin.com/in/brouer
David Miller
maintainerFrom: Jesper Dangaard Brouer <[email protected]>
Date: Wed, 04 Feb 2009 09:55:04 +0100
Quoted 11 lines · David Miller
Quoted 10 lines
From: Jesper Dangaard Brouer <[email protected]>
Date: Tue, 03 Feb 2009 14:38:20 +0100Quoted 6 lines
UDP: short packet: From 81.161.2.106:0 44063/1324 to 233.123.173.7:0
UDP: short packet: From 81.161.2.106:0 44063/1324 to 233.123.173.7:0
UDP: short packet: From 81.161.2.106:8304 27493/1324 to 233.123.173.7:24931
UDP: short packet: From 81.161.2.106:8304 27493/1324 to 233.123.173.7:24931
UDP: short packet: From 81.161.2.106:8304 27493/1324 to 233.123.173.7:24931
UDP: short packet: From 81.161.2.106:8304 27493/1324 to 233.123.173.7:24931
Quoted 4 lines
Quoted 1 line
The UDP header length field is garbage in all of these packets.
Yes, but this only happens on the NIU/netpune NIC. I works with the igb driver, I have both a 82575 and a 82576 NIC.
Right, and what's unique about NIU is that NIU won't prepull the protocol headers into the linear area on receive.
At the point when IPMR is dealing with potentially forwarding the frame, the UDP headers aren't yet pulled into the linear area. UDP input will do that with it's pskb_may_pull() call.
I think this is the critical bit, and some part of the IPMR code makes assumptions about all of the protocol headers being pulled into the linear SKB area when it executes.
I really don't have time to track this down, so what I suggest you do is add logging to UDP multicast packets at various locations looking for the UDP header length in the paged SKB area to go illegal. Move your probe points around to narrow down the culprit.
Good luck.
Quoted 14 lines · David Miller
From: Jesper Dangaard Brouer <[email protected]>
Date: Wed, 04 Feb 2009 09:55:04 +0100Quoted 11 lines · David Miller
Quoted 10 lines
From: Jesper Dangaard Brouer <[email protected]>
Date: Tue, 03 Feb 2009 14:38:20 +0100Quoted 6 lines
UDP: short packet: From 81.161.2.106:0 44063/1324 to 233.123.173.7:0
UDP: short packet: From 81.161.2.106:0 44063/1324 to 233.123.173.7:0
UDP: short packet: From 81.161.2.106:8304 27493/1324 to 233.123.173.7:24931
UDP: short packet: From 81.161.2.106:8304 27493/1324 to 233.123.173.7:24931
UDP: short packet: From 81.161.2.106:8304 27493/1324 to 233.123.173.7:24931
UDP: short packet: From 81.161.2.106:8304 27493/1324 to 233.123.173.7:24931
Quoted 7 lines
Quoted 4 lines
Quoted 1 line
The UDP header length field is garbage in all of these packets.
Yes, but this only happens on the NIU/netpune NIC. I works with the igb driver, I have both a 82575 and a 82576 NIC.
Right, and what's unique about NIU is that NIU won't prepull the protocol headers into the linear area on receive.
So thats the difference...
Quoted 3 lines
At the point when IPMR is dealing with potentially forwarding the frame, the UDP headers aren't yet pulled into the linear area. UDP input will do that with it's
pskb_may_pull()call.
Can I get a little code hint, where IPMR is dealing with potentially forwarding the frame?
Quoted 9 lines
I think this is the critical bit, and some part of the IPMR code makes assumptions about all of the protocol headers being pulled into the linear SKB area when it executes.
I really don't have time to track this down, so what I suggest you do is add logging to UDP multicast packets at various locations looking for the UDP header length in the paged SKB area to go illegal. Move your probe points around to narrow down the culprit.
Is there an easy way to test if the packet is corrupted? Can you recomment a way to test it?
Quoted 1 line
Good luck.
Thanks I'm going to need it ;-)
Thanks for you quick answer.
Signature · 7 lines
Med venlig hilsen / Best regards Jesper Brouer ComX Networks A/S Linux Network developer Cand. Scient Datalog / MSc. Author of http://adsl-optimizer.dk LinkedIn: http://www.linkedin.com/in/brouer
David Miller
maintainerFrom: Jesper Dangaard Brouer <[email protected]>
Date: Wed, 04 Feb 2009 10:32:06 +0100
Quoted 7 lines · David Miller
Quoted 3 lines
At the point when IPMR is dealing with potentially forwarding the frame, the UDP headers aren't yet pulled into the linear area. UDP input will do that with it's
pskb_may_pull()call.Can I get a little code hint, where IPMR is dealing with potentially forwarding the frame?
ip_mr_forward().
Quoted 2 lines
Is there an easy way to test if the packet is corrupted? Can you recomment a way to test it?
If UDP header length field is > 2000, you've got a problem.
Quoted 10 lines · David Miller
Quoted 1 line
Yes, but this only happens on the NIU/netpune NIC. I works with the
igb
Quoted 1 line
driver, I have both a 82575 and a 82576 NIC.
Right, and what's unique about NIU is that NIU won't prepull the protocol headers into the linear area on receive.
At the point when IPMR is dealing with potentially forwarding the frame, the UDP headers aren't yet pulled into the linear area. UDP input will do that with it's
pskb_may_pull()call.
I think I have found the problem. And its actually due to the pskb_may_pull() call in net/ipv4/udp.c:__udp4_lib_rcv().
pskb_may_pull() may alter the SKB and a pointer to the UDP header was stored before the pskb_may_pull() call.
I'll send a patch...
Signature · 7 lines
Med venlig hilsen / Best regards Jesper Brouer ComX Networks A/S Linux Network developer Cand. Scient Datalog / MSc. Author of http://adsl-optimizer.dk LinkedIn: http://www.linkedin.com/in/brouer
PATCH 1/1 Fix UDP short packet false positive
The UDP header pointer assignment must happen after calling pskb_may_pull(). As pskb_may_pull() can potentially alter the SKB buffer.
This was exposted by running multicast traffic through the NIU driver, as it won't prepull the protocol headers into the linear area on receive.
---
net/ipv4/udp.c | 3 ++- 1 files changed, 2 insertions(+), 1 deletions(-)
net/ipv4/udp.c +2 -1
@@ -1231,7 +1231,7 @@ int __udp4_lib_rcv(struct sk_buff *skb, struct udp_table *udptable, int proto) {struct sock *sk; -struct udphdr *uh =udp_hdr (skb); +struct udphdr *uh;unsigned short ulen;struct rtable *rt = (struct rtable *)skb->dst;__be32 saddr =ip_hdr (skb)->saddr;
@@ -1244,6 +1244,7 @@ int __udp4_lib_rcv(struct sk_buff *skb, struct udp_table *udptable,if (!pskb_may_pull (skb,sizeof (struct udphdr ))) goto drop;/* No space for header. */ + uh =udp_hdr (skb); ulen =ntohs (uh->len);if (ulen > skb->len) goto short_packet;
David Miller
maintainerRe: [PATCH] Fix UDP short packet false positive
From: Jesper Dangaard Brouer <[email protected]>
Date: Thu, 05 Feb 2009 13:47:07 +0100
Quoted 10 lines
The UDP header pointer assignment must happen after calling
pskb_may_pull(). Aspskb_may_pull()can potentially alter the SKB buffer.This was exposted by running multicast traffic through the NIU driver, as it won't prepull the protocol headers into the linear area on receive.
Signed-off-by Jesper Dangaard Brouer <[email protected]>
Excellent work!
Applied and queued up for -stable, thanks!
RFC PATCH 1/1 Fix UDP short packet false positive
Quoted 8 lines · David Miller
From: Jesper Dangaard Brouer <[email protected]>
Date: Thu, 05 Feb 2009 13:47:07 +0100Quoted 3 lines
The UDP header pointer assignment must happen after calling
pskb_may_pull(). Aspskb_may_pull()can potentially alter the SKB buffer.Excellent work!
Thanks :-)
I'm wondering if the ip_hdr() pointer can be changed by the pskb_may_pull(), but I assume it cannot as it should already be in the linear area... right?
Well the patch below, shows what I mean...
net/ipv4/udp.c +5 -2
@@ -1232,20 +1232,23 @@ int __udp4_lib_rcv(struct sk_buff *skb, struct udp_table *udptable, {struct sock *sk;struct udphdr *uh;unsigned short ulen;struct rtable *rt = (struct rtable *)skb->dst; -__be32 saddr =ip_hdr (skb)->saddr; -__be32 daddr =ip_hdr (skb)->daddr; +__be32 saddr; +__be32 daddr;struct net *net =dev_net (skb->dev);/* * Validate the packet. */ if (!pskb_may_pull (skb,sizeof (struct udphdr ))) goto drop;/* No space for header. */ + saddr =ip_hdr (skb)->saddr; + daddr =ip_hdr (skb)->daddr; + uh =udp_hdr (skb); ulen =ntohs (uh->len);if (ulen > skb->len) goto short_packet;
Signature · 7 lines
Med venlig hilsen / Best regards Jesper Brouer ComX Networks A/S Linux Network developer Cand. Scient Datalog / MSc. Author of http://adsl-optimizer.dk LinkedIn: http://www.linkedin.com/in/brouer
David Miller
maintainerRe: [RFC] [PATCH] Fix UDP short packet false positive
From: Jesper Dangaard Brouer <[email protected]>
Date: Fri, 06 Feb 2009 10:00:24 +0100
Quoted 10 lines · David Miller
Quoted 8 lines
From: Jesper Dangaard Brouer <[email protected]>
Date: Thu, 05 Feb 2009 13:47:07 +0100Quoted 3 lines
The UDP header pointer assignment must happen after calling
pskb_may_pull(). Aspskb_may_pull()can potentially alter the SKB buffer.Excellent work!
Quoted 7 lines
Thanks :-)
I'm wondering if the
ip_hdr()pointer can be changed by thepskb_may_pull(), but I assume it cannot as it should already be in the linear area... right?Well the patch below, shows what I mean...
It has the same potential problem, but in this case you'd only see corruption if the old skb->data buffer were reallocated by another user and written into very quickly (or poison'd by SLAB debugging).
Please respin this patch of your's with proper commit message and signoffs, thanks!
BTW, ipv6 udp gets all of this right :-)
PATCH 1/1 udp: Fix potential wrong ip_hdr(skb) pointers
Quoted 2 lines · David Miller
Please respin this patch of your's with proper commit message and signoffs, thanks!
Like the UDP header fix, pskb_may_pull() can potentially alter the SKB buffer. Thus the saddr and daddr, pointers may point to the old skb->data buffer.
I haven't seen corruptions, as its only seen if the old skb->data buffer were reallocated by another user and written into very quickly (or poison'd by SLAB debugging).
---
net/ipv4/udp.c | 6 ++++-- 1 files changed, 4 insertions(+), 2 deletions(-)
net/ipv4/udp.c +4 -2
@@ -1234,8 +1234,7 @@ int __udp4_lib_rcv(struct sk_buff *skb, struct udp_table *udptable,struct udphdr *uh;unsigned short ulen;struct rtable *rt = (struct rtable *)skb->dst; -__be32 saddr =ip_hdr (skb)->saddr; -__be32 daddr =ip_hdr (skb)->daddr; +__be32 saddr, daddr;struct net *net =dev_net (skb->dev);/*
@@ -1259,6 +1258,9 @@ int __udp4_lib_rcv(struct sk_buff *skb, struct udp_table *udptable,if (udp4_csum_init (skb, uh, proto)) goto csum_error; + saddr =ip_hdr (skb)->saddr; + daddr =ip_hdr (skb)->daddr; +if (rt->rt_flags & (RTCF_BROADCAST |RTCF_MULTICAST ))return __udp4_lib_mcast_deliver (net, skb, uh, saddr, daddr, udptable);
Re: [PATCH] udp: Fix potential wrong ip_hdr(skb) pointers
From: Jesper Dangaard Brouer <[email protected]>
Date: Fri, 06 Feb 2009 10:55:58 +0100
Quoted 8 lines · David Miller
Quoted 2 lines
Please respin this patch of your's with proper commit message and signoffs, thanks!
Like the UDP header fix,
pskb_may_pull()can potentially alter the SKB buffer. Thus the saddr and daddr, pointers may point to the oldskb->databuffer.
Quoted 5 lines
I haven't seen corruptions, as its only seen if the old
skb->databuffer were reallocated by another user and written into very quickly (or poison'd by SLAB debugging).Signed-off-by Jesper Dangaard Brouer <[email protected]>
Applied, thanks!
Re: [PATCH] udp: Fix potential wrong ip_hdr(skb) pointers
In reply to Jesper Dangaard BrouerQuoted 8 lines · Jesper Dangaard Brouer
Quoted 2 lines · David Miller
Please respin this patch of your's with proper commit message and signoffs, thanks!
Like the UDP header fix,
pskb_may_pull()can potentially alter the SKB buffer. Thus the saddr and daddr, pointers may point to the oldskb->databuffer.
I dont know... daddr and saddr are not pointers but integers.
Patch makes sense as a cleanup, but ChangeLog seems wrong ?
Quoted 37 lines
I haven't seen corruptions, as its only seen if the old
skb->databuffer were reallocated by another user and written into very quickly (or poison'd by SLAB debugging).Signed-off-by Jesper Dangaard Brouer <[email protected]>---
net/ipv4/udp.c | 6 ++++-- 1 files changed, 4 insertions(+), 2 deletions(-)net/ipv4/udp.c +4
-2@@ -1234,8 +1234,7 @@ int __udp4_lib_rcv(struct sk_buff *skb, struct udp_table *udptable,struct udphdr *uh;unsigned short ulen;struct rtable *rt = (struct rtable *)skb->dst; -__be32 saddr =ip_hdr (skb)->saddr; -__be32 daddr =ip_hdr (skb)->daddr; +__be32 saddr, daddr;struct net *net =dev_net (skb->dev);/* @@ -1259,6 +1258,9 @@ int __udp4_lib_rcv(struct sk_buff *skb, struct udp_table *udptable,if (udp4_csum_init (skb, uh, proto)) goto csum_error; + saddr =ip_hdr (skb)->saddr; + daddr =ip_hdr (skb)->daddr; +if (rt->rt_flags & (RTCF_BROADCAST |RTCF_MULTICAST ))return __udp4_lib_mcast_deliver (net, skb, uh, saddr, daddr, udptable);
Re: [PATCH] udp: Fix potential wrong ip_hdr(skb) pointers
Quoted 11 lines · Eric Dumazet
Quoted 8 lines · Jesper Dangaard Brouer
Quoted 2 lines · David Miller
Please respin this patch of your's with proper commit message and signoffs, thanks!
Like the UDP header fix,
pskb_may_pull()can potentially alter the SKB buffer. Thus the saddr and daddr, pointers may point to the oldskb->databuffer.I dont know... daddr and saddr are not pointers but integers.
Yes, you are right... its only in the ipv6 code these are pointers (which as DaveM mentioned handels it correctly).
Quoted 1 line
Patch makes sense as a cleanup, but ChangeLog seems wrong ?
Okay, lets view it as a cleanup... Its upto DaveM if he wants to fix the commit message (or ask me the correct it, revert and reapply...)
Quoted 39 lines
Quoted 37 lines
I haven't seen corruptions, as its only seen if the old
skb->databuffer were reallocated by another user and written into very quickly (or poison'd by SLAB debugging).Signed-off-by Jesper Dangaard Brouer <[email protected]>---
net/ipv4/udp.c | 6 ++++-- 1 files changed, 4 insertions(+), 2 deletions(-)net/ipv4/udp.c +4
-2@@ -1234,8 +1234,7 @@ int __udp4_lib_rcv(struct sk_buff *skb, struct udp_table *udptable,struct udphdr *uh;unsigned short ulen;struct rtable *rt = (struct rtable *)skb->dst; -__be32 saddr =ip_hdr (skb)->saddr; -__be32 daddr =ip_hdr (skb)->daddr; +__be32 saddr, daddr;struct net *net =dev_net (skb->dev);/* @@ -1259,6 +1258,9 @@ int __udp4_lib_rcv(struct sk_buff *skb, struct udp_table *udptable,if (udp4_csum_init (skb, uh, proto)) goto csum_error; + saddr =ip_hdr (skb)->saddr; + daddr =ip_hdr (skb)->daddr; +if (rt->rt_flags & (RTCF_BROADCAST |RTCF_MULTICAST ))return __udp4_lib_mcast_deliver (net, skb, uh, saddr, daddr, udptable);
Signature · 7 lines
Med venlig hilsen / Best regards Jesper Brouer ComX Networks A/S Linux Network developer Cand. Scient Datalog / MSc. Author of http://adsl-optimizer.dk LinkedIn: http://www.linkedin.com/in/brouer
David Miller
maintainerRe: [PATCH] udp: Fix potential wrong ip_hdr(skb) pointers
From: Jesper Dangaard Brouer <[email protected]>
Date: Fri, 06 Feb 2009 11:49:22 +0100
Quoted 13 lines · Eric Dumazet
Quoted 11 lines
Quoted 8 lines · Jesper Dangaard Brouer
Quoted 2 lines · David Miller
Please respin this patch of your's with proper commit message and signoffs, thanks!
Like the UDP header fix,
pskb_may_pull()can potentially alter the SKB buffer. Thus the saddr and daddr, pointers may point to the oldskb->databuffer.I dont know... daddr and saddr are not pointers but integers.
Quoted 7 lines
Yes, you are right... its only in the ipv6 code these are pointers (which as DaveM mentioned handels it correctly).
Quoted 1 line
Patch makes sense as a cleanup, but ChangeLog seems wrong ?
Okay, lets view it as a cleanup... Its upto DaveM if he wants to fix the commit message (or ask me the correct it, revert and reapply...)
I already pushed the commit out, rebasing the tree is not an option and a revert is super ugly so it's staying as-is.
Quoted 3 lines · David Miller
If the packet does get multicast forwarded, there is all kinds of funny code that mangles the packet, for example
ipmr_cache_report().
That one looks alright since it'll either allocate a new skb or at least do pskb_expand_head.
Cheers,
Signature · 4 lines
Visit Openswan at http://www.openswan.org/
Email: Herbert Xu ~{PmV>HI~} <[email protected]>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt