IPv4 multicast and mac-vlans acting weird on 3.0.4+

PATCH Applied Ben Greear 3 participants 17 messages Oct 5, 2011 – Oct 19, 2011

Subsystems (4)

Networking Network drivers IPv4/IPv6 Packet sockets

lore

Share & tools

lore

Ben Greear

author

2011-10-05 16:46 UTC #0 lore raw

This is on a hacked 3.0.4 kernel...

I am seeing an issue where an IPv4 mcast receiver will not receive a 1473 or larger byte mcast message, but will receive a 1472. The difference being that 1473 ends up being two packets on the wire. It works on 802.1Q VLANs, VETH interfaces and real Ethernet. It does not work on a mac-vlan hanging off the VETH.

I see packets received on the macvlan in tshark, and they appear correct. No obvious errors in the macvlan port stats or netstat -s,
and the 'ss' tool doesn't appear to support UDP sockets at all.

So, I'm about to go digging into the code, but if anyone has any suggestions for places to look, please let me know!

Thanks,
Ben

Signature · 2 lines

Ben Greear <[email protected]>
Candela Technologies Inc  http://www.candelatech.com

Eric Dumazet

2011-10-05 19:54 UTC #1 lore raw

Quoted 8 lines · Ben Greear

This is on a hacked 3.0.4 kernel...
I am seeing an issue where an IPv4 mcast receiver will not receive a 1473 or larger byte mcast message, but will receive a 1472. The difference being that 1473 ends up being two packets on the wire. It works on 802.1Q VLANs, VETH interfaces and real Ethernet. It does not work on a mac-vlan hanging off the VETH.

Quoted 7 lines

I see packets received on the macvlan in tshark, and they appear correct. No obvious errors in the macvlan port stats or netstat -s,
and the 'ss' tool doesn't appear to support UDP sockets at all.
So, I'm about to go digging into the code, but if anyone has any suggestions for places to look, please let me know!

Well, problem is defragmentation and macvlan cooperation.

Multicast messages are broadcasted on all macvlan ports.

But IP defrag will probably deliver a single final frame.

We probably need to handle defrag in macvlan before broadcasting to all ports.

Ben Greear

author

2011-10-05 20:09 UTC #2 lore raw

Quoted 17 lines · Eric Dumazet

Quoted 8 lines · Ben Greear
This is on a hacked 3.0.4 kernel...
I am seeing an issue where an IPv4 mcast receiver will not receive a 1473 or larger byte mcast message, but will receive a 1472. The difference being that 1473 ends up being two packets on the wire. It works on 802.1Q VLANs, VETH interfaces and real Ethernet. It does not work on a mac-vlan hanging off the VETH.
Quoted 7 lines
I see packets received on the macvlan in tshark, and they appear correct. No obvious errors in the macvlan port stats or netstat -s,
and the 'ss' tool doesn't appear to support UDP sockets at all.
So, I'm about to go digging into the code, but if anyone has any suggestions for places to look, please let me know!

Quoted 8 lines

Well, problem is defragmentation and macvlan cooperation.
Multicast messages are broadcasted on all macvlan ports.
But IP defrag will probably deliver a single final frame.
We probably need to handle defrag in macvlan before broadcasting to all ports.

I see packets get to this code in ip_input.c (line 467 or so), and that printk is mine of course.

	if ((dev && strcmp(dev->name, "rddVR10#0") == 0) ||
	    (dev && strcmp(dev->name, "rddVR10") == 0)) {
		printk("calling ip_rcv_finish through NF_HOOK, dev: %s, len: %i\n",
		       dev->name, skb->len);
	}

	return NF_HOOK(NFPROTO_IPV4, NF_INET_PRE_ROUTING, skb, dev, NULL,
		       ip_rcv_finish);

But, the macvlan packets never make it to the ip_rcv_finish method.

I do see a big and a little packet entering this code.

I have no firewall rules that I'm aware of, though there is some conn-track logic (though not associated with the mac-vlan interface):

[root@lec2010-ath9k-1 ~]# iptables -L
Chain INPUT (policy ACCEPT)
target prot opt source destination

Chain FORWARD (policy ACCEPT)
target prot opt source destination

Chain OUTPUT (policy ACCEPT)
target prot opt source destination

And, I just reproduced the problem on vanilla linus top-of-tree (3.1.0-rc9).

Thanks,
Ben

Signature · 2 lines

Ben Greear <[email protected]>
Candela Technologies Inc  http://www.candelatech.com

Eric Dumazet

2011-10-05 20:17 UTC #3 lore raw

Quoted 39 lines · Ben Greear

Quoted 17 lines · Eric Dumazet
Quoted 8 lines · Ben Greear
This is on a hacked 3.0.4 kernel...
I am seeing an issue where an IPv4 mcast receiver will not receive a 1473 or larger byte mcast message, but will receive a 1472. The difference being that 1473 ends up being two packets on the wire. It works on 802.1Q VLANs, VETH interfaces and real Ethernet. It does not work on a mac-vlan hanging off the VETH.
Quoted 7 lines
I see packets received on the macvlan in tshark, and they appear correct. No obvious errors in the macvlan port stats or netstat -s,
and the 'ss' tool doesn't appear to support UDP sockets at all.
So, I'm about to go digging into the code, but if anyone has any suggestions for places to look, please let me know!
Quoted 8 lines
Well, problem is defragmentation and macvlan cooperation.
Multicast messages are broadcasted on all macvlan ports.
But IP defrag will probably deliver a single final frame.
We probably need to handle defrag in macvlan before broadcasting to all ports.
I see packets get to this code in ip_input.c (line 467 or so), and that printk is mine of course.
	if ((dev && strcmp(dev->name, "rddVR10#0") == 0) ||
	    (dev && strcmp(dev->name, "rddVR10") == 0)) {
		printk("calling ip_rcv_finish through NF_HOOK, dev: %s, len: %i\n",
		       dev->name, skb->len);
	}

	return NF_HOOK(NFPROTO_IPV4, NF_INET_PRE_ROUTING, skb, dev, NULL,
		       ip_rcv_finish);

Quoted 7 lines

But, the macvlan packets never make it to the ip_rcv_finish method.
I do see a big and a little packet entering this code.
I have no firewall rules that I'm aware of, though there is some conn-track logic (though not associated with the mac-vlan interface):

Say you have 10 vlans on your eth0, how many times do you want one incoming multicast frame being delivered to your application listening on 0.0.0.0:port ?

10 or 1 ?

Ben Greear

author

2011-10-05 20:19 UTC #4 lore raw

Quoted 44 lines · Eric Dumazet

Quoted 43 lines · Ben Greear
Quoted 17 lines · Eric Dumazet
Quoted 8 lines · Ben Greear
This is on a hacked 3.0.4 kernel...
I am seeing an issue where an IPv4 mcast receiver will not receive a 1473 or larger byte mcast message, but will receive a 1472. The difference being that 1473 ends up being two packets on the wire. It works on 802.1Q VLANs, VETH interfaces and real Ethernet. It does not work on a mac-vlan hanging off the VETH.
Quoted 7 lines
I see packets received on the macvlan in tshark, and they appear correct. No obvious errors in the macvlan port stats or netstat -s,
and the 'ss' tool doesn't appear to support UDP sockets at all.
So, I'm about to go digging into the code, but if anyone has any suggestions for places to look, please let me know!
Quoted 8 lines
Well, problem is defragmentation and macvlan cooperation.
Multicast messages are broadcasted on all macvlan ports.
But IP defrag will probably deliver a single final frame.
We probably need to handle defrag in macvlan before broadcasting to all ports.
I see packets get to this code in ip_input.c (line 467 or so), and that printk is mine of course.
	if ((dev&&  strcmp(dev->name, "rddVR10#0") == 0) ||
	    (dev&&  strcmp(dev->name, "rddVR10") == 0)) {
		printk("calling ip_rcv_finish through NF_HOOK, dev: %s, len: %i\n",
		       dev->name, skb->len);
	}

	return NF_HOOK(NFPROTO_IPV4, NF_INET_PRE_ROUTING, skb, dev, NULL,
		       ip_rcv_finish);
But, the macvlan packets never make it to the ip_rcv_finish method.
I do see a big and a little packet entering this code.

Quoted 7 lines

Quoted 3 lines
I have no firewall rules that I'm aware of, though there is some conn-track logic (though not associated with the mac-vlan interface):
Say you have 10 vlans on your eth0, how many times do you want one incoming multicast frame being delivered to your application listening on 0.0.0.0:port ?

How would it work for two Ethernet devices on the same LAN? I'd say that mac-vlans should mimic that case.

And in my case, I'm binding hard to a device & IP address, so my app should get it once regardless.

Thanks,
Ben

Signature · 2 lines

Ben Greear <[email protected]>
Candela Technologies Inc  http://www.candelatech.com

Eric Dumazet

2011-10-05 20:31 UTC #5 lore raw

Quoted 53 lines · Ben Greear

Quoted 44 lines · Eric Dumazet
Quoted 43 lines · Ben Greear
Quoted 17 lines · Eric Dumazet
Quoted 8 lines · Ben Greear
This is on a hacked 3.0.4 kernel...
I am seeing an issue where an IPv4 mcast receiver will not receive a 1473 or larger byte mcast message, but will receive a 1472. The difference being that 1473 ends up being two packets on the wire. It works on 802.1Q VLANs, VETH interfaces and real Ethernet. It does not work on a mac-vlan hanging off the VETH.
Quoted 7 lines
I see packets received on the macvlan in tshark, and they appear correct. No obvious errors in the macvlan port stats or netstat -s,
and the 'ss' tool doesn't appear to support UDP sockets at all.
So, I'm about to go digging into the code, but if anyone has any suggestions for places to look, please let me know!
Quoted 8 lines
Well, problem is defragmentation and macvlan cooperation.
Multicast messages are broadcasted on all macvlan ports.
But IP defrag will probably deliver a single final frame.
We probably need to handle defrag in macvlan before broadcasting to all ports.
I see packets get to this code in ip_input.c (line 467 or so), and that printk is mine of course.
	if ((dev&&  strcmp(dev->name, "rddVR10#0") == 0) ||
	    (dev&&  strcmp(dev->name, "rddVR10") == 0)) {
		printk("calling ip_rcv_finish through NF_HOOK, dev: %s, len: %i\n",
		       dev->name, skb->len);
	}

	return NF_HOOK(NFPROTO_IPV4, NF_INET_PRE_ROUTING, skb, dev, NULL,
		       ip_rcv_finish);
But, the macvlan packets never make it to the ip_rcv_finish method.
I do see a big and a little packet entering this code.
Quoted 7 lines
Quoted 3 lines
I have no firewall rules that I'm aware of, though there is some conn-track logic (though not associated with the mac-vlan interface):
Say you have 10 vlans on your eth0, how many times do you want one incoming multicast frame being delivered to your application listening on 0.0.0.0:port ?

Quoted 6 lines

How would it work for two Ethernet devices on the same LAN? I'd say that mac-vlans should mimic that case.
And in my case, I'm binding hard to a device & IP address, so my app should get it once regardless.

OK, but before frame being delivered to your app, it must be re-assembled by net/ipv4/inet_fragment.c & net/ipv4/ip_fragment.c machinery.

This machinery uses :

static int ip4_frag_match(struct inet_frag_queue *q, void *a)

{
        struct ipq *qp;
        struct ip4_create_arg *arg = a;

        qp = container_of(q, struct ipq, q);
        return  qp->id == arg->iph->id &&
                        qp->saddr == arg->iph->saddr &&
                        qp->daddr == arg->iph->daddr &&
                        qp->protocol == arg->iph->protocol &&
                        qp->user == arg->user;
}

All frames broadcasted (because of multicast code in macvlan) on vlans have same saddr/daddr/protocol (and user).

So kernel will discard all redundant copies of frames and deliver one copy only to upper stack.

Check commit 7736d33f4262d437c5 (packet: Add pre-defragmentation support for ipv4 fanouts) for a possible hint :

We could perform the re-assembly in macvlan code, before doing the "broadcast the frame on all ports" part.

Ben Greear

author

2011-10-05 20:56 UTC #6 lore raw

Quoted 80 lines · Eric Dumazet

Quoted 53 lines · Ben Greear
Quoted 44 lines · Eric Dumazet
Quoted 43 lines · Ben Greear
Quoted 17 lines · Eric Dumazet
Quoted 8 lines · Ben Greear
This is on a hacked 3.0.4 kernel...
I am seeing an issue where an IPv4 mcast receiver will not receive a 1473 or larger byte mcast message, but will receive a 1472. The difference being that 1473 ends up being two packets on the wire. It works on 802.1Q VLANs, VETH interfaces and real Ethernet. It does not work on a mac-vlan hanging off the VETH.
Quoted 7 lines
I see packets received on the macvlan in tshark, and they appear correct. No obvious errors in the macvlan port stats or netstat -s,
and the 'ss' tool doesn't appear to support UDP sockets at all.
So, I'm about to go digging into the code, but if anyone has any suggestions for places to look, please let me know!
Quoted 8 lines
Well, problem is defragmentation and macvlan cooperation.
Multicast messages are broadcasted on all macvlan ports.
But IP defrag will probably deliver a single final frame.
We probably need to handle defrag in macvlan before broadcasting to all ports.
I see packets get to this code in ip_input.c (line 467 or so), and that printk is mine of course.
	if ((dev&&   strcmp(dev->name, "rddVR10#0") == 0) ||
	    (dev&&   strcmp(dev->name, "rddVR10") == 0)) {
		printk("calling ip_rcv_finish through NF_HOOK, dev: %s, len: %i\n",
		       dev->name, skb->len);
	}

	return NF_HOOK(NFPROTO_IPV4, NF_INET_PRE_ROUTING, skb, dev, NULL,
		       ip_rcv_finish);
But, the macvlan packets never make it to the ip_rcv_finish method.
I do see a big and a little packet entering this code.
Quoted 7 lines
Quoted 3 lines
I have no firewall rules that I'm aware of, though there is some conn-track logic (though not associated with the mac-vlan interface):
Say you have 10 vlans on your eth0, how many times do you want one incoming multicast frame being delivered to your application listening on 0.0.0.0:port ?
Quoted 6 lines
How would it work for two Ethernet devices on the same LAN? I'd say that mac-vlans should mimic that case.
And in my case, I'm binding hard to a device& IP address, so my app should get it once regardless.
OK, but before frame being delivered to your app, it must be re-assembled by net/ipv4/inet_fragment.c& net/ipv4/ip_fragment.c machinery.
This machinery uses :
static int ip4_frag_match(struct inet_frag_queue *q, void *a)
{
         struct ipq *qp;
         struct ip4_create_arg *arg = a;

         qp = container_of(q, struct ipq, q);
         return  qp->id == arg->iph->id&&
                         qp->saddr == arg->iph->saddr&&
                         qp->daddr == arg->iph->daddr&&
                         qp->protocol == arg->iph->protocol&&
                         qp->user == arg->user;
}

Quoted 2 lines

All frames broadcasted (because of multicast code in macvlan) on vlans have same saddr/daddr/protocol (and user).

Wouldn't you have the same problem with two real Ethernet interfaces on the same LAN, or two 802.1Q devices for that matter? The addrs will all be the same in that case too?

Also, if I have just a single mac-vlan active (the other 3 are 'ifconfig foo down'), I still see the problem with mcast.

From what you describe, I am thinking I may be hitting a different issue. Any ideas on how to figure out why exactly the NF_HOOK isn't calling the ip_rcv_finish method?

Quoted 8 lines

So kernel will discard all redundant copies of frames and deliver one copy only to upper stack.
Check commit 7736d33f4262d437c5 (packet: Add pre-defragmentation support for ipv4 fanouts) for a possible hint :
We could perform the re-assembly in macvlan code, before doing the "broadcast the frame on all ports" part.

Thanks,
Ben

Signature · 2 lines

Ben Greear <[email protected]>
Candela Technologies Inc  http://www.candelatech.com

Eric Dumazet

2011-10-05 21:36 UTC #7 lore raw

Quoted 4 lines · Ben Greear

Wouldn't you have the same problem with two real Ethernet interfaces on the same LAN, or two 802.1Q devices for that matter? The addrs will all be the same in that case too?

Usually multicast is coupled with routing.

A JOIN message from your app wont be sent on all interfaces...

But yes, we might have a similar issue with regular vlans.

Probably nobody noticed yet. Just say no to fragments :)

Quoted 3 lines

Also, if I have just a single mac-vlan active (the other 3 are 'ifconfig foo down'), I still see the problem with mcast.

Thats another bug : macvlan doesnt test IFF_UP on broadcasts, only for unicast messages. Please test following patch.

Quoted 4 lines

From what you describe, I am thinking I may be hitting a different
issue. Any ideas on how to figure out why exactly the NF_HOOK isn't calling the ip_rcv_finish method?

Really I believe I tried to explain the thing already...

ip_local_deliver() -> ip_defrag() :

[PATCH] macvlan: dont send frames on DOWN devices

Reported-by Ben Greear <[email protected]>

Signed-off-by Eric Dumazet <[email protected]>

---

drivers/net/macvlan.c +2 -1

index b100c90..94a0282 100644
@@ -145,7 +145,8 @@ static void macvlan_broadcast(struct sk_buff *skb,
 		hlist_for_each_entry_rcu(vlan, n, &port->vlan_hash[i], hlist) {
 			if (vlan->dev == src || !(vlan->mode & mode))
 				continue;
-
+			if (!(vlan->dev->flags & IFF_UP))
+				continue;
 			nskb = skb_clone(skb, GFP_ATOMIC);
 			err = macvlan_broadcast_one(nskb, vlan, eth,
 					 mode == MACVLAN_MODE_BRIDGE);

Ben Greear

author

2011-10-05 21:52 UTC #8 lore raw

Quoted 10 lines · Eric Dumazet

Quoted 4 lines · Ben Greear
Wouldn't you have the same problem with two real Ethernet interfaces on the same LAN, or two 802.1Q devices for that matter? The addrs will all be the same in that case too?
Usually multicast is coupled with routing.
A JOIN message from your app wont be sent on all interfaces...

It will be if you open two sockets and bind each one of them to a network device, at least as far as I can tell.

Quoted 4 lines

But yes, we might have a similar issue with regular vlans.
Probably nobody noticed yet. Just say no to fragments :)

Heh, it's regression testing time..we're trying all the weird stuff this week :)

Quoted 7 lines

Quoted 3 lines
Also, if I have just a single mac-vlan active (the other 3 are 'ifconfig foo down'), I still see the problem with mcast.
Thats another bug : macvlan doesnt test IFF_UP on broadcasts, only for unicast messages. Please test following patch.

Quoted 8 lines

Quoted 4 lines
From what you describe, I am thinking I may be hitting a different
issue. Any ideas on how to figure out why exactly the NF_HOOK isn't calling the ip_rcv_finish method?
Really I believe I tried to explain the thing already...
ip_local_deliver() -> ip_defrag() :

It seems that netfilter is reporting the pkt as NF_STOLEN, probably because of the nf_ct_ipv4_gather_frags (which ends up calling ip_defrag) logic in nf_defrag_ipv4.c, line 86 or so. I'm adding more debugging to verify this.

I'll try out your patch below shortly.

Thanks,
Ben

Quoted 27 lines

[PATCH] macvlan: dont send frames on DOWN devices
Reported-by Ben Greear<[email protected]>
Signed-off-by Eric Dumazet<[email protected]>
---
drivers/net/macvlan.c +2 -2
index b100c90..94a0282 100644
@@ -145,7 +145,8 @@ static void macvlan_broadcast(struct sk_buff *skb,
  		hlist_for_each_entry_rcu(vlan, n,&port->vlan_hash[i], hlist) {
  			if (vlan->dev == src || !(vlan->mode&  mode))
  				continue;
-
+			if (!(vlan->dev->flags&  IFF_UP))
+				continue;
  			nskb = skb_clone(skb, GFP_ATOMIC);
  			err = macvlan_broadcast_one(nskb, vlan, eth,
  					 mode == MACVLAN_MODE_BRIDGE);
 
 
--
To unsubscribe from this list: send the line "unsubscribe netdev" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

Signature · 2 lines

Ben Greear <[email protected]>
Candela Technologies Inc  http://www.candelatech.com

Ben Greear

author

2011-10-05 22:35 UTC #9 lore raw

Quoted 35 lines · Ben Greear

Quoted 10 lines · Eric Dumazet
Quoted 4 lines · Ben Greear
Wouldn't you have the same problem with two real Ethernet interfaces on the same LAN, or two 802.1Q devices for that matter? The addrs will all be the same in that case too?
Usually multicast is coupled with routing.
A JOIN message from your app wont be sent on all interfaces...
It will be if you open two sockets and bind each one of them to a network device, at least as far as I can tell.
Quoted 4 lines
But yes, we might have a similar issue with regular vlans.
Probably nobody noticed yet. Just say no to fragments :)
Heh, it's regression testing time..we're trying all the weird stuff this week :)
Quoted 12 lines
Quoted 3 lines
Also, if I have just a single mac-vlan active (the other 3 are 'ifconfig foo down'), I still see the problem with mcast.
Thats another bug : macvlan doesnt test IFF_UP on broadcasts, only for unicast messages. Please test following patch.
Quoted 4 lines
From what you describe, I am thinking I may be hitting a different issue. Any ideas on how to figure out why exactly the NF_HOOK isn't calling the ip_rcv_finish method?

Quoted 8 lines

Quoted 3 lines
Really I believe I tried to explain the thing already...
ip_local_deliver() -> ip_defrag() :
It seems that netfilter is reporting the pkt as NF_STOLEN, probably because of the nf_ct_ipv4_gather_frags (which ends up calling ip_defrag) logic in nf_defrag_ipv4.c, line 86 or so. I'm adding more debugging to verify this.

Ok, this is definitely the problem. Also, even if you have a single mac-vlan, you will still have this problem because the underlying device will get a copy first. So, your patch doesn't solve my particular problem, but it does appear to be correct.

If someone wants to cook up macvlan-ip-defrag patch I'll be happy to test it. But, as far as I can tell, this problem can happen on any two interfaces. The reason that some of mine work (.1q vlans) and macvlan didn't is probably because those were separated by some virtual network links that imparted extra delay...so the vlan consumed all its fragments and passed the complete pkt up the stack before the mac-vlan ever saw the initial frame.

With this in mind, it seems that using multiple udp multicast sockets bound to specific devices is fundamentally broken for fragmented packets.

I have no pressing need for this feature, so now that I better understand the problem I can just document it and move on to other things.

Thanks for all the help.

Ben

Signature · 2 lines

Ben Greear <[email protected]>
Candela Technologies Inc  http://www.candelatech.com

Eric Dumazet

2011-10-06 20:28 UTC #10 lore raw

PATCH 1/1 macvlan: handle fragmented multicast frames

Quoted 8 lines · Ben Greear

If someone wants to cook up macvlan-ip-defrag patch I'll be happy to test it. But, as far as I can tell, this problem can happen on any two interfaces. The reason that some of mine work (.1q vlans) and macvlan didn't is probably because those were separated by some virtual network links that imparted extra delay...so the vlan consumed all its fragments and passed the complete pkt up the stack before the mac-vlan ever saw the initial frame.

Quoted 9 lines

With this in mind, it seems that using multiple udp multicast sockets bound to specific devices is fundamentally broken for fragmented packets.
I have no pressing need for this feature, so now that I better understand the problem I can just document it and move on to other things.
Thanks for all the help.

Please test following patch (note I had no time to test it, sorry !)

Based on net-next tree, might apply on 3.0 kernel...

[PATCH net-next] macvlan: handle fragmented multicast frames

Fragmented multicast frames are delivered to a single macvlan port, because ip defrag logic considers other samples are redundant.

Implement a defrag step before trying to send the multicast frame.

Reported-by Ben Greear <[email protected]>

Signed-off-by Eric Dumazet <[email protected]>

---
 drivers/net/macvlan.c  |    3 +++
 include/net/ip.h       |    9 +++++++++
 net/ipv4/ip_fragment.c |   36 ++++++++++++++++++++++++++++++++++++
 net/packet/af_packet.c |   39 +--------------------------------------

drivers/net/macvlan.c +3 -0

index b100c90..40366eb 100644
@@ -169,6 +169,9 @@ static rx_handler_result_t macvlan_handle_frame(struct sk_buff **pskb)
 
 	port = macvlan_port_get_rcu(skb->dev);
 	if (is_multicast_ether_addr(eth->h_dest)) {
+		skb = ip_check_defrag(skb, IP_DEFRAG_MACVLAN);
+		if (!skb)
+			return RX_HANDLER_CONSUMED;
 		src = macvlan_hash_lookup(port, eth->h_source);
 		if (!src)
 			/* frame comes from an external address */

include/net/ip.h +9 -0

index aa76c7a..c7e066a 100644
@@ -406,9 +406,18 @@ enum ip_defrag_users {
 	IP_DEFRAG_VS_OUT,
 	IP_DEFRAG_VS_FWD,
 	IP_DEFRAG_AF_PACKET,
+	IP_DEFRAG_MACVLAN,
 };
 
 int ip_defrag(struct sk_buff *skb, u32 user);
+#ifdef CONFIG_INET
+struct sk_buff *ip_check_defrag(struct sk_buff *skb, u32 user);
+#else
+static inline struct sk_buff *ip_check_defrag(struct sk_buff *skb, u32 user)
+{
+	return skb;
+}
+#endif
 int ip_frag_mem(struct net *net);
 int ip_frag_nqueues(struct net *net);

net/ipv4/ip_fragment.c +36 -0

index 0e0ab98..763589a 100644
@@ -682,6 +682,42 @@ int ip_defrag(struct sk_buff *skb, u32 user)
 }
 EXPORT_SYMBOL(ip_defrag);
 
+struct sk_buff *ip_check_defrag(struct sk_buff *skb, u32 user)
+{
+	const struct iphdr *iph;
+	u32 len;
+
+	if (skb->protocol != htons(ETH_P_IP))
+		return skb;
+
+	if (!pskb_may_pull(skb, sizeof(struct iphdr)))
+		return skb;
+
+	iph = ip_hdr(skb);
+	if (iph->ihl < 5 || iph->version != 4)
+		return skb;
+	if (!pskb_may_pull(skb, iph->ihl*4))
+		return skb;
+	iph = ip_hdr(skb);
+	len = ntohs(iph->tot_len);
+	if (skb->len < len || len < (iph->ihl * 4))
+		return skb;
+
+	if (ip_is_fragment(ip_hdr(skb))) {
+		skb = skb_share_check(skb, GFP_ATOMIC);
+		if (skb) {
+			if (pskb_trim_rcsum(skb, len))
+				return skb;
+			memset(IPCB(skb), 0, sizeof(struct inet_skb_parm));
+			if (ip_defrag(skb, user))
+				return NULL;
+			skb->rxhash = 0;
+		}
+	}
+	return skb;
+}
+EXPORT_SYMBOL(ip_check_defrag);
+
 #ifdef CONFIG_SYSCTL
 static int zero;

net/packet/af_packet.c +1 ~~-38~~

index 25e68f5..ff9eed7 100644
@@ -1213,43 +1213,6 @@ static struct sock *fanout_demux_cpu(struct packet_fanout *f, struct sk_buff *sk
 	return f->arr[cpu % num];
 }
 
-static struct sk_buff *fanout_check_defrag(struct sk_buff *skb)
-{
-#ifdef CONFIG_INET
-	const struct iphdr *iph;
-	u32 len;
-
-	if (skb->protocol != htons(ETH_P_IP))
-		return skb;
-
-	if (!pskb_may_pull(skb, sizeof(struct iphdr)))
-		return skb;
-
-	iph = ip_hdr(skb);
-	if (iph->ihl < 5 || iph->version != 4)
-		return skb;
-	if (!pskb_may_pull(skb, iph->ihl*4))
-		return skb;
-	iph = ip_hdr(skb);
-	len = ntohs(iph->tot_len);
-	if (skb->len < len || len < (iph->ihl * 4))
-		return skb;
-
-	if (ip_is_fragment(ip_hdr(skb))) {
-		skb = skb_share_check(skb, GFP_ATOMIC);
-		if (skb) {
-			if (pskb_trim_rcsum(skb, len))
-				return skb;
-			memset(IPCB(skb), 0, sizeof(struct inet_skb_parm));
-			if (ip_defrag(skb, IP_DEFRAG_AF_PACKET))
-				return NULL;
-			skb->rxhash = 0;
-		}
-	}
-#endif
-	return skb;
-}
-
 static int packet_rcv_fanout(struct sk_buff *skb, struct net_device *dev,
 			     struct packet_type *pt, struct net_device *orig_dev)
 {

@@ -1268,7 +1231,7 @@ static int packet_rcv_fanout(struct sk_buff *skb, struct net_device *dev,
 	case PACKET_FANOUT_HASH:
 	default:
 		if (f->defrag) {
-			skb = fanout_check_defrag(skb);
+			skb = ip_check_defrag(skb, IP_DEFRAG_AF_PACKET);
 			if (!skb)
 				return 0;
 		}

Ben Greear

author

2011-10-07 16:44 UTC #11 lore raw

Re: [PATCH net-next] macvlan: handle fragmented multicast frames

Quoted 14 lines · Eric Dumazet

Quoted 12 lines · Ben Greear
If someone wants to cook up macvlan-ip-defrag patch I'll be happy to test it. But, as far as I can tell, this problem can happen on any two interfaces. The reason that some of mine work (.1q vlans) and macvlan didn't is probably because those were separated by some virtual network links that imparted extra delay...so the vlan consumed all its fragments and passed the complete pkt up the stack before the mac-vlan ever saw the initial frame.
With this in mind, it seems that using multiple udp multicast sockets bound to specific devices is fundamentally broken for fragmented packets.

Quoted 7 lines

Quoted 5 lines
I have no pressing need for this feature, so now that I better understand the problem I can just document it and move on to other things.
Thanks for all the help.
Please test following patch (note I had no time to test it, sorry !)

I hope to test this soon, but lots of other things piled up all at once, so might be a few days.

Thanks,
Ben

Quoted 9 lines

Based on net-next tree, might apply on 3.0 kernel...
[PATCH net-next] macvlan: handle fragmented multicast frames
Fragmented multicast frames are delivered to a single macvlan port, because ip defrag logic considers other samples are redundant.
Implement a defrag step before trying to send the multicast frame.

Signature · 2 lines

Ben Greear <[email protected]>
Candela Technologies Inc  http://www.candelatech.com

Ben Greear

author

2011-10-10 16:27 UTC #12 lore raw

Re: [PATCH net-next] macvlan: handle fragmented multicast frames

In reply to Eric Dumazet

Quoted 22 lines · Eric Dumazet

Quoted 8 lines · Ben Greear
If someone wants to cook up macvlan-ip-defrag patch I'll be happy to test it. But, as far as I can tell, this problem can happen on any two interfaces. The reason that some of mine work (.1q vlans) and macvlan didn't is probably because those were separated by some virtual network links that imparted extra delay...so the vlan consumed all its fragments and passed the complete pkt up the stack before the mac-vlan ever saw the initial frame.
Quoted 9 lines
With this in mind, it seems that using multiple udp multicast sockets bound to specific devices is fundamentally broken for fragmented packets.
I have no pressing need for this feature, so now that I better understand the problem I can just document it and move on to other things.
Thanks for all the help.
Please test following patch (note I had no time to test it, sorry !)

Quoted 8 lines

Based on net-next tree, might apply on 3.0 kernel...
[PATCH net-next] macvlan: handle fragmented multicast frames
Fragmented multicast frames are delivered to a single macvlan port, because ip defrag logic considers other samples are redundant.
Implement a defrag step before trying to send the multicast frame.

I applied this to Linus' top-of-tree this morning and it does appear to fix the problem for mac-vlans.

I do see this error, but I doubt it has anything to do with your patch:

device eth0 entered promiscuous mode
device rddVR10 entered promiscuous mode ADDRCONF(NETDEV_CHANGE): rddVR1b: link becomes ready

================================================
[ BUG: lock held when returning to user space! ]
------------------------------------------------
ip/3452 is leaving the kernel with locks still held!
1 lock held by ip/3452:
#0: (rcu_read_lock){.+.+..}, at: [<f8c5336f>] rcu_read_lock+0x0/0x26 [ipv6]
ADDRCONF(NETDEV_CHANGE): rddVR4b: link becomes ready ADDRCONF(NETDEV_CHANGE): rddVR5b: link becomes ready

I have no idea why it doesn't print out a more useful stack trace. It seems repeatable (2 of 2 reboots so far). I'm configuring a pretty complex virtual network, with veth devices, xorp instances running ipv4 and ipv6 routing protocols, etc.

This is a clean upstream kernel with no outside patches aside from your own.

Thanks,
Ben

Signature · 2 lines

Ben Greear <[email protected]>
Candela Technologies Inc  http://www.candelatech.com

Eric Dumazet

2011-10-10 16:41 UTC #13 lore raw

Re: [PATCH net-next] macvlan: handle fragmented multicast frames

Quoted 3 lines · Ben Greear

I applied this to Linus' top-of-tree this morning and it does appear to fix the problem for mac-vlans.

Thanks for testing

Quoted 17 lines

I do see this error, but I doubt it has anything to do with your patch:
device eth0 entered promiscuous mode
device rddVR10 entered promiscuous mode ADDRCONF(NETDEV_CHANGE): rddVR1b: link becomes ready
================================================
[ BUG: lock held when returning to user space! ]
------------------------------------------------
ip/3452 is leaving the kernel with locks still held!
1 lock held by ip/3452:
#0: (rcu_read_lock){.+.+..}, at: [<f8c5336f>] rcu_read_lock+0x0/0x26 [ipv6]
ADDRCONF(NETDEV_CHANGE): rddVR4b: link becomes ready ADDRCONF(NETDEV_CHANGE): rddVR5b: link becomes ready

Quoted 5 lines

I have no idea why it doesn't print out a more useful stack trace. It seems repeatable (2 of 2 reboots so far). I'm configuring a pretty complex virtual network, with veth devices, xorp instances running ipv4 and ipv6 routing protocols, etc.

Do you have LOCKDEP enabled ?

Quoted 2 lines

This is a clean upstream kernel with no outside patches aside from your own.

Hmm, it seems we have an rcu_read_unlock() missing...

Any idea what was done by this "ip" command ?

Ben Greear

author

2011-10-10 16:53 UTC #14 lore raw

Re: [PATCH net-next] macvlan: handle fragmented multicast frames

Quoted 25 lines · Eric Dumazet

Quoted 3 lines · Ben Greear
I applied this to Linus' top-of-tree this morning and it does appear to fix the problem for mac-vlans.
Thanks for testing
Quoted 17 lines
I do see this error, but I doubt it has anything to do with your patch:
device eth0 entered promiscuous mode
device rddVR10 entered promiscuous mode ADDRCONF(NETDEV_CHANGE): rddVR1b: link becomes ready
================================================
[ BUG: lock held when returning to user space! ]
------------------------------------------------
ip/3452 is leaving the kernel with locks still held! 1 lock held by ip/3452:
   #0:  (rcu_read_lock){.+.+..}, at: [<f8c5336f>] rcu_read_lock+0x0/0x26 [ipv6]
ADDRCONF(NETDEV_CHANGE): rddVR4b: link becomes ready ADDRCONF(NETDEV_CHANGE): rddVR5b: link becomes ready

Quoted 7 lines

Quoted 5 lines
I have no idea why it doesn't print out a more useful stack trace. It seems repeatable (2 of 2 reboots so far). I'm configuring a pretty complex virtual network, with veth devices, xorp instances running ipv4 and ipv6 routing protocols, etc.
Do you have LOCKDEP enabled ?

Yes, as far as I can tell:
[greearb@build-32 linux-2.6.p4s]$ grep LOCKDEP .config CONFIG_LOCKDEP_SUPPORT=y
CONFIG_LOCKDEP=y

And it doesn't appear to have turned itself off:

[root@lec2010-ath9k-1 ~]# dmesg|grep lockdep
RCU lockdep checking is enabled.
lockdep: fixing up alternatives.
[root@lec2010-ath9k-1 ~]#

I looked through the kernel debug section of the config, and it seems normal enough...

But, after this splat, if I run sysrq-d, then it says sysrq is off, maybe because the splat disabled it?

SysRq : Show Locks Held
INFO: lockdep is turned off.

sysrq-l does show backtraces, so the backtrace logic in general seems to work fine.

Quoted 7 lines

Quoted 3 lines
This is a clean upstream kernel with no outside patches aside from your own.
Hmm, it seems we have an rcu_read_unlock() missing...
Any idea what was done by this "ip" command ?

No, it's called multiple times by my user-space control logic. Basically,
it configures around 30 interfaces, some GRE, veth, mac-vlans, .1q vlans, normal ethernet, etc. Also, I have some ipv6 addrs configured on many of them.

And, setting up routing rules, for ipv4 and ipv6 for the virtual routers.

Thanks,
Ben

Signature · 2 lines

Ben Greear <[email protected]>
Candela Technologies Inc  http://www.candelatech.com

David Miller

maintainer Applied

2011-10-19 03:23 UTC #15 lore raw

Re: [PATCH net-next] macvlan: handle fragmented multicast frames

In reply to Eric Dumazet

From: Eric Dumazet <[email protected]>
Date: Thu, 06 Oct 2011 22:28:31 +0200

Quoted 9 lines

[PATCH net-next] macvlan: handle fragmented multicast frames
Fragmented multicast frames are delivered to a single macvlan port, because ip defrag logic considers other samples are redundant.
Implement a defrag step before trying to send the multicast frame.
Reported-by Ben Greear <[email protected]>
Signed-off-by Eric Dumazet <[email protected]>

Applied to net-next, thanks.

Eric Dumazet

2011-10-06 20:42 UTC #16 lore raw

In reply to Eric Dumazet

Quoted 43 lines · Eric Dumazet

Quoted 4 lines · Ben Greear
Wouldn't you have the same problem with two real Ethernet interfaces on the same LAN, or two 802.1Q devices for that matter? The addrs will all be the same in that case too?
Usually multicast is coupled with routing.
A JOIN message from your app wont be sent on all interfaces...
But yes, we might have a similar issue with regular vlans.
Probably nobody noticed yet. Just say no to fragments :)
Quoted 3 lines
Also, if I have just a single mac-vlan active (the other 3 are 'ifconfig foo down'), I still see the problem with mcast.
Thats another bug : macvlan doesnt test IFF_UP on broadcasts, only for unicast messages. Please test following patch.
Quoted 4 lines
From what you describe, I am thinking I may be hitting a different
issue. Any ideas on how to figure out why exactly the NF_HOOK isn't calling the ip_rcv_finish method?
Really I believe I tried to explain the thing already...
ip_local_deliver() -> ip_defrag() :
[PATCH] macvlan: dont send frames on DOWN devices
Reported-by Ben Greear <[email protected]>
Signed-off-by Eric Dumazet <[email protected]>
---
drivers/net/macvlan.c +0 -0
index b100c90..94a0282 100644

drivers/net/macvlan.c static void macvlan_broadcast(struct sk_buff *skb, · line 152

Quoted 9 lines

drivers/net/macvlan.c +2 -1

@@ -145,7 +145,8 @@ static void macvlan_broadcast(struct sk_buff *skb,
 		hlist_for_each_entry_rcu(vlan, n, &port->vlan_hash[i], hlist) {
 			if (vlan->dev == src || !(vlan->mode & mode))
 				continue;
-
+			if (!(vlan->dev->flags & IFF_UP))
+				continue;
 			nskb = skb_clone(skb, GFP_ATOMIC);
 			err = macvlan_broadcast_one(nskb, vlan, eth,
 					 mode == MACVLAN_MODE_BRIDGE);

This one is not needed.

When a port is down, its not in vlan_hash[] table anymore.

(Not sure why we perform the IFF_UP test for unicast frames.)