Skip to contentSkip to conversation

VLAN and ARP failure on tg3 drivers

PATCH In review Gertjan Hofman 5 participants 10 messages Oct 26, 2009 – Oct 28, 2009

Subsystems (3)
lore
Share & tools
lore
#0 lore raw

Dear Matt, Eric, Benny,

Sorry about the slow response to your fast replies. I think Benny is correct, the 'problem' lies in the fact that we were using a VLAN ID of 0, without knowing its special significance. User error.

I tested it with other VLAN id's (>0) and it appears to work fine. We are not entirely sure we understand  why it used to work with VLAN ID 0 on the Broadcom chips and still does with a number of different cards (with >2.6.27 kernels).  What is the 'correct' behaviour for this incorrect usage ? When a PC returns the ARP response to the machine with the BroadCom card, it will have the destination address of the VLAN device, but presumably the VLAN ID tag set to zero.  Hmmm. I can live with not knowing the answer I guess.

Thanks again,

Gertjan

Quoted 67 lines · Matt Carlson

From: Matt Carlson <[email protected]>
Subject: Re: VLAN and ARP failure on tg3 drivers
To: "Gertjan Hofman" <[email protected]>
Cc: "[email protected]" <[email protected]>
Date: Friday, October 23, 2009, 3:35 PM

Quoted 3 lines · Gertjan Hofman

Dear Kernel developers,

A couple of weeks ago we tried to migrate from a

2.6.24? kernel to a 2.6.29 kernel and noticed our VLAN application no longer works.? The problem is easy to replicate:

Quoted 3 lines

1. connect 2 PC's with a cross-over cable
2. set up a fixed IP address to both PC's? (say

192.168.0.[1,2])

Quoted 2 lines

3. create a vlan:? vconfig? add eth0 0.
4. set IP addresses on the VLAN devices? (say

192.168.1.[1,2])

Quoted 3 lines

5. try ping one machine from the other.

I tried to dig into the problem by using un-patched

kernel.org kernels with Ubuntu .config files.? Kernels up to 2.6.26 work fine, kernels after and including 2.6.27 fail. The problem is that ARP messages are being dropped. If the ARP table is updated by hand on each machine, the communication across the VLAN works fine.

Quoted 2 lines

At first I thought the kernel VLAN code was the

problem (we had an earlier issue with a regression in 2.6.24) but it looks like the problem is actually with the tg3 driver.? Our system uses Broadcom ethernet chips. I tried the same experiments with combination of boards that have Broadcom and none-Broadcom and the only time I see it fail is with the tg3? driver loaded.

Quoted 2 lines

Snooping with WireShark shows that a ARP request from

the non-Broadcom machine is seen and even answered, but never appears back on the network. If the Broadcom machine orginates the ARP message, it never arrives at the destination. I tried lowering the size of the MTU to 1492 as well as giving each VLAN device a different MAC. No deal.

Quoted 2 lines

I tried to look at tg3 patch changes from 2.6.26 to

2.6.27 but I am not familiar enough with the Git system to extract the appropiate changes.? I am a bit surprised that I am not seeing any references to this on the web, the combination of >2.6.27 kernels, Broadcom and VLAN cant be that uncommon.

Quoted 2 lines

I would be happy to provide more information and to

try tests if any one can suggest them.

Quoted 4 lines

Sincerely,

Gertjan

I don't see any reason why your setup should fail, but it doesn't hurt
to gather more info about the problem.

What device are you experiencing this problem with? Is management
firmware enabled?  (ethtool -i ethx)

#1 lore raw
Quoted 11 lines · Gertjan Hofman

Dear Matt, Eric, Benny,

Sorry about the slow response to your fast replies. I think Benny is correct, the 'problem' lies in the fact that we were using a VLAN ID of 0, without knowing its special significance. User error.

I tested it with other VLAN id's (>0) and it appears to work fine. We are not entirely sure we understand  why it used to work with VLAN ID 0 on the Broadcom chips and still does with a number of different cards (with >2.6.27 kernels).  What is the 'correct' behaviour for this incorrect usage ?

VLAN 0 isn't incorrect, it's just surprising. When you send a packet tagged with VLAN 0, it means that the packet should be interpreted as being the same VLAN as a completely untagged packet.

So in theory, if both ends are using VLAN 0 and you aren't using eth0 for anything, traffic should flow, at least if both ends are on the same kernel version. Feel free to debug why that isn't the case for you, of course...

/Benny

#2 lore raw
Quoted 14 lines · Benny Amorsen
Quoted 11 lines · Gertjan Hofman

Dear Matt, Eric, Benny,

Sorry about the slow response to your fast replies. I think Benny is correct, the 'problem' lies in the fact that we were using a VLAN ID of 0, without knowing its special significance. User error.

I tested it with other VLAN id's (>0) and it appears to work fine. We are not entirely sure we understand why it used to work with VLAN ID 0 on the Broadcom chips and still does with a number of different cards (with >2.6.27 kernels). What is the 'correct' behaviour for this incorrect usage ?

Quoted 9 lines

VLAN 0 isn't incorrect, it's just surprising. When you send a packet tagged with VLAN 0, it means that the packet should be interpreted as being the same VLAN as a completely untagged packet.

So in theory, if both ends are using VLAN 0 and you aren't using eth0 for anything, traffic should flow, at least if both ends are on the same kernel version. Feel free to debug why that isn't the case for you, of course...

VLAN id 0 is not usable on current kernel because we use 16 bits in skb to
store vlan_tci, and vlan_tci = 0 means there is no VLAN tagging.

We could use high order bit (0x8000) to tell if vlan tagging is set or not.

#3 lore raw

PATCH 1/1 vlan: allow VLAN ID 0 to be used

Quoted 6 lines · Eric Dumazet

VLAN id 0 is not usable on current kernel because we use 16 bits in skb to
store vlan_tci, and vlan_tci = 0 means there is no VLAN tagging.

We could use high order bit (0x8000) to tell if vlan tagging is set or not.

Here is the patch I cooked that permitted VLAN 0 to be used with tg3 (and other HW accelerated vlan nics I suppose)

[PATCH] vlan: allow VLAN ID 0 to be used

We currently use a 16 bit field (vlan_tci) to store VLAN ID on a skb.

0 value is used a special value, meaning VLAN ID not set. This forbids use of VLAN ID 0

As VLAN ID is 12 bits, we can use high order bit as a flag, and allow VLAN ID 0

---

include/linux/if_vlan.h +5 -4
index 7ff9af1..7dfcdb5 100644
@@ -105,8 +105,9 @@ static inline void vlan_group_set_device(struct vlan_group *vg,
 	array[vlan_id % VLAN_GROUP_ARRAY_PART_LEN] = dev;
 }
 
-#define vlan_tx_tag_present(__skb)	((__skb)->vlan_tci)
-#define vlan_tx_tag_get(__skb)		((__skb)->vlan_tci)
+#define VLAN_TAG_PRESENT		0x8000
+#define vlan_tx_tag_present(__skb)	((__skb)->vlan_tci & VLAN_TAG_PRESENT)
+#define vlan_tx_tag_get(__skb)		((__skb)->vlan_tci & 0x7fff)
 
 #if defined(CONFIG_VLAN_8021Q) || defined(CONFIG_VLAN_8021Q_MODULE)
 extern struct net_device *vlan_dev_real_dev(const struct net_device *dev);
@@ -231,7 +232,7 @@ static inline struct sk_buff *__vlan_put_tag(struct sk_buff *skb, u16 vlan_tci)
 static inline struct sk_buff *__vlan_hwaccel_put_tag(struct sk_buff *skb,
 						     u16 vlan_tci)
 {
-	skb->vlan_tci = vlan_tci;
+	skb->vlan_tci = VLAN_TAG_PRESENT | vlan_tci;
 	return skb;
 }
 
@@ -284,7 +285,7 @@ static inline int __vlan_hwaccel_get_tag(const struct sk_buff *skb,
 					 u16 *vlan_tci)
 {
 	if (vlan_tx_tag_present(skb)) {
-		*vlan_tci = skb->vlan_tci;
+		*vlan_tci = vlan_tx_tag_get(skb);
 		return 0;
 	} else {
 		*vlan_tci = 0;
maintainer
#4 lore raw

Re: [PATCH] vlan: allow VLAN ID 0 to be used

From: Eric Dumazet <[email protected]>
Date: Mon, 26 Oct 2009 17:13:58 +0100

Quoted 4 lines

[PATCH] vlan: allow VLAN ID 0 to be used

We currently use a 16 bit field (vlan_tci) to store VLAN ID on a skb.

Quoted 8 lines

0 value is used a special value, meaning VLAN ID not set. This forbids use of VLAN ID 0

As VLAN ID is 12 bits, we can use high order bit as a flag, and allow VLAN ID 0

This is going to need some more work.

IXGBE is already using the higher bits of ->vlan_tci internally, your change breaks that.

QLGE explicitly initializes skb->vlan_tci to zero, you'll need to make sure that's OK.

There is an explicit "if (skb->vlan_tci" (ie. zero vs. non-zero) test in net/core/dev.c:netif_receive_skb()

net/core/skbuff.c:__copy_skb_header() does a straight copy, you'll need to make sure that's still OK.

net/packet/af_packet.c:tpacket_rcv() and packet_recvmsg() report the skb->vlan_tci value to userspace, that's broken now as userspace doesn't expect that new bit to be there.

#5 lore raw

Re: [PATCH] vlan: allow VLAN ID 0 to be used

Quoted 27 lines · David Miller

From: Eric Dumazet <[email protected]>
Date: Mon, 26 Oct 2009 17:13:58 +0100

Quoted 4 lines

[PATCH] vlan: allow VLAN ID 0 to be used

We currently use a 16 bit field (vlan_tci) to store VLAN ID on a skb.

Quoted 8 lines

0 value is used a special value, meaning VLAN ID not set. This forbids use of VLAN ID 0

As VLAN ID is 12 bits, we can use high order bit as a flag, and allow VLAN ID 0

This is going to need some more work.

IXGBE is already using the higher bits of ->vlan_tci internally, your change breaks that.

QLGE explicitly initializes skb->vlan_tci to zero, you'll need to make sure that's OK.

There is an explicit "if (skb->vlan_tci" (ie. zero vs. non-zero) test in net/core/dev.c:netif_receive_skb()

Quoted 6 lines

net/core/skbuff.c:__copy_skb_header() does a straight copy, you'll need to make sure that's still OK.

net/packet/af_packet.c:tpacket_rcv() and packet_recvmsg() report the skb->vlan_tci value to userspace, that's broken now as userspace doesn't expect that new bit to be there.

Thanks a lot David for this extended review and guidelines

I hope I did not miss another important thing in this second version.

Again, tested on tg3 only, and on net-next-2.6 tree

[PATCH] vlan: allow null VLAN ID to be used

We currently use a 16 bit field (vlan_tci) to store VLAN ID/PRIO on a skb.

Null value is used as a special value, meaning vlan tagging not enabled. This forbids use of null vlan ID.

As pointed by David, some drivers use the 3 high order bits (PRIO)

As VLAN ID is 12 bits, we can use the remaining bit (CFI) as a flag, and allow null VLAN ID.

In case future code really wants to use VLAN_CFI_MASK, we'll have to use a bit outside of vlan_tci.

#define VLAN_PRIO_MASK         0xe000 /* Priority Code Point */
#define VLAN_PRIO_SHIFT        13
#define VLAN_CFI_MASK          0x1000 /* Canonical Format Indicator */
#define VLAN_TAG_PRESENT       VLAN_CFI_MASK
#define VLAN_VID_MASK          0x0fff /* VLAN Identifier */
---
 include/linux/if_vlan.h |   14 +++++++++-----
 net/8021q/vlan.h        |    2 +-
 net/8021q/vlan_dev.c    |    2 +-
 net/core/dev.c          |    2 +-
 net/packet/af_packet.c  |    5 +++--
 5 files changed, 15 insertions(+), 10 deletions(-)
include/linux/if_vlan.h +9 -5
index 7ff9af1..8898cbe 100644
@@ -63,7 +63,11 @@ static inline struct vlan_ethhdr *vlan_eth_hdr(const struct sk_buff *skb)
 	return (struct vlan_ethhdr *)skb_mac_header(skb);
 }
 
-#define VLAN_VID_MASK	0xfff
+#define VLAN_PRIO_MASK		0xe000 /* Priority Code Point */
+#define VLAN_PRIO_SHIFT		13
+#define VLAN_CFI_MASK		0x1000 /* Canonical Format Indicator */
+#define VLAN_TAG_PRESENT	VLAN_CFI_MASK
+#define VLAN_VID_MASK		0x0fff /* VLAN Identifier */
 
 /* found in socket.c */
 extern void vlan_ioctl_set(int (*hook)(struct net *, void __user *));
@@ -105,8 +109,8 @@ static inline void vlan_group_set_device(struct vlan_group *vg,
 	array[vlan_id % VLAN_GROUP_ARRAY_PART_LEN] = dev;
 }
 
-#define vlan_tx_tag_present(__skb)	((__skb)->vlan_tci)
-#define vlan_tx_tag_get(__skb)		((__skb)->vlan_tci)
+#define vlan_tx_tag_present(__skb)	((__skb)->vlan_tci & VLAN_TAG_PRESENT)
+#define vlan_tx_tag_get(__skb)		((__skb)->vlan_tci & ~VLAN_TAG_PRESENT)
 
 #if defined(CONFIG_VLAN_8021Q) || defined(CONFIG_VLAN_8021Q_MODULE)
 extern struct net_device *vlan_dev_real_dev(const struct net_device *dev);
@@ -231,7 +235,7 @@ static inline struct sk_buff *__vlan_put_tag(struct sk_buff *skb, u16 vlan_tci)
 static inline struct sk_buff *__vlan_hwaccel_put_tag(struct sk_buff *skb,
 						     u16 vlan_tci)
 {
-	skb->vlan_tci = vlan_tci;
+	skb->vlan_tci = VLAN_TAG_PRESENT | vlan_tci;
 	return skb;
 }
 
@@ -284,7 +288,7 @@ static inline int __vlan_hwaccel_get_tag(const struct sk_buff *skb,
 					 u16 *vlan_tci)
 {
 	if (vlan_tx_tag_present(skb)) {
-		*vlan_tci = skb->vlan_tci;
+		*vlan_tci = vlan_tx_tag_get(skb);
 		return 0;
 	} else {
 		*vlan_tci = 0;
net/8021q/vlan.h +1 -1
index 82570bc..4ade5ed 100644
@@ -89,7 +89,7 @@ static inline u32 vlan_get_ingress_priority(struct net_device *dev,
 {
 	struct vlan_dev_info *vip = vlan_dev_info(dev);
 
-	return vip->ingress_priority_map[(vlan_tci >> 13) & 0x7];
+	return vip->ingress_priority_map[(vlan_tci >> VLAN_PRIO_SHIFT) & 0x7];
 }
 
 #ifdef CONFIG_VLAN_8021Q_GVRP
net/8021q/vlan_dev.c +1 -1
index 4198ec5..e370197 100644
@@ -393,7 +393,7 @@ int vlan_dev_set_egress_priority(const struct net_device *dev,
 	struct vlan_dev_info *vlan = vlan_dev_info(dev);
 	struct vlan_priority_tci_mapping *mp = NULL;
 	struct vlan_priority_tci_mapping *np;
-	u32 vlan_qos = (vlan_prio << 13) & 0xE000;
+	u32 vlan_qos = (vlan_prio << VLAN_PRIO_SHIFT) & VLAN_PRIO_MASK;
 
 	/* See if a priority mapping exists.. */
 	mp = vlan->egress_priority_map[skb_prio & 0xF];
net/core/dev.c +1 -1
index fa88dcd..5ab8668 100644
@@ -2303,7 +2303,7 @@ int netif_receive_skb(struct sk_buff *skb)
 	if (!skb->tstamp.tv64)
 		net_timestamp(skb);
 
-	if (skb->vlan_tci && vlan_hwaccel_do_receive(skb))
+	if (vlan_tx_tag_present(skb) && vlan_hwaccel_do_receive(skb))
 		return NET_RX_SUCCESS;
 
 	/* if we've gotten here through NAPI, check netpoll */
net/packet/af_packet.c +3 -2
index ff752c6..33e68f2 100644
@@ -79,6 +79,7 @@
 #include <linux/module.h>
 #include <linux/init.h>
 #include <linux/mutex.h>
+#include <linux/if_vlan.h>
 
 #ifdef CONFIG_INET
 #include <net/inet_common.h>
@@ -766,7 +767,7 @@ static int tpacket_rcv(struct sk_buff *skb, struct net_device *dev,
 			getnstimeofday(&ts);
 		h.h2->tp_sec = ts.tv_sec;
 		h.h2->tp_nsec = ts.tv_nsec;
-		h.h2->tp_vlan_tci = skb->vlan_tci;
+		h.h2->tp_vlan_tci = vlan_tx_tag_get(skb);
 		hdrlen = sizeof(*h.h2);
 		break;
 	default:
@@ -1493,7 +1494,7 @@ static int packet_recvmsg(struct kiocb *iocb, struct socket *sock,
 		aux.tp_snaplen = skb->len;
 		aux.tp_mac = 0;
 		aux.tp_net = skb_network_offset(skb);
-		aux.tp_vlan_tci = skb->vlan_tci;
+		aux.tp_vlan_tci = vlan_tx_tag_get(skb);
 
 		put_cmsg(msg, SOL_PACKET, PACKET_AUXDATA, sizeof(aux), &aux);
 	}
maintainer
#6 lore raw

Re: [PATCH] vlan: allow VLAN ID 0 to be used

From: Eric Dumazet <[email protected]>
Date: Tue, 27 Oct 2009 02:34:57 +0100

Quoted 5 lines

I hope I did not miss another important thing in this second version.

Again, tested on tg3 only, and on net-next-2.6 tree

[PATCH] vlan: allow null VLAN ID to be used

Looks good, applied to net-next-2.6

Someone now needs to convert IXGBE to use VLAN_PRIO_MASK and VLAN_PRIO_SHIFT instead of it's internal macros.

#7 lore raw

Re: [PATCH] vlan: allow VLAN ID 0 to be used

In reply to Eric Dumazet
Quoted 9 lines · Eric Dumazet

Here is the patch I cooked that permitted VLAN 0 to be used with tg3 (and other HW accelerated vlan nics I suppose)

[PATCH] vlan: allow VLAN ID 0 to be used

We currently use a 16 bit field (vlan_tci) to store VLAN ID on a skb.

0 value is used a special value, meaning VLAN ID not set. This forbids use of VLAN ID 0

Are you sure you actually want to do this?

VLAN 0 IS special. Frames received on VLAN 0 should be treated just as if they had no VLAN tag at all, except that they have an 802.1p value.

Sending frames with VLAN 0 should have something to do with whether the sender wants to use 802.1p, which doesn't really have much to do with VLAN's at all...

It would be nice if the unsuspecting user was at least warned that their use of VLAN 0 is non-standard and may cause surprising results like leakage into the "native" VLAN. That could be done in /sbin/ip or /sbin/vconfig, of course.

/Benny

#8 lore raw

Re: [PATCH] vlan: allow VLAN ID 0 to be used

Quoted 17 lines · Benny Amorsen
Quoted 9 lines · Eric Dumazet

Here is the patch I cooked that permitted VLAN 0 to be used with tg3 (and other HW accelerated vlan nics I suppose)

[PATCH] vlan: allow VLAN ID 0 to be used

We currently use a 16 bit field (vlan_tci) to store VLAN ID on a skb.

0 value is used a special value, meaning VLAN ID not set. This forbids use of VLAN ID 0

Are you sure you actually want to do this?

VLAN 0 IS special. Frames received on VLAN 0 should be treated just as if they had no VLAN tag at all, except that they have an 802.1p value.

Quoted 9 lines

Sending frames with VLAN 0 should have something to do with whether the sender wants to use 802.1p, which doesn't really have much to do with VLAN's at all...

It would be nice if the unsuspecting user was at least warned that their use of VLAN 0 is non-standard and may cause surprising results like leakage into the "native" VLAN. That could be done in /sbin/ip or /sbin/vconfig, of course.

Quotting http://en.wikipedia.org/wiki/IEEE_802.1Q

VLAN Identifier (VID): a 12-bit field specifying the VLAN to which the frame belongs.
A value of 0 means that the frame doesn't belong to any VLAN; in this case the 802.1Q tag specifies only a priority and is referred to as a priority tag.
A value of hex FFF is reserved for implementation use.
All other values may be used as VLAN identifiers, allowing up to 4094 VLANs

So we expect to generate a 802.1Q frame, even with a VID=0 field.
Before patch, device sends a non 802.1Q frame, which is not what was wanted by user. (Maybe he wants to check its device/network is able to transport 1522 bytes frames, who knows...)

To use non tagged frames, user selects eth0 device, and to send tagged frames, he selects eth0.0

Now, maybe eth0 and eth0.0 should share same IP addresses, because incoming frame with ID=0 tag should be received by eth0 device, but I am not sure standard requires this.

#9 lore raw

Re: [PATCH] vlan: allow VLAN ID 0 to be used

Quoted 36 lines · Eric Dumazet
Quoted 16 lines · Benny Amorsen
Quoted 9 lines · Eric Dumazet

Here is the patch I cooked that permitted VLAN 0 to be used with tg3 (and other HW accelerated vlan nics I suppose)

[PATCH] vlan: allow VLAN ID 0 to be used

We currently use a 16 bit field (vlan_tci) to store VLAN ID on a skb.

0 value is used a special value, meaning VLAN ID not set. This forbids use of VLAN ID 0

Are you sure you actually want to do this?

VLAN 0 IS special. Frames received on VLAN 0 should be treated just as if they had no VLAN tag at all, except that they have an 802.1p value.

Quoted 9 lines

Sending frames with VLAN 0 should have something to do with whether the sender wants to use 802.1p, which doesn't really have much to do with VLAN's at all...

It would be nice if the unsuspecting user was at least warned that their use of VLAN 0 is non-standard and may cause surprising results like leakage into the "native" VLAN. That could be done in /sbin/ip or /sbin/vconfig, of course.

Quotting http://en.wikipedia.org/wiki/IEEE_802.1Q

VLAN Identifier (VID): a 12-bit field specifying the VLAN to which the frame belongs.
A value of 0 means that the frame doesn't belong to any VLAN; in this case the 802.1Q tag specifies only a priority and is referred to as a priority tag.
A value of hex FFF is reserved for implementation use.
All other values may be used as VLAN identifiers, allowing up to 4094 VLANs

Quoted 5 lines

So we expect to generate a 802.1Q frame, even with a VID=0 field.
Before patch, device sends a non 802.1Q frame, which is not what was wanted by user. (Maybe he wants to check its device/network is able to transport 1522 bytes frames, who knows...)

To use non tagged frames, user selects eth0 device, and to send tagged frames, he selects eth0.0

I agree. Quoting IEEE802.1Q:

0 The null VLAN ID. Indicates that the tag header contains only priority
information; no VLAN identifier is present in the frame. ...

which implies that its valid to use 0 as tag in the header.

Quoted 2 lines

Now, maybe eth0 and eth0.0 should share same IP addresses, because incoming frame with ID=0 tag should be received by eth0 device, but I am not sure standard requires this.

I don't think the standard makes any demands regarding this.

The current behaviour seems better to me since its symetrical to the output path, where the VLAN device is necessary to be able to specify a priority mapping.