VLAN and ARP failure on tg3 drivers
Subsystems (3)
Dear Matt, Eric, Benny,
Sorry about the slow response to your fast replies. I think Benny is correct, the 'problem' lies in the fact that we were using a VLAN ID of 0, without knowing its special significance. User error.
I tested it with other VLAN id's (>0) and it appears to work fine. We are not entirely sure we understand why it used to work with VLAN ID 0 on the Broadcom chips and still does with a number of different cards (with >2.6.27 kernels). What is the 'correct' behaviour for this incorrect usage ? When a PC returns the ARP response to the machine with the BroadCom card, it will have the destination address of the VLAN device, but presumably the VLAN ID tag set to zero. Hmmm. I can live with not knowing the answer I guess.
Thanks again,
Gertjan
Quoted 67 lines · Matt Carlson
From: Matt Carlson <[email protected]>
Subject: Re: VLAN and ARP failure on tg3 drivers
To: "Gertjan Hofman" <[email protected]>
Cc: "[email protected]" <[email protected]>
Date: Friday, October 23, 2009, 3:35 PMQuoted 3 lines · Gertjan Hofman
Dear Kernel developers,
A couple of weeks ago we tried to migrate from a
2.6.24? kernel to a 2.6.29 kernel and noticed our VLAN application no longer works.? The problem is easy to replicate:
Quoted 3 lines
1. connect 2 PC's with a cross-over cable
2. set up a fixed IP address to both PC's? (say192.168.0.[1,2])
Quoted 2 lines
3. create a vlan:? vconfig? add eth0 0.
4. set IP addresses on the VLAN devices? (say192.168.1.[1,2])
Quoted 3 lines
5. try ping one machine from the other.
I tried to dig into the problem by using un-patched
kernel.org kernels with Ubuntu .config files.? Kernels up to 2.6.26 work fine, kernels after and including 2.6.27 fail. The problem is that ARP messages are being dropped. If the ARP table is updated by hand on each machine, the communication across the VLAN works fine.
Quoted 2 lines
At first I thought the kernel VLAN code was the
problem (we had an earlier issue with a regression in 2.6.24) but it looks like the problem is actually with the tg3 driver.? Our system uses Broadcom ethernet chips. I tried the same experiments with combination of boards that have Broadcom and none-Broadcom and the only time I see it fail is with the tg3? driver loaded.
Quoted 2 lines
Snooping with WireShark shows that a ARP request from
the non-Broadcom machine is seen and even answered, but never appears back on the network. If the Broadcom machine orginates the ARP message, it never arrives at the destination. I tried lowering the size of the MTU to 1492 as well as giving each VLAN device a different MAC. No deal.
Quoted 2 lines
I tried to look at tg3 patch changes from 2.6.26 to
2.6.27 but I am not familiar enough with the Git system to extract the appropiate changes.? I am a bit surprised that I am not seeing any references to this on the web, the combination of >2.6.27 kernels, Broadcom and VLAN cant be that uncommon.
Quoted 2 lines
I would be happy to provide more information and to
try tests if any one can suggest them.
Quoted 4 lines
Sincerely,
Gertjan
I don't see any reason why your setup should fail, but it doesn't hurt
to gather more info about the problem.What device are you experiencing this problem with? Is management
firmware enabled? (ethtool -i ethx)
Quoted 11 lines · Gertjan Hofman
Dear Matt, Eric, Benny,
Sorry about the slow response to your fast replies. I think Benny is correct, the 'problem' lies in the fact that we were using a VLAN ID of 0, without knowing its special significance. User error.
I tested it with other VLAN id's (>0) and it appears to work fine. We are not entirely sure we understand why it used to work with VLAN ID 0 on the Broadcom chips and still does with a number of different cards (with >2.6.27 kernels). What is the 'correct' behaviour for this incorrect usage ?
VLAN 0 isn't incorrect, it's just surprising. When you send a packet tagged with VLAN 0, it means that the packet should be interpreted as being the same VLAN as a completely untagged packet.
So in theory, if both ends are using VLAN 0 and you aren't using eth0 for anything, traffic should flow, at least if both ends are on the same kernel version. Feel free to debug why that isn't the case for you, of course...
/Benny
Quoted 14 lines · Benny Amorsen
Quoted 11 lines · Gertjan Hofman
Dear Matt, Eric, Benny,
Sorry about the slow response to your fast replies. I think Benny is correct, the 'problem' lies in the fact that we were using a VLAN ID of 0, without knowing its special significance. User error.
I tested it with other VLAN id's (>0) and it appears to work fine. We are not entirely sure we understand why it used to work with VLAN ID 0 on the Broadcom chips and still does with a number of different cards (with >2.6.27 kernels). What is the 'correct' behaviour for this incorrect usage ?
Quoted 9 lines
VLAN 0 isn't incorrect, it's just surprising. When you send a packet tagged with VLAN 0, it means that the packet should be interpreted as being the same VLAN as a completely untagged packet.
So in theory, if both ends are using VLAN 0 and you aren't using eth0 for anything, traffic should flow, at least if both ends are on the same kernel version. Feel free to debug why that isn't the case for you, of course...
VLAN id 0 is not usable on current kernel because we use 16 bits in skb to
store vlan_tci, and vlan_tci = 0 means there is no VLAN tagging.
We could use high order bit (0x8000) to tell if vlan tagging is set or not.
PATCH 1/1 vlan: allow VLAN ID 0 to be used
Quoted 6 lines · Eric Dumazet
VLAN id 0 is not usable on current kernel because we use 16 bits in skb to
storevlan_tci, andvlan_tci= 0 means there is no VLAN tagging.We could use high order bit (
0x8000) to tell if vlan tagging is set or not.
Here is the patch I cooked that permitted VLAN 0 to be used with tg3 (and other HW accelerated vlan nics I suppose)
[PATCH] vlan: allow VLAN ID 0 to be used
We currently use a 16 bit field (vlan_tci) to store VLAN ID on a skb.
0 value is used a special value, meaning VLAN ID not set. This forbids use of VLAN ID 0
As VLAN ID is 12 bits, we can use high order bit as a flag, and allow VLAN ID 0
---
include/linux/if_vlan.h +5 -4
@@ -105,8 +105,9 @@ static inline void vlan_group_set_device(struct vlan_group *vg, array[vlan_id %VLAN_GROUP_ARRAY_PART_LEN ] = dev; } -#define vlan_tx_tag_present (__skb) ((__skb)->vlan_tci) -#define vlan_tx_tag_get (__skb) ((__skb)->vlan_tci) +#define VLAN_TAG_PRESENT 0x8000 +#define vlan_tx_tag_present (__skb) ((__skb)->vlan_tci & VLAN_TAG_PRESENT) +#define vlan_tx_tag_get (__skb) ((__skb)->vlan_tci & 0x7fff)#if defined(CONFIG_VLAN_8021Q ) || defined(CONFIG_VLAN_8021Q_MODULE )extern struct net_device *vlan_dev_real_dev (const struct net_device *dev);
@@ -231,7 +232,7 @@ static inline struct sk_buff *__vlan_put_tag(struct sk_buff *skb, u16 vlan_tci)static inline struct sk_buff *__vlan_hwaccel_put_tag (struct sk_buff *skb,u16 vlan_tci) { - skb->vlan_tci = vlan_tci; + skb->vlan_tci =VLAN_TAG_PRESENT | vlan_tci;return skb; }
@@ -284,7 +285,7 @@ static inline int __vlan_hwaccel_get_tag(const struct sk_buff *skb, u16 *vlan_tci) {if (vlan_tx_tag_present (skb)) { - *vlan_tci = skb->vlan_tci; + *vlan_tci =vlan_tx_tag_get (skb);return 0 ; }else { *vlan_tci =0 ;
David Miller
maintainerRe: [PATCH] vlan: allow VLAN ID 0 to be used
From: Eric Dumazet <[email protected]>
Date: Mon, 26 Oct 2009 17:13:58 +0100
Quoted 4 lines
[PATCH] vlan: allow VLAN ID 0 to be used
We currently use a 16 bit field (
vlan_tci) to store VLAN ID on a skb.
Quoted 8 lines
0 value is used a special value, meaning VLAN ID not set. This forbids use of VLAN ID 0
As VLAN ID is 12 bits, we can use high order bit as a flag, and allow VLAN ID 0
Reported-by Gertjan Hofman <[email protected]>Signed-off-by Eric Dumazet <[email protected]>
This is going to need some more work.
IXGBE is already using the higher bits of ->vlan_tci internally, your change breaks that.
QLGE explicitly initializes skb->vlan_tci to zero, you'll need to make sure that's OK.
There is an explicit "if (skb->vlan_tci" (ie. zero vs. non-zero) test in net/core/dev.c:netif_receive_skb()
net/core/skbuff.c:__copy_skb_header() does a straight copy, you'll need to make sure that's still OK.
net/packet/af_packet.c:tpacket_rcv() and packet_recvmsg() report the skb->vlan_tci value to userspace, that's broken now as userspace doesn't expect that new bit to be there.
Re: [PATCH] vlan: allow VLAN ID 0 to be used
Quoted 27 lines · David Miller
From: Eric Dumazet <[email protected]>
Date: Mon, 26 Oct 2009 17:13:58 +0100Quoted 4 lines
[PATCH] vlan: allow VLAN ID 0 to be used
We currently use a 16 bit field (
vlan_tci) to store VLAN ID on a skb.Quoted 8 lines
0 value is used a special value, meaning VLAN ID not set. This forbids use of VLAN ID 0
As VLAN ID is 12 bits, we can use high order bit as a flag, and allow VLAN ID 0
Reported-by Gertjan Hofman <[email protected]>Signed-off-by Eric Dumazet <[email protected]>This is going to need some more work.
IXGBE is already using the higher bits of ->vlan_tci internally, your change breaks that.
QLGE explicitly initializes
skb->vlan_tcito zero, you'll need to make sure that's OK.There is an explicit "if (
skb->vlan_tci" (ie. zero vs. non-zero) test in net/core/dev.c:netif_receive_skb()
Quoted 6 lines
net/core/skbuff.c:__copy_skb_header() does a straight copy, you'll need to make sure that's still OK.
net/packet/af_packet.c:tpacket_rcv() and
packet_recvmsg()report theskb->vlan_tcivalue to userspace, that's broken now as userspace doesn't expect that new bit to be there.
Thanks a lot David for this extended review and guidelines
I hope I did not miss another important thing in this second version.
Again, tested on tg3 only, and on net-next-2.6 tree
[PATCH] vlan: allow null VLAN ID to be used
We currently use a 16 bit field (vlan_tci) to store VLAN ID/PRIO on a skb.
Null value is used as a special value, meaning vlan tagging not enabled. This forbids use of null vlan ID.
As pointed by David, some drivers use the 3 high order bits (PRIO)
As VLAN ID is 12 bits, we can use the remaining bit (CFI) as a flag, and allow null VLAN ID.
In case future code really wants to use VLAN_CFI_MASK, we'll have to use a bit outside of vlan_tci.
#define VLAN_PRIO_MASK 0xe000 /* Priority Code Point */ #define VLAN_PRIO_SHIFT 13 #define VLAN_CFI_MASK 0x1000 /* Canonical Format Indicator */ #define VLAN_TAG_PRESENT VLAN_CFI_MASK #define VLAN_VID_MASK 0x0fff /* VLAN Identifier */
--- include/linux/if_vlan.h | 14 +++++++++----- net/8021q/vlan.h | 2 +- net/8021q/vlan_dev.c | 2 +- net/core/dev.c | 2 +- net/packet/af_packet.c | 5 +++-- 5 files changed, 15 insertions(+), 10 deletions(-)
include/linux/if_vlan.h +9 -5
@@ -63,7 +63,11 @@ static inline struct vlan_ethhdr *vlan_eth_hdr(const struct sk_buff *skb)return (struct vlan_ethhdr *)skb_mac_header (skb); } -#define VLAN_VID_MASK 0xfff +#define VLAN_PRIO_MASK 0xe000/* Priority Code Point */ +#define VLAN_PRIO_SHIFT 13 +#define VLAN_CFI_MASK 0x1000/* Canonical Format Indicator */ +#define VLAN_TAG_PRESENT VLAN_CFI_MASK +#define VLAN_VID_MASK 0x0fff/* VLAN Identifier */ /* found in socket.c */ extern void vlan_ioctl_set (int (*hook)(struct net *,void __user *));
@@ -105,8 +109,8 @@ static inline void vlan_group_set_device(struct vlan_group *vg, array[vlan_id %VLAN_GROUP_ARRAY_PART_LEN ] = dev; } -#define vlan_tx_tag_present (__skb) ((__skb)->vlan_tci) -#define vlan_tx_tag_get (__skb) ((__skb)->vlan_tci) +#define vlan_tx_tag_present (__skb) ((__skb)->vlan_tci & VLAN_TAG_PRESENT) +#define vlan_tx_tag_get (__skb) ((__skb)->vlan_tci & ~VLAN_TAG_PRESENT)#if defined(CONFIG_VLAN_8021Q ) || defined(CONFIG_VLAN_8021Q_MODULE )extern struct net_device *vlan_dev_real_dev (const struct net_device *dev);
@@ -231,7 +235,7 @@ static inline struct sk_buff *__vlan_put_tag(struct sk_buff *skb, u16 vlan_tci)static inline struct sk_buff *__vlan_hwaccel_put_tag (struct sk_buff *skb,u16 vlan_tci) { - skb->vlan_tci = vlan_tci; + skb->vlan_tci =VLAN_TAG_PRESENT | vlan_tci;return skb; }
@@ -284,7 +288,7 @@ static inline int __vlan_hwaccel_get_tag(const struct sk_buff *skb, u16 *vlan_tci) {if (vlan_tx_tag_present (skb)) { - *vlan_tci = skb->vlan_tci; + *vlan_tci =vlan_tx_tag_get (skb);return 0 ; }else { *vlan_tci =0 ;
net/8021q/vlan.h +1 -1
@@ -89,7 +89,7 @@ static inline u32 vlan_get_ingress_priority(struct net_device *dev, {struct vlan_dev_info *vip =vlan_dev_info (dev); -return vip->ingress_priority_map[(vlan_tci >>13 ) &0x7 ]; +return vip->ingress_priority_map[(vlan_tci >>VLAN_PRIO_SHIFT ) &0x7 ]; }#ifdef CONFIG_VLAN_8021Q_GVRP
net/8021q/vlan_dev.c +1 -1
@@ -393,7 +393,7 @@ int vlan_dev_set_egress_priority(const struct net_device *dev,struct vlan_dev_info *vlan =vlan_dev_info (dev);struct vlan_priority_tci_mapping *mp =NULL ;struct vlan_priority_tci_mapping *np; -u32 vlan_qos = (vlan_prio <<13 ) &0xE000 ; +u32 vlan_qos = (vlan_prio <<VLAN_PRIO_SHIFT ) &VLAN_PRIO_MASK ;/* See if a priority mapping exists.. */ mp = vlan->egress_priority_map[skb_prio &0xF ];
net/core/dev.c +1 -1
@@ -2303,7 +2303,7 @@ int netif_receive_skb(struct sk_buff *skb)if (!skb->tstamp.tv64)net_timestamp (skb); -if (skb->vlan_tci &&vlan_hwaccel_do_receive (skb)) +if (vlan_tx_tag_present (skb) &&vlan_hwaccel_do_receive (skb))return NET_RX_SUCCESS ;/* if we've gotten here through NAPI, check netpoll */
net/packet/af_packet.c +3 -2
@@ -79,6 +79,7 @@#include <linux/module.h> #include <linux/init.h> #include <linux/mutex.h> +#include <linux/if_vlan.h> #ifdef CONFIG_INET #include <net/inet_common.h>
@@ -766,7 +767,7 @@ static int tpacket_rcv(struct sk_buff *skb, struct net_device *dev,getnstimeofday (&ts); h.h2->tp_sec = ts.tv_sec; h.h2->tp_nsec = ts.tv_nsec; - h.h2->tp_vlan_tci = skb->vlan_tci; + h.h2->tp_vlan_tci =vlan_tx_tag_get (skb); hdrlen =sizeof (*h.h2);break ;default :
@@ -1493,7 +1494,7 @@ static int packet_recvmsg(struct kiocb *iocb, struct socket *sock, aux.tp_snaplen = skb->len; aux.tp_mac =0 ; aux.tp_net =skb_network_offset (skb); - aux.tp_vlan_tci = skb->vlan_tci; + aux.tp_vlan_tci =vlan_tx_tag_get (skb);put_cmsg (msg,SOL_PACKET ,PACKET_AUXDATA ,sizeof (aux), &aux); }
David Miller
maintainerRe: [PATCH] vlan: allow VLAN ID 0 to be used
From: Eric Dumazet <[email protected]>
Date: Tue, 27 Oct 2009 02:34:57 +0100
Quoted 5 lines
I hope I did not miss another important thing in this second version.
Again, tested on tg3 only, and on net-next-2.6 tree
[PATCH] vlan: allow null VLAN ID to be used
Looks good, applied to net-next-2.6
Someone now needs to convert IXGBE to use VLAN_PRIO_MASK and VLAN_PRIO_SHIFT instead of it's internal macros.
Re: [PATCH] vlan: allow VLAN ID 0 to be used
In reply to Eric DumazetQuoted 9 lines · Eric Dumazet
Here is the patch I cooked that permitted VLAN 0 to be used with tg3 (and other HW accelerated vlan nics I suppose)
[PATCH] vlan: allow VLAN ID 0 to be used
We currently use a 16 bit field (
vlan_tci) to store VLAN ID on a skb.0 value is used a special value, meaning VLAN ID not set. This forbids use of VLAN ID 0
Are you sure you actually want to do this?
VLAN 0 IS special. Frames received on VLAN 0 should be treated just as if they had no VLAN tag at all, except that they have an 802.1p value.
Sending frames with VLAN 0 should have something to do with whether the sender wants to use 802.1p, which doesn't really have much to do with VLAN's at all...
It would be nice if the unsuspecting user was at least warned that their use of VLAN 0 is non-standard and may cause surprising results like leakage into the "native" VLAN. That could be done in /sbin/ip or /sbin/vconfig, of course.
/Benny
Re: [PATCH] vlan: allow VLAN ID 0 to be used
Quoted 17 lines · Benny Amorsen
Quoted 9 lines · Eric Dumazet
Here is the patch I cooked that permitted VLAN 0 to be used with tg3 (and other HW accelerated vlan nics I suppose)
[PATCH] vlan: allow VLAN ID 0 to be used
We currently use a 16 bit field (
vlan_tci) to store VLAN ID on a skb.0 value is used a special value, meaning VLAN ID not set. This forbids use of VLAN ID 0
Are you sure you actually want to do this?
VLAN 0 IS special. Frames received on VLAN 0 should be treated just as if they had no VLAN tag at all, except that they have an 802.1p value.
Quoted 9 lines
Sending frames with VLAN 0 should have something to do with whether the sender wants to use 802.1p, which doesn't really have much to do with VLAN's at all...
It would be nice if the unsuspecting user was at least warned that their use of VLAN 0 is non-standard and may cause surprising results like leakage into the "native" VLAN. That could be done in
/sbin/ipor/sbin/vconfig, of course.
Quotting http://en.wikipedia.org/wiki/IEEE_802.1Q
VLAN Identifier (VID): a 12-bit field specifying the VLAN to which the frame belongs.
A value of 0 means that the frame doesn't belong to any VLAN; in this case the 802.1Q tag specifies only a priority and is referred to as a priority tag.
A value of hex FFF is reserved for implementation use.
All other values may be used as VLAN identifiers, allowing up to 4094 VLANs
So we expect to generate a 802.1Q frame, even with a VID=0 field.
Before patch, device sends a non 802.1Q frame, which is not what was wanted by user. (Maybe he wants to check its device/network is able to transport 1522 bytes frames, who knows...)
To use non tagged frames, user selects eth0 device, and to send tagged frames, he selects eth0.0
Now, maybe eth0 and eth0.0 should share same IP addresses, because incoming frame with ID=0 tag should be received by eth0 device, but I am not sure standard requires this.
Re: [PATCH] vlan: allow VLAN ID 0 to be used
Quoted 36 lines · Eric Dumazet
Quoted 16 lines · Benny Amorsen
Quoted 9 lines · Eric Dumazet
Here is the patch I cooked that permitted VLAN 0 to be used with tg3 (and other HW accelerated vlan nics I suppose)
[PATCH] vlan: allow VLAN ID 0 to be used
We currently use a 16 bit field (
vlan_tci) to store VLAN ID on a skb.0 value is used a special value, meaning VLAN ID not set. This forbids use of VLAN ID 0
Are you sure you actually want to do this?
VLAN 0 IS special. Frames received on VLAN 0 should be treated just as if they had no VLAN tag at all, except that they have an 802.1p value.
Quoted 9 lines
Sending frames with VLAN 0 should have something to do with whether the sender wants to use 802.1p, which doesn't really have much to do with VLAN's at all...
It would be nice if the unsuspecting user was at least warned that their use of VLAN 0 is non-standard and may cause surprising results like leakage into the "native" VLAN. That could be done in
/sbin/ipor/sbin/vconfig, of course.Quotting http://en.wikipedia.org/wiki/IEEE_802.1Q
VLAN Identifier (VID): a 12-bit field specifying the VLAN to which the frame belongs.
A value of 0 means that the frame doesn't belong to any VLAN; in this case the 802.1Q tag specifies only a priority and is referred to as a priority tag.
A value of hex FFF is reserved for implementation use.
All other values may be used as VLAN identifiers, allowing up to 4094 VLANs
Quoted 5 lines
So we expect to generate a 802.1Q frame, even with a VID=0 field.
Before patch, device sends a non 802.1Q frame, which is not what was wanted by user. (Maybe he wants to check its device/network is able to transport 1522 bytes frames, who knows...)To use non tagged frames, user selects eth0 device, and to send tagged frames, he selects eth0.0
I agree. Quoting IEEE802.1Q:
0 The null VLAN ID. Indicates that the tag header contains only priority
information; no VLAN identifier is present in the frame. ...
which implies that its valid to use 0 as tag in the header.
Quoted 2 lines
Now, maybe eth0 and eth0.0 should share same IP addresses, because incoming frame with ID=0 tag should be received by eth0 device, but I am not sure standard requires this.
I don't think the standard makes any demands regarding this.
The current behaviour seems better to me since its symetrical to the output path, where the VLAN device is necessary to be able to specify a priority mapping.