ixgbe and mac-vlans problem

DISCUSSION Ben Greear 4 participants 14 messages Apr 30, 2010 – May 7, 2010

Subsystems (2)

Networking Network drivers

lore

Share & tools

lore

Ben Greear

author

2010-04-30 17:33 UTC #0 lore raw

Hello!

We are seeing a strange problem with mac-vlans on top of an 82599 NIC (ixgbe) on a 2.6.31.y (plus patches) kernel.

Basically, we create 50 mac-vlans, with sequential MAC addresses and sequential IP addresses, and set up ip rules properly.

The issue is that only 10 or so of the mac-vlans receive other than broadcast packets. The ixgbe NIC doesn't show PROMISC mode.

If we manually turn on PROMISC mode on the physical ixgbe interface, then everything works properly. I'm guessing the ixgbe NIC must have some filters that mac-vlan is trying to enable w/out turning on full PROMISC mode. Is there any easy way to debug these filters to see if they are being properly configured?

Thanks,
Ben

Signature · 2 lines

Ben Greear <[email protected]>
Candela Technologies Inc  http://www.candelatech.com

Patrick McHardy

2010-04-30 17:49 UTC #1 lore raw

Quoted 6 lines · Ben Greear

Hello!
We are seeing a strange problem with mac-vlans on top of an 82599 NIC (ixgbe)
on a 2.6.31.y (plus patches) kernel.

Quoted 6 lines

Basically, we create 50 mac-vlans, with sequential MAC addresses and sequential
IP addresses, and set up ip rules properly.
The issue is that only 10 or so of the mac-vlans receive other than broadcast packets. The ixgbe NIC doesn't show PROMISC mode.

Not showing PROMISC is normal since this is going into promisc when too many unicast addresses are configured is handled internally by the drivers.

Quoted 5 lines

If we manually turn on PROMISC mode on the physical ixgbe interface, then everything works properly. I'm guessing the ixgbe NIC must have some filters that mac-vlan is trying to enable w/out turning on full PROMISC mode. Is there any easy way to debug these filters to see if they are being properly configured?

Adding "#define DEBUG" to ixgbe_common.c should print some information when unicast addresses are added.

Arnd Bergmann

2010-04-30 18:01 UTC #2 lore raw

In reply to Ben Greear

Quoted 5 lines · On Friday 30 April 2010 00:27:39 Ben Greear wrote:

Basically, we create 50 mac-vlans, with sequential MAC addresses and sequential IP addresses, and set up ip rules properly.
The issue is that only 10 or so of the mac-vlans receive other than broadcast packets. The ixgbe NIC doesn't show PROMISC mode.

I just took a brief look at the driver and noticed that 82599 should be able to handle 128 entries before going into promisc mode, while 82598 (the same driver) does 16.

Maybe the logic for >16 entries is wrong, so you could try forcing hw->mac.num_rar_entries to 16 for 82599 as well.

Arnd

Ben Greear

author

2010-04-30 18:10 UTC #3 lore raw

Quoted 7 lines · Arnd Bergmann

Quoted 5 lines · On Friday 30 April 2010 00:27:39 Ben Greear wrote:
Basically, we create 50 mac-vlans, with sequential MAC addresses and sequential IP addresses, and set up ip rules properly.
The issue is that only 10 or so of the mac-vlans receive other than broadcast packets. The ixgbe NIC doesn't show PROMISC mode.

Quoted 6 lines

I just took a brief look at the driver and noticed that 82599 should be able to handle 128 entries before going into promisc mode, while 82598 (the same driver) does 16.
Maybe the logic for>16 entries is wrong, so you could try forcing hw->mac.num_rar_entries to 16 for 82599 as well.

I think I was actually on an 825998 system when I saw it yesterday, but I have seen similar issues on 82599, though I didn't take time to debug it fully, so it could have been something else.

I will double-check the NIC chipset on the system that showed the problem yesterday.

Thanks,
Ben

Signature · 2 lines

Ben Greear <[email protected]>
Candela Technologies Inc  http://www.candelatech.com

Tantilov, Emil S

2010-04-30 21:13 UTC #4 lore raw

Quoted 15 lines · Ben Greear

Quoted 7 lines · Arnd Bergmann
Quoted 5 lines · On Friday 30 April 2010 00:27:39 Ben Greear wrote:
Basically, we create 50 mac-vlans, with sequential MAC addresses and sequential IP addresses, and set up ip rules properly.
The issue is that only 10 or so of the mac-vlans receive other than broadcast packets. The ixgbe NIC doesn't show PROMISC mode.
Quoted 6 lines
I just took a brief look at the driver and noticed that 82599 should be able to handle 128 entries before going into promisc mode, while 82598 (the same driver) does 16.
Maybe the logic for>16 entries is wrong, so you could try forcing hw->mac.num_rar_entries to 16 for 82599 as well.

Quoted 6 lines

I think I was actually on an 825998 system when I saw it yesterday, but I have seen similar issues on 82599, though I didn't take time to debug it fully, so it could have been something else.
I will double-check the NIC chipset on the system that showed the problem yesterday.

I ran a quick test in my setup with 82599 and was able to pass traffic on all 50 mac-vlans without issues. This is on net-next.

Thanks,
Emil

Ben Greear

author

2010-04-30 22:05 UTC #5 lore raw

Quoted 5 lines · Emil S

Quoted 2 lines
I will double-check the NIC chipset on the system that showed the problem yesterday.
I ran a quick test in my setup with 82599 and was able to pass traffic on all 50 mac-vlans without issues. This is on net-next.

I attempted this on an 82598 system, and it reproduces the problem for me, so I think that must be what the customer was using as well (I haven't gotten access to their system again, yet)

I created 50 mac-vlans on eth11, but it does NOT go PROMISC mode as far as I can tell. Only 14 of the mac-vlans are working. I'm using 64kbps UDP streams, 1k packets (ie, light load).

[root@i7-1qc-1 lanforge]# cat /sys/class/net/eth11/flags 0x1003

[root@i7-1qc-1 lanforge]# ip link show dev eth11
2: eth11: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000

    link/ether 00:e0:ed:11:25:12 brd ff:ff:ff:ff:ff:ff

I put it into promisc mode and now they all work:

[root@i7-1qc-1 lanforge]# ip link show dev eth11
2: eth11: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000

    link/ether 00:e0:ed:11:25:12 brd ff:ff:ff:ff:ff:ff

[root@i7-1qc-1 lanforge]# cat /sys/class/net/eth11/flags 0x1103

This is on a slightly modified 2.6.31.12 kernel. I haven't had a chance to try this on a more recent kernel yet.

lspci:
03:00.0 Ethernet controller: Intel Corporation 82598EB 10 Gigabit AF Dual Port Network Connection (rev 01) 03:00.1 Ethernet controller: Intel Corporation 82598EB 10 Gigabit AF Dual Port Network Connection

Please let me know if I can offer any additional information. It will probably take a few days before I can get the .34 kernel in testing.

Thanks,
Ben

Signature · 2 lines

Ben Greear <[email protected]>
Candela Technologies Inc  http://www.candelatech.com

Ben Greear

author

2010-04-30 22:20 UTC #6 lore raw

In reply to Tantilov, Emil S

Quoted 20 lines · Emil S

Quoted 19 lines · Ben Greear
Quoted 7 lines · Arnd Bergmann
Quoted 5 lines · On Friday 30 April 2010 00:27:39 Ben Greear wrote:
Basically, we create 50 mac-vlans, with sequential MAC addresses and sequential IP addresses, and set up ip rules properly.
The issue is that only 10 or so of the mac-vlans receive other than broadcast packets. The ixgbe NIC doesn't show PROMISC mode.
Quoted 6 lines
I just took a brief look at the driver and noticed that 82599 should be able to handle 128 entries before going into promisc mode, while 82598 (the same driver) does 16.
Maybe the logic for>16 entries is wrong, so you could try forcing hw->mac.num_rar_entries to 16 for 82599 as well.
I think I was actually on an 825998 system when I saw it yesterday, but I have seen similar issues on 82599, though I didn't take time to debug it fully, so it could have been something else.

Quoted 5 lines

Quoted 2 lines
I will double-check the NIC chipset on the system that showed the problem yesterday.
I ran a quick test in my setup with 82599 and was able to pass traffic on all 50 mac-vlans without issues. This is on net-next.

For an 82599 system, I can get 127 mac-vlans working out of 500 created.

That NIC also does not go PROMISC with lots (500) of mac-vlans.

Once I put it in promisc mode manually, it works fine.

So, I think whatever logic is supposed to put the NIC into promisc mode when it overflows it's lookup tables isn't working for ixgbe in 2.6.31.12.

Thanks,
Ben

Quoted 3 lines

Thanks,
Emil

Signature · 2 lines

Ben Greear <[email protected]>
Candela Technologies Inc  http://www.candelatech.com

Tantilov, Emil S

2010-04-30 22:26 UTC #7 lore raw

Quoted 30 lines · Ben Greear

Quoted 20 lines · Emil S
Quoted 19 lines · Ben Greear
Quoted 7 lines · Arnd Bergmann
Quoted 5 lines · On Friday 30 April 2010 00:27:39 Ben Greear wrote:
Basically, we create 50 mac-vlans, with sequential MAC addresses and sequential IP addresses, and set up ip rules properly.
The issue is that only 10 or so of the mac-vlans receive other than broadcast packets. The ixgbe NIC doesn't show PROMISC mode.
Quoted 6 lines
I just took a brief look at the driver and noticed that 82599 should be able to handle 128 entries before going into promisc mode, while 82598 (the same driver) does 16.
Maybe the logic for>16 entries is wrong, so you could try forcing hw->mac.num_rar_entries to 16 for 82599 as well.
I think I was actually on an 825998 system when I saw it yesterday, but I have seen similar issues on 82599, though I didn't take time to debug it fully, so it could have been something else.
Quoted 5 lines
Quoted 2 lines
I will double-check the NIC chipset on the system that showed the problem yesterday.
I ran a quick test in my setup with 82599 and was able to pass traffic on all 50 mac-vlans without issues. This is on net-next.
For an 82599 system, I can get 127 mac-vlans working out of 500 created.

Quoted 7 lines

That NIC also does not go PROMISC with lots (500) of mac-vlans.
Once I put it in promisc mode manually, it works fine.
So, I think whatever logic is supposed to put the NIC into promisc mode when it overflows it's lookup tables isn't working for ixgbe in 2.6.31.12.

Yeah, you're right. I was able to repro it.

We'll look into it.

Thanks,
Emil

Ben Greear

author

2010-05-06 16:23 UTC #8 lore raw

Quoted 13 lines · Emil S

Quoted 12 lines · Ben Greear
Quoted 2 lines · Emil S
I ran a quick test in my setup with 82599 and was able to pass traffic on all 50 mac-vlans without issues. This is on net-next.
For an 82599 system, I can get 127 mac-vlans working out of 500 created.
That NIC also does not go PROMISC with lots (500) of mac-vlans.
Once I put it in promisc mode manually, it works fine.

Quoted 7 lines

Quoted 3 lines
So, I think whatever logic is supposed to put the NIC into promisc mode when it overflows it's lookup tables isn't working for ixgbe in 2.6.31.12.
Yeah, you're right. I was able to repro it.
We'll look into it.

I'd be happy to test out a patch if you have one available.

If you don't expect to have one soon, please let me know and I'll add work-arounds to my code to throw ixgbe NICs into PROMISC mode manually.

Thanks,
Ben

Signature · 2 lines

Ben Greear <[email protected]>
Candela Technologies Inc  http://www.candelatech.com

Tantilov, Emil S

2010-05-06 17:51 UTC #9 lore raw

Quoted 22 lines · Ben Greear

Quoted 13 lines · Emil S
Quoted 12 lines · Ben Greear
Quoted 2 lines · Emil S
I ran a quick test in my setup with 82599 and was able to pass traffic on all 50 mac-vlans without issues. This is on net-next.
For an 82599 system, I can get 127 mac-vlans working out of 500 created.
That NIC also does not go PROMISC with lots (500) of mac-vlans.
Once I put it in promisc mode manually, it works fine.
Quoted 7 lines
Quoted 3 lines
So, I think whatever logic is supposed to put the NIC into promisc mode when it overflows it's lookup tables isn't working for ixgbe in 2.6.31.12.
Yeah, you're right. I was able to repro it.
We'll look into it.

Quoted 8 lines

I'd be happy to test out a patch if you have one available.
If you don't expect to have one soon, please let me know and I'll add work-arounds to my code to throw ixgbe NICs into PROMISC mode manually.
Thanks,
Ben

Hi Ben,

We do have a patch in testing (see attached). It may not apply cleanly as it is on top of some other patches currently in validation. Let me know if it works for you.

Thanks,
Emil

Introduce uc_set_promisc flag to better handle the enabling of promisc when the number of RARs is exceeded.

Signed-off-by Emil Tantilov <[email protected]>

 drivers/net/ixgbe/ixgbe_common.c |    5 ++++-
 drivers/net/ixgbe/ixgbe_main.c   |    8 ++++----
 drivers/net/ixgbe/ixgbe_type.h   |    1 +
 3 files changed, 9 insertions(+), 5 deletions(-)

drivers/net/ixgbe/ixgbe_common.c +4 -1

index ee42fd6..49775b6 100644
@@ -1397,14 +1397,17 @@ s32 ixgbe_update_uc_addr_list_generic(struct ixgbe_hw *hw,
 			fctrl = IXGBE_READ_REG(hw, IXGBE_FCTRL);
 			fctrl |= IXGBE_FCTRL_UPE;
 			IXGBE_WRITE_REG(hw, IXGBE_FCTRL, fctrl);
+			hw->addr_ctrl.uc_set_promisc = true;
 		}
 	} else {
 		/* only disable if set by overflow, not by user */
-		if (old_promisc_setting && !hw->addr_ctrl.user_set_promisc) {
+		if ((old_promisc_setting && hw->addr_ctrl.uc_set_promisc) &&
+		   !(hw->addr_ctrl.user_set_promisc)){
 			hw_dbg(hw, " Leaving address overflow promisc mode\n");
 			fctrl = IXGBE_READ_REG(hw, IXGBE_FCTRL);
 			fctrl &= ~IXGBE_FCTRL_UPE;
 			IXGBE_WRITE_REG(hw, IXGBE_FCTRL, fctrl);
+			hw->addr_ctrl.uc_set_promisc = false;
 		}
 	}

drivers/net/ixgbe/ixgbe_main.c +4 -4

index 577ac72..7bf3b40 100644
@@ -2970,8 +2970,8 @@ void ixgbe_set_rx_mode(struct net_device *netdev)
 
 	fctrl = IXGBE_READ_REG(hw, IXGBE_FCTRL);
 
-	if (netdev->flags & IFF_PROMISC) {
-		hw->addr_ctrl.user_set_promisc = 1;
+	if (netdev->flags & IFF_PROMISC){
+		hw->addr_ctrl.user_set_promisc = true;	
 		fctrl |= (IXGBE_FCTRL_UPE | IXGBE_FCTRL_MPE);
 		/* don't hardware filter vlans in promisc mode */
 		ixgbe_vlan_filter_disable(adapter);

@@ -2979,11 +2979,11 @@ void ixgbe_set_rx_mode(struct net_device *netdev)
 		if (netdev->flags & IFF_ALLMULTI) {
 			fctrl |= IXGBE_FCTRL_MPE;
 			fctrl &= ~IXGBE_FCTRL_UPE;
-		} else {
+		} else if (!hw->addr_ctrl.uc_set_promisc) {
 			fctrl &= ~(IXGBE_FCTRL_UPE | IXGBE_FCTRL_MPE);
 		}
 		ixgbe_vlan_filter_enable(adapter);
-		hw->addr_ctrl.user_set_promisc = 0;
+		hw->addr_ctrl.user_set_promisc = false;	
 	}
 
 	IXGBE_WRITE_REG(hw, IXGBE_FCTRL, fctrl);

drivers/net/ixgbe/ixgbe_type.h +1 -0

index 1c89cbb..38f26bb 100644
@@ -2285,6 +2285,7 @@ struct ixgbe_addr_filter_info {
 	u32 mc_addr_in_rar_count;
 	u32 mta_in_use;
 	u32 overflow_promisc;
+	bool uc_set_promisc;
 	bool user_set_promisc;
 };

Ben Greear

author

2010-05-06 20:49 UTC #10 lore raw

Quoted 3 lines · Emil S

Hi Ben,
We do have a patch in testing (see attached). It may not apply cleanly as it is on top of some other patches currently in validation. Let me know if it works for you.

It wasn't difficult to backport this patch to 2.6.31.12....

I just tested this on an 85998 NIC and 50 MAC-VLANs worked fine.

The NIC doesn't show as PROMISC in any way I can detect, but I guess it must actually be in PROMISC mode:

[root@i7-1qc-1 ~]# cat /sys/class/net/eth11/flags
0x1003

[root@i7-1qc-1 ~]# ip link show dev eth11
2: eth11: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000

    link/ether 00:e0:ed:11:25:12 brd ff:ff:ff:ff:ff:ff

Thanks,
Ben

Signature · 2 lines

Ben Greear <[email protected]>
Candela Technologies Inc  http://www.candelatech.com

Tantilov, Emil S

2010-05-07 00:06 UTC #11 lore raw

Quoted 8 lines · Ben Greear

Quoted 5 lines · Emil S
Hi Ben,
We do have a patch in testing (see attached). It may not apply cleanly as it is on top of some other patches currently in validation. Let me know if it works for you.

Quoted 6 lines

It wasn't difficult to backport this patch to 2.6.31.12....
I just tested this on an 85998 NIC and 50 MAC-VLANs worked fine.
The NIC doesn't show as PROMISC in any way I can detect, but I guess it must actually be in PROMISC mode:

Yes the interface is in promisc mode. The driver sets the FCTRL.UPE bit (unicast promisc mode) when the number of allowed rar_entries is exceeded.

Quoted 8 lines

[root@i7-1qc-1 ~]# cat /sys/class/net/eth11/flags
0x1003
[root@i7-1qc-1 ~]# ip link show dev eth11
2: eth11: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast
     state UP qlen 1000 link/ether 00:e0:ed:11:25:12 brd
ff:ff:ff:ff:ff:ff

The IFF_PROMISC flag is not set in this case. That's how the driver knows when the promisc mode is turned by the user.

Thanks,
Emil

Ben Greear

author

2010-05-07 03:12 UTC #12 lore raw

Quoted 11 lines · Emil S

Quoted 10 lines · Ben Greear
Quoted 5 lines · Emil S
Hi Ben,
We do have a patch in testing (see attached). It may not apply cleanly as it is on top of some other patches currently in validation. Let me know if it works for you.
It wasn't difficult to backport this patch to 2.6.31.12....

Quoted 7 lines

Quoted 4 lines
I just tested this on an 85998 NIC and 50 MAC-VLANs worked fine.
The NIC doesn't show as PROMISC in any way I can detect, but I guess it must actually be in PROMISC mode:
Yes the interface is in promisc mode. The driver sets the FCTRL.UPE bit (unicast promisc mode) when the number of allowed rar_entries is exceeded.

Is there any way to get this setting from ethtool or similar? It would be nice to know the actual PROMISC state of the NIC regardless of what user-space has or has not configured.

Thanks,
Ben

Signature · 2 lines

Ben Greear <[email protected]>
Candela Technologies Inc  http://www.candelatech.com

Tantilov, Emil S

2010-05-07 17:40 UTC #13 lore raw

Quoted 17 lines · Ben Greear

Quoted 16 lines · Emil S
Quoted 14 lines · Ben Greear
Quoted 5 lines · Emil S
Hi Ben,
We do have a patch in testing (see attached). It may not apply cleanly as it is on top of some other patches currently in validation. Let me know if it works for you.
It wasn't difficult to backport this patch to 2.6.31.12....
I just tested this on an 85998 NIC and 50 MAC-VLANs worked fine.
The NIC doesn't show as PROMISC in any way I can detect, but I guess it must actually be in PROMISC mode:

Quoted 7 lines

Quoted 3 lines
Yes the interface is in promisc mode. The driver sets the FCTRL.UPE bit (unicast promisc mode) when the number of allowed rar_entries is exceeded.
Is there any way to get this setting from ethtool or similar? It would be nice to know the actual PROMISC state of the NIC regardless of what user-space has or has not configured.

ethtool -d eth11 | grep -i promisc

Thanks,
Emil