Skip to contentSkip to conversation

behaviour question for igb on nehalem box

DISCUSSION Chris Friesen 4 participants 8 messages Oct 9, 2009 – Oct 16, 2009

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

Hi all,

I've got some general questions around the expected behaviour of the 82576 igb net device. (On a dual quad-core Nehalem box, if it matters.)

As a caveat, the box is running Centos 5.3 with their 2.6.18 kernel. It's using the 1.3.16-k2 igb driver though, which looks to be the one from mainline linux.

The igb driver is being loaded with no parameters specified. At driver init time, it's selecting 1 tx queue and 4 rx queues per device.

My first question is whether the number of queues makes sense. I couldn't figure out how this would happen since the rules for selecting the number of queues seems to be the same for rx and tx. Also, it's not clear to me why it's limiting itself to 4 rx queues when I have 8 physical cores (and 16 virtual ones with hyperthreading enabled).

My second question is around how the rx queues are mapped to interrupts.
According to /proc/interrupts there appears to be a 1:1 mapping between
queues and interrupts. However, I've set up at test with a given amount of traffic coming in to the device (from 4 different IP addresses and 4 ports). Under this scenario, "ethtool -S" shows the number of packets increasing for only rx queue 0, but I see the interrupt count going up for two interrupts.

My final question is around smp affinity for the rx and tx queue interrupts. Do I need to affine the interrupt for each rx queue to a single core to guarantee proper packet ordering, or can they be handled on arbitrary cores? Should the tx queue be affined to a particular core or left to be handled by all cores?

Thanks,

Chris

------------------------------------------------------------------------------
Come build with us! The BlackBerry(R) Developer Conference in SF, CA is the only developer event you need to attend this year. Jumpstart your developing skills, take BlackBerry mobile applications to market and stay ahead of the curve. Join us from November 9 - 12, 2009. Register now!
http://p.sf.net/sfu/devconference

#1 lore raw
Quoted 11 lines · Chris Friesen

I've got some general questions around the expected behaviour of the 82576 igb net device. (On a dual quad-core Nehalem box, if it matters.)

As a caveat, the box is running Centos 5.3 with their 2.6.18 kernel. It's using the 1.3.16-k2 igb driver though, which looks to be the one from mainline linux.

The igb driver is being loaded with no parameters specified. At driver init time, it's selecting 1 tx queue and 4 rx queues per device.

My first question is whether the number of queues makes sense. I

It does for this kernel, because 2.6.18 doesn't support multiple tx queues. The hardware supports RSS over receive queues, and the driver doesn't mention the multiple receive queues from the OS.

Quoted 4 lines

couldn't figure out how this would happen since the rules for selecting the number of queues seems to be the same for rx and tx. Also, it's not clear to me why it's limiting itself to 4 rx queues when I have 8 physical cores (and 16 virtual ones with hyperthreading enabled).

for gigabit more queues is not necessarily better, and MQ arguably isn't necessary at all for gigabit. However, it can help for some workloads when spreading out RX traffic. the hardware you have only supports 8 queues (rx and tx) and the driver is configured to only set up 4 max.

Quoted 7 lines

My second question is around how the rx queues are mapped to interrupts.
According to /proc/interrupts there appears to be a 1:1 mapping between
queues and interrupts. However, I've set up at test with a given amount of traffic coming in to the device (from 4 different IP addresses and 4 ports). Under this scenario, "ethtool -S" shows the number of packets increasing for only rx queue 0, but I see the interrupt count going up for two interrupts.

one transmit interrupt and one receive interrupt? RSS will spread the receive work out in a flow based way, based on ip/xDP header. Your test as described should be using more than one flow (and therefore more than one rx queue) unless you got caught out by the default arp_filter behavior (check arp -an).

Quoted 5 lines

My final question is around smp affinity for the rx and tx queue interrupts. Do I need to affine the interrupt for each rx queue to a single core to guarantee proper packet ordering, or can they be handled on arbitrary cores? Should the tx queue be affined to a particular core or left to be handled by all cores?

on RHEL5.3 you can use irqbalance, you shouldn't need to hand affine anything. Packets won't be received out of order unless you have the rx interrupts going to more that one cpu per queue. (smp_affinity mask has more than one bit set) RSS is doing flow steering.

going to a 2.6.27 or newer kernel will get you full tx multiqueue support.

Hope this helps,
Jesse

------------------------------------------------------------------------------
Come build with us! The BlackBerry(R) Developer Conference in SF, CA is the only developer event you need to attend this year. Jumpstart your developing skills, take BlackBerry mobile applications to market and stay ahead of the curve. Join us from November 9 - 12, 2009. Register now!
http://p.sf.net/sfu/devconference

#2 lore raw
Quoted 6 lines · Jesse
Quoted 2 lines · Chris Friesen

I've got some general questions around the expected behaviour of the 82576 igb net device. (On a dual quad-core Nehalem box, if it matters.)

the hardware you have only supports 8
queues (rx and tx) and the driver is configured to only set up 4 max.

The datasheet for the 82576 says 16 tx queues and 16 rx queues. Is that a typo or do we have the economy version?

Quoted 9 lines
Quoted 7 lines

My second question is around how the rx queues are mapped to interrupts.
According to /proc/interrupts there appears to be a 1:1 mapping between
queues and interrupts. However, I've set up at test with a given amount of traffic coming in to the device (from 4 different IP addresses and 4 ports). Under this scenario, "ethtool -S" shows the number of packets increasing for only rx queue 0, but I see the interrupt count going up for two interrupts.

one transmit interrupt and one receive interrupt?

No, two rx interrupts. (Can't remember if the tx interrupt was going up as well or no...was only looking at rx.)

Quoted 5 lines

RSS will spread the
receive work out in a flow based way, based on ip/xDP header. Your test as described should be using more than one flow (and therefore more than one rx queue) unless you got caught out by the default arp_filter behavior (check arp -an).

I was surprised as well since it didn't match what I expected. What's the story around the arp_filter? I just logged onto the test box and "arp -an" gives:

? (47.135.251.129) at 00:00:5E:00:01:08 [ether] on eth0

but I'm not sure that's worth anything since someone is running a test and it's currently using all four rx queues and all four rx interrupt counts are increasing. I'll have to see if they changed anything.

Quoted 1 line

Hope this helps,

That's great, thanks.

Chris

------------------------------------------------------------------------------
Come build with us! The BlackBerry(R) Developer Conference in SF, CA is the only developer event you need to attend this year. Jumpstart your developing skills, take BlackBerry mobile applications to market and stay ahead of the curve. Join us from November 9 - 12, 2009. Register now!
http://p.sf.net/sfu/devconference

#3 lore raw
Quoted 10 lines · Chris Friesen
Quoted 6 lines · Jesse
Quoted 2 lines · Chris Friesen

I've got some general questions around the expected behaviour of the 82576 igb net device. (On a dual quad-core Nehalem box, if it matters.)

the hardware you have only supports 8
queues (rx and tx) and the driver is configured to only set up 4 max.

The datasheet for the 82576 says 16 tx queues and 16 rx queues. Is that a typo or do we have the economy version?

Actually the limitation is due to the fact that there are only 10 interrupts available. On kernels that support TX multi-queue the number of queues would be 4 TX and 4 RX, which would consume 8 interrupts leaving 1 for the link status change and one unused.

However on the kernel you are using I don't believe multi-queue NAPI is enabled so you shouldn't have multiple RX queues either. On a 2.6.18 kernel you should have only 1 RX and 1 TX queue unless you are using the driver provided on e1000.sourceforge.net which uses fake netdevs to support multi-queue NAPI. I believe this may be a bug that was introduced when SR-IOV support was back-ported from the 2.6.30 kernel.

Quoted 11 lines
Quoted 8 lines
Quoted 7 lines

My second question is around how the rx queues are mapped to interrupts.
According to /proc/interrupts there appears to be a 1:1 mapping between
queues and interrupts. However, I've set up at test with a given amount of traffic coming in to the device (from 4 different IP addresses and 4 ports). Under this scenario, "ethtool -S" shows the number of packets increasing for only rx queue 0, but I see the interrupt count going up for two interrupts.

one transmit interrupt and one receive interrupt?

No, two rx interrupts. (Can't remember if the tx interrupt was going up as well or no...was only looking at rx.)

This may be due to the bug I mentioned above. Multiple RX queues shouldn't be present on the 2.6.18 kernel as I do not believe multi-queue NAPI has been back-ported and it could have negative effects.

Quoted 23 lines
Quoted 5 lines

RSS will spread the
receive work out in a flow based way, based on ip/xDP header. Your test as described should be using more than one flow (and therefore more than one rx queue) unless you got caught out by the default arp_filter behavior (check arp -an).

I was surprised as well since it didn't match what I expected. What's the story around the arp_filter? I just logged onto the test box and "arp -an" gives:

? (47.135.251.129) at 00:00:5E:00:01:08 [ether] on eth0

but I'm not sure that's worth anything since someone is running a test and it's currently using all four rx queues and all four rx interrupt counts are increasing. I'll have to see if they changed anything.

Quoted 1 line

Hope this helps,

That's great, thanks.

Chris

Quoted 10 lines

------------------------------------------------------------------------------
Come build with us! The BlackBerry(R) Developer Conference in SF, CA is the only developer event you need to attend this year. Jumpstart your developing skills, take BlackBerry mobile applications to market and stay ahead of the curve. Join us from November 9 - 12, 2009. Register now!
http://p.sf.net/sfu/devconference
_______________________________________________
E1000-devel mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/e1000-devel

------------------------------------------------------------------------------
Come build with us! The BlackBerry(R) Developer Conference in SF, CA is the only developer event you need to attend this year. Jumpstart your developing skills, take BlackBerry mobile applications to market and stay ahead of the curve. Join us from November 9 - 12, 2009. Register now!
http://p.sf.net/sfu/devconference

#4 lore raw
Quoted 15 lines · Alexander Duyck
Quoted 8 lines · Chris Friesen
Quoted 5 lines · Jesse
Quoted 2 lines · Chris Friesen

I've got some general questions around the expected behaviour of the 82576 igb net device. (On a dual quad-core Nehalem box, if it matters.)

the hardware you have only supports 8
queues (rx and tx) and the driver is configured to only set up 4 max.

The datasheet for the 82576 says 16 tx queues and 16 rx queues. Is that a typo or do we have the economy version?

Actually the limitation is due to the fact that there are only 10 interrupts available. On kernels that support TX multi-queue the number of queues would be 4 TX and 4 RX, which would consume 8 interrupts leaving 1 for the link status change and one unused.

Quoted 6 lines

However on the kernel you are using I don't believe multi-queue NAPI is enabled so you shouldn't have multiple RX queues either. On a 2.6.18 kernel you should have only 1 RX and 1 TX queue unless you are using the driver provided on e1000.sourceforge.net which uses fake netdevs to support multi-queue NAPI. I believe this may be a bug that was introduced when SR-IOV support was back-ported from the 2.6.30 kernel.

Actually after looking closer at the Redhat source it looks like they have done the fake netdev workaround in their own code so I guess igb driver in the RHEL kernel does support multiple RX queues.

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines

My second question is around how the rx queues are mapped to interrupts.
According to /proc/interrupts there appears to be a 1:1 mapping between
queues and interrupts. However, I've set up at test with a given amount of traffic coming in to the device (from 4 different IP addresses and 4 ports). Under this scenario, "ethtool -S" shows the number of packets increasing for only rx queue 0, but I see the interrupt count going up for two interrupts.

Quoted 7 lines
Quoted 3 lines
Quoted 1 line

one transmit interrupt and one receive interrupt?

No, two rx interrupts. (Can't remember if the tx interrupt was going up as well or no...was only looking at rx.)

This may be due to the bug I mentioned above. Multiple RX queues shouldn't be present on the 2.6.18 kernel as I do not believe multi-queue NAPI has been back-ported and it could have negative effects.

The odds of any 2 flows overlapping when you are only using 4 flows is pretty high, especially if the addresses/ports are close in range. You typically need something on the order of about 16 flows over a wide range of port numbers in order to get a good distribution.

Thanks,

Alex

------------------------------------------------------------------------------
Come build with us! The BlackBerry(R) Developer Conference in SF, CA is the only developer event you need to attend this year. Jumpstart your developing skills, take BlackBerry mobile applications to market and stay ahead of the curve. Join us from November 9 - 12, 2009. Register now!
http://p.sf.net/sfu/devconference

#5 lore raw
Quoted 4 lines · Alexander Duyck

The odds of any 2 flows overlapping when you are only using 4 flows is pretty high, especially if the addresses/ports are close in range. You typically need something on the order of about 16 flows over a wide range of port numbers in order to get a good distribution.

Yes, I realize this. However, I was surprised that we were seeing the packet count increasing for only one queue but the interrupt count increasing for more than one.

Also, if we really crank up the traffic levels the box apparently panics. They're working on getting a serial cable hooked up to it to get the debug information, so I don't really have much information on that part just yet.

We're going to try the out-of-tree drivers. Unfortunately it appears that the out-of-tree igb driver doesn't compile for this kernel. I suspect the compat code needs tweaking.

Chris

------------------------------------------------------------------------------
Come build with us! The BlackBerry(R) Developer Conference in SF, CA is the only developer event you need to attend this year. Jumpstart your developing skills, take BlackBerry mobile applications to market and stay ahead of the curve. Join us from November 9 - 12, 2009. Register now!
http://p.sf.net/sfu/devconference

#6 lore raw

I'm have just put together a Nehalem system (1 x Xeon), 2.6.30.8-64.fc11.x86_64, which has quad onboard 82576 and noticed during testing using just a single interface, that the RX queues on the other 3 were receiving interrupts - observed in /proc/interrupts.

Is this normal behaviour?

Regards,

Richard

------------------------------------------------------------------------------
Come build with us! The BlackBerry(R) Developer Conference in SF, CA is the only developer event you need to attend this year. Jumpstart your developing skills, take BlackBerry mobile applications to market and stay ahead of the curve. Join us from November 9 - 12, 2009. Register now!
http://p.sf.net/sfu/devconference

#7 lore raw
Quoted 6 lines · Richard Scobie

I'm have just put together a Nehalem system (1 x Xeon), 2.6.30.8-64.fc11.x86_64, which has quad onboard 82576 and noticed during testing using just a single interface, that the RX queues on the other 3 were receiving interrupts - observed in /proc/interrupts.

Is this normal behaviour?

Hi Richard,

This is normal, since we trigger an interrupt on every queue during our watchdog. So if the interface is up it should be triggering an interrupt on every queue every two seconds.

This has (and will probably continue to be) necessary in order to pick up any straggler packets due to any (extremely rare, but expected to occur) missed interrupts that could happen when traffic is running. It also is extremely useful if you're doing irq affinitization and/or debugging interrupts.

you can bring down the interfaces and they will stop interrupting.

Jesse