relationship between cpu_affinity and Packet RX Processing

Hi All,

I was reading some stuff on interrupts and irq lines today and I thought I'll expermient with the network rx path. To start with, I've a Virtual Machine running 3.8 linux kernel. My machine has 4 CPU cores, network (eth) interface is driven by pcnet_32 AMD driver and is tied to IRQ line #19. I started some network traffic and I notice that out of those 4 CPUs, only one of them is being used and despite changing the CPU affinity, I still don't see the other cores being used for this network traffic. So based on this behavior(please see the logs below), I've these following queries:
i) Does it mean that this network card do not have multiple Rx Queues ? ii) I think all the modern NICs must be implementing multiple Rx Queues and hence Can someone please point me to the simplest of such implemenation in any of the in-tree drivers ?
iii) I'm just doing a simple 'ping' to google with of big size packets, As I do not have a peer to use packetgen/netperf/iperf utilities.

ref: Comments in double quotes.
<logs>

$ uname -r
3.8.0-rc6-next-20130208

$ cat /proc/cpuinfo |grep processor
processor	: 0
processor	: 1
processor	: 2
processor	: 3
"Total 4 cpu cores"

$ cat /proc/interrupts|egrep 'eth0|CPU'
      	CPU0       CPU1       CPU2       CPU3       
19:     5103         74         33          5   IO-APIC-fasteoi 	eth0
"IRQ Line #19 for the network device"

$ lspci|grep -i ethernet
02:01.0 Ethernet controller: Advanced Micro Devices [AMD] 79c970 [PCnet32 LANCE]
(rev 10)

"AMD NIC"

$ lspci -s 02:01.0 -vvv|grep module
	Kernel modules: pcnet32

$ lsmod|grep pcnet32
pcnet32                40671  0
"driver"

# whoami
root

# cat /proc/irq/19/smp_affinity
03

# cat /proc/irq/19/affinity_hint 
00

"I think smp_affinity is a bit map, therefore for all the four cores to be utilized, all 4 bits should be set to 1, which leads to 15(0xf), hence Chaning the cpu affinity"

# echo 15 > /proc/irq/19/smp_affinity
# cat /proc/irq/19/smp_affinity
15 

"started network traffic here and monitoring it"

# cat /proc/interrupts|grep eth0
19:       5452         78         33          5   IO-APIC-fasteoi   eth0
# cat /proc/interrupts|grep eth0
19:       5488         78         35          5   IO-APIC-fasteoi   eth0
# cat /proc/interrupts|grep eth0
19:       5492         78         35          5   IO-APIC-fasteoi   eth0
# cat /proc/interrupts|grep eth0
19:       5500         78         35          5   IO-APIC-fasteoi   eth0
.................................
.........after some time.........
# cat /proc/interrupts|grep eth0
19:       6035         78         42          5   IO-APIC-fasteoi   eth0

"Most of the packets are still getting routed through CPU0 *Only*." <logs>

-Amit

Hi, Kumar,

1) Regarding rx queues: what does
"ls /sys/class/net/eth0/queues"
show ?

2) can you try setting affinity for working only with CPU3 and see what happens after some traffic is sent by cat /proc/interrupts?

regards,
Rami Rosen
http://ramirose.wix.com/ramirosen

$ ls /sys/class/net/eth0/queues
rx-0  tx-0

# echo 8 > /proc/irq/19/smp_affinity
# cat /proc/irq/19/smp_affinity
08

$ cat /proc/interrupts |grep eth0
19:      20635         78        387       1857   IO-APIC-fasteoi   eth0
$ cat /proc/interrupts |grep eth0
19:      20635         78        387       3041   IO-APIC-fasteoi   eth0

Now only, CPU3 is processing the packets.
So, i think, this driver(pcnet32) is not using multiple Rx Queue.

Hi,

1) You can be sure that it does not use more than one rx queue.

You can find more info about RX queues in
http://media.wix.com/ugd//295986_4ef6dbdf11fd0a7f74f09741b4b5b2ee.pdf

(I wrote it, a shameless plug...)

2) Does the irqbalance service, which is common in some distros, is running ? can you try to disable it and run traffic again and check /proc/interrupts afrer some time ?

3) regardless of 2 (meaning not stopping irqbalance if it was active) can you please try "taskset 02 ping www.google.com" and tell us whether interrupts were incremented on the second CPU ? (please send /proc/interrupts | grep eth0 before and after sending, let's say, 5 pings)?

Rgs,
Rami Rosen
http://ramirose.wix.com/ramirosen

My apologies for responding after a long time. I lost access to the machine on which this VM was running. Now running this VM on my laptop, which seem to have similar hardware configuration. Please find the details, which you had asked for below:(comment in quotes)

amit at ubuntu:~$ cat /proc/cpuinfo |grep processor
processor : 0
processor : 1
"2 CPU"

amit at ubuntu:~$ cat /proc/interrupts |egrep 'CPU|eth0'
CPU0 CPU1
19: 7887 31 IO-APIC-fasteoi eth0

amit at ubuntu:~$ ls -lah /sys/class/net/eth0/device/driver lrwxrwxrwx 1 root root 0 Mar 29 01:13 /sys/class/net/eth0/device/driver -> ../../../../bus/pci/drivers/pcnet32

amit at ubuntu:~$ lsmod|grep -i pcnet32
pcnet32 40671 0
"driver"

amit at ubuntu:~$ ls /sys/class/net/eth0/queues
rx-0 tx-0
"just one rx-queue"

amit at ubuntu:~$ sudo cat /proc/irq/19/smp_affinity 01
"tied to CPU0"

"disabling irqbalance"
amit at ubuntu:~$ status irqbalance
irqbalance start/running, process 1016

amit at ubuntu:~$ sudo stop irqbalance
irqbalance stop/waiting

amit at ubuntu:~$ taskset 02 ping www.google.com

"after sometime"
amit at ubuntu:~$ cat /proc/interrupts |egrep 'CPU|eth0'
CPU0 CPU1
19: 8046 31 IO-APIC-fasteoi eth0

"No increase in interrupt counts on CPU1"

"Now lets use both CPUs"
root at ubuntu:/home/amit# echo "3" >/proc/irq/19/smp_affinity root at ubuntu:/home/amit# cat /proc/irq/19/smp_affinity 03

"As before, start ping and expecting both CPUs to be used" amit at ubuntu:~$ taskset 02 ping www.google.com

root at ubuntu:/home/amit# cat /proc/interrupts |egrep 'CPU|eth0'
CPU0 CPU1
19: 9001 71 IO-APIC-fasteoi eth0

                      ^^^^

and yes, indeed..
So multiple Rx queues seem to have nothing to do with CPU affinity, as in this case I have a driver implementing a single Rx queue and I see that the rx packet processing happens through both of the CPUs.

-Amit

Thank you so much for sharing this.

Hi,
Stay tuned as I intend to put on the web an expanded document, including documentations about kernel internals (besides networking).

Regards,
Rami Rosen
http://ramirose.wix.com/ramirosen

Thanks a lot for this document Rami! I'm starting to study the kernel and it will greatly help me to figure out how to dig into the source code. Looking forward for the expanded version as well!

Best regards,

Paulo

iPhone????

30/03/2013 05:16?Rami Rosen <[email protected]> ??????:

Hi Amit,

I don't know for sure what linux does, but the NICs I've seen with multiple queues tend to select queues by hashing incoming packets based on source IP, sourse port, destination IP, destination port and (if TCP) protocol.

This is done because you generally want all packets for a given connection in the same queue.

For a single ping, always to the same destination, all packets will hash the same, and so wind up in the same queue.

--
Arlie

I think what you are referring to is called RSS(Recieve side scalling), which I think is a different feature. IIRC, windows ndis driver implements this feature.