results with diff affinity settings using ixgbe

DISCUSSION Paweł Staszewski 2 participants 2 messages Jan 27, 2009

Subsystems (2)

Networking Network drivers

lore

Share & tools

lore

Paweł Staszewski

author

2009-01-27 15:14 UTC #0 lore raw

I make some tests with affinity on my machine with some traffic load. If someone want look:

Traffic in all test was 350Mbit/s TX and the same on RX

RX queues on cpu6 / TX queues on cpu7 / intel e1000e on cpu4

Average:     CPU   %user   %nice    %sys %iowait    %irq   %soft  %steal   %idle    intr/s
Average:     all    0.05    0.00    0.00    0.00    1.89   10.03    0.00   88.03 221511.70
Average:       0    0.40    0.00    0.00    0.00    0.00    0.00    0.00   99.60      0.70
Average:       1    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00      0.80
Average:       2    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00      0.70
Average:       3    0.00    0.00    0.10    0.00    0.00    0.40    0.00   99.50      0.70
Average:       4    0.00    0.00    0.00    0.00    2.06   18.74    0.00   79.20  12870.50
Average:       5    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00      1.30
Average:       6    0.00    0.00    0.00    0.00    2.65   57.49    0.00   39.86  59410.60
Average:       7    0.00    0.00    0.00    0.00   10.50    5.00    0.00   84.50  53977.90

RX queues on cpu6 / TX queues on cpu7 / intel e1000e on cpu1

Average:     CPU   %user   %nice    %sys %iowait    %irq   %soft  %steal   %idle    intr/s
Average:     all    0.08    0.00    0.09    0.00    0.75   14.24    0.00   84.84 215755.20
Average:       0    0.50    0.00    0.00    0.00    0.00    0.00    0.00   99.50      1.30
Average:       1    0.00    0.00    0.53    0.00    2.34   36.91    0.00   60.21   9771.90
Average:       2    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00      1.40
Average:       3    0.00    0.00    0.00    0.00    0.00    0.20    0.00   99.80      1.30
Average:       4    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00      1.30
Average:       5    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00      1.80
Average:       6    0.00    0.00    0.20    0.00    2.65   78.51    0.00   18.64  54691.70
Average:       7    0.00    0.00    0.00    0.00    1.03    0.51    0.00   98.46  54060.60

All RX/TX queues from ixgbe on one cpu7 / intel e1000e on cpu1

Average:     CPU   %user   %nice    %sys %iowait    %irq   %soft  %steal   %idle    intr/s
Average:     all    0.07    0.00    0.10    0.00    1.19    9.44    0.00   89.19 210155.50
Average:       0    0.60    0.00    0.00    0.00    0.00    0.00    0.00   99.40      0.60
Average:       1    0.00    0.00    0.41    0.00    2.87   16.27    0.00   80.45  13045.30
Average:       2    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00      0.70
Average:       3    0.10    0.00    0.30    0.00    0.00    0.20    0.00   99.40      0.60
Average:       4    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00      0.70
Average:       5    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00      1.10
Average:       6    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00      0.60
Average:       7    0.00    0.00    0.10    0.00    6.77   60.00    0.00   33.13 107242.50

All RX/TX queues from ixgbe on one cpu7 / intel e1000e on cpu4

Average:     CPU   %user   %nice    %sys %iowait    %irq   %soft  %steal   %idle    intr/s
Average:     all    0.05    0.00    0.04    0.00    1.24    9.55    0.00   89.12 212186.90
Average:       0    0.40    0.00    0.00    0.00    0.00    0.00    0.00   99.60      2.20
Average:       1    0.10    0.00    0.00    0.00    0.00    0.00    0.00   99.90      2.30
Average:       2    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00      2.20
Average:       3    0.00    0.00    0.00    0.00    0.00    0.30    0.00   99.70      2.20
Average:       4    0.00    0.00    0.00    0.00    1.03   14.68    0.00   84.28  11965.50
Average:       5    0.00    0.00    0.10    0.00    0.00    0.00    0.00   99.90      2.80
Average:       6    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00      2.30
Average:       7    0.00    0.00    0.20    0.00    8.97   62.69    0.00   28.13 110943.50

Affinity set to "ffffffff" / and intel e1000e is on cpu4

Average:     CPU   %user   %nice    %sys %iowait    %irq   %soft  %steal   %idle    intr/s
Average:     all    0.10    0.00    0.14    0.01    1.16   15.07    0.00   83.52 185043.10
Average:       0    0.00    0.00    0.00    0.10    2.90   36.90    0.00   60.10   9008.60
Average:       1    0.00    0.00    0.10    0.00    2.30   28.36    0.00   69.24   9009.10
Average:       2    0.80    0.00    0.40    0.00    0.50   13.76    0.00   84.54   9008.90
Average:       3    0.00    0.00    0.20    0.00    0.20    4.61    0.00   94.98   9008.90
Average:       4    0.00    0.00    0.10    0.00    2.35   19.55    0.00   77.99  14142.10
Average:       5    0.00    0.00    0.10    0.00    0.20    3.85    0.00   95.84   9009.10
Average:       6    0.00    0.00    0.10    0.00    0.40    4.22    0.00   95.28   9008.90
Average:       7    0.00    0.00    0.10    0.00    0.51    8.69    0.00   90.70   9009.00

And if i split more RX/TX queues to different CPU-s i have more load done by:
CPU: Core 2, speed 2999.98 MHz (estimated)
Counted CPU_CLK_UNHALTED events (Clock cycles when not halted) with a unit mask of 0x00 (Unhalted core cycles) count 100000

samples  %        image name               app name                 symbol name
458001   29.5565  vmlinux                  vmlinux                  native_read_tsc

If all interrupts are on one CPU i have the best results
So for me the best is to put ixgbe work on one CPU or make affinity to balance over all cpus

But something is wrong with my intel e1000e
because if i put intel e1000e with ixgbe on the same CPU system have 100% cpu load and all hangs.

Eric Dumazet

2009-01-27 15:53 UTC #1 lore raw

Quoted 131 lines · Paweł Staszewski

I make some tests with affinity on my machine with some traffic load. If someone want look:
Traffic in all test was 350Mbit/s TX and the same on RX
RX queues on cpu6 / TX queues on cpu7 / intel e1000e on cpu4
Average: CPU %user %nice %sys %iowait %irq %soft
%steal %idle intr/s
Average: all 0.05 0.00 0.00 0.00 1.89 10.03
0.00 88.03 221511.70
Average: 0 0.40 0.00 0.00 0.00 0.00 0.00
0.00 99.60 0.70
Average: 1 0.00 0.00 0.00 0.00 0.00 0.00
0.00 100.00 0.80
Average: 2 0.00 0.00 0.00 0.00 0.00 0.00
0.00 100.00 0.70
Average: 3 0.00 0.00 0.10 0.00 0.00 0.40
0.00 99.50 0.70
Average: 4 0.00 0.00 0.00 0.00 2.06 18.74
0.00 79.20 12870.50
Average: 5 0.00 0.00 0.00 0.00 0.00 0.00
0.00 100.00 1.30
Average: 6 0.00 0.00 0.00 0.00 2.65 57.49
0.00 39.86 59410.60
Average: 7 0.00 0.00 0.00 0.00 10.50 5.00
0.00 84.50 53977.90
RX queues on cpu6 / TX queues on cpu7 / intel e1000e on cpu1
Average: CPU %user %nice %sys %iowait %irq %soft
%steal %idle intr/s
Average: all 0.08 0.00 0.09 0.00 0.75 14.24
0.00 84.84 215755.20
Average: 0 0.50 0.00 0.00 0.00 0.00 0.00
0.00 99.50 1.30
Average:       1    0.00    0.00    0.53    0.00    2.34   36.91   
0.00   60.21   9771.90
Average:       2    0.00    0.00    0.00    0.00    0.00    0.00   
0.00 100.00 1.40
Average: 3 0.00 0.00 0.00 0.00 0.00 0.20
0.00 99.80 1.30
Average: 4 0.00 0.00 0.00 0.00 0.00 0.00
0.00 100.00 1.30
Average: 5 0.00 0.00 0.00 0.00 0.00 0.00
0.00 100.00 1.80
Average: 6 0.00 0.00 0.20 0.00 2.65 78.51
0.00 18.64 54691.70
Average: 7 0.00 0.00 0.00 0.00 1.03 0.51
0.00 98.46 54060.60
All RX/TX queues from ixgbe on one cpu7 / intel e1000e on cpu1
Average: CPU %user %nice %sys %iowait %irq %soft
%steal %idle intr/s
Average: all 0.07 0.00 0.10 0.00 1.19 9.44
0.00 89.19 210155.50
Average: 0 0.60 0.00 0.00 0.00 0.00 0.00
0.00 99.40 0.60
Average: 1 0.00 0.00 0.41 0.00 2.87 16.27
0.00 80.45 13045.30
Average: 2 0.00 0.00 0.00 0.00 0.00 0.00
0.00 100.00 0.70
Average: 3 0.10 0.00 0.30 0.00 0.00 0.20
0.00 99.40 0.60
Average: 4 0.00 0.00 0.00 0.00 0.00 0.00
0.00 100.00 0.70
Average: 5 0.00 0.00 0.00 0.00 0.00 0.00
0.00 100.00 1.10
Average: 6 0.00 0.00 0.00 0.00 0.00 0.00
0.00 100.00 0.60
Average: 7 0.00 0.00 0.10 0.00 6.77 60.00
0.00 33.13 107242.50
All RX/TX queues from ixgbe on one cpu7 / intel e1000e on cpu4
Average: CPU %user %nice %sys %iowait %irq %soft
%steal %idle intr/s
Average: all 0.05 0.00 0.04 0.00 1.24 9.55
0.00 89.12 212186.90
Average: 0 0.40 0.00 0.00 0.00 0.00 0.00
0.00 99.60 2.20
Average: 1 0.10 0.00 0.00 0.00 0.00 0.00
0.00 99.90 2.30
Average: 2 0.00 0.00 0.00 0.00 0.00 0.00
0.00 100.00 2.20
Average: 3 0.00 0.00 0.00 0.00 0.00 0.30
0.00 99.70 2.20
Average: 4 0.00 0.00 0.00 0.00 1.03 14.68
0.00 84.28 11965.50
Average: 5 0.00 0.00 0.10 0.00 0.00 0.00
0.00 99.90 2.80
Average: 6 0.00 0.00 0.00 0.00 0.00 0.00
0.00 100.00 2.30
Average: 7 0.00 0.00 0.20 0.00 8.97 62.69
0.00 28.13 110943.50
Affinity set to "ffffffff" / and intel e1000e is on cpu4
Average: CPU %user %nice %sys %iowait %irq %soft
%steal %idle intr/s
Average: all 0.10 0.00 0.14 0.01 1.16 15.07
0.00 83.52 185043.10
Average:       0    0.00    0.00    0.00    0.10    2.90   36.90   
0.00   60.10   9008.60
Average:       1    0.00    0.00    0.10    0.00    2.30   28.36   
0.00   69.24   9009.10
Average:       2    0.80    0.00    0.40    0.00    0.50   13.76   
0.00   84.54   9008.90
Average:       3    0.00    0.00    0.20    0.00    0.20    4.61   
0.00   94.98   9008.90
Average:       4    0.00    0.00    0.10    0.00    2.35   19.55   
0.00   77.99  14142.10
Average:       5    0.00    0.00    0.10    0.00    0.20    3.85   
0.00   95.84   9009.10
Average:       6    0.00    0.00    0.10    0.00    0.40    4.22   
0.00   95.28   9008.90
Average:       7    0.00    0.00    0.10    0.00    0.51    8.69   
0.00   90.70   9009.00
And if i split more RX/TX queues to different CPU-s i have more load done by:
CPU: Core 2, speed 2999.98 MHz (estimated)
Counted CPU_CLK_UNHALTED events (Clock cycles when not halted) with a unit mask of 0x00 (Unhalted core cycles) count 100000
samples % image name app name
symbol name
458001 29.5565 vmlinux vmlinux
native_read_tsc

Quoted 3 lines

If all interrupts are on one CPU i have the best results
So for me the best is to put ixgbe work on one CPU or make affinity to balance over all cpus

I am confused by all these numbers. What is your goal exactly ?

If one CPU is able to do the whole workload, it will be faster too.

All other cpus are idle and wont fight with cache lines.

Quoted 4 lines

But something is wrong with my intel e1000e
because if i put intel e1000e with ixgbe on the same CPU system have 100% cpu load and all hangs.

Try to reproduce the bug and submit it ?