Setting cpu affinity for MSI-X on i.mx6 possible?

DISCUSSION Heiner Kallweit 3 participants 7 messages Oct 26, 2014 – Oct 28, 2014 PCI

Share & tools

Heiner Kallweit

author

2014-10-26 18:05 UTC #0 lore raw

I use a mini pc with i.mx6 dual and Intel igb based nic (I211) connected via PCIE.
igb driver uses MSI-X and creates two rx/tx queue pairs. However the IRQs of all queues are handled by the same CPU. Changing affinity is not possible. Obvious direct reason is that irq_chip PCI-MSI doesn't implement the callback for setting cpu affinity.
I'd appreciate a hint whether this is a general limitation of the Cortex-A9 / GICv1 / Synopsys Designware architecture or whether just a proper implementation is missing yet.

Rgds, Heiner

Lucas Stach

2014-10-27 08:52 UTC #1 lore raw

Hello Heiner,

Am Sonntag, den 26.10.2014, 19:05 +0100 schrieb Heiner Kallweit:

Quoted 8 lines

I use a mini pc with i.mx6 dual and Intel igb based nic (I211) connected via PCIE.
igb driver uses MSI-X and creates two rx/tx queue pairs. However the IRQs of all queues are handled by the same CPU. Changing affinity is not possible. Obvious direct reason is that irq_chip PCI-MSI doesn't implement the callback for setting cpu affinity.
I'd appreciate a hint whether this is a general limitation of the Cortex-A9 / GICv1 / Synopsys Designware architecture or whether just a proper implementation is missing yet.
Rgds, Heiner

The MSI controller on imx6 doesn't support MSI-X so igb falls back to regular MSI interrupts. Also all MSI interrupts are aggregated into a single GIC interrupt line. This means the best we could do is switch all PCIe MSI interrupts to some other CPU. Configuring the CPU affinity on a per interrupt basis is impossible on this hardware.

Regards,
Lucas

Signature · 2 lines

Pengutronix e.K.             | Lucas Stach                 |
Industrial Linux Solutions   | http://www.pengutronix.de/  |

Heiner Kallweit

author

2014-10-27 19:16 UTC #2 lore raw

Thanks, Lucas.

It's interesting however that the igb driver claims it's using MSI-X. Maybe the DW PCIE drivers offers MSI-X on the bus and just internally falls back to MSI?

Rgds, Heiner

[    4.165850] igb: Intel(R) Gigabit Ethernet Network Driver - version 5.2.15-k
[    4.172930] igb: Copyright (c) 2007-2014 Intel Corporation.
[    4.178638] igb 0000:01:00.0: enabling device (0140 -> 0142)
[    4.235789] igb 0000:01:00.0: added PHC on eth0
[    4.235793] igb 0000:01:00.0: Intel(R) Gigabit Ethernet Network Connection
[    4.235799] igb 0000:01:00.0: eth0: (PCIe:2.5Gb/s:Width x1) 00:01:c0:14:b1:58
[    4.235804] igb 0000:01:00.0: eth0: PBA No: FFFFFF-0FF
[    4.235810] igb 0000:01:00.0: Using MSI-X interrupts. 2 rx queue(s), 2 tx queue(s)

Am 27.10.2014 um 09:52 schrieb Lucas Stach:

Quoted 21 lines

Hello Heiner,
Am Sonntag, den 26.10.2014, 19:05 +0100 schrieb Heiner Kallweit:
Quoted 8 lines
I use a mini pc with i.mx6 dual and Intel igb based nic (I211) connected via PCIE.
igb driver uses MSI-X and creates two rx/tx queue pairs. However the IRQs of all queues are handled by the same CPU. Changing affinity is not possible. Obvious direct reason is that irq_chip PCI-MSI doesn't implement the callback for setting cpu affinity.
I'd appreciate a hint whether this is a general limitation of the Cortex-A9 / GICv1 / Synopsys Designware architecture or whether just a proper implementation is missing yet.
Rgds, Heiner
The MSI controller on imx6 doesn't support MSI-X so igb falls back to regular MSI interrupts. Also all MSI interrupts are aggregated into a single GIC interrupt line. This means the best we could do is switch all PCIe MSI interrupts to some other CPU. Configuring the CPU affinity on a per interrupt basis is impossible on this hardware.
Regards,
Lucas

Bjorn Helgaas

maintainer

2014-10-27 19:31 UTC #3 lore raw

Quoted 7 lines · Heiner Kallweit

Thanks, Lucas.
It's interesting however that the igb driver claims it's using MSI-X. Maybe the DW PCIE drivers offers MSI-X on the bus and just internally falls back to MSI?
Rgds, Heiner

Quoted 8 lines

[    4.165850] igb: Intel(R) Gigabit Ethernet Network Driver - version 5.2.15-k
[    4.172930] igb: Copyright (c) 2007-2014 Intel Corporation.
[    4.178638] igb 0000:01:00.0: enabling device (0140 -> 0142)
[    4.235789] igb 0000:01:00.0: added PHC on eth0
[    4.235793] igb 0000:01:00.0: Intel(R) Gigabit Ethernet Network Connection
[    4.235799] igb 0000:01:00.0: eth0: (PCIe:2.5Gb/s:Width x1) 00:01:c0:14:b1:58
[    4.235804] igb 0000:01:00.0: eth0: PBA No: FFFFFF-0FF
[    4.235810] igb 0000:01:00.0: Using MSI-X interrupts. 2 rx queue(s), 2 tx queue(s)

What does "lspci -vvs01:00.0" say? It *looks* like igb pays attention to whether enabling MSI-X worked, but that path in the driver is not completely trivial, and I got tired of tracing the code to verify it.

Bjorn

Heiner Kallweit

author

2014-10-27 19:38 UTC #4 lore raw

Am 27.10.2014 um 20:31 schrieb Bjorn Helgaas:

Quoted 17 lines · Heiner Kallweit

Quoted 7 lines

Thanks, Lucas.
It's interesting however that the igb driver claims it's using MSI-X. Maybe the DW PCIE drivers offers MSI-X on the bus and just internally falls back to MSI?
Rgds, Heiner

Quoted 8 lines

[    4.165850] igb: Intel(R) Gigabit Ethernet Network Driver - version 5.2.15-k
[    4.172930] igb: Copyright (c) 2007-2014 Intel Corporation.
[    4.178638] igb 0000:01:00.0: enabling device (0140 -> 0142)
[    4.235789] igb 0000:01:00.0: added PHC on eth0
[    4.235793] igb 0000:01:00.0: Intel(R) Gigabit Ethernet Network Connection
[    4.235799] igb 0000:01:00.0: eth0: (PCIe:2.5Gb/s:Width x1) 00:01:c0:14:b1:58
[    4.235804] igb 0000:01:00.0: eth0: PBA No: FFFFFF-0FF
[    4.235810] igb 0000:01:00.0: Using MSI-X interrupts. 2 rx queue(s), 2 tx queue(s)

Quoted 6 lines

What does "lspci -vvs01:00.0" say? It *looks* like igb pays attention to whether enabling MSI-X worked, but that path in the driver is not completely trivial, and I got tired of tracing the code to verify it.
Bjorn

Thanks, Bjorn. Here comes the full output of the lspci command.

[root at utilite ~]# lspci -vvs01:00.0
01:00.0 Ethernet controller: Intel Corporation I211 Gigabit Network Connection (rev 03)
Subsystem: Intel Corporation Device 0000
Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisINTx+
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0, Cache Line Size: 64 bytes
Interrupt: pin A routed to IRQ 155
Region 0: Memory at 01100000 (32-bit, non-prefetchable) [size=128K]
Region 2: I/O ports at 1000 [disabled] [size=32]
Region 3: Memory@01120000 (32-bit, non-prefetchable) [size=16K]
Capabilities: [40] Power Management version 3
Flags: PMEClk- DSI+ D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=1 PME-
Capabilities: [50] MSI: Enable- Count=1/1 Maskable+ 64bit+
Address: 0000000000000000 Data: 0000
Masking: 00000000 Pending: 00000000
Capabilities: [70] MSI-X: Enable+ Count=5 Masked-
Vector table: BAR=3 offset=00000000
PBA: BAR=3 offset=00002000
Capabilities: [a0] Express (v2) Endpoint, MSI 00
DevCap: MaxPayload 512 bytes, PhantFunc 0, Latency L0s <512ns, L1 <64us
ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset+
DevCtl: Report errors: Correctable+ Non-Fatal+ Fatal+ Unsupported+
RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+ FLReset-
MaxPayload 128 bytes, MaxReadReq 512 bytes
DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr+ TransPend-
LnkCap: Port #4, Speed 2.5GT/s, Width x1, ASPM L0s L1, Exit Latency L0s <2us, L1 <16us
ClockPM- Surprise- LLActRep- BwNot-
LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk+
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 2.5GT/s, Width x1, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
DevCap2: Completion Timeout: Range ABCD, TimeoutDis+, LTR-, OBFF Not Supported
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
LnkCtl2: Target Link Speed: 2.5GT/s, EnterCompliance- SpeedDis-
Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS- Compliance De-emphasis: -6dB
LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1-
EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
Capabilities: [100 v2] Advanced Error Reporting
UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
AERCap: First Error Pointer: 00, GenCap+ CGenEn- ChkCap+ ChkEn-
Capabilities: [140 v1] Device Serial Number 00-a0-c9-ff-ff-00-00-00
Capabilities: [1a0 v1] Transaction Processing Hints
Device specific mode supported
Steering table in TPH capability structure
Kernel driver in use: igb
Kernel modules: igb

Lucas Stach

2014-10-28 12:10 UTC #5 lore raw

In reply to Heiner Kallweit

Am Montag, den 27.10.2014, 20:16 +0100 schrieb Heiner Kallweit:

Quoted 5 lines

Thanks, Lucas.
It's interesting however that the igb driver claims it's using MSI-X. Maybe the DW PCIE drivers offers MSI-X on the bus and just internally falls back to MSI?

Are you running a upstream or downstream vendor kernel? On my upstream kernel igb properly falls back to MSI.

Regards,
Lucas

Quoted 39 lines

Rgds, Heiner
[    4.165850] igb: Intel(R) Gigabit Ethernet Network Driver - version 5.2.15-k
[    4.172930] igb: Copyright (c) 2007-2014 Intel Corporation.
[    4.178638] igb 0000:01:00.0: enabling device (0140 -> 0142)
[    4.235789] igb 0000:01:00.0: added PHC on eth0
[    4.235793] igb 0000:01:00.0: Intel(R) Gigabit Ethernet Network Connection
[    4.235799] igb 0000:01:00.0: eth0: (PCIe:2.5Gb/s:Width x1) 00:01:c0:14:b1:58
[    4.235804] igb 0000:01:00.0: eth0: PBA No: FFFFFF-0FF
[    4.235810] igb 0000:01:00.0: Using MSI-X interrupts. 2 rx queue(s), 2 tx queue(s)
Am 27.10.2014 um 09:52 schrieb Lucas Stach:
Quoted 12 lines
Hello Heiner,
Am Sonntag, den 26.10.2014, 19:05 +0100 schrieb Heiner Kallweit:
Quoted 8 lines
I use a mini pc with i.mx6 dual and Intel igb based nic (I211) connected via PCIE.
igb driver uses MSI-X and creates two rx/tx queue pairs. However the IRQs of all queues are handled by the same CPU. Changing affinity is not possible. Obvious direct reason is that irq_chip PCI-MSI doesn't implement the callback for setting cpu affinity.
I'd appreciate a hint whether this is a general limitation of the Cortex-A9 / GICv1 / Synopsys Designware architecture or whether just a proper implementation is missing yet.
Rgds, Heiner
Quoted 9 lines
The MSI controller on imx6 doesn't support MSI-X so igb falls back to regular MSI interrupts. Also all MSI interrupts are aggregated into a single GIC interrupt line. This means the best we could do is switch all PCIe MSI interrupts to some other CPU. Configuring the CPU affinity on a per interrupt basis is impossible on this hardware.
Regards,
Lucas
--
To unsubscribe from this list: send the line "unsubscribe linux-pci" in the body of a message to majordomo at vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

Signature · 2 lines

Pengutronix e.K.             | Lucas Stach                 |
Industrial Linux Solutions   | http://www.pengutronix.de/  |

Heiner Kallweit

author

2014-10-28 18:20 UTC #6 lore raw

I'm using the latest vanilla linux-next kernel.

Rgds, Heiner

Am 28.10.2014 um 13:10 schrieb Lucas Stach:

Quoted 52 lines

Am Montag, den 27.10.2014, 20:16 +0100 schrieb Heiner Kallweit:
Quoted 5 lines
Thanks, Lucas.
It's interesting however that the igb driver claims it's using MSI-X. Maybe the DW PCIE drivers offers MSI-X on the bus and just internally falls back to MSI?
Are you running a upstream or downstream vendor kernel? On my upstream kernel igb properly falls back to MSI.
Regards,
Lucas
Quoted 39 lines
Rgds, Heiner
[    4.165850] igb: Intel(R) Gigabit Ethernet Network Driver - version 5.2.15-k
[    4.172930] igb: Copyright (c) 2007-2014 Intel Corporation.
[    4.178638] igb 0000:01:00.0: enabling device (0140 -> 0142)
[    4.235789] igb 0000:01:00.0: added PHC on eth0
[    4.235793] igb 0000:01:00.0: Intel(R) Gigabit Ethernet Network Connection
[    4.235799] igb 0000:01:00.0: eth0: (PCIe:2.5Gb/s:Width x1) 00:01:c0:14:b1:58
[    4.235804] igb 0000:01:00.0: eth0: PBA No: FFFFFF-0FF
[    4.235810] igb 0000:01:00.0: Using MSI-X interrupts. 2 rx queue(s), 2 tx queue(s)
Am 27.10.2014 um 09:52 schrieb Lucas Stach:
Quoted 12 lines
Hello Heiner,
Am Sonntag, den 26.10.2014, 19:05 +0100 schrieb Heiner Kallweit:
Quoted 8 lines
I use a mini pc with i.mx6 dual and Intel igb based nic (I211) connected via PCIE.
igb driver uses MSI-X and creates two rx/tx queue pairs. However the IRQs of all queues are handled by the same CPU. Changing affinity is not possible. Obvious direct reason is that irq_chip PCI-MSI doesn't implement the callback for setting cpu affinity.
I'd appreciate a hint whether this is a general limitation of the Cortex-A9 / GICv1 / Synopsys Designware architecture or whether just a proper implementation is missing yet.
Rgds, Heiner
Quoted 9 lines
The MSI controller on imx6 doesn't support MSI-X so igb falls back to regular MSI interrupts. Also all MSI interrupts are aggregated into a single GIC interrupt line. This means the best we could do is switch all PCIe MSI interrupts to some other CPU. Configuring the CPU affinity on a per interrupt basis is impossible on this hardware.
Regards,
Lucas
--
To unsubscribe from this list: send the line "unsubscribe linux-pci" in the body of a message to majordomo at vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html