New: The smp_affinity cannot work correctly when PCI passthrough device using msi/msi-x with KVM
[Bug 50891] The smp_affinity cannot work correctly on guest os when PCI passthrough device using msi/msi-x with KVM
In reply to [email protected]https://bugzilla.kernel.org/show_bug.cgi?id=50891
liyi <[email protected]> changed:
What |Removed |Added
----------------------------------------------------------------------------
Summary|The smp_affinity cannot |The smp_affinity cannot
|work correctly when PCI |work correctly on guest os
|passthrough device using |when PCI passthrough device
|msi/msi-x with KVM |using msi/msi-x with KVMSignature · 4 lines
Configure bugmail: https://bugzilla.kernel.org/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug. You are watching the assignee of the bug.
https://bugzilla.kernel.org/show_bug.cgi?id=50891
Summary: The smp_affinity cannot work correctly when PCI
passthrough device using msi/msi-x with KVM
Product: Virtualization
Version: unspecified
Kernel Version: 3.7-rc6
Platform: All
OS/Version: Linux
Tree: Mainline
Status: NEW
Severity: normal
Priority: P1
Component: kvm
AssignedTo: [email protected]
ReportedBy: [email protected]
CC: [email protected]
Regression: No
1: passthrough a netcard (Brodcom BCM5716S) to the guest os
2: ifup the netcard, the card will use msi-x interrupt default, and close the irqbalance service
3: echo 4 > cat /proc/irq/NETCARDIRQ/smp_affinity, so we assume the vcpu2 handle the irq.
4: we have set <vcpupin vcpu='2' cpuset='1'/> and set the irq kvm:pci-bus to the pcpu1 on the host.
we think this configure will reduce the ipi interrupt when inject interrupt to the guest os. but this irq is not only handle on vcpu2. it is not expect。
Signature · 4 lines
Configure bugmail: https://bugzilla.kernel.org/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug. You are watching the assignee of the bug.
[Bug 50891] The smp_affinity cannot work correctly on guest os when PCI passthrough device using msi/msi-x with KVM
https://bugzilla.kernel.org/show_bug.cgi?id=50891
Alex Williamson <[email protected]> changed:
What |Removed |Added
----------------------------------------------------------------------------
CC| |[email protected]
--- Comment #1 from Alex Williamson <[email protected]> 2012-11-26 19:32:15 --- MSI-X SMP affinity should be working, MSI SMP affinity is not currently implemented. Please clarify whether the device in question is actually making use of MSI or MSI-X. Thanks.
Signature · 4 lines
Configure bugmail: https://bugzilla.kernel.org/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug. You are watching the assignee of the bug.
[Bug 50891] The smp_affinity cannot work correctly on guest os when PCI passthrough device using msi/msi-x with KVM
In reply to [email protected]https://bugzilla.kernel.org/show_bug.cgi?id=50891
--- Comment #2 from liyi <[email protected]> 2012-11-27 00:57:28 --- sorry, i am not clearly it.
1:i am sure the device using the MSI-X, the test is failed.
check the attribute, entry->msi_attrib.is_msix is 1.
pls, the qemu kvm version is 1.2.
also, when using the virtio driver, i find the the nercard using the msi-x, but the test is ok.
and i have test intel 82599 SR-IOV, passthrough the VF to the guest os, the test is failed using the msi-x as the BCM5716S.
Signature · 4 lines
Configure bugmail: https://bugzilla.kernel.org/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug. You are watching the assignee of the bug.
[Bug 50891] The smp_affinity cannot work correctly on guest os when PCI passthrough device using msi/msi-x with KVM
In reply to [email protected]https://bugzilla.kernel.org/show_bug.cgi?id=50891
--- Comment #3 from Alex Williamson <[email protected]> 2012-11-27 18:13:07 --- I tested a BCM5716 on 3.7.0-rc7 with both qemu-kvm-1.2.0 and current qemu.git using pci-assign. MSI-X pinning works exactly as expected. Note that Linux MSI affinity is setup lazily on the next interrupt for a vector, so it's normal that after setting the affinity for a vector that you might see a single interrupt on another CPU before the interrupt is moved. Also note that setting the affinity in the guest only changes the affinity of the virtual interrupt to the guest, the physical interrupt affinity must be separately configured on the host. Perhaps the steps your missing in comment 0 above is to disable irqbalance in the host and set the irq affinity of the kvm interrupts in the host.
Signature · 4 lines
Configure bugmail: https://bugzilla.kernel.org/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug. You are watching the assignee of the bug.
[Bug 50891] The smp_affinity cannot work correctly on guest os when PCI passthrough device using msi/msi-x with KVM
In reply to [email protected]https://bugzilla.kernel.org/show_bug.cgi?id=50891
--- Comment #4 from liyi <[email protected]> 2012-11-29 15:38:10 --- 1: i have disable the irqbalance in the host, and the irq affinity on the host is ok.you can see the 4 step in comment 0.
2: you see in comment 0, we have set irq affinity on the host is pcpu1. and so reduce the ipi interrupt, so we want to set the irq affinity at the vcpu2 on the guest os according <vcpupin vcpu='2' cpuset='1'/>, but the irq affinity on the guest os when using msi-x is failed.
Signature · 4 lines
Configure bugmail: https://bugzilla.kernel.org/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug. You are watching the assignee of the bug.
[Bug 50891] The smp_affinity cannot work correctly on guest os when PCI passthrough device using msi/msi-x with KVM
In reply to [email protected]https://bugzilla.kernel.org/show_bug.cgi?id=50891
--- Comment #5 from Alex Williamson <[email protected]> 2012-11-29 16:10:55 --- (In reply to comment #4)
Quoted 7 lines
1: i have disable the irqbalance in the host, and the irq affinity on the host is ok.you can see the 4 step in comment 0.
2: you see in comment 0, we have set irq affinity on the host is pcpu1. and so reduce the ipi interrupt, so we want to set the irq affinity at the vcpu2 on the guest os according <vcpupin vcpu='2' cpuset='1'/>, but the irq affinity on the guest os when using msi-x is failed.
How does it fail? When I test this, I look at /proc/interrupts on the guest and the interrupt count on the row with the pinned IRQ only increases in the column under the target CPU. Perhaps if you included some logs and exact commands used I could better understand where things are going wrong. Thanks.
Signature · 4 lines
Configure bugmail: https://bugzilla.kernel.org/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug. You are watching the assignee of the bug.
[Bug 50891] The smp_affinity cannot work correctly on guest os when PCI passthrough device using msi/msi-x with KVM
In reply to [email protected]https://bugzilla.kernel.org/show_bug.cgi?id=50891
--- Comment #6 from liyi <[email protected]> 2012-11-29 16:21:16 --- 1:i have set the irq (associate the passthrough device) to the vcpu2 on the guest os.but look at /proc/interrupts on the guest and the interrupt count with the pinned IRQ increases a lot of vcpus except the vcpu0.
2 my qemu kvm version is 1.2 , and have not add the patch
http://www.spinics.net/lists/kvm/msg83109.html from, maybe it will help me. are you sure that your qemu-kvm is from thegit://git.kernel.org/pub/scm/virt/kvm/qemu-kvm.git?
Signature · 4 lines
Configure bugmail: https://bugzilla.kernel.org/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug. You are watching the assignee of the bug.
[Bug 50891] The smp_affinity cannot work correctly on guest os when PCI passthrough device using msi/msi-x with KVM
In reply to [email protected]https://bugzilla.kernel.org/show_bug.cgi?id=50891
--- Comment #7 from Alex Williamson <[email protected]> 2012-11-29 16:33:38 --- (In reply to comment #6)
Quoted 3 lines
1:i have set the irq (associate the passthrough device) to the vcpu2 on the guest os.but look at
/proc/interruptson the guest and the interrupt count with the pinned IRQ increases a lot of vcpus except the vcpu0.
Perhaps double check that irqbalance is not running the in guest, ps aux | grep irqbalance.
Quoted 4 lines
2 my qemu kvm version is 1.2 , and have not add the patch
http://www.spinics.net/lists/kvm/msg83109.html from, maybe it will help me. are you sure that your qemu-kvm is from thegit://git.kernel.org/pub/scm/virt/kvm/qemu-kvm.git?
Those patches are against git://git.qemu.org/qemu.git I believe the BCM5716 will try to use MSI-X interrupts, not MSI. Please provide lspci -vvv of the device in the guest and we can verify what interrupt mode it's working in. What is the guest operating system? Thanks.
Signature · 4 lines
Configure bugmail: https://bugzilla.kernel.org/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug. You are watching the assignee of the bug.
[Bug 50891] The smp_affinity cannot work correctly on guest os when PCI passthrough device using msi/msi-x with KVM
In reply to [email protected]https://bugzilla.kernel.org/show_bug.cgi?id=50891
--- Comment #8 from liyi <[email protected]> 2012-11-29 17:52:47 --- ok,i am sure the irqbalance is not running in the guest.
you are right, BCM5716S using the MSI-X interrupts default.
1:i have set the smp_affinity cannot work correctly using the MSI-X. but the test is ok after insmod the bnx2 with disable_msi=1.
2: lspci -vvv:
00:06.0 Ethernet controller: Broadcom Corporation NetXtreme II BCM5709 Gigabit Ethernet (rev 20)
Subsystem: Dell PowerEdge R710 BCM5709 Gigabit Ethernet
Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr-
Stepping- SERR- FastB2B- DisINTx+
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort-
<MAbort- >SERR- <PERR- INTx-
Latency: 0
Interrupt: pin A routed to IRQ 11
Region 0: Memory at f6000000 (32-bit, non-prefetchable) [size=32M]
Capabilities: [50] Vital Product Data <?>
Capabilities: [ac] Express (v2) Endpoint, MSI 00
DevCap: MaxPayload 512 bytes, PhantFunc 0, Latency L0s <4us, L1
<64us
ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset-
DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+ MaxPayload 128 bytes, MaxReadReq 512 bytes
DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
LnkCap: Port #0, Speed 2.5GT/s, Width x4, ASPM L0s L1, Latency L0
<4us, L1 <4us
ClockPM- Suprise- LLActRep- BwNot-
LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- Retrain- CommClk-
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 2.5GT/s, Width x4, TrErr- Train- SlotClk- DLActive-
BWMgmt- ABWMgmt-
DevCap2: Completion Timeout: Range ABCD, TimeoutDis+
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-
LnkCtl2: Target Link Speed: 2.5GT/s, EnterCompliance- SpeedDis-,
Selectable De-emphasis: -6dB
Transmit Margin: Normal Operating Range, EnterModifiedCompliance-
ComplianceSOS-
Compliance De-emphasis: -6dB
LnkSta2: Current De-emphasis Level: -6dB
Capabilities: [48] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mAPME(D0-,D1-,D2-,D3hot-,D3cold-)
Status: D0 PME-Enable- DSel=0 DScale=0 PME-
Capabilities: [a0] MSI-X: Enable+ Mask- TabSize=9
Vector table: BAR=0 offset=0000c000
PBA: BAR=0 offset=0000e000
Capabilities: [58] Message Signalled Interrupts: Mask- 64bit- Count=1/16
Enable-
Address: 00000000 Data: 0000
Kernel driver in use: bnx2
Kernel modules: bnx2
00:06.1 Ethernet controller: Broadcom Corporation NetXtreme II BCM5709 Gigabit Ethernet (rev 20)
Subsystem: Dell PowerEdge R710 BCM5709 Gigabit Ethernet
Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr-
Stepping- SERR- FastB2B- DisINTx+
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort-
<MAbort- >SERR- <PERR- INTx-
Latency: 0
Interrupt: pin B routed to IRQ 11
Region 0: Memory at f8000000 (32-bit, non-prefetchable) [size=32M]
Capabilities: [50] Vital Product Data <?>
Capabilities: [ac] Express (v2) Endpoint, MSI 00
DevCap: MaxPayload 512 bytes, PhantFunc 0, Latency L0s <4us, L1
<64us
ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset-
DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+ MaxPayload 128 bytes, MaxReadReq 512 bytes
DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
LnkCap: Port #0, Speed 2.5GT/s, Width x4, ASPM L0s L1, Latency L0
<4us, L1 <4us
ClockPM- Suprise- LLActRep- BwNot-
LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- Retrain- CommClk-
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 2.5GT/s, Width x4, TrErr- Train- SlotClk- DLActive-
BWMgmt- ABWMgmt-
DevCap2: Completion Timeout: Range ABCD, TimeoutDis+
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-
LnkCtl2: Target Link Speed: 2.5GT/s, EnterCompliance- SpeedDis-,
Selectable De-emphasis: -6dB
Transmit Margin: Normal Operating Range, EnterModifiedCompliance-
ComplianceSOS-
Compliance De-emphasis: -6dB
LnkSta2: Current De-emphasis Level: -6dB
Capabilities: [48] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mAPME(D0-,D1-,D2-,D3hot-,D3cold-)
Status: D0 PME-Enable- DSel=0 DScale=0 PME-
Capabilities: [a0] MSI-X: Enable+ Mask- TabSize=9
Vector table: BAR=0 offset=0000c000
PBA: BAR=0 offset=0000e000
Capabilities: [58] Message Signalled Interrupts: Mask- 64bit- Count=1/16
Enable-
Address: 00000000 Data: 0000
Kernel driver in use: bnx2
Kernel modules: bnx2
the guest os: 3.7-rc6 (the same as host)
Signature · 4 lines
Configure bugmail: https://bugzilla.kernel.org/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug. You are watching the assignee of the bug.
[Bug 50891] The smp_affinity cannot work correctly on guest os when PCI passthrough device using msi/msi-x with KVM
In reply to [email protected]https://bugzilla.kernel.org/show_bug.cgi?id=50891
--- Comment #9 from Alex Williamson <[email protected]> 2012-11-29 18:08:44 --- (In reply to comment #8)
Quoted 9 lines
ok,i am sure the irqbalance is not running in the guest. you are right, BCM5716S using the MSI-X interrupts default.
1:i have set the
smp_affinitycannot work correctly using the MSI-X. but the test is ok after insmod the bnx2 with disable_msi=1.2: lspci -vvv:
00:06.0 Ethernet controller: Broadcom Corporation NetXtreme II BCM5709 Gigabit Ethernet (rev 20)
...
Quoted 2 lines
00:06.1 Ethernet controller: Broadcom Corporation NetXtreme II BCM5709 Gigabit Ethernet (rev 20)
Where's the BCM5716?
Signature · 4 lines
Configure bugmail: https://bugzilla.kernel.org/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug. You are watching the assignee of the bug.
[Bug 50891] The smp_affinity cannot work correctly on guest os when PCI passthrough device using msi/msi-x with KVM
In reply to [email protected]https://bugzilla.kernel.org/show_bug.cgi?id=50891
--- Comment #10 from liyi <[email protected]> 2012-11-29 18:18:21 --- sorry, i find the test on BCM5709 is ok.
now, i cannot access my physical machine (with BCM5709S,or intel 82599), i will give the debug info asap.
Thanks
Signature · 4 lines
Configure bugmail: https://bugzilla.kernel.org/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug. You are watching the assignee of the bug.
[Bug 50891] The smp_affinity cannot work correctly on guest os when PCI passthrough device using msi/msi-x with KVM
In reply to [email protected]https://bugzilla.kernel.org/show_bug.cgi?id=50891
--- Comment #11 from liyi <[email protected]> 2012-11-29 18:20:59 --- please omit the BCM5709S.
.... i have BCM5709 BCM5716S intel82599 environment, but now , cannot access the BCM5716S and intel82599.
now, the test on BCM5709 is ok, but the test on BCM5716S intel82599 before is failed.
Signature · 4 lines
Configure bugmail: https://bugzilla.kernel.org/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug. You are watching the assignee of the bug.
[Bug 50891] The smp_affinity cannot work correctly on guest os when PCI passthrough device using msi/msi-x with KVM
In reply to [email protected]https://bugzilla.kernel.org/show_bug.cgi?id=50891
--- Comment #12 from Alex Williamson <[email protected]> 2012-11-29 18:55:46 --- (In reply to comment #11)
Quoted 7 lines
please omit the BCM5709S.
.... i have BCM5709 BCM5716S intel82599 environment, but now , cannot access the BCM5716S and intel82599.
now, the test on BCM5709 is ok, but the test on BCM5716S intel82599 before is failed.
Are you saying MSI-X affinity works as expected on the BCM5709?
Signature · 4 lines
Configure bugmail: https://bugzilla.kernel.org/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug. You are watching the assignee of the bug.
[Bug 50891] The smp_affinity cannot work correctly on guest os when PCI passthrough device using msi/msi-x with KVM
In reply to [email protected]https://bugzilla.kernel.org/show_bug.cgi?id=50891
--- Comment #13 from liyi <[email protected]> 2012-11-29 19:12:47 --- yes.
Signature · 4 lines
Configure bugmail: https://bugzilla.kernel.org/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug. You are watching the assignee of the bug.
[Bug 50891] The smp_affinity cannot work correctly on guest os when PCI passthrough device using msi/msi-x with KVM
In reply to [email protected]https://bugzilla.kernel.org/show_bug.cgi?id=50891
liyi <[email protected]> changed:
What |Removed |Added
----------------------------------------------------------------------------
Status|NEW |RESOLVED
Resolution| |INVALID--- Comment #14 from liyi <[email protected]> 2012-12-03 13:18:21 --- sorry to disturb all of you. this issue is fixed at the latest qemu-kvm version
Signature · 4 lines
Configure bugmail: https://bugzilla.kernel.org/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug. You are watching the assignee of the bug.