PCI passthrough problem
If this isn't the right place to ask, any pointers to the correct place are appreciated...
I'm trying to see if I can get PCI passthrough working for a video capture card (Hauppauge Colossus 1x PCIe) under a Windows XP guest (32 -bit). Things appear to be somewhat working (Windows is seeing the device, the drivers successfully installed, and device manager indicates everything is working) however when I fire up the capture application, it is not able to find the device despite Windows recognizing it (no errors, it just doesn't 'see' any installed capture devices). There is also a secondary capture/viewer application that won't even install due to not being able to find a capture card. Since that wasn't the behavior when running it natively under Windows, I'm assuming that the issue is related to PCI passthrough but it's difficult to be certain since I'm not seeing any errors beyond the capture applications not being able to find the device.
Some details on my setup: i7-2600 running on a Q77 motherboard with VT -d enabled in bios. I'm running Debian Linux (testing) with qemu-kvm 1:2.4+dfsg-3 and am passing intel_iommu=on as a kernel parameter on boot. These are the main details I can think of to share, but if there is any additional info that would be useful, please let me know and I'll be happy to provide it.
Having read through a few different posts around the 'net on how to do PCI passthrough (pretty much everything I've found was discussing GPUs, and almost always on a different distro), the only thing that jumps out as a potential problem is that the card in question does not appear to support FLR. However, I'm not clear if that's an absolute requirement for PCI passthrough or something that is specific to GPU support?
Beyond that, I'm at a loss as to what the issue could be...
Thanks,
Phil
Quoted 17 lines · Phil (list)
If this isn't the right place to ask, any pointers to the correct place are appreciated...
I'm trying to see if I can get PCI passthrough working for a video capture card (Hauppauge Colossus 1x PCIe) under a Windows XP guest (32 -bit). Things appear to be somewhat working (Windows is seeing the device, the drivers successfully installed, and device manager indicates everything is working) however when I fire up the capture application, it is not able to find the device despite Windows recognizing it (no errors, it just doesn't 'see' any installed capture devices). There is also a secondary capture/viewer application that won't even install due to not being able to find a capture card. Since that wasn't the behavior when running it natively under Windows, I'm assuming that the issue is related to PCI passthrough but it's difficult to be certain since I'm not seeing any errors beyond the capture applications not being able to find the device.
I think you need to find out if the problem follows the program,
the card, or the passthrough thingie. For instance, is there any other program you can run to see if it sees the card? If you can't think of anything, you could run a, say, ubuntu/fedora livecd (start you vm client and tell it to boot from iso) and see if it can see and use the card.
Quoted 21 lines
Some details on my setup: i7-2600 running on a Q77 motherboard with VT -d enabled in bios. I'm running Debian Linux (testing) with qemu-kvm 1:2.4+dfsg-3 and am passing intel_iommu=on as a kernel parameter on boot. These are the main details I can think of to share, but if there is any additional info that would be useful, please let me know and I'll be happy to provide it.
Having read through a few different posts around the 'net on how to do PCI passthrough (pretty much everything I've found was discussing GPUs, and almost always on a different distro), the only thing that jumps out as a potential problem is that the card in question does not appear to support FLR. However, I'm not clear if that's an absolute requirement for PCI passthrough or something that is specific to GPU support?
Beyond that, I'm at a loss as to what the issue could be...Thanks,
Phil
--
To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 25 lines · Mauricio Tavares
Quoted 14 lines · Phil (list)
If this isn't the right place to ask, any pointers to the correct place
are appreciated...I'm trying to see if I can get PCI passthrough working for a video capture card (Hauppauge Colossus 1x PCIe) under a Windows XP guest (32
-bit). Things appear to be somewhat working (Windows is seeing the device, the drivers successfully installed, and device manager indicates everything is working) however when I fire up the capture application, it is not able to find the device despite Windows recognizing it (no errors, it just doesn't 'see' any installed capture
devices). There is also a secondary capture/viewer applicationQuoted 9 lines
that
won't even install due to not being able to find a capture card.
Since
that wasn't the behavior when running it natively under Windows, I'm
assuming that the issue is related to PCI passthrough but it's difficult to be certain since I'm not seeing any errors beyond the capture applications not being able to find the device.
Quoted 10 lines
I think you need to find out if the problem follows the
program,
the card, or the passthrough thingie. For instance, is there any other
program you can run to see if it sees the card? If you can't think of anything, you could run a, say, ubuntu/fedora livecd (start you vm client and tell it to boot from iso) and see if it can see and use the
card.
I only have the two capture apps that came with the card as I don't really use Windows for much other than this card anymore.
To try to verify that everything is fine from a hardware / Windows driver standpoint: I took a spare drive and performed a bare metal Win XP install, installed the drivers, and then the capture software (i.e. the same sequence and software versions as I used in the VM) and everything works properly (i.e. both capture applications were able to
detect and use the capture card as expected). Other than using a
different hard drive, all other system hardware was identical. So that would seem to rule out everything from the hardware through to the Windows applications and leave it back in the realm of kvm/PCI passthrough.
Unfortunately, no Linux drivers exist for this card (i.e. the reason I'm attempting to use it under Windows in a VM) so any other Linux distro would have about the same level of support in that it would recognize that the PCI card exists but then not be able to do anything with it. If you're thinking that there is a problem with version of kvm in Debian, I would be open to trying another distro if that would help troubleshoot it. I'm also reasonably comfortable navigating around kvm, it's the PCI passthrough functionality that is new to me.
Quoted 29 lines
Quoted 29 lines
Some details on my setup: i7-2600 running on a Q77 motherboard with VT
(quoted diff) +0
-2-d enabled in bios. I'm running Debian Linux (testing) with qemu -kvm1:2.4+dfsg-3 and am passing intel_iommu=on as a kernel parameter on boot. These are the main details I can think of to share, but if there
is any additional info that would be useful, please let me know and I'll be happy to provide it.Having read through a few different posts around the 'net on how to do
PCI passthrough (pretty much everything I've found was discussing GPUs,
and almost always on a different distro), the only thing that jumps out
as a potential problem is that the card in question does not appear to
support FLR. However, I'm not clear if that's an absolute requirement
for PCI passthrough or something that is specific to GPU support?
Beyond that, I'm at a loss as to what the issue could be...Thanks,
Phil
--
To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 51 lines · Phil (list)
Quoted 25 lines · Mauricio Tavares
Quoted 14 lines · Phil (list)
If this isn't the right place to ask, any pointers to the correct place
are appreciated...I'm trying to see if I can get PCI passthrough working for a video capture card (Hauppauge Colossus 1x PCIe) under a Windows XP guest (32
-bit). Things appear to be somewhat working (Windows is seeing the device, the drivers successfully installed, and device manager indicates everything is working) however when I fire up the capture application, it is not able to find the device despite Windows recognizing it (no errors, it just doesn't 'see' any installed capture
devices). There is also a secondary capture/viewer applicationQuoted 9 lines
that
won't even install due to not being able to find a capture card.
Since
that wasn't the behavior when running it natively under Windows, I'm
assuming that the issue is related to PCI passthrough but it's difficult to be certain since I'm not seeing any errors beyond the capture applications not being able to find the device.Quoted 10 lines
I think you need to find out if the problem follows the
program,
the card, or the passthrough thingie. For instance, is there any other
program you can run to see if it sees the card? If you can't think of anything, you could run a, say, ubuntu/fedora livecd (start you vm client and tell it to boot from iso) and see if it can see and use the
card.I only have the two capture apps that came with the card as I don't really use Windows for much other than this card anymore.
To try to verify that everything is fine from a hardware / Windows driver standpoint: I took a spare drive and performed a bare metal Win XP install, installed the drivers, and then the capture software (i.e. the same sequence and software versions as I used in the VM) and everything works properly (i.e. both capture applications were able to
detect and use the capture card as expected). Other than using a
different hard drive, all other system hardware was identical. So that would seem to rule out everything from the hardware through to the Windows applications and leave it back in the realm of kvm/PCI passthrough.
Quoted 8 lines
Unfortunately, no Linux drivers exist for this card (i.e. the reason I'm attempting to use it under Windows in a VM) so any other Linux distro would have about the same level of support in that it would recognize that the PCI card exists but then not be able to do anything with it. If you're thinking that there is a problem with version of kvm in Debian, I would be open to trying another distro if that would help troubleshoot it. I'm also reasonably comfortable navigating around kvm, it's the PCI passthrough functionality that is new to me.
Are you using vfio to do the device assignment or legacy KVM device assignment. If the latter, try the former. Since you're using a 32-bit Windows guest, what CPU model are you exposing to the VM? Windows can be rather particular about enabling MSI for devices if the processor model seen by the VM is too old (does the device support MSI?). '-cpu host' might help or "host-passthrough" if using libvirt. You can look in /proc/interrupts on the host and see if you're getting interrupts (non-zero count on at least one of the CPUs for the interrupt associated with the device). If instead the device is using INTx interrupts, interrupt masking might be broken. You can try using the nointxmask=1 module option to vfio-pci, for force masking at the APIC rather than the device, but be forewarned that you'll need to make the interrupt for the device exclusive, either by locating it in a slot where it won't share interrupts or unloading drivers from devices sharing the interrupt line.
There's always the chance that the device is simply not compatible with PCI device assignment. We do rely on some degree of good behavior on the part of the device. Some environments also expect to find the device behind a PCIe root port, which is not the topology we expose on the default 440fx VM chipset. It's possible that such devices might work on the Q35 chipset or by placing the device behind a pci-bridge to fool the software. It's really hard to tell what might be wrong, especially since the driver appears to work and only the application fails, and it's all proprietary code within the black box of a VM. Thanks,
Alex
Quoted 81 lines · Alex Williamson
Quoted 66 lines · Phil (list)
Quoted 33 lines · Mauricio Tavares
Quoted 22 lines · Phil (list)
If this isn't the right place to ask, any pointers to the correct
place
are appreciated...I'm trying to see if I can get PCI passthrough working for a video
capture card (Hauppauge Colossus 1x PCIe) under a Windows XP guest
(32
-bit). Things appear to be somewhat working (Windows is seeing the
device, the drivers successfully installed, and device manager indicates everything is working) however when I fire up the capture
application, it is not able to find the device despite Windows recognizing it (no errors, it just doesn't 'see' any installed capture
devices). There is also a secondary capture/viewer application that
won't even install due to not being able to find a capture card.Quoted 9 lines
Since
that wasn't the behavior when running it natively under Windows,
I'm
assuming that the issue is related to PCI passthrough but it's difficult to be certain since I'm not seeing any errors beyond the
capture applications not being able to find the device.Quoted 13 lines
I think you need to find out if the problem follows the
program,
the card, or the passthrough thingie. For instance, is there any other
program you can run to see if it sees the card? If you can't think of
anything, you could run a, say, ubuntu/fedora livecd (start you vm
client and tell it to boot from iso) and see if it can see and use
the
card.I only have the two capture apps that came with the card as I don't really use Windows for much other than this card anymore.
To try to verify that everything is fine from a hardware / Windows driver standpoint: I took a spare drive and performed a bare metal Win
XP install, installed the drivers, and then the capture software (i.e.
the same sequence and software versions as I used in the VM) and everything works properly (i.e. both capture applications were able to
detect and use the capture card as expected). Other than using a
different hard drive, all other system hardware was identical. So that
would seem to rule out everything from the hardware through to the Windows applications and leave it back in the realm of kvm/PCI passthrough.Quoted 13 lines
Unfortunately, no Linux drivers exist for this card (i.e. the reason
I'm attempting to use it under Windows in a VM) so any other Linux distro would have about the same level of support in that it would recognize that the PCI card exists but then not be able to do anything
with it. If you're thinking that there is a problem with version of
kvm in Debian, I would be open to trying another distro if that would
help troubleshoot it. I'm also reasonably comfortable navigating around kvm, it's the PCI passthrough functionality that is new to me.
Quoted 2 lines
Are you using vfio to do the device assignment or legacy KVM device assignment. If the latter, try the former.
vfio
Quoted 7 lines
Since you're using a 32-bit
Windows guest, what CPU model are you exposing to the VM? Windows can
be rather particular about enabling MSI for devices if the processor model seen by the VM is too old (does the device support MSI?). ' -cpu
host' might help or "host-passthrough" if using libvirt.
I am passing through the host CPU (Sandy Bridge... which is about a decade newer than Win XP though.) If I'm reading the lspci output correctly, the card supports MSI but it is not enabled (which seems to make sense as your comment indicates that this would be something that Windows, not Linux, would enable?):
02:00.0 Multimedia controller: ViXS Systems, Inc. Device 3000
Subsystem: Hauppauge computer works Inc. Device d180
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop-
ParErr- Stepping- SERR+ FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort-
<TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0, Cache Line Size: 32 bytes Interrupt: pin A routed to IRQ 16 Region 0: Memory at f2100000 (64-bit, prefetchable) [size=1M] Region 2: Memory at f7100000 (32-bit, non-prefetchable)
[size=64K]
Capabilities: [40] Power Management version 3
Flags: PMEClk- DSI- D1+ D2- AuxCurrent=375mAPME(D0+,D1+,D2-,D3hot+,D3cold-)
Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
Capabilities: [50] MSI: Enable- Count=1/1 Maskable- 64bit+
Address: 0000000000000000 Data: 0000
Capabilities: [70] Express (v2) Endpoint, MSI 00
DevCap: MaxPayload 128 bytes, PhantFunc 0,
Latency L0s <256ns, L1 <16us
ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset-
DevCtl: Report errors: Correctable- Non-Fatal-
Fatal- Unsupported-
RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop- MaxPayload 128 bytes, MaxReadReq 512 bytes
DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq-
AuxPwr- TransPend-
LnkCap: Port #0, Speed 2.5GT/s, Width x1, ASPM
L0s L1, Exit Latency L0s <256ns, L1 <16us
ClockPM+ Surprise- LLActRep- BwNot- ASPMOptComp
-
LnkCtl: ASPM Disabled; RCB 64 bytes Disabled-
CommClk+
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 2.5GT/s, Width x1, TrErr- Train-
SlotClk+ DLActive- BWMgmt- ABWMgmt-
DevCap2: Completion Timeout: Not Supported,
TimeoutDis+, LTR-, OBFF Not Supported
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-,
LTR-, OBFF Disabled
LnkCtl2: Target Link Speed: 2.5GT/s, EnterCompliance-
SpeedDis-
Transmit Margin: Normal Operating Range,
EnterModifiedCompliance- ComplianceSOS-
Compliance De-emphasis: -6dB
LnkSta2: Current De-emphasis Level: -6dB,
EqualizationComplete-, EqualizationPhase1-
EqualizationPhase2-, EqualizationPhase3-,
LinkEqualizationRequest-
Capabilities: [100 v1] Advanced Error Reporting
UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt-
UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt-
UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt-
UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout
- NonFatalErr-
CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout
- NonFatalErr+
AERCap: First Error Pointer: 00, GenCap+ CGenEn-
ChkCap+ ChkEn-
Capabilities: [140 v1] Virtual Channel Caps: LPEVC=0 RefClk=100ns PATEntryBits=1 Arb: Fixed- WRR32- WRR64- WRR128- Ctrl: ArbSelect=Fixed Status: InProgress- VC0: Caps: PATOffset=00 MaxTimeSlots=1 RejSnoopTrans- Arb: Fixed- WRR32- WRR64- WRR128-
TWRR128- WRR256-
Ctrl: Enable+ ID=0 ArbSelect=Fixed
TC/VC=01
Status: NegoPending- InProgress-
Kernel driver in use: vfio-pci
Quoted 5 lines
You can look
in/proc/interruptson the host and see if you're getting interrupts (non-zero count on at least one of the CPUs for the interrupt associated
with the device).
It appears to be getting interrupts but I assume there's no way to tell which device (usb1 or vfio-intx) they are from:
16: 654
6309 0 0 0 0 0 0
IR-IO-APIC 16-fasteoi ehci_hcd:usb1, vfio-intx(0000:02:00.0)
Quoted 2 lines
If instead the device is using INTx interrupts,
interrupt masking might be broken.
Am I correct in interpreting this statement along with the above output to mean that it's using INTx interrupts?
Quoted 10 lines
You can try using the nointxmask=1
module option to vfio-pci, for force masking at the APIC rather than the
device, but be forewarned that you'll need to make the interrupt for the
device exclusive, either by locating it in a slot where it won't share
interrupts or unloading drivers from devices sharing the interrupt line.
Given the above, it appears to be sharing the interrupt with one of the USB controllers. If your advice still stands, I can swap a couple of cards and see if I can get an exclusive interrupt or try to get everything off of that USB controller so that I can disable it. I'd appreciate your thoughts on if I would be better off trying this or your next suggestion first...
Quoted 10 lines
There's always the chance that the device is simply not compatible with
PCI device assignment. We do rely on some degree of good behavior on the part of the device. Some environments also expect to find the device behind a PCIe root port, which is not the topology we expose on
the default 440fx VM chipset. It's possible that such devices might work on the Q35 chipset or by placing the device behind a pci-bridge to
fool the software.
After doing a bit of reading up on 440fx vs Q35, this looks promising:
since this is a PCIe card, the Q35 chipset seems likely to be needed.
The only question I have is how to get Win XP to install using it. (on
newer machines you need to set the bios to legacy/emulation mode in a couple of places and I'm not sure yet how / if there's a way to 'fake it' for Windows with Q35) Are you aware of any docs / posts that deal with this scenario?
Quoted 3 lines
It's really hard to tell what might be wrong,
especially since the driver appears to work and only the application fails, and it's all proprietary code within the black box of a VM.
One additional detail I've found on closer examination: it appears that the driver installation did not complete fully in the VM, Windows just thinks it did. On the bare metal installation, there are two installed device drivers: one for the capture device (which doesn't have any hardware resources listed), one for the encoder (which does list hardware resources)... but both functions are performed on the card (it captures an audio/video input and then hardware encodes it to either mpeg 2 or h.264) so the first driver is not just a pseudo/software driver. In the VM, only the encoder is being installed. This explains a lot re: why the capture applications are having problems seeing the card. Is it possible that there's some unreported hardware resource that needs to be passed through that isn't being identified as a result of seeing it as a PCI rather than a PCIe device? (I'm thinking it could be something along the lines of the PCIe extended configuration space)
Quoted 4 lines
Thanks,
Alex
Thank you,
Phil