vfio-pci: dvb-s2 pcie card stopped working after a short time

DISCUSSION Wolfgang Nothdurft 3 participants 3 messages Apr 9, 2015 KVM

lore

Share & tools

lore

Wolfgang Nothdurft

author

2015-04-09 13:46 UTC #0 lore raw

Hi,

I'm using kvm + libvirt + pci pass-through (vfio-pci) for virtualizing my mythtv server.

    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x03' 
function='0x0'/>
    </hostdev>

I had no problems with my previous dvb cards at least no big known problems.

With my new dvb-s2 card (DVBSky S952 V3), the card stopped working after recording 1 or 2 hour.

After than the dvbstream from the card is broken and I have to reload the driver.

MythTV Log shows:

Apr  7 19:19:10 mythtv mythlogserver: mythbackend[2087]: E DVBRead recorders/dtvrecorder.cpp:855 (FindH264Keyframes) DTVRec[1]: PES start code not found in TS packet with PUSI set
Apr  7 19:19:12 mythtv mythlogserver: mythbackend[2087]: E DVBRead mpeg/mpegstreamdata.cpp:364 (AssemblePSIP) MPEGStream[1](0x7f9bd412b898): Error: offset>181, pes length & current cannot be queried

On the kvm host the card works properly. With my old cards I got these problems sporadically, so this was not a big deal for me. But now it is unusable.

/proc/interrupts and lspci -vv from the kvmhost:

29: 1061275 1065204 1060158 1066001 PCI-MSI-edge
SMI_PCIE

01:00.0 Multimedia video controller: Spin Master Ltd. Device 3038 (rev 01)
Subsystem: DVBSky Device 0552
Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop-
ParErr- Stepping- SERR- FastB2B- DisINTx+
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort-
<TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0, Cache Line Size: 32 bytes
Interrupt: pin A routed to IRQ 29
Region 0: Memory at d0100000 (32-bit, non-prefetchable) [size=4K]
Capabilities: [40] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA
PME(D0-,D1-,D2-,D3hot-,D3cold-)
Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
Capabilities: [50] MSI: Enable+ Count=1/16 Maskable- 64bit+
Address: 00000000fee0f00c Data: 4127
Capabilities: [70] Express (v1) Endpoint, MSI 00
DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s
<64ns, L1 <1us
ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset-
DevCtl: Report errors: Correctable- Non-Fatal- Fatal-
Unsupported-
RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop- MaxPayload 128 bytes, MaxReadReq 512 bytes
DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq-
AuxPwr- TransPend-

                LnkCap: Port #0, Speed 2.5GT/s, Width x1, ASPM L0s,

Latency L0 unlimited, L1 unlimited
ClockPM- Surprise- LLActRep- BwNot-
LnkCtl: ASPM Disabled; RCB 128 bytes Disabled- Retrain-
CommClk+
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 2.5GT/s, Width x1, TrErr- Train- SlotClk+
DLActive- BWMgmt- ABWMgmt-
Kernel driver in use: SMI PCIe driver
Kernel modules: smipcie

/proc/interrupts from kvmhost when the vm is started:

29: 0 0 0 0 PCI-MSI-edge
vfio-msi[0](0000:01:00.0)

/proc/interrupts and lspci -vv from the vm

25: 0 0 PCI-MSI-edge SMI_PCIE

02:03.0 Multimedia video controller: Spin Master Ltd. Device 3038 (rev 01)
Subsystem: DVBSky Device 0552
Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop-
ParErr- Stepping- SERR+ FastB2B- DisINTx+
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort-
<TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0, Cache Line Size: 32 bytes
Interrupt: pin A routed to IRQ 25
Region 0: Memory at fe660000 (32-bit, non-prefetchable) [size=4K]
Capabilities: [40] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA
PME(D0-,D1-,D2-,D3hot-,D3cold-)
Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
Capabilities: [50] MSI: Enable+ Count=1/16 Maskable- 64bit+
Address: 00000000fee0300c Data: 41a1
Capabilities: [70] Express (v1) Endpoint, MSI 00
DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s
<64ns, L1 <1us
ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset-
DevCtl: Report errors: Correctable- Non-Fatal- Fatal-
Unsupported-
RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop- MaxPayload 128 bytes, MaxReadReq 512 bytes
DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq-
AuxPwr- TransPend-

                LnkCap: Port #0, Speed 2.5GT/s, Width x1, ASPM L0s,

I have tried several things, all with the same result.

* Swapping PCIe Port
* using kvm-based-passthrough instead of vfio-pci
* starting qemu direct and using bus=pcie.0 ( -device vfio-pci,host=01:00.0,id=hostdev0,bus=pcie.0,addr=0x3)

Board: Supermicro C2SBC-Q
dvb-s2: DVBSky S952 V3
System: Gentoo
Kernel: 3.19.0-gentoo (on both host and vm)
qemu: 2.1.2
libvirt: 1.2.10

Is there anything I can do to find and solve this problem?
Or is this simply a bad combination of hardware components and I will stuck here?

Thanks
Wolfgang

Alex Williamson

2015-04-09 15:40 UTC #1 lore raw

Quoted 79 lines · Wolfgang Nothdurft

Hi,
I'm using kvm + libvirt + pci pass-through (vfio-pci) for virtualizing my mythtv server.
     <hostdev mode='subsystem' type='pci' managed='yes'>
       <driver name='vfio'/>
       <source>
         <address domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
       </source>
       <address type='pci' domain='0x0000' bus='0x02' slot='0x03' 
function='0x0'/>
     </hostdev>
I had no problems with my previous dvb cards at least no big known problems.
With my new dvb-s2 card (DVBSky S952 V3), the card stopped working after recording 1 or 2 hour.
After than the dvbstream from the card is broken and I have to reload the driver.
MythTV Log shows:
Apr  7 19:19:10 mythtv mythlogserver: mythbackend[2087]: E DVBRead 
recorders/dtvrecorder.cpp:855 (FindH264Keyframes) DTVRec[1]: PES start 
code not found in TS packet with PUSI set
Apr  7 19:19:12 mythtv mythlogserver: mythbackend[2087]: E DVBRead 
mpeg/mpegstreamdata.cpp:364 (AssemblePSIP) MPEGStream[1](0x7f9bd412b898): Error: offset>181, pes length & current cannot be queried
On the kvm host the card works properly. With my old cards I got these problems sporadically, so this was not a big deal for me. But now it is unusable.
/proc/interrupts and lspci -vv from the kvmhost:
29: 1061275 1065204 1060158 1066001 PCI-MSI-edge
SMI_PCIE
01:00.0 Multimedia video controller: Spin Master Ltd. Device 3038 (rev 01)
Subsystem: DVBSky Device 0552
Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop-
ParErr- Stepping- SERR- FastB2B- DisINTx+
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort-
<TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0, Cache Line Size: 32 bytes
Interrupt: pin A routed to IRQ 29
Region 0: Memory at d0100000 (32-bit, non-prefetchable) [size=4K]
Capabilities: [40] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA
PME(D0-,D1-,D2-,D3hot-,D3cold-)
Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
Capabilities: [50] MSI: Enable+ Count=1/16 Maskable- 64bit+
Address: 00000000fee0f00c Data: 4127
Capabilities: [70] Express (v1) Endpoint, MSI 00
DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s
<64ns, L1 <1us
ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset-
DevCtl: Report errors: Correctable- Non-Fatal- Fatal-
Unsupported-
RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop- MaxPayload 128 bytes, MaxReadReq 512 bytes
DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq-
AuxPwr- TransPend-
                 LnkCap: Port #0, Speed 2.5GT/s, Width x1, ASPM L0s, 
Latency L0 unlimited, L1 unlimited
ClockPM- Surprise- LLActRep- BwNot-
LnkCtl: ASPM Disabled; RCB 128 bytes Disabled- Retrain-
CommClk+
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 2.5GT/s, Width x1, TrErr- Train- SlotClk+
DLActive- BWMgmt- ABWMgmt-
Kernel driver in use: SMI PCIe driver
Kernel modules: smipcie

Quoted 8 lines

/proc/interrupts from kvmhost when the vm is started:
29: 0 0 0 0 PCI-MSI-edge
vfio-msi[0](0000:01:00.0)
/proc/interrupts and lspci -vv from the vm
25: 0 0 PCI-MSI-edge SMI_PCIE

Are you suggesting with these zero counts that interrupts aren't working, or do they increment normally when the card is in use?

Quoted 51 lines

02:03.0 Multimedia video controller: Spin Master Ltd. Device 3038 (rev 01)
Subsystem: DVBSky Device 0552
Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop-
ParErr- Stepping- SERR+ FastB2B- DisINTx+
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort-
<TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0, Cache Line Size: 32 bytes
Interrupt: pin A routed to IRQ 25
Region 0: Memory at fe660000 (32-bit, non-prefetchable) [size=4K]
Capabilities: [40] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA
PME(D0-,D1-,D2-,D3hot-,D3cold-)
Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
Capabilities: [50] MSI: Enable+ Count=1/16 Maskable- 64bit+
Address: 00000000fee0300c Data: 41a1
Capabilities: [70] Express (v1) Endpoint, MSI 00
DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s
<64ns, L1 <1us
ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset-
DevCtl: Report errors: Correctable- Non-Fatal- Fatal-
Unsupported-
RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop- MaxPayload 128 bytes, MaxReadReq 512 bytes
DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq-
AuxPwr- TransPend-
                 LnkCap: Port #0, Speed 2.5GT/s, Width x1, ASPM L0s, 
Latency L0 unlimited, L1 unlimited
ClockPM- Surprise- LLActRep- BwNot-
LnkCtl: ASPM Disabled; RCB 128 bytes Disabled- Retrain-
CommClk+
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 2.5GT/s, Width x1, TrErr- Train- SlotClk+
DLActive- BWMgmt- ABWMgmt-
Kernel driver in use: SMI PCIe driver
Kernel modules: smipcie
I have tried several things, all with the same result.
* Swapping PCIe Port
* using kvm-based-passthrough instead of vfio-pci
* starting qemu direct and using bus=pcie.0 ( -device vfio-pci,host=01:00.0,id=hostdev0,bus=pcie.0,addr=0x3)
Board: Supermicro C2SBC-Q
dvb-s2: DVBSky S952 V3
System: Gentoo
Kernel: 3.19.0-gentoo (on both host and vm)
qemu: 2.1.2
libvirt: 1.2.10

Quoted 3 lines

Is there anything I can do to find and solve this problem? Or is this simply a bad combination of hardware components and I will stuck here?

If neither vfio-pci nor legacy kvm based assignment work reliably, then it sounds like there's some quirkiness to this card's operation that hasn't been figured out. Does the old card work in the same configuration? Do the old and new cards use the same driver? Is there any relevant dmesg output in either the host or the guest? If you can kick the new card to work again by reloading the driver in the guest, that would seem to indicate some lack of robustness in the driver itself. Thanks,

Alex

Wolfgang Nothdurft

2015-04-09 16:22 UTC #2 lore raw

Am 09.04.2015 um 17:40 schrieb Alex Williamson:

Quoted 149 lines · Wolfgang Nothdurft

Quoted 79 lines
Hi,
I'm using kvm + libvirt + pci pass-through (vfio-pci) for virtualizing my mythtv server.
      <hostdev mode='subsystem' type='pci' managed='yes'>
        <driver name='vfio'/>
        <source>
          <address domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
        </source>
        <address type='pci' domain='0x0000' bus='0x02' slot='0x03'
function='0x0'/>
      </hostdev>
I had no problems with my previous dvb cards at least no big known problems.
With my new dvb-s2 card (DVBSky S952 V3), the card stopped working after recording 1 or 2 hour.
After than the dvbstream from the card is broken and I have to reload the driver.
MythTV Log shows:
Apr  7 19:19:10 mythtv mythlogserver: mythbackend[2087]: E DVBRead
recorders/dtvrecorder.cpp:855 (FindH264Keyframes) DTVRec[1]: PES start
code not found in TS packet with PUSI set
Apr  7 19:19:12 mythtv mythlogserver: mythbackend[2087]: E DVBRead
mpeg/mpegstreamdata.cpp:364 (AssemblePSIP) MPEGStream[1](0x7f9bd412b898): Error: offset>181, pes length & current cannot be queried
On the kvm host the card works properly. With my old cards I got these problems sporadically, so this was not a big deal for me. But now it is unusable.
/proc/interrupts and lspci -vv from the kvmhost:
   29:    1061275    1065204    1060158    1066001   PCI-MSI-edge
SMI_PCIE
01:00.0 Multimedia video controller: Spin Master Ltd. Device 3038 (rev 01)
Subsystem: DVBSky Device 0552
Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop-
ParErr- Stepping- SERR- FastB2B- DisINTx+
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort-
<TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0, Cache Line Size: 32 bytes
Interrupt: pin A routed to IRQ 29
Region 0: Memory at d0100000 (32-bit, non-prefetchable) [size=4K]
Capabilities: [40] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA
PME(D0-,D1-,D2-,D3hot-,D3cold-)
Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
Capabilities: [50] MSI: Enable+ Count=1/16 Maskable- 64bit+
Address: 00000000fee0f00c Data: 4127
Capabilities: [70] Express (v1) Endpoint, MSI 00
DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s
<64ns, L1 <1us
ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset-
DevCtl: Report errors: Correctable- Non-Fatal- Fatal-
Unsupported-
RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop- MaxPayload 128 bytes, MaxReadReq 512 bytes
DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq-
AuxPwr- TransPend-
                  LnkCap: Port #0, Speed 2.5GT/s, Width x1, ASPM L0s,
Latency L0 unlimited, L1 unlimited
ClockPM- Surprise- LLActRep- BwNot-
LnkCtl: ASPM Disabled; RCB 128 bytes Disabled- Retrain-
CommClk+
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 2.5GT/s, Width x1, TrErr- Train- SlotClk+
DLActive- BWMgmt- ABWMgmt-
Kernel driver in use: SMI PCIe driver
Kernel modules: smipcie
Quoted 8 lines
/proc/interrupts from kvmhost when the vm is started:
29: 0 0 0 0 PCI-MSI-edge
vfio-msi[0](0000:01:00.0)
/proc/interrupts and lspci -vv from the vm
25: 0 0 PCI-MSI-edge SMI_PCIE
Are you suggesting with these zero counts that interrupts aren't working, or do they increment normally when the card is in use?
Quoted 51 lines
02:03.0 Multimedia video controller: Spin Master Ltd. Device 3038 (rev 01)
Subsystem: DVBSky Device 0552
Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop-
ParErr- Stepping- SERR+ FastB2B- DisINTx+
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort-
<TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0, Cache Line Size: 32 bytes
Interrupt: pin A routed to IRQ 25
Region 0: Memory at fe660000 (32-bit, non-prefetchable) [size=4K]
Capabilities: [40] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA
PME(D0-,D1-,D2-,D3hot-,D3cold-)
Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
Capabilities: [50] MSI: Enable+ Count=1/16 Maskable- 64bit+
Address: 00000000fee0300c Data: 41a1
Capabilities: [70] Express (v1) Endpoint, MSI 00
DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s
<64ns, L1 <1us
ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset-
DevCtl: Report errors: Correctable- Non-Fatal- Fatal-
Unsupported-
RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop- MaxPayload 128 bytes, MaxReadReq 512 bytes
DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq-
AuxPwr- TransPend-
                  LnkCap: Port #0, Speed 2.5GT/s, Width x1, ASPM L0s,
Latency L0 unlimited, L1 unlimited
ClockPM- Surprise- LLActRep- BwNot-
LnkCtl: ASPM Disabled; RCB 128 bytes Disabled- Retrain-
CommClk+
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 2.5GT/s, Width x1, TrErr- Train- SlotClk+
DLActive- BWMgmt- ABWMgmt-
Kernel driver in use: SMI PCIe driver
Kernel modules: smipcie
I have tried several things, all with the same result.
* Swapping PCIe Port
* using kvm-based-passthrough instead of vfio-pci
* starting qemu direct and using bus=pcie.0 ( -device vfio-pci,host=01:00.0,id=hostdev0,bus=pcie.0,addr=0x3)
Board: Supermicro C2SBC-Q
dvb-s2: DVBSky S952 V3
System: Gentoo
Kernel: 3.19.0-gentoo (on both host and vm)
qemu: 2.1.2
libvirt: 1.2.10
Quoted 3 lines
Is there anything I can do to find and solve this problem? Or is this simply a bad combination of hardware components and I will stuck here?

Quoted 8 lines

If neither vfio-pci nor legacy kvm based assignment work reliably, then it sounds like there's some quirkiness to this card's operation that hasn't been figured out. Does the old card work in the same configuration? Do the old and new cards use the same driver? Is there any relevant dmesg output in either the host or the guest? If you can kick the new card to work again by reloading the driver in the guest, that would seem to indicate some lack of robustness in the driver itself. Thanks,

so it's not hopeless :)

The old card works in exact this configuration, but uses an other driver. The driver for the new card is newish and was first introduced in 3.19. I tested the media-build driver from dvbsky with the same result. But I was wondering, because the card was running stable with the same driver on the bare host for days, so I thought it would be due to virtualization.

The only relevant dmesg message is

kvm: zapping shadow pages for mmio generation wraparound

on the host when I start the vm.

One thing to mention is, that I need allow_unsafe_interrupts to get the passthrough to work.

Thanks
Wolfgang