Skip to contentSkip to conversation

Failed to PCI passthrough SSD with SMI SM2262 controller.

Bug 202055 [email protected] 1 participant 6 messages Apr 8, 2019 – Dec 29, 2020 1 message not in the archive KVM

lore
Share & tools
lore
#1 lore raw
In reply to a message not in the archive

https://bugzilla.kernel.org/show_bug.cgi?id=202055

--- Comment #42 from Alex Williamson ([email protected]) --- It seems there's a partial workaround available since QEMU v2.12 hiding under our noses. That version adds support for relocating the MSI-X vector table on vfio-pci devices, which recreates the MSI-X MMIO space elsewhere on the device.
A side-effect of this is that the vector table and PBA are properly sized so
as not to collide. The size of the tables remains wrong, but this only becomes a problem if the nvme code attempts to allocate >16 vectors, which requires >15 vCPU (or host CPUs if the device is returned to host drivers after being assigned)(nvme appears to allocate 1 admin queue, plus a queue per CPU, each making use of an IRQ vector). The QEMU vfio-pci device option is x-msix-relocation= which allows specifying the bar to use for the MSI-X tables, ex. bar0...bar5. Since this device uses a 64bit bar0, we can either extend that BAR or choose another, excluding bar1, which is consumed by the upper half of bar0. For instance, I tested with:

<domain type='kvm' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'>
...

    <hostdev mode='subsystem' type='pci' managed='yes'>
      <source>
        <address domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
      </source>
      <alias name='ua-sm2262'/>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x00'
function='0x0'/>
    </hostdev>
...
  <qemu:commandline>
    <qemu:arg value='-set'/>
    <qemu:arg value='device.ua-sm2262.x-msix-relocation=bar2'/>

</qemu:commandline>

(NB: "ua-" is a required prefix when specifying an alias)

A new virtual BAR appears in the guest hosting the MSI-X table and QEMU starts normally so long as the guest doesn't exceed 15 vCPUs.

The vCPU/pCPU count limitations are obviously not ideal, but hopefully this provides some degree of workaround for typical configurations.

Signature · 2 lines
You are receiving this mail because:
You are watching the assignee of the bug.
#2 lore raw
In reply to a message not in the archive

https://bugzilla.kernel.org/show_bug.cgi?id=202055

--- Comment #43 from LimeTech ([email protected]) --- (In reply to Alex Williamson from comment #42)

Quoted 31 lines

It seems there's a partial workaround available since QEMU v2.12 hiding under our noses. That version adds support for relocating the MSI-X vector table on vfio-pci devices, which recreates the MSI-X MMIO space elsewhere on the device. A side-effect of this is that the vector table and PBA are properly sized so as not to collide. The size of the tables remains wrong, but this only becomes a problem if the nvme code attempts to allocate >16 vectors, which requires >15 vCPU (or host CPUs if the device is returned to host drivers after being assigned)(nvme appears to allocate 1 admin queue, plus a queue per CPU, each making use of an IRQ vector). The QEMU vfio-pci device option is x-msix-relocation= which allows specifying the bar to use for the MSI-X tables, ex. bar0...bar5. Since this device uses a 64bit bar0, we can either extend that BAR or choose another, excluding bar1, which is consumed by the upper half of bar0. For instance, I tested with:

<domain type='kvm' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'>
...

    <hostdev mode='subsystem' type='pci' managed='yes'>
      <source>
        <address domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
      </source>
      <alias name='ua-sm2262'/>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x00'
function='0x0'/>
    </hostdev>
...
  <qemu:commandline>
    <qemu:arg value='-set'/>
    <qemu:arg value='device.ua-sm2262.x-msix-relocation=bar2'/>

</qemu:commandline>

Quoted 7 lines

(NB: "ua-" is a required prefix when specifying an alias)

A new virtual BAR appears in the guest hosting the MSI-X table and QEMU starts normally so long as the guest doesn't exceed 15 vCPUs.

The vCPU/pCPU count limitations are obviously not ideal, but hopefully this provides some degree of workaround for typical configurations.

Hi Alex, are you saying the above coding in the VM xml is all that is necessary (with noted vCPU/pCPU count limitations) to successfully pass-through sm2262-based controllers, without above kernel patch? Or is a kernel patch also necessary (if so, which one)?

thx,
-tom

Signature · 2 lines
You are receiving this mail because:
You are watching the assignee of the bug.
#3 lore raw
In reply to a message not in the archive

https://bugzilla.kernel.org/show_bug.cgi?id=202055

--- Comment #44 from Alex Williamson ([email protected]) --- (In reply to LimeTech from comment #43)

Quoted 5 lines

Hi Alex, are you saying the above coding in the VM xml is all that is necessary (with noted vCPU/pCPU count limitations) to successfully pass-through sm2262-based controllers, without above kernel patch? Or is a kernel patch also necessary (if so, which one)?

Yes, without kernel patch.

Signature · 2 lines
You are receiving this mail because:
You are watching the assignee of the bug.
#4 lore raw
In reply to a message not in the archive

https://bugzilla.kernel.org/show_bug.cgi?id=202055

FCL ([email protected]) changed:

What |Removed |Added
----------------------------------------------------------------------------

                 CC|                            |[email protected]
                   |                            |m

--- Comment #46 from FCL ([email protected]) --- Hey gents!

Trying to apply this to a proxmox VM, any advice on how to do so? I'm running on top of the 5.4.x linux kernel. I'm experiencing the same bug as in topic name, using the SMI 2263 controller. (Adata as above)

Anything I can provide in terms of debugging data? Otherwise, any advice on how to use the above non kernel patch? also comfortable doing a kernel recompile if thats the better solution. Will be going to a FreeBSD (FreeNAS 11.3) VM with 4 threads and 72GB of ram alongside an LSI SAS controller.

Platform is a Dual Xeon 5670 on a HP ml350 G6 mother board.

Also attached but on a different VM is a Radeon 5600XT, an intel USB controller and an intel Nic to a windows VM

Signature · 2 lines
You are receiving this mail because:
You are watching the assignee of the bug.
#5 lore raw
In reply to a message not in the archive

https://bugzilla.kernel.org/show_bug.cgi?id=202055

--- Comment #47 from FCL ([email protected]) --- (In reply to FCL from comment #46)

Quoted 11 lines

Hey gents!

Trying to apply this to a proxmox VM, any advice on how to do so? I'm running on top of the 5.4.x linux kernel. I'm experiencing the same bug as in topic name, using the SMI 2263 controller. (Adata as above)

Anything I can provide in terms of debugging data? Otherwise, any advice on how to use the above non kernel patch? also comfortable doing a kernel recompile if thats the better solution. Will be going to a FreeBSD (FreeNAS 11.3) VM with 4 threads and 72GB of ram alongside an LSI SAS controller.

Quoted 4 lines

Platform is a Dual Xeon 5670 on a HP ml350 G6 mother board.

Also attached but on a different VM is a Radeon 5600XT, an intel USB controller and an intel Nic to a windows VM

Found the solution without the need for a kernel recompile. in my Qemu conf file added:
args: -set device.hostpci1.x-msix-relocation=bar2

Where hostpci1 is
hostpci1: 08:00.0

Signature · 2 lines
You are receiving this mail because:
You are watching the assignee of the bug.
#6 lore raw
In reply to a message not in the archive

https://bugzilla.kernel.org/show_bug.cgi?id=202055

[email protected] changed:

What |Removed |Added
----------------------------------------------------------------------------
CC| |[email protected]

--- Comment #48 from [email protected] ---
For Intel SSD 660p series, the latest firmware is required for pass-through to work without issues. The latest one should be from mid 2020 at the time of writing this.

Not even any workarounds or special settings for qemu/libvirt are needed, just a simple pass-through setup as with any other PCIe device.

Signature · 4 lines
You may reply to this email to add a comment.

You are receiving this mail because:
You are watching the assignee of the bug.