ath11k and vfio-pci support
Subsystems (3)
Hi,
For a while now I've been using vfio-pci to pass through wireless hardware to a Qemu VM. This makes testing kernel changes quick and easy compared to swapping the host kernel for reach iterative change. So far I've had very few issues doing this, maybe I've just been lucky... I tried doing this with an ath11k (WCN6855) card and ran into issues with the driver starting up. I'm wondering if its a configuration issue or just a lack of support by ath11k? The card works just fine when I use it on my host machine. Based on the logs it may not even be related to ath11k as wmi-bmof seems to fail first, but I'm not familiar with anything at the PCI level so I've got no idea whats going on.
My host machine is running 5.19
Booting into a 6.2 kernel with Qemu
Below is a partial kernel log of relevant messages:
[ 0.318120] ath11k_pci 0000:00:06.0: BAR 0: assigned [mem 0x100000000-0x1001fffff 64bit]
[ 0.319006] ath11k_pci 0000:00:06.0: MSI vectors: 1
[ 0.319075] ath11k_pci 0000:00:06.0: wcn6855 hw2.1
[ 0.475326] mhi mhi0: Requested to power ON
[ 0.475333] mhi mhi0: Power on setup success
[ 1.440375] Driver 'wmi-bmof' was unable to register with bus_type 'wmi' because the bus was not initialized.
[ 1.459667] ath11k_pci 0000:00:06.0: chip_id 0x12 chip_family 0xb board_id 0xff soc_id 0x400c1211
[ 1.459674] ath11k_pci 0000:00:06.0: fw_version 0x1109996e fw_build_timestamp 2023-12-19 11:11 fw_build_id WLAN.HSP.1.1-03125-QCAHSPSWPL_V1_V2_SILICONZ_LITE-3.6510.36 [ 1.528705] ath11k_pci 0000:00:06.0: leaving PCI ASPM disabled to avoid MHI M2 problems
[ 2.537835] ath11k_pci 0000:00:06.0: failed to receive control response completion, polling..
[ 3.562151] ath11k_pci 0000:00:06.0: Service connect timeout
[ 3.563511] ath11k_pci 0000:00:06.0: failed to connect to HTT: -110
[ 3.565375] ath11k_pci 0000:00:06.0: failed to start core: -110
[ 3.776067] ath11k_pci 0000:00:06.0: ignore reset dev flags 0x4000
[ 13.802149] ath11k_pci 0000:00:06.0: failed to wait wlan mode request (mode 4): -110
[ 13.802163] ath11k_pci 0000:00:06.0: qmi failed to send wlan mode off: -110
+ ath11k list
Quoted 11 lines · James Prestwood
For a while now I've been using vfio-pci to pass through wireless hardware to a Qemu VM. This makes testing kernel changes quick and easy compared to swapping the host kernel for reach iterative change. So far I've had very few issues doing this, maybe I've just been lucky... I tried doing this with an ath11k (WCN6855) card and ran into issues with the driver starting up. I'm wondering if its a configuration issue or just a lack of support by ath11k? The card works just fine when I use it on my host machine. Based on the logs it may not even be related to ath11k as wmi-bmof seems to fail first, but I'm not familiar with anything at the PCI level so I've got no idea whats going on.
There's a bug report about this:
https://bugzilla.kernel.org/show_bug.cgi?id=216055
But I have also no idea what is causing this, I guess we are doing something wrong with the PCI communication? That reminds me, you could try this in case that helps:
https://patchwork.kernel.org/project/linux-wireless/patch/[email protected]/
Signature · 3 lines
https://patchwork.kernel.org/project/linux-wireless/list/ https://wireless.wiki.kernel.org/en/developers/documentation/submittingpatches
Hi Kalle,
Quoted 15 lines · Kalle Valo
+ ath11k list
Quoted 11 lines · James Prestwood
For a while now I've been using vfio-pci to pass through wireless hardware to a Qemu VM. This makes testing kernel changes quick and easy compared to swapping the host kernel for reach iterative change. So far I've had very few issues doing this, maybe I've just been lucky... I tried doing this with an ath11k (WCN6855) card and ran into issues with the driver starting up. I'm wondering if its a configuration issue or just a lack of support by ath11k? The card works just fine when I use it on my host machine. Based on the logs it may not even be related to ath11k as wmi-bmof seems to fail first, but I'm not familiar with anything at the PCI level so I've got no idea whats going on.
Quoted 3 lines
There's a bug report about this:
Yes I saw that, but apparently didn't read past the first comment... I'm not seeing any of the DMAR faults but the ath-releated ones look the same. Good to know its not just me.
Quoted 6 lines
But I have also no idea what is causing this, I guess we are doing something wrong with the PCI communication? That reminds me, you could try this in case that helps:
https://patchwork.kernel.org/project/linux-wireless/patch/[email protected]/
Heh, I saw this pop up a day after I sent this and was wondering. Is this something I'd need on the host kernel, guest, or both?
Thanks,
James
Quoted 8 lines · James Prestwood
Quoted 5 lines
But I have also no idea what is causing this, I guess we are doing something wrong with the PCI communication? That reminds me, you could try this in case that helps:
https://patchwork.kernel.org/project/linux-wireless/patch/[email protected]/
Heh, I saw this pop up a day after I sent this and was wondering. Is this something I'd need on the host kernel, guest, or both?
On the guest where ath11k is running. I'm not optimistic that this would solve your issue, I suspect there can be also other bugs, but good to know if the patch changes anything.
Signature · 3 lines
https://patchwork.kernel.org/project/linux-wireless/list/ https://wireless.wiki.kernel.org/en/developers/documentation/submittingpatches
Hi Kalle,
Quoted 6 lines · Kalle Valo
Quoted 4 lines · James Prestwood
Quoted 4 lines
But I have also no idea what is causing this, I guess we are doing something wrong with the PCI communication? That reminds me, you could try this in case that helps:
Quoted 6 lines
Quoted 3 lines
Heh, I saw this pop up a day after I sent this and was wondering. Is this something I'd need on the host kernel, guest, or both?
On the guest where ath11k is running. I'm not optimistic that this would solve your issue, I suspect there can be also other bugs, but good to know if the patch changes anything.
Looks the same here, didn't seem to change anything based on the kernel logs.
Thanks,
James
Quoted 16 lines · James Prestwood
Hi Kalle,
Quoted 6 lines · Kalle Valo
Quoted 4 lines · James Prestwood
Quoted 4 lines
But I have also no idea what is causing this, I guess we are doing something wrong with the PCI communication? That reminds me, you could try this in case that helps:
Quoted 6 lines
Quoted 3 lines
Heh, I saw this pop up a day after I sent this and was wondering. Is this something I'd need on the host kernel, guest, or both?
On the guest where ath11k is running. I'm not optimistic that this would solve your issue, I suspect there can be also other bugs, but good to know if the patch changes anything.
Quoted 6 lines
Looks the same here, didn't seem to change anything based on the kernel logs.
Thanks,
James
Could you try this?
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/drivers/net/wireless/ath/ath11k/pci.c?id=39564b475ac5a589e6c22c43a08cbd283c295d2c
Quoted 18 lines · Baochen Qiang
Quoted 17 lines · James Prestwood
Hi Kalle,
Quoted 14 lines · Kalle Valo
Quoted 8 lines · James Prestwood
Quoted 5 lines
But I have also no idea what is causing this, I guess we are doing something wrong with the PCI communication? That reminds me, you could try this in case that helps:
https://patchwork.kernel.org/project/linux-wireless/patch/[email protected]/
Heh, I saw this pop up a day after I sent this and was wondering. Is this something I'd need on the host kernel, guest, or both?
On the guest where ath11k is running. I'm not optimistic that this would solve your issue, I suspect there can be also other bugs, but good to know if the patch changes anything.
Quoted 6 lines
Quoted 3 lines
Looks the same here, didn't seem to change anything based on the kernel logs.
Could you try this?
This reminds me, I assumed James was testing with ath.git master branch (which has that commit) but I never checked that. So for testing please always use the master branch to get the latest and greatest ath11k:
https://git.kernel.org/pub/scm/linux/kernel/git/kvalo/ath.git/
There's a quite long delay from ath.git to official releases.
Signature · 3 lines
https://patchwork.kernel.org/project/linux-wireless/list/ https://wireless.wiki.kernel.org/en/developers/documentation/submittingpatches
Hi Kalle, Baochen,
Quoted 22 lines · Kalle Valo
Quoted 20 lines · Baochen Qiang
Quoted 8 lines · James Prestwood
Hi Kalle,
Quoted 6 lines · Kalle Valo
Quoted 4 lines · James Prestwood
Quoted 4 lines
But I have also no idea what is causing this, I guess we are doing something wrong with the PCI communication? That reminds me, you could try this in case that helps:
Quoted 9 lines
Quoted 6 lines
Quoted 3 lines
Heh, I saw this pop up a day after I sent this and was wondering. Is this something I'd need on the host kernel, guest, or both?
On the guest where ath11k is running. I'm not optimistic that this would solve your issue, I suspect there can be also other bugs, but good to know if the patch changes anything.
Looks the same here, didn't seem to change anything based on the kernel logs.
Could you try this?
Quoted 8 lines
This reminds me, I assumed James was testing with ath.git master branch (which has that commit) but I never checked that. So for testing please always use the master branch to get the latest and greatest ath11k:
https://git.kernel.org/pub/scm/linux/kernel/git/kvalo/ath.git/
There's a quite long delay from ath.git to official releases.
Good to know, and I was not in fact using that branch. Rebuilt from ath.git/master but still roughly the same behavior. There does appear to be more output now though, specifically a firmware crash:
[ 2.281721] ath11k_pci 0000:00:06.0: failed to receive control response completion, polling.. [ 2.282101] ip (65) used greatest stack depth: 12464 bytes left
[ 3.306039] ath11k_pci 0000:00:06.0: Service connect timeout
[ 3.307588] ath11k_pci 0000:00:06.0: failed to connect to HTT: -110
[ 3.309286] ath11k_pci 0000:00:06.0: failed to start core: -110
[ 3.519637] ath11k_pci 0000:00:06.0: firmware crashed: MHI_CB_EE_RDDM
[ 3.519678] ath11k_pci 0000:00:06.0: ignore reset dev flags 0x4000
[ 3.627087] ath11k_pci 0000:00:06.0: firmware crashed: MHI_CB_EE_RDDM
[ 3.627129] ath11k_pci 0000:00:06.0: ignore reset dev flags 0x4000
[ 13.802105] ath11k_pci 0000:00:06.0: failed to wait wlan mode request (mode 4): -110 [ 13.802175] ath11k_pci 0000:00:06.0: qmi failed to send wlan mode off: -110
Thanks,
James
Quoted 38 lines · James Prestwood
Hi Kalle, Baochen,
Quoted 22 lines · Kalle Valo
Quoted 20 lines · Baochen Qiang
Quoted 8 lines · James Prestwood
Hi Kalle,
Quoted 6 lines · Kalle Valo
Quoted 4 lines · James Prestwood
Quoted 4 lines
But I have also no idea what is causing this, I guess we are doing something wrong with the PCI communication? That reminds me, you could try this in case that helps:
Quoted 9 lines
Quoted 6 lines
Quoted 3 lines
Heh, I saw this pop up a day after I sent this and was wondering. Is this something I'd need on the host kernel, guest, or both?
On the guest where ath11k is running. I'm not optimistic that this would solve your issue, I suspect there can be also other bugs, but good to know if the patch changes anything.
Looks the same here, didn't seem to change anything based on the kernel logs.
Could you try this?
Quoted 8 lines
This reminds me, I assumed James was testing with ath.git master branch (which has that commit) but I never checked that. So for testing please always use the master branch to get the latest and greatest ath11k:
https://git.kernel.org/pub/scm/linux/kernel/git/kvalo/ath.git/
There's a quite long delay from ath.git to official releases.
Good to know, and I was not in fact using that branch. Rebuilt from ath.git/master but still roughly the same behavior. There does appear to be more output now though, specifically a firmware crash:
Quoted 14 lines
[ 2.281721]
ath11k_pci0000:00:06.0: failed to receive control response completion, polling..
[ 2.282101] ip (65) used greatest stack depth: 12464 bytes left [ 3.306039]ath11k_pci0000:00:06.0: Service connect timeout [ 3.307588]ath11k_pci0000:00:06.0: failed to connect to HTT: -110 [ 3.309286]ath11k_pci0000:00:06.0: failed to start core: -110 [ 3.519637]ath11k_pci0000:00:06.0: firmware crashed:MHI_CB_EE_RDDM[ 3.519678]ath11k_pci0000:00:06.0: ignore reset dev flags0x4000[ 3.627087]ath11k_pci0000:00:06.0: firmware crashed:MHI_CB_EE_RDDM[ 3.627129]ath11k_pci0000:00:06.0: ignore reset dev flags0x4000[ 13.802105]ath11k_pci0000:00:06.0: failed to wait wlan mode request (mode 4): -110
[ 13.802175]ath11k_pci0000:00:06.0: qmi failed to send wlan mode
off: -110
Ok, that's progress now. Can you try next try the iommu patch[1] we talked about earlier? It's already in master-pending branch (along with other pending patches) so you can use that branch if you want.
[1] https://patchwork.kernel.org/project/linux-wireless/patch/[email protected]/
Signature · 3 lines
https://patchwork.kernel.org/project/linux-wireless/list/ https://wireless.wiki.kernel.org/en/developers/documentation/submittingpatches
Quoted 53 lines · Kalle Valo
Quoted 37 lines · James Prestwood
Hi Kalle, Baochen,
Quoted 22 lines · Kalle Valo
Quoted 20 lines · Baochen Qiang
Quoted 8 lines · James Prestwood
Hi Kalle,
Quoted 6 lines · Kalle Valo
Quoted 4 lines · James Prestwood
Quoted 4 lines
But I have also no idea what is causing this, I guess we are doing something wrong with the PCI communication? That reminds me, you could try this in case that helps:
Quoted 9 lines
Quoted 6 lines
Quoted 3 lines
Heh, I saw this pop up a day after I sent this and was wondering. Is this something I'd need on the host kernel, guest, or both?
On the guest where ath11k is running. I'm not optimistic that this would solve your issue, I suspect there can be also other bugs, but good to know if the patch changes anything.
Looks the same here, didn't seem to change anything based on the kernel logs.
Could you try this?
Quoted 8 lines
This reminds me, I assumed James was testing with ath.git master branch (which has that commit) but I never checked that. So for testing please always use the master branch to get the latest and greatest ath11k:
https://git.kernel.org/pub/scm/linux/kernel/git/kvalo/ath.git/
There's a quite long delay from ath.git to official releases.
Good to know, and I was not in fact using that branch. Rebuilt from ath.git/master but still roughly the same behavior. There does appear to be more output now though, specifically a firmware crash:
Quoted 14 lines
[ 2.281721]
ath11k_pci0000:00:06.0: failed to receive control response completion, polling..
[ 2.282101] ip (65) used greatest stack depth: 12464 bytes left [ 3.306039]ath11k_pci0000:00:06.0: Service connect timeout [ 3.307588]ath11k_pci0000:00:06.0: failed to connect to HTT: -110 [ 3.309286]ath11k_pci0000:00:06.0: failed to start core: -110 [ 3.519637]ath11k_pci0000:00:06.0: firmware crashed:MHI_CB_EE_RDDM[ 3.519678]ath11k_pci0000:00:06.0: ignore reset dev flags0x4000[ 3.627087]ath11k_pci0000:00:06.0: firmware crashed:MHI_CB_EE_RDDM[ 3.627129]ath11k_pci0000:00:06.0: ignore reset dev flags0x4000[ 13.802105]ath11k_pci0000:00:06.0: failed to wait wlan mode request (mode 4): -110
[ 13.802175]ath11k_pci0000:00:06.0: qmi failed to send wlan mode
off: -110
Quoted 5 lines
Ok, that's progress now. Can you try next try the iommu patch[1] we talked about earlier? It's already in master-pending branch (along with other pending patches) so you can use that branch if you want.
[1] https://patchwork.kernel.org/project/linux-wireless/patch/[email protected]/
Same result unfortunately, tried both with just [1] applied to ath.git and at HEAD of master-pending.
Thanks,
James
Quoted 59 lines · James Prestwood
Quoted 57 lines · Kalle Valo
Quoted 37 lines · James Prestwood
Hi Kalle, Baochen,
Quoted 22 lines · Kalle Valo
Quoted 20 lines · Baochen Qiang
Quoted 8 lines · James Prestwood
Hi Kalle,
Quoted 6 lines · Kalle Valo
Quoted 4 lines · James Prestwood
Quoted 4 lines
But I have also no idea what is causing this, I guess we are doing something wrong with the PCI communication? That reminds me, you could try this in case that helps:
Quoted 9 lines
Quoted 6 lines
Quoted 3 lines
Heh, I saw this pop up a day after I sent this and was wondering. Is this something I'd need on the host kernel, guest, or both?
On the guest where ath11k is running. I'm not optimistic that this would solve your issue, I suspect there can be also other bugs, but good to know if the patch changes anything.
Looks the same here, didn't seem to change anything based on the kernel logs.
Could you try this?
Quoted 8 lines
This reminds me, I assumed James was testing with ath.git master branch (which has that commit) but I never checked that. So for testing please always use the master branch to get the latest and greatest ath11k:
https://git.kernel.org/pub/scm/linux/kernel/git/kvalo/ath.git/
There's a quite long delay from ath.git to official releases.
Good to know, and I was not in fact using that branch. Rebuilt from ath.git/master but still roughly the same behavior. There does appear to be more output now though, specifically a firmware crash:
Quoted 14 lines
[ 2.281721]
ath11k_pci0000:00:06.0: failed to receive control response completion, polling..
[ 2.282101] ip (65) used greatest stack depth: 12464 bytes left [ 3.306039]ath11k_pci0000:00:06.0: Service connect timeout [ 3.307588]ath11k_pci0000:00:06.0: failed to connect to HTT: -110 [ 3.309286]ath11k_pci0000:00:06.0: failed to start core: -110 [ 3.519637]ath11k_pci0000:00:06.0: firmware crashed:MHI_CB_EE_RDDM[ 3.519678]ath11k_pci0000:00:06.0: ignore reset dev flags0x4000[ 3.627087]ath11k_pci0000:00:06.0: firmware crashed:MHI_CB_EE_RDDM[ 3.627129]ath11k_pci0000:00:06.0: ignore reset dev flags0x4000[ 13.802105]ath11k_pci0000:00:06.0: failed to wait wlan mode request (mode 4): -110
[ 13.802175]ath11k_pci0000:00:06.0: qmi failed to send wlan mode
off: -110Ok, that's progress now. Can you try next try the iommu patch[1] we talked about earlier? It's already in master-pending branch (along with other pending patches) so you can use that branch if you want.
Quoted 7 lines
Same result unfortunately, tried both with just [1] applied to ath.git and at HEAD of master-pending.
Thanks,
James
Strange that still fails. Are you now seeing this error in your host or your Qemu? or both?
Could you share your test steps? And if you can share please be as detailed as possible since I'm not familiar with passing WLAN hardware to a VM using vfio-pci.
Quoted 2 lines
Hi,
Quoted 66 lines · Baochen Qiang
Quoted 63 lines · James Prestwood
Quoted 60 lines · Kalle Valo
Quoted 39 lines · James Prestwood
Hi Kalle, Baochen,
Quoted 25 lines · Kalle Valo
Quoted 16 lines · Baochen Qiang
Quoted 15 lines · James Prestwood
Hi Kalle,
Quoted 13 lines · Kalle Valo
Quoted 8 lines · James Prestwood
Quoted 6 lines
But I have also no idea what is causing this, I guess we are doing something wrong with the PCI communication? That reminds me, you could try this in case that helps:
https://patchwork.kernel.org/project/linux-wireless/patch/[email protected]/
Heh, I saw this pop up a day after I sent this and was wondering. Is this something I'd need on the host kernel, guest, or both?
On the guest where ath11k is running. I'm not optimistic that this would solve your issue, I suspect there can be also other bugs, but good to know if the patch changes anything.
Quoted 7 lines
Quoted 3 lines
Looks the same here, didn't seem to change anything based on the kernel logs.
Could you try this?
Quoted 7 lines
This reminds me, I assumed James was testing with ath.git master branch (which has that commit) but I never checked that. So for testing please always use the master branch to get the latest and greatest ath11k:
https://git.kernel.org/pub/scm/linux/kernel/git/kvalo/ath.git/
There's a quite long delay from ath.git to official releases.
Good to know, and I was not in fact using that branch. Rebuilt from ath.git/master but still roughly the same behavior. There does appear to be more output now though, specifically a firmware crash:
Quoted 14 lines
[ 2.281721]
ath11k_pci0000:00:06.0: failed to receive control response completion, polling..
[ 2.282101] ip (65) used greatest stack depth: 12464 bytes left [ 3.306039]ath11k_pci0000:00:06.0: Service connect timeout [ 3.307588]ath11k_pci0000:00:06.0: failed to connect to HTT: -110 [ 3.309286]ath11k_pci0000:00:06.0: failed to start core: -110 [ 3.519637]ath11k_pci0000:00:06.0: firmware crashed:MHI_CB_EE_RDDM[ 3.519678]ath11k_pci0000:00:06.0: ignore reset dev flags0x4000[ 3.627087]ath11k_pci0000:00:06.0: firmware crashed:MHI_CB_EE_RDDM[ 3.627129]ath11k_pci0000:00:06.0: ignore reset dev flags0x4000[ 13.802105]ath11k_pci0000:00:06.0: failed to wait wlan mode request (mode 4): -110
[ 13.802175]ath11k_pci0000:00:06.0: qmi failed to send wlan mode
off: -110Ok, that's progress now. Can you try next try the iommu patch[1] we talked about earlier? It's already in master-pending branch (along with other pending patches) so you can use that branch if you want.
[1] https://patchwork.kernel.org/project/linux-wireless/patch/[email protected]/
Quoted 7 lines
Quoted 5 lines
Same result unfortunately, tried both with just [1] applied to ath.git and at HEAD of master-pending.
Thanks,
James
Strange that still fails. Are you now seeing this error in your host or your Qemu? or both?
Could you share your test steps? And if you can share please be as detailed as possible since I'm not familiar with passing WLAN hardware to a VM using vfio-pci.
Just in Qemu, the hardware works fine on my host machine.
I basically follow this guide to set it up, its written in the context of GPUs/libvirt but the host setup is exactly the same. By no means do you need to read it all, once you set the vfio-pci.ids and see your unclaimed adapter you can stop:
https://wiki.archlinux.org/title/PCI_passthrough_via_OVMF
In short you should be able to set the following host kernel options and reboot (assuming your motherboard/hardware is compatible):
intel_iommu=on iommu=pt vfio-pci.ids=17cb:1103
Obviously change the device/vendor IDs to whatever ath11k hw you have. Once the host is rebooted you should see your wlan adapter as UNCLAIMED, showing the driver in use as vfio-pci. If not, its likely your motherboard just isn't compatible, the device has to be in its own IOMMU group (you could try switching PCI ports if this is the case).
I then build a "kvm_guest.config" kernel with the driver/firmware for ath11k and boot into that with the following Qemu options:
-enable-kvm -device -vfio-pci,host=<PCI address>
If it seems easier you could also utilize IWD's test-runner which handles launching the Qemu kernel automatically, detecting any vfio-devices and passes them through and mounts some useful host folders into the VM. Its actually a very good general purpose tool for kernel testing, not just for IWD:
https://git.kernel.org/pub/scm/network/wireless/iwd.git/tree/doc/test-runner.txt
Once set up you can just run test-runner with a few flags and you'll boot into a shell:
./tools/test-runner -k <kernel-image> --hw --start /bin/bash
Please reach out if you have questions, thanks for looking into this.
Quoted 93 lines · James Prestwood
Hi,
Quoted 64 lines · Baochen Qiang
Quoted 61 lines · James Prestwood
Quoted 58 lines · Kalle Valo
Quoted 37 lines · James Prestwood
Hi Kalle, Baochen,
Quoted 22 lines · Kalle Valo
Quoted 20 lines · Baochen Qiang
Quoted 8 lines · James Prestwood
Hi Kalle,
Quoted 6 lines · Kalle Valo
Quoted 4 lines · James Prestwood
Quoted 4 lines
But I have also no idea what is causing this, I guess we are doing something wrong with the PCI communication? That reminds me, you could try this in case that helps:
Quoted 9 lines
Quoted 6 lines
Quoted 3 lines
Heh, I saw this pop up a day after I sent this and was wondering. Is this something I'd need on the host kernel, guest, or both?
On the guest where ath11k is running. I'm not optimistic that this would solve your issue, I suspect there can be also other bugs, but good to know if the patch changes anything.
Looks the same here, didn't seem to change anything based on the kernel logs.
Could you try this?
Quoted 8 lines
This reminds me, I assumed James was testing with ath.git master branch (which has that commit) but I never checked that. So for testing please always use the master branch to get the latest and greatest ath11k:
https://git.kernel.org/pub/scm/linux/kernel/git/kvalo/ath.git/
There's a quite long delay from ath.git to official releases.
Good to know, and I was not in fact using that branch. Rebuilt from ath.git/master but still roughly the same behavior. There does appear to be more output now though, specifically a firmware crash:
Quoted 14 lines
[ 2.281721]
ath11k_pci0000:00:06.0: failed to receive control response completion, polling..
[ 2.282101] ip (65) used greatest stack depth: 12464 bytes left [ 3.306039]ath11k_pci0000:00:06.0: Service connect timeout [ 3.307588]ath11k_pci0000:00:06.0: failed to connect to HTT: -110 [ 3.309286]ath11k_pci0000:00:06.0: failed to start core: -110 [ 3.519637]ath11k_pci0000:00:06.0: firmware crashed:MHI_CB_EE_RDDM[ 3.519678]ath11k_pci0000:00:06.0: ignore reset dev flags0x4000[ 3.627087]ath11k_pci0000:00:06.0: firmware crashed:MHI_CB_EE_RDDM[ 3.627129]ath11k_pci0000:00:06.0: ignore reset dev flags0x4000[ 13.802105]ath11k_pci0000:00:06.0: failed to wait wlan mode request (mode 4): -110
[ 13.802175]ath11k_pci0000:00:06.0: qmi failed to send wlan mode
off: -110Ok, that's progress now. Can you try next try the iommu patch[1] we talked about earlier? It's already in master-pending branch (along with other pending patches) so you can use that branch if you want.
[1] https://patchwork.kernel.org/project/linux-wireless/patch/[email protected]/
Quoted 7 lines
Quoted 5 lines
Same result unfortunately, tried both with just [1] applied to ath.git and at HEAD of master-pending.
Thanks,
James
Strange that still fails. Are you now seeing this error in your host or your Qemu? or both?
Could you share your test steps? And if you can share please be as detailed as possible since I'm not familiar with passing WLAN hardware to a VM using vfio-pci.Just in Qemu, the hardware works fine on my host machine.
I basically follow this guide to set it up, its written in the context of GPUs/libvirt but the host setup is exactly the same. By no means do you need to read it all, once you set the vfio-pci.ids and see your unclaimed adapter you can stop:
https://wiki.archlinux.org/title/PCI_passthrough_via_OVMF
In short you should be able to set the following host kernel options and reboot (assuming your motherboard/hardware is compatible):
intel_iommu=on iommu=pt vfio-pci.ids=17cb:1103
Obviously change the device/vendor IDs to whatever ath11k hw you have. Once the host is rebooted you should see your wlan adapter as UNCLAIMED, showing the driver in use as vfio-pci. If not, its likely your motherboard just isn't compatible, the device has to be in its own IOMMU group (you could try switching PCI ports if this is the case).
I then build a "kvm_guest.config" kernel with the driver/firmware for ath11k and boot into that with the following Qemu options:
-enable-kvm -device -vfio-pci,host=<PCI address>
If it seems easier you could also utilize IWD's test-runner which handles launching the Qemu kernel automatically, detecting any vfio-devices and passes them through and mounts some useful host folders into the VM. Its actually a very good general purpose tool for kernel testing, not just for IWD:
Quoted 8 lines
https://git.kernel.org/pub/scm/network/wireless/iwd.git/tree/doc/test-runner.txt
Once set up you can just run test-runner with a few flags and you'll boot into a shell:
./tools/test-runner-k <kernel-image> --hw --start/bin/bashPlease reach out if you have questions, thanks for looking into this.
Thanks for these details. I reproduced this issue by following your guide.
Seems the root cause is that the MSI vector assigned to WCN6855 in qemu is different with that in host. In my case the MSI vector in qemu is [Address: fee00000 Data: 0020] while in host it is [Address: fee00578 Data: 0000]. So in qemu ath11k configures MSI vector [Address: fee00000 Data: 0020] to WCN6855 hardware/firmware, and firmware uses that vector to fire interrupts to host/qemu. However host IOMMU doesn't know that vector because the real vector is [Address: fee00578 Data: 0000], as a result host blocks that interrupt and reports an error, see below log:
[ 1414.206069] DMAR: DRHD: handling fault status reg 2 [ 1414.206081] DMAR: [INTR-REMAP] Request device [02:00.0] fault index 0x0 [fault reason 0x25] Blocked a compatibility format interrupt request [ 1414.210334] DMAR: DRHD: handling fault status reg 2 [ 1414.210342] DMAR: [INTR-REMAP] Request device [02:00.0] fault index 0x0 [fault reason 0x25] Blocked a compatibility format interrupt request [ 1414.212496] DMAR: DRHD: handling fault status reg 2 [ 1414.212503] DMAR: [INTR-REMAP] Request device [02:00.0] fault index 0x0 [fault reason 0x25] Blocked a compatibility format interrupt request [ 1414.214600] DMAR: DRHD: handling fault status reg 2
While I don't think there is a way for qemu/ath11k to get the real MSI vector from host, I will try to read the vfio code to check further. Before that, to unblock you, a possible hack is to hard code the MSI vector in qemu to the same as in host, on condition that the MSI vector doesn't change. In my case, the change looks like
drivers/net/wireless/ath/ath11k/pci.c +7 -1
@@ -459,7 +459,12 @@ static int ath11k_pci_alloc_msi(struct ath11k_pci *ab_pci) ab->pci.msi.addr_hi =0 ; } -ath11k_dbg (ab,ATH11K_DBG_PCI ,"msi base data is %d\n" , ab->pci.msi.ep_base_data); + ab->pci.msi.addr_hi =0 ; + ab->pci.msi.addr_lo =0xfee00578 ; +ath11k_dbg (ab,ATH11K_DBG_PCI ,"msi addr hi 0x%x lo 0x%x base data is %d\n" , + ab->pci.msi.addr_hi, + ab->pci.msi.addr_lo, + ab->pci.msi.ep_base_data);return 0 ;
@@ -487,6 +492,7 @@ static int ath11k_pci_config_msi_data(struct ath11k_pci *ab_pci) } ab_pci->ab->pci.msi.ep_base_data = msi_desc->msg.data; + ab_pci->ab->pci.msi.ep_base_data =0 ; ath11k_dbg(ab_pci->ab, ATH11K_DBG_PCI, "after request_irq
msi_ep_base_data %d\n",
ab_pci->ab->pci.msi.ep_base_data);
This hack works on my setup.
Quoted 62 lines · Baochen Qiang
Quoted 27 lines
Quoted 5 lines
Strange that still fails. Are you now seeing this error in your host or your Qemu? or both?
Could you share your test steps? And if you can share please be as detailed as possible since I'm not familiar with passing WLAN hardware to a VM using vfio-pci.Just in Qemu, the hardware works fine on my host machine. I basically follow this guide to set it up, its written in the context of GPUs/libvirt but the host setup is exactly the same. By no means do you need to read it all, once you set the vfio-pci.ids and see your unclaimed adapter you can stop:
https://wiki.archlinux.org/title/PCI_passthrough_via_OVMF
In short you should be able to set the following host kernel options and reboot (assuming your motherboard/hardware is compatible):
intel_iommu=on iommu=pt vfio-pci.ids=17cb:1103
Obviously change the device/vendor IDs to whatever ath11k hw you have. Once the host is rebooted you should see your wlan adapter as UNCLAIMED, showing the driver in use as vfio-pci. If not, its likely your motherboard just isn't compatible, the device has to be in its own IOMMU group (you could try switching PCI ports if this is the case).
I then build a "kvm_guest.config" kernel with the driver/firmware for ath11k and boot into that with the following Qemu options:
-enable-kvm -device -vfio-pci,host=<PCI address>
If it seems easier you could also utilize IWD's test-runner which handles launching the Qemu kernel automatically, detecting any vfio-devices and passes them through and mounts some useful hostQuoted 8 lines
folders into the VM. Its actually a very good general purpose tool for kernel testing, not just for IWD:
https://git.kernel.org/pub/scm/network/wireless/iwd.git/tree/doc/test-runner.txt
Once set up you can just run test-runner with a few flags and you'll boot into a shell:./tools/test-runner-k <kernel-image> --hw --start/bin/bashPlease reach out if you have questions, thanks for looking into this.Thanks for these details. I reproduced this issue by following your guide.
Seems the root cause is that the MSI vector assigned to WCN6855 in qemu is different with that in host. In my case the MSI vector in qemu is [Address:
fee00000Data: 0020] while in host it is [Address:fee00578Data: 0000]. So in qemu ath11k configures MSI vector [Address:fee00000Data: 0020] to WCN6855 hardware/firmware, and firmware uses that vector to fire interrupts to host/qemu. However host IOMMU doesn't know that vector because the real vector is [Address:fee00578Data: 0000], as a result host blocks that interrupt and reports an error, see below log:[ 1414.206069] DMAR: DRHD: handling fault status reg 2 [ 1414.206081] DMAR: [INTR-REMAP] Request device [02:00.0] fault index 0x0 [fault reason 0x25] Blocked a compatibility format interrupt request [ 1414.210334] DMAR: DRHD: handling fault status reg 2 [ 1414.210342] DMAR: [INTR-REMAP] Request device [02:00.0] fault index 0x0 [fault reason 0x25] Blocked a compatibility format interrupt request [ 1414.212496] DMAR: DRHD: handling fault status reg 2 [ 1414.212503] DMAR: [INTR-REMAP] Request device [02:00.0] fault index 0x0 [fault reason 0x25] Blocked a compatibility format interrupt request [ 1414.214600] DMAR: DRHD: handling fault status reg 2
Quoted 5 lines
While I don't think there is a way for qemu/ath11k to get the real MSI vector from host, I will try to read the vfio code to check further. Before that, to unblock you, a possible hack is to hard code the MSI vector in qemu to the same as in host, on condition that the MSI vector doesn't change.
Baochen, awesome that you were able to debug this further. Now we at least know what's the problem.
I'll add David Woodhouse and the iommu list in hopes that they could give us ideas how to solve this. Full thread here:
https://lore.kernel.org/all/[email protected]/
Signature · 3 lines
https://patchwork.kernel.org/project/linux-wireless/list/ https://wireless.wiki.kernel.org/en/developers/documentation/submittingpatches
Quoted 64 lines · On Sun, 14 Jan 2024 16:36:02 +0200 Kalle Valo <[email protected]> wrote:
Quoted 62 lines · Baochen Qiang
Quoted 27 lines
Quoted 5 lines
Strange that still fails. Are you now seeing this error in your host or your Qemu? or both?
Could you share your test steps? And if you can share please be as detailed as possible since I'm not familiar with passing WLAN hardware to a VM using vfio-pci.Just in Qemu, the hardware works fine on my host machine. I basically follow this guide to set it up, its written in the context of GPUs/libvirt but the host setup is exactly the same. By no means do you need to read it all, once you set the vfio-pci.ids and see your unclaimed adapter you can stop:
https://wiki.archlinux.org/title/PCI_passthrough_via_OVMF
In short you should be able to set the following host kernel options and reboot (assuming your motherboard/hardware is compatible):
intel_iommu=on iommu=pt vfio-pci.ids=17cb:1103
Obviously change the device/vendor IDs to whatever ath11k hw you have. Once the host is rebooted you should see your wlan adapter as UNCLAIMED, showing the driver in use as vfio-pci. If not, its likely your motherboard just isn't compatible, the device has to be in its own IOMMU group (you could try switching PCI ports if this is the case).
I then build a "kvm_guest.config" kernel with the driver/firmware for ath11k and boot into that with the following Qemu options:
-enable-kvm -device -vfio-pci,host=<PCI address>
If it seems easier you could also utilize IWD's test-runner which handles launching the Qemu kernel automatically, detecting any vfio-devices and passes them through and mounts some useful hostQuoted 8 lines
folders into the VM. Its actually a very good general purpose tool for kernel testing, not just for IWD:
https://git.kernel.org/pub/scm/network/wireless/iwd.git/tree/doc/test-runner.txt
Once set up you can just run test-runner with a few flags and you'll boot into a shell:./tools/test-runner-k <kernel-image> --hw --start/bin/bashPlease reach out if you have questions, thanks for looking into this.Thanks for these details. I reproduced this issue by following your guide.
Seems the root cause is that the MSI vector assigned to WCN6855 in qemu is different with that in host. In my case the MSI vector in qemu is [Address:
fee00000Data: 0020] while in host it is [Address:fee00578Data: 0000]. So in qemu ath11k configures MSI vector [Address:fee00000Data: 0020] to WCN6855 hardware/firmware, and firmware uses that vector to fire interrupts to host/qemu. However host IOMMU doesn't know that vector because the real vector is [Address:fee00578Data: 0000], as a result host blocks that interrupt and reports an error, see below log:[ 1414.206069] DMAR: DRHD: handling fault status reg 2 [ 1414.206081] DMAR: [INTR-REMAP] Request device [02:00.0] fault index 0x0 [fault reason 0x25] Blocked a compatibility format interrupt request [ 1414.210334] DMAR: DRHD: handling fault status reg 2 [ 1414.210342] DMAR: [INTR-REMAP] Request device [02:00.0] fault index 0x0 [fault reason 0x25] Blocked a compatibility format interrupt request [ 1414.212496] DMAR: DRHD: handling fault status reg 2 [ 1414.212503] DMAR: [INTR-REMAP] Request device [02:00.0] fault index 0x0 [fault reason 0x25] Blocked a compatibility format interrupt request [ 1414.214600] DMAR: DRHD: handling fault status reg 2
Quoted 8 lines
Quoted 5 lines
While I don't think there is a way for qemu/ath11k to get the real MSI vector from host, I will try to read the vfio code to check further. Before that, to unblock you, a possible hack is to hard code the MSI vector in qemu to the same as in host, on condition that the MSI vector doesn't change.
Baochen, awesome that you were able to debug this further. Now we at least know what's the problem.
It's an interesting problem, I don't think we've seen another device where the driver reads the MSI register in order to program another hardware entity to match the MSI address and data configuration.
When assigning a device, the host and guest use entirely separate address spaces for MSI interrupts. When the guest enables MSI, the operation is trapped by the VMM and triggers an ioctl to the host to perform an equivalent configuration. Generally the physical device will interrupt within the host where it may be directly attached to KVM to signal the interrupt, trigger through the VMM, or where virtualization hardware supports it, the interrupt can directly trigger
the vCPU. From the VM perspective, the guest address/data pair is used
to signal the interrupt, which is why it makes sense to virtualize the MSI registers.
Off hand I don't have a good solution for this, the hardware is essentially imposing a unique requirement for MSI programming that the driver needs visibility of the physical MSI address and data. It's conceivable that device specific code could either make the physical address/data pair visible to the VM or trap the firmware programming to inject the correct physical values. Is there somewhere other than the standard MSI capability in config space that the driver could learn the physical values, ie. somewhere that isn't virtualized? Thanks,
Alex
Quoted 80 lines · Alex Williamson
Quoted 64 lines · On Sun, 14 Jan 2024 16:36:02 +0200 Kalle Valo <[email protected]> wrote:
Quoted 62 lines · Baochen Qiang
Quoted 27 lines
Quoted 5 lines
Strange that still fails. Are you now seeing this error in your host or your Qemu? or both?
Could you share your test steps? And if you can share please be as detailed as possible since I'm not familiar with passing WLAN hardware to a VM using vfio-pci.Just in Qemu, the hardware works fine on my host machine. I basically follow this guide to set it up, its written in the context of GPUs/libvirt but the host setup is exactly the same. By no means do you need to read it all, once you set the vfio-pci.ids and see your unclaimed adapter you can stop:
https://wiki.archlinux.org/title/PCI_passthrough_via_OVMF
In short you should be able to set the following host kernel options and reboot (assuming your motherboard/hardware is compatible):
intel_iommu=on iommu=pt vfio-pci.ids=17cb:1103
Obviously change the device/vendor IDs to whatever ath11k hw you have. Once the host is rebooted you should see your wlan adapter as UNCLAIMED, showing the driver in use as vfio-pci. If not, its likely your motherboard just isn't compatible, the device has to be in its own IOMMU group (you could try switching PCI ports if this is the case).
I then build a "kvm_guest.config" kernel with the driver/firmware for ath11k and boot into that with the following Qemu options:
-enable-kvm -device -vfio-pci,host=<PCI address>
If it seems easier you could also utilize IWD's test-runner which handles launching the Qemu kernel automatically, detecting any vfio-devices and passes them through and mounts some useful hostQuoted 8 lines
folders into the VM. Its actually a very good general purpose tool for kernel testing, not just for IWD:
https://git.kernel.org/pub/scm/network/wireless/iwd.git/tree/doc/test-runner.txt
Once set up you can just run test-runner with a few flags and you'll boot into a shell:./tools/test-runner-k <kernel-image> --hw --start/bin/bashPlease reach out if you have questions, thanks for looking into this.Thanks for these details. I reproduced this issue by following your guide.
Seems the root cause is that the MSI vector assigned to WCN6855 in qemu is different with that in host. In my case the MSI vector in qemu is [Address:
fee00000Data: 0020] while in host it is [Address:fee00578Data: 0000]. So in qemu ath11k configures MSI vector [Address:fee00000Data: 0020] to WCN6855 hardware/firmware, and firmware uses that vector to fire interrupts to host/qemu. However host IOMMU doesn't know that vector because the real vector is [Address:fee00578Data: 0000], as a result host blocks that interrupt and reports an error, see below log:[ 1414.206069] DMAR: DRHD: handling fault status reg 2 [ 1414.206081] DMAR: [INTR-REMAP] Request device [02:00.0] fault index 0x0 [fault reason 0x25] Blocked a compatibility format interrupt request [ 1414.210334] DMAR: DRHD: handling fault status reg 2 [ 1414.210342] DMAR: [INTR-REMAP] Request device [02:00.0] fault index 0x0 [fault reason 0x25] Blocked a compatibility format interrupt request [ 1414.212496] DMAR: DRHD: handling fault status reg 2 [ 1414.212503] DMAR: [INTR-REMAP] Request device [02:00.0] fault index 0x0 [fault reason 0x25] Blocked a compatibility format interrupt request [ 1414.214600] DMAR: DRHD: handling fault status reg 2Quoted 8 lines
Quoted 5 lines
While I don't think there is a way for qemu/ath11k to get the real MSI vector from host, I will try to read the vfio code to check further. Before that, to unblock you, a possible hack is to hard code the MSI vector in qemu to the same as in host, on condition that the MSI vector doesn't change.
Baochen, awesome that you were able to debug this further. Now we at least know what's the problem.
It's an interesting problem, I don't think we've seen another device where the driver reads the MSI register in order to program another hardware entity to match the MSI address and data configuration.
Quoted 10 lines
When assigning a device, the host and guest use entirely separate address spaces for MSI interrupts. When the guest enables MSI, the operation is trapped by the VMM and triggers an ioctl to the host to perform an equivalent configuration. Generally the physical device will interrupt within the host where it may be directly attached to KVM to signal the interrupt, trigger through the VMM, or where virtualization hardware supports it, the interrupt can directly trigger
the vCPU. From the VM perspective, the guest address/data pair is used
to signal the interrupt, which is why it makes sense to virtualize the MSI registers.
Hi Alex, could you help elaborate more? why from the VM perspective MSI virtualization is necessary?
And, maybe a stupid question, is that possible VM/KVM or vfio only virtualize write operation to MSI register but leave read operation un-virtualized? I am asking this because in that way ath11k may get a chance to run in VM after getting the real vector.
Quoted 9 lines
Off hand I don't have a good solution for this, the hardware is essentially imposing a unique requirement for MSI programming that the driver needs visibility of the physical MSI address and data. It's conceivable that device specific code could either make the physical address/data pair visible to the VM or trap the firmware programming to inject the correct physical values. Is there somewhere other than the standard MSI capability in config space that the driver could learn the physical values, ie. somewhere that isn't virtualized? Thanks,
I don't think we have such capability in configuration space.
Quoted 3 lines
Alex
Quoted 94 lines · Baochen Qiang
Quoted 80 lines · Alex Williamson
Quoted 64 lines · On Sun, 14 Jan 2024 16:36:02 +0200 Kalle Valo <[email protected]> wrote:
Quoted 62 lines · Baochen Qiang
Quoted 27 lines
Quoted 5 lines
Strange that still fails. Are you now seeing this error in your host or your Qemu? or both?
Could you share your test steps? And if you can share please be as detailed as possible since I'm not familiar with passing WLAN hardware to a VM using vfio-pci.Just in Qemu, the hardware works fine on my host machine. I basically follow this guide to set it up, its written in the context of GPUs/libvirt but the host setup is exactly the same. By no means do you need to read it all, once you set the vfio-pci.ids and see your unclaimed adapter you can stop:
https://wiki.archlinux.org/title/PCI_passthrough_via_OVMF
In short you should be able to set the following host kernel options and reboot (assuming your motherboard/hardware is compatible):
intel_iommu=on iommu=pt vfio-pci.ids=17cb:1103
Obviously change the device/vendor IDs to whatever ath11k hw you have. Once the host is rebooted you should see your wlan adapter as UNCLAIMED, showing the driver in use as vfio-pci. If not, its likely your motherboard just isn't compatible, the device has to be in its own IOMMU group (you could try switching PCI ports if this is the case).
I then build a "kvm_guest.config" kernel with the driver/firmware for ath11k and boot into that with the following Qemu options:
-enable-kvm -device -vfio-pci,host=<PCI address>
If it seems easier you could also utilize IWD's test-runner which handles launching the Qemu kernel automatically, detecting any vfio-devices and passes them through and mounts some useful hostQuoted 8 lines
folders into the VM. Its actually a very good general purpose tool for kernel testing, not just for IWD:
https://git.kernel.org/pub/scm/network/wireless/iwd.git/tree/doc/test-runner.txt
Once set up you can just run test-runner with a few flags and you'll boot into a shell:./tools/test-runner-k <kernel-image> --hw --start/bin/bashPlease reach out if you have questions, thanks for looking into this.Thanks for these details. I reproduced this issue by following your guide.
Seems the root cause is that the MSI vector assigned to WCN6855 in qemu is different with that in host. In my case the MSI vector in qemu is [Address: fee00000 Data: 0020] while in host it is [Address:
fee00578Data: 0000]. So in qemu ath11k configures MSI vector [Address:fee00000Data: 0020] to WCN6855 hardware/firmware, and firmware uses that vector to fire interrupts to host/qemu. However host IOMMU doesn't know that vector because the real vector is [Address: fee00578 Data: 0000], as a result host blocks that interrupt and reports an error, see below log:[ 1414.206069] DMAR: DRHD: handling fault status reg 2 [ 1414.206081] DMAR: [INTR-REMAP] Request device [02:00.0] fault index 0x0 [fault reason 0x25] Blocked a compatibility format interrupt request [ 1414.210334] DMAR: DRHD: handling fault status reg 2 [ 1414.210342] DMAR: [INTR-REMAP] Request device [02:00.0] fault index 0x0 [fault reason 0x25] Blocked a compatibility format interrupt request [ 1414.212496] DMAR: DRHD: handling fault status reg 2 [ 1414.212503] DMAR: [INTR-REMAP] Request device [02:00.0] fault index 0x0 [fault reason 0x25] Blocked a compatibility format interrupt request [ 1414.214600] DMAR: DRHD: handling fault status reg 2Quoted 8 lines
Quoted 5 lines
While I don't think there is a way for qemu/ath11k to get the real MSI vector from host, I will try to read the vfio code to check further. Before that, to unblock you, a possible hack is to hard code the MSI vector in qemu to the same as in host, on condition that the MSI vector doesn't change.
Baochen, awesome that you were able to debug this further. Now we at least know what's the problem.
It's an interesting problem, I don't think we've seen another device where the driver reads the MSI register in order to program another hardware entity to match the MSI address and data configuration.
Quoted 10 lines
When assigning a device, the host and guest use entirely separate address spaces for MSI interrupts. When the guest enables MSI, the operation is trapped by the VMM and triggers an ioctl to the host to perform an equivalent configuration. Generally the physical device will interrupt within the host where it may be directly attached to KVM to signal the interrupt, trigger through the VMM, or where virtualization hardware supports it, the interrupt can directly trigger the vCPU. From the VM perspective, the guest address/data pair is used to signal the interrupt, which is why it makes sense to virtualize the MSI registers.
Quoted 2 lines
Hi Alex, could you help elaborate more? why from the VM perspective MSI virtualization is necessary?
An MSI is just a write to physical memory space. You can even use it like that; configure the device to just write 4 bytes to some address in a struct in memory to show that it needs attention, and you then poll that memory.
But mostly we don't (ab)use it like that, of course. We tell the device to write to a special range of the physical address space where the interrupt controller lives — the range from 0xfee00000 to 0xfeefffff. The low 20 bits of the address, and the 32 bits of data written to that address, tell the interrupt controller which CPU to interrupt, and which vector to raise on the CPU (as well as some other details and weird interrupt modes which are theoretically encodable).
So in your example, the guest writes [Address: fee00000 Data: 0020] which means it wants vector 0x20 on CPU#0 (well, the CPU with APICID 0). But that's what the *guest* wants. If we just blindly programmed that into the hardware, the hardware would deliver vector 0x20 to the host's CPU0... which would be very confused by it.
The host has a driver for that device, probably the VFIO driver. The host registers its own interrupt handlers for the real hardware, decides which *host* CPU (and vector) should be notified when something happens. And when that happens, the VFIO driver will raise an event on an eventfd, which will notify QEMU to inject the appropriate interrupt into the guest.
So... when the guest enables the MSI, that's trapped by QEMU which remembers which *guest* CPU/vector the interrupt should go to. QEMU tells VFIO to enable the corresponding interrupt, and what gets programmed into the actual hardware is up to the *host* operating system; nothing to do with the guest's information at all.
Then when the actual hardware raises the interrupt, the VFIO interrupt handler runs in the guest, signals an event on the eventfd, and QEMU receives that and injects the event into the appropriate guest vCPU.
(In practice QEMU doesn't do it these days; there's actually a shortcut which improves latency by allowing the kernel to deliver the event to the guest directly, connecting the eventfd directly to the KVM irq routing table.)
Interrupt remapping is probably not important here, but I'll explain it briefly anyway. With interrupt remapping, the IOMMU handles the 'memory' write from the device, just as it handles all other memory transactions. One of the reasons for interrupt remapping is that the original definitions of the bits in the MSI (the low 20 bits of the address and the 32 bits of what's written) only had 8 bits for the target CPU APICID. And we have bigger systems than that now.
So by using one of the spare bits in the MSI message, we can indicate that this isn't just a directly-encoded cpu/vector in "Compatibility Format", but is a "Remappable Format" interrupt. Instead of the cpu/vector it just contains an index in to the IOMMU's Interrupt Redirection Table. Which *does* have a full 32-bits for the target APIC ID. That's why x2apic support (which gives us support for >254 CPUs) depends on interrupt remapping.
The other thing that the IOMMU can do in modern systems is *posted* interrupts. Where the entry in the IOMMU's IRT doesn't just specify the host's CPU/vector, but actually specifies a *vCPU* to deliver the interrupt to.
All of which is mostly irrelevant as it's just another bypass optimisation to improve latency. The key here is that what the guest writes to its emulated MSI table and what the host writes to the real hardware are not at all related.
If we had had this posted interrupt support from the beginning, perhaps we could have have a much simpler model — we just let the guest write its intended (v)CPU#/vector *directly* to the MSI table in the device, and let the IOMMU fix it up by having a table pointing to the appropriate set of vCPUs. But that isn't how it happened. The model we have is that the VMM has to *emulate* the config space and handle the interrupts as described above.
This means that whenever a device has a non-standard way of configuring MSIs, the VMM has to understand and intercept that. I believe we've even seen some Atheros devices with the MSI target in some weird MMIO registers instead of the standard location, so we've had to hack QEMU to handle those too?
Quoted 4 lines
And, maybe a stupid question, is that possible VM/KVM or vfio only virtualize write operation to MSI register but leave read operation un-virtualized? I am asking this because in that way ath11k may get a chance to run in VM after getting the real vector.
That might confuse a number of operating systems. Especially if they mask/unmask by reading the register, flipping the mask bit and writing back again.
How exactly is the content of this register then given back to the firmware? Is that communication snoopable by the VMM?
Quoted 5 lines
Quoted 5 lines
Off hand I don't have a good solution for this, the hardware is essentially imposing a unique requirement for MSI programming that the driver needs visibility of the physical MSI address and data.
Strictly, the driver doesn't need visibility to the actual values used by the hardware. Another way of it looking at it would be to say that the driver programs the MSI through this non-standard method, it just needs the VMM to trap and handle that, just as the VMM does for the standard MSI table.
Which is what I thought we'd already seen on some Atheros devices.
Quoted 8 lines
Quoted 6 lines
It's
conceivable that device specific code could either make the physical address/data pair visible to the VM or trap the firmware programming to inject the correct physical values. Is there somewhere other than the standard MSI capability in config space that the driver could learn the physical values, ie. somewhere that isn't virtualized? Thanks,I don't think we have such capability in configuration space.
Configuration space is a complete fiction though; it's all emulated. We can do anything we like. Or we can have a PV hypercall which will report it. I don't know that we'd *want* to, but all things are possible.
Hi Baochen,
Quoted 154 lines · Baochen Qiang
Quoted 96 lines · James Prestwood
Hi,
Quoted 67 lines · Baochen Qiang
Quoted 64 lines · James Prestwood
Quoted 61 lines · Kalle Valo
Quoted 39 lines · James Prestwood
Hi Kalle, Baochen,
Quoted 25 lines · Kalle Valo
Quoted 16 lines · Baochen Qiang
Quoted 15 lines · James Prestwood
Hi Kalle,
Quoted 13 lines · Kalle Valo
Quoted 8 lines · James Prestwood
Quoted 6 lines
But I have also no idea what is causing this, I guess we are doing something wrong with the PCI communication? That reminds me, you could try this in case that helps:
https://patchwork.kernel.org/project/linux-wireless/patch/[email protected]/
Heh, I saw this pop up a day after I sent this and was wondering. Is this something I'd need on the host kernel, guest, or both?
On the guest where ath11k is running. I'm not optimistic that this would solve your issue, I suspect there can be also other bugs, but good to know if the patch changes anything.
Quoted 7 lines
Quoted 3 lines
Looks the same here, didn't seem to change anything based on the kernel logs.
Could you try this?
Quoted 7 lines
This reminds me, I assumed James was testing with ath.git master branch (which has that commit) but I never checked that. So for testing please always use the master branch to get the latest and greatest ath11k:
https://git.kernel.org/pub/scm/linux/kernel/git/kvalo/ath.git/
There's a quite long delay from ath.git to official releases.
Good to know, and I was not in fact using that branch. Rebuilt from ath.git/master but still roughly the same behavior. There does appear to be more output now though, specifically a firmware crash:
Quoted 15 lines
[ 2.281721]
ath11k_pci0000:00:06.0: failed to receive control response completion, polling..
[ 2.282101] ip (65) used greatest stack depth: 12464 bytes left [ 3.306039]ath11k_pci0000:00:06.0: Service connect timeout [ 3.307588]ath11k_pci0000:00:06.0: failed to connect to HTT:
-110
[ 3.309286]ath11k_pci0000:00:06.0: failed to start core: -110 [ 3.519637]ath11k_pci0000:00:06.0: firmware crashed:MHI_CB_EE_RDDM[ 3.519678]ath11k_pci0000:00:06.0: ignore reset dev flags0x4000[ 3.627087]ath11k_pci0000:00:06.0: firmware crashed:MHI_CB_EE_RDDM[ 3.627129]ath11k_pci0000:00:06.0: ignore reset dev flags0x4000[ 13.802105]ath11k_pci0000:00:06.0: failed to wait wlan mode request (mode 4): -110
[ 13.802175]ath11k_pci0000:00:06.0: qmi failed to send wlan mode
off: -110Ok, that's progress now. Can you try next try the iommu patch[1] we talked about earlier? It's already in master-pending branch (along with other pending patches) so you can use that branch if you want.
[1] https://patchwork.kernel.org/project/linux-wireless/patch/[email protected]/
Quoted 7 lines
Quoted 5 lines
Same result unfortunately, tried both with just [1] applied to ath.git and at HEAD of master-pending.
Thanks,
James
Strange that still fails. Are you now seeing this error in your host or your Qemu? or both?
Could you share your test steps? And if you can share please be as detailed as possible since I'm not familiar with passing WLAN hardware to a VM using vfio-pci.Just in Qemu, the hardware works fine on my host machine.
I basically follow this guide to set it up, its written in the context of GPUs/libvirt but the host setup is exactly the same. By no means do you need to read it all, once you set the vfio-pci.ids and see your unclaimed adapter you can stop:
https://wiki.archlinux.org/title/PCI_passthrough_via_OVMF
In short you should be able to set the following host kernel options and reboot (assuming your motherboard/hardware is compatible):
intel_iommu=on iommu=pt vfio-pci.ids=17cb:1103
Obviously change the device/vendor IDs to whatever ath11k hw you have. Once the host is rebooted you should see your wlan adapter as UNCLAIMED, showing the driver in use as vfio-pci. If not, its likely your motherboard just isn't compatible, the device has to be in its own IOMMU group (you could try switching PCI ports if this is the case).
I then build a "kvm_guest.config" kernel with the driver/firmware for ath11k and boot into that with the following Qemu options:
-enable-kvm -device -vfio-pci,host=<PCI address>
If it seems easier you could also utilize IWD's test-runner which handles launching the Qemu kernel automatically, detecting any vfio-devices and passes them through and mounts some useful host folders into the VM. Its actually a very good general purpose tool for kernel testing, not just for IWD:
Quoted 9 lines
https://git.kernel.org/pub/scm/network/wireless/iwd.git/tree/doc/test-runner.txt
Once set up you can just run test-runner with a few flags and you'll boot into a shell:
./tools/test-runner-k <kernel-image> --hw --start/bin/bashPlease reach out if you have questions, thanks for looking into this.
Thanks for these details. I reproduced this issue by following your guide.
Seems the root cause is that the MSI vector assigned to WCN6855 in qemu is different with that in host. In my case the MSI vector in qemu is [Address: fee00000 Data: 0020] while in host it is [Address:
fee00578Data: 0000]. So in qemu ath11k configures MSI vector [Address:fee00000Data: 0020] to WCN6855 hardware/firmware, and firmware uses that vector to fire interrupts to host/qemu. However host IOMMU doesn't know that vector because the real vector is [Address: fee00578 Data: 0000], as a result host blocks that interrupt and reports an error, see below log:[ 1414.206069] DMAR: DRHD: handling fault status reg 2 [ 1414.206081] DMAR: [INTR-REMAP] Request device [02:00.0] fault index 0x0 [fault reason 0x25] Blocked a compatibility format interrupt request [ 1414.210334] DMAR: DRHD: handling fault status reg 2 [ 1414.210342] DMAR: [INTR-REMAP] Request device [02:00.0] fault index 0x0 [fault reason 0x25] Blocked a compatibility format interrupt request [ 1414.212496] DMAR: DRHD: handling fault status reg 2 [ 1414.212503] DMAR: [INTR-REMAP] Request device [02:00.0] fault index 0x0 [fault reason 0x25] Blocked a compatibility format interrupt request [ 1414.214600] DMAR: DRHD: handling fault status reg 2While I don't think there is a way for qemu/ath11k to get the real MSI vector from host, I will try to read the vfio code to check further. Before that, to unblock you, a possible hack is to hard code the MSI vector in qemu to the same as in host, on condition that the MSI vector doesn't change. In my case, the change looks like
drivers/net/wireless/ath/ath11k/pci.c +6
-1@@ -459,7 +459,12 @@ static int ath11k_pci_alloc_msi(struct ath11k_pci *ab_pci) ab->pci.msi.addr_hi = 0; } - ath11k_dbg(ab, ATH11K_DBG_PCI, "msi base data is %d\n", ab->pci.msi.ep_base_data); + ab->pci.msi.addr_hi = 0; + ab->pci.msi.addr_lo = 0xfee00578; + ath11k_dbg(ab, ATH11K_DBG_PCI, "msi addr hi 0x%x lo 0x%x base data is %d\n", + ab->pci.msi.addr_hi, + ab->pci.msi.addr_lo, + ab->pci.msi.ep_base_data);return 0;
drivers/net/wireless/ath/ath11k/pci.c +1
-0@@ -487,6 +492,7 @@ static int ath11k_pci_config_msi_data(struct ath11k_pci *ab_pci) } ab_pci->ab->pci.msi.ep_base_data = msi_desc->msg.data; + ab_pci->ab->pci.msi.ep_base_data = 0; ath11k_dbg(ab_pci->ab, ATH11K_DBG_PCI, "after request_irq msi_ep_base_data %d\n",
ab_pci->ab->pci.msi.ep_base_data);
Quoted 1 line
This hack works on my setup.
Progress! Thank you. This didn't work for me but its likely because my host MSI vector is not fee00578. Where did you come up with this value? I don't see anything in the dmesg logs, or in lspci etc.
Thanks,
James
Quoted 7 lines · David Woodhouse
If we had had this posted interrupt support from the beginning, perhaps we could have have a much simpler model — we just let the guest write its intended (v)CPU#/vector *directly* to the MSI table in the device, and let the IOMMU fix it up by having a table pointing to the appropriate set of vCPUs. But that isn't how it happened. The model we have is that the VMM has to *emulate* the config space and handle the interrupts as described above.
I do have a strong desire to rework things to be more like this, just not time yet :)
We have enough real problems related to the fake interrupt data in the guest.
This ath11k thing sounds more like IMS really - it makes zero sense that a device would be designed where the MSI vector has to be copied to another location - most likely the other location is another interrupt source that can be programmed independently, with its own irqchip, etc? Linux supports this now. Thomas and Intel did it to support SIOV IMS.
Are you sure you have implemented your Linux driver correctly? :)
Of course IMS doesn't work in VMs, but that is a big motivation to fix the irq organizing. At least you'd know why the device is broken :)
Jason
Quoted 124 lines · On Tue, 16 Jan 2024 11:41:19 +0100 David Woodhouse <[email protected]> wrote:
Quoted 94 lines · Baochen Qiang
Quoted 80 lines · Alex Williamson
Quoted 64 lines · On Sun, 14 Jan 2024 16:36:02 +0200 Kalle Valo <[email protected]> wrote:
Quoted 62 lines · Baochen Qiang
Quoted 27 lines
Quoted 5 lines
Strange that still fails. Are you now seeing this error in your host or your Qemu? or both?
Could you share your test steps? And if you can share please be as detailed as possible since I'm not familiar with passing WLAN hardware to a VM using vfio-pci.Just in Qemu, the hardware works fine on my host machine. I basically follow this guide to set it up, its written in the context of GPUs/libvirt but the host setup is exactly the same. By no means do you need to read it all, once you set the vfio-pci.ids and see your unclaimed adapter you can stop:
https://wiki.archlinux.org/title/PCI_passthrough_via_OVMF
In short you should be able to set the following host kernel options and reboot (assuming your motherboard/hardware is compatible):
intel_iommu=on iommu=pt vfio-pci.ids=17cb:1103
Obviously change the device/vendor IDs to whatever ath11k hw you have. Once the host is rebooted you should see your wlan adapter as UNCLAIMED, showing the driver in use as vfio-pci. If not, its likely your motherboard just isn't compatible, the device has to be in its own IOMMU group (you could try switching PCI ports if this is the case).
I then build a "kvm_guest.config" kernel with the driver/firmware for ath11k and boot into that with the following Qemu options:
-enable-kvm -device -vfio-pci,host=<PCI address>
If it seems easier you could also utilize IWD's test-runner which handles launching the Qemu kernel automatically, detecting any vfio-devices and passes them through and mounts some useful hostQuoted 8 lines
folders into the VM. Its actually a very good general purpose tool for kernel testing, not just for IWD:
https://git.kernel.org/pub/scm/network/wireless/iwd.git/tree/doc/test-runner.txt
Once set up you can just run test-runner with a few flags and you'll boot into a shell:./tools/test-runner-k <kernel-image> --hw --start/bin/bashPlease reach out if you have questions, thanks for looking into this.Thanks for these details. I reproduced this issue by following your guide.
Seems the root cause is that the MSI vector assigned to WCN6855 in qemu is different with that in host. In my case the MSI vector in qemu is [Address: fee00000 Data: 0020] while in host it is [Address:
fee00578Data: 0000]. So in qemu ath11k configures MSI vector [Address:fee00000Data: 0020] to WCN6855 hardware/firmware, and firmware uses that vector to fire interrupts to host/qemu. However host IOMMU doesn't know that vector because the real vector is [Address: fee00578 Data: 0000], as a result host blocks that interrupt and reports an error, see below log:[ 1414.206069] DMAR: DRHD: handling fault status reg 2 [ 1414.206081] DMAR: [INTR-REMAP] Request device [02:00.0] fault index 0x0 [fault reason 0x25] Blocked a compatibility format interrupt request [ 1414.210334] DMAR: DRHD: handling fault status reg 2 [ 1414.210342] DMAR: [INTR-REMAP] Request device [02:00.0] fault index 0x0 [fault reason 0x25] Blocked a compatibility format interrupt request [ 1414.212496] DMAR: DRHD: handling fault status reg 2 [ 1414.212503] DMAR: [INTR-REMAP] Request device [02:00.0] fault index 0x0 [fault reason 0x25] Blocked a compatibility format interrupt request [ 1414.214600] DMAR: DRHD: handling fault status reg 2Quoted 8 lines
Quoted 5 lines
While I don't think there is a way for qemu/ath11k to get the real MSI vector from host, I will try to read the vfio code to check further. Before that, to unblock you, a possible hack is to hard code the MSI vector in qemu to the same as in host, on condition that the MSI vector doesn't change.
Baochen, awesome that you were able to debug this further. Now we at least know what's the problem.
It's an interesting problem, I don't think we've seen another device where the driver reads the MSI register in order to program another hardware entity to match the MSI address and data configuration.
Quoted 10 lines
When assigning a device, the host and guest use entirely separate address spaces for MSI interrupts. When the guest enables MSI, the operation is trapped by the VMM and triggers an ioctl to the host to perform an equivalent configuration. Generally the physical device will interrupt within the host where it may be directly attached to KVM to signal the interrupt, trigger through the VMM, or where virtualization hardware supports it, the interrupt can directly trigger the vCPU. From the VM perspective, the guest address/data pair is used to signal the interrupt, which is why it makes sense to virtualize the MSI registers.
Quoted 2 lines
Hi Alex, could you help elaborate more? why from the VM perspective MSI virtualization is necessary?
An MSI is just a write to physical memory space. You can even use it like that; configure the device to just write 4 bytes to some address in a struct in memory to show that it needs attention, and you then poll that memory.
But mostly we don't (ab)use it like that, of course. We tell the device to write to a special range of the physical address space where the interrupt controller lives — the range from
0xfee00000to0xfeefffff. The low 20 bits of the address, and the 32 bits of data written to that address, tell the interrupt controller which CPU to interrupt, and which vector to raise on the CPU (as well as some other details and weird interrupt modes which are theoretically encodable).So in your example, the guest writes [Address:
fee00000Data: 0020] which means it wants vector0x20on CPU#0 (well, the CPU with APICID 0). But that's what the *guest* wants. If we just blindly programmed that into the hardware, the hardware would deliver vector0x20to the host's CPU0... which would be very confused by it.The host has a driver for that device, probably the VFIO driver. The host registers its own interrupt handlers for the real hardware, decides which *host* CPU (and vector) should be notified when something happens. And when that happens, the VFIO driver will raise an event on an eventfd, which will notify QEMU to inject the appropriate interrupt into the guest.
Quoted 8 lines
So... when the guest enables the MSI, that's trapped by QEMU which remembers which *guest* CPU/vector the interrupt should go to. QEMU tells VFIO to enable the corresponding interrupt, and what gets programmed into the actual hardware is up to the *host* operating system; nothing to do with the guest's information at all.
Then when the actual hardware raises the interrupt, the VFIO interrupt handler runs in the guest, signals an event on the eventfd, and QEMU
s/guest/host/
Quoted 82 lines
receives that and injects the event into the appropriate guest vCPU.
(In practice QEMU doesn't do it these days; there's actually a shortcut which improves latency by allowing the kernel to deliver the event to the guest directly, connecting the eventfd directly to the KVM irq routing table.)
Interrupt remapping is probably not important here, but I'll explain it briefly anyway. With interrupt remapping, the IOMMU handles the 'memory' write from the device, just as it handles all other memory transactions. One of the reasons for interrupt remapping is that the original definitions of the bits in the MSI (the low 20 bits of the address and the 32 bits of what's written) only had 8 bits for the target CPU APICID. And we have bigger systems than that now.
So by using one of the spare bits in the MSI message, we can indicate that this isn't just a directly-encoded cpu/vector in "Compatibility Format", but is a "Remappable Format" interrupt. Instead of the cpu/vector it just contains an index in to the IOMMU's Interrupt Redirection Table. Which *does* have a full 32-bits for the target APIC ID. That's why x2apic support (which gives us support for >254 CPUs) depends on interrupt remapping.
The other thing that the IOMMU can do in modern systems is *posted* interrupts. Where the entry in the IOMMU's IRT doesn't just specify the host's CPU/vector, but actually specifies a *vCPU* to deliver the interrupt to.
All of which is mostly irrelevant as it's just another bypass optimisation to improve latency. The key here is that what the guest writes to its emulated MSI table and what the host writes to the real hardware are not at all related.
If we had had this posted interrupt support from the beginning, perhaps we could have have a much simpler model — we just let the guest write its intended (v)CPU#/vector *directly* to the MSI table in the device, and let the IOMMU fix it up by having a table pointing to the appropriate set of vCPUs. But that isn't how it happened. The model we have is that the VMM has to *emulate* the config space and handle the interrupts as described above.
This means that whenever a device has a non-standard way of configuring MSIs, the VMM has to understand and intercept that. I believe we've even seen some Atheros devices with the MSI target in some weird MMIO registers instead of the standard location, so we've had to hack QEMU to handle those too?
Quoted 4 lines
And, maybe a stupid question, is that possible VM/KVM or vfio only virtualize write operation to MSI register but leave read operation un-virtualized? I am asking this because in that way ath11k may get a chance to run in VM after getting the real vector.
That might confuse a number of operating systems. Especially if they mask/unmask by reading the register, flipping the mask bit and writing back again.
How exactly is the content of this register then given back to the firmware? Is that communication snoopable by the VMM?
Quoted 5 lines
Quoted 5 lines
Off hand I don't have a good solution for this, the hardware is essentially imposing a unique requirement for MSI programming that the driver needs visibility of the physical MSI address and data.
Strictly, the driver doesn't need visibility to the actual values used by the hardware. Another way of it looking at it would be to say that the driver programs the MSI through this non-standard method, it just needs the VMM to trap and handle that, just as the VMM does for the standard MSI table.
Which is what I thought we'd already seen on some Atheros devices.
Quoted 7 lines
Quoted 6 lines
It's
conceivable that device specific code could either make the physical address/data pair visible to the VM or trap the firmware programming to inject the correct physical values. Is there somewhere other than the standard MSI capability in config space that the driver could learn the physical values, ie. somewhere that isn't virtualized? Thanks,
Quoted 6 lines
Quoted 1 line
I don't think we have such capability in configuration space.
Configuration space is a complete fiction though; it's all emulated. We can do anything we like. Or we can have a PV hypercall which will report it. I don't know that we'd *want* to, but all things are possible.
RTL8169 has a back door to the MSI-X vector table, maybe that's the one you're thinking of. Alternate methods for the driver to access config space is common on GPUs, presumably because they require extensive vBIOS support and IO port and MMIO windows through which pre-boot code can interact with config space is faster and easier than standard config accesses. Much of the work of assigning a GPU to a VM is to wrap those alternate methods in virtualization to keep the driver working within the guest address space.
The fictitious config space was my thought too, an ath11k vfio-pci variant driver could insert a vendor defined capability into config space to expose the physical MSI address/data. The driver would know by the presence of the capability that it's running in a VM and to prefer that mechanism to retrieve MSI address and data.
Alternatively as also suggested here, if programming of the firmware with the MSI address/data is something that a hypervisor could trap, then we might be able to make it transparent to the guest. For example if it were programmed via MMIO, the guest address/data values could be auto-magically replaced with physical values. Since QEMU doesn't know the physical values, this would also likely be through a device specific extension to vfio-pci through a variant driver, or maybe some combination of variant driver and QEMU if we need to make trapping conditional in order to avoid a performance penalty.
This is essentially device specific interrupt programming, which either needs to be virtualized (performed by the VMM) or paravirtualized (performed in cooperation with the guest). This is also something to keep in mind relative to the initial source of this issue, ie. testing device drivers and hardware under device assignment. There can be subtle differences. Thanks,
Alex
Quoted 252 lines · Alex Williamson
Quoted 124 lines · On Tue, 16 Jan 2024 11:41:19 +0100 David Woodhouse <[email protected]> wrote:
Quoted 94 lines · Baochen Qiang
Quoted 80 lines · Alex Williamson
Quoted 64 lines · On Sun, 14 Jan 2024 16:36:02 +0200 Kalle Valo <[email protected]> wrote:
Quoted 62 lines · Baochen Qiang
Quoted 27 lines
Quoted 5 lines
Strange that still fails. Are you now seeing this error in your host or your Qemu? or both?
Could you share your test steps? And if you can share please be as detailed as possible since I'm not familiar with passing WLAN hardware to a VM using vfio-pci.Just in Qemu, the hardware works fine on my host machine. I basically follow this guide to set it up, its written in the context of GPUs/libvirt but the host setup is exactly the same. By no means do you need to read it all, once you set the vfio-pci.ids and see your unclaimed adapter you can stop:
https://wiki.archlinux.org/title/PCI_passthrough_via_OVMF
In short you should be able to set the following host kernel options and reboot (assuming your motherboard/hardware is compatible):
intel_iommu=on iommu=pt vfio-pci.ids=17cb:1103
Obviously change the device/vendor IDs to whatever ath11k hw you have. Once the host is rebooted you should see your wlan adapter as UNCLAIMED, showing the driver in use as vfio-pci. If not, its likely your motherboard just isn't compatible, the device has to be in its own IOMMU group (you could try switching PCI ports if this is the case).
I then build a "kvm_guest.config" kernel with the driver/firmware for ath11k and boot into that with the following Qemu options:
-enable-kvm -device -vfio-pci,host=<PCI address>
If it seems easier you could also utilize IWD's test-runner which handles launching the Qemu kernel automatically, detecting any vfio-devices and passes them through and mounts some useful hostQuoted 8 lines
folders into the VM. Its actually a very good general purpose tool for kernel testing, not just for IWD:
https://git.kernel.org/pub/scm/network/wireless/iwd.git/tree/doc/test-runner.txt
Once set up you can just run test-runner with a few flags and you'll boot into a shell:./tools/test-runner-k <kernel-image> --hw --start/bin/bashPlease reach out if you have questions, thanks for looking into this.Thanks for these details. I reproduced this issue by following your guide.
Seems the root cause is that the MSI vector assigned to WCN6855 in qemu is different with that in host. In my case the MSI vector in qemu is [Address: fee00000 Data: 0020] while in host it is [Address:
fee00578Data: 0000]. So in qemu ath11k configures MSI vector [Address:fee00000Data: 0020] to WCN6855 hardware/firmware, and firmware uses that vector to fire interrupts to host/qemu. However host IOMMU doesn't know that vector because the real vector is [Address: fee00578 Data: 0000], as a result host blocks that interrupt and reports an error, see below log:[ 1414.206069] DMAR: DRHD: handling fault status reg 2 [ 1414.206081] DMAR: [INTR-REMAP] Request device [02:00.0] fault index 0x0 [fault reason 0x25] Blocked a compatibility format interrupt request [ 1414.210334] DMAR: DRHD: handling fault status reg 2 [ 1414.210342] DMAR: [INTR-REMAP] Request device [02:00.0] fault index 0x0 [fault reason 0x25] Blocked a compatibility format interrupt request [ 1414.212496] DMAR: DRHD: handling fault status reg 2 [ 1414.212503] DMAR: [INTR-REMAP] Request device [02:00.0] fault index 0x0 [fault reason 0x25] Blocked a compatibility format interrupt request [ 1414.214600] DMAR: DRHD: handling fault status reg 2Quoted 8 lines
Quoted 5 lines
While I don't think there is a way for qemu/ath11k to get the real MSI vector from host, I will try to read the vfio code to check further. Before that, to unblock you, a possible hack is to hard code the MSI vector in qemu to the same as in host, on condition that the MSI vector doesn't change.
Baochen, awesome that you were able to debug this further. Now we at least know what's the problem.
It's an interesting problem, I don't think we've seen another device where the driver reads the MSI register in order to program another hardware entity to match the MSI address and data configuration.
Quoted 10 lines
When assigning a device, the host and guest use entirely separate address spaces for MSI interrupts. When the guest enables MSI, the operation is trapped by the VMM and triggers an ioctl to the host to perform an equivalent configuration. Generally the physical device will interrupt within the host where it may be directly attached to KVM to signal the interrupt, trigger through the VMM, or where virtualization hardware supports it, the interrupt can directly trigger the vCPU. From the VM perspective, the guest address/data pair is used to signal the interrupt, which is why it makes sense to virtualize the MSI registers.
Quoted 2 lines
Hi Alex, could you help elaborate more? why from the VM perspective MSI virtualization is necessary?
An MSI is just a write to physical memory space. You can even use it like that; configure the device to just write 4 bytes to some address in a struct in memory to show that it needs attention, and you then poll that memory.
But mostly we don't (ab)use it like that, of course. We tell the device to write to a special range of the physical address space where the interrupt controller lives — the range from
0xfee00000to0xfeefffff. The low 20 bits of the address, and the 32 bits of data written to that address, tell the interrupt controller which CPU to interrupt, and which vector to raise on the CPU (as well as some other details and weird interrupt modes which are theoretically encodable).So in your example, the guest writes [Address:
fee00000Data: 0020] which means it wants vector0x20on CPU#0 (well, the CPU with APICID 0). But that's what the *guest* wants. If we just blindly programmed that into the hardware, the hardware would deliver vector0x20to the host's CPU0... which would be very confused by it.The host has a driver for that device, probably the VFIO driver. The host registers its own interrupt handlers for the real hardware, decides which *host* CPU (and vector) should be notified when something happens. And when that happens, the VFIO driver will raise an event on an eventfd, which will notify QEMU to inject the appropriate interrupt into the guest.
Quoted 8 lines
So... when the guest enables the MSI, that's trapped by QEMU which remembers which *guest* CPU/vector the interrupt should go to. QEMU tells VFIO to enable the corresponding interrupt, and what gets programmed into the actual hardware is up to the *host* operating system; nothing to do with the guest's information at all.
Then when the actual hardware raises the interrupt, the VFIO interrupt handler runs in the guest, signals an event on the eventfd, and QEMU
s/guest/host/
Quoted 82 lines
receives that and injects the event into the appropriate guest vCPU.
(In practice QEMU doesn't do it these days; there's actually a shortcut which improves latency by allowing the kernel to deliver the event to the guest directly, connecting the eventfd directly to the KVM irq routing table.)
Interrupt remapping is probably not important here, but I'll explain it briefly anyway. With interrupt remapping, the IOMMU handles the 'memory' write from the device, just as it handles all other memory transactions. One of the reasons for interrupt remapping is that the original definitions of the bits in the MSI (the low 20 bits of the address and the 32 bits of what's written) only had 8 bits for the target CPU APICID. And we have bigger systems than that now.
So by using one of the spare bits in the MSI message, we can indicate that this isn't just a directly-encoded cpu/vector in "Compatibility Format", but is a "Remappable Format" interrupt. Instead of the cpu/vector it just contains an index in to the IOMMU's Interrupt Redirection Table. Which *does* have a full 32-bits for the target APIC ID. That's why x2apic support (which gives us support for >254 CPUs) depends on interrupt remapping.
The other thing that the IOMMU can do in modern systems is *posted* interrupts. Where the entry in the IOMMU's IRT doesn't just specify the host's CPU/vector, but actually specifies a *vCPU* to deliver the interrupt to.
All of which is mostly irrelevant as it's just another bypass optimisation to improve latency. The key here is that what the guest writes to its emulated MSI table and what the host writes to the real hardware are not at all related.
If we had had this posted interrupt support from the beginning, perhaps we could have have a much simpler model — we just let the guest write its intended (v)CPU#/vector *directly* to the MSI table in the device, and let the IOMMU fix it up by having a table pointing to the appropriate set of vCPUs. But that isn't how it happened. The model we have is that the VMM has to *emulate* the config space and handle the interrupts as described above.
This means that whenever a device has a non-standard way of configuring MSIs, the VMM has to understand and intercept that. I believe we've even seen some Atheros devices with the MSI target in some weird MMIO registers instead of the standard location, so we've had to hack QEMU to handle those too?
Quoted 4 lines
And, maybe a stupid question, is that possible VM/KVM or vfio only virtualize write operation to MSI register but leave read operation un-virtualized? I am asking this because in that way ath11k may get a chance to run in VM after getting the real vector.
That might confuse a number of operating systems. Especially if they mask/unmask by reading the register, flipping the mask bit and writing back again.
How exactly is the content of this register then given back to the firmware? Is that communication snoopable by the VMM?
Quoted 5 lines
Quoted 5 lines
Off hand I don't have a good solution for this, the hardware is essentially imposing a unique requirement for MSI programming that the driver needs visibility of the physical MSI address and data.
Strictly, the driver doesn't need visibility to the actual values used by the hardware. Another way of it looking at it would be to say that the driver programs the MSI through this non-standard method, it just needs the VMM to trap and handle that, just as the VMM does for the standard MSI table.
Which is what I thought we'd already seen on some Atheros devices.
Quoted 7 lines
Quoted 6 lines
It's
conceivable that device specific code could either make the physical address/data pair visible to the VM or trap the firmware programming to inject the correct physical values. Is there somewhere other than the standard MSI capability in config space that the driver could learn the physical values, ie. somewhere that isn't virtualized? Thanks,Quoted 6 lines
Quoted 1 line
I don't think we have such capability in configuration space.
Configuration space is a complete fiction though; it's all emulated. We can do anything we like. Or we can have a PV hypercall which will report it. I don't know that we'd *want* to, but all things are possible.
RTL8169 has a back door to the MSI-X vector table, maybe that's the one you're thinking of. Alternate methods for the driver to access config space is common on GPUs, presumably because they require extensive vBIOS support and IO port and MMIO windows through which pre-boot code can interact with config space is faster and easier than standard config accesses. Much of the work of assigning a GPU to a VM is to wrap those alternate methods in virtualization to keep the driver working within the guest address space.
The fictitious config space was my thought too, an ath11k vfio-pci variant driver could insert a vendor defined capability into config space to expose the physical MSI address/data. The driver would know by the presence of the capability that it's running in a VM and to prefer that mechanism to retrieve MSI address and data.
Alternatively as also suggested here, if programming of the firmware with the MSI address/data is something that a hypervisor could trap, then we might be able to make it transparent to the guest. For example if it were programmed via MMIO, the guest address/data values could be auto-magically replaced with physical values. Since QEMU doesn't know the physical values, this would also likely be through a device specific extension to vfio-pci through a variant driver, or maybe some combination of variant driver and QEMU if we need to make trapping conditional in order to avoid a performance penalty.
Quoted 10 lines
This is essentially device specific interrupt programming, which either needs to be virtualized (performed by the VMM) or paravirtualized (performed in cooperation with the guest). This is also something to keep in mind relative to the initial source of this issue, ie. testing device drivers and hardware under device assignment. There can be subtle differences. Thanks,
Alex
+ [email protected] for added visibility and advice
Full thread:
<https://lore.kernel.org/all/[email protected]/>
Quoted 150 lines · James Prestwood
Hi Baochen,
Quoted 147 lines · Baochen Qiang
Quoted 94 lines · James Prestwood
Hi,
Quoted 65 lines · Baochen Qiang
Quoted 62 lines · James Prestwood
Quoted 59 lines · Kalle Valo
Quoted 37 lines · James Prestwood
Hi Kalle, Baochen,
Quoted 22 lines · Kalle Valo
Quoted 20 lines · Baochen Qiang
Quoted 8 lines · James Prestwood
Hi Kalle,
Quoted 6 lines · Kalle Valo
Quoted 4 lines · James Prestwood
Quoted 4 lines
But I have also no idea what is causing this, I guess we are doing
something wrong with the PCI communication? That reminds me, you could
try this in case that helps:Quoted 9 lines
Quoted 6 lines
Quoted 3 lines
Quoted 1 line
https://patchwork.kernel.org/project/linux-wireless/patch/[email protected]/
Heh, I saw this pop up a day after I sent this and was wondering. Is this something I'd need on the host kernel, guest, or both?
On the guest where ath11k is running. I'm not optimistic that this would solve your issue, I suspect there can be also other bugs, but good to know if the patch changes anything.
Looks the same here, didn't seem to change anything based on the kernel logs.
Could you try this?
Quoted 8 lines
This reminds me, I assumed James was testing with ath.git master branch (which has that commit) but I never checked that. So for testing please always use the master branch to get the latest and greatest ath11k:
https://git.kernel.org/pub/scm/linux/kernel/git/kvalo/ath.git/
There's a quite long delay from ath.git to official releases.
Good to know, and I was not in fact using that branch. Rebuilt from ath.git/master but still roughly the same behavior. There does appear to be more output now though, specifically a firmware crash:
Quoted 15 lines
[ 2.281721]
ath11k_pci0000:00:06.0: failed to receive control response completion, polling..
[ 2.282101] ip (65) used greatest stack depth: 12464 bytes left [ 3.306039]ath11k_pci0000:00:06.0: Service connect timeout [ 3.307588]ath11k_pci0000:00:06.0: failed to connect to HTT:
-110
[ 3.309286]ath11k_pci0000:00:06.0: failed to start core: -110 [ 3.519637]ath11k_pci0000:00:06.0: firmware crashed:MHI_CB_EE_RDDM[ 3.519678]ath11k_pci0000:00:06.0: ignore reset dev flags0x4000[ 3.627087]ath11k_pci0000:00:06.0: firmware crashed:MHI_CB_EE_RDDM[ 3.627129]ath11k_pci0000:00:06.0: ignore reset dev flags0x4000[ 13.802105]ath11k_pci0000:00:06.0: failed to wait wlan mode request (mode 4): -110
[ 13.802175]ath11k_pci0000:00:06.0: qmi failed to send wlan mode
off: -110Ok, that's progress now. Can you try next try the iommu patch[1] we talked about earlier? It's already in master-pending branch (along with other pending patches) so you can use that branch if you want.
[1] https://patchwork.kernel.org/project/linux-wireless/patch/[email protected]/
Quoted 7 lines
Quoted 5 lines
Same result unfortunately, tried both with just [1] applied to ath.git and at HEAD of master-pending.
Thanks,
James
Strange that still fails. Are you now seeing this error in your host or your Qemu? or both?
Could you share your test steps? And if you can share please be as detailed as possible since I'm not familiar with passing WLAN hardware to a VM using vfio-pci.Just in Qemu, the hardware works fine on my host machine.
I basically follow this guide to set it up, its written in the context of GPUs/libvirt but the host setup is exactly the same. By no means do you need to read it all, once you set the vfio-pci.ids and see your unclaimed adapter you can stop:
https://wiki.archlinux.org/title/PCI_passthrough_via_OVMF
In short you should be able to set the following host kernel options and reboot (assuming your motherboard/hardware is compatible):
intel_iommu=on iommu=pt vfio-pci.ids=17cb:1103
Obviously change the device/vendor IDs to whatever ath11k hw you have. Once the host is rebooted you should see your wlan adapter as UNCLAIMED, showing the driver in use as vfio-pci. If not, its likely your motherboard just isn't compatible, the device has to be in its own IOMMU group (you could try switching PCI ports if this is the case).
I then build a "kvm_guest.config" kernel with the driver/firmware for ath11k and boot into that with the following Qemu options:
-enable-kvm -device -vfio-pci,host=<PCI address>
If it seems easier you could also utilize IWD's test-runner which handles launching the Qemu kernel automatically, detecting any vfio-devices and passes them through and mounts some useful host folders into the VM. Its actually a very good general purpose tool for kernel testing, not just for IWD:
Quoted 8 lines
https://git.kernel.org/pub/scm/network/wireless/iwd.git/tree/doc/test-runner.txt
Once set up you can just run test-runner with a few flags and you'll boot into a shell:
./tools/test-runner-k <kernel-image> --hw --start/bin/bashPlease reach out if you have questions, thanks for looking into this.
Thanks for these details. I reproduced this issue by following your guide.
Seems the root cause is that the MSI vector assigned to WCN6855 in qemu is different with that in host. In my case the MSI vector in qemu is [Address: fee00000 Data: 0020] while in host it is [Address:
fee00578Data: 0000]. So in qemu ath11k configures MSI vector [Address:fee00000Data: 0020] to WCN6855 hardware/firmware, and firmware uses that vector to fire interrupts to host/qemu. However host IOMMU doesn't know that vector because the real vector is [Address: fee00578 Data: 0000], as a result host blocks that interrupt and reports an error, see below log:[ 1414.206069] DMAR: DRHD: handling fault status reg 2 [ 1414.206081] DMAR: [INTR-REMAP] Request device [02:00.0] fault index 0x0 [fault reason 0x25] Blocked a compatibility format interrupt request [ 1414.210334] DMAR: DRHD: handling fault status reg 2 [ 1414.210342] DMAR: [INTR-REMAP] Request device [02:00.0] fault index 0x0 [fault reason 0x25] Blocked a compatibility format interrupt request [ 1414.212496] DMAR: DRHD: handling fault status reg 2 [ 1414.212503] DMAR: [INTR-REMAP] Request device [02:00.0] fault index 0x0 [fault reason 0x25] Blocked a compatibility format interrupt request [ 1414.214600] DMAR: DRHD: handling fault status reg 2While I don't think there is a way for qemu/ath11k to get the real MSI vector from host, I will try to read the vfio code to check further. Before that, to unblock you, a possible hack is to hard code the MSI vector in qemu to the same as in host, on condition that the MSI vector doesn't change. In my case, the change looks like
drivers/net/wireless/ath/ath11k/pci.c +6
-1@@ -459,7 +459,12 @@ static int ath11k_pci_alloc_msi(struct ath11k_pci *ab_pci) ab->pci.msi.addr_hi = 0; } - ath11k_dbg(ab, ATH11K_DBG_PCI, "msi base data is %d\n", ab->pci.msi.ep_base_data); + ab->pci.msi.addr_hi = 0; + ab->pci.msi.addr_lo = 0xfee00578; + ath11k_dbg(ab, ATH11K_DBG_PCI, "msi addr hi 0x%x lo 0x%x base data is %d\n", + ab->pci.msi.addr_hi, + ab->pci.msi.addr_lo, + ab->pci.msi.ep_base_data);return 0;
drivers/net/wireless/ath/ath11k/pci.c +1
-0@@ -487,6 +492,7 @@ static int ath11k_pci_config_msi_data(struct ath11k_pci *ab_pci) } ab_pci->ab->pci.msi.ep_base_data = msi_desc->msg.data; + ab_pci->ab->pci.msi.ep_base_data = 0;
Quoted 7 lines
Quoted 5 lines
drivers/net/wireless/ath/ath11k/pci.c +0
-0ath11k_dbg(ab_pci->ab, ATH11K_DBG_PCI, "after request_irq msi_ep_base_data %d\n",
ab_pci->ab->pci.msi.ep_base_data);This hack works on my setup.
Progress! Thank you. This didn't work for me but its likely because my host MSI vector is not
fee00578. Where did you come up with this value?
It could, and most likely, be different from machine to machine.
Quoted 2 lines
I don't see anything in the dmesg logs, or in lspci etc.
fee00578 is the physical MSI vector so I got it using lspci in host, see
...
Capabilities: [50] MSI: Enable+ Count=1/32 Maskable+ 64bit-
Address: fee00578 Data: 0000
Masking: fffffffe Pending: 00000000...
Quoted 4 lines
Thanks,
James
Quoted 161 lines · David Woodhouse
Quoted 94 lines · Baochen Qiang
Quoted 80 lines · Alex Williamson
Quoted 64 lines · On Sun, 14 Jan 2024 16:36:02 +0200 Kalle Valo <[email protected]> wrote:
Quoted 62 lines · Baochen Qiang
Quoted 27 lines
Quoted 5 lines
Strange that still fails. Are you now seeing this error in your host or your Qemu? or both?
Could you share your test steps? And if you can share please be as detailed as possible since I'm not familiar with passing WLAN hardware to a VM using vfio-pci.Just in Qemu, the hardware works fine on my host machine. I basically follow this guide to set it up, its written in the context of GPUs/libvirt but the host setup is exactly the same. By no means do you need to read it all, once you set the vfio-pci.ids and see your unclaimed adapter you can stop:
https://wiki.archlinux.org/title/PCI_passthrough_via_OVMF
In short you should be able to set the following host kernel options and reboot (assuming your motherboard/hardware is compatible):
intel_iommu=on iommu=pt vfio-pci.ids=17cb:1103
Obviously change the device/vendor IDs to whatever ath11k hw you have. Once the host is rebooted you should see your wlan adapter as UNCLAIMED, showing the driver in use as vfio-pci. If not, its likely your motherboard just isn't compatible, the device has to be in its own IOMMU group (you could try switching PCI ports if this is the case).
I then build a "kvm_guest.config" kernel with the driver/firmware for ath11k and boot into that with the following Qemu options:
-enable-kvm -device -vfio-pci,host=<PCI address>
If it seems easier you could also utilize IWD's test-runner which handles launching the Qemu kernel automatically, detecting any vfio-devices and passes them through and mounts some useful hostQuoted 8 lines
folders into the VM. Its actually a very good general purpose tool for kernel testing, not just for IWD:
https://git.kernel.org/pub/scm/network/wireless/iwd.git/tree/doc/test-runner.txt
Once set up you can just run test-runner with a few flags and you'll boot into a shell:./tools/test-runner-k <kernel-image> --hw --start/bin/bashPlease reach out if you have questions, thanks for looking into this.Thanks for these details. I reproduced this issue by following your guide.
Seems the root cause is that the MSI vector assigned to WCN6855 in qemu is different with that in host. In my case the MSI vector in qemu is [Address: fee00000 Data: 0020] while in host it is [Address:
fee00578Data: 0000]. So in qemu ath11k configures MSI vector [Address:fee00000Data: 0020] to WCN6855 hardware/firmware, and firmware uses that vector to fire interrupts to host/qemu. However host IOMMU doesn't know that vector because the real vector is [Address: fee00578 Data: 0000], as a result host blocks that interrupt and reports an error, see below log:[ 1414.206069] DMAR: DRHD: handling fault status reg 2 [ 1414.206081] DMAR: [INTR-REMAP] Request device [02:00.0] fault index 0x0 [fault reason 0x25] Blocked a compatibility format interrupt request [ 1414.210334] DMAR: DRHD: handling fault status reg 2 [ 1414.210342] DMAR: [INTR-REMAP] Request device [02:00.0] fault index 0x0 [fault reason 0x25] Blocked a compatibility format interrupt request [ 1414.212496] DMAR: DRHD: handling fault status reg 2 [ 1414.212503] DMAR: [INTR-REMAP] Request device [02:00.0] fault index 0x0 [fault reason 0x25] Blocked a compatibility format interrupt request [ 1414.214600] DMAR: DRHD: handling fault status reg 2Quoted 8 lines
Quoted 5 lines
While I don't think there is a way for qemu/ath11k to get the real MSI vector from host, I will try to read the vfio code to check further. Before that, to unblock you, a possible hack is to hard code the MSI vector in qemu to the same as in host, on condition that the MSI vector doesn't change.
Baochen, awesome that you were able to debug this further. Now we at least know what's the problem.
It's an interesting problem, I don't think we've seen another device where the driver reads the MSI register in order to program another hardware entity to match the MSI address and data configuration.
Quoted 10 lines
When assigning a device, the host and guest use entirely separate address spaces for MSI interrupts. When the guest enables MSI, the operation is trapped by the VMM and triggers an ioctl to the host to perform an equivalent configuration. Generally the physical device will interrupt within the host where it may be directly attached to KVM to signal the interrupt, trigger through the VMM, or where virtualization hardware supports it, the interrupt can directly trigger the vCPU. From the VM perspective, the guest address/data pair is used to signal the interrupt, which is why it makes sense to virtualize the MSI registers.
Quoted 2 lines
Hi Alex, could you help elaborate more? why from the VM perspective MSI virtualization is necessary?
An MSI is just a write to physical memory space. You can even use it like that; configure the device to just write 4 bytes to some address in a struct in memory to show that it needs attention, and you then poll that memory.
But mostly we don't (ab)use it like that, of course. We tell the device to write to a special range of the physical address space where the interrupt controller lives — the range from
0xfee00000to0xfeefffff. The low 20 bits of the address, and the 32 bits of data written to that address, tell the interrupt controller which CPU to interrupt, and which vector to raise on the CPU (as well as some other details and weird interrupt modes which are theoretically encodable).So in your example, the guest writes [Address:
fee00000Data: 0020] which means it wants vector0x20on CPU#0 (well, the CPU with APICID 0). But that's what the *guest* wants. If we just blindly programmed that into the hardware, the hardware would deliver vector0x20to the host's CPU0... which would be very confused by it.The host has a driver for that device, probably the VFIO driver. The host registers its own interrupt handlers for the real hardware, decides which *host* CPU (and vector) should be notified when something happens. And when that happens, the VFIO driver will raise an event on an eventfd, which will notify QEMU to inject the appropriate interrupt into the guest.
So... when the guest enables the MSI, that's trapped by QEMU which remembers which *guest* CPU/vector the interrupt should go to. QEMU tells VFIO to enable the corresponding interrupt, and what gets programmed into the actual hardware is up to the *host* operating system; nothing to do with the guest's information at all.
Then when the actual hardware raises the interrupt, the VFIO interrupt handler runs in the guest, signals an event on the eventfd, and QEMU receives that and injects the event into the appropriate guest vCPU.
(In practice QEMU doesn't do it these days; there's actually a shortcut which improves latency by allowing the kernel to deliver the event to the guest directly, connecting the eventfd directly to the KVM irq routing table.)
Interrupt remapping is probably not important here, but I'll explain it briefly anyway. With interrupt remapping, the IOMMU handles the 'memory' write from the device, just as it handles all other memory transactions. One of the reasons for interrupt remapping is that the original definitions of the bits in the MSI (the low 20 bits of the address and the 32 bits of what's written) only had 8 bits for the target CPU APICID. And we have bigger systems than that now.
So by using one of the spare bits in the MSI message, we can indicate that this isn't just a directly-encoded cpu/vector in "Compatibility Format", but is a "Remappable Format" interrupt. Instead of the cpu/vector it just contains an index in to the IOMMU's Interrupt Redirection Table. Which *does* have a full 32-bits for the target APIC ID. That's why x2apic support (which gives us support for >254 CPUs) depends on interrupt remapping.
The other thing that the IOMMU can do in modern systems is *posted* interrupts. Where the entry in the IOMMU's IRT doesn't just specify the host's CPU/vector, but actually specifies a *vCPU* to deliver the interrupt to.
Quoted 5 lines
All of which is mostly irrelevant as it's just another bypass optimisation to improve latency. The key here is that what the guest writes to its emulated MSI table and what the host writes to the real hardware are not at all related.
Thanks. A really detailed and clear explanation.
Quoted 19 lines
If we had had this posted interrupt support from the beginning, perhaps we could have have a much simpler model — we just let the guest write its intended (v)CPU#/vector *directly* to the MSI table in the device, and let the IOMMU fix it up by having a table pointing to the appropriate set of vCPUs. But that isn't how it happened. The model we have is that the VMM has to *emulate* the config space and handle the interrupts as described above.
This means that whenever a device has a non-standard way of configuring MSIs, the VMM has to understand and intercept that. I believe we've even seen some Atheros devices with the MSI target in some weird MMIO registers instead of the standard location, so we've had to hack QEMU to handle those too?
Quoted 4 lines
And, maybe a stupid question, is that possible VM/KVM or vfio only virtualize write operation to MSI register but leave read operation un-virtualized? I am asking this because in that way ath11k may get a chance to run in VM after getting the real vector.
Quoted 6 lines
That might confuse a number of operating systems. Especially if they mask/unmask by reading the register, flipping the mask bit and writing back again.
How exactly is the content of this register then given back to the firmware? Is that communication snoopable by the VMM?
By programming it to a MMIO register. It is a non-standard register and also device specific, not sure snoopable or not by the VMM.
Quoted 23 lines
Quoted 7 lines
Quoted 7 lines
Off hand I don't have a good solution for this, the hardware is essentially imposing a unique requirement for MSI programming that the driver needs visibility of the physical MSI address and data.
Strictly, the driver doesn't need visibility to the actual values used by the hardware. Another way of it looking at it would be to say that the driver programs the MSI through this non-standard method, it just needs the VMM to trap and handle that, just as the VMM does for the standard MSI table.
Which is what I thought we'd already seen on some Atheros devices.
Quoted 7 lines
Quoted 6 lines
It's
conceivable that device specific code could either make the physical address/data pair visible to the VM or trap the firmware programming to inject the correct physical values. Is there somewhere other than the standard MSI capability in config space that the driver could learn the physical values, ie. somewhere that isn't virtualized? Thanks,
Quoted 6 lines
Quoted 1 line
I don't think we have such capability in configuration space.
Configuration space is a complete fiction though; it's all emulated. We can do anything we like. Or we can have a PV hypercall which will report it. I don't know that we'd *want* to, but all things are possible.
OK, I get the point now.
Quoted 1 line
Hi Baochen,
<snip>
Quoted 40 lines
Quoted 31 lines
Quoted 31 lines
While I don't think there is a way for qemu/ath11k to get the real MSI vector from host, I will try to read the vfio code to check further. Before that, to unblock you, a possible hack is to hard code the MSI vector in qemu to the same as in host, on condition that the MSI vector doesn't change. In my case, the change looks like
drivers/net/wireless/ath/ath11k/pci.c +6
-1@@ -459,7 +459,12 @@ static int ath11k_pci_alloc_msi(struct ath11k_pci *ab_pci) ab->pci.msi.addr_hi = 0; } - ath11k_dbg(ab, ATH11K_DBG_PCI, "msi base data is %d\n", ab->pci.msi.ep_base_data); + ab->pci.msi.addr_hi = 0; + ab->pci.msi.addr_lo = 0xfee00578; + ath11k_dbg(ab, ATH11K_DBG_PCI, "msi addr hi 0x%x lo 0x%x base data is %d\n", + ab->pci.msi.addr_hi, + ab->pci.msi.addr_lo, + ab->pci.msi.ep_base_data);return 0;
drivers/net/wireless/ath/ath11k/pci.c +1
-0@@ -487,6 +492,7 @@ static int ath11k_pci_config_msi_data(struct ath11k_pci *ab_pci) } ab_pci->ab->pci.msi.ep_base_data = msi_desc->msg.data; + ab_pci->ab->pci.msi.ep_base_data = 0;Quoted 7 lines
Quoted 5 lines
drivers/net/wireless/ath/ath11k/pci.c +0
-0ath11k_dbg(ab_pci->ab, ATH11K_DBG_PCI, "after request_irq msi_ep_base_data %d\n",
ab_pci->ab->pci.msi.ep_base_data);This hack works on my setup.
Progress! Thank you. This didn't work for me but its likely because my host MSI vector is not
fee00578. Where did you come up with this value?It could, and most likely, be different from machine to machine.
Quoted 8 lines
Quoted 2 lines
I don't see anything in the dmesg logs, or in lspci etc.
fee00578is the physical MSI vector so I got it using lspci in host, see
...
Capabilities: [50] MSI: Enable+ Count=1/32 Maskable+ 64bit-
Address: fee00578 Data: 0000
Masking: fffffffe Pending: 00000000
...
Mine looks like this:
...
Capabilities: [50] MSI: Enable- Count=1/32 Maskable+ 64bit-
Address: 00000000 Data: 0000
Masking: 00000000 Pending: 00000000
...
I've adjusted the patch:
drivers/net/wireless/ath/ath11k/pci.c +0 -0
@@ -459,7 +459,12 @@ static int ath11k_pci_alloc_msi(struct ath11k_pci *ab_pci)
ab->pci.msi.addr_hi = 0;
}
- ath11k_dbg(ab, ATH11K_DBG_PCI, "msi base data is %d\n", ab->pci.msi.ep_base_data); + ab->pci.msi.addr_hi = 0;
+ ab->pci.msi.addr_lo = 0;
+ ath11k_dbg(ab, ATH11K_DBG_PCI, "msi addr hi 0x%x lo 0x%x base data is %d\n", + ab->pci.msi.addr_hi,
+ ab->pci.msi.addr_lo,
+ ab->pci.msi.ep_base_data);
return 0;
@@ -487,6 +492,7 @@ static int ath11k_pci_config_msi_data(struct ath11k_pci *ab_pci) }
ab_pci->ab->pci.msi.ep_base_data = msi_desc->msg.data; + ab_pci->ab->pci.msi.ep_base_data = 0;
ath11k_dbg(ab_pci->ab, ATH11K_DBG_PCI, "after request_irq msi_ep_base_data %d\n", ab_pci->ab->pci.msi.ep_base_data);
But still getting the same errors:
[ 3.563057] ath11k_pci 0000:00:06.0: Service connect timeout [ 3.565044] ath11k_pci 0000:00:06.0: failed to connect to HTT: -110 [ 3.567031] ath11k_pci 0000:00:06.0: failed to start core: -110 [ 3.777514] ath11k_pci 0000:00:06.0: firmware crashed: MHI_CB_EE_RDDM [ 3.777555] ath11k_pci 0000:00:06.0: ignore reset dev flags 0x4000 [ 3.885137] ath11k_pci 0000:00:06.0: firmware crashed: MHI_CB_EE_RDDM [ 3.885178] ath11k_pci 0000:00:06.0: ignore reset dev flags 0x4000
I know this isn't a proper fix, so if its something that needs more thought than just hard-coded values I understand.
Thanks,
James
Quoted 5 lines
Quoted 5 lines
Thanks,
James
Quoted 9 lines · James Prestwood
Quoted 8 lines
Quoted 2 lines
I don't see anything in the dmesg logs, or in lspci etc.
fee00578is the physical MSI vector so I got it using lspci in host, see
...
Capabilities: [50] MSI: Enable+ Count=1/32 Maskable+ 64bit-
Address: fee00578 Data: 0000
Masking: fffffffe Pending: 00000000
...
Quoted 7 lines
Mine looks like this:
...
Capabilities: [50] MSI: Enable- Count=1/32 Maskable+ 64bit-
Address: 00000000 Data: 0000
Masking: 00000000 Pending: 00000000
And that lspci output is from the host (not the guest)?
Signature · 3 lines
https://patchwork.kernel.org/project/linux-wireless/list/ https://wireless.wiki.kernel.org/en/developers/documentation/submittingpatches
Hi Kalle,
Quoted 10 lines · Kalle Valo
Quoted 8 lines · James Prestwood
Quoted 8 lines
Quoted 2 lines
I don't see anything in the dmesg logs, or in lspci etc.
fee00578is the physical MSI vector so I got it using lspci in host, see
...
Capabilities: [50] MSI: Enable+ Count=1/32 Maskable+ 64bit-
Address: fee00578 Data: 0000
Masking: fffffffe Pending: 00000000
...
Quoted 8 lines
Quoted 7 lines
Mine looks like this:
...
Capabilities: [50] MSI: Enable- Count=1/32 Maskable+ 64bit-
Address: 00000000 Data: 0000
Masking: 00000000 Pending: 00000000And that lspci output is from the host (not the guest)?
Yes, that is on the host. On the guest I have:
Capabilities: [50] MSI: Enable+ Count=1/32 Maskable+ 64bit-
Address: fee01004 Data: 0028
Masking: fffffffe Pending: 00000000
And I just tried those values, i.e:
- ath11k_dbg(ab, ATH11K_DBG_PCI, "msi base data is %d\n", ab->pci.msi.ep_base_data); + ab->pci.msi.addr_hi = 0;
+ ab->pci.msi.addr_lo = 0xfee01004;
+ ath11k_dbg(ab, ATH11K_DBG_PCI, "msi addr hi 0x%x lo 0x%x base data is %d\n", + ab->pci.msi.addr_hi,
+ ab->pci.msi.addr_lo,
+ ab->pci.msi.ep_base_data);
Though wasn't sure for "Data"/ep_base_data since the kernel uses a u32 but lspci appears to show a 16 bit value. I tried:
+ ab_pci->ab->pci.msi.ep_base_data = 0; + ab_pci->ab->pci.msi.ep_base_data = 0x0028; + ab_pci->ab->pci.msi.ep_base_data = 0x0028000;
(maybe thats entirely the wrong change)
Quoted 1 line
Quoted 53 lines · James Prestwood
Hi Baochen,
<snip>
Quoted 40 lines
Quoted 31 lines
Quoted 31 lines
While I don't think there is a way for qemu/ath11k to get the real MSI vector from host, I will try to read the vfio code to check further. Before that, to unblock you, a possible hack is to hard code the MSI vector in qemu to the same as in host, on condition that the MSI vector doesn't change. In my case, the change looks like
drivers/net/wireless/ath/ath11k/pci.c +6
-1@@ -459,7 +459,12 @@ static int ath11k_pci_alloc_msi(struct ath11k_pci *ab_pci) ab->pci.msi.addr_hi = 0; } - ath11k_dbg(ab, ATH11K_DBG_PCI, "msi base data is %d\n", ab->pci.msi.ep_base_data); + ab->pci.msi.addr_hi = 0; + ab->pci.msi.addr_lo = 0xfee00578; + ath11k_dbg(ab, ATH11K_DBG_PCI, "msi addr hi 0x%x lo 0x%x base data is %d\n", + ab->pci.msi.addr_hi, + ab->pci.msi.addr_lo, + ab->pci.msi.ep_base_data);return 0;
drivers/net/wireless/ath/ath11k/pci.c +1
-0@@ -487,6 +492,7 @@ static int ath11k_pci_config_msi_data(struct ath11k_pci *ab_pci) } ab_pci->ab->pci.msi.ep_base_data = msi_desc->msg.data; + ab_pci->ab->pci.msi.ep_base_data = 0;Quoted 7 lines
Quoted 5 lines
drivers/net/wireless/ath/ath11k/pci.c +0
-0ath11k_dbg(ab_pci->ab, ATH11K_DBG_PCI, "after request_irq msi_ep_base_data %d\n",
ab_pci->ab->pci.msi.ep_base_data);This hack works on my setup.
Progress! Thank you. This didn't work for me but its likely because my host MSI vector is not
fee00578. Where did you come up with this value?It could, and most likely, be different from machine to machine.
Quoted 8 lines
Quoted 2 lines
I don't see anything in the dmesg logs, or in lspci etc.
fee00578is the physical MSI vector so I got it using lspci in host, see
...
Capabilities: [50] MSI: Enable+ Count=1/32 Maskable+ 64bit-
Address: fee00578 Data: 0000
Masking: fffffffe Pending: 00000000
...
Quoted 7 lines
Mine looks like this:
...
Capabilities: [50] MSI: Enable- Count=1/32 Maskable+ 64bit-
Address: 00000000 Data: 0000
Masking: 00000000 Pending: 00000000
I guess you got this output when qemu/ath11k is not running, either qemu not started or ath11k module not loaded, right? Since MSI is not enabled in this output.
Sorry I should have made it clear:
step1: start qemu and load ath11k without changing anything, then you get those errors;
step2: keep qemu running and ath11k there, don't unload it. Then check lspci output in host to get the MSI vector.
step3: Back in qemu, hard code MSI vector to the one you get in step2, then rebuild and reload ath11k.
Quoted 57 lines
...
I've adjusted the patch:
drivers/net/wireless/ath/ath11k/pci.c +6
-1@@ -459,7 +459,12 @@ static int ath11k_pci_alloc_msi(struct ath11k_pci *ab_pci) ab->pci.msi.addr_hi = 0; } - ath11k_dbg(ab, ATH11K_DBG_PCI, "msi base data is %d\n", ab->pci.msi.ep_base_data); + ab->pci.msi.addr_hi = 0; + ab->pci.msi.addr_lo = 0; + ath11k_dbg(ab, ATH11K_DBG_PCI, "msi addr hi 0x%x lo 0x%x base data is %d\n", + ab->pci.msi.addr_hi, + ab->pci.msi.addr_lo, + ab->pci.msi.ep_base_data);return 0;
drivers/net/wireless/ath/ath11k/pci.c +1
-0@@ -487,6 +492,7 @@ static int ath11k_pci_config_msi_data(struct ath11k_pci *ab_pci) } ab_pci->ab->pci.msi.ep_base_data = msi_desc->msg.data; + ab_pci->ab->pci.msi.ep_base_data = 0; ath11k_dbg(ab_pci->ab, ATH11K_DBG_PCI, "after request_irq
msi_ep_base_data%d\n",
ab_pci->ab->pci.msi.ep_base_data);But still getting the same errors:
[ 3.563057]
ath11k_pci0000:00:06.0: Service connect timeout [ 3.565044]ath11k_pci0000:00:06.0: failed to connect to HTT: -110 [ 3.567031]ath11k_pci0000:00:06.0: failed to start core: -110 [ 3.777514]ath11k_pci0000:00:06.0: firmware crashed:MHI_CB_EE_RDDM[ 3.777555]ath11k_pci0000:00:06.0: ignore reset dev flags0x4000[ 3.885137]ath11k_pci0000:00:06.0: firmware crashed:MHI_CB_EE_RDDM[ 3.885178]ath11k_pci0000:00:06.0: ignore reset dev flags0x4000I know this isn't a proper fix, so if its something that needs more thought than just hard-coded values I understand.
Thanks,
James
Quoted 5 lines
Quoted 5 lines
Thanks,
James
Hi Baochen,
I guess you got this output when qemu/ath11k is not running, either qemu not started or ath11k module not loaded, right? Since MSI is not enabled in this output.
Quoted 6 lines
Sorry I should have made it clear:
step1: start qemu and load ath11k without changing anything, then you get those errors;
step2: keep qemu running and ath11k there, don't unload it. Then check lspci output in host to get the MSI vector.
step3: Back in qemu, hard code MSI vector to the one you get in step2, then rebuild and reload ath11k.
Thank you for the clarification, this worked like a charm!
Thanks,
James
Quoted 9 lines · James Prestwood
Quoted 7 lines
Sorry I should have made it clear:
step1: start qemu and load ath11k without changing anything, then you get those errors;
step2: keep qemu running and ath11k there, don't unload it. Then check lspci output in host to get the MSI vector.
step3: Back in qemu, hard code MSI vector to the one you get in step2, then rebuild and reload ath11k.Thank you for the clarification, this worked like a charm!
Very nice, huge thanks to Baochen for finding the workaround! James, let us know how it works for you in the long run. This would definitely help automating ath11k testing so I'm very interested about this.
Signature · 3 lines
https://patchwork.kernel.org/project/linux-wireless/list/ https://wireless.wiki.kernel.org/en/developers/documentation/submittingpatches
Hi Kalle,
Quoted 9 lines · Kalle Valo
Quoted 7 lines · James Prestwood
Quoted 7 lines
Sorry I should have made it clear:
step1: start qemu and load ath11k without changing anything, then you get those errors;
step2: keep qemu running and ath11k there, don't unload it. Then check lspci output in host to get the MSI vector.
step3: Back in qemu, hard code MSI vector to the one you get in step2, then rebuild and reload ath11k.
Quoted 4 lines
Quoted 1 line
Thank you for the clarification, this worked like a charm!
Very nice, huge thanks to Baochen for finding the workaround! James, let us know how it works for you in the long run. This would definitely help automating ath11k testing so I'm very interested about this.
Yes thank you Baochen!
You mean using PCI pass-through? Yes, its immensely useful for testing kernel changes. No swapping host kernels/reboots and you can build a very minimal kernel, mine takes ~10 min to build initially then iterative changes take ~10 seconds.
We have also integrated this into IWD's autotest framework so you can, in theory, run tests you normally run using mac80211_hwsim but using real hardware (assuming you aren't needing to modify frames). I did at one point have a system set up with 4-5 wifi cards and could run python tests against them, bring up an AP, connect, etc. It was pretty neat.
Thanks,
James
Hi Kalle/Baochen,
Quoted 9 lines · Kalle Valo
Quoted 7 lines · James Prestwood
Quoted 7 lines
Sorry I should have made it clear:
step1: start qemu and load ath11k without changing anything, then you get those errors;
step2: keep qemu running and ath11k there, don't unload it. Then check lspci output in host to get the MSI vector.
step3: Back in qemu, hard code MSI vector to the one you get in step2, then rebuild and reload ath11k.
Quoted 4 lines
Quoted 1 line
Thank you for the clarification, this worked like a charm!
Very nice, huge thanks to Baochen for finding the workaround! James, let us know how it works for you in the long run. This would definitely help automating ath11k testing so I'm very interested about this.
It appears this hack isn't a reliable solution long term. It did survive several reboots but then I was back to the same behavior. Looked at the MSI vector and it had changed:
It went from fee00558 to feee00538
But at least I can just modify this and get it working again.
Thanks,
James
Quoted 1 line
Quoted 12 lines · James Prestwood
Hi Kalle/Baochen,
Quoted 9 lines · Kalle Valo
Quoted 7 lines · James Prestwood
Quoted 7 lines
Sorry I should have made it clear:
step1: start qemu and load ath11k without changing anything, then you get those errors;
step2: keep qemu running and ath11k there, don't unload it. Then check lspci output in host to get the MSI vector.
step3: Back in qemu, hard code MSI vector to the one you get in step2, then rebuild and reload ath11k.
Quoted 9 lines
Quoted 4 lines
Quoted 1 line
Thank you for the clarification, this worked like a charm!
Very nice, huge thanks to Baochen for finding the workaround! James, let us know how it works for you in the long run. This would definitely help automating ath11k testing so I'm very interested about this.
It appears this hack isn't a reliable solution long term. It did survive several reboots but then I was back to the same behavior. Looked at the MSI vector and it had changed:
It went from
fee00558tofeee00538
Yes, the MSI vector allocation is up to the kernel, and several factors are considered when doing this. Among them, the most important are the IRQ load balance and available vectors on a specific CPU. As a result, the vector changes, not only when you request it (as you already see), but also in run time.
Quoted 7 lines
But at least I can just modify this and get it working again.
Thanks,
James
Quoted 1 line
Given the "ewww" patch to make a module parameter workaround for this:
https://lore.kernel.org/linux-wireless/[email protected]/T/#u
Quoted 3 lines · David Woodhouse
How exactly is the content of this register then given back to the firmware? Is that communication snoopable by the VMM?
If I'm reading the code correctly, it's just a write to a register in a memory mapped region, possibly in multiple locations (different queues or something). Possibly indirect (see __ath11k_pcic_write32), but someone would have to trace it or know the HW better to understand which locations it's written to.
But yeah seems totally feasible to just translate that back in the VMM.
Quoted 10 lines
Quoted 4 lines
Quoted 4 lines
Off hand I don't have a good solution for this, the hardware is essentially imposing a unique requirement for MSI programming that the driver needs visibility of the physical MSI address and data.
Strictly, the driver doesn't need visibility to the actual values used by the hardware. Another way of it looking at it would be to say that the driver programs the MSI through this non-standard method, it just needs the VMM to trap and handle that, just as the VMM does for the standard MSI table.
Indeed. Much better than having a module parameter.
Quoted 1 line
Which is what I thought we'd already seen on some Atheros devices.
It probably also affects ath12k, seems similar.
johannes
PATCH 1/1 vfio/pci: Create feature to disable MSI virtualization
In reply to James Prestwoodvfio-pci has always virtualized the MSI address and data registers as MSI programming is performed through the SET_IRQS ioctl. Often this virtualization is not used, and in specific cases can be unhelpful.
One such case where the virtualization is a hinderance is when the device contains an onboard interrupt controller programmed by the guest driver. Userspace VMMs have a chance to quirk this programming, injecting the host physical MSI information, but only if the userspace driver can get access to the host physical address and data registers.
This introduces a device feature which allows the userspace driver to disable virtualization of the MSI capability address and data registers in order to provide read-only access the the physical values.
--- drivers/vfio/pci/vfio_pci_config.c | 26 ++++++++++++++++++++++++++ drivers/vfio/pci/vfio_pci_core.c | 21 +++++++++++++++++++++ drivers/vfio/pci/vfio_pci_priv.h | 1 + include/uapi/linux/vfio.h | 14 ++++++++++++++ 4 files changed, 62 insertions(+)
drivers/vfio/pci/vfio_pci_config.c +26 -0
@@ -1259,6 +1259,32 @@ static int vfio_msi_cap_len(struct vfio_pci_core_device *vdev, u8 pos)return len; } +/* Disable virtualization of the MSI address and data fields */ +int vfio_pci_msi_novirt (struct vfio_pci_core_device *vdev) +{ +struct pci_dev *pdev = vdev->pdev; +struct perm_bits *perm = vdev->msi_perm; +u16 flags; +int ret; + +if (!perm) +return -EINVAL ; + + ret =pci_read_config_word (pdev, pdev->msi_cap +PCI_MSI_FLAGS , &flags); +if (ret) +return pcibios_err_to_errno (ret); + +p_setd (perm,PCI_MSI_ADDRESS_LO ,NO_VIRT ,NO_WRITE ); +if (flags &PCI_MSI_FLAGS_64BIT ) { +p_setd (perm,PCI_MSI_ADDRESS_HI ,NO_VIRT ,NO_WRITE ); +p_setw (perm,PCI_MSI_DATA_64 , (u16 )NO_VIRT , (u16 )NO_WRITE ); + }else { +p_setw (perm,PCI_MSI_DATA_32 , (u16 )NO_VIRT , (u16 )NO_WRITE ); + } + +return 0 ; +} +/* Determine extended capability length for VC (2 & 9) and MFVC */ static int vfio_vc_cap_len (struct vfio_pci_core_device *vdev,u16 pos) {
drivers/vfio/pci/vfio_pci_core.c +21 -0
@@ -1518,6 +1518,24 @@ static int vfio_pci_core_feature_token(struct vfio_device *device, u32 flags,return 0 ; } +static int vfio_pci_core_feature_msi_novirt (struct vfio_device *device, +u32 flags,void __user *arg, +size_t argsz) +{ +struct vfio_pci_core_device *vdev = + container_of(device, struct vfio_pci_core_device, vdev); +int ret; + +if (!vdev->msi_perm) +return -ENOTTY ; + + ret =vfio_check_feature (flags, argsz,VFIO_DEVICE_FEATURE_SET ,0 ); +if (ret !=1 ) +return ret; + +return vfio_pci_msi_novirt (vdev); +} +int vfio_pci_core_ioctl_feature (struct vfio_device *device,u32 flags,void __user *arg,size_t argsz) {
@@ -1531,6 +1549,9 @@ int vfio_pci_core_ioctl_feature(struct vfio_device *device, u32 flags,return vfio_pci_core_pm_exit (device, flags, arg, argsz);case VFIO_DEVICE_FEATURE_PCI_VF_TOKEN :return vfio_pci_core_feature_token (device, flags, arg, argsz); +case VFIO_DEVICE_FEATURE_PCI_MSI_NOVIRT : +return vfio_pci_core_feature_msi_novirt (device, flags, + arg, argsz);default :return -ENOTTY ; }
drivers/vfio/pci/vfio_pci_priv.h +1 -0
@@ -53,6 +53,7 @@ int vfio_pci_ioeventfd(struct vfio_pci_core_device *vdev, loff_t offset,int vfio_pci_init_perm_bits (void );void vfio_pci_uninit_perm_bits (void ); +int vfio_pci_msi_novirt (struct vfio_pci_core_device *vdev);int vfio_config_init (struct vfio_pci_core_device *vdev);void vfio_config_free (struct vfio_pci_core_device *vdev);
include/uapi/linux/vfio.h +14 -0
@@ -1458,6 +1458,20 @@ struct vfio_device_feature_bus_master { };#define VFIO_DEVICE_FEATURE_BUS_MASTER 10 +/** +* Toggle virtualization of PCI MSI address and data fields off. By default +* vfio-pci-core based drivers virtualize the MSI address and data fields of +* the MSI capability to emulate direct access to the device, ie. writes are +* allowed and buffered where subsequent reads return the buffered data. +* VMMs often virtualize these registers anyway and there are cases in user- +* space where having access to the host MSI fields can be useful, such as +* quirking an embedded interrupt controller on the device to generate physical +* MSI interrupts. Upon VFIO_DEVICE_FEATURE_SET of the PCI_MSI_NOVIRT feature +* this virtualization is disabled, reads of the MSI address and data fields +* will return the physical values and writes are dropped. +*/ +#define VFIO_DEVICE_FEATURE_PCI_MSI_NOVIRT 11 +/* -------- API for Type1 VFIO IOMMU -------- */ /**
Signature · 1 line
2.45.2
PATCH 1/1 vfio/pci-quirks: Quirk for ath wireless
In reply to James PrestwoodThese devices have an embedded interrupt controller which is programmed with guest physical MSI address/data, which doesn't work. We need vfio-pci kernel support to provide a device feature which disables virtualization of the MSI capability registers. Then we can do brute force testing for writes matching the MSI address, from which we can infer writes of the MSI data, replacing each with host physical values.
This has only been tested on ath11k (0x1103), ath12k support is speculative and requires testing. Note that Windows guest drivers make use of multi-vector MSI which requires interrupt remapping support in the host.
NB. The #define for the new vfio feature is temporary for RFC/RFT, a formal proposal will include a proper linux-headers update.
--- hw/vfio/pci-quirks.c | 236 +++++++++++++++++++++++++++++++++++++++++++ hw/vfio/trace-events | 3 + 2 files changed, 239 insertions(+)
hw/vfio/pci-quirks.c +236 -0
@@ -23,6 +23,7 @@#include "qapi/visitor.h" #include <sys/ioctl.h> #include "hw/nvram/fw_cfg.h" +#include "hw/pci/msi.h" #include "hw/qdev-properties.h" #include "pci.h" #include "trace.h"
@@ -1159,6 +1160,240 @@ static void vfio_probe_rtl8168_bar2_quirk(VFIOPCIDevice *vdev, int nr)trace_vfio_quirk_rtl8168_probe (vdev->vbasedev.name); } +#define PCI_VENDOR_ID_QCOM 0x17cb + +/* +* ath11k wireless adapters do not support INTx and appear to have an interrupt +* controller embedded into the hardware. By default the interrupt controller +* is programmed with the MSI guest physical address, which doesn't work. +* Instead we need to trap and insert the host physical address and data. +* +* By default vfio-pci virtualizes the MSI address and data registers, providing +* a writable buffer, where reads simply return the buffer. QEMU writes +* everything through to hardware, so this only holds the guest MSI address. +* Therefore we first need to invoke a device feature that disables this +* emulation of the MSI address and data registers, allowing access to the host +* physical address and data. +* +* Next, where do these values get written? If we disable mmap support and +* trace accesses, we get this: +* +* # grep -A2 region0.*0xfee00000 typescript +* vfio_region_write (0000:01:00.0:region0+0x80048, 0xfee00000, 4) +* vfio_region_write (0000:01:00.0:region0+0x8004c, 0x100, 4) +* vfio_region_write (0000:01:00.0:region0+0x80050, 0x21, 4) +* -- +* vfio_region_write (0000:01:00.0:region0+0x83048, 0xfee00000, 4) +* vfio_region_write (0000:01:00.0:region0+0x8304c, 0x100, 4) +* vfio_region_write (0000:01:00.0:region0+0x83050, 0x21, 4) +* -- +* vfio_region_write (0000:01:00.0:region0+0x830a0, 0xfee00000, 4) +* vfio_region_write (0000:01:00.0:region0+0x830a4, 0x100, 4) +* vfio_region_write (0000:01:00.0:region0+0x830a8, 0x21, 4) +* -- +* vfio_region_write (0000:01:00.0:region0+0x85048, 0xfee00000, 4) +* vfio_region_write (0000:01:00.0:region0+0x8504c, 0x100, 4) +* vfio_region_write (0000:01:00.0:region0+0x85050, 0x21, 4) +* -- +* vfio_region_write (0000:01:00.0:region0+0x850a0, 0xfee00000, 4) +* vfio_region_write (0000:01:00.0:region0+0x850a4, 0x100, 4) +* vfio_region_write (0000:01:00.0:region0+0x850a8, 0x21, 4) +* -- +* vfio_region_write (0000:01:00.0:region0+0x86048, 0xfee00000, 4) +* vfio_region_write (0000:01:00.0:region0+0x8604c, 0x100, 4) +* vfio_region_write (0000:01:00.0:region0+0x86050, 0x21, 4) +* -- +* vfio_region_write (0000:01:00.0:region0+0x8b048, 0xfee00000, 4) +* vfio_region_write (0000:01:00.0:region0+0x8b04c, 0x100, 4) +* vfio_region_write (0000:01:00.0:region0+0x8b050, 0x21, 4) +* -- +* vfio_region_write (0000:01:00.0:region0+0x8b0a0, 0xfee00000, 4) +* vfio_region_write (0000:01:00.0:region0+0x8b0a4, 0x100, 4) +* vfio_region_write (0000:01:00.0:region0+0x8b0a8, 0x21, 4) +* -- +* vfio_region_write (0000:01:00.0:region0+0x8e048, 0xfee00000, 4) +* vfio_region_write (0000:01:00.0:region0+0x8e04c, 0x100, 4) +* vfio_region_write (0000:01:00.0:region0+0x8e050, 0x21, 4) +* -- +* vfio_region_write (0000:01:00.0:region0+0xb4968, 0xfee00000, 4) +* vfio_region_write (0000:01:00.0:region0+0xb496c, 0x100, 4) +* vfio_region_write (0000:01:00.0:region0+0xb4970, 0x21, 4) +* -- +* vfio_region_write (0000:01:00.0:region0+0xb4a70, 0xfee00000, 4) +* vfio_region_write (0000:01:00.0:region0+0xb4a74, 0x100, 4) +* vfio_region_write (0000:01:00.0:region0+0xb4a78, 0x21, 4) +* -- +* vfio_region_write (0000:01:00.0:region0+0xb849c, 0xfee00000, 4) +* vfio_region_write (0000:01:00.0:region0+0xb84a0, 0x100, 4) +* vfio_region_write (0000:01:00.0:region0+0xb84a4, 0x21, 4) +* -- +* vfio_region_write (0000:01:00.0:region0+0xb85a4, 0xfee00000, 4) +* vfio_region_write (0000:01:00.0:region0+0xb85a8, 0x100, 4) +* vfio_region_write (0000:01:00.0:region0+0xb85ac, 0x21, 4) +* +* We can find in this example trace that the MSI capability is programmed as: +* +* vfio_pci_write_config (0000:01:00.0, @0x54, 0xfee00000, len=0x4) +* vfio_pci_write_config (0000:01:00.0, @0x58, 0x21, len=0x2) +* +* This is a 32-bit MSI capability based at 0x50, so the MSI address is +* 0xfee00000 with data 0x21. So we see writes of the MSI address, followed +* 8-bytes later by what appears to be the MSI data. There's no obvious +* pattern in the device address where these are being written. +* +* We therefore come up with a really crude quirk that looks for values +* written to the ATH11K_PCI_WINDOW (defined in Linux driver as starting at +* 0x80000 with an 18-bit mask, ie. 256k) that match the guest MSI address. +* When found we replace the data with the host physical address and set a +* cookie to match the MSI data write, again replacing with the host value and +* clearing the cookie. +* +* Amazingly, this seems sufficient to work, and the trapped window only seems +* to be used during initialization, so this should introduce minimal overhead. +* +* The Windows driver makes use of multi-vector MSI, where our sanity test +* of the MSI data value must then mask off the vector offset for comparison +* and add it back to the host base data value on write. +* +* Only 4- and 8-byte accesses are observed to the PCI window, and MSI access +* are only observed with 4-byte width. +*/ + +// Temporary, include updated Linux headers +#define VFIO_DEVICE_FEATURE_PCI_MSI_NOVIRT 11 + +typedef struct VFIOQAthQuirk { +VFIOPCIDevice *vdev; +uint32_t pci_window_base; +uint32_t msi_data_addr; +}VFIOQAthQuirk ; + +static uint64_t vfio_ath_quirk_read (void *opaque,hwaddr addr,unsigned size) +{ +VFIOQAthQuirk *ath = opaque; +VFIOPCIDevice *vdev = ath->vdev; + +return vfio_region_read (&vdev->bars[0 ].region, + addr + ath->pci_window_base, size); +} + +static void vfio_ath_quirk_write (void *opaque,hwaddr addr, +uint64_t data,unsigned size) +{ +VFIOQAthQuirk *ath = opaque; +VFIOPCIDevice *vdev = ath->vdev; + +if (size ==4 &&msi_enabled (&vdev->pdev)) { +uint32_t phys =0 ; + +if (data ==pci_get_long (vdev->pdev.config + + vdev->pdev.msi_cap +PCI_MSI_ADDRESS_LO )) { +pread (vdev->vbasedev.fd, &phys,4 , vdev->config_offset + + vdev->pdev.msi_cap +PCI_MSI_ADDRESS_LO ); +trace_vfio_quirk_ath_write_address (vdev->vbasedev.name, + addr + ath->pci_window_base, + data, phys); + data = phys; + ath->msi_data_addr = addr +8 ; + }else if (ath->msi_data_addr && ath->msi_data_addr == addr) { +uint32_t mask =msi_nr_vectors_allocated (&vdev->pdev) -1 ; +uint32_t nr = data & mask; + +if ((data & ~mask) ==pci_get_word (vdev->pdev.config + + vdev->pdev.msi_cap + +PCI_MSI_DATA_32 )) { +pread (vdev->vbasedev.fd, &phys,2 , vdev->config_offset + + vdev->pdev.msi_cap +PCI_MSI_DATA_32 ); + phys += nr; +trace_vfio_quirk_ath_write_data (vdev->vbasedev.name, + addr + ath->pci_window_base, + data, phys); + data = phys; + } + ath->msi_data_addr =0 ; + } + } + +vfio_region_write (&vdev->bars[0 ].region, addr + ath->pci_window_base, + data, size); +} + +static const MemoryRegionOps vfio_ath_quirk = { + .read = vfio_ath_quirk_read, + .write = vfio_ath_quirk_write, + .valid = { + .min_access_size =4 , + .max_access_size =8 , + .unaligned = false, + }, + .endianness =DEVICE_LITTLE_ENDIAN , +}; + +static bool vfio_pci_msi_novirt (VFIOPCIDevice *vdev) +{ +uint64_t buf[DIV_ROUND_UP (sizeof (struct vfio_device_feature ), +sizeof (uint64_t ))] = {}; +struct vfio_device_feature *feature = (struct vfio_device_feature *)buf; + + feature->argsz =sizeof (buf); + feature->flags =VFIO_DEVICE_FEATURE_SET | +VFIO_DEVICE_FEATURE_PCI_MSI_NOVIRT ; + +return !ioctl (vdev->vbasedev.fd,VFIO_DEVICE_FEATURE , feature); +} + +static void vfio_probe_ath_bar0_quirk (VFIOPCIDevice *vdev,int nr) +{ +VFIOQuirk *quirk; +VFIOQAthQuirk *ath; +uint32_t pci_window_base, pci_window_size; + +if (nr !=0 || + vdev->vendor_id !=PCI_VENDOR_ID_QCOM || !msi_present (&vdev->pdev)) { +return ; + } + +switch (vdev->device_id) { +case 0x1101 :/* Untested */ +case 0x1103 : +case 0x1104 :/* Untested */ +/* Devices claimed by ath11k_pci_id_table in Linux driver as of v6.10 */ + pci_window_base =0x80000 ;/* ATH11K_PCI_WINDOW_START */ + pci_window_size =0x40000 ;/* ATH11K_PCI_WINDOW_RANGE_MASK (256k)*/ +break ; +case 0x1107 :/* Untested */ +case 0x1109 :/* Untested */ +/* Devices claimed by ath12k_pci_id_table in Linux driver as of v6.10 */ + pci_window_base =0x1e00000 ;/* PCI_BAR_WINDOW0_BASE */ + pci_window_size =0x80000 ;/* ~PCI_BAR_WINDOW0_END (512k) */ +break ; +default : +return ; + } + +if (!vfio_pci_msi_novirt (vdev)) { +warn_report ("Found device matching Atheros wireless quirk, but host " +"does not support vfio device feature required for quirk. " +"Device is known not to work with device assignment " +"without this quirk. Please update host kernel." ); +return ; + } + + quirk =vfio_quirk_alloc (1 ); + quirk->data = ath =g_malloc0 (sizeof (*ath)); + ath->vdev = vdev; + ath->pci_window_base = pci_window_base; + +memory_region_init_io (&quirk->mem[0 ],OBJECT (vdev), &vfio_ath_quirk, + ath,"vfio-ath-quirk" , pci_window_size); +memory_region_add_subregion_overlap (vdev->bars[nr].region.mem, + pci_window_base, &quirk->mem[0 ],1 ); + +QLIST_INSERT_HEAD (&vdev->bars[nr].quirks, quirk, next); + +trace_vfio_quirk_ath_bar0_probe (vdev->vbasedev.name); +} +#define IGD_ASLS 0xfc/* ASL Storage Register */ /*
@@ -1261,6 +1496,7 @@ void vfio_bar_quirk_setup(VFIOPCIDevice *vdev, int nr)#ifdef CONFIG_VFIO_IGD vfio_probe_igd_bar4_quirk (vdev, nr);#endif +vfio_probe_ath_bar0_quirk (vdev, nr); }void vfio_bar_quirk_exit (VFIOPCIDevice *vdev,int nr)
hw/vfio/trace-events +3 -0
@@ -82,6 +82,9 @@ vfio_ioeventfd_exit(const char *name, uint64_t addr, unsigned size, uint64_t dat vfio_ioeventfd_handler(const char *name, uint64_t addr, unsigned size, uint64_t data) "%s+0x%"PRIx64"[%d] -> 0x%"PRIx64 vfio_ioeventfd_init(const char *name, uint64_t addr, unsigned size, uint64_t data, bool vfio) "%s+0x%"PRIx64"[%d]:0x%"PRIx64" vfio:%d" vfio_pci_igd_opregion_enabled(const char *name) "%s" +vfio_quirk_ath_bar0_probe(const char *name) "%s" +vfio_quirk_ath_write_address(const char *name, uint64_t addr, uint64_t data, uint32_t phys) "%s[0x%"PRIx64"]: Replacing MSI address 0x%"PRIx64" with value 0x%x" +vfio_quirk_ath_write_data(const char *name, uint64_t addr, uint64_t data, uint32_t phys) "%s[0x%"PRIx64"]: Replacing MSI data 0x%"PRIx64" with value 0x%x" # igd.c vfio_pci_igd_bar4_write(const char *name, uint32_t index, uint32_t data, uint32_t base) "%s [0x%03x] 0x%08x -> 0x%08x"
Signature · 1 line
2.45.2
Re: [PATCH RFC/RFT] vfio/pci: Create feature to disable MSI virtualization
In reply to Alex WilliamsonQuoted 10 lines · Alex Williamson
vfio-pci has always virtualized the MSI address and data registers as MSI programming is performed through the
SET_IRQSioctl. Often this virtualization is not used, and in specific cases can be unhelpful.One such case where the virtualization is a hinderance is when the device contains an onboard interrupt controller programmed by the guest driver. Userspace VMMs have a chance to quirk this programming, injecting the host physical MSI information, but only if the userspace driver can get access to the host physical address and data registers.
Quoted 3 lines
This introduces a device feature which allows the userspace driver to disable virtualization of the MSI capability address and data registers in order to provide read-only access the the physical values.
Personally, I very much dislike this. Encouraging such hacky driver use of the interrupt subsystem is not a good direction. Enabling this in VMs will further complicate fixing the IRQ usages in these drivers over the long run.
If the device has it's own interrupt sources then the device needs to create an irq_chip and related and hook them up properly. Not hackily read the MSI-X registers and write them someplace else.
Thomas Gleixner has done alot of great work recently to clean this up.
So if you imagine the driver is fixed, then this is not necessary.
Howver, it will still not work in a VM. Making IMS and non-MSI interrupt controlers work within VMs is still something that needs to be done.
Jason
Re: [PATCH RFC/RFT] vfio/pci-quirks: Quirk for ath wireless
In reply to Alex WilliamsonQuoted 11 lines · Alex Williamson
These devices have an embedded interrupt controller which is programmed with guest physical MSI address/data, which doesn't work. We need vfio-pci kernel support to provide a device feature which disables virtualization of the MSI capability registers. Then we can do brute force testing for writes matching the MSI address, from which we can infer writes of the MSI data, replacing each with host physical values.
This has only been tested on ath11k (
0x1103), ath12k support is speculative and requires testing. Note that Windows guest drivers make use of multi-vector MSI which requires interrupt remapping support in the host.
The way it is really supposed to work, is that the guest itself controls/knows the MSI addr/data pairs and the interrupt remapping HW makes that delegation safe since all the interrupt processing will be qualified by the RID.
Then the guest can make up the unique interrupts for MSI and any internal "IMS" sources and we just let the guest directly write the MSI/MSI-X and any IMS values however it wants.
This hackery to capture and substitute the IMS programming is neat and will solve this one device, but there are more IMS style devices in the pipeline than will really need a full solution.
Quoted 3 lines
+ * The Windows driver makes use of multi-vector MSI, where our sanity test
+ * of the MSI data value must then mask off the vector offset for comparison
+ * and add it back to the host base data value on write.
But is that really enough? If the vector offset is newly created then that means the VM built a new interrupt that needs setup to be routed into the VM?? Is that why you say it "requires interrupt remapping support" because that setup is happening implicitly on x86?
It looks like Windows is acting as I said Linux should, with a "irq_chip" and so on to get the unique interrupt source a proper unique addr/data pair...
Jason
Re: [PATCH RFC/RFT] vfio/pci: Create feature to disable MSI virtualization
In reply to Jason GunthorpeQuoted 20 lines · Jason Gunthorpe
Quoted 10 lines · Alex Williamson
vfio-pci has always virtualized the MSI address and data registers as MSI programming is performed through the
SET_IRQSioctl. Often this virtualization is not used, and in specific cases can be unhelpful.One such case where the virtualization is a hinderance is when the device contains an onboard interrupt controller programmed by the guest driver. Userspace VMMs have a chance to quirk this programming, injecting the host physical MSI information, but only if the userspace driver can get access to the host physical address and data registers.
Quoted 3 lines
This introduces a device feature which allows the userspace driver to disable virtualization of the MSI capability address and data registers in order to provide read-only access the the physical values.
Personally, I very much dislike this. Encouraging such hacky driver use of the interrupt subsystem is not a good direction. Enabling this in VMs will further complicate fixing the IRQ usages in these drivers over the long run.
Quoted 7 lines
If the device has it's own interrupt sources then the device needs to create an
irq_chipand related and hook them up properly. Not hackily read the MSI-X registers and write them someplace else.Thomas Gleixner has done alot of great work recently to clean this up.
So if you imagine the driver is fixed, then this is not necessary.
Yes. I looked at the at11k driver when I was reworking the PCI/MSI subsystem and that's a perfect candidate for a proper device specific interrupt domain to replace the horrible MSI hackery it has.
Quoted 3 lines
Howver, it will still not work in a VM. Making IMS and non-MSI interrupt controlers work within VMs is still something that needs to be done.
Sure, but we really want to do that in a generic way and not based on ad hoc workarounds.
Did the debate around this go anywhere?
Thanks,
tglx
Re: [PATCH RFC/RFT] vfio/pci-quirks: Quirk for ath wireless
In reply to Jason GunthorpeQuoted 18 lines · On Tue, 13 Aug 2024 13:43:41 -0300 Jason Gunthorpe <[email protected]> wrote:
Quoted 11 lines · Alex Williamson
These devices have an embedded interrupt controller which is programmed with guest physical MSI address/data, which doesn't work. We need vfio-pci kernel support to provide a device feature which disables virtualization of the MSI capability registers. Then we can do brute force testing for writes matching the MSI address, from which we can infer writes of the MSI data, replacing each with host physical values.
This has only been tested on ath11k (
0x1103), ath12k support is speculative and requires testing. Note that Windows guest drivers make use of multi-vector MSI which requires interrupt remapping support in the host.The way it is really supposed to work, is that the guest itself controls/knows the MSI addr/data pairs and the interrupt remapping HW makes that delegation safe since all the interrupt processing will be qualified by the RID.
Quoted 7 lines
Then the guest can make up the unique interrupts for MSI and any internal "IMS" sources and we just let the guest directly write the MSI/MSI-X and any IMS values however it wants.
This hackery to capture and substitute the IMS programming is neat and will solve this one device, but there are more IMS style devices in the pipeline than will really need a full solution.
How does the guest know to write a remappable vector format? How does the guest know the host interrupt architecture? For example why would an aarch64 guest program an MSI vector of 0xfee... if the host is x86?
The idea of guest owning the physical MSI address space sounds great, but is it practical? Is it something that would be accomplished while this device is still relevant?
Quoted 4 lines
Quoted 3 lines
+ * The Windows driver makes use of multi-vector MSI, where our sanity test
+ * of the MSI data value must then mask off the vector offset for comparison
+ * and add it back to the host base data value on write.
Quoted 8 lines
But is that really enough? If the vector offset is newly created then that means the VM built a new interrupt that needs setup to be routed into the VM?? Is that why you say it "requires interrupt remapping support" because that setup is happening implicitly on x86?
It looks like Windows is acting as I said Linux should, with a "
irq_chip" and so on to get the unique interrupt source a proper unique addr/data pair...
The Windows driver is just programming the MSI capability to use 16 vectors. We configure those vectors on the host at the time the capability is written. Whereas the Linux driver is only using a single vector and therefore writing the same MSI address and data at the locations noted in the trace, the Windows driver is writing different data values at different locations to make use of those vectors. This note is simply describing that we can't directly write the physical data value into the device, we need to determine which vector offset the guest is using and provide the same offset from the host data register value.
I don't know that interrupt remapping is specifically required, but the MSI domain needs to support MSI_FLAG_MULTI_PCI_MSI and AFAIK that's only available with interrupt remapping on x86, ie. pci_alloc_irq_vectors() with max_vecs >1 and PCI_IRQ_MSI flags needs to work on the host to mirror the guest MSI configuration. Thanks,
Alex
Re: [PATCH RFC/RFT] vfio/pci: Create feature to disable MSI virtualization
In reply to Jason GunthorpeQuoted 11 lines · On Tue, 13 Aug 2024 13:30:53 -0300 Jason Gunthorpe <[email protected]> wrote:
Quoted 10 lines · Alex Williamson
vfio-pci has always virtualized the MSI address and data registers as MSI programming is performed through the
SET_IRQSioctl. Often this virtualization is not used, and in specific cases can be unhelpful.One such case where the virtualization is a hinderance is when the device contains an onboard interrupt controller programmed by the guest driver. Userspace VMMs have a chance to quirk this programming, injecting the host physical MSI information, but only if the userspace driver can get access to the host physical address and data registers.
Quoted 8 lines
Quoted 3 lines
This introduces a device feature which allows the userspace driver to disable virtualization of the MSI capability address and data registers in order to provide read-only access the the physical values.
Personally, I very much dislike this. Encouraging such hacky driver use of the interrupt subsystem is not a good direction. Enabling this in VMs will further complicate fixing the IRQ usages in these drivers over the long run.
Clearly these _guest_ drivers are doing this regardless of the interfaces provided by vfio, so I don't see how we're encouraging hacky driver behavior, especially when it comes to Windows guest drivers.
Quoted 3 lines
If the device has it's own interrupt sources then the device needs to create an
irq_chipand related and hook them up properly. Not hackily read the MSI-X registers and write them someplace else.
This is how the hardware works, regardless of whether the guest driver represents the hardware using an irq_chip.
Quoted 3 lines
Thomas Gleixner has done alot of great work recently to clean this up.
So if you imagine the driver is fixed, then this is not necessary.
How so? Regardless of the guest driver structure, something is writing the MSI address and data values elsewhere in the device. AFAICT the only way to avoid needing to fixup those values is to give the guest ownership of the address space as you suggested in the other patch. That also seems to have a pile of issues though.
Quoted 3 lines
Howver, it will still not work in a VM. Making IMS and non-MSI interrupt controlers work within VMs is still something that needs to be done.
Making it work in a VM is sort of the point here. Thanks,
Alex
Re: [PATCH RFC/RFT] vfio/pci: Create feature to disable MSI virtualization
Quoted 8 lines · Alex Williamson
Quoted 4 lines
Personally, I very much dislike this. Encouraging such hacky driver use of the interrupt subsystem is not a good direction. Enabling this in VMs will further complicate fixing the IRQ usages in these drivers over the long run.
Clearly these _guest_ drivers are doing this regardless of the interfaces provided by vfio, so I don't see how we're encouraging hacky driver behavior, especially when it comes to Windows guest drivers.
Because people will then say the Linux driver can't be fixed to properly use an irq_domain/etc as the only option that works in VMs will be the hacky copy from MSI-X approach :\
Quoted 5 lines
Quoted 3 lines
Thomas Gleixner has done alot of great work recently to clean this up.
So if you imagine the driver is fixed, then this is not necessary.
How so?
Because if the driver is properly using the new irq_domain/etc infrastructure to model its additional interrupt source then this patch won't make it work in the VM anyhow, so it is not necessary..
Your other patch would be the only short term answer.
Jason
Re: [PATCH RFC/RFT] vfio/pci-quirks: Quirk for ath wireless
In reply to Alex WilliamsonQuoted 3 lines · Alex Williamson
How does the guest know to write a remappable vector format? How does the guest know the host interrupt architecture? For example why would an aarch64 guest program an MSI vector of
0xfee... if the host is x86?
All excellent questions.
Emulating real interrupt controllers in the VM is probably impossible in every scenario. But certainly x86 emulating x86 and ARM emulating ARM would be usefully achievable.
hyperv did a neat thing where their remapping driver seems to make VMM traps and looks kind of like the VMM gives it the platform specific addr/data pair.
It is a big ugly problem for sure, and we definately have painted ourselves into a corner where the OS has no idea if IMS techniques work properly or it is broken. :( :(
But I think there may not be a terribly impossible path where at least the guest could be offered a, say, virtio-irq in addition to the existing platform controllers that would process IMS for it.
Quoted 2 lines
The idea of guest owning the physical MSI address space sounds great, but is it practical?
In many cases yes, it is, but more importantly it is the only sane way to support these IMS like techniques broadly since IMS is by definition not generally trappable.
Quoted 2 lines
Is it something that would be accomplished while
this device is still relevant?
I don't know, I fear not. But it keeps coming up. Too many things don't work right with the trapping approach, including this.
Quoted 10 lines
The Windows driver is just programming the MSI capability to use 16 vectors. We configure those vectors on the host at the time the capability is written. Whereas the Linux driver is only using a single vector and therefore writing the same MSI address and data at the locations noted in the trace, the Windows driver is writing different data values at different locations to make use of those vectors. This note is simply describing that we can't directly write the physical data value into the device, we need to determine which vector offset the guest is using and provide the same offset from the host data register value.
I see, it seems to be assuming also that these extra interrupt sources are generating the same MSI message as the main MSI, not something else. That is more a SW quirk of Windows, I expect. I don't think Linux would do that..
This is probably the only way to approach this, trap and emulate the places in the device that program additional interrupt sources and do a full MSI-like flow to set them up in the kernel.
Jason
Re: [PATCH RFC/RFT] vfio/pci: Create feature to disable MSI virtualization
In reply to Thomas GleixnerQuoted 8 lines · Thomas Gleixner
Quoted 3 lines
Howver, it will still not work in a VM. Making IMS and non-MSI interrupt controlers work within VMs is still something that needs to be done.
Sure, but we really want to do that in a generic way and not based on ad hoc workarounds.
Did the debate around this go anywhere?
No, it got stuck on the impossible situation that there is no existing way for the VM to have any idea if IMS will work or is broken. Recall Intel was planning to "solve" this by sticking a DVSEC in their virtual config space that said to turn off IMS :\
So using IMS in the real world looked impractical and interest faded a bit.
But the underlying reasons for IMS haven't gone away and more work is coming that will bring it up again...
Jason
Re: [PATCH RFC/RFT] vfio/pci: Create feature to disable MSI virtualization
In reply to Jason GunthorpeQuoted 7 lines · On Tue, 13 Aug 2024 20:16:42 -0300 Jason Gunthorpe <[email protected]> wrote:
Quoted 5 lines · Alex Williamson
Quoted 4 lines
Personally, I very much dislike this. Encouraging such hacky driver use of the interrupt subsystem is not a good direction. Enabling this in VMs will further complicate fixing the IRQ usages in these drivers over the long run.
Quoted 7 lines
Quoted 3 lines
Clearly these _guest_ drivers are doing this regardless of the interfaces provided by vfio, so I don't see how we're encouraging hacky driver behavior, especially when it comes to Windows guest drivers.
Because people will then say the Linux driver can't be fixed to properly use an irq_domain/etc as the only option that works in VMs will be the hacky copy from MSI-X approach :\
Ironically QEMU already has direct access to the MSI-X vector table in MMIO space and could implement this type of quirk with no kernel changes. It's MSI that is now blocked by virtualization of the address and data registers. Note also that QEMU is still virtualizing these registers, the values seen in the guest are unchanged. It's only the VMM that can bypass that virtualization to see the host values.
Let's imagine the guest driver does change to implement an irq_domain. How does that fundamentally change the problem for the VMM that guest MSI values are being written to other portions of the device? The guest driver can have whatever architecture it wants (we don't know the architecture of the Windows driver) but we still need to trap writes of the guest MSI address/data and replace it with host values.
Quoted 11 lines
Quoted 5 lines
Quoted 3 lines
Thomas Gleixner has done alot of great work recently to clean this up.
So if you imagine the driver is fixed, then this is not necessary.
How so?
Because if the driver is properly using the new irq_domain/etc infrastructure to model its additional interrupt source then this patch won't make it work in the VM anyhow, so it is not necessary..
Your other patch would be the only short term answer.
The QEMU patch relies on this kernel patch in order to be able to access the host physical MSI address and data values through the vfio interface. Otherwise QEMU has no host values with which to patch-up guest values. As noted above, this does not provide any visible change to a QEMU guest, it only enables QEMU to implement the quirk in the other patch. Thanks,
Alex
Re: [PATCH RFC/RFT] vfio/pci: Create feature to disable MSI virtualization
Quoted 3 lines · Alex Williamson
Let's imagine the guest driver does change to implement an
irq_domain. How does that fundamentally change the problem for the VMM that guest MSI values are being written to other portions of the device?
If changed to irq_domain the VM will write addr/data pairs into those special register that are unique to that interrupt source and will not re-use values already set in the MSI table.
This means the VMM doesn't get any value from inspecting the MSI table because the value it needs won't be there, and alos that no interrupt routing will have been setup. The VMM must call VFIO_DEVICE_SET_IRQS to setup the unique routing.
These two patches are avoiding VFIO_DEVICE_SET_IRQS based on the assumption that the VM will re-use a addr/data pair already setup in the MSI table. Invalidating that assumption is the fundamental change irq_domain in the VM will make.
Quoted 4 lines
The guest driver can have whatever architecture it wants (we don't know the architecture of the Windows driver) but we still need to trap writes of the guest MSI address/data and replace it with host values.
Yes you do. But the wrinkle is you can't just assume one of the existing MSI entries is a valid replacement and copy from the MSI table. That works right now only because the Linux/Windows driver is re-using a MSI vector in the IMS registers.
I suggest the general path is something like:
1) A vfio variant driver sets up an irq_domain for the additional interrupt source registers
2) Somehow wire up VFIO_DEVICE_SET_IRQS so it can target vectors in the additional interrupt domain
3) Have the VMM trap writes to the extra interrupt source registers and execute VFIO_DEVICE_SET_IRQS
4) IRQ layer will setup an appropriate unique IRQ and route it to the guest/whatever just like MSI. Callbacks into the variant driver's irq_domain will program the HW registers.
Basically exactly the same flow as MSI, except instead of targetting a vector in the PCI core's MSI irq_domain it targets a vector in the variant driver's IMS IRQ domain.
Then we don't make any assumptions about how the VM is using these interrupt vectors, and crucially, SET_IRQs is called for every interrupt source and we rely on the kernel to produce the correct addr/data pair. No need for copying addr/data pairs from MSI tables.
Quoted 3 lines
As noted above, this does not provide any visible change to a QEMU guest, it only enables QEMU to implement the quirk in the other patch.
I see, I definitely didn't understand that it only reaches qemu from the commit message..
Jason
Re: [PATCH RFC/RFT] vfio/pci-quirks: Quirk for ath wireless
In reply to Jason GunthorpeQuoted 48 lines · On Tue, 13 Aug 2024 20:37:24 -0300 Jason Gunthorpe <[email protected]> wrote:
Quoted 3 lines · Alex Williamson
How does the guest know to write a remappable vector format? How does the guest know the host interrupt architecture? For example why would an aarch64 guest program an MSI vector of
0xfee... if the host is x86?All excellent questions.
Emulating real interrupt controllers in the VM is probably impossible in every scenario. But certainly x86 emulating x86 and ARM emulating ARM would be usefully achievable.
hyperv did a neat thing where their remapping driver seems to make VMM traps and looks kind of like the VMM gives it the platform specific addr/data pair.
It is a big ugly problem for sure, and we definately have painted ourselves into a corner where the OS has no idea if IMS techniques work properly or it is broken. :( :(
But I think there may not be a terribly impossible path where at least the guest could be offered a, say, virtio-irq in addition to the existing platform controllers that would process IMS for it.
Quoted 2 lines
The idea of guest owning the physical MSI address space sounds great, but is it practical?
In many cases yes, it is, but more importantly it is the only sane way to support these IMS like techniques broadly since IMS is by definition not generally trappable.
Quoted 2 lines
Is it something that would be accomplished while
this device is still relevant?I don't know, I fear not. But it keeps coming up. Too many things don't work right with the trapping approach, including this.
Quoted 10 lines
The Windows driver is just programming the MSI capability to use 16 vectors. We configure those vectors on the host at the time the capability is written. Whereas the Linux driver is only using a single vector and therefore writing the same MSI address and data at the locations noted in the trace, the Windows driver is writing different data values at different locations to make use of those vectors. This note is simply describing that we can't directly write the physical data value into the device, we need to determine which vector offset the guest is using and provide the same offset from the host data register value.
Quoted 8 lines
I see, it seems to be assuming also that these extra interrupt sources are generating the same MSI message as the main MSI, not something else. That is more a SW quirk of Windows, I expect. I don't think Linux would do that..
This is probably the only way to approach this, trap and emulate the places in the device that program additional interrupt sources and do a full MSI-like flow to set them up in the kernel.
Your last sentence here seems to agree with this approach, but everything else suggests disapproval, so I don't know where you're going here.
I have no specs for this device, nor any involvement from the device vendor, so the idea of creating a vfio-pci variant driver to setup an irq_domain and augment a device specific SET_IRQs ioctls not only sounds tremendously more complicated (host and VMM), it's simply not possible with the knowledge we have at hand. Making this device work in a VM is dead in the water if that's the bar to achieve.
I observe that the device configures MSI vectors and then writes that same vector address/data elsewhere into the device. Whether the device can trigger those vectors based only on the MSI capability programming and a secondary source piggybacks on those vectors or if this is just a hack by Qualcomm to use an MSI capability to acquire some vectors which are exclusively used by the secondary hardware, I have no idea. Who can even say if this is just a cost saving measure that a PCI config space is slapped into a platform device and there's simply no hw/fw support to push the vector data into the hardware and the driver bridges the gap.
The solution here is arguably fragile, we're relying on having a sufficiently unique MSI address that we can recognize writes with that value in order to both replace it with the host value and mark the location of the data register. If someone with some hardware insight wants to come along and provide a reference for static locations of these writes, I'd certainly welcome it. My sample size is one, which is why this is posted as an RFT.
I do not believe that introducing a vfio device feature that disables virtualization of the MSI address/data _only_ at the vfio interface (not to a QEMU VM) provides some implicit support of this device behavior. These values are already available to a privileged user in the host and the same is available for an MSI-X use case by directly reading the MSI-X vector table. The only point of the vfio device feature is that we need a vehicle to expose the MSI phsyical address/data values through he vfio channel, without additional host privileges. The virtualized values are essentially unused by QEMU, so why not give QEMU a way to turn off the virtualization to expose the host values. Thanks,
Alex
Re: [PATCH RFC/RFT] vfio/pci-quirks: Quirk for ath wireless
Quoted 7 lines · Alex Williamson
Quoted 3 lines
This is probably the only way to approach this, trap and emulate the places in the device that program additional interrupt sources and do a full MSI-like flow to set them up in the kernel.
Your last sentence here seems to agree with this approach, but everything else suggests disapproval, so I don't know where you're going here.
Trapping and emulating is fine.
My concern is really only about skipping SET_IRQ.
That works because of the assumption that the IMS sources are going to re-use addr/data pairs setup in the MSI CAP.
That assumption is frail, and won't work at all under the proper IMS support Linux now has.
I really don't want to go down the road and have someone tell Thomas he can't convert the Linux driver to use irq_domain IMS because it will break this stuff here.
Quoted 5 lines
I have no specs for this device, nor any involvement from the device vendor, so the idea of creating a vfio-pci variant driver to setup an
irq_domainand augment a device specificSET_IRQsioctls not only sounds tremendously more complicated (host and VMM), it's simply not possible with the knowledge we have at hand.
It seems like you have reverse engineered alot of the necessary information though??
Maybe there is a more generic approach than a variant driver. If you wanted to use IMS from userspace generically you could imagine some kind of IMS focused "SET_IRQ" in generic VFIO. Where we'd create the needed irq_domains and pass the addr/data pair back to userspace?
Quoted 6 lines
I observe that the device configures MSI vectors and then writes that same vector address/data elsewhere into the device. Whether the device can trigger those vectors based only on the MSI capability programming and a secondary source piggybacks on those vectors or if this is just a hack by Qualcomm to use an MSI capability to acquire some vectors which are exclusively used by the secondary hardware, I have no idea.
Well at least that should be testable - but it seems crazy if the device has registers for an addr/data pair and then somehow doesn't use the values that get put in them??
Copying from the MSI is almost certainly a SW hack because IMS support has never really existed in an OS until now. I think your guess for why it is like this is pretty good.
Quoted 6 lines
I do not believe that introducing a vfio device feature that disables virtualization of the MSI address/data _only_ at the vfio interface (not to a QEMU VM) provides some implicit support of this device behavior. These values are already available to a privileged user in the host and the same is available for an MSI-X use case by directly reading the MSI-X vector table.
To be clear, I'm not really worried about showing the data to userspace.
Userspace just shouldn't be using it to implement an IMS technique!
Jason
Re: [PATCH RFC/RFT] vfio/pci: Create feature to disable MSI virtualization
In reply to Thomas GleixnerQuoted 25 lines · Thomas Gleixner
Quoted 24 lines · Jason Gunthorpe
Quoted 10 lines · Alex Williamson
vfio-pci has always virtualized the MSI address and data registers as MSI programming is performed through the
SET_IRQSioctl. Often this virtualization is not used, and in specific cases can be unhelpful.One such case where the virtualization is a hinderance is when the device contains an onboard interrupt controller programmed by the guest driver. Userspace VMMs have a chance to quirk this programming, injecting the host physical MSI information, but only if the userspace driver can get access to the host physical address and data registers.
Quoted 3 lines
This introduces a device feature which allows the userspace driver to disable virtualization of the MSI capability address and data registers in order to provide read-only access the the physical values.
Personally, I very much dislike this. Encouraging such hacky driver use of the interrupt subsystem is not a good direction. Enabling this in VMs will further complicate fixing the IRQ usages in these drivers over the long run.
If the device has it's own interrupt sources then the device needs to create an
irq_chipand related and hook them up properly. Not hackily read the MSI-X registers and write them someplace else.
Quoted 7 lines
Quoted 3 lines
Thomas Gleixner has done alot of great work recently to clean this up.
So if you imagine the driver is fixed, then this is not necessary.
Yes. I looked at the at11k driver when I was reworking the PCI/MSI subsystem and that's a perfect candidate for a proper device specific interrupt domain to replace the horrible MSI hackery it has.
The ath11k hacks may be awful, but in their defence, that's because the whole way the hardware works is awful.
Q: With PCI passthrough to a guest, how does the guest OS tell the device where to do DMA?
A: The guest OS just hands the device a guest physical address and the IOMMU does the rest. Nothing 'intercedes' between the guest and the device to mess with that address.
Q: MSIs are just DMA. So with PCI passthrough to a guest, how does the guest OS configure the device's MSIs?
<fantasy>
A: The guest OS just hands the device a standard MSI message encoding the target guest APIC ID and vector (etc.), and the IOMMU does the rest. Nothing 'intercedes' between the guest and the device to mess with that MSI message.
And thus ath11k didn't need to do *any* hacks to work around a stupid hardware design with the VMM snooping on stuff it ideally shouldn't have had any business touching in the first place.
Posted interrupts are almost the *default* because the IOMMU receives a <source-id, vCPU APIC ID, vector> tuple on the bus. If receiving an interrupt for a vCPU which isn't currently running, that's when the IOMMU sets a bit in a table somewhere and notifies the host OS.
All that special case MSI handling and routing code that I had nightmares about because it fell through a wormhole from a parallel universe, doesn't exist.
And look, DPDK drivers which run in polling mode and 'abuse' MSIs by using real memory addresses and asking the device to "write <these> 32 bits to <this> structure if you want attention" just work nicely in virtual machines too, just as they do on real hardware. </fantasy>
/me wakes up...
Shit.
And we have to enable this Interrupt Remapping crap even to address more than 255 CPUs *without* virtualization? Even a *guest* has to see a virtual IOMMU and enable Interrupt Remapping to be able to use more than 255 vCPUs? Even though there were a metric shitload of spare bits in the MSI message we could have used¹.
Wait, so that means we have to offer an IOMMU with *DMA* remapping to guests, which means 2-stage translations and/or massive overhead, just for that guest to be able to use >255 vCPUs?
Screw you all, I'm going back to bed.
¹ And *should* use, if we ever do something similar like, say, expand
the vector# space past 8 bits. Intel and AMD take note.
Re: [PATCH RFC/RFT] vfio/pci: Create feature to disable MSI virtualization
Quoted 7 lines · David Woodhouse
<fantasy>
A: The guest OS just hands the device a standard MSI message encoding the target guest APIC ID and vector (etc.), and the IOMMU does the rest. Nothing 'intercedes' between the guest and the device to mess with that MSI message./me wakes up...
Well, I share your dream at least. :\
Have the VMM shadow the virtual interrupt remapping tables and assign it to the phyiscal remapping so that the physical addr/data pair doesn't change. Driving interrupt routing fully via the remapping HW and not via MSI interception.
IIRC Alex had a patch series for qemu to capture and rewrite the ath non-standard MSI locations, so virtualization worries should not block moving ath to use the device-specific MSI..
Jason
Re: [PATCH RFC/RFT] vfio/pci: Create feature to disable MSI virtualization
Quoted 4 lines · Jason Gunthorpe
IIRC Alex had a patch series for qemu to capture and rewrite the ath non-standard MSI locations, so virtualization worries should not block moving ath to use the device-specific MSI..
In the absence of the IOMMU in our shared fantasy, this seems like the best approach.
Even for passed-through PCI hardware, the config space is largely a fiction. As are the MSI(-X) vectors. So although it's a device-specific quirk, this seems like the best approach.
Probably best for the hypervisor to have some way to *advertise* that it's handling this though, as guests also might want to run on hypervisors which don't.
Re: [PATCH RFC/RFT] vfio/pci: Create feature to disable MSI virtualization
Quoted 3 lines · David Woodhouse
Probably best for the hypervisor to have some way to *advertise* that it's handling this though, as guests also might want to run on hypervisors which don't.
If the hypervisor doesn't properly virtualize the device it shouldn't assign it to a VM to start with :\
Intel looked at the question of advertising clean interrupt remapping when trying to virtualize IMS and it didn't seem so great.
Bare metal machines need to work, so any test they could think of adding would either fail on bare metal or fail on existing VMs.
VMM's have taken the approach of not telling the guest they are in VMs and then also not implementing the bare metal HW behaviors with full fidelity. So we have no way to discover that the VMM is, in fact, emulating broken "hw".
Thus we get this push that all kernels need to accomodate the worst VMM behaviors :(
Jason
Re: [PATCH RFC/RFT] vfio/pci-quirks: Quirk for ath wireless
In reply to Jason GunthorpeQuoted 55 lines · Jason Gunthorpe
Quoted 6 lines · Alex Williamson
Quoted 3 lines
This is probably the only way to approach this, trap and emulate the places in the device that program additional interrupt sources and do a full MSI-like flow to set them up in the kernel.
Your last sentence here seems to agree with this approach, but everything else suggests disapproval, so I don't know where you're going here.
Trapping and emulating is fine.
My concern is really only about skipping
SET_IRQ.That works because of the assumption that the IMS sources are going to re-use addr/data pairs setup in the MSI CAP.
That assumption is frail, and won't work at all under the proper IMS support Linux now has.
I really don't want to go down the road and have someone tell Thomas he can't convert the Linux driver to use
irq_domainIMS because it will break this stuff here.Quoted 5 lines
I have no specs for this device, nor any involvement from the device vendor, so the idea of creating a vfio-pci variant driver to setup an
irq_domainand augment a device specificSET_IRQsioctls not only sounds tremendously more complicated (host and VMM), it's simply not possible with the knowledge we have at hand.It seems like you have reverse engineered alot of the necessary information though??
Maybe there is a more generic approach than a variant driver. If you wanted to use IMS from userspace generically you could imagine some kind of IMS focused "
SET_IRQ" in generic VFIO. Where we'd create the neededirq_domainsand pass the addr/data pair back to userspace?Quoted 6 lines
I observe that the device configures MSI vectors and then writes that same vector address/data elsewhere into the device. Whether the device can trigger those vectors based only on the MSI capability programming and a secondary source piggybacks on those vectors or if this is just a hack by Qualcomm to use an MSI capability to acquire some vectors which are exclusively used by the secondary hardware, I have no idea.
Well at least that should be testable - but it seems crazy if the device has registers for an addr/data pair and then somehow doesn't use the values that get put in them??
Copying from the MSI is almost certainly a SW hack because IMS support has never really existed in an OS until now. I think your guess for why it is like this is pretty good.
Quoted 6 lines
I do not believe that introducing a vfio device feature that disables virtualization of the MSI address/data _only_ at the vfio interface (not to a QEMU VM) provides some implicit support of this device behavior. These values are already available to a privileged user in the host and the same is available for an MSI-X use case by directly reading the MSI-X vector table.
Quoted 6 lines
To be clear, I'm not really worried about showing the data to userspace.
Userspace just shouldn't be using it to implement an IMS technique!
Jason
I see this thread went stale. Wondering if there was ever a final conclusion if this could get fixed for ath11k or not. I tried again on a recent kernel, 6.17, and see the same behavior.
Thanks,
James
Re: [PATCH RFC/RFT] vfio/pci-quirks: Quirk for ath wireless
Quoted 61 lines · James Prestwood
Quoted 60 lines · Jason Gunthorpe
Quoted 6 lines · Alex Williamson
Quoted 3 lines
This is probably the only way to approach this, trap and emulate the places in the device that program additional interrupt sources and do a full MSI-like flow to set them up in the kernel.
Your last sentence here seems to agree with this approach, but everything else suggests disapproval, so I don't know where you're going here.
Trapping and emulating is fine.
My concern is really only about skipping
SET_IRQ.That works because of the assumption that the IMS sources are going to re-use addr/data pairs setup in the MSI CAP.
That assumption is frail, and won't work at all under the proper IMS support Linux now has.
I really don't want to go down the road and have someone tell Thomas he can't convert the Linux driver to use
irq_domainIMS because it will break this stuff here.Quoted 5 lines
I have no specs for this device, nor any involvement from the device vendor, so the idea of creating a vfio-pci variant driver to setup an
irq_domainand augment a device specificSET_IRQsioctls not only sounds tremendously more complicated (host and VMM), it's simply not possible with the knowledge we have at hand.It seems like you have reverse engineered alot of the necessary information though??
Maybe there is a more generic approach than a variant driver. If you wanted to use IMS from userspace generically you could imagine some kind of IMS focused "
SET_IRQ" in generic VFIO. Where we'd create the neededirq_domainsand pass the addr/data pair back to userspace?Quoted 6 lines
I observe that the device configures MSI vectors and then writes that same vector address/data elsewhere into the device. Whether the device can trigger those vectors based only on the MSI capability programming and a secondary source piggybacks on those vectors or if this is just a hack by Qualcomm to use an MSI capability to acquire some vectors which are exclusively used by the secondary hardware, I have no idea.
Well at least that should be testable - but it seems crazy if the device has registers for an addr/data pair and then somehow doesn't use the values that get put in them??
Copying from the MSI is almost certainly a SW hack because IMS support has never really existed in an OS until now. I think your guess for why it is like this is pretty good.
Quoted 6 lines
I do not believe that introducing a vfio device feature that disables virtualization of the MSI address/data _only_ at the vfio interface (not to a QEMU VM) provides some implicit support of this device behavior. These values are already available to a privileged user in the host and the same is available for an MSI-X use case by directly reading the MSI-X vector table.
To be clear, I'm not really worried about showing the data to userspace.
Userspace just shouldn't be using it to implement an IMS technique!
Quoted 8 lines
Quoted 1 line
Jason
I see this thread went stale. Wondering if there was ever a final conclusion if this could get fixed for ath11k or not. I tried again on a recent kernel, 6.17, and see the same behavior.
Thanks,
James
I addition, I've looked at various kernel version and see no time where there was ever a "hw/vfio/pci-quirks.c" file in the tree. I tried many versions between 6.17 and 5.11, I don't see the "hw" directory at all.
I'd like to try this patch out, but might need some guidance on what kernel version this was meant for and if files may have shuffled around.
Thanks,
James