virtio-mmio: add xenbus probing
RFC PATCH 1/1 virtio-mmio: add xenbus probing
The experimental virtio-mmio support for Xen was initially developed on aarch64, so device trees were used to configure the mmio devices, with arbitrary vGIC interrupts used by the hypervisor. On x86_64 however, the only reasonable way to interrupt the guest is over Xen event channels, which can only be acquired by children of xenbus, the virtual bus driven by Xen's configuration database, XenStore. It is also a more convenient and "Xen-ish" way to provision devices.
Implement a xenbus client for virtio-mmio which negotiates an event channel and provides it as a platform IRQ to the virtio-mmio driver.
---
Hi,
I've been working on porting virtio-mmio support from Arm to x86_64, with the goal of running vhost-user-gpu to power Wayland/GPU integration for Qubes OS. (I'm aware of various proposals for alternative virtio transports but virtio-mmio seems to be the only one that *is* upstream already and just Works..) Setting up virtio-mmio through xenbus, initially motivated just by event channels being the only real way to get interrupts working on HVM, turned out to generally be quite pleasant and nice :)
I'd like to get some early feedback for this patch, particularly the general stuff:
* is this whole thing acceptable in general?
* should it be extracted into a different file?
* (from the Xen side) any input on the xenstore keys, what goes where?
* anything else to keep in mind?
It does seem simple enough, so hopefully this can be done?
The corresponding userspace-side WIP is available at:
https://github.com/QubesOS/xen-vhost-frontend
And the required DMOP for firing the evtchn events will be sent to xen-devel shortly as well.
Thanks,
~val
--- drivers/virtio/Kconfig | 7 ++ drivers/virtio/virtio_mmio.c | 177 ++++++++++++++++++++++++++++++++++- 2 files changed, 183 insertions(+), 1 deletion(-)
drivers/virtio/Kconfig +7 -0
@@ -171,6 +171,13 @@ config VIRTIO_MMIO_CMDLINE_DEVICES If unsure, say 'N'. +config VIRTIO_MMIO_XENBUS + bool "Memory mapped virtio devices parameter parsing" + depends on VIRTIO_MMIO && XEN + select XEN_XENBUS_FRONTEND + help + Allow virtio-mmio devices instantiation for Xen guests via xenbus. + config VIRTIO_DMA_SHARED_BUFFER tristate depends on DMA_SHARED_BUFFER
drivers/virtio/virtio_mmio.c +176 -1
195 lines not on this page · raw message
Signature · 1 line
2.53.0
Hello,
Le 29/04/2026 à 16:18, Val Packett a écrit :
Quoted 18 lines
The experimental virtio-mmio support for Xen was initially developed on aarch64, so device trees were used to configure the mmio devices, with arbitrary vGIC interrupts used by the hypervisor. On
x86_64however, the only reasonable way to interrupt the guest is over Xen event channels, which can only be acquired by children of xenbus, the virtual bus driven by Xen's configuration database, XenStore. It is also a more convenient and "Xen-ish" way to provision devices.Implement a xenbus client for virtio-mmio which negotiates an event channel and provides it as a platform IRQ to the virtio-mmio driver.
Signed-off-by Val Packett <[email protected]>---
Hi,
Quoted 7 lines
I've been working on porting virtio-mmio support from Arm to
x86_64, with the goal of running vhost-user-gpu to power Wayland/GPU integration for Qubes OS. (I'm aware of various proposals for alternative virtio transports but virtio-mmio seems to be the only one that *is* upstream already and just Works..) Setting up virtio-mmio through xenbus, initially motivated just by event channels being the only real way to get interrupts working on HVM, turned out to generally be quite pleasant and nice :)
Is it HVM specific, or can we also make it work for PVH (we can actually attach a ioreq server to PVH guests) ?
Quoted 9 lines
I'd like to get some early feedback for this patch, particularly the general stuff:
* is this whole thing acceptable in general?
* should it be extracted into a different file?
* (from the Xen side) any input on the xenstore keys, what goes where?
* anything else to keep in mind?
Quoted 7 lines
It does seem simple enough, so hopefully this can be done?
The corresponding userspace-side WIP is available at:
https://github.com/QubesOS/xen-vhost-frontendAnd the required DMOP for firing the evtchn events will be sent to xen-devel shortly as well.
Could that be done through evtchn_send (or its userland counterpart) ?
Quoted 14 lines
Thanks,
~val--- drivers/virtio/Kconfig | 7 ++ drivers/virtio/virtio_mmio.c | 177 ++++++++++++++++++++++++++++++++++- 2 files changed, 183 insertions(+), 1 deletion(-)drivers/virtio/Kconfig +0
-0
drivers/virtio/Kconfig config VIRTIO_MMIO_CMDLINE_DEVICES · line 175
Quoted 5 lines
drivers/virtio/Kconfig +2
-0@@ -171,6 +171,13 @@ config VIRTIO_MMIO_CMDLINE_DEVICES If unsure, say 'N'. +config VIRTIO_MMIO_XENBUS + bool "Memory mapped virtio devices parameter parsing"
that text seems to miss the xenbus aspect
Quoted 151 lines
drivers/virtio/Kconfig +5
-0+ depends on VIRTIO_MMIO && XEN + select XEN_XENBUS_FRONTEND + help + Allow virtio-mmio devices instantiation for Xen guests via xenbus. + config VIRTIO_DMA_SHARED_BUFFER tristate depends on DMA_SHARED_BUFFERdrivers/virtio/virtio_mmio.c +128
-0@@ -70,6 +70,11 @@#include <uapi/linux/virtio_mmio.h> #include <linux/virtio_ring.h> +#ifdef CONFIG_VIRTIO_MMIO_XENBUS +#include <xen/xen.h> +#include <xen/xenbus.h> +#include <xen/events.h> +#endif /* The alignment to use between consumer and producer parts of vring. @@ -810,13 +815,183 @@ static struct platform_driver virtio_mmio_driver = { }, }; +#ifdef CONFIG_VIRTIO_MMIO_XENBUS +struct virtio_mmio_xen_info { +struct resource resources[2 ]; +unsigned int evtchn; +struct platform_device *pdev; +}; + +static int virtio_mmio_xen_probe (struct xenbus_device *dev, +const struct xenbus_device_id *id) +{ +int err; +long long base, size; +char *mem; +struct virtio_mmio_xen_info *info; +struct xenbus_transaction xbt; + +/* TODO: allocate an unused address here and pass it to the host instead */ + err =xenbus_scanf (XBT_NIL , dev->otherend,"base" ,"0x%llx" , + &base); +if (err <0 ) { +xenbus_dev_fatal (dev, err,"reading base" ); +return -EINVAL ; + } + + mem =xenbus_read (XBT_NIL , dev->otherend,"size" ,NULL ); +if (XENBUS_IS_ERR_READ (mem)) +return PTR_ERR (mem); + size =memparse (mem,NULL ); +kfree (mem); + + info =kzalloc_obj (*info); +if (!info) { +xenbus_dev_fatal (dev, -ENOMEM ,"allocating info structure" ); +return -ENOMEM ; + } + + info->resources[0 ].flags =IORESOURCE_MEM ; + info->resources[0 ].start = base; + info->resources[0 ].end = base + size -1 ; + + err =xenbus_alloc_evtchn (dev, &info->evtchn); +if (err) { +xenbus_dev_fatal (dev, err,"xenbus_alloc_evtchn" ); + goto error_info; + } + + err =bind_evtchn_to_irq (info->evtchn); +if (err <=0 ) { +xenbus_dev_fatal (dev, err,"bind_evtchn_to_irq" ); + goto error_evtchan; + } + + info->resources[1 ].flags =IORESOURCE_IRQ ; + info->resources[1 ].start = info->resources[1 ].end = err; + +again: + err =xenbus_transaction_start (&xbt); +if (err) { +xenbus_dev_fatal (dev, err,"starting transaction" ); + goto error_irq; + } + + err =xenbus_printf (xbt, dev->nodename,"event-channel" ,"%u" , + info->evtchn); +if (err) { +xenbus_transaction_end (xbt,1 ); +xenbus_dev_fatal (dev, err,"%s" ,"writing event-channel" ); + goto error_irq; + } + + err =xenbus_transaction_end (xbt,0 ); +if (err) { +if (err == -EAGAIN ) + goto again; +xenbus_dev_fatal (dev, err,"completing transaction" ); + goto error_irq; + } + +dev_set_drvdata (&dev->dev, info); +xenbus_switch_state (dev, XenbusStateInitialised); +return 0 ; + +error_irq: +unbind_from_irqhandler (info->resources[1 ].start, info); +error_evtchan: +xenbus_free_evtchn (dev, info->evtchn); +error_info: +kfree (info); + +return err; +} + +static void virtio_mmio_xen_backend_changed (struct xenbus_device *dev, +enum xenbus_state backend_state) +{ +struct virtio_mmio_xen_info *info =dev_get_drvdata (&dev->dev); + +switch (backend_state) { +case XenbusStateInitialising: +case XenbusStateInitWait: +case XenbusStateInitialised: +case XenbusStateReconfiguring: +case XenbusStateReconfigured: +case XenbusStateUnknown: +break ; + +case XenbusStateConnected: +if (dev->state != XenbusStateInitialised) { +dev_warn (&dev->dev,"state %d on connect" , dev->state); +break ; + } + info->pdev =platform_device_register_resndata (&dev->dev, +"virtio-mmio" ,PLATFORM_DEVID_AUTO , + info->resources,ARRAY_SIZE (info->resources),NULL ,0 ); +xenbus_switch_state (dev, XenbusStateConnected); +break ; + +case XenbusStateClosed: +if (dev->state == XenbusStateClosed) +break ; + fallthrough;/* Missed the backend's Closing state. */ +case XenbusStateClosing: +platform_device_unregister (info->pdev);
drivers/virtio/virtio_mmio.c static struct platform_driver virtio_mmio_driver = { · line 950
Quoted 10 lines
drivers/virtio/virtio_mmio.c +10
-0+xenbus_switch_state (dev, XenbusStateClosed); +break ; + +default : +xenbus_dev_fatal (dev, -EINVAL ,"saw state %d at frontend" , + backend_state); +break ; + } +} +
In some way, we're defining a new "PV driver" which is a virtio-mmio one, I guess we can eventually specific some form of protocol that backend/frontend would need to follow ?
Quoted 37 lines
drivers/virtio/virtio_mmio.c +32
-1+static void virtio_mmio_xen_remove (struct xenbus_device *dev) +{ +struct virtio_mmio_xen_info *info =dev_get_drvdata (&dev->dev); + +kfree (info); +dev_set_drvdata (&dev->dev,NULL ); +} + +static const struct xenbus_device_id virtio_mmio_xen_ids[] = { + {"virtio" }, + {"" }, +}; + +static struct xenbus_driver virtio_mmio_xen_driver = { + .ids = virtio_mmio_xen_ids, + .probe = virtio_mmio_xen_probe, + .otherend_changed = virtio_mmio_xen_backend_changed, + .remove = virtio_mmio_xen_remove, +}; +#endif +static int __init virtio_mmio_init (void ) { -return platform_driver_register (&virtio_mmio_driver); +int ret; + + ret =platform_driver_register (&virtio_mmio_driver); +if (ret) +return ret; + +#ifdef CONFIG_VIRTIO_MMIO_XENBUS +if (xen_domain ()) + ret =xenbus_register_frontend (&virtio_mmio_xen_driver); +#endif> + + return ret; }
drivers/virtio/virtio_mmio.c static struct platform_driver virtio_mmio_driver = { · line 997
Quoted 10 lines
drivers/virtio/virtio_mmio.c +5
-0static void __exit virtio_mmio_exit(void) { +#ifdef CONFIG_VIRTIO_MMIO_XENBUS +if (xen_domain ()) +xenbus_unregister_driver (&virtio_mmio_xen_driver); +#endif +platform_driver_unregister (&virtio_mmio_driver);vm_unregister_cmdline_devices (); }
--
Teddy Astie | Vates XCP-ng Developer
XCP-ng & Xen Orchestra - Vates solutions
web: https://vates.tech
Quoted 12 lines · Teddy Astie
Hello,
Le 29/04/2026 à 16:18, Val Packett a écrit :
Quoted 9 lines
[…]
I've been working on porting virtio-mmio support from Arm to
x86_64, with the goal of running vhost-user-gpu to power Wayland/GPU integration for Qubes OS. (I'm aware of various proposals for alternative virtio transports but virtio-mmio seems to be the only one that *is* upstream already and just Works..) Setting up virtio-mmio through xenbus, initially motivated just by event channels being the only real way to get interrupts working on HVM, turned out to generally be quite pleasant and nice :)
Quoted 2 lines
Is it HVM specific, or can we also make it work for PVH (we can actually attach a ioreq server to PVH guests) ?
Sorry, typo, I did mean PVH of course!
I've been testing this with PVH guests + PV dom0, with my PV alloc_ioreq fix:
https://lore.kernel.org/all/[email protected]/
(Time to resend that one as a non-RFC I guess…)
HVM actually does have legacy ISA interrupts (which are often used with virtio-mmio on KVM), funnily enough, and I've tried firing those from a DMOP but that silly thing didn't work properly.
Quoted 8 lines
Quoted 8 lines
I'd like to get some early feedback for this patch, particularly the general stuff:
* is this whole thing acceptable in general?
* should it be extracted into a different file?
* (from the Xen side) any input on the xenstore keys, what goes where?
* anything else to keep in mind?
Quoted 8 lines
Quoted 7 lines
It does seem simple enough, so hopefully this can be done?
The corresponding userspace-side WIP is available at:
https://github.com/QubesOS/xen-vhost-frontendAnd the required DMOP for firing the evtchn events will be sent to xen-devel shortly as well.
Could that be done through
evtchn_send(or its userland counterpart) ?
Actually, yes… The use of DMOPs is only dictated by the current Linux privcmd.c code (the irqfds created by the kernel react to events by executing HYPERVISOR_dm_op with a stored operation), we can avoid the need to modify Xen by simply expanding the privcmd driver to make "evtchn fds". Sounds good, will do.
Quoted 8 lines
Quoted 8 lines
[..]
drivers/virtio/Kconfig +0
-0@@ -171,6 +171,13 @@ config VIRTIO_MMIO_CMDLINE_DEVICES
Quoted 5 lines
drivers/virtio/Kconfig config VIRTIO_MMIO_CMDLINE_DEVICES · line 175
Quoted 4 lines
drivers/virtio/Kconfig +2
-0If unsure, say 'N'. +config VIRTIO_MMIO_XENBUS + bool "Memory mapped virtio devices parameter parsing"that text seems to miss the xenbus aspect
Yep, didn't change that yet, ack
Quoted 4 lines
Quoted 1 line
[..]
In some way, we're defining a new "PV driver" which is a virtio-mmio one, I guess we can eventually specific some form of protocol that backend/frontend would need to follow ?
Right, Jürgen mentioned documenting the keys in the xenstore-paths doc.. would the entire "protocol" (keys + state transition logic) fit into that?
The keys are currently derived from the initial Arm prototype which wasn't actually using xenbus properly (the guest driver was configured by a device tree node, but the ioreq server used xenstore keys, without properly transitioning between states).
Thanks,
~val
Le 30/04/2026 à 06:06, Val Packett a écrit :
Quoted 47 lines · Teddy Astie
Quoted 14 lines
Hello,
Le 29/04/2026 à 16:18, Val Packett a écrit :
Quoted 11 lines
[…]
I've been working on porting virtio-mmio support from Arm to
x86_64, with the goal of running vhost-user-gpu to power Wayland/GPU integration for Qubes OS. (I'm aware of various proposals for alternative virtio transports but virtio-mmio seems to be the only one that *is* upstream already and just Works..) Setting up virtio-mmio through xenbus, initially
motivated just by event channels being the only real way to get interrupts
working on HVM, turned out to generally be quite pleasant and nice :)Quoted 2 lines
Is it HVM specific, or can we also make it work for PVH (we can actually attach a ioreq server to PVH guests) ?
Sorry, typo, I did mean PVH of course!
I've been testing this with PVH guests + PV dom0, with my PV
alloc_ioreqfix:
https://lore.kernel.org/all/20251126062124.117425-1-
[email protected]/(Time to resend that one as a non-RFC I guess…)
HVM actually does have legacy ISA interrupts (which are often used with virtio-mmio on KVM), funnily enough, and I've tried firing those from a DMOP but that silly thing didn't work properly.
Quoted 15 lines
Quoted 15 lines
I'd like to get some early feedback for this patch, particularly the general stuff:
* is this whole thing acceptable in general?
* should it be extracted into a different file?
* (from the Xen side) any input on the xenstore keys, what goes where?
* anything else to keep in mind?It does seem simple enough, so hopefully this can be done?
The corresponding userspace-side WIP is available at:
https://github.com/QubesOS/xen-vhost-frontendAnd the required DMOP for firing the evtchn events will be sent to xen-devel shortly as well.
Quoted 8 lines
Quoted 1 line
Could that be done through
evtchn_send(or its userland counterpart) ?Actually, yes… The use of DMOPs is only dictated by the current Linux
privcmd.ccode (the irqfds created by the kernel react to events by executingHYPERVISOR_dm_opwith a stored operation), we can avoid the need to modify Xen by simply expanding the privcmd driver to make "evtchn fds". Sounds good, will do.
Given that the event channel used by device models is exposed through ioreq.vp_eport ("evtchn for notifications to/from device model"). I don't think you need to expand the privcmd interface, and you should be able to do this instead :
open /dev/xen/evtchn
perform IOCTL_EVTCHN_BIND_INTERDOMAIN (for each guest vCPU)
with remote_domain=guest_domid, remote_port=ioreq.vp_eport
Then interact with the event channel through IOCTL_EVTCHN_NOTIFY (with local port given by IOCTL_EVTCHN_BIND_INTERDOMAIN) and read/write on the file descriptor.
I have some experimental Rust code to work with event channels [1], but I think you can find similar code in multiples places.
[1]
https://github.com/TSnake41/rust-vmm-xen/blob/redesign-proposal/xen/src/event/mod.rs
https://github.com/TSnake41/rust-vmm-xen/blob/redesign-proposal/xen-unix/src/event/mod.rs
Quoted 20 lines
Quoted 11 lines
Quoted 10 lines
[..]
drivers/virtio/Kconfig +0
-0@@ -171,6 +171,13 @@ config VIRTIO_MMIO_CMDLINE_DEVICESIf unsure, say 'N'.
+configVIRTIO_MMIO_XENBUS
+ bool "Memory mapped virtio devices parameter parsing"that text seems to miss the xenbus aspect
Yep, didn't change that yet, ack
Quoted 4 lines
Quoted 1 line
[..]
In some way, we're defining a new "PV driver" which is a virtio-mmio one, I guess we can eventually specific some form of protocol that backend/frontend would need to follow ?
Right, Jürgen mentioned documenting the keys in the xenstore-paths doc.. would the entire "protocol" (keys + state transition logic) fit into that?
Quoted 10 lines
The keys are currently derived from the initial Arm prototype which wasn't actually using xenbus properly (the guest driver was configured by a device tree node, but the ioreq server used xenstore keys, without properly transitioning between states).
Thanks,
~val
Teddy
--
Teddy Astie | Vates XCP-ng Developer
XCP-ng & Xen Orchestra - Vates solutions
web: https://vates.tech
Quoted 58 lines · Teddy Astie
Le 30/04/2026 à 06:06, Val Packett a écrit :
Quoted 43 lines · Teddy Astie
Quoted 14 lines
Hello,
Le 29/04/2026 à 16:18, Val Packett a écrit :
Quoted 11 lines
[…]
I've been working on porting virtio-mmio support from Arm to
x86_64, with the goal of running vhost-user-gpu to power Wayland/GPU integration for Qubes OS. (I'm aware of various proposals for alternative virtio transports but virtio-mmio seems to be the only one that *is* upstream already and just Works..) Setting up virtio-mmio through xenbus, initially
motivated just by event channels being the only real way to get interrupts
working on HVM, turned out to generally be quite pleasant and nice :)Quoted 2 lines
Is it HVM specific, or can we also make it work for PVH (we can actually attach a ioreq server to PVH guests) ?
Sorry, typo, I did mean PVH of course!
I've been testing this with PVH guests + PV dom0, with my PV
alloc_ioreqfix:
https://lore.kernel.org/all/20251126062124.117425-1-
[email protected]/(Time to resend that one as a non-RFC I guess…)
HVM actually does have legacy ISA interrupts (which are often used with virtio-mmio on KVM), funnily enough, and I've tried firing those from a DMOP but that silly thing didn't work properly.
Quoted 13 lines
Quoted 13 lines
I'd like to get some early feedback for this patch, particularly the general stuff:
* is this whole thing acceptable in general?
* should it be extracted into a different file?
* (from the Xen side) any input on the xenstore keys, what goes where?
* anything else to keep in mind?It does seem simple enough, so hopefully this can be done?
The corresponding userspace-side WIP is available at:
https://github.com/QubesOS/xen-vhost-frontendQuoted 9 lines
Quoted 3 lines
Quoted 2 lines
And the required DMOP for firing the evtchn events will be sent to xen-devel shortly as well.
Could that be done through
evtchn_send(or its userland counterpart) ?Actually, yes… The use of DMOPs is only dictated by the current Linux
privcmd.ccode (the irqfds created by the kernel react to events by executingHYPERVISOR_dm_opwith a stored operation), we can avoid the need to modify Xen by simply expanding the privcmd driver to make "evtchn fds". Sounds good, will do.Given that the event channel used by device models is exposed through ioreq.vp_eport ("evtchn for notifications to/from device model"). I don't think you need to expand the privcmd interface, and you should be able to do this instead :
Quoted 7 lines
open
/dev/xen/evtchn
performIOCTL_EVTCHN_BIND_INTERDOMAIN(for each guest vCPU)
with remote_domain=guest_domid, remote_port=ioreq.vp_eportThen interact with the event channel through
IOCTL_EVTCHN_NOTIFY(with local port given byIOCTL_EVTCHN_BIND_INTERDOMAIN) and read/write on the file descriptor.
So the reason there's currently an ioctl to bind an eventfd to fire a stored DMOP is that the whole idea is to (efficiently!) support generic, hypervisor-neutral device server implementations via the vhost-user protocol.
Now of course, the current implementation isn't *entirely* hypervisor-neutral as e.g. the vm-memory Rust crate (inside of the "neutral" vhost-user device servers) does need to be built with the xen feature. But still, that's how it works. What can be made generic is generic.
xen-vhost-frontend, which is the thing that integrates these with Xen, actually used to handle the interrupts in userspace[1] by firing the DMOP itself (which is where I could "just replace that with IOCTL_EVTCHN_NOTIFY") but that was offloaded to the kernel with the introduction of IOCTL_PRIVCMD_IRQFD[2], similarly to KVM_IRQFD.
Switching back to handling the eventfd in userspace would be a literal deoptimization :)
While throwing away the whole generic layer to do a fully integrated use-case-specific thing sounds more difficult/tedious than this, and not necessarily desirable in general.
[1]: https://github.com/vireshk/xen-vhost-frontend/commit/06d59035f8a387c0f600931d09dfaa27b80ede7f
[2]: https://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git/commit/?id=f8941e6c4c712948663ec5d7bbb546f1a0f4e3f6
~val
Le 30/04/2026 à 10:51, Val Packett a écrit :
Quoted 80 lines · Teddy Astie
Quoted 60 lines
Le 30/04/2026 à 06:06, Val Packett a écrit :
Quoted 45 lines · Teddy Astie
Quoted 15 lines
Hello,
Le 29/04/2026 à 16:18, Val Packett a écrit :
Quoted 12 lines
[…]
I've been working on porting virtio-mmio support from Arm to
x86_64, with the goal of running vhost-user-gpu to power Wayland/GPU integration
for Qubes OS. (I'm aware of various proposals for alternative virtio transports but virtio-mmio seems to be the only one that *is* upstream already and just Works..) Setting up virtio-mmio through xenbus, initially
motivated just by event channels being the only real way to get interrupts
working on HVM, turned out to generally be quite pleasant and nice :)Quoted 3 lines
Is it HVM specific, or can we also make it work for PVH (we can actually
attach a ioreq server to PVH guests) ?Sorry, typo, I did mean PVH of course!
I've been testing this with PVH guests + PV dom0, with my PV
alloc_ioreqfix:
https://lore.kernel.org/all/20251126062124.117425-1-
[email protected]/(Time to resend that one as a non-RFC I guess…)
HVM actually does have legacy ISA interrupts (which are often used with virtio-mmio on KVM), funnily enough, and I've tried firing those from a DMOP but that silly thing didn't work properly.
Quoted 13 lines
Quoted 13 lines
I'd like to get some early feedback for this patch, particularly the general stuff:
* is this whole thing acceptable in general?
* should it be extracted into a different file?
* (from the Xen side) any input on the xenstore keys, what goes where?
* anything else to keep in mind?It does seem simple enough, so hopefully this can be done?
The corresponding userspace-side WIP is available at:
https://github.com/QubesOS/xen-vhost-frontendQuoted 9 lines
Quoted 3 lines
Quoted 2 lines
And the required DMOP for firing the evtchn events will be sent to xen-devel shortly as well.
Could that be done through
evtchn_send(or its userland counterpart) ?Actually, yes… The use of DMOPs is only dictated by the current Linux
privcmd.ccode (the irqfds created by the kernel react to events by executingHYPERVISOR_dm_opwith a stored operation), we can avoid the need to modify Xen by simply expanding the privcmd driver to make "evtchn fds". Sounds good, will do.Given that the event channel used by device models is exposed through ioreq.vp_eport ("evtchn for notifications to/from device model"). I don't think you need to expand the privcmd interface, and you should be able to do this instead :
Quoted 7 lines
open
/dev/xen/evtchn
performIOCTL_EVTCHN_BIND_INTERDOMAIN(for each guest vCPU) with remote_domain=guest_domid, remote_port=ioreq.vp_eportThen interact with the event channel through
IOCTL_EVTCHN_NOTIFY(with local port given byIOCTL_EVTCHN_BIND_INTERDOMAIN) and read/write on the file descriptor.So the reason there's currently an ioctl to bind an eventfd to fire a stored DMOP is that the whole idea is to (efficiently!) support generic, hypervisor-neutral device server implementations via the vhost-user protocol.
Now of course, the current implementation isn't *entirely* hypervisor- neutral as e.g. the vm-memory Rust crate (inside of the "neutral" vhost- user device servers) does need to be built with the
xenfeature. But still, that's how it works. What can be made generic is generic.
Quoted 6 lines
xen-vhost-frontend, which is the thing that integrates these with Xen, actually used to handle the interrupts in userspace[1] by firing the DMOP itself (which is where I could "just replace that with
IOCTL_EVTCHN_NOTIFY") but that was offloaded to the kernel with the introduction of IOCTL_PRIVCMD_IRQFD[2], similarly toKVM_IRQFD.
I think what would be preferable for your usecase would be to have a way to bind a event channel with a eventfd object, which should be a primitive that lives in the evtchn device.
The current interface kinda assume that you're looking to emulate a completely emulated virtio device with no Xen specifics, it looks like it's not exactly what you're implementing.
As you actually plan to switch to using event channels for notifying the guest, I think it would be preferable to do the same the other way (event channels to notify the host) so you only have event channels to worry about here.
Quoted 6 lines
Switching back to handling the eventfd in userspace would be a literal deoptimization :)
Quoted 1 line
While throwing away the whole generic layer to do a fully integrated
use-case-specific thing sounds more difficult/tedious than this, and not necessarily desirable in general.
Quoted 8 lines
[1]: https://github.com/vireshk/xen-vhost-frontend/
commit/06d59035f8a387c0f600931d09dfaa27b80ede7f
[2]: https://git.kernel.org/pub/scm/linux/kernel/git/next/linux-
next.git/commit/?id=f8941e6c4c712948663ec5d7bbb546f1a0f4e3f6~val
--
Teddy Astie | Vates XCP-ng Developer
XCP-ng & Xen Orchestra - Vates solutions
web: https://vates.tech
Quoted 54 lines · Teddy Astie
Le 30/04/2026 à 10:51, Val Packett a écrit :
Quoted 47 lines · Teddy Astie
Quoted 29 lines
Le 30/04/2026 à 06:06, Val Packett a écrit :
Quoted 14 lines
[..]
Quoted 13 lines
Quoted 13 lines
I'd like to get some early feedback for this patch, particularly the general stuff:
* is this whole thing acceptable in general?
* should it be extracted into a different file?
* (from the Xen side) any input on the xenstore keys, what goes where?
* anything else to keep in mind?It does seem simple enough, so hopefully this can be done?
The corresponding userspace-side WIP is available at:
https://github.com/QubesOS/xen-vhost-frontendQuoted 9 lines
Quoted 3 lines
Quoted 2 lines
And the required DMOP for firing the evtchn events will be sent to xen-devel shortly as well.
Could that be done through
evtchn_send(or its userland counterpart) ?Actually, yes… The use of DMOPs is only dictated by the current Linux
privcmd.ccode (the irqfds created by the kernel react to events by executingHYPERVISOR_dm_opwith a stored operation), we can avoid the need to modify Xen by simply expanding the privcmd driver to make "evtchn fds". Sounds good, will do.Given that the event channel used by device models is exposed through ioreq.vp_eport ("evtchn for notifications to/from device model"). I don't think you need to expand the privcmd interface, and you should be able to do this instead :
Quoted 7 lines
open
/dev/xen/evtchn
performIOCTL_EVTCHN_BIND_INTERDOMAIN(for each guest vCPU)
with remote_domain=guest_domid, remote_port=ioreq.vp_eportThen interact with the event channel through
IOCTL_EVTCHN_NOTIFY(with local port given byIOCTL_EVTCHN_BIND_INTERDOMAIN) and read/write on the file descriptor.So the reason there's currently an ioctl to bind an eventfd to fire a stored DMOP is that the whole idea is to (efficiently!) support generic, hypervisor-neutral device server implementations via the vhost-user protocol.
Now of course, the current implementation isn't *entirely* hypervisor- neutral as e.g. the vm-memory Rust crate (inside of the "neutral" vhost- user device servers) does need to be built with the
xenfeature. But still, that's how it works. What can be made generic is generic.Quoted 6 lines
xen-vhost-frontend, which is the thing that integrates these with Xen, actually used to handle the interrupts in userspace[1] by firing the DMOP itself (which is where I could "just replace that with
IOCTL_EVTCHN_NOTIFY") but that was offloaded to the kernel with the introduction of IOCTL_PRIVCMD_IRQFD[2], similarly toKVM_IRQFD.
Quoted 3 lines
I think what would be preferable for your usecase would be to have a way to bind a event channel with a eventfd object, which should be a primitive that lives in the evtchn device.
Yeah, it would be an ioctl on the evtchn device, definitely. I wasn't being exact when I said "extend privcmd", sorry. I just meant "handling it on the Linux side" generally!
Quoted 3 lines
The current interface kinda assume that you're looking to emulate a completely emulated virtio device with no Xen specifics, it looks like it's not exactly what you're implementing.
It's already implemented, and I'm not looking to change it much, just to make it work on x86_64. The only thing that wasn't already compatible was firing the host-to-guest interrupt, because on x86_64 we don't have anything like the (v)GIC with its massive arbitrary IRQ number space. Event channels are the only way to interrupt a PVH guest, hence using xenbus in the guest to provision the device.
Quoted 4 lines
As you actually plan to switch to using event channels for notifying the guest, I think it would be preferable to do the same the other way (event channels to notify the host) so you only have event channels to worry about here.
The other direction is already implemented perfectly well in IOCTL_PRIVCMD_IOEVENTFD. The MMIO area is set up like so:
- ioreq is mapped with IOCTL_PRIVCMD_MMAP_RESOURCE(XENMEM_resource_ioreq_server, ..);
- vp_eport event channels (per cpu) are bound to the current domain via IOCTL_EVTCHN_BIND_INTERDOMAIN;
- those are passed, along with the ioreq page itself, to IOCTL_PRIVCMD_IOEVENTFD to get an eventfd that fires when a virtqueue is ready;
- which is an eventfd that xen-vhost-frontend passes to the vhost-user device server.
So for this direction, it's not a 1:1 mapping but rather a specific contraption designed to efficiently handle this use case:
- when an ioreq event channel (for any of the vcpus) fires,
- the kernel handler (ioeventfd_interrupt) checks if it's specifically an IOREQ_WRITE write to the VIRTIO_MMIO_QUEUE_NOTIFY offset,
- and if so, it signals the eventfd for any virtqueue that has new data (waking the generic device server which has the eventfd, so bypassing xen-vhost-frontend), pings the guest back via evtchn, and returns IRQ_HANDLED;
- otherwise the request is handled in userspace by xen-vhost-frontend (virtio configuration register access).
It just works :)
~val
Some minor details from the Xen side of things:
Quoted 5 lines · Val Packett
The experimental virtio-mmio support for Xen was initially developed on aarch64, so device trees were used to configure the mmio devices, with arbitrary vGIC interrupts used by the hypervisor. On
x86_64however, the only reasonable way to interrupt the guest is over Xen event channels, which can only be acquired by children of xenbus,
More exact: interdomain event channels need to be connected to a xenbus device. But you are needing those, so for your use case the above statement is correct.
Quoted 21 lines
the virtual bus driven by Xen's configuration database, XenStore. It is also a more convenient and "Xen-ish" way to provision devices.
Implement a xenbus client for virtio-mmio which negotiates an event channel and provides it as a platform IRQ to the virtio-mmio driver.
Signed-off-by Val Packett <[email protected]>---
Hi,
I've been working on porting virtio-mmio support from Arm to
x86_64, with the goal of running vhost-user-gpu to power Wayland/GPU integration for Qubes OS. (I'm aware of various proposals for alternative virtio transports but virtio-mmio seems to be the only one that *is* upstream already and just Works..) Setting up virtio-mmio through xenbus, initially motivated just by event channels being the only real way to get interrupts working on HVM, turned out to generally be quite pleasant and nice :)
Quoted 6 lines
I'd like to get some early feedback for this patch, particularly the general stuff:
* is this whole thing acceptable in general?
* should it be extracted into a different file?
* (from the Xen side) any input on the xenstore keys, what goes where?
You should add some documentation in the Xen source tree regarding the Xenstore keys (see docs/misc/xenstore-paths.pandoc there).
Quoted 63 lines
* anything else to keep in mind?
It does seem simple enough, so hopefully this can be done?
The corresponding userspace-side WIP is available at:
https://github.com/QubesOS/xen-vhost-frontendAnd the required DMOP for firing the evtchn events will be sent to xen-devel shortly as well.
Thanks,
~val--- drivers/virtio/Kconfig | 7 ++ drivers/virtio/virtio_mmio.c | 177 ++++++++++++++++++++++++++++++++++- 2 files changed, 183 insertions(+), 1 deletion(-)drivers/virtio/Kconfig +7
-0@@ -171,6 +171,13 @@ config VIRTIO_MMIO_CMDLINE_DEVICES If unsure, say 'N'. +config VIRTIO_MMIO_XENBUS + bool "Memory mapped virtio devices parameter parsing" + depends on VIRTIO_MMIO && XEN + select XEN_XENBUS_FRONTEND + help + Allow virtio-mmio devices instantiation for Xen guests via xenbus. + config VIRTIO_DMA_SHARED_BUFFER tristate depends on DMA_SHARED_BUFFERdrivers/virtio/virtio_mmio.c +12
-0@@ -70,6 +70,11 @@#include <uapi/linux/virtio_mmio.h> #include <linux/virtio_ring.h> +#ifdef CONFIG_VIRTIO_MMIO_XENBUS +#include <xen/xen.h> +#include <xen/xenbus.h> +#include <xen/events.h> +#endif /* The alignment to use between consumer and producer parts of vring. @@ -810,13 +815,183 @@ static struct platform_driver virtio_mmio_driver = { }, }; +#ifdef CONFIG_VIRTIO_MMIO_XENBUS +struct virtio_mmio_xen_info { +struct resource resources[2 ]; +unsigned int evtchn; +struct platform_device *pdev; +}; +
drivers/virtio/virtio_mmio.c static struct platform_driver virtio_mmio_driver = { · line 834
Quoted 10 lines
drivers/virtio/virtio_mmio.c +10
-0+static int virtio_mmio_xen_probe (struct xenbus_device *dev, +const struct xenbus_device_id *id) +{ +int err; +long long base, size; +char *mem; +struct virtio_mmio_xen_info *info; +struct xenbus_transaction xbt; + +/* TODO: allocate an unused address here and pass it to the host instead */
Indeed.
Quoted 30 lines
drivers/virtio/virtio_mmio.c +30
-0+ err =xenbus_scanf (XBT_NIL , dev->otherend,"base" ,"0x%llx" , + &base); +if (err <0 ) { +xenbus_dev_fatal (dev, err,"reading base" ); +return -EINVAL ; + } + + mem =xenbus_read (XBT_NIL , dev->otherend,"size" ,NULL ); +if (XENBUS_IS_ERR_READ (mem)) +return PTR_ERR (mem); + size =memparse (mem,NULL ); +kfree (mem); + + info =kzalloc_obj (*info); +if (!info) { +xenbus_dev_fatal (dev, -ENOMEM ,"allocating info structure" ); +return -ENOMEM ; + } + + info->resources[0 ].flags =IORESOURCE_MEM ; + info->resources[0 ].start = base; + info->resources[0 ].end = base + size -1 ; + + err =xenbus_alloc_evtchn (dev, &info->evtchn); +if (err) { +xenbus_dev_fatal (dev, err,"xenbus_alloc_evtchn" ); + goto error_info; + } + + err =bind_evtchn_to_irq (info->evtchn);
drivers/virtio/virtio_mmio.c static struct platform_driver virtio_mmio_driver = { · line 874
Quoted 10 lines
drivers/virtio/virtio_mmio.c +10
-0+if (err <=0 ) { +xenbus_dev_fatal (dev, err,"bind_evtchn_to_irq" ); + goto error_evtchan; + } + + info->resources[1 ].flags =IORESOURCE_IRQ ; + info->resources[1 ].start = info->resources[1 ].end = err; + +again: + err =xenbus_transaction_start (&xbt);
No need to use a Xenstore transaction here. The written node(s) are regarded to be valid only after calling xenbus_switch_state() to set the frontend state to XenbusStateInitialised.
drivers/virtio/virtio_mmio.c static struct platform_driver virtio_mmio_driver = { · line 881
Quoted 7 lines
drivers/virtio/virtio_mmio.c +7
-0+if (err) { +xenbus_dev_fatal (dev, err,"starting transaction" ); + goto error_irq; + } + + err =xenbus_printf (xbt, dev->nodename,"event-channel" ,"%u" , + info->evtchn);
With allocation of the base address you'd want to write it to another node, of course.
Quoted 69 lines
drivers/virtio/virtio_mmio.c +69
-0+if (err) { +xenbus_transaction_end (xbt,1 ); +xenbus_dev_fatal (dev, err,"%s" ,"writing event-channel" ); + goto error_irq; + } + + err =xenbus_transaction_end (xbt,0 ); +if (err) { +if (err == -EAGAIN ) + goto again; +xenbus_dev_fatal (dev, err,"completing transaction" ); + goto error_irq; + } + +dev_set_drvdata (&dev->dev, info); +xenbus_switch_state (dev, XenbusStateInitialised); +return 0 ; + +error_irq: +unbind_from_irqhandler (info->resources[1 ].start, info); +error_evtchan: +xenbus_free_evtchn (dev, info->evtchn); +error_info: +kfree (info); + +return err; +} + +static void virtio_mmio_xen_backend_changed (struct xenbus_device *dev, +enum xenbus_state backend_state) +{ +struct virtio_mmio_xen_info *info =dev_get_drvdata (&dev->dev); + +switch (backend_state) { +case XenbusStateInitialising: +case XenbusStateInitWait: +case XenbusStateInitialised: +case XenbusStateReconfiguring: +case XenbusStateReconfigured: +case XenbusStateUnknown: +break ; + +case XenbusStateConnected: +if (dev->state != XenbusStateInitialised) { +dev_warn (&dev->dev,"state %d on connect" , dev->state); +break ; + } + info->pdev =platform_device_register_resndata (&dev->dev, +"virtio-mmio" ,PLATFORM_DEVID_AUTO , + info->resources,ARRAY_SIZE (info->resources),NULL ,0 ); +xenbus_switch_state (dev, XenbusStateConnected); +break ; + +case XenbusStateClosed: +if (dev->state == XenbusStateClosed) +break ; + fallthrough;/* Missed the backend's Closing state. */ +case XenbusStateClosing: +platform_device_unregister (info->pdev); +xenbus_switch_state (dev, XenbusStateClosed); +break ; + +default : +xenbus_dev_fatal (dev, -EINVAL ,"saw state %d at frontend" , + backend_state); +break ; + } +} +
drivers/virtio/virtio_mmio.c static struct platform_driver virtio_mmio_driver = { · line 960
Quoted 10 lines
drivers/virtio/virtio_mmio.c +10
-0+static void virtio_mmio_xen_remove (struct xenbus_device *dev) +{ +struct virtio_mmio_xen_info *info =dev_get_drvdata (&dev->dev); + +kfree (info); +dev_set_drvdata (&dev->dev,NULL ); +} + +static const struct xenbus_device_id virtio_mmio_xen_ids[] = { + {"virtio" },
Please use "virtio-mmio" here, as I could imagine "virtio-pci" devices, too.
Juergen
Quoted 38 lines
drivers/virtio/virtio_mmio.c +28
-1+ { "" }, +}; + +static struct xenbus_driver virtio_mmio_xen_driver = { + .ids = virtio_mmio_xen_ids, + .probe = virtio_mmio_xen_probe, + .otherend_changed = virtio_mmio_xen_backend_changed, + .remove = virtio_mmio_xen_remove, +}; +#endif + static int __init virtio_mmio_init(void) { -return platform_driver_register (&virtio_mmio_driver); + int ret; + + ret = platform_driver_register(&virtio_mmio_driver); + if (ret) + return ret; + +#ifdef CONFIG_VIRTIO_MMIO_XENBUS + if (xen_domain()) + ret = xenbus_register_frontend(&virtio_mmio_xen_driver); +#endif + + return ret; } static void __exit virtio_mmio_exit(void) { +#ifdef CONFIG_VIRTIO_MMIO_XENBUS + if (xen_domain()) + xenbus_unregister_driver(&virtio_mmio_xen_driver); +#endif + platform_driver_unregister(&virtio_mmio_driver); vm_unregister_cmdline_devices(); }
Quoted 37 lines · Jürgen Groß
Some minor details from the Xen side of things:
Quoted 5 lines · Val Packett
The experimental virtio-mmio support for Xen was initially developed on aarch64, so device trees were used to configure the mmio devices, with arbitrary vGIC interrupts used by the hypervisor. On
x86_64however, the only reasonable way to interrupt the guest is over Xen event channels, which can only be acquired by children of xenbus,More exact: interdomain event channels need to be connected to a xenbus device. But you are needing those, so for your use case the above statement is correct.
Quoted 24 lines
the virtual bus driven by Xen's configuration database, XenStore. It is also a more convenient and "Xen-ish" way to provision devices.
Implement a xenbus client for virtio-mmio which negotiates an event channel and provides it as a platform IRQ to the virtio-mmio driver.
Signed-off-by Val Packett <[email protected]>---
Hi,
I've been working on porting virtio-mmio support from Arm to
x86_64, with the goal of running vhost-user-gpu to power Wayland/GPU integration for Qubes OS. (I'm aware of various proposals for alternative virtio transports but virtio-mmio seems to be the only one that *is* upstream already and just Works..) Setting up virtio-mmio through xenbus, initially motivated just by event channels being the only real way to get interrupts working on HVM, turned out to generally be quite pleasant and nice :)I'd like to get some early feedback for this patch, particularly the general stuff:
Quoted 6 lines
Quoted 3 lines
* is this whole thing acceptable in general?
* should it be extracted into a different file?
* (from the Xen side) any input on the xenstore keys, what goes where?You should add some documentation in the Xen source tree regarding the Xenstore keys (see docs/misc/xenstore-paths.pandoc there).
Ack, thanks!
Quoted 8 lines
Quoted 4 lines
[…]
+again:
+ err =xenbus_transaction_start(&xbt);No need to use a Xenstore transaction here. The written node(s) are regarded to be valid only after calling
xenbus_switch_state()to set the frontend state to XenbusStateInitialised.
Oh, I assumed transactions were required for writing from the kernel to work at all…
Quoted 6 lines
Quoted 4 lines
[..]
+static const struct
xenbus_device_idvirtio_mmio_xen_ids[] = { + { "virtio" },Please use "virtio-mmio" here, as I could imagine "virtio-pci" devices, too.
Ack. Would actually also distinguish it from the initial Arm proof-of-concept version…
Thanks,
~val