Skip to contentSkip to conversation

Add support for MSI injection on Arm

PATCH In review Mykyta Poturai 6 participants 23 messages Jan 14, 2024 – Mar 31, 2025

lore
Share & tools
lore
#0 lore raw

cover Add support for MSI injection on Arm

This series adds the base support for MSI injection on Arm. This is needed to streamline virtio-pci interrupt triggering.

With this patches, MSIs can be triggered in guests by issuing the new DM op, inject_msi2. This op is similar to inject_msi, but it allows to specify the source id of the MSI.

We chose the approach of adding a new DM op instead of using the pad field of inject_msi because we have no clear way of distinguishing between set and unset pad fields. New implementations also adds flags field to clearly specify if the SBDF is set.

Patches were tested on QEMU with QEMU virtio-pci backends, with virtio-pci patches and patches for ITS support for DomUs applied.

Virtio-pci patches:
https://github.com/xen-troops/xen/tree/xen-4.18-xt0.2

ITS patches:
https://github.com/stewdk/xen/commits/pcie-passthrough-arm-vpci-v11

Mykyta Poturai (2):
arm: vgic: Add the ability to trigger MSIs from the Hypervisor xen/dm: arm: Introduce inject_msi2 DM op

 tools/include/xendevicemodel.h               | 14 ++++++++
 tools/libs/devicemodel/core.c                | 22 ++++++++++++
 tools/libs/devicemodel/libxendevicemodel.map |  5 +++
 xen/arch/arm/dm.c                            | 15 +++++++++
 xen/arch/arm/include/asm/vgic.h              | 11 ++++++
 xen/arch/arm/vgic-v3-its.c                   | 35 ++++++++++++++++++++
 xen/arch/x86/hvm/dm.c                        | 13 ++++++++
 xen/include/public/hvm/dm_op.h               | 12 +++++++
 8 files changed, 127 insertions(+)
Signature · 1 line
2.34.1
#1 lore raw

PATCH 1/2 arm: vgic: Add the ability to trigger MSIs from the Hypervisor

Add the vgic_its_trigger_msi() function to the vgic interface. This function allows to inject MSIs from the Hypervisor to the guest. Which is useful for userspace PCI backend drivers.

---
 xen/arch/arm/include/asm/vgic.h | 11 +++++++++++
 xen/arch/arm/vgic-v3-its.c      | 35 +++++++++++++++++++++++++++++++++
 2 files changed, 46 insertions(+)
xen/arch/arm/include/asm/vgic.h +11 -0
index 922779ce14..4695743848 100644
@@ -317,6 +317,17 @@ extern bool vgic_migrate_irq(struct vcpu *old, struct vcpu *new, unsigned int ir
 extern void vgic_check_inflight_irqs_pending(struct domain *d, struct vcpu *v,
                                              unsigned int rank, uint32_t r);
 
+#ifdef CONFIG_HAS_ITS
+int vgic_its_trigger_msi(struct domain *d, paddr_t doorbell_address,
+                                u32 devid, u32 eventid);
+#else
+static inline int vgic_its_trigger_msi(struct domain *d, paddr_t doorbell_address,
+                                u32 devid, u32 eventid)
+{
+    return -EOPNOTSUPP;
+}
+#endif /* CONFIG_HAS_ITS */
+
 #endif /* !CONFIG_NEW_VGIC */
 
 /*** Common VGIC functions used by Xen arch code ****/
xen/arch/arm/vgic-v3-its.c +35 -0
index 70b5aeb822..683a378f6e 100644
@@ -1484,6 +1484,41 @@ static int vgic_v3_its_init_virtual(struct domain *d, paddr_t guest_addr,
     return 0;
 }
 
+int vgic_its_trigger_msi(struct domain *d, paddr_t doorbell_address,
+                                u32 devid, u32 eventid)
+{
+    struct vcpu *vcpu;
+    struct virt_its *pos, *temp;
+    struct virt_its *its = NULL;
+    uint32_t vlpi;
+    bool ret;
+
+    list_for_each_entry_safe( pos, temp, &d->arch.vgic.vits_list, vits_list )
+    {
+        if ( pos->doorbell_address == doorbell_address )
+        {
+            its = pos;
+            break;
+        }
+    }
+
+    if ( !its )
+        return -EINVAL;
+
+    spin_lock(&its->its_lock);
+    ret = read_itte(its, devid, eventid, &vcpu, &vlpi);
+    spin_unlock(&its->its_lock);
+    if ( !ret )
+        return -1;
+
+    if ( vlpi == INVALID_LPI )
+        return -1;
+
+    vgic_vcpu_inject_lpi(its->d, vlpi);
+
+    return 0;
+}
+
 unsigned int vgic_v3_its_count(const struct domain *d)
 {
     struct host_its *hw_its;
Signature · 1 line
2.34.1
#2 lore raw

Re: [PATCH 1/2] arm: vgic: Add the ability to trigger MSIs from the Hypervisor

Quoted 59 lines · Mykyta Poturai

Add the vgic_its_trigger_msi() function to the vgic interface. This function allows to inject MSIs from the Hypervisor to the guest. Which is useful for userspace PCI backend drivers.

---
 xen/arch/arm/include/asm/vgic.h | 11 +++++++++++
 xen/arch/arm/vgic-v3-its.c      | 35 +++++++++++++++++++++++++++++++++
 2 files changed, 46 insertions(+)
xen/arch/arm/include/asm/vgic.h +11 -0
index 922779ce14..4695743848 100644
@@ -317,6 +317,17 @@ extern bool vgic_migrate_irq(struct vcpu *old, struct vcpu *new, unsigned int ir
 extern void vgic_check_inflight_irqs_pending(struct domain *d, struct vcpu *v,
                                              unsigned int rank, uint32_t r);
 
+#ifdef CONFIG_HAS_ITS
+int vgic_its_trigger_msi(struct domain *d, paddr_t doorbell_address,
+                                u32 devid, u32 eventid);
+#else
+static inline int vgic_its_trigger_msi(struct domain *d, paddr_t doorbell_address,
+                                u32 devid, u32 eventid)
+{
+    return -EOPNOTSUPP;
+}
+#endif /* CONFIG_HAS_ITS */
+
 #endif /* !CONFIG_NEW_VGIC */
 
 /*** Common VGIC functions used by Xen arch code ****/
xen/arch/arm/vgic-v3-its.c +19 -0
index 70b5aeb822..683a378f6e 100644
@@ -1484,6 +1484,41 @@ static int vgic_v3_its_init_virtual(struct domain *d, paddr_t guest_addr,
     return 0;
 }
 
+int vgic_its_trigger_msi(struct domain *d, paddr_t doorbell_address,
+                                u32 devid, u32 eventid)
+{
+    struct vcpu *vcpu;
+    struct virt_its *pos, *temp;
+    struct virt_its *its = NULL;
+    uint32_t vlpi;
+    bool ret;
+
+    list_for_each_entry_safe( pos, temp, &d->arch.vgic.vits_list, vits_list )
+    {
+        if ( pos->doorbell_address == doorbell_address )
+        {
+            its = pos;
+            break;
+        }
+    }
+
+    if ( !its )

xen/arch/arm/vgic-v3-its.c static int vgic_v3_its_init_virtual(struct domain *d, paddr_t guest_addr, · line 1515

Quoted 10 lines
xen/arch/arm/vgic-v3-its.c +10 -0
+        return -EINVAL;
+
+    spin_lock(&its->its_lock);
+    ret = read_itte(its, devid, eventid, &vcpu, &vlpi);
+    spin_unlock(&its->its_lock);
+    if ( !ret )
+        return -1;
+
+    if ( vlpi == INVALID_LPI )
+        return -1;

We need a better error code, maybe EINVAL or ENOENT ?

Other than that, it looks OK

Quoted 11 lines
xen/arch/arm/vgic-v3-its.c +5 -0
+    vgic_vcpu_inject_lpi(its->d, vlpi);
+
+    return 0;
+}
+
 unsigned int vgic_v3_its_count(const struct domain *d)
 {
     struct host_its *hw_its;
Signature · 1 line
2.34.1
#3 lore raw

Re: [PATCH 1/2] arm: vgic: Add the ability to trigger MSIs from the Hypervisor

Hi,

Quoted 64 lines · Stefano Stabellini
Quoted 63 lines · Mykyta Poturai

Add the vgic_its_trigger_msi() function to the vgic interface. This function allows to inject MSIs from the Hypervisor to the guest. Which is useful for userspace PCI backend drivers.

---
  xen/arch/arm/include/asm/vgic.h | 11 +++++++++++
  xen/arch/arm/vgic-v3-its.c      | 35 +++++++++++++++++++++++++++++++++
  2 files changed, 46 insertions(+)
xen/arch/arm/include/asm/vgic.h +11 -0
index 922779ce14..4695743848 100644
@@ -317,6 +317,17 @@ extern bool vgic_migrate_irq(struct vcpu *old, struct vcpu *new, unsigned int ir
  extern void vgic_check_inflight_irqs_pending(struct domain *d, struct vcpu *v,
                                               unsigned int rank, uint32_t r);
 
+#ifdef CONFIG_HAS_ITS
+int vgic_its_trigger_msi(struct domain *d, paddr_t doorbell_address,
+                                u32 devid, u32 eventid);
+#else
+static inline int vgic_its_trigger_msi(struct domain *d, paddr_t doorbell_address,
+                                u32 devid, u32 eventid)
+{
+    return -EOPNOTSUPP;
+}
+#endif /* CONFIG_HAS_ITS */
+
  #endif /* !CONFIG_NEW_VGIC */
 
  /*** Common VGIC functions used by Xen arch code ****/
xen/arch/arm/vgic-v3-its.c +23 -0
index 70b5aeb822..683a378f6e 100644
@@ -1484,6 +1484,41 @@ static int vgic_v3_its_init_virtual(struct domain *d, paddr_t guest_addr,
      return 0;
  }
 
+int vgic_its_trigger_msi(struct domain *d, paddr_t doorbell_address,
+                                u32 devid, u32 eventid)
+{
+    struct vcpu *vcpu;
+    struct virt_its *pos, *temp;
+    struct virt_its *its = NULL;
+    uint32_t vlpi;
+    bool ret;
+
+    list_for_each_entry_safe( pos, temp, &d->arch.vgic.vits_list, vits_list )
+    {
+        if ( pos->doorbell_address == doorbell_address )
+        {
+            its = pos;
+            break;
+        }
+    }
+
+    if ( !its )
+        return -EINVAL;
+
+    spin_lock(&its->its_lock);
+    ret = read_itte(its, devid, eventid, &vcpu, &vlpi);
Quoted 8 lines

xen/arch/arm/vgic-v3-its.c static int vgic_v3_its_init_virtual(struct domain *d, paddr_t guest_addr, · line 1515

Quoted 6 lines
xen/arch/arm/vgic-v3-its.c +6 -0
+    spin_unlock(&its->its_lock);
+    if ( !ret )
+        return -1;
+
+    if ( vlpi == INVALID_LPI )
+        return -1;

We need a better error code, maybe EINVAL or ENOENT ?

EINVAL tends to be overloaded. I would prefer ENOENT.

Cheers,

Signature · 1 line
Julien Grall
#4 lore raw

Re: [PATCH 1/2] arm: vgic: Add the ability to trigger MSIs from the Hypervisor

In reply to Mykyta Poturai

Hi,

Quoted 59 lines · Mykyta Poturai

Add the vgic_its_trigger_msi() function to the vgic interface. This function allows to inject MSIs from the Hypervisor to the guest. Which is useful for userspace PCI backend drivers.

---
  xen/arch/arm/include/asm/vgic.h | 11 +++++++++++
  xen/arch/arm/vgic-v3-its.c      | 35 +++++++++++++++++++++++++++++++++
  2 files changed, 46 insertions(+)
xen/arch/arm/include/asm/vgic.h +11 -0
index 922779ce14..4695743848 100644
@@ -317,6 +317,17 @@ extern bool vgic_migrate_irq(struct vcpu *old, struct vcpu *new, unsigned int ir
  extern void vgic_check_inflight_irqs_pending(struct domain *d, struct vcpu *v,
                                               unsigned int rank, uint32_t r);
 
+#ifdef CONFIG_HAS_ITS
+int vgic_its_trigger_msi(struct domain *d, paddr_t doorbell_address,
+                                u32 devid, u32 eventid);
+#else
+static inline int vgic_its_trigger_msi(struct domain *d, paddr_t doorbell_address,
+                                u32 devid, u32 eventid)
+{
+    return -EOPNOTSUPP;
+}
+#endif /* CONFIG_HAS_ITS */
+
  #endif /* !CONFIG_NEW_VGIC */
 
  /*** Common VGIC functions used by Xen arch code ****/
xen/arch/arm/vgic-v3-its.c +19 -0
index 70b5aeb822..683a378f6e 100644
@@ -1484,6 +1484,41 @@ static int vgic_v3_its_init_virtual(struct domain *d, paddr_t guest_addr,
      return 0;
  }
 
+int vgic_its_trigger_msi(struct domain *d, paddr_t doorbell_address,
+                                u32 devid, u32 eventid)
+{
+    struct vcpu *vcpu;
+    struct virt_its *pos, *temp;
+    struct virt_its *its = NULL;
+    uint32_t vlpi;
+    bool ret;
+
+    list_for_each_entry_safe( pos, temp, &d->arch.vgic.vits_list, vits_list )
+    {
+        if ( pos->doorbell_address == doorbell_address )
+        {
+            its = pos;
+            break;
+        }
+    }
+
+    if ( !its )

xen/arch/arm/vgic-v3-its.c static int vgic_v3_its_init_virtual(struct domain *d, paddr_t guest_addr, · line 1515

Quoted 10 lines
xen/arch/arm/vgic-v3-its.c +10 -0
+        return -EINVAL;
+
+    spin_lock(&its->its_lock);
+    ret = read_itte(its, devid, eventid, &vcpu, &vlpi);
+    spin_unlock(&its->its_lock);
+    if ( !ret )
+        return -1;
+
+    if ( vlpi == INVALID_LPI )
+        return -1;

Reading the code, I think you want to use get_event_pending_irq(). This will return the associated pending_irq. Then you can...

xen/arch/arm/vgic-v3-its.c static int vgic_v3_its_init_virtual(struct domain *d, paddr_t guest_addr, · line 1517

Quoted 2 lines
xen/arch/arm/vgic-v3-its.c +2 -0
+
+    vgic_vcpu_inject_lpi(its->d, vlpi);

... open-code vgic_vcpu_inject_lpi(). This would avoid access the guest memory (done by read_itte) and reduce to just one lookup (today you are doing two: read_itte() and irq_to_pending()).

Cheers,

Signature · 1 line
Julien Grall
#5 lore raw

PATCH 2/2 xen/dm: arm: Introduce inject_msi2 DM op

In reply to Mykyta Poturai

Add the second version of inject_msi DM op, which allows to specify the source_id of an MSI interrupt. This is needed for correct MSI injection on ARM.

It would not be safe to include the source_id in the original inject_msi in the pad field, because we have no way to know if it is set or not.

---
 tools/include/xendevicemodel.h               | 14 +++++++++++++
 tools/libs/devicemodel/core.c                | 22 ++++++++++++++++++++
 tools/libs/devicemodel/libxendevicemodel.map |  5 +++++
 xen/arch/arm/dm.c                            | 15 +++++++++++++
 xen/arch/x86/hvm/dm.c                        | 13 ++++++++++++
 xen/include/public/hvm/dm_op.h               | 12 +++++++++++
 6 files changed, 81 insertions(+)
tools/include/xendevicemodel.h +14 -0
index 797e0c6b29..4833e55bce 100644
@@ -236,6 +236,20 @@ int xendevicemodel_inject_msi(
     xendevicemodel_handle *dmod, domid_t domid, uint64_t msi_addr,
     uint32_t msi_data);
 
+/**
+ * This function injects an MSI into a guest.
+ *
+ * @parm dmod a handle to an open devicemodel interface.
+ * @parm domid the domain id to be serviced
+ * @parm msi_addr the MSI address (0xfeexxxxx)
+ * @parm source_id the PCI SBDF of the source device
+ * @parm msi_data the MSI data
+ * @return 0 on success, -1 on failure.
+*/
+int xendevicemodel_inject_msi2(
+    xendevicemodel_handle *dmod, domid_t domid, uint64_t msi_addr, uint32_t source_id,
+    uint32_t msi_data, unsigned int source_id_valid);
+
 /**
  * This function enables tracking of changes in the VRAM area.
  *
tools/libs/devicemodel/core.c +22 -0
index 8e619eeb0a..17ad00c5d9 100644
@@ -448,6 +448,28 @@ int xendevicemodel_set_irq_level(
     return xendevicemodel_op(dmod, domid, 1, &op, sizeof(op));
 }
 
+int xendevicemodel_inject_msi2(
+    xendevicemodel_handle *dmod, domid_t domid, uint64_t msi_addr, uint32_t source_id,
+    uint32_t msi_data, unsigned int source_id_valid)
+{
+    struct xen_dm_op op;
+    struct xen_dm_op_inject_msi2 *data;
+
+    memset(&op, 0, sizeof(op));
+
+    op.op = XEN_DMOP_inject_msi2;
+    data = &op.u.inject_msi2;
+
+    data->addr = msi_addr;
+    data->data = msi_data;
+    if ( source_id_valid ) {
+        data->source_id = source_id;
+        data->flags = XEN_DMOP_MSI_SOURCE_ID_VALID;
+    }
+
+    return xendevicemodel_op(dmod, domid, 1, &op, sizeof(op));
+}
+
 int xendevicemodel_set_pci_link_route(
     xendevicemodel_handle *dmod, domid_t domid, uint8_t link, uint8_t irq)
 {
tools/libs/devicemodel/libxendevicemodel.map +5 -0
index f7f9e3d932..aa05768642 100644
@@ -44,3 +44,8 @@ VERS_1.4 {
 		xendevicemodel_set_irq_level;
 		xendevicemodel_nr_vcpus;
 } VERS_1.3;
+
+VERS_1.5 {
+	global:
+		xendevicemodel_inject_msi2;
+} VERS_1.4;
xen/arch/arm/dm.c +15 -0
index 5569efa121..c45e196561 100644
@@ -27,6 +27,7 @@ int dm_op(const struct dmop_args *op_args)
         [XEN_DMOP_set_ioreq_server_state]           = sizeof(struct xen_dm_op_set_ioreq_server_state),
         [XEN_DMOP_destroy_ioreq_server]             = sizeof(struct xen_dm_op_destroy_ioreq_server),
         [XEN_DMOP_set_irq_level]                    = sizeof(struct xen_dm_op_set_irq_level),
+        [XEN_DMOP_inject_msi2]                      = sizeof(struct xen_dm_op_inject_msi2),
         [XEN_DMOP_nr_vcpus]                         = sizeof(struct xen_dm_op_nr_vcpus),
     };
 
@@ -112,6 +113,20 @@ int dm_op(const struct dmop_args *op_args)
         break;
     }
 
+    case XEN_DMOP_inject_msi2:
+    {
+        const struct xen_dm_op_inject_msi2 *data =
+            &op.u.inject_msi2;
+
+        if ( !(data->flags & XEN_DMOP_MSI_SOURCE_ID_VALID) )
+        {
+            rc = -EINVAL;
+            break;
+        }
+        rc = vgic_its_trigger_msi(d, data->addr, data->source_id, data->data);
+        break;
+
+    }
     case XEN_DMOP_nr_vcpus:
     {
         struct xen_dm_op_nr_vcpus *data = &op.u.nr_vcpus;
xen/arch/x86/hvm/dm.c +13 -0
index 462691f91d..a4a0e3dff9 100644
@@ -344,6 +344,7 @@ int dm_op(const struct dmop_args *op_args)
         [XEN_DMOP_set_mem_type]                     = sizeof(struct xen_dm_op_set_mem_type),
         [XEN_DMOP_inject_event]                     = sizeof(struct xen_dm_op_inject_event),
         [XEN_DMOP_inject_msi]                       = sizeof(struct xen_dm_op_inject_msi),
+        [XEN_DMOP_inject_msi2]                      = sizeof(struct xen_dm_op_inject_msi2),
         [XEN_DMOP_map_mem_type_to_ioreq_server]     = sizeof(struct xen_dm_op_map_mem_type_to_ioreq_server),
         [XEN_DMOP_remote_shutdown]                  = sizeof(struct xen_dm_op_remote_shutdown),
         [XEN_DMOP_relocate_memory]                  = sizeof(struct xen_dm_op_relocate_memory),
@@ -539,6 +540,18 @@ int dm_op(const struct dmop_args *op_args)
         break;
     }
 
+    case XEN_DMOP_inject_msi2:
+    {
+        const struct xen_dm_op_inject_msi2 *data =
+            &op.u.inject_msi2;
+
+        if ( !(data->flags & XEN_DMOP_MSI_SOURCE_ID_VALID) )
+            printk(XENLOG_WARNING "XEN_DMOP_inject_msi2: source_id is ignored\n");
+
+        rc = hvm_inject_msi(d, data->addr, data->data);
+        break;
+    }
+
     case XEN_DMOP_remote_shutdown:
     {
         const struct xen_dm_op_remote_shutdown *data =
xen/include/public/hvm/dm_op.h +12 -0
index fa98551914..da2ce4a7f7 100644
@@ -444,6 +444,17 @@ struct xen_dm_op_nr_vcpus {
 };
 typedef struct xen_dm_op_nr_vcpus xen_dm_op_nr_vcpus_t;
 
+#define XEN_DMOP_inject_msi2 21
+#define XEN_DMOP_MSI_SOURCE_ID_VALID (1u << 0)
+
+struct xen_dm_op_inject_msi2 {
+    uint64_aligned_t addr;
+    uint32_t data;
+    uint32_t source_id; /* PCI SBDF */
+    uint32_t flags;
+};
+typedef struct xen_dm_op_inject_msi2 xen_dm_op_inject_msi2_t;
+
 struct xen_dm_op {
     uint32_t op;
     uint32_t pad;
@@ -463,6 +474,7 @@ struct xen_dm_op {
         xen_dm_op_set_mem_type_t set_mem_type;
         xen_dm_op_inject_event_t inject_event;
         xen_dm_op_inject_msi_t inject_msi;
+        xen_dm_op_inject_msi2_t inject_msi2;
         xen_dm_op_map_mem_type_to_ioreq_server_t map_mem_type_to_ioreq_server;
         xen_dm_op_remote_shutdown_t remote_shutdown;
         xen_dm_op_relocate_memory_t relocate_memory;
Signature · 1 line
2.34.1
#6 lore raw

Re: [PATCH 2/2] xen/dm: arm: Introduce inject_msi2 DM op

Quoted 6 lines · Mykyta Poturai

Add the second version of inject_msi DM op, which allows to specify the source_id of an MSI interrupt. This is needed for correct MSI injection on ARM.

It would not be safe to include the source_id in the original inject_msi in the pad field, because we have no way to know if it is set or not.

At least on x86 I think 0 could serve as a "not valid" indicator, as that's always a host bridge, which would never be handed through to guests. Can't speak for Arm, as I don't know whether 00:00.0 always being a host bridge (or unpopulated) is a universal requirement of the spec. Therefore this may be insufficient justification for adding a new sub-op.

Quoted 14 lines
xen/arch/arm/dm.c +1 -0
@@ -27,6 +27,7 @@ int dm_op(const struct dmop_args *op_args)
         [XEN_DMOP_set_ioreq_server_state]           = sizeof(struct xen_dm_op_set_ioreq_server_state),
         [XEN_DMOP_destroy_ioreq_server]             = sizeof(struct xen_dm_op_destroy_ioreq_server),
         [XEN_DMOP_set_irq_level]                    = sizeof(struct xen_dm_op_set_irq_level),
+        [XEN_DMOP_inject_msi2]                      = sizeof(struct xen_dm_op_inject_msi2),
         [XEN_DMOP_nr_vcpus]                         = sizeof(struct xen_dm_op_nr_vcpus),
     };
 
@@ -112,6 +113,20 @@ int dm_op(const struct dmop_args *op_args)
         break;
     }
 

xen/arch/arm/dm.c int dm_op(const struct dmop_args *op_args) · line 125

Quoted 10 lines
xen/arch/arm/dm.c +10 -0
+    case XEN_DMOP_inject_msi2:
+    {
+        const struct xen_dm_op_inject_msi2 *data =
+            &op.u.inject_msi2;
+
+        if ( !(data->flags & XEN_DMOP_MSI_SOURCE_ID_VALID) )
+        {
+            rc = -EINVAL;
+            break;
+        }

While I can see the reason for this check here, ...

xen/arch/arm/dm.c int dm_op(const struct dmop_args *op_args) · line 549

Quoted 11 lines
xen/arch/arm/dm.c +7 -0
@@ -539,6 +540,18 @@ int dm_op(const struct dmop_args *op_args)
         break;
     }
 
+    case XEN_DMOP_inject_msi2:
+    {
+        const struct xen_dm_op_inject_msi2 *data =
+            &op.u.inject_msi2;
+
+        if ( !(data->flags & XEN_DMOP_MSI_SOURCE_ID_VALID) )
+            printk(XENLOG_WARNING "XEN_DMOP_inject_msi2: source_id is ignored\n");

... I don't understand what's going on here: If the flag isn't set, it's obvious that the field is to be ignored. Is the conditional inverted?

Also in both cases you will need to check that all other flags fields are clear, or else we won't be able to give them meaning down the road.

Finally such a message, if really warranted, wants to use gprintk().

xen/include/public/hvm/dm_op.h struct xen_dm_op_nr_vcpus { · line 453

Quoted 13 lines
xen/include/public/hvm/dm_op.h +7 -0
@@ -444,6 +444,17 @@ struct xen_dm_op_nr_vcpus {
 };
 typedef struct xen_dm_op_nr_vcpus xen_dm_op_nr_vcpus_t;
 
+#define XEN_DMOP_inject_msi2 21
+#define XEN_DMOP_MSI_SOURCE_ID_VALID (1u << 0)
+
+struct xen_dm_op_inject_msi2 {
+    uint64_aligned_t addr;
+    uint32_t data;
+    uint32_t source_id; /* PCI SBDF */

Since the comment says SBDF (not BDF), how are multiple segments handled here? At least on x86 (VT-d and V-i) source IDs are only 16 bits iirc, and are local to the respective segment. It would feel wrong to use a 32-bit quantity there; IOW I'd rather see this as being two 16-bit fields (segment and source_id).

xen/include/public/hvm/dm_op.h struct xen_dm_op_nr_vcpus { · line 455

Quoted 2 lines
xen/include/public/hvm/dm_op.h +2 -0
+    uint32_t flags;
+};

Just like in struct xen_dm_op_inject_msi padding wants making explicit, and then wants checking for being zero.

Jan

#7 lore raw

Re: [PATCH 2/2] xen/dm: arm: Introduce inject_msi2 DM op

Quoted 7 lines · Jan Beulich
Quoted 6 lines · Mykyta Poturai
xen/include/public/hvm/dm_op.h +0 -0
@@ -444,6 +444,17 @@ struct xen_dm_op_nr_vcpus {
  };
  typedef struct xen_dm_op_nr_vcpus xen_dm_op_nr_vcpus_t;
 
Quoted 14 lines

xen/include/public/hvm/dm_op.h struct xen_dm_op_nr_vcpus { · line 453

Quoted 7 lines
xen/include/public/hvm/dm_op.h +7 -0
+#define XEN_DMOP_inject_msi2 21
+#define XEN_DMOP_MSI_SOURCE_ID_VALID (1u << 0)
+
+struct xen_dm_op_inject_msi2 {
+    uint64_aligned_t addr;
+    uint32_t data;
+    uint32_t source_id; /* PCI SBDF */

Since the comment says SBDF (not BDF), how are multiple segments handled here? At least on x86 (VT-d and V-i) source IDs are only 16 bits iirc, and are local to the respective segment. It would feel wrong to use a 32-bit quantity there; IOW I'd rather see this as being two 16-bit fields (segment and source_id).

Hi Jan,

I'm planning on resuming this series in the near future and want to clarify the DM op interface.

Wouldn't it be better to keep things consistent and use XEN_DMOP_PCI_SBDF as it's done in xen_dm_op_ioreq_server_range? Also with this, the value can be easily casted to pci_sbdf_t later and split to segment and BDF if needed.

Quoted 7 lines

xen/include/public/hvm/dm_op.h struct xen_dm_op_nr_vcpus { · line 455

Quoted 2 lines
xen/include/public/hvm/dm_op.h +2 -0
+    uint32_t flags;
+};

Just like in struct xen_dm_op_inject_msi padding wants making explicit, and then wants checking for being zero.

Jan

Will do.

Signature · 1 line
Mykyta
#8 lore raw

Re: [PATCH 2/2] xen/dm: arm: Introduce inject_msi2 DM op

Quoted 22 lines · Mykyta Poturai
Quoted 7 lines · Jan Beulich
Quoted 6 lines · Mykyta Poturai
xen/include/public/hvm/dm_op.h +0 -0
@@ -444,6 +444,17 @@ struct xen_dm_op_nr_vcpus {
  };
  typedef struct xen_dm_op_nr_vcpus xen_dm_op_nr_vcpus_t;
 
Quoted 13 lines

xen/include/public/hvm/dm_op.h struct xen_dm_op_nr_vcpus { · line 453

Quoted 7 lines
xen/include/public/hvm/dm_op.h +7 -0
+#define XEN_DMOP_inject_msi2 21
+#define XEN_DMOP_MSI_SOURCE_ID_VALID (1u << 0)
+
+struct xen_dm_op_inject_msi2 {
+    uint64_aligned_t addr;
+    uint32_t data;
+    uint32_t source_id; /* PCI SBDF */

Since the comment says SBDF (not BDF), how are multiple segments handled here? At least on x86 (VT-d and V-i) source IDs are only 16 bits iirc, and are local to the respective segment. It would feel wrong to use a 32-bit quantity there; IOW I'd rather see this as being two 16-bit fields (segment and source_id).

Quoted 7 lines

I'm planning on resuming this series in the near future and want to clarify the DM op interface.

Wouldn't it be better to keep things consistent and use XEN_DMOP_PCI_SBDF as it's done in xen_dm_op_ioreq_server_range? Also with this, the value can be easily casted to pci_sbdf_t later and split to segment and BDF if needed.

The essence of my earlier comment is: Naming, contents, and comments need to be in sync.

I question though that "casting to pci_sbdf_t" is technically possible. Nor am I convinced that it would be desirable to do so if it was possible (or if "casting" was intended to mean something else than what this is in C). See my recent comments on some of Andrii's patches [1][2].

Jan

[1] https://lists.xen.org/archives/html/xen-devel/2025-03/msg01294.html
[2] https://lists.xen.org/archives/html/xen-devel/2025-03/msg01301.html

#9 lore raw

Re: [PATCH 2/2] xen/dm: arm: Introduce inject_msi2 DM op

Quoted 39 lines · Jan Beulich
Quoted 22 lines · Mykyta Poturai
Quoted 7 lines · Jan Beulich
Quoted 6 lines · Mykyta Poturai
xen/include/public/hvm/dm_op.h +0 -0
@@ -444,6 +444,17 @@ struct xen_dm_op_nr_vcpus {
   };
   typedef struct xen_dm_op_nr_vcpus xen_dm_op_nr_vcpus_t;
 
Quoted 13 lines

xen/include/public/hvm/dm_op.h struct xen_dm_op_nr_vcpus { · line 453

Quoted 7 lines
xen/include/public/hvm/dm_op.h +7 -0
+#define XEN_DMOP_inject_msi2 21
+#define XEN_DMOP_MSI_SOURCE_ID_VALID (1u << 0)
+
+struct xen_dm_op_inject_msi2 {
+    uint64_aligned_t addr;
+    uint32_t data;
+    uint32_t source_id; /* PCI SBDF */

Since the comment says SBDF (not BDF), how are multiple segments handled here? At least on x86 (VT-d and V-i) source IDs are only 16 bits iirc, and are local to the respective segment. It would feel wrong to use a 32-bit quantity there; IOW I'd rather see this as being two 16-bit fields (segment and source_id).

Quoted 7 lines

I'm planning on resuming this series in the near future and want to clarify the DM op interface.

Wouldn't it be better to keep things consistent and use XEN_DMOP_PCI_SBDF as it's done in xen_dm_op_ioreq_server_range? Also with this, the value can be easily casted to pci_sbdf_t later and split to segment and BDF if needed.

The essence of my earlier comment is: Naming, contents, and comments need to be in sync.

I question though that "casting to pci_sbdf_t" is technically possible. Nor am I convinced that it would be desirable to do so if it was possible (or if "casting" was intended to mean something else than what this is in C). See my recent comments on some of Andrii's patches [1][2].

Quoted 4 lines

Jan

[1] https://lists.xen.org/archives/html/xen-devel/2025-03/msg01294.html
[2] https://lists.xen.org/archives/html/xen-devel/2025-03/msg01301.html

Would something like this be okay then?

struct xen_dm_op_inject_msi2 {
     /* IN - MSI data (lower 32 bits) */
     uint32_t data;
     /* IN - ITS devid of the device triggering the interrupt */
     uint32_t source_id;
     uint32_t flags;
     uint32_t pad;
     /* IN - MSI address */
     uint64_aligned_t addr;
};

Added padding and explained source_id purpose better. --
Mykyta

#10 lore raw

Re: [PATCH 2/2] xen/dm: arm: Introduce inject_msi2 DM op

Quoted 58 lines · Mykyta Poturai
Quoted 39 lines · Jan Beulich
Quoted 22 lines · Mykyta Poturai
Quoted 7 lines · Jan Beulich
Quoted 6 lines · Mykyta Poturai
xen/include/public/hvm/dm_op.h +0 -0
@@ -444,6 +444,17 @@ struct xen_dm_op_nr_vcpus {
   };
   typedef struct xen_dm_op_nr_vcpus xen_dm_op_nr_vcpus_t;
 
Quoted 13 lines

xen/include/public/hvm/dm_op.h struct xen_dm_op_nr_vcpus { · line 453

Quoted 7 lines
xen/include/public/hvm/dm_op.h +7 -0
+#define XEN_DMOP_inject_msi2 21
+#define XEN_DMOP_MSI_SOURCE_ID_VALID (1u << 0)
+
+struct xen_dm_op_inject_msi2 {
+    uint64_aligned_t addr;
+    uint32_t data;
+    uint32_t source_id; /* PCI SBDF */

Since the comment says SBDF (not BDF), how are multiple segments handled here? At least on x86 (VT-d and V-i) source IDs are only 16 bits iirc, and are local to the respective segment. It would feel wrong to use a 32-bit quantity there; IOW I'd rather see this as being two 16-bit fields (segment and source_id).

Quoted 7 lines

I'm planning on resuming this series in the near future and want to clarify the DM op interface.

Wouldn't it be better to keep things consistent and use XEN_DMOP_PCI_SBDF as it's done in xen_dm_op_ioreq_server_range? Also with this, the value can be easily casted to pci_sbdf_t later and split to segment and BDF if needed.

The essence of my earlier comment is: Naming, contents, and comments need to be in sync.

I question though that "casting to pci_sbdf_t" is technically possible. Nor am I convinced that it would be desirable to do so if it was possible (or if "casting" was intended to mean something else than what this is in C). See my recent comments on some of Andrii's patches [1][2].

Quoted 4 lines

Jan

[1] https://lists.xen.org/archives/html/xen-devel/2025-03/msg01294.html
[2] https://lists.xen.org/archives/html/xen-devel/2025-03/msg01301.html

Would something like this be okay then?

struct xen_dm_op_inject_msi2 {
     /* IN - MSI data (lower 32 bits) */
     uint32_t data;
     /* IN - ITS devid of the device triggering the interrupt */
     uint32_t source_id;
     uint32_t flags;
     uint32_t pad;
     /* IN - MSI address */
     uint64_aligned_t addr;
};
Quoted 1 line

Added padding and explained source_id purpose better.

I fear the comment is far from making clear what layout source_id is to have, hence also leaving open whether a segment number is included there. Of course the issue here may be that I have no clue what "ITS devid" means or implies. What is clear is that "ITS devid" is meaningless on x86, for example.

I'm further puzzled by "(lower 32 bits)" - isn't the data written to trigger an MSI always a 32-bit quantity?

Jan

#11 lore raw

Re: [PATCH 2/2] xen/dm: arm: Introduce inject_msi2 DM op

Quoted 59 lines · Jan Beulich
Quoted 58 lines · Mykyta Poturai
Quoted 39 lines · Jan Beulich
Quoted 22 lines · Mykyta Poturai
Quoted 7 lines · Jan Beulich
Quoted 6 lines · Mykyta Poturai
xen/include/public/hvm/dm_op.h +0 -0
@@ -444,6 +444,17 @@ struct xen_dm_op_nr_vcpus {
    };
    typedef struct xen_dm_op_nr_vcpus xen_dm_op_nr_vcpus_t;
 
Quoted 13 lines

xen/include/public/hvm/dm_op.h struct xen_dm_op_nr_vcpus { · line 453

Quoted 7 lines
xen/include/public/hvm/dm_op.h +7 -0
+#define XEN_DMOP_inject_msi2 21
+#define XEN_DMOP_MSI_SOURCE_ID_VALID (1u << 0)
+
+struct xen_dm_op_inject_msi2 {
+    uint64_aligned_t addr;
+    uint32_t data;
+    uint32_t source_id; /* PCI SBDF */

Since the comment says SBDF (not BDF), how are multiple segments handled here? At least on x86 (VT-d and V-i) source IDs are only 16 bits iirc, and are local to the respective segment. It would feel wrong to use a 32-bit quantity there; IOW I'd rather see this as being two 16-bit fields (segment and source_id).

Quoted 7 lines

I'm planning on resuming this series in the near future and want to clarify the DM op interface.

Wouldn't it be better to keep things consistent and use XEN_DMOP_PCI_SBDF as it's done in xen_dm_op_ioreq_server_range? Also with this, the value can be easily casted to pci_sbdf_t later and split to segment and BDF if needed.

The essence of my earlier comment is: Naming, contents, and comments need to be in sync.

I question though that "casting to pci_sbdf_t" is technically possible. Nor am I convinced that it would be desirable to do so if it was possible (or if "casting" was intended to mean something else than what this is in C). See my recent comments on some of Andrii's patches [1][2].

Quoted 4 lines

Jan

[1] https://lists.xen.org/archives/html/xen-devel/2025-03/msg01294.html
[2] https://lists.xen.org/archives/html/xen-devel/2025-03/msg01301.html

Would something like this be okay then?

struct xen_dm_op_inject_msi2 {
      /* IN - MSI data (lower 32 bits) */
      uint32_t data;
      /* IN - ITS devid of the device triggering the interrupt */
      uint32_t source_id;
      uint32_t flags;
      uint32_t pad;
      /* IN - MSI address */
      uint64_aligned_t addr;
};
Quoted 7 lines
Quoted 1 line

Added padding and explained source_id purpose better.

I fear the comment is far from making clear what layout source_id is to have, hence also leaving open whether a segment number is included there. Of course the issue here may be that I have no clue what "ITS devid" means or implies. What is clear is that "ITS devid" is meaningless on x86, for example.

ITS devid is implementation defined. Its size is also implementation defined but up to 32 bits.

Quotes from Arm Base System Architecture[1]:

Quoted 1 line

The system designer assigns a requester a unique StreamID to device

traffic input to the SMMU.

Quoted 1 line

If a requester is a bridge from a different interconnect with an

originator ID, like a PCIe RequesterID, and devices on that interconnect might need to send MSIs, the originator ID is used to generate a DeviceID. The function to generate the DeviceID should be an identity or a simple offset.

On the Xen's side it is used as an offset into the ITS translation tables and is sourced from msi-map nodes in the device tree.

Practically for PCI the requester ID is usually the SBDF. Where the segment is used to find the host bridge node that contains the msi-map node with defined offsets but it is also included as part of an id. That's why it was originally called SBDF in the comment. I don't know if there is a better way to describe what it is concisely in the comments.

Quoted 4 lines

I'm further puzzled by "(lower 32 bits)" - isn't the data written to trigger an MSI always a 32-bit quantity?

Jan

Hmm, it actually is, I copied this line from xen_dm_op_inject_msi, not sure why it is specified like that there. But I'll remove this part.

[1]: https://developer.arm.com/documentation/den0094/latest/

--
Mykyta

#12 lore raw

Re: [PATCH 2/2] xen/dm: arm: Introduce inject_msi2 DM op

Quoted 80 lines · Mykyta Poturai
Quoted 59 lines · Jan Beulich
Quoted 58 lines · Mykyta Poturai
Quoted 39 lines · Jan Beulich
Quoted 22 lines · Mykyta Poturai
Quoted 7 lines · Jan Beulich
Quoted 6 lines · Mykyta Poturai
xen/include/public/hvm/dm_op.h +0 -0
@@ -444,6 +444,17 @@ struct xen_dm_op_nr_vcpus {
    };
    typedef struct xen_dm_op_nr_vcpus xen_dm_op_nr_vcpus_t;
 
Quoted 13 lines

xen/include/public/hvm/dm_op.h struct xen_dm_op_nr_vcpus { · line 453

Quoted 7 lines
xen/include/public/hvm/dm_op.h +7 -0
+#define XEN_DMOP_inject_msi2 21
+#define XEN_DMOP_MSI_SOURCE_ID_VALID (1u << 0)
+
+struct xen_dm_op_inject_msi2 {
+    uint64_aligned_t addr;
+    uint32_t data;
+    uint32_t source_id; /* PCI SBDF */

Since the comment says SBDF (not BDF), how are multiple segments handled here? At least on x86 (VT-d and V-i) source IDs are only 16 bits iirc, and are local to the respective segment. It would feel wrong to use a 32-bit quantity there; IOW I'd rather see this as being two 16-bit fields (segment and source_id).

Quoted 7 lines

I'm planning on resuming this series in the near future and want to clarify the DM op interface.

Wouldn't it be better to keep things consistent and use XEN_DMOP_PCI_SBDF as it's done in xen_dm_op_ioreq_server_range? Also with this, the value can be easily casted to pci_sbdf_t later and split to segment and BDF if needed.

The essence of my earlier comment is: Naming, contents, and comments need to be in sync.

I question though that "casting to pci_sbdf_t" is technically possible. Nor am I convinced that it would be desirable to do so if it was possible (or if "casting" was intended to mean something else than what this is in C). See my recent comments on some of Andrii's patches [1][2].

Quoted 4 lines

Jan

[1] https://lists.xen.org/archives/html/xen-devel/2025-03/msg01294.html
[2] https://lists.xen.org/archives/html/xen-devel/2025-03/msg01301.html

Would something like this be okay then?

struct xen_dm_op_inject_msi2 {
      /* IN - MSI data (lower 32 bits) */
      uint32_t data;
      /* IN - ITS devid of the device triggering the interrupt */
      uint32_t source_id;
      uint32_t flags;
      uint32_t pad;
      /* IN - MSI address */
      uint64_aligned_t addr;
};
Quoted 7 lines
Quoted 1 line

Added padding and explained source_id purpose better.

I fear the comment is far from making clear what layout source_id is to have, hence also leaving open whether a segment number is included there. Of course the issue here may be that I have no clue what "ITS devid" means or implies. What is clear is that "ITS devid" is meaningless on x86, for example.

ITS devid is implementation defined. Its size is also implementation defined but up to 32 bits.

Quotes from Arm Base System Architecture[1]:

Quoted 1 line

The system designer assigns a requester a unique StreamID to device

traffic input to the SMMU.

Quoted 1 line

If a requester is a bridge from a different interconnect with an

originator ID, like a PCIe RequesterID, and devices on that interconnect might need to send MSIs, the originator ID is used to generate a DeviceID. The function to generate the DeviceID should be an identity or a simple offset.

Quoted 8 lines

On the Xen's side it is used as an offset into the ITS translation tables and is sourced from msi-map nodes in the device tree.

Practically for PCI the requester ID is usually the SBDF. Where the segment is used to find the host bridge node that contains the msi-map node with defined offsets but it is also included as part of an id. That's why it was originally called SBDF in the comment. I don't know if there is a better way to describe what it is concisely in the comments.

If this is to be usable for other architectures too, it may need to be a union to put there. With appropriate comments for each member.

Jan

#13 lore raw

Re: [PATCH 2/2] xen/dm: arm: Introduce inject_msi2 DM op

On 19.03.25 14:37, Jan Beulich wrote:

Hello Jan, all.

Quoted 84 lines · Mykyta Poturai
Quoted 83 lines · Jan Beulich
Quoted 59 lines · Mykyta Poturai
Quoted 58 lines · Jan Beulich
Quoted 39 lines · Mykyta Poturai
Quoted 22 lines · Jan Beulich
Quoted 7 lines · Mykyta Poturai
Quoted 6 lines
xen/include/public/hvm/dm_op.h +0 -0
@@ -444,6 +444,17 @@ struct xen_dm_op_nr_vcpus {
     };
     typedef struct xen_dm_op_nr_vcpus xen_dm_op_nr_vcpus_t;
 
Quoted 13 lines

xen/include/public/hvm/dm_op.h struct xen_dm_op_nr_vcpus { · line 453

Quoted 7 lines
xen/include/public/hvm/dm_op.h +7 -0
+#define XEN_DMOP_inject_msi2 21
+#define XEN_DMOP_MSI_SOURCE_ID_VALID (1u << 0)
+
+struct xen_dm_op_inject_msi2 {
+    uint64_aligned_t addr;
+    uint32_t data;
+    uint32_t source_id; /* PCI SBDF */

Since the comment says SBDF (not BDF), how are multiple segments handled here? At least on x86 (VT-d and V-i) source IDs are only 16 bits iirc, and are local to the respective segment. It would feel wrong to use a 32-bit quantity there; IOW I'd rather see this as being two 16-bit fields (segment and source_id).

Quoted 7 lines

I'm planning on resuming this series in the near future and want to clarify the DM op interface.

Wouldn't it be better to keep things consistent and use XEN_DMOP_PCI_SBDF as it's done in xen_dm_op_ioreq_server_range? Also with this, the value can be easily casted to pci_sbdf_t later and split to segment and BDF if needed.

The essence of my earlier comment is: Naming, contents, and comments need to be in sync.

I question though that "casting to pci_sbdf_t" is technically possible. Nor am I convinced that it would be desirable to do so if it was possible (or if "casting" was intended to mean something else than what this is in C). See my recent comments on some of Andrii's patches [1][2].

Quoted 4 lines

Jan

[1] https://lists.xen.org/archives/html/xen-devel/2025-03/msg01294.html
[2] https://lists.xen.org/archives/html/xen-devel/2025-03/msg01301.html

Would something like this be okay then?

struct xen_dm_op_inject_msi2 {
       /* IN - MSI data (lower 32 bits) */
       uint32_t data;
       /* IN - ITS devid of the device triggering the interrupt */
       uint32_t source_id;
       uint32_t flags;
       uint32_t pad;
       /* IN - MSI address */
       uint64_aligned_t addr;
};
Quoted 7 lines
Quoted 1 line

Added padding and explained source_id purpose better.

I fear the comment is far from making clear what layout source_id is to have, hence also leaving open whether a segment number is included there. Of course the issue here may be that I have no clue what "ITS devid" means or implies. What is clear is that "ITS devid" is meaningless on x86, for example.

ITS devid is implementation defined. Its size is also implementation defined but up to 32 bits.

Quotes from Arm Base System Architecture[1]:

Quoted 1 line

The system designer assigns a requester a unique StreamID to device

traffic input to the SMMU.

Quoted 1 line

If a requester is a bridge from a different interconnect with an

originator ID, like a PCIe RequesterID, and devices on that interconnect might need to send MSIs, the originator ID is used to generate a DeviceID. The function to generate the DeviceID should be an identity or a simple offset.

On the Xen's side it is used as an offset into the ITS translation tables and is sourced from msi-map nodes in the device tree.

Quoted 8 lines
Quoted 5 lines

Practically for PCI the requester ID is usually the SBDF. Where the segment is used to find the host bridge node that contains the msi-map node with defined offsets but it is also included as part of an id. That's why it was originally called SBDF in the comment. I don't know if there is a better way to describe what it is concisely in the comments.

If this is to be usable for other architectures too, it may need to be a union to put there. With appropriate comments for each member.

If I got correctly what is wrote in current thread (and in RFC version where it was an attempt to create just Arm64's counterpart of XEN_DMOP_inject_msi), my understanding (maybe not precise/correct, since I am not quite familiar with x86 details) that we would need something like that:

/*
* XEN_DMOP_inject_msi2: An enhanced version of the sub-ob to inject an MSI
* for an emulated device, which allows specifying
* the ID of the device triggering the MSI
(source ID).
*
* The source ID is specified by a pair of <segment> and <source_id>.
* If <flags> does not contain XEN_DMOP_MSI_SOURCE_ID_VALID then source ID
* is invalid and should be ignored.
*/
#define XEN_DMOP_inject_msi 21

struct xen_dm_op_inject_msi2 {
     /* IN - MSI data */
     uint32_t data;
     /* IN - next two fields form an ID of the device triggering the MSI */
     uint16_t segment; /* The segment number */
     uint16_t source_id; /* The source ID that is local to segment (PCI 
BDF) */
     /* IN - types of source ID */
     uint32_t flags;
#define XEN_DMOP_MSI_SOURCE_ID_VALID (1u << 0)
     uint32_t pad;
     /* IN - MSI address */
     uint64_aligned_t addr;
};

This is arch agnostic sub-op without the (obvious) specifics of the underlying MSI controller. The sub-ob, I hope, will be suitable on both Arm64 soon (segment + source_id provide vSBDF, then it will possible to locate vITS and devid) and on x86 in future (for the vIOMMU needs).

Would you be ok with that in general? Please share you opinion.

Quoted 3 lines

Jan

#14 lore raw

Re: [PATCH 2/2] xen/dm: arm: Introduce inject_msi2 DM op

Quoted 130 lines · Oleksandr Tyshchenko
Quoted 84 lines · Jan Beulich
Quoted 83 lines · Mykyta Poturai
Quoted 59 lines · Jan Beulich
Quoted 58 lines · Mykyta Poturai
Quoted 39 lines · Jan Beulich
Quoted 22 lines · Mykyta Poturai
Quoted 7 lines · Jan Beulich
Quoted 6 lines · Mykyta Poturai
xen/include/public/hvm/dm_op.h +0 -0
@@ -444,6 +444,17 @@ struct xen_dm_op_nr_vcpus {
     };
     typedef struct xen_dm_op_nr_vcpus xen_dm_op_nr_vcpus_t;
 
Quoted 13 lines

xen/include/public/hvm/dm_op.h struct xen_dm_op_nr_vcpus { · line 453

Quoted 7 lines
xen/include/public/hvm/dm_op.h +7 -0
+#define XEN_DMOP_inject_msi2 21
+#define XEN_DMOP_MSI_SOURCE_ID_VALID (1u << 0)
+
+struct xen_dm_op_inject_msi2 {
+    uint64_aligned_t addr;
+    uint32_t data;
+    uint32_t source_id; /* PCI SBDF */

Since the comment says SBDF (not BDF), how are multiple segments handled here? At least on x86 (VT-d and V-i) source IDs are only 16 bits iirc, and are local to the respective segment. It would feel wrong to use a 32-bit quantity there; IOW I'd rather see this as being two 16-bit fields (segment and source_id).

Quoted 7 lines

I'm planning on resuming this series in the near future and want to clarify the DM op interface.

Wouldn't it be better to keep things consistent and use XEN_DMOP_PCI_SBDF as it's done in xen_dm_op_ioreq_server_range? Also with this, the value can be easily casted to pci_sbdf_t later and split to segment and BDF if needed.

The essence of my earlier comment is: Naming, contents, and comments need to be in sync.

I question though that "casting to pci_sbdf_t" is technically possible. Nor am I convinced that it would be desirable to do so if it was possible (or if "casting" was intended to mean something else than what this is in C). See my recent comments on some of Andrii's patches [1][2].

Quoted 4 lines

Jan

[1] https://lists.xen.org/archives/html/xen-devel/2025-03/msg01294.html
[2] https://lists.xen.org/archives/html/xen-devel/2025-03/msg01301.html

Would something like this be okay then?

struct xen_dm_op_inject_msi2 {
       /* IN - MSI data (lower 32 bits) */
       uint32_t data;
       /* IN - ITS devid of the device triggering the interrupt */
       uint32_t source_id;
       uint32_t flags;
       uint32_t pad;
       /* IN - MSI address */
       uint64_aligned_t addr;
};
Quoted 7 lines
Quoted 1 line

Added padding and explained source_id purpose better.

I fear the comment is far from making clear what layout source_id is to have, hence also leaving open whether a segment number is included there. Of course the issue here may be that I have no clue what "ITS devid" means or implies. What is clear is that "ITS devid" is meaningless on x86, for example.

ITS devid is implementation defined. Its size is also implementation defined but up to 32 bits.

Quotes from Arm Base System Architecture[1]:

Quoted 1 line

The system designer assigns a requester a unique StreamID to device

traffic input to the SMMU.

Quoted 1 line

If a requester is a bridge from a different interconnect with an

originator ID, like a PCIe RequesterID, and devices on that interconnect might need to send MSIs, the originator ID is used to generate a DeviceID. The function to generate the DeviceID should be an identity or a simple offset.

On the Xen's side it is used as an offset into the ITS translation tables and is sourced from msi-map nodes in the device tree.

Quoted 8 lines
Quoted 5 lines

Practically for PCI the requester ID is usually the SBDF. Where the segment is used to find the host bridge node that contains the msi-map node with defined offsets but it is also included as part of an id. That's why it was originally called SBDF in the comment. I don't know if there is a better way to describe what it is concisely in the comments.

If this is to be usable for other architectures too, it may need to be a union to put there. With appropriate comments for each member.

If I got correctly what is wrote in current thread (and in RFC version where it was an attempt to create just Arm64's counterpart of XEN_DMOP_inject_msi), my understanding (maybe not precise/correct, since I am not quite familiar with x86 details) that we would need something like that:

/*
* XEN_DMOP_inject_msi2: An enhanced version of the sub-ob to inject an MSI
* for an emulated device, which allows specifying
* the ID of the device triggering the MSI
(source ID).
*
* The source ID is specified by a pair of <segment> and <source_id>.
* If <flags> does not contain XEN_DMOP_MSI_SOURCE_ID_VALID then source ID
* is invalid and should be ignored.
*/
#define XEN_DMOP_inject_msi 21

struct xen_dm_op_inject_msi2 {
      /* IN - MSI data */
      uint32_t data;
      /* IN - next two fields form an ID of the device triggering the MSI */
      uint16_t segment; /* The segment number */
      uint16_t source_id; /* The source ID that is local to segment (PCI 
BDF) */
      /* IN - types of source ID */
      uint32_t flags;
#define XEN_DMOP_MSI_SOURCE_ID_VALID (1u << 0)
      uint32_t pad;
      /* IN - MSI address */
      uint64_aligned_t addr;
};
Quoted 6 lines

This is arch agnostic sub-op without the (obvious) specifics of the underlying MSI controller. The sub-ob, I hope, will be suitable on both Arm64 soon (segment + source_id provide vSBDF, then it will possible to locate vITS and devid) and on x86 in future (for the vIOMMU needs).

Would you be ok with that in general? Please share you opinion.

Yes, this looks plausible. In the context of things like VMD (using software established segment numbers wider than 16 bits) I wonder though whether we wouldn't better make segment and source ID 32-bit fields from the beginning. Out-of-range values would of course need rejecting then.

Jan

#15 lore raw

Re: [PATCH 2/2] xen/dm: arm: Introduce inject_msi2 DM op

On 20.03.25 09:30, Jan Beulich wrote:

Hello Jan, Mykyta, all

Quoted 131 lines · Oleksandr Tyshchenko
Quoted 130 lines · Jan Beulich
Quoted 84 lines · Mykyta Poturai
Quoted 83 lines · Jan Beulich
Quoted 59 lines · Mykyta Poturai
Quoted 58 lines · Jan Beulich
Quoted 39 lines · Mykyta Poturai
Quoted 22 lines · Jan Beulich
Quoted 7 lines · Mykyta Poturai
Quoted 6 lines
xen/include/public/hvm/dm_op.h +0 -0
@@ -444,6 +444,17 @@ struct xen_dm_op_nr_vcpus {
      };
      typedef struct xen_dm_op_nr_vcpus xen_dm_op_nr_vcpus_t;
 
Quoted 13 lines

xen/include/public/hvm/dm_op.h struct xen_dm_op_nr_vcpus { · line 453

Quoted 7 lines
xen/include/public/hvm/dm_op.h +7 -0
+#define XEN_DMOP_inject_msi2 21
+#define XEN_DMOP_MSI_SOURCE_ID_VALID (1u << 0)
+
+struct xen_dm_op_inject_msi2 {
+    uint64_aligned_t addr;
+    uint32_t data;
+    uint32_t source_id; /* PCI SBDF */

Since the comment says SBDF (not BDF), how are multiple segments handled here? At least on x86 (VT-d and V-i) source IDs are only 16 bits iirc, and are local to the respective segment. It would feel wrong to use a 32-bit quantity there; IOW I'd rather see this as being two 16-bit fields (segment and source_id).

Quoted 7 lines

I'm planning on resuming this series in the near future and want to clarify the DM op interface.

Wouldn't it be better to keep things consistent and use XEN_DMOP_PCI_SBDF as it's done in xen_dm_op_ioreq_server_range? Also with this, the value can be easily casted to pci_sbdf_t later and split to segment and BDF if needed.

The essence of my earlier comment is: Naming, contents, and comments need to be in sync.

I question though that "casting to pci_sbdf_t" is technically possible. Nor am I convinced that it would be desirable to do so if it was possible (or if "casting" was intended to mean something else than what this is in C). See my recent comments on some of Andrii's patches [1][2].

Quoted 4 lines

Jan

[1] https://lists.xen.org/archives/html/xen-devel/2025-03/msg01294.html
[2] https://lists.xen.org/archives/html/xen-devel/2025-03/msg01301.html

Would something like this be okay then?

struct xen_dm_op_inject_msi2 {
        /* IN - MSI data (lower 32 bits) */
        uint32_t data;
        /* IN - ITS devid of the device triggering the interrupt */
        uint32_t source_id;
        uint32_t flags;
        uint32_t pad;
        /* IN - MSI address */
        uint64_aligned_t addr;
};
Quoted 7 lines
Quoted 1 line

Added padding and explained source_id purpose better.

I fear the comment is far from making clear what layout source_id is to have, hence also leaving open whether a segment number is included there. Of course the issue here may be that I have no clue what "ITS devid" means or implies. What is clear is that "ITS devid" is meaningless on x86, for example.

ITS devid is implementation defined. Its size is also implementation defined but up to 32 bits.

Quotes from Arm Base System Architecture[1]:

Quoted 1 line

The system designer assigns a requester a unique StreamID to device

traffic input to the SMMU.

Quoted 1 line

If a requester is a bridge from a different interconnect with an

originator ID, like a PCIe RequesterID, and devices on that interconnect might need to send MSIs, the originator ID is used to generate a DeviceID. The function to generate the DeviceID should be an identity or a simple offset.

On the Xen's side it is used as an offset into the ITS translation tables and is sourced from msi-map nodes in the device tree.

Quoted 8 lines
Quoted 5 lines

Practically for PCI the requester ID is usually the SBDF. Where the segment is used to find the host bridge node that contains the msi-map node with defined offsets but it is also included as part of an id. That's why it was originally called SBDF in the comment. I don't know if there is a better way to describe what it is concisely in the comments.

If this is to be usable for other architectures too, it may need to be a union to put there. With appropriate comments for each member.

If I got correctly what is wrote in current thread (and in RFC version where it was an attempt to create just Arm64's counterpart of XEN_DMOP_inject_msi), my understanding (maybe not precise/correct, since I am not quite familiar with x86 details) that we would need something like that:

/*
* XEN_DMOP_inject_msi2: An enhanced version of the sub-ob to inject an MSI
* for an emulated device, which allows specifying
* the ID of the device triggering the MSI
(source ID).
*
* The source ID is specified by a pair of <segment> and <source_id>.
* If <flags> does not contain XEN_DMOP_MSI_SOURCE_ID_VALID then source ID
* is invalid and should be ignored.
*/
#define XEN_DMOP_inject_msi 21

struct xen_dm_op_inject_msi2 {
       /* IN - MSI data */
       uint32_t data;
       /* IN - next two fields form an ID of the device triggering the MSI */
       uint16_t segment; /* The segment number */
       uint16_t source_id; /* The source ID that is local to segment (PCI
BDF) */
       /* IN - types of source ID */
       uint32_t flags;
#define XEN_DMOP_MSI_SOURCE_ID_VALID (1u << 0)
       uint32_t pad;
       /* IN - MSI address */
       uint64_aligned_t addr;
};
Quoted 8 lines
Quoted 6 lines

This is arch agnostic sub-op without the (obvious) specifics of the underlying MSI controller. The sub-ob, I hope, will be suitable on both Arm64 soon (segment + source_id provide vSBDF, then it will possible to locate vITS and devid) and on x86 in future (for the vIOMMU needs).

Would you be ok with that in general? Please share you opinion.

Yes, this looks plausible.

Jan, thanks

In the context of things like VMD (using

Quoted 3 lines

software established segment numbers wider than 16 bits) I wonder though whether we wouldn't better make segment and source ID 32-bit fields from the beginning.

Keeping in mind that dm_op hypercall is stable and segment can be >= 0x10000, probably yes, makes sense to use 32-bit from the very beginning.

Out-of-range values would of course need

Quoted 1 line

rejecting then.

yes, sure

***

Mykyta, are you ok with the proposed adjustments to the sub-op structure?

Quoted 2 lines

Jan

#16 lore raw

Re: [PATCH 2/2] xen/dm: arm: Introduce inject_msi2 DM op

Quoted 61 lines · Oleksandr Tyshchenko
Quoted 40 lines
Quoted 40 lines

If I got correctly what is wrote in current thread (and in RFC version where it was an attempt to create just Arm64's counterpart of XEN_DMOP_inject_msi), my understanding (maybe not precise/correct, since I am not quite familiar with x86 details) that we would need something like that:

/*
* XEN_DMOP_inject_msi2: An enhanced version of the sub-ob to
inject an MSI
* for an emulated device, which allows
specifying
* the ID of the device triggering the MSI
(source ID).
*
* The source ID is specified by a pair of <segment> and <source_id>.
* If <flags> does not contain XEN_DMOP_MSI_SOURCE_ID_VALID then
source ID
* is invalid and should be ignored.
*/
#define XEN_DMOP_inject_msi 21

struct xen_dm_op_inject_msi2 {
       /* IN - MSI data */
       uint32_t data;
       /* IN - next two fields form an ID of the device triggering 

the MSI */
uint16_t segment; /* The segment number */ uint16_t source_id; /* The source ID that is local to segment
(PCI
BDF) */

       /* IN - types of source ID */
       uint32_t flags;
#define XEN_DMOP_MSI_SOURCE_ID_VALID (1u << 0)
       uint32_t pad;
       /* IN - MSI address */
       uint64_aligned_t addr;
};
Quoted 8 lines
Quoted 6 lines

This is arch agnostic sub-op without the (obvious) specifics of the underlying MSI controller. The sub-ob, I hope, will be suitable on both Arm64 soon (segment + source_id provide vSBDF, then it will possible to locate vITS and devid) and on x86 in future (for the vIOMMU needs).

Would you be ok with that in general? Please share you opinion.

Yes, this looks plausible.

Jan, thanks

In the context of things like VMD (using

Quoted 3 lines

software established segment numbers wider than 16 bits) I wonder though whether we wouldn't better make segment and source ID 32-bit fields from the beginning.

Keeping in mind that dm_op hypercall is stable and segment can be >= 0x10000, probably yes, makes sense to use 32-bit from the very beginning.

Quoted 11 lines

Out-of-range values would of course need

Quoted 1 line

rejecting then.

yes, sure

***

Mykyta, are you ok with the proposed adjustments to the sub-op structure?

Quoted 2 lines

Jan

Yes, seems fine for me. I will use it in the next version.

Signature · 1 line
Mykyta
#17 lore raw

Re: [PATCH 2/2] xen/dm: arm: Introduce inject_msi2 DM op

In reply to Mykyta Poturai
Quoted 28 lines · Mykyta Poturai

Add the second version of inject_msi DM op, which allows to specify the source_id of an MSI interrupt. This is needed for correct MSI injection on ARM.

It would not be safe to include the source_id in the original inject_msi in the pad field, because we have no way to know if it is set or not.

---
 tools/include/xendevicemodel.h               | 14 +++++++++++++
 tools/libs/devicemodel/core.c                | 22 ++++++++++++++++++++
 tools/libs/devicemodel/libxendevicemodel.map |  5 +++++
 xen/arch/arm/dm.c                            | 15 +++++++++++++
 xen/arch/x86/hvm/dm.c                        | 13 ++++++++++++
 xen/include/public/hvm/dm_op.h               | 12 +++++++++++
 6 files changed, 81 insertions(+)
tools/include/xendevicemodel.h +3 -0
index 797e0c6b29..4833e55bce 100644
@@ -236,6 +236,20 @@ int xendevicemodel_inject_msi(
     xendevicemodel_handle *dmod, domid_t domid, uint64_t msi_addr,
     uint32_t msi_data);
 
+/**
+ * This function injects an MSI into a guest.
+ *

tools/include/xendevicemodel.h int xendevicemodel_inject_msi( · line 251

Quoted 10 lines
tools/include/xendevicemodel.h +10 -0
+ * @parm dmod a handle to an open devicemodel interface.
+ * @parm domid the domain id to be serviced
+ * @parm msi_addr the MSI address (0xfeexxxxx)
+ * @parm source_id the PCI SBDF of the source device
+ * @parm msi_data the MSI data
+ * @return 0 on success, -1 on failure.
+*/
+int xendevicemodel_inject_msi2(
+    xendevicemodel_handle *dmod, domid_t domid, uint64_t msi_addr, uint32_t source_id,
+    uint32_t msi_data, unsigned int source_id_valid);

What is "source_id_valid"? It is not described in the comment. Also, it should be a fixed size int. I agree with Jan that we could reuse xendevicemodel_inject_msi by assuing that PCI BDF zero is invalid.

Quoted 146 lines
tools/include/xendevicemodel.h +0 -0
 /**
  * This function enables tracking of changes in the VRAM area.
  *
tools/libs/devicemodel/core.c +22 -0
index 8e619eeb0a..17ad00c5d9 100644
@@ -448,6 +448,28 @@ int xendevicemodel_set_irq_level(
     return xendevicemodel_op(dmod, domid, 1, &op, sizeof(op));
 }
 
+int xendevicemodel_inject_msi2(
+    xendevicemodel_handle *dmod, domid_t domid, uint64_t msi_addr, uint32_t source_id,
+    uint32_t msi_data, unsigned int source_id_valid)
+{
+    struct xen_dm_op op;
+    struct xen_dm_op_inject_msi2 *data;
+
+    memset(&op, 0, sizeof(op));
+
+    op.op = XEN_DMOP_inject_msi2;
+    data = &op.u.inject_msi2;
+
+    data->addr = msi_addr;
+    data->data = msi_data;
+    if ( source_id_valid ) {
+        data->source_id = source_id;
+        data->flags = XEN_DMOP_MSI_SOURCE_ID_VALID;
+    }
+
+    return xendevicemodel_op(dmod, domid, 1, &op, sizeof(op));
+}
+
 int xendevicemodel_set_pci_link_route(
     xendevicemodel_handle *dmod, domid_t domid, uint8_t link, uint8_t irq)
 {
tools/libs/devicemodel/libxendevicemodel.map +5 -0
index f7f9e3d932..aa05768642 100644
@@ -44,3 +44,8 @@ VERS_1.4 {
 		xendevicemodel_set_irq_level;
 		xendevicemodel_nr_vcpus;
 } VERS_1.3;
+
+VERS_1.5 {
+	global:
+		xendevicemodel_inject_msi2;
+} VERS_1.4;
xen/arch/arm/dm.c +15 -0
index 5569efa121..c45e196561 100644
@@ -27,6 +27,7 @@ int dm_op(const struct dmop_args *op_args)
         [XEN_DMOP_set_ioreq_server_state]           = sizeof(struct xen_dm_op_set_ioreq_server_state),
         [XEN_DMOP_destroy_ioreq_server]             = sizeof(struct xen_dm_op_destroy_ioreq_server),
         [XEN_DMOP_set_irq_level]                    = sizeof(struct xen_dm_op_set_irq_level),
+        [XEN_DMOP_inject_msi2]                      = sizeof(struct xen_dm_op_inject_msi2),
         [XEN_DMOP_nr_vcpus]                         = sizeof(struct xen_dm_op_nr_vcpus),
     };
 
@@ -112,6 +113,20 @@ int dm_op(const struct dmop_args *op_args)
         break;
     }
 
+    case XEN_DMOP_inject_msi2:
+    {
+        const struct xen_dm_op_inject_msi2 *data =
+            &op.u.inject_msi2;
+
+        if ( !(data->flags & XEN_DMOP_MSI_SOURCE_ID_VALID) )
+        {
+            rc = -EINVAL;
+            break;
+        }
+        rc = vgic_its_trigger_msi(d, data->addr, data->source_id, data->data);
+        break;
+
+    }
     case XEN_DMOP_nr_vcpus:
     {
         struct xen_dm_op_nr_vcpus *data = &op.u.nr_vcpus;
xen/arch/x86/hvm/dm.c +13 -0
index 462691f91d..a4a0e3dff9 100644
@@ -344,6 +344,7 @@ int dm_op(const struct dmop_args *op_args)
         [XEN_DMOP_set_mem_type]                     = sizeof(struct xen_dm_op_set_mem_type),
         [XEN_DMOP_inject_event]                     = sizeof(struct xen_dm_op_inject_event),
         [XEN_DMOP_inject_msi]                       = sizeof(struct xen_dm_op_inject_msi),
+        [XEN_DMOP_inject_msi2]                      = sizeof(struct xen_dm_op_inject_msi2),
         [XEN_DMOP_map_mem_type_to_ioreq_server]     = sizeof(struct xen_dm_op_map_mem_type_to_ioreq_server),
         [XEN_DMOP_remote_shutdown]                  = sizeof(struct xen_dm_op_remote_shutdown),
         [XEN_DMOP_relocate_memory]                  = sizeof(struct xen_dm_op_relocate_memory),
@@ -539,6 +540,18 @@ int dm_op(const struct dmop_args *op_args)
         break;
     }
 
+    case XEN_DMOP_inject_msi2:
+    {
+        const struct xen_dm_op_inject_msi2 *data =
+            &op.u.inject_msi2;
+
+        if ( !(data->flags & XEN_DMOP_MSI_SOURCE_ID_VALID) )
+            printk(XENLOG_WARNING "XEN_DMOP_inject_msi2: source_id is ignored\n");
+
+        rc = hvm_inject_msi(d, data->addr, data->data);
+        break;
+    }
+
     case XEN_DMOP_remote_shutdown:
     {
         const struct xen_dm_op_remote_shutdown *data =
xen/include/public/hvm/dm_op.h +12 -0
index fa98551914..da2ce4a7f7 100644
@@ -444,6 +444,17 @@ struct xen_dm_op_nr_vcpus {
 };
 typedef struct xen_dm_op_nr_vcpus xen_dm_op_nr_vcpus_t;
 
+#define XEN_DMOP_inject_msi2 21
+#define XEN_DMOP_MSI_SOURCE_ID_VALID (1u << 0)
+
+struct xen_dm_op_inject_msi2 {
+    uint64_aligned_t addr;
+    uint32_t data;
+    uint32_t source_id; /* PCI SBDF */
+    uint32_t flags;
+};
+typedef struct xen_dm_op_inject_msi2 xen_dm_op_inject_msi2_t;
+
 struct xen_dm_op {
     uint32_t op;
     uint32_t pad;
@@ -463,6 +474,7 @@ struct xen_dm_op {
         xen_dm_op_set_mem_type_t set_mem_type;
         xen_dm_op_inject_event_t inject_event;
         xen_dm_op_inject_msi_t inject_msi;
+        xen_dm_op_inject_msi2_t inject_msi2;
         xen_dm_op_map_mem_type_to_ioreq_server_t map_mem_type_to_ioreq_server;
         xen_dm_op_remote_shutdown_t remote_shutdown;
         xen_dm_op_relocate_memory_t relocate_memory;
Signature · 1 line
2.34.1
#18 lore raw

Re: [PATCH 2/2] xen/dm: arm: Introduce inject_msi2 DM op

Quoted 2 lines · Stefano Stabellini
Quoted 2 lines

I agree with Jan that we could reuse xendevicemodel_inject_msi by assuing that PCI BDF zero is invalid.

I've already explained why that will break future x86.  (See the other thread off this patch for specifics).

When we add vIOMMUs to guests, we *must* have correct source ids.  0 is ambiguous, so not safe to reuse.

Allocating a new subop now is the only way to not shoot in the foot later.

~Andrew

#19 lore raw

Re: [PATCH 2/2] xen/dm: arm: Introduce inject_msi2 DM op

Quoted 4 lines · Andrew Cooper
Quoted 2 lines · Stefano Stabellini
Quoted 2 lines

I agree with Jan that we could reuse xendevicemodel_inject_msi by assuing that PCI BDF zero is invalid.

Quoted 8 lines

I've already explained why that will break future x86.  (See the other thread off this patch for specifics).

When we add vIOMMUs to guests, we *must* have correct source ids.  0 is ambiguous, so not safe to reuse.

Allocating a new subop now is the only way to not shoot in the foot later.

OK, what you wrote makes sense to me.

#20 lore raw

Re: [PATCH 2/2] xen/dm: arm: Introduce inject_msi2 DM op

In reply to Mykyta Poturai
Quoted 5 lines · Mykyta Poturai
tools/include/xendevicemodel.h +0 -0
index 797e0c6b29..4833e55bce 100644

tools/include/xendevicemodel.h int xendevicemodel_inject_msi( · line 244

Quoted 9 lines
tools/include/xendevicemodel.h +6 -0
@@ -236,6 +236,20 @@ int xendevicemodel_inject_msi(
     xendevicemodel_handle *dmod, domid_t domid, uint64_t msi_addr,
     uint32_t msi_data);
 
+/**
+ * This function injects an MSI into a guest.
+ *
+ * @parm dmod a handle to an open devicemodel interface.
+ * @parm domid the domain id to be serviced
+ * @parm msi_addr the MSI address (0xfeexxxxx)

This 0xfeexxxxx is an x86-ism which I doubt is correct for ARM.  I'd suggest dropping it from the comment.

tools/include/xendevicemodel.h int xendevicemodel_inject_msi( · line 251

Quoted 7 lines
tools/include/xendevicemodel.h +7 -0
+ * @parm source_id the PCI SBDF of the source device
+ * @parm msi_data the MSI data
+ * @return 0 on success, -1 on failure.
+*/
+int xendevicemodel_inject_msi2(
+    xendevicemodel_handle *dmod, domid_t domid, uint64_t msi_addr, uint32_t source_id,
+    uint32_t msi_data, unsigned int source_id_valid);

The Source ID is always valid when making this call.  It is only within the hypervisor itself that we may need to worry about the source ID being invalid.

This means you don't have the flags field, and as a consequence, there's no padding to worry about.

Also, the msi_ prefix to address and data are redundant.  Either drop them, or put a prefix on source_id too, because we shouldn't be inconsistent here.

Quoted 22 lines
tools/include/xendevicemodel.h +1 -0
+
 /**
  * This function enables tracking of changes in the VRAM area.
  *
tools/libs/devicemodel/core.c +10 -0
index 8e619eeb0a..17ad00c5d9 100644
@@ -448,6 +448,28 @@ int xendevicemodel_set_irq_level(
     return xendevicemodel_op(dmod, domid, 1, &op, sizeof(op));
 }
 
+int xendevicemodel_inject_msi2(
+    xendevicemodel_handle *dmod, domid_t domid, uint64_t msi_addr, uint32_t source_id,
+    uint32_t msi_data, unsigned int source_id_valid)
+{
+    struct xen_dm_op op;
+    struct xen_dm_op_inject_msi2 *data;
+
+    memset(&op, 0, sizeof(op));
+
+    op.op = XEN_DMOP_inject_msi2;

tools/libs/devicemodel/core.c int xendevicemodel_set_irq_level( · line 470

Quoted 10 lines
tools/libs/devicemodel/core.c +10 -0
+    data = &op.u.inject_msi2;
+
+    data->addr = msi_addr;
+    data->data = msi_data;
+    if ( source_id_valid ) {
+        data->source_id = source_id;
+        data->flags = XEN_DMOP_MSI_SOURCE_ID_VALID;
+    }
+
+    return xendevicemodel_op(dmod, domid, 1, &op, sizeof(op));

{
    struct xen_dm_op op = {
        .op = XEN_DMOP_inject_msi2,
        .u.inject_msi2 = {
            .addr = msi_addr,
            .data = msi_data,
            .source_id = source_id,
        },
    };

    return xendevicemodel_op(dmod, domid, 1, &op, sizeof(op)); }

Quoted 7 lines
xen/arch/x86/hvm/dm.c +0 -0
index 462691f91d..a4a0e3dff9 100644
@@ -539,6 +540,18 @@ int dm_op(const struct dmop_args *op_args)
         break;
     }

xen/arch/x86/hvm/dm.c int dm_op(const struct dmop_args *op_args) · line 551

Quoted 10 lines
xen/arch/x86/hvm/dm.c +9 -0
 
+    case XEN_DMOP_inject_msi2:
+    {
+        const struct xen_dm_op_inject_msi2 *data =
+            &op.u.inject_msi2;
+
+        if ( !(data->flags & XEN_DMOP_MSI_SOURCE_ID_VALID) )
+            printk(XENLOG_WARNING "XEN_DMOP_inject_msi2: source_id is ignored\n");
+
+        rc = hvm_inject_msi(d, data->addr, data->data);

You need a prep patch adding a source id parameter into hvm_inject_msi().

The XEN_DMOP_inject_msi case can probably pass in 0 in the short term, and it can probably be discarded internally.

As I said before, the source id doesn't matter until we start trying to expose vIOMMUs to guests, at which point I suspect the easiest option will simply to be to reject XEN_DMOP_inject_msi against a domain with a vIOMMU and force Qemu/whatever in dom0 to use XEN_DMOP_inject_msi2.

~Andrew

#21 lore raw

Re: [PATCH 2/2] xen/dm: arm: Introduce inject_msi2 DM op

Quoted 21 lines · Andrew Cooper
Quoted 5 lines · Mykyta Poturai
tools/include/xendevicemodel.h +0 -0
index 797e0c6b29..4833e55bce 100644

tools/include/xendevicemodel.h int xendevicemodel_inject_msi( · line 245

Quoted 10 lines
tools/include/xendevicemodel.h +7 -0
@@ -236,6 +236,20 @@ int xendevicemodel_inject_msi(
      xendevicemodel_handle *dmod, domid_t domid, uint64_t msi_addr,
      uint32_t msi_data);
 
+ * @parm source_id the PCI SBDF of the source device
+ * @parm msi_data the MSI data
+ * @return 0 on success, -1 on failure.
+*/
+int xendevicemodel_inject_msi2(
+    xendevicemodel_handle *dmod, domid_t domid, uint64_t msi_addr, uint32_t source_id,
+    uint32_t msi_data, unsigned int source_id_valid);

The Source ID is always valid when making this call. It is only within the hypervisor itself that we may need to worry about the source ID being invalid.

Quoted 3 lines

This means you don't have the flags field, and as a consequence, there's no padding to worry about.

Hi Andrew,
I was thinking that the plan is to eventually deprecate the inject_msi call and have only the new one with different behaviors depending on source_id_valid flag.
Do you mean we are leaving both of them indefinitely and should treat source id as a valid one implicitly from the fact that the new call is being issued?

Signature · 1 line
Mykyta
#22 lore raw
In reply to Mykyta Poturai

Hi,

Quoted 7 lines · Mykyta Poturai

This series adds the base support for MSI injection on Arm. This is needed to streamline virtio-pci interrupt triggering.

With this patches, MSIs can be triggered in guests by issuing the new DM op, inject_msi2. This op is similar to inject_msi, but it allows to specify the source id of the MSI.

Quoted 7 lines

We chose the approach of adding a new DM op instead of using the pad field of inject_msi because we have no clear way of distinguishing between set and unset pad fields. New implementations also adds flags field to clearly specify if the SBDF is set.

Patches were tested on QEMU with QEMU virtio-pci backends, with virtio-pci patches and patches for ITS support for DomUs applied.

I have seen some virtio-pci patches on the ML. But nothing about ITS support in the DomUs (at least in recent months).

It would feel odd to me to even consider merging this series before the ITS support for DomUs is merged. So what's the plan for it?

Cheers,

Signature · 1 line
Julien Grall