Skip to contentSkip to conversation

Remaining patches for dynamic node programming using overlay dtbo

PATCH NAK Henry Wang 6 participants 64 messages Apr 24, 2024 – May 11, 2024

lore
Share & tools
lore

Henry Wang

author
#0 lore raw

cover Remaining patches for dynamic node programming using overlay dtbo

Hi all,

This is the remaining series for the full functional "dynamic node programming using overlay dtbo" feature. The first part [1] has already been merged.

Quoting from the original series, the first part has already made Xen aware of new device tree node which means updating the dt_host with overlay node information, and in this series, the goal is to map IRQ and IOMMU during runtime, where we will do the actual IOMMU and IRQ mapping to a running domain and will call unmap_mmio_regions() to remove the mapping.

Also, documentation of the "dynamic node programming using overlay dtbo" feature is added.

Patch 1 is a fix of [1] which is noticed during my local test, details please see the commit message.

Gitlab CI for this series can be found in [2].

[1] https://lore.kernel.org/xen-devel/[email protected]/
[2] https://gitlab.com/xen-project/people/henryw/xen/-/pipelines/1265297506

Henry Wang (1):
xen/commom/dt-overlay: Fix missing lock when remove the device

Vikram Garhwal (14):
xen/arm/gic: Enable interrupt assignment to running VM xen/arm: Always enable IOMMU
tools/libs/light: Always enable IOMMU
tools/libs/light: Increase nr_spi to 160
rangeset: Move struct range and struct rangeset to headerfile xen/overlay: Enable device tree overlay assignment to running domains
tools: Add domain_id and expert mode for overlay operations tools/libs/light: Modify dtbo to domU linux dtbo format tools/xl: Share overlay with domU
tools/helpers: Add get_overlay
get_overlay: remove domU overlay
xl/overlay: add remove operation to xenstore
add a domU script to fetch overlays and applying them to linux
docs: add device tree overlay documentation

 docs/misc/arm/overlay.txt           | 172 +++++++++
 tools/helpers/Makefile              |   8 +
 tools/helpers/get_overlay.c         | 507 ++++++++++++++++++++++++++
 tools/helpers/get_overlay.sh        |  81 +++++
 tools/include/libxl.h               |   8 +-
 tools/include/xenctrl.h             |   5 +-
 tools/libs/ctrl/xc_dt_overlay.c     |   7 +-
 tools/libs/light/libxl_arm.c        |   9 +-
 tools/libs/light/libxl_dt_overlay.c |  90 ++++-
 tools/xl/Makefile                   |   2 +-
 tools/xl/xl_cmdtable.c              |   2 +-
 tools/xl/xl_vmcontrol.c             | 539 +++++++++++++++++++++++++++-
 xen/arch/arm/device.c               |   8 +-
 xen/arch/arm/dom0less-build.c       |   3 +-
 xen/arch/arm/domain_build.c         |   2 +-
 xen/arch/arm/gic.c                  |   4 +
 xen/arch/arm/include/asm/setup.h    |   3 +-
 xen/common/dt-overlay.c             | 223 ++++++++++--
 xen/common/rangeset.c               |  31 +-
 xen/include/public/sysctl.h         |   4 +-
 xen/include/xen/rangeset.h          |  32 +-
 21 files changed, 1654 insertions(+), 86 deletions(-)
 create mode 100644 docs/misc/arm/overlay.txt
 create mode 100644 tools/helpers/get_overlay.c
 create mode 100755 tools/helpers/get_overlay.sh
Signature · 1 line
2.34.1
author
#1 lore raw

PATCH 1/15 xen/commom/dt-overlay: Fix missing lock when remove the device

If CONFIG_DEBUG=y, below assertion will be triggered:
(XEN) Assertion 'rw_is_locked(&dt_host_lock)' failed at drivers/passthrough/device_tree.c:146 (XEN) ----[ Xen-4.19-unstable  arm64  debug=y  Not tainted ]----
(XEN) CPU:    0
(XEN) PC:     00000a0000257418 iommu_remove_dt_device+0x8c/0xd4
(XEN) LR:     00000a00002573a0
(XEN) SP:     00008000fff7fb30
(XEN) CPSR:   0000000000000249 MODE:64-bit EL2h (Hypervisor, handler)
[...]

(XEN) Xen call trace:
(XEN)    [<00000a0000257418>] iommu_remove_dt_device+0x8c/0xd4 (PC)
(XEN)    [<00000a00002573a0>] iommu_remove_dt_device+0x14/0xd4 (LR)
(XEN)    [<00000a000020797c>] dt-overlay.c#remove_node_resources+0x8c/0x90 (XEN)    [<00000a0000207f14>] dt-overlay.c#remove_nodes+0x524/0x648
(XEN)    [<00000a0000208460>] dt_overlay_sysctl+0x428/0xc68
(XEN)    [<00000a00002707f8>] arch_do_sysctl+0x1c/0x2c
(XEN)    [<00000a0000230b40>] do_sysctl+0x96c/0x9ec
(XEN)    [<00000a0000271e08>] traps.c#do_trap_hypercall+0x1e8/0x288
(XEN)    [<00000a0000273490>] do_trap_guest_sync+0x448/0x63c
(XEN)    [<00000a000025c480>] entry.o#guest_sync_slowpath+0xa8/0xd8
(XEN)
(XEN)
(XEN) ****************************************
(XEN) Panic on CPU 0:
(XEN) Assertion 'rw_is_locked(&dt_host_lock)' failed at drivers/passthrough/device_tree.c:146 (XEN) ****************************************

This is because iommu_remove_dt_device() is called without taking the dt_host_lock. Fix the issue by taking and releasing the lock properly.

Fixes 7e5c4a8b86f1 ("xen/arm: Implement device tree node removal functionalities")
---
 xen/common/dt-overlay.c | 5 +++++
 1 file changed, 5 insertions(+)
xen/common/dt-overlay.c +5 -0
index 1b197381f6..39e4ba59dd 100644
@@ -381,9 +381,14 @@ static int remove_node_resources(struct dt_device_node *device_node)
     {
         if ( dt_device_is_protected(device_node) )
         {
+            write_lock(&dt_host_lock);
             rc = iommu_remove_dt_device(device_node);
             if ( rc < 0 )
+            {
+                write_unlock(&dt_host_lock);
                 return rc;
+            }
+            write_unlock(&dt_host_lock);
         }
     }
 
Signature · 1 line
2.34.1
#2 lore raw

Re: [PATCH 01/15] xen/commom/dt-overlay: Fix missing lock when remove the device

xen/common/dt-overlay.c static int remove_node_resources(struct dt_device_node *device_node) · line 385

Quoted 8 lines · Henry Wang
xen/common/dt-overlay.c +1 -0
@@ -381,9 +381,14 @@ static int remove_node_resources(struct dt_device_node *device_node)
     {
         if ( dt_device_is_protected(device_node) )
         {
+            write_lock(&dt_host_lock);
             rc = iommu_remove_dt_device(device_node);

Any particular reason you add two call sites to the unlock function, instead of putting it here?

Jan

Quoted 9 lines
xen/common/dt-overlay.c +4 -0
             if ( rc < 0 )
+            {
+                write_unlock(&dt_host_lock);
                 return rc;
+            }
+            write_unlock(&dt_host_lock);
         }
     }
 
author
#3 lore raw

Re: [PATCH 01/15] xen/commom/dt-overlay: Fix missing lock when remove the device

Hi Jan,

Quoted 11 lines · Jan Beulich

xen/common/dt-overlay.c static int remove_node_resources(struct dt_device_node *device_node) · line 385

Quoted 8 lines · Henry Wang
xen/common/dt-overlay.c +1 -0
@@ -381,9 +381,14 @@ static int remove_node_resources(struct dt_device_node *device_node)
      {
          if ( dt_device_is_protected(device_node) )
          {
+            write_lock(&dt_host_lock);
              rc = iommu_remove_dt_device(device_node);

Any particular reason you add two call sites to the unlock function, instead of putting it here?

Oh...you are correct. It is indeed better to put the unlock here. If this is the only comment for this patch, can I respin this only patch as a v1.1 or would one of the committers be ok to fix on commit? Sorry for the trouble and thanks for the suggestion.

Kind regards,
Henry

Quoted 11 lines

Jan

Quoted 9 lines
xen/common/dt-overlay.c +4 -0
              if ( rc < 0 )
+            {
+                write_unlock(&dt_host_lock);
                  return rc;
+            }
+            write_unlock(&dt_host_lock);
          }
      }
 
author
#4 lore raw

PATCH 2/15 xen/arm/gic: Enable interrupt assignment to running VM

In reply to Henry Wang

From: Vikram Garhwal <[email protected]>

Enable interrupt assign/remove for running VMs in CONFIG_OVERLAY_DTB.

Currently, irq_route and mapping is only allowed at the domain creation. Adding exception for CONFIG_OVERLAY_DTB.

---
 xen/arch/arm/gic.c | 4 ++++
 1 file changed, 4 insertions(+)
xen/arch/arm/gic.c +4 -0
index 44c40e86de..a775f886ed 100644
@@ -140,8 +140,10 @@ int gic_route_irq_to_guest(struct domain *d, unsigned int virq,
      * back to the physical IRQ. To prevent get unsync, restrict the
      * routing to when the Domain is been created.
      */
+#ifndef CONFIG_OVERLAY_DTB
     if ( d->creation_finished )
         return -EBUSY;
+#endif
 
     ret = vgic_connect_hw_irq(d, NULL, virq, desc, true);
     if ( ret )
@@ -171,8 +173,10 @@ int gic_remove_irq_from_guest(struct domain *d, unsigned int virq,
      * Removing an interrupt while the domain is running may have
      * undesirable effect on the vGIC emulation.
      */
+#ifndef CONFIG_OVERLAY_DTB
     if ( !d->is_dying )
         return -EBUSY;
+#endif
 
     desc->handler->shutdown(desc);
 
Signature · 1 line
2.34.1
#5 lore raw

Re: [PATCH 02/15] xen/arm/gic: Enable interrupt assignment to running VM

Hi Henry,

Quoted 6 lines · Henry Wang

From: Vikram Garhwal <[email protected]>

Enable interrupt assign/remove for running VMs in CONFIG_OVERLAY_DTB.

Currently, irq_route and mapping is only allowed at the domain creation. Adding exception for CONFIG_OVERLAY_DTB.

AFAICT, this is mostly reverting b8577547236f ("xen/arm: Restrict when a physical IRQ can be routed/removed from/to a domain").

Quoted 13 lines
---
  xen/arch/arm/gic.c | 4 ++++
  1 file changed, 4 insertions(+)
xen/arch/arm/gic.c +0 -0
index 44c40e86de..a775f886ed 100644

xen/arch/arm/gic.c int gic_route_irq_to_guest(struct domain *d, unsigned int virq, · line 142

Quoted 3 lines
xen/arch/arm/gic.c +0 -0
@@ -140,8 +140,10 @@ int gic_route_irq_to_guest(struct domain *d, unsigned int virq,
       * back to the physical IRQ. To prevent get unsync, restrict the
       * routing to when the Domain is been created.
       */

The above comment explains why the check was added. But the commit message doesn't explain why this can be disregarded for your use-case.

Looking at the history, I don't think you can simply remove the checks.

Regardless that...

xen/arch/arm/gic.c int gic_route_irq_to_guest(struct domain *d, unsigned int virq, · line 143

Quoted 1 line
xen/arch/arm/gic.c +1 -0
+#ifndef CONFIG_OVERLAY_DTB

... I am against such #ifdef. A distros may want to have OVERLAY_DTB enabled, yet the user will not use it.

Instead, you want to remove the check once the code can properly handle routing an IRQ the domain is created or ...

Quoted 7 lines
xen/arch/arm/gic.c +1 -0
      if ( d->creation_finished )
          return -EBUSY;
+#endif
 
      ret = vgic_connect_hw_irq(d, NULL, virq, desc, true);
      if ( ret )

xen/arch/arm/gic.c int gic_remove_irq_from_guest(struct domain *d, unsigned int virq, · line 179

Quoted 7 lines
xen/arch/arm/gic.c +2 -0
@@ -171,8 +173,10 @@ int gic_remove_irq_from_guest(struct domain *d, unsigned int virq,
       * Removing an interrupt while the domain is running may have
       * undesirable effect on the vGIC emulation.
       */
+#ifndef CONFIG_OVERLAY_DTB
      if ( !d->is_dying )
          return -EBUSY;
+#endif

... removed before they domain is destroyed.

xen/arch/arm/gic.c int gic_remove_irq_from_guest(struct domain *d, unsigned int virq, · line 182

Quoted 3 lines
xen/arch/arm/gic.c +0 -0
      desc->handler->shutdown(desc);
 

Cheers,

Signature · 1 line
Julien Grall
author
#6 lore raw

Re: [PATCH 02/15] xen/arm/gic: Enable interrupt assignment to running VM

Hi Julien,

Quoted 42 lines · Julien Grall

Hi Henry,

Quoted 6 lines · Henry Wang

From: Vikram Garhwal <[email protected]>

Enable interrupt assign/remove for running VMs in CONFIG_OVERLAY_DTB.

Currently, irq_route and mapping is only allowed at the domain creation. Adding exception for CONFIG_OVERLAY_DTB.

AFAICT, this is mostly reverting b8577547236f ("xen/arm: Restrict when a physical IRQ can be routed/removed from/to a domain").

Quoted 13 lines

---
  xen/arch/arm/gic.c | 4 ++++

  1 file changed, 4 insertions(+)
xen/arch/arm/gic.c +0 -0
index 44c40e86de..a775f886ed 100644

xen/arch/arm/gic.c int gic_route_irq_to_guest(struct domain *d, · line 143

Quoted 4 lines
xen/arch/arm/gic.c +0 -0
@@ -140,8 +140,10 @@ int gic_route_irq_to_guest(struct domain *d, 
 unsigned int virq,
        * back to the physical IRQ. To prevent get unsync, restrict the
        * routing to when the Domain is been created.
        */

The above comment explains why the check was added. But the commit message doesn't explain why this can be disregarded for your use-case.

Looking at the history, I don't think you can simply remove the checks.

Regardless that...

xen/arch/arm/gic.c int gic_route_irq_to_guest(struct domain *d, · line 144

Quoted 1 line
xen/arch/arm/gic.c +1 -0
+#ifndef CONFIG_OVERLAY_DTB

... I am against such #ifdef. A distros may want to have OVERLAY_DTB enabled, yet the user will not use it.

Instead, you want to remove the check once the code can properly handle routing an IRQ the domain is created or ...

Quoted 16 lines
Quoted 6 lines
xen/arch/arm/gic.c +1 -0
       if ( d->creation_finished )
           return -EBUSY;
+#endif
         ret = vgic_connect_hw_irq(d, NULL, virq, desc, true);
       if ( ret )

xen/arch/arm/gic.c int gic_remove_irq_from_guest(struct domain *d, · line 180

Quoted 8 lines
xen/arch/arm/gic.c +2 -0
@@ -171,8 +173,10 @@ int gic_remove_irq_from_guest(struct domain *d, 
 unsigned int virq,
        * Removing an interrupt while the domain is running may have
        * undesirable effect on the vGIC emulation.
        */
+#ifndef CONFIG_OVERLAY_DTB
       if ( !d->is_dying )
           return -EBUSY;
+#endif

... removed before they domain is destroyed.

Thanks for your feeedback. After checking the b8577547236f commit message I think I now understand your point. Do you have any suggestion about how can I properly add the support to route/remove the IRQ to running domains? Thanks.

Kind regards,
Henry

Quoted 5 lines

xen/arch/arm/gic.c int gic_remove_irq_from_guest(struct domain *d, unsigned int virq, · line 181

Quoted 2 lines
xen/arch/arm/gic.c +0 -0
 desc->handler->shutdown(desc);

Cheers,

#7 lore raw

Re: [PATCH 02/15] xen/arm/gic: Enable interrupt assignment to running VM

Hi,

Quoted 62 lines · Henry Wang

Hi Julien,

Quoted 42 lines · Julien Grall

Hi Henry,

Quoted 6 lines · Henry Wang

From: Vikram Garhwal <[email protected]>

Enable interrupt assign/remove for running VMs in CONFIG_OVERLAY_DTB.

Currently, irq_route and mapping is only allowed at the domain creation. Adding exception for CONFIG_OVERLAY_DTB.

AFAICT, this is mostly reverting b8577547236f ("xen/arm: Restrict when a physical IRQ can be routed/removed from/to a domain").

Quoted 13 lines

---
  xen/arch/arm/gic.c | 4 ++++

  1 file changed, 4 insertions(+)
xen/arch/arm/gic.c +0 -0
index 44c40e86de..a775f886ed 100644

xen/arch/arm/gic.c int gic_route_irq_to_guest(struct domain *d, · line 143

Quoted 4 lines
xen/arch/arm/gic.c +0 -0
@@ -140,8 +140,10 @@ int gic_route_irq_to_guest(struct domain *d, 
 unsigned int virq,
        * back to the physical IRQ. To prevent get unsync, restrict the
        * routing to when the Domain is been created.
        */

The above comment explains why the check was added. But the commit message doesn't explain why this can be disregarded for your use-case.

Looking at the history, I don't think you can simply remove the checks.

Regardless that...

xen/arch/arm/gic.c int gic_route_irq_to_guest(struct domain *d, · line 144

Quoted 1 line
xen/arch/arm/gic.c +1 -0
+#ifndef CONFIG_OVERLAY_DTB

... I am against such #ifdef. A distros may want to have OVERLAY_DTB enabled, yet the user will not use it.

Instead, you want to remove the check once the code can properly handle routing an IRQ the domain is created or ...

Quoted 16 lines
Quoted 6 lines
xen/arch/arm/gic.c +1 -0
       if ( d->creation_finished )
           return -EBUSY;
+#endif
         ret = vgic_connect_hw_irq(d, NULL, virq, desc, true);
       if ( ret )

xen/arch/arm/gic.c int gic_remove_irq_from_guest(struct domain *d, · line 180

Quoted 8 lines
xen/arch/arm/gic.c +2 -0
@@ -171,8 +173,10 @@ int gic_remove_irq_from_guest(struct domain *d, 
 unsigned int virq,
        * Removing an interrupt while the domain is running may have
        * undesirable effect on the vGIC emulation.
        */
+#ifndef CONFIG_OVERLAY_DTB
       if ( !d->is_dying )
           return -EBUSY;
+#endif

... removed before they domain is destroyed.

Quoted 1 line

Thanks for your feeedback. After checking the b8577547236f commit message I think I now understand your point. Do you have any suggestion about how can I properly add the support to route/remove the IRQ to running domains? Thanks.

I haven't really look at that code in quite a while. I think we need to make sure that the virtual and physical IRQ state matches at the time we do the routing.

I am undecided on whether we want to simply prevent the action to happen or try to reset the state.

There is also the question of what to do if the guest is enabling the vIRQ before it is routed.

Overall, someone needs to spend some time reading the code and then make a proposal (this could be just documentation if we believe it is safe to do). Both the current vGIC and the new one may need an update.

Cheers,

Signature · 1 line
Julien Grall
author
#8 lore raw

Re: [PATCH 02/15] xen/arm/gic: Enable interrupt assignment to running VM

Hi Julien,

Sorry for the late reply,

Quoted 66 lines · Julien Grall

Hi,

Quoted 63 lines · Henry Wang

Hi Julien,

Quoted 42 lines · Julien Grall

Hi Henry,

Quoted 6 lines · Henry Wang

From: Vikram Garhwal <[email protected]>

Enable interrupt assign/remove for running VMs in CONFIG_OVERLAY_DTB.

Currently, irq_route and mapping is only allowed at the domain creation. Adding exception for CONFIG_OVERLAY_DTB.

AFAICT, this is mostly reverting b8577547236f ("xen/arm: Restrict when a physical IRQ can be routed/removed from/to a domain").

Quoted 13 lines

---
  xen/arch/arm/gic.c | 4 ++++

  1 file changed, 4 insertions(+)
xen/arch/arm/gic.c +0 -0
index 44c40e86de..a775f886ed 100644

xen/arch/arm/gic.c int gic_route_irq_to_guest(struct domain *d, · line 143

Quoted 4 lines
xen/arch/arm/gic.c +0 -0
@@ -140,8 +140,10 @@ int gic_route_irq_to_guest(struct domain *d, 
 unsigned int virq,
        * back to the physical IRQ. To prevent get unsync, restrict the
        * routing to when the Domain is been created.
        */

The above comment explains why the check was added. But the commit message doesn't explain why this can be disregarded for your use-case.

Looking at the history, I don't think you can simply remove the checks.

Regardless that...

xen/arch/arm/gic.c int gic_route_irq_to_guest(struct domain *d, · line 144

Quoted 1 line
xen/arch/arm/gic.c +1 -0
+#ifndef CONFIG_OVERLAY_DTB

... I am against such #ifdef. A distros may want to have OVERLAY_DTB enabled, yet the user will not use it.

Instead, you want to remove the check once the code can properly handle routing an IRQ the domain is created or ...

Quoted 16 lines
Quoted 6 lines
xen/arch/arm/gic.c +1 -0
       if ( d->creation_finished )
           return -EBUSY;
+#endif
         ret = vgic_connect_hw_irq(d, NULL, virq, desc, true);
       if ( ret )

xen/arch/arm/gic.c int gic_remove_irq_from_guest(struct domain · line 180

Quoted 8 lines
xen/arch/arm/gic.c +2 -0
@@ -171,8 +173,10 @@ int gic_remove_irq_from_guest(struct domain 
 *d, unsigned int virq,
        * Removing an interrupt while the domain is running may have
        * undesirable effect on the vGIC emulation.
        */
+#ifndef CONFIG_OVERLAY_DTB
       if ( !d->is_dying )
           return -EBUSY;
+#endif

... removed before they domain is destroyed.

Thanks for your feeedback. After checking the b8577547236f commit message I think I now understand your point. Do you have any suggestion about how can I properly add the support to route/remove the IRQ to running domains? Thanks.

I spent some time going through the GIC/vGIC code and had some discussions with Stefano and Stewart during the last couple of days, let me see if I can describe the use case properly now to continue the discussion:

We have some use cases that requires assigning devices to domains after domain boot time. For example, suppose there is an FPGA on the board which can simulate a device, and the bitstream for the FPGA is provided and programmed after domain boot. So we need a way to assign the device to the running domain. This series tries to implement this use case by using device tree overlay - users can firstly add the overlay to Xen dtb, assign the device in the overlay to a domain by the xl command, then apply the overlay to Linux.

Quoted 5 lines

I haven't really look at that code in quite a while. I think we need to make sure that the virtual and physical IRQ state matches at the time we do the routing.

I am undecided on whether we want to simply prevent the action to happen or try to reset the state.

There is also the question of what to do if the guest is enabling the vIRQ before it is routed.

Sorry for bothering, would you mind elaborating a bit more about the two cases that you mentioned above? Commit b8577547236f ("xen/arm: Restrict when a physical IRQ can be routed/removed from/to a domain") only said there will be undesirable effects, so I am not sure if I understand the concerns raised above and the consequences of these two use cases. I am probably wrong, I think when we add the overlay, we are probably fine as the interrupt is not being used before. Also since we only load the device driver after the IRQ is routed to the guest, I am not sure the guest can enable the vIRQ before it is routed.

Kind regards,
Henry

Quoted 4 lines

Overall, someone needs to spend some time reading the code and then make a proposal (this could be just documentation if we believe it is safe to do). Both the current vGIC and the new one may need an update.

Cheers,

#9 lore raw

Re: [PATCH 02/15] xen/arm/gic: Enable interrupt assignment to running VM

Hi Henry,

Quoted 6 lines · Henry Wang
Quoted 1 line · Julien Grall
Quoted 1 line

Thanks for your feeedback. After checking the b8577547236f commit message I think I now understand your point. Do you have any suggestion about how can I properly add the support to route/remove the IRQ to running domains? Thanks.

I spent some time going through the GIC/vGIC code and had some discussions with Stefano and Stewart during the last couple of days, let me see if I can describe the use case properly now to continue the discussion:

We have some use cases that requires assigning devices to domains after domain boot time. For example, suppose there is an FPGA on the board which can simulate a device, and the bitstream for the FPGA is provided and programmed after domain boot. So we need a way to assign the device to the running domain. This series tries to implement this use case by using device tree overlay - users can firstly add the overlay to Xen dtb, assign the device in the overlay to a domain by the xl command, then apply the overlay to Linux.

Thanks for the description! This helps to understand your goal :).

Quoted 8 lines
Quoted 6 lines

I haven't really look at that code in quite a while. I think we need to make sure that the virtual and physical IRQ state matches at the time we do the routing.

I am undecided on whether we want to simply prevent the action to happen or try to reset the state.

There is also the question of what to do if the guest is enabling the vIRQ before it is routed.

Sorry for bothering, would you mind elaborating a bit more about the two cases that you mentioned above? Commit b8577547236f ("xen/arm: Restrict when a physical IRQ can be routed/removed from/to a domain") only said there will be undesirable effects, so I am not sure if I understand the concerns raised above and the consequences of these two use cases.

I will try to explain them below after I answer the rest.

Quoted 1 line

I am probably wrong, I think when we add the overlay, we are probably fine as the interrupt is not being used before.

What if the DT overlay is unloaded and then reloaded? Wouldn't the same interrupt be re-used? As a more generic case, this could also be a new bitstream for the FPGA.

But even if the interrupt is brand new every time for the DT overlay, you are effectively relaxing the check for every user (such as XEN_DOMCTL_bind_pt_irq). So the interrupt re-use case needs to be taken into account.

Quoted 1 line

Also since we only load the device driver after the IRQ is routed to the guest,

This is what a well-behave guest will do. However, we need to think what will happen if a guest misbehaves. I am not concerned about a guest only impacting itself, I am more concerned about the case where the rest of the system is impacted.

Quoted 1 line

I am not sure the guest can enable the vIRQ before it is routed.

Xen allows the guest to enable a vIRQ even if there is no pIRQ assigned. Thanksfully, it looks like the vgic_connect_hw_irq(), in both the current and new vGIC, will return an error if we are trying to route a pIRQ to an already enabled vIRQ.

But we need to investigate all the possible scenarios to make sure that any inconsistencies between the physical state and virtual state (including the LRs) will not result to bigger problem.

The one that comes to my mind is: The physical interrupt is de-assigned from the guest before it was EOIed. In this case, the interrupt will still be in the LR with the HW bit set. This would allow the guest to EOI the interrupt even if it is routed to someone else. It is unclear what would be the impact on the other guest.

Cheers,

Signature · 1 line
Julien Grall
author
#10 lore raw

Re: [PATCH 02/15] xen/arm/gic: Enable interrupt assignment to running VM

Hi Julien,

Quoted 11 lines · Julien Grall

Hi Henry,

Quoted 6 lines · Henry Wang
Quoted 1 line · Julien Grall
Quoted 1 line

Thanks for your feeedback. After checking the b8577547236f commit message I think I now understand your point. Do you have any suggestion about how can I properly add the support to route/remove the IRQ to running domains? Thanks.

I spent some time going through the GIC/vGIC code and had some discussions with Stefano and Stewart during the last couple of days, let me see if I can describe the use case properly now to continue the discussion:

We have some use cases that requires assigning devices to domains after domain boot time. For example, suppose there is an FPGA on the board which can simulate a device, and the bitstream for the FPGA is provided and programmed after domain boot. So we need a way to assign the device to the running domain. This series tries to implement this use case by using device tree overlay - users can firstly add the overlay to Xen dtb, assign the device in the overlay to a domain by the xl command, then apply the overlay to Linux.

Thanks for the description! This helps to understand your goal :).

Thank you very much for spending your time on discussing this and provide these valuable comments!

Quoted 9 lines
Quoted 8 lines
Quoted 6 lines

I haven't really look at that code in quite a while. I think we need to make sure that the virtual and physical IRQ state matches at the time we do the routing.

I am undecided on whether we want to simply prevent the action to happen or try to reset the state.

There is also the question of what to do if the guest is enabling the vIRQ before it is routed.

Sorry for bothering, would you mind elaborating a bit more about the two cases that you mentioned above? Commit b8577547236f ("xen/arm: Restrict when a physical IRQ can be routed/removed from/to a domain") only said there will be undesirable effects, so I am not sure if I understand the concerns raised above and the consequences of these two use cases.

Quoted 7 lines

I will try to explain them below after I answer the rest.

Quoted 1 line

I am probably wrong, I think when we add the overlay, we are probably fine as the interrupt is not being used before.

What if the DT overlay is unloaded and then reloaded? Wouldn't the same interrupt be re-used? As a more generic case, this could also be a new bitstream for the FPGA.

But even if the interrupt is brand new every time for the DT overlay, you are effectively relaxing the check for every user (such as XEN_DOMCTL_bind_pt_irq). So the interrupt re-use case needs to be taken into account.

I agree. I think IIUC, with your explanation here and below, could we simplify the problem to how to properly handle the removal of the IRQ from a running guest, if we always properly remove and clean up the information when remove the IRQ from the guest? In this way, the IRQ can always be viewed as a brand new one when we add it back. Then the only corner case that we need to take care of would be...

Quoted 11 lines
Quoted 1 line

Also since we only load the device driver after the IRQ is routed to the guest,

This is what a well-behave guest will do. However, we need to think what will happen if a guest misbehaves. I am not concerned about a guest only impacting itself, I am more concerned about the case where the rest of the system is impacted.

Quoted 1 line

I am not sure the guest can enable the vIRQ before it is routed.

Xen allows the guest to enable a vIRQ even if there is no pIRQ assigned. Thanksfully, it looks like the vgic_connect_hw_irq(), in both the current and new vGIC, will return an error if we are trying to route a pIRQ to an already enabled vIRQ.

But we need to investigate all the possible scenarios to make sure that any inconsistencies between the physical state and virtual state (including the LRs) will not result to bigger problem.

The one that comes to my mind is: The physical interrupt is de-assigned from the guest before it was EOIed. In this case, the interrupt will still be in the LR with the HW bit set. This would allow the guest to EOI the interrupt even if it is routed to someone else. It is unclear what would be the impact on the other guest.

...same as this case, i.e.
test_bit(_IRQ_INPROGRESS, &desc->status) || !test_bit(_IRQ_DISABLED, &desc->status)) when we try to remove the IRQ from a running domain.

we have 3 possible states which can be read from LR for this case : active, pending, pending and active.
- I don't think we can do anything about the active state, so we should return -EBUSY and reject the whole operation of removing the IRQ from running guest, and user can always retry this operation.
- For the pending (and active) case, can we clear the LR and point the LR for the pending_irq to invalid?

Kind regards,
Henry

Quoted 3 lines

Cheers,

#11 lore raw

Re: [PATCH 02/15] xen/arm/gic: Enable interrupt assignment to running VM

Hi Henry,

Quoted 26 lines · Henry Wang
Quoted 11 lines · Julien Grall

Hi Henry,

Quoted 6 lines · Henry Wang
Quoted 1 line · Julien Grall
Quoted 1 line

Thanks for your feeedback. After checking the b8577547236f commit message I think I now understand your point. Do you have any suggestion about how can I properly add the support to route/remove the IRQ to running domains? Thanks.

I spent some time going through the GIC/vGIC code and had some discussions with Stefano and Stewart during the last couple of days, let me see if I can describe the use case properly now to continue the discussion:

We have some use cases that requires assigning devices to domains after domain boot time. For example, suppose there is an FPGA on the board which can simulate a device, and the bitstream for the FPGA is provided and programmed after domain boot. So we need a way to assign the device to the running domain. This series tries to implement this use case by using device tree overlay - users can firstly add the overlay to Xen dtb, assign the device in the overlay to a domain by the xl command, then apply the overlay to Linux.

Thanks for the description! This helps to understand your goal :).

Thank you very much for spending your time on discussing this and provide these valuable comments!

Quoted 11 lines
Quoted 8 lines
Quoted 6 lines

I haven't really look at that code in quite a while. I think we need to make sure that the virtual and physical IRQ state matches at the time we do the routing.

I am undecided on whether we want to simply prevent the action to happen or try to reset the state.

There is also the question of what to do if the guest is enabling the vIRQ before it is routed.

Sorry for bothering, would you mind elaborating a bit more about the two cases that you mentioned above? Commit b8577547236f ("xen/arm: Restrict when a physical IRQ can be routed/removed from/to a domain") only said there will be undesirable effects, so I am not sure if I understand the concerns raised above and the consequences of these two use cases.

I will try to explain them below after I answer the rest.

Quoted 7 lines
Quoted 5 lines
Quoted 1 line

I am probably wrong, I think when we add the overlay, we are probably fine as the interrupt is not being used before.

What if the DT overlay is unloaded and then reloaded? Wouldn't the same interrupt be re-used? As a more generic case, this could also be a new bitstream for the FPGA.

But even if the interrupt is brand new every time for the DT overlay, you are effectively relaxing the check for every user (such as XEN_DOMCTL_bind_pt_irq). So the interrupt re-use case needs to be taken into account.

I agree. I think IIUC, with your explanation here and below, could we simplify the problem to how to properly handle the removal of the IRQ from a running guest, if we always properly remove and clean up the information when remove the IRQ from the guest? In this way, the IRQ can always be viewed as a brand new one when we add it back.

If we can make sure the virtual IRQ and physical IRQ is cleaned then yes.

Quoted 1 line

Then the only corner case that we need to take care of would be...

Can you clarify whether you say the "only corner case" because you looked at the code? Or is it just because I mentioned only one?

Quoted 7 lines
Quoted 7 lines
Quoted 2 lines

Also since we only load the device driver after the IRQ is routed to the guest,

This is what a well-behave guest will do. However, we need to think what will happen if a guest misbehaves. I am not concerned about a guest only impacting itself, I am more concerned about the case where the rest of the system is impacted.

Quoted 1 line

I am not sure the guest can enable the vIRQ before it is routed.

Quoted 8 lines
Quoted 5 lines

Xen allows the guest to enable a vIRQ even if there is no pIRQ assigned. Thanksfully, it looks like the vgic_connect_hw_irq(), in both the current and new vGIC, will return an error if we are trying to route a pIRQ to an already enabled vIRQ.

But we need to investigate all the possible scenarios to make sure that any inconsistencies between the physical state and virtual state (including the LRs) will not result to bigger problem.

The one that comes to my mind is: The physical interrupt is de-assigned from the guest before it was EOIed. In this case, the interrupt will still be in the LR with the HW bit set. This would allow the guest to EOI the interrupt even if it is routed to someone else. It is unclear what would be the impact on the other guest.

...same as this case, i.e.
test_bit(_IRQ_INPROGRESS, &desc->status) || !test_bit(_IRQ_DISABLED, &desc->status)) when we try to remove the IRQ from a running domain.

We already call ->shutdown() which will disable the IRQ. So don't we only need to take care of _IRQ_INPROGRESS?

[...]

Quoted 3 lines

we have 3 possible states which can be read from LR for this case : active, pending, pending and active.
- I don't think we can do anything about the active state, so we should
return -EBUSY and reject the whole operation of removing the IRQ from running guest, and user can always retry this operation.

This would mean a malicious/buggy guest would be able to prevent a device to be de-assigned. This is not a good idea in particular when the domain is dying.

That said, I think you can handle this case. The LR has a bit to indicate whether the pIRQ needs to be EOIed. You can clear it and this would prevent the guest to touch the pIRQ. There might be other clean-up to do in the vGIC datastructure.

Anyway, we don't have to handle removing an active IRQ when the domain is still running (although we do when the domain is destroying). But I think this would need to be solved before the feature is (security) supported.

Quoted 1 line

- For the pending (and active) case,

Shouldn't the pending and active case handled the same way as the active case?

Quoted 1 line

can we clear the LR and point the LR for the pending_irq to invalid?

LRs can be cleared. You will need to find which vCPU was used for the injection and then pause it so the LR can be safely updated.

There will also be some private state to clear. I don't know how easy it will be. However, we decided to not do anything for ICPENDR (which requires a similar behavior) as this was complex (?) to do with the existing vGIC.

I vaguely remember we had some discussions on the ML. I didn't look for them though.

Anyway, same as above, this could possibly handled later on. But this would probably need to be solved before the feature is (security supported).

Cheers,

Signature · 1 line
Julien Grall
author
#12 lore raw

Re: [PATCH 02/15] xen/arm/gic: Enable interrupt assignment to running VM

Hi Julien,

Quoted 5 lines · Julien Grall

Hi Henry,

Quoted 4 lines
Quoted 3 lines

What if the DT overlay is unloaded and then reloaded? Wouldn't the same interrupt be re-used? As a more generic case, this could also be a new bitstream for the FPGA.

But even if the interrupt is brand new every time for the DT overlay, you are effectively relaxing the check for every user (such as XEN_DOMCTL_bind_pt_irq). So the interrupt re-use case needs to be taken into account.

Quoted 7 lines
Quoted 1 line

I agree. I think IIUC, with your explanation here and below, could we simplify the problem to how to properly handle the removal of the IRQ from a running guest, if we always properly remove and clean up the information when remove the IRQ from the guest? In this way, the IRQ can always be viewed as a brand new one when we add it back.

If we can make sure the virtual IRQ and physical IRQ is cleaned then yes.

Quoted 1 line

Then the only corner case that we need to take care of would be...

Can you clarify whether you say the "only corner case" because you looked at the code? Or is it just because I mentioned only one?

Well, I indeed checked the code and to my best knowledge the corner case that you pointed out would be the only one I can think of.

Quoted 10 lines
Quoted 8 lines
Quoted 5 lines

Xen allows the guest to enable a vIRQ even if there is no pIRQ assigned. Thanksfully, it looks like the vgic_connect_hw_irq(), in both the current and new vGIC, will return an error if we are trying to route a pIRQ to an already enabled vIRQ.

But we need to investigate all the possible scenarios to make sure that any inconsistencies between the physical state and virtual state (including the LRs) will not result to bigger problem.

The one that comes to my mind is: The physical interrupt is de-assigned from the guest before it was EOIed. In this case, the interrupt will still be in the LR with the HW bit set. This would allow the guest to EOI the interrupt even if it is routed to someone else. It is unclear what would be the impact on the other guest.

...same as this case, i.e.
test_bit(_IRQ_INPROGRESS, &desc->status) || !test_bit(_IRQ_DISABLED, &desc->status)) when we try to remove the IRQ from a running domain.

We already call ->shutdown() which will disable the IRQ. So don't we only need to take care of _IRQ_INPROGRESS?

Yes you are correct.

Quoted 7 lines
Quoted 3 lines

we have 3 possible states which can be read from LR for this case : active, pending, pending and active.
- I don't think we can do anything about the active state, so we
should return -EBUSY and reject the whole operation of removing the IRQ from running guest, and user can always retry this operation.

This would mean a malicious/buggy guest would be able to prevent a device to be de-assigned. This is not a good idea in particular when the domain is dying.

That said, I think you can handle this case. The LR has a bit to indicate whether the pIRQ needs to be EOIed. You can clear it and this would prevent the guest to touch the pIRQ. There might be other clean-up to do in the vGIC datastructure.

I probably misunderstood this sentence, do you mean the EOI bit in the pINTID field? I think this bit is only available when the HW bit of LR is 0, but in our case the HW is supposed to be 1 (as indicated as your previous comment). Would you mind clarifying a bit more? Thanks!

Quoted 5 lines

Anyway, we don't have to handle removing an active IRQ when the domain is still running (although we do when the domain is destroying). But I think this would need to be solved before the feature is (security) supported.

Quoted 1 line

- For the pending (and active) case,

Shouldn't the pending and active case handled the same way as the active case?

Sorry, yes you are correct.

Kind regards,
Henry

#13 lore raw

Re: [PATCH 02/15] xen/arm/gic: Enable interrupt assignment to running VM

Hi Henry,

Quoted 8 lines · Henry Wang
Quoted 7 lines · Julien Grall

Hi Henry,

Quoted 5 lines
Quoted 3 lines

What if the DT overlay is unloaded and then reloaded? Wouldn't the same interrupt be re-used? As a more generic case, this could also be a new bitstream for the FPGA.

But even if the interrupt is brand new every time for the DT overlay, you are effectively relaxing the check for every user (such as XEN_DOMCTL_bind_pt_irq). So the interrupt re-use case needs to be taken into account.

I agree. I think IIUC, with your explanation here and below, could we simplify the problem to how to properly handle the removal of the IRQ from a running guest, if we always properly remove and clean up the information when remove the IRQ from the guest? In this way, the IRQ can always be viewed as a brand new one when we add it back.

Quoted 7 lines
Quoted 5 lines

If we can make sure the virtual IRQ and physical IRQ is cleaned then yes.

Quoted 1 line

Then the only corner case that we need to take care of would be...

Can you clarify whether you say the "only corner case" because you looked at the code? Or is it just because I mentioned only one?

Well, I indeed checked the code and to my best knowledge the corner case that you pointed out would be the only one I can think of.

Ok :). I was just checking you had a look as well.

[...]

Quoted 9 lines
Quoted 7 lines
Quoted 3 lines

we have 3 possible states which can be read from LR for this case : active, pending, pending and active.
- I don't think we can do anything about the active state, so we
should return -EBUSY and reject the whole operation of removing the IRQ from running guest, and user can always retry this operation.

This would mean a malicious/buggy guest would be able to prevent a device to be de-assigned. This is not a good idea in particular when the domain is dying.

That said, I think you can handle this case. The LR has a bit to indicate whether the pIRQ needs to be EOIed. You can clear it and this would prevent the guest to touch the pIRQ. There might be other clean-up to do in the vGIC datastructure.

I probably misunderstood this sentence, do you mean the EOI bit in the pINTID field? I think this bit is only available when the HW bit of LR is 0, but in our case the HW is supposed to be 1 (as indicated as your previous comment). Would you mind clarifying a bit more? Thanks!

You are right, ICH_LR.HW will be 1 for physical IRQ routed to a guest. What I was trying to explain is this bit could be cleared (with ICH_LR.pINTD adjusted).

Cheers,

Signature · 1 line
Julien Grall
author
#14 lore raw

Re: [PATCH 02/15] xen/arm/gic: Enable interrupt assignment to running VM

Hi Julien,

Quoted 7 lines · Julien Grall

Hi Henry,
[...]

Quoted 4 lines
Quoted 4 lines
Quoted 3 lines

we have 3 possible states which can be read from LR for this case : active, pending, pending and active.
- I don't think we can do anything about the active state, so we
should return -EBUSY and reject the whole operation of removing the IRQ from running guest, and user can always retry this operation.

Quoted 7 lines
Quoted 5 lines
Quoted 3 lines

This would mean a malicious/buggy guest would be able to prevent a device to be de-assigned. This is not a good idea in particular when the domain is dying.

That said, I think you can handle this case. The LR has a bit to indicate whether the pIRQ needs to be EOIed. You can clear it and this would prevent the guest to touch the pIRQ. There might be other clean-up to do in the vGIC datastructure.

I probably misunderstood this sentence, do you mean the EOI bit in the pINTID field? I think this bit is only available when the HW bit of LR is 0, but in our case the HW is supposed to be 1 (as indicated as your previous comment). Would you mind clarifying a bit more? Thanks!

You are right, ICH_LR.HW will be 1 for physical IRQ routed to a guest. What I was trying to explain is this bit could be cleared (with ICH_LR.pINTD adjusted).

Thank you for all the discussions. Based on that, would below diff make sense to you? I did a test of the dynamic dtbo adding/removing with a ethernet device with this patch applied. Test steps are:
(1) Use xl dt-overlay to add the ethernet device to Xen device tree and assign it to dom0.
(2) Create a domU.
(3) Use xl dt-overlay to de-assign the device from dom0 and assign it to domU.
(4) Destroy the domU.

The ethernet device is functional in the domain respectively when it is attached to a domain and I don't see errors when I destroy domU. But honestly I think the case we talked about is a quite unusual case so I am not sure if it was hit during my test.

```
diff --git a/xen/arch/arm/gic.c b/xen/arch/arm/gic.c
index a775f886ed..d3f9cd2299 100644
--- a/xen/arch/arm/gic.c
+++ b/xen/arch/arm/gic.c
@@ -135,16 +135,6 @@ int gic_route_irq_to_guest(struct domain *d, unsigned int virq,
     ASSERT(virq < vgic_num_irqs(d));
     ASSERT(!is_lpi(virq));

-    /*
-     * When routing an IRQ to guest, the virtual state is not synced
-     * back to the physical IRQ. To prevent get unsync, restrict the
-     * routing to when the Domain is been created.
-     */
-#ifndef CONFIG_OVERLAY_DTB
-    if ( d->creation_finished )
-        return -EBUSY;
-#endif
-
     ret = vgic_connect_hw_irq(d, NULL, virq, desc, true);
     if ( ret )
         return ret;
@@ -169,20 +159,40 @@ int gic_remove_irq_from_guest(struct domain *d, unsigned int virq,
     ASSERT(test_bit(_IRQ_GUEST, &desc->status));
     ASSERT(!is_lpi(virq));

-    /*
-     * Removing an interrupt while the domain is running may have
-     * undesirable effect on the vGIC emulation.
-     */
-#ifndef CONFIG_OVERLAY_DTB
-    if ( !d->is_dying )
-        return -EBUSY;
-#endif
-
     desc->handler->shutdown(desc);

     /* EOI the IRQ if it has not been done by the guest */
     if ( test_bit(_IRQ_INPROGRESS, &desc->status) )
+    {
+        /*
+         * Handle the LR where the physical interrupt is de-assigned from the
+         * guest before it was EOIed
+         */
+        struct vcpu *v_target = vgic_get_target_vcpu(d->vcpu[0], virq);
+        struct vgic_irq_rank *rank = vgic_rank_irq(v_target, virq);
+        struct pending_irq *p = irq_to_pending(v_target, virq);
+        unsigned long flags;
+
+        spin_lock_irqsave(&v_target->arch.vgic.lock, flags);
+        /* LR allocated for the IRQ */
+        if ( test_bit(GIC_IRQ_GUEST_ACTIVE, &p->status) &&
+             test_bit(GIC_IRQ_GUEST_VISIBLE, &p->status) )
+        {
+            gic_hw_ops->clear_lr(p->lr);
+            clear_bit(p->lr, &v_target->arch.lr_mask);
+
+            clear_bit(GIC_IRQ_GUEST_VISIBLE, &p->status);
+            clear_bit(GIC_IRQ_GUEST_ACTIVE, &p->status);
+            p->lr = GIC_INVALID_LR;
+        }
+        spin_unlock_irqrestore(&v_target->arch.vgic.lock, flags);
+
+        vgic_lock_rank(v_target, rank, flags);
+        vgic_disable_irqs(v_target, (~rank->ienable) & rank->ienable, rank->index);
+        vgic_unlock_rank(v_target, rank, flags);
+
         gic_hw_ops->deactivate_irq(desc);
+    }
     clear_bit(_IRQ_INPROGRESS, &desc->status);

     ret = vgic_connect_hw_irq(d, NULL, virq, desc, false);
```

Kind regards,
Henry

Quoted 3 lines

Cheers,

#15 lore raw

Re: [PATCH 02/15] xen/arm/gic: Enable interrupt assignment to running VM

Hi,

Quoted 16 lines · Henry Wang
Quoted 7 lines · Julien Grall

Hi Henry,
[...]

Quoted 4 lines
Quoted 4 lines
Quoted 3 lines

we have 3 possible states which can be read from LR for this case : active, pending, pending and active.
- I don't think we can do anything about the active state, so we
should return -EBUSY and reject the whole operation of removing the IRQ from running guest, and user can always retry this operation.

Quoted 7 lines
Quoted 5 lines
Quoted 3 lines

This would mean a malicious/buggy guest would be able to prevent a device to be de-assigned. This is not a good idea in particular when the domain is dying.

That said, I think you can handle this case. The LR has a bit to indicate whether the pIRQ needs to be EOIed. You can clear it and this would prevent the guest to touch the pIRQ. There might be other clean-up to do in the vGIC datastructure.

I probably misunderstood this sentence, do you mean the EOI bit in the pINTID field? I think this bit is only available when the HW bit of LR is 0, but in our case the HW is supposed to be 1 (as indicated as your previous comment). Would you mind clarifying a bit more? Thanks!

You are right, ICH_LR.HW will be 1 for physical IRQ routed to a guest. What I was trying to explain is this bit could be cleared (with ICH_LR.pINTD adjusted).

Quoted 7 lines

Thank you for all the discussions. Based on that, would below diff make sense to you? I did a test of the dynamic dtbo adding/removing with a ethernet device with this patch applied. Test steps are:
(1) Use xl dt-overlay to add the ethernet device to Xen device tree and assign it to dom0.
(2) Create a domU.
(3) Use xl dt-overlay to de-assign the device from dom0 and assign it to domU.
(4) Destroy the domU.

The ethernet device is functional in the domain respectively when it is attached to a domain and I don't see errors when I destroy domU. But honestly I think the case we talked about is a quite unusual case so I am not sure if it was hit during my test.

Correct, they are not errors that will happen in normal operations. You will want to tweak Linux or use XTF to trigger them.

We want to test the state active, pending and both together before the physical interrupt is routed and also removed.

Quoted 12 lines

```

xen/arch/arm/gic.c +0 -1
index a775f886ed..d3f9cd2299 100644
@@ -135,16 +135,6 @@ int gic_route_irq_to_guest(struct domain *d, 
 unsigned int virq,
      ASSERT(virq < vgic_num_irqs(d));
      ASSERT(!is_lpi(virq));
 
-    /*

xen/arch/arm/gic.c int gic_route_irq_to_guest(struct domain *d, · line 139

Quoted 10 lines
xen/arch/arm/gic.c +0 -9
-     * When routing an IRQ to guest, the virtual state is not synced
-     * back to the physical IRQ. To prevent get unsync, restrict the
-     * routing to when the Domain is been created.
-     */
-#ifndef CONFIG_OVERLAY_DTB
-    if ( d->creation_finished )
-        return -EBUSY;
-#endif
-
      ret = vgic_connect_hw_irq(d, NULL, virq, desc, true);

This is checking if the interrupt is already enabled. Do we also need to check for active/pending?

Quoted 11 lines
xen/arch/arm/gic.c +0 -0
      if ( ret )

         return ret;

xen/arch/arm/gic.c +0 -4
@@ -169,20 +159,40 @@ int gic_remove_irq_from_guest(struct domain *d, 
 unsigned int virq,
      ASSERT(test_bit(_IRQ_GUEST, &desc->status));
      ASSERT(!is_lpi(virq));
 
-    /*
-     * Removing an interrupt while the domain is running may have
-     * undesirable effect on the vGIC emulation.
-     */

xen/arch/arm/gic.c int gic_remove_irq_from_guest(struct domain *d, · line 167

Quoted 10 lines
xen/arch/arm/gic.c +1 -5
-#ifndef CONFIG_OVERLAY_DTB
-    if ( !d->is_dying )
-        return -EBUSY;
-#endif
-
      desc->handler->shutdown(desc);
 
      /* EOI the IRQ if it has not been done by the guest */
      if ( test_bit(_IRQ_INPROGRESS, &desc->status) )
+    {

I assume this is just a PoC state, but I just want to point out that this will not work with the new vGIC (some of the functions doesn't exist there).

Quoted 6 lines
xen/arch/arm/gic.c +2 -0
+        /*
+         * Handle the LR where the physical interrupt is de-assigned 

from the
+         * guest before it was EOIed
+         */
+        struct vcpu *v_target = vgic_get_target_vcpu(d->vcpu[0], virq);

This will return a vCPU from the current affinity. This may not be where the interrupt was injected. From a brief look, I can't tell whether we have an easy way to know where the interrupt was injected (other than the pending_irq is in the list lr_queue/inflight)

Quoted 10 lines

+        struct vgic_irq_rank *rank = vgic_rank_irq(v_target, virq); +        struct pending_irq *p = irq_to_pending(v_target, virq); +        unsigned long flags;
+
+        spin_lock_irqsave(&v_target->arch.vgic.lock, flags); +        /* LR allocated for the IRQ */
+        if ( test_bit(GIC_IRQ_GUEST_ACTIVE, &p->status) && +             test_bit(GIC_IRQ_GUEST_VISIBLE, &p->status) ) +        {
+            gic_hw_ops->clear_lr(p->lr);

This works on the current pCPU LR. However, the vCPU may not run on this pCPU.

Quoted 5 lines

+            clear_bit(p->lr, &v_target->arch.lr_mask); +
+            clear_bit(GIC_IRQ_GUEST_VISIBLE, &p->status); +            clear_bit(GIC_IRQ_GUEST_ACTIVE, &p->status); +            p->lr = GIC_INVALID_LR;

You also need to remove 'p' from the various list (e.g. inflight/lr_queue).

But as I wrote previously, I think it would be much easier if we simply clear the HW bit in the LR. So we don't have to try to mess up with the vGIC internal state which is quite complex.

I think it could be done as part of vgic_connect_hw_irq(). But I haven't fully investigate it.

Quoted 7 lines

+        }
+        spin_unlock_irqrestore(&v_target->arch.vgic.lock, flags); +
+        vgic_lock_rank(v_target, rank, flags);
+        vgic_disable_irqs(v_target, (~rank->ienable) & rank->ienable, rank->index);
+        vgic_unlock_rank(v_target, rank, flags);

Why do you need to call vgic_disable_irqs()?

Quoted 5 lines

+
         gic_hw_ops->deactivate_irq(desc);
+    }
     clear_bit(_IRQ_INPROGRESS, &desc->status);

Quoted 10 lines

     ret = vgic_connect_hw_irq(d, NULL, virq, desc, false);
```

Kind regards,
Henry

Quoted 3 lines

Cheers,

Cheers,

Signature · 1 line
Julien Grall
author
#16 lore raw

Re: [PATCH 02/15] xen/arm/gic: Enable interrupt assignment to running VM

Hi Julien,

Quoted 16 lines · Julien Grall

Hi,

Quoted 13 lines · Henry Wang
Quoted 12 lines · Julien Grall

Hi Henry,
[...]
```

xen/arch/arm/gic.c +0 -0
index a775f886ed..d3f9cd2299 100644
@@ -135,16 +135,6 @@ int gic_route_irq_to_guest(struct domain *d, 
 unsigned int virq,
       ASSERT(virq < vgic_num_irqs(d));
       ASSERT(!is_lpi(virq));
 
Quoted 13 lines
Quoted 11 lines

xen/arch/arm/gic.c int gic_route_irq_to_guest(struct domain *d, · line 139

Quoted 11 lines
xen/arch/arm/gic.c +0 -10
-    /*
-     * When routing an IRQ to guest, the virtual state is not synced
-     * back to the physical IRQ. To prevent get unsync, restrict the
-     * routing to when the Domain is been created.
-     */
-#ifndef CONFIG_OVERLAY_DTB
-    if ( d->creation_finished )
-        return -EBUSY;
-#endif
-
       ret = vgic_connect_hw_irq(d, NULL, virq, desc, true);

This is checking if the interrupt is already enabled. Do we also need to check for active/pending?

Thank you for raising this! I assume you meant this?
@@ -444,7 +444,9 @@ int vgic_connect_hw_irq(struct domain *d, struct vcpu *v, unsigned int virq, {
         /* The VIRQ should not be already enabled by the guest */
         if ( !p->desc &&
-             !test_bit(GIC_IRQ_GUEST_ENABLED, &p->status) )
+             !test_bit(GIC_IRQ_GUEST_ENABLED, &p->status) &&
+             !test_bit(GIC_IRQ_GUEST_ACTIVE, &p->status) &&
+             !test_bit(GIC_IRQ_GUEST_VISIBLE, &p->status) )
             p->desc = desc;
         else
             ret = -EBUSY;

I think adding the check for active/pending check at the time of routing the IRQ makes sense, so I will add them (both for old and new vGIC implementation).

Quoted 18 lines
Quoted 18 lines

      if ( ret )
          return ret;

(quoted diff) +0 -9
@@ -169,20 +159,40 @@ int gic_remove_irq_from_guest(struct domain *d, 
 unsigned int virq,
       ASSERT(test_bit(_IRQ_GUEST, &desc->status));
       ASSERT(!is_lpi(virq));
 
-    /*
-     * Removing an interrupt while the domain is running may have
-     * undesirable effect on the vGIC emulation.
-     */
-#ifndef CONFIG_OVERLAY_DTB
-    if ( !d->is_dying )
-        return -EBUSY;
-#endif
-
       desc->handler->shutdown(desc);
 
Quoted 5 lines

Review int gic_remove_irq_from_guest(struct domain *d, · line 167

Quoted 3 lines
(quoted diff) +1 -0
       /* EOI the IRQ if it has not been done by the guest */
       if ( test_bit(_IRQ_INPROGRESS, &desc->status) )
+    {

I assume this is just a PoC state, but I just want to point out that this will not work with the new vGIC (some of the functions doesn't exist there).

Thank you. Yes currently we can discuss for the old vGIC implementation. After we reach the final conclusion I will do the changes for both old and new vGIC.

Quoted 8 lines
Quoted 6 lines
(quoted diff) +2 -0
+        /*
+         * Handle the LR where the physical interrupt is de-assigned 

from the
+         * guest before it was EOIed
+         */
+        struct vcpu *v_target = vgic_get_target_vcpu(d->vcpu[0], virq);

This will return a vCPU from the current affinity. This may not be where the interrupt was injected. From a brief look, I can't tell whether we have an easy way to know where the interrupt was injected (other than the pending_irq is in the list lr_queue/inflight)

I doubt if we need to handle more than this - I think if the pending_irq is not in the lr_queue/inflight list, it would not belong to the corner case we are talking about (?).

Quoted 9 lines
Quoted 7 lines

+        }
+ spin_unlock_irqrestore(&v_target->arch.vgic.lock, flags); +
+        vgic_lock_rank(v_target, rank, flags);
+        vgic_disable_irqs(v_target, (~rank->ienable) & rank->ienable, rank->index);
+        vgic_unlock_rank(v_target, rank, flags);

Why do you need to call vgic_disable_irqs()?

I will drop this part.

Kind regards,
Henry

#17 lore raw

Re: [PATCH 02/15] xen/arm/gic: Enable interrupt assignment to running VM

Hi Henry,

Quoted 10 lines · Henry Wang
Quoted 8 lines
Quoted 6 lines

+        /*
+         * Handle the LR where the physical interrupt is de-assigned from the
+         * guest before it was EOIed
+         */
+        struct vcpu *v_target = vgic_get_target_vcpu(d->vcpu[0], virq);

This will return a vCPU from the current affinity. This may not be where the interrupt was injected. From a brief look, I can't tell whether we have an easy way to know where the interrupt was injected (other than the pending_irq is in the list lr_queue/inflight)

I doubt if we need to handle more than this - I think if the pending_irq is not in the lr_queue/inflight list, it would not belong to the corner case we are talking about (?).

I didn't suggest we would need to handle the case where the pending_irq is not any of the queues. I was pointing out that I think we don't directly store the vCPU ID where we injected the IRQ. Instead, the pending_irq is just in list, so we will possibly need to store the vCPU ID for convenience.

Cheers,

Signature · 1 line
Julien Grall
author
#18 lore raw

Re: [PATCH 02/15] xen/arm/gic: Enable interrupt assignment to running VM

Hi Julien,

Quoted 11 lines · Julien Grall

Hi Henry,

Quoted 8 lines · Henry Wang
Quoted 8 lines
Quoted 7 lines

+        /*
+         * Handle the LR where the physical interrupt is de-assigned from the
+         * guest before it was EOIed
+         */
+        struct vcpu *v_target = vgic_get_target_vcpu(d->vcpu[0], virq);

Quoted 5 lines
Quoted 3 lines
Quoted 1 line

This will return a vCPU from the current affinity. This may not be where the interrupt was injected. From a brief look, I can't tell whether we have an easy way to know where the interrupt was injected (other than the pending_irq is in the list lr_queue/inflight)

I doubt if we need to handle more than this - I think if the pending_irq is not in the lr_queue/inflight list, it would not belong to the corner case we are talking about (?).

I didn't suggest we would need to handle the case where the pending_irq is not any of the queues. I was pointing out that I think we don't directly store the vCPU ID where we injected the IRQ. Instead, the pending_irq is just in list, so we will possibly need to store the vCPU ID for convenience.

Sorry for misunderstanding. Yeah you are definitely correct. Also thank you so much for the suggestion! Before seeing this suggestion, I was struggling in finding the correct vCPU by "for_each_vcpus" and comparison... but now I realized your suggestion is way more clever :)

Kind regards,
Henry

author
#19 lore raw

PATCH 3/15 xen/arm: Always enable IOMMU

In reply to Henry Wang

From: Vikram Garhwal <[email protected]>

For overlay with iommu functionality to work with running VMs, we need to enable IOMMU by default for the domains.

---
 xen/arch/arm/dom0less-build.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
xen/arch/arm/dom0less-build.c +2 -1
index fb63ec6fd1..2d1fd1e214 100644
@@ -894,7 +894,8 @@ void __init create_domUs(void)
             panic("Missing property 'cpus' for domain %s\n",
                   dt_node_name(node));
 
-        if ( dt_find_compatible_node(node, NULL, "multiboot,device-tree") &&
+        if ( (IS_ENABLED(CONFIG_OVERLAY_DTB) ||
+              dt_find_compatible_node(node, NULL, "multiboot,device-tree")) &&
              iommu_enabled )
             d_cfg.flags |= XEN_DOMCTL_CDF_iommu;
 
Signature · 1 line
2.34.1
#20 lore raw

Re: [PATCH 03/15] xen/arm: Always enable IOMMU

Hi Henry,

Quoted 17 lines · Henry Wang

From: Vikram Garhwal <[email protected]>

For overlay with iommu functionality to work with running VMs, we need to enable IOMMU by default for the domains.

---
  xen/arch/arm/dom0less-build.c | 3 ++-
  1 file changed, 2 insertions(+), 1 deletion(-)
xen/arch/arm/dom0less-build.c +0 -0
index fb63ec6fd1..2d1fd1e214 100644

xen/arch/arm/dom0less-build.c void __init create_domUs(void) · line 897

Quoted 5 lines
xen/arch/arm/dom0less-build.c +1 -1
@@ -894,7 +894,8 @@ void __init create_domUs(void)
              panic("Missing property 'cpus' for domain %s\n",
                    dt_node_name(node));
 
-        if ( dt_find_compatible_node(node, NULL, "multiboot,device-tree") &&
+        if ( (IS_ENABLED(CONFIG_OVERLAY_DTB) ||

Similar to the first patch, building Xen with the DTB overlay doesn't mean the user will want to use it (think of distros that may want to provide a generic Xen).

Instead, we should introduce a new DT property "passthrough" that would indicate whether the IOMMU should be used.

To be futureproof, I would match the values used by xl.cfg (see docs/man/xl.cfg.5.pod.in).

xen/arch/arm/dom0less-build.c void __init create_domUs(void) · line 901

Quoted 4 lines
xen/arch/arm/dom0less-build.c +1 -0
+              dt_find_compatible_node(node, NULL, "multiboot,device-tree")) &&
               iommu_enabled )
              d_cfg.flags |= XEN_DOMCTL_CDF_iommu;
 

Cheers,

Signature · 1 line
Julien Grall
author
#21 lore raw

Re: [PATCH 03/15] xen/arm: Always enable IOMMU

Hi Julien,

Quoted 27 lines · Julien Grall

Hi Henry,

Quoted 19 lines · Henry Wang

From: Vikram Garhwal <[email protected]>

For overlay with iommu functionality to work with running VMs, we need to enable IOMMU by default for the domains.

---
  xen/arch/arm/dom0less-build.c | 3 ++-

  1 file changed, 2 insertions(+), 1 deletion(-)
xen/arch/arm/dom0less-build.c +0 -0
index fb63ec6fd1..2d1fd1e214 100644
@@ -894,7 +894,8 @@ void __init create_domUs(void)
               panic("Missing property 'cpus' for domain %s\n",
                     dt_node_name(node));
Quoted 2 lines

  -        if ( dt_find_compatible_node(node, NULL, "multiboot,device-tree") && +        if ( (IS_ENABLED(CONFIG_OVERLAY_DTB) ||

Similar to the first patch, building Xen with the DTB overlay doesn't mean the user will want to use it (think of distros that may want to provide a generic Xen).

Quoted 3 lines

Instead, we should introduce a new DT property "passthrough" that would indicate whether the IOMMU should be used.

To be futureproof, I would match the values used by xl.cfg (see docs/man/xl.cfg.5.pod.in).

That sounds good. I can introduce a new DT property as suggested. Thanks for the suggestion!

Kind regards,
Henry

Quoted 7 lines
Quoted 4 lines

+ dt_find_compatible_node(node, NULL, "multiboot,device-tree")) && iommu_enabled )
              d_cfg.flags |= XEN_DOMCTL_CDF_iommu;

Cheers,

author
#22 lore raw

PATCH 4/15 tools/libs/light: Always enable IOMMU

In reply to Henry Wang

From: Vikram Garhwal <[email protected]>

For overlay with iommu functionality to work with running VMs, we need to enable IOMMU when iomem presents for the domains.

---
 tools/libs/light/libxl_arm.c | 6 ++++++
 1 file changed, 6 insertions(+)
tools/libs/light/libxl_arm.c +6 -0
index 1cb89fa584..dd5c9f4917 100644
@@ -222,6 +222,12 @@ int libxl__arch_domain_prepare_config(libxl__gc *gc,
         config->arch.sve_vl = d_config->b_info.arch_arm.sve_vl / 128U;
     }
 
+#ifdef LIBXL_HAVE_DT_OVERLAY
+    if (d_config->b_info.num_iomem) {
+        config->flags |= XEN_DOMCTL_CDF_iommu;
+    }
+#endif
+
     return 0;
 }
 
Signature · 1 line
2.34.1
#23 lore raw

Re: [PATCH 04/15] tools/libs/light: Always enable IOMMU

Quoted 14 lines · Henry Wang

For overlay with iommu functionality to work with running VMs, we need to enable IOMMU when iomem presents for the domains.

---
 tools/libs/light/libxl_arm.c | 6 ++++++
 1 file changed, 6 insertions(+)
tools/libs/light/libxl_arm.c +0 -0
index 1cb89fa584..dd5c9f4917 100644

tools/libs/light/libxl_arm.c int libxl__arch_domain_prepare_config(libxl__gc *gc, · line 225

Quoted 4 lines
tools/libs/light/libxl_arm.c +1 -0
@@ -222,6 +222,12 @@ int libxl__arch_domain_prepare_config(libxl__gc *gc,
         config->arch.sve_vl = d_config->b_info.arch_arm.sve_vl / 128U;
     }
 
+#ifdef LIBXL_HAVE_DT_OVERLAY

libxl_arm.c is only build on Arm, so this should be defined, so no need to check.

tools/libs/light/libxl_arm.c int libxl__arch_domain_prepare_config(libxl__gc *gc, · line 227

Quoted 2 lines
tools/libs/light/libxl_arm.c +2 -0
+    if (d_config->b_info.num_iomem) {
+        config->flags |= XEN_DOMCTL_CDF_iommu;

Is this doing the same thing as the previous patch?

Thanks,

Signature · 1 line
Anthony PERARD
author
#24 lore raw

Re: [PATCH 04/15] tools/libs/light: Always enable IOMMU

Hi Anthony,

Quoted 18 lines · Anthony PERARD
Quoted 17 lines · Henry Wang

For overlay with iommu functionality to work with running VMs, we need to enable IOMMU when iomem presents for the domains.

---
  tools/libs/light/libxl_arm.c | 6 ++++++
  1 file changed, 6 insertions(+)
tools/libs/light/libxl_arm.c +0 -0
index 1cb89fa584..dd5c9f4917 100644
@@ -222,6 +222,12 @@ int libxl__arch_domain_prepare_config(libxl__gc *gc,
          config->arch.sve_vl = d_config->b_info.arch_arm.sve_vl / 128U;
      }
 
Quoted 3 lines

tools/libs/light/libxl_arm.c int libxl__arch_domain_prepare_config(libxl__gc *gc, · line 225

Quoted 1 line
tools/libs/light/libxl_arm.c +1 -0
+#ifdef LIBXL_HAVE_DT_OVERLAY

libxl_arm.c is only build on Arm, so this should be defined, so no need to check.

Ah sure, I was just thought in the future RISC-V/PPC may have the same, but you are correct. I will remove the check.

Quoted 3 lines

tools/libs/light/libxl_arm.c int libxl__arch_domain_prepare_config(libxl__gc *gc, · line 227

Quoted 2 lines
tools/libs/light/libxl_arm.c +2 -0
+    if (d_config->b_info.num_iomem) {
+        config->flags |= XEN_DOMCTL_CDF_iommu;

Is this doing the same thing as the previous patch?

I think so, yes, we need the IOMMU flag to be set if we want to assign a device from a DT node protected by IOMMU.

Kind regards,
Henry

Quoted 3 lines

Thanks,

author
#25 lore raw

PATCH 5/15 tools/libs/light: Increase nr_spi to 160

In reply to Henry Wang

From: Vikram Garhwal <[email protected]>

Increase number of spi to 160 i.e. gic_number_lines() for Xilinx ZynqMP - 32. This was done to allocate and assign IRQs to a running domain.

---
 tools/libs/light/libxl_arm.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
tools/libs/light/libxl_arm.c +2 -1
index dd5c9f4917..50dbd0f2a9 100644
@@ -181,7 +181,8 @@ int libxl__arch_domain_prepare_config(libxl__gc *gc,
 
     LOG(DEBUG, "Configure the domain");
 
-    config->arch.nr_spis = nr_spis;
+    /* gic_number_lines() is 192 for Xilinx ZynqMP. min nr_spis = 192 - 32. */
+    config->arch.nr_spis = MAX(nr_spis, 160);
     LOG(DEBUG, " - Allocate %u SPIs", nr_spis);
 
     switch (d_config->b_info.arch_arm.gic_version) {
Signature · 1 line
2.34.1
#26 lore raw

Re: [PATCH 05/15] tools/libs/light: Increase nr_spi to 160

Quoted 15 lines · Henry Wang

Increase number of spi to 160 i.e. gic_number_lines() for Xilinx ZynqMP - 32. This was done to allocate and assign IRQs to a running domain.

---
 tools/libs/light/libxl_arm.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
tools/libs/light/libxl_arm.c +0 -0
index dd5c9f4917..50dbd0f2a9 100644

tools/libs/light/libxl_arm.c int libxl__arch_domain_prepare_config(libxl__gc *gc, · line 185

Quoted 6 lines
tools/libs/light/libxl_arm.c +2 -1
@@ -181,7 +181,8 @@ int libxl__arch_domain_prepare_config(libxl__gc *gc,
 
     LOG(DEBUG, "Configure the domain");
 
-    config->arch.nr_spis = nr_spis;
+    /* gic_number_lines() is 192 for Xilinx ZynqMP. min nr_spis = 192 - 32. */
+    config->arch.nr_spis = MAX(nr_spis, 160);

Is there a way that that Xen or libxl could find out what the minimum number of SPI needs to be? Are we going to have to increase that minimum number every time a new platform comes along?

It doesn't appear that libxl is using that nr_spis value and it is probably just given to Xen. So my guess is that Xen could simply take care of the minimum value, gic_number_lines() seems to be a Xen function.

Thanks,

Signature · 1 line
Anthony PERARD
author
#27 lore raw

Re: [PATCH 05/15] tools/libs/light: Increase nr_spi to 160

Hi Anthony,

(+Arm maintainers)

Quoted 17 lines · Anthony PERARD
Quoted 16 lines · Henry Wang

Increase number of spi to 160 i.e. gic_number_lines() for Xilinx ZynqMP - 32. This was done to allocate and assign IRQs to a running domain.

---
  tools/libs/light/libxl_arm.c | 3 ++-
  1 file changed, 2 insertions(+), 1 deletion(-)
tools/libs/light/libxl_arm.c +0 -0
index dd5c9f4917..50dbd0f2a9 100644
@@ -181,7 +181,8 @@ int libxl__arch_domain_prepare_config(libxl__gc *gc,
 
Quoted 7 lines

tools/libs/light/libxl_arm.c int libxl__arch_domain_prepare_config(libxl__gc *gc, · line 185

Quoted 5 lines
tools/libs/light/libxl_arm.c +2 -1
      LOG(DEBUG, "Configure the domain");
 
-    config->arch.nr_spis = nr_spis;
+    /* gic_number_lines() is 192 for Xilinx ZynqMP. min nr_spis = 192 - 32. */
+    config->arch.nr_spis = MAX(nr_spis, 160);

Is there a way that that Xen or libxl could find out what the minimum number of SPI needs to be?

I am afraid currently there is none.

Quoted 7 lines

Are we going to have to increase that minimum
number every time a new platform comes along?

It doesn't appear that libxl is using that nr_spis value and it is probably just given to Xen. So my guess is that Xen could simply take care of the minimum value, gic_number_lines() seems to be a Xen function.

Xen will take care of the value of nr_spis for dom0 in create_dom0() dom0_cfg.arch.nr_spis = min(gic_number_lines(), (unsigned int) 992) - 32; and also for dom0less domUs in create_domUs().

However, it looks like Xen will not take care of the mininum value for libxl guests, the value from config->arch.nr_spis in guest config file will be directly passed to the domain_vgic_init() function from arch_domain_create().

I agree with you that we shouldn't just bump the number everytime when we have a new platform. Therefore, would it be a good idea to move the logic in this patch to arch_sanitise_domain_config()?

Kind regards,
Henry

Quoted 3 lines

Thanks,

#28 lore raw

Re: [PATCH 05/15] tools/libs/light: Increase nr_spi to 160

Hi,

Quoted 35 lines · Henry Wang
Quoted 11 lines · Anthony PERARD
Quoted 10 lines · Henry Wang

Increase number of spi to 160 i.e. gic_number_lines() for Xilinx ZynqMP - 32. This was done to allocate and assign IRQs to a running domain.

---
  tools/libs/light/libxl_arm.c | 3 ++-

  1 file changed, 2 insertions(+), 1 deletion(-)
Quoted 12 lines
Quoted 10 lines
tools/libs/light/libxl_arm.c +1 -1
index dd5c9f4917..50dbd0f2a9 100644
@@ -181,7 +181,8 @@ int libxl__arch_domain_prepare_config(libxl__gc *gc,
       LOG(DEBUG, "Configure the domain");
-    config->arch.nr_spis = nr_spis;
+    /* gic_number_lines() is 192 for Xilinx ZynqMP. min nr_spis = 

192 - 32. */
+    config->arch.nr_spis = MAX(nr_spis, 160);

Is there a way that that Xen or libxl could find out what the minimum number of SPI needs to be?

I am afraid currently there is none.

Quoted 7 lines

Are we going to have to increase that minimum
number every time a new platform comes along?

It doesn't appear that libxl is using that nr_spis value and it is probably just given to Xen. So my guess is that Xen could simply take care of the minimum value, gic_number_lines() seems to be a Xen function.

Quoted 7 lines

Xen will take care of the value of nr_spis for dom0 in create_dom0() dom0_cfg.arch.nr_spis = min(gic_number_lines(), (unsigned int) 992) - 32; and also for dom0less domUs in create_domUs().

However, it looks like Xen will not take care of the mininum value for libxl guests, the value from config->arch.nr_spis in guest config file will be directly passed to the domain_vgic_init() function from arch_domain_create().

I agree with you that we shouldn't just bump the number everytime when we have a new platform. Therefore, would it be a good idea to move the logic in this patch to arch_sanitise_domain_config()?

Xen domains are supposed to be platform agnostics and therefore the numbers of SPIs should not be based on the HW.

Furthermore, with your proposal we would end up to allocate data structure for N SPIs when a domain may never needs any SPIs (such as if passthrough is not in-use). This is more likely for domain created by the toolstack than from Xen directly.

Instead, we should introduce a new XL configuration to let the user decide the number of SPIs. I would suggest to name "nr_spis" to match the DT bindings.

Cheers,

Signature · 1 line
Julien Grall
author
#29 lore raw

Re: [PATCH 05/15] tools/libs/light: Increase nr_spi to 160

Hi Julien,

Quoted 43 lines · Julien Grall

Hi,

Quoted 40 lines · Henry Wang
Quoted 11 lines · Anthony PERARD
Quoted 10 lines · Henry Wang

Increase number of spi to 160 i.e. gic_number_lines() for Xilinx ZynqMP - 32. This was done to allocate and assign IRQs to a running domain.

---
  tools/libs/light/libxl_arm.c | 3 ++-

  1 file changed, 2 insertions(+), 1 deletion(-)
Quoted 13 lines
Quoted 11 lines
tools/libs/light/libxl_arm.c +1 -1
index dd5c9f4917..50dbd0f2a9 100644
@@ -181,7 +181,8 @@ int libxl__arch_domain_prepare_config(libxl__gc 
 *gc,
       LOG(DEBUG, "Configure the domain");
-    config->arch.nr_spis = nr_spis;
+    /* gic_number_lines() is 192 for Xilinx ZynqMP. min nr_spis = 

192 - 32. */
+    config->arch.nr_spis = MAX(nr_spis, 160);

Is there a way that that Xen or libxl could find out what the minimum number of SPI needs to be?

I am afraid currently there is none.

Quoted 7 lines

Are we going to have to increase that minimum
number every time a new platform comes along?

It doesn't appear that libxl is using that nr_spis value and it is probably just given to Xen. So my guess is that Xen could simply take care of the minimum value, gic_number_lines() seems to be a Xen function.

Xen will take care of the value of nr_spis for dom0 in create_dom0() dom0_cfg.arch.nr_spis = min(gic_number_lines(), (unsigned int) 992) - 32; and also for dom0less domUs in create_domUs().

Quoted 7 lines
Quoted 3 lines

However, it looks like Xen will not take care of the mininum value for libxl guests, the value from config->arch.nr_spis in guest config file will be directly passed to the domain_vgic_init() function from arch_domain_create().

I agree with you that we shouldn't just bump the number everytime when we have a new platform. Therefore, would it be a good idea to move the logic in this patch to arch_sanitise_domain_config()?

Xen domains are supposed to be platform agnostics and therefore the numbers of SPIs should not be based on the HW.

Furthermore, with your proposal we would end up to allocate data structure for N SPIs when a domain may never needs any SPIs (such as if passthrough is not in-use). This is more likely for domain created by the toolstack than from Xen directly.

Agreed on both comments.

Quoted 1 line

Instead, we should introduce a new XL configuration to let the user decide the number of SPIs. I would suggest to name "nr_spis" to match the DT bindings.

Sure, I will introduce a new xl config for this to replace this patch. Thank you for the suggestion.

Kind regards,
Henry

Quoted 3 lines

Cheers,

author
#30 lore raw

PATCH 6/15 rangeset: Move struct range and struct rangeset to headerfile

In reply to Henry Wang

From: Vikram Garhwal <[email protected]>

Move struct range, rangeset and removed static from first_range and next_range(). IRQs and IOMEMs for nodes are stored as rangeset in the dynamic node addition part. While removing the nodes we need to access every IRQ and IOMEM ranges to unmap IRQ and IOMEM from the domain.

---
 xen/common/rangeset.c      | 31 ++-----------------------------
 xen/include/xen/rangeset.h | 32 +++++++++++++++++++++++++++++++-
 2 files changed, 33 insertions(+), 30 deletions(-)
xen/common/rangeset.c +2 -29
index b75590f907..d3f4297e41 100644
@@ -12,31 +12,6 @@
 #include <xen/rangeset.h>
 #include <xsm/xsm.h>
 
-/* An inclusive range [s,e] and pointer to next range in ascending order. */
-struct range {
-    struct list_head list;
-    unsigned long s, e;
-};
-
-struct rangeset {
-    /* Owning domain and threaded list of rangesets. */
-    struct list_head rangeset_list;
-    struct domain   *domain;
-
-    /* Ordered list of ranges contained in this set, and protecting lock. */
-    struct list_head range_list;
-
-    /* Number of ranges that can be allocated */
-    long             nr_ranges;
-    rwlock_t         lock;
-
-    /* Pretty-printing name. */
-    char             name[32];
-
-    /* RANGESETF flags. */
-    unsigned int     flags;
-};
-
 /*****************************
  * Private range functions hide the underlying linked-list implemnetation.
  */
@@ -57,8 +32,7 @@ static struct range *find_range(
     return x;
 }
 
-/* Return the lowest range in the set r, or NULL if r is empty. */
-static struct range *first_range(
+struct range *first_range(
     struct rangeset *r)
 {
     if ( list_empty(&r->range_list) )
@@ -66,8 +40,7 @@ static struct range *first_range(
     return list_entry(r->range_list.next, struct range, list);
 }
 
-/* Return range following x in ascending order, or NULL if x is the highest. */
-static struct range *next_range(
+struct range *next_range(
     struct rangeset *r, struct range *x)
 {
     if ( x->list.next == &r->range_list )
xen/include/xen/rangeset.h +31 -1
index 96c9180825..cd80fd9179 100644
@@ -13,7 +13,37 @@
 #include <xen/types.h>
 
 struct domain;
-struct rangeset;
+
+struct rangeset {
+    /* Owning domain and threaded list of rangesets. */
+    struct list_head rangeset_list;
+    struct domain   *domain;
+
+    /* Ordered list of ranges contained in this set, and protecting lock. */
+    struct list_head range_list;
+
+    /* Number of ranges that can be allocated */
+    long             nr_ranges;
+    rwlock_t         lock;
+
+    /* Pretty-printing name. */
+    char             name[32];
+
+    /* RANGESETF flags. */
+    unsigned int     flags;
+};
+
+/* An inclusive range [s,e] and pointer to next range in ascending order. */
+struct range {
+    struct list_head list;
+    unsigned long s, e;
+};
+
+/* Return the lowest range in the set r, or NULL if r is empty. */
+struct range *first_range(struct rangeset *r);
+
+/* Return range following x in ascending order, or NULL if x is the highest. */
+struct range *next_range(struct rangeset *r, struct range *x);
 
 /*
  * Initialise/destroy per-domain rangeset information.
Signature · 1 line
2.34.1
#31 lore raw

Re: [PATCH 06/15] rangeset: Move struct range and struct rangeset to headerfile

Quoted 3 lines · Henry Wang

From: Vikram Garhwal <[email protected]>

Move struct range, rangeset and removed static from first_range and next_range().

NAK, for going against what we do elsewhere (limiting exposure of internals). At least as long as the justification isn't any better than ...

Quoted 3 lines

IRQs and IOMEMs for nodes are stored as rangeset in the dynamic node addition part. While removing the nodes we need to access every IRQ and IOMEM ranges to unmap IRQ and IOMEM from the domain.

... this. You're aware of rangeset_report_ranges() and rangeset_consume_ranges(), aren't you? If neither is suitable for your purpose, can you explain what you need in addition?

Jan

author
#32 lore raw

Re: [PATCH 06/15] rangeset: Move struct range and struct rangeset to headerfile

Hi Jan,

Quoted 7 lines · Jan Beulich
Quoted 3 lines · Henry Wang

From: Vikram Garhwal <[email protected]>

Move struct range, rangeset and removed static from first_range and next_range().

NAK, for going against what we do elsewhere (limiting exposure of internals). At least as long as the justification isn't any better than ...

Quoted 6 lines
Quoted 3 lines

IRQs and IOMEMs for nodes are stored as rangeset in the dynamic node addition part. While removing the nodes we need to access every IRQ and IOMEM ranges to unmap IRQ and IOMEM from the domain.

... this. You're aware of rangeset_report_ranges() and rangeset_consume_ranges(), aren't you? If neither is suitable for your purpose, can you explain what you need in addition?

I understand your concern. I will check if I can refactor this patch using the suggested helpers. Thanks!

Kind regards,
Henry

Quoted 2 lines

Jan

author
#33 lore raw

PATCH 7/15 xen/overlay: Enable device tree overlay assignment to running domains

In reply to Henry Wang

From: Vikram Garhwal <[email protected]>

Following changes are done to enable dtbo assignment to a running vm with given domain_id:
1. Support for irq map and unmap for running domain. We store the IRQ nums for each overlay and while removing those IRQ nums are removed.
2. Support iommu assign/reassign to running domains.
3. Added "map_nodes" options to support two modes:
a. Add the nodes in Xen device tree and map the nodes to given VM b. Just add them in xen device tree without mapping.

Change device.c: handle_device() function with input domain_mapping flag.

---
 xen/arch/arm/device.c            |   8 +-
 xen/arch/arm/domain_build.c      |   2 +-
 xen/arch/arm/include/asm/setup.h |   3 +-
 xen/common/dt-overlay.c          | 218 ++++++++++++++++++++++++++-----
 xen/include/public/sysctl.h      |   4 +-
 5 files changed, 199 insertions(+), 36 deletions(-)
xen/arch/arm/device.c +5 -3
index 3e02cff008..df5035f9a8 100644
@@ -317,7 +317,8 @@ static int map_device_children(const struct dt_device_node *dev,
  *  - Map the IRQs and iomem regions to DOM0
  */
 int handle_device(struct domain *d, struct dt_device_node *dev, p2m_type_t p2mt,
-                  struct rangeset *iomem_ranges, struct rangeset *irq_ranges)
+                  struct rangeset *iomem_ranges, struct rangeset *irq_ranges,
+                  bool device_mapping)
 {
     unsigned int naddr;
     unsigned int i;
@@ -334,9 +335,10 @@ int handle_device(struct domain *d, struct dt_device_node *dev, p2m_type_t p2mt,
     struct map_range_data mr_data = {
         .d = d,
         .p2mt = p2mt,
-        .skip_mapping = !own_device ||
+        .skip_mapping = (!own_device ||
                         (is_pci_passthrough_enabled() &&
-                        (device_get_class(dev) == DEVICE_PCI_HOSTBRIDGE)),
+                        (device_get_class(dev) == DEVICE_PCI_HOSTBRIDGE))) &&
+                        !device_mapping,
         .iomem_ranges = iomem_ranges,
         .irq_ranges = irq_ranges
     };
xen/arch/arm/domain_build.c +1 -1
index 54232ed4cb..a525aed175 100644
@@ -1699,7 +1699,7 @@ static int __init handle_node(struct domain *d, struct kernel_info *kinfo,
                "WARNING: Path %s is reserved, skip the node as we may re-use the path.\n",
                path);
 
-    res = handle_device(d, node, p2mt, NULL, NULL);
+    res = handle_device(d, node, p2mt, NULL, NULL, false);
     if ( res)
         return res;
 
xen/arch/arm/include/asm/setup.h +2 -1
index d15a88d2e0..d6d8c28d02 100644
@@ -172,7 +172,8 @@ u32 device_tree_get_u32(const void *fdt, int node,
                         const char *prop_name, u32 dflt);
 
 int handle_device(struct domain *d, struct dt_device_node *dev, p2m_type_t p2mt,
-                  struct rangeset *iomem_ranges, struct rangeset *irq_ranges);
+                  struct rangeset *iomem_ranges, struct rangeset *irq_ranges,
+                  bool device_mapping);
 
 int map_device_irqs_to_domain(struct domain *d, struct dt_device_node *dev,
                               bool need_mapping, struct rangeset *irq_ranges);
xen/common/dt-overlay.c +188 -30

359 lines not on this page · raw message

xen/include/public/sysctl.h +3 -1
index 9b19679cae..f5435f8890 100644
@@ -1197,7 +1197,9 @@ struct xen_sysctl_dt_overlay {
 #define XEN_SYSCTL_DT_OVERLAY_ADD                   1
 #define XEN_SYSCTL_DT_OVERLAY_REMOVE                2
     uint8_t overlay_op;                     /* IN: Add or remove. */
-    uint8_t pad[3];                         /* IN: Must be zero. */
+    bool domain_mapping;                    /* IN: True of False. */
+    uint8_t pad[2];                         /* IN: Must be zero. */
+    uint32_t domain_id;
 };
 #endif
 
Signature · 1 line
2.34.1
#34 lore raw

Re: [PATCH 07/15] xen/overlay: Enable device tree overlay assignment to running domains

xen/include/public/sysctl.h struct xen_sysctl_dt_overlay { · line 1203

Quoted 11 lines · Henry Wang
xen/include/public/sysctl.h +3 -1
@@ -1197,7 +1197,9 @@ struct xen_sysctl_dt_overlay {
 #define XEN_SYSCTL_DT_OVERLAY_ADD                   1
 #define XEN_SYSCTL_DT_OVERLAY_REMOVE                2
     uint8_t overlay_op;                     /* IN: Add or remove. */
-    uint8_t pad[3];                         /* IN: Must be zero. */
+    bool domain_mapping;                    /* IN: True of False. */
+    uint8_t pad[2];                         /* IN: Must be zero. */
+    uint32_t domain_id;
 };

If you merely re-purposed padding fields, all would be fine without bumping the interface version. Yet you don't, albeit for an unclear
reason: Why uint32_t rather than domid_t? And on top of that - why a separate boolean when you could use e.g. DOMID_INVALID to indicate "no domain mapping"?

That said - anything taking a domain ID is certainly suspicious in a sysctl. Judging from the description you really mean this to be a domctl. Anything else will require extra justification.

Jan

author
#35 lore raw

Re: [PATCH 07/15] xen/overlay: Enable device tree overlay assignment to running domains

Hi Jan, Julien, Stefano,

Quoted 17 lines · Jan Beulich

xen/include/public/sysctl.h struct xen_sysctl_dt_overlay { · line 1203

Quoted 11 lines · Henry Wang
xen/include/public/sysctl.h +3 -1
@@ -1197,7 +1197,9 @@ struct xen_sysctl_dt_overlay {
  #define XEN_SYSCTL_DT_OVERLAY_ADD                   1
  #define XEN_SYSCTL_DT_OVERLAY_REMOVE                2
      uint8_t overlay_op;                     /* IN: Add or remove. */
-    uint8_t pad[3];                         /* IN: Must be zero. */
+    bool domain_mapping;                    /* IN: True of False. */
+    uint8_t pad[2];                         /* IN: Must be zero. */
+    uint32_t domain_id;
  };

If you merely re-purposed padding fields, all would be fine without bumping the interface version. Yet you don't, albeit for an unclear
reason: Why uint32_t rather than domid_t? And on top of that - why a separate boolean when you could use e.g. DOMID_INVALID to indicate "no domain mapping"?

I think both of your suggestion make great sense. I will follow the suggestion in v2.

Quoted 3 lines

That said - anything taking a domain ID is certainly suspicious in a sysctl. Judging from the description you really mean this to be a domctl. Anything else will require extra justification.

I also think a domctl is better. I had a look at the history of the already merged series, it looks like in the first version of merged part 1 [1], the hypercall was implemented as the domctl in the beginning but later in v2 changed to sysctl. I think this makes sense as the scope of that time is just to make Xen aware of the device tree node via Xen device tree.

However this is now a problem for the current part where the scope (and the end goal) is extended to assign the added device to Linux Dom0/DomU via device tree overlays. I am not sure which way is better, should we repurposing the sysctl to domctl or maybe add another domctl (I am worrying about the duplication because basically we need the same sysctl functionality but now with a domid in it)? What do you think?

@Stefano: Since I am not 100% if I understand the whole story behind this feature, would you mind checking if I am providing correct information above and sharing your opinions on this? Thank you very much!

[1] https://lore.kernel.org/xen-devel/[email protected]/

Kind regards,
Henry

Quoted 1 line

Jan

#36 lore raw

Re: [PATCH 07/15] xen/overlay: Enable device tree overlay assignment to running domains

Quoted 35 lines · Henry Wang

Hi Jan, Julien, Stefano,

Quoted 17 lines · Jan Beulich

xen/include/public/sysctl.h struct xen_sysctl_dt_overlay { · line 1203

Quoted 11 lines · Henry Wang
xen/include/public/sysctl.h +3 -1
@@ -1197,7 +1197,9 @@ struct xen_sysctl_dt_overlay {
  #define XEN_SYSCTL_DT_OVERLAY_ADD                   1
  #define XEN_SYSCTL_DT_OVERLAY_REMOVE                2
      uint8_t overlay_op;                     /* IN: Add or remove. */
-    uint8_t pad[3];                         /* IN: Must be zero. */
+    bool domain_mapping;                    /* IN: True of False. */
+    uint8_t pad[2];                         /* IN: Must be zero. */
+    uint32_t domain_id;
  };

If you merely re-purposed padding fields, all would be fine without bumping the interface version. Yet you don't, albeit for an unclear
reason: Why uint32_t rather than domid_t? And on top of that - why a separate boolean when you could use e.g. DOMID_INVALID to indicate "no domain mapping"?

I think both of your suggestion make great sense. I will follow the suggestion in v2.

Quoted 3 lines

That said - anything taking a domain ID is certainly suspicious in a sysctl. Judging from the description you really mean this to be a domctl. Anything else will require extra justification.

I also think a domctl is better. I had a look at the history of the already merged series, it looks like in the first version of merged part 1 [1], the hypercall was implemented as the domctl in the beginning but later in v2 changed to sysctl. I think this makes sense as the scope of that time is just to make Xen aware of the device tree node via Xen device tree.

Quoted 6 lines

However this is now a problem for the current part where the scope (and the end goal) is extended to assign the added device to Linux Dom0/DomU via device tree overlays. I am not sure which way is better, should we repurposing the sysctl to domctl or maybe add another domctl (I am worrying about the duplication because basically we need the same sysctl functionality but now with a domid in it)? What do you think?

I'm taking it that what is a sysctl right now legitimately is. Therefore folding both into domctl would at least be bending the rules of what should be sysctl and what domctl. It would need clarifying what (fake) domain such a (folded) domctl ought to operate on for the case that's currently a sysctl. That then may (or may not) be justification for such folding.

Jan

Quoted 12 lines

@Stefano: Since I am not 100% if I understand the whole story behind this feature, would you mind checking if I am providing correct information above and sharing your opinions on this? Thank you very much!

[1]
https://lore.kernel.org/xen-devel/[email protected]/

Kind regards,
Henry

Quoted 1 line

Jan

#37 lore raw

Re: [PATCH 07/15] xen/overlay: Enable device tree overlay assignment to running domains

In reply to Henry Wang
Quoted 1 line · Henry Wang

Hi Jan, Julien, Stefano,

Hi Henry,

Quoted 21 lines · Jan Beulich
Quoted 9 lines · Henry Wang
Quoted 8 lines
xen/include/public/sysctl.h +0 -0
@@ -1197,7 +1197,9 @@ struct xen_sysctl_dt_overlay {
   #define XEN_SYSCTL_DT_OVERLAY_ADD                   1
   #define XEN_SYSCTL_DT_OVERLAY_REMOVE                2

      uint8_t overlay_op;                     /* IN: Add or remove. */ -    uint8_t pad[3];                         /* IN: Must be zero. */ +    bool domain_mapping;                    /* IN: True of False. */

Quoted 8 lines
Quoted 3 lines

+    uint8_t pad[2];                         /* IN: Must be zero. */ +    uint32_t domain_id;
  };

If you merely re-purposed padding fields, all would be fine without bumping the interface version. Yet you don't, albeit for an unclear
reason: Why uint32_t rather than domid_t? And on top of that - why a separate boolean when you could use e.g. DOMID_INVALID to indicate "no domain mapping"?

I think both of your suggestion make great sense. I will follow the suggestion in v2.

Quoted 7 lines
Quoted 3 lines

That said - anything taking a domain ID is certainly suspicious in a sysctl. Judging from the description you really mean this to be a domctl. Anything else will require extra justification.

I also think a domctl is better. I had a look at the history of the already merged series, it looks like in the first version of merged part 1 [1], the hypercall was implemented as the domctl in the beginning but later in v2 changed to sysctl. I think this makes sense as the scope of that time is just to make Xen aware of the device tree node via Xen device tree.

However this is now a problem for the current part where the scope (and the end goal) is extended to assign the added device to Linux Dom0/DomU via device tree overlays. I am not sure which way is better, should we repurposing the sysctl to domctl or maybe add another domctl (I am worrying about the duplication because basically we need the same sysctl functionality but now with a domid in it)? What do you think?

I am not entirely sure this is a good idea to try to add the device in Xen and attach it to the guests at the same time. Imagine the following situation:

1) Add and attach devices
2) The domain is rebooted
3) Detach and remove devices

After step 2, you technically have a new domain. You could have also a case where this is a completely different guest. So the flow would look a little bit weird (you create the DT overlay with domain A but remove with domain B).

So, at the moment, it feels like the add/attach (resp detech/remove) operations should happen separately.

Can you clarify why you want to add devices to Xen and attach to a guest within a single hypercall?

Cheers,

Signature · 1 line
Julien Grall
author
#38 lore raw

Re: [PATCH 07/15] xen/overlay: Enable device tree overlay assignment to running domains

Hi Julien,

Quoted 40 lines · Julien Grall
Quoted 1 line · Henry Wang

Hi Jan, Julien, Stefano,

Hi Henry,

Quoted 21 lines · Jan Beulich
Quoted 9 lines · Henry Wang
Quoted 8 lines
xen/include/public/sysctl.h +0 -0
@@ -1197,7 +1197,9 @@ struct xen_sysctl_dt_overlay {
   #define XEN_SYSCTL_DT_OVERLAY_ADD                   1
   #define XEN_SYSCTL_DT_OVERLAY_REMOVE                2

      uint8_t overlay_op;                     /* IN: Add or remove. */ -    uint8_t pad[3];                         /* IN: Must be zero. */ +    bool domain_mapping;                    /* IN: True of False. */

Quoted 8 lines
Quoted 3 lines

+    uint8_t pad[2];                         /* IN: Must be zero. */ +    uint32_t domain_id;
  };

If you merely re-purposed padding fields, all would be fine without bumping the interface version. Yet you don't, albeit for an unclear
reason: Why uint32_t rather than domid_t? And on top of that - why a separate boolean when you could use e.g. DOMID_INVALID to indicate "no domain mapping"?

I think both of your suggestion make great sense. I will follow the suggestion in v2.

Quoted 7 lines
Quoted 3 lines

That said - anything taking a domain ID is certainly suspicious in a sysctl. Judging from the description you really mean this to be a domctl. Anything else will require extra justification.

I also think a domctl is better. I had a look at the history of the already merged series, it looks like in the first version of merged part 1 [1], the hypercall was implemented as the domctl in the beginning but later in v2 changed to sysctl. I think this makes sense as the scope of that time is just to make Xen aware of the device tree node via Xen device tree.

However this is now a problem for the current part where the scope (and the end goal) is extended to assign the added device to Linux Dom0/DomU via device tree overlays. I am not sure which way is better, should we repurposing the sysctl to domctl or maybe add another domctl (I am worrying about the duplication because basically we need the same sysctl functionality but now with a domid in it)? What do you think?

I am not entirely sure this is a good idea to try to add the device in Xen and attach it to the guests at the same time. Imagine the following situation:

1) Add and attach devices
2) The domain is rebooted
3) Detach and remove devices

Quoted 5 lines

After step 2, you technically have a new domain. You could have also a case where this is a completely different guest. So the flow would look a little bit weird (you create the DT overlay with domain A but remove with domain B).

So, at the moment, it feels like the add/attach (resp detech/remove) operations should happen separately.

Can you clarify why you want to add devices to Xen and attach to a guest within a single hypercall?

Sorry I don't know if there is any specific thoughts on the design of using a single hypercall to do both add devices to Xen device tree and assign the device to the guest. In fact seeing your above comments, I think separating these two functionality to two xl commands using separated hypercalls would indeed be a better idea. Thank you for the suggestion!

To make sure I understand correctly, would you mind confirming if below actions for v2 make sense to you? Thanks!
- Only use the XEN_SYSCTL_DT_OVERLAY_{ADD, REMOVE} sysctls to add/remove overlay to Xen device tree
- Introduce the xl dt-overlay attach <domid> command and respective domctls to do the device assignment for the overlay to domain.

Kind regards,
Henry

Quoted 3 lines

Cheers,

#39 lore raw

Re: [PATCH 07/15] xen/overlay: Enable device tree overlay assignment to running domains

Quoted 1 line · Henry Wang

Hi Julien,

Hi Henry,

Quoted 37 lines · Julien Grall
Quoted 36 lines · Henry Wang
Quoted 1 line

Hi Jan, Julien, Stefano,

Hi Henry,

Quoted 21 lines · Jan Beulich
Quoted 9 lines · Henry Wang
Quoted 8 lines
xen/include/public/sysctl.h +0 -0
@@ -1197,7 +1197,9 @@ struct xen_sysctl_dt_overlay {
   #define XEN_SYSCTL_DT_OVERLAY_ADD                   1
   #define XEN_SYSCTL_DT_OVERLAY_REMOVE                2

      uint8_t overlay_op;                     /* IN: Add or remove. */ -    uint8_t pad[3];                         /* IN: Must be zero. */ +    bool domain_mapping;                    /* IN: True of False. */

Quoted 8 lines
Quoted 3 lines

+    uint8_t pad[2];                         /* IN: Must be zero. */ +    uint32_t domain_id;
  };

If you merely re-purposed padding fields, all would be fine without bumping the interface version. Yet you don't, albeit for an unclear
reason: Why uint32_t rather than domid_t? And on top of that - why a separate boolean when you could use e.g. DOMID_INVALID to indicate "no domain mapping"?

I think both of your suggestion make great sense. I will follow the suggestion in v2.

Quoted 7 lines
Quoted 3 lines

That said - anything taking a domain ID is certainly suspicious in a sysctl. Judging from the description you really mean this to be a domctl. Anything else will require extra justification.

I also think a domctl is better. I had a look at the history of the already merged series, it looks like in the first version of merged part 1 [1], the hypercall was implemented as the domctl in the beginning but later in v2 changed to sysctl. I think this makes sense as the scope of that time is just to make Xen aware of the device tree node via Xen device tree.

However this is now a problem for the current part where the scope (and the end goal) is extended to assign the added device to Linux Dom0/DomU via device tree overlays. I am not sure which way is better, should we repurposing the sysctl to domctl or maybe add another domctl (I am worrying about the duplication because basically we need the same sysctl functionality but now with a domid in it)? What do you think?

I am not entirely sure this is a good idea to try to add the device in Xen and attach it to the guests at the same time. Imagine the following situation:

Quoted 7 lines
Quoted 7 lines

1) Add and attach devices
2) The domain is rebooted
3) Detach and remove devices

After step 2, you technically have a new domain. You could have also a case where this is a completely different guest. So the flow would look a little bit weird (you create the DT overlay with domain A but remove with domain B).

So, at the moment, it feels like the add/attach (resp detech/remove) operations should happen separately.

Thinking a bit more about it, there is another problem with the single hypercall appproach. The MMIOs will be mapped 1:1 to the guest. These region may clash with other part of the layout for domain created by the toolstack and dom0less (if the 1:1 option has not been enabled).

I guess for that add, it would be possible to specify the mapping in the Device-Tree. But that would not work for the removal (this may be a different domain).

On a somewhat similar topic, the number of IRQs supported by the vGIC is fixed at boot. How would that work with this patch?

Quoted 8 lines
Quoted 2 lines

Can you clarify why you want to add devices to Xen and attach to a guest within a single hypercall?

Sorry I don't know if there is any specific thoughts on the design of using a single hypercall to do both add devices to Xen device tree and assign the device to the guest. In fact seeing your above comments, I think separating these two functionality to two xl commands using separated hypercalls would indeed be a better idea. Thank you for the suggestion!

To make sure I understand correctly, would you mind confirming if below actions for v2 make sense to you? Thanks!
- Only use the XEN_SYSCTL_DT_OVERLAY_{ADD, REMOVE} sysctls to add/remove
overlay to Xen device tree

Note that this would attach the devices to dom0 first. Maybe this is why it was decided to merge the two operations? An option would be to allow the devices to be attached to no-one.

Quoted 2 lines

- Introduce the xl dt-overlay attach <domid> command and respective domctls to do the device assignment for the overlay to domain.

We already have domctls to route IRQs and map MMIOs. So do we actually need new domctls?

Cheers,

Signature · 1 line
Julien Grall
author
#40 lore raw

Re: [PATCH 07/15] xen/overlay: Enable device tree overlay assignment to running domains

Hi Julien,

Quoted 49 lines · Julien Grall
Quoted 1 line · Henry Wang

Hi Julien,

Hi Henry,

Quoted 44 lines · Julien Grall
Quoted 43 lines · Henry Wang
Quoted 1 line

Hi Jan, Julien, Stefano,

Hi Henry,

Quoted 22 lines · Jan Beulich
Quoted 10 lines · Henry Wang
Quoted 9 lines
xen/include/public/sysctl.h +0 -0
@@ -1197,7 +1197,9 @@ struct xen_sysctl_dt_overlay {
   #define XEN_SYSCTL_DT_OVERLAY_ADD                   1
   #define XEN_SYSCTL_DT_OVERLAY_REMOVE                2

      uint8_t overlay_op;                     /* IN: Add or remove. */ -    uint8_t pad[3];                         /* IN: Must be zero. */ +    bool domain_mapping;                    /* IN: True of False. */

Quoted 8 lines
Quoted 3 lines

+    uint8_t pad[2];                         /* IN: Must be zero. */ +    uint32_t domain_id;
  };

If you merely re-purposed padding fields, all would be fine without bumping the interface version. Yet you don't, albeit for an unclear
reason: Why uint32_t rather than domid_t? And on top of that - why a separate boolean when you could use e.g. DOMID_INVALID to indicate "no domain mapping"?

I think both of your suggestion make great sense. I will follow the suggestion in v2.

Quoted 7 lines
Quoted 3 lines

That said - anything taking a domain ID is certainly suspicious in a sysctl. Judging from the description you really mean this to be a domctl. Anything else will require extra justification.

I also think a domctl is better. I had a look at the history of the already merged series, it looks like in the first version of merged part 1 [1], the hypercall was implemented as the domctl in the beginning but later in v2 changed to sysctl. I think this makes sense as the scope of that time is just to make Xen aware of the device tree node via Xen device tree.

However this is now a problem for the current part where the scope (and the end goal) is extended to assign the added device to Linux Dom0/DomU via device tree overlays. I am not sure which way is better, should we repurposing the sysctl to domctl or maybe add another domctl (I am worrying about the duplication because basically we need the same sysctl functionality but now with a domid in it)? What do you think?

I am not entirely sure this is a good idea to try to add the device in Xen and attach it to the guests at the same time. Imagine the following situation:

1) Add and attach devices
2) The domain is rebooted
3) Detach and remove devices

After step 2, you technically have a new domain. You could have also a case where this is a completely different guest. So the flow would look a little bit weird (you create the DT overlay with domain A but remove with domain B).

Quoted 8 lines
Quoted 1 line
Quoted 1 line

So, at the moment, it feels like the add/attach (resp detech/remove) operations should happen separately.

Thinking a bit more about it, there is another problem with the single hypercall appproach. The MMIOs will be mapped 1:1 to the guest. These region may clash with other part of the layout for domain created by the toolstack and dom0less (if the 1:1 option has not been enabled).

I guess for that add, it would be possible to specify the mapping in the Device-Tree. But that would not work for the removal (this may be a different domain).

On a somewhat similar topic, the number of IRQs supported by the vGIC is fixed at boot. How would that work with this patch?

Seeing your comment here I now realized patch #5 is to address this issue. But I think we need to have a complete rework of the original patch to make the feature portable. We can continue the discussion in patch 5.

Quoted 9 lines
Quoted 8 lines
Quoted 2 lines

Can you clarify why you want to add devices to Xen and attach to a guest within a single hypercall?

Sorry I don't know if there is any specific thoughts on the design of using a single hypercall to do both add devices to Xen device tree and assign the device to the guest. In fact seeing your above comments, I think separating these two functionality to two xl commands using separated hypercalls would indeed be a better idea. Thank you for the suggestion!

To make sure I understand correctly, would you mind confirming if below actions for v2 make sense to you? Thanks!
- Only use the XEN_SYSCTL_DT_OVERLAY_{ADD, REMOVE} sysctls to
add/remove overlay to Xen device tree

Quoted 6 lines

Note that this would attach the devices to dom0 first. Maybe this is why it was decided to merge the two operations? An option would be to allow the devices to be attached to no-one.

Quoted 2 lines

- Introduce the xl dt-overlay attach <domid> command and respective domctls to do the device assignment for the overlay to domain.

We already have domctls to route IRQs and map MMIOs. So do we actually need new domctls?

No I don't think so, like you and Stefano said in the other thread, I think I need to split the command to different hypercalls instead of only one hypercall and reuse the existing domctl.

Kind regards,
Henry

Quoted 3 lines

Cheers,

#41 lore raw

Re: [PATCH 07/15] xen/overlay: Enable device tree overlay assignment to running domains

In reply to Henry Wang
Quoted 73 lines · Henry Wang

Hi Julien,

Quoted 58 lines · Julien Grall
Quoted 1 line · Henry Wang

Hi Jan, Julien, Stefano,

Hi Henry,

Quoted 33 lines · Jan Beulich
Quoted 9 lines · Henry Wang
Quoted 8 lines
xen/include/public/sysctl.h +0 -0
@@ -1197,7 +1197,9 @@ struct xen_sysctl_dt_overlay {
   #define XEN_SYSCTL_DT_OVERLAY_ADD                   1
   #define XEN_SYSCTL_DT_OVERLAY_REMOVE                2

      uint8_t overlay_op;                     /* IN: Add or remove. */ -    uint8_t pad[3];                         /* IN: Must be zero. */ +    bool domain_mapping;                    /* IN: True of False. */

Quoted 8 lines
Quoted 3 lines

+    uint8_t pad[2];                         /* IN: Must be zero. */ +    uint32_t domain_id;
  };

If you merely re-purposed padding fields, all would be fine without bumping the interface version. Yet you don't, albeit for an unclear
reason: Why uint32_t rather than domid_t? And on top of that - why a separate boolean when you could use e.g. DOMID_INVALID to indicate "no domain mapping"?

I think both of your suggestion make great sense. I will follow the suggestion in v2.

Quoted 3 lines

That said - anything taking a domain ID is certainly suspicious in a sysctl. Judging from the description you really mean this to be a domctl. Anything else will require extra justification.

I also think a domctl is better. I had a look at the history of the already merged series, it looks like in the first version of merged part 1 [1], the hypercall was implemented as the domctl in the beginning but later in v2 changed to sysctl. I think this makes sense as the scope of that time is just to make Xen aware of the device tree node via Xen device tree.

Quoted 6 lines

However this is now a problem for the current part where the scope (and the end goal) is extended to assign the added device to Linux Dom0/DomU via device tree overlays. I am not sure which way is better, should we repurposing the sysctl to domctl or maybe add another domctl (I am worrying about the duplication because basically we need the same sysctl functionality but now with a domid in it)? What do you think?

I am not entirely sure this is a good idea to try to add the device in Xen and attach it to the guests at the same time. Imagine the following situation:

1) Add and attach devices
2) The domain is rebooted
3) Detach and remove devices

After step 2, you technically have a new domain. You could have also a case where this is a completely different guest. So the flow would look a little bit weird (you create the DT overlay with domain A but remove with domain B).

Quoted 5 lines

So, at the moment, it feels like the add/attach (resp detech/remove) operations should happen separately.

Can you clarify why you want to add devices to Xen and attach to a guest within a single hypercall?

Sorry I don't know if there is any specific thoughts on the design of using a single hypercall to do both add devices to Xen device tree and assign the device to the guest. In fact seeing your above comments, I think separating these two functionality to two xl commands using separated hypercalls would indeed be a better idea. Thank you for the suggestion!

Quoted 6 lines

To make sure I understand correctly, would you mind confirming if below actions for v2 make sense to you? Thanks!
- Only use the XEN_SYSCTL_DT_OVERLAY_{ADD, REMOVE} sysctls to add/remove overlay to Xen device tree
- Introduce the xl dt-overlay attach <domid> command and respective domctls to do the device assignment for the overlay to domain.

I think two hypercalls is OK. The original idea was to have a single xl command to do the operation for user convenience (even that is not a hard requirement) but that can result easily in two hypercalls.

author
#42 lore raw

Re: [PATCH 07/15] xen/overlay: Enable device tree overlay assignment to running domains

Hi Stefano, Julien,

Quoted 75 lines · Stefano Stabellini
Quoted 68 lines · Henry Wang

Hi Julien,

Quoted 54 lines · Julien Grall
Quoted 1 line · Henry Wang

Hi Jan, Julien, Stefano,

Hi Henry,

Quoted 31 lines · Jan Beulich
Quoted 9 lines · Henry Wang
Quoted 8 lines
xen/include/public/sysctl.h +1 -1
@@ -1197,7 +1197,9 @@ struct xen_sysctl_dt_overlay {
   #define XEN_SYSCTL_DT_OVERLAY_ADD                   1
   #define XEN_SYSCTL_DT_OVERLAY_REMOVE                2
       uint8_t overlay_op;                     /* IN: Add or remove. */
-    uint8_t pad[3];                         /* IN: Must be zero. */
+    bool domain_mapping;                    /* IN: True of False. */
Quoted 8 lines

xen/include/public/sysctl.h struct xen_sysctl_dt_overlay { · line 1203

Quoted 3 lines
xen/include/public/sysctl.h +2 -0
+    uint8_t pad[2];                         /* IN: Must be zero. */
+    uint32_t domain_id;
   };

If you merely re-purposed padding fields, all would be fine without bumping the interface version. Yet you don't, albeit for an unclear
reason: Why uint32_t rather than domid_t? And on top of that - why a separate boolean when you could use e.g. DOMID_INVALID to indicate "no domain mapping"?

I think both of your suggestion make great sense. I will follow the suggestion in v2.

Quoted 3 lines

That said - anything taking a domain ID is certainly suspicious in a sysctl. Judging from the description you really mean this to be a domctl. Anything else will require extra justification.

I also think a domctl is better. I had a look at the history of the already merged series, it looks like in the first version of merged part 1 [1], the hypercall was implemented as the domctl in the beginning but later in v2 changed to sysctl. I think this makes sense as the scope of that time is just to make Xen aware of the device tree node via Xen device tree.

Quoted 6 lines

However this is now a problem for the current part where the scope (and the end goal) is extended to assign the added device to Linux Dom0/DomU via device tree overlays. I am not sure which way is better, should we repurposing the sysctl to domctl or maybe add another domctl (I am worrying about the duplication because basically we need the same sysctl functionality but now with a domid in it)? What do you think?

I am not entirely sure this is a good idea to try to add the device in Xen and attach it to the guests at the same time. Imagine the following situation:

1) Add and attach devices
2) The domain is rebooted
3) Detach and remove devices

After step 2, you technically have a new domain. You could have also a case where this is a completely different guest. So the flow would look a little bit weird (you create the DT overlay with domain A but remove with domain B).

Quoted 5 lines

So, at the moment, it feels like the add/attach (resp detech/remove) operations should happen separately.

Can you clarify why you want to add devices to Xen and attach to a guest within a single hypercall?

Sorry I don't know if there is any specific thoughts on the design of using a single hypercall to do both add devices to Xen device tree and assign the device to the guest. In fact seeing your above comments, I think separating these two functionality to two xl commands using separated hypercalls would indeed be a better idea. Thank you for the suggestion!

Quoted 6 lines

To make sure I understand correctly, would you mind confirming if below actions for v2 make sense to you? Thanks!
- Only use the XEN_SYSCTL_DT_OVERLAY_{ADD, REMOVE} sysctls to add/remove overlay to Xen device tree
- Introduce the xl dt-overlay attach <domid> command and respective domctls to do the device assignment for the overlay to domain.

Quoted 3 lines

I think two hypercalls is OK. The original idea was to have a single xl command to do the operation for user convenience (even that is not a hard requirement) but that can result easily in two hypercalls.

Ok, sounds good. I will break the command to two hypercalls and try to reuse the existing domctls for assign/remove IRQ/MMIO ranges.

Kind regards,
Henry

author
#43 lore raw

PATCH 8/15 tools: Add domain_id and expert mode for overlay operations

In reply to Henry Wang

From: Vikram Garhwal <[email protected]>

Add domain_id and expert mode for overlay assignment. This enables dynamic programming of nodes during runtime.

Take the opportunity to fix the name mismatch in the xl command, the command name should be "dt-overlay" instead of "dt_overlay".

---
 tools/include/libxl.h               |  8 +++++--
 tools/include/xenctrl.h             |  5 +++--
 tools/libs/ctrl/xc_dt_overlay.c     |  7 ++++--
 tools/libs/light/libxl_dt_overlay.c | 17 +++++++++++----
 tools/xl/xl_vmcontrol.c             | 34 ++++++++++++++++++++++++++---
 5 files changed, 58 insertions(+), 13 deletions(-)
tools/include/libxl.h +6 -2
index 62cb07dea6..59a3e1b37c 100644
@@ -2549,8 +2549,12 @@ libxl_device_pci *libxl_device_pci_list(libxl_ctx *ctx, uint32_t domid,
 void libxl_device_pci_list_free(libxl_device_pci* list, int num);
 
 #if defined(__arm__) || defined(__aarch64__)
-int libxl_dt_overlay(libxl_ctx *ctx, void *overlay,
-                     uint32_t overlay_size, uint8_t overlay_op);
+#define LIBXL_DT_OVERLAY_ADD                   1
+#define LIBXL_DT_OVERLAY_REMOVE                2
+
+int libxl_dt_overlay(libxl_ctx *ctx, uint32_t domain_id, void *overlay,
+                     uint32_t overlay_size, uint8_t overlay_op, bool auto_mode,
+                     bool domain_mapping);
 #endif
 
 /*
tools/include/xenctrl.h +3 -2
index 2ef8b4e054..3861d0a23f 100644
@@ -2654,8 +2654,9 @@ int xc_domain_cacheflush(xc_interface *xch, uint32_t domid,
                          xen_pfn_t start_pfn, xen_pfn_t nr_pfns);
 
 #if defined(__arm__) || defined(__aarch64__)
-int xc_dt_overlay(xc_interface *xch, void *overlay_fdt,
-                  uint32_t overlay_fdt_size, uint8_t overlay_op);
+int xc_dt_overlay(xc_interface *xch, uint32_t domain_id, void *overlay_fdt,
+                  uint32_t overlay_fdt_size, uint8_t overlay_op,
+                  bool domain_mapping);
 #endif
 
 /* Compat shims */
tools/libs/ctrl/xc_dt_overlay.c +5 -2
index c2224c4d15..a1dc549915 100644
@@ -20,15 +20,18 @@
 
 #include "xc_private.h"
 
-int xc_dt_overlay(xc_interface *xch, void *overlay_fdt,
-                  uint32_t overlay_fdt_size, uint8_t overlay_op)
+int xc_dt_overlay(xc_interface *xch, uint32_t domid, void *overlay_fdt,
+                  uint32_t overlay_fdt_size, uint8_t overlay_op,
+                  bool domain_mapping)
 {
     int err;
     struct xen_sysctl sysctl = {
         .cmd = XEN_SYSCTL_dt_overlay,
         .u.dt_overlay = {
+            .domain_id = domid,
             .overlay_op = overlay_op,
             .overlay_fdt_size = overlay_fdt_size,
+            .domain_mapping = domain_mapping,
         }
     };
 
tools/libs/light/libxl_dt_overlay.c +13 -4
index a6c709a6dc..cdb62b28cf 100644
@@ -41,8 +41,9 @@ static int check_overlay_fdt(libxl__gc *gc, void *fdt, size_t size)
     return 0;
 }
 
-int libxl_dt_overlay(libxl_ctx *ctx, void *overlay_dt, uint32_t overlay_dt_size,
-                     uint8_t overlay_op)
+int libxl_dt_overlay(libxl_ctx *ctx, uint32_t domid, void *overlay_dt,
+                     uint32_t overlay_dt_size, uint8_t overlay_op,
+                     bool auto_mode, bool domain_mapping)
 {
     int rc;
     int r;
@@ -57,10 +58,18 @@ int libxl_dt_overlay(libxl_ctx *ctx, void *overlay_dt, uint32_t overlay_dt_size,
         rc = 0;
     }
 
-    r = xc_dt_overlay(ctx->xch, overlay_dt, overlay_dt_size, overlay_op);
+    /* Check if user entered a valid domain id. */
+    rc = libxl_domain_info(CTX, NULL, domid);
+    if (rc == ERROR_DOMAIN_NOTFOUND) {
+        LOGD(ERROR, domid, "Non-existant domain.");
+        return ERROR_FAIL;
+    }
+
+    r = xc_dt_overlay(ctx->xch, domid, overlay_dt, overlay_dt_size, overlay_op,
+                      domain_mapping);
 
     if (r) {
-        LOG(ERROR, "%s: Adding/Removing overlay dtb failed.", __func__);
+        LOG(ERROR, "domain%d: Adding/Removing overlay dtb failed.", domid);
         rc = ERROR_FAIL;
     }
 
tools/xl/xl_vmcontrol.c +31 -3
index 98f6bd2e76..9674383ec3 100644
@@ -1270,21 +1270,48 @@ int main_dt_overlay(int argc, char **argv)
 {
     const char *overlay_ops = NULL;
     const char *overlay_config_file = NULL;
+    uint32_t domain_id = 0;
     void *overlay_dtb = NULL;
     int rc;
+    bool auto_mode = true;
+    bool domain_mapping = false;
     uint8_t op;
     int overlay_dtb_size = 0;
     const int overlay_add_op = 1;
     const int overlay_remove_op = 2;
 
-    if (argc < 2) {
-        help("dt_overlay");
+    if (argc < 3) {
+        help("dt-overlay");
         return EXIT_FAILURE;
     }
 
+    if (argc > 5) {
+        fprintf(stderr, "Too many arguments\n");
+        return ERROR_FAIL;
+    }
+
     overlay_ops = argv[1];
     overlay_config_file = argv[2];
 
+    if (!strcmp(argv[argc - 1], "-e"))
+        auto_mode = false;
+
+    if (argc == 4 || !auto_mode) {
+        domain_id = find_domain(argv[argc-1]);
+        domain_mapping = true;
+    }
+
+    if (argc == 5 || !auto_mode) {
+        domain_id = find_domain(argv[argc-2]);
+        domain_mapping = true;
+    }
+
+    /* User didn't prove any overlay operation. */
+    if (overlay_ops == NULL) {
+        fprintf(stderr, "No overlay operation mode provided\n");
+        return ERROR_FAIL;
+    }
+
     if (strcmp(overlay_ops, "add") == 0)
         op = overlay_add_op;
     else if (strcmp(overlay_ops, "remove") == 0)
@@ -1309,7 +1336,8 @@ int main_dt_overlay(int argc, char **argv)
         return ERROR_FAIL;
     }
 
-    rc = libxl_dt_overlay(ctx, overlay_dtb, overlay_dtb_size, op);
+    rc = libxl_dt_overlay(ctx, domain_id, overlay_dtb, overlay_dtb_size, op,
+                          auto_mode, domain_mapping);
 
     free(overlay_dtb);
 
Signature · 1 line
2.34.1
#44 lore raw

Re: [PATCH 08/15] tools: Add domain_id and expert mode for overlay operations

Quoted 7 lines · Henry Wang

From: Vikram Garhwal <[email protected]>

Add domain_id and expert mode for overlay assignment. This enables dynamic programming of nodes during runtime.

Take the opportunity to fix the name mismatch in the xl command, the command name should be "dt-overlay" instead of "dt_overlay".

I don't like much these unrelated / opportunistic changes in a patch, I'd rather have a separate patch. And in this case, if it was on a separate patch, that separated patch could gain: Fixes: 61765a07e3d8 ("tools/xl: Add new xl command overlay for device tree overlay support") and potentially backported.

Quoted 17 lines
---
 tools/include/libxl.h               |  8 +++++--
 tools/include/xenctrl.h             |  5 +++--
 tools/libs/ctrl/xc_dt_overlay.c     |  7 ++++--
 tools/libs/light/libxl_dt_overlay.c | 17 +++++++++++----
 tools/xl/xl_vmcontrol.c             | 34 ++++++++++++++++++++++++++---
 5 files changed, 58 insertions(+), 13 deletions(-)
tools/include/libxl.h +0 -0
index 62cb07dea6..59a3e1b37c 100644
@@ -2549,8 +2549,12 @@ libxl_device_pci *libxl_device_pci_list(libxl_ctx *ctx, uint32_t domid,
 void libxl_device_pci_list_free(libxl_device_pci* list, int num);

tools/include/libxl.h libxl_device_pci *libxl_device_pci_list(libxl_ctx *ctx, uint32_t domid, · line 2557

Quoted 10 lines
tools/include/libxl.h +6 -2
 
 #if defined(__arm__) || defined(__aarch64__)
-int libxl_dt_overlay(libxl_ctx *ctx, void *overlay,
-                     uint32_t overlay_size, uint8_t overlay_op);
+#define LIBXL_DT_OVERLAY_ADD                   1
+#define LIBXL_DT_OVERLAY_REMOVE                2
+
+int libxl_dt_overlay(libxl_ctx *ctx, uint32_t domain_id, void *overlay,
+                     uint32_t overlay_size, uint8_t overlay_op, bool auto_mode,
+                     bool domain_mapping);

Sorry, you cannot change the API of an existing libxl function without providing something backward compatible. We have already a few example
of this changes in libxl.h, e.g.: fded24ea8315 ("libxl: Make libxl_set_vcpuonline async") So, providing a wrapper called libxl_dt_overlay_0x041800() which call
the new function.

Quoted 8 lines
tools/include/libxl.h +0 -0
 #endif
 
 /*
tools/libs/light/libxl_dt_overlay.c +0 -0
index a6c709a6dc..cdb62b28cf 100644

tools/libs/light/libxl_dt_overlay.c int libxl_dt_overlay(libxl_ctx *ctx, void *overlay_dt, uint32_t overlay_dt_size, · line 63

Quoted 7 lines
tools/libs/light/libxl_dt_overlay.c +3 -1
@@ -57,10 +58,18 @@ int libxl_dt_overlay(libxl_ctx *ctx, void *overlay_dt, uint32_t overlay_dt_size,
         rc = 0;
     }
 
-    r = xc_dt_overlay(ctx->xch, overlay_dt, overlay_dt_size, overlay_op);
+    /* Check if user entered a valid domain id. */
+    rc = libxl_domain_info(CTX, NULL, domid);
+    if (rc == ERROR_DOMAIN_NOTFOUND) {

Why do you check specifically for "domain not found", what about other error?

tools/libs/light/libxl_dt_overlay.c int libxl_dt_overlay(libxl_ctx *ctx, void *overlay_dt, uint32_t overlay_dt_size, · line 65

Quoted 2 lines
tools/libs/light/libxl_dt_overlay.c +2 -0
+        LOGD(ERROR, domid, "Non-existant domain.");
+        return ERROR_FAIL;

Use goto out, and you can let the function return ERROR_DOMAIN_NOTFOUND if that the error, we can just propagate the rc from libxl_domain_info().

tools/libs/light/libxl_dt_overlay.c int libxl_dt_overlay(libxl_ctx *ctx, void *overlay_dt, uint32_t overlay_dt_size, · line 72

Quoted 8 lines
tools/libs/light/libxl_dt_overlay.c +5 -1
+    }
+
+    r = xc_dt_overlay(ctx->xch, domid, overlay_dt, overlay_dt_size, overlay_op,
+                      domain_mapping);
 
     if (r) {
-        LOG(ERROR, "%s: Adding/Removing overlay dtb failed.", __func__);
+        LOG(ERROR, "domain%d: Adding/Removing overlay dtb failed.", domid);

You could replace the macro by LOGD, instead of handwriting "domain%d".

Quoted 38 lines
tools/libs/light/libxl_dt_overlay.c +0 -0
         rc = ERROR_FAIL;
     }
 
tools/xl/xl_vmcontrol.c +12 -2
index 98f6bd2e76..9674383ec3 100644
@@ -1270,21 +1270,48 @@ int main_dt_overlay(int argc, char **argv)
 {
     const char *overlay_ops = NULL;
     const char *overlay_config_file = NULL;
+    uint32_t domain_id = 0;
     void *overlay_dtb = NULL;
     int rc;
+    bool auto_mode = true;
+    bool domain_mapping = false;
     uint8_t op;
     int overlay_dtb_size = 0;
     const int overlay_add_op = 1;
     const int overlay_remove_op = 2;
 
-    if (argc < 2) {
-        help("dt_overlay");
+    if (argc < 3) {
+        help("dt-overlay");
         return EXIT_FAILURE;
     }
 
+    if (argc > 5) {
+        fprintf(stderr, "Too many arguments\n");
+        return ERROR_FAIL;
+    }
+
     overlay_ops = argv[1];
     overlay_config_file = argv[2];
 
+    if (!strcmp(argv[argc - 1], "-e"))
+        auto_mode = false;

tools/xl/xl_vmcontrol.c int main_dt_overlay(int argc, char **argv) · line 1307

Quoted 10 lines
tools/xl/xl_vmcontrol.c +10 -0
+
+    if (argc == 4 || !auto_mode) {
+        domain_id = find_domain(argv[argc-1]);
+        domain_mapping = true;
+    }
+
+    if (argc == 5 || !auto_mode) {
+        domain_id = find_domain(argv[argc-2]);
+        domain_mapping = true;
+    }

Sorry, I can't review that changes, this needs a change in the help message of dt-overlay, and something in the man page. (and that argument parsing looks convoluted).

tools/xl/xl_vmcontrol.c int main_dt_overlay(int argc, char **argv) · line 1309

Quoted 2 lines
tools/xl/xl_vmcontrol.c +2 -0
+
+    /* User didn't prove any overlay operation. */

I guess you meant "provide" instead of prove. But the comment can be discarded, it doesn't explain anything useful that the error message doesn't already explain.

tools/xl/xl_vmcontrol.c int main_dt_overlay(int argc, char **argv) · line 1312

Quoted 3 lines
tools/xl/xl_vmcontrol.c +3 -0
+    if (overlay_ops == NULL) {
+        fprintf(stderr, "No overlay operation mode provided\n");
+        return ERROR_FAIL;

That should be EXIT_FAILURE instead. (and I realise that the function already return ERROR_FAIL by mistake in several places. (ERROR_FAIL is a libxl error return value of -3, which isn't really a good exit value for CLI programmes.))

Thanks,

Signature · 1 line
Anthony PERARD
author
#45 lore raw

Re: [PATCH 08/15] tools: Add domain_id and expert mode for overlay operations

Hi Anthony,

Quoted 6 lines · Anthony PERARD
Quoted 5 lines · Henry Wang

From: Vikram Garhwal <[email protected]>

Add domain_id and expert mode for overlay assignment. This enables dynamic programming of nodes during runtime.

Quoted 7 lines
Quoted 2 lines

Take the opportunity to fix the name mismatch in the xl command, the command name should be "dt-overlay" instead of "dt_overlay".

I don't like much these unrelated / opportunistic changes in a patch, I'd rather have a separate patch. And in this case, if it was on a separate patch, that separated patch could gain: Fixes: 61765a07e3d8 ("tools/xl: Add new xl command overlay for device tree overlay support") and potentially backported.

Ok. I can split this part to a separated commit.

Quoted 18 lines
Quoted 18 lines
---
  tools/include/libxl.h               |  8 +++++--
  tools/include/xenctrl.h             |  5 +++--
  tools/libs/ctrl/xc_dt_overlay.c     |  7 ++++--
  tools/libs/light/libxl_dt_overlay.c | 17 +++++++++++----
  tools/xl/xl_vmcontrol.c             | 34 ++++++++++++++++++++++++++---
  5 files changed, 58 insertions(+), 13 deletions(-)
tools/include/libxl.h +0 -0
index 62cb07dea6..59a3e1b37c 100644
@@ -2549,8 +2549,12 @@ libxl_device_pci *libxl_device_pci_list(libxl_ctx *ctx, uint32_t domid,
  void libxl_device_pci_list_free(libxl_device_pci* list, int num);
 
Quoted 14 lines

tools/include/libxl.h libxl_device_pci *libxl_device_pci_list(libxl_ctx *ctx, uint32_t domid, · line 2557

Quoted 9 lines
tools/include/libxl.h +6 -2
  #if defined(__arm__) || defined(__aarch64__)
-int libxl_dt_overlay(libxl_ctx *ctx, void *overlay,
-                     uint32_t overlay_size, uint8_t overlay_op);
+#define LIBXL_DT_OVERLAY_ADD                   1
+#define LIBXL_DT_OVERLAY_REMOVE                2
+
+int libxl_dt_overlay(libxl_ctx *ctx, uint32_t domain_id, void *overlay,
+                     uint32_t overlay_size, uint8_t overlay_op, bool auto_mode,
+                     bool domain_mapping);

Sorry, you cannot change the API of an existing libxl function without providing something backward compatible. We have already a few example
of this changes in libxl.h, e.g.: fded24ea8315 ("libxl: Make libxl_set_vcpuonline async") So, providing a wrapper called libxl_dt_overlay_0x041800() which call
the new function.

Ok, I will add an wrapper.

Quoted 11 lines
Quoted 11 lines
tools/include/libxl.h +0 -0
  #endif
 
  /*
tools/libs/light/libxl_dt_overlay.c +0 -0
index a6c709a6dc..cdb62b28cf 100644
@@ -57,10 +58,18 @@ int libxl_dt_overlay(libxl_ctx *ctx, void *overlay_dt, uint32_t overlay_dt_size,
          rc = 0;
      }
 
Quoted 6 lines

tools/libs/light/libxl_dt_overlay.c int libxl_dt_overlay(libxl_ctx *ctx, void *overlay_dt, uint32_t overlay_dt_size, · line 63

Quoted 4 lines
tools/libs/light/libxl_dt_overlay.c +3 -1
-    r = xc_dt_overlay(ctx->xch, overlay_dt, overlay_dt_size, overlay_op);
+    /* Check if user entered a valid domain id. */
+    rc = libxl_domain_info(CTX, NULL, domid);
+    if (rc == ERROR_DOMAIN_NOTFOUND) {

Why do you check specifically for "domain not found", what about other error?

I agree this is indeed very confusing...I will rewrite this part properly in the next version.

Quoted 5 lines

tools/libs/light/libxl_dt_overlay.c int libxl_dt_overlay(libxl_ctx *ctx, void *overlay_dt, uint32_t overlay_dt_size, · line 65

Quoted 2 lines
tools/libs/light/libxl_dt_overlay.c +2 -0
+        LOGD(ERROR, domid, "Non-existant domain.");
+        return ERROR_FAIL;

Use goto out, and you can let the function return ERROR_DOMAIN_NOTFOUND if that the error, we can just propagate the rc from libxl_domain_info().

Sure, will do the suggested way.

Quoted 9 lines

tools/libs/light/libxl_dt_overlay.c int libxl_dt_overlay(libxl_ctx *ctx, void *overlay_dt, uint32_t overlay_dt_size, · line 72

Quoted 8 lines
tools/libs/light/libxl_dt_overlay.c +5 -1
+    }
+
+    r = xc_dt_overlay(ctx->xch, domid, overlay_dt, overlay_dt_size, overlay_op,
+                      domain_mapping);
 
      if (r) {
-        LOG(ERROR, "%s: Adding/Removing overlay dtb failed.", __func__);
+        LOG(ERROR, "domain%d: Adding/Removing overlay dtb failed.", domid);

You could replace the macro by LOGD, instead of handwriting "domain%d".

Great suggestion. I will use LOGD.

Quoted 36 lines
Quoted 36 lines
tools/libs/light/libxl_dt_overlay.c +0 -0
          rc = ERROR_FAIL;
      }
 
tools/xl/xl_vmcontrol.c +10 -2
index 98f6bd2e76..9674383ec3 100644
@@ -1270,21 +1270,48 @@ int main_dt_overlay(int argc, char **argv)
  {
      const char *overlay_ops = NULL;
      const char *overlay_config_file = NULL;
+    uint32_t domain_id = 0;
      void *overlay_dtb = NULL;
      int rc;
+    bool auto_mode = true;
+    bool domain_mapping = false;
      uint8_t op;
      int overlay_dtb_size = 0;
      const int overlay_add_op = 1;
      const int overlay_remove_op = 2;
 
-    if (argc < 2) {
-        help("dt_overlay");
+    if (argc < 3) {
+        help("dt-overlay");
          return EXIT_FAILURE;
      }
 
+    if (argc > 5) {
+        fprintf(stderr, "Too many arguments\n");
+        return ERROR_FAIL;
+    }
+
      overlay_ops = argv[1];
      overlay_config_file = argv[2];
 
Quoted 15 lines

tools/xl/xl_vmcontrol.c int main_dt_overlay(int argc, char **argv) · line 1307

Quoted 12 lines
tools/xl/xl_vmcontrol.c +12 -0
+    if (!strcmp(argv[argc - 1], "-e"))
+        auto_mode = false;
+
+    if (argc == 4 || !auto_mode) {
+        domain_id = find_domain(argv[argc-1]);
+        domain_mapping = true;
+    }
+
+    if (argc == 5 || !auto_mode) {
+        domain_id = find_domain(argv[argc-2]);
+        domain_mapping = true;
+    }

Sorry, I can't review that changes, this needs a change in the help message of dt-overlay, and something in the man page. (and that argument parsing looks convoluted).

I will rework this part to see if I can make it better.

Quoted 5 lines

tools/xl/xl_vmcontrol.c int main_dt_overlay(int argc, char **argv) · line 1309

Quoted 2 lines
tools/xl/xl_vmcontrol.c +2 -0
+
+    /* User didn't prove any overlay operation. */

I guess you meant "provide" instead of prove. But the comment can be discarded, it doesn't explain anything useful that the error message doesn't already explain.

I think your comment is correct, I will just drop it.

Quoted 7 lines

tools/xl/xl_vmcontrol.c int main_dt_overlay(int argc, char **argv) · line 1312

Quoted 3 lines
tools/xl/xl_vmcontrol.c +3 -0
+    if (overlay_ops == NULL) {
+        fprintf(stderr, "No overlay operation mode provided\n");
+        return ERROR_FAIL;

That should be EXIT_FAILURE instead. (and I realise that the function already return ERROR_FAIL by mistake in several places. (ERROR_FAIL is a libxl error return value of -3, which isn't really a good exit value for CLI programmes.))

Thanks. Will use EXIT_FAILURE.

Kind regards,
Henry

Quoted 3 lines

Thanks,

author
#46 lore raw

PATCH 9/15 tools/libs/light: Modify dtbo to domU linux dtbo format

In reply to Henry Wang

From: Vikram Garhwal <[email protected]>

Add support for modifying dtbo to make it suitable for DomU Linux. This is done for '-e expert' mode.

---
 tools/libs/light/libxl_dt_overlay.c | 73 +++++++++++++++++++++++++++++
 1 file changed, 73 insertions(+)
tools/libs/light/libxl_dt_overlay.c +73 -0
index cdb62b28cf..eaf11a0f9c 100644
@@ -17,6 +17,7 @@
 #include "libxl_internal.h"
 #include <libfdt.h>
 #include <xenctrl.h>
+#include <xen/device_tree_defs.h>
 
 static int check_overlay_fdt(libxl__gc *gc, void *fdt, size_t size)
 {
@@ -41,6 +42,69 @@ static int check_overlay_fdt(libxl__gc *gc, void *fdt, size_t size)
     return 0;
 }
 
+static int modify_overlay_for_domU(libxl__gc *gc, void *overlay_dt_domU,
+                                   size_t size)
+{
+    int rc = 0;
+    int virtual_interrupt_parent = GUEST_PHANDLE_GIC;
+    const struct fdt_property *fdt_prop_node = NULL;
+    int overlay;
+    int prop_len = 0;
+    int subnode = 0;
+    int fragment;
+    const char *prop_name;
+    const char *target_path = "/";
+
+    fdt_for_each_subnode(fragment, overlay_dt_domU, 0) {
+        prop_name = fdt_getprop(overlay_dt_domU, fragment, "target-path",
+                                &prop_len);
+        if (prop_name == NULL) {
+            LOG(ERROR, "target-path property not found\n");
+            rc = ERROR_FAIL;
+            goto err;
+        }
+
+        /* Change target path for domU dtb. */
+        rc = fdt_setprop_string(overlay_dt_domU, fragment, "target-path",
+                                target_path);
+        if (rc) {
+            LOG(ERROR, "Setting interrupt parent property failed for %s\n",
+                prop_name);
+            goto err;
+        }
+
+        overlay = fdt_subnode_offset(overlay_dt_domU, fragment, "__overlay__");
+
+        fdt_for_each_subnode(subnode, overlay_dt_domU, overlay)
+        {
+            const char *node_name = fdt_get_name(overlay_dt_domU, subnode,
+                                                 NULL);
+
+            fdt_prop_node = fdt_getprop(overlay_dt_domU, subnode,
+                                        "interrupt-parent", &prop_len);
+            if (fdt_prop_node == NULL) {
+                LOG(DETAIL, "%s property not found for %s. Skip to next node\n",
+                    "interrupt-parent", node_name);
+                continue;
+            }
+
+            rc = fdt_setprop_inplace_u32(overlay_dt_domU, subnode,
+                                         "interrupt-parent",
+                                         virtual_interrupt_parent);
+            if (rc) {
+                LOG(ERROR, "Setting interrupt parent property failed for %s\n",
+                    "interrupt-parent");
+                goto err;
+            }
+        }
+    }
+
+return 0;
+
+err:
+    return rc;
+}
+
 int libxl_dt_overlay(libxl_ctx *ctx, uint32_t domid, void *overlay_dt,
                      uint32_t overlay_dt_size, uint8_t overlay_op,
                      bool auto_mode, bool domain_mapping)
@@ -73,6 +137,15 @@ int libxl_dt_overlay(libxl_ctx *ctx, uint32_t domid, void *overlay_dt,
         rc = ERROR_FAIL;
     }
 
+    /*
+     * auto_mode doesn't apply to dom0 as dom0 can get the physical
+     * description of the hardware.
+     */
+    if (domid && auto_mode) {
+        if (overlay_op == LIBXL_DT_OVERLAY_ADD)
+            rc = modify_overlay_for_domU(gc, overlay_dt, overlay_dt_size);
+    }
+
 out:
     GC_FREE;
     return rc;
Signature · 1 line
2.34.1
#47 lore raw

Re: [PATCH 09/15] tools/libs/light: Modify dtbo to domU linux dtbo format

Quoted 16 lines · Henry Wang
tools/libs/light/libxl_dt_overlay.c +8 -0
index cdb62b28cf..eaf11a0f9c 100644
@@ -41,6 +42,69 @@ static int check_overlay_fdt(libxl__gc *gc, void *fdt, size_t size)
     return 0;
 }
 
+static int modify_overlay_for_domU(libxl__gc *gc, void *overlay_dt_domU,
+                                   size_t size)
+{
+    int rc = 0;
+    int virtual_interrupt_parent = GUEST_PHANDLE_GIC;
+    const struct fdt_property *fdt_prop_node = NULL;
+    int overlay;
+    int prop_len = 0;

tools/libs/light/libxl_dt_overlay.c static int check_overlay_fdt(libxl__gc *gc, void *fdt, size_t size) · line 62

Quoted 10 lines
tools/libs/light/libxl_dt_overlay.c +10 -0
+    int subnode = 0;
+    int fragment;
+    const char *prop_name;
+    const char *target_path = "/";
+
+    fdt_for_each_subnode(fragment, overlay_dt_domU, 0) {
+        prop_name = fdt_getprop(overlay_dt_domU, fragment, "target-path",
+                                &prop_len);
+        if (prop_name == NULL) {
+            LOG(ERROR, "target-path property not found\n");

LOG* macros already takes care of adding \n, no need to add an extra one.

tools/libs/light/libxl_dt_overlay.c static int check_overlay_fdt(libxl__gc *gc, void *fdt, size_t size) · line 68

Quoted 6 lines
tools/libs/light/libxl_dt_overlay.c +6 -0
+            rc = ERROR_FAIL;
+            goto err;
+        }
+
+        /* Change target path for domU dtb. */
+        rc = fdt_setprop_string(overlay_dt_domU, fragment, "target-path",

fdt_setprop_string() isn't a libxl function, store the return value in a variable named r instead.

Quoted 9 lines
tools/libs/light/libxl_dt_overlay.c +9 -0
+                                target_path);
+        if (rc) {
+            LOG(ERROR, "Setting interrupt parent property failed for %s\n",
+                prop_name);
+            goto err;
+        }
+
+        overlay = fdt_subnode_offset(overlay_dt_domU, fragment, "__overlay__");
+

tools/libs/light/libxl_dt_overlay.c static int check_overlay_fdt(libxl__gc *gc, void *fdt, size_t size) · line 87

Quoted 10 lines
tools/libs/light/libxl_dt_overlay.c +10 -0
+        fdt_for_each_subnode(subnode, overlay_dt_domU, overlay)
+        {
+            const char *node_name = fdt_get_name(overlay_dt_domU, subnode,
+                                                 NULL);
+
+            fdt_prop_node = fdt_getprop(overlay_dt_domU, subnode,
+                                        "interrupt-parent", &prop_len);
+            if (fdt_prop_node == NULL) {
+                LOG(DETAIL, "%s property not found for %s. Skip to next node\n",
+                    "interrupt-parent", node_name);

Why do you have "interrupt-parent" in a separate argument? Do you meant to do something like

    const char *some_name = "interrupt-parent";

and use that in the 4 different places that this string is used? (Using a variable mean that we (or the compiler) can make sure that they are all spelled correctly.

Quoted 5 lines
tools/libs/light/libxl_dt_overlay.c +5 -0
+                continue;
+            }
+
+            rc = fdt_setprop_inplace_u32(overlay_dt_domU, subnode,
+                                         "interrupt-parent",

tools/libs/light/libxl_dt_overlay.c static int check_overlay_fdt(libxl__gc *gc, void *fdt, size_t size) · line 102

Quoted 10 lines
tools/libs/light/libxl_dt_overlay.c +10 -0
+                                         virtual_interrupt_parent);
+            if (rc) {
+                LOG(ERROR, "Setting interrupt parent property failed for %s\n",
+                    "interrupt-parent");
+                goto err;
+            }
+        }
+    }
+
+return 0;

Missed indentation.

tools/libs/light/libxl_dt_overlay.c static int check_overlay_fdt(libxl__gc *gc, void *fdt, size_t size) · line 105

Quoted 3 lines
tools/libs/light/libxl_dt_overlay.c +3 -0
+
+err:
+    return rc;

A few things, looks like rc is always going to be ERROR_FAIL here, unless you find an libxl_error code that better describe the error, so you could forgo the rc variable.

Also, if you don't need to clean up anything in the function or have a generic error message, you could simply "return " instead of using the "goto" style.

Quoted 6 lines
tools/libs/light/libxl_dt_overlay.c +2 -0
+}
+
 int libxl_dt_overlay(libxl_ctx *ctx, uint32_t domid, void *overlay_dt,
                      uint32_t overlay_dt_size, uint8_t overlay_op,
                      bool auto_mode, bool domain_mapping)

tools/libs/light/libxl_dt_overlay.c int libxl_dt_overlay(libxl_ctx *ctx, uint32_t domid, void *overlay_dt, · line 145

Quoted 9 lines
tools/libs/light/libxl_dt_overlay.c +6 -0
@@ -73,6 +137,15 @@ int libxl_dt_overlay(libxl_ctx *ctx, uint32_t domid, void *overlay_dt,
         rc = ERROR_FAIL;
     }
 
+    /*
+     * auto_mode doesn't apply to dom0 as dom0 can get the physical
+     * description of the hardware.
+     */
+    if (domid && auto_mode) {
+        if (overlay_op == LIBXL_DT_OVERLAY_ADD)

Shouldn't libxl complain if the operation is different?

tools/libs/light/libxl_dt_overlay.c int libxl_dt_overlay(libxl_ctx *ctx, uint32_t domid, void *overlay_dt, · line 151

Quoted 6 lines
tools/libs/light/libxl_dt_overlay.c +3 -0
+            rc = modify_overlay_for_domU(gc, overlay_dt, overlay_dt_size);
+    }
+
 out:
     GC_FREE;
     return rc;

Thanks,

Signature · 1 line
Anthony PERARD
author
#48 lore raw

Re: [PATCH 09/15] tools/libs/light: Modify dtbo to domU linux dtbo format

Hi Anthony,

Quoted 21 lines · Anthony PERARD
Quoted 20 lines · Henry Wang
tools/libs/light/libxl_dt_overlay.c +12 -0
index cdb62b28cf..eaf11a0f9c 100644
@@ -41,6 +42,69 @@ static int check_overlay_fdt(libxl__gc *gc, void *fdt, size_t size)
      return 0;
  }
 
+static int modify_overlay_for_domU(libxl__gc *gc, void *overlay_dt_domU,
+                                   size_t size)
+{
+    int rc = 0;
+    int virtual_interrupt_parent = GUEST_PHANDLE_GIC;
+    const struct fdt_property *fdt_prop_node = NULL;
+    int overlay;
+    int prop_len = 0;
+    int subnode = 0;
+    int fragment;
+    const char *prop_name;
+    const char *target_path = "/";
Quoted 8 lines

tools/libs/light/libxl_dt_overlay.c static int check_overlay_fdt(libxl__gc *gc, void *fdt, size_t size) · line 62

Quoted 6 lines
tools/libs/light/libxl_dt_overlay.c +6 -0
+
+    fdt_for_each_subnode(fragment, overlay_dt_domU, 0) {
+        prop_name = fdt_getprop(overlay_dt_domU, fragment, "target-path",
+                                &prop_len);
+        if (prop_name == NULL) {
+            LOG(ERROR, "target-path property not found\n");

LOG* macros already takes care of adding \n, no need to add an extra one.

Sure, I will remove the "\n".

Quoted 9 lines

tools/libs/light/libxl_dt_overlay.c static int check_overlay_fdt(libxl__gc *gc, void *fdt, size_t size) · line 68

Quoted 7 lines
tools/libs/light/libxl_dt_overlay.c +6 -0
+            rc = ERROR_FAIL;
+            goto err;
+        }
+
+        /* Change target path for domU dtb. */
+        rc = fdt_setprop_string(overlay_dt_domU, fragment, "target-path",

fdt_setprop_string() isn't a libxl function, store the return value in a variable named r instead.'

Thanks for spotting this. Will change it to r.

Quoted 21 lines
Quoted 9 lines
tools/libs/light/libxl_dt_overlay.c +9 -0
+                                target_path);
+        if (rc) {
+            LOG(ERROR, "Setting interrupt parent property failed for %s\n",
+                prop_name);
+            goto err;
+        }
+
+        overlay = fdt_subnode_offset(overlay_dt_domU, fragment, "__overlay__");
+

tools/libs/light/libxl_dt_overlay.c static int check_overlay_fdt(libxl__gc *gc, void *fdt, size_t size) · line 87

Quoted 10 lines
tools/libs/light/libxl_dt_overlay.c +10 -0
+        fdt_for_each_subnode(subnode, overlay_dt_domU, overlay)
+        {
+            const char *node_name = fdt_get_name(overlay_dt_domU, subnode,
+                                                 NULL);
+
+            fdt_prop_node = fdt_getprop(overlay_dt_domU, subnode,
+                                        "interrupt-parent", &prop_len);
+            if (fdt_prop_node == NULL) {
+                LOG(DETAIL, "%s property not found for %s. Skip to next node\n",
+                    "interrupt-parent", node_name);

Why do you have "interrupt-parent" in a separate argument? Do you meant to do something like

Quoted 4 lines
     const char *some_name = "interrupt-parent";

and use that in the 4 different places that this string is used? (Using a variable mean that we (or the compiler) can make sure that they are all spelled correctly.

Great suggestion! I will do this way.

Quoted 8 lines
Quoted 8 lines
tools/libs/light/libxl_dt_overlay.c +8 -0
+                continue;
+            }
+
+            rc = fdt_setprop_inplace_u32(overlay_dt_domU, subnode,
+                                         "interrupt-parent",
+                                         virtual_interrupt_parent);
+            if (rc) {
+                LOG(ERROR, "Setting interrupt parent property failed for %s\n",
Quoted 8 lines

tools/libs/light/libxl_dt_overlay.c static int check_overlay_fdt(libxl__gc *gc, void *fdt, size_t size) · line 102

Quoted 7 lines
tools/libs/light/libxl_dt_overlay.c +7 -0
+                    "interrupt-parent");
+                goto err;
+            }
+        }
+    }
+
+return 0;

Missed indentation.

Will correct it.

Quoted 10 lines

tools/libs/light/libxl_dt_overlay.c static int check_overlay_fdt(libxl__gc *gc, void *fdt, size_t size) · line 105

Quoted 3 lines
tools/libs/light/libxl_dt_overlay.c +3 -0
+
+err:
+    return rc;

A few things, looks like rc is always going to be ERROR_FAIL here, unless you find an libxl_error code that better describe the error, so you could forgo the rc variable.

Also, if you don't need to clean up anything in the function or have a generic error message, you could simply "return " instead of using the "goto" style.

Sure, I will simply use return because I don't really think there is anything to be cleaned up.

Quoted 9 lines
Quoted 9 lines
tools/libs/light/libxl_dt_overlay.c +2 -0
+}
+
  int libxl_dt_overlay(libxl_ctx *ctx, uint32_t domid, void *overlay_dt,
                       uint32_t overlay_dt_size, uint8_t overlay_op,
                       bool auto_mode, bool domain_mapping)
@@ -73,6 +137,15 @@ int libxl_dt_overlay(libxl_ctx *ctx, uint32_t domid, void *overlay_dt,
          rc = ERROR_FAIL;
      }
 
Quoted 7 lines

tools/libs/light/libxl_dt_overlay.c int libxl_dt_overlay(libxl_ctx *ctx, uint32_t domid, void *overlay_dt, · line 145

Quoted 6 lines
tools/libs/light/libxl_dt_overlay.c +6 -0
+    /*
+     * auto_mode doesn't apply to dom0 as dom0 can get the physical
+     * description of the hardware.
+     */
+    if (domid && auto_mode) {
+        if (overlay_op == LIBXL_DT_OVERLAY_ADD)

Shouldn't libxl complain if the operation is different?

I will add corresponding error handling code here. Thanks!

Kind regards,
Henry

Quoted 8 lines

tools/libs/light/libxl_dt_overlay.c int libxl_dt_overlay(libxl_ctx *ctx, uint32_t domid, void *overlay_dt, · line 151

Quoted 6 lines
tools/libs/light/libxl_dt_overlay.c +3 -0
+            rc = modify_overlay_for_domU(gc, overlay_dt, overlay_dt_size);
+    }
+
  out:
      GC_FREE;
      return rc;

Thanks,

author
#49 lore raw

PATCH 10/15 tools/xl: Share overlay with domU

In reply to Henry Wang

From: Vikram Garhwal <[email protected]>

Add domid to enable assigning the node to a running VMs.

Add expert mode option to share the overlay dtbo with domU. In this mode dom0 communicates with domid domain to share the overlay dtbo. This is done via xenstore.

---
 tools/xl/Makefile       |   2 +-
 tools/xl/xl_cmdtable.c  |   2 +-
 tools/xl/xl_vmcontrol.c | 343 +++++++++++++++++++++++++++++++++++++++-
 3 files changed, 342 insertions(+), 5 deletions(-)
tools/xl/Makefile +1 -1
index d742e96a5b..122dfb9346 100644
@@ -32,7 +32,7 @@ $(XL_OBJS): CFLAGS += -include $(XEN_ROOT)/tools/config.h # libxl_json.h needs i
 all: xl
 
 xl: $(XL_OBJS)
-	$(CC) $(LDFLAGS) -o $@ $(XL_OBJS) $(LDLIBS_libxenutil) $(LDLIBS_libxenlight) $(LDLIBS_libxentoollog) -lyajl $(APPEND_LDFLAGS)
+	$(CC) $(LDFLAGS) -o $@ $(XL_OBJS) $(LDLIBS_libxenutil) $(LDLIBS_libxenlight) $(LDLIBS_libxentoollog) $(LDLIBS_libxenstore) $(LDLIBS_libxengnttab) -lyajl $(APPEND_LDFLAGS)
 
 .PHONY: install
 install: all
tools/xl/xl_cmdtable.c +1 -1
index 62bdb2aeaa..2531e8517b 100644
@@ -635,7 +635,7 @@ const struct cmd_spec cmd_table[] = {
     { "dt-overlay",
       &main_dt_overlay, 0, 1,
       "Add/Remove a device tree overlay",
-      "add/remove <.dtbo>"
+      "add/remove <.dtbo> domain_id -e[expert_mode]",
       "-h print this help\n"
     },
 #endif
tools/xl/xl_vmcontrol.c +340 -3

369 lines not on this page · raw message

Signature · 1 line
2.34.1
author
#50 lore raw

PATCH 11/15 tools/helpers: Add get_overlay

In reply to Henry Wang

From: Vikram Garhwal <[email protected]>

This user level application copies the overlay dtbo shared by dom0 while doing overlay node assignment operation. It uses xenstore to communicate with dom0. More information on the protocol is writtien in docs/misc/overlay.txt file.

---
 tools/helpers/Makefile      |   8 +
 tools/helpers/get_overlay.c | 393 ++++++++++++++++++++++++++++++++++++
 2 files changed, 401 insertions(+)
 create mode 100644 tools/helpers/get_overlay.c
tools/helpers/Makefile +8 -0
index 09590eb5b6..dfe17ef269 100644
@@ -12,6 +12,7 @@ TARGETS += init-xenstore-domain
 endif
 ifeq ($(CONFIG_ARM),y)
 TARGETS += init-dom0less
+TARGETS += get_overlay
 endif
 endif
 
@@ -39,6 +40,9 @@ $(INIT_DOM0LESS_OBJS): CFLAGS += $(CFLAGS_libxenctrl)
 $(INIT_DOM0LESS_OBJS): CFLAGS += $(CFLAGS_libxenevtchn)
 init-dom0less: LDLIBS += $(call xenlibs-ldlibs,ctrl evtchn toollog store light guest foreignmemory)
 
+SHARE_OVERLAY_OBJS = get_overlay.o
+$(SHARE_OVERLAY_OBJS): CFLAGS += $(CFLAGS_libxengnttab) $(CFLAGS_libxenstore) $(CFLAGS_libxenctrl)
+
 .PHONY: all
 all: $(TARGETS)
 
@@ -51,6 +55,10 @@ init-xenstore-domain: $(INIT_XENSTORE_DOMAIN_OBJS)
 init-dom0less: $(INIT_DOM0LESS_OBJS)
 	$(CC) $(LDFLAGS) -o $@ $(INIT_DOM0LESS_OBJS) $(LDLIBS) $(APPEND_LDFLAGS)
 
+get_overlay: $(SHARE_OVERLAY_OBJS)
+	$(CC) $(LDFLAGS) -o $@ $< $(LDLIBS_libxenvchan) $(LDLIBS_libxenstore) $(LDLIBS_libxenctrl) $(LDLIBS_libxengnttab) $(APPEND_LDFLAGS)
+
+
 .PHONY: install
 install: all
 	$(INSTALL_DIR) $(DESTDIR)$(LIBEXEC_BIN)
tools/helpers/get_overlay.c new file +393 -0

393 lines not on this page · raw message

Signature · 1 line
2.34.1
#51 lore raw

Re: [PATCH 11/15] tools/helpers: Add get_overlay

Quoted 6 lines · Henry Wang

From: Vikram Garhwal <[email protected]>

This user level application copies the overlay dtbo shared by dom0 while doing overlay node assignment operation. It uses xenstore to communicate with dom0. More information on the protocol is writtien in docs/misc/overlay.txt file.

Quoted 8 lines
---
 tools/helpers/Makefile      |   8 +
 tools/helpers/get_overlay.c | 393 ++++++++++++++++++++++++++++++++++++
 2 files changed, 401 insertions(+)
 create mode 100644 tools/helpers/get_overlay.c

As mentioned before on various occasions - new files preferably use dashes as separators in preference to underscores. You not doing so is particularly puzzling seeing ...

tools/helpers/Makefile TARGETS += init-xenstore-domain · line 15

Quoted 7 lines
tools/helpers/Makefile +1 -0
@@ -12,6 +12,7 @@ TARGETS += init-xenstore-domain
 endif
 ifeq ($(CONFIG_ARM),y)
 TARGETS += init-dom0less
+TARGETS += get_overlay

... patch context here (demonstrating a whopping 3 dashes used in similar cases).

Jan

author
#52 lore raw

Re: [PATCH 11/15] tools/helpers: Add get_overlay

Hi Jan,

Quoted 19 lines · Jan Beulich
Quoted 6 lines · Henry Wang

From: Vikram Garhwal <[email protected]>

This user level application copies the overlay dtbo shared by dom0 while doing overlay node assignment operation. It uses xenstore to communicate with dom0. More information on the protocol is writtien in docs/misc/overlay.txt file.

Quoted 8 lines
---
  tools/helpers/Makefile      |   8 +
  tools/helpers/get_overlay.c | 393 ++++++++++++++++++++++++++++++++++++
  2 files changed, 401 insertions(+)

create mode 100644 tools/helpers/get_overlay.c

As mentioned before on various occasions - new files preferably use dashes as separators in preference to underscores. You not doing so is particularly puzzling seeing ...

Quoted 9 lines

tools/helpers/Makefile TARGETS += init-xenstore-domain · line 15

Quoted 7 lines
tools/helpers/Makefile +1 -0
@@ -12,6 +12,7 @@ TARGETS += init-xenstore-domain
  endif
  ifeq ($(CONFIG_ARM),y)
  TARGETS += init-dom0less
+TARGETS += get_overlay

... patch context here (demonstrating a whopping 3 dashes used in similar cases).

I am not very sure why Vikram used "_" in the original patch. However I agree you are correct. Since I am currently doing the follow up of this series, I will use "-" in v2 as suggested. Thanks.

Kind regards,
Henry

Quoted 2 lines

Jan

#53 lore raw

Re: [PATCH 11/15] tools/helpers: Add get_overlay

Quoted 32 lines · Henry Wang

Hi Jan,

Quoted 19 lines · Jan Beulich
Quoted 6 lines · Henry Wang

From: Vikram Garhwal <[email protected]>

This user level application copies the overlay dtbo shared by dom0 while doing overlay node assignment operation. It uses xenstore to communicate with dom0. More information on the protocol is writtien in docs/misc/overlay.txt file.

Quoted 8 lines

---
  tools/helpers/Makefile      |   8 + tools/helpers/get_overlay.c | 393 ++++++++++++++++++++++++++++++++++++

  2 files changed, 401 insertions(+)

  create mode 100644 tools/helpers/get_overlay.c

As mentioned before on various occasions - new files preferably use dashes as separators in preference to underscores. You not doing so is particularly puzzling seeing ...

Quoted 9 lines
Quoted 7 lines
tools/helpers/Makefile +0 -0
@@ -12,6 +12,7 @@ TARGETS += init-xenstore-domain

  endif
  ifeq ($(CONFIG_ARM),y)
  TARGETS += init-dom0less
+TARGETS += get_overlay

... patch context here (demonstrating a whopping 3 dashes used in similar cases).

Quoted 1 line

I am not very sure why Vikram used "_" in the original patch. However I agree you are correct. Since I am currently doing the follow up of this series, I will use "-" in v2 as suggested. Thanks.

Please also add tools/helpers/get-overlay to .gitignore

author
#54 lore raw

Re: [PATCH 11/15] tools/helpers: Add get_overlay

Hi Stewart,

Quoted 32 lines · Stewart Hildebrand
Quoted 31 lines · Henry Wang

Hi Jan,

Quoted 28 lines · Jan Beulich
Quoted 6 lines · Henry Wang

From: Vikram Garhwal <[email protected]>

This user level application copies the overlay dtbo shared by dom0 while doing overlay node assignment operation. It uses xenstore to communicate with dom0. More information on the protocol is writtien in docs/misc/overlay.txt file.

Quoted 8 lines
---
   tools/helpers/Makefile      |   8 +
   tools/helpers/get_overlay.c | 393 ++++++++++++++++++++++++++++++++++++
   2 files changed, 401 insertions(+)

  create mode 100644 tools/helpers/get_overlay.c

As mentioned before on various occasions - new files preferably use dashes as separators in preference to underscores. You not doing so is particularly puzzling seeing ...

tools/helpers/Makefile TARGETS += init-xenstore-domain · line 15

Quoted 7 lines
tools/helpers/Makefile +1 -0
@@ -12,6 +12,7 @@ TARGETS += init-xenstore-domain
   endif
   ifeq ($(CONFIG_ARM),y)
   TARGETS += init-dom0less
+TARGETS += get_overlay

... patch context here (demonstrating a whopping 3 dashes used in similar cases).

Quoted 2 lines
Quoted 1 line

I am not very sure why Vikram used "_" in the original patch. However I agree you are correct. Since I am currently doing the follow up of this series, I will use "-" in v2 as suggested. Thanks.

Please also add tools/helpers/get-overlay to .gitignore

Thanks for the reminder! Yes sure I will add it.

Kind regards,
Henry

author
#55 lore raw

PATCH 12/15 get_overlay: remove domU overlay

In reply to Henry Wang

From: Vikram Garhwal <[email protected]>

Retrieve 4 new parameters from xenstore: overlay name, type, whether it is a partial overlay and operation. Operation can be "add" or "remove".

Add correspond to existing mode of operation. Remove introduces support for removing an overlay from a domU.

---
 tools/helpers/get_overlay.c | 132 +++++++++++++++++++++++++++++++++---
 1 file changed, 123 insertions(+), 9 deletions(-)
tools/helpers/get_overlay.c +123 -9
index ca3007570e..daa697ca04 100644
@@ -66,6 +66,33 @@ retry_transaction:
     snprintf(ref, sizeof(ref), "%s", "not_ready");
     snprintf(buf, sizeof(buf), "%s/sender-status", xs_base);
 
+    if (!xs_write(xs, xs_trans, buf, ref, strlen(ref)))
+        goto fail_xs_transaction;
+    if (!xs_set_permissions(xs, xs_trans, buf, perms, 2))
+        goto fail_xs_transaction;
+
+    /* Create overlay-name node. */
+    snprintf(ref, sizeof(ref), "%s", "overlay_node");
+    snprintf(buf, sizeof(buf), "%s/overlay-name", xs_base);
+
+    if (!xs_write(xs, xs_trans, buf, ref, strlen(ref)))
+        goto fail_xs_transaction;
+    if (!xs_set_permissions(xs, xs_trans, buf, perms, 2))
+        goto fail_xs_transaction;
+
+    /* Create overlay-type node. */
+    snprintf(ref, sizeof(ref), "%s", "type");
+    snprintf(buf, sizeof(buf), "%s/overlay-type", xs_base);
+
+    if (!xs_write(xs, xs_trans, buf, ref, strlen(ref)))
+        goto fail_xs_transaction;
+    if (!xs_set_permissions(xs, xs_trans, buf, perms, 2))
+        goto fail_xs_transaction;
+
+    /* Create overlay-partial node. */
+    snprintf(ref, sizeof(ref), "%d", 0);
+    snprintf(buf, sizeof(buf), "%s/overlay-partial", xs_base);
+
     if (!xs_write(xs, xs_trans, buf, ref, strlen(ref)))
         goto fail_xs_transaction;
     if (!xs_set_permissions(xs, xs_trans, buf, perms, 2))
@@ -174,7 +201,7 @@ static bool wait_for_status(struct xs_handle *xs, int fd, char *status_path,
 }
 
 static bool write_page_ref(struct xs_handle *xs, uint32_t *page_ref,
-                           uint32_t num_pages, char *path)
+                           uint32_t num_pages, const char *path)
 {
     xs_transaction_t xs_trans = XBT_NULL;
     char buf[128];
@@ -249,12 +276,69 @@ retry_transaction:
     return true;
 }
 
+static char *get_overlay_ops(struct xs_handle *xs, const char *xs_path)
+{
+    char buf[128];
+    char *ref = NULL;
+    unsigned int len;
+
+    snprintf(buf, sizeof(buf), "%s/overlay-operation", xs_path);
+
+    ref = xs_read(xs, XBT_NULL, buf, &len);
+
+    return ref;
+}
+static char *get_overlay_name(struct xs_handle *xs, const char *xs_path)
+{
+    char buf[128];
+    char *ref = NULL;
+    unsigned int len;
+
+    snprintf(buf, sizeof(buf), "%s/overlay-name", xs_path);
+
+    ref = xs_read(xs, XBT_NULL, buf, &len);
+
+    return ref;
+}
+
+static char *get_overlay_type(struct xs_handle *xs, const char *xs_path)
+{
+    char buf[128];
+    char *ref = NULL;
+    unsigned int len;
+
+    snprintf(buf, sizeof(buf), "%s/overlay-type", xs_path);
+
+    ref = xs_read(xs, XBT_NULL, buf, &len);
+
+    return ref;
+}
+
+static bool get_overlay_partial(struct xs_handle *xs, const char *xs_path)
+{
+    char buf[128];
+    char *ref = NULL;
+    unsigned int len;
+
+    snprintf(buf, sizeof(buf), "%s/overlay-partial", xs_path);
+
+    ref = xs_read(xs, XBT_NULL, buf, &len);
+
+    if (ref) {
+        bool is_partial = atoi(ref);
+        free(ref);
+        return is_partial;
+    }
+
+    return false;
+}
+
 int main(int argc, char **argv)
 {
     void *buffer = NULL;
     int domain ;
     uint32_t *page_refs = NULL;
-    FILE *fptr;
+    FILE *fptr = NULL;
     int dtbo_size = 0;
     const char *path = "data/overlay";
     char receiver_status_path[64] = { };
@@ -263,7 +347,11 @@ int main(int argc, char **argv)
     int rc = 0;
     int fd = 0;
     uint32_t num_pages = 0;
-    xengntshr_handle *gntshr;
+    xengntshr_handle *gntshr = NULL;
+    char *overlay_ops = NULL;
+    char *name = NULL;
+    char *type = NULL;
+    bool is_partial = false;
 
     if (argc < 2) {
        fprintf(stderr,"Please enter domain_id.\n");
@@ -357,16 +445,33 @@ int main(int argc, char **argv)
         goto out;
     }
 
-    if ((fptr = fopen("overlay.dtbo","wb")) == NULL) {
-        fprintf(stderr,"Error! opening file");
+    overlay_ops = get_overlay_ops(xs, path);
+    name = get_overlay_name(xs, path);
+    type = get_overlay_type(xs, path);
+    is_partial = get_overlay_partial(xs, path);
+
+    if (overlay_ops == NULL || name == NULL || type == NULL)
         goto out;
-    }
 
-    printf("Writing to file overlay.dtbo.\n");
+    printf("%s %s %s", overlay_ops, name, type);
+    if (is_partial)
+        printf(" %d", is_partial);
+
+    printf("\n");
 
-    fwrite(buffer, dtbo_size, 1, fptr);
+    if (!strcmp(overlay_ops, "add")) {
 
-    printf("Done writing to file overlay.dtbo \n");
+        if ((fptr = fopen("overlay.dtbo","wb")) == NULL) {
+            fprintf(stderr,"Error! opening file");
+            goto out;
+        }
+
+        printf("Writing to file overlay.dtbo.\n");
+
+        fwrite(buffer, dtbo_size, 1, fptr);
+
+        printf("Done writing to file overlay.dtbo \n");
+    }
 
 out:
     if (fptr)
@@ -375,6 +480,15 @@ out:
     if (page_refs)
         free(page_refs);
 
+    if (overlay_ops)
+        free(overlay_ops);
+
+    if (name)
+        free(name);
+
+    if (type)
+        free(type);
+
     if (xs) {
         close(fd);
 
Signature · 1 line
2.34.1
author
#56 lore raw

PATCH 13/15 xl/overlay: add remove operation to xenstore

In reply to Henry Wang

From: Vikram Garhwal <[email protected]>

Add 3 new command line parameters to the xl overlay command: overlay name, type and partial. Pass these paramters to the domU via xenstore.

Also introduce support for "operation" in xenstore: it can be "add" or "remove". In case of "remove", the overlay is to be removed from the domU device tree.

---
 tools/xl/xl_vmcontrol.c | 184 +++++++++++++++++++++++++++++++++++++---
 1 file changed, 173 insertions(+), 11 deletions(-)
tools/xl/xl_vmcontrol.c +173 -11
index 2bf76dd389..ddd6e9e370 100644
@@ -1466,8 +1466,123 @@ static uint32_t get_num_pages(struct xs_handle *xs, const char *xs_path)
     return num_pages;
 }
 
+static bool write_overlay_operation(struct xs_handle *xs, char *operation,
+                               char *path)
+{
+    xs_transaction_t xs_trans = XBT_NULL;
+    char buf[128];
+    char ref[64];
+
+retry_transaction:
+    xs_trans = xs_transaction_start(xs);
+    if (!xs_trans)
+        return false;
+
+    snprintf(ref, sizeof(ref), "%s", operation);
+    snprintf(buf, sizeof(buf), "%s/overlay-operation", path);
+
+    if (!xs_write(xs, xs_trans, buf, ref, strlen(ref)))
+        return false;
+
+    if (!xs_transaction_end(xs, xs_trans, 0)) {
+        if (errno == EAGAIN)
+            goto retry_transaction;
+        else
+            return false;
+    }
+
+    return true;
+}
+
+static bool write_overlay_name(struct xs_handle *xs, char *name,
+                               char *path)
+{
+    xs_transaction_t xs_trans = XBT_NULL;
+    char buf[128];
+    char ref[64];
+
+retry_transaction:
+    xs_trans = xs_transaction_start(xs);
+    if (!xs_trans)
+        return false;
+
+    snprintf(ref, sizeof(ref), "%s", name);
+    snprintf(buf, sizeof(buf), "%s/overlay-name", path);
+
+    if (!xs_write(xs, xs_trans, buf, ref, strlen(ref)))
+        return false;
+
+    if (!xs_transaction_end(xs, xs_trans, 0)) {
+        if (errno == EAGAIN)
+            goto retry_transaction;
+        else
+            return false;
+    }
+
+    return true;
+}
+
+static bool write_overlay_type(struct xs_handle *xs, char *type,
+                               char *path)
+{
+    xs_transaction_t xs_trans = XBT_NULL;
+    char buf[128];
+    char ref[64];
+
+retry_transaction:
+    xs_trans = xs_transaction_start(xs);
+    if (!xs_trans)
+        return false;
+
+    snprintf(ref, sizeof(ref), "%s", type);
+    snprintf(buf, sizeof(buf), "%s/overlay-type", path);
+
+    if (!xs_write(xs, xs_trans, buf, ref, strlen(ref)))
+        return false;
+
+    if (!xs_transaction_end(xs, xs_trans, 0)) {
+        if (errno == EAGAIN)
+            goto retry_transaction;
+        else
+            return false;
+    }
+
+    return true;
+}
+
+static bool write_overlay_partial(struct xs_handle *xs, bool is_partial,
+                                  char *path)
+{
+    xs_transaction_t xs_trans = XBT_NULL;
+    char buf[128];
+    char ref[4];
+
+retry_transaction:
+    xs_trans = xs_transaction_start(xs);
+    if (!xs_trans)
+        return false;
+
+    snprintf(ref, sizeof(ref), "%d", is_partial);
+    snprintf(buf, sizeof(buf), "%s/overlay-partial", path);
+
+    if (!xs_write(xs, xs_trans, buf, ref, strlen(ref)))
+        return false;
+
+    if (!xs_transaction_end(xs, xs_trans, 0)) {
+        if (errno == EAGAIN)
+            goto retry_transaction;
+        else
+            return false;
+    }
+
+    return true;
+}
+
+
 static int share_overlay_with_domu(void *overlay_dt_domU, int overlay_dt_size,
-                                   int domain_id)
+                                   int domain_id, char *overlay_ops,
+                                   char *overlay_name,
+                                   char *overlay_type, bool is_overlay_partial)
 {
     struct xs_handle *xs = NULL;
     char *path = NULL;
@@ -1574,6 +1689,34 @@ static int share_overlay_with_domu(void *overlay_dt_domU, int overlay_dt_size,
         goto out;
     }
 
+    /* write overlay ops */
+    if (!write_overlay_operation(xs, overlay_ops, path)) {
+        err = ERROR_FAIL;
+        fprintf(stderr,"Writing overlay_ops ready failed\n");
+        goto out;
+    }
+
+    /* Write the overlay-name. */
+    if (!write_overlay_name(xs, overlay_name, path)) {
+        err = ERROR_FAIL;
+        fprintf(stderr,"Writing overlay_name ready failed\n");
+        goto out;
+    }
+
+    /* Write the overlay-type. */
+    if (!write_overlay_type(xs, overlay_type, path)) {
+        err = ERROR_FAIL;
+        fprintf(stderr,"Writing overlay_type ready failed\n");
+        goto out;
+    }
+
+    /* Write the overlay-partial. */
+    if (!write_overlay_partial(xs, is_overlay_partial, path)) {
+        err = ERROR_FAIL;
+        fprintf(stderr,"Writing overlay_partial ready failed\n");
+        goto out;
+    }
+
     /* Write the status "done". */
     if (!write_status(xs, "done", sender_status_path)) {
         fprintf(stderr,"Writing status DONE failed\n");
@@ -1611,13 +1754,16 @@ int main_dt_overlay(int argc, char **argv)
     int overlay_dtb_size = 0;
     const int overlay_add_op = 1;
     const int overlay_remove_op = 2;
+    char *overlay_name = "overlay";
+    char *overlay_type = "normal";
+    bool is_overlay_partial = false;
 
     if (argc < 3) {
         help("dt-overlay");
         return EXIT_FAILURE;
     }
 
-    if (argc > 5) {
+    if (argc > 7) {
         fprintf(stderr, "Too many arguments\n");
         return ERROR_FAIL;
     }
@@ -1625,17 +1771,22 @@ int main_dt_overlay(int argc, char **argv)
     overlay_ops = argv[1];
     overlay_config_file = argv[2];
 
-    if (!strcmp(argv[argc - 1], "-e"))
-        auto_mode = false;
-
-    if (argc == 4 || !auto_mode) {
+    if (argc == 4 ) {
         domain_id = find_domain(argv[argc-1]);
         domain_mapping = true;
-    }
-
-    if (argc == 5 || !auto_mode) {
-        domain_id = find_domain(argv[argc-2]);
+    } else if (argc == 5 && !strcmp(argv[4], "-e")) {
+        domain_id = find_domain(argv[3]);
+        auto_mode = false;
         domain_mapping = true;
+    } else if (argc == 7) {
+        domain_id = find_domain(argv[3]);
+        domain_mapping = true;
+        overlay_name = argv[4];
+        overlay_type = argv[5];
+        is_overlay_partial = atoi(argv[6]);
+    } else {
+        fprintf(stderr, "Invalid arguments\n");
+        return ERROR_FAIL;
     }
 
     /* User didn't prove any overlay operation. */
@@ -1678,7 +1829,18 @@ int main_dt_overlay(int argc, char **argv)
 
     if (domain_id && auto_mode && (op == LIBXL_DT_OVERLAY_ADD))
         rc = share_overlay_with_domu(overlay_dtb, overlay_dtb_size,
-                                     domain_id);
+                                     domain_id, "add", overlay_name,
+                                     overlay_type, is_overlay_partial);
+
+    if (rc) {
+        free(overlay_dtb);
+        return rc;
+    }
+
+    if (domain_id && auto_mode && (op == LIBXL_DT_OVERLAY_REMOVE))
+        rc = share_overlay_with_domu(overlay_dtb, overlay_dtb_size,
+                                     domain_id, "remove", overlay_name,
+                                     overlay_type, is_overlay_partial);
 
     free(overlay_dtb);
     return rc;
Signature · 1 line
2.34.1
author
#57 lore raw

PATCH 14/15 add a domU script to fetch overlays and applying them to linux

In reply to Henry Wang

From: Vikram Garhwal <[email protected]>

Introduce a shell script that runs in the background and calls get_overlay to retrive overlays and add them (or remove them) to Linux device tree (running as a domU).

---
 tools/helpers/Makefile       |  2 +-
 tools/helpers/get_overlay.sh | 81 ++++++++++++++++++++++++++++++++++++
 2 files changed, 82 insertions(+), 1 deletion(-)
 create mode 100755 tools/helpers/get_overlay.sh
tools/helpers/Makefile +1 -1
index dfe17ef269..2d0558aeb8 100644
@@ -58,7 +58,6 @@ init-dom0less: $(INIT_DOM0LESS_OBJS)
 get_overlay: $(SHARE_OVERLAY_OBJS)
 	$(CC) $(LDFLAGS) -o $@ $< $(LDLIBS_libxenvchan) $(LDLIBS_libxenstore) $(LDLIBS_libxenctrl) $(LDLIBS_libxengnttab) $(APPEND_LDFLAGS)
 
-
 .PHONY: install
 install: all
 	$(INSTALL_DIR) $(DESTDIR)$(LIBEXEC_BIN)
@@ -67,6 +66,7 @@ install: all
 .PHONY: uninstall
 uninstall:
 	for i in $(TARGETS); do rm -f $(DESTDIR)$(LIBEXEC_BIN)/$$i; done
+	$(RM) $(DESTDIR)$(LIBEXEC_BIN)/get_overlay.sh
 
 .PHONY: clean
 clean:
tools/helpers/get_overlay.sh new file +81 -0

81 lines not on this page · raw message

Signature · 1 line
2.34.1
#58 lore raw

Re: [PATCH 14/15] add a domU script to fetch overlays and applying them to linux

Quoted 6 lines · Henry Wang

From: Vikram Garhwal <[email protected]>

Introduce a shell script that runs in the background and calls get_overlay to retrive overlays and add them (or remove them) to Linux device tree (running as a domU).

Quoted 8 lines
---
 tools/helpers/Makefile       |  2 +-
 tools/helpers/get_overlay.sh | 81 ++++++++++++++++++++++++++++++++++++
 2 files changed, 82 insertions(+), 1 deletion(-)
 create mode 100755 tools/helpers/get_overlay.sh

Besides the same naming issue as in the earlier patch, the script also looks very Linux-ish. Yet ...

Quoted 11 lines
tools/helpers/Makefile +0 -1
@@ -58,7 +58,6 @@ init-dom0less: $(INIT_DOM0LESS_OBJS)
 get_overlay: $(SHARE_OVERLAY_OBJS)
 	$(CC) $(LDFLAGS) -o $@ $< $(LDLIBS_libxenvchan) $(LDLIBS_libxenstore) $(LDLIBS_libxenctrl) $(LDLIBS_libxengnttab) $(APPEND_LDFLAGS)
 
-
 .PHONY: install
 install: all
 	$(INSTALL_DIR) $(DESTDIR)$(LIBEXEC_BIN)

tools/helpers/Makefile install: all · line 72

Quoted 7 lines
tools/helpers/Makefile +1 -0
@@ -67,6 +66,7 @@ install: all
 .PHONY: uninstall
 uninstall:
 	for i in $(TARGETS); do rm -f $(DESTDIR)$(LIBEXEC_BIN)/$$i; done
+	$(RM) $(DESTDIR)$(LIBEXEC_BIN)/get_overlay.sh
 
 .PHONY: clean
 clean:

... you touching only the uninstall target, it's not even clear to me how (and under what conditions) the script is going to make it into $(DESTDIR)$(LIBEXEC_BIN)/. Did you mean to add to $(TARGETS), perhaps, alongside the earlier added get-overlay binary?

Jan

author
#59 lore raw

Re: [PATCH 14/15] add a domU script to fetch overlays and applying them to linux

Hi Jan,

Quoted 15 lines · Jan Beulich
Quoted 6 lines · Henry Wang

From: Vikram Garhwal <[email protected]>

Introduce a shell script that runs in the background and calls get_overlay to retrive overlays and add them (or remove them) to Linux device tree (running as a domU).

Quoted 8 lines
---
  tools/helpers/Makefile       |  2 +-
  tools/helpers/get_overlay.sh | 81 ++++++++++++++++++++++++++++++++++++
  2 files changed, 82 insertions(+), 1 deletion(-)

create mode 100755 tools/helpers/get_overlay.sh

Quoted 2 lines

Besides the same naming issue as in the earlier patch, the script also looks very Linux-ish. Yet ...

I will fix the naming issue in v2. Would you mind elaborating a bit more about the "Linux-ish" concern? I guess this is because the original use case is on Linux, should I do anything about this?

Quoted 16 lines
Quoted 16 lines
tools/helpers/Makefile +1 -1
@@ -58,7 +58,6 @@ init-dom0less: $(INIT_DOM0LESS_OBJS)
  get_overlay: $(SHARE_OVERLAY_OBJS)
  	$(CC) $(LDFLAGS) -o $@ $< $(LDLIBS_libxenvchan) $(LDLIBS_libxenstore) $(LDLIBS_libxenctrl) $(LDLIBS_libxengnttab) $(APPEND_LDFLAGS)
 
-
  .PHONY: install
  install: all
  	$(INSTALL_DIR) $(DESTDIR)$(LIBEXEC_BIN)
@@ -67,6 +66,7 @@ install: all
  .PHONY: uninstall
  uninstall:
  	for i in $(TARGETS); do rm -f $(DESTDIR)$(LIBEXEC_BIN)/$$i; done
+	$(RM) $(DESTDIR)$(LIBEXEC_BIN)/get_overlay.sh
 
Quoted 6 lines

tools/helpers/Makefile install: all · line 72

Quoted 2 lines
tools/helpers/Makefile +0 -0
  .PHONY: clean
  clean:

... you touching only the uninstall target, it's not even clear to me how (and under what conditions) the script is going to make it into $(DESTDIR)$(LIBEXEC_BIN)/. Did you mean to add to $(TARGETS), perhaps, alongside the earlier added get-overlay binary?

You are right, I think the get-overlay binary and this script should be installed if DTB overlay is supported. Checking the code, I found LIBXL_HAVE_DT_OVERLAY which can indicate if we have this feature supported in libxl. Do you think it is a good idea to use it to install these two files in Makefile? Thanks.

Kind regards,
Henry

Quoted 2 lines

Jan

#60 lore raw

Re: [PATCH 14/15] add a domU script to fetch overlays and applying them to linux

Quoted 16 lines · Henry Wang
Quoted 15 lines · Jan Beulich
Quoted 14 lines · Henry Wang

From: Vikram Garhwal <[email protected]>

Introduce a shell script that runs in the background and calls get_overlay to retrive overlays and add them (or remove them) to Linux device tree (running as a domU).

---
  tools/helpers/Makefile       |  2 +-
  tools/helpers/get_overlay.sh | 81 ++++++++++++++++++++++++++++++++++++
  2 files changed, 82 insertions(+), 1 deletion(-)

create mode 100755 tools/helpers/get_overlay.sh

Quoted 6 lines
Quoted 2 lines

Besides the same naming issue as in the earlier patch, the script also looks very Linux-ish. Yet ...

I will fix the naming issue in v2. Would you mind elaborating a bit more about the "Linux-ish" concern? I guess this is because the original use case is on Linux, should I do anything about this?

Well, the script won't work on other than Linux, will it? Therefore ...

Quoted 16 lines
Quoted 16 lines
Quoted 16 lines
tools/helpers/Makefile +1 -1
@@ -58,7 +58,6 @@ init-dom0less: $(INIT_DOM0LESS_OBJS)
  get_overlay: $(SHARE_OVERLAY_OBJS)
  	$(CC) $(LDFLAGS) -o $@ $< $(LDLIBS_libxenvchan) $(LDLIBS_libxenstore) $(LDLIBS_libxenctrl) $(LDLIBS_libxengnttab) $(APPEND_LDFLAGS)
 
-
  .PHONY: install
  install: all
  	$(INSTALL_DIR) $(DESTDIR)$(LIBEXEC_BIN)
@@ -67,6 +66,7 @@ install: all
  .PHONY: uninstall
  uninstall:
  	for i in $(TARGETS); do rm -f $(DESTDIR)$(LIBEXEC_BIN)/$$i; done
+	$(RM) $(DESTDIR)$(LIBEXEC_BIN)/get_overlay.sh
 
Quoted 6 lines
Quoted 6 lines

tools/helpers/Makefile install: all · line 72

Quoted 2 lines
tools/helpers/Makefile +0 -0
  .PHONY: clean
  clean:

... you touching only the uninstall target, it's not even clear to me how (and under what conditions) the script is going to make it into $(DESTDIR)$(LIBEXEC_BIN)/. Did you mean to add to $(TARGETS), perhaps, alongside the earlier added get-overlay binary?

... it first of needs to become clear under what conditions it is actually going to be installed.

Quoted 5 lines

You are right, I think the get-overlay binary and this script should be installed if DTB overlay is supported. Checking the code, I found LIBXL_HAVE_DT_OVERLAY which can indicate if we have this feature supported in libxl. Do you think it is a good idea to use it to install these two files in Makefile? Thanks.

Counter question: If it's not going to be installed, how are people going to make use of it? If the script is intended for manual use only, I think that would want saying in the description. Yet then I couldn't see why the uninstall goal would need modifying.

As to LIBXL_HAVE_DT_OVERLAY - that's not accessible from a Makefile, I guess?

Jan

author
#61 lore raw

Re: [PATCH 14/15] add a domU script to fetch overlays and applying them to linux

Hi Jan,

Quoted 54 lines · Jan Beulich
Quoted 16 lines · Henry Wang
Quoted 15 lines · Jan Beulich
Quoted 14 lines · Henry Wang

From: Vikram Garhwal <[email protected]>

Introduce a shell script that runs in the background and calls get_overlay to retrive overlays and add them (or remove them) to Linux device tree (running as a domU).

---
   tools/helpers/Makefile       |  2 +-
   tools/helpers/get_overlay.sh | 81 ++++++++++++++++++++++++++++++++++++
   2 files changed, 82 insertions(+), 1 deletion(-)

create mode 100755 tools/helpers/get_overlay.sh

Quoted 5 lines
Quoted 2 lines

Besides the same naming issue as in the earlier patch, the script also looks very Linux-ish. Yet ...

I will fix the naming issue in v2. Would you mind elaborating a bit more about the "Linux-ish" concern? I guess this is because the original use case is on Linux, should I do anything about this?

Well, the script won't work on other than Linux, will it? Therefore ...

Quoted 16 lines
Quoted 16 lines
Quoted 16 lines
tools/helpers/Makefile +1 -1
@@ -58,7 +58,6 @@ init-dom0less: $(INIT_DOM0LESS_OBJS)
   get_overlay: $(SHARE_OVERLAY_OBJS)
   	$(CC) $(LDFLAGS) -o $@ $< $(LDLIBS_libxenvchan) $(LDLIBS_libxenstore) $(LDLIBS_libxenctrl) $(LDLIBS_libxengnttab) $(APPEND_LDFLAGS)
 
-
   .PHONY: install
   install: all
   	$(INSTALL_DIR) $(DESTDIR)$(LIBEXEC_BIN)
@@ -67,6 +66,7 @@ install: all
   .PHONY: uninstall
   uninstall:
   	for i in $(TARGETS); do rm -f $(DESTDIR)$(LIBEXEC_BIN)/$$i; done
+	$(RM) $(DESTDIR)$(LIBEXEC_BIN)/get_overlay.sh
 
Quoted 6 lines
Quoted 6 lines

tools/helpers/Makefile install: all · line 72

Quoted 2 lines
tools/helpers/Makefile +0 -0
   .PHONY: clean
   clean:

... you touching only the uninstall target, it's not even clear to me how (and under what conditions) the script is going to make it into $(DESTDIR)$(LIBEXEC_BIN)/. Did you mean to add to $(TARGETS), perhaps, alongside the earlier added get-overlay binary?

... it first of needs to become clear under what conditions it is actually going to be installed.

Quoted 5 lines

You are right, I think the get-overlay binary and this script should be installed if DTB overlay is supported. Checking the code, I found LIBXL_HAVE_DT_OVERLAY which can indicate if we have this feature supported in libxl. Do you think it is a good idea to use it to install these two files in Makefile? Thanks.

Quoted 4 lines

Counter question: If it's not going to be installed, how are people going to make use of it? If the script is intended for manual use only, I think that would want saying in the description. Yet then I couldn't see why the uninstall goal would need modifying.

Checking the code again, I feel like this is a mistake actually. I think this script should be installed together with the get-overlay application as the script actually calls get-overlay. The uninstall goal should remain untouched. I will fix this in v2.

Quoted 2 lines

As to LIBXL_HAVE_DT_OVERLAY - that's not accessible from a Makefile, I guess?

Yes.

Kind regards,
Henry

Quoted 2 lines

Jan

author
#62 lore raw

PATCH 15/15 docs: add device tree overlay documentation

In reply to Henry Wang

From: Vikram Garhwal <[email protected]>

---
 docs/misc/arm/overlay.txt | 172 ++++++++++++++++++++++++++++++++++++++
 1 file changed, 172 insertions(+)
 create mode 100644 docs/misc/arm/overlay.txt
docs/misc/arm/overlay.txt new file +172 -0
new file mode 100644
index 0000000000..0351f82a19
@@ -0,0 +1,172 @@
+# Device Tree Overlays support in Xen
+
+Xen now supports dynamic device assignment to running domains i.e.
+adding/removing nodes (using .dtbo) in Xen device tree, and assigning
+them to a running domain with given $domid.
+
+Xen supports two modes of operation: normal mode and expert mode for
+assigning nodes to domU. More on this below.
+
+Dynamic node assignment works in following ways:
+
+1. Xen tools check the dtbo given and parse all other user provided arguments
+2. Xen tools pass the dtbo to Xen hypervisor via hypercall.
+3. Xen hypervisor applies the dtbo to Xen device tree and assign the
+   dbto's node resources to the user-provided $domid.
+4. For normal mode,  Xen tools share the modified dtbo with domU. domU needs
+   to run get_overlay.sh to get the dtbo from dom0 and apply the
+   overlay. get_overlay.sh uses get_overlay application for data transfer
+   between dom0 and domU.
+
+
+# Overlay Sharing protocol based on Xenstore
+
+The overlay sharing protocol with domU works in the following ways:
+
+1. get_overlay creates Xenstore path data/overlay and creates the
+   following nodes under data/overlay path:
+       a. receiver-status
+       b. sender-status
+       c. overlay-size
+       d. overlay-name
+       e. overlay-type
+       f. overlay-partial
+   and write "waiting" on receiver-status and "not_ready" to sender_status.
+
+2. libxl waits for "waiting" status on receiver-status, then writes
+   "overlay-size" with dtbo size and "ready" on "sender-status".
+
+3. get_overlay waits for "sender-status" to "ready", then allocates the
+   pages, next it shares the pages with dom0 (the page-ref num) by creating
+   page-ref node under /data/overlay and finally writes "page-refs" to
+   "receiver_status".
+
+4. libxl waits for "receiver-status" to become "page-refs" and copies
+   the data to buffer allocated with page_refs. libxl also writes the
+   "overlay-name", "overlay-type", and "overlay-partial" nodes with
+   user-provided information.  Lastly, libxl writes "done" to
+   "sender-status".
+
+6. get_overlay waits for "sender-status" to be "done".
+
+7. get_overlay copies the data and writes it to file.
+
+8. Finally, get_overlay unshares the pages with dom0.
+
+Note: get_overlay application needs two drivers xen_gntdev and xen_gntalloc in
+Linux. These can be loaded using modprobe xen_gntalloc and modprobe xen_gntdev.
+
+
+# Examples
+
+Here are a few examples on how to use it.
+
+## Dom0 device add
+
+For assigning a device tree overlay to dom0, enter the following:
+
+    (dom0) xl dt-overlay add overlay.dtbo 0
+
+This will allocate the devices mentioned in overlay.dtbo to Xen device
+tree and will assign these devices to dom0.
+
+Next, if the user wants to add the same device tree overlay to dom0
+Linux, execute the following:
+
+    (dom0) mkdir -p /sys/kernel/config/device-tree/overlays/new_overlay
+    (dom0) cat overlay.dtbo > /sys/kernel/config/device-tree/overlays/new_overlay/dtbo
+
+Finally if needed, the relevant Linux kernel drive can be loaded using:
+
+    (dom0) modprobe module_name.ko
+
+
+## Dom0 device remove
+
+For removing the device from dom0, do the following:
+
+    (dom0) xl dt-overlay remove overlay.dtbo
+
+NOTE: The user is expected to unload any Linux kernel modules which
+might be accessing the devices in overlay.dtbo. Removing devices without
+unloading the modules might result in a crash.
+
+The following is an incorrect sequence:
+
+    (dom0) xl dt-overlay add overlay.dtbo 0
+    (dom0) xl dt-overlay remove overlay.dtbo
+
+The last command only removed the nodes from the Xen dtb but it did not
+deassigning irq/iommus from dom0. This will result in unhandled
+behavior. The correct sequence is to deassign the nodes from dom0:
+
+    (dom0) xl dt-overlay remove overlay.dtbo 0
+
+
+## DomU device add/remove
+
+There are two modes supported for domU use cases: expert mode and normal
+mode.
+
+
+### Expert mode
+
+All the nodes in dtbo will be assigned to a domain; the user will need
+to prepare the dtb for the domU. User will also need to modprobe the
+relevant drivers.
+
+Example for domU device add:
+
+    (dom0) xl dt-overlay add overlay.dtbo $domid
+    (dom0) xl console $domid  # to access $domid console
+
+Next, if the user needs to modify/prepare the overlay.dtbo suitable for
+the domU:
+
+    (domU) mkdir -p /sys/kernel/config/device-tree/overlays/new_overlay
+    (domU) cat overlay_linux.dtbo > /sys/kernel/config/device-tree/overlays/new_overlay/dtbo
+
+Finally, if needed, the relevant Linux kernel drive can be probed:
+
+    (domU) modprobe module_name.ko
+
+Example for domU overlay remove:
+
+    (dom0) xl dt-overlay remove overlay.dtbo $domid
+
+
+### Normal mode
+
+Libxl modifies the dtbo suitable for the domU. Currently, it does basic
+modifications like updating "target-path" and "interrupt-parent" to make
+them compatible with the domU device tree. Please note that this might
+not work for nodes which need more complex adjustments. The user needs
+to make any required changes for complex overlays and modprobe the
+required Linux modules.
+
+For normal mode, the user is also required to input below three parameters:
+
+a. overlay_node: the name
+b. overlay_type: whether the nodes are fpga nodes or normal nodes
+c. partial or full overlay type: only needed for fpga overlays
+
+Example for domU overlay add:
+
+    (dom0) xl dt-overlay add overlay.dtbo $domid overlay_node_name overlay_type is_partial
+    (dom0) xl console $domid  # go to $domid console
+    (domU) ./usr/lib/xen/bin/get_overlay.sh
+
+Finally if needed, the relevant Linux kernel drive can be loaded:
+
+    (domU) modprobe module_name.ko
+
+Example for domU overlay remove:
+
+    (dom0) xl dt-overlay remove overlay.dtbo $domid overlay_node_name overlay_type is_partial
+
+The get_overlay.sh script automates the following:
+
+a. gets the modified overlay.dtbo from dom0 using Xenstore transactions
+b. applies overlay.dtbo to the domU Linux device tree depending on what
+   type of overlay it is
+c. removes the overlay nodes from device tree when the user requests it
Signature · 1 line
2.34.1
#63 lore raw
In reply to Henry Wang
Quoted 27 lines · Henry Wang

Hi all,

This is the remaining series for the full functional "dynamic node programming using overlay dtbo" feature. The first part [1] has already been merged.

Quoting from the original series, the first part has already made Xen aware of new device tree node which means updating the dt_host with overlay node information, and in this series, the goal is to map IRQ and IOMMU during runtime, where we will do the actual IOMMU and IRQ mapping to a running domain and will call unmap_mmio_regions() to remove the mapping.

Also, documentation of the "dynamic node programming using overlay dtbo" feature is added.

Patch 1 is a fix of [1] which is noticed during my local test, details please see the commit message.

Gitlab CI for this series can be found in [2].

[1] https://lore.kernel.org/xen-devel/[email protected]/
[2] https://gitlab.com/xen-project/people/henryw/xen/-/pipelines/1265297506

Henry Wang (1):
xen/commom/dt-overlay: Fix missing lock when remove the device

Quoted 15 lines

Vikram Garhwal (14):
xen/arm/gic: Enable interrupt assignment to running VM xen/arm: Always enable IOMMU
tools/libs/light: Always enable IOMMU
tools/libs/light: Increase nr_spi to 160
rangeset: Move struct range and struct rangeset to headerfile xen/overlay: Enable device tree overlay assignment to running domains
tools: Add domain_id and expert mode for overlay operations tools/libs/light: Modify dtbo to domU linux dtbo format tools/xl: Share overlay with domU
tools/helpers: Add get_overlay
get_overlay: remove domU overlay
xl/overlay: add remove operation to xenstore
add a domU script to fetch overlays and applying them to linux
docs: add device tree overlay documentation

For all the replies I sent, Vikram's addresses - Xilinx or AMD - bounced. What's the value of Cc-ing dead addresses?

Jan