Skip to contentSkip to conversation

ARM: VGIC/GIC separation cleanups

PATCH In review Andre Przywara 2 participants 15 messages Feb 5, 2018 – Feb 6, 2018

lore
Share & tools
lore
#0 lore raw

v4 cover ARM: VGIC/GIC separation cleanups

Hi,

an updated version, focussing on the complaints Julien raised about the locking on assigned hardware IRQs. Patch 5/8 has been changed to address his concerns.
I dropped the tag on this patch, also on 2/8 since I moved the per-CPU variable declaration into gic.h, as requested. Not sure if that is what people wanted, so please have a look.
The rest of that patch is unchanged, and I checked that the code is still correct even after the rebase to staging.

Please test and apply.

Cheers,
Andre

Changelov v3->v4
01: added tags
02: remove extra header files, move per-CPU variable into gic.h 03, 04: unchanged
05: rework locking, rework dropping hardware IRQ connection 06: add ASSERT
07: unchanged
08: removed read_mostly attribute

Changelog v2->v3:
01: reworked
02: rebased to match staging, added tag
03, 07: added tag
08: new, as suggested by Julien

================
By the original VGIC design, Xen differentiates between the actual VGIC emulation on one hand and the GIC hardware accesses on the other. It seems there were some deviations from that scheme (over time?), so at the moment we end up happily accessing VGIC specific data structures like struct pending_irq and struct vgic_irq_rank from pure GIC files like gic.c or even irq.c (try: git grep -l struct\ pending_irq xen/arch/arm). But any future VGIC rework will depend on a clean separation, so this series tries to clean this up.
After this series there are no more references to VGIC structures from GIC files, at least for non-ITS code. The ITS is a beast own its own (blame the author) and will be addressed later.

Andre Przywara (8):
ARM: VGIC: drop unneeded gic_restore_pending_irqs()
ARM: VGIC: split gic.c to observe hardware/virtual GIC separation
ARM: VGIC: split up gic_dump_info() to cover virtual part separately
ARM: VGIC: rework events_need_delivery()
ARM: VGIC: factor out vgic_connect_hw_irq()
ARM: VGIC: factor out vgic_get_hw_irq_desc()
ARM: VGIC: rework gicv[23]_update_lr to not use pending_irq
ARM: make nr_irqs a constant

 xen/arch/arm/Makefile       |   1 +
 xen/arch/arm/domain.c       |   1 +
 xen/arch/arm/gic-v2.c       |  14 +-
 xen/arch/arm/gic-v3.c       |  12 +-
 xen/arch/arm/gic-vgic.c     | 461 ++++++++++++++++++++++++++++++++++++++++++++
 xen/arch/arm/gic.c          | 423 ++--------------------------------------
 xen/arch/arm/irq.c          |   9 +-
 xen/arch/arm/vgic.c         |  11 ++
 xen/include/asm-arm/event.h |  13 +-
 xen/include/asm-arm/gic.h   |   8 +-
 xen/include/asm-arm/irq.h   |   5 +-
 xen/include/asm-arm/vgic.h  |   6 +
 12 files changed, 521 insertions(+), 443 deletions(-)
 create mode 100644 xen/arch/arm/gic-vgic.c
Signature · 7 lines
2.14.1


_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
#1 lore raw

PATCH v4 1/8 ARM: VGIC: drop unneeded gic_restore_pending_irqs()

In gic_restore_pending_irqs() we push our pending virtual IRQs into the list registers. This function is called once from gic_inject(), just before we return to the guest, but also in gic_restore_state(), when we context-switch a VCPU. Having a closer look it turns out that the later call is not needed, since we will always call gic_inject() anyway. So remove that call (and the forward declaration) to streamline this interface and make separating the GIC from the VGIC world later.

---
 xen/arch/arm/gic.c | 4 ----
 1 file changed, 4 deletions(-)
xen/arch/arm/gic.c +0 -4
index bac8ada2bb..721a17a9d7 100644
@@ -36,8 +36,6 @@
 #include <asm/vgic.h>
 #include <asm/acpi.h>
 
-static void gic_restore_pending_irqs(struct vcpu *v);
-
 static DEFINE_PER_CPU(uint64_t, lr_mask);
 
 #define lr_all_full() (this_cpu(lr_mask) == ((1 << gic_hw_ops->info->nr_lrs) - 1))
@@ -91,8 +89,6 @@ void gic_restore_state(struct vcpu *v)
     gic_hw_ops->restore_state(v);
 
     isb();
-
-    gic_restore_pending_irqs(v);
 }
 
 /* desc->irq needs to be disabled before calling this function */
Signature · 7 lines
2.14.1


_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
#2 lore raw

PATCH v4 2/8 ARM: VGIC: split gic.c to observe hardware/virtual GIC separation

In reply to Andre Przywara

Currently gic.c holds code to handle hardware IRQs as well as code to bridge VGIC requests to the GIC virtualization hardware. Despite being named gic.c, this file reaches into the VGIC and uses data structures describing virtual IRQs.
To improve abstraction, move the VGIC functions into a separate file, so that gic.c does what it says on the tin.

---
 xen/arch/arm/Makefile     |   1 +
 xen/arch/arm/gic-vgic.c   | 396 ++++++++++++++++++++++++++++++++++++++++++++++
 xen/arch/arm/gic.c        | 363 +-----------------------------------------
 xen/include/asm-arm/gic.h |   3 +
 4 files changed, 402 insertions(+), 361 deletions(-)
 create mode 100644 xen/arch/arm/gic-vgic.c
xen/arch/arm/Makefile +1 -0
index 30a2a6500a..41d7366527 100644
@@ -16,6 +16,7 @@ obj-y += domain_build.o
 obj-y += domctl.o
 obj-$(EARLY_PRINTK) += early_printk.o
 obj-y += gic.o
+obj-y += gic-vgic.o
 obj-y += gic-v2.o
 obj-$(CONFIG_HAS_GICV3) += gic-v3.o
 obj-$(CONFIG_HAS_ITS) += gic-v3-its.o
xen/arch/arm/gic-vgic.c new file +396 -0

396 lines not on this page · raw message

xen/arch/arm/gic.c +2 -361

378 lines not on this page · raw message

xen/include/asm-arm/gic.h +3 -0
index 587a14f8b9..71e5354427 100644
@@ -220,6 +220,8 @@ enum gic_version {
     GIC_V3,
 };
 
+DECLARE_PER_CPU(uint64_t, lr_mask);
+
 extern enum gic_version gic_hw_version(void);
 
 /* Program the IRQ type into the GIC */
@@ -372,6 +374,7 @@ struct gic_hw_operations {
     void (*do_LPI)(unsigned int lpi);
 };
 
+extern const struct gic_hw_operations *gic_hw_ops;
 void register_gic_ops(const struct gic_hw_operations *ops);
 int gic_make_hwdom_dt_node(const struct domain *d,
                            const struct dt_device_node *gic,
Signature · 7 lines
2.14.1


_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Acked-by on v4 2/8
#3 lore raw

Re: [PATCH v4 2/8] ARM: VGIC: split gic.c to observe hardware/virtual GIC separation

Hi,

Quoted 19 lines · Andre Przywara
xen/arch/arm/gic-vgic.c new file +13 -0
new file mode 100644
index 0000000000..263b430075
@@ -0,0 +1,396 @@
+/*
+ * xen/arch/arm/gic-vgic.c
+ *
+ * ARM Generic Interrupt Controller virtualization support
+ *
+ * Tim Deegan <[email protected]>
+ * Copyright (c) 2011 Citrix Systems.
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *

xen/arch/arm/gic-vgic.c line 23

Quoted 10 lines
xen/arch/arm/gic-vgic.c new file +10 -0
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ */
+
+#include <xen/lib.h>
+#include <xen/irq.h>
+#include <xen/sched.h>
+#include <xen/errno.h>

Please order them alphabetically. With that:

Cheers,

Signature · 6 lines
Julien Grall

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
#4 lore raw

PATCH v4 3/8 ARM: VGIC: split up gic_dump_info() to cover virtual part separately

In reply to Andre Przywara

Currently gic_dump_info() not only dumps the hardware state of the GIC, but also the VGIC internal virtual IRQ lists.
Split the latter off and move it into gic-vgic.c to observe the abstraction.

---
 xen/arch/arm/domain.c     |  1 +
 xen/arch/arm/gic-vgic.c   | 11 +++++++++++
 xen/arch/arm/gic.c        | 12 ------------
 xen/include/asm-arm/gic.h |  1 +
 4 files changed, 13 insertions(+), 12 deletions(-)
xen/arch/arm/domain.c +1 -0
index eb8c8f6176..a010443bfd 100644
@@ -941,6 +941,7 @@ long arch_do_vcpu_op(int cmd, struct vcpu *v, XEN_GUEST_HANDLE_PARAM(void) arg)
 void arch_dump_vcpu_info(struct vcpu *v)
 {
     gic_dump_info(v);
+    gic_dump_vgic_info(v);
 }
 
 void vcpu_mark_events_pending(struct vcpu *v)
xen/arch/arm/gic-vgic.c +11 -0
index 263b430075..1d5744ecc8 100644
@@ -386,6 +386,17 @@ void gic_inject(void)
         gic_hw_ops->update_hcr_status(GICH_HCR_UIE, true);
 }
 
+void gic_dump_vgic_info(struct vcpu *v)
+{
+    struct pending_irq *p;
+
+    list_for_each_entry ( p, &v->arch.vgic.inflight_irqs, inflight )
+        printk("Inflight irq=%u lr=%u\n", p->irq, p->lr);
+
+    list_for_each_entry( p, &v->arch.vgic.lr_pending, lr_queue )
+        printk("Pending irq=%d\n", p->irq);
+}
+
 /*
  * Local variables:
  * mode: C
xen/arch/arm/gic.c +0 -12
index 04e6d66b69..4cb74d449e 100644
@@ -443,20 +443,8 @@ static void maintenance_interrupt(int irq, void *dev_id, struct cpu_user_regs *r
 
 void gic_dump_info(struct vcpu *v)
 {
-    struct pending_irq *p;
-
     printk("GICH_LRs (vcpu %d) mask=%"PRIx64"\n", v->vcpu_id, v->arch.lr_mask);
     gic_hw_ops->dump_state(v);
-
-    list_for_each_entry ( p, &v->arch.vgic.inflight_irqs, inflight )
-    {
-        printk("Inflight irq=%u lr=%u\n", p->irq, p->lr);
-    }
-
-    list_for_each_entry( p, &v->arch.vgic.lr_pending, lr_queue )
-    {
-        printk("Pending irq=%d\n", p->irq);
-    }
 }
 
 void init_maintenance_interrupt(void)
xen/include/asm-arm/gic.h +1 -0
index 71e5354427..1a142d6e9f 100644
@@ -287,6 +287,7 @@ extern void send_SGI_allbutself(enum gic_sgi sgi);
 
 /* print useful debug info */
 extern void gic_dump_info(struct vcpu *v);
+extern void gic_dump_vgic_info(struct vcpu *v);
 
 /* Number of interrupt lines */
 extern unsigned int gic_number_lines(void);
Signature · 7 lines
2.14.1


_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
#5 lore raw

PATCH v4 4/8 ARM: VGIC: rework events_need_delivery()

In reply to Andre Przywara

In event.h we very deeply dive into the VGIC to learn if an event for a guest is pending.
Rework that function to abstract the VGIC specific part out. Also reorder the queries there, as we only actually need to check for the event channel if there are no other pending IRQs.

---
 xen/arch/arm/vgic.c         | 11 +++++++++++
 xen/include/asm-arm/event.h | 13 +++----------
 xen/include/asm-arm/vgic.h  |  2 ++
 3 files changed, 16 insertions(+), 10 deletions(-)
xen/arch/arm/vgic.c +11 -0
index 6e933a86d3..9921769b15 100644
@@ -593,6 +593,17 @@ void arch_evtchn_inject(struct vcpu *v)
     vgic_vcpu_inject_irq(v, v->domain->arch.evtchn_irq);
 }
 
+bool vgic_evtchn_irq_pending(struct vcpu *v)
+{
+    struct pending_irq *p;
+
+    p = irq_to_pending(v, v->domain->arch.evtchn_irq);
+    /* Does not work for LPIs. */
+    ASSERT(!is_lpi(v->domain->arch.evtchn_irq));
+
+    return list_empty(&p->inflight);
+}
+
 bool vgic_emulate(struct cpu_user_regs *regs, union hsr hsr)
 {
     struct vcpu *v = current;
xen/include/asm-arm/event.h +3 -10
index 2b20d1aa15..e8c2a6cb44 100644
@@ -16,12 +16,6 @@ static inline int vcpu_event_delivery_is_enabled(struct vcpu *v)
 
 static inline int local_events_need_delivery_nomask(void)
 {
-    struct pending_irq *p = irq_to_pending(current,
-                                           current->domain->arch.evtchn_irq);
-
-    /* Does not work for LPIs. */
-    ASSERT(!is_lpi(current->domain->arch.evtchn_irq));
-
     /* XXX: if the first interrupt has already been delivered, we should
      * check whether any other interrupts with priority higher than the
      * one in GICV_IAR are in the lr_pending queue or in the LR
@@ -33,11 +27,10 @@ static inline int local_events_need_delivery_nomask(void)
     if ( gic_events_need_delivery() )
         return 1;
 
-    if ( vcpu_info(current, evtchn_upcall_pending) &&
-        list_empty(&p->inflight) )
-        return 1;
+    if ( !vcpu_info(current, evtchn_upcall_pending) )
+        return 0;
 
-    return 0;
+    return vgic_evtchn_irq_pending(current);
 }
 
 static inline int local_events_need_delivery(void)
xen/include/asm-arm/vgic.h +2 -0
index 2a93a7bef9..22c8502c95 100644
@@ -218,6 +218,8 @@ extern void register_vgic_ops(struct domain *d, const struct vgic_ops *ops);
 int vgic_v2_init(struct domain *d, int *mmio_count);
 int vgic_v3_init(struct domain *d, int *mmio_count);
 
+bool vgic_evtchn_irq_pending(struct vcpu *v);
+
 extern int domain_vgic_register(struct domain *d, int *mmio_count);
 extern int vcpu_vgic_free(struct vcpu *v);
 extern bool vgic_to_sgi(struct vcpu *v, register_t sgir,
Signature · 7 lines
2.14.1


_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
#6 lore raw

PATCH v4 5/8 ARM: VGIC: factor out vgic_connect_hw_irq()

In reply to Andre Przywara

At the moment we happily access VGIC internal data structures like the rank and struct pending_irq in gic.c, which should be VGIC agnostic.

Factor out a new function vgic_connect_hw_irq(), which allows a virtual IRQ to be connected to a hardware IRQ (using the hw bit in the LR).

This removes said accesses to VGIC data structures and improves abstraction.

---
 xen/arch/arm/gic-vgic.c    | 36 ++++++++++++++++++++++++++++++++++++
 xen/arch/arm/gic.c         | 44 ++++++++++----------------------------------
 xen/include/asm-arm/vgic.h |  2 ++
 3 files changed, 48 insertions(+), 34 deletions(-)
xen/arch/arm/gic-vgic.c +36 -0
index 1d5744ecc8..fff7c01ee8 100644
@@ -397,6 +397,42 @@ void gic_dump_vgic_info(struct vcpu *v)
         printk("Pending irq=%d\n", p->irq);
 }
 
+int vgic_connect_hw_irq(struct domain *d, struct vcpu *v, unsigned int virq,
+                        struct irq_desc *desc, bool connect)
+{
+    unsigned long flags;
+    /* Use vcpu0 to retrieve the pending_irq struct. Given that we only
+     * route SPIs to guests, it doesn't make any difference. */
+    struct vcpu *v_target = vgic_get_target_vcpu(d->vcpu[0], virq);
+    struct vgic_irq_rank *rank = vgic_rank_irq(v_target, virq);
+    struct pending_irq *p = irq_to_pending(v_target, virq);
+    int ret = 0;
+
+    ASSERT(connect && desc);
+
+    /* We are taking to rank lock to prevent parallel connections. */
+    vgic_lock_rank(v_target, rank, flags);
+
+    if ( connect )
+    {
+        /* The VIRQ should not be already enabled by the guest */
+        if ( !p->desc &&
+             !test_bit(GIC_IRQ_GUEST_ENABLED, &p->status) )
+            p->desc = desc;
+        else
+            ret = -EBUSY;
+    }
+    else
+    {
+        if ( !desc || p->desc == desc )
+            p->desc = NULL;
+    }
+
+    vgic_unlock_rank(v_target, rank, flags);
+
+    return ret;
+}
+
 /*
  * Local variables:
  * mode: C
xen/arch/arm/gic.c +10 -34

85 lines not on this page · raw message

xen/include/asm-arm/vgic.h +2 -0
index 22c8502c95..fda082395b 100644
@@ -219,6 +219,8 @@ int vgic_v2_init(struct domain *d, int *mmio_count);
 int vgic_v3_init(struct domain *d, int *mmio_count);
 
 bool vgic_evtchn_irq_pending(struct vcpu *v);
+int vgic_connect_hw_irq(struct domain *d, struct vcpu *v, unsigned int virq,
+                        struct irq_desc *desc, bool connect);
 
 extern int domain_vgic_register(struct domain *d, int *mmio_count);
 extern int vcpu_vgic_free(struct vcpu *v);
Signature · 7 lines
2.14.1


_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
#7 lore raw

Re: [PATCH v4 5/8] ARM: VGIC: factor out vgic_connect_hw_irq()

Hi Andre,

Quoted 7 lines · Andre Przywara

At the moment we happily access VGIC internal data structures like the rank and struct pending_irq in gic.c, which should be VGIC agnostic.

Factor out a new function vgic_connect_hw_irq(), which allows a virtual IRQ to be connected to a hardware IRQ (using the hw bit in the LR).

This removes said accesses to VGIC data structures and improves abstraction.

I was expecting some explanation regarding the new locking order in the commit message.

Quoted 13 lines
---
  xen/arch/arm/gic-vgic.c    | 36 ++++++++++++++++++++++++++++++++++++
  xen/arch/arm/gic.c         | 44 ++++++++++----------------------------------
  xen/include/asm-arm/vgic.h |  2 ++
  3 files changed, 48 insertions(+), 34 deletions(-)
xen/arch/arm/gic-vgic.c +0 -0
index 1d5744ecc8..fff7c01ee8 100644

xen/arch/arm/gic-vgic.c void gic_dump_vgic_info(struct vcpu *v) · line 405

Quoted 9 lines
xen/arch/arm/gic-vgic.c +6 -0
@@ -397,6 +397,42 @@ void gic_dump_vgic_info(struct vcpu *v)
          printk("Pending irq=%d\n", p->irq);
  }
 
+int vgic_connect_hw_irq(struct domain *d, struct vcpu *v, unsigned int virq,
+                        struct irq_desc *desc, bool connect)
+{
+    unsigned long flags;
+    /* Use vcpu0 to retrieve the pending_irq struct. Given that we only
+     * route SPIs to guests, it doesn't make any difference. */

Please fix the coding style as requested in v3.

xen/arch/arm/gic-vgic.c void gic_dump_vgic_info(struct vcpu *v) · line 411

Quoted 6 lines
xen/arch/arm/gic-vgic.c +6 -0
+    struct vcpu *v_target = vgic_get_target_vcpu(d->vcpu[0], virq);
+    struct vgic_irq_rank *rank = vgic_rank_irq(v_target, virq);
+    struct pending_irq *p = irq_to_pending(v_target, virq);
+    int ret = 0;
+
+    ASSERT(connect && desc);

I am not sure why you allow desc to be non-NULL when disconnecting it (see more below).

Quoted 6 lines
xen/arch/arm/gic-vgic.c +6 -0
+
+    /* We are taking to rank lock to prevent parallel connections. */
+    vgic_lock_rank(v_target, rank, flags);
+
+    if ( connect )
+    {

xen/arch/arm/gic-vgic.c void gic_dump_vgic_info(struct vcpu *v) · line 427

Quoted 10 lines
xen/arch/arm/gic-vgic.c +10 -0
+        /* The VIRQ should not be already enabled by the guest */
+        if ( !p->desc &&
+             !test_bit(GIC_IRQ_GUEST_ENABLED, &p->status) )
+            p->desc = desc;
+        else
+            ret = -EBUSY;
+    }
+    else
+    {
+        if ( !desc || p->desc == desc )

From a quick glance, no caller will have desc is NULL. Even if it was, it will not harm because p->desc will be set to NULL.

xen/arch/arm/gic-vgic.c void gic_dump_vgic_info(struct vcpu *v) · line 429

Quoted 2 lines
xen/arch/arm/gic-vgic.c +2 -0
+            p->desc = NULL;
+    }

But likely you want to return an error if p->desc != desc as this is a user input error. Ignoring it is a pretty bad.

xen/arch/arm/gic-vgic.c void gic_dump_vgic_info(struct vcpu *v) · line 438

Quoted 9 lines
xen/arch/arm/gic-vgic.c +6 -0
+
+    vgic_unlock_rank(v_target, rank, flags);
+
+    return ret;
+}
+
  /*
   * Local variables:
   * mode: C

Cheers,

Signature · 6 lines
Julien Grall

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
#8 lore raw

Re: [PATCH v4 5/8] ARM: VGIC: factor out vgic_connect_hw_irq()

Quoted 6 lines · Julien Grall

Hi Andre,

Quoted 3 lines · Andre Przywara

At the moment we happily access VGIC internal data structures like the rank and struct pending_irq in gic.c, which should be VGIC agnostic.

Quoted 6 lines
Quoted 4 lines

Factor out a new function vgic_connect_hw_irq(), which allows a virtual IRQ to be connected to a hardware IRQ (using the hw bit in the LR).

This removes said accesses to VGIC data structures and improves abstraction.

I was expecting some explanation regarding the new locking order in the commit message.

From a quick chat with Andre on IRC, it looks like figuring out the current locking might be tricky. So I would be happy if we only point out the change of locking (+ a word on the new locking) in the commit message.

This would help in case we find a regression later on.

Cheers,

Signature · 6 lines
Julien Grall

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
#9 lore raw

Re: [PATCH v4 5/8] ARM: VGIC: factor out vgic_connect_hw_irq()

In reply to Julien Grall

Hi,

Quoted 6 lines · Julien Grall

Hi Andre,

Quoted 3 lines · Andre Przywara

At the moment we happily access VGIC internal data structures like the rank and struct pending_irq in gic.c, which should be VGIC agnostic.

Quoted 8 lines
Quoted 5 lines

Factor out a new function vgic_connect_hw_irq(), which allows a virtual IRQ to be connected to a hardware IRQ (using the hw bit in the LR).

This removes said accesses to VGIC data structures and improves abstraction.

I was expecting some explanation regarding the new locking order in the commit message.

Well, there is no real change in the new locking order. We enter both gic_route_irq_to_guest() and gic_remove_irq_from_guest() with the desc lock held, then take the rank lock at some point in time and drop it again. The only change is how long we hold the rank lock, but this should have no effect, since we don't manipulate any virtual IRQ properties (rank or not) in the code lines that are now no longer under the lock.
I can try to summarise this in the commit message.

Quoted 8 lines
Quoted 8 lines

---
  xen/arch/arm/gic-vgic.c    | 36 ++++++++++++++++++++++++++++++++++++ xen/arch/arm/gic.c         | 44
++++++++++----------------------------------
  xen/include/asm-arm/vgic.h |  2 ++

  3 files changed, 48 insertions(+), 34 deletions(-)
Quoted 16 lines
Quoted 7 lines
xen/arch/arm/gic-vgic.c +0 -0
index 1d5744ecc8..fff7c01ee8 100644
@@ -397,6 +397,42 @@ void gic_dump_vgic_info(struct vcpu *v)
           printk("Pending irq=%d\n", p->irq);
   }
Quoted 7 lines

  +int vgic_connect_hw_irq(struct domain *d, struct vcpu *v, unsigned int virq,
+                        struct irq_desc *desc, bool connect) +{
+    unsigned long flags;
+    /* Use vcpu0 to retrieve the pending_irq struct. Given that we only +     * route SPIs to guests, it doesn't make any difference. */

Please fix the coding style as requested in v3.

Ah, sorry, I forgot that over the rewrite.

Quoted 9 lines
Quoted 6 lines

+    struct vcpu *v_target = vgic_get_target_vcpu(d->vcpu[0], virq); +    struct vgic_irq_rank *rank = vgic_rank_irq(v_target, virq); +    struct pending_irq *p = irq_to_pending(v_target, virq); +    int ret = 0;
+
+    ASSERT(connect && desc);

I am not sure why you allow desc to be non-NULL when disconnecting it (see more below).

I consider passing the desc pointer in addition to the vIRQ number redundant in case we want to drop the association. The only reason we do this is to work around the somewhat opposite locking order. So this is a property of the existing code, which I consider at least somewhat dodgy.

In order to allow changing this in the future (for instance with the new VGIC), I introduced the "connect" parameter and decided to make "desc" optional in case "connect" is false.
If we give it anyway, we check for a match, which is what we do with the current code. So we keep this molly guard.

Quoted 10 lines
Quoted 10 lines
xen/arch/arm/gic-vgic.c +10 -0
+
+    /* We are taking to rank lock to prevent parallel connections. */
+    vgic_lock_rank(v_target, rank, flags);
+
+    if ( connect )
+    {
+        /* The VIRQ should not be already enabled by the guest */
+        if ( !p->desc &&
+             !test_bit(GIC_IRQ_GUEST_ENABLED, &p->status) )
+            p->desc = desc;
Quoted 8 lines
Quoted 6 lines

+        else
+            ret = -EBUSY;
+    }
+    else
+    {
+        if ( !desc || p->desc == desc )

From a quick glance, no caller will have desc is NULL.

Yes, for now.

Quoted 7 lines

Even if it was,
it will not harm because p->desc will be set to NULL.

Quoted 2 lines

+            p->desc = NULL;
+    }

But likely you want to return an error if p->desc != desc as this is a user input error. Ignoring it is a pretty bad.

Right, good point. Actually this is what I had in mind, but somehow managed to derail it. Will fix it.

Thanks for the review!
Andre.

Quoted 12 lines
Quoted 9 lines
xen/arch/arm/gic-vgic.c +6 -0
+
+    vgic_unlock_rank(v_target, rank, flags);
+
+    return ret;
+}
+

  /*
   * Local variables:
   * mode: C

Cheers,

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

#10 lore raw

Re: [PATCH v4 5/8] ARM: VGIC: factor out vgic_connect_hw_irq()

Quoted 18 lines · Andre Przywara

Hi,

Quoted 6 lines · Julien Grall

Hi Andre,

Quoted 3 lines · Andre Przywara

At the moment we happily access VGIC internal data structures like the rank and struct pending_irq in gic.c, which should be VGIC agnostic.

Quoted 8 lines
Quoted 5 lines

Factor out a new function vgic_connect_hw_irq(), which allows a virtual IRQ to be connected to a hardware IRQ (using the hw bit in the LR).

This removes said accesses to VGIC data structures and improves abstraction.

I was expecting some explanation regarding the new locking order in the commit message.

Quoted 8 lines

Well, there is no real change in the new locking order. We enter both gic_route_irq_to_guest() and gic_remove_irq_from_guest() with the desc lock held, then take the rank lock at some point in time and drop it again. The only change is how long we hold the rank lock, but this should have no effect, since we don't manipulate any virtual IRQ properties (rank or not) in the code lines that are now no longer under the lock.
I can try to summarise this in the commit message.

Sorry, somehow I wrote locking order when I was meant to say about how long you take the lock.

Something along what you say above looks good to me. What I want to avoid is running into a problem and wondering why the locking was modified.

Cheers,

Quoted 96 lines
Quoted 9 lines
Quoted 9 lines
---
   xen/arch/arm/gic-vgic.c    | 36 ++++++++++++++++++++++++++++++++++++
   xen/arch/arm/gic.c         | 44
++++++++++----------------------------------
   xen/include/asm-arm/vgic.h |  2 ++
   3 files changed, 48 insertions(+), 34 deletions(-)
Quoted 16 lines
Quoted 5 lines
xen/arch/arm/gic-vgic.c +0 -0
index 1d5744ecc8..fff7c01ee8 100644

xen/arch/arm/gic-vgic.c void gic_dump_vgic_info(struct vcpu *v) · line 405

Quoted 9 lines
xen/arch/arm/gic-vgic.c +5 -0
@@ -397,6 +397,42 @@ void gic_dump_vgic_info(struct vcpu *v)
           printk("Pending irq=%d\n", p->irq);
   }
   +int vgic_connect_hw_irq(struct domain *d, struct vcpu *v, unsigned
 int virq,
+                        struct irq_desc *desc, bool connect)
+{
+    unsigned long flags;
+    /* Use vcpu0 to retrieve the pending_irq struct. Given that we only
+     * route SPIs to guests, it doesn't make any difference. */

Please fix the coding style as requested in v3.

Ah, sorry, I forgot that over the rewrite.

Quoted 9 lines

xen/arch/arm/gic-vgic.c void gic_dump_vgic_info(struct vcpu *v) · line 411

Quoted 6 lines
xen/arch/arm/gic-vgic.c +6 -0
+    struct vcpu *v_target = vgic_get_target_vcpu(d->vcpu[0], virq);
+    struct vgic_irq_rank *rank = vgic_rank_irq(v_target, virq);
+    struct pending_irq *p = irq_to_pending(v_target, virq);
+    int ret = 0;
+
+    ASSERT(connect && desc);

I am not sure why you allow desc to be non-NULL when disconnecting it (see more below).

I consider passing the desc pointer in addition to the vIRQ number redundant in case we want to drop the association. The only reason we do this is to work around the somewhat opposite locking order. So this is a property of the existing code, which I consider at least somewhat dodgy.

In order to allow changing this in the future (for instance with the new VGIC), I introduced the "connect" parameter and decided to make "desc" optional in case "connect" is false.
If we give it anyway, we check for a match, which is what we do with the current code. So we keep this molly guard.

Quoted 10 lines
Quoted 10 lines
xen/arch/arm/gic-vgic.c +10 -0
+
+    /* We are taking to rank lock to prevent parallel connections. */
+    vgic_lock_rank(v_target, rank, flags);
+
+    if ( connect )
+    {
+        /* The VIRQ should not be already enabled by the guest */
+        if ( !p->desc &&
+             !test_bit(GIC_IRQ_GUEST_ENABLED, &p->status) )
+            p->desc = desc;
Quoted 8 lines

xen/arch/arm/gic-vgic.c void gic_dump_vgic_info(struct vcpu *v) · line 427

Quoted 6 lines
xen/arch/arm/gic-vgic.c +6 -0
+        else
+            ret = -EBUSY;
+    }
+    else
+    {
+        if ( !desc || p->desc == desc )

From a quick glance, no caller will have desc is NULL.

Yes, for now.

Quoted 7 lines

Even if it was,
it will not harm because p->desc will be set to NULL.

xen/arch/arm/gic-vgic.c void gic_dump_vgic_info(struct vcpu *v) · line 429

Quoted 2 lines
xen/arch/arm/gic-vgic.c +2 -0
+            p->desc = NULL;
+    }

But likely you want to return an error if p->desc != desc as this is a user input error. Ignoring it is a pretty bad.

Right, good point. Actually this is what I had in mind, but somehow managed to derail it. Will fix it.

Thanks for the review!
Andre.

Quoted 12 lines

xen/arch/arm/gic-vgic.c void gic_dump_vgic_info(struct vcpu *v) · line 438

Quoted 9 lines
xen/arch/arm/gic-vgic.c +6 -0
+
+    vgic_unlock_rank(v_target, rank, flags);
+
+    return ret;
+}
+
   /*
    * Local variables:
    * mode: C

Cheers,

Signature · 6 lines
Julien Grall

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
#11 lore raw

PATCH v4 6/8 ARM: VGIC: factor out vgic_get_hw_irq_desc()

In reply to Andre Przywara

At the moment we happily access the VGIC internal struct pending_irq (which describes a virtual IRQ) in irq.c.
Factor out the actually needed functionality to learn the associated hardware IRQ and move that into gic-vgic.c to improve abstraction.

---
 xen/arch/arm/gic-vgic.c    | 17 +++++++++++++++++
 xen/arch/arm/irq.c         |  7 ++-----
 xen/include/asm-arm/vgic.h |  2 ++
 3 files changed, 21 insertions(+), 5 deletions(-)
xen/arch/arm/gic-vgic.c +17 -0
index fff7c01ee8..632c411565 100644
@@ -397,6 +397,23 @@ void gic_dump_vgic_info(struct vcpu *v)
         printk("Pending irq=%d\n", p->irq);
 }
 
+struct irq_desc *vgic_get_hw_irq_desc(struct domain *d, struct vcpu *v,
+                                      unsigned int virq)
+{
+    struct pending_irq *p;
+
+    ASSERT(!v && virq >= 32);
+
+    if ( !v )
+        v = d->vcpu[0];
+
+    p = irq_to_pending(v, virq);
+    if ( !p )
+        return NULL;
+
+    return p->desc;
+}
+
 int vgic_connect_hw_irq(struct domain *d, struct vcpu *v, unsigned int virq,
                         struct irq_desc *desc, bool connect)
 {
xen/arch/arm/irq.c +2 -5
index 7f133de549..62103a20e3 100644
@@ -534,19 +534,16 @@ int release_guest_irq(struct domain *d, unsigned int virq)
     struct irq_desc *desc;
     struct irq_guest *info;
     unsigned long flags;
-    struct pending_irq *p;
     int ret;
 
     /* Only SPIs are supported */
     if ( virq < NR_LOCAL_IRQS || virq >= vgic_num_irqs(d) )
         return -EINVAL;
 
-    p = spi_to_pending(d, virq);
-    if ( !p->desc )
+    desc = vgic_get_hw_irq_desc(d, NULL, virq);
+    if ( !desc )
         return -EINVAL;
 
-    desc = p->desc;
-
     spin_lock_irqsave(&desc->lock, flags);
 
     ret = -EINVAL;
xen/include/asm-arm/vgic.h +2 -0
index fda082395b..6ea9f140a7 100644
@@ -219,6 +219,8 @@ int vgic_v2_init(struct domain *d, int *mmio_count);
 int vgic_v3_init(struct domain *d, int *mmio_count);
 
 bool vgic_evtchn_irq_pending(struct vcpu *v);
+struct irq_desc *vgic_get_hw_irq_desc(struct domain *d, struct vcpu *v,
+                                      unsigned int virq);
 int vgic_connect_hw_irq(struct domain *d, struct vcpu *v, unsigned int virq,
                         struct irq_desc *desc, bool connect);
 
Signature · 7 lines
2.14.1


_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Reviewed-by on v4 6/8
#12 lore raw

Re: [PATCH v4 6/8] ARM: VGIC: factor out vgic_get_hw_irq_desc()

Hi Andre,

Quoted 7 lines · Andre Przywara

At the moment we happily access the VGIC internal struct pending_irq (which describes a virtual IRQ) in irq.c.
Factor out the actually needed functionality to learn the associated hardware IRQ and move that into gic-vgic.c to improve abstraction.

Cheers,

Quoted 74 lines
---
  xen/arch/arm/gic-vgic.c    | 17 +++++++++++++++++
  xen/arch/arm/irq.c         |  7 ++-----
  xen/include/asm-arm/vgic.h |  2 ++
  3 files changed, 21 insertions(+), 5 deletions(-)
xen/arch/arm/gic-vgic.c +17 -0
index fff7c01ee8..632c411565 100644
@@ -397,6 +397,23 @@ void gic_dump_vgic_info(struct vcpu *v)
          printk("Pending irq=%d\n", p->irq);
  }
 
+struct irq_desc *vgic_get_hw_irq_desc(struct domain *d, struct vcpu *v,
+                                      unsigned int virq)
+{
+    struct pending_irq *p;
+
+    ASSERT(!v && virq >= 32);
+
+    if ( !v )
+        v = d->vcpu[0];
+
+    p = irq_to_pending(v, virq);
+    if ( !p )
+        return NULL;
+
+    return p->desc;
+}
+
  int vgic_connect_hw_irq(struct domain *d, struct vcpu *v, unsigned int virq,
                          struct irq_desc *desc, bool connect)
  {
xen/arch/arm/irq.c +2 -5
index 7f133de549..62103a20e3 100644
@@ -534,19 +534,16 @@ int release_guest_irq(struct domain *d, unsigned int virq)
      struct irq_desc *desc;
      struct irq_guest *info;
      unsigned long flags;
-    struct pending_irq *p;
      int ret;
 
      /* Only SPIs are supported */
      if ( virq < NR_LOCAL_IRQS || virq >= vgic_num_irqs(d) )
          return -EINVAL;
 
-    p = spi_to_pending(d, virq);
-    if ( !p->desc )
+    desc = vgic_get_hw_irq_desc(d, NULL, virq);
+    if ( !desc )
          return -EINVAL;
 
-    desc = p->desc;
-
      spin_lock_irqsave(&desc->lock, flags);
 
      ret = -EINVAL;
xen/include/asm-arm/vgic.h +2 -0
index fda082395b..6ea9f140a7 100644
@@ -219,6 +219,8 @@ int vgic_v2_init(struct domain *d, int *mmio_count);
  int vgic_v3_init(struct domain *d, int *mmio_count);
 
  bool vgic_evtchn_irq_pending(struct vcpu *v);
+struct irq_desc *vgic_get_hw_irq_desc(struct domain *d, struct vcpu *v,
+                                      unsigned int virq);
  int vgic_connect_hw_irq(struct domain *d, struct vcpu *v, unsigned int virq,
                          struct irq_desc *desc, bool connect);
 
Signature · 6 lines
Julien Grall

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
#13 lore raw

PATCH v4 7/8 ARM: VGIC: rework gicv[23]_update_lr to not use pending_irq

In reply to Andre Przywara

The functions to actually populate a list register were accessing the VGIC internal pending_irq struct, although they should be abstracting from that.
Break the needed information down to remove the reference to pending_irq from gic-v[23].c.

---
 xen/arch/arm/gic-v2.c     | 14 +++++++-------
 xen/arch/arm/gic-v3.c     | 12 ++++++------
 xen/arch/arm/gic-vgic.c   |  3 ++-
 xen/include/asm-arm/gic.h |  4 ++--
 xen/include/asm-arm/irq.h |  3 +++
 5 files changed, 20 insertions(+), 16 deletions(-)
xen/arch/arm/gic-v2.c +7 -7
index 511c8d7294..2b271ba322 100644
@@ -428,8 +428,8 @@ static void gicv2_disable_interface(void)
     spin_unlock(&gicv2.lock);
 }
 
-static void gicv2_update_lr(int lr, const struct pending_irq *p,
-                            unsigned int state)
+static void gicv2_update_lr(int lr, unsigned int virq, uint8_t priority,
+                            unsigned int hw_irq, unsigned int state)
 {
     uint32_t lr_reg;
 
@@ -437,12 +437,12 @@ static void gicv2_update_lr(int lr, const struct pending_irq *p,
     BUG_ON(lr < 0);
 
     lr_reg = (((state & GICH_V2_LR_STATE_MASK) << GICH_V2_LR_STATE_SHIFT)  |
-              ((GIC_PRI_TO_GUEST(p->priority) & GICH_V2_LR_PRIORITY_MASK)
-                                             << GICH_V2_LR_PRIORITY_SHIFT) |
-              ((p->irq & GICH_V2_LR_VIRTUAL_MASK) << GICH_V2_LR_VIRTUAL_SHIFT));
+              ((GIC_PRI_TO_GUEST(priority) & GICH_V2_LR_PRIORITY_MASK)
+                                          << GICH_V2_LR_PRIORITY_SHIFT) |
+              ((virq & GICH_V2_LR_VIRTUAL_MASK) << GICH_V2_LR_VIRTUAL_SHIFT));
 
-    if ( p->desc != NULL )
-        lr_reg |= GICH_V2_LR_HW | ((p->desc->irq & GICH_V2_LR_PHYSICAL_MASK )
+    if ( hw_irq != INVALID_IRQ )
+        lr_reg |= GICH_V2_LR_HW | ((hw_irq & GICH_V2_LR_PHYSICAL_MASK )
                                    << GICH_V2_LR_PHYSICAL_SHIFT);
 
     writel_gich(lr_reg, GICH_LR + lr * 4);
xen/arch/arm/gic-v3.c +6 -6
index 072345c6f9..25c30bb9ea 100644
@@ -966,8 +966,8 @@ static void gicv3_disable_interface(void)
     spin_unlock(&gicv3.lock);
 }
 
-static void gicv3_update_lr(int lr, const struct pending_irq *p,
-                            unsigned int state)
+static void gicv3_update_lr(int lr, unsigned int virq, uint8_t priority,
+                            unsigned int hw_irq, unsigned int state)
 {
     uint64_t val = 0;
 
@@ -983,11 +983,11 @@ static void gicv3_update_lr(int lr, const struct pending_irq *p,
     if ( current->domain->arch.vgic.version == GIC_V3 )
         val |= GICH_LR_GRP1;
 
-    val |= ((uint64_t)p->priority & 0xff) << GICH_LR_PRIORITY_SHIFT;
-    val |= ((uint64_t)p->irq & GICH_LR_VIRTUAL_MASK) << GICH_LR_VIRTUAL_SHIFT;
+    val |= (uint64_t)priority << GICH_LR_PRIORITY_SHIFT;
+    val |= ((uint64_t)virq & GICH_LR_VIRTUAL_MASK) << GICH_LR_VIRTUAL_SHIFT;
 
-   if ( p->desc != NULL )
-       val |= GICH_LR_HW | (((uint64_t)p->desc->irq & GICH_LR_PHYSICAL_MASK)
+   if ( hw_irq != INVALID_IRQ )
+       val |= GICH_LR_HW | (((uint64_t)hw_irq & GICH_LR_PHYSICAL_MASK)
                            << GICH_LR_PHYSICAL_SHIFT);
 
     gicv3_ich_write_lr(lr, val);
xen/arch/arm/gic-vgic.c +2 -1
index 632c411565..cad30a929e 100644
@@ -38,7 +38,8 @@ static inline void gic_set_lr(int lr, struct pending_irq *p,
 
     clear_bit(GIC_IRQ_GUEST_PRISTINE_LPI, &p->status);
 
-    gic_hw_ops->update_lr(lr, p, state);
+    gic_hw_ops->update_lr(lr, p->irq, p->priority,
+                          p->desc ? p->desc->irq : INVALID_IRQ, state);
 
     set_bit(GIC_IRQ_GUEST_VISIBLE, &p->status);
     clear_bit(GIC_IRQ_GUEST_QUEUED, &p->status);
xen/include/asm-arm/gic.h +2 -2
index 1a142d6e9f..497f195bc1 100644
@@ -344,8 +344,8 @@ struct gic_hw_operations {
     /* Disable CPU physical and virtual interfaces */
     void (*disable_interface)(void);
     /* Update LR register with state and priority */
-    void (*update_lr)(int lr, const struct pending_irq *pending_irq,
-                      unsigned int state);
+    void (*update_lr)(int lr, unsigned int virq, uint8_t priority,
+                      unsigned int hw_irq, unsigned int state);
     /* Update HCR status register */
     void (*update_hcr_status)(uint32_t flag, bool set);
     /* Clear LR register */
xen/include/asm-arm/irq.h +3 -0
index abc8f06a13..0d110ecb08 100644
@@ -31,6 +31,9 @@ struct arch_irq_desc {
 /* LPIs are always numbered starting at 8192, so 0 is a good invalid case. */
 #define INVALID_LPI     0
 
+/* This is a spurious interrupt ID which never makes it into the GIC code. */
+#define INVALID_IRQ     1023
+
 extern unsigned int nr_irqs;
 #define nr_static_irqs NR_IRQS
 #define arch_hwdom_irqs(domid) NR_IRQS
Signature · 7 lines
2.14.1


_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
#14 lore raw

PATCH v4 8/8 ARM: make nr_irqs a constant

In reply to Andre Przywara

On ARM the maximum number of IRQs is a constant, but we share it being a variable to match x86. Since we are not supposed to alter it, let's mark it as "const" to avoid accidental change.

---
 xen/arch/arm/irq.c        | 2 +-
 xen/include/asm-arm/irq.h | 2 +-
 2 files changed, 2 insertions(+), 2 deletions(-)
xen/arch/arm/irq.c +1 -1
index 62103a20e3..29af10e82c 100644
@@ -27,7 +27,7 @@
 #include <asm/gic.h>
 #include <asm/vgic.h>
 
-unsigned int __read_mostly nr_irqs = NR_IRQS;
+const unsigned int nr_irqs = NR_IRQS;
 
 static unsigned int local_irqs_type[NR_LOCAL_IRQS];
 static DEFINE_SPINLOCK(local_irqs_type_lock);
xen/include/asm-arm/irq.h +1 -1
index 0d110ecb08..9d55e9b122 100644
@@ -34,7 +34,7 @@ struct arch_irq_desc {
 /* This is a spurious interrupt ID which never makes it into the GIC code. */
 #define INVALID_IRQ     1023
 
-extern unsigned int nr_irqs;
+extern const unsigned int nr_irqs;
 #define nr_static_irqs NR_IRQS
 #define arch_hwdom_irqs(domid) NR_IRQS
 
Signature · 7 lines
2.14.1


_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel