Skip to contentSkip to conversation

Xen: FIFO-based event channel ABI

PATCH Superseded David Vrabel 6 participants 23 messages Sep 27, 2013 – Oct 1, 2013

v4 of 9 Superseded Latest is v9 →

lore
Share & tools
lore
#0 lore raw

v4 cover Xen: FIFO-based event channel ABI

This is a complete implementation of the hypervisor and xl toolstack parts of the FIFO-based event channel ABI described in this design document:

http://xenbits.xen.org/people/dvrabel/event-channels-F.pdf

Changes in draft F are:

- READY field in the control block is now 32-bits (so guests only need to support atomic bit ops on 32-bit words). This is only a documentation change as the implementation already used a uint32_t.

- DOMCTL_set_max_evtchn replaces EVTCHNOP_set_limit.

- DomUs default to unlimited number of event channels requiring the toolstack to set a limit.

The toolstack defaults to limiting guests to 127 event channels if the event_channels option is omitted. This means the minimum amount of both Xen heap and global mapping space is used regardless of which ABI is used. If this is considered too restrictive a limit, 1023 would be another sensible default (limits the guest to a single event array page but 5 xenheap pages for the struct evtchns).

An updated version of the Linux patch series is not quite ready yet. There is one remaining issue but fixing this will not require any changes to the hypervisor ABI or implementation. The remaining issue requires preallocating space in the evtchn-to-irq map as this cannot be expanded in pirq_startup() (since this function cannot return a failure).

The latest Linux changes can be found in the orochi-v4w branch of:

git://xenbits.xen.org/people/dvrabel/linux.git

Patch 1-4 do some preparatory work for supporting alternate ABIs.

Patch 5 expands the number of evtchn objects a domain may have by changing how they are allocated.

Patch 6 adds the public ABI.

Patch 7 adds the EVTCHNOP_set_priority implementation. This will return -ENOSYS for ABIs that do not support priority.

Patch 8 adds the FIFO-based ABI implementation.

Patch 9-10 adds the DOMCTL_set_max_evtchn implementation and adds a function to libxc. This will also work with the 2-level ABI.

Patch 11 add the event_channels configuration option to xl and libxl bits needed for this.

Changes in v4:

- Updates for Draft F of the design.
- DOMCTL_set_max_evtchn replaces EVTCHNOP_set_limit.
- Hypervisor defaults to unlimited event channels for for DomU.
- Optimized memory allocation for struct evtchn's when fewer than 128 are required (see patch 5).
- Added event_channels option to the xl domain configuration file and plumbed this through libxl_build_info. Defaults to 127.

Changes in v3:

- Updates for Draft E of the design.
- Store priority in struct evtchn.
- Implement set_priority with generic code + hook.
- Implement set_limit and add libxc function.
- Add ABI specific output to 'e' debug key.

Changes in v2:

- Updates for Draft D of the design.
- 130,000+ event channels are now supported.
- event_port.c -> event_2l.c and only contains 2l functions.
- Addresses various review comments
- int -> unsigned in lots of places
- use write_atomic() to set HEAD
- removed MAX_EVTCHNS
- evtchn_ops are const.
- Pack struct evtchns better to reduce memory needed.

#1 lore raw

PATCH v4 1/11 debug: remove some event channel info from the 'i' and 'q' debug keys

From: David Vrabel <[email protected]>

The 'i' key would always use VCPU0's selector word when printing the event channel state. Remove the incorrect output as a subsequent change will add the (correct) information to the 'e' key instead.

When dumping domain information, printing the state of the VIRQ_DEBUG port is redundant -- this information is available via the 'e' key.

---
 xen/arch/x86/irq.c      |    5 +----
 xen/common/keyhandler.c |   11 ++---------
 2 files changed, 3 insertions(+), 13 deletions(-)
xen/arch/x86/irq.c +1 -4
index c61cc46..7f547ff 100644
@@ -2262,14 +2262,11 @@ static void dump_irqs(unsigned char key)
                 d = action->guest[i];
                 pirq = domain_irq_to_pirq(d, irq);
                 info = pirq_info(d, pirq);
-                printk("%u:%3d(%c%c%c%c)",
+                printk("%u:%3d(%c%c%c)",
                        d->domain_id, pirq,
                        (test_bit(info->evtchn,
                                  &shared_info(d, evtchn_pending)) ?
                         'P' : '-'),
-                       (test_bit(info->evtchn / BITS_PER_EVTCHN_WORD(d),
-                                 &vcpu_info(d->vcpu[0], evtchn_pending_sel)) ?
-                        'S' : '-'),
                        (test_bit(info->evtchn, &shared_info(d, evtchn_mask)) ?
                         'M' : '-'),
                        (info->masked ? 'M' : '-'));
xen/common/keyhandler.c +2 -9
index b9ad1b5..8e4b3f8 100644
@@ -310,16 +310,9 @@ static void dump_domains(unsigned char key)
     {
         for_each_vcpu ( d, v )
         {
-            printk("Notifying guest %d:%d (virq %d, port %d, stat %d/%d/%d)\n",
+            printk("Notifying guest %d:%d (virq %d, port %d)\n",
                    d->domain_id, v->vcpu_id,
-                   VIRQ_DEBUG, v->virq_to_evtchn[VIRQ_DEBUG],
-                   test_bit(v->virq_to_evtchn[VIRQ_DEBUG], 
-                            &shared_info(d, evtchn_pending)),
-                   test_bit(v->virq_to_evtchn[VIRQ_DEBUG], 
-                            &shared_info(d, evtchn_mask)),
-                   test_bit(v->virq_to_evtchn[VIRQ_DEBUG] /
-                            BITS_PER_EVTCHN_WORD(d),
-                            &vcpu_info(v, evtchn_pending_sel)));
+                   VIRQ_DEBUG, v->virq_to_evtchn[VIRQ_DEBUG]);
             send_guest_vcpu_virq(v, VIRQ_DEBUG);
         }
     }
Signature · 1 line
1.7.2.5
#2 lore raw

PATCH v4 2/11 evtchn: refactor low-level event channel port ops

In reply to David Vrabel

From: David Vrabel <[email protected]>

Use functions for the low-level event channel port operations (set/clear pending, unmask, is_pending and is_masked).

Group these functions into a struct evtchn_port_op so they can be replaced by alternate implementations (for different ABIs) on a per-domain basis.

---
 xen/arch/x86/irq.c         |   11 ++---
 xen/common/Makefile        |    1 +
 xen/common/event_2l.c      |   99 ++++++++++++++++++++++++++++++++++++++++++++
 xen/common/event_channel.c |   87 +++++++++++++++------------------------
 xen/common/schedule.c      |    3 +-
 xen/include/xen/event.h    |   45 ++++++++++++++++++++
 xen/include/xen/sched.h    |    4 ++
 7 files changed, 189 insertions(+), 61 deletions(-)
 create mode 100644 xen/common/event_2l.c
xen/arch/x86/irq.c +5 -6
index 7f547ff..53fe9e3 100644
@@ -1474,7 +1474,7 @@ int pirq_guest_unmask(struct domain *d)
         {
             pirq = pirqs[i]->pirq;
             if ( pirqs[i]->masked &&
-                 !test_bit(pirqs[i]->evtchn, &shared_info(d, evtchn_mask)) )
+                 !evtchn_port_is_masked(d, evtchn_from_port(d, pirqs[i]->evtchn)) )
                 pirq_guest_eoi(pirqs[i]);
         }
     } while ( ++pirq < d->nr_pirqs && n == ARRAY_SIZE(pirqs) );
@@ -2222,6 +2222,7 @@ static void dump_irqs(unsigned char key)
     int i, irq, pirq;
     struct irq_desc *desc;
     irq_guest_action_t *action;
+    struct evtchn *evtchn;
     struct domain *d;
     const struct pirq *info;
     unsigned long flags;
@@ -2262,13 +2263,11 @@ static void dump_irqs(unsigned char key)
                 d = action->guest[i];
                 pirq = domain_irq_to_pirq(d, irq);
                 info = pirq_info(d, pirq);
+                evtchn = evtchn_from_port(d, info->evtchn);
                 printk("%u:%3d(%c%c%c)",
                        d->domain_id, pirq,
-                       (test_bit(info->evtchn,
-                                 &shared_info(d, evtchn_pending)) ?
-                        'P' : '-'),
-                       (test_bit(info->evtchn, &shared_info(d, evtchn_mask)) ?
-                        'M' : '-'),
+                       (evtchn_port_is_pending(d, evtchn) ? 'P' : '-'),
+                       (evtchn_port_is_masked(d, evtchn) ? 'M' : '-'),
                        (info->masked ? 'M' : '-'));
                 if ( i != action->nr_guests )
                     printk(",");
xen/common/Makefile +1 -0
index 5486140..0a3a367 100644
@@ -5,6 +5,7 @@ obj-y += cpupool.o
 obj-$(HAS_DEVICE_TREE) += device_tree.o
 obj-y += domctl.o
 obj-y += domain.o
+obj-y += event_2l.o
 obj-y += event_channel.o
 obj-y += grant_table.o
 obj-y += irq.o
xen/common/event_2l.c new file +99 -0

99 lines not on this page · raw message

xen/common/event_channel.c +33 -54

132 lines not on this page · raw message

xen/common/schedule.c +2 -1
index a8398bd..7e6884d 100644
@@ -34,6 +34,7 @@
 #include <xen/multicall.h>
 #include <xen/cpu.h>
 #include <xen/preempt.h>
+#include <xen/event.h>
 #include <public/sched.h>
 #include <xsm/xsm.h>
 
@@ -751,7 +752,7 @@ static long do_poll(struct sched_poll *sched_poll)
             goto out;
 
         rc = 0;
-        if ( test_bit(port, &shared_info(d, evtchn_pending)) )
+        if ( evtchn_port_is_pending(d, evtchn_from_port(d, port)) )
             goto out;
     }
 
xen/include/xen/event.h +45 -0

49 lines not on this page · raw message

xen/include/xen/sched.h +4 -0
index 0013a8d..fb9cf11 100644
@@ -66,6 +66,7 @@ struct evtchn
     u8  state;             /* ECS_* */
     u8  xen_consumer;      /* Consumer in Xen, if any? (0 = send to guest) */
     u16 notify_vcpu_id;    /* VCPU for local delivery notification */
+    u32 port;
     union {
         struct {
             domid_t remote_domid;
@@ -238,6 +239,8 @@ struct mem_event_per_domain
     struct mem_event_domain access;
 };
 
+struct evtchn_port_ops;
+
 struct domain
 {
     domid_t          domain_id;
@@ -271,6 +274,7 @@ struct domain
     /* Event channel information. */
     struct evtchn   *evtchn[NR_EVTCHN_BUCKETS];
     spinlock_t       event_lock;
+    const struct evtchn_port_ops *evtchn_port_ops;
 
     struct grant_table *grant_table;
 
Signature · 1 line
1.7.2.5
#3 lore raw

PATCH v4 3/11 evtchn: print ABI specific state with the 'e' debug key

In reply to David Vrabel

From: David Vrabel <[email protected]>

In the output of the 'e' debug key, print some ABI specific state in addition to the (p)ending and (m)asked bits.

For the 2-level ABI, print the state of that event's selector bit. e.g.,

(XEN) port [p/m/s]
(XEN) 1 [0/0/1]: s=3 n=0 x=0 d=0 p=74
(XEN) 2 [0/0/1]: s=3 n=0 x=0 d=0 p=75

---
 xen/common/event_2l.c      |   10 ++++++++++
 xen/common/event_channel.c |    8 +++++---
 xen/include/xen/event.h    |    7 +++++++
 3 files changed, 22 insertions(+), 3 deletions(-)
xen/common/event_2l.c +10 -0
index 18c0c6e..b7b152c 100644
@@ -74,6 +74,15 @@ static bool_t evtchn_2l_is_masked(struct domain *d,
     return test_bit(evtchn->port, &shared_info(d, evtchn_mask));
 }
 
+static void evtchn_2l_print_state(struct domain *d,
+                                  const struct evtchn *evtchn)
+{
+    struct vcpu *v = d->vcpu[evtchn->notify_vcpu_id];
+
+    printk("%d", !!test_bit(evtchn->port / BITS_PER_EVTCHN_WORD(d),
+                            &vcpu_info(v, evtchn_pending_sel)));
+}
+
 static const struct evtchn_port_ops evtchn_port_ops_2l =
 {
     .set_pending   = evtchn_2l_set_pending,
@@ -81,6 +90,7 @@ static const struct evtchn_port_ops evtchn_port_ops_2l =
     .unmask        = evtchn_2l_unmask,
     .is_pending    = evtchn_2l_is_pending,
     .is_masked     = evtchn_2l_is_masked,
+    .print_state   = evtchn_2l_print_state,
 };
 
 void evtchn_2l_init(struct domain *d)
xen/common/event_channel.c +5 -3
index 618ced0..51d59b8 100644
@@ -1232,7 +1232,7 @@ static void domain_dump_evtchn_info(struct domain *d)
                          d->poll_mask, d->max_vcpus);
     printk("Event channel information for domain %d:\n"
            "Polling vCPUs: {%s}\n"
-           "    port [p/m]\n", d->domain_id, keyhandler_scratch);
+           "    port [p/m/s]\n", d->domain_id, keyhandler_scratch);
 
     spin_lock(&d->event_lock);
 
@@ -1247,10 +1247,12 @@ static void domain_dump_evtchn_info(struct domain *d)
         if ( chn->state == ECS_FREE )
             continue;
 
-        printk("    %4u [%d/%d]: s=%d n=%d x=%d",
+        printk("    %4u [%d/%d/",
                port,
                !!evtchn_port_is_pending(d, chn),
-               !!evtchn_port_is_masked(d, chn),
+               !!evtchn_port_is_masked(d, chn));
+        evtchn_port_print_state(d, chn);
+        printk("]: s=%d n=%d x=%d",
                chn->state, chn->notify_vcpu_id, chn->xen_consumer);
 
         switch ( chn->state )
xen/include/xen/event.h +7 -0
index 7522f4e..90410e0 100644
@@ -115,6 +115,7 @@ struct evtchn_port_ops {
     void (*unmask)(struct domain *d, struct evtchn *evtchn);
     bool_t (*is_pending)(struct domain *d, const struct evtchn *evtchn);
     bool_t (*is_masked)(struct domain *d, const struct evtchn *evtchn);
+    void (*print_state)(struct domain *d, const struct evtchn *evtchn);
 };
 
 static inline void evtchn_port_set_pending(struct vcpu *v,
@@ -147,4 +148,10 @@ static inline bool_t evtchn_port_is_masked(struct domain *d,
     return d->evtchn_port_ops->is_masked(d, evtchn);
 }
 
+static inline void evtchn_port_print_state(struct domain *d,
+                                           const struct evtchn *evtchn)
+{
+    d->evtchn_port_ops->print_state(d, evtchn);
+}
+
 #endif /* __XEN_EVENT_H__ */
Signature · 1 line
1.7.2.5
#4 lore raw

PATCH v4 4/11 evtchn: use a per-domain variable for the max number of event channels

In reply to David Vrabel

From: David Vrabel <[email protected]>

Instead of the MAX_EVTCHNS(d) macro, use d->max_evtchns instead. This avoids having to repeatedly check the ABI type.

---
 xen/common/event_2l.c      |    1 +
 xen/common/event_channel.c |    4 ++--
 xen/common/schedule.c      |    2 +-
 xen/include/xen/event.h    |    2 +-
 xen/include/xen/sched.h    |    2 +-
 5 files changed, 6 insertions(+), 5 deletions(-)
xen/common/event_2l.c +1 -0
index b7b152c..ecdcdaf 100644
@@ -96,6 +96,7 @@ static const struct evtchn_port_ops evtchn_port_ops_2l =
 void evtchn_2l_init(struct domain *d)
 {
     d->evtchn_port_ops = &evtchn_port_ops_2l;
+    d->max_evtchns = BITS_PER_EVTCHN_WORD(d) * BITS_PER_EVTCHN_WORD(d);
 }
 
 /*
xen/common/event_channel.c +2 -2
index 51d59b8..5b5df88 100644
@@ -134,7 +134,7 @@ static int get_free_port(struct domain *d)
         if ( evtchn_from_port(d, port)->state == ECS_FREE )
             return port;
 
-    if ( port == MAX_EVTCHNS(d) )
+    if ( port == d->max_evtchns )
         return -ENOSPC;
 
     chn = xzalloc_array(struct evtchn, EVTCHNS_PER_BUCKET);
@@ -1236,7 +1236,7 @@ static void domain_dump_evtchn_info(struct domain *d)
 
     spin_lock(&d->event_lock);
 
-    for ( port = 1; port < MAX_EVTCHNS(d); ++port )
+    for ( port = 1; port < d->max_evtchns; ++port )
     {
         const struct evtchn *chn;
         char *ssid;
xen/common/schedule.c +1 -1
index 7e6884d..a5a0010 100644
@@ -748,7 +748,7 @@ static long do_poll(struct sched_poll *sched_poll)
             goto out;
 
         rc = -EINVAL;
-        if ( port >= MAX_EVTCHNS(d) )
+        if ( port >= d->max_evtchns )
             goto out;
 
         rc = 0;
xen/include/xen/event.h +1 -1
index 90410e0..302a904 100644
@@ -73,7 +73,7 @@ void notify_via_xen_event_channel(struct domain *ld, int lport);
 #define bucket_from_port(d,p) \
     ((d)->evtchn[(p)/EVTCHNS_PER_BUCKET])
 #define port_is_valid(d,p)    \
-    (((p) >= 0) && ((p) < MAX_EVTCHNS(d)) && \
+    (((p) >= 0) && ((p) < (d)->max_evtchns) &&  \
      (bucket_from_port(d,p) != NULL))
 #define evtchn_from_port(d,p) \
     (&(bucket_from_port(d,p))[(p)&(EVTCHNS_PER_BUCKET-1)])
xen/include/xen/sched.h +1 -1
index fb9cf11..532dd46 100644
@@ -50,7 +50,6 @@ extern struct domain *dom0;
 #else
 #define BITS_PER_EVTCHN_WORD(d) (has_32bit_shinfo(d) ? 32 : BITS_PER_XEN_ULONG)
 #endif
-#define MAX_EVTCHNS(d) (BITS_PER_EVTCHN_WORD(d) * BITS_PER_EVTCHN_WORD(d))
 #define EVTCHNS_PER_BUCKET 128
 #define NR_EVTCHN_BUCKETS  (NR_EVENT_CHANNELS / EVTCHNS_PER_BUCKET)
 
@@ -273,6 +272,7 @@ struct domain
 
     /* Event channel information. */
     struct evtchn   *evtchn[NR_EVTCHN_BUCKETS];
+    unsigned         max_evtchns;
     spinlock_t       event_lock;
     const struct evtchn_port_ops *evtchn_port_ops;
 
Signature · 1 line
1.7.2.5
#5 lore raw

PATCH v4 5/11 evtchn: allow many more evtchn objects to be allocated per domain

In reply to David Vrabel

From: David Vrabel <[email protected]>

Expand the number of event channels that can be supported internally by altering now struct evtchn's are allocated.

The objects are indexed using a two level scheme of groups and buckets (instead of only buckets). Each group is a page of bucket pointers. Each bucket is a page-sized array of struct evtchn's.

The optimal number of evtchns per bucket is calculated at compile time.

If XSM is not enabled, struct evtchn is 16 bytes and each bucket contains 256, requiring only 1 group of 512 pointers for 2^17 (131,072) event channels. With XSM enabled, struct evtchn is 24 bytes, each bucket contains 128 and 2 groups are required.

For the common case of a domain with only a few event channels, instead of requiring an additional allocation for the group page, the first bucket is indexed directly.

As a consequence of this, struct domain shrinks by at least 232 bytes as 32 bucket pointers are replaced with 1 bucket pointer and (at most) 2 group pointers.

[ Based on a patch from Wei Liu with improvements from Malcolm Crossley. ]

---
 xen/common/event_channel.c |   82 ++++++++++++++++++++++++++++++++++----------
 xen/include/xen/event.h    |   40 ++++++++++++++++-----
 xen/include/xen/sched.h    |   21 ++++++++++--
 3 files changed, 113 insertions(+), 30 deletions(-)
xen/common/event_channel.c +64 -18
index 5b5df88..fe9bba2 100644
@@ -121,11 +121,47 @@ static int virq_is_global(uint32_t virq)
 }
 
 
+static struct evtchn *alloc_evtchn_bucket(struct domain *d, unsigned port)
+{
+    struct evtchn *chn;
+    unsigned i;
+
+    chn = xzalloc_array(struct evtchn, EVTCHNS_PER_BUCKET);
+    if ( !chn )
+        return NULL;
+
+    for ( i = 0; i < EVTCHNS_PER_BUCKET; i++ )
+    {
+        if ( xsm_alloc_security_evtchn(&chn[i]) )
+        {
+            while ( i-- )
+                xsm_free_security_evtchn(&chn[i]);
+            xfree(chn);
+            return NULL;
+        }
+        chn[i].port = port + i;
+    }
+    return chn;
+}
+
+static void free_evtchn_bucket(struct domain *d, struct evtchn *bucket)
+{
+    unsigned i;
+
+    if ( !bucket )
+        return;
+
+    for ( i = 0; i < EVTCHNS_PER_BUCKET; i++ )
+        xsm_free_security_evtchn(bucket + i);
+
+    xfree(bucket);
+}
+
 static int get_free_port(struct domain *d)
 {
     struct evtchn *chn;
+    struct evtchn **grp;
     int            port;
-    int            i, j;
 
     if ( d->is_dying )
         return -EINVAL;
@@ -137,22 +173,17 @@ static int get_free_port(struct domain *d)
     if ( port == d->max_evtchns )
         return -ENOSPC;
 
-    chn = xzalloc_array(struct evtchn, EVTCHNS_PER_BUCKET);
-    if ( unlikely(chn == NULL) )
-        return -ENOMEM;
-
-    for ( i = 0; i < EVTCHNS_PER_BUCKET; i++ )
+    if ( !group_from_port(d, port) )
     {
-        if ( xsm_alloc_security_evtchn(&chn[i]) )
-        {
-            for ( j = 0; j < i; j++ )
-                xsm_free_security_evtchn(&chn[j]);
-            xfree(chn);
+        grp = xzalloc_array(struct evtchn *, BUCKETS_PER_GROUP);
+        if ( !grp )
             return -ENOMEM;
-        }
-        chn[i].port = port + i;
+        group_from_port(d, port) = grp;
     }
 
+    chn = alloc_evtchn_bucket(d, port);
+    if ( !chn )
+        return -ENOMEM;
     bucket_from_port(d, port) = chn;
 
     return port;
@@ -1152,15 +1183,25 @@ int evtchn_init(struct domain *d)
 {
     evtchn_2l_init(d);
 
+    d->evtchn = alloc_evtchn_bucket(d, 0);
+    if ( !d->evtchn )
+        return -ENOMEM;
+
     spin_lock_init(&d->event_lock);
     if ( get_free_port(d) != 0 )
+    {
+        free_evtchn_bucket(d, d->evtchn);
         return -EINVAL;
+    }
     evtchn_from_port(d, 0)->state = ECS_RESERVED;
 
 #if MAX_VIRT_CPUS > BITS_PER_LONG
     d->poll_mask = xmalloc_array(unsigned long, BITS_TO_LONGS(MAX_VIRT_CPUS));
     if ( !d->poll_mask )
+    {
+        free_evtchn_bucket(d, d->evtchn);
         return -ENOMEM;
+    }
     bitmap_zero(d->poll_mask, MAX_VIRT_CPUS);
 #endif
 
@@ -1170,7 +1211,7 @@ int evtchn_init(struct domain *d)
 
 void evtchn_destroy(struct domain *d)
 {
-    int i;
+    unsigned i, j;
 
     /* After this barrier no new event-channel allocations can occur. */
     BUG_ON(!d->is_dying);
@@ -1185,12 +1226,17 @@ void evtchn_destroy(struct domain *d)
 
     /* Free all event-channel buckets. */
     spin_lock(&d->event_lock);
-    for ( i = 0; i < NR_EVTCHN_BUCKETS; i++ )
+    for ( i = 0; i < NR_EVTCHN_GROUPS; i++ )
     {
-        xsm_free_security_evtchn(d->evtchn[i]);
-        xfree(d->evtchn[i]);
-        d->evtchn[i] = NULL;
+        if ( !d->evtchn_group[i] )
+            continue;
+        for ( j = 0; j < BUCKETS_PER_GROUP; j++ )
+            free_evtchn_bucket(d, d->evtchn_group[i][j]);
+        xfree(d->evtchn_group[i]);
+        d->evtchn_group[i] = NULL;
     }
+    free_evtchn_bucket(d, d->evtchn);
+    d->evtchn = NULL;
     spin_unlock(&d->event_lock);
 
     clear_global_virq_handlers(d);
xen/include/xen/event.h +30 -8
index 302a904..9d1a8c4 100644
@@ -69,15 +69,37 @@ int guest_enabled_event(struct vcpu *v, uint32_t virq);
 /* Notify remote end of a Xen-attached event channel.*/
 void notify_via_xen_event_channel(struct domain *ld, int lport);
 
-/* Internal event channel object accessors */
-#define bucket_from_port(d,p) \
-    ((d)->evtchn[(p)/EVTCHNS_PER_BUCKET])
-#define port_is_valid(d,p)    \
-    (((p) >= 0) && ((p) < (d)->max_evtchns) &&  \
-     (bucket_from_port(d,p) != NULL))
-#define evtchn_from_port(d,p) \
-    (&(bucket_from_port(d,p))[(p)&(EVTCHNS_PER_BUCKET-1)])
+/*
+ * Internal event channel object storage.
+ *
+ * The objects (struct evtchn) are indexed using a two level scheme of
+ * groups and buckets.  Each group is a page of bucket pointers.  Each
+ * bucket is a page-sized array of struct evtchn's.
+ *
+ * The first bucket is directly accessed via d->evtchn.
+ */
+#define group_from_port(d, p) \
+    ((d)->evtchn_group[(p) / EVTCHNS_PER_GROUP])
+#define bucket_from_port(d, p) \
+    ((group_from_port(d, p))[((p) % EVTCHNS_PER_GROUP) / EVTCHNS_PER_BUCKET])
 
+static inline bool_t port_is_valid(struct domain *d, unsigned p)
+{
+    if ( p >= d->max_evtchns )
+        return 0;
+    if ( !d->evtchn )
+        return 0;
+    if ( p < EVTCHNS_PER_BUCKET )
+        return 1;
+    return group_from_port(d, p) != NULL && bucket_from_port(d, p) != NULL;
+}
+
+static inline struct evtchn *evtchn_from_port(struct domain *d, unsigned p)
+{
+    if ( p < EVTCHNS_PER_BUCKET )
+        return &d->evtchn[p];
+    return bucket_from_port(d, p) + (p % EVTCHNS_PER_BUCKET);
+}
 
 /* Wait on a Xen-attached event channel. */
 #define wait_on_xen_event_channel(port, condition)                      \
xen/include/xen/sched.h +18 -3
index 532dd46..48ab812 100644
@@ -50,8 +50,22 @@ extern struct domain *dom0;
 #else
 #define BITS_PER_EVTCHN_WORD(d) (has_32bit_shinfo(d) ? 32 : BITS_PER_XEN_ULONG)
 #endif
-#define EVTCHNS_PER_BUCKET 128
-#define NR_EVTCHN_BUCKETS  (NR_EVENT_CHANNELS / EVTCHNS_PER_BUCKET)
+
+#define BUCKETS_PER_GROUP  (PAGE_SIZE/sizeof(struct evtchn *))
+/* Round size of struct evtchn up to power of 2 size */
+#define __RDU2(x)   (       (x) | (   (x) >> 1))
+#define __RDU4(x)   ( __RDU2(x) | ( __RDU2(x) >> 2))
+#define __RDU8(x)   ( __RDU4(x) | ( __RDU4(x) >> 4))
+#define __RDU16(x)  ( __RDU8(x) | ( __RDU8(x) >> 8))
+#define __RDU32(x)  (__RDU16(x) | (__RDU16(x) >>16))
+#define next_power_of_2(x)      (__RDU32((x)-1) + 1)
+
+/* Maximum number of event channels for any ABI. */
+#define MAX_NR_EVTCHNS NR_EVENT_CHANNELS
+
+#define EVTCHNS_PER_BUCKET (PAGE_SIZE / next_power_of_2(sizeof(struct evtchn)))
+#define EVTCHNS_PER_GROUP  (BUCKETS_PER_GROUP * EVTCHNS_PER_BUCKET)
+#define NR_EVTCHN_GROUPS   DIV_ROUND_UP(MAX_NR_EVTCHNS, EVTCHNS_PER_GROUP)
 
 struct evtchn
 {
@@ -271,7 +285,8 @@ struct domain
     spinlock_t       rangesets_lock;
 
     /* Event channel information. */
-    struct evtchn   *evtchn[NR_EVTCHN_BUCKETS];
+    struct evtchn   *evtchn;                         /* first bucket only */
+    struct evtchn  **evtchn_group[NR_EVTCHN_GROUPS]; /* all other buckets */
     unsigned         max_evtchns;
     spinlock_t       event_lock;
     const struct evtchn_port_ops *evtchn_port_ops;
Signature · 1 line
1.7.2.5
#6 lore raw

PATCH v4 6/11 evtchn: add FIFO-based event channel ABI

In reply to David Vrabel

From: David Vrabel <[email protected]>

Add the event channel hypercall sub-ops and the definitions for the shared data structures for the FIFO-based event channel ABI.

The design document for this new ABI is available here:

http://xenbits.xen.org/people/dvrabel/event-channels-F.pdf

In summary, events are reported using a per-domain shared event array of event words. Each event word has PENDING, LINKED and MASKED bits and a LINK field for pointing to the next event in the event queue.

There are 16 event queues (with different priorities) per-VCPU.

Key advantages of this new ABI include:

- Support for over 100,000 events (2^17).
- 16 different event priorities.
- Improved fairness in event latency through the use of FIFOs.

---
 xen/include/public/event_channel.h |   75 ++++++++++++++++++++++++++++++++++++
 xen/include/public/xen.h           |    6 ++-
 xen/include/xen/sched.h            |    2 +-
 3 files changed, 80 insertions(+), 3 deletions(-)
xen/include/public/event_channel.h +75 -0
index 472efdb..4a53484 100644
@@ -71,6 +71,9 @@
 #define EVTCHNOP_bind_vcpu        8
 #define EVTCHNOP_unmask           9
 #define EVTCHNOP_reset           10
+#define EVTCHNOP_init_control    11
+#define EVTCHNOP_expand_array    12
+#define EVTCHNOP_set_priority    13
 /* ` } */
 
 typedef uint32_t evtchn_port_t;
@@ -258,6 +261,43 @@ struct evtchn_reset {
 typedef struct evtchn_reset evtchn_reset_t;
 
 /*
+ * EVTCHNOP_init_control: initialize the control block for the FIFO ABI.
+ *
+ * Note: any events that are currently pending will not be resent and
+ * will be lost.  Guests should call this before binding any event to
+ * avoid losing any events.
+ */
+struct evtchn_init_control {
+    /* IN parameters. */
+    uint64_t control_gfn;
+    uint32_t offset;
+    uint32_t vcpu;
+    /* OUT parameters. */
+    uint8_t link_bits;
+    uint8_t _pad[7];
+};
+typedef struct evtchn_init_control evtchn_init_control_t;
+
+/*
+ * EVTCHNOP_expand_array: add an additional page to the event array.
+ */
+struct evtchn_expand_array {
+    /* IN parameters. */
+    uint64_t array_gfn;
+};
+typedef struct evtchn_expand_array evtchn_expand_array_t;
+
+/*
+ * EVTCHNOP_set_priority: set the priority for an event channel.
+ */
+struct evtchn_set_priority {
+    /* IN parameters. */
+    uint32_t port;
+    uint32_t priority;
+};
+typedef struct evtchn_set_priority evtchn_set_priority_t;
+
+/*
  * ` enum neg_errnoval
  * ` HYPERVISOR_event_channel_op_compat(struct evtchn_op *op)
  * `
@@ -281,6 +321,41 @@ struct evtchn_op {
 typedef struct evtchn_op evtchn_op_t;
 DEFINE_XEN_GUEST_HANDLE(evtchn_op_t);
 
+/*
+ * 2-level ABI
+ */
+
+#define EVTCHN_2L_NR_CHANNELS (sizeof(xen_ulong_t) * sizeof(xen_ulong_t) * 64)
+
+/*
+ * FIFO ABI
+ */
+
+/* Events may have priorities from 0 (highest) to 15 (lowest). */
+#define EVTCHN_FIFO_PRIORITY_MAX     0
+#define EVTCHN_FIFO_PRIORITY_DEFAULT 7
+#define EVTCHN_FIFO_PRIORITY_MIN     15
+
+#define EVTCHN_FIFO_MAX_QUEUES (EVTCHN_FIFO_PRIORITY_MIN + 1)
+
+typedef uint32_t event_word_t;
+
+#define EVTCHN_FIFO_PENDING 31
+#define EVTCHN_FIFO_MASKED  30
+#define EVTCHN_FIFO_LINKED  29
+
+#define EVTCHN_FIFO_LINK_BITS 17
+#define EVTCHN_FIFO_LINK_MASK ((1 << EVTCHN_FIFO_LINK_BITS) - 1)
+
+#define EVTCHN_FIFO_NR_CHANNELS (1 << EVTCHN_FIFO_LINK_BITS)
+
+struct evtchn_fifo_control_block {
+    uint32_t ready;
+    uint32_t _rsvd;
+    uint32_t head[EVTCHN_FIFO_MAX_QUEUES];
+};
+typedef struct evtchn_fifo_control_block evtchn_fifo_control_block_t;
+
 #endif /* __XEN_PUBLIC_EVENT_CHANNEL_H__ */
 
 /*
xen/include/public/xen.h +4 -2
index b50bd05..8c5697e 100644
@@ -552,11 +552,13 @@ struct multicall_entry {
 typedef struct multicall_entry multicall_entry_t;
 DEFINE_XEN_GUEST_HANDLE(multicall_entry_t);
 
+#if __XEN_INTERFACE_VERSION__ < 0x00040400
 /*
- * Event channel endpoints per domain:
+ * Event channel endpoints per domain (when using the 2-level ABI):
  *  1024 if a long is 32 bits; 4096 if a long is 64 bits.
  */
-#define NR_EVENT_CHANNELS (sizeof(xen_ulong_t) * sizeof(xen_ulong_t) * 64)
+#define NR_EVENT_CHANNELS EVTCHN_2L_NR_CHANNELS
+#endif
 
 struct vcpu_time_info {
     /*
xen/include/xen/sched.h +1 -1
index 48ab812..4107c41 100644
@@ -61,7 +61,7 @@ extern struct domain *dom0;
 #define next_power_of_2(x)      (__RDU32((x)-1) + 1)
 
 /* Maximum number of event channels for any ABI. */
-#define MAX_NR_EVTCHNS NR_EVENT_CHANNELS
+#define MAX_NR_EVTCHNS EVTCHN_2L_NR_CHANNELS
 
 #define EVTCHNS_PER_BUCKET (PAGE_SIZE / next_power_of_2(sizeof(struct evtchn)))
 #define EVTCHNS_PER_GROUP  (BUCKETS_PER_GROUP * EVTCHNS_PER_BUCKET)
Signature · 1 line
1.7.2.5
#7 lore raw

PATCH v4 7/11 evtchn: implement EVTCHNOP_set_priority and add the set_priority hook

In reply to David Vrabel

From: David Vrabel <[email protected]>

Implement EVTCHNOP_set_priority. A new set_priority hook added to struct evtchn_port_ops will do the ABI specific validation and setup.

If an ABI does not provide a set_priority hook (as is the case of the 2-level ABI), the sub-op will return -ENOSYS.

---
 xen/common/event_channel.c |   29 +++++++++++++++++++++++++++++
 xen/include/xen/event.h    |   11 +++++++++++
 2 files changed, 40 insertions(+), 0 deletions(-)
xen/common/event_channel.c +29 -0
index fe9bba2..9b5f710 100644
@@ -955,6 +955,27 @@ out:
     return rc;
 }
 
+static long evtchn_set_priority(const struct evtchn_set_priority *set_priority)
+{
+    struct domain *d = current->domain;
+    unsigned port = set_priority->port;
+    long ret;
+
+    spin_lock(&d->event_lock);
+
+    if ( !port_is_valid(d, port) )
+    {
+        spin_unlock(&d->event_lock);
+        return -EINVAL;
+    }
+
+    ret = evtchn_port_set_priority(d, evtchn_from_port(d, port),
+                                   set_priority->priority);
+
+    spin_unlock(&d->event_lock);
+
+    return ret;
+}
 
 long do_event_channel_op(int cmd, XEN_GUEST_HANDLE_PARAM(void) arg)
 {
@@ -1064,6 +1085,14 @@ long do_event_channel_op(int cmd, XEN_GUEST_HANDLE_PARAM(void) arg)
         break;
     }
 
+    case EVTCHNOP_set_priority: {
+        struct evtchn_set_priority set_priority;
+        if ( copy_from_guest(&set_priority, arg, 1) != 0 )
+            return -EFAULT;
+        rc = evtchn_set_priority(&set_priority);
+        break;
+    }
+
     default:
         rc = -ENOSYS;
         break;
xen/include/xen/event.h +11 -0
index 9d1a8c4..16e7ed0 100644
@@ -137,6 +137,8 @@ struct evtchn_port_ops {
     void (*unmask)(struct domain *d, struct evtchn *evtchn);
     bool_t (*is_pending)(struct domain *d, const struct evtchn *evtchn);
     bool_t (*is_masked)(struct domain *d, const struct evtchn *evtchn);
+    int (*set_priority)(struct domain *d, struct evtchn *evtchn,
+                        unsigned priority);
     void (*print_state)(struct domain *d, const struct evtchn *evtchn);
 };
 
@@ -170,6 +172,15 @@ static inline bool_t evtchn_port_is_masked(struct domain *d,
     return d->evtchn_port_ops->is_masked(d, evtchn);
 }
 
+static inline int evtchn_port_set_priority(struct domain *d,
+                                           struct evtchn *evtchn,
+                                           unsigned priority)
+{
+    if ( !d->evtchn_port_ops->set_priority )
+        return -ENOSYS;
+    return d->evtchn_port_ops->set_priority(d, evtchn, priority);
+}
+
 static inline void evtchn_port_print_state(struct domain *d,
                                            const struct evtchn *evtchn)
 {
Signature · 1 line
1.7.2.5
#8 lore raw

PATCH v4 8/11 evtchn: add FIFO-based event channel hypercalls and port ops

In reply to David Vrabel

From: David Vrabel <[email protected]>

Add the implementation for the FIFO-based event channel ABI. The new hypercall sub-ops (EVTCHNOP_init_control, EVTCHNOP_expand_array) and the required evtchn_ops (set_pending, unmask, etc.).

---
 xen/common/Makefile          |    1 +
 xen/common/event_channel.c   |   21 ++
 xen/common/event_fifo.c      |  455 ++++++++++++++++++++++++++++++++++++++++++
 xen/include/xen/event_fifo.h |   53 +++++
 xen/include/xen/sched.h      |    6 +-
 5 files changed, 535 insertions(+), 1 deletions(-)
 create mode 100644 xen/common/event_fifo.c
 create mode 100644 xen/include/xen/event_fifo.h
xen/common/Makefile +1 -0
index 0a3a367..533b603 100644
@@ -7,6 +7,7 @@ obj-y += domctl.o
 obj-y += domain.o
 obj-y += event_2l.o
 obj-y += event_channel.o
+obj-y += event_fifo.o
 obj-y += grant_table.o
 obj-y += irq.o
 obj-y += kernel.o
xen/common/event_channel.c +21 -0
index 9b5f710..2c90a66 100644
@@ -26,6 +26,7 @@
 #include <xen/compat.h>
 #include <xen/guest_access.h>
 #include <xen/keyhandler.h>
+#include <xen/event_fifo.h>
 #include <asm/current.h>
 
 #include <public/xen.h>
@@ -1085,6 +1086,24 @@ long do_event_channel_op(int cmd, XEN_GUEST_HANDLE_PARAM(void) arg)
         break;
     }
 
+    case EVTCHNOP_init_control: {
+        struct evtchn_init_control init_control;
+        if ( copy_from_guest(&init_control, arg, 1) != 0 )
+            return -EFAULT;
+        rc = evtchn_fifo_init_control(&init_control);
+        if ( !rc && __copy_to_guest(arg, &init_control, 1) )
+            rc = -EFAULT;
+        break;
+    }
+
+    case EVTCHNOP_expand_array: {
+        struct evtchn_expand_array expand_array;
+        if ( copy_from_guest(&expand_array, arg, 1) != 0 )
+            return -EFAULT;
+        rc = evtchn_fifo_expand_array(&expand_array);
+        break;
+    }
+
     case EVTCHNOP_set_priority: {
         struct evtchn_set_priority set_priority;
         if ( copy_from_guest(&set_priority, arg, 1) != 0 )
@@ -1269,6 +1288,8 @@ void evtchn_destroy(struct domain *d)
     spin_unlock(&d->event_lock);
 
     clear_global_virq_handlers(d);
+
+    evtchn_fifo_destroy(d);
 }
 
 
xen/common/event_fifo.c new file +455 -0

455 lines not on this page · raw message

xen/include/xen/event_fifo.h new file +53 -0

53 lines not on this page · raw message

xen/include/xen/sched.h +5 -1
index 4107c41..ae653e6 100644
@@ -61,7 +61,7 @@ extern struct domain *dom0;
 #define next_power_of_2(x)      (__RDU32((x)-1) + 1)
 
 /* Maximum number of event channels for any ABI. */
-#define MAX_NR_EVTCHNS EVTCHN_2L_NR_CHANNELS
+#define MAX_NR_EVTCHNS MAX(EVTCHN_2L_NR_CHANNELS, EVTCHN_FIFO_NR_CHANNELS)
 
 #define EVTCHNS_PER_BUCKET (PAGE_SIZE / next_power_of_2(sizeof(struct evtchn)))
 #define EVTCHNS_PER_GROUP  (BUCKETS_PER_GROUP * EVTCHNS_PER_BUCKET)
@@ -95,6 +95,7 @@ struct evtchn
         } pirq;        /* state == ECS_PIRQ */
         u16 virq;      /* state == ECS_VIRQ */
     } u;
+    u8 priority;
 #ifdef FLASK_ENABLE
     void *ssid;
 #endif
@@ -209,6 +210,8 @@ struct vcpu
     /* Guest-specified relocation of vcpu_info. */
     unsigned long vcpu_info_mfn;
 
+    struct evtchn_fifo_vcpu *evtchn_fifo;
+
     struct arch_vcpu arch;
 };
 
@@ -290,6 +293,7 @@ struct domain
     unsigned         max_evtchns;
     spinlock_t       event_lock;
     const struct evtchn_port_ops *evtchn_port_ops;
+    struct evtchn_fifo_domain *evtchn_fifo;
 
     struct grant_table *grant_table;
 
Signature · 1 line
1.7.2.5
#9 lore raw

Re: [PATCH 08/11] evtchn: add FIFO-based event channel hypercalls and port ops

Quoted 7 lines
Quoted 1 line
Quoted 1 line · David Vrabel
xen/common/event_fifo.c new file +6 -0
+static int map_guest_page(struct domain *d, uint64_t gfn,
+                          struct page_info **page, void **virt)
+{
+    struct page_info *p;
+
+    p = get_page_from_gfn(d, gfn, NULL, P2M_ALLOC);

xen/common/event_fifo.c line 229

Quoted 10 lines
xen/common/event_fifo.c new file +10 -0
+    if ( !p )
+        return -EINVAL;
+
+    if ( !get_page_type(p, PGT_writable_page) )
+    {
+        put_page(p);
+        return -EINVAL;
+    }
+
+    *virt = map_domain_page_global(gfn);

Since this one returns page aligned addresses, ...

Quoted 5 lines
xen/common/event_fifo.c new file +5 -0
+    if ( !*virt )
+    {
+        put_page_and_type(p);
+        return -ENOMEM;
+    }

xen/common/event_fifo.c line 244

Quoted 10 lines
xen/common/event_fifo.c new file +10 -0
+    *page = p;
+    return 0;
+}
+
+static void unmap_guest_page(struct page_info *page, void *virt)
+{
+    if ( page == NULL )
+        return;
+
+    unmap_domain_page_global(virt);

... this one expects page aligned addresses, but the way you use it below doesn't guarantee that (see b0581b92 ("x86: make map_domain_page_global() a simple wrapper around vmap()")).

Quoted 16 lines
xen/common/event_fifo.c new file +16 -0
+static int setup_control_block(struct vcpu *v, uint64_t gfn, uint32_t offset)
+{
+    struct domain *d = v->domain;
+    struct evtchn_fifo_vcpu *efv;
+    struct page_info *page;
+    void *virt;
+    unsigned i;
+    int rc;
+
+    if ( v->evtchn_fifo )
+        return -EINVAL;
+
+    efv = xzalloc(struct evtchn_fifo_vcpu);
+    if ( efv == NULL )
+        return -ENOMEM;
+

xen/common/event_fifo.c line 290

Quoted 10 lines
xen/common/event_fifo.c new file +10 -0
+    rc = map_guest_page(d, gfn, &page, &virt);
+    if ( rc < 0 )
+    {
+        xfree(efv);
+        return rc;
+    }
+
+    v->evtchn_fifo = efv;
+
+    v->evtchn_fifo->cb_page       = page;

There's no real need to store this if you only need it for freeing later. x86 at least has domain_page_map_to_mfn() to recover the MFN (and thus page) from the mapping, and if ARM doesn't they should implement it.

xen/include/xen/event_fifo.h line 17

Quoted 6 lines
xen/include/xen/event_fifo.h new file +6 -0
+struct evtchn_fifo_queue {
+    uint32_t *head; /* points into control block */
+    uint32_t tail;
+    spinlock_t lock;
+    uint8_t priority;
+};

spinlock_t being at least 32 bits in size you win nothing with this ordering, whereas if "tail" and "priority" would be adjacent the structure size might decrease for certain cases (namely when alignof(spinlock_t) > 4).

Jan

#10 lore raw

PATCH v4 9/11 xen: Add DOMCTL to limit the number of event channels a domain may use

In reply to David Vrabel

From: David Vrabel <[email protected]>

Add XEN_DOMCTL_set_max_evtchn which may be used during domain creation to set the maximum event channel port a domain may use. This may be used to limit the amount of Xen resources (global mapping space and xenheap) that a domain may use for event channels.

A domain that does not have a limit set may use all the event channels supported by the event channel ABI in use.

Cc Daniel De Graaf <[email protected]>
---
 tools/flask/policy/policy/mls                |    2 +-
 tools/flask/policy/policy/modules/xen/xen.if |    2 +-
 tools/flask/policy/policy/modules/xen/xen.te |    2 +-
 xen/common/domctl.c                          |    8 ++++++++
 xen/common/event_channel.c                   |    7 ++++++-
 xen/include/public/domctl.h                  |   15 ++++++++++++++-
 xen/include/xen/sched.h                      |    1 +
 xen/xsm/flask/hooks.c                        |    3 +++
 xen/xsm/flask/policy/access_vectors          |    2 ++
 9 files changed, 37 insertions(+), 5 deletions(-)
tools/flask/policy/policy/mls +1 -1
index 9290a76..fb603cd 100644
@@ -74,7 +74,7 @@ mlsconstrain domain { getaffinity getdomaininfo getvcpuinfo getvcpucontext getad
 	((l1 dom l2) or (t1 == mls_priv));
 
 # all the domain "write" ops
-mlsconstrain domain { setvcpucontext pause unpause resume create max_vcpus destroy setaffinity scheduler setdomainmaxmem setdomainhandle setdebugging hypercall settime set_target shutdown setaddrsize trigger setextvcpucontext }
+mlsconstrain domain { setvcpucontext pause unpause resume create max_vcpus destroy setaffinity scheduler setdomainmaxmem setdomainhandle setdebugging hypercall settime set_target shutdown setaddrsize trigger setextvcpucontext set_max_evtchn }
 	((l1 eq l2) or (t1 == mls_priv));
 
 # This is incomplete - similar constraints must be written for all classes
tools/flask/policy/policy/modules/xen/xen.if +1 -1
index 97af0a8..63e40f0 100644
@@ -48,7 +48,7 @@ define(`create_domain_common', `
 	allow $1 $2:domain { create max_vcpus setdomainmaxmem setaddrsize
 			getdomaininfo hypercall setvcpucontext setextvcpucontext
 			getscheduler getvcpuinfo getvcpuextstate getaddrsize
-			getaffinity setaffinity };
+			getaffinity setaffinity set_max_evtchn };
 	allow $1 $2:domain2 { set_cpuid settsc setscheduler setclaim };
 	allow $1 $2:security check_context;
 	allow $1 $2:shadow enable;
tools/flask/policy/policy/modules/xen/xen.te +1 -1
index c89ce28..5f9de5c 100644
@@ -73,7 +73,7 @@ allow dom0_t dom0_t:domain {
 	getdomaininfo getvcpuinfo getvcpucontext setdomainmaxmem setdomainhandle
 	setdebugging hypercall settime setaddrsize getaddrsize trigger
 	getextvcpucontext setextvcpucontext getvcpuextstate setvcpuextstate
-	getpodtarget setpodtarget set_misc_info set_virq_handler
+	getpodtarget setpodtarget set_misc_info set_virq_handler set_max_evtchn
 };
 allow dom0_t dom0_t:domain2 {
 	set_cpuid gettsc settsc setscheduler
xen/common/domctl.c +8 -0
index 9760d50..bffe8d8 100644
@@ -863,6 +863,14 @@ long do_domctl(XEN_GUEST_HANDLE_PARAM(xen_domctl_t) u_domctl)
     }
     break;
 
+    case XEN_DOMCTL_set_max_evtchn:
+    {
+        d->max_evtchn_port = min_t(unsigned,
+                                   op->u.set_max_evtchn.max_port,
+                                   INT_MAX);
+    }
+    break;
+
     default:
         ret = arch_do_domctl(op, d, u_domctl);
         break;
xen/common/event_channel.c +6 -1
index 2c90a66..8e430e2 100644
@@ -168,10 +168,14 @@ static int get_free_port(struct domain *d)
         return -EINVAL;
 
     for ( port = 0; port_is_valid(d, port); port++ )
+    {
+        if ( port > d->max_evtchn_port )
+            return -ENOSPC;
         if ( evtchn_from_port(d, port)->state == ECS_FREE )
             return port;
+    }
 
-    if ( port == d->max_evtchns )
+    if ( port == d->max_evtchns || port > d->max_evtchn_port )
         return -ENOSPC;
 
     if ( !group_from_port(d, port) )
@@ -1230,6 +1234,7 @@ void evtchn_check_pollers(struct domain *d, unsigned port)
 int evtchn_init(struct domain *d)
 {
     evtchn_2l_init(d);
+    d->max_evtchn_port = INT_MAX;
 
     d->evtchn = alloc_evtchn_bucket(d, 0);
     if ( !d->evtchn )
xen/include/public/domctl.h +14 -1
index 4c5b2bb..ed9155a 100644
@@ -36,7 +36,7 @@
 #include "grant_table.h"
 #include "hvm/save.h"
 
-#define XEN_DOMCTL_INTERFACE_VERSION 0x00000009
+#define XEN_DOMCTL_INTERFACE_VERSION 0x0000000a
 
 /*
  * NB. xen_domctl.domain is an IN/OUT parameter for this operation.
@@ -852,6 +852,17 @@ struct xen_domctl_set_broken_page_p2m {
 typedef struct xen_domctl_set_broken_page_p2m xen_domctl_set_broken_page_p2m_t;
 DEFINE_XEN_GUEST_HANDLE(xen_domctl_set_broken_page_p2m_t);
 
+/*
+ * XEN_DOMCTL_set_max_evtchn: sets the maximum event channel port
+ * number the guest may use.  Use this limit the amount of resources
+ * (global mapping space, xenheap) a guest may use for event channels.
+ */
+struct xen_domctl_set_max_evtchn {
+    uint32_t max_port;
+};
+typedef struct xen_domctl_set_max_evtchn xen_domctl_set_max_evtchn_t;
+DEFINE_XEN_GUEST_HANDLE(xen_domctl_set_max_evtchn_t);
+
 struct xen_domctl {
     uint32_t cmd;
 #define XEN_DOMCTL_createdomain                   1
@@ -920,6 +931,7 @@ struct xen_domctl {
 #define XEN_DOMCTL_set_broken_page_p2m           67
 #define XEN_DOMCTL_setnodeaffinity               68
 #define XEN_DOMCTL_getnodeaffinity               69
+#define XEN_DOMCTL_set_max_evtchn                70
 #define XEN_DOMCTL_gdbsx_guestmemio            1000
 #define XEN_DOMCTL_gdbsx_pausevcpu             1001
 #define XEN_DOMCTL_gdbsx_unpausevcpu           1002
@@ -975,6 +987,7 @@ struct xen_domctl {
         struct xen_domctl_set_access_required access_required;
         struct xen_domctl_audit_p2m         audit_p2m;
         struct xen_domctl_set_virq_handler  set_virq_handler;
+        struct xen_domctl_set_max_evtchn    set_max_evtchn;
         struct xen_domctl_gdbsx_memio       gdbsx_guest_memio;
         struct xen_domctl_set_broken_page_p2m set_broken_page_p2m;
         struct xen_domctl_gdbsx_pauseunp_vcpu gdbsx_pauseunp_vcpu;
xen/include/xen/sched.h +1 -0
index ae653e6..bca381c 100644
@@ -291,6 +291,7 @@ struct domain
     struct evtchn   *evtchn;                         /* first bucket only */
     struct evtchn  **evtchn_group[NR_EVTCHN_GROUPS]; /* all other buckets */
     unsigned         max_evtchns;
+    unsigned         max_evtchn_port;
     spinlock_t       event_lock;
     const struct evtchn_port_ops *evtchn_port_ops;
     struct evtchn_fifo_domain *evtchn_fifo;
xen/xsm/flask/hooks.c +3 -0
index fa0589a..548df47 100644
@@ -727,6 +727,9 @@ static int flask_domctl(struct domain *d, int cmd)
     case XEN_DOMCTL_audit_p2m:
         return current_has_perm(d, SECCLASS_HVM, HVM__AUDIT_P2M);
 
+    case XEN_DOMCTL_set_max_evtchn:
+        return current_has_perm(d, SECCLASS_DOMAIN, DOMAIN__SET_MAX_EVTCHN);
+
     default:
         printk("flask_domctl: Unknown op %d\n", cmd);
         return -EPERM;
xen/xsm/flask/policy/access_vectors +2 -0
index 5dfe13b..03a8e64 100644
@@ -157,6 +157,8 @@ class domain
     set_misc_info
 # XEN_DOMCTL_set_virq_handler
     set_virq_handler
+# XEN_DOMCTL_set_max_evtchn
+    set_max_evtchn
 }
 
 # This is a continuation of class domain, since only 32 permissions can be
Signature · 1 line
1.7.2.5
Reviewed-by on v4 9/11
#11 lore raw

Re: [PATCH 09/11] xen: Add DOMCTL to limit the number of event channels a domain may use

xen/include/public/domctl.h line 39

Quoted 9 lines
Quoted 1 line
Quoted 1 line · David Vrabel
xen/include/public/domctl.h +1 -1
@@ -36,7 +36,7 @@
 #include "grant_table.h"
 #include "hvm/save.h"
 
-#define XEN_DOMCTL_INTERFACE_VERSION 0x00000009
+#define XEN_DOMCTL_INTERFACE_VERSION 0x0000000a

The mere addition of a new sub-hypercall does not make an interface version bump necessary.

With that fixed

for the non-XSM parts of the patch.

Jan

#12 lore raw

Re: [PATCH 09/11] xen: Add DOMCTL to limit the number of event channels a domain may use

In reply to David Vrabel
Quoted 13 lines · David Vrabel

From: David Vrabel <[email protected]>

Add XEN_DOMCTL_set_max_evtchn which may be used during domain creation to set the maximum event channel port a domain may use. This may be used to limit the amount of Xen resources (global mapping space and xenheap) that a domain may use for event channels.

A domain that does not have a limit set may use all the event channels supported by the event channel ABI in use.

Cc Daniel De Graaf <[email protected]>

---

[...]

Quoted 13 lines
xen/xsm/flask/policy/access_vectors +2 -0
index 5dfe13b..03a8e64 100644
@@ -157,6 +157,8 @@ class domain
      set_misc_info
  # XEN_DOMCTL_set_virq_handler
      set_virq_handler
+# XEN_DOMCTL_set_max_evtchn
+    set_max_evtchn
  }
 
  # This is a continuation of class domain, since only 32 permissions can be
Quoted 2 lines
  # defined per class
  class domain2

The new domctl access vector must be added to the "domain2" class, not "domain" which is full (already has 32 items). While the hypervisor compilation does not currently report this as an error, attempting to compile the policy (tools/flask/policy) will report it.

Signature · 2 lines
Daniel De Graaf
National Security Agency
#13 lore raw

PATCH v4 10/11 libxc: add xc_domain_set_max_evtchn()

In reply to David Vrabel

From: David Vrabel <[email protected]>

Add xc_domain_set_max_evtchn(), a wrapper around the DOMCTL_set_max_evtchn hypercall.

Cc Ian Campbell <[email protected]>
Cc Ian Jackson <[email protected]>
---
 tools/libxc/xc_domain.c |   11 +++++++++++
 tools/libxc/xenctrl.h   |   12 ++++++++++++
 2 files changed, 23 insertions(+), 0 deletions(-)
tools/libxc/xc_domain.c +11 -0
index 81316d3..2cea6e3 100644
@@ -1766,6 +1766,17 @@ int xc_domain_set_virq_handler(xc_interface *xch, uint32_t domid, int virq)
     return do_domctl(xch, &domctl);
 }
 
+int xc_domain_set_max_evtchn(xc_interface *xch, uint32_t domid,
+                             uint32_t max_port)
+{
+    DECLARE_DOMCTL;
+
+    domctl.cmd = XEN_DOMCTL_set_max_evtchn;
+    domctl.domain = domid;
+    domctl.u.set_max_evtchn.max_port = max_port;
+    return do_domctl(xch, &domctl);
+}
+
 /*
  * Local variables:
  * mode: C
tools/libxc/xenctrl.h +12 -0
index 58d51f3..8cf3f3b 100644
@@ -847,6 +847,18 @@ int xc_domain_set_access_required(xc_interface *xch,
  */
 int xc_domain_set_virq_handler(xc_interface *xch, uint32_t domid, int virq);
 
+/**
+ * Set the maximum event channel port a domain may bind.
+ *
+ * This does not affect ports that are already bound.
+ *
+ * @param xch a handle to an open hypervisor interface
+ * @param domid the domain id
+ * @param max_port maximum port number
+ */
+int xc_domain_set_max_evtchn(xc_interface *xch, uint32_t domid,
+                             uint32_t max_port);
+
 /*
  * CPUPOOL MANAGEMENT FUNCTIONS
  */
Signature · 1 line
1.7.2.5
Acked-by on v4 10/11
#14 lore raw

Re: [PATCH 10/11] libxc: add xc_domain_set_max_evtchn()

Quoted 6 lines · David Vrabel

From: David Vrabel <[email protected]>

Add xc_domain_set_max_evtchn(), a wrapper around the DOMCTL_set_max_evtchn hypercall.

I'm happy for this to be committed by whoever takes the hypervisor side...

Quoted 51 lines
Cc Ian Jackson <[email protected]>
---
 tools/libxc/xc_domain.c |   11 +++++++++++
 tools/libxc/xenctrl.h   |   12 ++++++++++++
 2 files changed, 23 insertions(+), 0 deletions(-)
tools/libxc/xc_domain.c +11 -0
index 81316d3..2cea6e3 100644
@@ -1766,6 +1766,17 @@ int xc_domain_set_virq_handler(xc_interface *xch, uint32_t domid, int virq)
     return do_domctl(xch, &domctl);
 }
 
+int xc_domain_set_max_evtchn(xc_interface *xch, uint32_t domid,
+                             uint32_t max_port)
+{
+    DECLARE_DOMCTL;
+
+    domctl.cmd = XEN_DOMCTL_set_max_evtchn;
+    domctl.domain = domid;
+    domctl.u.set_max_evtchn.max_port = max_port;
+    return do_domctl(xch, &domctl);
+}
+
 /*
  * Local variables:
  * mode: C
tools/libxc/xenctrl.h +12 -0
index 58d51f3..8cf3f3b 100644
@@ -847,6 +847,18 @@ int xc_domain_set_access_required(xc_interface *xch,
  */
 int xc_domain_set_virq_handler(xc_interface *xch, uint32_t domid, int virq);
 
+/**
+ * Set the maximum event channel port a domain may bind.
+ *
+ * This does not affect ports that are already bound.
+ *
+ * @param xch a handle to an open hypervisor interface
+ * @param domid the domain id
+ * @param max_port maximum port number
+ */
+int xc_domain_set_max_evtchn(xc_interface *xch, uint32_t domid,
+                             uint32_t max_port);
+
 /*
  * CPUPOOL MANAGEMENT FUNCTIONS
  */
#15 lore raw

PATCH v4 11/11 libxl, xl: add event_channels option to xl configuration file

In reply to David Vrabel

From: David Vrabel <[email protected]>

Add the 'event_channels' option to the xl configuration file to limit the number of event channels that domain may use.

Plumb this option through to libxl via a new libxl_build_info field and call xc_domain_set_max_evtchn() in the post build stage of domain creation.

A new LIBXL_HAVE_BUILDINFO_EVENT_CHANNELS #define indicates that this new field is available.

The default value of 127 limits the domain to uses the minimum amount of Xen resources (xenheap and global mapping pages) regardless of event channel ABI that may be used by a guest.

Cc Ian Campbell <[email protected]>
Cc Ian Jackson <[email protected]>
---
 docs/man/xl.cfg.pod.5       |   12 ++++++++++++
 tools/libxl/libxl.h         |    5 +++++
 tools/libxl/libxl_create.c  |    3 +++
 tools/libxl/libxl_dom.c     |    4 ++++
 tools/libxl/libxl_types.idl |    1 +
 tools/libxl/xl_cmdimpl.c    |    3 +++
 6 files changed, 28 insertions(+), 0 deletions(-)
docs/man/xl.cfg.pod.5 +12 -0
index 769767b..3c7dd28 100644
@@ -572,6 +572,18 @@ Allow a guest to access specific physical IRQs.
 It is recommended to use this option only for trusted VMs under
 administrator control.
 
+=item B<event_channels=N>
+
+Limit the guest to using a most N event channels (PV interrupts).
+Guests use hypervisor resources for each event channel they use.
+
+The default of 127 should be sufficient for typical guests and means
+the guest uses the lowest amout of hypervisor resources.  The maximum
+value depends what the guest supports.  Guests supporting the
+FIFO-based event channel ABI support up to 131,071 event channels.
+Other guests are limited to 4095 (64-bit x86 and ARM) or 1023 (32-bit
+x86).
+
 =back
 
 =head2 Paravirtualised (PV) Guest Specific Options
tools/libxl/libxl.h +5 -0
index 4cab294..30712c2 100644
@@ -90,6 +90,11 @@
 #define LIBXL_HAVE_BUILDINFO_HVM_VENDOR_DEVICE 1
 
 /*
+ * The libxl_domain_build_info has the event_channels field.
+ */
+#define LIBXL_HAVE_BUILDINFO_EVENT_CHANNELS 1
+
+/*
  * libxl ABI compatibility
  *
  * The only guarantee which libxl makes regarding ABI compatibility
tools/libxl/libxl_create.c +3 -0
index 7567238..806e25c 100644
@@ -208,6 +208,9 @@ int libxl__domain_build_info_setdefault(libxl__gc *gc,
 
     libxl_defbool_setdefault(&b_info->disable_migrate, false);
 
+    if (!b_info->event_channels)
+        b_info->event_channels = 127;
+
     switch (b_info->type) {
     case LIBXL_DOMAIN_TYPE_HVM:
         if (b_info->shadow_memkb == LIBXL_MEMKB_DEFAULT)
tools/libxl/libxl_dom.c +4 -0
index 6e2252a..c905c74 100644
@@ -268,6 +268,10 @@ int libxl__build_post(libxl__gc *gc, uint32_t domid,
     if (rc)
         return rc;
 
+    rc = xc_domain_set_max_evtchn(ctx->xch, domid, info->event_channels);
+    if (rc)
+        return rc;
+
     libxl_cpuid_apply_policy(ctx, domid);
     if (info->cpuid != NULL)
         libxl_cpuid_set(ctx, domid, info->cpuid);
tools/libxl/libxl_types.idl +1 -0
index 049dbb5..7bf517d 100644
@@ -308,6 +308,7 @@ libxl_domain_build_info = Struct("domain_build_info",[
     ("irqs",             Array(uint32, "num_irqs")),
     ("iomem",            Array(libxl_iomem_range, "num_iomem")),
     ("claim_mode",	     libxl_defbool),
+    ("event_channels",   uint32),
     ("u", KeyedUnion(None, libxl_domain_type, "type",
                 [("hvm", Struct(None, [("firmware",         string),
                                        ("bios",             libxl_bios_type),
tools/libxl/xl_cmdimpl.c +3 -0
index 3d7eaad..7931bb9 100644
@@ -813,6 +813,9 @@ static void parse_config_data(const char *config_source,
     if (!xlu_cfg_get_long (config, "videoram", &l, 0))
         b_info->video_memkb = l * 1024;
 
+    if (!xlu_cfg_get_long(config, "event_channels", &l, 0))
+        b_info->event_channels = l;
+
     switch(b_info->type) {
     case LIBXL_DOMAIN_TYPE_HVM:
         if (!xlu_cfg_get_string (config, "kernel", &buf, 0))
Signature · 1 line
1.7.2.5
#16 lore raw

Re: [PATCH 11/11] libxl, xl: add event_channels option to xl configuration file

Quoted 5 lines · David Vrabel

From: David Vrabel <[email protected]>

Add the 'event_channels' option to the xl configuration file to limit the number of event channels that domain may use.

Quoted 8 lines

Plumb this option through to libxl via a new libxl_build_info field and call xc_domain_set_max_evtchn() in the post build stage of domain creation.

A new LIBXL_HAVE_BUILDINFO_EVENT_CHANNELS #define indicates that this new field is available.

The default value of 127 limits the domain to uses the minimum amount

"to use"

Quoted 20 lines

of Xen resources (xenheap and global mapping pages) regardless of event channel ABI that may be used by a guest.

Cc Ian Campbell <[email protected]>
Cc Ian Jackson <[email protected]>
---
 docs/man/xl.cfg.pod.5       |   12 ++++++++++++
 tools/libxl/libxl.h         |    5 +++++
 tools/libxl/libxl_create.c  |    3 +++
 tools/libxl/libxl_dom.c     |    4 ++++
 tools/libxl/libxl_types.idl |    1 +
 tools/libxl/xl_cmdimpl.c    |    3 +++
 6 files changed, 28 insertions(+), 0 deletions(-)
docs/man/xl.cfg.pod.5 +0 -0
index 769767b..3c7dd28 100644

docs/man/xl.cfg.pod.5 Allow a guest to access specific physical IRQs. · line 575

Quoted 4 lines
docs/man/xl.cfg.pod.5 +1 -0
@@ -572,6 +572,18 @@ Allow a guest to access specific physical IRQs.
 It is recommended to use this option only for trusted VMs under
 administrator control.
 
+=item B<event_channels=N>

Since this is actually a limit perhaps apply s/event_channels/max_&/ throughout?

docs/man/xl.cfg.pod.5 Allow a guest to access specific physical IRQs. · line 581

Quoted 6 lines
docs/man/xl.cfg.pod.5 +6 -0
+
+Limit the guest to using a most N event channels (PV interrupts).
+Guests use hypervisor resources for each event channel they use.
+
+The default of 127 should be sufficient for typical guests and means
+the guest uses the lowest amout of hypervisor resources.  The maximum

"amount"

docs/man/xl.cfg.pod.5 Allow a guest to access specific physical IRQs. · line 585

Quoted 4 lines
docs/man/xl.cfg.pod.5 +4 -0
+value depends what the guest supports.  Guests supporting the
+FIFO-based event channel ABI support up to 131,071 event channels.
+Other guests are limited to 4095 (64-bit x86 and ARM) or 1023 (32-bit
+x86).

Does this setting really restrict the classical ABI to 127? TBH I'm fine if it does, just curious.

Quoted 39 lines
docs/man/xl.cfg.pod.5 +1 -0
+
 =back
 
 =head2 Paravirtualised (PV) Guest Specific Options
tools/libxl/libxl.h +5 -0
index 4cab294..30712c2 100644
@@ -90,6 +90,11 @@
 #define LIBXL_HAVE_BUILDINFO_HVM_VENDOR_DEVICE 1
 
 /*
+ * The libxl_domain_build_info has the event_channels field.
+ */
+#define LIBXL_HAVE_BUILDINFO_EVENT_CHANNELS 1
+
+/*
  * libxl ABI compatibility
  *
  * The only guarantee which libxl makes regarding ABI compatibility
tools/libxl/libxl_create.c +3 -0
index 7567238..806e25c 100644
@@ -208,6 +208,9 @@ int libxl__domain_build_info_setdefault(libxl__gc *gc,
 
     libxl_defbool_setdefault(&b_info->disable_migrate, false);
 
+    if (!b_info->event_channels)
+        b_info->event_channels = 127;
+
     switch (b_info->type) {
     case LIBXL_DOMAIN_TYPE_HVM:
         if (b_info->shadow_memkb == LIBXL_MEMKB_DEFAULT)
tools/libxl/libxl_dom.c +0 -0
index 6e2252a..c905c74 100644

tools/libxl/libxl_dom.c int libxl__build_post(libxl__gc *gc, uint32_t domid, · line 273

Quoted 6 lines
tools/libxl/libxl_dom.c +3 -0
@@ -268,6 +268,10 @@ int libxl__build_post(libxl__gc *gc, uint32_t domid,
     if (rc)
         return rc;
 
+    rc = xc_domain_set_max_evtchn(ctx->xch, domid, info->event_channels);
+    if (rc)
+        return rc;

Is there anything sensible we can log here? Would be nice to give some clue as to why the domain creation failed...

Quoted 30 lines
tools/libxl/libxl_dom.c +1 -0
+
     libxl_cpuid_apply_policy(ctx, domid);
     if (info->cpuid != NULL)
         libxl_cpuid_set(ctx, domid, info->cpuid);
tools/libxl/libxl_types.idl +1 -0
index 049dbb5..7bf517d 100644
@@ -308,6 +308,7 @@ libxl_domain_build_info = Struct("domain_build_info",[
     ("irqs",             Array(uint32, "num_irqs")),
     ("iomem",            Array(libxl_iomem_range, "num_iomem")),
     ("claim_mode",	     libxl_defbool),
+    ("event_channels",   uint32),
     ("u", KeyedUnion(None, libxl_domain_type, "type",
                 [("hvm", Struct(None, [("firmware",         string),
                                        ("bios",             libxl_bios_type),
tools/libxl/xl_cmdimpl.c +3 -0
index 3d7eaad..7931bb9 100644
@@ -813,6 +813,9 @@ static void parse_config_data(const char *config_source,
     if (!xlu_cfg_get_long (config, "videoram", &l, 0))
         b_info->video_memkb = l * 1024;
 
+    if (!xlu_cfg_get_long(config, "event_channels", &l, 0))
+        b_info->event_channels = l;
+
     switch(b_info->type) {
     case LIBXL_DOMAIN_TYPE_HVM:
         if (!xlu_cfg_get_string (config, "kernel", &buf, 0))
#17 lore raw

Re: [PATCH 11/11] libxl, xl: add event_channels option to xl configuration file

Quoted 9 lines · Ian Campbell

docs/man/xl.cfg.pod.5 Allow a guest to access specific physical IRQs. · line 585

Quoted 4 lines · David Vrabel
docs/man/xl.cfg.pod.5 +4 -0
+value depends what the guest supports.  Guests supporting the
+FIFO-based event channel ABI support up to 131,071 event channels.
+Other guests are limited to 4095 (64-bit x86 and ARM) or 1023 (32-bit
+x86).

Does this setting really restrict the classical ABI to 127? TBH I'm fine if it does, just curious.

Yes, the limit applies regardless of the ABI that a guest uses.

With the 2-level ABI it doesn't save much Xen resources though (1 xen heap page per 128 events).

David

Reviewed-by on the v4 series
#18 lore raw
In reply to David Vrabel
Quoted 10 lines
Quoted 1 line
Quoted 1 line · David Vrabel

Patch 1-4 do some preparatory work for supporting alternate ABIs.

Patch 5 expands the number of evtchn objects a domain may have by changing how they are allocated.

Patch 6 adds the public ABI.

Patch 7 adds the EVTCHNOP_set_priority implementation. This will return -ENOSYS for ABIs that do not support priority.

Up to here

despite there still being a few minor coding style issues. One thing - as it is quite wide spread throughout the series - you may want to adjust is the use of "unsigned" when generally we use "unsigned int".

Jan

#19 lore raw
In reply to David Vrabel
Quoted 17 lines · David Vrabel

This is a complete implementation of the hypervisor and xl toolstack parts of the FIFO-based event channel ABI described in this design document:

http://xenbits.xen.org/people/dvrabel/event-channels-F.pdf

Changes in draft F are:

- READY field in the control block is now 32-bits (so guests only need to support atomic bit ops on 32-bit words). This is only a documentation change as the implementation already used a uint32_t.

- DOMCTL_set_max_evtchn replaces EVTCHNOP_set_limit.

- DomUs default to unlimited number of event channels requiring the toolstack to set a limit.

Quoted 6 lines

The toolstack defaults to limiting guests to 127 event channels if the event_channels option is omitted. This means the minimum amount of both Xen heap and global mapping space is used regardless of which ABI is used. If this is considered too restrictive a limit, 1023 would be another sensible default (limits the guest to a single event array page but 5 xenheap pages for the struct evtchns).

I would say 1023 (so the same value as the existing event mechanism) would be a sensible default.

Quoted 65 lines

An updated version of the Linux patch series is not quite ready yet. There is one remaining issue but fixing this will not require any changes to the hypervisor ABI or implementation. The remaining issue requires preallocating space in the evtchn-to-irq map as this cannot be expanded in pirq_startup() (since this function cannot return a failure).

The latest Linux changes can be found in the orochi-v4w branch of:

git://xenbits.xen.org/people/dvrabel/linux.git

Patch 1-4 do some preparatory work for supporting alternate ABIs.

Patch 5 expands the number of evtchn objects a domain may have by changing how they are allocated.

Patch 6 adds the public ABI.

Patch 7 adds the EVTCHNOP_set_priority implementation. This will return -ENOSYS for ABIs that do not support priority.

Patch 8 adds the FIFO-based ABI implementation.

Patch 9-10 adds the DOMCTL_set_max_evtchn implementation and adds a function to libxc. This will also work with the 2-level ABI.

Patch 11 add the event_channels configuration option to xl and libxl bits needed for this.

Changes in v4:

- Updates for Draft F of the design.
- DOMCTL_set_max_evtchn replaces EVTCHNOP_set_limit.
- Hypervisor defaults to unlimited event channels for for DomU.
- Optimized memory allocation for struct evtchn's when fewer than 128 are required (see patch 5).
- Added event_channels option to the xl domain configuration file and plumbed this through libxl_build_info. Defaults to 127.

Changes in v3:

- Updates for Draft E of the design.
- Store priority in struct evtchn.
- Implement set_priority with generic code + hook.
- Implement set_limit and add libxc function.
- Add ABI specific output to 'e' debug key.

Changes in v2:

- Updates for Draft D of the design.
- 130,000+ event channels are now supported.
- event_port.c -> event_2l.c and only contains 2l functions.
- Addresses various review comments
- int -> unsigned in lots of places
- use write_atomic() to set HEAD
- removed MAX_EVTCHNS
- evtchn_ops are const.
- Pack struct evtchns better to reduce memory needed.

_______________________________________________
Xen-devel mailing list
[email protected]
http://lists.xen.org/xen-devel

#20 lore raw
Quoted 26 lines
Quoted 1 line
Quoted 1 line · Konrad Rzeszutek Wilk
Quoted 17 lines · David Vrabel

This is a complete implementation of the hypervisor and xl toolstack parts of the FIFO-based event channel ABI described in this design document:

http://xenbits.xen.org/people/dvrabel/event-channels-F.pdf

Changes in draft F are:

- READY field in the control block is now 32-bits (so guests only need to support atomic bit ops on 32-bit words). This is only a documentation change as the implementation already used a uint32_t.

- DOMCTL_set_max_evtchn replaces EVTCHNOP_set_limit.

- DomUs default to unlimited number of event channels requiring the toolstack to set a limit.

Quoted 6 lines

The toolstack defaults to limiting guests to 127 event channels if the event_channels option is omitted. This means the minimum amount of both Xen heap and global mapping space is used regardless of which ABI is used. If this is considered too restrictive a limit, 1023 would be another sensible default (limits the guest to a single event array page but 5 xenheap pages for the struct evtchns).

Quoted 2 lines

I would say 1023 (so the same value as the existing event mechanism) would be a sensible default.

That's the existing 32-bit default; 64-bit has 4095 (yet that surely would be needlessly high as the new default).

Jan

#21 lore raw
Quoted 26 lines · Jan Beulich
Quoted 26 lines
Quoted 1 line
Quoted 1 line · Konrad Rzeszutek Wilk
Quoted 23 lines · David Vrabel

This is a complete implementation of the hypervisor and xl toolstack parts of the FIFO-based event channel ABI described in this design document:

http://xenbits.xen.org/people/dvrabel/event-channels-F.pdf

Changes in draft F are:

- READY field in the control block is now 32-bits (so guests only need to support atomic bit ops on 32-bit words). This is only a documentation change as the implementation already used a uint32_t.

- DOMCTL_set_max_evtchn replaces EVTCHNOP_set_limit.

- DomUs default to unlimited number of event channels requiring the toolstack to set a limit.

The toolstack defaults to limiting guests to 127 event channels if the event_channels option is omitted. This means the minimum amount of both Xen heap and global mapping space is used regardless of which ABI is used. If this is considered too restrictive a limit, 1023 would be another sensible default (limits the guest to a single event array page but 5 xenheap pages for the struct evtchns).

Quoted 5 lines
Quoted 2 lines

I would say 1023 (so the same value as the existing event mechanism) would be a sensible default.

That's the existing 32-bit default; 64-bit has 4095 (yet that surely would be needlessly high as the new default).

127 is too little I think. For example for every VCPU there are 6 events being consumed (VIRQ_TIMER, VIRQ_DEBUG, CALLFUNCSINGLE, CALLFUNC, RESCHED and IRQWORK). If you launch a 32 VCPU guest you are already at 224.

Then there is the blk event channel, the tx/rx of the vif. With the possibility of per-cpu tx/rx of vifs you would have 2*VCPU, so now we are at 288.

If you want even more LUNS (say 16), you are at 304.

1023 being the universal value looks OK to me.

Quoted 3 lines

Jan

#22 lore raw
Quoted 28 lines
Quoted 1 line
Quoted 1 line · Konrad Rzeszutek Wilk
Quoted 26 lines · Jan Beulich
Quoted 26 lines
Quoted 1 line
Quoted 1 line · Konrad Rzeszutek Wilk
Quoted 23 lines · David Vrabel

This is a complete implementation of the hypervisor and xl toolstack parts of the FIFO-based event channel ABI described in this design document:

http://xenbits.xen.org/people/dvrabel/event-channels-F.pdf

Changes in draft F are:

- READY field in the control block is now 32-bits (so guests only need to support atomic bit ops on 32-bit words). This is only a documentation change as the implementation already used a uint32_t.

- DOMCTL_set_max_evtchn replaces EVTCHNOP_set_limit.

- DomUs default to unlimited number of event channels requiring the toolstack to set a limit.

The toolstack defaults to limiting guests to 127 event channels if the event_channels option is omitted. This means the minimum amount of both Xen heap and global mapping space is used regardless of which ABI is used. If this is considered too restrictive a limit, 1023 would be another sensible default (limits the guest to a single event array page but 5 xenheap pages for the struct evtchns).

Quoted 7 lines
Quoted 5 lines
Quoted 2 lines

I would say 1023 (so the same value as the existing event mechanism) would be a sensible default.

That's the existing 32-bit default; 64-bit has 4095 (yet that surely would be needlessly high as the new default).

127 is too little I think.

I agree; I'm fine with defaulting to 1023 (I only wanted to point out that other than you claimed this is lower than the 2-level default on 64-bit guests). Perhaps the tools could even be intelligent enough to make the default depend on the vCPU count of the guest.

Quoted 3 lines

For example for every VCPU there are 6 events
being consumed (VIRQ_TIMER, VIRQ_DEBUG, CALLFUNCSINGLE, CALLFUNC, RESCHED and IRQWORK). If you launch a 32 VCPU guest you are already at 224.

Because you waste them - all the IPI flavors could collectively do with just one event channel per vCPU (as our more recent kernels do).

You of course need a separate one for the timer, and you forgot the spin lock polling one. Whether the VIRQ_DEBUG one is always necessary I'm not sure - I would think the kernel should by default avoid registering it.

Jan