XSA-343 followup patches
v5 cover XSA-343 followup patches
The patches for XSA-343 produced some fallout, especially the event channel locking has shown to be problematic.
Patch 1 is targeting fifo event channels for avoiding any races for the case that the fifo queue has been changed for a specific event channel.
The second patch is modifying the per event channel locking scheme in order to avoid deadlocks and problems due to the event channel lock having been changed to require IRQs off by the XSA-343 patches.
Changes in V5:
- moved evtchn_write_[un]lock() to event_channel.c (Jan Beulich)
- used normal read_lock() in some cases (Jan Beulich)
Changes in V4:
- switched to real rwlock
Changes in V3:
- addressed comments
Juergen Gross (2):
xen/events: access last_priority and last_vcpu_id together xen/evtchn: rework per event channel lock
xen/arch/x86/irq.c | 6 +- xen/arch/x86/pv/shim.c | 9 +-- xen/common/event_channel.c | 144 +++++++++++++++++++++---------------- xen/common/event_fifo.c | 25 +++++-- xen/include/xen/event.h | 32 ++++++--- xen/include/xen/sched.h | 8 ++- 6 files changed, 136 insertions(+), 88 deletions(-)
Signature · 1 line
2.26.2
PATCH v5 1/2 xen/events: access last_priority and last_vcpu_id together
The queue for a fifo event is depending on the vcpu_id and the priority of the event. When sending an event it might happen the event needs to change queues and the old queue needs to be kept for keeping the links between queue elements intact. For this purpose the event channel contains last_priority and last_vcpu_id values elements for being able to identify the old queue.
In order to avoid races always access last_priority and last_vcpu_id with a single atomic operation avoiding any inconsistencies.
--- xen/common/event_fifo.c | 25 +++++++++++++++++++------ xen/include/xen/sched.h | 3 +-- 2 files changed, 20 insertions(+), 8 deletions(-)
xen/common/event_fifo.c +19 -6
@@ -42,6 +42,14 @@ struct evtchn_fifo_domain {unsigned int num_evtchns; }; +union evtchn_fifo_lastq { +uint32_t raw; +struct { +uint8_t last_priority; +uint16_t last_vcpu_id; + }; +}; +static inline event_word_t *evtchn_fifo_word_from_port (const struct domain *d,unsigned int port) {
@@ -86,16 +94,18 @@ static struct evtchn_fifo_queue *lock_old_queue(const struct domain *d,struct vcpu *v;struct evtchn_fifo_queue *q, *old_q;unsigned int try; +union evtchn_fifo_lastq lastq;for ( try =0 ; try <3 ; try++ ) { - v = d->vcpu[evtchn->last_vcpu_id]; - old_q = &v->evtchn_fifo->queue[evtchn->last_priority]; + lastq.raw =read_atomic (&evtchn->fifo_lastq); + v = d->vcpu[lastq.last_vcpu_id]; + old_q = &v->evtchn_fifo->queue[lastq.last_priority];spin_lock_irqsave (&old_q->lock, *flags); - v = d->vcpu[evtchn->last_vcpu_id]; - q = &v->evtchn_fifo->queue[evtchn->last_priority]; + v = d->vcpu[lastq.last_vcpu_id]; + q = &v->evtchn_fifo->queue[lastq.last_priority];if ( old_q == q )return old_q;
@@ -246,8 +256,11 @@ static void evtchn_fifo_set_pending(struct vcpu *v, struct evtchn *evtchn)/* Moved to a different queue? */ if ( old_q != q ) { - evtchn->last_vcpu_id = v->vcpu_id; - evtchn->last_priority = q->priority; +union evtchn_fifo_lastq lastq = { }; + + lastq.last_vcpu_id = v->vcpu_id; + lastq.last_priority = q->priority; +write_atomic (&evtchn->fifo_lastq, lastq.raw);spin_unlock_irqrestore (&old_q->lock, flags);spin_lock_irqsave (&q->lock, flags);
xen/include/xen/sched.h +1 -2
@@ -114,8 +114,7 @@ struct evtchnu16 virq;/* state == ECS_VIRQ */ } u;u8 priority; -u8 last_priority; -u16 last_vcpu_id; +u32 fifo_lastq;/* Data for fifo events identifying last queue. */ #ifdef CONFIG_XSM union { #ifdef XSM_NEED_GENERIC_EVTCHN_SSID
Signature · 1 line
2.26.2
PATCH v5 2/2 xen/evtchn: rework per event channel lock
In reply to Juergen GrossCurrently the lock for a single event channel needs to be taken with interrupts off, which causes deadlocks in some cases.
Rework the per event channel lock to be non-blocking for the case of sending an event and removing the need for disabling interrupts for taking the lock.
The lock is needed for avoiding races between event channel state changes (creation, closing, binding) against normal operations (set pending, [un]masking, priority changes).
Use a rwlock, but with some restrictions:
- normal operations use read_trylock(), in case of not obtaining the lock the operation is omitted or a default state is returned
- closing an event channel is using write_lock(), with ASSERT()ing that the lock is taken as writer only when the state of the event channel is either before or after the locked region appropriate (either free or unbound).
---
V4:
- switch to rwlock
- add ASSERT() to verify correct write_lock() usage
V3:
- corrected a copy-and-paste error (Jan Beulich)
- corrected unlocking in two cases (Jan Beulich)
- renamed evtchn_read_trylock() (Jan Beulich)
- added some comments and an ASSERT() for evtchn_write_lock()
- set EVENT_WRITE_LOCK_INC to INT_MIN
V2:
- added needed barriers
--- xen/arch/x86/irq.c | 6 +- xen/arch/x86/pv/shim.c | 9 +-- xen/common/event_channel.c | 144 +++++++++++++++++++++---------------- xen/include/xen/event.h | 32 ++++++--- xen/include/xen/sched.h | 5 +- 5 files changed, 116 insertions(+), 80 deletions(-)
xen/arch/x86/irq.c +2 -4
@@ -2495,14 +2495,12 @@ static void dump_irqs(unsigned char key) pirq =domain_irq_to_pirq (d, irq); info =pirq_info (d, pirq); evtchn =evtchn_from_port (d, info->evtchn); -local_irq_disable (); -if (spin_trylock (&evtchn->lock) ) +if (evtchn_read_trylock (evtchn) ) { pending =evtchn_is_pending (d, evtchn); masked =evtchn_is_masked (d, evtchn); -spin_unlock (&evtchn->lock); +evtchn_read_unlock (evtchn); } -local_irq_enable (); printk("d%d:%3d(%c%c%c)%c", d->domain_id, pirq, "-P?"[pending], "-M?"[masked], info->masked ? 'M' : '-',
xen/arch/x86/pv/shim.c +5 -4
@@ -660,11 +660,12 @@ void pv_shim_inject_evtchn(unsigned int port)if (port_is_valid (guest, port) ) {struct evtchn *chn =evtchn_from_port (guest, port); -unsigned long flags; -spin_lock_irqsave (&chn->lock, flags); -evtchn_port_set_pending (guest, chn->notify_vcpu_id, chn); -spin_unlock_irqrestore (&chn->lock, flags); +if (evtchn_read_trylock (chn) ) + { +evtchn_port_set_pending (guest, chn->notify_vcpu_id, chn); +evtchn_read_unlock (chn); + } } }
xen/common/event_channel.c +83 -61
389 lines not on this page · raw message
xen/include/xen/event.h +22 -10
@@ -105,6 +105,16 @@ void notify_via_xen_event_channel(struct domain *ld, int lport);#define bucket_from_port (d, p) \ ((group_from_port(d, p))[((p) % EVTCHNS_PER_GROUP) / EVTCHNS_PER_BUCKET]) +static inline bool evtchn_read_trylock (struct evtchn *evtchn) +{ +return read_trylock (&evtchn->lock); +} + +static inline void evtchn_read_unlock (struct evtchn *evtchn) +{ +read_unlock (&evtchn->lock); +} +static inline bool_t port_is_valid (struct domain *d,unsigned int p) {if ( p >=read_atomic (&d->valid_evtchns) )
@@ -234,12 +244,13 @@ static inline bool evtchn_is_masked(const struct domain *d,static inline bool evtchn_port_is_masked (struct domain *d,evtchn_port_t port) {struct evtchn *evtchn =evtchn_from_port (d, port); -bool rc; -unsigned long flags; +bool rc = true; -spin_lock_irqsave (&evtchn->lock, flags); - rc =evtchn_is_masked (d, evtchn); -spin_unlock_irqrestore (&evtchn->lock, flags); +if (evtchn_read_trylock (evtchn) ) + { + rc =evtchn_is_masked (d, evtchn); +evtchn_read_unlock (evtchn); + }return rc; }
@@ -252,12 +263,13 @@ static inline int evtchn_port_poll(struct domain *d, evtchn_port_t port)if (port_is_valid (d, port) ) {struct evtchn *evtchn =evtchn_from_port (d, port); -unsigned long flags; -spin_lock_irqsave (&evtchn->lock, flags); -if (evtchn_usable (evtchn) ) - rc =evtchn_is_pending (d, evtchn); -spin_unlock_irqrestore (&evtchn->lock, flags); +if (evtchn_read_trylock (evtchn) ) + { +if (evtchn_usable (evtchn) ) + rc =evtchn_is_pending (d, evtchn); +evtchn_read_unlock (evtchn); + } }return rc;
xen/include/xen/sched.h +4 -1
@@ -85,7 +85,7 @@ extern domid_t hardware_domid;struct evtchn { -spinlock_t lock; +rwlock_t lock;#define ECS_FREE 0/* Channel is available for use. */ #define ECS_RESERVED 1/* Channel is reserved. */ #define ECS_UNBOUND 2/* Channel is waiting to bind to a remote domain. */
@@ -114,6 +114,9 @@ struct evtchnu16 virq;/* state == ECS_VIRQ */ } u;u8 priority; +#ifndef NDEBUG +u8 old_state;/* State when taking lock in write mode. */ +#endif u32 fifo_lastq;/* Data for fifo events identifying last queue. */ #ifdef CONFIG_XSM union {
Signature · 1 line
2.26.2
Re: [PATCH v5 2/2] xen/evtchn: rework per event channel lock
Quoted 13 lines · Juergen Gross
Currently the lock for a single event channel needs to be taken with interrupts off, which causes deadlocks in some cases.
Rework the per event channel lock to be non-blocking for the case of sending an event and removing the need for disabling interrupts for taking the lock.
The lock is needed for avoiding races between event channel state changes (creation, closing, binding) against normal operations (set pending, [un]masking, priority changes).
Use a rwlock, but with some restrictions:
Quoted 7 lines
- normal operations use
read_trylock(), in case of not obtaining the lock the operation is omitted or a default state is returned- closing an event channel is using
write_lock(), with ASSERT()ing that the lock is taken as writer only when the state of the event channel is either before or after the locked region appropriate (either free or unbound).
This has become stale, and may have been incomplete already before:
- Normal operations now may use two diffeent approaches. Which one is to be used when would want writing down here.
- write_lock() use goes beyond closing.
Quoted 9 lines
xen/arch/x86/irq.c +1
-2@@ -2495,14 +2495,12 @@ static void dump_irqs(unsigned char key) pirq =domain_irq_to_pirq (d, irq); info =pirq_info (d, pirq); evtchn =evtchn_from_port (d, info->evtchn); -local_irq_disable (); -if (spin_trylock (&evtchn->lock) ) +if (evtchn_read_trylock (evtchn) )
xen/arch/x86/irq.c static void dump_irqs(unsigned char key) · line 2506
Quoted 10 lines
xen/arch/x86/irq.c +1
-2{ pending =evtchn_is_pending (d, evtchn); masked =evtchn_is_masked (d, evtchn); -spin_unlock (&evtchn->lock); +evtchn_read_unlock (evtchn); } -local_irq_enable (); printk("d%d:%3d(%c%c%c)%c", d->domain_id, pirq, "-P?"[pending], "-M?"[masked], info->masked ? 'M' : '-',
Using trylock here has a reason different from that in sending functions, aiui. Please say so in the description, to justify exposure of evtchn_read_lock().
Quoted 6 lines
xen/arch/x86/pv/shim.c +0
-0@@ -660,11 +660,12 @@ void pv_shim_inject_evtchn(unsigned int port)if (port_is_valid (guest, port) ) {struct evtchn *chn =evtchn_from_port (guest, port);
xen/arch/x86/pv/shim.c void pv_shim_inject_evtchn(unsigned int port) · line 668
Quoted 10 lines
xen/arch/x86/pv/shim.c +5
-4-unsigned long flags; -spin_lock_irqsave (&chn->lock, flags); -evtchn_port_set_pending (guest, chn->notify_vcpu_id, chn); -spin_unlock_irqrestore (&chn->lock, flags); +if (evtchn_read_trylock (chn) ) + { +evtchn_port_set_pending (guest, chn->notify_vcpu_id, chn); +evtchn_read_unlock (chn); + }
Does this need trylock?
Quoted 8 lines
xen/arch/x86/pv/shim.c +0
-1@@ -1068,15 +1088,16 @@ int evtchn_unmask(unsigned int port) {struct domain *d = current->domain;struct evtchn *evtchn; -unsigned long flags;if (unlikely (!port_is_valid (d, port)) )return -EINVAL ;
xen/arch/x86/pv/shim.c int evtchn_unmask(unsigned int port) · line 1100
Quoted 10 lines
xen/arch/x86/pv/shim.c +5
-3evtchn =evtchn_from_port (d, port); -spin_lock_irqsave (&evtchn->lock, flags); -evtchn_port_unmask (d, evtchn); -spin_unlock_irqrestore (&evtchn->lock, flags); +if (evtchn_read_trylock (evtchn) ) + { +evtchn_port_unmask (d, evtchn); +evtchn_read_unlock (evtchn); + }
I think this one could as well use plain read_lock().
Quoted 28 lines
xen/arch/x86/pv/shim.c +6
-9@@ -234,12 +244,13 @@ static inline bool evtchn_is_masked(const struct domain *d,static inline bool evtchn_port_is_masked (struct domain *d,evtchn_port_t port) {struct evtchn *evtchn =evtchn_from_port (d, port); -bool rc; -unsigned long flags; +bool rc = true; -spin_lock_irqsave (&evtchn->lock, flags); - rc =evtchn_is_masked (d, evtchn); -spin_unlock_irqrestore (&evtchn->lock, flags); +if (evtchn_read_trylock (evtchn) ) + { + rc =evtchn_is_masked (d, evtchn); +evtchn_read_unlock (evtchn); + }return rc; }@@ -252,12 +263,13 @@ static inline int evtchn_port_poll(struct domain *d, evtchn_port_t port)if (port_is_valid (d, port) ) {struct evtchn *evtchn =evtchn_from_port (d, port); -unsigned long flags; -spin_lock_irqsave (&evtchn->lock, flags); -if (evtchn_usable (evtchn) ) - rc =evtchn_is_pending (d, evtchn);
xen/arch/x86/pv/shim.c static inline int evtchn_port_poll(struct domain *d, evtchn_port_t port) · line 275
Quoted 10 lines
xen/arch/x86/pv/shim.c +6
-1-spin_unlock_irqrestore (&evtchn->lock, flags); +if (evtchn_read_trylock (evtchn) ) + { +if (evtchn_usable (evtchn) ) + rc =evtchn_is_pending (d, evtchn); +evtchn_read_unlock (evtchn); + } }return rc;
At least for the latter I suppose it should also be plain read_lock(). We ought to keep the exceptions to where they're actually needed, I think.
Jan
Re: [PATCH v5 2/2] xen/evtchn: rework per event channel lock
Quoted 22 lines · Jan Beulich
Quoted 13 lines · Juergen Gross
Currently the lock for a single event channel needs to be taken with interrupts off, which causes deadlocks in some cases.
Rework the per event channel lock to be non-blocking for the case of sending an event and removing the need for disabling interrupts for taking the lock.
The lock is needed for avoiding races between event channel state changes (creation, closing, binding) against normal operations (set pending, [un]masking, priority changes).
Use a rwlock, but with some restrictions:
Quoted 7 lines
- normal operations use
read_trylock(), in case of not obtaining the lock the operation is omitted or a default state is returned- closing an event channel is using
write_lock(), with ASSERT()ing that the lock is taken as writer only when the state of the event channel is either before or after the locked region appropriate (either free or unbound).
Quoted 4 lines
This has become stale, and may have been incomplete already before:
- Normal operations now may use two diffeent approaches. Which one is to be used when would want writing down here.
-write_lock()use goes beyond closing.
Yes, you are right.
Quoted 16 lines
Quoted 16 lines
xen/arch/x86/irq.c +2
-3@@ -2495,14 +2495,12 @@ static void dump_irqs(unsigned char key) pirq =domain_irq_to_pirq (d, irq); info =pirq_info (d, pirq); evtchn =evtchn_from_port (d, info->evtchn); -local_irq_disable (); -if (spin_trylock (&evtchn->lock) ) +if (evtchn_read_trylock (evtchn) ) { pending =evtchn_is_pending (d, evtchn); masked =evtchn_is_masked (d, evtchn); -spin_unlock (&evtchn->lock); +evtchn_read_unlock (evtchn); }
Quoted 8 lines
xen/arch/x86/irq.c static void dump_irqs(unsigned char key) · line 2506
Quoted 4 lines
xen/arch/x86/irq.c +0
-1-local_irq_enable (); printk("d%d:%3d(%c%c%c)%c", d->domain_id, pirq, "-P?"[pending], "-M?"[masked], info->masked ? 'M' : '-',Using trylock here has a reason different from that in sending functions, aiui. Please say so in the description, to justify exposure of
evtchn_read_lock().
Okay.
Quoted 9 lines
Quoted 9 lines
xen/arch/x86/pv/shim.c +0
-1@@ -660,11 +660,12 @@ void pv_shim_inject_evtchn(unsigned int port)if (port_is_valid (guest, port) ) {struct evtchn *chn =evtchn_from_port (guest, port); -unsigned long flags;
Quoted 10 lines
xen/arch/x86/pv/shim.c void pv_shim_inject_evtchn(unsigned int port) · line 668
Quoted 8 lines
xen/arch/x86/pv/shim.c +5
-3-spin_lock_irqsave (&chn->lock, flags); -evtchn_port_set_pending (guest, chn->notify_vcpu_id, chn); -spin_unlock_irqrestore (&chn->lock, flags); +if (evtchn_read_trylock (chn) ) + { +evtchn_port_set_pending (guest, chn->notify_vcpu_id, chn); +evtchn_read_unlock (chn); + }Does this need trylock?
It is called directly from the event upcall, so interrupts should be off here. Without trylock this would result in check_lock() triggering.
Quoted 10 lines
Quoted 10 lines
xen/arch/x86/pv/shim.c +0
-1@@ -1068,15 +1088,16 @@ int evtchn_unmask(unsigned int port) {struct domain *d = current->domain;struct evtchn *evtchn; -unsigned long flags;if (unlikely (!port_is_valid (d, port)) )return -EINVAL ;
Quoted 11 lines
xen/arch/x86/pv/shim.c int evtchn_unmask(unsigned int port) · line 1100
Quoted 9 lines
xen/arch/x86/pv/shim.c +5
-3evtchn =evtchn_from_port (d, port); -spin_lock_irqsave (&evtchn->lock, flags); -evtchn_port_unmask (d, evtchn); -spin_unlock_irqrestore (&evtchn->lock, flags); +if (evtchn_read_trylock (evtchn) ) + { +evtchn_port_unmask (d, evtchn); +evtchn_read_unlock (evtchn); + }I think this one could as well use plain
read_lock().
Oh, indeed.
Quoted 38 lines
Quoted 38 lines
xen/arch/x86/pv/shim.c +12
-10@@ -234,12 +244,13 @@ static inline bool evtchn_is_masked(const struct domain *d,static inline bool evtchn_port_is_masked (struct domain *d,evtchn_port_t port) {struct evtchn *evtchn =evtchn_from_port (d, port); -bool rc; -unsigned long flags; +bool rc = true; -spin_lock_irqsave (&evtchn->lock, flags); - rc =evtchn_is_masked (d, evtchn); -spin_unlock_irqrestore (&evtchn->lock, flags); +if (evtchn_read_trylock (evtchn) ) + { + rc =evtchn_is_masked (d, evtchn); +evtchn_read_unlock (evtchn); + }return rc; }@@ -252,12 +263,13 @@ static inline int evtchn_port_poll(struct domain *d, evtchn_port_t port)if (port_is_valid (d, port) ) {struct evtchn *evtchn =evtchn_from_port (d, port); -unsigned long flags; -spin_lock_irqsave (&evtchn->lock, flags); -if (evtchn_usable (evtchn) ) - rc =evtchn_is_pending (d, evtchn); -spin_unlock_irqrestore (&evtchn->lock, flags); +if (evtchn_read_trylock (evtchn) ) + { +if (evtchn_usable (evtchn) ) + rc =evtchn_is_pending (d, evtchn); +evtchn_read_unlock (evtchn); + } }
Quoted 5 lines
xen/arch/x86/pv/shim.c static inline int evtchn_port_poll(struct domain *d, evtchn_port_t port) · line 275
Quoted 1 line
xen/arch/x86/pv/shim.c +0
-0return rc;At least for the latter I suppose it should also be plain
read_lock(). We ought to keep the exceptions to where they're actually needed, I think.
I think both instances can be switched.
Juergen
Re: [PATCH v5 2/2] xen/evtchn: rework per event channel lock
Quoted 21 lines · Jürgen Groß
Quoted 9 lines · Jan Beulich
Quoted 8 lines · Juergen Gross
xen/arch/x86/pv/shim.c +0
-1@@ -660,11 +660,12 @@ void pv_shim_inject_evtchn(unsigned int port)if (port_is_valid (guest, port) ) {struct evtchn *chn =evtchn_from_port (guest, port); -unsigned long flags;Quoted 10 lines
xen/arch/x86/pv/shim.c void pv_shim_inject_evtchn(unsigned int port) · line 668
Quoted 8 lines
xen/arch/x86/pv/shim.c +5
-3-spin_lock_irqsave (&chn->lock, flags); -evtchn_port_set_pending (guest, chn->notify_vcpu_id, chn); -spin_unlock_irqrestore (&chn->lock, flags); +if (evtchn_read_trylock (chn) ) + { +evtchn_port_set_pending (guest, chn->notify_vcpu_id, chn); +evtchn_read_unlock (chn); + }Does this need trylock?
Quoted 2 lines
It is called directly from the event upcall, so interrupts should be off here. Without trylock this would result in
check_lock()triggering.
Hmm, right. Since the trylock function needs exposing anyway, this isn't much of a problem, the more that it fits the pattern of being a send.
Jan
Quoted 17 lines · Juergen Gross
The patches for XSA-343 produced some fallout, especially the event channel locking has shown to be problematic.
Patch 1 is targeting fifo event channels for avoiding any races for the case that the fifo queue has been changed for a specific event channel.
The second patch is modifying the per event channel locking scheme in order to avoid deadlocks and problems due to the event channel lock having been changed to require IRQs off by the XSA-343 patches.
Changes in V5:
- moved evtchn_write_[un]lock() toevent_channel.c(Jan Beulich)
- used normalread_lock()in some cases (Jan Beulich)Changes in V4:
- switched to real rwlock
Quoted 6 lines
Changes in V3:
- addressed commentsJuergen Gross (2):
xen/events: accesslast_priorityandlast_vcpu_idtogether xen/evtchn: rework per event channel lock
Didn't you mean to add a 3rd patch here to drop the 2nd call to xsm_evtchn_send() again?
Jan
Quoted 21 lines · Jan Beulich
Quoted 20 lines · Juergen Gross
The patches for XSA-343 produced some fallout, especially the event channel locking has shown to be problematic.
Patch 1 is targeting fifo event channels for avoiding any races for the case that the fifo queue has been changed for a specific event channel.
The second patch is modifying the per event channel locking scheme in order to avoid deadlocks and problems due to the event channel lock having been changed to require IRQs off by the XSA-343 patches.
Changes in V5:
- moved evtchn_write_[un]lock() toevent_channel.c(Jan Beulich)
- used normalread_lock()in some cases (Jan Beulich)Changes in V4:
- switched to real rwlockChanges in V3:
- addressed comments
Quoted 6 lines
Quoted 3 lines
Juergen Gross (2):
xen/events: accesslast_priorityandlast_vcpu_idtogether xen/evtchn: rework per event channel lockDidn't you mean to add a 3rd patch here to drop the 2nd call to
xsm_evtchn_send()again?
I wanted to that after this series has been accepted, but I can include it here, of course.
Juergen