xen/rcu: let rcu work better with core scheduling
PATCH v4 2/6 xen/rcu: don't use stop_machine_run() for rcu_barrier()
In reply to Juergen GrossToday rcu_barrier() is calling stop_machine_run() to synchronize all physical cpus in order to ensure all pending rcu calls have finished when returning.
As stop_machine_run() is using tasklets this requires scheduling of idle vcpus on all cpus imposing the need to call rcu_barrier() on idle cpus only in case of core scheduling being active, as otherwise a scheduling deadlock would occur.
There is no need at all to do the syncing of the cpus in tasklets, as rcu activity is started in __do_softirq() called whenever softirq activity is allowed. So rcu_barrier() can easily be modified to use softirq for synchronization of the cpus no longer requiring any scheduling activity.
As there already is a rcu softirq reuse that for the synchronization.
Remove the barrier element from struct rcu_data as it isn't used.
Finally switch rcu_barrier() to return void as it now can never fail.
---
V2:
- add recursion detection
V3:
- fix races (Igor Druzhinin)
--- xen/common/rcupdate.c | 85 +++++++++++++++++++++++++++++++--------------- xen/include/xen/rcupdate.h | 2 +- 2 files changed, 59 insertions(+), 28 deletions(-)
xen/common/rcupdate.c +58 -27
@@ -83,7 +83,6 @@ struct rcu_data {struct rcu_head **donetail;long blimit;/* Upper limit on a processed batch */ int cpu; -struct rcu_head barrier;long last_rs_qlen;/* qlen during the last resched */ /* 3) idle CPUs handling */
@@ -91,6 +90,7 @@ struct rcu_data {bool idle_timer_active;bool process_callbacks; +bool barrier_active; };/*
@@ -143,51 +143,75 @@ static int qhimark = 10000;static int qlowmark =100 ;static int rsinterval =1000 ; -struct rcu_barrier_data { -struct rcu_head head; -atomic_t *cpu_count; -}; +/* +* rcu_barrier() handling: +* cpu_count holds the number of cpu required to finish barrier handling. +* Cpus are synchronized via softirq mechanism. rcu_barrier() is regarded to +* be active if cpu_count is not zero. In case rcu_barrier() is called on +* multiple cpus it is enough to check for cpu_count being not zero on entry +* and to call process_pending_softirqs() in a loop until cpu_count drops to +* zero, as syncing has been requested already and we don't need to sync +* multiple times. +* In order to avoid hangs when rcu_barrier() is called mutiple times on the +* same cpu in fast sequence and a slave cpu couldn't drop out of the +* barrier handling fast enough a second counter done_count is needed. +*/ +static atomic_t cpu_count =ATOMIC_INIT (0 ); +static atomic_t done_count =ATOMIC_INIT (0 );static void rcu_barrier_callback (struct rcu_head *head) { -struct rcu_barrier_data *data =container_of ( - head, struct rcu_barrier_data, head); -atomic_inc (data->cpu_count); +atomic_dec (&cpu_count); } -static int rcu_barrier_action (void *_cpu_count) +static void rcu_barrier_action (void ) { -struct rcu_barrier_data data = { .cpu_count = _cpu_count }; - -ASSERT (!local_irq_is_enabled ()); -local_irq_enable (); +struct rcu_head head;/* * When callback is executed, all previously-queued RCU work on this CPU -* is completed. When all CPUs have executed their callback, data.cpu_count -* will have been incremented to include every online CPU. +* is completed. When all CPUs have executed their callback, cpu_count +* will have been decremented to 0. */ -call_rcu (&data.head, rcu_barrier_callback); +call_rcu (&head, rcu_barrier_callback); -while (atomic_read (data.cpu_count) !=num_online_cpus () ) +while (atomic_read (&cpu_count) ) {process_pending_softirqs ();cpu_relax (); } -local_irq_disable (); - -return 0 ; +atomic_dec (&done_count); } -/* -* As rcu_barrier() is using stop_machine_run() it is allowed to be used in -* idle context only (see comment for stop_machine_run()). -*/ -int rcu_barrier (void ) +void rcu_barrier (void ) { -atomic_t cpu_count =ATOMIC_INIT (0 ); -return stop_machine_run (rcu_barrier_action, &cpu_count,NR_CPUS ); +unsigned int n_cpus; + +while ( !get_cpu_maps () ) + { +process_pending_softirqs (); +if ( !atomic_read (&cpu_count) ) +return ; + +cpu_relax (); + } + + n_cpus =num_online_cpus (); + +if (atomic_cmpxchg (&cpu_count,0 , n_cpus) ==0 ) + { +atomic_add (n_cpus, &done_count); +cpumask_raise_softirq (&cpu_online_map,RCU_SOFTIRQ ); + } + +while (atomic_read (&done_count) ) + { +process_pending_softirqs (); +cpu_relax (); + } + +put_cpu_maps (); }/* Is batch a before batch b ? */
@@ -426,6 +450,13 @@ static void rcu_process_callbacks(void) rdp->process_callbacks = false;__rcu_process_callbacks (&rcu_ctrlblk, rdp); } + +if (atomic_read (&cpu_count) && !rdp->barrier_active ) + { + rdp->barrier_active = true; +rcu_barrier_action (); + rdp->barrier_active = false; + } }static int __rcu_pending (struct rcu_ctrlblk *rcp,struct rcu_data *rdp)
xen/include/xen/rcupdate.h +1 -1
@@ -143,7 +143,7 @@ void rcu_check_callbacks(int cpu);void call_rcu (struct rcu_head *head,void (*func)(struct rcu_head *head)); -int rcu_barrier (void ); +void rcu_barrier (void );void rcu_idle_enter (unsigned int cpu);void rcu_idle_exit (unsigned int cpu);
Signature · 7 lines
2.16.4 _______________________________________________ Xen-devel mailing list [email protected] https://lists.xenproject.org/mailman/listinfo/xen-devel
PATCH v4 3/6 xen: add process_pending_softirqs_norcu() for keyhandlers
In reply to Juergen GrossSome keyhandlers are calling process_pending_softirqs() while holding a rcu_read_lock(). This is wrong, as process_pending_softirqs() might activate rcu calls which should not happen inside a rcu_read_lock().
For that purpose add process_pending_softirqs_norcu() which will not do any rcu activity and use this for keyhandlers.
---
V3:
- add RCU_SOFTIRQ to ignore in process_pending_softirqs_norcu() (Roger Pau Monné)
--- xen/arch/x86/mm/p2m-ept.c | 2 +- xen/arch/x86/numa.c | 4 ++-- xen/common/keyhandler.c | 6 +++--- xen/common/softirq.c | 17 +++++++++++++---- xen/drivers/passthrough/amd/pci_amd_iommu.c | 2 +- xen/drivers/passthrough/vtd/iommu.c | 2 +- xen/drivers/vpci/msi.c | 4 ++-- xen/include/xen/softirq.h | 2 ++ 8 files changed, 25 insertions(+), 14 deletions(-)
xen/arch/x86/mm/p2m-ept.c +1 -1
@@ -1344,7 +1344,7 @@ static void ept_dump_p2m_table(unsigned char key) c ?: ept_entry->ipat ? '!' : ' ');if ( !(record_counter++ %100 ) ) -process_pending_softirqs (); +process_pending_softirqs_norcu (); }unmap_domain_page (table); }
xen/arch/x86/numa.c +2 -2
@@ -418,7 +418,7 @@ static void dump_numa(unsigned char key)printk ("Memory location of each domain:\n" );for_each_domain ( d ) { -process_pending_softirqs (); +process_pending_softirqs_norcu ();printk ("Domain %u (total: %u):\n" , d->domain_id,domain_tot_pages (d));
@@ -462,7 +462,7 @@ static void dump_numa(unsigned char key)for ( j =0 ; j < d->max_vcpus; j++ ) {if ( !(j &0x3f ) ) -process_pending_softirqs (); +process_pending_softirqs_norcu ();if ( vnuma->vcpu_to_vnode[j] == i ) {
xen/common/keyhandler.c +3 -3
@@ -263,7 +263,7 @@ static void dump_domains(unsigned char key) {unsigned int i; -process_pending_softirqs (); +process_pending_softirqs_norcu ();printk ("General information for domain %u:\n" , d->domain_id); printk(" refcnt=%d dying=%d pause_count=%d\n",
@@ -307,7 +307,7 @@ static void dump_domains(unsigned char key) for_each_sched_unit_vcpu ( unit, v ) {if ( !(v->vcpu_id &0x3f ) ) -process_pending_softirqs (); +process_pending_softirqs_norcu (); printk(" VCPU%d: CPU%d [has=%c] poll=%d " "upcall_pend=%02x upcall_mask=%02x ",
@@ -337,7 +337,7 @@ static void dump_domains(unsigned char key) for_each_vcpu ( d, v ) {if ( !(v->vcpu_id &0x3f ) ) -process_pending_softirqs (); +process_pending_softirqs_norcu (); printk("Notifying guest %d:%d (virq %d, port %d)\n", d->domain_id, v->vcpu_id,
xen/common/softirq.c +13 -4
@@ -25,7 +25,7 @@ static softirq_handler softirq_handlers[NR_SOFTIRQS]; static DEFINE_PER_CPU(cpumask_t, batch_mask); static DEFINE_PER_CPU(unsigned int, batching); -static void __do_softirq (unsigned long ignore_mask) +static void __do_softirq (unsigned long ignore_mask,bool rcu_allowed) {unsigned int i, cpu;unsigned long pending;
@@ -38,7 +38,7 @@ static void __do_softirq(unsigned long ignore_mask)*/ cpu =smp_processor_id (); -if (rcu_pending (cpu) ) +if ( rcu_allowed &&rcu_pending (cpu) )rcu_check_callbacks (cpu); if ( ((pending = (softirq_pending(cpu) & ~ignore_mask)) == 0)
@@ -55,13 +55,22 @@ void process_pending_softirqs(void) {ASSERT (!in_irq () &&local_irq_is_enabled ());/* Do not enter scheduler as it can preempt the calling context. */ -__do_softirq ((1ul <<SCHEDULE_SOFTIRQ ) | (1ul <<SCHED_SLAVE_SOFTIRQ )); +__do_softirq ((1ul <<SCHEDULE_SOFTIRQ ) | (1ul <<SCHED_SLAVE_SOFTIRQ ), + true); +} + +void process_pending_softirqs_norcu (void ) +{ +ASSERT (!in_irq () &&local_irq_is_enabled ()); +/* Do not enter scheduler as it can preempt the calling context. */ +__do_softirq ((1ul <<SCHEDULE_SOFTIRQ ) | (1ul <<SCHED_SLAVE_SOFTIRQ ) | + (1ul <<RCU_SOFTIRQ ), false); }void do_softirq (void ) {ASSERT_NOT_IN_ATOMIC (); -__do_softirq (0 ); +__do_softirq (0 , true); }void open_softirq (int nr,softirq_handler handler)
xen/drivers/passthrough/amd/pci_amd_iommu.c +1 -1
@@ -587,7 +587,7 @@ static void amd_dump_p2m_table_level(struct page_info* pg, int level,struct amd_iommu_pte *pde = &table_vaddr[index];if ( !(index %2 ) ) -process_pending_softirqs (); +process_pending_softirqs_norcu ();if ( !pde->pr )continue ;
xen/drivers/passthrough/vtd/iommu.c +1 -1
@@ -2646,7 +2646,7 @@ static void vtd_dump_p2m_table_level(paddr_t pt_maddr, int level, paddr_t gpa,for ( i =0 ; i <PTE_NUM ; i++ ) {if ( !(i %2 ) ) -process_pending_softirqs (); +process_pending_softirqs_norcu (); pte = &pt_vaddr[i];if ( !dma_pte_present (*pte) )
xen/drivers/vpci/msi.c +2 -2
@@ -321,13 +321,13 @@ void vpci_dump_msi(void)* holding the lock. */ printk ("unable to print all MSI-X entries: %d\n" , rc); -process_pending_softirqs (); +process_pending_softirqs_norcu ();continue ; } }spin_unlock (&pdev->vpci->lock); -process_pending_softirqs (); +process_pending_softirqs_norcu (); } }rcu_read_unlock (&domlist_read_lock);
xen/include/xen/softirq.h +2 -0
@@ -37,7 +37,9 @@ void cpu_raise_softirq_batch_finish(void);* Process pending softirqs on this CPU. This should be called periodically * when performing work that prevents softirqs from running in a timely manner. * Use this instead of do_softirq() when you do not want to be preempted. +* The norcu variant is to be used while holding a read_rcu_lock(). */ void process_pending_softirqs (void ); +void process_pending_softirqs_norcu (void );#endif /* __XEN_SOFTIRQ_H__ */
Signature · 7 lines
2.16.4 _______________________________________________ Xen-devel mailing list [email protected] https://lists.xenproject.org/mailman/listinfo/xen-devel
PATCH v4 4/6 xen/rcu: fix rcu_lock_domain()
In reply to Juergen Grossrcu_lock_domain() misuses the domain structure as rcu lock, which is working only as long as rcu_read_lock() isn't evaluating the lock.
Fix that by adding a rcu lock to struct domain and use that for rcu_lock_domain().
--- xen/common/domain.c | 1 + xen/include/xen/rcupdate.h | 1 + xen/include/xen/sched.h | 6 ++++-- 3 files changed, 6 insertions(+), 2 deletions(-)
xen/common/domain.c +1 -0
@@ -398,6 +398,7 @@ struct domain *domain_create(domid_t domid, goto fail;atomic_set (&d->refcnt,1 ); +RCU_READ_LOCK_INIT (&d->rcu_lock);spin_lock_init_prof (d, domain_lock);spin_lock_init_prof (d, page_alloc_lock);spin_lock_init (&d->hypercall_deadlock_mutex);
xen/include/xen/rcupdate.h +1 -0
@@ -65,6 +65,7 @@ int rcu_needs_cpu(int cpu);struct _rcu_read_lock {};typedef struct _rcu_read_lock rcu_read_lock_t ;#define DEFINE_RCU_READ_LOCK (x) rcu_read_lock_t x +#define RCU_READ_LOCK_INIT (x)/** * rcu_read_lock - mark the beginning of an RCU read-side critical section.
xen/include/xen/sched.h +4 -2
@@ -348,6 +348,8 @@ struct domainshared_info_t *shared_info;/* shared data area */ +rcu_read_lock_t rcu_lock; +spinlock_t domain_lock;spinlock_t page_alloc_lock;/* protects all the following fields */
@@ -634,13 +636,13 @@ int rcu_lock_live_remote_domain_by_id(domid_t dom, struct domain **d);static inline void rcu_unlock_domain (struct domain *d) {if ( d != current->domain ) -rcu_read_unlock (d); +rcu_read_unlock (&d->rcu_lock); }static inline struct domain *rcu_lock_domain (struct domain *d) {if ( d != current->domain ) -rcu_read_lock (d); +rcu_read_lock (&d->rcu_lock);return d; }
Signature · 7 lines
2.16.4 _______________________________________________ Xen-devel mailing list [email protected] https://lists.xenproject.org/mailman/listinfo/xen-devel
v4 cover xen/rcu: let rcu work better with core scheduling
Today the RCU handling in Xen is affecting scheduling in several ways. It is raising sched softirqs without any real need and it requires tasklets for rcu_barrier(), which interacts badly with core scheduling.
This small series repairs those issues.
Additionally some ASSERT()s are added for verification of sane rcu handling. In order to avoid those triggering right away the obvious violations are fixed. This includes making rcu locking functions type safe.
Changes in V4:
- patch 5: use barrier()
Changes in V3:
- type safe locking functions (functions instead of macros)
- per-lock debug additions
- new patches 4 and 6
- fixed races
Changes in V2:
- use get_cpu_maps() in rcu_barrier() handling
- avoid recursion in rcu_barrier() handling
- new patches 3 and 4
Juergen Gross (6):
xen/rcu: use rcu softirq for forcing quiescent state xen/rcu: don't use stop_machine_run() for rcu_barrier()
xen: add process_pending_softirqs_norcu() for keyhandlers xen/rcu: fix rcu_lock_domain()
xen/rcu: add assertions to debug build
xen/rcu: add per-lock counter in debug builds
xen/arch/x86/mm/p2m-ept.c | 2 +- xen/arch/x86/numa.c | 4 +- xen/common/domain.c | 1 + xen/common/keyhandler.c | 6 +- xen/common/multicall.c | 1 + xen/common/preempt.c | 5 +- xen/common/rcupdate.c | 109 ++++++++++++++++++++-------- xen/common/softirq.c | 19 ++++- xen/common/wait.c | 1 + xen/drivers/passthrough/amd/pci_amd_iommu.c | 2 +- xen/drivers/passthrough/vtd/iommu.c | 2 +- xen/drivers/vpci/msi.c | 4 +- xen/include/xen/rcupdate.h | 76 +++++++++++++++---- xen/include/xen/sched.h | 6 +- xen/include/xen/softirq.h | 2 + 15 files changed, 180 insertions(+), 60 deletions(-)
Signature · 7 lines
2.16.4 _______________________________________________ Xen-devel mailing list [email protected] https://lists.xenproject.org/mailman/listinfo/xen-devel
PATCH v4 6/6 xen/rcu: add per-lock counter in debug builds
Add a lock specific counter to rcu read locks in debug builds. This allows to test for matching lock/unlock calls.
--- xen/include/xen/rcupdate.h | 42 +++++++++++++++++++++++++++--------------- 1 file changed, 27 insertions(+), 15 deletions(-)
xen/include/xen/rcupdate.h +27 -15
@@ -36,21 +36,33 @@#include <xen/spinlock.h> #include <xen/cpumask.h> #include <xen/percpu.h> +#include <asm/atomic.h> #define __rcu#ifndef NDEBUG +/* * Lock type for passing to rcu_read_{lock,unlock}. */ +struct _rcu_read_lock { +atomic_t cnt; +}; +typedef struct _rcu_read_lock rcu_read_lock_t ; +#define DEFINE_RCU_READ_LOCK (x) rcu_read_lock_t x = { .cnt = ATOMIC_INIT(0) } +#define RCU_READ_LOCK_INIT (x) atomic_set(&(x)->cnt, 0) + DECLARE_PER_CPU(unsigned int, rcu_lock_cnt); -static inline void rcu_quiesce_disable (void ) +static inline void rcu_quiesce_disable (rcu_read_lock_t *lock) {this_cpu (rcu_lock_cnt)++; +atomic_inc (&lock->cnt);barrier (); } -static inline void rcu_quiesce_enable (void ) +static inline void rcu_quiesce_enable (rcu_read_lock_t *lock) {barrier (); +ASSERT (atomic_read (&lock->cnt)); +atomic_dec (&lock->cnt);this_cpu (rcu_lock_cnt)--; }
@@ -60,8 +72,17 @@ static inline bool rcu_quiesce_allowed(void) }#else -static inline void rcu_quiesce_disable (void ) { } -static inline void rcu_quiesce_enable (void ) { } +/* +* Dummy lock type for passing to rcu_read_{lock,unlock}. Currently exists +* only to document the reason for rcu_read_lock() critical sections. +*/ +struct _rcu_read_lock {}; +typedef struct _rcu_read_lock rcu_read_lock_t ; +#define DEFINE_RCU_READ_LOCK (x) rcu_read_lock_t x +#define RCU_READ_LOCK_INIT (x) + +static inline void rcu_quiesce_disable (rcu_read_lock_t *lock) { } +static inline void rcu_quiesce_enable (rcu_read_lock_t *lock) { }static inline bool rcu_quiesce_allowed (void ) {return true;
@@ -88,15 +109,6 @@ struct rcu_head {int rcu_pending (int cpu);int rcu_needs_cpu (int cpu); -/* -* Dummy lock type for passing to rcu_read_{lock,unlock}. Currently exists -* only to document the reason for rcu_read_lock() critical sections. -*/ -struct _rcu_read_lock {}; -typedef struct _rcu_read_lock rcu_read_lock_t ; -#define DEFINE_RCU_READ_LOCK (x) rcu_read_lock_t x -#define RCU_READ_LOCK_INIT (x) -/** * rcu_read_lock - mark the beginning of an RCU read-side critical section. *
@@ -125,7 +137,7 @@ typedef struct _rcu_read_lock rcu_read_lock_t;*/ static inline void rcu_read_lock (rcu_read_lock_t *lock) { -rcu_quiesce_disable (); +rcu_quiesce_disable (lock); }/**
@@ -136,7 +148,7 @@ static inline void rcu_read_lock(rcu_read_lock_t *lock)static inline void rcu_read_unlock (rcu_read_lock_t *lock) {ASSERT (!rcu_quiesce_allowed ()); -rcu_quiesce_enable (); +rcu_quiesce_enable (lock); }/*
Signature · 7 lines
2.16.4 _______________________________________________ Xen-devel mailing list [email protected] https://lists.xenproject.org/mailman/listinfo/xen-devel
PATCH v4 1/6 xen/rcu: use rcu softirq for forcing quiescent state
In reply to Juergen GrossAs rcu callbacks are processed in __do_softirq() there is no need to use the scheduling softirq for forcing quiescent state. Any other softirq would do the job and the scheduling one is the most expensive.
So use the already existing rcu softirq for that purpose. For telling apart why the rcu softirq was raised add a flag for the current usage.
--- xen/common/rcupdate.c | 20 +++++++++++++++++--- 1 file changed, 17 insertions(+), 3 deletions(-)
xen/common/rcupdate.c +17 -3
@@ -89,6 +89,8 @@ struct rcu_data {/* 3) idle CPUs handling */ struct timer idle_timer;bool idle_timer_active; + +bool process_callbacks; };/*
@@ -198,7 +200,7 @@ static void force_quiescent_state(struct rcu_data *rdp, struct rcu_ctrlblk *rcp) {cpumask_t cpumask; -raise_softirq (SCHEDULE_SOFTIRQ ); +raise_softirq (RCU_SOFTIRQ );if (unlikely (rdp->qlen - rdp->last_rs_qlen > rsinterval)) { rdp->last_rs_qlen = rdp->qlen;/*
@@ -206,7 +208,7 @@ static void force_quiescent_state(struct rcu_data *rdp,* rdp->cpu is the current cpu. */ cpumask_andnot (&cpumask, &rcp->cpumask,cpumask_of (rdp->cpu)); -cpumask_raise_softirq (&cpumask,SCHEDULE_SOFTIRQ ); +cpumask_raise_softirq (&cpumask,RCU_SOFTIRQ ); } }
@@ -263,7 +265,10 @@ static void rcu_do_batch(struct rcu_data *rdp)if (!rdp->donelist) rdp->donetail = &rdp->donelist;else + { + rdp->process_callbacks = true;raise_softirq (RCU_SOFTIRQ ); + } }/*
@@ -414,7 +419,13 @@ static void __rcu_process_callbacks(struct rcu_ctrlblk *rcp,static void rcu_process_callbacks (void ) { -__rcu_process_callbacks (&rcu_ctrlblk, &this_cpu (rcu_data)); +struct rcu_data *rdp = &this_cpu (rcu_data); + +if ( rdp->process_callbacks ) + { + rdp->process_callbacks = false; +__rcu_process_callbacks (&rcu_ctrlblk, rdp); + } }static int __rcu_pending (struct rcu_ctrlblk *rcp,struct rcu_data *rdp)
@@ -522,6 +533,9 @@ static void rcu_idle_timer_handler(void* data)void rcu_check_callbacks (int cpu) { +struct rcu_data *rdp = &this_cpu (rcu_data); + + rdp->process_callbacks = true;raise_softirq (RCU_SOFTIRQ ); }
Signature · 7 lines
2.16.4 _______________________________________________ Xen-devel mailing list [email protected] https://lists.xenproject.org/mailman/listinfo/xen-devel
PATCH v4 5/6 xen/rcu: add assertions to debug build
In reply to Juergen GrossXen's RCU implementation relies on no softirq handling taking place while being in a RCU critical section. Add ASSERT()s in debug builds in order to catch any violations.
For that purpose modify rcu_read_[un]lock() to use a dedicated percpu counter instead of preempt_[en|dis]able() as this enables to test that condition in __do_softirq() (ASSERT_NOT_IN_ATOMIC() is not usable there due to __cpu_up() calling process_pending_softirqs() while holding the cpu hotplug lock).
Dropping the now no longer needed #include of preempt.h in rcupdate.h requires adding it in some sources.
While at it switch the rcu_read_[un]lock() implementation to static inline functions instead of macros.
---
V3:
- add barriers to rcu_[en|dis]able() (Roger Pau Monné)
- add rcu_quiesce_allowed() to ASSERT_NOT_IN_ATOMIC (Roger Pau Monné)
- convert macros to static inline functions
- add sanity check in rcu_read_unlock()
V4:
- use barrier() in rcu_[en|dis]able() (Julien Grall)
--- xen/common/multicall.c | 1 + xen/common/preempt.c | 5 ++++- xen/common/rcupdate.c | 4 ++++ xen/common/softirq.c | 2 ++ xen/common/wait.c | 1 + xen/include/xen/rcupdate.h | 45 +++++++++++++++++++++++++++++++++++++++++---- 6 files changed, 53 insertions(+), 5 deletions(-)
xen/common/multicall.c +1 -0
@@ -10,6 +10,7 @@#include <xen/multicall.h> #include <xen/guest_access.h> #include <xen/perfc.h> +#include <xen/preempt.h> #include <xen/trace.h> #include <asm/current.h> #include <asm/hardirq.h>
xen/common/preempt.c +4 -1
@@ -21,13 +21,15 @@#include <xen/preempt.h> #include <xen/irq.h> +#include <xen/rcupdate.h> #include <asm/system.h> DEFINE_PER_CPU(unsigned int, __preempt_count);bool_t in_atomic (void ) { -return preempt_count () ||in_irq () || !local_irq_is_enabled (); +return preempt_count () ||in_irq () || !local_irq_is_enabled () || + !rcu_quiesce_allowed (); }#ifndef NDEBUG
@@ -36,5 +38,6 @@ void ASSERT_NOT_IN_ATOMIC(void)ASSERT (!preempt_count ());ASSERT (!in_irq ());ASSERT (local_irq_is_enabled ()); + ASSERT(rcu_quiesce_allowed()); }#endif
xen/common/rcupdate.c +4 -0
@@ -46,6 +46,10 @@#include <xen/cpu.h> #include <xen/stop_machine.h> +#ifndef NDEBUG +DEFINE_PER_CPU(unsigned int, rcu_lock_cnt); +#endif +/* Global control variables for rcupdate callback mechanism. */ static struct rcu_ctrlblk { long cur; /* Current batch number. */
xen/common/softirq.c +2 -0
@@ -30,6 +30,8 @@ static void __do_softirq(unsigned long ignore_mask, bool rcu_allowed)unsigned int i, cpu;unsigned long pending; +ASSERT (!rcu_allowed ||rcu_quiesce_allowed ()); +for ( ; ; ) {/*
xen/common/wait.c +1 -0
@@ -19,6 +19,7 @@* along with this program; If not, see <http://www.gnu.org/licenses/>. */ +#include <xen/preempt.h> #include <xen/sched.h> #include <xen/softirq.h> #include <xen/wait.h>
xen/include/xen/rcupdate.h +41 -4
69 lines not on this page · raw message
Signature · 7 lines
2.16.4 _______________________________________________ Xen-devel mailing list [email protected] https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [Xen-devel] [PATCH v4 2/6] xen/rcu: don't use stop_machine_run() for rcu_barrier()
In reply to Juergen GrossQuoted 13 lines · Juergen Gross
xen/common/rcupdate.c +5
-4@@ -143,51 +143,75 @@ static int qhimark = 10000;static int qlowmark =100 ;static int rsinterval =1000 ; -struct rcu_barrier_data { -struct rcu_head head; -atomic_t *cpu_count; -}; +/* +* rcu_barrier() handling: +* cpu_count holds the number of cpu required to finish barrier handling. +* Cpus are synchronized via softirq mechanism. rcu_barrier() is regarded to +* be active if cpu_count is not zero. In case rcu_barrier() is called on
xen/common/rcupdate.c static int qhimark = 10000; · line 160
Quoted 10 lines
xen/common/rcupdate.c +10
-0+* multiple cpus it is enough to check for cpu_count being not zero on entry +* and to call process_pending_softirqs() in a loop until cpu_count drops to +* zero, as syncing has been requested already and we don't need to sync +* multiple times. +* In order to avoid hangs when rcu_barrier() is called mutiple times on the +* same cpu in fast sequence and a slave cpu couldn't drop out of the +* barrier handling fast enough a second counter done_count is needed. +*/ +static atomic_t cpu_count =ATOMIC_INIT (0 ); +static atomic_t done_count =ATOMIC_INIT (0 );
From its use below this looks more like "todo_count" or "pending_count".
Quoted 14 lines
xen/common/rcupdate.c +11
-2+void rcu_barrier (void ) { -atomic_t cpu_count =ATOMIC_INIT (0 ); -return stop_machine_run (rcu_barrier_action, &cpu_count,NR_CPUS ); +unsigned int n_cpus; + +while ( !get_cpu_maps () ) + { +process_pending_softirqs (); +if ( !atomic_read (&cpu_count) ) +return ; + +cpu_relax (); + }
xen/common/rcupdate.c static int qhimark = 10000; · line 208
Quoted 10 lines
xen/common/rcupdate.c +10
-0+ + n_cpus =num_online_cpus (); + +if (atomic_cmpxchg (&cpu_count,0 , n_cpus) ==0 ) + { +atomic_add (n_cpus, &done_count); +cpumask_raise_softirq (&cpu_online_map,RCU_SOFTIRQ ); + } + +while (atomic_read (&done_count) )
Don't you leave a window for races here, in that done_count gets set to non-zero only after setting cpu_count? A CPU losing the cmpxchg attempt above may observe done_count still being zero, and hence exit without waiting for the count to actually _drop_ to zero.
Jan
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [Xen-devel] [PATCH v4 2/6] xen/rcu: don't use stop_machine_run() for rcu_barrier()
Quoted 50 lines · Jan Beulich
Quoted 13 lines · Juergen Gross
xen/common/rcupdate.c +5
-4@@ -143,51 +143,75 @@ static int qhimark = 10000;static int qlowmark =100 ;static int rsinterval =1000 ; -struct rcu_barrier_data { -struct rcu_head head; -atomic_t *cpu_count; -}; +/* +* rcu_barrier() handling: +* cpu_count holds the number of cpu required to finish barrier handling. +* Cpus are synchronized via softirq mechanism. rcu_barrier() is regarded to +* be active if cpu_count is not zero. In case rcu_barrier() is called on xen/common/rcupdate.c static int qhimark = 10000; · line 160
Quoted 10 lines
xen/common/rcupdate.c +10
-0+* multiple cpus it is enough to check for cpu_count being not zero on entry +* and to call process_pending_softirqs() in a loop until cpu_count drops to +* zero, as syncing has been requested already and we don't need to sync +* multiple times. +* In order to avoid hangs when rcu_barrier() is called mutiple times on the +* same cpu in fast sequence and a slave cpu couldn't drop out of the +* barrier handling fast enough a second counter done_count is needed. +*/ +static atomic_t cpu_count =ATOMIC_INIT (0 ); +static atomic_t done_count =ATOMIC_INIT (0 );From its use below this looks more like "
todo_count" or
"pending_count".Quoted 22 lines
xen/common/rcupdate.c +19
-2+void rcu_barrier (void ) { -atomic_t cpu_count =ATOMIC_INIT (0 ); -return stop_machine_run (rcu_barrier_action, &cpu_count,NR_CPUS ); +unsigned int n_cpus; + +while ( !get_cpu_maps () ) + { +process_pending_softirqs (); +if ( !atomic_read (&cpu_count) ) +return ; + +cpu_relax (); + } + + n_cpus =num_online_cpus (); + +if (atomic_cmpxchg (&cpu_count,0 , n_cpus) ==0 ) + { +atomic_add (n_cpus, &done_count); +cpumask_raise_softirq (&cpu_online_map,RCU_SOFTIRQ ); + }
Quoted 8 lines
Review static int qhimark = 10000; · line 182
Quoted 2 lines
(quoted diff) +2
-0+ + while ( atomic_read(&done_count) )Don't you leave a window for races here, in that
done_countgets set to non-zero only after settingcpu_count? A CPU losing the cmpxchg attempt above may observedone_countstill being zero, and hence exit without waiting for the count to actually _drop_ to zero.
This can only be a cpu not having joined the barrier handling, so it will do that later.
Juergen
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [Xen-devel] [PATCH v4 2/6] xen/rcu: don't use stop_machine_run() for rcu_barrier()
Quoted 33 lines · Jürgen Groß
Quoted 23 lines · Jan Beulich
Quoted 22 lines · Juergen Gross
xen/common/rcupdate.c +19
-2+void rcu_barrier (void ) { -atomic_t cpu_count =ATOMIC_INIT (0 ); -return stop_machine_run (rcu_barrier_action, &cpu_count,NR_CPUS ); +unsigned int n_cpus; + +while ( !get_cpu_maps () ) + { +process_pending_softirqs (); +if ( !atomic_read (&cpu_count) ) +return ; + +cpu_relax (); + } + + n_cpus =num_online_cpus (); + +if (atomic_cmpxchg (&cpu_count,0 , n_cpus) ==0 ) + { +atomic_add (n_cpus, &done_count); +cpumask_raise_softirq (&cpu_online_map,RCU_SOFTIRQ ); + }Quoted 8 lines
Review
Quoted 2 lines
(quoted diff) +2
-0+ + while ( atomic_read(&done_count) )Don't you leave a window for races here, in that
done_countgets set to non-zero only after settingcpu_count? A CPU losing the cmpxchg attempt above may observedone_countstill being zero, and hence exit without waiting for the count to actually _drop_ to zero.
Quoted 2 lines
This can only be a cpu not having joined the barrier handling, so it will do that later.
I'm afraid I don't understand - if two CPUs independently call rcu_barrier(), neither should fall through here without waiting at all, I would think?
Jan
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [Xen-devel] [PATCH v4 3/6] xen: add process_pending_softirqs_norcu() for keyhandlers
In reply to Juergen Grossxen/common/softirq.c static softirq_handler softirq_handlers[NR_SOFTIRQS]; · line 28
Quoted 8 lines · Juergen Gross
xen/common/softirq.c +1
-1@@ -25,7 +25,7 @@ static softirq_handler softirq_handlers[NR_SOFTIRQS]; static DEFINE_PER_CPU(cpumask_t, batch_mask); static DEFINE_PER_CPU(unsigned int, batching); -static void __do_softirq (unsigned long ignore_mask) +static void __do_softirq (unsigned long ignore_mask,bool rcu_allowed)
Why the separate bool? Can't you ...
xen/common/softirq.c static void __do_softirq(unsigned long ignore_mask) · line 41
Quoted 6 lines
xen/common/softirq.c +1
-1@@ -38,7 +38,7 @@ static void __do_softirq(unsigned long ignore_mask)*/ cpu =smp_processor_id (); -if (rcu_pending (cpu) ) +if ( rcu_allowed &&rcu_pending (cpu) )
... check !(ignore_mask & RCU_SOFTIRQ) here?
Quoted 5 lines
xen/common/softirq.c +0
-1@@ -55,13 +55,22 @@ void process_pending_softirqs(void) {ASSERT (!in_irq () &&local_irq_is_enabled ());/* Do not enter scheduler as it can preempt the calling context. */ -__do_softirq ((1ul <<SCHEDULE_SOFTIRQ ) | (1ul <<SCHED_SLAVE_SOFTIRQ ));
xen/common/softirq.c void process_pending_softirqs(void) · line 67
Quoted 10 lines
xen/common/softirq.c +10
-0+__do_softirq ((1ul <<SCHEDULE_SOFTIRQ ) | (1ul <<SCHED_SLAVE_SOFTIRQ ), + true); +} + +void process_pending_softirqs_norcu (void ) +{ +ASSERT (!in_irq () &&local_irq_is_enabled ()); +/* Do not enter scheduler as it can preempt the calling context. */ +__do_softirq ((1ul <<SCHEDULE_SOFTIRQ ) | (1ul <<SCHED_SLAVE_SOFTIRQ ) | + (1ul <<RCU_SOFTIRQ ), false);
I guess the comment here also wants to mention RCU?
xen/drivers/passthrough/amd/pci_amd_iommu.c static void amd_dump_p2m_table_level(struct page_info* pg, int level, · line 590
Quoted 8 lines
xen/drivers/passthrough/amd/pci_amd_iommu.c +1
-1@@ -587,7 +587,7 @@ static void amd_dump_p2m_table_level(struct page_info* pg, int level,struct amd_iommu_pte *pde = &table_vaddr[index];if ( !(index %2 ) ) -process_pending_softirqs (); +process_pending_softirqs_norcu ();
At the example of this - the property of holding an RCU lock is entirely invisible here, as it's the generic iommu_dump_p2m_table() which acquires it. This suggest to me that going forward breaking this is going to be very likely. Couldn't process_pending_softirqs() exclude RCU handling when finding preempt_count() to return non-zero?
Jan
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [Xen-devel] [PATCH v4 2/6] xen/rcu: don't use stop_machine_run() for rcu_barrier()
In reply to Jan BeulichQuoted 34 lines · Jan Beulich
Quoted 33 lines · Jürgen Groß
Quoted 31 lines · Jan Beulich
Quoted 14 lines · Juergen Gross
xen/common/rcupdate.c +11
-2+void rcu_barrier (void ) { -atomic_t cpu_count =ATOMIC_INIT (0 ); -return stop_machine_run (rcu_barrier_action, &cpu_count,NR_CPUS ); +unsigned int n_cpus; + +while ( !get_cpu_maps () ) + { +process_pending_softirqs (); +if ( !atomic_read (&cpu_count) ) +return ; + +cpu_relax (); + }xen/common/rcupdate.c static int qhimark = 10000; · line 208
Quoted 10 lines
xen/common/rcupdate.c +10
-0+ + n_cpus =num_online_cpus (); + +if (atomic_cmpxchg (&cpu_count,0 , n_cpus) ==0 ) + { +atomic_add (n_cpus, &done_count); +cpumask_raise_softirq (&cpu_online_map,RCU_SOFTIRQ ); + } + +while (atomic_read (&done_count) )Don't you leave a window for races here, in that
done_countgets set to non-zero only after settingcpu_count? A CPU losing the cmpxchg attempt above may observedone_countstill being zero, and hence exit without waiting for the count to actually _drop_ to zero.
Quoted 6 lines
Quoted 2 lines
This can only be a cpu not having joined the barrier handling, so it will do that later.
I'm afraid I don't understand - if two CPUs independently call
rcu_barrier(), neither should fall through here without waiting at all, I would think?
Oh, good catch!
I have thought more about this problem and I think using counters only for doing rendezvous accounting is rather risky. I'll have a try using a cpumask instead.
Juergen
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [Xen-devel] [PATCH v4 3/6] xen: add process_pending_softirqs_norcu() for keyhandlers
In reply to Jan BeulichQuoted 12 lines · Jan Beulich
xen/common/softirq.c static softirq_handler softirq_handlers[NR_SOFTIRQS]; · line 28
Quoted 8 lines · Juergen Gross
xen/common/softirq.c +1
-1@@ -25,7 +25,7 @@ static softirq_handler softirq_handlers[NR_SOFTIRQS]; static DEFINE_PER_CPU(cpumask_t, batch_mask); static DEFINE_PER_CPU(unsigned int, batching); -static void __do_softirq (unsigned long ignore_mask) +static void __do_softirq (unsigned long ignore_mask,bool rcu_allowed)Why the separate bool? Can't you ...
Quoted 8 lines
xen/common/softirq.c static void __do_softirq(unsigned long ignore_mask) · line 41
Quoted 6 lines
xen/common/softirq.c +1
-1@@ -38,7 +38,7 @@ static void __do_softirq(unsigned long ignore_mask)*/ cpu =smp_processor_id (); -if (rcu_pending (cpu) ) +if ( rcu_allowed &&rcu_pending (cpu) )... check !(ignore_mask &
RCU_SOFTIRQ) here?
Good idea.
Quoted 10 lines
Quoted 10 lines
xen/common/softirq.c +4
-1@@ -55,13 +55,22 @@ void process_pending_softirqs(void) {ASSERT (!in_irq () &&local_irq_is_enabled ());/* Do not enter scheduler as it can preempt the calling context. */ -__do_softirq ((1ul <<SCHEDULE_SOFTIRQ ) | (1ul <<SCHED_SLAVE_SOFTIRQ )); +__do_softirq ((1ul <<SCHEDULE_SOFTIRQ ) | (1ul <<SCHED_SLAVE_SOFTIRQ ), + true); +} +
Quoted 8 lines
xen/common/softirq.c void process_pending_softirqs(void) · line 67
Quoted 6 lines
xen/common/softirq.c +6
-0+void process_pending_softirqs_norcu (void ) +{ +ASSERT (!in_irq () &&local_irq_is_enabled ()); +/* Do not enter scheduler as it can preempt the calling context. */ +__do_softirq ((1ul <<SCHEDULE_SOFTIRQ ) | (1ul <<SCHED_SLAVE_SOFTIRQ ) | + (1ul <<RCU_SOFTIRQ ), false);I guess the comment here also wants to mention RCU?
Yes.
Quoted 6 lines
Quoted 6 lines
xen/drivers/passthrough/amd/pci_amd_iommu.c +0
-0@@ -587,7 +587,7 @@ static void amd_dump_p2m_table_level(struct page_info* pg, int level,struct amd_iommu_pte *pde = &table_vaddr[index];
Quoted 10 lines
xen/drivers/passthrough/amd/pci_amd_iommu.c static void amd_dump_p2m_table_level(struct page_info* pg, int level, · line 590
Quoted 3 lines
xen/drivers/passthrough/amd/pci_amd_iommu.c +1
-1if ( !(index %2 ) ) -process_pending_softirqs (); +process_pending_softirqs_norcu ();At the example of this - the property of holding an RCU lock is entirely invisible here, as it's the generic
iommu_dump_p2m_table()which acquires it. This suggest to me that going forward breaking this is going to be very likely. Couldn'tprocess_pending_softirqs()exclude RCU handling when findingpreempt_count()to return non-zero?
This can be done, but then the non-debug build would require to have non-empty rcu lock functions.
An alternative would be to ASSERT() no rcu lock being held in process_pending_softirqs() or rcu_check_callbacks() which would catch the problematic cases in debug builds.
Juergen
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [Xen-devel] [PATCH v4 3/6] xen: add process_pending_softirqs_norcu() for keyhandlers
Quoted 18 lines · Jürgen Groß
Quoted 6 lines · Jan Beulich
Quoted 5 lines · Juergen Gross
xen/drivers/passthrough/amd/pci_amd_iommu.c +0
-0@@ -587,7 +587,7 @@ static void amd_dump_p2m_table_level(struct page_info* pg, int level,struct amd_iommu_pte *pde = &table_vaddr[index];Quoted 10 lines
xen/drivers/passthrough/amd/pci_amd_iommu.c static void amd_dump_p2m_table_level(struct page_info* pg, int level, · line 590
Quoted 3 lines
xen/drivers/passthrough/amd/pci_amd_iommu.c +1
-1if ( !(index %2 ) ) -process_pending_softirqs (); +process_pending_softirqs_norcu ();At the example of this - the property of holding an RCU lock is entirely invisible here, as it's the generic
iommu_dump_p2m_table()which acquires it. This suggest to me that going forward breaking this is going to be very likely. Couldn'tprocess_pending_softirqs()exclude RCU handling when findingpreempt_count()to return non-zero?
Quoted 2 lines
This can be done, but then the non-debug build would require to have non-empty rcu lock functions.
I guess I don't understand - I see only one version of them:
#define rcu_read_lock(x) ({ ((void)(x)); preempt_disable(); })
#define rcu_read_unlock(x) ({ ((void)(x)); preempt_enable(); })
Same for the preempt count adjustment operations.
Jan
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [Xen-devel] [PATCH v4 3/6] xen: add process_pending_softirqs_norcu() for keyhandlers
Quoted 22 lines · Jan Beulich
Quoted 18 lines · Jürgen Groß
Quoted 6 lines · Jan Beulich
Quoted 5 lines · Juergen Gross
xen/drivers/passthrough/amd/pci_amd_iommu.c +0
-0@@ -587,7 +587,7 @@ static void amd_dump_p2m_table_level(struct page_info* pg, int level,struct amd_iommu_pte *pde = &table_vaddr[index];Quoted 10 lines
xen/drivers/passthrough/amd/pci_amd_iommu.c static void amd_dump_p2m_table_level(struct page_info* pg, int level, · line 590
Quoted 3 lines
xen/drivers/passthrough/amd/pci_amd_iommu.c +1
-1if ( !(index %2 ) ) -process_pending_softirqs (); +process_pending_softirqs_norcu ();At the example of this - the property of holding an RCU lock is entirely invisible here, as it's the generic
iommu_dump_p2m_table()which acquires it. This suggest to me that going forward breaking this is going to be very likely. Couldn'tprocess_pending_softirqs()exclude RCU handling when findingpreempt_count()to return non-zero?Quoted 2 lines
This can be done, but then the non-debug build would require to have non-empty rcu lock functions.
Quoted 6 lines
I guess I don't understand - I see only one version of them:
#define
rcu_read_lock(x)({ ((void)(x));preempt_disable(); })
#definercu_read_unlock(x)({ ((void)(x));preempt_enable(); })Same for the preempt count adjustment operations.
See patch 5.
Juergen
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [Xen-devel] [PATCH v4 3/6] xen: add process_pending_softirqs_norcu() for keyhandlers
Quoted 23 lines · Jürgen Groß
Quoted 22 lines · Jan Beulich
Quoted 20 lines · Jürgen Groß
Quoted 6 lines · Jan Beulich
Quoted 5 lines · Juergen Gross
xen/drivers/passthrough/amd/pci_amd_iommu.c +0
-0@@ -587,7 +587,7 @@ static void amd_dump_p2m_table_level(struct page_info* pg, int level,struct amd_iommu_pte *pde = &table_vaddr[index];Quoted 10 lines
xen/drivers/passthrough/amd/pci_amd_iommu.c static void amd_dump_p2m_table_level(struct page_info* pg, int level, · line 590
Quoted 3 lines
xen/drivers/passthrough/amd/pci_amd_iommu.c +1
-1if ( !(index %2 ) ) -process_pending_softirqs (); +process_pending_softirqs_norcu ();At the example of this - the property of holding an RCU lock is entirely invisible here, as it's the generic
iommu_dump_p2m_table()which acquires it. This suggest to me that going forward breaking this is going to be very likely. Couldn'tprocess_pending_softirqs()exclude RCU handling when findingpreempt_count()to return non-zero?This can be done, but then the non-debug build would require to have non-empty rcu lock functions.
Quoted 8 lines
Quoted 6 lines
I guess I don't understand - I see only one version of them:
#define
rcu_read_lock(x)({ ((void)(x));preempt_disable(); })
#definercu_read_unlock(x)({ ((void)(x));preempt_enable(); })Same for the preempt count adjustment operations.
See patch 5.
Which I haven't looked at yet, and which I also shouldn't need to look at to understand the patch here. If this is a preparatory change rather than some form of fix or improvement, then the description should say so.
Jan
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [Xen-devel] [PATCH v4 3/6] xen: add process_pending_softirqs_norcu() for keyhandlers
Quoted 29 lines · Jan Beulich
Quoted 28 lines · Jürgen Groß
Quoted 27 lines · Jan Beulich
Quoted 18 lines · Jürgen Groß
Quoted 6 lines · Jan Beulich
Quoted 5 lines · Juergen Gross
xen/drivers/passthrough/amd/pci_amd_iommu.c +0
-0@@ -587,7 +587,7 @@ static void amd_dump_p2m_table_level(struct page_info* pg, int level,struct amd_iommu_pte *pde = &table_vaddr[index];Quoted 10 lines
xen/drivers/passthrough/amd/pci_amd_iommu.c static void amd_dump_p2m_table_level(struct page_info* pg, int level, · line 590
Quoted 3 lines
xen/drivers/passthrough/amd/pci_amd_iommu.c +1
-1if ( !(index %2 ) ) -process_pending_softirqs (); +process_pending_softirqs_norcu ();At the example of this - the property of holding an RCU lock is entirely invisible here, as it's the generic
iommu_dump_p2m_table()which acquires it. This suggest to me that going forward breaking this is going to be very likely. Couldn'tprocess_pending_softirqs()exclude RCU handling when findingpreempt_count()to return non-zero?Quoted 2 lines
This can be done, but then the non-debug build would require to have non-empty rcu lock functions.
I guess I don't understand - I see only one version of them:
#define
rcu_read_lock(x)({ ((void)(x));preempt_disable(); })
#definercu_read_unlock(x)({ ((void)(x));preempt_enable(); })
Quoted 8 lines
Quoted 3 lines
Quoted 1 line
Same for the preempt count adjustment operations.
See patch 5.
Which I haven't looked at yet, and which I also shouldn't need to look at to understand the patch here. If this is a preparatory change rather than some form of fix or improvement, then the description should say so.
This was just meant as an answer to your question regarding considering preempt_count(). Just changing this patch here and then undoing the change again in patch 5 is no option IMO, so I gave a hint why using preempt_count() might be a bad idea.
Juergen
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [Xen-devel] [PATCH v4 3/6] xen: add process_pending_softirqs_norcu() for keyhandlers
Quoted 39 lines · Jürgen Groß
Quoted 29 lines · Jan Beulich
Quoted 28 lines · Jürgen Groß
Quoted 27 lines · Jan Beulich
Quoted 18 lines · Jürgen Groß
Quoted 6 lines · Jan Beulich
Quoted 5 lines · Juergen Gross
xen/drivers/passthrough/amd/pci_amd_iommu.c +0
-0@@ -587,7 +587,7 @@ static void amd_dump_p2m_table_level(struct page_info* pg, int level,struct amd_iommu_pte *pde = &table_vaddr[index];Quoted 10 lines
xen/drivers/passthrough/amd/pci_amd_iommu.c static void amd_dump_p2m_table_level(struct page_info* pg, int level, · line 590
Quoted 3 lines
xen/drivers/passthrough/amd/pci_amd_iommu.c +1
-1if ( !(index %2 ) ) -process_pending_softirqs (); +process_pending_softirqs_norcu ();At the example of this - the property of holding an RCU lock is entirely invisible here, as it's the generic
iommu_dump_p2m_table()which acquires it. This suggest to me that going forward breaking this is going to be very likely. Couldn'tprocess_pending_softirqs()exclude RCU handling when findingpreempt_count()to return non-zero?Quoted 2 lines
This can be done, but then the non-debug build would require to have non-empty rcu lock functions.
I guess I don't understand - I see only one version of them:
#define
rcu_read_lock(x)({ ((void)(x));preempt_disable(); })
#definercu_read_unlock(x)({ ((void)(x));preempt_enable(); })Quoted 8 lines
Quoted 3 lines
Quoted 1 line
Same for the preempt count adjustment operations.
See patch 5.
Which I haven't looked at yet, and which I also shouldn't need to look at to understand the patch here. If this is a preparatory change rather than some form of fix or improvement, then the description should say so.
Quoted 4 lines
This was just meant as an answer to your question regarding considering
preempt_count(). Just changing this patch here and then undoing the change again in patch 5 is no option IMO, so I gave a hint why usingpreempt_count()might be a bad idea.
I've briefly looked at patch 5, and I don't see why the counter you introduce there couldn't also be maintained in non-debug builds.
Jan
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [Xen-devel] [PATCH v4 4/6] xen/rcu: fix rcu_lock_domain()
In reply to Juergen GrossQuoted 7 lines · Juergen Gross
rcu_lock_domain()misuses the domain structure as rcu lock, which is working only as long asrcu_read_lock()isn't evaluating the lock.Fix that by adding a rcu lock to struct domain and use that for
rcu_lock_domain().Signed-off-by Juergen Gross <[email protected]>
I guess this one is independent of patches 2 and 3, and hence could go in together with patch 1?
Jan
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [Xen-devel] [PATCH v4 4/6] xen/rcu: fix rcu_lock_domain()
Quoted 7 lines · Jan Beulich
Quoted 6 lines · Juergen Gross
rcu_lock_domain()misuses the domain structure as rcu lock, which is working only as long asrcu_read_lock()isn't evaluating the lock.Fix that by adding a rcu lock to struct domain and use that for
rcu_lock_domain().
Quoted 6 lines
Quoted 1 line
Signed-off-by Juergen Gross <[email protected]>Reviewed-by Jan Beulich <[email protected]>I guess this one is independent of patches 2 and 3, and hence could go in together with patch 1?
Yes.
Juergen
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [Xen-devel] [PATCH v4 3/6] xen: add process_pending_softirqs_norcu() for keyhandlers
In reply to Jan BeulichQuoted 40 lines · Jan Beulich
Quoted 39 lines · Jürgen Groß
Quoted 37 lines · Jan Beulich
Quoted 23 lines · Jürgen Groß
Quoted 22 lines · Jan Beulich
Quoted 20 lines · Jürgen Groß
Quoted 6 lines · Jan Beulich
Quoted 5 lines · Juergen Gross
xen/drivers/passthrough/amd/pci_amd_iommu.c +0
-0@@ -587,7 +587,7 @@ static void amd_dump_p2m_table_level(struct page_info* pg, int level,struct amd_iommu_pte *pde = &table_vaddr[index];Quoted 10 lines
xen/drivers/passthrough/amd/pci_amd_iommu.c static void amd_dump_p2m_table_level(struct page_info* pg, int level, · line 590
Quoted 3 lines
xen/drivers/passthrough/amd/pci_amd_iommu.c +1
-1if ( !(index %2 ) ) -process_pending_softirqs (); +process_pending_softirqs_norcu ();At the example of this - the property of holding an RCU lock is entirely invisible here, as it's the generic
iommu_dump_p2m_table()which acquires it. This suggest to me that going forward breaking this is going to be very likely. Couldn'tprocess_pending_softirqs()exclude RCU handling when findingpreempt_count()to return non-zero?This can be done, but then the non-debug build would require to have non-empty rcu lock functions.
Quoted 8 lines
Quoted 6 lines
I guess I don't understand - I see only one version of them:
#define
rcu_read_lock(x)({ ((void)(x));preempt_disable(); })
#definercu_read_unlock(x)({ ((void)(x));preempt_enable(); })Same for the preempt count adjustment operations.
See patch 5.
Which I haven't looked at yet, and which I also shouldn't need to look at to understand the patch here. If this is a preparatory change rather than some form of fix or improvement, then the description should say so.
Quoted 7 lines
Quoted 4 lines
This was just meant as an answer to your question regarding considering
preempt_count(). Just changing this patch here and then undoing the change again in patch 5 is no option IMO, so I gave a hint why usingpreempt_count()might be a bad idea.I've briefly looked at patch 5, and I don't see why the counter you introduce there couldn't also be maintained in non-debug builds.
Okay. I'll modify patches 3 and 5 accordingly.
Juergen
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [Xen-devel] [PATCH v4 6/6] xen/rcu: add per-lock counter in debug builds
In reply to Juergen GrossQuoted 2 lines · Juergen Gross
Add a lock specific counter to rcu read locks in debug builds. This allows to test for matching lock/unlock calls.
Similar checking doesn't exist for e.g. spin locks iirc, and hence I think you want to spend the word on the "why" here.
Jan
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [Xen-devel] [PATCH v4 6/6] xen/rcu: add per-lock counter in debug builds
Quoted 6 lines · Jan Beulich
Quoted 2 lines · Juergen Gross
Add a lock specific counter to rcu read locks in debug builds. This allows to test for matching lock/unlock calls.
Similar checking doesn't exist for e.g. spin locks iirc, and hence I think you want to spend the word on the "why" here.
With spinlock debugging turned on there is such a check in rel_lock():
The locking cpu has to match and on unlock the locking cpu is set to SPINLOCK_NO_CPU.
I can add something like:
"This will help top avoid cases like the one fixed by commit
98ed1f43cc2c89 where different rcu read locks were referenced in the lock and unlock calls."
Juergen
_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel