Skip to contentSkip to conversation

xen/rcu: let rcu work better with core scheduling

PATCH Superseded Juergen Gross 4 participants 16 messages Feb 17, 2020

v1 of 8 Superseded Latest is v8 →

lore
Share & tools
lore
#0 lore raw

cover xen/rcu: let rcu work better with core scheduling

Today the RCU handling in Xen is affecting scheduling in several ways. It is raising sched softirqs without any real need and it requires tasklets for rcu_barrier(), which interacts badly with core scheduling.

This small series repairs those issues.

Juergen Gross (2):
xen/rcu: use rcu softirq for forcing quiescent state xen/rcu: don't use stop_machine_run() for rcu_barrier()

 xen/common/rcupdate.c      | 69 ++++++++++++++++++++++++++++++----------------
 xen/include/xen/rcupdate.h |  2 +-
 2 files changed, 46 insertions(+), 25 deletions(-)
Signature · 7 lines
2.16.4


_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
#1 lore raw

PATCH 1/2 xen/rcu: use rcu softirq for forcing quiescent state

As rcu callbacks are processed in __do_softirq() there is no need to use the scheduling softirq for forcing quiescent state. Any other softirq would do the job and the scheduling one is the most expensive.

So use the already existing rcu softirq for that purpose. For telling apart why the rcu softirq was raised add a flag for the current usage.

---
 xen/common/rcupdate.c | 20 +++++++++++++++++---
 1 file changed, 17 insertions(+), 3 deletions(-)
xen/common/rcupdate.c +17 -3
index 91d4ad0fd8..079ea9d8a1 100644
@@ -89,6 +89,8 @@ struct rcu_data {
     /* 3) idle CPUs handling */
     struct timer idle_timer;
     bool idle_timer_active;
+
+    bool            process_callbacks;
 };
 
 /*
@@ -194,7 +196,7 @@ static void force_quiescent_state(struct rcu_data *rdp,
                                   struct rcu_ctrlblk *rcp)
 {
     cpumask_t cpumask;
-    raise_softirq(SCHEDULE_SOFTIRQ);
+    raise_softirq(RCU_SOFTIRQ);
     if (unlikely(rdp->qlen - rdp->last_rs_qlen > rsinterval)) {
         rdp->last_rs_qlen = rdp->qlen;
         /*
@@ -202,7 +204,7 @@ static void force_quiescent_state(struct rcu_data *rdp,
          * rdp->cpu is the current cpu.
          */
         cpumask_andnot(&cpumask, &rcp->cpumask, cpumask_of(rdp->cpu));
-        cpumask_raise_softirq(&cpumask, SCHEDULE_SOFTIRQ);
+        cpumask_raise_softirq(&cpumask, RCU_SOFTIRQ);
     }
 }
 
@@ -259,7 +261,10 @@ static void rcu_do_batch(struct rcu_data *rdp)
     if (!rdp->donelist)
         rdp->donetail = &rdp->donelist;
     else
+    {
+        rdp->process_callbacks = true;
         raise_softirq(RCU_SOFTIRQ);
+    }
 }
 
 /*
@@ -410,7 +415,13 @@ static void __rcu_process_callbacks(struct rcu_ctrlblk *rcp,
 
 static void rcu_process_callbacks(void)
 {
-    __rcu_process_callbacks(&rcu_ctrlblk, &this_cpu(rcu_data));
+    struct rcu_data *rdp = &this_cpu(rcu_data);
+
+    if ( rdp->process_callbacks )
+    {
+        rdp->process_callbacks = false;
+        __rcu_process_callbacks(&rcu_ctrlblk, rdp);
+    }
 }
 
 static int __rcu_pending(struct rcu_ctrlblk *rcp, struct rcu_data *rdp)
@@ -518,6 +529,9 @@ static void rcu_idle_timer_handler(void* data)
 
 void rcu_check_callbacks(int cpu)
 {
+    struct rcu_data *rdp = &this_cpu(rcu_data);
+
+    rdp->process_callbacks = true;
     raise_softirq(RCU_SOFTIRQ);
 }
 
Signature · 7 lines
2.16.4


_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
#2 lore raw

PATCH 2/2 xen/rcu: don't use stop_machine_run() for rcu_barrier()

In reply to Juergen Gross

Today rcu_barrier() is calling stop_machine_run() to synchronize all physical cpus in order to ensure all pending rcu calls have finished when returning.

As stop_machine_run() is using tasklets this requires scheduling of idle vcpus on all cpus imposing the need to call rcu_barrier() on idle cpus only in case of core scheduling being active, as otherwise a scheduling deadlock would occur.

There is no need at all to do the syncing of the cpus in tasklets, as rcu activity is started in __do_softirq() called whenever softirq activity is allowed. So rcu_barrier() can easily be modified to use softirq for synchronization of the cpus no longer requiring any scheduling activity.

As there already is a rcu softirq reuse that for the synchronization.

Finally switch rcu_barrier() to return void as it now can never fail.

---
 xen/common/rcupdate.c      | 49 ++++++++++++++++++++++++++--------------------
 xen/include/xen/rcupdate.h |  2 +-
 2 files changed, 29 insertions(+), 22 deletions(-)
xen/common/rcupdate.c +28 -21
index 079ea9d8a1..1f02a804e3 100644
@@ -143,47 +143,51 @@ static int qhimark = 10000;
 static int qlowmark = 100;
 static int rsinterval = 1000;
 
-struct rcu_barrier_data {
-    struct rcu_head head;
-    atomic_t *cpu_count;
-};
+/*
+ * rcu_barrier() handling:
+ * cpu_count holds the number of cpu required to finish barrier handling.
+ * Cpus are synchronized via softirq mechanism. rcu_barrier() is regarded to
+ * be active if cpu_count is not zero. In case rcu_barrier() is called on
+ * multiple cpus it is enough to check for cpu_count being not zero on entry
+ * and to call process_pending_softirqs() in a loop until cpu_count drops to
+ * zero, as syncing has been requested already and we don't need to sync
+ * multiple times.
+ */
+static atomic_t cpu_count = ATOMIC_INIT(0);
 
 static void rcu_barrier_callback(struct rcu_head *head)
 {
-    struct rcu_barrier_data *data = container_of(
-        head, struct rcu_barrier_data, head);
-    atomic_inc(data->cpu_count);
+    atomic_dec(&cpu_count);
 }
 
-static int rcu_barrier_action(void *_cpu_count)
+static void rcu_barrier_action(void)
 {
-    struct rcu_barrier_data data = { .cpu_count = _cpu_count };
-
-    ASSERT(!local_irq_is_enabled());
-    local_irq_enable();
+    struct rcu_head head;
 
     /*
      * When callback is executed, all previously-queued RCU work on this CPU
      * is completed. When all CPUs have executed their callback, data.cpu_count
      * will have been incremented to include every online CPU.
      */
-    call_rcu(&data.head, rcu_barrier_callback);
+    call_rcu(&head, rcu_barrier_callback);
 
-    while ( atomic_read(data.cpu_count) != num_online_cpus() )
+    while ( atomic_read(&cpu_count) )
     {
         process_pending_softirqs();
         cpu_relax();
     }
-
-    local_irq_disable();
-
-    return 0;
 }
 
-int rcu_barrier(void)
+void rcu_barrier(void)
 {
-    atomic_t cpu_count = ATOMIC_INIT(0);
-    return stop_machine_run(rcu_barrier_action, &cpu_count, NR_CPUS);
+    if ( !atomic_cmpxchg(&cpu_count, 0, num_online_cpus()) )
+        cpumask_raise_softirq(&cpu_online_map, RCU_SOFTIRQ);
+
+    while ( atomic_read(&cpu_count) )
+    {
+        process_pending_softirqs();
+        cpu_relax();
+    }
 }
 
 /* Is batch a before batch b ? */
@@ -422,6 +426,9 @@ static void rcu_process_callbacks(void)
         rdp->process_callbacks = false;
         __rcu_process_callbacks(&rcu_ctrlblk, rdp);
     }
+
+    if ( atomic_read(&cpu_count) )
+        rcu_barrier_action();
 }
 
 static int __rcu_pending(struct rcu_ctrlblk *rcp, struct rcu_data *rdp)
xen/include/xen/rcupdate.h +1 -1
index 174d058113..87f35b7704 100644
@@ -143,7 +143,7 @@ void rcu_check_callbacks(int cpu);
 void call_rcu(struct rcu_head *head, 
               void (*func)(struct rcu_head *head));
 
-int rcu_barrier(void);
+void rcu_barrier(void);
 
 void rcu_idle_enter(unsigned int cpu);
 void rcu_idle_exit(unsigned int cpu);
Signature · 7 lines
2.16.4


_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
#3 lore raw

Re: [Xen-devel] [PATCH 2/2] xen/rcu: don't use stop_machine_run() for rcu_barrier()

Hi Juergen,

Quoted 82 lines · Juergen Gross

Today rcu_barrier() is calling stop_machine_run() to synchronize all physical cpus in order to ensure all pending rcu calls have finished when returning.

As stop_machine_run() is using tasklets this requires scheduling of idle vcpus on all cpus imposing the need to call rcu_barrier() on idle cpus only in case of core scheduling being active, as otherwise a scheduling deadlock would occur.

There is no need at all to do the syncing of the cpus in tasklets, as rcu activity is started in __do_softirq() called whenever softirq activity is allowed. So rcu_barrier() can easily be modified to use softirq for synchronization of the cpus no longer requiring any scheduling activity.

As there already is a rcu softirq reuse that for the synchronization.

Finally switch rcu_barrier() to return void as it now can never fail.

---
  xen/common/rcupdate.c      | 49 ++++++++++++++++++++++++++--------------------
  xen/include/xen/rcupdate.h |  2 +-
  2 files changed, 29 insertions(+), 22 deletions(-)
xen/common/rcupdate.c +16 -16
index 079ea9d8a1..1f02a804e3 100644
@@ -143,47 +143,51 @@ static int qhimark = 10000;
  static int qlowmark = 100;
  static int rsinterval = 1000;
 
-struct rcu_barrier_data {
-    struct rcu_head head;
-    atomic_t *cpu_count;
-};
+/*
+ * rcu_barrier() handling:
+ * cpu_count holds the number of cpu required to finish barrier handling.
+ * Cpus are synchronized via softirq mechanism. rcu_barrier() is regarded to
+ * be active if cpu_count is not zero. In case rcu_barrier() is called on
+ * multiple cpus it is enough to check for cpu_count being not zero on entry
+ * and to call process_pending_softirqs() in a loop until cpu_count drops to
+ * zero, as syncing has been requested already and we don't need to sync
+ * multiple times.
+ */
+static atomic_t cpu_count = ATOMIC_INIT(0);
 
  static void rcu_barrier_callback(struct rcu_head *head)
  {
-    struct rcu_barrier_data *data = container_of(
-        head, struct rcu_barrier_data, head);
-    atomic_inc(data->cpu_count);
+    atomic_dec(&cpu_count);
  }
 
-static int rcu_barrier_action(void *_cpu_count)
+static void rcu_barrier_action(void)
  {
-    struct rcu_barrier_data data = { .cpu_count = _cpu_count };
-
-    ASSERT(!local_irq_is_enabled());
-    local_irq_enable();
+    struct rcu_head head;
 
      /*
       * When callback is executed, all previously-queued RCU work on this CPU
       * is completed. When all CPUs have executed their callback, data.cpu_count
       * will have been incremented to include every online CPU.
       */
-    call_rcu(&data.head, rcu_barrier_callback);
+    call_rcu(&head, rcu_barrier_callback);
 
-    while ( atomic_read(data.cpu_count) != num_online_cpus() )
+    while ( atomic_read(&cpu_count) )
      {
          process_pending_softirqs();
          cpu_relax();
      }
-
-    local_irq_disable();

xen/common/rcupdate.c static int qhimark = 10000; · line 183

Quoted 10 lines
xen/common/rcupdate.c +2 -5
-
-    return 0;
  }
 
-int rcu_barrier(void)
+void rcu_barrier(void)
  {
-    atomic_t cpu_count = ATOMIC_INIT(0);
-    return stop_machine_run(rcu_barrier_action, &cpu_count, NR_CPUS);
+    if ( !atomic_cmpxchg(&cpu_count, 0, num_online_cpus()) )

What does prevent the cpu_online_map to change under your feet? Shouldn't you grab the lock via get_cpu_maps()?

Quoted 26 lines
xen/common/rcupdate.c +10 -0
+        cpumask_raise_softirq(&cpu_online_map, RCU_SOFTIRQ);
+
+    while ( atomic_read(&cpu_count) )
+    {
+        process_pending_softirqs();
+        cpu_relax();
+    }
  }
 
  /* Is batch a before batch b ? */
@@ -422,6 +426,9 @@ static void rcu_process_callbacks(void)
          rdp->process_callbacks = false;
          __rcu_process_callbacks(&rcu_ctrlblk, rdp);
      }
+
+    if ( atomic_read(&cpu_count) )
+        rcu_barrier_action();
  }
 
  static int __rcu_pending(struct rcu_ctrlblk *rcp, struct rcu_data *rdp)
xen/include/xen/rcupdate.h +0 -0
index 174d058113..87f35b7704 100644

xen/include/xen/rcupdate.h void rcu_check_callbacks(int cpu); · line 149

Quoted 8 lines
xen/include/xen/rcupdate.h +1 -1
@@ -143,7 +143,7 @@ void rcu_check_callbacks(int cpu);
  void call_rcu(struct rcu_head *head,
                void (*func)(struct rcu_head *head));
 
-int rcu_barrier(void);
+void rcu_barrier(void);
 
  void rcu_idle_enter(unsigned int cpu);
  void rcu_idle_exit(unsigned int cpu);

Cheers,

Signature · 6 lines
Julien Grall

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
#4 lore raw

Re: [Xen-devel] [PATCH 2/2] xen/rcu: don't use stop_machine_run() for rcu_barrier()

Quoted 10 lines · Julien Grall

Hi Juergen,

Quoted 5 lines · Juergen Gross

+void rcu_barrier(void)
  {
-    atomic_t cpu_count = ATOMIC_INIT(0);
-    return stop_machine_run(rcu_barrier_action, &cpu_count, NR_CPUS); +    if ( !atomic_cmpxchg(&cpu_count, 0, num_online_cpus()) )

What does prevent the cpu_online_map to change under your feet? Shouldn't you grab the lock via get_cpu_maps()?

Oh, indeed.

This in turn will require a modification of the logic to detect parallel calls on multiple cpus.

Juergen

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

#5 lore raw

Re: [Xen-devel] [PATCH 2/2] xen/rcu: don't use stop_machine_run() for rcu_barrier()

Quoted 10 lines · Jürgen Groß
Quoted 9 lines · Julien Grall

Hi Juergen,

Quoted 5 lines · Juergen Gross

+void rcu_barrier(void)
  {
-    atomic_t cpu_count = ATOMIC_INIT(0);
-    return stop_machine_run(rcu_barrier_action, &cpu_count, NR_CPUS); +    if ( !atomic_cmpxchg(&cpu_count, 0, num_online_cpus()) )

Quoted 7 lines
Quoted 2 lines

What does prevent the cpu_online_map to change under your feet? Shouldn't you grab the lock via get_cpu_maps()?

Oh, indeed.

This in turn will require a modification of the logic to detect parallel calls on multiple cpus.

If you pick my patch to turn that into a rw lock you shouldn't worry about parallel calls I think, but the lock acquisition can still fail if there's a CPU plug/unplug going on:

https://lists.xenproject.org/archives/html/xen-devel/2020-02/msg00940.html

Roger.

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

#6 lore raw

Re: [Xen-devel] [PATCH 2/2] xen/rcu: don't use stop_machine_run() for rcu_barrier()

Quoted 16 lines · Roger Pau Monné
Quoted 15 lines · Jürgen Groß
Quoted 11 lines · Julien Grall

Hi Juergen,

Quoted 5 lines · Juergen Gross

+void rcu_barrier(void)
  {
-    atomic_t cpu_count = ATOMIC_INIT(0);
-    return stop_machine_run(rcu_barrier_action, &cpu_count, NR_CPUS); +    if ( !atomic_cmpxchg(&cpu_count, 0, num_online_cpus()) )

What does prevent the cpu_online_map to change under your feet? Shouldn't you grab the lock via get_cpu_maps()?

Oh, indeed.

Quoted 8 lines
Quoted 2 lines

This in turn will require a modification of the logic to detect parallel calls on multiple cpus.

If you pick my patch to turn that into a rw lock you shouldn't worry about parallel calls I think, but the lock acquisition can still fail if there's a CPU plug/unplug going on:

https://lists.xenproject.org/archives/html/xen-devel/2020-02/msg00940.html

Thanks, but letting rcu_barrier() fail is a no go, so I still need to handle that case (I mean the case failing to get the lock). And handling of parallel calls is not needed to be functional correct, but to avoid not necessary cpu synchronization (each parallel call detected can just wait until the master has finished and then return).

BTW - The recursive spinlock today would allow for e.g. rcu_barrier() to be called inside a CPU plug/unplug section. Your rwlock is removing that possibility. Any chance that could be handled?

Juergen

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

#7 lore raw

Re: [Xen-devel] [PATCH 2/2] xen/rcu: don't use stop_machine_run() for rcu_barrier()

Quoted 32 lines · Jürgen Groß
Quoted 16 lines · Roger Pau Monné
Quoted 15 lines · Jürgen Groß
Quoted 11 lines · Julien Grall

Hi Juergen,

Quoted 5 lines · Juergen Gross

+void rcu_barrier(void)
  {
-    atomic_t cpu_count = ATOMIC_INIT(0);
-    return stop_machine_run(rcu_barrier_action, &cpu_count, NR_CPUS); +    if ( !atomic_cmpxchg(&cpu_count, 0, num_online_cpus()) )

What does prevent the cpu_online_map to change under your feet? Shouldn't you grab the lock via get_cpu_maps()?

Oh, indeed.

Quoted 8 lines
Quoted 2 lines

This in turn will require a modification of the logic to detect parallel calls on multiple cpus.

If you pick my patch to turn that into a rw lock you shouldn't worry about parallel calls I think, but the lock acquisition can still fail if there's a CPU plug/unplug going on:

https://lists.xenproject.org/archives/html/xen-devel/2020-02/msg00940.html

Thanks, but letting rcu_barrier() fail is a no go, so I still need to handle that case (I mean the case failing to get the lock). And handling of parallel calls is not needed to be functional correct, but to avoid not necessary cpu synchronization (each parallel call detected can just wait until the master has finished and then return).

Quoted 3 lines

BTW - The recursive spinlock today would allow for e.g. rcu_barrier() to be called inside a CPU plug/unplug section. Your rwlock is removing that possibility. Any chance that could be handled?

While this might be interesting for the rcu stuff, it certainly isn't for other pieces also relying on the cpu maps lock.

Ie: get_cpu_maps must fail when called by send_IPI_mask if there's a CPU plug/unplug operation going on, even if it's on the same pCPU that's holding the lock in write mode.

I guess you could add a pCPU variable to record whether the current pCPU is in the middle of a CPU plug/unplug operation (and hence has the maps locked in write mode) and avoid taking the lock in that case?

Roger.

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

#8 lore raw

Re: [Xen-devel] [PATCH 2/2] xen/rcu: don't use stop_machine_run() for rcu_barrier()

Quoted 37 lines · Roger Pau Monné
Quoted 32 lines · Jürgen Groß
Quoted 16 lines · Roger Pau Monné
Quoted 15 lines · Jürgen Groß
Quoted 11 lines · Julien Grall

Hi Juergen,

Quoted 5 lines · Juergen Gross

+void rcu_barrier(void)
  {
-    atomic_t cpu_count = ATOMIC_INIT(0);
-    return stop_machine_run(rcu_barrier_action, &cpu_count, NR_CPUS); +    if ( !atomic_cmpxchg(&cpu_count, 0, num_online_cpus()) )

What does prevent the cpu_online_map to change under your feet? Shouldn't you grab the lock via get_cpu_maps()?

Oh, indeed.

Quoted 8 lines
Quoted 2 lines

This in turn will require a modification of the logic to detect parallel calls on multiple cpus.

If you pick my patch to turn that into a rw lock you shouldn't worry about parallel calls I think, but the lock acquisition can still fail if there's a CPU plug/unplug going on:

https://lists.xenproject.org/archives/html/xen-devel/2020-02/msg00940.html

Thanks, but letting rcu_barrier() fail is a no go, so I still need to handle that case (I mean the case failing to get the lock). And handling of parallel calls is not needed to be functional correct, but to avoid not necessary cpu synchronization (each parallel call detected can just wait until the master has finished and then return).

Quoted 3 lines

BTW - The recursive spinlock today would allow for e.g. rcu_barrier() to be called inside a CPU plug/unplug section. Your rwlock is removing that possibility. Any chance that could be handled?

Quoted 6 lines

While this might be interesting for the rcu stuff, it certainly isn't for other pieces also relying on the cpu maps lock.

Ie: get_cpu_maps must fail when called by send_IPI_mask if there's a CPU plug/unplug operation going on, even if it's on the same pCPU that's holding the lock in write mode.

Sure? How is cpu_down() working then? It is calling stop_machine_run() which is using send_IPI_mask()...

Juergen

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

#9 lore raw

Re: [Xen-devel] [PATCH 2/2] xen/rcu: don't use stop_machine_run() for rcu_barrier()

Quoted 38 lines · Jürgen Groß
Quoted 37 lines · Roger Pau Monné
Quoted 35 lines · Jürgen Groß
Quoted 16 lines · Roger Pau Monné
Quoted 15 lines · Jürgen Groß
Quoted 11 lines · Julien Grall

Hi Juergen,

Quoted 5 lines · Juergen Gross

+void rcu_barrier(void)
  {
-    atomic_t cpu_count = ATOMIC_INIT(0);
-    return stop_machine_run(rcu_barrier_action, &cpu_count, NR_CPUS); +    if ( !atomic_cmpxchg(&cpu_count, 0, num_online_cpus()) )

What does prevent the cpu_online_map to change under your feet? Shouldn't you grab the lock via get_cpu_maps()?

Oh, indeed.

Quoted 8 lines
Quoted 2 lines

This in turn will require a modification of the logic to detect parallel calls on multiple cpus.

If you pick my patch to turn that into a rw lock you shouldn't worry about parallel calls I think, but the lock acquisition can still fail if there's a CPU plug/unplug going on:

https://lists.xenproject.org/archives/html/xen-devel/2020-02/msg00940.html

Thanks, but letting rcu_barrier() fail is a no go, so I still need to handle that case (I mean the case failing to get the lock). And handling of parallel calls is not needed to be functional correct, but to avoid not necessary cpu synchronization (each parallel call detected can just wait until the master has finished and then return).

BTW - The recursive spinlock today would allow for e.g. rcu_barrier() to be called inside a CPU plug/unplug section. Your rwlock is removing that possibility. Any chance that could be handled?

Quoted 8 lines
Quoted 6 lines

While this might be interesting for the rcu stuff, it certainly isn't for other pieces also relying on the cpu maps lock.

Ie: get_cpu_maps must fail when called by send_IPI_mask if there's a CPU plug/unplug operation going on, even if it's on the same pCPU that's holding the lock in write mode.

Sure? How is cpu_down() working then?

send_IPI_mask failing to acquire the cpu maps lock prevents it from using the APIC shorthand, which is what we want in that case.

Quoted 2 lines

It is calling stop_machine_run()
which is using send_IPI_mask()...

Xen should avoid using the APIC shorthand in that case, which I don't think it's happening now, as the lock is recursive.

Thanks, Roger.

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

#10 lore raw

Re: [Xen-devel] [PATCH 2/2] xen/rcu: don't use stop_machine_run() for rcu_barrier()

Quoted 48 lines · Roger Pau Monné
Quoted 38 lines · Jürgen Groß
Quoted 37 lines · Roger Pau Monné
Quoted 35 lines · Jürgen Groß
Quoted 16 lines · Roger Pau Monné
Quoted 15 lines · Jürgen Groß
Quoted 11 lines · Julien Grall

Hi Juergen,

Quoted 5 lines · Juergen Gross

+void rcu_barrier(void)
  {
-    atomic_t cpu_count = ATOMIC_INIT(0);
-    return stop_machine_run(rcu_barrier_action, &cpu_count, NR_CPUS); +    if ( !atomic_cmpxchg(&cpu_count, 0, num_online_cpus()) )

What does prevent the cpu_online_map to change under your feet? Shouldn't you grab the lock via get_cpu_maps()?

Oh, indeed.

Quoted 8 lines
Quoted 2 lines

This in turn will require a modification of the logic to detect parallel calls on multiple cpus.

If you pick my patch to turn that into a rw lock you shouldn't worry about parallel calls I think, but the lock acquisition can still fail if there's a CPU plug/unplug going on:

https://lists.xenproject.org/archives/html/xen-devel/2020-02/msg00940.html

Thanks, but letting rcu_barrier() fail is a no go, so I still need to handle that case (I mean the case failing to get the lock). And handling of parallel calls is not needed to be functional correct, but to avoid not necessary cpu synchronization (each parallel call detected can just wait until the master has finished and then return).

BTW - The recursive spinlock today would allow for e.g. rcu_barrier() to be called inside a CPU plug/unplug section. Your rwlock is removing that possibility. Any chance that could be handled?

Quoted 8 lines
Quoted 6 lines

While this might be interesting for the rcu stuff, it certainly isn't for other pieces also relying on the cpu maps lock.

Ie: get_cpu_maps must fail when called by send_IPI_mask if there's a CPU plug/unplug operation going on, even if it's on the same pCPU that's holding the lock in write mode.

Sure? How is cpu_down() working then?

Quoted 8 lines

send_IPI_mask failing to acquire the cpu maps lock prevents it from using the APIC shorthand, which is what we want in that case.

Quoted 2 lines

It is calling stop_machine_run()
which is using send_IPI_mask()...

Xen should avoid using the APIC shorthand in that case, which I don't think it's happening now, as the lock is recursive.

In fact the code area where this is true is much smaller than that protected by the lock.

Basically only __cpu_disable() and __cpu_up() (on x86) are critical in this regard.

Juergen

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

#11 lore raw

Re: [Xen-devel] [PATCH 2/2] xen/rcu: don't use stop_machine_run() for rcu_barrier()

In reply to Juergen Gross
Quoted 15 lines · Juergen Gross

Today rcu_barrier() is calling stop_machine_run() to synchronize all physical cpus in order to ensure all pending rcu calls have finished when returning.

As stop_machine_run() is using tasklets this requires scheduling of idle vcpus on all cpus imposing the need to call rcu_barrier() on idle cpus only in case of core scheduling being active, as otherwise a scheduling deadlock would occur.

There is no need at all to do the syncing of the cpus in tasklets, as rcu activity is started in __do_softirq() called whenever softirq activity is allowed. So rcu_barrier() can easily be modified to use softirq for synchronization of the cpus no longer requiring any scheduling activity.

Quoted 4 lines

As there already is a rcu softirq reuse that for the synchronization.

Finally switch rcu_barrier() to return void as it now can never fail.

Would this implementation guarantee progress as previous implementation guaranteed?

Igor

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

#12 lore raw

Re: [Xen-devel] [PATCH 2/2] xen/rcu: don't use stop_machine_run() for rcu_barrier()

Quoted 16 lines · Igor Druzhinin
Quoted 15 lines · Juergen Gross

Today rcu_barrier() is calling stop_machine_run() to synchronize all physical cpus in order to ensure all pending rcu calls have finished when returning.

As stop_machine_run() is using tasklets this requires scheduling of idle vcpus on all cpus imposing the need to call rcu_barrier() on idle cpus only in case of core scheduling being active, as otherwise a scheduling deadlock would occur.

There is no need at all to do the syncing of the cpus in tasklets, as rcu activity is started in __do_softirq() called whenever softirq activity is allowed. So rcu_barrier() can easily be modified to use softirq for synchronization of the cpus no longer requiring any scheduling activity.

Quoted 7 lines
Quoted 4 lines

As there already is a rcu softirq reuse that for the synchronization.

Finally switch rcu_barrier() to return void as it now can never fail.

Would this implementation guarantee progress as previous implementation guaranteed?

Yes.

Juergen

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

#13 lore raw

Re: [Xen-devel] [PATCH 2/2] xen/rcu: don't use stop_machine_run() for rcu_barrier()

Quoted 19 lines · Jürgen Groß
Quoted 18 lines · Igor Druzhinin
Quoted 17 lines · Juergen Gross

Today rcu_barrier() is calling stop_machine_run() to synchronize all physical cpus in order to ensure all pending rcu calls have finished when returning.

As stop_machine_run() is using tasklets this requires scheduling of idle vcpus on all cpus imposing the need to call rcu_barrier() on idle cpus only in case of core scheduling being active, as otherwise a scheduling deadlock would occur.

There is no need at all to do the syncing of the cpus in tasklets, as rcu activity is started in __do_softirq() called whenever softirq activity is allowed. So rcu_barrier() can easily be modified to use softirq for synchronization of the cpus no longer requiring any scheduling activity.

As there already is a rcu softirq reuse that for the synchronization.

Quoted 7 lines
Quoted 5 lines
Quoted 2 lines

Finally switch rcu_barrier() to return void as it now can never fail.

Would this implementation guarantee progress as previous implementation guaranteed?

Yes.

Thanks, I'll put it to test today to see if it solves our use case.

Igor

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

#14 lore raw

Re: [Xen-devel] [PATCH 2/2] xen/rcu: don't use stop_machine_run() for rcu_barrier()

Quoted 23 lines · Igor Druzhinin
Quoted 22 lines · Jürgen Groß
Quoted 21 lines · Igor Druzhinin
Quoted 19 lines · Juergen Gross

Today rcu_barrier() is calling stop_machine_run() to synchronize all physical cpus in order to ensure all pending rcu calls have finished when returning.

As stop_machine_run() is using tasklets this requires scheduling of idle vcpus on all cpus imposing the need to call rcu_barrier() on idle cpus only in case of core scheduling being active, as otherwise a scheduling deadlock would occur.

There is no need at all to do the syncing of the cpus in tasklets, as rcu activity is started in __do_softirq() called whenever softirq activity is allowed. So rcu_barrier() can easily be modified to use softirq for synchronization of the cpus no longer requiring any scheduling activity.

As there already is a rcu softirq reuse that for the synchronization.

Finally switch rcu_barrier() to return void as it now can never fail.

Quoted 6 lines
Quoted 4 lines
Quoted 2 lines

Would this implementation guarantee progress as previous implementation guaranteed?

Yes.

Thanks, I'll put it to test today to see if it solves our use case.

Just manually tried it - gives infinite (up to stack size) trace like:

(XEN) [    1.496520]    [<ffff82d08022e435>] F softirq.c#__do_softirq+0x85/0x90
(XEN) [    1.496561]    [<ffff82d08022e475>] F process_pending_softirqs+0x35/0x37
(XEN) [    1.496600]    [<ffff82d080221101>] F rcupdate.c#rcu_process_callbacks+0x1df/0x1f6
(XEN) [    1.496643]    [<ffff82d08022e435>] F softirq.c#__do_softirq+0x85/0x90
(XEN) [    1.496685]    [<ffff82d08022e475>] F process_pending_softirqs+0x35/0x37
(XEN) [    1.496726]    [<ffff82d080221101>] F rcupdate.c#rcu_process_callbacks+0x1df/0x1f6
(XEN) [    1.496766]    [<ffff82d08022e435>] F softirq.c#__do_softirq+0x85/0x90
(XEN) [    1.496806]    [<ffff82d08022e475>] F process_pending_softirqs+0x35/0x37
(XEN) [    1.496847]    [<ffff82d080221101>] F rcupdate.c#rcu_process_callbacks+0x1df/0x1f6
(XEN) [    1.496887]    [<ffff82d08022e435>] F softirq.c#__do_softirq+0x85/0x90
(XEN) [    1.496927]    [<ffff82d08022e475>] F process_pending_softirqs+0x35/0x37

Igor

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

#15 lore raw

Re: [Xen-devel] [PATCH 2/2] xen/rcu: don't use stop_machine_run() for rcu_barrier()

Quoted 33 lines · Igor Druzhinin
Quoted 23 lines · Igor Druzhinin
Quoted 22 lines · Jürgen Groß
Quoted 21 lines · Igor Druzhinin
Quoted 19 lines · Juergen Gross

Today rcu_barrier() is calling stop_machine_run() to synchronize all physical cpus in order to ensure all pending rcu calls have finished when returning.

As stop_machine_run() is using tasklets this requires scheduling of idle vcpus on all cpus imposing the need to call rcu_barrier() on idle cpus only in case of core scheduling being active, as otherwise a scheduling deadlock would occur.

There is no need at all to do the syncing of the cpus in tasklets, as rcu activity is started in __do_softirq() called whenever softirq activity is allowed. So rcu_barrier() can easily be modified to use softirq for synchronization of the cpus no longer requiring any scheduling activity.

As there already is a rcu softirq reuse that for the synchronization.

Finally switch rcu_barrier() to return void as it now can never fail.

Quoted 6 lines
Quoted 4 lines
Quoted 2 lines

Would this implementation guarantee progress as previous implementation guaranteed?

Yes.

Thanks, I'll put it to test today to see if it solves our use case.

Just manually tried it - gives infinite (up to stack size) trace like:

Quoted 11 lines
(XEN) [    1.496520]    [<ffff82d08022e435>] F softirq.c#__do_softirq+0x85/0x90
(XEN) [    1.496561]    [<ffff82d08022e475>] F process_pending_softirqs+0x35/0x37
(XEN) [    1.496600]    [<ffff82d080221101>] F rcupdate.c#rcu_process_callbacks+0x1df/0x1f6
(XEN) [    1.496643]    [<ffff82d08022e435>] F softirq.c#__do_softirq+0x85/0x90
(XEN) [    1.496685]    [<ffff82d08022e475>] F process_pending_softirqs+0x35/0x37
(XEN) [    1.496726]    [<ffff82d080221101>] F rcupdate.c#rcu_process_callbacks+0x1df/0x1f6
(XEN) [    1.496766]    [<ffff82d08022e435>] F softirq.c#__do_softirq+0x85/0x90
(XEN) [    1.496806]    [<ffff82d08022e475>] F process_pending_softirqs+0x35/0x37
(XEN) [    1.496847]    [<ffff82d080221101>] F rcupdate.c#rcu_process_callbacks+0x1df/0x1f6
(XEN) [    1.496887]    [<ffff82d08022e435>] F softirq.c#__do_softirq+0x85/0x90
(XEN) [    1.496927]    [<ffff82d08022e475>] F process_pending_softirqs+0x35/0x37

Interesting I didn't run into this problem. Obviously I managed to forget handling the case of recursion.

Juergen

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel