stop_machine: Remove stop_cpus_lock and lg_double_lock/unlock()

Please see V2. It is much simpler, and it doesn't need cond_resched().

To me this looks better than changing stop_cpus() to take all stopper locks at once. Because this actually turns stopper->lock into another lglock.

Yes, with this patch cpu_stop_queue_two_works() spins in busy-wait loop if it races with stop_cpus(). But lg_double_lock() spins too, and performance-wise I think this change is a win.

To simplify the review, let me show the code with this patch applied. The patch simply adds "bool stop_cpus_in_progress" set by queue_stop_cpus_work() and checked by cpu_stop_queue_two_works().

	static int cpu_stop_queue_two_works(int cpu1, struct cpu_stop_work *work1,
					    int cpu2, struct cpu_stop_work *work2)
	{
		struct cpu_stopper *stopper1 = per_cpu_ptr(&cpu_stopper, cpu1);
		struct cpu_stopper *stopper2 = per_cpu_ptr(&cpu_stopper, cpu2);
		int err;
	retry:
		spin_lock_irq(&stopper1->lock);
		spin_lock_nested(&stopper2->lock, SINGLE_DEPTH_NESTING);

		err = -ENOENT;
		if (!stopper1->enabled || !stopper2->enabled)
			goto unlock;
		/*
		 * Ensure that if we race with __stop_cpus() the stoppers won't get
		 * queued up in reverse order leading to system deadlock.
		 *
		 * We can't miss stop_cpus_in_progress if queue_stop_cpus_work() has
		 * queued a work on cpu1 but not on cpu2, we hold both locks.
		 *
		 * It can be falsely true but it is safe to spin until it is cleared,
		 * queue_stop_cpus_work() does everything under preempt_disable().
		 */
		err = -EDEADLK;
		if (unlikely(stop_cpus_in_progress))
				goto unlock;

		err = 0;
		__cpu_stop_queue_work(stopper1, work1);
		__cpu_stop_queue_work(stopper2, work2);
	unlock:
		spin_unlock(&stopper2->lock);
		spin_unlock_irq(&stopper1->lock);

		if (unlikely(err == -EDEADLK)) {
			while (stop_cpus_in_progress)
				cpu_relax();
			goto retry;
		}
		return err;
	}

	static bool queue_stop_cpus_work(const struct cpumask *cpumask,
					 cpu_stop_fn_t fn, void *arg,
					 struct cpu_stop_done *done)
	{
		struct cpu_stop_work *work;
		unsigned int cpu;
		bool queued = false;

		/*
		 * Disable preemption while queueing to avoid getting
		 * preempted by a stopper which might wait for other stoppers
		 * to enter @fn which can lead to deadlock.
		 */
		preempt_disable();
		stop_cpus_in_progress = true;
		for_each_cpu(cpu, cpumask) {
			work = &per_cpu(cpu_stopper.stop_work, cpu);
			work->fn = fn;
			work->arg = arg;
			work->done = done;
			if (cpu_stop_queue_work(cpu, work))
				queued = true;
		}
		stop_cpus_in_progress = false;
		preempt_enable();

		return queued;
	}

Oleg.

stop_two_cpus() and stop_cpus() use stop_cpus_lock to avoid the deadlock, we need to ensure that the stopper functions can't be queued "backwards" from one another. This doesn't look nice; if we use lglock then we do not really need stopper->lock, cpu_stop_queue_work() could use lg_local_lock() under local_irq_save().

OTOH it would be even better to avoid lglock in stop_machine.c and remove lg_double_lock(). This patch adds "bool stop_cpus_in_progress" set/cleared by queue_stop_cpus_work(), and changes cpu_stop_queue_two_works() to busy wait until it is cleared.

queue_stop_cpus_work() sets stop_cpus_in_progress = T lockless, but after it queues a work on CPU1 it must be visible to stop_two_cpus(CPU1, CPU2) which checks it under the same lock. And since stop_two_cpus() holds the 2nd lock too, queue_stop_cpus_work() can not clear stop_cpus_in_progress if it is also going to queue a work on CPU2, it needs to take that 2nd lock to do this.

---
 include/linux/lglock.h  |    5 -----
 kernel/locking/lglock.c |   22 ----------------------
 kernel/stop_machine.c   |   42 ++++++++++++++++++++++++++----------------
 3 files changed, 26 insertions(+), 43 deletions(-)

index c92ebd1..0081f00 100644
@@ -52,15 +52,10 @@ struct lglock {
 	static struct lglock name = { .lock = &name ## _lock }
 
 void lg_lock_init(struct lglock *lg, char *name);
-
 void lg_local_lock(struct lglock *lg);
 void lg_local_unlock(struct lglock *lg);
 void lg_local_lock_cpu(struct lglock *lg, int cpu);
 void lg_local_unlock_cpu(struct lglock *lg, int cpu);
-
-void lg_double_lock(struct lglock *lg, int cpu1, int cpu2);
-void lg_double_unlock(struct lglock *lg, int cpu1, int cpu2);
-
 void lg_global_lock(struct lglock *lg);
 void lg_global_unlock(struct lglock *lg);
 

index 951cfcd..86ae2ae 100644
@@ -60,28 +60,6 @@ void lg_local_unlock_cpu(struct lglock *lg, int cpu)
 }
 EXPORT_SYMBOL(lg_local_unlock_cpu);
 
-void lg_double_lock(struct lglock *lg, int cpu1, int cpu2)
-{
-	BUG_ON(cpu1 == cpu2);
-
-	/* lock in cpu order, just like lg_global_lock */
-	if (cpu2 < cpu1)
-		swap(cpu1, cpu2);
-
-	preempt_disable();
-	lock_acquire_shared(&lg->lock_dep_map, 0, 0, NULL, _RET_IP_);
-	arch_spin_lock(per_cpu_ptr(lg->lock, cpu1));
-	arch_spin_lock(per_cpu_ptr(lg->lock, cpu2));
-}
-
-void lg_double_unlock(struct lglock *lg, int cpu1, int cpu2)
-{
-	lock_release(&lg->lock_dep_map, 1, _RET_IP_);
-	arch_spin_unlock(per_cpu_ptr(lg->lock, cpu1));
-	arch_spin_unlock(per_cpu_ptr(lg->lock, cpu2));
-	preempt_enable();
-}
-
 void lg_global_lock(struct lglock *lg)
 {
 	int i;

1.5.5.1

Hello, Oleg.

Yeah, removing stopper->lock would be nice.

Isn't this a lot more subtler than the other direction? Unless there's a clear performance advantage to removing stopper->lock, using lglock for both stop_two and stop_machine seems like an easier-to-follow approach to me.

Thanks.

tejun

The idea is to kill lglock. There's only two users, this and fs/locks.c for which I have patches -- which are being benchmarked 'now-ish' :-)

Hello,

Ah, I see. Please disregard my comment. The patch looks good to me. Please feel free to add

Thanks.

tejun