Skip to contentSkip to conversation

wait queue head; locking considerations

RFC In review Paul Gortmaker 2 participants 3 messages Dec 6, 2013

Subsystems (14)
lore
Share & tools
lore
#0 lore raw

cover wait queue head; locking considerations

Peter pointed out[1] that the simple wait queues in RT could run any number of wakeups with the wait queue head lock held.

It turns out that this isnt really specific to RT, but it is mainline too -- all the __wake_up_XYZ that in turn use the __wake_up_common boiler plate end up looping over all the callbacks (normally ttwu) while holding the head lock.

I'm not sure exactly whether contention is a serious issue, but the possibility exists for someone to use a non-default callback (i.e. not ttwu) and have some considerable amount of computation going on while holding the head lock.

I entertained several ideas, none of which I am really happy with (except perhaps #4).

1) Limit the number of callbacks; via a MAX_WORK or similar. This would require bubbling up return values (some of which were just recently removed) and relying on the holder of the lock to take appropriate action (unlock, breathe, restart.) Plus, it doesn't really solve anything if someone installs a non default callback that takes forever.

2) Try and defer the callbacks temporarily somehow until the waitqueue head lock is released. I experimented with this, via creating a snapshot of the waiters, and running that on the next head unlock, but it was ugly, lockdep didn't like it at all, etc. Plus it isn't clear we can delay any without causing additional latency.

3) Declare locked wake ups evil, and try and determine if we can somehow fix the users in a way that we can phase them out. Perhaps by somehow cascading the head lock down into the individual waitqueue_t elements?

4) Do nothing. It isn't really a significant problem.

In any case, while attempting #2, I did abstract out the locking and unlocking. Maybe this makes sense regardless, since we already hide the spinlock init in wait queue head init; we don't use raw list operations, but instead use add_to_waitqueue() abstractions. In fact it seems only the locking doesn't have waitq in its function name.

It does help highlight those users who are manually manipulating their waitq head lock, and perhaps slightly improve readability. It passes x86/_64 allmodconfig, but if we think we want to use this, I'll need to build test the other arches too.

Thoughts?

Paul.
--
[1] http://marc.info/?l=linux-kernel&m=138089860308430&w=2

Paul Gortmaker (1):
waitq: abstract out locking/unlocking waitq head operations

 drivers/gpu/drm/radeon/radeon_sa.c                 | 18 ++++-----
 drivers/iio/industrialio-event.c                   | 26 ++++++-------
 .../lustre/lustre/libcfs/linux/linux-prim.c        |  4 +-
 drivers/usb/gadget/f_fs.c                          | 42 ++++++++++-----------
 fs/eventfd.c                                       | 32 ++++++++--------
 fs/eventpoll.c                                     |  4 +-
 fs/nilfs2/segment.c                                |  4 +-
 fs/timerfd.c                                       | 26 ++++++-------
 include/linux/wait.h                               | 36 +++++++++++++++---
 kernel/sched/completion.c                          | 24 ++++++------
 kernel/sched/core.c                                |  8 ++--
 kernel/sched/wait.c                                | 44 +++++++++++-----------
 mm/filemap.c                                       |  4 +-
 net/sunrpc/sched.c                                 |  4 +-
 14 files changed, 150 insertions(+), 126 deletions(-)
Signature · 1 line
1.8.5.rc3
#1 lore raw

PATCH 1/1 waitq: abstract out locking/unlocking waitq head operations

We abstract out the wait queue head locking operations. This is done in order to be able to add future functionality to them in a single place. As a side benefit, it happens to increase code readability.

We rename the wait queue lock entry to "__lock" which ensures that anyone who tries to use the old wq->lock or wq.lock will fail, and hence be forced into being aware of this change.

---

[Still needs build coverage testing on non-x86]

 drivers/gpu/drm/radeon/radeon_sa.c                 | 18 ++++-----
 drivers/iio/industrialio-event.c                   | 26 ++++++-------
 .../lustre/lustre/libcfs/linux/linux-prim.c        |  4 +-
 drivers/usb/gadget/f_fs.c                          | 42 ++++++++++-----------
 fs/eventfd.c                                       | 32 ++++++++--------
 fs/eventpoll.c                                     |  4 +-
 fs/nilfs2/segment.c                                |  4 +-
 fs/timerfd.c                                       | 26 ++++++-------
 include/linux/wait.h                               | 36 +++++++++++++++---
 kernel/sched/completion.c                          | 24 ++++++------
 kernel/sched/core.c                                |  8 ++--
 kernel/sched/wait.c                                | 44 +++++++++++-----------
 mm/filemap.c                                       |  4 +-
 net/sunrpc/sched.c                                 |  4 +-
 14 files changed, 150 insertions(+), 126 deletions(-)
drivers/gpu/drm/radeon/radeon_sa.c +9 -9
index f0bac68..b2d6701 100644
@@ -330,7 +330,7 @@ int radeon_sa_bo_new(struct radeon_device *rdev,
 	INIT_LIST_HEAD(&(*sa_bo)->olist);
 	INIT_LIST_HEAD(&(*sa_bo)->flist);
 
-	spin_lock(&sa_manager->wq.lock);
+	waitq_lock(&sa_manager->wq);
 	do {
 		for (i = 0; i < RADEON_NUM_RINGS; ++i) {
 			fences[i] = NULL;
@@ -342,16 +342,16 @@ int radeon_sa_bo_new(struct radeon_device *rdev,
 
 			if (radeon_sa_bo_try_alloc(sa_manager, *sa_bo,
 						   size, align)) {
-				spin_unlock(&sa_manager->wq.lock);
+				waitq_unlock(&sa_manager->wq);
 				return 0;
 			}
 
 			/* see if we can skip over some allocations */
 		} while (radeon_sa_bo_next_hole(sa_manager, fences, tries));
 
-		spin_unlock(&sa_manager->wq.lock);
+		waitq_unlock(&sa_manager->wq);
 		r = radeon_fence_wait_any(rdev, fences, false);
-		spin_lock(&sa_manager->wq.lock);
+		waitq_lock(&sa_manager->wq);
 		/* if we have nothing to wait for block */
 		if (r == -ENOENT && block) {
 			r = wait_event_interruptible_locked(
@@ -365,7 +365,7 @@ int radeon_sa_bo_new(struct radeon_device *rdev,
 
 	} while (!r);
 
-	spin_unlock(&sa_manager->wq.lock);
+	waitq_unlock(&sa_manager->wq);
 	kfree(*sa_bo);
 	*sa_bo = NULL;
 	return r;
@@ -381,7 +381,7 @@ void radeon_sa_bo_free(struct radeon_device *rdev, struct radeon_sa_bo **sa_bo,
 	}
 
 	sa_manager = (*sa_bo)->manager;
-	spin_lock(&sa_manager->wq.lock);
+	waitq_lock(&sa_manager->wq);
 	if (fence && !radeon_fence_signaled(fence)) {
 		(*sa_bo)->fence = radeon_fence_ref(fence);
 		list_add_tail(&(*sa_bo)->flist,
@@ -390,7 +390,7 @@ void radeon_sa_bo_free(struct radeon_device *rdev, struct radeon_sa_bo **sa_bo,
 		radeon_sa_bo_remove_locked(*sa_bo);
 	}
 	wake_up_all_locked(&sa_manager->wq);
-	spin_unlock(&sa_manager->wq.lock);
+	waitq_unlock(&sa_manager->wq);
 	*sa_bo = NULL;
 }
 
@@ -400,7 +400,7 @@ void radeon_sa_bo_dump_debug_info(struct radeon_sa_manager *sa_manager,
 {
 	struct radeon_sa_bo *i;
 
-	spin_lock(&sa_manager->wq.lock);
+	waitq_lock(&sa_manager->wq);
 	list_for_each_entry(i, &sa_manager->olist, olist) {
 		if (&i->olist == sa_manager->hole) {
 			seq_printf(m, ">");
@@ -415,6 +415,6 @@ void radeon_sa_bo_dump_debug_info(struct radeon_sa_manager *sa_manager,
 		}
 		seq_printf(m, "\n");
 	}
-	spin_unlock(&sa_manager->wq.lock);
+	waitq_unlock(&sa_manager->wq);
 }
 #endif
drivers/iio/industrialio-event.c +13 -13
index c10eab6..2523fbb 100644
@@ -50,7 +50,7 @@ int iio_push_event(struct iio_dev *indio_dev, u64 ev_code, s64 timestamp)
 	int copied;
 
 	/* Does anyone care? */
-	spin_lock_irqsave(&ev_int->wait.lock, flags);
+	waitq_lock_irqsave(&ev_int->wait, flags);
 	if (test_bit(IIO_BUSY_BIT_POS, &ev_int->flags)) {
 
 		ev.id = ev_code;
@@ -60,7 +60,7 @@ int iio_push_event(struct iio_dev *indio_dev, u64 ev_code, s64 timestamp)
 		if (copied != 0)
 			wake_up_locked_poll(&ev_int->wait, POLLIN);
 	}
-	spin_unlock_irqrestore(&ev_int->wait.lock, flags);
+	waitq_unlock_irqrestore(&ev_int->wait, flags);
 
 	return 0;
 }
@@ -81,10 +81,10 @@ static unsigned int iio_event_poll(struct file *filep,
 
 	poll_wait(filep, &ev_int->wait, wait);
 
-	spin_lock_irq(&ev_int->wait.lock);
+	waitq_lock_irq(&ev_int->wait);
 	if (!kfifo_is_empty(&ev_int->det_events))
 		events = POLLIN | POLLRDNORM;
-	spin_unlock_irq(&ev_int->wait.lock);
+	waitq_unlock_irq(&ev_int->wait);
 
 	return events;
 }
@@ -105,7 +105,7 @@ static ssize_t iio_event_chrdev_read(struct file *filep,
 	if (count < sizeof(struct iio_event_data))
 		return -EINVAL;
 
-	spin_lock_irq(&ev_int->wait.lock);
+	waitq_lock_irq(&ev_int->wait);
 	if (kfifo_is_empty(&ev_int->det_events)) {
 		if (filep->f_flags & O_NONBLOCK) {
 			ret = -EAGAIN;
@@ -127,7 +127,7 @@ static ssize_t iio_event_chrdev_read(struct file *filep,
 	ret = kfifo_to_user(&ev_int->det_events, buf, count, &copied);
 
 error_unlock:
-	spin_unlock_irq(&ev_int->wait.lock);
+	waitq_unlock_irq(&ev_int->wait);
 
 	return ret ? ret : copied;
 }
@@ -137,7 +137,7 @@ static int iio_event_chrdev_release(struct inode *inode, struct file *filep)
 	struct iio_dev *indio_dev = filep->private_data;
 	struct iio_event_interface *ev_int = indio_dev->event_interface;
 
-	spin_lock_irq(&ev_int->wait.lock);
+	waitq_lock_irq(&ev_int->wait);
 	__clear_bit(IIO_BUSY_BIT_POS, &ev_int->flags);
 	/*
 	 * In order to maintain a clean state for reopening,
@@ -145,7 +145,7 @@ static int iio_event_chrdev_release(struct inode *inode, struct file *filep)
 	 * any new __iio_push_event calls running.
 	 */
 	kfifo_reset_out(&ev_int->det_events);
-	spin_unlock_irq(&ev_int->wait.lock);
+	waitq_unlock_irq(&ev_int->wait);
 
 	iio_device_put(indio_dev);
 
@@ -168,20 +168,20 @@ int iio_event_getfd(struct iio_dev *indio_dev)
 	if (ev_int == NULL)
 		return -ENODEV;
 
-	spin_lock_irq(&ev_int->wait.lock);
+	waitq_lock_irq(&ev_int->wait);
 	if (__test_and_set_bit(IIO_BUSY_BIT_POS, &ev_int->flags)) {
-		spin_unlock_irq(&ev_int->wait.lock);
+		waitq_unlock_irq(&ev_int->wait);
 		return -EBUSY;
 	}
-	spin_unlock_irq(&ev_int->wait.lock);
+	waitq_unlock_irq(&ev_int->wait);
 	iio_device_get(indio_dev);
 
 	fd = anon_inode_getfd("iio:event", &iio_event_chrdev_fileops,
 				indio_dev, O_RDONLY | O_CLOEXEC);
 	if (fd < 0) {
-		spin_lock_irq(&ev_int->wait.lock);
+		waitq_lock_irq(&ev_int->wait);
 		__clear_bit(IIO_BUSY_BIT_POS, &ev_int->flags);
-		spin_unlock_irq(&ev_int->wait.lock);
+		waitq_unlock_irq(&ev_int->wait);
 		iio_device_put(indio_dev);
 	}
 	return fd;
drivers/staging/lustre/lustre/libcfs/linux/linux-prim.c +2 -2
index cc9829f..1794447 100644
@@ -74,9 +74,9 @@ add_wait_queue_exclusive_head(wait_queue_head_t *waitq, wait_queue_t *link)
 {
 	unsigned long flags;
 
-	spin_lock_irqsave(&LINUX_WAITQ_HEAD(waitq)->lock, flags);
+	waitq_lock_irqsave(LINUX_WAITQ_HEAD(waitq), flags);
 	__add_wait_queue_exclusive(LINUX_WAITQ_HEAD(waitq), LINUX_WAITQ(link));
-	spin_unlock_irqrestore(&LINUX_WAITQ_HEAD(waitq)->lock, flags);
+	waitq_unlock_irqrestore(LINUX_WAITQ_HEAD(waitq), flags);
 }
 EXPORT_SYMBOL(add_wait_queue_exclusive_head);
 
drivers/usb/gadget/f_fs.c +21 -21

150 lines not on this page · raw message

fs/eventfd.c +16 -16
index 35470d9..2888013 100644
@@ -55,13 +55,13 @@ __u64 eventfd_signal(struct eventfd_ctx *ctx, __u64 n)
 {
 	unsigned long flags;
 
-	spin_lock_irqsave(&ctx->wqh.lock, flags);
+	waitq_lock_irqsave(&ctx->wqh, flags);
 	if (ULLONG_MAX - ctx->count < n)
 		n = ULLONG_MAX - ctx->count;
 	ctx->count += n;
 	if (waitqueue_active(&ctx->wqh))
 		wake_up_locked_poll(&ctx->wqh, POLLIN);
-	spin_unlock_irqrestore(&ctx->wqh.lock, flags);
+	waitq_unlock_irqrestore(&ctx->wqh, flags);
 
 	return n;
 }
@@ -122,14 +122,14 @@ static unsigned int eventfd_poll(struct file *file, poll_table *wait)
 
 	poll_wait(file, &ctx->wqh, wait);
 
-	spin_lock_irqsave(&ctx->wqh.lock, flags);
+	waitq_lock_irqsave(&ctx->wqh, flags);
 	if (ctx->count > 0)
 		events |= POLLIN;
 	if (ctx->count == ULLONG_MAX)
 		events |= POLLERR;
 	if (ULLONG_MAX - 1 > ctx->count)
 		events |= POLLOUT;
-	spin_unlock_irqrestore(&ctx->wqh.lock, flags);
+	waitq_unlock_irqrestore(&ctx->wqh, flags);
 
 	return events;
 }
@@ -158,12 +158,12 @@ int eventfd_ctx_remove_wait_queue(struct eventfd_ctx *ctx, wait_queue_t *wait,
 {
 	unsigned long flags;
 
-	spin_lock_irqsave(&ctx->wqh.lock, flags);
+	waitq_lock_irqsave(&ctx->wqh, flags);
 	eventfd_ctx_do_read(ctx, cnt);
 	__remove_wait_queue(&ctx->wqh, wait);
 	if (*cnt != 0 && waitqueue_active(&ctx->wqh))
 		wake_up_locked_poll(&ctx->wqh, POLLOUT);
-	spin_unlock_irqrestore(&ctx->wqh.lock, flags);
+	waitq_unlock_irqrestore(&ctx->wqh, flags);
 
 	return *cnt != 0 ? 0 : -EAGAIN;
 }
@@ -188,7 +188,7 @@ ssize_t eventfd_ctx_read(struct eventfd_ctx *ctx, int no_wait, __u64 *cnt)
 	ssize_t res;
 	DECLARE_WAITQUEUE(wait, current);
 
-	spin_lock_irq(&ctx->wqh.lock);
+	waitq_lock_irq(&ctx->wqh);
 	*cnt = 0;
 	res = -EAGAIN;
 	if (ctx->count > 0)
@@ -205,9 +205,9 @@ ssize_t eventfd_ctx_read(struct eventfd_ctx *ctx, int no_wait, __u64 *cnt)
 				res = -ERESTARTSYS;
 				break;
 			}
-			spin_unlock_irq(&ctx->wqh.lock);
+			waitq_unlock_irq(&ctx->wqh);
 			schedule();
-			spin_lock_irq(&ctx->wqh.lock);
+			waitq_lock_irq(&ctx->wqh);
 		}
 		__remove_wait_queue(&ctx->wqh, &wait);
 		__set_current_state(TASK_RUNNING);
@@ -217,7 +217,7 @@ ssize_t eventfd_ctx_read(struct eventfd_ctx *ctx, int no_wait, __u64 *cnt)
 		if (waitqueue_active(&ctx->wqh))
 			wake_up_locked_poll(&ctx->wqh, POLLOUT);
 	}
-	spin_unlock_irq(&ctx->wqh.lock);
+	waitq_unlock_irq(&ctx->wqh);
 
 	return res;
 }
@@ -253,7 +253,7 @@ static ssize_t eventfd_write(struct file *file, const char __user *buf, size_t c
 		return -EFAULT;
 	if (ucnt == ULLONG_MAX)
 		return -EINVAL;
-	spin_lock_irq(&ctx->wqh.lock);
+	waitq_lock_irq(&ctx->wqh);
 	res = -EAGAIN;
 	if (ULLONG_MAX - ctx->count > ucnt)
 		res = sizeof(ucnt);
@@ -269,9 +269,9 @@ static ssize_t eventfd_write(struct file *file, const char __user *buf, size_t c
 				res = -ERESTARTSYS;
 				break;
 			}
-			spin_unlock_irq(&ctx->wqh.lock);
+			waitq_unlock_irq(&ctx->wqh);
 			schedule();
-			spin_lock_irq(&ctx->wqh.lock);
+			waitq_lock_irq(&ctx->wqh);
 		}
 		__remove_wait_queue(&ctx->wqh, &wait);
 		__set_current_state(TASK_RUNNING);
@@ -281,7 +281,7 @@ static ssize_t eventfd_write(struct file *file, const char __user *buf, size_t c
 		if (waitqueue_active(&ctx->wqh))
 			wake_up_locked_poll(&ctx->wqh, POLLIN);
 	}
-	spin_unlock_irq(&ctx->wqh.lock);
+	waitq_unlock_irq(&ctx->wqh);
 
 	return res;
 }
@@ -292,10 +292,10 @@ static int eventfd_show_fdinfo(struct seq_file *m, struct file *f)
 	struct eventfd_ctx *ctx = f->private_data;
 	int ret;
 
-	spin_lock_irq(&ctx->wqh.lock);
+	waitq_lock_irq(&ctx->wqh);
 	ret = seq_printf(m, "eventfd-count: %16llx\n",
 			 (unsigned long long)ctx->count);
-	spin_unlock_irq(&ctx->wqh.lock);
+	waitq_unlock_irq(&ctx->wqh);
 
 	return ret;
 }
fs/eventpoll.c +2 -2
index 8b5e258..17594db 100644
@@ -474,9 +474,9 @@ static inline void ep_wake_up_nested(wait_queue_head_t *wqueue,
 {
 	unsigned long flags;
 
-	spin_lock_irqsave_nested(&wqueue->lock, flags, subclass);
+	waitq_lock_irqsave_nested(wqueue, flags, subclass);
 	wake_up_locked_poll(wqueue, events);
-	spin_unlock_irqrestore(&wqueue->lock, flags);
+	waitq_unlock_irqrestore(wqueue, flags);
 }
 #else
 static inline void ep_wake_up_nested(wait_queue_head_t *wqueue,
fs/nilfs2/segment.c +2 -2
index 9f6b486..7560a36 100644
@@ -2106,7 +2106,7 @@ static void nilfs_segctor_wakeup(struct nilfs_sc_info *sci, int err)
 	struct nilfs_segctor_wait_request *wrq, *n;
 	unsigned long flags;
 
-	spin_lock_irqsave(&sci->sc_wait_request.lock, flags);
+	waitq_lock_irqsave(&sci->sc_wait_request, flags);
 	list_for_each_entry_safe(wrq, n, &sci->sc_wait_request.task_list,
 				 wq.task_list) {
 		if (!atomic_read(&wrq->done) &&
@@ -2120,7 +2120,7 @@ static void nilfs_segctor_wakeup(struct nilfs_sc_info *sci, int err)
 				     0, NULL);
 		}
 	}
-	spin_unlock_irqrestore(&sci->sc_wait_request.lock, flags);
+	waitq_unlock_irqrestore(&sci->sc_wait_request, flags);
 }
 
 /**
fs/timerfd.c +13 -13
index 9293121..16791b2 100644
@@ -60,11 +60,11 @@ static void timerfd_triggered(struct timerfd_ctx *ctx)
 {
 	unsigned long flags;
 
-	spin_lock_irqsave(&ctx->wqh.lock, flags);
+	waitq_lock_irqsave(&ctx->wqh, flags);
 	ctx->expired = 1;
 	ctx->ticks++;
 	wake_up_locked(&ctx->wqh);
-	spin_unlock_irqrestore(&ctx->wqh.lock, flags);
+	waitq_unlock_irqrestore(&ctx->wqh, flags);
 }
 
 static enum hrtimer_restart timerfd_tmrproc(struct hrtimer *htmr)
@@ -100,13 +100,13 @@ void timerfd_clock_was_set(void)
 	list_for_each_entry_rcu(ctx, &cancel_list, clist) {
 		if (!ctx->might_cancel)
 			continue;
-		spin_lock_irqsave(&ctx->wqh.lock, flags);
+		waitq_lock_irqsave(&ctx->wqh, flags);
 		if (ctx->moffs.tv64 != moffs.tv64) {
 			ctx->moffs.tv64 = KTIME_MAX;
 			ctx->ticks++;
 			wake_up_locked(&ctx->wqh);
 		}
-		spin_unlock_irqrestore(&ctx->wqh.lock, flags);
+		waitq_unlock_irqrestore(&ctx->wqh, flags);
 	}
 	rcu_read_unlock();
 }
@@ -221,10 +221,10 @@ static unsigned int timerfd_poll(struct file *file, poll_table *wait)
 
 	poll_wait(file, &ctx->wqh, wait);
 
-	spin_lock_irqsave(&ctx->wqh.lock, flags);
+	waitq_lock_irqsave(&ctx->wqh, flags);
 	if (ctx->ticks)
 		events |= POLLIN;
-	spin_unlock_irqrestore(&ctx->wqh.lock, flags);
+	waitq_unlock_irqrestore(&ctx->wqh, flags);
 
 	return events;
 }
@@ -238,7 +238,7 @@ static ssize_t timerfd_read(struct file *file, char __user *buf, size_t count,
 
 	if (count < sizeof(ticks))
 		return -EINVAL;
-	spin_lock_irq(&ctx->wqh.lock);
+	waitq_lock_irq(&ctx->wqh);
 	if (file->f_flags & O_NONBLOCK)
 		res = -EAGAIN;
 	else
@@ -278,7 +278,7 @@ static ssize_t timerfd_read(struct file *file, char __user *buf, size_t count,
 		ctx->expired = 0;
 		ctx->ticks = 0;
 	}
-	spin_unlock_irq(&ctx->wqh.lock);
+	waitq_unlock_irq(&ctx->wqh);
 	if (ticks)
 		res = put_user(ticks, (u64 __user *) buf) ? -EFAULT: sizeof(ticks);
 	return res;
@@ -370,7 +370,7 @@ static int do_timerfd_settime(int ufd, int flags,
 	 * it to the new values.
 	 */
 	for (;;) {
-		spin_lock_irq(&ctx->wqh.lock);
+		waitq_lock_irq(&ctx->wqh);
 
 		if (isalarm(ctx)) {
 			if (alarm_try_to_cancel(&ctx->t.alarm) >= 0)
@@ -379,7 +379,7 @@ static int do_timerfd_settime(int ufd, int flags,
 			if (hrtimer_try_to_cancel(&ctx->t.tmr) >= 0)
 				break;
 		}
-		spin_unlock_irq(&ctx->wqh.lock);
+		waitq_unlock_irq(&ctx->wqh);
 		cpu_relax();
 	}
 
@@ -404,7 +404,7 @@ static int do_timerfd_settime(int ufd, int flags,
 	 */
 	ret = timerfd_setup(ctx, flags, new);
 
-	spin_unlock_irq(&ctx->wqh.lock);
+	waitq_unlock_irq(&ctx->wqh);
 	fdput(f);
 	return ret;
 }
@@ -418,7 +418,7 @@ static int do_timerfd_gettime(int ufd, struct itimerspec *t)
 		return ret;
 	ctx = f.file->private_data;
 
-	spin_lock_irq(&ctx->wqh.lock);
+	waitq_lock_irq(&ctx->wqh);
 	if (ctx->expired && ctx->tintv.tv64) {
 		ctx->expired = 0;
 
@@ -436,7 +436,7 @@ static int do_timerfd_gettime(int ufd, struct itimerspec *t)
 	}
 	t->it_value = ktime_to_timespec(timerfd_get_remaining(ctx));
 	t->it_interval = ktime_to_timespec(ctx->tintv);
-	spin_unlock_irq(&ctx->wqh.lock);
+	waitq_unlock_irq(&ctx->wqh);
 	fdput(f);
 	return 0;
 }
include/linux/wait.h +30 -6
index eaa00b1..bc45ddb 100644
@@ -33,11 +33,35 @@ struct wait_bit_queue {
 };
 
 struct __wait_queue_head {
-	spinlock_t		lock;
+	spinlock_t		__lock;		/* Use wrapper for access */
 	struct list_head	task_list;
 };
 typedef struct __wait_queue_head wait_queue_head_t;
 
+#define waitq_lock_irqsave_nested(qh, f, sc)				\
+	spin_lock_irqsave_nested(&(qh)->__lock, f, sc)
+
+#define waitq_lock_irqsave(qh, f)					\
+	spin_lock_irqsave(&(qh)->__lock, f)
+
+#define waitq_unlock_irqrestore(qh, f)					\
+	spin_unlock_irqrestore(&(qh)->__lock, f)
+
+#define waitq_lock_irq(qh)						\
+	spin_lock_irq(&(qh)->__lock)
+
+#define waitq_unlock_irq(qh)						\
+	spin_unlock_irq(&(qh)->__lock)
+
+#define waitq_lock(qh)							\
+	spin_lock(&(qh)->__lock)
+
+#define waitq_unlock(qh)						\
+	spin_unlock(&(qh)->__lock)
+
+#define waitq_lock_init(qh)						\
+	spin_lock_init(&(qh)->__lock)
+
 struct task_struct;
 
 /*
@@ -53,7 +77,7 @@ struct task_struct;
 	wait_queue_t name = __WAITQUEUE_INITIALIZER(name, tsk)
 
 #define __WAIT_QUEUE_HEAD_INITIALIZER(name) {				\
-	.lock		= __SPIN_LOCK_UNLOCKED(name.lock),		\
+	.__lock		= __SPIN_LOCK_UNLOCKED(name.__lock),		\
 	.task_list	= { &(name).task_list, &(name).task_list } }
 
 #define DECLARE_WAIT_QUEUE_HEAD(name) \
@@ -465,14 +489,14 @@ do {									\
 			break;						\
 		}							\
 		if (irq)						\
-			spin_unlock_irq(&(wq).lock);			\
+			waitq_unlock_irq(&(wq));			\
 		else							\
-			spin_unlock(&(wq).lock);			\
+			waitq_unlock(&(wq));				\
 		schedule();						\
 		if (irq)						\
-			spin_lock_irq(&(wq).lock);			\
+			waitq_lock_irq(&(wq));				\
 		else							\
-			spin_lock(&(wq).lock);				\
+			waitq_lock(&(wq));				\
 	} while (!(condition));						\
 	__remove_wait_queue(&(wq), &__wait);				\
 	__set_current_state(TASK_RUNNING);				\
kernel/sched/completion.c +12 -12
index a63f4dc..6d174f5 100644
@@ -30,10 +30,10 @@ void complete(struct completion *x)
 {
 	unsigned long flags;
 
-	spin_lock_irqsave(&x->wait.lock, flags);
+	waitq_lock_irqsave(&x->wait, flags);
 	x->done++;
 	__wake_up_locked(&x->wait, TASK_NORMAL, 1);
-	spin_unlock_irqrestore(&x->wait.lock, flags);
+	waitq_unlock_irqrestore(&x->wait, flags);
 }
 EXPORT_SYMBOL(complete);
 
@@ -50,10 +50,10 @@ void complete_all(struct completion *x)
 {
 	unsigned long flags;
 
-	spin_lock_irqsave(&x->wait.lock, flags);
+	waitq_lock_irqsave(&x->wait, flags);
 	x->done += UINT_MAX/2;
 	__wake_up_locked(&x->wait, TASK_NORMAL, 0);
-	spin_unlock_irqrestore(&x->wait.lock, flags);
+	waitq_unlock_irqrestore(&x->wait, flags);
 }
 EXPORT_SYMBOL(complete_all);
 
@@ -71,9 +71,9 @@ do_wait_for_common(struct completion *x,
 				break;
 			}
 			__set_current_state(state);
-			spin_unlock_irq(&x->wait.lock);
+			waitq_unlock_irq(&x->wait);
 			timeout = action(timeout);
-			spin_lock_irq(&x->wait.lock);
+			waitq_lock_irq(&x->wait);
 		} while (!x->done && timeout);
 		__remove_wait_queue(&x->wait, &wait);
 		if (!x->done)
@@ -89,9 +89,9 @@ __wait_for_common(struct completion *x,
 {
 	might_sleep();
 
-	spin_lock_irq(&x->wait.lock);
+	waitq_lock_irq(&x->wait);
 	timeout = do_wait_for_common(x, action, timeout, state);
-	spin_unlock_irq(&x->wait.lock);
+	waitq_unlock_irq(&x->wait);
 	return timeout;
 }
 
@@ -267,12 +267,12 @@ bool try_wait_for_completion(struct completion *x)
 	unsigned long flags;
 	int ret = 1;
 
-	spin_lock_irqsave(&x->wait.lock, flags);
+	waitq_lock_irqsave(&x->wait, flags);
 	if (!x->done)
 		ret = 0;
 	else
 		x->done--;
-	spin_unlock_irqrestore(&x->wait.lock, flags);
+	waitq_unlock_irqrestore(&x->wait, flags);
 	return ret;
 }
 EXPORT_SYMBOL(try_wait_for_completion);
@@ -290,10 +290,10 @@ bool completion_done(struct completion *x)
 	unsigned long flags;
 	int ret = 1;
 
-	spin_lock_irqsave(&x->wait.lock, flags);
+	waitq_lock_irqsave(&x->wait, flags);
 	if (!x->done)
 		ret = 0;
-	spin_unlock_irqrestore(&x->wait.lock, flags);
+	waitq_unlock_irqrestore(&x->wait, flags);
 	return ret;
 }
 EXPORT_SYMBOL(completion_done);
kernel/sched/core.c +4 -4
index e85cda2..c826ae6 100644
@@ -2711,13 +2711,13 @@ sleep_on_common(wait_queue_head_t *q, int state, long timeout)
 
 	__set_current_state(state);
 
-	spin_lock_irqsave(&q->lock, flags);
+	waitq_lock_irqsave(q, flags);
 	__add_wait_queue(q, &wait);
-	spin_unlock(&q->lock);
+	waitq_unlock(q);
 	timeout = schedule_timeout(timeout);
-	spin_lock_irq(&q->lock);
+	waitq_lock_irq(q);
 	__remove_wait_queue(q, &wait);
-	spin_unlock_irqrestore(&q->lock, flags);
+	waitq_unlock_irqrestore(q, flags);
 
 	return timeout;
 }
kernel/sched/wait.c +22 -22

132 lines not on this page · raw message

mm/filemap.c +2 -2
index b7749a9..ba9ef44 100644
@@ -587,9 +587,9 @@ void add_page_wait_queue(struct page *page, wait_queue_t *waiter)
 	wait_queue_head_t *q = page_waitqueue(page);
 	unsigned long flags;
 
-	spin_lock_irqsave(&q->lock, flags);
+	waitq_lock_irqsave(q, flags);
 	__add_wait_queue(q, waiter);
-	spin_unlock_irqrestore(&q->lock, flags);
+	waitq_unlock_irqrestore(q, flags);
 }
 EXPORT_SYMBOL_GPL(add_page_wait_queue);
 
net/sunrpc/sched.c +2 -2
index ff3cc4b..79120be 100644
@@ -293,12 +293,12 @@ static int rpc_complete_task(struct rpc_task *task)
 
 	trace_rpc_task_complete(task->tk_client, task, NULL);
 
-	spin_lock_irqsave(&wq->lock, flags);
+	waitq_lock_irqsave(wq, flags);
 	clear_bit(RPC_TASK_ACTIVE, &task->tk_runstate);
 	ret = atomic_dec_and_test(&task->tk_count);
 	if (waitqueue_active(wq))
 		__wake_up_locked_key(wq, TASK_NORMAL, &k);
-	spin_unlock_irqrestore(&wq->lock, flags);
+	waitq_unlock_irqrestore(wq, flags);
 	return ret;
 }
 
Signature · 1 line
1.8.5.rc3
#2 lore raw
In reply to Paul Gortmaker

Hi,

06.12.2013, 19:24, "Paul Gortmaker" <[email protected]>:

Quoted 7 lines

Peter pointed out[1] that the simple wait queues in RT could run any number of wakeups with the wait queue head lock held.

It turns out that this isnt really specific to RT, but it is mainline too -- all the __wake_up_XYZ that in turn use the __wake_up_common boiler plate end up looping over all the callbacks (normally ttwu) while holding the head lock.

<snip>

One more problem with wait queues in RT case.

Wait queues moving from spinlocks to rt_mutexes makes timerfd not irqsafe on RT. This makes it unusable for some user tasks.

Quoted 8 lines

 fs/timerfd.c                                       | 26 ++++++------- include/linux/wait.h                               | 36 +++++++++++++++--- kernel/sched/completion.c                          | 24 ++++++------ kernel/sched/core.c                                |  8 ++-- kernel/sched/wait.c                                | 44 +++++++++++----------- mm/filemap.c                                       |  4 +- net/sunrpc/sched.c                                 |  4 +-

 14 files changed, 150 insertions(+), 126 deletions(-)