wait queue head; locking considerations
Subsystems (14)
cover wait queue head; locking considerations
Peter pointed out[1] that the simple wait queues in RT could run any number of wakeups with the wait queue head lock held.
It turns out that this isnt really specific to RT, but it is mainline too -- all the __wake_up_XYZ that in turn use the __wake_up_common boiler plate end up looping over all the callbacks (normally ttwu) while holding the head lock.
I'm not sure exactly whether contention is a serious issue, but the possibility exists for someone to use a non-default callback (i.e. not ttwu) and have some considerable amount of computation going on while holding the head lock.
I entertained several ideas, none of which I am really happy with (except perhaps #4).
1) Limit the number of callbacks; via a MAX_WORK or similar. This would require bubbling up return values (some of which were just recently removed) and relying on the holder of the lock to take appropriate action (unlock, breathe, restart.) Plus, it doesn't really solve anything if someone installs a non default callback that takes forever.
2) Try and defer the callbacks temporarily somehow until the waitqueue head lock is released. I experimented with this, via creating a snapshot of the waiters, and running that on the next head unlock, but it was ugly, lockdep didn't like it at all, etc. Plus it isn't clear we can delay any without causing additional latency.
3) Declare locked wake ups evil, and try and determine if we can somehow fix the users in a way that we can phase them out. Perhaps by somehow cascading the head lock down into the individual waitqueue_t elements?
4) Do nothing. It isn't really a significant problem.
In any case, while attempting #2, I did abstract out the locking and unlocking. Maybe this makes sense regardless, since we already hide the spinlock init in wait queue head init; we don't use raw list operations, but instead use add_to_waitqueue() abstractions. In fact it seems only the locking doesn't have waitq in its function name.
It does help highlight those users who are manually manipulating their waitq head lock, and perhaps slightly improve readability. It passes x86/_64 allmodconfig, but if we think we want to use this, I'll need to build test the other arches too.
Thoughts?
Paul.
--
[1] http://marc.info/?l=linux-kernel&m=138089860308430&w=2
Paul Gortmaker (1):
waitq: abstract out locking/unlocking waitq head operations
drivers/gpu/drm/radeon/radeon_sa.c | 18 ++++----- drivers/iio/industrialio-event.c | 26 ++++++------- .../lustre/lustre/libcfs/linux/linux-prim.c | 4 +- drivers/usb/gadget/f_fs.c | 42 ++++++++++----------- fs/eventfd.c | 32 ++++++++-------- fs/eventpoll.c | 4 +- fs/nilfs2/segment.c | 4 +- fs/timerfd.c | 26 ++++++------- include/linux/wait.h | 36 +++++++++++++++--- kernel/sched/completion.c | 24 ++++++------ kernel/sched/core.c | 8 ++-- kernel/sched/wait.c | 44 +++++++++++----------- mm/filemap.c | 4 +- net/sunrpc/sched.c | 4 +- 14 files changed, 150 insertions(+), 126 deletions(-)
Signature · 1 line
1.8.5.rc3
PATCH 1/1 waitq: abstract out locking/unlocking waitq head operations
We abstract out the wait queue head locking operations. This is done in order to be able to add future functionality to them in a single place. As a side benefit, it happens to increase code readability.
We rename the wait queue lock entry to "__lock" which ensures that anyone who tries to use the old wq->lock or wq.lock will fail, and hence be forced into being aware of this change.
---
[Still needs build coverage testing on non-x86]
drivers/gpu/drm/radeon/radeon_sa.c | 18 ++++----- drivers/iio/industrialio-event.c | 26 ++++++------- .../lustre/lustre/libcfs/linux/linux-prim.c | 4 +- drivers/usb/gadget/f_fs.c | 42 ++++++++++----------- fs/eventfd.c | 32 ++++++++-------- fs/eventpoll.c | 4 +- fs/nilfs2/segment.c | 4 +- fs/timerfd.c | 26 ++++++------- include/linux/wait.h | 36 +++++++++++++++--- kernel/sched/completion.c | 24 ++++++------ kernel/sched/core.c | 8 ++-- kernel/sched/wait.c | 44 +++++++++++----------- mm/filemap.c | 4 +- net/sunrpc/sched.c | 4 +- 14 files changed, 150 insertions(+), 126 deletions(-)
drivers/gpu/drm/radeon/radeon_sa.c +9 -9
@@ -330,7 +330,7 @@ int radeon_sa_bo_new(struct radeon_device *rdev,INIT_LIST_HEAD (&(*sa_bo)->olist);INIT_LIST_HEAD (&(*sa_bo)->flist); -spin_lock (&sa_manager->wq.lock); +waitq_lock (&sa_manager->wq); do { for (i = 0; i < RADEON_NUM_RINGS; ++i) { fences[i] = NULL;
@@ -342,16 +342,16 @@ int radeon_sa_bo_new(struct radeon_device *rdev,if (radeon_sa_bo_try_alloc (sa_manager, *sa_bo, size, align)) { -spin_unlock (&sa_manager->wq.lock); +waitq_unlock (&sa_manager->wq);return 0 ; }/* see if we can skip over some allocations */ }while (radeon_sa_bo_next_hole (sa_manager, fences, tries)); -spin_unlock (&sa_manager->wq.lock); +waitq_unlock (&sa_manager->wq); r =radeon_fence_wait_any (rdev, fences, false); -spin_lock (&sa_manager->wq.lock); +waitq_lock (&sa_manager->wq);/* if we have nothing to wait for block */ if (r == -ENOENT && block) { r = wait_event_interruptible_locked(
@@ -365,7 +365,7 @@ int radeon_sa_bo_new(struct radeon_device *rdev, }while (!r); -spin_unlock (&sa_manager->wq.lock); +waitq_unlock (&sa_manager->wq);kfree (*sa_bo); *sa_bo =NULL ;return r;
@@ -381,7 +381,7 @@ void radeon_sa_bo_free(struct radeon_device *rdev, struct radeon_sa_bo **sa_bo, } sa_manager = (*sa_bo)->manager; -spin_lock (&sa_manager->wq.lock); +waitq_lock (&sa_manager->wq);if (fence && !radeon_fence_signaled (fence)) { (*sa_bo)->fence =radeon_fence_ref (fence); list_add_tail(&(*sa_bo)->flist,
@@ -390,7 +390,7 @@ void radeon_sa_bo_free(struct radeon_device *rdev, struct radeon_sa_bo **sa_bo,radeon_sa_bo_remove_locked (*sa_bo); }wake_up_all_locked (&sa_manager->wq); -spin_unlock (&sa_manager->wq.lock); +waitq_unlock (&sa_manager->wq); *sa_bo =NULL ; }
@@ -400,7 +400,7 @@ void radeon_sa_bo_dump_debug_info(struct radeon_sa_manager *sa_manager, {struct radeon_sa_bo *i; -spin_lock (&sa_manager->wq.lock); +waitq_lock (&sa_manager->wq);list_for_each_entry (i, &sa_manager->olist, olist) {if (&i->olist == sa_manager->hole) {seq_printf (m,">" );
@@ -415,6 +415,6 @@ void radeon_sa_bo_dump_debug_info(struct radeon_sa_manager *sa_manager, }seq_printf (m,"\n" ); } -spin_unlock (&sa_manager->wq.lock); +waitq_unlock (&sa_manager->wq); }#endif
drivers/iio/industrialio-event.c +13 -13
@@ -50,7 +50,7 @@ int iio_push_event(struct iio_dev *indio_dev, u64 ev_code, s64 timestamp)int copied;/* Does anyone care? */ -spin_lock_irqsave (&ev_int->wait.lock, flags); +waitq_lock_irqsave (&ev_int->wait, flags);if (test_bit (IIO_BUSY_BIT_POS , &ev_int->flags)) { ev.id = ev_code;
@@ -60,7 +60,7 @@ int iio_push_event(struct iio_dev *indio_dev, u64 ev_code, s64 timestamp)if (copied !=0 )wake_up_locked_poll (&ev_int->wait,POLLIN ); } -spin_unlock_irqrestore (&ev_int->wait.lock, flags); +waitq_unlock_irqrestore (&ev_int->wait, flags);return 0 ; }
@@ -81,10 +81,10 @@ static unsigned int iio_event_poll(struct file *filep,poll_wait (filep, &ev_int->wait, wait); -spin_lock_irq (&ev_int->wait.lock); +waitq_lock_irq (&ev_int->wait);if (!kfifo_is_empty (&ev_int->det_events)) events =POLLIN |POLLRDNORM ; -spin_unlock_irq (&ev_int->wait.lock); +waitq_unlock_irq (&ev_int->wait);return events; }
@@ -105,7 +105,7 @@ static ssize_t iio_event_chrdev_read(struct file *filep,if (count <sizeof (struct iio_event_data ))return -EINVAL ; -spin_lock_irq (&ev_int->wait.lock); +waitq_lock_irq (&ev_int->wait);if (kfifo_is_empty (&ev_int->det_events)) {if (filep->f_flags &O_NONBLOCK ) { ret = -EAGAIN ;
@@ -127,7 +127,7 @@ static ssize_t iio_event_chrdev_read(struct file *filep, ret =kfifo_to_user (&ev_int->det_events, buf, count, &copied); error_unlock: -spin_unlock_irq (&ev_int->wait.lock); +waitq_unlock_irq (&ev_int->wait);return ret ? ret : copied; }
@@ -137,7 +137,7 @@ static int iio_event_chrdev_release(struct inode *inode, struct file *filep)struct iio_dev *indio_dev = filep->private_data;struct iio_event_interface *ev_int = indio_dev->event_interface; -spin_lock_irq (&ev_int->wait.lock); +waitq_lock_irq (&ev_int->wait);__clear_bit (IIO_BUSY_BIT_POS , &ev_int->flags);/* * In order to maintain a clean state for reopening,
@@ -145,7 +145,7 @@ static int iio_event_chrdev_release(struct inode *inode, struct file *filep)* any new __iio_push_event calls running. */ kfifo_reset_out (&ev_int->det_events); -spin_unlock_irq (&ev_int->wait.lock); +waitq_unlock_irq (&ev_int->wait);iio_device_put (indio_dev);
@@ -168,20 +168,20 @@ int iio_event_getfd(struct iio_dev *indio_dev)if (ev_int ==NULL )return -ENODEV ; -spin_lock_irq (&ev_int->wait.lock); +waitq_lock_irq (&ev_int->wait);if (__test_and_set_bit (IIO_BUSY_BIT_POS , &ev_int->flags)) { -spin_unlock_irq (&ev_int->wait.lock); +waitq_unlock_irq (&ev_int->wait);return -EBUSY ; } -spin_unlock_irq (&ev_int->wait.lock); +waitq_unlock_irq (&ev_int->wait);iio_device_get (indio_dev); fd =anon_inode_getfd ("iio:event" , &iio_event_chrdev_fileops, indio_dev,O_RDONLY |O_CLOEXEC );if (fd <0 ) { -spin_lock_irq (&ev_int->wait.lock); +waitq_lock_irq (&ev_int->wait);__clear_bit (IIO_BUSY_BIT_POS , &ev_int->flags); -spin_unlock_irq (&ev_int->wait.lock); +waitq_unlock_irq (&ev_int->wait);iio_device_put (indio_dev); }return fd;
drivers/staging/lustre/lustre/libcfs/linux/linux-prim.c +2 -2
@@ -74,9 +74,9 @@ add_wait_queue_exclusive_head(wait_queue_head_t *waitq, wait_queue_t *link) {unsigned long flags; -spin_lock_irqsave (&LINUX_WAITQ_HEAD (waitq)->lock, flags); +waitq_lock_irqsave (LINUX_WAITQ_HEAD (waitq), flags);__add_wait_queue_exclusive (LINUX_WAITQ_HEAD (waitq),LINUX_WAITQ (link)); -spin_unlock_irqrestore (&LINUX_WAITQ_HEAD (waitq)->lock, flags); +waitq_unlock_irqrestore (LINUX_WAITQ_HEAD (waitq), flags); }EXPORT_SYMBOL (add_wait_queue_exclusive_head);
drivers/usb/gadget/f_fs.c +21 -21
150 lines not on this page · raw message
fs/eventfd.c +16 -16
@@ -55,13 +55,13 @@ __u64 eventfd_signal(struct eventfd_ctx *ctx, __u64 n) {unsigned long flags; -spin_lock_irqsave (&ctx->wqh.lock, flags); +waitq_lock_irqsave (&ctx->wqh, flags);if (ULLONG_MAX - ctx->count < n) n =ULLONG_MAX - ctx->count; ctx->count += n;if (waitqueue_active (&ctx->wqh))wake_up_locked_poll (&ctx->wqh,POLLIN ); -spin_unlock_irqrestore (&ctx->wqh.lock, flags); +waitq_unlock_irqrestore (&ctx->wqh, flags);return n; }
@@ -122,14 +122,14 @@ static unsigned int eventfd_poll(struct file *file, poll_table *wait)poll_wait (file, &ctx->wqh, wait); -spin_lock_irqsave (&ctx->wqh.lock, flags); +waitq_lock_irqsave (&ctx->wqh, flags);if (ctx->count >0 ) events |=POLLIN ;if (ctx->count ==ULLONG_MAX ) events |=POLLERR ;if (ULLONG_MAX -1 > ctx->count) events |=POLLOUT ; -spin_unlock_irqrestore (&ctx->wqh.lock, flags); +waitq_unlock_irqrestore (&ctx->wqh, flags);return events; }
@@ -158,12 +158,12 @@ int eventfd_ctx_remove_wait_queue(struct eventfd_ctx *ctx, wait_queue_t *wait, {unsigned long flags; -spin_lock_irqsave (&ctx->wqh.lock, flags); +waitq_lock_irqsave (&ctx->wqh, flags);eventfd_ctx_do_read (ctx, cnt);__remove_wait_queue (&ctx->wqh, wait);if (*cnt !=0 &&waitqueue_active (&ctx->wqh))wake_up_locked_poll (&ctx->wqh,POLLOUT ); -spin_unlock_irqrestore (&ctx->wqh.lock, flags); +waitq_unlock_irqrestore (&ctx->wqh, flags);return *cnt !=0 ?0 : -EAGAIN ; }
@@ -188,7 +188,7 @@ ssize_t eventfd_ctx_read(struct eventfd_ctx *ctx, int no_wait, __u64 *cnt)ssize_t res;DECLARE_WAITQUEUE (wait, current); -spin_lock_irq (&ctx->wqh.lock); +waitq_lock_irq (&ctx->wqh); *cnt =0 ; res = -EAGAIN ;if (ctx->count >0 )
@@ -205,9 +205,9 @@ ssize_t eventfd_ctx_read(struct eventfd_ctx *ctx, int no_wait, __u64 *cnt) res = -ERESTARTSYS ;break ; } -spin_unlock_irq (&ctx->wqh.lock); +waitq_unlock_irq (&ctx->wqh);schedule (); -spin_lock_irq (&ctx->wqh.lock); +waitq_lock_irq (&ctx->wqh); }__remove_wait_queue (&ctx->wqh, &wait);__set_current_state (TASK_RUNNING );
@@ -217,7 +217,7 @@ ssize_t eventfd_ctx_read(struct eventfd_ctx *ctx, int no_wait, __u64 *cnt)if (waitqueue_active (&ctx->wqh))wake_up_locked_poll (&ctx->wqh,POLLOUT ); } -spin_unlock_irq (&ctx->wqh.lock); +waitq_unlock_irq (&ctx->wqh);return res; }
@@ -253,7 +253,7 @@ static ssize_t eventfd_write(struct file *file, const char __user *buf, size_t creturn -EFAULT ;if (ucnt ==ULLONG_MAX )return -EINVAL ; -spin_lock_irq (&ctx->wqh.lock); +waitq_lock_irq (&ctx->wqh); res = -EAGAIN ;if (ULLONG_MAX - ctx->count > ucnt) res =sizeof (ucnt);
@@ -269,9 +269,9 @@ static ssize_t eventfd_write(struct file *file, const char __user *buf, size_t c res = -ERESTARTSYS ;break ; } -spin_unlock_irq (&ctx->wqh.lock); +waitq_unlock_irq (&ctx->wqh);schedule (); -spin_lock_irq (&ctx->wqh.lock); +waitq_lock_irq (&ctx->wqh); }__remove_wait_queue (&ctx->wqh, &wait);__set_current_state (TASK_RUNNING );
@@ -281,7 +281,7 @@ static ssize_t eventfd_write(struct file *file, const char __user *buf, size_t cif (waitqueue_active (&ctx->wqh))wake_up_locked_poll (&ctx->wqh,POLLIN ); } -spin_unlock_irq (&ctx->wqh.lock); +waitq_unlock_irq (&ctx->wqh);return res; }
@@ -292,10 +292,10 @@ static int eventfd_show_fdinfo(struct seq_file *m, struct file *f)struct eventfd_ctx *ctx = f->private_data;int ret; -spin_lock_irq (&ctx->wqh.lock); +waitq_lock_irq (&ctx->wqh); ret =seq_printf (m,"eventfd-count: %16llx\n" , (unsigned long long )ctx->count); -spin_unlock_irq (&ctx->wqh.lock); +waitq_unlock_irq (&ctx->wqh);return ret; }
fs/eventpoll.c +2 -2
@@ -474,9 +474,9 @@ static inline void ep_wake_up_nested(wait_queue_head_t *wqueue, {unsigned long flags; -spin_lock_irqsave_nested (&wqueue->lock, flags, subclass); +waitq_lock_irqsave_nested (wqueue, flags, subclass);wake_up_locked_poll (wqueue, events); -spin_unlock_irqrestore (&wqueue->lock, flags); +waitq_unlock_irqrestore (wqueue, flags); }#else static inline void ep_wake_up_nested(wait_queue_head_t *wqueue,
fs/nilfs2/segment.c +2 -2
@@ -2106,7 +2106,7 @@ static void nilfs_segctor_wakeup(struct nilfs_sc_info *sci, int err)struct nilfs_segctor_wait_request *wrq, *n;unsigned long flags; -spin_lock_irqsave (&sci->sc_wait_request.lock, flags); +waitq_lock_irqsave (&sci->sc_wait_request, flags);list_for_each_entry_safe (wrq, n, &sci->sc_wait_request.task_list, wq.task_list) { if (!atomic_read(&wrq->done) &&
@@ -2120,7 +2120,7 @@ static void nilfs_segctor_wakeup(struct nilfs_sc_info *sci, int err) 0, NULL); } } -spin_unlock_irqrestore (&sci->sc_wait_request.lock, flags); +waitq_unlock_irqrestore (&sci->sc_wait_request, flags); }/**
fs/timerfd.c +13 -13
@@ -60,11 +60,11 @@ static void timerfd_triggered(struct timerfd_ctx *ctx) {unsigned long flags; -spin_lock_irqsave (&ctx->wqh.lock, flags); +waitq_lock_irqsave (&ctx->wqh, flags); ctx->expired =1 ; ctx->ticks++;wake_up_locked (&ctx->wqh); -spin_unlock_irqrestore (&ctx->wqh.lock, flags); +waitq_unlock_irqrestore (&ctx->wqh, flags); }static enum hrtimer_restart timerfd_tmrproc (struct hrtimer *htmr)
@@ -100,13 +100,13 @@ void timerfd_clock_was_set(void)list_for_each_entry_rcu (ctx, &cancel_list, clist) {if (!ctx->might_cancel)continue ; -spin_lock_irqsave (&ctx->wqh.lock, flags); +waitq_lock_irqsave (&ctx->wqh, flags);if (ctx->moffs.tv64 != moffs.tv64) { ctx->moffs.tv64 =KTIME_MAX ; ctx->ticks++;wake_up_locked (&ctx->wqh); } -spin_unlock_irqrestore (&ctx->wqh.lock, flags); +waitq_unlock_irqrestore (&ctx->wqh, flags); }rcu_read_unlock (); }
@@ -221,10 +221,10 @@ static unsigned int timerfd_poll(struct file *file, poll_table *wait)poll_wait (file, &ctx->wqh, wait); -spin_lock_irqsave (&ctx->wqh.lock, flags); +waitq_lock_irqsave (&ctx->wqh, flags);if (ctx->ticks) events |=POLLIN ; -spin_unlock_irqrestore (&ctx->wqh.lock, flags); +waitq_unlock_irqrestore (&ctx->wqh, flags);return events; }
@@ -238,7 +238,7 @@ static ssize_t timerfd_read(struct file *file, char __user *buf, size_t count,if (count <sizeof (ticks))return -EINVAL ; -spin_lock_irq (&ctx->wqh.lock); +waitq_lock_irq (&ctx->wqh);if (file->f_flags &O_NONBLOCK ) res = -EAGAIN ;else
@@ -278,7 +278,7 @@ static ssize_t timerfd_read(struct file *file, char __user *buf, size_t count, ctx->expired =0 ; ctx->ticks =0 ; } -spin_unlock_irq (&ctx->wqh.lock); +waitq_unlock_irq (&ctx->wqh);if (ticks) res =put_user (ticks, (u64 __user *) buf) ? -EFAULT :sizeof (ticks);return res;
@@ -370,7 +370,7 @@ static int do_timerfd_settime(int ufd, int flags,* it to the new values. */ for (;;) { -spin_lock_irq (&ctx->wqh.lock); +waitq_lock_irq (&ctx->wqh);if (isalarm (ctx)) {if (alarm_try_to_cancel (&ctx->t.alarm) >=0 )
@@ -379,7 +379,7 @@ static int do_timerfd_settime(int ufd, int flags,if (hrtimer_try_to_cancel (&ctx->t.tmr) >=0 )break ; } -spin_unlock_irq (&ctx->wqh.lock); +waitq_unlock_irq (&ctx->wqh);cpu_relax (); }
@@ -404,7 +404,7 @@ static int do_timerfd_settime(int ufd, int flags,*/ ret =timerfd_setup (ctx, flags, new); -spin_unlock_irq (&ctx->wqh.lock); +waitq_unlock_irq (&ctx->wqh);fdput (f);return ret; }
@@ -418,7 +418,7 @@ static int do_timerfd_gettime(int ufd, struct itimerspec *t)return ret; ctx = f.file->private_data; -spin_lock_irq (&ctx->wqh.lock); +waitq_lock_irq (&ctx->wqh);if (ctx->expired && ctx->tintv.tv64) { ctx->expired =0 ;
@@ -436,7 +436,7 @@ static int do_timerfd_gettime(int ufd, struct itimerspec *t) } t->it_value =ktime_to_timespec (timerfd_get_remaining (ctx)); t->it_interval =ktime_to_timespec (ctx->tintv); -spin_unlock_irq (&ctx->wqh.lock); +waitq_unlock_irq (&ctx->wqh);fdput (f);return 0 ; }
include/linux/wait.h +30 -6
@@ -33,11 +33,35 @@ struct wait_bit_queue { };struct __wait_queue_head { -spinlock_t lock; +spinlock_t __lock;/* Use wrapper for access */ struct list_head task_list; };typedef struct __wait_queue_head wait_queue_head_t ; +#define waitq_lock_irqsave_nested (qh, f, sc) \ + spin_lock_irqsave_nested(&(qh)->__lock, f, sc) + +#define waitq_lock_irqsave (qh, f) \ + spin_lock_irqsave(&(qh)->__lock, f) + +#define waitq_unlock_irqrestore (qh, f) \ + spin_unlock_irqrestore(&(qh)->__lock, f) + +#define waitq_lock_irq (qh) \ + spin_lock_irq(&(qh)->__lock) + +#define waitq_unlock_irq (qh) \ + spin_unlock_irq(&(qh)->__lock) + +#define waitq_lock (qh) \ + spin_lock(&(qh)->__lock) + +#define waitq_unlock (qh) \ + spin_unlock(&(qh)->__lock) + +#define waitq_lock_init (qh) \ + spin_lock_init(&(qh)->__lock) +struct task_struct ;/*
@@ -53,7 +77,7 @@ struct task_struct; wait_queue_t name = __WAITQUEUE_INITIALIZER(name, tsk)#define __WAIT_QUEUE_HEAD_INITIALIZER (name) { \ - .lock = __SPIN_LOCK_UNLOCKED(name.lock), \ + .__lock = __SPIN_LOCK_UNLOCKED(name.__lock), \ .task_list = { &(name).task_list, &(name).task_list } }#define DECLARE_WAIT_QUEUE_HEAD (name) \
@@ -465,14 +489,14 @@ do { \break ; \ } \if (irq) \ -spin_unlock_irq (&(wq).lock); \ +waitq_unlock_irq (&(wq)); \else \ -spin_unlock (&(wq).lock); \ +waitq_unlock (&(wq)); \schedule (); \if (irq) \ -spin_lock_irq (&(wq).lock); \ +waitq_lock_irq (&(wq)); \else \ -spin_lock (&(wq).lock); \ +waitq_lock (&(wq)); \ }while (!(condition)); \__remove_wait_queue (&(wq), &__wait); \__set_current_state (TASK_RUNNING ); \
kernel/sched/completion.c +12 -12
@@ -30,10 +30,10 @@ void complete(struct completion *x) {unsigned long flags; -spin_lock_irqsave (&x->wait.lock, flags); +waitq_lock_irqsave (&x->wait, flags); x->done++;__wake_up_locked (&x->wait,TASK_NORMAL ,1 ); -spin_unlock_irqrestore (&x->wait.lock, flags); +waitq_unlock_irqrestore (&x->wait, flags); }EXPORT_SYMBOL (complete);
@@ -50,10 +50,10 @@ void complete_all(struct completion *x) {unsigned long flags; -spin_lock_irqsave (&x->wait.lock, flags); +waitq_lock_irqsave (&x->wait, flags); x->done +=UINT_MAX /2 ;__wake_up_locked (&x->wait,TASK_NORMAL ,0 ); -spin_unlock_irqrestore (&x->wait.lock, flags); +waitq_unlock_irqrestore (&x->wait, flags); }EXPORT_SYMBOL (complete_all);
@@ -71,9 +71,9 @@ do_wait_for_common(struct completion *x,break ; }__set_current_state (state); -spin_unlock_irq (&x->wait.lock); +waitq_unlock_irq (&x->wait); timeout =action (timeout); -spin_lock_irq (&x->wait.lock); +waitq_lock_irq (&x->wait); }while (!x->done && timeout);__remove_wait_queue (&x->wait, &wait);if (!x->done)
@@ -89,9 +89,9 @@ __wait_for_common(struct completion *x, {might_sleep (); -spin_lock_irq (&x->wait.lock); +waitq_lock_irq (&x->wait); timeout =do_wait_for_common (x, action, timeout, state); -spin_unlock_irq (&x->wait.lock); +waitq_unlock_irq (&x->wait);return timeout; }
@@ -267,12 +267,12 @@ bool try_wait_for_completion(struct completion *x)unsigned long flags;int ret =1 ; -spin_lock_irqsave (&x->wait.lock, flags); +waitq_lock_irqsave (&x->wait, flags);if (!x->done) ret =0 ;else x->done--; -spin_unlock_irqrestore (&x->wait.lock, flags); +waitq_unlock_irqrestore (&x->wait, flags);return ret; }EXPORT_SYMBOL (try_wait_for_completion);
@@ -290,10 +290,10 @@ bool completion_done(struct completion *x)unsigned long flags;int ret =1 ; -spin_lock_irqsave (&x->wait.lock, flags); +waitq_lock_irqsave (&x->wait, flags);if (!x->done) ret =0 ; -spin_unlock_irqrestore (&x->wait.lock, flags); +waitq_unlock_irqrestore (&x->wait, flags);return ret; }EXPORT_SYMBOL (completion_done);
kernel/sched/core.c +4 -4
@@ -2711,13 +2711,13 @@ sleep_on_common(wait_queue_head_t *q, int state, long timeout)__set_current_state (state); -spin_lock_irqsave (&q->lock, flags); +waitq_lock_irqsave (q, flags);__add_wait_queue (q, &wait); -spin_unlock (&q->lock); +waitq_unlock (q); timeout =schedule_timeout (timeout); -spin_lock_irq (&q->lock); +waitq_lock_irq (q);__remove_wait_queue (q, &wait); -spin_unlock_irqrestore (&q->lock, flags); +waitq_unlock_irqrestore (q, flags);return timeout; }
kernel/sched/wait.c +22 -22
132 lines not on this page · raw message
mm/filemap.c +2 -2
@@ -587,9 +587,9 @@ void add_page_wait_queue(struct page *page, wait_queue_t *waiter)wait_queue_head_t *q =page_waitqueue (page);unsigned long flags; -spin_lock_irqsave (&q->lock, flags); +waitq_lock_irqsave (q, flags);__add_wait_queue (q, waiter); -spin_unlock_irqrestore (&q->lock, flags); +waitq_unlock_irqrestore (q, flags); }EXPORT_SYMBOL_GPL (add_page_wait_queue);
net/sunrpc/sched.c +2 -2
@@ -293,12 +293,12 @@ static int rpc_complete_task(struct rpc_task *task)trace_rpc_task_complete (task->tk_client, task,NULL ); -spin_lock_irqsave (&wq->lock, flags); +waitq_lock_irqsave (wq, flags);clear_bit (RPC_TASK_ACTIVE , &task->tk_runstate); ret =atomic_dec_and_test (&task->tk_count);if (waitqueue_active (wq))__wake_up_locked_key (wq,TASK_NORMAL , &k); -spin_unlock_irqrestore (&wq->lock, flags); +waitq_unlock_irqrestore (wq, flags);return ret; }
Signature · 1 line
1.8.5.rc3
Hi,
06.12.2013, 19:24, "Paul Gortmaker" <[email protected]>:
Quoted 7 lines
Peter pointed out[1] that the simple wait queues in RT could run any number of wakeups with the wait queue head lock held.
It turns out that this isnt really specific to RT, but it is mainline too -- all the
__wake_up_XYZthat in turn use the__wake_up_commonboiler plate end up looping over all the callbacks (normally ttwu) while holding the head lock.
<snip>
One more problem with wait queues in RT case.
Wait queues moving from spinlocks to rt_mutexes makes timerfd not irqsafe on RT. This makes it unusable for some user tasks.
Quoted 8 lines
fs/timerfd.c | 26 ++++++-------
include/linux/wait.h| 36 +++++++++++++++---kernel/sched/completion.c| 24 ++++++------kernel/sched/core.c| 8 ++--kernel/sched/wait.c| 44 +++++++++++-----------mm/filemap.c| 4 +-net/sunrpc/sched.c| 4 +-14 files changed, 150 insertions(+), 126 deletions(-)