Skip to contentSkip to conversation

rust: sync: add WaitQueue infrastructure

PATCH In review Danilo Krummrich 4 participants 18 messages Nov 17

Subsystems (3)
lore
Share & tools
lore

Danilo Krummrich

author reviewer
#0 lore raw

v3 cover rust: sync: add WaitQueue infrastructure

Add a WaitQueue abstraction wrapping struct wait_queue_head, providing wait_event()-style methods that take a condition closure directly.

This is needed for a (subsequent) DRM patch series to wait for in-flight file close operations during driver unbind. The Tyr driver also needs it for "CSF firmware responses and other GPU-driven events" [1].

Convert CondVar (and PollCondVar) to use WaitQueue internally, removing duplicate code using WaitQueue as the base primitive and add wait_interruptible_freezable_nonexclusive() for subsequent binder usage.

This series is based on FUJITA Tomonori's CondVar timeout conversion [2].

Changes in v3:
- Use Delta<Jiffy> for timeouts and treat negative durations as zero.
- Add might_sleep() and caller tracking to the wait_event() methods.
- Gate the WaitQueue KUnit tests behind a dedicated Kconfig option.

Changes in v2:
- Drop the already merged workqueue patch.
- Make as_raw() public.
- Split error types and use bindings constants for direct C calls.
- Avoid condition rechecks after success.
- Check the condition before preparing a waiter.
- Add non-exclusive wait support, including a CondVar method for subsequent Binder conversion.
- Handle zero and expired timeouts before interruption; extend KUnit coverage.

[1] https://lore.kernel.org/dri-devel/[email protected]/
[2] https://lore.kernel.org/rust-for-linux/[email protected]/

Danilo Krummrich (5):
rust: task: add safe schedule_timeout() wrapper
rust: sync: add WaitQueue infrastructure
rust: sync: convert CondVar and PollCondVar to use WaitQueue
rust: sync: condvar: use task::schedule_timeout()
rust: sync: condvar: add non-exclusive freezable waiting

 rust/kernel/Kconfig.test    |  10 +
 rust/kernel/sync.rs         |   8 +
 rust/kernel/sync/condvar.rs | 154 +++++-----
 rust/kernel/sync/poll.rs    |  19 +-
 rust/kernel/sync/wait.rs    | 546 ++++++++++++++++++++++++++++++++++++
 rust/kernel/task.rs         |  23 ++
 6 files changed, 664 insertions(+), 96 deletions(-)
 create mode 100644 rust/kernel/sync/wait.rs

base-commit: 60fff02883397e17439c16f0e72b2f8c5f6eb791

Signature · 1 line
2.56.1
author reviewer
#1 lore raw

PATCH v3 1/5 rust: task: add safe schedule_timeout() wrapper

Add a safe wrapper around schedule_timeout() that takes and returns Delta<Jiffy>, treating negative durations as zero.

---
 rust/kernel/task.rs | 23 +++++++++++++++++++++++
 1 file changed, 23 insertions(+)
rust/kernel/task.rs +23 -0
index 3336df493dec..2fc0607ba9b7 100644
@@ -10,6 +10,10 @@
     pid_namespace::PidNamespace,
     prelude::*,
     sync::aref::ARef,
+    time::{
+        Delta,
+        Jiffy, //
+    },
     types::{NotThreadSafe, Opaque},
 };
 use core::{
@@ -20,6 +24,25 @@
 /// A sentinel value used for infinite timeouts.
 pub const MAX_SCHEDULE_TIMEOUT: c_long = c_long::MAX;
 
+/// Sleeps for the given timeout, or until woken.
+///
+/// Returns the remaining time, or zero if the timeout expired. The task state should be set before
+/// calling this.
+///
+/// A negative timeout is treated as zero.
+///
+/// A timeout of [`MAX_SCHEDULE_TIMEOUT`] jiffies means an indefinite wait. In that case, the
+/// returned duration is also [`MAX_SCHEDULE_TIMEOUT`] jiffies.
+#[inline]
+pub fn schedule_timeout(timeout: Delta<Jiffy>) -> Delta<Jiffy> {
+    let timeout = timeout.as_jiffies().max(0);
+
+    // SAFETY: `schedule_timeout()` is always safe to call.
+    let remaining = unsafe { bindings::schedule_timeout(timeout) };
+
+    Delta::from_jiffies(remaining)
+}
+
 /// Bitmask for tasks that are sleeping in an interruptible state.
 pub const TASK_INTERRUPTIBLE: c_int = bindings::TASK_INTERRUPTIBLE as c_int;
 /// Bitmask for tasks that are sleeping in an uninterruptible state.
Signature · 1 line
2.56.1
#2 lore raw

Re: [PATCH v3 1/5] rust: task: add safe schedule_timeout() wrapper

rust/kernel/task.rs line 38

Quoted 2 lines · On Tue, 17 Nov 2026 01:14:42 +0100 Danilo Krummrich <[email protected]> wrote:
rust/kernel/task.rs +2 -0
+pub fn schedule_timeout(timeout: Delta<Jiffy>) -> Delta<Jiffy> {
+    let timeout = timeout.as_jiffies().max(0);

In earlier reviews of my patches, Gary asked me to use the function syntax or clamp() for this instead of the method syntax [1][2]:

    let timeout = isize::max(timeout.as_jiffies(), 0);

[1] https://lore.kernel.org/rust-for-linux/[email protected]/
[2] https://lore.kernel.org/rust-for-linux/[email protected]/

maintainer
#3 lore raw

Re: [PATCH v3 1/5] rust: task: add safe schedule_timeout() wrapper

Quoted 10 lines · FUJITA Tomonori

rust/kernel/task.rs line 38

Quoted 2 lines · On Tue, 17 Nov 2026 01:14:42 +0100 Danilo Krummrich <[email protected]> wrote:
rust/kernel/task.rs +2 -0
+pub fn schedule_timeout(timeout: Delta<Jiffy>) -> Delta<Jiffy> {
+    let timeout = timeout.as_jiffies().max(0);

In earlier reviews of my patches, Gary asked me to use the function syntax or clamp() for this instead of the method syntax [1][2]:

    let timeout = isize::max(timeout.as_jiffies(), 0);

Yes, please do. I'm okay with either isize::max or cmp::max.

Best,
Gary

Quoted 3 lines

[1] https://lore.kernel.org/rust-for-linux/[email protected]/
[2] https://lore.kernel.org/rust-for-linux/[email protected]/

author reviewer
#4 lore raw

Re: [PATCH v3 1/5] rust: task: add safe schedule_timeout() wrapper

Quoted 7 lines · Gary Guo
Quoted 6 lines · FUJITA Tomonori
Quoted 2 lines · On Tue, 17 Nov 2026 01:14:42 +0100 Danilo Krummrich <[email protected]> wrote:
rust/kernel/task.rs +2 -0
+pub fn schedule_timeout(timeout: Delta<Jiffy>) -> Delta<Jiffy> {
+    let timeout = timeout.as_jiffies().max(0);
Quoted 6 lines
Quoted 4 lines

In earlier reviews of my patches, Gary asked me to use the function syntax or clamp() for this instead of the method syntax [1][2]:

    let timeout = isize::max(timeout.as_jiffies(), 0);

Yes, please do. I'm okay with either isize::max or cmp::max.

The examples in [1][2] both seemed to be able to avoid a cast with this, which isn't the case for this patch. There also doesn't seem to be any ambiguity in the extression used in this patch. So I don't see why changing it would be better.

Thanks,
Danilo

Quoted 2 lines
Quoted 2 lines

[1] https://lore.kernel.org/rust-for-linux/[email protected]/
[2] https://lore.kernel.org/rust-for-linux/[email protected]/

maintainer
#5 lore raw

Re: [PATCH v3 1/5] rust: task: add safe schedule_timeout() wrapper

Quoted 11 lines · Danilo Krummrich
Quoted 10 lines · Gary Guo
Quoted 9 lines · FUJITA Tomonori

rust/kernel/task.rs line 38

Quoted 2 lines · On Tue, 17 Nov 2026 01:14:42 +0100 Danilo Krummrich <[email protected]> wrote:
rust/kernel/task.rs +2 -0
+pub fn schedule_timeout(timeout: Delta<Jiffy>) -> Delta<Jiffy> {
+    let timeout = timeout.as_jiffies().max(0);

In earlier reviews of my patches, Gary asked me to use the function syntax or clamp() for this instead of the method syntax [1][2]:

Quoted 8 lines
Quoted 3 lines
Quoted 1 line
    let timeout = isize::max(timeout.as_jiffies(), 0);

Yes, please do. I'm okay with either isize::max or cmp::max.

The examples in [1][2] both seemed to be able to avoid a cast with this, which isn't the case for this patch. There also doesn't seem to be any ambiguity in the extression used in this patch. So I don't see why changing it would be better.

The ambiguity is "the maximum value returned will be 0" and "the minimum value returned will be 0". The answer is always the latter but I'd avoid the mental gymnastics.

Best,
Gary

author reviewer
#6 lore raw

Re: [PATCH v3 1/5] rust: task: add safe schedule_timeout() wrapper

Quoted 16 lines · Gary Guo
Quoted 15 lines · Danilo Krummrich
Quoted 13 lines · Gary Guo
Quoted 10 lines · FUJITA Tomonori

rust/kernel/task.rs line 38

Quoted 2 lines · On Tue, 17 Nov 2026 01:14:42 +0100 Danilo Krummrich <[email protected]> wrote:
rust/kernel/task.rs +2 -0
+pub fn schedule_timeout(timeout: Delta<Jiffy>) -> Delta<Jiffy> {
+    let timeout = timeout.as_jiffies().max(0);

In earlier reviews of my patches, Gary asked me to use the function syntax or clamp() for this instead of the method syntax [1][2]:

    let timeout = isize::max(timeout.as_jiffies(), 0);

Yes, please do. I'm okay with either isize::max or cmp::max.

Quoted 8 lines
Quoted 4 lines

The examples in [1][2] both seemed to be able to avoid a cast with this, which isn't the case for this patch. There also doesn't seem to be any ambiguity in the extression used in this patch. So I don't see why changing it would be better.

The ambiguity is "the maximum value returned will be 0" and "the minimum value returned will be 0". The answer is always the latter but I'd avoid the mental gymnastics.

To me clearly both

	a.max(b)	(1)

	max(a, b)	(2)

read as the maximum of a and b.

IIUC, what you are saying is that people may read (1) as

	a.at_max(b)

which would indeed be equivalent to min(a, b).

If this is indeed commonly the case I'd suggest to raise this to become a common rule (ideally with a lint), such that we avoid this to be discussed in reviews.

Personally, I prefer (1) for the expression above, but I don't care too much, so I'm fine to change it. (In case I don't need to respin I'll change it on apply.)

Thanks,
Danilo

author reviewer
#8 lore raw

PATCH v3 2/5 rust: sync: add WaitQueue infrastructure

In reply to Danilo Krummrich

Implement a wait queue wrapping the kernel's struct wait_queue_head, with wait_event()-style methods that take a condition closure directly.

The API mirrors the C wait_event() family:

- wait_event()
- wait_event_interruptible()
- wait_event_timeout()
- wait_event_interruptible_timeout()
- wake_up() / wake_up_all() / wake_up_sync()

The interruptible variant returns Result<(), Interrupted>, the timeout variant Result<(), TimedOut>, and the combined variant Result<(), WaitError>. These errors map to ERESTARTSYS (signal) or ETIMEDOUT (timeout).

---
 rust/kernel/Kconfig.test |  10 +
 rust/kernel/sync.rs      |   8 +
 rust/kernel/sync/wait.rs | 549 +++++++++++++++++++++++++++++++++++++++
 3 files changed, 567 insertions(+)
 create mode 100644 rust/kernel/sync/wait.rs
rust/kernel/Kconfig.test +10 -0
index 3bf4194f28c7..7a1a464b6b29 100644
@@ -104,4 +104,14 @@ config RUST_TIME_KUNIT_TEST
 
 	  If unsure, say N.
 
+config RUST_WAITQUEUE_KUNIT_TEST
+	bool "KUnit tests for the Rust wait queue API" if !KUNIT_ALL_TESTS
+	default KUNIT_ALL_TESTS
+	help
+	  This option enables KUnit tests for the Rust wait queue API.
+	  These are only for development and testing, not for regular
+	  kernel use cases.
+
+	  If unsure, say N.
+
 endif
rust/kernel/sync.rs +8 -0
index ee372c048a8a..5f7ffd828259 100644
@@ -19,6 +19,7 @@
 mod refcount;
 mod set_once;
 pub mod srcu;
+mod wait;
 
 pub use arc::{Arc, ArcBorrow, UniqueArc};
 pub use completion::Completion;
@@ -41,3 +42,10 @@
 pub use refcount::Refcount;
 pub use set_once::SetOnce;
 pub use srcu::Srcu;
+pub use wait::{
+    new_waitqueue,
+    Interrupted,
+    TimedOut,
+    WaitError,
+    WaitQueue, //
+};
rust/kernel/sync/wait.rs new file +549 -0

549 lines not on this page · raw message

Signature · 1 line
2.56.1
maintainer
#9 lore raw
Quoted 99 lines · Danilo Krummrich

Implement a wait queue wrapping the kernel's struct wait_queue_head, with wait_event()-style methods that take a condition closure directly.

The API mirrors the C wait_event() family:

- wait_event()
- wait_event_interruptible()
- wait_event_timeout()
- wait_event_interruptible_timeout()
- wake_up() / wake_up_all() / wake_up_sync()

The interruptible variant returns Result<(), Interrupted>, the timeout variant Result<(), TimedOut>, and the combined variant Result<(), WaitError>. These errors map to ERESTARTSYS (signal) or ETIMEDOUT (timeout).

---
 rust/kernel/Kconfig.test |  10 +
 rust/kernel/sync.rs      |   8 +
 rust/kernel/sync/wait.rs | 549 +++++++++++++++++++++++++++++++++++++++
 3 files changed, 567 insertions(+)
 create mode 100644 rust/kernel/sync/wait.rs
rust/kernel/Kconfig.test +10 -0
index 3bf4194f28c7..7a1a464b6b29 100644
@@ -104,4 +104,14 @@ config RUST_TIME_KUNIT_TEST
 
 	  If unsure, say N.
 
+config RUST_WAITQUEUE_KUNIT_TEST
+	bool "KUnit tests for the Rust wait queue API" if !KUNIT_ALL_TESTS
+	default KUNIT_ALL_TESTS
+	help
+	  This option enables KUnit tests for the Rust wait queue API.
+	  These are only for development and testing, not for regular
+	  kernel use cases.
+
+	  If unsure, say N.
+
 endif
rust/kernel/sync.rs +8 -0
index ee372c048a8a..5f7ffd828259 100644
@@ -19,6 +19,7 @@
 mod refcount;
 mod set_once;
 pub mod srcu;
+mod wait;
 
 pub use arc::{Arc, ArcBorrow, UniqueArc};
 pub use completion::Completion;
@@ -41,3 +42,10 @@
 pub use refcount::Refcount;
 pub use set_once::SetOnce;
 pub use srcu::Srcu;
+pub use wait::{
+    new_waitqueue,
+    Interrupted,
+    TimedOut,
+    WaitError,
+    WaitQueue, //
+};
rust/kernel/sync/wait.rs new file +27 -0
new file mode 100644
index 000000000000..abbd5466ff44
@@ -0,0 +1,549 @@
+// SPDX-License-Identifier: GPL-2.0
+// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+
+//! Wait queue.
+//!
+//! C header: [`include/linux/wait.h`](srctree/include/linux/wait.h)
+
+use super::LockClassKey;
+use crate::{
+    prelude::*,
+    str::CStr,
+    task,
+    time::{
+        Delta,
+        Jiffy, //
+    },
+    types::Opaque, //
+};
+
+use core::{
+    pin::Pin,
+    ptr, //
+};
+
+/// Creates a [`WaitQueue`] initialiser with the given name and a newly-created lock class.
+#[macro_export]
+macro_rules! new_waitqueue {

rust/kernel/sync/wait.rs line 37

Quoted 10 lines
rust/kernel/sync/wait.rs new file +10 -0
+    ($($name:literal)?) => {
+        $crate::sync::WaitQueue::new(
+            $crate::optional_name!($($name)?),
+            $crate::static_lock_class!(),
+        )
+    };
+}
+pub use new_waitqueue;
+
+/// Exposes the kernel's [`struct wait_queue_head`] as a Rust wait queue.

The first paragraph is rendered as rustdoc short item description so please have something more descriptive and leave the C equivalence to other pragraphs.

Best,
Gary

Quoted 45 lines
rust/kernel/sync/wait.rs new file +45 -0
+///
+/// A `WaitQueue` allows a thread to sleep until a caller-supplied condition becomes true,
+/// re-checking the condition on each wake-up. This matches the C `wait_event()` family of macros.
+///
+/// For waiting with a lock guard (the condition variable pattern), use [`CondVar`](super::CondVar)
+/// instead.
+///
+/// Instances of `WaitQueue` need a lock class and to be pinned. The recommended way to create such
+/// instances is with the [`pin_init!`] and [`new_waitqueue!`] macros.
+///
+/// # Examples
+///
+/// ```
+/// use kernel::sync::{
+///     atomic::{
+///         Atomic,
+///         Relaxed,
+///     },
+///     new_waitqueue,
+///     WaitQueue,
+/// };
+///
+/// #[pin_data]
+/// pub struct Example {
+///     value: Atomic<i32>,
+///     #[pin]
+///     queue: WaitQueue,
+/// }
+///
+/// fn wait_for_value(e: &Example, v: i32) {
+///     e.queue.wait_event(|| e.value.load(Relaxed) == v);
+/// }
+///
+/// fn set_value(e: &Example, v: i32) {
+///     e.value.store(v, Relaxed);
+///     e.queue.wake_up();
+/// }
+/// ```
+///
+/// [`struct wait_queue_head`]: srctree/include/linux/wait.h
+#[pin_data]
+pub struct WaitQueue {
+    #[pin]
+    wait_queue_head: Opaque<bindings::wait_queue_head>,
+}
maintainer Reviewed-by on v3 2/5
#10 lore raw
In reply to Danilo Krummrich
Quoted 11 lines · Danilo Krummrich

Implement a wait queue wrapping the kernel's struct wait_queue_head, with wait_event()-style methods that take a condition closure directly.

The API mirrors the C wait_event() family:

- wait_event()
- wait_event_interruptible()
- wait_event_timeout()
- wait_event_interruptible_timeout()
- wake_up() / wake_up_all() / wake_up_sync()

Quoted 6 lines

The interruptible variant returns Result<(), Interrupted>, the timeout variant Result<(), TimedOut>, and the combined variant Result<(), WaitError>. These errors map to ERESTARTSYS (signal) or ETIMEDOUT (timeout).

Thanks for addressing the feedback from v1!

Alice

author reviewer
#11 lore raw

PATCH v3 3/5 rust: sync: convert CondVar and PollCondVar to use WaitQueue

In reply to Danilo Krummrich

CondVar currently wraps a raw wait_queue_head directly. Now that WaitQueue provides the same primitive, convert CondVar to use it instead. This removes duplicate init, pinning, Send/Sync, and wake-up code from CondVar.

Keep the wait logic in CondVar since it needs to release and reacquire a lock guard around the sleep, which WaitQueue's condition-based wait_event() methods do not handle. Use wait_once_exclusive() to perform a single prepare_to_wait_exclusive / finish_wait cycle with a caller-provided schedule function to implement the lock-aware sleep.

Adapt PollCondVar accordingly.

---
 rust/kernel/sync/condvar.rs | 89 +++++++++++--------------------------
 rust/kernel/sync/poll.rs    | 19 ++++----
 rust/kernel/sync/wait.rs    |  2 -
 3 files changed, 33 insertions(+), 77 deletions(-)
rust/kernel/sync/condvar.rs +25 -64
index 4be2c3bf8802..2deeaf2038e4 100644
@@ -5,21 +5,29 @@
 //! This module allows Rust code to use the kernel's [`struct wait_queue_head`] as a condition
 //! variable.
 
-use super::{lock::Backend, lock::Guard, LockClassKey};
+use super::{
+    lock::Backend,
+    lock::Guard,
+    LockClassKey,
+    WaitQueue, //
+};
+
 use crate::{
-    ffi::{c_int, c_long},
-    str::{CStr, CStrExt as _},
+    prelude::*,
+    str::CStr,
     task::{
-        MAX_SCHEDULE_TIMEOUT, TASK_FREEZABLE, TASK_INTERRUPTIBLE, TASK_NORMAL, TASK_UNINTERRUPTIBLE,
+        MAX_SCHEDULE_TIMEOUT,
+        TASK_FREEZABLE,
+        TASK_INTERRUPTIBLE,
+        TASK_UNINTERRUPTIBLE, //
     },
     time::{
         Delta,
         Jiffy, //
     },
-    types::Opaque,
 };
-use core::{marker::PhantomPinned, pin::Pin, ptr};
-use pin_init::{pin_data, pin_init, PinInit};
+
+use core::pin::Pin;
 
 /// Creates a [`CondVar`] initialiser with the given name and a newly-created lock class.
 #[macro_export]
@@ -84,33 +92,14 @@ macro_rules! new_condvar {
 #[pin_data]
 pub struct CondVar {
     #[pin]
-    pub(crate) wait_queue_head: Opaque<bindings::wait_queue_head>,
-
-    /// A condvar needs to be pinned because it contains a [`struct list_head`] that is
-    /// self-referential, so it cannot be safely moved once it is initialised.
-    ///
-    /// [`struct list_head`]: srctree/include/linux/types.h
-    #[pin]
-    _pin: PhantomPinned,
+    pub(crate) wq: WaitQueue,
 }
 
-// SAFETY: `CondVar` only uses a `struct wait_queue_head`, which is safe to use on any thread.
-unsafe impl Send for CondVar {}
-
-// SAFETY: `CondVar` only uses a `struct wait_queue_head`, which is safe to use on multiple threads
-// concurrently.
-unsafe impl Sync for CondVar {}
-
 impl CondVar {
     /// Constructs a new condvar initialiser.
     pub fn new(name: &'static CStr, key: Pin<&'static LockClassKey>) -> impl PinInit<Self> {
         pin_init!(Self {
-            _pin: PhantomPinned,
-            // SAFETY: `slot` is valid while the closure is called and both `name` and `key` have
-            // static lifetimes so they live indefinitely.
-            wait_queue_head <- Opaque::ffi_init(|slot| unsafe {
-                bindings::__init_waitqueue_head(slot, name.as_char_ptr(), key.as_ptr())
-            }),
+            wq <- WaitQueue::new(name, key),
         })
     }
 
@@ -120,23 +109,10 @@ fn wait_internal<T: ?Sized, B: Backend>(
         guard: &mut Guard<'_, T, B>,
         timeout_in_jiffies: c_long,
     ) -> c_long {
-        let wait = Opaque::<bindings::wait_queue_entry>::uninit();
-
-        // SAFETY: `wait` points to valid memory.
-        unsafe { bindings::init_wait(wait.get()) };
-
-        // SAFETY: Both `wait` and `wait_queue_head` point to valid memory.
-        unsafe {
-            bindings::prepare_to_wait_exclusive(self.wait_queue_head.get(), wait.get(), wait_state)
-        };
-
-        // SAFETY: Switches to another thread. The timeout can be any number.
-        let ret = guard.do_unlocked(|| unsafe { bindings::schedule_timeout(timeout_in_jiffies) });
-
-        // SAFETY: Both `wait` and `wait_queue_head` point to valid memory.
-        unsafe { bindings::finish_wait(self.wait_queue_head.get(), wait.get()) };
-
-        ret
+        self.wq.wait_once_exclusive(wait_state, || {
+            // SAFETY: Switches to another thread. The timeout can be any number.
+            guard.do_unlocked(|| unsafe { bindings::schedule_timeout(timeout_in_jiffies) })
+        })
     }
 
     /// Releases the lock and waits for a notification in uninterruptible mode.
@@ -211,28 +187,13 @@ pub fn wait_interruptible_timeout<T: ?Sized, B: Backend>(
         }
     }
 
-    /// Calls the kernel function to notify the appropriate number of threads.
-    fn notify(&self, count: c_int) {
-        // SAFETY: `wait_queue_head` points to valid memory.
-        unsafe {
-            bindings::__wake_up(
-                self.wait_queue_head.get(),
-                TASK_NORMAL,
-                count,
-                ptr::null_mut(),
-            )
-        };
-    }
-
     /// Calls the kernel function to notify one thread synchronously.
     ///
     /// This method behaves like `notify_one`, except that it hints to the scheduler that the
-    /// current thread is about to go to sleep, so it should schedule the target thread on the same
-    /// CPU.
+    /// current thread expects to sleep soon.
     #[inline]
     pub fn notify_sync(&self) {
-        // SAFETY: `wait_queue_head` points to valid memory.
-        unsafe { bindings::__wake_up_sync(self.wait_queue_head.get(), TASK_NORMAL) };
+        self.wq.wake_up_sync();
     }
 
     /// Wakes a single waiter up, if any.
@@ -241,7 +202,7 @@ pub fn notify_sync(&self) {
     /// completely (as opposed to automatically waking up the next waiter).
     #[inline]
     pub fn notify_one(&self) {
-        self.notify(1);
+        self.wq.wake_up();
     }
 
     /// Wakes all waiters up, if any.
@@ -250,7 +211,7 @@ pub fn notify_one(&self) {
     /// completely (as opposed to automatically waking up the next waiter).
     #[inline]
     pub fn notify_all(&self) {
-        self.notify(0);
+        self.wq.wake_up_all();
     }
 }
 
rust/kernel/sync/poll.rs +8 -11
index dc40bfaa57e6..fbd754a0bc54 100644
@@ -64,11 +64,11 @@ pub fn register_wait(&self, file: &File, cv: &PollCondVar) {
         // * `file.as_ptr()` references a valid file for the duration of this call.
         // * `self.table` is null or references a valid poll_table for the duration of this call.
         // * Since `PollCondVar` is pinned, its destructor is guaranteed to run before the memory
-        //   containing `cv.wait_queue_head` is invalidated. Since the destructor clears all
-        //   waiters and then waits for an rcu grace period, it's guaranteed that
-        //   `cv.wait_queue_head` remains valid for at least an rcu grace period after the removal
-        //   of the last waiter.
-        unsafe { bindings::poll_wait(file.as_ptr(), cv.wait_queue_head.get(), self.table) }
+        //   containing the wait queue head is invalidated. Since the destructor clears all
+        //   waiters and then waits for an rcu grace period, it's guaranteed that the wait queue
+        //   head remains valid for at least an rcu grace period after the removal of the last
+        //   waiter.
+        unsafe { bindings::poll_wait(file.as_ptr(), cv.wq.as_raw(), self.table) }
     }
 }
 
@@ -105,9 +105,7 @@ impl PinnedDrop for PollCondVar {
     #[inline]
     fn drop(self: Pin<&mut Self>) {
         // Clear anything registered using `register_wait`.
-        //
-        // SAFETY: The pointer points at a valid `wait_queue_head`.
-        unsafe { bindings::__wake_up_pollfree(self.inner.wait_queue_head.get()) };
+        self.inner.wq.wake_up_pollfree();
 
         // Wait for epoll items to be properly removed.
         synchronize_rcu();
@@ -165,14 +163,13 @@ fn drop(&mut self) {
         // SAFETY: ManuallyDrop::take ok because not already taken.
         let boxed = unsafe { ManuallyDrop::take(&mut self.inner) };
 
+        boxed.inner.inner.wq.wake_up_pollfree();
+
         // SAFETY: The code below frees the box without calling the actual destructor of the type,
         // but it's okay because it re-implements the destructor using `kfree_rcu()` in place of
         // `synchronize_rcu()`.
         let ptr = KBox::into_raw(unsafe { Pin::into_inner_unchecked(boxed) });
 
-        // SAFETY: The pointer points at a valid `wait_queue_head`.
-        unsafe { bindings::__wake_up_pollfree((*ptr).inner.inner.wait_queue_head.get()) };
-
         // SAFETY: This was allocated using `KBox::pin_init`, so it can be freed with `kvfree`.
         unsafe { bindings::kvfree_call_rcu((*ptr).rcu.get(), ptr.cast::<ffi::c_void>()) };
     }
rust/kernel/sync/wait.rs +0 -2
index abbd5466ff44..819dbd0eb3a0 100644
@@ -238,7 +238,6 @@ pub(super) fn wait_once<F, R>(&self, wait_state: c_int, schedule_fn: F) -> R
     /// Performs a single exclusive prepare-to-wait / finish-wait cycle, calling `schedule_fn`
     /// in between.
     #[inline]
-    #[expect(unused)]
     pub(super) fn wait_once_exclusive<F, R>(&self, wait_state: c_int, schedule_fn: F) -> R
     where
         F: FnOnce() -> R,
@@ -322,7 +321,6 @@ pub fn wake_up_sync(&self) {
     /// Used when a wait queue is about to be freed, to ensure epoll items are properly removed.
     /// Matches C's `wake_up_pollfree()`.
     #[inline]
-    #[expect(unused)]
     pub(super) fn wake_up_pollfree(&self) {
         // SAFETY: `wait_queue_head` points to valid memory.
         unsafe { bindings::__wake_up_pollfree(self.wait_queue_head.get()) };
Signature · 1 line
2.56.1
maintainer Reviewed-by on v3 3/5
#12 lore raw

Re: [PATCH v3 3/5] rust: sync: convert CondVar and PollCondVar to use WaitQueue

Quoted 11 lines · Danilo Krummrich

CondVar currently wraps a raw wait_queue_head directly. Now that WaitQueue provides the same primitive, convert CondVar to use it instead. This removes duplicate init, pinning, Send/Sync, and wake-up code from CondVar.

Keep the wait logic in CondVar since it needs to release and reacquire a lock guard around the sleep, which WaitQueue's condition-based wait_event() methods do not handle. Use wait_once_exclusive() to perform a single prepare_to_wait_exclusive / finish_wait cycle with a caller-provided schedule function to implement the lock-aware sleep.

Quoted 3 lines

Adapt PollCondVar accordingly.

author reviewer
#13 lore raw

PATCH v3 4/5 rust: sync: condvar: use task::schedule_timeout()

In reply to Danilo Krummrich

Convert CondVar::wait_internal() from calling bindings::schedule_timeout() directly to using the safe task::schedule_timeout() wrapper. This removes the unsafe call from CondVar.

---
 rust/kernel/sync/condvar.rs | 42 ++++++++++++++++++-------------------
 1 file changed, 20 insertions(+), 22 deletions(-)
rust/kernel/sync/condvar.rs +20 -22
index 2deeaf2038e4..dae07dc140c5 100644
@@ -16,7 +16,7 @@
     prelude::*,
     str::CStr,
     task::{
-        MAX_SCHEDULE_TIMEOUT,
+        self,
         TASK_FREEZABLE,
         TASK_INTERRUPTIBLE,
         TASK_UNINTERRUPTIBLE, //
@@ -107,11 +107,10 @@ fn wait_internal<T: ?Sized, B: Backend>(
         &self,
         wait_state: c_int,
         guard: &mut Guard<'_, T, B>,
-        timeout_in_jiffies: c_long,
-    ) -> c_long {
+        timeout: Delta<Jiffy>,
+    ) -> Delta<Jiffy> {
         self.wq.wait_once_exclusive(wait_state, || {
-            // SAFETY: Switches to another thread. The timeout can be any number.
-            guard.do_unlocked(|| unsafe { bindings::schedule_timeout(timeout_in_jiffies) })
+            guard.do_unlocked(|| task::schedule_timeout(timeout))
         })
     }
 
@@ -122,7 +121,11 @@ fn wait_internal<T: ?Sized, B: Backend>(
     /// [`CondVar::notify_one`] or [`CondVar::notify_all`]. Note that it may also wake up
     /// spuriously.
     pub fn wait<T: ?Sized, B: Backend>(&self, guard: &mut Guard<'_, T, B>) {
-        self.wait_internal(TASK_UNINTERRUPTIBLE, guard, MAX_SCHEDULE_TIMEOUT);
+        self.wait_internal(
+            TASK_UNINTERRUPTIBLE,
+            guard,
+            Delta::from_jiffies(task::MAX_SCHEDULE_TIMEOUT),
+        );
     }
 
     /// Releases the lock and waits for a notification in interruptible mode.
@@ -133,7 +136,11 @@ pub fn wait<T: ?Sized, B: Backend>(&self, guard: &mut Guard<'_, T, B>) {
     /// Returns whether there is a signal pending.
     #[must_use = "wait_interruptible returns if a signal is pending, so the caller must check the return value"]
     pub fn wait_interruptible<T: ?Sized, B: Backend>(&self, guard: &mut Guard<'_, T, B>) -> bool {
-        self.wait_internal(TASK_INTERRUPTIBLE, guard, MAX_SCHEDULE_TIMEOUT);
+        self.wait_internal(
+            TASK_INTERRUPTIBLE,
+            guard,
+            Delta::from_jiffies(task::MAX_SCHEDULE_TIMEOUT),
+        );
         crate::current!().signal_pending()
     }
 
@@ -151,7 +158,7 @@ pub fn wait_interruptible_freezable<T: ?Sized, B: Backend>(
         self.wait_internal(
             TASK_INTERRUPTIBLE | TASK_FREEZABLE,
             guard,
-            MAX_SCHEDULE_TIMEOUT,
+            Delta::from_jiffies(task::MAX_SCHEDULE_TIMEOUT),
         );
         crate::current!().signal_pending()
     }
@@ -169,21 +176,12 @@ pub fn wait_interruptible_timeout<T: ?Sized, B: Backend>(
         guard: &mut Guard<'_, T, B>,
         timeout: Delta<Jiffy>,
     ) -> CondVarTimeoutResult {
-        let jiffies = timeout.as_jiffies();
-        let res = self.wait_internal(
-            TASK_INTERRUPTIBLE,
-            guard,
-            jiffies.clamp(0, MAX_SCHEDULE_TIMEOUT),
-        );
+        let remaining = self.wait_internal(TASK_INTERRUPTIBLE, guard, timeout);
 
-        match (res, crate::current!().signal_pending()) {
-            (remainder, true) => CondVarTimeoutResult::Signal {
-                remainder: Delta::from_jiffies(remainder),
-            },
-            (0, false) => CondVarTimeoutResult::Timeout,
-            (remainder, false) => CondVarTimeoutResult::Woken {
-                remainder: Delta::from_jiffies(remainder),
-            },
+        match (remaining, current!().signal_pending()) {
+            (remainder, true) => CondVarTimeoutResult::Signal { remainder },
+            (Delta::ZERO, false) => CondVarTimeoutResult::Timeout,
+            (remainder, false) => CondVarTimeoutResult::Woken { remainder },
         }
     }
 
Signature · 1 line
2.56.1
maintainer Reviewed-by on v3 4/5
#14 lore raw

Re: [PATCH v3 4/5] rust: sync: condvar: use task::schedule_timeout()

rust/kernel/sync/condvar.rs fn wait_internal<T: ?Sized, B: Backend>( · line 129

Quoted 11 lines · Danilo Krummrich
rust/kernel/sync/condvar.rs +5 -1
@@ -122,7 +121,11 @@ fn wait_internal<T: ?Sized, B: Backend>(
     /// [`CondVar::notify_one`] or [`CondVar::notify_all`]. Note that it may also wake up
     /// spuriously.
     pub fn wait<T: ?Sized, B: Backend>(&self, guard: &mut Guard<'_, T, B>) {
-        self.wait_internal(TASK_UNINTERRUPTIBLE, guard, MAX_SCHEDULE_TIMEOUT);
+        self.wait_internal(
+            TASK_UNINTERRUPTIBLE,
+            guard,
+            Delta::from_jiffies(task::MAX_SCHEDULE_TIMEOUT),
+        );
     }

Now that CondVar uses task::schedule_timeout(), all remaining uses of task::MAX_SCHEDULE_TIMEOUT wrap it in Delta::from_jiffies(...), so we could also change MAX_SCHEDULE_TIMEOUT itself to Delta<Jiffy> (either if you spin a v4 or as a follow-up). Either way:

Alice

#15 lore raw

Re: [PATCH v3 4/5] rust: sync: condvar: use task::schedule_timeout()

Quoted 18 lines · On Tue, 17 Nov 2026 14:16:52 +0000 Alice Ryhl <[email protected]> wrote:

rust/kernel/sync/condvar.rs fn wait_internal<T: ?Sized, B: Backend>( · line 129

Quoted 11 lines · Danilo Krummrich
rust/kernel/sync/condvar.rs +5 -1
@@ -122,7 +121,11 @@ fn wait_internal<T: ?Sized, B: Backend>(
     /// [`CondVar::notify_one`] or [`CondVar::notify_all`]. Note that it may also wake up
     /// spuriously.
     pub fn wait<T: ?Sized, B: Backend>(&self, guard: &mut Guard<'_, T, B>) {
-        self.wait_internal(TASK_UNINTERRUPTIBLE, guard, MAX_SCHEDULE_TIMEOUT);
+        self.wait_internal(
+            TASK_UNINTERRUPTIBLE,
+            guard,
+            Delta::from_jiffies(task::MAX_SCHEDULE_TIMEOUT),
+        );
     }

Now that CondVar uses task::schedule_timeout(), all remaining uses of task::MAX_SCHEDULE_TIMEOUT wrap it in Delta::from_jiffies(...), so we could also change MAX_SCHEDULE_TIMEOUT itself to Delta<Jiffy> (either if you spin a v4 or as a follow-up). Either way:

Quoted 1 line

serdev users need this value too. I was going to add Delta::<Jiffy>::MAX for it, but this looks better.

author reviewer
#16 lore raw

PATCH v3 5/5 rust: sync: condvar: add non-exclusive freezable waiting

In reply to Danilo Krummrich

CondVar currently registers all sleeping tasks as exclusive waiters. Add wait_interruptible_freezable_nonexclusive() for callers that need non-exclusive waits, using WaitQueue::wait_once().

Document that notify_one() and notify_sync() also wake all non-exclusive waiters.

This provides non-exclusive waiting support for a subsequent Binder conversion.

---
 rust/kernel/sync/condvar.rs | 29 +++++++++++++++++++++++++++--
 rust/kernel/sync/wait.rs    |  1 -
 2 files changed, 27 insertions(+), 3 deletions(-)
rust/kernel/sync/condvar.rs +27 -2
index dae07dc140c5..89f1a31afb89 100644
@@ -163,6 +163,31 @@ pub fn wait_interruptible_freezable<T: ?Sized, B: Backend>(
         crate::current!().signal_pending()
     }
 
+    /// Releases the lock and waits non-exclusively in interruptible and freezable mode.
+    ///
+    /// Like [`CondVar::wait_interruptible_freezable`], but registers the current task as a
+    /// non-exclusive waiter.
+    ///
+    /// The lock is reacquired before returning. The wait may also end due to a signal or a
+    /// spurious wakeup.
+    ///
+    /// No lock other than the one released through `guard` should be held, since the task may
+    /// be frozen while sleeping.
+    ///
+    /// Returns whether there is a signal pending.
+    #[must_use = "the caller must check whether a signal is pending"]
+    pub fn wait_interruptible_freezable_nonexclusive<T: ?Sized, B: Backend>(
+        &self,
+        guard: &mut Guard<'_, T, B>,
+    ) -> bool {
+        self.wq.wait_once(TASK_INTERRUPTIBLE | TASK_FREEZABLE, || {
+            guard.do_unlocked(|| {
+                task::schedule_timeout(Delta::from_jiffies(task::MAX_SCHEDULE_TIMEOUT))
+            })
+        });
+        crate::current!().signal_pending()
+    }
+
     /// Releases the lock and waits for a notification in interruptible mode.
     ///
     /// Atomically releases the given lock (whose ownership is proven by the guard) and puts the
@@ -185,7 +210,7 @@ pub fn wait_interruptible_timeout<T: ?Sized, B: Backend>(
         }
     }
 
-    /// Calls the kernel function to notify one thread synchronously.
+    /// Wakes all non-exclusive waiters and one exclusive waiter synchronously.
     ///
     /// This method behaves like `notify_one`, except that it hints to the scheduler that the
     /// current thread expects to sleep soon.
@@ -194,7 +219,7 @@ pub fn notify_sync(&self) {
         self.wq.wake_up_sync();
     }
 
-    /// Wakes a single waiter up, if any.
+    /// Wakes all non-exclusive waiters and one exclusive waiter, if any.
     ///
     /// This is not 'sticky' in the sense that if no thread is waiting, the notification is lost
     /// completely (as opposed to automatically waking up the next waiter).
rust/kernel/sync/wait.rs +0 -1
index 819dbd0eb3a0..8de7aed89e0c 100644
@@ -227,7 +227,6 @@ fn wait_event_timeout_internal<F: FnMut() -> bool>(
     /// Performs a single non-exclusive prepare-to-wait / finish-wait cycle, calling `schedule_fn`
     /// in between. Non-exclusive waiters are all woken by [`wake_up()`](Self::wake_up).
     #[inline]
-    #[expect(unused)]
     pub(super) fn wait_once<F, R>(&self, wait_state: c_int, schedule_fn: F) -> R
     where
         F: FnOnce() -> R,
Signature · 1 line
2.56.1
maintainer Reviewed-by on v3 5/5
#17 lore raw

Re: [PATCH v3 5/5] rust: sync: condvar: add non-exclusive freezable waiting

Quoted 11 lines · Danilo Krummrich

CondVar currently registers all sleeping tasks as exclusive waiters. Add wait_interruptible_freezable_nonexclusive() for callers that need non-exclusive waits, using WaitQueue::wait_once().

Document that notify_one() and notify_sync() also wake all non-exclusive waiters.

This provides non-exclusive waiting support for a subsequent Binder conversion.