workqueue: OwnedQueue and ScopedWork
Subsystems (3)
Alice Ryhl
reviewerv4 cover workqueue: OwnedQueue and ScopedWork
This series extends the Rust workqueue bindings with the ability to create custom workqueues (OwnedQueue), and introduces ScopedWork, a work item type that cancels synchronously on drop, allowing drivers to use workqueues with lifetime-bound data.
This is a continuation of Danilo's v2 [4], which picked up my earlier series [1] and Onur's ScopedQueue work [2].
Patch 1/6 of v2, "rust: workqueue: replace deprecated system_wq with system_{percpu,dfl}_wq", is a fix that was sent separately [3] and landed upstream in v7.4-rc1.
The soundness of this series depends on the ordered workqueue cancellation fix series [5].
[1] https://lore.kernel.org/all/[email protected]
[2] https://lore.kernel.org/all/[email protected]
[3] https://lore.kernel.org/all/[email protected]
[4] https://lore.kernel.org/all/[email protected]
[5] https://lore.kernel.org/r/[email protected]
---
Changes in v4:
- Rebase on v7.4-rc3
- ScopedWork: bind ScopedWork<'wq, T> to &'wq Queue at construction and replace Queue::enqueue_scoped() with ScopedWork::enqueue() so a work
item cannot be migrated across workqueues
- Add patch to pass object_is_on_stack() in init_work_with_key() and
call destroy_work_on_stack() when Work is dropped
- ScopedWork: call destroy_work_on_stack() on drop for CONFIG_DEBUG_OBJECTS_WORK support
- ScopedWork: initialize data before work_struct in ScopedWork::new() so init_work_with_key() is not called if the data initializer fails
- ScopedWork: add INVARIANT comments in ScopedWork::new()
- ScopedWork: require Sync on ScopedWorkItem
- Add Queue::as_raw()
- Pick up Reviewed-by tag from Philipp
- Link to v3: https://lore.kernel.org/r/[email protected]
Changes in v3:
- Rebase on v7.3-rc7
- Split out the system_wq patch and send it separately as a fix [3]; this series no longer depends on it
- Drop ScopedQueue: with the ScopedWork changes below it is no longer needed to enqueue non-'static work items, and it has no other user
- ScopedWork: make ScopedWork::new() unsafe instead of Queue::enqueue_scoped(), which is now a safe method accepting &ScopedWork<T> on any queue
- ScopedWork: drop the WorkItem/WorkItemPointer/HasWork impls and the public ScopedWorkRef type; embed the work_struct directly with a dedicated callback, and keep the Opaque-wrapped payload private
- ScopedWork: drop T on drop (Opaque does not run destructors)
- ScopedWork: propagate the initializer's error type from new()
- Builder: new_power_efficient() uses WQ_PERCPU | WQ_POWER_EFFICIENT; remove the now pointless percpu() modifier
- Pick up Reviewed-by tags from Daniel
- Link to v2: https://lore.kernel.org/all/[email protected]
Changes in v2:
- Merge scoped_work.rs and scoped_queue.rs into scoped.rs
- Introduce ScopedWorkRef<T> and wrap it in Opaque inside ScopedWork to avoid &mut aliasing in drop
- Pass &ScopedWorkRef<Self> to ScopedWorkItem::run to enable self-re-enqueue
- Add Deref<Target = Queue> for ScopedQueue; remove duplicate enqueue_scoped()
- Add Send + Sync for OwnedQueue
- Fix stale workqueue::system() references in docs
- Drop Work::cancel_work_sync()
- Make sysfs() available on ordered workqueues
- Add #[inline] annotations throughout
---
Alice Ryhl (4):
rust: workqueue: support stack-allocated work items
rust: workqueue: restrict delayed work to global wqs
rust: workqueue: create workqueue subdirectory
rust: workqueue: add creation of workqueues
Danilo Krummrich (1):
rust: workqueue: add ScopedWork for non-'static work items
MAINTAINERS | 1 +
rust/helpers/workqueue.c | 16 +-
rust/kernel/workqueue/builder.rs | 370 ++++++++++++++++++++++
rust/kernel/{workqueue.rs => workqueue/mod.rs} | 86 +++++-
rust/kernel/workqueue/scoped.rs | 410 +++++++++++++++++++++++++
5 files changed, 874 insertions(+), 9 deletions(-)---
base-commit: aebb7274476403040d582c61478df8f4a7bf9cd3
change-id: 20261012-rust-workqueue-658cd0fbe770
Best regards,
Signature · 1 line
Alice Ryhl <[email protected]>
Alice Ryhl
reviewerPATCH v4 1/5 rust: workqueue: support stack-allocated work items
Pass object_is_on_stack(work) to __init_work() in rust_helper_init_work_with_key() instead of taking an explicit onstack parameter, and call destroy_work_on_stack() when Work is dropped. This ensures CONFIG_DEBUG_OBJECTS_WORK properly tracks and cleans up stack-allocated work items without requiring separate constructors.
--- rust/helpers/workqueue.c | 9 +++++++-- rust/kernel/workqueue.rs | 13 ++++++++++--- 2 files changed, 17 insertions(+), 5 deletions(-)
rust/helpers/workqueue.c +7 -2
@@ -1,16 +1,21 @@// SPDX-License-Identifier: GPL-2.0 +#include <linux/sched/task_stack.h> #include <linux/workqueue.h> __rust_helper void rust_helper_init_work_with_key(struct work_struct *work,work_func_t func, -bool onstack,const char *name,struct lock_class_key *key) { -__init_work (work, onstack); +__init_work (work,object_is_on_stack (work)); work->data = (atomic_long_t )WORK_DATA_INIT ();lockdep_init_map (&work->lockdep_map, name, key,0 );INIT_LIST_HEAD (&work->entry); work->func = func; } + +__rust_helper void rust_helper_destroy_work_on_stack(struct work_struct *work) +{ +destroy_work_on_stack (work); +}
rust/kernel/workqueue.rs +10 -3
@@ -481,7 +481,7 @@ pub trait WorkItem<const ID: u64 = 0> {/// This is a helper type used to associate a `work_struct` with the [`WorkItem`] that uses it. /// /// [`run`]: WorkItemPointer::run -#[pin_data] +#[pin_data( PinnedDrop )] #[repr(transparent)] pub struct Work <T : ?Sized ,const ID :u64 =0 > { #[pin]
@@ -513,7 +513,6 @@ pub fn new(name: &'static CStr, key: Pin<&'static LockClassKey>) -> impl PinInit bindings::init_work_with_key ( slot,Some (T ::Pointer ::run), -false , name.as_char_ptr (), key.as_ptr (), )
@@ -539,6 +538,15 @@ pub unsafe fn raw_get(ptr: *const Self) -> *mut bindings::work_struct { } } +#[pinned_drop] +impl <T : ?Sized ,const ID :u64 >PinnedDrop for Work <T ,ID > { +fn drop (self:Pin <&mut Self >) { +// SAFETY: `self.work.get()` points to a valid `work_struct` initialized in `Work::new`, +// and the work item is not currently queued because `Self` is being dropped. +unsafe { bindings::destroy_work_on_stack (self.work.get ()) } + } +} +/// Declares that a type contains a [`Work<T, ID>`]. /// /// The intended way of using this trait is via the [`impl_has_work!`] macro. You can use the macro
@@ -686,7 +694,6 @@ pub fn new( bindings::init_work_with_key ( core::ptr::addr_of_mut!((*slot).work),Some (T ::Pointer ::run), -false , work_name.as_char_ptr (), work_key.as_ptr (), )
Signature · 1 line
2.56.1.708.g77570a6a1d-goog
Alice Ryhl
reviewerPATCH v4 2/5 rust: workqueue: restrict delayed work to global wqs
In reply to Alice RyhlWhen a workqueue is shut down, delayed work that is pending but not scheduled does not get properly cleaned up, so it's not safe to use enqueue_delayed on a workqueue that might be destroyed. To fix this, restricted enqueue_delayed to static queues.
This may be fixed in the future by an approach along the lines of [1].
--- rust/kernel/workqueue.rs | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-)
rust/kernel/workqueue.rs +8 -1
@@ -302,8 +302,15 @@ pub fn enqueue<W, const ID: u64>(&self, w: W) -> W::EnqueueOutput/// /// This may fail if the work item is already enqueued in a workqueue. /// +/// This is only valid for global workqueues (with static lifetimes) because those are the only +/// ones that outlive all possible delayed work items. +/// /// The work item will be submitted using `WORK_CPU_UNBOUND`. -pub fn enqueue_delayed <W ,const ID :u64 >(&self, w:W , delay:Jiffies ) ->W ::EnqueueOutput +pub fn enqueue_delayed <W ,const ID :u64 >( + &'static self, + w:W , + delay:Jiffies , + ) ->W ::EnqueueOutput where W :RawDelayedWorkItem <ID > +Send + 'static, {
Signature · 1 line
2.56.1.708.g77570a6a1d-goog
Alice Ryhl
reviewerPATCH v4 3/5 rust: workqueue: create workqueue subdirectory
In reply to Alice RyhlThe following patch will implement a workqueue builder in a separate file. To prepare for that, create a rust/kernel/workqueue subdirectory and move the existing file.
---
MAINTAINERS | 1 +
rust/kernel/{workqueue.rs => workqueue/mod.rs} | 0
2 files changed, 1 insertion(+)MAINTAINERS +1 -0
@@ -30299,6 +30299,7 @@ F: Documentation/core-api/workqueue.rst F: include/linux/workqueue.h F: kernel/workqueue.c F: kernel/workqueue_internal.h +F: rust/kernel/workqueue/ WWAN DRIVERS M: Loic Poulain <[email protected]>
rust/kernel/workqueue/mod.rs renamed from rust/kernel/workqueue.rs +0 -0
Signature · 1 line
2.56.1.708.g77570a6a1d-goog
Alice Ryhl
reviewerPATCH v4 4/5 rust: workqueue: add creation of workqueues
In reply to Alice RyhlCreating workqueues is needed by various GPU drivers. Not only does it give you better control over execution, it also allows devices to ensure that all tasks have exited before the device is unbound (or similar) by running the workqueue destructor.
--- rust/helpers/workqueue.c | 7 + rust/kernel/workqueue/builder.rs | 370 +++++++++++++++++++++++++++++++++++++++ rust/kernel/workqueue/mod.rs | 51 +++++- 3 files changed, 425 insertions(+), 3 deletions(-)
rust/helpers/workqueue.c +7 -0
@@ -19,3 +19,10 @@ __rust_helper void rust_helper_destroy_work_on_stack(struct work_struct *work) {destroy_work_on_stack (work); } + +__rust_helper +struct workqueue_struct *rust_helper_alloc_workqueue(const char *fmt, unsigned int flags, +int max_active,const void *data) +{ +return alloc_workqueue (fmt, flags, max_active, data); +}
rust/kernel/workqueue/builder.rs new file +370 -0
370 lines not on this page · raw message
rust/kernel/workqueue/mod.rs +48 -3
75 lines not on this page · raw message
Signature · 1 line
2.56.1.708.g77570a6a1d-goog
Alice Ryhl
reviewerPATCH v4 5/5 rust: workqueue: add ScopedWork for non-'static work items
In reply to Alice RyhlFrom: Danilo Krummrich <[email protected]>
Add ScopedWork<'wq, T>, a work item wrapper bound to a specific &'wq Queue whose destructor calls cancel_work_sync(), allowing T to carry non-'static lifetimes. Ownership of the data is not transferred to the workqueue; instead, the synchronous cancellation on drop guarantees the work function is not running when the data is freed.
Each ScopedWork<'wq, T> is bound to a single Queue at construction and enqueued via ScopedWork::enqueue() to uphold the non-reentrance conditions documented in Documentation/core-api/workqueue.rst: queuing an in-flight work item onto a different workqueue violates those conditions, in which case cancel_work_sync() only waits for execution on the last queued workqueue and may return while an earlier execution is still running.
Because the cancellation guarantee only holds if the destructor runs before the bound Queue or any lifetime in T expires, ScopedWork::new() (and thus new_scoped_work!) is unsafe and requires that the resulting value is never forgotten and is dropped before 'wq or T's lifetimes end. In return, ScopedWork::enqueue() is a safe method.
ScopedWork does not use the Work/WorkItem machinery: it embeds the work_struct directly and registers its own C callback, which recovers the &ScopedWork<'_, T> via container_of!() and passes it to ScopedWorkItem::run(). The work_struct, queue pointer, and user data live in a private Inner<T> that is wrapped in Opaque as a whole, so that the &mut Self held by the destructor while cancel_work_sync() waits does not claim uniqueness over memory (including padding) that a worker thread may still be reading through &ScopedWork<'_, T>.
--- rust/kernel/workqueue/mod.rs | 13 ++ rust/kernel/workqueue/scoped.rs | 410 ++++++++++++++++++++++++++++++++++++++++ 2 files changed, 423 insertions(+)
rust/kernel/workqueue/mod.rs +13 -0
@@ -212,6 +212,13 @@mod builder;pub use self::builder::Builder ; +mod scoped; +pub use self::scoped::{ + new_scoped_work, +ScopedWork , +ScopedWorkItem ,// +}; +/// Creates a [`Work`] initialiser with the given name and a newly-created lock class. #[macro_export] macro_rules! new_work {
@@ -275,6 +282,12 @@ pub unsafe fn from_raw<'a>(ptr: *const bindings::workqueue_struct) -> &'a Queueunsafe { &*ptr.cast ::<Queue >() } } +/// Obtain the raw `struct workqueue_struct` pointer. +#[inline] +pub fn as_raw (&self) -> *mut bindings::workqueue_struct { + self.0 .get () + } +/// Enqueues a work item. /// /// This may fail if the work item is already enqueued in a workqueue.
rust/kernel/workqueue/scoped.rs new file +410 -0
410 lines not on this page · raw message
Signature · 1 line
2.56.1.708.g77570a6a1d-goog
Gary Guo
reviewerRe: [PATCH v4 1/5] rust: workqueue: support stack-allocated work items
In reply to Alice RyhlQuoted 22 lines · Alice Ryhl
Pass
object_is_on_stack(work)to__init_work()inrust_helper_init_work_with_key()instead of taking an explicitonstackparameter, and calldestroy_work_on_stack()whenWorkis dropped. This ensuresCONFIG_DEBUG_OBJECTS_WORKproperly tracks and cleans up stack-allocated work items without requiring separate constructors.Fixes 7324b88975c5 ("rust: workqueue: add helper for defining work_struct fields")Signed-off-by Alice Ryhl <[email protected]>--- rust/helpers/workqueue.c | 9 +++++++-- rust/kernel/workqueue.rs | 13 ++++++++++--- 2 files changed, 17 insertions(+), 5 deletions(-)rust/helpers/workqueue.c +1
-0@@ -1,16 +1,21 @@// SPDX-License-Identifier: GPL-2.0 +#include <linux/sched/task_stack.h>
rust/helpers/workqueue.c line 11
Quoted 10 lines
rust/helpers/workqueue.c +1
-2#include <linux/workqueue.h> __rust_helper void rust_helper_init_work_with_key(struct work_struct *work,work_func_t func, -bool onstack,const char *name,struct lock_class_key *key) { -__init_work (work, onstack); +__init_work (work,object_is_on_stack (work));
Is this fully optimized out if !CONFIG_DEBUG_OBJECTS_WORK?
Best,
Gary
Quoted 55 lines
rust/helpers/workqueue.c +5
-0work->data = (atomic_long_t )WORK_DATA_INIT ();lockdep_init_map (&work->lockdep_map, name, key,0 );INIT_LIST_HEAD (&work->entry); work->func = func; } + +__rust_helper void rust_helper_destroy_work_on_stack(struct work_struct *work) +{ +destroy_work_on_stack (work); +}rust/kernel/workqueue.rs +10
-3@@ -481,7 +481,7 @@ pub trait WorkItem<const ID: u64 = 0> {/// This is a helper type used to associate a `work_struct` with the [`WorkItem`] that uses it. /// /// [`run`]: WorkItemPointer::run -#[pin_data] +#[pin_data( PinnedDrop )] #[repr(transparent)] pub struct Work <T : ?Sized ,const ID :u64 =0 > { #[pin]@@ -513,7 +513,6 @@ pub fn new(name: &'static CStr, key: Pin<&'static LockClassKey>) -> impl PinInit bindings::init_work_with_key ( slot,Some (T ::Pointer ::run), -false , name.as_char_ptr (), key.as_ptr (), )@@ -539,6 +538,15 @@ pub unsafe fn raw_get(ptr: *const Self) -> *mut bindings::work_struct { } } +#[pinned_drop] +impl <T : ?Sized ,const ID :u64 >PinnedDrop for Work <T ,ID > { +fn drop (self:Pin <&mut Self >) { +// SAFETY: `self.work.get()` points to a valid `work_struct` initialized in `Work::new`, +// and the work item is not currently queued because `Self` is being dropped. +unsafe { bindings::destroy_work_on_stack (self.work.get ()) } + } +} +/// Declares that a type contains a [`Work<T, ID>`]. /// /// The intended way of using this trait is via the [`impl_has_work!`] macro. You can use the macro @@ -686,7 +694,6 @@ pub fn new( bindings::init_work_with_key ( core::ptr::addr_of_mut!((*slot).work),Some (T ::Pointer ::run), -false , work_name.as_char_ptr (), work_key.as_ptr (), )
Re: [PATCH v4 3/5] rust: workqueue: create workqueue subdirectory
In reply to Alice RyhlQuoted 7 lines · Alice Ryhl
The following patch will implement a workqueue builder in a separate file. To prepare for that, create a
rust/kernel/workqueuesubdirectory and move the existing file.Reviewed-by Daniel Almeida <[email protected]>Reviewed-by Philipp Stanner <[email protected]>Signed-off-by Alice Ryhl <[email protected]>
Quoted 4 lines
---
MAINTAINERS | 1 +
rust/kernel/{workqueue.rs => workqueue/mod.rs} | 0
2 files changed, 1 insertion(+)Alice Ryhl
reviewerRe: [PATCH v4 1/5] rust: workqueue: support stack-allocated work items
In reply to Gary GuoQuoted 11 lines · Gary Guo
rust/helpers/workqueue.c line 11
Quoted 8 lines · Alice Ryhl
__rust_helpervoid rust_helper_init_work_with_key(structwork_struct*work,rust/helpers/workqueue.c +1
-2work_func_t func, - bool onstack, const char *name, struct lock_class_key *key) { -__init_work (work ,onstack ); +__init_work (work ,object_is_on_stack (work ));Is this fully optimized out if !CONFIG_DEBUG_OBJECTS_WORK?
Yes, on normal architectures (checked x86_64 and arm64 with Clang and GCC): get_current() omits volatile and current->stack is a plain load, so DCE removes all of object_is_on_stack(work) once the empty __init_work() stub is inlined.
The one exception is ARCH=um with CONFIG_SMP=y, where get_current() calls an external uml_curr_cpu() function without __attribute_const__ (and under CONFIG_RUST_INLINE_HELPERS, the dead mrs/movq %gs:... inline asm stays in helpers.bc until backend codegen drops it). I can wrap it in if (IS_ENABLED(CONFIG_DEBUG_OBJECTS_WORK)) if I send a v5.
Alice