Skip to contentSkip to conversation

workqueue: Implement BH workqueue and convert several tasklet users

PATCH In review Tejun Heo 14 participants 59 messages Jan 30, 2024 – May 27, 2025

Subsystems (7)
lore
Share & tools
lore

Tejun Heo

author maintainer
#0 lore raw

PATCH 1/1 workqueue: Implement BH workqueue and convert several tasklet users

Hello,

The only generic interface to execute asynchronously in the BH context is tasklet; however, it's marked deprecated and has some design flaws such as the execution code accessing the tasklet item after the execution is complete which can lead to subtle use-after-free in certain usage scenarios and less-developed flush and cancel mechanisms.

Mikulas Patocka recently reported that dm-crypt and dm-crypt are affected by the access-after-completion issue and suggested adding TASKLET_STATE_ONESHOT flag which selectively removes post-completion access while significantly limiting how the tasklet can be used in the following thread:

http://lkml.kernel.org/r/[email protected]

Linus didn't like the approach and suggested extending workqueue to support execution from atomic context:

http://lkml.kernel.org/r/CAHk-=wjDW53w4-YcSmgKC5RruiRLHmJ1sXeYdp_ZgVoBw=5byA@mail.gmail.com

As suggested, this patchset implements BH workqueues which are like regular workqueues but executes work items in the BH (softirq) context and converts several tasklet users.

- The name bh is used instead of the suggested atomic as it's more in line with widely used execution context interface - local_bh_enable/disable() and friends.

- The system default BH workqueues - system_bh_wq and system_bh_highpri_wq - are provided. As queue-wide flushing doesn't exist in tasklet, all existing tasklet users should be able to use the system BH workqueues without creating their own.

- BH workqueues currently use tasklet to run the work items to avoid priority inversions involving tasklet_hi and WQ_BH | WQ_HIGHPRI. Once all tasklet users are converted, tasklet code can be removed and BH workqueues can take over its softirqs.

This patchset is on top of wq/for-6.9 (aae17ebb53c ("workqueue: Avoid using isolated cpus' timers on queue_delayed_work")) and contains the following eight patches.

0001-workqueue-Update-lock-debugging-code.patch 0002-workqueue-Factor-out-init_cpu_worker_pool.patch 0003-workqueue-Implement-BH-workqueues-to-eventually-repl.patch 0004-backtracetest-Convert-from-tasklet-to-BH-workqueue.patch 0005-usb-core-hcd-Convert-from-tasklet-to-BH-workqueue.patch 0006-net-tcp-tsq-Convert-from-tasklet-to-BH-workqueue.patch 0007-dm-crypt-Convert-from-tasklet-to-BH-workqueue.patch 0008-dm-verity-Convert-from-tasklet-to-BH-workqueue.patch

0001-0003 prepare and implement BH workqueues.

0004-0008 convert some tasklet users to BH workqueue. The conversions are relatively straightforward but are in descending order of confidence.

The patchset is also available in the following git branch.

git://git.kernel.org/pub/scm/linux/kernel/git/tj/wq.git wq-bh-v1

diffstat follows. Thanks.

 Documentation/core-api/workqueue.rst |   29 ++++-
 drivers/md/dm-crypt.c                |   36 -------
 drivers/md/dm-verity-target.c        |    8 -
 drivers/md/dm-verity.h               |    2
 drivers/usb/core/hcd.c               |   23 ++--
 include/linux/usb/hcd.h              |    2
 include/linux/workqueue.h            |    9 +
 include/net/tcp.h                    |    2
 kernel/backtracetest.c               |   18 +--
 kernel/workqueue.c                   |  312 ++++++++++++++++++++++++++++++++++++++++++++++++--------------
 kernel/workqueue_internal.h          |    3
 net/ipv4/tcp.c                       |    2
 net/ipv4/tcp_output.c                |   36 +++----
 tools/workqueue/wq_dump.py           |   11 +-
 14 files changed, 335 insertions(+), 158 deletions(-)

--
tejun

author maintainer
#1 lore raw

PATCH 1/8 workqueue: Update lock debugging code

These changes are in preparation of BH workqueue which will execute work items from BH context.

- Update lock and RCU depth checks in process_one_work() so that it remembers and checks against the starting depths and prints out the depth changes.

- Factor out lockdep annotations in the flush paths into touch_{wq|work}_lockdep_map(). The work->lockdep_map touching is moved from __flush_work() to its callee - start_flush_work(). This brings it closer to the wq counterpart and will allow testing the associated wq's flags which will be needed to support BH workqueues. This is not expected to cause any functional changes.

---
 kernel/workqueue.c | 51 ++++++++++++++++++++++++++++++----------------
 1 file changed, 34 insertions(+), 17 deletions(-)
kernel/workqueue.c +34 -17
index 9221a4c57ae1..3f2081bd05a4 100644
@@ -2954,6 +2954,7 @@ __acquires(&pool->lock)
 	struct pool_workqueue *pwq = get_work_pwq(work);
 	struct worker_pool *pool = worker->pool;
 	unsigned long work_data;
+	int lockdep_start_depth, rcu_start_depth;
 #ifdef CONFIG_LOCKDEP
 	/*
 	 * It is permissible to free the struct work_struct from
@@ -3016,6 +3017,8 @@ __acquires(&pool->lock)
 	pwq->stats[PWQ_STAT_STARTED]++;
 	raw_spin_unlock_irq(&pool->lock);
 
+	rcu_start_depth = rcu_preempt_depth();
+	lockdep_start_depth = lockdep_depth(current);
 	lock_map_acquire(&pwq->wq->lockdep_map);
 	lock_map_acquire(&lockdep_map);
 	/*
@@ -3051,12 +3054,15 @@ __acquires(&pool->lock)
 	lock_map_release(&lockdep_map);
 	lock_map_release(&pwq->wq->lockdep_map);
 
-	if (unlikely(in_atomic() || lockdep_depth(current) > 0 ||
-		     rcu_preempt_depth() > 0)) {
-		pr_err("BUG: workqueue leaked lock or atomic: %s/0x%08x/%d/%d\n"
-		       "     last function: %ps\n",
-		       current->comm, preempt_count(), rcu_preempt_depth(),
-		       task_pid_nr(current), worker->current_func);
+	if (unlikely((worker->task && in_atomic()) ||
+		     lockdep_depth(current) != lockdep_start_depth ||
+		     rcu_preempt_depth() != rcu_start_depth)) {
+		pr_err("BUG: workqueue leaked atomic, lock or RCU: %s[%d]\n"
+		       "     preempt=0x%08x lock=%d->%d RCU=%d->%d workfn=%ps\n",
+		       current->comm, task_pid_nr(current), preempt_count(),
+		       lockdep_start_depth, lockdep_depth(current),
+		       rcu_start_depth, rcu_preempt_depth(),
+		       worker->current_func);
 		debug_show_held_locks(current);
 		dump_stack();
 	}
@@ -3538,6 +3544,19 @@ static bool flush_workqueue_prep_pwqs(struct workqueue_struct *wq,
 	return wait;
 }
 
+static void touch_wq_lockdep_map(struct workqueue_struct *wq)
+{
+	lock_map_acquire(&wq->lockdep_map);
+	lock_map_release(&wq->lockdep_map);
+}
+
+static void touch_work_lockdep_map(struct work_struct *work,
+				   struct workqueue_struct *wq)
+{
+	lock_map_acquire(&work->lockdep_map);
+	lock_map_release(&work->lockdep_map);
+}
+
 /**
  * __flush_workqueue - ensure that any scheduled work has run to completion.
  * @wq: workqueue to flush
@@ -3557,8 +3576,7 @@ void __flush_workqueue(struct workqueue_struct *wq)
 	if (WARN_ON(!wq_online))
 		return;
 
-	lock_map_acquire(&wq->lockdep_map);
-	lock_map_release(&wq->lockdep_map);
+	touch_wq_lockdep_map(wq);
 
 	mutex_lock(&wq->mutex);
 
@@ -3757,6 +3775,7 @@ static bool start_flush_work(struct work_struct *work, struct wq_barrier *barr,
 	struct worker *worker = NULL;
 	struct worker_pool *pool;
 	struct pool_workqueue *pwq;
+	struct workqueue_struct *wq;
 
 	might_sleep();
 
@@ -3780,11 +3799,14 @@ static bool start_flush_work(struct work_struct *work, struct wq_barrier *barr,
 		pwq = worker->current_pwq;
 	}
 
-	check_flush_dependency(pwq->wq, work);
+	wq = pwq->wq;
+	check_flush_dependency(wq, work);
 
 	insert_wq_barrier(pwq, barr, work, worker);
 	raw_spin_unlock_irq(&pool->lock);
 
+	touch_work_lockdep_map(work, wq);
+
 	/*
 	 * Force a lock recursion deadlock when using flush_work() inside a
 	 * single-threaded or rescuer equipped workqueue.
@@ -3794,11 +3816,9 @@ static bool start_flush_work(struct work_struct *work, struct wq_barrier *barr,
 	 * workqueues the deadlock happens when the rescuer stalls, blocking
 	 * forward progress.
 	 */
-	if (!from_cancel &&
-	    (pwq->wq->saved_max_active == 1 || pwq->wq->rescuer)) {
-		lock_map_acquire(&pwq->wq->lockdep_map);
-		lock_map_release(&pwq->wq->lockdep_map);
-	}
+	if (!from_cancel && (wq->saved_max_active == 1 || wq->rescuer))
+		touch_wq_lockdep_map(wq);
+
 	rcu_read_unlock();
 	return true;
 already_gone:
@@ -3817,9 +3837,6 @@ static bool __flush_work(struct work_struct *work, bool from_cancel)
 	if (WARN_ON(!work->func))
 		return false;
 
-	lock_map_acquire(&work->lockdep_map);
-	lock_map_release(&work->lockdep_map);
-
 	if (start_flush_work(work, &barr, from_cancel)) {
 		wait_for_completion(&barr.done);
 		destroy_work_on_stack(&barr.work);
Signature · 1 line
2.43.0
author maintainer
#2 lore raw

PATCH 2/8 workqueue: Factor out init_cpu_worker_pool()

In reply to Tejun Heo

Factor out init_cpu_worker_pool() from workqueue_init_early(). This is pure reorganization in preparation of BH workqueue support.

---
 kernel/workqueue.c | 32 ++++++++++++++++++--------------
 1 file changed, 18 insertions(+), 14 deletions(-)
kernel/workqueue.c +18 -14
index 3f2081bd05a4..f93554e479c4 100644
@@ -7135,6 +7135,22 @@ static void __init restrict_unbound_cpumask(const char *name, const struct cpuma
 	cpumask_and(wq_unbound_cpumask, wq_unbound_cpumask, mask);
 }
 
+static void __init init_cpu_worker_pool(struct worker_pool *pool, int cpu, int nice)
+{
+	BUG_ON(init_worker_pool(pool));
+	pool->cpu = cpu;
+	cpumask_copy(pool->attrs->cpumask, cpumask_of(cpu));
+	cpumask_copy(pool->attrs->__pod_cpumask, cpumask_of(cpu));
+	pool->attrs->nice = nice;
+	pool->attrs->affn_strict = true;
+	pool->node = cpu_to_node(cpu);
+
+	/* alloc pool ID */
+	mutex_lock(&wq_pool_mutex);
+	BUG_ON(worker_pool_assign_id(pool));
+	mutex_unlock(&wq_pool_mutex);
+}
+
 /**
  * workqueue_init_early - early init for workqueue subsystem
  *
@@ -7195,20 +7211,8 @@ void __init workqueue_init_early(void)
 		struct worker_pool *pool;
 
 		i = 0;
-		for_each_cpu_worker_pool(pool, cpu) {
-			BUG_ON(init_worker_pool(pool));
-			pool->cpu = cpu;
-			cpumask_copy(pool->attrs->cpumask, cpumask_of(cpu));
-			cpumask_copy(pool->attrs->__pod_cpumask, cpumask_of(cpu));
-			pool->attrs->nice = std_nice[i++];
-			pool->attrs->affn_strict = true;
-			pool->node = cpu_to_node(cpu);
-
-			/* alloc pool ID */
-			mutex_lock(&wq_pool_mutex);
-			BUG_ON(worker_pool_assign_id(pool));
-			mutex_unlock(&wq_pool_mutex);
-		}
+		for_each_cpu_worker_pool(pool, cpu)
+			init_cpu_worker_pool(pool, cpu, std_nice[i++]);
 	}
 
 	/* create default unbound and ordered wq attrs */
Signature · 1 line
2.43.0
author maintainer
#3 lore raw

PATCH 3/8 workqueue: Implement BH workqueues to eventually replace tasklets

In reply to Tejun Heo

The only generic interface to execute asynchronously in the BH context is tasklet; however, it's marked deprecated and has some design flaws such as the execution code accessing the tasklet item after the execution is complete which can lead to subtle use-after-free in certain usage scenarios and less-developed flush and cancel mechanisms.

This patch implements BH workqueues which share the same semantics and features of regular workqueues but execute their work items in the softirq context. As there is always only one BH execution context per CPU, none of the concurrency management mechanisms applies and a BH workqueue can be thought of as a convenience wrapper around softirq.

Except for the inability to sleep while executing and lack of max_active adjustments, BH workqueues and work items should behave the same as regular workqueues and work items.

Currently, the execution is hooked to tasklet[_hi]. However, the goal is to convert all tasklet users over to BH workqueues. Once the conversion is complete, tasklet can be removed and BH workqueues can directly take over the tasklet softirqs.

system_bh[_highpri]_wq are added. As queue-wide flushing doesn't exist in tasklet, all existing tasklet users should be able to use the system BH workqueues without creating their own.

---
 Documentation/core-api/workqueue.rst |  29 +++-
 include/linux/workqueue.h            |   9 ++
 kernel/workqueue.c                   | 231 ++++++++++++++++++++++-----
 kernel/workqueue_internal.h          |   3 +
 tools/workqueue/wq_dump.py           |  11 +-
 5 files changed, 237 insertions(+), 46 deletions(-)
Documentation/core-api/workqueue.rst +24 -5
index 33c4539155d9..2d6af6c4665c 100644
@@ -77,10 +77,12 @@ wants a function to be executed asynchronously it has to set up a work
 item pointing to that function and queue that work item on a
 workqueue.
 
-Special purpose threads, called worker threads, execute the functions
-off of the queue, one after the other.  If no work is queued, the
-worker threads become idle.  These worker threads are managed in so
-called worker-pools.
+A work item can be executed in either a thread or the BH (softirq) context.
+
+For threaded workqueues, special purpose threads, called [k]workers, execute
+the functions off of the queue, one after the other. If no work is queued,
+the worker threads become idle. These worker threads are managed in
+worker-pools.
 
 The cmwq design differentiates between the user-facing workqueues that
 subsystems and drivers queue work items on and the backend mechanism
@@ -91,6 +93,12 @@ for high priority ones, for each possible CPU and some extra
 worker-pools to serve work items queued on unbound workqueues - the
 number of these backing pools is dynamic.
 
+BH workqueues use the same framework. However, as there can only be one
+concurrent execution context, there's no need to worry about concurrency.
+Each per-CPU BH worker pool contains only one pseudo worker which represents
+the BH execution context. A BH workqueue can be considered a convenience
+interface to softirq.
+
 Subsystems and drivers can create and queue work items through special
 workqueue API functions as they see fit. They can influence some
 aspects of the way the work items are executed by setting flags on the
@@ -106,7 +114,7 @@ unless specifically overridden, a work item of a bound workqueue will
 be queued on the worklist of either normal or highpri worker-pool that
 is associated to the CPU the issuer is running on.
 
-For any worker pool implementation, managing the concurrency level
+For any thread pool implementation, managing the concurrency level
 (how many execution contexts are active) is an important issue.  cmwq
 tries to keep the concurrency at a minimal but sufficient level.
 Minimal to save resources and sufficient in that the system is used at
@@ -164,6 +172,17 @@ resources, scheduled and executed.
 ``flags``
 ---------
 
+``WQ_BH``
+  BH workqueues can be considered a convenience interface to softirq. BH
+  workqueues are always per-CPU and all BH work items are executed in the
+  queueing CPU's softirq context in the queueing order.
+
+  All BH workqueues must have 0 ``max_active`` and ``WQ_HIGHPRI`` is the
+  only allowed additional flag.
+
+  BH work items cannot sleep. All other features such as delayed queueing,
+  flushing and canceling are supported.
+
 ``WQ_UNBOUND``
   Work items queued to an unbound wq are served by the special
   worker-pools which host workers which are not bound to any
include/linux/workqueue.h +9 -0
index 232baea90a1d..3ac044691246 100644
@@ -353,6 +353,7 @@ static inline unsigned int work_static(struct work_struct *work) { return 0; }
  * Documentation/core-api/workqueue.rst.
  */
 enum wq_flags {
+	WQ_BH			= 1 << 0, /* execute in bottom half (softirq) context */
 	WQ_UNBOUND		= 1 << 1, /* not bound to any cpu */
 	WQ_FREEZABLE		= 1 << 2, /* freeze during suspend */
 	WQ_MEM_RECLAIM		= 1 << 3, /* may be used for memory reclaim */
@@ -392,6 +393,9 @@ enum wq_flags {
 	__WQ_ORDERED		= 1 << 17, /* internal: workqueue is ordered */
 	__WQ_LEGACY		= 1 << 18, /* internal: create*_workqueue() */
 	__WQ_ORDERED_EXPLICIT	= 1 << 19, /* internal: alloc_ordered_workqueue() */
+
+	/* BH wq only allows the following flags */
+	__WQ_BH_ALLOWS		= WQ_BH | WQ_HIGHPRI,
 };
 
 enum wq_consts {
@@ -434,6 +438,9 @@ enum wq_consts {
  * they are same as their non-power-efficient counterparts - e.g.
  * system_power_efficient_wq is identical to system_wq if
  * 'wq_power_efficient' is disabled.  See WQ_POWER_EFFICIENT for more info.
+ *
+ * system_bh[_highpri]_wq are convenience interface to softirq. BH work items
+ * are executed in the queueing CPU's BH context in the queueing order.
  */
 extern struct workqueue_struct *system_wq;
 extern struct workqueue_struct *system_highpri_wq;
@@ -442,6 +449,8 @@ extern struct workqueue_struct *system_unbound_wq;
 extern struct workqueue_struct *system_freezable_wq;
 extern struct workqueue_struct *system_power_efficient_wq;
 extern struct workqueue_struct *system_freezable_power_efficient_wq;
+extern struct workqueue_struct *system_bh_wq;
+extern struct workqueue_struct *system_bh_highpri_wq;
 
 /**
  * alloc_workqueue - allocate a workqueue
kernel/workqueue.c +192 -39

403 lines not on this page · raw message

kernel/workqueue_internal.h +3 -0
index f6275944ada7..8da306b76ec9 100644
@@ -10,6 +10,7 @@
 
 #include <linux/workqueue.h>
 #include <linux/kthread.h>
+#include <linux/interrupt.h>
 #include <linux/preempt.h>
 
 struct worker_pool;
@@ -42,6 +43,8 @@ struct worker {
 	struct list_head	scheduled;	/* L: scheduled works */
 
 	struct task_struct	*task;		/* I: worker task */
+	struct tasklet_struct	bh_tasklet;	/* I: tasklet for bh pool */
+
 	struct worker_pool	*pool;		/* A: the associated pool */
 						/* L: for rescuers */
 	struct list_head	node;		/* A: anchored at pool->workers */
tools/workqueue/wq_dump.py +9 -2
index bd381511bd9a..d29b918306b4 100644
@@ -79,7 +79,9 @@ args = parser.parse_args()
 wq_type_len = 9
 
 def wq_type_str(wq):
-    if wq.flags & WQ_UNBOUND:
+    if wq.flags & WQ_BH:
+        return f'{"bh":{wq_type_len}}'
+    elif wq.flags & WQ_UNBOUND:
         if wq.flags & WQ_ORDERED:
             return f'{"ordered":{wq_type_len}}'
         else:
@@ -97,6 +99,7 @@ wq_pod_types            = prog['wq_pod_types']
 wq_affn_dfl             = prog['wq_affn_dfl']
 wq_affn_names           = prog['wq_affn_names']
 
+WQ_BH                   = prog['WQ_BH']
 WQ_UNBOUND              = prog['WQ_UNBOUND']
 WQ_ORDERED              = prog['__WQ_ORDERED']
 WQ_MEM_RECLAIM          = prog['WQ_MEM_RECLAIM']
@@ -107,6 +110,8 @@ WQ_AFFN_CACHE           = prog['WQ_AFFN_CACHE']
 WQ_AFFN_NUMA            = prog['WQ_AFFN_NUMA']
 WQ_AFFN_SYSTEM          = prog['WQ_AFFN_SYSTEM']
 
+POOL_BH                 = prog['POOL_BH']
+
 WQ_NAME_LEN             = prog['WQ_NAME_LEN'].value_()
 cpumask_str_len         = len(cpumask_str(wq_unbound_cpumask))
 
@@ -151,10 +156,12 @@ max_ref_len = 0
 
 for pi, pool in idr_for_each(worker_pool_idr):
     pool = drgn.Object(prog, 'struct worker_pool', address=pool)
-    print(f'pool[{pi:0{max_pool_id_len}}] ref={pool.refcnt.value_():{max_ref_len}} nice={pool.attrs.nice.value_():3} ', end='')
+    print(f'pool[{pi:0{max_pool_id_len}}] flags=0x{pool.flags.value_():02x} ref={pool.refcnt.value_():{max_ref_len}} nice={pool.attrs.nice.value_():3} ', end='')
     print(f'idle/workers={pool.nr_idle.value_():3}/{pool.nr_workers.value_():3} ', end='')
     if pool.cpu >= 0:
         print(f'cpu={pool.cpu.value_():3}', end='')
+        if pool.flags & POOL_BH:
+            print(' bh', end='')
     else:
         print(f'cpus={cpumask_str(pool.attrs.cpumask)}', end='')
         print(f' pod_cpus={cpumask_str(pool.attrs.__pod_cpumask)}', end='')
Signature · 1 line
2.43.0
#4 lore raw

Re: [PATCH 3/8] workqueue: Implement BH workqueues to eventually replace tasklets

Quoted 4 lines · Tejun Heo

This patch implements BH workqueues which share the same semantics and features of regular workqueues but execute their work items in the softirq context.

Thanks for doing this. Honestly, while I felt this was a natural thing to do and would clean things up, every time I look at the workqueue code I just shudder and go "I'm sure Tejun can work this out".

Patches look fine to me - I'd love to actually have the dm-crypt people (and networking people, for that matter) verify that there are no performance gotchas from the slightly heavier (but more generic) workqueue interfaces.

Linus

reviewer
#5 lore raw

Re: [PATCH 3/8] workqueue: Implement BH workqueues to eventually replace tasklets

In reply to Tejun Heo

Hello, Tejun

kernel/workqueue.c static bool kick_pool(struct worker_pool *pool) · line 1220

Quoted 11 lines · Tejun Heo
kernel/workqueue.c +7 -0
@@ -1184,6 +1211,14 @@ static bool kick_pool(struct worker_pool *pool)
        if (!need_more_worker(pool) || !worker)
                return false;
 
+       if (pool->flags & POOL_BH) {
+               if (pool->attrs->nice == HIGHPRI_NICE_LEVEL)
+                       tasklet_hi_schedule(&worker->bh_tasklet);
+               else
+                       tasklet_schedule(&worker->bh_tasklet);
+               return true;
+       }

I think it is more straightforward to call bh_worker_taskletfn[_hi]() in tasklet_action() and tasklet_hi_action() rather than add a worker->bh_tasklet.

raise_softirq_irqoff() can be used here (kick_pool()) instead.

As the changelog said, once the conversion is complete, tasklet can be removed and BH workqueues can directly take over the tasklet softirqs, in which case, then, bh_worker_taskletfn[_hi]() can directly take over the tasklet_action() and tasklet_hi_action().

Quoted 10 lines
kernel/workqueue.c +1 -2
+
        p = worker->task;
 
 #ifdef CONFIG_SMP
@@ -1663,8 +1698,16 @@ static bool pwq_tryinc_nr_active(struct pool_workqueue *pwq, bool fill)
        lockdep_assert_held(&pool->lock);
 
        if (!nna) {
-               /* per-cpu workqueue, pwq->nr_active is sufficient */
-               obtained = pwq->nr_active < READ_ONCE(wq->max_active);

kernel/workqueue.c static bool pwq_tryinc_nr_active(struct pool_workqueue *pwq, bool fill) · line 1710

Quoted 10 lines
kernel/workqueue.c +10 -0
+               /*
+                * BH workqueues always share a single execution context per CPU
+                * and don't impose any max_active limit, so tryinc always
+                * succeeds. For a per-cpu workqueue, checking pwq->nr_active is
+                * sufficient.
+                */
+               if (wq->flags & WQ_BH)
+                       obtained = true;
+               else
+                       obtained = pwq->nr_active < READ_ONCE(wq->max_active);

I think wq->max_active can be forced to be UINT_MAX or ULONG_MAX in the max_active management code to avoid a branch here.

Quoted 44 lines
kernel/workqueue.c +20 -16
                goto out;
        }
 
@@ -2599,27 +2642,31 @@ static struct worker *create_worker(struct worker_pool *pool)
 
        worker->id = id;
 
-       if (pool->cpu >= 0)
-               snprintf(id_buf, sizeof(id_buf), "%d:%d%s", pool->cpu, id,
-                        pool->attrs->nice < 0  ? "H" : "");
-       else
-               snprintf(id_buf, sizeof(id_buf), "u%d:%d", pool->id, id);
-
-       worker->task = kthread_create_on_node(worker_thread, worker, pool->node,
-                                             "kworker/%s", id_buf);
-       if (IS_ERR(worker->task)) {
-               if (PTR_ERR(worker->task) == -EINTR) {
-                       pr_err("workqueue: Interrupted when creating a worker thread \"kworker/%s\"\n",
-                              id_buf);
-               } else {
-                       pr_err_once("workqueue: Failed to create a worker thread: %pe",
-                                   worker->task);
+       if (pool->flags & POOL_BH) {
+               tasklet_setup(&worker->bh_tasklet, bh_worker_taskletfn);
+       } else {
+               if (pool->cpu >= 0)
+                       snprintf(id_buf, sizeof(id_buf), "%d:%d%s", pool->cpu, id,
+                                pool->attrs->nice < 0  ? "H" : "");
+               else
+                       snprintf(id_buf, sizeof(id_buf), "u%d:%d", pool->id, id);
+
+               worker->task = kthread_create_on_node(worker_thread, worker,
+                                       pool->node, "kworker/%s", id_buf);
+               if (IS_ERR(worker->task)) {
+                       if (PTR_ERR(worker->task) == -EINTR) {
+                               pr_err("workqueue: Interrupted when creating a worker thread \"kworker/%s\"\n",
+                                      id_buf);
+                       } else {
+                               pr_err_once("workqueue: Failed to create a worker thread: %pe",
+                                           worker->task);
+                       }
+                       goto fail;
                }
-               goto fail;

kernel/workqueue.c static struct worker *create_worker(struct worker_pool *pool) · line 2672

Quoted 10 lines
kernel/workqueue.c +3 -3
-       }
 
-       set_user_nice(worker->task, pool->attrs->nice);
-       kthread_bind_mask(worker->task, pool_allowed_cpus(pool));
+               set_user_nice(worker->task, pool->attrs->nice);
+               kthread_bind_mask(worker->task, pool_allowed_cpus(pool));
+       }
 
        /* successful, attach the worker to the pool */
        worker_attach_to_pool(worker, pool);

worker_attach_to_pool() and worker_detach_from_pool also access to worker->task with kthread_set_per_cpu() and luckily to_kthread() checks the NULL pointer for it.

IMO, it is better to avoid calling kthread_set_per_cpu() for bh workers.

kernel/workqueue.c static void pr_cont_pool_info(struct worker_pool *pool) · line 5739

Quoted 11 lines
kernel/workqueue.c +6 -1
@@ -5605,7 +5731,12 @@ static void pr_cont_pool_info(struct worker_pool *pool)
        pr_cont(" cpus=%*pbl", nr_cpumask_bits, pool->attrs->cpumask);
        if (pool->node != NUMA_NO_NODE)
                pr_cont(" node=%d", pool->node);
-       pr_cont(" flags=0x%x nice=%d", pool->flags, pool->attrs->nice);
+       pr_cont(" flags=0x%x", pool->flags);
+       if (pool->flags & POOL_BH)
+               pr_cont(" bh%s",
+                       pool->attrs->nice == HIGHPRI_NICE_LEVEL ? "-hi" : "");
+       else
+               pr_cont(" nice=%d", pool->attrs->nice);

There are also some "worker->task" in show_pwq(), show_one_worker_pool(), and show_cpu_pool_hog() needing taking care of.

Thanks
Lai

author maintainer
#6 lore raw

Re: [PATCH 3/8] workqueue: Implement BH workqueues to eventually replace tasklets

Hello, Lai.

Quoted 17 lines · Lai Jiangshan

kernel/workqueue.c static bool kick_pool(struct worker_pool *pool) · line 1220

Quoted 11 lines · Tejun Heo
kernel/workqueue.c +7 -0
@@ -1184,6 +1211,14 @@ static bool kick_pool(struct worker_pool *pool)
        if (!need_more_worker(pool) || !worker)
                return false;
 
+       if (pool->flags & POOL_BH) {
+               if (pool->attrs->nice == HIGHPRI_NICE_LEVEL)
+                       tasklet_hi_schedule(&worker->bh_tasklet);
+               else
+                       tasklet_schedule(&worker->bh_tasklet);
+               return true;
+       }

I think it is more straightforward to call bh_worker_taskletfn[_hi]() in tasklet_action() and tasklet_hi_action() rather than add a worker->bh_tasklet.

Quoted 6 lines

raise_softirq_irqoff() can be used here (kick_pool()) instead.

As the changelog said, once the conversion is complete, tasklet can be removed and BH workqueues can directly take over the tasklet softirqs, in which case, then, bh_worker_taskletfn[_hi]() can directly take over the tasklet_action() and tasklet_hi_action().

Hmmm.... maybe. Yeah, that'd also make it a tiny bit cheaper for hot paths. Lemme see how that looks.

Quoted 2 lines

I think wq->max_active can be forced to be UINT_MAX or ULONG_MAX in the max_active management code to avoid a branch here.

Good point. Will update.

Quoted 5 lines

worker_attach_to_pool() and worker_detach_from_pool also access to worker->task with kthread_set_per_cpu() and luckily to_kthread() checks the NULL pointer for it.

IMO, it is better to avoid calling kthread_set_per_cpu() for bh workers.

Note that BH worker pools are always DISASSOCIATED, so worker_attach_to_pool() shouldn't call kthread_set_per_cpu(). Also, BH workers and worker pools are never destroyed, so worker_detach_from_pool() shouldn't be called either. I'll add WARN_ONs to clarify.

Quoted 6 lines
Quoted 6 lines
kernel/workqueue.c +1 -1
@@ -5605,7 +5731,12 @@ static void pr_cont_pool_info(struct worker_pool *pool)
        pr_cont(" cpus=%*pbl", nr_cpumask_bits, pool->attrs->cpumask);
        if (pool->node != NUMA_NO_NODE)
                pr_cont(" node=%d", pool->node);
-       pr_cont(" flags=0x%x nice=%d", pool->flags, pool->attrs->nice);
+       pr_cont(" flags=0x%x", pool->flags);
Quoted 8 lines

kernel/workqueue.c static void pr_cont_pool_info(struct worker_pool *pool) · line 5739

Quoted 5 lines
kernel/workqueue.c +5 -0
+       if (pool->flags & POOL_BH)
+               pr_cont(" bh%s",
+                       pool->attrs->nice == HIGHPRI_NICE_LEVEL ? "-hi" : "");
+       else
+               pr_cont(" nice=%d", pool->attrs->nice);

There are also some "worker->task" in show_pwq(), show_one_worker_pool(), and show_cpu_pool_hog() needing taking care of.

Ah, right, I'll hunt them down.

Thanks.

Signature · 1 line
tejun
author maintainer
#7 lore raw

PATCH v2 3/8 workqueue: Implement BH workqueues to eventually replace tasklets

In reply to Tejun Heo

The only generic interface to execute asynchronously in the BH context is tasklet; however, it's marked deprecated and has some design flaws such as the execution code accessing the tasklet item after the execution is complete which can lead to subtle use-after-free in certain usage scenarios and less-developed flush and cancel mechanisms.

This patch implements BH workqueues which share the same semantics and features of regular workqueues but execute their work items in the softirq context. As there is always only one BH execution context per CPU, none of the concurrency management mechanisms applies and a BH workqueue can be thought of as a convenience wrapper around softirq.

Except for the inability to sleep while executing and lack of max_active adjustments, BH workqueues and work items should behave the same as regular workqueues and work items.

Currently, the execution is hooked to tasklet[_hi]. However, the goal is to convert all tasklet users over to BH workqueues. Once the conversion is complete, tasklet can be removed and BH workqueues can directly take over the tasklet softirqs.

system_bh[_highpri]_wq are added. As queue-wide flushing doesn't exist in tasklet, all existing tasklet users should be able to use the system BH workqueues without creating their own workqueues.

v2: - Instead of using tasklets, hook directly into its softirq action
functions - tasklet[_hi]_action(). This is slightly cheaper and closer to the eventual code structure we want to arrive at. Suggested by Lai.

- Lai also pointed out several places which need NULL worker->task handling or can use clarification. Updated.

---
Hello,

The git branch is also updated. Once Lai acks, I'll land the workqueue part in wq/for-6.9 and create a separate branch for conversion. I'll apply the safe / acked ones only.

Thanks.

 Documentation/core-api/workqueue.rst |  29 ++-
 include/linux/workqueue.h            |  11 +
 kernel/softirq.c                     |   3 +
 kernel/workqueue.c                   | 290 ++++++++++++++++++++++-----
 tools/workqueue/wq_dump.py           |  11 +-
 5 files changed, 284 insertions(+), 60 deletions(-)
Documentation/core-api/workqueue.rst +24 -5
index 33c4539155d9..2d6af6c4665c 100644
@@ -77,10 +77,12 @@ wants a function to be executed asynchronously it has to set up a work
 item pointing to that function and queue that work item on a
 workqueue.
 
-Special purpose threads, called worker threads, execute the functions
-off of the queue, one after the other.  If no work is queued, the
-worker threads become idle.  These worker threads are managed in so
-called worker-pools.
+A work item can be executed in either a thread or the BH (softirq) context.
+
+For threaded workqueues, special purpose threads, called [k]workers, execute
+the functions off of the queue, one after the other. If no work is queued,
+the worker threads become idle. These worker threads are managed in
+worker-pools.
 
 The cmwq design differentiates between the user-facing workqueues that
 subsystems and drivers queue work items on and the backend mechanism
@@ -91,6 +93,12 @@ for high priority ones, for each possible CPU and some extra
 worker-pools to serve work items queued on unbound workqueues - the
 number of these backing pools is dynamic.
 
+BH workqueues use the same framework. However, as there can only be one
+concurrent execution context, there's no need to worry about concurrency.
+Each per-CPU BH worker pool contains only one pseudo worker which represents
+the BH execution context. A BH workqueue can be considered a convenience
+interface to softirq.
+
 Subsystems and drivers can create and queue work items through special
 workqueue API functions as they see fit. They can influence some
 aspects of the way the work items are executed by setting flags on the
@@ -106,7 +114,7 @@ unless specifically overridden, a work item of a bound workqueue will
 be queued on the worklist of either normal or highpri worker-pool that
 is associated to the CPU the issuer is running on.
 
-For any worker pool implementation, managing the concurrency level
+For any thread pool implementation, managing the concurrency level
 (how many execution contexts are active) is an important issue.  cmwq
 tries to keep the concurrency at a minimal but sufficient level.
 Minimal to save resources and sufficient in that the system is used at
@@ -164,6 +172,17 @@ resources, scheduled and executed.
 ``flags``
 ---------
 
+``WQ_BH``
+  BH workqueues can be considered a convenience interface to softirq. BH
+  workqueues are always per-CPU and all BH work items are executed in the
+  queueing CPU's softirq context in the queueing order.
+
+  All BH workqueues must have 0 ``max_active`` and ``WQ_HIGHPRI`` is the
+  only allowed additional flag.
+
+  BH work items cannot sleep. All other features such as delayed queueing,
+  flushing and canceling are supported.
+
 ``WQ_UNBOUND``
   Work items queued to an unbound wq are served by the special
   worker-pools which host workers which are not bound to any
include/linux/workqueue.h +11 -0
index 232baea90a1d..283d7891b4c4 100644
@@ -353,6 +353,7 @@ static inline unsigned int work_static(struct work_struct *work) { return 0; }
  * Documentation/core-api/workqueue.rst.
  */
 enum wq_flags {
+	WQ_BH			= 1 << 0, /* execute in bottom half (softirq) context */
 	WQ_UNBOUND		= 1 << 1, /* not bound to any cpu */
 	WQ_FREEZABLE		= 1 << 2, /* freeze during suspend */
 	WQ_MEM_RECLAIM		= 1 << 3, /* may be used for memory reclaim */
@@ -392,6 +393,9 @@ enum wq_flags {
 	__WQ_ORDERED		= 1 << 17, /* internal: workqueue is ordered */
 	__WQ_LEGACY		= 1 << 18, /* internal: create*_workqueue() */
 	__WQ_ORDERED_EXPLICIT	= 1 << 19, /* internal: alloc_ordered_workqueue() */
+
+	/* BH wq only allows the following flags */
+	__WQ_BH_ALLOWS		= WQ_BH | WQ_HIGHPRI,
 };
 
 enum wq_consts {
@@ -434,6 +438,9 @@ enum wq_consts {
  * they are same as their non-power-efficient counterparts - e.g.
  * system_power_efficient_wq is identical to system_wq if
  * 'wq_power_efficient' is disabled.  See WQ_POWER_EFFICIENT for more info.
+ *
+ * system_bh[_highpri]_wq are convenience interface to softirq. BH work items
+ * are executed in the queueing CPU's BH context in the queueing order.
  */
 extern struct workqueue_struct *system_wq;
 extern struct workqueue_struct *system_highpri_wq;
@@ -442,6 +449,10 @@ extern struct workqueue_struct *system_unbound_wq;
 extern struct workqueue_struct *system_freezable_wq;
 extern struct workqueue_struct *system_power_efficient_wq;
 extern struct workqueue_struct *system_freezable_power_efficient_wq;
+extern struct workqueue_struct *system_bh_wq;
+extern struct workqueue_struct *system_bh_highpri_wq;
+
+void workqueue_softirq_action(bool highpri);
 
 /**
  * alloc_workqueue - allocate a workqueue
kernel/softirq.c +3 -0
index 210cf5f8d92c..547d282548a8 100644
@@ -27,6 +27,7 @@
 #include <linux/tick.h>
 #include <linux/irq.h>
 #include <linux/wait_bit.h>
+#include <linux/workqueue.h>
 
 #include <asm/softirq_stack.h>
 
@@ -802,11 +803,13 @@ static void tasklet_action_common(struct softirq_action *a,
 
 static __latent_entropy void tasklet_action(struct softirq_action *a)
 {
+	workqueue_softirq_action(false);
 	tasklet_action_common(a, this_cpu_ptr(&tasklet_vec), TASKLET_SOFTIRQ);
 }
 
 static __latent_entropy void tasklet_hi_action(struct softirq_action *a)
 {
+	workqueue_softirq_action(true);
 	tasklet_action_common(a, this_cpu_ptr(&tasklet_hi_vec), HI_SOFTIRQ);
 }
 
kernel/workqueue.c +237 -53

503 lines not on this page · raw message

tools/workqueue/wq_dump.py +9 -2
index bd381511bd9a..d29b918306b4 100644
@@ -79,7 +79,9 @@ args = parser.parse_args()
 wq_type_len = 9
 
 def wq_type_str(wq):
-    if wq.flags & WQ_UNBOUND:
+    if wq.flags & WQ_BH:
+        return f'{"bh":{wq_type_len}}'
+    elif wq.flags & WQ_UNBOUND:
         if wq.flags & WQ_ORDERED:
             return f'{"ordered":{wq_type_len}}'
         else:
@@ -97,6 +99,7 @@ wq_pod_types            = prog['wq_pod_types']
 wq_affn_dfl             = prog['wq_affn_dfl']
 wq_affn_names           = prog['wq_affn_names']
 
+WQ_BH                   = prog['WQ_BH']
 WQ_UNBOUND              = prog['WQ_UNBOUND']
 WQ_ORDERED              = prog['__WQ_ORDERED']
 WQ_MEM_RECLAIM          = prog['WQ_MEM_RECLAIM']
@@ -107,6 +110,8 @@ WQ_AFFN_CACHE           = prog['WQ_AFFN_CACHE']
 WQ_AFFN_NUMA            = prog['WQ_AFFN_NUMA']
 WQ_AFFN_SYSTEM          = prog['WQ_AFFN_SYSTEM']
 
+POOL_BH                 = prog['POOL_BH']
+
 WQ_NAME_LEN             = prog['WQ_NAME_LEN'].value_()
 cpumask_str_len         = len(cpumask_str(wq_unbound_cpumask))
 
@@ -151,10 +156,12 @@ max_ref_len = 0
 
 for pi, pool in idr_for_each(worker_pool_idr):
     pool = drgn.Object(prog, 'struct worker_pool', address=pool)
-    print(f'pool[{pi:0{max_pool_id_len}}] ref={pool.refcnt.value_():{max_ref_len}} nice={pool.attrs.nice.value_():3} ', end='')
+    print(f'pool[{pi:0{max_pool_id_len}}] flags=0x{pool.flags.value_():02x} ref={pool.refcnt.value_():{max_ref_len}} nice={pool.attrs.nice.value_():3} ', end='')
     print(f'idle/workers={pool.nr_idle.value_():3}/{pool.nr_workers.value_():3} ', end='')
     if pool.cpu >= 0:
         print(f'cpu={pool.cpu.value_():3}', end='')
+        if pool.flags & POOL_BH:
+            print(' bh', end='')
     else:
         print(f'cpus={cpumask_str(pool.attrs.cpumask)}', end='')
         print(f' pod_cpus={cpumask_str(pool.attrs.__pod_cpumask)}', end='')
Signature · 1 line
2.43.0
author maintainer
#9 lore raw

PATCH v3 3/8 workqueue: Implement BH workqueues to eventually replace tasklets

In reply to Tejun Heo

From 4cb1ef64609f9b0254184b2947824f4b46ccab22 Mon Sep 17 00:00:00 2001
From: Tejun Heo <[email protected]>
Date: Sun, 4 Feb 2024 11:28:06 -1000

The only generic interface to execute asynchronously in the BH context is tasklet; however, it's marked deprecated and has some design flaws such as the execution code accessing the tasklet item after the execution is complete which can lead to subtle use-after-free in certain usage scenarios and less-developed flush and cancel mechanisms.

This patch implements BH workqueues which share the same semantics and features of regular workqueues but execute their work items in the softirq context. As there is always only one BH execution context per CPU, none of the concurrency management mechanisms applies and a BH workqueue can be thought of as a convenience wrapper around softirq.

Except for the inability to sleep while executing and lack of max_active adjustments, BH workqueues and work items should behave the same as regular workqueues and work items.

Currently, the execution is hooked to tasklet[_hi]. However, the goal is to convert all tasklet users over to BH workqueues. Once the conversion is complete, tasklet can be removed and BH workqueues can directly take over the tasklet softirqs.

system_bh[_highpri]_wq are added. As queue-wide flushing doesn't exist in tasklet, all existing tasklet users should be able to use the system BH workqueues without creating their own workqueues.

v3: - Add missing interrupt.h include.

v2: - Instead of using tasklets, hook directly into its softirq action
functions - tasklet[_hi]_action(). This is slightly cheaper and closer to the eventual code structure we want to arrive at. Suggested by Lai.

- Lai also pointed out several places which need NULL worker->task handling or can use clarification. Updated.

---
 Documentation/core-api/workqueue.rst |  29 ++-
 include/linux/workqueue.h            |  11 +
 kernel/softirq.c                     |   3 +
 kernel/workqueue.c                   | 291 ++++++++++++++++++++++-----
 tools/workqueue/wq_dump.py           |  11 +-
 5 files changed, 285 insertions(+), 60 deletions(-)
Documentation/core-api/workqueue.rst +24 -5
index 33c4539155d9..2d6af6c4665c 100644
@@ -77,10 +77,12 @@ wants a function to be executed asynchronously it has to set up a work
 item pointing to that function and queue that work item on a
 workqueue.
 
-Special purpose threads, called worker threads, execute the functions
-off of the queue, one after the other.  If no work is queued, the
-worker threads become idle.  These worker threads are managed in so
-called worker-pools.
+A work item can be executed in either a thread or the BH (softirq) context.
+
+For threaded workqueues, special purpose threads, called [k]workers, execute
+the functions off of the queue, one after the other. If no work is queued,
+the worker threads become idle. These worker threads are managed in
+worker-pools.
 
 The cmwq design differentiates between the user-facing workqueues that
 subsystems and drivers queue work items on and the backend mechanism
@@ -91,6 +93,12 @@ for high priority ones, for each possible CPU and some extra
 worker-pools to serve work items queued on unbound workqueues - the
 number of these backing pools is dynamic.
 
+BH workqueues use the same framework. However, as there can only be one
+concurrent execution context, there's no need to worry about concurrency.
+Each per-CPU BH worker pool contains only one pseudo worker which represents
+the BH execution context. A BH workqueue can be considered a convenience
+interface to softirq.
+
 Subsystems and drivers can create and queue work items through special
 workqueue API functions as they see fit. They can influence some
 aspects of the way the work items are executed by setting flags on the
@@ -106,7 +114,7 @@ unless specifically overridden, a work item of a bound workqueue will
 be queued on the worklist of either normal or highpri worker-pool that
 is associated to the CPU the issuer is running on.
 
-For any worker pool implementation, managing the concurrency level
+For any thread pool implementation, managing the concurrency level
 (how many execution contexts are active) is an important issue.  cmwq
 tries to keep the concurrency at a minimal but sufficient level.
 Minimal to save resources and sufficient in that the system is used at
@@ -164,6 +172,17 @@ resources, scheduled and executed.
 ``flags``
 ---------
 
+``WQ_BH``
+  BH workqueues can be considered a convenience interface to softirq. BH
+  workqueues are always per-CPU and all BH work items are executed in the
+  queueing CPU's softirq context in the queueing order.
+
+  All BH workqueues must have 0 ``max_active`` and ``WQ_HIGHPRI`` is the
+  only allowed additional flag.
+
+  BH work items cannot sleep. All other features such as delayed queueing,
+  flushing and canceling are supported.
+
 ``WQ_UNBOUND``
   Work items queued to an unbound wq are served by the special
   worker-pools which host workers which are not bound to any
include/linux/workqueue.h +11 -0
index 232baea90a1d..283d7891b4c4 100644
@@ -353,6 +353,7 @@ static inline unsigned int work_static(struct work_struct *work) { return 0; }
  * Documentation/core-api/workqueue.rst.
  */
 enum wq_flags {
+	WQ_BH			= 1 << 0, /* execute in bottom half (softirq) context */
 	WQ_UNBOUND		= 1 << 1, /* not bound to any cpu */
 	WQ_FREEZABLE		= 1 << 2, /* freeze during suspend */
 	WQ_MEM_RECLAIM		= 1 << 3, /* may be used for memory reclaim */
@@ -392,6 +393,9 @@ enum wq_flags {
 	__WQ_ORDERED		= 1 << 17, /* internal: workqueue is ordered */
 	__WQ_LEGACY		= 1 << 18, /* internal: create*_workqueue() */
 	__WQ_ORDERED_EXPLICIT	= 1 << 19, /* internal: alloc_ordered_workqueue() */
+
+	/* BH wq only allows the following flags */
+	__WQ_BH_ALLOWS		= WQ_BH | WQ_HIGHPRI,
 };
 
 enum wq_consts {
@@ -434,6 +438,9 @@ enum wq_consts {
  * they are same as their non-power-efficient counterparts - e.g.
  * system_power_efficient_wq is identical to system_wq if
  * 'wq_power_efficient' is disabled.  See WQ_POWER_EFFICIENT for more info.
+ *
+ * system_bh[_highpri]_wq are convenience interface to softirq. BH work items
+ * are executed in the queueing CPU's BH context in the queueing order.
  */
 extern struct workqueue_struct *system_wq;
 extern struct workqueue_struct *system_highpri_wq;
@@ -442,6 +449,10 @@ extern struct workqueue_struct *system_unbound_wq;
 extern struct workqueue_struct *system_freezable_wq;
 extern struct workqueue_struct *system_power_efficient_wq;
 extern struct workqueue_struct *system_freezable_power_efficient_wq;
+extern struct workqueue_struct *system_bh_wq;
+extern struct workqueue_struct *system_bh_highpri_wq;
+
+void workqueue_softirq_action(bool highpri);
 
 /**
  * alloc_workqueue - allocate a workqueue
kernel/softirq.c +3 -0
index 210cf5f8d92c..547d282548a8 100644
@@ -27,6 +27,7 @@
 #include <linux/tick.h>
 #include <linux/irq.h>
 #include <linux/wait_bit.h>
+#include <linux/workqueue.h>
 
 #include <asm/softirq_stack.h>
 
@@ -802,11 +803,13 @@ static void tasklet_action_common(struct softirq_action *a,
 
 static __latent_entropy void tasklet_action(struct softirq_action *a)
 {
+	workqueue_softirq_action(false);
 	tasklet_action_common(a, this_cpu_ptr(&tasklet_vec), TASKLET_SOFTIRQ);
 }
 
 static __latent_entropy void tasklet_hi_action(struct softirq_action *a)
 {
+	workqueue_softirq_action(true);
 	tasklet_action_common(a, this_cpu_ptr(&tasklet_hi_vec), HI_SOFTIRQ);
 }
 
kernel/workqueue.c +238 -53

510 lines not on this page · raw message

tools/workqueue/wq_dump.py +9 -2
index bd381511bd9a..d29b918306b4 100644
@@ -79,7 +79,9 @@ args = parser.parse_args()
 wq_type_len = 9
 
 def wq_type_str(wq):
-    if wq.flags & WQ_UNBOUND:
+    if wq.flags & WQ_BH:
+        return f'{"bh":{wq_type_len}}'
+    elif wq.flags & WQ_UNBOUND:
         if wq.flags & WQ_ORDERED:
             return f'{"ordered":{wq_type_len}}'
         else:
@@ -97,6 +99,7 @@ wq_pod_types            = prog['wq_pod_types']
 wq_affn_dfl             = prog['wq_affn_dfl']
 wq_affn_names           = prog['wq_affn_names']
 
+WQ_BH                   = prog['WQ_BH']
 WQ_UNBOUND              = prog['WQ_UNBOUND']
 WQ_ORDERED              = prog['__WQ_ORDERED']
 WQ_MEM_RECLAIM          = prog['WQ_MEM_RECLAIM']
@@ -107,6 +110,8 @@ WQ_AFFN_CACHE           = prog['WQ_AFFN_CACHE']
 WQ_AFFN_NUMA            = prog['WQ_AFFN_NUMA']
 WQ_AFFN_SYSTEM          = prog['WQ_AFFN_SYSTEM']
 
+POOL_BH                 = prog['POOL_BH']
+
 WQ_NAME_LEN             = prog['WQ_NAME_LEN'].value_()
 cpumask_str_len         = len(cpumask_str(wq_unbound_cpumask))
 
@@ -151,10 +156,12 @@ max_ref_len = 0
 
 for pi, pool in idr_for_each(worker_pool_idr):
     pool = drgn.Object(prog, 'struct worker_pool', address=pool)
-    print(f'pool[{pi:0{max_pool_id_len}}] ref={pool.refcnt.value_():{max_ref_len}} nice={pool.attrs.nice.value_():3} ', end='')
+    print(f'pool[{pi:0{max_pool_id_len}}] flags=0x{pool.flags.value_():02x} ref={pool.refcnt.value_():{max_ref_len}} nice={pool.attrs.nice.value_():3} ', end='')
     print(f'idle/workers={pool.nr_idle.value_():3}/{pool.nr_workers.value_():3} ', end='')
     if pool.cpu >= 0:
         print(f'cpu={pool.cpu.value_():3}', end='')
+        if pool.flags & POOL_BH:
+            print(' bh', end='')
     else:
         print(f'cpus={cpumask_str(pool.attrs.cpumask)}', end='')
         print(f' pod_cpus={cpumask_str(pool.attrs.__pod_cpumask)}', end='')
Signature · 1 line
2.43.0
#10 lore raw

Re: [PATCH v3 3/8] workqueue: Implement BH workqueues to eventually replace tasklets

On Sun, 4 Feb 2024 11:28:06 -1000 Tejun Heo <[email protected]>

Quoted 25 lines
kernel/workqueue.c +25 -0
+static void bh_worker(struct worker *worker)
+{
+	struct worker_pool *pool = worker->pool;
+	int nr_restarts = BH_WORKER_RESTARTS;
+	unsigned long end = jiffies + BH_WORKER_JIFFIES;
+
+	raw_spin_lock_irq(&pool->lock);
+	worker_leave_idle(worker);
+
+	/*
+	 * This function follows the structure of worker_thread(). See there for
+	 * explanations on each step.
+	 */
+	if (!need_more_worker(pool))
+		goto done;
+
+	WARN_ON_ONCE(!list_empty(&worker->scheduled));
+	worker_clr_flags(worker, WORKER_PREP | WORKER_REBOUND);
+
+	do {
+		struct work_struct *work =
+			list_first_entry(&pool->worklist,
+					 struct work_struct, entry);
+
+		if (assign_work(work, worker, NULL))

kernel/workqueue.c static int rescuer_thread(void *__rescuer) · line 3451

Quoted 10 lines
kernel/workqueue.c +10 -0
+			process_scheduled_works(worker);
+	} while (keep_working(pool) &&
+		 --nr_restarts && time_before(jiffies, end));
+
+	worker_set_flags(worker, WORKER_PREP);
+done:
+	worker_enter_idle(worker);
+	kick_pool(pool);
+	raw_spin_unlock_irq(&pool->lock);
+}

I see no need to exec bh works for 2ms with irq disabled.

author maintainer
#11 lore raw

Re: [PATCH v3 3/8] workqueue: Implement BH workqueues to eventually replace tasklets

Quoted 1 line · Hillf Danton

I see no need to exec bh works for 2ms with irq disabled.

The irqs are enabled and locks are released while work items are executing.

Thanks.

Signature · 1 line
tejun
#12 lore raw

Re: [PATCH v3 3/8] workqueue: Implement BH workqueues to eventually replace tasklets

In reply to Tejun Heo
Quoted 208 lines · Tejun Heo
Quoted 1 line

From 4cb1ef64609f9b0254184b2947824f4b46ccab22 Mon Sep 17 00:00:00 2001

From: Tejun Heo <[email protected]>
Date: Sun, 4 Feb 2024 11:28:06 -1000

The only generic interface to execute asynchronously in the BH context is tasklet; however, it's marked deprecated and has some design flaws such as the execution code accessing the tasklet item after the execution is complete which can lead to subtle use-after-free in certain usage scenarios and less-developed flush and cancel mechanisms.

This patch implements BH workqueues which share the same semantics and features of regular workqueues but execute their work items in the softirq context. As there is always only one BH execution context per CPU, none of the concurrency management mechanisms applies and a BH workqueue can be thought of as a convenience wrapper around softirq.

Except for the inability to sleep while executing and lack of max_active adjustments, BH workqueues and work items should behave the same as regular workqueues and work items.

Currently, the execution is hooked to tasklet[_hi]. However, the goal is to convert all tasklet users over to BH workqueues. Once the conversion is complete, tasklet can be removed and BH workqueues can directly take over the tasklet softirqs.

system_bh[_highpri]_wq are added. As queue-wide flushing doesn't exist in tasklet, all existing tasklet users should be able to use the system BH workqueues without creating their own workqueues.

v3: - Add missing interrupt.h include.

v2: - Instead of using tasklets, hook directly into its softirq action
functions - tasklet[_hi]_action(). This is slightly cheaper and closer to the eventual code structure we want to arrive at. Suggested by Lai.

- Lai also pointed out several places which need NULL worker->task handling or can use clarification. Updated.

---
 Documentation/core-api/workqueue.rst |  29 ++-
 include/linux/workqueue.h            |  11 +
 kernel/softirq.c                     |   3 +
 kernel/workqueue.c                   | 291 ++++++++++++++++++++++-----
 tools/workqueue/wq_dump.py           |  11 +-
 5 files changed, 285 insertions(+), 60 deletions(-)
Documentation/core-api/workqueue.rst +24 -5
index 33c4539155d9..2d6af6c4665c 100644
@@ -77,10 +77,12 @@ wants a function to be executed asynchronously it has to set up a work
 item pointing to that function and queue that work item on a
 workqueue.
 
-Special purpose threads, called worker threads, execute the functions
-off of the queue, one after the other.  If no work is queued, the
-worker threads become idle.  These worker threads are managed in so
-called worker-pools.
+A work item can be executed in either a thread or the BH (softirq) context.
+
+For threaded workqueues, special purpose threads, called [k]workers, execute
+the functions off of the queue, one after the other. If no work is queued,
+the worker threads become idle. These worker threads are managed in
+worker-pools.
 
 The cmwq design differentiates between the user-facing workqueues that
 subsystems and drivers queue work items on and the backend mechanism
@@ -91,6 +93,12 @@ for high priority ones, for each possible CPU and some extra
 worker-pools to serve work items queued on unbound workqueues - the
 number of these backing pools is dynamic.
 
+BH workqueues use the same framework. However, as there can only be one
+concurrent execution context, there's no need to worry about concurrency.
+Each per-CPU BH worker pool contains only one pseudo worker which represents
+the BH execution context. A BH workqueue can be considered a convenience
+interface to softirq.
+
 Subsystems and drivers can create and queue work items through special
 workqueue API functions as they see fit. They can influence some
 aspects of the way the work items are executed by setting flags on the
@@ -106,7 +114,7 @@ unless specifically overridden, a work item of a bound workqueue will
 be queued on the worklist of either normal or highpri worker-pool that
 is associated to the CPU the issuer is running on.
 
-For any worker pool implementation, managing the concurrency level
+For any thread pool implementation, managing the concurrency level
 (how many execution contexts are active) is an important issue.  cmwq
 tries to keep the concurrency at a minimal but sufficient level.
 Minimal to save resources and sufficient in that the system is used at
@@ -164,6 +172,17 @@ resources, scheduled and executed.
 ``flags``
 ---------
 
+``WQ_BH``
+  BH workqueues can be considered a convenience interface to softirq. BH
+  workqueues are always per-CPU and all BH work items are executed in the
+  queueing CPU's softirq context in the queueing order.
+
+  All BH workqueues must have 0 ``max_active`` and ``WQ_HIGHPRI`` is the
+  only allowed additional flag.
+
+  BH work items cannot sleep. All other features such as delayed queueing,
+  flushing and canceling are supported.
+
 ``WQ_UNBOUND``
   Work items queued to an unbound wq are served by the special
   worker-pools which host workers which are not bound to any
include/linux/workqueue.h +11 -0
index 232baea90a1d..283d7891b4c4 100644
@@ -353,6 +353,7 @@ static inline unsigned int work_static(struct work_struct *work) { return 0; }
  * Documentation/core-api/workqueue.rst.
  */
 enum wq_flags {
+	WQ_BH			= 1 << 0, /* execute in bottom half (softirq) context */
 	WQ_UNBOUND		= 1 << 1, /* not bound to any cpu */
 	WQ_FREEZABLE		= 1 << 2, /* freeze during suspend */
 	WQ_MEM_RECLAIM		= 1 << 3, /* may be used for memory reclaim */
@@ -392,6 +393,9 @@ enum wq_flags {
 	__WQ_ORDERED		= 1 << 17, /* internal: workqueue is ordered */
 	__WQ_LEGACY		= 1 << 18, /* internal: create*_workqueue() */
 	__WQ_ORDERED_EXPLICIT	= 1 << 19, /* internal: alloc_ordered_workqueue() */
+
+	/* BH wq only allows the following flags */
+	__WQ_BH_ALLOWS		= WQ_BH | WQ_HIGHPRI,
 };
 
 enum wq_consts {
@@ -434,6 +438,9 @@ enum wq_consts {
  * they are same as their non-power-efficient counterparts - e.g.
  * system_power_efficient_wq is identical to system_wq if
  * 'wq_power_efficient' is disabled.  See WQ_POWER_EFFICIENT for more info.
+ *
+ * system_bh[_highpri]_wq are convenience interface to softirq. BH work items
+ * are executed in the queueing CPU's BH context in the queueing order.
  */
 extern struct workqueue_struct *system_wq;
 extern struct workqueue_struct *system_highpri_wq;
@@ -442,6 +449,10 @@ extern struct workqueue_struct *system_unbound_wq;
 extern struct workqueue_struct *system_freezable_wq;
 extern struct workqueue_struct *system_power_efficient_wq;
 extern struct workqueue_struct *system_freezable_power_efficient_wq;
+extern struct workqueue_struct *system_bh_wq;
+extern struct workqueue_struct *system_bh_highpri_wq;
+
+void workqueue_softirq_action(bool highpri);
 
 /**
  * alloc_workqueue - allocate a workqueue
kernel/softirq.c +3 -0
index 210cf5f8d92c..547d282548a8 100644
@@ -27,6 +27,7 @@
 #include <linux/tick.h>
 #include <linux/irq.h>
 #include <linux/wait_bit.h>
+#include <linux/workqueue.h>
 
 #include <asm/softirq_stack.h>
 
@@ -802,11 +803,13 @@ static void tasklet_action_common(struct softirq_action *a,
 
 static __latent_entropy void tasklet_action(struct softirq_action *a)
 {
+	workqueue_softirq_action(false);
 	tasklet_action_common(a, this_cpu_ptr(&tasklet_vec), TASKLET_SOFTIRQ);
 }
 
 static __latent_entropy void tasklet_hi_action(struct softirq_action *a)
 {
+	workqueue_softirq_action(true);
 	tasklet_action_common(a, this_cpu_ptr(&tasklet_hi_vec), HI_SOFTIRQ);
 }
 
kernel/workqueue.c +6 -1
index 767971a29c7a..78b4b992e1a3 100644
@@ -29,6 +29,7 @@
 #include <linux/kernel.h>
 #include <linux/sched.h>
 #include <linux/init.h>
+#include <linux/interrupt.h>
 #include <linux/signal.h>
 #include <linux/completion.h>
 #include <linux/workqueue.h>
@@ -72,8 +73,12 @@ enum worker_pool_flags {
 	 * Note that DISASSOCIATED should be flipped only while holding
 	 * wq_pool_attach_mutex to avoid changing binding state while
 	 * worker_attach_to_pool() is in progress.
+	 *
+	 * As there can only be one concurrent BH execution context per CPU, a
+	 * BH pool is per-CPU and always DISASSOCIATED.
 	 */
-	POOL_MANAGER_ACTIVE	= 1 << 0,	/* being managed */
+	POOL_BH			= 1 << 0,	/* is a BH pool */
+	POOL_MANAGER_ACTIVE	= 1 << 1,	/* being managed */
 	POOL_DISASSOCIATED	= 1 << 2,	/* cpu can't serve workers */
 };
 

kernel/workqueue.c enum wq_internal_consts { · line 129

Quoted 10 lines
kernel/workqueue.c +7 -0
@@ -115,6 +120,14 @@ enum wq_internal_consts {
 	WQ_NAME_LEN		= 32,
 };
 
+/*
+ * We don't want to trap softirq for too long. See MAX_SOFTIRQ_TIME and
+ * MAX_SOFTIRQ_RESTART in kernel/softirq.c. These are macros because
+ * msecs_to_jiffies() can't be an initializer.
+ */
+#define BH_WORKER_JIFFIES	msecs_to_jiffies(2)
+#define BH_WORKER_RESTARTS	10

Sorry, late to the party, but I wonder how this play along with cpu hotplug? Say we've queued a lot BH_WORK on a CPU, and we offline that cpu, wouldn't that end up with a few BH_WORK left on that CPU not being executed?

[Cc Thomas]

Regards,
Boqun

Quoted 1 line
kernel/workqueue.c +1 -0
+

[..]

author maintainer
#13 lore raw

Re: [PATCH v3 3/8] workqueue: Implement BH workqueues to eventually replace tasklets

Hello,

Quoted 4 lines · Boqun Feng

Sorry, late to the party, but I wonder how this play along with cpu hotplug? Say we've queued a lot BH_WORK on a CPU, and we offline that cpu, wouldn't that end up with a few BH_WORK left on that CPU not being executed?

Ah, good point. tasklets get transferred out of offline CPU. Workqueue counterpart doesn't do that. I'll fix that.

Thanks.

Signature · 1 line
tejun
author maintainer
#14 lore raw

PATCH 1/1 workqueue: Drain BH work items on hot-unplugged CPUs

In reply to Boqun Feng

Boqun pointed out that workqueues aren't handling BH work items on offlined CPUs. Unlike tasklet which transfers out the pending tasks from CPUHP_SOFTIRQ_DEAD, BH workqueue would just leave them pending which is problematic. Note that this behavior is specific to BH workqueues as the non-BH per-CPU workers just become unbound when the CPU goes offline.

This patch fixes the issue by draining the pending BH work items from an offlined CPU from CPUHP_SOFTIRQ_DEAD. Because work items carry more context, it's not as easy to transfer the pending work items from one pool to another. Instead, run BH work items which execute the offlined pools on an online CPU.

Note that this assumes that no further BH work items will be queued on the offlined CPUs. This assumption is shared with tasklet and should be fine for conversions. However, this issue also exists for per-CPU workqueues which will just keep executing work items queued after CPU offline on unbound workers and workqueue should reject per-CPU and BH work items queued on offline CPUs. This will be addressed separately later.

---
 include/linux/workqueue.h |    1 
 kernel/softirq.c          |    2 +
 kernel/workqueue.c        |   91 ++++++++++++++++++++++++++++++++++++++++++++--
 3 files changed, 91 insertions(+), 3 deletions(-)
include/linux/workqueue.h +1 -0
@@ -458,6 +458,7 @@ extern struct workqueue_struct *system_b
 extern struct workqueue_struct *system_bh_highpri_wq;
 
 void workqueue_softirq_action(bool highpri);
+void workqueue_softirq_dead(unsigned int cpu);
 
 /**
  * alloc_workqueue - allocate a workqueue
kernel/softirq.c +2 -0
@@ -932,6 +932,8 @@ static void run_ksoftirqd(unsigned int c
 #ifdef CONFIG_HOTPLUG_CPU
 static int takeover_tasklets(unsigned int cpu)
 {
+	workqueue_softirq_dead(cpu);
+
 	/* CPU is dead, so no lock needed. */
 	local_irq_disable();
 
kernel/workqueue.c +88 -3

127 lines not on this page · raw message

author maintainer
#15 lore raw

Re: [PATCH for-6.9] workqueue: Drain BH work items on hot-unplugged CPUs

Quoted 19 lines · Tejun Heo

Boqun pointed out that workqueues aren't handling BH work items on offlined CPUs. Unlike tasklet which transfers out the pending tasks from CPUHP_SOFTIRQ_DEAD, BH workqueue would just leave them pending which is problematic. Note that this behavior is specific to BH workqueues as the non-BH per-CPU workers just become unbound when the CPU goes offline.

This patch fixes the issue by draining the pending BH work items from an offlined CPU from CPUHP_SOFTIRQ_DEAD. Because work items carry more context, it's not as easy to transfer the pending work items from one pool to another. Instead, run BH work items which execute the offlined pools on an online CPU.

Note that this assumes that no further BH work items will be queued on the offlined CPUs. This assumption is shared with tasklet and should be fine for conversions. However, this issue also exists for per-CPU workqueues which will just keep executing work items queued after CPU offline on unbound workers and workqueue should reject per-CPU and BH work items queued on offline CPUs. This will be addressed separately later.

Quoted 3 lines

Applying this to wq/for-6.9.

Thanks.

Signature · 1 line
tejun
Reviewed-by
#16 lore raw

Re: [PATCH for-6.9] workqueue: Drain BH work items on hot-unplugged CPUs

Quoted 20 lines · Tejun Heo
Quoted 19 lines · Tejun Heo

Boqun pointed out that workqueues aren't handling BH work items on offlined CPUs. Unlike tasklet which transfers out the pending tasks from CPUHP_SOFTIRQ_DEAD, BH workqueue would just leave them pending which is problematic. Note that this behavior is specific to BH workqueues as the non-BH per-CPU workers just become unbound when the CPU goes offline.

This patch fixes the issue by draining the pending BH work items from an offlined CPU from CPUHP_SOFTIRQ_DEAD. Because work items carry more context, it's not as easy to transfer the pending work items from one pool to another. Instead, run BH work items which execute the offlined pools on an online CPU.

Note that this assumes that no further BH work items will be queued on the offlined CPUs. This assumption is shared with tasklet and should be fine for conversions. However, this issue also exists for per-CPU workqueues which will just keep executing work items queued after CPU offline on unbound workers and workqueue should reject per-CPU and BH work items queued on offline CPUs. This will be addressed separately later.

Quoted 6 lines
Quoted 3 lines

Applying this to wq/for-6.9.

FWIW,

(I took a look yesterday, but hasn't gotten the time to reply..)

Regards,
Boqun

Quoted 4 lines

Thanks.

Signature · 1 line
tejun
author maintainer
#17 lore raw

PATCH 4/8 backtracetest: Convert from tasklet to BH workqueue

In reply to Tejun Heo

The only generic interface to execute asynchronously in the BH context is tasklet; however, it's marked deprecated and has some design flaws. To replace tasklets, BH workqueue support was recently added. A BH workqueue behaves similarly to regular workqueues except that the queued work items are executed in the BH context.

This patch converts backtracetest from tasklet to BH workqueue.

- Replace "irq" with "bh" in names and message to better reflect what's happening.

- Replace completion usage with a flush_work() call.

Cc Arjan van de Ven <[email protected]>
---
 kernel/backtracetest.c | 18 +++++++-----------
 1 file changed, 7 insertions(+), 11 deletions(-)
kernel/backtracetest.c +7 -11
index 370217dd7e39..a4181234232b 100644
@@ -21,24 +21,20 @@ static void backtrace_test_normal(void)
 	dump_stack();
 }
 
-static DECLARE_COMPLETION(backtrace_work);
-
-static void backtrace_test_irq_callback(unsigned long data)
+static void backtrace_test_bh_workfn(struct work_struct *work)
 {
 	dump_stack();
-	complete(&backtrace_work);
 }
 
-static DECLARE_TASKLET_OLD(backtrace_tasklet, &backtrace_test_irq_callback);
+static DECLARE_WORK(backtrace_bh_work, &backtrace_test_bh_workfn);
 
-static void backtrace_test_irq(void)
+static void backtrace_test_bh(void)
 {
-	pr_info("Testing a backtrace from irq context.\n");
+	pr_info("Testing a backtrace from BH context.\n");
 	pr_info("The following trace is a kernel self test and not a bug!\n");
 
-	init_completion(&backtrace_work);
-	tasklet_schedule(&backtrace_tasklet);
-	wait_for_completion(&backtrace_work);
+	queue_work(system_bh_wq, &backtrace_bh_work);
+	flush_work(&backtrace_bh_work);
 }
 
 #ifdef CONFIG_STACKTRACE
@@ -65,7 +61,7 @@ static int backtrace_regression_test(void)
 	pr_info("====[ backtrace testing ]===========\n");
 
 	backtrace_test_normal();
-	backtrace_test_irq();
+	backtrace_test_bh();
 	backtrace_test_saved();
 
 	pr_info("====[ end of backtrace testing ]====\n");
Signature · 1 line
2.43.0
author maintainer
#18 lore raw

PATCH 5/8 usb: core: hcd: Convert from tasklet to BH workqueue

In reply to Tejun Heo

The only generic interface to execute asynchronously in the BH context is tasklet; however, it's marked deprecated and has some design flaws. To replace tasklets, BH workqueue support was recently added. A BH workqueue behaves similarly to regular workqueues except that the queued work items are executed in the BH context.

This patch converts usb hcd from tasklet to BH workqueue.

Cc Greg Kroah-Hartman <[email protected]>
Cc Alan Stern <[email protected]>
---
 drivers/usb/core/hcd.c  | 23 ++++++++++++-----------
 include/linux/usb/hcd.h |  2 +-
 2 files changed, 13 insertions(+), 12 deletions(-)
drivers/usb/core/hcd.c +12 -11
index 12b6dfeaf658..edf74458474a 100644
@@ -1664,9 +1664,10 @@ static void __usb_hcd_giveback_urb(struct urb *urb)
 	usb_put_urb(urb);
 }
 
-static void usb_giveback_urb_bh(struct tasklet_struct *t)
+static void usb_giveback_urb_bh(struct work_struct *work)
 {
-	struct giveback_urb_bh *bh = from_tasklet(bh, t, bh);
+	struct giveback_urb_bh *bh =
+		container_of(work, struct giveback_urb_bh, bh);
 	struct list_head local_list;
 
 	spin_lock_irq(&bh->lock);
@@ -1691,9 +1692,9 @@ static void usb_giveback_urb_bh(struct tasklet_struct *t)
 	spin_lock_irq(&bh->lock);
 	if (!list_empty(&bh->head)) {
 		if (bh->high_prio)
-			tasklet_hi_schedule(&bh->bh);
+			queue_work(system_bh_highpri_wq, &bh->bh);
 		else
-			tasklet_schedule(&bh->bh);
+			queue_work(system_bh_wq, &bh->bh);
 	}
 	bh->running = false;
 	spin_unlock_irq(&bh->lock);
@@ -1706,7 +1707,7 @@ static void usb_giveback_urb_bh(struct tasklet_struct *t)
  * @status: completion status code for the URB.
  *
  * Context: atomic. The completion callback is invoked in caller's context.
- * For HCDs with HCD_BH flag set, the completion callback is invoked in tasklet
+ * For HCDs with HCD_BH flag set, the completion callback is invoked in BH
  * context (except for URBs submitted to the root hub which always complete in
  * caller's context).
  *
@@ -1725,7 +1726,7 @@ void usb_hcd_giveback_urb(struct usb_hcd *hcd, struct urb *urb, int status)
 	struct giveback_urb_bh *bh;
 	bool running;
 
-	/* pass status to tasklet via unlinked */
+	/* pass status to BH via unlinked */
 	if (likely(!urb->unlinked))
 		urb->unlinked = status;
 
@@ -1747,9 +1748,9 @@ void usb_hcd_giveback_urb(struct usb_hcd *hcd, struct urb *urb, int status)
 	if (running)
 		;
 	else if (bh->high_prio)
-		tasklet_hi_schedule(&bh->bh);
+		queue_work(system_bh_highpri_wq, &bh->bh);
 	else
-		tasklet_schedule(&bh->bh);
+		queue_work(system_bh_wq, &bh->bh);
 }
 EXPORT_SYMBOL_GPL(usb_hcd_giveback_urb);
 
@@ -2540,7 +2541,7 @@ static void init_giveback_urb_bh(struct giveback_urb_bh *bh)
 
 	spin_lock_init(&bh->lock);
 	INIT_LIST_HEAD(&bh->head);
-	tasklet_setup(&bh->bh, usb_giveback_urb_bh);
+	INIT_WORK(&bh->bh, usb_giveback_urb_bh);
 }
 
 struct usb_hcd *__usb_create_hcd(const struct hc_driver *driver,
@@ -2926,7 +2927,7 @@ int usb_add_hcd(struct usb_hcd *hcd,
 			&& device_can_wakeup(&hcd->self.root_hub->dev))
 		dev_dbg(hcd->self.controller, "supports USB remote wakeup\n");
 
-	/* initialize tasklets */
+	/* initialize BHs */
 	init_giveback_urb_bh(&hcd->high_prio_bh);
 	hcd->high_prio_bh.high_prio = true;
 	init_giveback_urb_bh(&hcd->low_prio_bh);
@@ -3036,7 +3037,7 @@ void usb_remove_hcd(struct usb_hcd *hcd)
 	mutex_unlock(&usb_bus_idr_lock);
 
 	/*
-	 * tasklet_kill() isn't needed here because:
+	 * flush_work() isn't needed here because:
 	 * - driver's disconnect() called from usb_disconnect() should
 	 *   make sure its URBs are completed during the disconnect()
 	 *   callback
include/linux/usb/hcd.h +1 -1
index 00724b4f6e12..f698aac71de3 100644
@@ -55,7 +55,7 @@ struct giveback_urb_bh {
 	bool high_prio;
 	spinlock_t lock;
 	struct list_head  head;
-	struct tasklet_struct bh;
+	struct work_struct bh;
 	struct usb_host_endpoint *completing_ep;
 };
 
Signature · 1 line
2.43.0
maintainer Acked-by on v1 5/8
#19 lore raw

Re: [PATCH 5/8] usb: core: hcd: Convert from tasklet to BH workqueue

Quoted 8 lines · Tejun Heo

The only generic interface to execute asynchronously in the BH context is tasklet; however, it's marked deprecated and has some design flaws. To replace tasklets, BH workqueue support was recently added. A BH workqueue behaves similarly to regular workqueues except that the queued work items are executed in the BH context.

This patch converts usb hcd from tasklet to BH workqueue.

Quoted 8 lines
Cc Greg Kroah-Hartman <[email protected]>
Cc Alan Stern <[email protected]>
---
 drivers/usb/core/hcd.c  | 23 ++++++++++++-----------
 include/linux/usb/hcd.h |  2 +-
 2 files changed, 13 insertions(+), 12 deletions(-)
Acked-by on v1 5/8
#20 lore raw

Re: [PATCH 5/8] usb: core: hcd: Convert from tasklet to BH workqueue

In reply to Tejun Heo
Quoted 7 lines · Tejun Heo

The only generic interface to execute asynchronously in the BH context is tasklet; however, it's marked deprecated and has some design flaws. To replace tasklets, BH workqueue support was recently added. A BH workqueue behaves similarly to regular workqueues except that the queued work items are executed in the BH context.

This patch converts usb hcd from tasklet to BH workqueue.

In the past this tasklet removal was held up by Mauro's device not properly streaming - hopefully this no longer the case. Cc'ing.

https://lore.kernel.org/all/[email protected]/

Quoted 5 lines
Cc Greg Kroah-Hartman <[email protected]>
Cc Alan Stern <[email protected]>
Quoted 107 lines

---
drivers/usb/core/hcd.c | 23 ++++++++++++----------- include/linux/usb/hcd.h | 2 +-

2 files changed, 13 insertions(+), 12 deletions(-)
drivers/usb/core/hcd.c +12 -11
index 12b6dfeaf658..edf74458474a 100644
@@ -1664,9 +1664,10 @@ static void __usb_hcd_giveback_urb(struct urb *urb)
 	usb_put_urb(urb);
 }
 
-static void usb_giveback_urb_bh(struct tasklet_struct *t)
+static void usb_giveback_urb_bh(struct work_struct *work)
 {
-	struct giveback_urb_bh *bh = from_tasklet(bh, t, bh);
+	struct giveback_urb_bh *bh =
+		container_of(work, struct giveback_urb_bh, bh);
 	struct list_head local_list;
 
 	spin_lock_irq(&bh->lock);
@@ -1691,9 +1692,9 @@ static void usb_giveback_urb_bh(struct tasklet_struct *t)
 	spin_lock_irq(&bh->lock);
 	if (!list_empty(&bh->head)) {
 		if (bh->high_prio)
-			tasklet_hi_schedule(&bh->bh);
+			queue_work(system_bh_highpri_wq, &bh->bh);
 		else
-			tasklet_schedule(&bh->bh);
+			queue_work(system_bh_wq, &bh->bh);
 	}
 	bh->running = false;
 	spin_unlock_irq(&bh->lock);
@@ -1706,7 +1707,7 @@ static void usb_giveback_urb_bh(struct tasklet_struct *t)
 * @status: completion status code for the URB.
 *
 * Context: atomic. The completion callback is invoked in caller's context.
- * For HCDs with HCD_BH flag set, the completion callback is invoked in tasklet
+ * For HCDs with HCD_BH flag set, the completion callback is invoked in BH
 * context (except for URBs submitted to the root hub which always complete in
 * caller's context).
 *
@@ -1725,7 +1726,7 @@ void usb_hcd_giveback_urb(struct usb_hcd *hcd, struct urb *urb, int status)
 	struct giveback_urb_bh *bh;
 	bool running;
 
-	/* pass status to tasklet via unlinked */
+	/* pass status to BH via unlinked */
 	if (likely(!urb->unlinked))
 		urb->unlinked = status;
 
@@ -1747,9 +1748,9 @@ void usb_hcd_giveback_urb(struct usb_hcd *hcd, struct urb *urb, int status)
 	if (running)
 		;
 	else if (bh->high_prio)
-		tasklet_hi_schedule(&bh->bh);
+		queue_work(system_bh_highpri_wq, &bh->bh);
 	else
-		tasklet_schedule(&bh->bh);
+		queue_work(system_bh_wq, &bh->bh);
 }
 EXPORT_SYMBOL_GPL(usb_hcd_giveback_urb);
 
@@ -2540,7 +2541,7 @@ static void init_giveback_urb_bh(struct giveback_urb_bh *bh)
 
 	spin_lock_init(&bh->lock);
 	INIT_LIST_HEAD(&bh->head);
-	tasklet_setup(&bh->bh, usb_giveback_urb_bh);
+	INIT_WORK(&bh->bh, usb_giveback_urb_bh);
 }
 
 struct usb_hcd *__usb_create_hcd(const struct hc_driver *driver,
@@ -2926,7 +2927,7 @@ int usb_add_hcd(struct usb_hcd *hcd,
 			&& device_can_wakeup(&hcd->self.root_hub->dev))
 		dev_dbg(hcd->self.controller, "supports USB remote wakeup\n");
 
-	/* initialize tasklets */
+	/* initialize BHs */
 	init_giveback_urb_bh(&hcd->high_prio_bh);
 	hcd->high_prio_bh.high_prio = true;
 	init_giveback_urb_bh(&hcd->low_prio_bh);
@@ -3036,7 +3037,7 @@ void usb_remove_hcd(struct usb_hcd *hcd)
 	mutex_unlock(&usb_bus_idr_lock);
 
 	/*
-	 * tasklet_kill() isn't needed here because:
+	 * flush_work() isn't needed here because:
 	 * - driver's disconnect() called from usb_disconnect() should
 	 *   make sure its URBs are completed during the disconnect()
 	 *   callback
include/linux/usb/hcd.h +1 -2
index 00724b4f6e12..f698aac71de3 100644
@@ -55,7 +55,7 @@ struct giveback_urb_bh {
 	bool high_prio;
 	spinlock_t lock;
 	struct list_head  head;
-	struct tasklet_struct bh;
+	struct work_struct bh;
 	struct usb_host_endpoint *completing_ep;
 };
 
--

2.43.0

#21 lore raw

Re: [PATCH 5/8] usb: core: hcd: Convert from tasklet to BH workqueue

Quoted 5 lines · Davidlohr Bueso

In the past this tasklet removal was held up by Mauro's device not properly streaming - hopefully this no longer the case. Cc'ing.

https://lore.kernel.org/all/[email protected]/

Oh, lovely - an actual use-case where the old tasklet code has known requirements.

Mauro - the BH workqueue should provide the same kind of latency as the tasklets, and it would be good to validate early that yes, this workqueue conversion works well in practice. Since you have an actual real-life test-case, could you give it a try?

You can find the work in

git://git.kernel.org/pub/scm/linux/kernel/git/tj/wq.git
refs/heads/for-6.9-bh-conversions

although it's possible that Tejun has a newer version in some other branch. Tejun - maybe point Mauro at something he can try out if you have updated the conversion since?

Linus

author maintainer
#22 lore raw

Re: [PATCH 5/8] usb: core: hcd: Convert from tasklet to BH workqueue

Hello,

Quoted 6 lines · Linus Torvalds

git://git.kernel.org/pub/scm/linux/kernel/git/tj/wq.git
refs/heads/for-6.9-bh-conversions

although it's possible that Tejun has a newer version in some other branch. Tejun - maybe point Mauro at something he can try out if you have updated the conversion since?

Just pushed out the following branch for testing.

git://git.kernel.org/pub/scm/linux/kernel/git/tj/wq.git for-6.9-bh-conversions-test

It's the same branch but combined with the current linus#master to avoid the rc1 wonkiness.

Thanks.

Signature · 1 line
tejun
#23 lore raw

Re: [PATCH 5/8] usb: core: hcd: Convert from tasklet to BH workqueue

In reply to Linus Torvalds
Quoted 4 lines · Linus Torvalds

Mauro - the BH workqueue should provide the same kind of latency as the tasklets, and it would be good to validate early that yes, this workqueue conversion works well in practice. Since you have an actual real-life test-case, could you give it a try?

In general I think it's worth pointing out that future conversions should still aim for an equivalent in task context, and now with disable/enable_work a lot opens up for regular wq conversions. If users/maintainers shout about latency, then use BH wq.

Thanks,
Davidlohr

author maintainer
#24 lore raw

PATCH 6/8 net: tcp: tsq: Convert from tasklet to BH workqueue

In reply to Tejun Heo

The only generic interface to execute asynchronously in the BH context is tasklet; however, it's marked deprecated and has some design flaws. To replace tasklets, BH workqueue support was recently added. A BH workqueue behaves similarly to regular workqueues except that the queued work items are executed in the BH context.

This patch converts TCP Small Queues implementation from tasklet to BH workqueue.

Semantically, this is an equivalent conversion and there shouldn't be any user-visible behavior changes. While workqueue's queueing and execution paths are a bit heavier than tasklet's, unless the work item is being queued every packet, the difference hopefully shouldn't matter.

My experience with the networking stack is very limited and this patch definitely needs attention from someone who actually understands networking.

Cc Eric Dumazet <[email protected]>
Cc "David S. Miller" <[email protected]> (maintainer:NETWORKING [IPv4/IPv6])
Cc David Ahern <[email protected]>
Cc Jakub Kicinski <[email protected]> (maintainer:NETWORKING [GENERAL])
Cc Paolo Abeni <[email protected]>
Cc [email protected] (open list:NETWORKING [TCP])
---
 include/net/tcp.h     |  2 +-
 net/ipv4/tcp.c        |  2 +-
 net/ipv4/tcp_output.c | 36 ++++++++++++++++++------------------
 3 files changed, 20 insertions(+), 20 deletions(-)
include/net/tcp.h +1 -1
index dd78a1181031..89f3702be47a 100644
@@ -324,7 +324,7 @@ extern struct proto tcp_prot;
 #define TCP_DEC_STATS(net, field)	SNMP_DEC_STATS((net)->mib.tcp_statistics, field)
 #define TCP_ADD_STATS(net, field, val)	SNMP_ADD_STATS((net)->mib.tcp_statistics, field, val)
 
-void tcp_tasklet_init(void);
+void tcp_tsq_work_init(void);
 
 int tcp_v4_err(struct sk_buff *skb, u32);
 
net/ipv4/tcp.c +1 -1
index 1baa484d2190..d085ee5642fe 100644
@@ -4772,6 +4772,6 @@ void __init tcp_init(void)
 	tcp_v4_init();
 	tcp_metrics_init();
 	BUG_ON(tcp_register_congestion_control(&tcp_reno) != 0);
-	tcp_tasklet_init();
+	tcp_tsq_work_init();
 	mptcp_init();
 }
net/ipv4/tcp_output.c +18 -18
index e3167ad96567..d11be6eebb6e 100644
@@ -1049,15 +1049,15 @@ static unsigned int tcp_established_options(struct sock *sk, struct sk_buff *skb
  * needs to be reallocated in a driver.
  * The invariant being skb->truesize subtracted from sk->sk_wmem_alloc
  *
- * Since transmit from skb destructor is forbidden, we use a tasklet
+ * Since transmit from skb destructor is forbidden, we use a BH work item
  * to process all sockets that eventually need to send more skbs.
- * We use one tasklet per cpu, with its own queue of sockets.
+ * We use one work item per cpu, with its own queue of sockets.
  */
-struct tsq_tasklet {
-	struct tasklet_struct	tasklet;
+struct tsq_work {
+	struct work_struct	work;
 	struct list_head	head; /* queue of tcp sockets */
 };
-static DEFINE_PER_CPU(struct tsq_tasklet, tsq_tasklet);
+static DEFINE_PER_CPU(struct tsq_work, tsq_work);
 
 static void tcp_tsq_write(struct sock *sk)
 {
@@ -1087,14 +1087,14 @@ static void tcp_tsq_handler(struct sock *sk)
 	bh_unlock_sock(sk);
 }
 /*
- * One tasklet per cpu tries to send more skbs.
- * We run in tasklet context but need to disable irqs when
+ * One work item per cpu tries to send more skbs.
+ * We run in BH context but need to disable irqs when
  * transferring tsq->head because tcp_wfree() might
  * interrupt us (non NAPI drivers)
  */
-static void tcp_tasklet_func(struct tasklet_struct *t)
+static void tcp_tsq_workfn(struct work_struct *work)
 {
-	struct tsq_tasklet *tsq = from_tasklet(tsq,  t, tasklet);
+	struct tsq_work *tsq = container_of(work, struct tsq_work, work);
 	LIST_HEAD(list);
 	unsigned long flags;
 	struct list_head *q, *n;
@@ -1164,15 +1164,15 @@ void tcp_release_cb(struct sock *sk)
 }
 EXPORT_SYMBOL(tcp_release_cb);
 
-void __init tcp_tasklet_init(void)
+void __init tcp_tsq_work_init(void)
 {
 	int i;
 
 	for_each_possible_cpu(i) {
-		struct tsq_tasklet *tsq = &per_cpu(tsq_tasklet, i);
+		struct tsq_work *tsq = &per_cpu(tsq_work, i);
 
 		INIT_LIST_HEAD(&tsq->head);
-		tasklet_setup(&tsq->tasklet, tcp_tasklet_func);
+		INIT_WORK(&tsq->work, tcp_tsq_workfn);
 	}
 }
 
@@ -1186,11 +1186,11 @@ void tcp_wfree(struct sk_buff *skb)
 	struct sock *sk = skb->sk;
 	struct tcp_sock *tp = tcp_sk(sk);
 	unsigned long flags, nval, oval;
-	struct tsq_tasklet *tsq;
+	struct tsq_work *tsq;
 	bool empty;
 
 	/* Keep one reference on sk_wmem_alloc.
-	 * Will be released by sk_free() from here or tcp_tasklet_func()
+	 * Will be released by sk_free() from here or tcp_tsq_workfn()
 	 */
 	WARN_ON(refcount_sub_and_test(skb->truesize - 1, &sk->sk_wmem_alloc));
 
@@ -1212,13 +1212,13 @@ void tcp_wfree(struct sk_buff *skb)
 		nval = (oval & ~TSQF_THROTTLED) | TSQF_QUEUED;
 	} while (!try_cmpxchg(&sk->sk_tsq_flags, &oval, nval));
 
-	/* queue this socket to tasklet queue */
+	/* queue this socket to BH workqueue */
 	local_irq_save(flags);
-	tsq = this_cpu_ptr(&tsq_tasklet);
+	tsq = this_cpu_ptr(&tsq_work);
 	empty = list_empty(&tsq->head);
 	list_add(&tp->tsq_node, &tsq->head);
 	if (empty)
-		tasklet_schedule(&tsq->tasklet);
+		queue_work(system_bh_wq, &tsq->work);
 	local_irq_restore(flags);
 	return;
 out:
@@ -2623,7 +2623,7 @@ static bool tcp_small_queue_check(struct sock *sk, const struct sk_buff *skb,
 	if (refcount_read(&sk->sk_wmem_alloc) > limit) {
 		/* Always send skb if rtx queue is empty or has one skb.
 		 * No need to wait for TX completion to call us back,
-		 * after softirq/tasklet schedule.
+		 * after softirq schedule.
 		 * This helps when TX completions are delayed too much.
 		 */
 		if (tcp_rtx_queue_empty_or_single_skb(sk))
Signature · 1 line
2.43.0
author maintainer
#25 lore raw

Re: [PATCH 6/8] net: tcp: tsq: Convert from tasklet to BH workqueue

Hello,

Quoted 9 lines · Tejun Heo

The only generic interface to execute asynchronously in the BH context is tasklet; however, it's marked deprecated and has some design flaws. To replace tasklets, BH workqueue support was recently added. A BH workqueue behaves similarly to regular workqueues except that the queued work items are executed in the BH context.

This patch converts TCP Small Queues implementation from tasklet to BH workqueue.

Quoted 7 lines

Semantically, this is an equivalent conversion and there shouldn't be any user-visible behavior changes. While workqueue's queueing and execution paths are a bit heavier than tasklet's, unless the work item is being queued every packet, the difference hopefully shouldn't matter.

My experience with the networking stack is very limited and this patch definitely needs attention from someone who actually understands networking.

On Jakub's recommendation, I asked David Wei to perform production memcache benchmark on the backported conversion patch. There was no discernible difference before and after. Given that this is likely as hot as it gets for the path on a real workloal, the conversions shouldn't hopefully be noticeable in terms of performance impact.

Jakub, I'd really appreciate if you could ack. David, would it be okay if I add your Tested-by?

Thanks.

Signature · 1 line
tejun
#26 lore raw

Re: [PATCH 6/8] net: tcp: tsq: Convert from tasklet to BH workqueue

Quoted 21 lines · Tejun Heo

Hello,

Quoted 9 lines · Tejun Heo

The only generic interface to execute asynchronously in the BH context is tasklet; however, it's marked deprecated and has some design flaws. To replace tasklets, BH workqueue support was recently added. A BH workqueue behaves similarly to regular workqueues except that the queued work items are executed in the BH context.

This patch converts TCP Small Queues implementation from tasklet to BH workqueue.

Quoted 7 lines

Semantically, this is an equivalent conversion and there shouldn't be any user-visible behavior changes. While workqueue's queueing and execution paths are a bit heavier than tasklet's, unless the work item is being queued every packet, the difference hopefully shouldn't matter.

My experience with the networking stack is very limited and this patch definitely needs attention from someone who actually understands networking.

Quoted 8 lines

On Jakub's recommendation, I asked David Wei to perform production memcache benchmark on the backported conversion patch. There was no discernible difference before and after. Given that this is likely as hot as it gets for the path on a real workloal, the conversions shouldn't hopefully be noticeable in terms of performance impact.

Jakub, I'd really appreciate if you could ack. David, would it be okay if I add your Tested-by?

I presume memcache benchmark is using small RPC ?

TSQ matters for high BDP, and is very time sensitive.

Things like slow TX completions (firing from napi poll, BH context) can hurt TSQ.

If we add on top of these slow TX completions, an additional work queue overhead, I really am not sure...

I would recommend tests with pfifo_fast qdisc (not FQ which has a special override for TSQ limits)

Eventually we could add in TCP a measure of the time lost because of TSQ, regardless of the kick implementation (tasklet or workqueue). Measuring the delay between when a tcp socket got tcp_wfree approval to deliver more packets, and time it finally delivered these packets could be implemented with a bpftrace program.

#27 lore raw

Re: [PATCH 6/8] net: tcp: tsq: Convert from tasklet to BH workqueue

Quoted 35 lines · Eric Dumazet

!-------------------------------------------------------------------|
This Message Is From an External Sender

|-------------------------------------------------------------------!

Quoted 29 lines · Tejun Heo

Hello,

Quoted 9 lines · Tejun Heo

The only generic interface to execute asynchronously in the BH context is tasklet; however, it's marked deprecated and has some design flaws. To replace tasklets, BH workqueue support was recently added. A BH workqueue behaves similarly to regular workqueues except that the queued work items are executed in the BH context.

This patch converts TCP Small Queues implementation from tasklet to BH workqueue.

Quoted 7 lines

Semantically, this is an equivalent conversion and there shouldn't be any user-visible behavior changes. While workqueue's queueing and execution paths are a bit heavier than tasklet's, unless the work item is being queued every packet, the difference hopefully shouldn't matter.

My experience with the networking stack is very limited and this patch definitely needs attention from someone who actually understands networking.

On Jakub's recommendation, I asked David Wei to perform production memcache benchmark on the backported conversion patch. There was no discernible difference before and after. Given that this is likely as hot as it gets for the path on a real workloal, the conversions shouldn't hopefully be noticeable in terms of performance impact.

Jakub, I'd really appreciate if you could ack. David, would it be okay if I add your Tested-by?

Yes, that's fine.

Quoted 2 lines

I presume memcache benchmark is using small RPC ?

It is not a benchmark but a prod shadow, but yes the requests are small.

Quoted 17 lines

TSQ matters for high BDP, and is very time sensitive.

Things like slow TX completions (firing from napi poll, BH context) can hurt TSQ.

If we add on top of these slow TX completions, an additional work queue overhead, I really am not sure...

I would recommend tests with pfifo_fast qdisc (not FQ which has a special override for TSQ limits)

Eventually we could add in TCP a measure of the time lost because of TSQ, regardless of the kick implementation (tasklet or workqueue). Measuring the delay between when a tcp socket got tcp_wfree approval to deliver more packets, and time it finally delivered these packets could be implemented with a bpftrace program.

author maintainer
#28 lore raw

Re: [PATCH 6/8] net: tcp: tsq: Convert from tasklet to BH workqueue

In reply to Eric Dumazet

Hello, Eric. How have you been?

On Fri, Feb 16, 2024 at 09:23:00AM +0100, Eric Dumazet wrote:
...

Quoted 7 lines

TSQ matters for high BDP, and is very time sensitive.

Things like slow TX completions (firing from napi poll, BH context) can hurt TSQ.

If we add on top of these slow TX completions, an additional work queue overhead, I really am not sure...

Just to be sure, the workqueue here is executing in the same softirq context as tasklets. This isn't the usual workqueue which has to go through the scheduler. The only difference would be that workqueue does a bit more work (e.g. to manage the currenty executing hashtable) than tasklet. It's unlikely to show noticeable latency penalty in any practical case although the extra overhead would likely be visible in targeted microbenches where all that happens is scheduling and running noop work items.

Quoted 2 lines

I would recommend tests with pfifo_fast qdisc (not FQ which has a special override for TSQ limits)

David, do you think this is something we can do?

Quoted 5 lines

Eventually we could add in TCP a measure of the time lost because of TSQ, regardless of the kick implementation (tasklet or workqueue). Measuring the delay between when a tcp socket got tcp_wfree approval to deliver more packets, and time it finally delivered these packets could be implemented with a bpftrace program.

I don't have enough context here but it sounds like you are worried about adding latency in that path. This conversion is unlikely to make a noticeable difference there. The interface and sementics are workqueue but the work items are being executed exactly the same way from the same softirqs as tasklets. Would testing with pfifo_fast be sufficient to dispel your concern?

Thanks.

Signature · 1 line
tejun
#29 lore raw

Re: [PATCH 6/8] net: tcp: tsq: Convert from tasklet to BH workqueue

In reply to Tejun Heo
Quoted 18 lines · Tejun Heo

The only generic interface to execute asynchronously in the BH context is tasklet; however, it's marked deprecated and has some design flaws. To replace tasklets, BH workqueue support was recently added. A BH workqueue behaves similarly to regular workqueues except that the queued work items are executed in the BH context.

This patch converts TCP Small Queues implementation from tasklet to BH workqueue.

Semantically, this is an equivalent conversion and there shouldn't be any user-visible behavior changes. While workqueue's queueing and execution paths are a bit heavier than tasklet's, unless the work item is being queued every packet, the difference hopefully shouldn't matter.

My experience with the networking stack is very limited and this patch definitely needs attention from someone who actually understands networking.

Quoted 7 lines
Cc Eric Dumazet <[email protected]>
Cc "David S. Miller" <[email protected]> (maintainer:NETWORKING [IPv4/IPv6])
Cc David Ahern <[email protected]>
Cc Jakub Kicinski <[email protected]> (maintainer:NETWORKING [GENERAL])
Cc Paolo Abeni <[email protected]>
Cc [email protected] (open list:NETWORKING [TCP])

Hi Tejun,

Sorry to revive the old thread! I noticed this change because I've been doing an investigation around TSQ recently. I'm very cautious about the change in the core/sensitive part of the networking area because it might affect some corner cases beyond our limited test, even though I've tested many rounds and no regression results (including the latency between tcp_wfree and tcp_tsq_handler) show up. My main concern is what the exact benefit/improvement it could bring with the change applied since your BH workqueue commit[1] says the tasklet mechanism has some flaws. I'd like to see if I can reproduce/verify it.

[1]: https://git.kernel.org/pub/scm/linux/kernel/git/netdev/net-next.git/commit/?id=4cb1ef64609f9

Thanks,
Jason

Quoted 138 lines
---
 include/net/tcp.h     |  2 +-
 net/ipv4/tcp.c        |  2 +-
 net/ipv4/tcp_output.c | 36 ++++++++++++++++++------------------
 3 files changed, 20 insertions(+), 20 deletions(-)
include/net/tcp.h +1 -1
index dd78a1181031..89f3702be47a 100644
@@ -324,7 +324,7 @@ extern struct proto tcp_prot;
 #define TCP_DEC_STATS(net, field)      SNMP_DEC_STATS((net)->mib.tcp_statistics, field)
 #define TCP_ADD_STATS(net, field, val) SNMP_ADD_STATS((net)->mib.tcp_statistics, field, val)
 
-void tcp_tasklet_init(void);
+void tcp_tsq_work_init(void);
 
 int tcp_v4_err(struct sk_buff *skb, u32);
 
net/ipv4/tcp.c +1 -1
index 1baa484d2190..d085ee5642fe 100644
@@ -4772,6 +4772,6 @@ void __init tcp_init(void)
        tcp_v4_init();
        tcp_metrics_init();
        BUG_ON(tcp_register_congestion_control(&tcp_reno) != 0);
-       tcp_tasklet_init();
+       tcp_tsq_work_init();
        mptcp_init();
 }
net/ipv4/tcp_output.c +18 -19
index e3167ad96567..d11be6eebb6e 100644
@@ -1049,15 +1049,15 @@ static unsigned int tcp_established_options(struct sock *sk, struct sk_buff *skb
  * needs to be reallocated in a driver.
  * The invariant being skb->truesize subtracted from sk->sk_wmem_alloc
  *
- * Since transmit from skb destructor is forbidden, we use a tasklet
+ * Since transmit from skb destructor is forbidden, we use a BH work item
  * to process all sockets that eventually need to send more skbs.
- * We use one tasklet per cpu, with its own queue of sockets.
+ * We use one work item per cpu, with its own queue of sockets.
  */
-struct tsq_tasklet {
-       struct tasklet_struct   tasklet;
+struct tsq_work {
+       struct work_struct      work;
        struct list_head        head; /* queue of tcp sockets */
 };
-static DEFINE_PER_CPU(struct tsq_tasklet, tsq_tasklet);
+static DEFINE_PER_CPU(struct tsq_work, tsq_work);
 
 static void tcp_tsq_write(struct sock *sk)
 {
@@ -1087,14 +1087,14 @@ static void tcp_tsq_handler(struct sock *sk)
        bh_unlock_sock(sk);
 }
 /*
- * One tasklet per cpu tries to send more skbs.
- * We run in tasklet context but need to disable irqs when
+ * One work item per cpu tries to send more skbs.
+ * We run in BH context but need to disable irqs when
  * transferring tsq->head because tcp_wfree() might
  * interrupt us (non NAPI drivers)
  */
-static void tcp_tasklet_func(struct tasklet_struct *t)
+static void tcp_tsq_workfn(struct work_struct *work)
 {
-       struct tsq_tasklet *tsq = from_tasklet(tsq,  t, tasklet);
+       struct tsq_work *tsq = container_of(work, struct tsq_work, work);
        LIST_HEAD(list);
        unsigned long flags;
        struct list_head *q, *n;
@@ -1164,15 +1164,15 @@ void tcp_release_cb(struct sock *sk)
 }
 EXPORT_SYMBOL(tcp_release_cb);
 
-void __init tcp_tasklet_init(void)
+void __init tcp_tsq_work_init(void)
 {
        int i;
 
        for_each_possible_cpu(i) {
-               struct tsq_tasklet *tsq = &per_cpu(tsq_tasklet, i);
+               struct tsq_work *tsq = &per_cpu(tsq_work, i);
 
                INIT_LIST_HEAD(&tsq->head);
-               tasklet_setup(&tsq->tasklet, tcp_tasklet_func);
+               INIT_WORK(&tsq->work, tcp_tsq_workfn);
        }
 }
 
@@ -1186,11 +1186,11 @@ void tcp_wfree(struct sk_buff *skb)
        struct sock *sk = skb->sk;
        struct tcp_sock *tp = tcp_sk(sk);
        unsigned long flags, nval, oval;
-       struct tsq_tasklet *tsq;
+       struct tsq_work *tsq;
        bool empty;
 
        /* Keep one reference on sk_wmem_alloc.
-        * Will be released by sk_free() from here or tcp_tasklet_func()
+        * Will be released by sk_free() from here or tcp_tsq_workfn()
         */
        WARN_ON(refcount_sub_and_test(skb->truesize - 1, &sk->sk_wmem_alloc));
 
@@ -1212,13 +1212,13 @@ void tcp_wfree(struct sk_buff *skb)
                nval = (oval & ~TSQF_THROTTLED) | TSQF_QUEUED;
        } while (!try_cmpxchg(&sk->sk_tsq_flags, &oval, nval));
 
-       /* queue this socket to tasklet queue */
+       /* queue this socket to BH workqueue */
        local_irq_save(flags);
-       tsq = this_cpu_ptr(&tsq_tasklet);
+       tsq = this_cpu_ptr(&tsq_work);
        empty = list_empty(&tsq->head);
        list_add(&tp->tsq_node, &tsq->head);
        if (empty)
-               tasklet_schedule(&tsq->tasklet);
+               queue_work(system_bh_wq, &tsq->work);
        local_irq_restore(flags);
        return;
 out:
@@ -2623,7 +2623,7 @@ static bool tcp_small_queue_check(struct sock *sk, const struct sk_buff *skb,
        if (refcount_read(&sk->sk_wmem_alloc) > limit) {
                /* Always send skb if rtx queue is empty or has one skb.
                 * No need to wait for TX completion to call us back,
-                * after softirq/tasklet schedule.
+                * after softirq schedule.
                 * This helps when TX completions are delayed too much.
                 */
                if (tcp_rtx_queue_empty_or_single_skb(sk))
--

2.43.0

author maintainer
#30 lore raw

Re: [PATCH 6/8] net: tcp: tsq: Convert from tasklet to BH workqueue

Hello,

Quoted 10 lines · Jason Xing

Sorry to revive the old thread! I noticed this change because I've been doing an investigation around TSQ recently. I'm very cautious about the change in the core/sensitive part of the networking area because it might affect some corner cases beyond our limited test, even though I've tested many rounds and no regression results (including the latency between tcp_wfree and tcp_tsq_handler) show up. My main concern is what the exact benefit/improvement it could bring with the change applied since your BH workqueue commit[1] says the tasklet mechanism has some flaws. I'd like to see if I can reproduce/verify it.

There won't be any behavioral benefits. It's mostly that it'd be great to get rid of tasklets with something which is more generic, so if BH workqueue doesn't regress, we want to keep moving users to BH workqueue until all tasklet users are gone and then remove tasklet.

Thanks.

Signature · 1 line
tejun
Reviewed-by
#31 lore raw

Re: [PATCH 6/8] net: tcp: tsq: Convert from tasklet to BH workqueue

Quoted 15 lines · Tejun Heo

Hello,

Quoted 10 lines · Jason Xing

Sorry to revive the old thread! I noticed this change because I've been doing an investigation around TSQ recently. I'm very cautious about the change in the core/sensitive part of the networking area because it might affect some corner cases beyond our limited test, even though I've tested many rounds and no regression results (including the latency between tcp_wfree and tcp_tsq_handler) show up. My main concern is what the exact benefit/improvement it could bring with the change applied since your BH workqueue commit[1] says the tasklet mechanism has some flaws. I'd like to see if I can reproduce/verify it.

Quoted 4 lines

There won't be any behavioral benefits. It's mostly that it'd be great to get rid of tasklets with something which is more generic, so if BH workqueue doesn't regress, we want to keep moving users to BH workqueue until all tasklet users are gone and then remove tasklet.

I see. So far I haven't see any side effect especially with testing bulk transfer/multi-thread transfer, so if you're willing to repost it after the merge window, please feel free to add:

Thanks,
Jason

author maintainer
#32 lore raw

PATCH 7/8 dm-crypt: Convert from tasklet to BH workqueue

In reply to Tejun Heo

The only generic interface to execute asynchronously in the BH context is tasklet; however, it's marked deprecated and has some design flaws. To replace tasklets, BH workqueue support was recently added. A BH workqueue behaves similarly to regular workqueues except that the queued work items are executed in the BH context.

This patch converts dm-crypt from tasklet to BH workqueue.

Like a regular workqueue, a BH workqueue allows freeing the currently executing work item. Converting from tasklet to BH workqueue removes the need for deferring bio_endio() again to a work item, which was buggy anyway.

I tested this lightly with "--perf-no_read_workqueue --perf-no_write_workqueue" + some code modifications, but would really -appreciate if someone who knows the code base better could take a look.

---
 drivers/md/dm-crypt.c | 36 ++----------------------------------
 1 file changed, 2 insertions(+), 34 deletions(-)
drivers/md/dm-crypt.c +2 -34
index 855b482cbff1..619c762d4072 100644
@@ -73,11 +73,8 @@ struct dm_crypt_io {
 	struct bio *base_bio;
 	u8 *integrity_metadata;
 	bool integrity_metadata_from_pool:1;
-	bool in_tasklet:1;
 
 	struct work_struct work;
-	struct tasklet_struct tasklet;
-
 	struct convert_context ctx;
 
 	atomic_t io_pending;
@@ -1762,7 +1759,6 @@ static void crypt_io_init(struct dm_crypt_io *io, struct crypt_config *cc,
 	io->ctx.r.req = NULL;
 	io->integrity_metadata = NULL;
 	io->integrity_metadata_from_pool = false;
-	io->in_tasklet = false;
 	atomic_set(&io->io_pending, 0);
 }
 
@@ -1771,13 +1767,6 @@ static void crypt_inc_pending(struct dm_crypt_io *io)
 	atomic_inc(&io->io_pending);
 }
 
-static void kcryptd_io_bio_endio(struct work_struct *work)
-{
-	struct dm_crypt_io *io = container_of(work, struct dm_crypt_io, work);
-
-	bio_endio(io->base_bio);
-}
-
 /*
  * One of the bios was finished. Check for completion of
  * the whole request and correctly clean up the buffer.
@@ -1800,21 +1789,6 @@ static void crypt_dec_pending(struct dm_crypt_io *io)
 		kfree(io->integrity_metadata);
 
 	base_bio->bi_status = error;
-
-	/*
-	 * If we are running this function from our tasklet,
-	 * we can't call bio_endio() here, because it will call
-	 * clone_endio() from dm.c, which in turn will
-	 * free the current struct dm_crypt_io structure with
-	 * our tasklet. In this case we need to delay bio_endio()
-	 * execution to after the tasklet is done and dequeued.
-	 */
-	if (io->in_tasklet) {
-		INIT_WORK(&io->work, kcryptd_io_bio_endio);
-		queue_work(cc->io_queue, &io->work);
-		return;
-	}
-
 	bio_endio(base_bio);
 }
 
@@ -2246,11 +2220,6 @@ static void kcryptd_crypt(struct work_struct *work)
 		kcryptd_crypt_write_convert(io);
 }
 
-static void kcryptd_crypt_tasklet(unsigned long work)
-{
-	kcryptd_crypt((struct work_struct *)work);
-}
-
 static void kcryptd_queue_crypt(struct dm_crypt_io *io)
 {
 	struct crypt_config *cc = io->cc;
@@ -2263,9 +2232,8 @@ static void kcryptd_queue_crypt(struct dm_crypt_io *io)
 		 * it is being executed with irqs disabled.
 		 */
 		if (in_hardirq() || irqs_disabled()) {
-			io->in_tasklet = true;
-			tasklet_init(&io->tasklet, kcryptd_crypt_tasklet, (unsigned long)&io->work);
-			tasklet_schedule(&io->tasklet);
+			INIT_WORK(&io->work, kcryptd_crypt);
+			queue_work(system_bh_wq, &io->work);
 			return;
 		}
 
Signature · 1 line
2.43.0
#33 lore raw

Re: [PATCH 7/8] dm-crypt: Convert from tasklet to BH workqueue

drivers/md/dm-crypt.c static void kcryptd_queue_crypt(struct dm_crypt_io *io)

Quoted 3 lines · Tejun Heo
drivers/md/dm-crypt.c +0 -0
@@ -2263,9 +2232,8 @@ static void kcryptd_queue_crypt(struct dm_crypt_io *io)

/*
* in_hardirq(): Crypto API's skcipher_walk_first() refuses to work in hard IRQ context.
* irqs_disabled(): the kernel may run some IO completion from the idle thread, but
* it is being executed with irqs disabled.
*/

drivers/md/dm-crypt.c static void kcryptd_queue_crypt(struct dm_crypt_io *io) · line 2236

Quoted 8 lines
drivers/md/dm-crypt.c +2 -3
 		 * it is being executed with irqs disabled.
 		 */
 		if (in_hardirq() || irqs_disabled()) {
-			io->in_tasklet = true;
-			tasklet_init(&io->tasklet, kcryptd_crypt_tasklet, (unsigned long)&io->work);
-			tasklet_schedule(&io->tasklet);
+			INIT_WORK(&io->work, kcryptd_crypt);
+			queue_work(system_bh_wq, &io->work);

Why do we need the tasklet here in the first place? Couldn't we use workqueue? As per comment, the request originates in hardirq and then it is moved to tasklet. Couldn't it be moved to workqueue regardless?

drivers/md/dm-crypt.c static void kcryptd_queue_crypt(struct dm_crypt_io *io) · line 2239

Quoted 3 lines
drivers/md/dm-crypt.c +0 -0
 			return;
 		}
 

Sebastian

author maintainer
#34 lore raw

Re: [PATCH 7/8] dm-crypt: Convert from tasklet to BH workqueue

Hello,

Quoted 11 lines · Sebastian Andrzej Siewior

drivers/md/dm-crypt.c static void kcryptd_queue_crypt(struct dm_crypt_io *io) · line 2236

Quoted 6 lines · Tejun Heo
drivers/md/dm-crypt.c +2 -3
 		if (in_hardirq() || irqs_disabled()) {
-			io->in_tasklet = true;
-			tasklet_init(&io->tasklet, kcryptd_crypt_tasklet, (unsigned long)&io->work);
-			tasklet_schedule(&io->tasklet);
+			INIT_WORK(&io->work, kcryptd_crypt);
+			queue_work(system_bh_wq, &io->work);

Why do we need the tasklet here in the first place? Couldn't we use workqueue? As per comment, the request originates in hardirq and then it is moved to tasklet. Couldn't it be moved to workqueue regardless?

Yes, you can and if you replace that system_bh_wq with system_wq, or system_highpri_wq, everything should be fine in terms of correctness. However, that would mean that the work item now would always incur a scheduling latency which can be lengthy in certain circumstances. Whether that's an actual problem for the subsystem at hand, I have no idea.

Thanks.

Signature · 1 line
tejun
maintainer Reviewed-by on v1 7/8
#35 lore raw

Re: [PATCH 7/8] dm-crypt: Convert from tasklet to BH workqueue

In reply to Tejun Heo

This seems OK.

Quoted 114 lines · Tejun Heo

The only generic interface to execute asynchronously in the BH context is tasklet; however, it's marked deprecated and has some design flaws. To replace tasklets, BH workqueue support was recently added. A BH workqueue behaves similarly to regular workqueues except that the queued work items are executed in the BH context.

This patch converts dm-crypt from tasklet to BH workqueue.

Like a regular workqueue, a BH workqueue allows freeing the currently executing work item. Converting from tasklet to BH workqueue removes the need for deferring bio_endio() again to a work item, which was buggy anyway.

I tested this lightly with "--perf-no_read_workqueue --perf-no_write_workqueue" + some code modifications, but would really -appreciate if someone who knows the code base better could take a look.

---
 drivers/md/dm-crypt.c | 36 ++----------------------------------
 1 file changed, 2 insertions(+), 34 deletions(-)
drivers/md/dm-crypt.c +2 -34
index 855b482cbff1..619c762d4072 100644
@@ -73,11 +73,8 @@ struct dm_crypt_io {
 	struct bio *base_bio;
 	u8 *integrity_metadata;
 	bool integrity_metadata_from_pool:1;
-	bool in_tasklet:1;
 
 	struct work_struct work;
-	struct tasklet_struct tasklet;
-
 	struct convert_context ctx;
 
 	atomic_t io_pending;
@@ -1762,7 +1759,6 @@ static void crypt_io_init(struct dm_crypt_io *io, struct crypt_config *cc,
 	io->ctx.r.req = NULL;
 	io->integrity_metadata = NULL;
 	io->integrity_metadata_from_pool = false;
-	io->in_tasklet = false;
 	atomic_set(&io->io_pending, 0);
 }
 
@@ -1771,13 +1767,6 @@ static void crypt_inc_pending(struct dm_crypt_io *io)
 	atomic_inc(&io->io_pending);
 }
 
-static void kcryptd_io_bio_endio(struct work_struct *work)
-{
-	struct dm_crypt_io *io = container_of(work, struct dm_crypt_io, work);
-
-	bio_endio(io->base_bio);
-}
-
 /*
  * One of the bios was finished. Check for completion of
  * the whole request and correctly clean up the buffer.
@@ -1800,21 +1789,6 @@ static void crypt_dec_pending(struct dm_crypt_io *io)
 		kfree(io->integrity_metadata);
 
 	base_bio->bi_status = error;
-
-	/*
-	 * If we are running this function from our tasklet,
-	 * we can't call bio_endio() here, because it will call
-	 * clone_endio() from dm.c, which in turn will
-	 * free the current struct dm_crypt_io structure with
-	 * our tasklet. In this case we need to delay bio_endio()
-	 * execution to after the tasklet is done and dequeued.
-	 */
-	if (io->in_tasklet) {
-		INIT_WORK(&io->work, kcryptd_io_bio_endio);
-		queue_work(cc->io_queue, &io->work);
-		return;
-	}
-
 	bio_endio(base_bio);
 }
 
@@ -2246,11 +2220,6 @@ static void kcryptd_crypt(struct work_struct *work)
 		kcryptd_crypt_write_convert(io);
 }
 
-static void kcryptd_crypt_tasklet(unsigned long work)
-{
-	kcryptd_crypt((struct work_struct *)work);
-}
-
 static void kcryptd_queue_crypt(struct dm_crypt_io *io)
 {
 	struct crypt_config *cc = io->cc;
@@ -2263,9 +2232,8 @@ static void kcryptd_queue_crypt(struct dm_crypt_io *io)
 		 * it is being executed with irqs disabled.
 		 */
 		if (in_hardirq() || irqs_disabled()) {
-			io->in_tasklet = true;
-			tasklet_init(&io->tasklet, kcryptd_crypt_tasklet, (unsigned long)&io->work);
-			tasklet_schedule(&io->tasklet);
+			INIT_WORK(&io->work, kcryptd_crypt);
+			queue_work(system_bh_wq, &io->work);
 			return;
 		}
 
Signature · 1 line
2.43.0
author maintainer
#36 lore raw

PATCH 8/8 dm-verity: Convert from tasklet to BH workqueue

In reply to Tejun Heo

The only generic interface to execute asynchronously in the BH context is tasklet; however, it's marked deprecated and has some design flaws. To replace tasklets, BH workqueue support was recently added. A BH workqueue behaves similarly to regular workqueues except that the queued work items are executed in the BH context.

This patch converts dm-verity from tasklet to BH workqueue.

This is a minimal conversion which doesn't rename the related names including the "try_verify_in_tasklet" option. If this patch is applied, a follow-up patch would be necessary. I couldn't decide whether the option name would need to be updated too.

Only compile tested. I don't know how to verity.

Cc Alasdair Kergon <[email protected]>
Cc Mike Snitzer <[email protected]>
Cc Mikulas Patocka <[email protected]>
---
 drivers/md/dm-verity-target.c | 8 ++++----
 drivers/md/dm-verity.h        | 2 +-
 2 files changed, 5 insertions(+), 5 deletions(-)
drivers/md/dm-verity-target.c +4 -4
index 14e58ae70521..911261de2d08 100644
@@ -645,9 +645,9 @@ static void verity_work(struct work_struct *w)
 	verity_finish_io(io, errno_to_blk_status(verity_verify_io(io)));
 }
 
-static void verity_tasklet(unsigned long data)
+static void verity_bh_work(struct work_struct *w)
 {
-	struct dm_verity_io *io = (struct dm_verity_io *)data;
+	struct dm_verity_io *io = container_of(w, struct dm_verity_io, bh_work);
 	int err;
 
 	io->in_tasklet = true;
@@ -675,8 +675,8 @@ static void verity_end_io(struct bio *bio)
 	}
 
 	if (static_branch_unlikely(&use_tasklet_enabled) && io->v->use_tasklet) {
-		tasklet_init(&io->tasklet, verity_tasklet, (unsigned long)io);
-		tasklet_schedule(&io->tasklet);
+		INIT_WORK(&io->bh_work, verity_bh_work);
+		queue_work(system_bh_wq, &io->bh_work);
 	} else {
 		INIT_WORK(&io->work, verity_work);
 		queue_work(io->v->verify_wq, &io->work);
drivers/md/dm-verity.h +1 -1
index f9d522c870e6..7c16f834f31a 100644
@@ -83,7 +83,7 @@ struct dm_verity_io {
 	struct bvec_iter iter;
 
 	struct work_struct work;
-	struct tasklet_struct tasklet;
+	struct work_struct bh_work;
 
 	/*
 	 * Three variably-size fields follow this struct:
Signature · 1 line
2.43.0
maintainer
#37 lore raw

Re: [PATCH 8/8] dm-verity: Convert from tasklet to BH workqueue

Quoted 6 lines · Tejun Heo

The only generic interface to execute asynchronously in the BH context is tasklet; however, it's marked deprecated and has some design flaws. To replace tasklets, BH workqueue support was recently added. A BH workqueue behaves similarly to regular workqueues except that the queued work items are executed in the BH context.

Quoted 8 lines

This patch converts dm-verity from tasklet to BH workqueue.

This is a minimal conversion which doesn't rename the related names including the "try_verify_in_tasklet" option. If this patch is applied, a follow-up patch would be necessary. I couldn't decide whether the option name would need to be updated too.

Only compile tested. I don't know how to verity.

Download the cryptsetup package with "git clone
https://gitlab.com/cryptsetup/cryptsetup" and run the testsuite:
./autogen.sh && ./configure && make && make check

Quoted 42 lines
Cc Alasdair Kergon <[email protected]>
Cc Mike Snitzer <[email protected]>
Cc Mikulas Patocka <[email protected]>
---
 drivers/md/dm-verity-target.c | 8 ++++----
 drivers/md/dm-verity.h        | 2 +-
 2 files changed, 5 insertions(+), 5 deletions(-)
drivers/md/dm-verity-target.c +4 -4
index 14e58ae70521..911261de2d08 100644
@@ -645,9 +645,9 @@ static void verity_work(struct work_struct *w)
 	verity_finish_io(io, errno_to_blk_status(verity_verify_io(io)));
 }
 
-static void verity_tasklet(unsigned long data)
+static void verity_bh_work(struct work_struct *w)
 {
-	struct dm_verity_io *io = (struct dm_verity_io *)data;
+	struct dm_verity_io *io = container_of(w, struct dm_verity_io, bh_work);
 	int err;
 
 	io->in_tasklet = true;
@@ -675,8 +675,8 @@ static void verity_end_io(struct bio *bio)
 	}
 
 	if (static_branch_unlikely(&use_tasklet_enabled) && io->v->use_tasklet) {
-		tasklet_init(&io->tasklet, verity_tasklet, (unsigned long)io);
-		tasklet_schedule(&io->tasklet);
+		INIT_WORK(&io->bh_work, verity_bh_work);
+		queue_work(system_bh_wq, &io->bh_work);
 	} else {
 		INIT_WORK(&io->work, verity_work);
 		queue_work(io->v->verify_wq, &io->work);
drivers/md/dm-verity.h +0 -0
index f9d522c870e6..7c16f834f31a 100644

drivers/md/dm-verity.h struct dm_verity_io { · line 89

Quoted 8 lines
drivers/md/dm-verity.h +1 -1
@@ -83,7 +83,7 @@ struct dm_verity_io {
 	struct bvec_iter iter;
 
 	struct work_struct work;
-	struct tasklet_struct tasklet;
+	struct work_struct bh_work;
 
 	/*
 	 * Three variably-size fields follow this struct:

Do we really need two separate work_structs here? They are never submitted concurrently, so I think that one would be enough. Or, am I missing something?

Mikulas

author maintainer
#38 lore raw

Re: [PATCH 8/8] dm-verity: Convert from tasklet to BH workqueue

Hello,

Quoted 7 lines · Mikulas Patocka
Quoted 7 lines
drivers/md/dm-verity.h +1 -1
@@ -83,7 +83,7 @@ struct dm_verity_io {
 	struct bvec_iter iter;
 
 	struct work_struct work;
-	struct tasklet_struct tasklet;
+	struct work_struct bh_work;
 
Quoted 6 lines

drivers/md/dm-verity.h struct dm_verity_io { · line 89

Quoted 2 lines
drivers/md/dm-verity.h +0 -0
 	/*
 	 * Three variably-size fields follow this struct:

Do we really need two separate work_structs here? They are never submitted concurrently, so I think that one would be enough. Or, am I missing something?

I don't know, so just did the dumb thing. If the caller always guarantees that the work items are never queued at the same time, reusing is fine. However, the followings might be useful to keep on mind:

- work_struct is pretty small - 4 pointers.

- INIT_WORK() on a queued work item isn't gonna be pretty.

- Flushing and no-concurrent-execution guarantee are broken on INIT_WORK(). e.g. If you queue_work(), INIT_WORK(), flush_work(), the flush isn't actually going to wait for the work item to finish. Also, if you do queue_work(), INIT_WORK(), queue_work(), the two queued work item instances may end up running concurrently.

Muxing a single work item carries more risks of subtle bugs, but in some cases, the way it's used is clear (e.g. sequential chaining) and that's fine.

Thanks.

Signature · 1 line
tejun
maintainer
#39 lore raw

Re: [PATCH 8/8] dm-verity: Convert from tasklet to BH workqueue

Quoted 31 lines · Tejun Heo

Hello,

Quoted 7 lines · Mikulas Patocka
Quoted 7 lines
drivers/md/dm-verity.h +1 -1
@@ -83,7 +83,7 @@ struct dm_verity_io {
 	struct bvec_iter iter;
 
 	struct work_struct work;
-	struct tasklet_struct tasklet;
+	struct work_struct bh_work;
 
Quoted 6 lines

drivers/md/dm-verity.h struct dm_verity_io { · line 89

Quoted 2 lines
drivers/md/dm-verity.h +0 -0
 	/*
 	 * Three variably-size fields follow this struct:

Do we really need two separate work_structs here? They are never submitted concurrently, so I think that one would be enough. Or, am I missing something?

I don't know, so just did the dumb thing. If the caller always guarantees that the work items are never queued at the same time, reusing is fine. However, the followings might be useful to keep on mind:

- work_struct is pretty small - 4 pointers.

- INIT_WORK() on a queued work item isn't gonna be pretty.

- Flushing and no-concurrent-execution guarantee are broken on INIT_WORK(). e.g. If you queue_work(), INIT_WORK(), flush_work(), the flush isn't actually going to wait for the work item to finish. Also, if you do queue_work(), INIT_WORK(), queue_work(), the two queued work item instances may end up running concurrently.

Quoted 3 lines

Muxing a single work item carries more risks of subtle bugs, but in some cases, the way it's used is clear (e.g. sequential chaining) and that's fine.

The code doesn't call INIT_WORK() on a queued work item and it doesn't flush the workqueue (it destroys it only in a situation when there are no work items running) so I think it's safe to use just one work_struct.

Mikulas

#40 lore raw

Re: [PATCH 8/8] dm-verity: Convert from tasklet to BH workqueue

In reply to Tejun Heo
Quoted 3 lines · Tejun Heo

I don't know, so just did the dumb thing. If the caller always guarantees that the work items are never queued at the same time, reusing is fine.

So the reason I thought it would be a good cleanup to introduce that "atomic" workqueue thing (now "bh") was that this case literally has a switch between "use tasklets' or "use workqueues".

So it's not even about "reusing" the workqueue, it's literally a matter of making it always just use workqueues, and the switch then becomes just *which* workqueue to use - system or bh.

In fact, I suspect there is very little reason ever to *not* just use the bh one, and even the switch could be removed.

Because I think the only reason the "workqueue of tasklet" choice existed in the first place was that workqueues were the "proper" data structure, and the tasklet case was added later as a latency hack, and everybody knew that tasklets were deprecated.

Linus

author maintainer
#41 lore raw

Re: [PATCH 8/8] dm-verity: Convert from tasklet to BH workqueue

Hello, Linus.

Quoted 5 lines · Linus Torvalds
Quoted 3 lines · Tejun Heo

I don't know, so just did the dumb thing. If the caller always guarantees that the work items are never queued at the same time, reusing is fine.

Quoted 7 lines

So the reason I thought it would be a good cleanup to introduce that "atomic" workqueue thing (now "bh") was that this case literally has a switch between "use tasklets' or "use workqueues".

So it's not even about "reusing" the workqueue, it's literally a matter of making it always just use workqueues, and the switch then becomes just *which* workqueue to use - system or bh.

Yeah, that's how the dm-crypt got converted. The patch just before this one. This one probably can be converted the same way. I don't see the work item being re-initialized. It probably is better to initialize the work item together with the enclosing struct and then just queue it when needed.

Mikulas, I couldn't decide what to do with the "try_verify_in_tasklet" option and just decided to do the minimal thing hoping that someone more familiar with the code can take over the actual conversion. How much of user interface commitment is that? Should it be renamed or would it be better to leave it be?

Thanks.

Signature · 1 line
tejun
maintainer
#42 lore raw

Re: [PATCH 8/8] dm-verity: Convert from tasklet to BH workqueue

Quoted 12 lines · Tejun Heo

Hello, Linus.

Quoted 9 lines · Linus Torvalds
Quoted 3 lines · Tejun Heo

I don't know, so just did the dumb thing. If the caller always guarantees that the work items are never queued at the same time, reusing is fine.

So the reason I thought it would be a good cleanup to introduce that "atomic" workqueue thing (now "bh") was that this case literally has a switch between "use tasklets' or "use workqueues".

Quoted 8 lines
Quoted 3 lines

So it's not even about "reusing" the workqueue, it's literally a matter of making it always just use workqueues, and the switch then becomes just *which* workqueue to use - system or bh.

Yeah, that's how the dm-crypt got converted. The patch just before this one. This one probably can be converted the same way. I don't see the work item being re-initialized. It probably is better to initialize the work item together with the enclosing struct and then just queue it when needed.

Sounds good.

Quoted 5 lines

Mikulas, I couldn't decide what to do with the "try_verify_in_tasklet" option and just decided to do the minimal thing hoping that someone more familiar with the code can take over the actual conversion. How much of user interface commitment is that? Should it be renamed or would it be better to leave it be?

cryptsetup did add support for it, so I think it worthwhile to preserve the option; but it'd be fine to have it just be a backward compatible alias for a more appropriately named option?

Mike

maintainer
#43 lore raw

Re: [PATCH 8/8] dm-verity: Convert from tasklet to BH workqueue

Quoted 32 lines · Mike Snitzer
Quoted 12 lines · Tejun Heo

Hello, Linus.

Quoted 9 lines · Linus Torvalds
Quoted 3 lines · Tejun Heo

I don't know, so just did the dumb thing. If the caller always guarantees that the work items are never queued at the same time, reusing is fine.

So the reason I thought it would be a good cleanup to introduce that "atomic" workqueue thing (now "bh") was that this case literally has a switch between "use tasklets' or "use workqueues".

Quoted 8 lines
Quoted 3 lines

So it's not even about "reusing" the workqueue, it's literally a matter of making it always just use workqueues, and the switch then becomes just *which* workqueue to use - system or bh.

Yeah, that's how the dm-crypt got converted. The patch just before this one. This one probably can be converted the same way. I don't see the work item being re-initialized. It probably is better to initialize the work item together with the enclosing struct and then just queue it when needed.

Sounds good.

Quoted 5 lines

Mikulas, I couldn't decide what to do with the "try_verify_in_tasklet" option and just decided to do the minimal thing hoping that someone more familiar with the code can take over the actual conversion. How much of user interface commitment is that? Should it be renamed or would it be better to leave it be?

Quoted 3 lines

cryptsetup did add support for it, so I think it worthwhile to preserve the option; but it'd be fine to have it just be a backward compatible alias for a more appropriately named option?

Hey Tejun,

I'm not sure where things stand with the 6.9 workqueue changes to add BH workqueue. I had a look at your various branches and I'm not seeing where you might have staged any conversion patches (like this dm-verity one).

I just staged various unrelated dm-verity and dm-crypt 6.8 fixes from Mikulas that I'll be sending to Linus later this week (for v6.8-rc6). Those changes required rebasing 'dm-6.9' because of conflicts, here is the dm-6.9 branch:
https://git.kernel.org/pub/scm/linux/kernel/git/device-mapper/linux-dm.git/log/?h=dm-6.9

So we'll definitely need to rebase your changes on dm-6.9 to convert dm-crypt and dm-verity over to your BH workqueue. Are you OK with doing that or would you prefer I merge some 6.9 workqueue branch that you have into dm-6.9? And then Mikulas and I work to make the required DM target conversion changes?

However you'd like to do it: please let me know where you have the latest 6.9 code the adds BH workqueue (and if you have DM target conversion code that reflects your latest).

Thanks,
Mike

author maintainer
#44 lore raw

Re: [PATCH 8/8] dm-verity: Convert from tasklet to BH workqueue

Hello, Mike.

Quoted 4 lines · Mike Snitzer

I'm not sure where things stand with the 6.9 workqueue changes to add BH workqueue. I had a look at your various branches and I'm not seeing where you might have staged any conversion patches (like this dm-verity one).

Yeah, the branch is for-6.9-bh-conversions in the wq tree but I haven't queued the DM patches yet. Wanted to make sure the perf signal from TCP conversion is okay first. FWIW, while Eric still has concerns, the initial test didn't show any appreciable regression with production memcache workload on our side.

Quoted 11 lines

I just staged various unrelated dm-verity and dm-crypt 6.8 fixes from Mikulas that I'll be sending to Linus later this week (for v6.8-rc6). Those changes required rebasing 'dm-6.9' because of conflicts, here is the dm-6.9 branch:
https://git.kernel.org/pub/scm/linux/kernel/git/device-mapper/linux-dm.git/log/?h=dm-6.9

So we'll definitely need to rebase your changes on dm-6.9 to convert dm-crypt and dm-verity over to your BH workqueue. Are you OK with doing that or would you prefer I merge some 6.9 workqueue branch that you have into dm-6.9? And then Mikulas and I work to make the required DM target conversion changes?

Oh, if you don't mind, it'd be best if you could pull wq/for-6.9 into dm-6.9 and do the conversions there.

Thank you.

Signature · 1 line
tejun
maintainer
#45 lore raw

Re: [PATCH 8/8] dm-verity: Convert from tasklet to BH workqueue

Quoted 26 lines · Tejun Heo

Hello, Mike.

Quoted 4 lines · Mike Snitzer

I'm not sure where things stand with the 6.9 workqueue changes to add BH workqueue. I had a look at your various branches and I'm not seeing where you might have staged any conversion patches (like this dm-verity one).

Yeah, the branch is for-6.9-bh-conversions in the wq tree but I haven't queued the DM patches yet. Wanted to make sure the perf signal from TCP conversion is okay first. FWIW, while Eric still has concerns, the initial test didn't show any appreciable regression with production memcache workload on our side.

Quoted 11 lines

I just staged various unrelated dm-verity and dm-crypt 6.8 fixes from Mikulas that I'll be sending to Linus later this week (for v6.8-rc6). Those changes required rebasing 'dm-6.9' because of conflicts, here is the dm-6.9 branch:
https://git.kernel.org/pub/scm/linux/kernel/git/device-mapper/linux-dm.git/log/?h=dm-6.9

So we'll definitely need to rebase your changes on dm-6.9 to convert dm-crypt and dm-verity over to your BH workqueue. Are you OK with doing that or would you prefer I merge some 6.9 workqueue branch that you have into dm-6.9? And then Mikulas and I work to make the required DM target conversion changes?

Quoted 4 lines

Oh, if you don't mind, it'd be best if you could pull wq/for-6.9 into dm-6.9 and do the conversions there.

Thank you.

I've rebased and made the changes available in this dm-6.9-bh-wq branch:
https://git.kernel.org/pub/scm/linux/kernel/git/device-mapper/linux-dm.git/log/?h=dm-6.9-bh-wq

I left them both DM conversion commits attributed to use despite the rebase (and churn of renames I did in the dm-verity commit); hopefully you're fine with that but if not I can split the renames out.

I successfully tested using the cryptsetup testsuite ('make check'). And I've also added these changes to linux-next, via linux-dm.git's 'for-next':
https://git.kernel.org/pub/scm/linux/kernel/git/device-mapper/linux-dm.git/log/?h=for-next

Mikulas, feel free to review/test and possibly add another patch for dm-verity that only uses a single work_struct in struct dm_verity_io

Thanks,
Mike

author maintainer
#46 lore raw

Re: [PATCH 8/8] dm-verity: Convert from tasklet to BH workqueue

Quoted 6 lines · Mike Snitzer

I've rebased and made the changes available in this dm-6.9-bh-wq branch:
https://git.kernel.org/pub/scm/linux/kernel/git/device-mapper/linux-dm.git/log/?h=dm-6.9-bh-wq

I left them both DM conversion commits attributed to use despite the rebase (and churn of renames I did in the dm-verity commit); hopefully you're fine with that but if not I can split the renames out.

Looks good to me.

Thanks.

Signature · 1 line
tejun
maintainer
#47 lore raw

Re: [PATCH 8/8] dm-verity: Convert from tasklet to BH workqueue

In reply to Linus Torvalds
Quoted 5 lines · Linus Torvalds
Quoted 3 lines · Tejun Heo

I don't know, so just did the dumb thing. If the caller always guarantees that the work items are never queued at the same time, reusing is fine.

Quoted 7 lines

So the reason I thought it would be a good cleanup to introduce that "atomic" workqueue thing (now "bh") was that this case literally has a switch between "use tasklets' or "use workqueues".

So it's not even about "reusing" the workqueue, it's literally a matter of making it always just use workqueues, and the switch then becomes just *which* workqueue to use - system or bh.

DM generally always use dedicated workqueues instead of the system.

The dm-crypt tasklet's completion path did punt to the workqueue otherwise there was use-after-free of the per-bio-data that included the tasklet. And for verity there was fallback to workqueue if tasklet-based verification failed. Didn't inspire confidence.

Quoted 7 lines

In fact, I suspect there is very little reason ever to *not* just use the bh one, and even the switch could be removed.

Because I think the only reason the "workqueue of tasklet" choice existed in the first place was that workqueues were the "proper" data structure, and the tasklet case was added later as a latency hack, and everybody knew that tasklets were deprecated.

Correct, abusing tasklets was a very contrived latency optimization. Happy to see it all go away! (hindsight: it never should have gone in).

Mike

author maintainer
#48 lore raw
In reply to Tejun Heo

Oops, forgot to cc Lai.

Quoted 15 lines · Tejun Heo

Hello,

The only generic interface to execute asynchronously in the BH context is tasklet; however, it's marked deprecated and has some design flaws such as the execution code accessing the tasklet item after the execution is complete which can lead to subtle use-after-free in certain usage scenarios and less-developed flush and cancel mechanisms.

Mikulas Patocka recently reported that dm-crypt and dm-crypt are affected by the access-after-completion issue and suggested adding TASKLET_STATE_ONESHOT flag which selectively removes post-completion access while significantly limiting how the tasklet can be used in the following thread:

http://lkml.kernel.org/r/[email protected]

Quoted 8 lines

Linus didn't like the approach and suggested extending workqueue to support execution from atomic context:

http://lkml.kernel.org/r/CAHk-=wjDW53w4-YcSmgKC5RruiRLHmJ1sXeYdp_ZgVoBw=5byA@mail.gmail.com

As suggested, this patchset implements BH workqueues which are like regular workqueues but executes work items in the BH (softirq) context and converts several tasklet users.

Lai, the patchset is at:

http://lkml.kernel.org/r/[email protected]

I'd really appreciate if you can take a look.

Thanks.

Signature · 1 line
tejun
#49 lore raw
In reply to Tejun Heo
Quoted 1 line · Tejun Heo

Hello,

Hi,

Quoted 13 lines

As suggested, this patchset implements BH workqueues which are like regular workqueues but executes work items in the BH (softirq) context and converts several tasklet users.

- The name bh is used instead of the suggested atomic as it's more in line with widely used execution context interface - local_bh_enable/disable() and friends.

- The system default BH workqueues - system_bh_wq and system_bh_highpri_wq - are provided. As queue-wide flushing doesn't exist in tasklet, all existing tasklet users should be able to use the system BH workqueues without creating their own.

Quoted 4 lines

- BH workqueues currently use tasklet to run the work items to avoid priority inversions involving tasklet_hi and WQ_BH | WQ_HIGHPRI. Once all tasklet users are converted, tasklet code can be removed and BH workqueues can take over its softirqs.

If one context creates multiple work item which are then moved to tasklet I don't see the difference vs workqueue with a bh_disable() around it.
Looking at the USB changes, I would prefer to see it converted to threaded interrupts instead of using tasklet or workqueue. Both approaches (current tasklet, suggested workqueue) lose the original context where the request was created. Having threaded interrupts would allow to keep everything in the same "context" so you could prioritize according to your needs.

Sebastian

author maintainer
#50 lore raw

Hello, Sebastian.

Quoted 3 lines · Sebastian Andrzej Siewior

If one context creates multiple work item which are then moved to tasklet I don't see the difference vs workqueue with a bh_disable() around it.

The main difference is that it avoids scheduling latencies in scenarios where softirq isn't heavily loaded.

Quoted 6 lines

Looking at the USB changes, I would prefer to see it converted to threaded interrupts instead of using tasklet or workqueue. Both approaches (current tasklet, suggested workqueue) lose the original context where the request was created. Having threaded interrupts would allow to keep everything in the same "context" so you could prioritize according to your needs.

That's great. If threaded IRQs or even regular workqueues are suitable, that should be fine. This conversion is just targeted at the use cases which can benefit from executing in the softirq context.

Thanks.

Signature · 1 line
tejun
Tested-by on v1 1/1
#51 lore raw
In reply to Tejun Heo
Quoted 49 lines

The only generic interface to execute asynchronously in the BH context is tasklet; however, it's marked deprecated and has some design flaws such as the execution code accessing the tasklet item after the execution is complete which can lead to subtle use-after-free in certain usage scenarios and less-developed flush and cancel mechanisms.

Mikulas Patocka recently reported that dm-crypt and dm-crypt are affected by the access-after-completion issue and suggested adding TASKLET_STATE_ONESHOT flag which selectively removes post-completion access while significantly limiting how the tasklet can be used in the following thread:

http://lkml.kernel.org/r/[email protected]

Linus didn't like the approach and suggested extending workqueue to support execution from atomic context:

http://lkml.kernel.org/r/CAHk-=wjDW53w4-YcSmgKC5RruiRLHmJ1sXeYdp_ZgVoBw=5byA@mail.gmail.com

As suggested, this patchset implements BH workqueues which are like regular workqueues but executes work items in the BH (softirq) context and converts several tasklet users.

- The name bh is used instead of the suggested atomic as it's more in line with widely used execution context interface - local_bh_enable/disable() and friends.

- The system default BH workqueues - system_bh_wq and system_bh_highpri_wq - are provided. As queue-wide flushing doesn't exist in tasklet, all existing tasklet users should be able to use the system BH workqueues without creating their own.

- BH workqueues currently use tasklet to run the work items to avoid priority inversions involving tasklet_hi and WQ_BH | WQ_HIGHPRI. Once all tasklet users are converted, tasklet code can be removed and BH workqueues can take over its softirqs.

This patchset is on top of wq/for-6.9 (aae17ebb53c ("workqueue: Avoid using isolated cpus' timers on queue_delayed_work")) and contains the following eight patches.

0001-workqueue-Update-lock-debugging-code.patch 0002-workqueue-Factor-out-init_cpu_worker_pool.patch 0003-workqueue-Implement-BH-workqueues-to-eventually-repl.patch 0004-backtracetest-Convert-from-tasklet-to-BH-workqueue.patch 0005-usb-core-hcd-Convert-from-tasklet-to-BH-workqueue.patch 0006-net-tcp-tsq-Convert-from-tasklet-to-BH-workqueue.patch 0007-dm-crypt-Convert-from-tasklet-to-BH-workqueue.patch 0008-dm-verity-Convert-from-tasklet-to-BH-workqueue.patch

Quoted 9 lines

0001-0003 prepare and implement BH workqueues.

0004-0008 convert some tasklet users to BH workqueue. The conversions are relatively straightforward but are in descending order of confidence.

The patchset is also available in the following git branch.

git://git.kernel.org/pub/scm/linux/kernel/git/tj/wq.git wq-bh-v1

Thank you for doing this work.

Tested the series, things look good. I am going to run LTP and Unixbench on the kernel.

Thanks.

Quoted 20 lines

diffstat follows. Thanks.

 Documentation/core-api/workqueue.rst |   29 ++++-
 drivers/md/dm-crypt.c                |   36 -------
 drivers/md/dm-verity-target.c        |    8 -
 drivers/md/dm-verity.h               |    2
 drivers/usb/core/hcd.c               |   23 ++--
 include/linux/usb/hcd.h              |    2
 include/linux/workqueue.h            |    9 +
 include/net/tcp.h                    |    2
 kernel/backtracetest.c               |   18 +--
 kernel/workqueue.c                   |  312 ++++++++++++++++++++++++++++++++++++++++++++++++--------------
 kernel/workqueue_internal.h          |    3
 net/ipv4/tcp.c                       |    2
 net/ipv4/tcp_output.c                |   36 +++----
 tools/workqueue/wq_dump.py           |   11 +-
 14 files changed, 335 insertions(+), 158 deletions(-)

--
tejun

Signature · 1 line
       - Allen
author maintainer
#52 lore raw
In reply to Tejun Heo
Quoted 8 lines · Tejun Heo

0001-workqueue-Update-lock-debugging-code.patch 0002-workqueue-Factor-out-init_cpu_worker_pool.patch 0003-workqueue-Implement-BH-workqueues-to-eventually-repl.patch 0004-backtracetest-Convert-from-tasklet-to-BH-workqueue.patch 0005-usb-core-hcd-Convert-from-tasklet-to-BH-workqueue.patch 0006-net-tcp-tsq-Convert-from-tasklet-to-BH-workqueue.patch 0007-dm-crypt-Convert-from-tasklet-to-BH-workqueue.patch 0008-dm-verity-Convert-from-tasklet-to-BH-workqueue.patch

Applied 0001-0003 to wq/for-6.9. Applied 0004-0005 to wq/for-6.9-bh-conversions. Will proceed on 0006 and other conversions after more perf testing.

Thanks.

Signature · 1 line
tejun
#53 lore raw
Quoted 10 lines · Tejun Heo
Quoted 8 lines

0001-workqueue-Update-lock-debugging-code.patch 0002-workqueue-Factor-out-init_cpu_worker_pool.patch 0003-workqueue-Implement-BH-workqueues-to-eventually-repl.patch 0004-backtracetest-Convert-from-tasklet-to-BH-workqueue.patch 0005-usb-core-hcd-Convert-from-tasklet-to-BH-workqueue.patch 0006-net-tcp-tsq-Convert-from-tasklet-to-BH-workqueue.patch 0007-dm-crypt-Convert-from-tasklet-to-BH-workqueue.patch 0008-dm-verity-Convert-from-tasklet-to-BH-workqueue.patch

Quoted 6 lines

Applied 0001-0003 to wq/for-6.9. Applied 0004-0005 to wq/for-6.9-bh-conversions. Will proceed on 0006 and other conversions after more perf testing.

Thanks.

Thank you. I am basing my work on the branch you have pushed.(or-6.9-bh-conversions)
https://git.kernel.org/pub/scm/linux/kernel/git/tj/wq.git/log/?h=for-6.9-bh-conversions

In the order of priority, I have started converting drivers/media/*, drivers/dma/* followed by drivers/net/*
which constitutes the majority. Putting my plan out here so that the work is not duplicated.
I will write back in a day and share the branch for review.

W.r.t the conversion, there are drivers which call tasklet_[disable/enable](), which I suppose
can be ignored in the case of workqueues, I am not entirely sure if this is the right approach.
Please correct me if I am wrong.

Thanks,

- Allen

author maintainer
#54 lore raw

Hello, Allen.

Quoted 9 lines · Allen

Thank you. I am basing my work on the branch you have pushed.(or-6.9-bh-conversions)
https://git.kernel.org/pub/scm/linux/kernel/git/tj/wq.git/log/?h=for-6.9-bh-conversions

In the order of priority, I have started converting drivers/media/*, drivers/dma/* followed by drivers/net/*
which constitutes the majority. Putting my plan out here so that the work is not duplicated.
I will write back in a day and share the branch for review.

That's great. Thanks.

Quoted 5 lines

W.r.t the conversion, there are drivers which call tasklet_[disable/enable](), which I suppose
can be ignored in the case of workqueues, I am not entirely sure if this is the right approach.
Please correct me if I am wrong.

I don't think we can ignore them. I was just looking at tasklet_kill() and thought we're good because that seemed to map well to cancel_work_sync(). workqueue doesn't have the counterpart for tasklet_[disable/enable](). I'll look through them and think on it.

Thanks.

Signature · 1 line
tejun
#55 lore raw

Tejun,

Quoted 19 lines · Allen
Quoted 9 lines

Thank you. I am basing my work on the branch you have pushed.(or-6.9-bh-conversions)
https://git.kernel.org/pub/scm/linux/kernel/git/tj/wq.git/log/?h=for-6.9-bh-conversions

In the order of priority, I have started converting drivers/media/*, drivers/dma/* followed by drivers/net/*
which constitutes the majority. Putting my plan out here so that the work is not duplicated.
I will write back in a day and share the branch for review.

That's great. Thanks.

Quoted 5 lines

W.r.t the conversion, there are drivers which call tasklet_[disable/enable](), which I suppose
can be ignored in the case of workqueues, I am not entirely sure if this is the right approach.
Please correct me if I am wrong.

Quoted 5 lines

I don't think we can ignore them. I was just looking at tasklet_kill() and thought we're good because that seemed to map well to cancel_work_sync(). workqueue doesn't have the counterpart for tasklet_[disable/enable](). I'll look through them and think on it.

Okay, I will look into it too. I have these rough and completely untested functions. All I am trying to do is to match what tasklets are currently doing.

static inline void workqueue_disable(struct work_struct *work)

{
     cancel_work_sync(work);
     flush_workqueue(system_bh_wq);
     smp_mb();
}

static inline void workqueue_enable(struct work_struct *work)

{
    smp_mb__before_atomic();

// atomic_inc(&work->data);
}

I have to figure out a better way to handle atomic_inc() in the enable function.

Thanks.

- Allen

#56 lore raw

Tejun,

Quoted 5 lines
Quoted 5 lines · Allen
Quoted 4 lines

Thank you. I am basing my work on the branch you have pushed.(or-6.9-bh-conversions)
https://git.kernel.org/pub/scm/linux/kernel/git/tj/wq.git/log/?h=for-6.9-bh-conversions

Quoted 8 lines
Quoted 8 lines
Quoted 5 lines

In the order of priority, I have started converting drivers/media/*, drivers/dma/* followed by drivers/net/*
which constitutes the majority. Putting my plan out here so that the work is not duplicated.
I will write back in a day and share the branch for review.

That's great. Thanks.

I have the first set of conversions ready. Kindly review the patchset
before I post it to the list. These convert drivers/dma/* to use the new Atomic WQ mechanism.

Based on
https://git.kernel.org/pub/scm/linux/kernel/git/tj/wq.git for-6.9-bh-conversions

https://github.com/allenpais/for-6.9-bh-conversions

I am holding on to the patch that converts drivers/media/*, as I haven't found
a right way to replace tasklet_[disable/enable] api's. The rest should be ready in a day or two.

Thanks,
- Allen

author maintainer
#57 lore raw
Quoted 5 lines · Allen

https://github.com/allenpais/for-6.9-bh-conversions

I am holding on to the patch that converts drivers/media/*, as I haven't found
a right way to replace tasklet_[disable/enable] api's. The rest should be ready in a day or two.

Yeah, we'll need to add something to workqueue to support that. As for the rest, looking at the code, I think tasklet_kill() should be converted to cancel_work_sync(), not flush_work().

Thanks.

Signature · 1 line
tejun
#58 lore raw
Quoted 4 lines · Tejun Heo
Quoted 2 lines · Allen

https://github.com/allenpais/for-6.9-bh-conversions

Quoted 8 lines
Quoted 3 lines

I am holding on to the patch that converts drivers/media/*, as I haven't found
a right way to replace tasklet_[disable/enable] api's. The rest should be ready in a day or two.

Yeah, we'll need to add something to workqueue to support that. As for the rest, looking at the code, I think tasklet_kill() should be converted to cancel_work_sync(), not flush_work().

Ah, Thanks for pointing that out. I will update it and get that fixed.

Thanks for the review.

Signature · 1 line
       - Allen