bio completion in task enhancements / experiments
Subsystems (5)
Hi all,
this series builds on top of:
Subject: [PATCH RFC v5 0/3] block: enable RWF_DONTCACHE for block devices
which I fixed up to apply to linux-next. If you want to seriously review or test this, you're best off using the git branch here:
https://git.infradead.org/?p=users/hch/misc.git;a=shortlog;h=refs/heads/bio-task-completion
it first makes the complete in task interface more flexible so that it can also be used from inside the ->bi_end_io handlers, which we'll need for a few uses cases. The second patch fixes the offload condition, the next two then convert to uses in iomap added in the current merge window over to the interface.
The last patch plays with the implementation and reuses concepts from erofs to reduce the completion latency at the expense of more always alive threads.
There's a few other places that could benefit from this, like erofs decompression, PI verification in the block and file systems paths, or fscrypt decryption.
Diffstat:
block/bio.c | 93 ++++++++++++++++++++++++++++++++++++++++++++ block/fops.c | 5 +- fs/buffer.c | 25 ++++++++++- fs/iomap/bio.c | 44 -------------------- fs/iomap/ioend.c | 53 +++---------------------- fs/xfs/xfs_aops.c | 4 - include/linux/bio.h | 28 +++++++++++++ include/linux/blk_types.h | 6 ++ include/linux/buffer_head.h | 5 ++ 9 files changed, 165 insertions(+), 98 deletions(-)
PATCH 1/8 block: add BIO_COMPLETE_IN_TASK for task-context completion
From: Tal Zussman <[email protected]>
Some bio completion handlers need to run in task context but bio_endio() can be called from IRQ context (e.g. buffer_head writeback). Add a BIO_COMPLETE_IN_TASK flag that bio submitters can set to request task-context completion of their bi_end_io callback.
When bio_endio() sees this flag and is running in non-task context, it queues the bio to a per-cpu lockless list and schedules a delayed work item to call bi_end_io() from task context. The delayed work uses a 1-jiffie delay to allow batches of completions to accumulate before processing. A CPU hotplug dead callback drains any remaining bios from the departing CPU's batch.
This will be used to enable RWF_DONTCACHE for block devices, and could be used for other subsystems like fscrypt that need task-context bio completion.
--- block/bio.c | 83 ++++++++++++++++++++++++++++++++++++++- include/linux/blk_types.h | 7 +++- 2 files changed, 88 insertions(+), 2 deletions(-)
block/bio.c +82 -1
@@ -19,6 +19,7 @@#include <linux/blk-crypto.h> #include <linux/xarray.h> #include <linux/kmemleak.h> +#include <linux/llist.h> #include <trace/events/block.h> #include "blk.h"
@@ -1716,6 +1717,51 @@ void bio_check_pages_dirty(struct bio *bio) }EXPORT_SYMBOL_GPL (bio_check_pages_dirty); +struct bio_complete_batch { +struct llist_head list; +struct delayed_work work; +int cpu; +}; + +static DEFINE_PER_CPU(struct bio_complete_batch, bio_complete_batch); +static struct workqueue_struct *bio_complete_wq; + +static void bio_complete_work_fn (struct work_struct *w) +{ +struct delayed_work *dw =to_delayed_work (w); +struct bio_complete_batch *batch = + container_of(dw, struct bio_complete_batch, work); +struct llist_node *node; +struct bio *bio, *next; + +do { + node =llist_del_all (&batch->list); +if (!node) +break ; + + node =llist_reverse_order (node); +llist_for_each_entry_safe (bio, next, node, bi_llist) + bio->bi_end_io (bio); + +if (need_resched ()) { +if (!llist_empty (&batch->list)) +mod_delayed_work_on (batch->cpu, + bio_complete_wq, + &batch->work,0 ); +break ; + } + }while (1 ); +} + +static void bio_queue_completion (struct bio *bio) +{ +struct bio_complete_batch *batch =this_cpu_ptr (&bio_complete_batch); + +if (llist_add (&bio->bi_llist, &batch->list)) +mod_delayed_work_on (batch->cpu, bio_complete_wq, + &batch->work,1 ); +} +static inline bool bio_remaining_done (struct bio *bio) {/*
@@ -1790,7 +1836,9 @@ void bio_endio(struct bio *bio) }#endif -if (bio->bi_end_io) +if (!in_task () &&bio_flagged (bio,BIO_COMPLETE_IN_TASK )) +bio_queue_completion (bio); +else if (bio->bi_end_io) bio->bi_end_io (bio); }EXPORT_SYMBOL (bio_endio);
@@ -1976,6 +2024,24 @@ int bioset_init(struct bio_set *bs, }EXPORT_SYMBOL (bioset_init); +/* +* Drain a dead CPU's deferred bio completions. +*/ +static int bio_complete_batch_cpu_dead (unsigned int cpu) +{ +struct bio_complete_batch *batch = +per_cpu_ptr (&bio_complete_batch, cpu); +struct llist_node *node; +struct bio *bio, *next; + + node =llist_del_all (&batch->list); + node =llist_reverse_order (node); +llist_for_each_entry_safe (bio, next, node, bi_llist) + bio->bi_end_io (bio); + +return 0 ; +} +static int __init init_bio (void ) {int i;
@@ -1990,6 +2056,21 @@ static int __init init_bio(void) SLAB_HWCACHE_ALIGN | SLAB_PANIC, NULL); } +for_each_possible_cpu (i) { +struct bio_complete_batch *batch = +per_cpu_ptr (&bio_complete_batch, i); + +init_llist_head (&batch->list); +INIT_DELAYED_WORK (&batch->work, bio_complete_work_fn); + batch->cpu = i; + } + + bio_complete_wq =alloc_workqueue ("bio_complete" ,WQ_MEM_RECLAIM ,0 ); +if (!bio_complete_wq) +panic ("bio: can't allocate bio_complete workqueue\n" ); + +cpuhp_setup_state (CPUHP_BP_PREPARE_DYN ,"block/bio:complete:dead" , +NULL , bio_complete_batch_cpu_dead);cpuhp_setup_state_multi (CPUHP_BIO_DEAD ,"block/bio:dead" ,NULL , bio_cpu_dead);
include/linux/blk_types.h +6 -1
@@ -11,6 +11,7 @@#include <linux/device.h> #include <linux/ktime.h> #include <linux/rw_hint.h> +#include <linux/llist.h> struct bio_set ;struct bio ;
@@ -208,7 +209,10 @@ typedef unsigned int blk_qc_t;* stacking drivers) */ struct bio { -struct bio *bi_next;/* request queue link */ +union { +struct bio *bi_next;/* request queue link */ +struct llist_node bi_llist;/* deferred completion */ + };struct block_device *bi_bdev;blk_opf_t bi_opf;/* bottom bits REQ_OP, top bits * req_flags.
@@ -322,6 +326,7 @@ enum {BIO_REMAPPED ,BIO_ZONE_WRITE_PLUGGING ,/* bio handled through zone write plugging */ BIO_EMULATES_ZONE_APPEND ,/* bio emulates a zone append operation */ +BIO_COMPLETE_IN_TASK ,/* complete bi_end_io() in task context */ BIO_FLAG_LAST };
Signature · 1 line
2.47.3
PATCH 2/8 iomap: use BIO_COMPLETE_IN_TASK for dropbehind writeback
In reply to Christoph HellwigFrom: Tal Zussman <[email protected]>
Set BIO_COMPLETE_IN_TASK on iomap writeback bios when a dropbehind folio is added. This ensures that bi_end_io runs in task context, where folio_end_dropbehind() can safely invalidate folios.
With the bio layer now handling task-context deferral generically, IOMAP_IOEND_DONTCACHE is no longer needed, as XFS no longer needs to route DONTCACHE ioends through its completion workqueue. Remove the flag and its NOMERGE entry.
Without the NOMERGE, regular I/Os that get merged with a dropbehind folio will also have their completion deferred to task context.
--- fs/iomap/ioend.c | 5 +++-- fs/xfs/xfs_aops.c | 4 ---- include/linux/iomap.h | 6 +----- 3 files changed, 4 insertions(+), 11 deletions(-)
fs/iomap/ioend.c +3 -2
@@ -237,8 +237,6 @@ ssize_t iomap_add_to_ioend(struct iomap_writepage_ctx *wpc, struct folio *folio,if (wpc->iomap.flags &IOMAP_F_SHARED ) ioend_flags |=IOMAP_IOEND_SHARED ; -if (folio_test_dropbehind (folio)) - ioend_flags |=IOMAP_IOEND_DONTCACHE ;if (pos == wpc->iomap.offset && (wpc->iomap.flags &IOMAP_F_BOUNDARY )) ioend_flags |=IOMAP_IOEND_BOUNDARY ;
@@ -255,6 +253,9 @@ ssize_t iomap_add_to_ioend(struct iomap_writepage_ctx *wpc, struct folio *folio,if (!bio_add_folio (&ioend->io_bio, folio, map_len, poff)) goto new_ioend; +if (folio_test_dropbehind (folio)) +bio_set_flag (&ioend->io_bio,BIO_COMPLETE_IN_TASK ); +/* * Clamp io_offset and io_size to the incore EOF so that ondisk * file size updates in the ioend completion are byte-accurate.
fs/xfs/xfs_aops.c +0 -4
@@ -511,10 +511,6 @@ xfs_ioend_needs_wq_completion(if (ioend->io_flags & (IOMAP_IOEND_UNWRITTEN |IOMAP_IOEND_SHARED ))return true; -/* Page cache invalidation cannot be done in irq context. */ -if (ioend->io_flags &IOMAP_IOEND_DONTCACHE ) -return true; -return false; }
include/linux/iomap.h +1 -5
@@ -399,16 +399,12 @@ sector_t iomap_bmap(struct address_space *mapping, sector_t bno,#define IOMAP_IOEND_BOUNDARY (1U << 2)/* is direct I/O */ #define IOMAP_IOEND_DIRECT (1U << 3) -/* is DONTCACHE I/O */ -#define IOMAP_IOEND_DONTCACHE (1U << 4) -/* * Flags that if set on either ioend prevent the merge of two ioends. * (IOMAP_IOEND_BOUNDARY also prevents merges, but only one-way) */ #define IOMAP_IOEND_NOMERGE_FLAGS \ - (IOMAP_IOEND_SHARED | IOMAP_IOEND_UNWRITTEN | IOMAP_IOEND_DIRECT | \ - IOMAP_IOEND_DONTCACHE) + (IOMAP_IOEND_SHARED | IOMAP_IOEND_UNWRITTEN | IOMAP_IOEND_DIRECT)/* * Structure for writeback I/O completions.
Signature · 1 line
2.47.3
PATCH 3/8 block: enable RWF_DONTCACHE for block devices
In reply to Christoph HellwigFrom: Tal Zussman <[email protected]>
Block device buffered reads and writes already pass through filemap_read() and iomap_file_buffered_write() respectively, both of which handle IOCB_DONTCACHE. Enable RWF_DONTCACHE for block device files by setting FOP_DONTCACHE in def_blk_fops.
For CONFIG_BUFFER_HEAD=y paths, add block_write_begin_iocb() which threads the kiocb through so that buffer_head-based I/O can use DONTCACHE behavior. The existing block_write_begin() is preserved as a wrapper that passes a NULL iocb. Set BIO_COMPLETE_IN_TASK in submit_bh_wbc() when the folio has dropbehind so that buffer_head writeback completions get deferred to task context.
CONFIG_BUFFER_HEAD=n paths are handled by the previously added iomap BIO_COMPLETE_IN_TASK support.
This support is useful for databases that operate on raw block devices, among other userspace applications.
--- block/fops.c | 5 +++-- fs/buffer.c | 22 +++++++++++++++++++--- include/linux/buffer_head.h | 3 +++ 3 files changed, 25 insertions(+), 5 deletions(-)
block/fops.c +3 -2
@@ -504,7 +504,8 @@ static int blkdev_write_begin(const struct kiocb *iocb, unsigned len, struct folio **foliop, void **fsdata) { - return block_write_begin(mapping, pos, len, foliop, blkdev_get_block); + return block_write_begin_iocb(iocb, mapping, pos, len, foliop, +blkdev_get_block ); } static int blkdev_write_end(const struct kiocb *iocb,
@@ -966,7 +967,7 @@ const struct file_operations def_blk_fops = { .splice_write = iter_file_splice_write, .fallocate = blkdev_fallocate, .uring_cmd = blkdev_uring_cmd, - .fop_flags = FOP_BUFFER_RASYNC, + .fop_flags = FOP_BUFFER_RASYNC | FOP_DONTCACHE, };static __init int blkdev_init (void )
fs/buffer.c +19 -3
@@ -2131,14 +2131,19 @@ EXPORT_SYMBOL(block_commit_write);* * The filesystem needs to handle block truncation upon failure. */ -int block_write_begin (struct address_space *mapping,loff_t pos,unsigned len, +int block_write_begin_iocb (const struct kiocb *iocb, +struct address_space *mapping,loff_t pos,unsigned len,struct folio **foliop,get_block_t *get_block) {pgoff_t index = pos >>PAGE_SHIFT ; +fgf_t fgp_flags =FGP_WRITEBEGIN ;struct folio *folio;int status; - folio =__filemap_get_folio (mapping, index,FGP_WRITEBEGIN , +if (iocb && iocb->ki_flags &IOCB_DONTCACHE ) + fgp_flags |=FGP_DONTCACHE ; + + folio =__filemap_get_folio (mapping, index, fgp_flags,mapping_gfp_mask (mapping));if (IS_ERR (folio))return PTR_ERR (folio);
@@ -2153,6 +2158,13 @@ int block_write_begin(struct address_space *mapping, loff_t pos, unsigned len, *foliop = folio;return status; } + +int block_write_begin (struct address_space *mapping,loff_t pos,unsigned len, +struct folio **foliop,get_block_t *get_block) +{ +return block_write_begin_iocb (NULL , mapping, pos, len, foliop, + get_block); +}EXPORT_SYMBOL (block_write_begin); int block_write_end(loff_t pos, unsigned len, unsigned copied,
@@ -2481,7 +2493,8 @@ int cont_write_begin(const struct kiocb *iocb, struct address_space *mapping, (*bytes)++; } -return block_write_begin (mapping, pos, len, foliop, get_block); +return block_write_begin_iocb (iocb, mapping, pos, len, foliop, + get_block); }EXPORT_SYMBOL (cont_write_begin);
@@ -2711,6 +2724,9 @@ static void submit_bh_wbc(blk_opf_t opf, struct buffer_head *bh,if (IS_ENABLED (CONFIG_FS_ENCRYPTION ))buffer_set_crypto_ctx (bio, bh,GFP_NOIO ); +if (folio_test_dropbehind (bh->b_folio)) +bio_set_flag (bio,BIO_COMPLETE_IN_TASK ); + bio->bi_iter.bi_sector = bh->b_blocknr * (bh->b_size >>9 ); bio->bi_write_hint = write_hint;
include/linux/buffer_head.h +3 -0
@@ -260,6 +260,9 @@ int block_read_full_folio(struct folio *, get_block_t *);bool block_is_partially_uptodate (struct folio *,size_t from,size_t count);int block_write_begin (struct address_space *mapping,loff_t pos,unsigned len,struct folio **foliop,get_block_t *get_block); +int block_write_begin_iocb (const struct kiocb *iocb, +struct address_space *mapping,loff_t pos,unsigned len, +struct folio **foliop,get_block_t *get_block);int __block_write_begin (struct folio *folio,loff_t pos,unsigned len,get_block_t *get_block);int block_write_end (loff_t pos,unsigned len,unsigned copied,struct folio *);
Signature · 1 line
2.47.3
PATCH 4/8 FOLD: block: change the defer in task context interface to be procedural
In reply to Christoph HellwigReplace the bio-flag based interface with an explicit bio_complete_in_task() API. The advantage is that this can also be called from inside the ->bi_end_io callback and thus dynamically. This will be important to use it for fserror reporting.
--- block/bio.c | 7 +++---- fs/buffer.c | 5 ++++- fs/iomap/ioend.c | 11 ++++++++--- include/linux/bio.h | 17 +++++++++++++++++ include/linux/blk_types.h | 1 - include/linux/buffer_head.h | 2 ++ include/linux/iomap.h | 6 +++++- 7 files changed, 39 insertions(+), 10 deletions(-)
block/bio.c +3 -4
@@ -1753,7 +1753,7 @@ static void bio_complete_work_fn(struct work_struct *w) }while (1 ); } -static void bio_queue_completion (struct bio *bio) +void __bio_complete_in_task (struct bio *bio) {struct bio_complete_batch *batch =this_cpu_ptr (&bio_complete_batch);
@@ -1761,6 +1761,7 @@ static void bio_queue_completion(struct bio *bio)mod_delayed_work_on (batch->cpu, bio_complete_wq, &batch->work,1 ); } +EXPORT_SYMBOL_GPL (__bio_complete_in_task);static inline bool bio_remaining_done (struct bio *bio) {
@@ -1836,9 +1837,7 @@ void bio_endio(struct bio *bio) }#endif -if (!in_task () &&bio_flagged (bio,BIO_COMPLETE_IN_TASK )) -bio_queue_completion (bio); -else if (bio->bi_end_io) +if (bio->bi_end_io) bio->bi_end_io (bio); }EXPORT_SYMBOL (bio_endio);
fs/buffer.c +4 -1
@@ -2673,6 +2673,9 @@ static void end_bio_bh_io_sync(struct bio *bio) {struct buffer_head *bh = bio->bi_private; +if (buffer_dropbehind (bh) &&bio_complete_in_task (bio)) +return ; +if (unlikely (bio_flagged (bio,BIO_QUIET )))set_bit (BH_Quiet, &bh->b_state);
@@ -2725,7 +2728,7 @@ static void submit_bh_wbc(blk_opf_t opf, struct buffer_head *bh,buffer_set_crypto_ctx (bio, bh,GFP_NOIO );if (folio_test_dropbehind (bh->b_folio)) -bio_set_flag (bio,BIO_COMPLETE_IN_TASK ); +set_buffer_dropbehind (bh); bio->bi_iter.bi_sector = bh->b_blocknr * (bh->b_size >>9 ); bio->bi_write_hint = write_hint;
fs/iomap/ioend.c +8 -3
@@ -117,6 +117,12 @@ static void ioend_writeback_end_bio(struct bio *bio) {struct iomap_ioend *ioend =iomap_ioend_from_bio (bio); +/* Page cache invalidation cannot be done in irq context. */ +if (ioend->io_flags &IOMAP_IOEND_DONTCACHE ) { +if (bio_complete_in_task (bio)) +return ; + } + ioend->io_error =blk_status_to_errno (bio->bi_status);if (ioend->io_error) {iomap_fail_ioend_buffered (ioend);
@@ -237,6 +243,8 @@ ssize_t iomap_add_to_ioend(struct iomap_writepage_ctx *wpc, struct folio *folio,if (wpc->iomap.flags &IOMAP_F_SHARED ) ioend_flags |=IOMAP_IOEND_SHARED ; +if (folio_test_dropbehind (folio)) + ioend_flags |=IOMAP_IOEND_DONTCACHE ;if (pos == wpc->iomap.offset && (wpc->iomap.flags &IOMAP_F_BOUNDARY )) ioend_flags |=IOMAP_IOEND_BOUNDARY ;
@@ -253,9 +261,6 @@ ssize_t iomap_add_to_ioend(struct iomap_writepage_ctx *wpc, struct folio *folio,if (!bio_add_folio (&ioend->io_bio, folio, map_len, poff)) goto new_ioend; -if (folio_test_dropbehind (folio)) -bio_set_flag (&ioend->io_bio,BIO_COMPLETE_IN_TASK ); -/* * Clamp io_offset and io_size to the incore EOF so that ondisk * file size updates in the ioend completion are byte-accurate.
include/linux/bio.h +17 -0
@@ -375,6 +375,23 @@ static inline struct bio *bio_alloc(struct block_device *bdev,void submit_bio (struct bio *bio); +void __bio_complete_in_task (struct bio *bio); + +/** +* bio_complete_in_task - ensure a bio is complete in preemptible task context +* @bio: bio to complete +* +* If called from non-task context, offload the bio completion to worker thread +* and return %true. Else return %false and do nothing. +*/ +static inline bool bio_complete_in_task (struct bio *bio) +{ +if (in_task ()) +return false; +__bio_complete_in_task (bio); +return true; +} +extern void bio_endio (struct bio *);static inline void bio_io_error (struct bio *bio)
include/linux/blk_types.h +0 -1
@@ -326,7 +326,6 @@ enum {BIO_REMAPPED ,BIO_ZONE_WRITE_PLUGGING ,/* bio handled through zone write plugging */ BIO_EMULATES_ZONE_APPEND ,/* bio emulates a zone append operation */ -BIO_COMPLETE_IN_TASK ,/* complete bi_end_io() in task context */ BIO_FLAG_LAST };
include/linux/buffer_head.h +2 -0
@@ -35,6 +35,7 @@ enum bh_state_bits { BH_Prio,/* Buffer should be submitted with REQ_PRIO */ BH_Defer_Completion,/* Defer AIO completion to workqueue */ BH_Migrate,/* Buffer is being migrated (norefs) */ + BH_Dropbehind,/* drop pages on IO completion */ BH_PrivateStart,/* not a state bit, but the first bit available* for private allocation by other entities
@@ -136,6 +137,7 @@ BUFFER_FNS(Unwritten, unwritten) BUFFER_FNS(Meta, meta) BUFFER_FNS(Prio, prio) BUFFER_FNS(Defer_Completion, defer_completion) +BUFFER_FNS(Dropbehind, dropbehind) static __always_inline void set_buffer_uptodate(struct buffer_head *bh) {
include/linux/iomap.h +5 -1
@@ -399,12 +399,16 @@ sector_t iomap_bmap(struct address_space *mapping, sector_t bno,#define IOMAP_IOEND_BOUNDARY (1U << 2)/* is direct I/O */ #define IOMAP_IOEND_DIRECT (1U << 3) +/* is DONTCACHE I/O */ +#define IOMAP_IOEND_DONTCACHE (1U << 4) +/* * Flags that if set on either ioend prevent the merge of two ioends. * (IOMAP_IOEND_BOUNDARY also prevents merges, but only one-way) */ #define IOMAP_IOEND_NOMERGE_FLAGS \ - (IOMAP_IOEND_SHARED | IOMAP_IOEND_UNWRITTEN | IOMAP_IOEND_DIRECT) + (IOMAP_IOEND_SHARED | IOMAP_IOEND_UNWRITTEN | IOMAP_IOEND_DIRECT | \ + IOMAP_IOEND_DONTCACHE)/* * Structure for writeback I/O completions.
Signature · 1 line
2.47.3
Re: [PATCH 4/8] FOLD: block: change the defer in task context interface to be procedural
block/bio.c void bio_endio(struct bio *bio) · line 1841
Quoted 9 lines · Christoph Hellwig
block/bio.c +1
-3@@ -1836,9 +1837,7 @@ void bio_endio(struct bio *bio) }#endif -if (!in_task () &&bio_flagged (bio,BIO_COMPLETE_IN_TASK )) -bio_queue_completion (bio); -else if (bio->bi_end_io) +if (bio->bi_end_io) bio->bi_end_io (bio);
What I liked about this before is that we had one central place that needed to be changed. This change means that every bi_end_io now needs to check whether the BIO can be completed in its context.
fs/buffer.c static void end_bio_bh_io_sync(struct bio *bio) · line 2677
Quoted 7 lines
fs/buffer.c +3
-0+++ b/fs/buffer.c@@ -2673,6 +2673,9 @@ static void end_bio_bh_io_sync(struct bio *bio) {struct buffer_head *bh = bio->bi_private; +if (buffer_dropbehind (bh) &&bio_complete_in_task (bio)) +return ;
I really don't like this. It assumes there's only one reason to complete in task context -- whether the buffer belongs to a dropbehind folio. I want there to be other reasons. Why would you introduce the new BH_dropbehind flag instead of checking BIO_COMPLETE_IN_TASK?
fs/iomap/ioend.c static void ioend_writeback_end_bio(struct bio *bio) · line 124
Quoted 7 lines
fs/iomap/ioend.c +5
-0struct iomap_ioend *ioend =iomap_ioend_from_bio (bio); +/* Page cache invalidation cannot be done in irq context. */ +if (ioend->io_flags &IOMAP_IOEND_DONTCACHE ) { +if (bio_complete_in_task (bio)) +return ; + }
I thought we agreed to kill off IOMAP_IOEND_DONTCACHE?
Re: [PATCH 4/8] FOLD: block: change the defer in task context interface to be procedural
Quoted 5 lines · Matthew Wilcox
Quoted 4 lines · Christoph Hellwig
(quoted diff) +0
-0@@ -1836,9 +1837,7 @@ void bio_endio(struct bio *bio) } #endif
Quoted 9 lines
block/bio.c void bio_endio(struct bio *bio) · line 1841
Quoted 5 lines
block/bio.c +1
-3-if (!in_task () &&bio_flagged (bio,BIO_COMPLETE_IN_TASK )) -bio_queue_completion (bio); -else if (bio->bi_end_io) +if (bio->bi_end_io) bio->bi_end_io (bio);What I liked about this before is that we had one central place that needed to be changed. This change means that every
bi_end_ionow needs to check whether the BIO can be completed in its context.
Yes. On the other hand we can actually use it when we don't know if we need to offload beforehand, which enabls the two later conversions and probably more.
Quoted 6 lines
Quoted 6 lines
fs/buffer.c +1
-0+++ b/fs/buffer.c@@ -2673,6 +2673,9 @@ static void end_bio_bh_io_sync(struct bio *bio) {struct buffer_head *bh = bio->bi_private;
Quoted 7 lines
fs/buffer.c static void end_bio_bh_io_sync(struct bio *bio) · line 2677
Quoted 2 lines
fs/buffer.c +2
-0+if (buffer_dropbehind (bh) &&bio_complete_in_task (bio)) +return ;I really don't like this. It assumes there's only one reason to complete in task context -- whether the buffer belongs to a dropbehind folio. I want there to be other reasons. Why would you introduce the new
BH_dropbehindflag instead of checkingBIO_COMPLETE_IN_TASK?
Because there's no BIO_COMPLETE_IN_TASK? left.
Quoted 9 lines
fs/iomap/ioend.c static void ioend_writeback_end_bio(struct bio *bio) · line 124
Quoted 7 lines
fs/iomap/ioend.c +5
-0struct iomap_ioend *ioend =iomap_ioend_from_bio (bio); +/* Page cache invalidation cannot be done in irq context. */ +if (ioend->io_flags &IOMAP_IOEND_DONTCACHE ) { +if (bio_complete_in_task (bio)) +return ; + }I thought we agreed to kill off
IOMAP_IOEND_DONTCACHE?
Only IFF we don't need it. With this version there is no way to just remove it.
Re: [PATCH 4/8] FOLD: block: change the defer in task context interface to be procedural
Quoted 16 lines · Christoph Hellwig
Quoted 5 lines · Matthew Wilcox
Quoted 4 lines · Christoph Hellwig
(quoted diff) +0
-0@@ -1836,9 +1837,7 @@ void bio_endio(struct bio *bio) } #endifQuoted 9 lines
block/bio.c void bio_endio(struct bio *bio) · line 1841
Quoted 5 lines
block/bio.c +1
-3-if (!in_task () &&bio_flagged (bio,BIO_COMPLETE_IN_TASK )) -bio_queue_completion (bio); -else if (bio->bi_end_io) +if (bio->bi_end_io) bio->bi_end_io (bio);What I liked about this before is that we had one central place that needed to be changed. This change means that every
bi_end_ionow needs to check whether the BIO can be completed in its context.
Quoted 3 lines
Yes. On the other hand we can actually use it when we don't know if we need to offload beforehand, which enabls the two later conversions and probably more.
I don't understand why we need to remove _this_ way to defer completions to take context in order to _add_ the ability to defer completions inside the bi_end_io handler.
Re: [PATCH 4/8] FOLD: block: change the defer in task context interface to be procedural
Quoted 7 lines · Matthew Wilcox
Quoted 3 lines
Yes. On the other hand we can actually use it when we don't know if we need to offload beforehand, which enabls the two later conversions and probably more.
I don't understand why we need to remove _this_ way to defer completions to take context in order to _add_ the ability to defer completions inside the
bi_end_iohandler.
I don't need to, but we'd better have very strong reasons for two ways to do the same thing, and I don't really see it here. But I can give it a try if the consensus is that we want to way to do it, and we're not worried about people getting the bio-flag one wrong as we could set it from ->bi_end_io, but it would not do anything.
Re: [PATCH 4/8] FOLD: block: change the defer in task context interface to be procedural
Quoted 9 lines · Christoph Hellwig
Quoted 7 lines · Matthew Wilcox
Quoted 3 lines
Yes. On the other hand we can actually use it when we don't know if we need to offload beforehand, which enabls the two later conversions and probably more.
I don't understand why we need to remove _this_ way to defer completions to take context in order to _add_ the ability to defer completions inside the
bi_end_iohandler.
Quoted 5 lines
I don't need to, but we'd better have very strong reasons for two ways to do the same thing, and I don't really see it here. But I can give it a try if the consensus is that we want to way to do it, and we're not worried about people getting the bio-flag one wrong as we could set it from ->bi_end_io, but it would not do anything.
I want to make all writebacks complete in task context so we can change the i_pages.xa_lock from being irq-disabling to a plain spinlock.
PATCH 5/8 FOLD: don't use in_task() to decide for offloading
In reply to Christoph HellwigAs described in commit c99fab6e80b76, some block drivers might call into ->bi_end_io from non-preemptible context. Copy and past the logic from that commit, although having a core helper for it would be nicer.
--- include/linux/bio.h | 19 +++++++++++++++---- 1 file changed, 15 insertions(+), 4 deletions(-)
include/linux/bio.h +15 -4
@@ -375,6 +375,16 @@ static inline struct bio *bio_alloc(struct block_device *bdev,void submit_bio (struct bio *bio); +/* Offload from atomic contexts to minimize scheduling overhead */ +static inline bool bio_in_atomic (void ) +{ +if (IS_ENABLED (CONFIG_PREEMPTION ) &&rcu_preempt_depth ()) +return true; +if (!IS_ENABLED (CONFIG_PREEMPT_COUNT )) +return true; +return !preemptible (); +} +void __bio_complete_in_task (struct bio *bio);/**
@@ -386,10 +396,11 @@ void __bio_complete_in_task(struct bio *bio);*/ static inline bool bio_complete_in_task (struct bio *bio) { -if (in_task ()) -return false; -__bio_complete_in_task (bio); -return true; +if (bio_in_atomic ()) { +__bio_complete_in_task (bio); +return true; + } +return false; }extern void bio_endio (struct bio *);
Signature · 1 line
2.47.3
PATCH 6/8 iomap: use bio_complete_in_task for buffered read errors
In reply to Christoph HellwigReplace out own hand-crafted complete in task context scheme with the generic block code.
--- fs/iomap/bio.c | 44 +------------------------------------------- 1 file changed, 1 insertion(+), 43 deletions(-)
fs/iomap/bio.c +1 -43
@@ -9,9 +9,6 @@#include "internal.h" #include "trace.h" -static DEFINE_SPINLOCK (failed_read_lock); -static struct bio_list failed_read_list =BIO_EMPTY_LIST ; -static u32 __iomap_read_end_io (struct bio *bio,int error) {struct folio_iter fi;
@@ -27,49 +24,10 @@ static u32 __iomap_read_end_io(struct bio *bio, int error)return folio_count; } -static void -iomap_fail_reads ( -struct work_struct *work) -{ -struct bio *bio; -struct bio_list tmp =BIO_EMPTY_LIST ; -unsigned long flags; - -spin_lock_irqsave (&failed_read_lock, flags); -bio_list_merge_init (&tmp, &failed_read_list); -spin_unlock_irqrestore (&failed_read_lock, flags); - -while ((bio =bio_list_pop (&tmp)) !=NULL ) { -__iomap_read_end_io (bio,blk_status_to_errno (bio->bi_status)); -cond_resched (); - } -} - -static DECLARE_WORK(failed_read_work, iomap_fail_reads); - -static void iomap_fail_buffered_read (struct bio *bio) -{ -unsigned long flags; - -/* -* Bounce I/O errors to a workqueue to avoid nested i_lock acquisitions -* in the fserror code. The caller no longer owns the bio reference -* after the spinlock drops. -*/ -spin_lock_irqsave (&failed_read_lock, flags); -if (bio_list_empty (&failed_read_list)) -WARN_ON_ONCE (!schedule_work (&failed_read_work)); -bio_list_add (&failed_read_list, bio); -spin_unlock_irqrestore (&failed_read_lock, flags); -} -static void iomap_read_end_io (struct bio *bio) { -if (bio->bi_status) { -iomap_fail_buffered_read (bio); +if (bio->bi_status &&bio_complete_in_task (bio))return ; - } -__iomap_read_end_io (bio,0 ); }
Signature · 1 line
2.47.3
Re: [PATCH 6/8] iomap: use bio_complete_in_task for buffered read errors
Quoted 61 lines · Christoph Hellwig
Replace out own hand-crafted complete in task context scheme with the generic block code.
Signed-off-by Christoph Hellwig <[email protected]>--- fs/iomap/bio.c | 44 +------------------------------------------- 1 file changed, 1 insertion(+), 43 deletions(-)fs/iomap/bio.c +0
-38@@ -9,9 +9,6 @@#include "internal.h" #include "trace.h" -static DEFINE_SPINLOCK (failed_read_lock); -static struct bio_list failed_read_list =BIO_EMPTY_LIST ; -static u32 __iomap_read_end_io (struct bio *bio,int error) {struct folio_iter fi;@@ -27,49 +24,10 @@ static u32 __iomap_read_end_io(struct bio *bio, int error)return folio_count; } -static void -iomap_fail_reads ( -struct work_struct *work) -{ -struct bio *bio; -struct bio_list tmp =BIO_EMPTY_LIST ; -unsigned long flags; - -spin_lock_irqsave (&failed_read_lock, flags); -bio_list_merge_init (&tmp, &failed_read_list); -spin_unlock_irqrestore (&failed_read_lock, flags); - -while ((bio =bio_list_pop (&tmp)) !=NULL ) { -__iomap_read_end_io (bio,blk_status_to_errno (bio->bi_status)); -cond_resched (); - } -} - -static DECLARE_WORK(failed_read_work, iomap_fail_reads); - -static void iomap_fail_buffered_read (struct bio *bio) -{ -unsigned long flags; - -/* -* Bounce I/O errors to a workqueue to avoid nested i_lock acquisitions -* in the fserror code. The caller no longer owns the bio reference -* after the spinlock drops. -*/ -spin_lock_irqsave (&failed_read_lock, flags); -if (bio_list_empty (&failed_read_list)) -WARN_ON_ONCE (!schedule_work (&failed_read_work)); -bio_list_add (&failed_read_list, bio); -spin_unlock_irqrestore (&failed_read_lock, flags); -}
fs/iomap/bio.c static u32 __iomap_read_end_io(struct bio *bio, int error) · line 31
Quoted 10 lines
fs/iomap/bio.c +1
-5-static void iomap_read_end_io (struct bio *bio) { -if (bio->bi_status) { -iomap_fail_buffered_read (bio); +if (bio->bi_status &&bio_complete_in_task (bio))return ; - } -__iomap_read_end_io (bio,0 );
I think this should be something like:
__iomap_read_end_io(bio, blk_status_to_errno(bio->bi_status));
Otherwise the error is never propagated when iomap_read_end_io() is later called from task context. Maybe worth rearranging slightly to avoid the blk_status_to_errno() call for non-error I/Os?
I was planning on carrying this and 7/8 in the next version of my series so that the procedural version has users, but I'll send without for now.
Quoted 5 lines
fs/iomap/bio.c +0
-0}Signature · 1 line
2.47.3
PATCH 7/8 iomap: use bio_complete_in_task for buffered write completions
In reply to Christoph HellwigReplace out own hand-crafted complete in task context scheme with the generic block code.
--- fs/iomap/ioend.c | 53 +++++------------------------------------------- 1 file changed, 5 insertions(+), 48 deletions(-)
fs/iomap/ioend.c +5 -48
@@ -72,63 +72,20 @@ static u32 iomap_finish_ioend_buffered_write(struct iomap_ioend *ioend)return folio_count; } -static DEFINE_SPINLOCK (failed_ioend_lock); -static LIST_HEAD (failed_ioend_list); - -static void -iomap_fail_ioends ( -struct work_struct *work) -{ -struct iomap_ioend *ioend; -struct list_head tmp; -unsigned long flags; - -spin_lock_irqsave (&failed_ioend_lock, flags); -list_replace_init (&failed_ioend_list, &tmp); -spin_unlock_irqrestore (&failed_ioend_lock, flags); - - while ((ioend = list_first_entry_or_null(&tmp, struct iomap_ioend, - io_list))) { -list_del_init (&ioend->io_list); -iomap_finish_ioend_buffered_write (ioend); -cond_resched (); - } -} - -static DECLARE_WORK(failed_ioend_work, iomap_fail_ioends); - -static void iomap_fail_ioend_buffered (struct iomap_ioend *ioend) -{ -unsigned long flags; - -/* -* Bounce I/O errors to a workqueue to avoid nested i_lock acquisitions -* in the fserror code. The caller no longer owns the ioend reference -* after the spinlock drops. -*/ -spin_lock_irqsave (&failed_ioend_lock, flags); -if (list_empty (&failed_ioend_list)) -WARN_ON_ONCE (!schedule_work (&failed_ioend_work)); -list_add_tail (&ioend->io_list, &failed_ioend_list); -spin_unlock_irqrestore (&failed_ioend_lock, flags); -} -static void ioend_writeback_end_bio (struct bio *bio) {struct iomap_ioend *ioend =iomap_ioend_from_bio (bio); -/* Page cache invalidation cannot be done in irq context. */ -if (ioend->io_flags &IOMAP_IOEND_DONTCACHE ) { +/* +* Page cache invalidation and error reporting cannot be done in irq +* context. +*/ +if ((ioend->io_flags &IOMAP_IOEND_DONTCACHE ) || bio->bi_status) {if (bio_complete_in_task (bio))return ; } ioend->io_error =blk_status_to_errno (bio->bi_status); -if (ioend->io_error) { -iomap_fail_ioend_buffered (ioend); -return ; - } -iomap_finish_ioend_buffered_write (ioend); }
Signature · 1 line
2.47.3
PATCH 8/8 RFC: use a TASK_FIFO kthread for read completion support
In reply to Christoph HellwigCommit 3fffb589b9a6 ("erofs: add per-cpu threads for decompression as an option") explains why workqueue aren't great for low-latency completion handling. Switch to a per-cpu kthread to handle it instead. This code is based on the erofs code in the above commit, but further simplified by directly using a kthread instead of a kthread_work.
--- block/bio.c | 117 +++++++++++++++++++++++++++++----------------------- 1 file changed, 65 insertions(+), 52 deletions(-)
block/bio.c +65 -52
@@ -19,7 +19,7 @@#include <linux/blk-crypto.h> #include <linux/xarray.h> #include <linux/kmemleak.h> -#include <linux/llist.h> +#include <linux/freezer.h> #include <trace/events/block.h> #include "blk.h"
@@ -1718,51 +1718,83 @@ void bio_check_pages_dirty(struct bio *bio)EXPORT_SYMBOL_GPL (bio_check_pages_dirty);struct bio_complete_batch { -struct llist_head list; -struct delayed_work work; -int cpu; +spinlock_t lock; +struct bio_list bios; +struct task_struct *worker; }; static DEFINE_PER_CPU(struct bio_complete_batch, bio_complete_batch); -static struct workqueue_struct *bio_complete_wq; -static void bio_complete_work_fn (struct work_struct *w) +static bool bio_try_complete_batch (struct bio_complete_batch *batch) { -struct delayed_work *dw =to_delayed_work (w); -struct bio_complete_batch *batch = - container_of(dw, struct bio_complete_batch, work); -struct llist_node *node; -struct bio *bio, *next; +struct bio_list bios; +unsigned long flags; +struct bio *bio; -do { - node =llist_del_all (&batch->list); -if (!node) -break ; +spin_lock_irqsave (&batch->lock, flags); + bios = batch->bios; +bio_list_init (&batch->bios); +spin_unlock_irqrestore (&batch->lock, flags); - node =llist_reverse_order (node); -llist_for_each_entry_safe (bio, next, node, bi_llist) - bio->bi_end_io (bio); +if (bio_list_empty (&bios)) +return false; -if (need_resched ()) { -if (!llist_empty (&batch->list)) -mod_delayed_work_on (batch->cpu, - bio_complete_wq, - &batch->work,0 ); -break ; - } - }while (1 ); +__set_current_state (TASK_RUNNING ); +while ((bio =bio_list_pop (&bios))) + bio->bi_end_io (bio); +return true; +} + +static int bio_complete_thread (void *private) +{ +struct bio_complete_batch *batch = private; + +for (;;) { +set_current_state (TASK_INTERRUPTIBLE ); +if (!bio_try_complete_batch (batch)) +schedule (); + } + +return 0 ; }void __bio_complete_in_task (struct bio *bio) { -struct bio_complete_batch *batch =this_cpu_ptr (&bio_complete_batch); +struct bio_complete_batch *batch; +unsigned long flags; +bool wake; + +get_cpu (); + batch =this_cpu_ptr (&bio_complete_batch); +spin_lock_irqsave (&batch->lock, flags); + wake =bio_list_empty (&batch->bios); +bio_list_add (&batch->bios, bio); +spin_unlock_irqrestore (&batch->lock, flags); +put_cpu (); -if (llist_add (&bio->bi_llist, &batch->list)) -mod_delayed_work_on (batch->cpu, bio_complete_wq, - &batch->work,1 ); +if (wake) +wake_up_process (batch->worker); }EXPORT_SYMBOL_GPL (__bio_complete_in_task); +static void __init bio_complete_batch_init (int cpu) +{ +struct bio_complete_batch *batch = +per_cpu_ptr (&bio_complete_batch, cpu); +struct task_struct *worker; + + worker =kthread_create_on_cpu (bio_complete_thread, +per_cpu_ptr (&bio_complete_batch, cpu), + cpu,"bio_worker/%u" ); +if (IS_ERR (worker)) +panic ("bio: can't create kthread_work" ); +sched_set_fifo_low (worker); + +spin_lock_init (&batch->lock); +bio_list_init (&batch->bios); + batch->worker = worker; +} +static inline bool bio_remaining_done (struct bio *bio) {/*
@@ -2028,16 +2060,7 @@ EXPORT_SYMBOL(bioset_init);*/ static int bio_complete_batch_cpu_dead (unsigned int cpu) { -struct bio_complete_batch *batch = -per_cpu_ptr (&bio_complete_batch, cpu); -struct llist_node *node; -struct bio *bio, *next; - - node =llist_del_all (&batch->list); - node =llist_reverse_order (node); -llist_for_each_entry_safe (bio, next, node, bi_llist) - bio->bi_end_io (bio); - +bio_try_complete_batch (per_cpu_ptr (&bio_complete_batch, cpu));return 0 ; }
@@ -2055,18 +2078,8 @@ static int __init init_bio(void) SLAB_HWCACHE_ALIGN | SLAB_PANIC, NULL); } -for_each_possible_cpu (i) { -struct bio_complete_batch *batch = -per_cpu_ptr (&bio_complete_batch, i); - -init_llist_head (&batch->list); -INIT_DELAYED_WORK (&batch->work, bio_complete_work_fn); - batch->cpu = i; - } - - bio_complete_wq =alloc_workqueue ("bio_complete" ,WQ_MEM_RECLAIM ,0 ); -if (!bio_complete_wq) -panic ("bio: can't allocate bio_complete workqueue\n" ); +for_each_possible_cpu (i) +bio_complete_batch_init (i);cpuhp_setup_state (CPUHP_BP_PREPARE_DYN ,"block/bio:complete:dead" ,NULL , bio_complete_batch_cpu_dead);
Signature · 1 line
2.47.3
Re: [PATCH 8/8] RFC: use a TASK_FIFO kthread for read completion support
Quoted 16 lines · Christoph Hellwig
Commit
3fffb589b9a6("erofs: add per-cpu threads for decompression as an option") explains why workqueue aren't great for low-latency completion handling. Switch to a per-cpu kthread to handle it instead. This code is based on the erofs code in the above commit, but further simplified by directly using a kthread instead of akthread_work.Signed-off-by Christoph Hellwig <[email protected]>--- block/bio.c | 117 +++++++++++++++++++++++++++++----------------------- 1 file changed, 65 insertions(+), 52 deletions(-)block/bio.c +0
-0
block/bio.c line 22
Quoted 5 lines
block/bio.c +1
-1@@ -19,7 +19,7 @@#include <linux/blk-crypto.h> #include <linux/xarray.h> #include <linux/kmemleak.h> -#include <linux/llist.h> +#include <linux/freezer.h>
Why freezer.h and not kthread.h?
block/bio.c void bio_check_pages_dirty(struct bio *bio) · line 1721
Quoted 7 lines
block/bio.c +0
-1#include <trace/events/block.h> #include "blk.h" @@ -1718,51 +1718,83 @@ void bio_check_pages_dirty(struct bio *bio)EXPORT_SYMBOL_GPL (bio_check_pages_dirty);struct bio_complete_batch { -struct llist_head list;
If we go with this approach, we should remove the newly-added bi_llist from struct bio too.
Quoted 150 lines
block/bio.c +64
-50-struct delayed_work work; -int cpu; +spinlock_t lock; +struct bio_list bios; +struct task_struct *worker; }; static DEFINE_PER_CPU(struct bio_complete_batch, bio_complete_batch); -static struct workqueue_struct *bio_complete_wq; -static void bio_complete_work_fn (struct work_struct *w) +static bool bio_try_complete_batch (struct bio_complete_batch *batch) { -struct delayed_work *dw =to_delayed_work (w); -struct bio_complete_batch *batch = - container_of(dw, struct bio_complete_batch, work); -struct llist_node *node; -struct bio *bio, *next; +struct bio_list bios; +unsigned long flags; +struct bio *bio; -do { - node =llist_del_all (&batch->list); -if (!node) -break ; +spin_lock_irqsave (&batch->lock, flags); + bios = batch->bios; +bio_list_init (&batch->bios); +spin_unlock_irqrestore (&batch->lock, flags); - node =llist_reverse_order (node); -llist_for_each_entry_safe (bio, next, node, bi_llist) - bio->bi_end_io (bio); +if (bio_list_empty (&bios)) +return false; -if (need_resched ()) { -if (!llist_empty (&batch->list)) -mod_delayed_work_on (batch->cpu, - bio_complete_wq, - &batch->work,0 ); -break ; - } - }while (1 ); +__set_current_state (TASK_RUNNING ); +while ((bio =bio_list_pop (&bios))) + bio->bi_end_io (bio); +return true; +} + +static int bio_complete_thread (void *private) +{ +struct bio_complete_batch *batch = private; + +for (;;) { +set_current_state (TASK_INTERRUPTIBLE ); +if (!bio_try_complete_batch (batch)) +schedule (); + } + +return 0 ; }void __bio_complete_in_task (struct bio *bio) { -struct bio_complete_batch *batch =this_cpu_ptr (&bio_complete_batch); +struct bio_complete_batch *batch; +unsigned long flags; +bool wake; + +get_cpu (); + batch =this_cpu_ptr (&bio_complete_batch); +spin_lock_irqsave (&batch->lock, flags); + wake =bio_list_empty (&batch->bios); +bio_list_add (&batch->bios, bio); +spin_unlock_irqrestore (&batch->lock, flags); +put_cpu (); -if (llist_add (&bio->bi_llist, &batch->list)) -mod_delayed_work_on (batch->cpu, bio_complete_wq, - &batch->work,1 ); +if (wake) +wake_up_process (batch->worker); }EXPORT_SYMBOL_GPL (__bio_complete_in_task); +static void __init bio_complete_batch_init (int cpu) +{ +struct bio_complete_batch *batch = +per_cpu_ptr (&bio_complete_batch, cpu); +struct task_struct *worker; + + worker =kthread_create_on_cpu (bio_complete_thread, +per_cpu_ptr (&bio_complete_batch, cpu), + cpu,"bio_worker/%u" ); +if (IS_ERR (worker)) +panic ("bio: can't create kthread_work" ); +sched_set_fifo_low (worker); + +spin_lock_init (&batch->lock); +bio_list_init (&batch->bios); + batch->worker = worker; +} +static inline bool bio_remaining_done (struct bio *bio) {/* @@ -2028,16 +2060,7 @@ EXPORT_SYMBOL(bioset_init);*/ static int bio_complete_batch_cpu_dead (unsigned int cpu) { -struct bio_complete_batch *batch = -per_cpu_ptr (&bio_complete_batch, cpu); -struct llist_node *node; -struct bio *bio, *next; - - node =llist_del_all (&batch->list); - node =llist_reverse_order (node); -llist_for_each_entry_safe (bio, next, node, bi_llist) - bio->bi_end_io (bio); - +bio_try_complete_batch (per_cpu_ptr (&bio_complete_batch, cpu));return 0 ; }@@ -2055,18 +2078,8 @@ static int __init init_bio(void) SLAB_HWCACHE_ALIGN | SLAB_PANIC, NULL); } -for_each_possible_cpu (i) { -struct bio_complete_batch *batch = -per_cpu_ptr (&bio_complete_batch, i); - -init_llist_head (&batch->list); -INIT_DELAYED_WORK (&batch->work, bio_complete_work_fn); - batch->cpu = i; - } - - bio_complete_wq =alloc_workqueue ("bio_complete" ,WQ_MEM_RECLAIM ,0 ); -if (!bio_complete_wq) -panic ("bio: can't allocate bio_complete workqueue\n" ); +for_each_possible_cpu (i) +bio_complete_batch_init (i);cpuhp_setup_state (CPUHP_BP_PREPARE_DYN ,"block/bio:complete:dead" ,NULL , bio_complete_batch_cpu_dead);Signature · 1 line
2.47.3
Re: [PATCH 8/8] RFC: use a TASK_FIFO kthread for read completion support
Quoted 4 lines · Tal Zussman
block/bio.c line 22
Quoted 2 lines
block/bio.c +1
-1-#include <linux/llist.h> +#include <linux/freezer.h> Why
freezer.hand notkthread.h?
I needed freezer.h to try to make the thread freezable, but that didn't work out. kthread.h seems to be pulled in implicitly.
Quoted 5 lines
block/bio.c void bio_check_pages_dirty(struct bio *bio) · line 1721
Quoted 2 lines
struct bio_complete_batch {block/bio.c +0
-1-struct llist_head list;If we go with this approach, we should remove the newly-added
bi_llistfrom struct bio too.
Yes.
Re: [PATCH 8/8] RFC: use a TASK_FIFO kthread for read completion support
In reply to Christoph HellwigQuoted 7 lines · Christoph Hellwig
Commit
3fffb589b9a6("erofs: add per-cpu threads for decompression as an option") explains why workqueue aren't great for low-latency completion handling. Switch to a per-cpu kthread to handle it instead. This code is based on the erofs code in the above commit, but further simplified by directly using a kthread instead of akthread_work.Signed-off-by Christoph Hellwig <[email protected]>
Can we please not go back to the (bad) old days of individual subsystems needing their own set of per-cpu kernel tasks just sitting around idle most of of the time? The whole point of the workqueue infrastructure was to get rid of this widely repeated anti-pattern.
If there's a latency problem with workqueue scheduling, then we should be fixing that problem rather than working around it in every subsystem that thinkgs it has a workqueue scheduling latency issue...
-Dave.
Signature · 2 lines
Dave Chinner [email protected]
Re: [PATCH 8/8] RFC: use a TASK_FIFO kthread for read completion support
Quoted 15 lines · Dave Chinner
Quoted 7 lines · Christoph Hellwig
Commit
3fffb589b9a6("erofs: add per-cpu threads for decompression as an option") explains why workqueue aren't great for low-latency completion handling. Switch to a per-cpu kthread to handle it instead. This code is based on the erofs code in the above commit, but further simplified by directly using a kthread instead of akthread_work.Signed-off-by Christoph Hellwig <[email protected]>Can we please not go back to the (bad) old days of individual subsystems needing their own set of per-cpu kernel tasks just sitting around idle most of of the time? The whole point of the workqueue infrastructure was to get rid of this widely repeated anti-pattern.
Quoted 4 lines
If there's a latency problem with workqueue scheduling, then we should be fixing that problem rather than working around it in every subsystem that thinkgs it has a workqueue scheduling latency issue...
It has been "fixed" but never actually get fixed:
https://lore.kernel.org/r/CAB=BE-QaNBn1cVK6c7LM2cLpH_Ck_9SYw-YDYEnNrtwfoyu81Q@mail.gmail.com
and workqueues don't have any plan to introduce RT threads;
If Sandeep has more time, I hope he could have more time to test since I don't work on Android anymore: In principle, I still think RT thread is needed somewhere for such usage since lowest latencies is needed.
Compared to the scheduling latency issues, interested users don't care "individual subsystems needing their own set of per-cpu kernel tasks just sitting around idle most of of the time". If end users care it more, they can just turn it off by Kconfig.
If you think it's unclean to block subsystems, I'm fine to leave it as-is. I'm not sure how it impacts to others:
either some subsystems implement RT threads (workqueue or block), or leave it as-is; Otherwise there is always such latency issues and unacceptable for end users.
Thanks,
Gao Xiang
Quoted 3 lines
-Dave.
Re: [PATCH 8/8] RFC: use a TASK_FIFO kthread for read completion support
Quoted 39 lines · Gao Xiang
Quoted 15 lines · Dave Chinner
Quoted 7 lines · Christoph Hellwig
Commit
3fffb589b9a6("erofs: add per-cpu threads for decompression as an option") explains why workqueue aren't great for low-latency completion handling. Switch to a per-cpu kthread to handle it instead. This code is based on the erofs code in the above commit, but further simplified by directly using a kthread instead of akthread_work.Signed-off-by Christoph Hellwig <[email protected]>Can we please not go back to the (bad) old days of individual subsystems needing their own set of per-cpu kernel tasks just sitting around idle most of of the time? The whole point of the workqueue infrastructure was to get rid of this widely repeated anti-pattern.
Quoted 4 lines
If there's a latency problem with workqueue scheduling, then we should be fixing that problem rather than working around it in every subsystem that thinkgs it has a workqueue scheduling latency issue...
It has been "fixed" but never actually get fixed:
https://lore.kernel.org/r/CAB=BE-QaNBn1cVK6c7LM2cLpH_Ck_9SYw-YDYEnNrtwfoyu81Q@mail.gmail.comand workqueues don't have any plan to introduce RT threads;
If Sandeep has more time, I hope he could have more time to test since I don't work on Android anymore: In principle, I still think RT thread is needed somewhere for such usage since lowest latencies is needed.
Compared to the scheduling latency issues, interested users don't care "individual subsystems needing their own set of per-cpu kernel tasks just sitting around idle most of of the time". If end users care it more, they can just turn it off by
Kconfig.
Quoted 4 lines
If you think it's unclean to block subsystems, I'm fine to leave it as-is. I'm not sure how it impacts to others:
either some subsystems implement RT threads (workqueue or block), or leave it as-is; Otherwise there is always such
^ Implement dedicated queues somewhere using sched_set_fifo_low() for such latency-sensitive work.
Quoted 10 lines
latency issues and unacceptable for end users.
Thanks,
Gao XiangQuoted 3 lines
-Dave.
Re: [PATCH 8/8] RFC: use a TASK_FIFO kthread for read completion support
In reply to Gao XiangQuoted 23 lines · Gao Xiang
Quoted 15 lines · Dave Chinner
Quoted 7 lines · Christoph Hellwig
Commit
3fffb589b9a6("erofs: add per-cpu threads for decompression as an option") explains why workqueue aren't great for low-latency completion handling. Switch to a per-cpu kthread to handle it instead. This code is based on the erofs code in the above commit, but further simplified by directly using a kthread instead of akthread_work.Signed-off-by Christoph Hellwig <[email protected]>Can we please not go back to the (bad) old days of individual subsystems needing their own set of per-cpu kernel tasks just sitting around idle most of of the time? The whole point of the workqueue infrastructure was to get rid of this widely repeated anti-pattern.
Quoted 4 lines
If there's a latency problem with workqueue scheduling, then we should be fixing that problem rather than working around it in every subsystem that thinkgs it has a workqueue scheduling latency issue...
Quoted 4 lines
It has been "fixed" but never actually get fixed:
https://lore.kernel.org/r/CAB=BE-QaNBn1cVK6c7LM2cLpH_Ck_9SYw-YDYEnNrtwfoyu81Q@mail.gmail.comand workqueues don't have any plan to introduce RT threads;
They don't need to (or should) introduce RT threads. Per-cpu kernel threads already get priority over normal user tasks on scheduling decisions. However, they do not pre-empt running kernel tasks of the same priority.
In general, kernel threads should not use RT scheduling at all - if the kernel uses RT prioprity tasks then that can interfere with user scheduled RT tasks. This is especially true in this case where a non-RT tasks issue the IO, and the IO completion is then scheduled with RT priority. IOWs, any unprivileged user can now impact the processing time available to, and the response latency of, other RT scheduled tasks the system is running.
Tejun asked Sandeep if setting the workqueue thread priority to -19 through sysfs (i.e. making them higher priority than normal kernel threads) had the same effect on latency as using a dedicated per-cpu RT task thread. THere was no followup.
In theory, this should provide the same benefit, because what RT scheduling is doing is pre-empting any user and kernel task that was running when the interrupt was delivered to execute the completion task immediately.
Setting the workqueue to use kernel threads of a higher scheduler prioirty should do the same thing, without the need to use dedicated per-cpu RT threads.
Quoted 4 lines
If Sandeep has more time, I hope he could have more time to test since I don't work on Android anymore: In principle, I still think RT thread is needed somewhere for such usage since lowest latencies is needed.
All that is needed is for the kworker thread to be scheduled to run immeidately after the interrupt that scheduled the work exits. This does not require dedicated per-cpu kernel tasks or RT scheduling....
Quoted 5 lines
Compared to the scheduling latency issues, interested users don't care "individual subsystems needing their own set of per-cpu kernel tasks just sitting around idle most of of the time". If end users care it more, they can just turn it off by
Kconfig.
Distros enable all these subsystems all the time, so saying "turn it off via kconfig" is not a viable mitigation strategy. Proliferation of dedicated per-CPU worker task pools is a known problem, and we really don't want to regress back to those days when a typical system had thousands of dedicated per-cpu work queues that largely did nothing most of the time.
-Dave.
Signature · 2 lines
Dave Chinner [email protected]
Re: [PATCH 8/8] RFC: use a TASK_FIFO kthread for read completion support
Quoted 34 lines · Dave Chinner
Quoted 23 lines · Gao Xiang
Quoted 15 lines · Dave Chinner
Quoted 7 lines · Christoph Hellwig
Commit
3fffb589b9a6("erofs: add per-cpu threads for decompression as an option") explains why workqueue aren't great for low-latency completion handling. Switch to a per-cpu kthread to handle it instead. This code is based on the erofs code in the above commit, but further simplified by directly using a kthread instead of akthread_work.Signed-off-by Christoph Hellwig <[email protected]>Can we please not go back to the (bad) old days of individual subsystems needing their own set of per-cpu kernel tasks just sitting around idle most of of the time? The whole point of the workqueue infrastructure was to get rid of this widely repeated anti-pattern.
Quoted 4 lines
If there's a latency problem with workqueue scheduling, then we should be fixing that problem rather than working around it in every subsystem that thinkgs it has a workqueue scheduling latency issue...
Quoted 4 lines
It has been "fixed" but never actually get fixed:
https://lore.kernel.org/r/CAB=BE-QaNBn1cVK6c7LM2cLpH_Ck_9SYw-YDYEnNrtwfoyu81Q@mail.gmail.comand workqueues don't have any plan to introduce RT threads;
They don't need to (or should) introduce RT threads. Per-cpu kernel threads already get priority over normal user tasks on scheduling decisions. However, they do not pre-empt running kernel tasks of the same priority.
Quoted 7 lines
In general, kernel threads should not use RT scheduling at all - if the kernel uses RT prioprity tasks then that can interfere with user scheduled RT tasks. This is especially true in this case where a non-RT tasks issue the IO, and the IO completion is then scheduled with RT priority. IOWs, any unprivileged user can now impact the processing time available to, and the response latency of, other RT scheduled tasks the system is running.
All softirq IO completion already works like this although softirq tasks are not strictly called "RT tasks" (i.e. a non-RT task issues the IO, and the softirq IO completion will interrupt all ongoing tasks).
Basically what we want is to get a non-atomic context instead of using the current softirq context for read post-processing and switch to the task context immediately as you said, because:
- Our post-processing needs to work in task contexts since advanced features like compression deduplication need it;
- Even regardless of our specific requirement needing task contexts, using a dedicated task context for read post-processing is much better than run in the original softirq context:
- Algorithmic work could take extra time (especially slow LZMA algorithm could take milliseconds on low devices (however, we need a common workflow for all algorithms, including fast algorithms like lz4) and verify work for example); and long processing time will interfere with other remaining softirq tasks like sound-playback / network softirqs;
- If it is then deferred to softirqd, it just makes this latency issue _worse_.
Thus, if there is another dedicated mechanism which can provide a lightweight task context and is scheduled to run immediately after the softirq: that would fit our requirement and I believe it's useful for other various use cases, but currently there is no such clean infra; RT threads can just fulfill our requirement in a less elegant way.
Quoted 5 lines
Tejun asked Sandeep if setting the workqueue thread priority to -19 through sysfs (i.e. making them higher priority than normal kernel threads) had the same effect on latency as using a dedicated per-cpu RT task thread. THere was no followup.
I think the issue is that people are not already working on the same topic:
- Unlike large subsystems like XFS, people don't already work on EROFS unless they have new requirements or urgent production issues;
- The original latency issue was already considered as "done" in 2023, and I'm not sure if Sandeep could have the bandwidth to pause his current work and test more setups according to this ongoing discussion in 2026.
Quoted 5 lines
In theory, this should provide the same benefit, because what RT scheduling is doing is pre-empting any user and kernel task that was running when the interrupt was delivered to execute the completion task immediately.
But anyway, I think nice -19 can be evaluated if Sandeep have time, but such nice value should be set by the filesystem instead of the userspace since the reason is as above.
Additionally, we also need a way to set nice if we decide to switch to the new BIO_COMPLETE_IN_TASK approach as well if the nice way really works.
Quoted 5 lines
Setting the workqueue to use kernel threads of a higher scheduler prioirty should do the same thing, without the need to use dedicated per-cpu RT threads.
Quoted 8 lines
Quoted 4 lines
If Sandeep has more time, I hope he could have more time to test since I don't work on Android anymore: In principle, I still think RT thread is needed somewhere for such usage since lowest latencies is needed.
All that is needed is for the kworker thread to be scheduled to run immeidately after the interrupt that scheduled the work exits. This does not require dedicated per-cpu kernel tasks or RT scheduling....
Right and not sure, it needs some latency evaluation with heavy background/foreground app pressures.
Quoted 7 lines
Quoted 6 lines
Compared to the scheduling latency issues, interested users don't care "individual subsystems needing their own set of per-cpu kernel tasks just sitting around idle most of of the time". If end users care it more, they can just turn it off by
Kconfig.
Quoted 6 lines
Distros enable all these subsystems all the time, so saying "turn it off via kconfig" is not a viable mitigation strategy. Proliferation of dedicated per-CPU worker task pools is a known problem, and we really don't want to regress back to those days when a typical system had thousands of dedicated per-cpu work queues that largely did nothing most of the time.
That kconfig is off by default, but all Android vendors turn it on, and that latency issue is particularly vital for Android ecosystem.
Currently RT threads won't be used for other use cases even desktop use cases: Android workloads are pretty harsh even compared to desktops for example.
Thanks,
Gao Xiang
Quoted 3 lines
-Dave.
Re: [PATCH 8/8] RFC: use a TASK_FIFO kthread for read completion support
Quoted 25 lines · Gao Xiang
All softirq IO completion already works like this although softirq tasks are not strictly called "RT tasks" (i.e. a non-RT task issues the IO, and the softirq IO completion will interrupt all ongoing tasks).
Basically what we want is to get a non-atomic context instead of using the current softirq context for read post-processing and switch to the task context immediately as you said, because:
- Our post-processing needs to work in task contexts since advanced features like compression deduplication need it;
- Even regardless of our specific requirement needing task contexts, using a dedicated task context for read post-processing is much better than run in the original softirq context:
- Algorithmic work could take extra time (especially slow LZMA algorithm could take milliseconds on low devices (however, we need a common workflow for all algorithms, including fast algorithms like lz4) and verify work for example); and long processing time will interfere with other remaining softirq tasks like sound-playback / network softirqs;
Quoted 2 lines
- If it is then deferred to softirqd, it just makes this latency issue _worse_.
Yes, and the same applies to a lot of other things. A very similar algorithmic issue is the checksum validation, be that T10-PI or file system native checksums. We don't want to run them from soft/hardirq context obviously, but we really need them to preempt other work on the cpu, and avoid scheduling latency. The case that started this is a bit different - folio invalidation mostly needs user context to take sleeping locks, but those are usually uncontented. Again, getting this work done ASAP as readers are synchronously waiting is important.
Quoted 8 lines
Quoted 4 lines
Tejun asked Sandeep if setting the workqueue thread priority to -19 through sysfs (i.e. making them higher priority than normal kernel threads) had the same effect on latency as using a dedicated per-cpu RT task thread. THere was no followup.
I think the issue is that people are not already working on the same topic:
Quoted 8 lines
- Unlike large subsystems like XFS, people don't already work on EROFS unless they have new requirements or urgent production issues;
- The original latency issue was already considered as "done" in 2023, and I'm not sure if Sandeep could have the bandwidth to pause his current work and test more setups according to this ongoing discussion in 2026.
Which unfortunately might explain the sad state of Android. You really need dedicated people around and help to improve core infrastructure, instead of adding random Kconfig choices.
Re: [PATCH 8/8] RFC: use a TASK_FIFO kthread for read completion support
Quoted 1 line · Christoph Hellwig
On Tue, Apr 14, 2026 at 10:23:13AM +0800, Gao Xiang wrote:
...
Quoted 13 lines
Quoted 13 lines
Quoted 5 lines
Tejun asked Sandeep if setting the workqueue thread priority to -19 through sysfs (i.e. making them higher priority than normal kernel threads) had the same effect on latency as using a dedicated per-cpu RT task thread. THere was no followup.
I think the issue is that people are not already working on the same topic:
- Unlike large subsystems like XFS, people don't already work on EROFS unless they have new requirements or urgent production issues;
Quoted 8 lines
Quoted 4 lines
- The original latency issue was already considered as "done" in 2023, and I'm not sure if Sandeep could have the bandwidth to pause his current work and test more setups according to this ongoing discussion in 2026.
Which unfortunately might explain the sad state of Android. You really need dedicated people around and help to improve core infrastructure, instead of adding random
Kconfigchoices.
Sigh, let's not regard it as a technical problem, and it's more of a non-technical development resource issue among different resource groups & vendors.
The detailed background is much much more complex than I could write all down explicitly here, but at least I will try to contact with Sandeep and more related people to test again in 2026 if a cleaner approach can be worked out.
Thanks,
Gao Xiang
Re: [PATCH 8/8] RFC: use a TASK_FIFO kthread for read completion support
In reply to Christoph HellwigQuoted 29 lines · Christoph Hellwig
Quoted 25 lines · Gao Xiang
All softirq IO completion already works like this although softirq tasks are not strictly called "RT tasks" (i.e. a non-RT task issues the IO, and the softirq IO completion will interrupt all ongoing tasks).
Basically what we want is to get a non-atomic context instead of using the current softirq context for read post-processing and switch to the task context immediately as you said, because:
- Our post-processing needs to work in task contexts since advanced features like compression deduplication need it;
- Even regardless of our specific requirement needing task contexts, using a dedicated task context for read post-processing is much better than run in the original softirq context:
- Algorithmic work could take extra time (especially slow LZMA algorithm could take milliseconds on low devices (however, we need a common workflow for all algorithms, including fast algorithms like lz4) and verify work for example); and long processing time will interfere with other remaining softirq tasks like sound-playback / network softirqs;
Quoted 2 lines
- If it is then deferred to softirqd, it just makes this latency issue _worse_.
Quoted 9 lines
Yes, and the same applies to a lot of other things. A very similar algorithmic issue is the checksum validation, be that T10-PI or file system native checksums. We don't want to run them from soft/hardirq context obviously, but we really need them to preempt other work on the cpu, and avoid scheduling latency. The case that started this is a bit different - folio invalidation mostly needs user context to take sleeping locks, but those are usually uncontented. Again, getting this work done ASAP as readers are synchronously waiting is important.
Not sure why I ended up here but looking at the patch I don't see how is different to the kworker solution that is removed. Unless you fiddle with the thread priority in userland.
The get_cpu() + spin_lock usage breaks on PREEMPT_RT.
The NOHZ_FULL camp might not be happy about putting load on isolated CPUs. I remember there was some effort to avoid those CPUs.
Sebastian
Re: [PATCH 8/8] RFC: use a TASK_FIFO kthread for read completion support
Quoted 1 line · Sebastian Andrzej Siewior
Not sure why I ended up here
Because this is very similar to ksoftirqd and softirq.c doesn't have a maintainer, and you were the last one doing substantial changes to it. And you're also very knowledgeable about PREEMPT_RT, which has a bit of a history of low-latency user context helpers :)
Re: [PATCH 8/8] RFC: use a TASK_FIFO kthread for read completion support
In reply to Gao XiangQuoted 20 lines
Basically what we want is to get a non-atomic context instead of using the current softirq context for read post-processing and switch to the task context immediately as you said, because:
- Our post-processing needs to work in task contexts since
advanced features like compression deduplication need it;- Even regardless of our specific requirement needing task contexts, using a dedicated task context for read post-processing is much better than run in the original softirq context:
- Algorithmic work could take extra time (especially slow LZMA algorithm could take milliseconds on low devices (however, we need a common workflow for all algorithms, including fast algorithms like lz4) and verify work for example); and long processing time will interfere with other remaining softirq tasks like sound-playback / network softirqs;
Quoted 2 lines
- If it is then deferred to softirqd, it just makes this latency issue _worse_.
Yes, this was the reasoning when I narrowed it down to per-CPU kthread approach. Patch has helped reduce app launch times which has a direct user impact.
Quoted 5 lines
But anyway, I think nice -19 can be evaluated if Sandeep have time, but such nice value should be set by the filesystem instead of the userspace since the reason is as above.
I will find some time to re-do this again, including setting nice -19.
-Sandeep.
Re: [PATCH 8/8] RFC: use a TASK_FIFO kthread for read completion support
In reply to Dave ChinnerQuoted 34 lines · On Tue, 14 Apr 2026 10:58:16 +1000 Dave Chinner <[email protected]> wrote:
Quoted 23 lines · Gao Xiang
Quoted 15 lines · Dave Chinner
Quoted 7 lines · Christoph Hellwig
Commit
3fffb589b9a6("erofs: add per-cpu threads for decompression as an option") explains why workqueue aren't great for low-latency completion handling. Switch to a per-cpu kthread to handle it instead. This code is based on the erofs code in the above commit, but further simplified by directly using a kthread instead of akthread_work.Signed-off-by Christoph Hellwig <[email protected]>Can we please not go back to the (bad) old days of individual subsystems needing their own set of per-cpu kernel tasks just sitting around idle most of of the time? The whole point of the workqueue infrastructure was to get rid of this widely repeated anti-pattern.
Quoted 4 lines
If there's a latency problem with workqueue scheduling, then we should be fixing that problem rather than working around it in every subsystem that thinkgs it has a workqueue scheduling latency issue...
Quoted 4 lines
It has been "fixed" but never actually get fixed:
https://lore.kernel.org/r/CAB=BE-QaNBn1cVK6c7LM2cLpH_Ck_9SYw-YDYEnNrtwfoyu81Q@mail.gmail.comand workqueues don't have any plan to introduce RT threads;
They don't need to (or should) introduce RT threads. Per-cpu kernel threads already get priority over normal user tasks on scheduling decisions. However, they do not pre-empt running kernel tasks of the same priority.
Quoted 7 lines
In general, kernel threads should not use RT scheduling at all - if the kernel uses RT prioprity tasks then that can interfere with user scheduled RT tasks. This is especially true in this case where a non-RT tasks issue the IO, and the IO completion is then scheduled with RT priority. IOWs, any unprivileged user can now impact the processing time available to, and the response latency of, other RT scheduled tasks the system is running.
But might not an IO for a use RT task be sat behind it in the completion queue?
To avoid priority inversion you need to process the completions. The non-RT task won't be rescheduled to issue a later request. The only other way is to force that completion work be done in the context of the thread that issued the request.
Quoted 6 lines
Tejun asked Sandeep if setting the workqueue thread priority to -19 through sysfs (i.e. making them higher priority than normal kernel threads) had the same effect on latency as using a dedicated per-cpu RT task thread. THere was no followup.
Quoted 8 lines
In theory, this should provide the same benefit, because what RT scheduling is doing is pre-empting any user and kernel task that was running when the interrupt was delivered to execute the completion task immediately.
Setting the workqueue to use kernel threads of a higher scheduler prioirty should do the same thing, without the need to use dedicated per-cpu RT threads.
We had a related issue with the network stack's use of softint and threaded napi when trying to receive very high packet rate UDP traffic. (IIRC something like 500000 200byte RTP audio packets every second.) There is an absolute requirement to run the ethernet rx processing in order to avoid dropping packets.
Now normally the code runs as 'softint' making it higher priority that any user process, but under load it switches to using 'normal priority' kernel threads which are basically low priority. Similarly 'threaded napi' uses 'normal priority' threads. The only we made it work was to manually change the threads to be a low 'RT FIFO' priority so they got scheduled in preference to all user processes.
A system may need some user RT threads with a lower priority than these kernel threads and others with a higher priority.
So defaulting to a middling RT priority may be best.
David
Re: [PATCH 8/8] RFC: use a TASK_FIFO kthread for read completion support
In reply to Dave ChinnerQuoted 10 lines · Dave Chinner
Can we please not go back to the (bad) old days of individual subsystems needing their own set of per-cpu kernel tasks just sitting around idle most of of the time? The whole point of the workqueue infrastructure was to get rid of this widely repeated anti-pattern.
If there's a latency problem with workqueue scheduling, then we should be fixing that problem rather than working around it in every subsystem that thinkgs it has a workqueue scheduling latency issue...
Fixing the workqueue scheduling would be nice, but the attempts so far failed.
In addition to that for a lot of theses cases workqueues are actually a surprisingly bad fit - we have items we want to queue up an one single function to call on all of them. So the overhead should be a list item (which often can be singly linked) in the object, while the workqueue also adds flags and a function pointer, bloating the size. We often work around this by having a single work_struct work on multiple objects, but that just increases the amount of work that needs to be done, including atomics and scheduling.
Last but not least bio completion isn't really any random subsystem. Block I/O completion is important enough that we have an even more epensive softirq allocated to it. I agree that the dynamic workqueue-style workers are a much better choise for most use cases, though.
Re: [PATCH 8/8] RFC: use a TASK_FIFO kthread for read completion support
[adding the workqueue maintainers that I should have added]
Quoted 24 lines · Christoph Hellwig
Quoted 10 lines · Dave Chinner
Can we please not go back to the (bad) old days of individual subsystems needing their own set of per-cpu kernel tasks just sitting around idle most of of the time? The whole point of the workqueue infrastructure was to get rid of this widely repeated anti-pattern.
If there's a latency problem with workqueue scheduling, then we should be fixing that problem rather than working around it in every subsystem that thinkgs it has a workqueue scheduling latency issue...
Fixing the workqueue scheduling would be nice, but the attempts so far failed.
In addition to that for a lot of theses cases workqueues are actually a surprisingly bad fit - we have items we want to queue up an one single function to call on all of them. So the overhead should be a list item (which often can be singly linked) in the object, while the workqueue also adds flags and a function pointer, bloating the size. We often work around this by having a single
work_structwork on multiple objects, but that just increases the amount of work that needs to be done, including atomics and scheduling.
Quoted 5 lines
Last but not least bio completion isn't really any random subsystem. Block I/O completion is important enough that we have an even more epensive softirq allocated to it. I agree that the dynamic workqueue-style workers are a much better choise for most use cases, though.
---end quoted text---