Skip to contentSkip to conversation

block: enable RWF_DONTCACHE for block devices

PATCH Applied Tal Zussman 5 participants 22 messages Jul 30 – Aug 30

Subsystems (6)
lore
Share & tools
lore
#0 lore raw

v7 cover block: enable RWF_DONTCACHE for block devices

Add support for using RWF_DONTCACHE with block devices.

Dropbehind pruning needs to be done in non-IRQ context, but block devices complete writeback in IRQ context. To fix this, we defer dropbehind invalidation to task context. Add infrastructure that lets bi_end_io callbacks run from a worker, in two forms:

1. BIO_COMPLETE_IN_TASK, a bio flag the submitter sets when it knows upfront that the callback needs task context, as in the dropbehind writeback paths.

2. bio_complete_in_task(), a helper that callbacks can invoke from bi_end_io() when the decision to defer is dynamic, as in iomap fserror reporting.

These queue the bio to a per-CPU list and schedule a per-CPU work item to run the completions from task context.

Patch 1 moves erofs's atomic context detection into the block layer as bio_in_atomic() and converts erofs to use it.

Patch 2 adds the block layer task-context completion infrastructure, with both the flag and the procedural helper. This builds on top of suggestions by Matthew and Christoph: the procedural helper and bio_in_atomic() come from Christoph's "bio completion in task enhancements / experiments" series [1]. The procedural helper is unused in this series, but has multiple users planned [1][2].

Patch 3 wires BIO_COMPLETE_IN_TASK into iomap writeback for dropbehind folios, removes IOMAP_IOEND_DONTCACHE, and removes the DONTCACHE workqueue deferral from XFS.

Patch 4 sets up DONTCACHE support for buffer-head-based I/O by setting BIO_COMPLETE_IN_TASK in __bh_submit() for the CONFIG_BUFFER_HEAD path.

Patch 5 enables RWF_DONTCACHE for block devices based on the previous support. This support is useful for databases that operate on raw block devices, among other userspace applications.

I tested this (with CONFIG_BUFFER_HEAD=y) for reads and writes on a single block device on a VM, so results may be noisy.

Reads were tested on the root partition with a 45GB range (~2x RAM). Writes were tested on a disabled swap partition (~1GB) in a memcg of size 244MB to force reclaim pressure.

Results:

===== READS (/dev/nvme0n1p2) =====
sec normal MB/s dontcache MB/s
---- ------------ --------------

   1        1098.6          1609.0
   2        1270.3          1506.6
   3        1093.3          1576.5
   4        1141.8          2393.9
   5        1365.3          2793.8
   6        1324.6          2065.9
   7         879.6          1920.7
   8        1434.1          1662.4
   9        1184.9          1857.9
  10        1166.4          1702.8
  11        1161.4          1653.4
  12        1086.9          1555.4
  13        1198.5          1718.9
  14        1111.9          1752.2

---- ------------ --------------
avg 1173.7 1828.8 (+56%)

==== WRITES (/dev/nvme0n1p3) =====
sec normal MB/s dontcache MB/s
---- ------------ --------------

   1         692.4          9297.7
   2        4810.8          9342.8
   3        5221.7          2955.2
   4         396.7          8488.3
   5        7249.2          9249.3
   6        6695.4          1376.2
   7         122.9          9125.8
   8        5486.5          9414.7
   9        6921.5          8743.5
  10          27.9          8997.8

---- ------------ --------------
avg 3762.5 7699.1 (+105%)

[1]: https://lore.kernel.org/all/[email protected]/
[2]: https://lore.kernel.org/linux-block/[email protected]/

---
Changes in v7:
- Rebase on v7.2-rc5.
- 1/5: New patch. Move the atomic context detection into the block
layer as bio_in_atomic() and convert erofs to use it, per Christoph.
- 2/5: Drop the 1-jiffie completion delay and schedule the completion
work immediately. Use a plain work item instead of delayed_work
accordingly, per Jan and Christoph.
- 2/5: Drop the local_lock and use local IRQ disabling instead to
resolve PREEMPT_RT concerns raised by Sashiko.
- 2/5: Drop the need_resched() requeue from the work function loop, per Christoph and Hillf.
- 2/5: Use BIO_COMPLETE_IN_TASK as a sentinel in bio_complete_in_task()
so a callback running from the worker does not re-defer, per Sashiko.
- 2/5: Requeue pending completions when a CPU comes back online, per
Sashiko.
- 4/5: Check for op_is_write() before deferring to avoid deferring read completions, per Sashiko.
- 4/5: Set BIO_COMPLETE_IN_TASK in __bh_submit() for dropbehind write
bios instead of introducing block_write_begin_iocb() and threading the
kiocb through buffer_head writeback. This was dead code.
- 5/5: Drop the blkdev_write_begin() conversion accordingly.
- Link to v6: https://lore.kernel.org/r/[email protected]

Changes in v6:
- Remove RFC tag
- Rebase on v7.1-rc3.
- 1/4: Revert to using a bio_list, per Jens.
- 1/4: Restructure and simplify work function loop.
- 1/4: Expose both the flag and procedural version, in order to allow
static and dynamic deferral decisions, per conversation with Matthew
and Christoph at LSFMM.
- 1/4: Use bio_in_atomic() predicate, per Christoph.
- 1/4: Use the CPU hot-unplug protocol from mm/vmstat.c, to take into
account use of delayed_work.
- 1/4: Mark the workqueue WQ_PERCPU.
- 1/4: Add comments.
- 3/4 and 4/4: Split into two patches, per Christoph.
- 3/4: Drop the cont_write_begin() change. Block devices don't go
through cont_write_begin(), so it was out of scope and was left over
from v1.
- Link to v5: https://lore.kernel.org/r/[email protected]

Changes in v5:
- 1/3: Replace local_lock + bio_list with struct llist, per Dave.
- 1/3: Use delayed_work with 1-jiffie delay, per Dave.
- 1/3: Add dedicated workqueue to avoid deadlocks, per Christoph.
- 1/3: Restructure work function as do/while loop and only schedule work originally when the list was previously empty, per Jens.
- 2/3: Delete IOMAP_IOEND_DONTCACHE and its NOMERGE entry, per Matthew
and Christoph.
- Link to v4: https://lore.kernel.org/r/[email protected]

Changes in v4:
- 1/3: Move dropbehind deferral from folio-level to bio-level using BIO_COMPLETE_IN_TASK, per Matthew and Jan.
- 1/3: Work function yields on need_resched() to avoid hogging the CPU,
per Jan.
- 2/3: New patch. Set BIO_COMPLETE_IN_TASK on iomap writeback bios for DONTCACHE folios, removing the need for XFS-specific workqueue
deferral.
- 3/3: Set BIO_COMPLETE_IN_TASK in submit_bh_wbc() for buffer_head
path.
- 3/3: Update commit message to mention CONFIG_BUFFER_HEAD=n path.
- Link to v3: https://lore.kernel.org/r/[email protected]

Changes in v3:
- 1/2: Convert dropbehind deferral to per-CPU folio_batches protected by local_lock using per-CPU work items, to reduce contention, per Jens.
- 1/2: Call folio_end_dropbehind_irq() directly from
folio_end_writeback(), per Jens.
- 1/2: Add CPU hotplug dead callback to drain the departing CPU's folio
batch.
- 2/2: Introduce block_write_begin_iocb(), per Christoph.
- 2/2: Dropped R-b due to changes.
- Link to v2: https://lore.kernel.org/r/[email protected]

Changes in v2:
- Add R-b from Jan Kara for 2/2.
- Add patch to defer dropbehind completion from IRQ context via a work
item (1/2).
- Add initial performance numbers to cover letter.
- Link to v1: https://lore.kernel.org/r/[email protected]

---
Tal Zussman (5):
block: introduce bio_in_atomic()
block: add task-context bio completion infrastructure
iomap: use BIO_COMPLETE_IN_TASK for dropbehind writeback
buffer: set BIO_COMPLETE_IN_TASK for dropbehind writeback
block: enable RWF_DONTCACHE for block devices

 block/bio.c               | 132 +++++++++++++++++++++++++++++++++++++++++++++-
 block/fops.c              |   2 +-
 fs/buffer.c               |   3 ++
 fs/erofs/zdata.c          |  11 +---
 fs/iomap/ioend.c          |   5 +-
 fs/xfs/xfs_aops.c         |   4 --
 include/linux/bio.h       |  39 ++++++++++++++
 include/linux/blk_types.h |   1 +
 include/linux/iomap.h     |   5 +-
 9 files changed, 180 insertions(+), 22 deletions(-)

---
base-commit: f5098b6bae761e346ebcd9da7f95622c04733cff
change-id: 20260218-blk-dontcache-338133dd045e

Best regards,

Signature · 1 line
Tal Zussman <[email protected]>
#1 lore raw

PATCH v7 1/5 block: introduce bio_in_atomic()

Move the atomic context detection logic from erofs's z_erofs_in_atomic() into the block layer as bio_in_atomic(). This helper returns true when the current context is unsafe for sleeping bio completion handlers (e.g., hard/soft IRQ, preempt-disabled).

The logic was originally added to erofs in commit c99fab6e80b7 ("erofs:
fix atomic context detection when !CONFIG_DEBUG_LOCK_ALLOC"). A subsequent patch will use it in the block layer's bio completion infrastructure, so move it to include/linux/bio.h where both subsystems can share it.

Convert erofs to call the new bio_in_atomic() directly.

---
 fs/erofs/zdata.c    | 11 +----------
 include/linux/bio.h | 15 +++++++++++++++
 2 files changed, 16 insertions(+), 10 deletions(-)
fs/erofs/zdata.c +1 -10
index 74520e910259..f796bb3ef53b 100644
@@ -1427,15 +1427,6 @@ static void z_erofs_decompressqueue_kthread_work(struct kthread_work *work)
 }
 #endif
 
-/* Use (kthread_)work in atomic contexts to minimize scheduling overhead */
-static inline bool z_erofs_in_atomic(void)
-{
-	if (IS_ENABLED(CONFIG_PREEMPTION) && rcu_preempt_depth())
-		return true;
-	if (!IS_ENABLED(CONFIG_PREEMPT_COUNT))
-		return true;
-	return !preemptible();
-}
 
 static void z_erofs_decompress_kickoff(struct z_erofs_decompressqueue *io,
 				       int bios)
@@ -1452,7 +1443,7 @@ static void z_erofs_decompress_kickoff(struct z_erofs_decompressqueue *io,
 
 	if (atomic_add_return(bios, &io->pending_bios))
 		return;
-	if (z_erofs_in_atomic()) {
+	if (bio_in_atomic()) {
 		/* See `sync_decompress` in sysfs-fs-erofs for more details */
 		if (sbi->sync_decompress == EROFS_SYNC_DECOMPRESS_AUTO)
 			sbi->sync_decompress = EROFS_SYNC_DECOMPRESS_FORCE_ON;
include/linux/bio.h +15 -0
index 8f33f717b14f..62a983758e09 100644
@@ -368,6 +368,21 @@ static inline struct bio *bio_alloc(struct block_device *bdev,
 
 void submit_bio(struct bio *bio);
 
+/**
+ * bio_in_atomic - check if the current context is unsafe for bio completion
+ *
+ * Return: %true in atomic contexts (e.g. hard/soft IRQ, preempt-disabled);
+ * %false when a bio can be safely completed in the current context.
+ */
+static inline bool bio_in_atomic(void)
+{
+	if (IS_ENABLED(CONFIG_PREEMPTION) && rcu_preempt_depth())
+		return true;
+	if (!IS_ENABLED(CONFIG_PREEMPT_COUNT))
+		return true;
+	return !preemptible();
+}
+
 extern void bio_endio(struct bio *);
 
 /**
Signature · 1 line
2.39.5
reviewer Reviewed-by on v7 1/5
#3 lore raw

Re: [PATCH v7 1/5] block: introduce bio_in_atomic()

In reply to Tal Zussman
Quoted 11 lines · Tal Zussman

Move the atomic context detection logic from erofs's z_erofs_in_atomic() into the block layer as bio_in_atomic(). This helper returns true when the current context is unsafe for sleeping bio completion handlers (e.g., hard/soft IRQ, preempt-disabled).

The logic was originally added to erofs in commit c99fab6e80b7 ("erofs:
fix atomic context detection when !CONFIG_DEBUG_LOCK_ALLOC"). A subsequent patch will use it in the block layer's bio completion infrastructure, so move it to include/linux/bio.h where both subsystems can share it.

Quoted 4 lines

Convert erofs to call the new bio_in_atomic() directly.

Looks good. Feel free to add:

Honza

Quoted 64 lines
---
 fs/erofs/zdata.c    | 11 +----------
 include/linux/bio.h | 15 +++++++++++++++
 2 files changed, 16 insertions(+), 10 deletions(-)
fs/erofs/zdata.c +1 -10
index 74520e910259..f796bb3ef53b 100644
@@ -1427,15 +1427,6 @@ static void z_erofs_decompressqueue_kthread_work(struct kthread_work *work)
 }
 #endif
 
-/* Use (kthread_)work in atomic contexts to minimize scheduling overhead */
-static inline bool z_erofs_in_atomic(void)
-{
-	if (IS_ENABLED(CONFIG_PREEMPTION) && rcu_preempt_depth())
-		return true;
-	if (!IS_ENABLED(CONFIG_PREEMPT_COUNT))
-		return true;
-	return !preemptible();
-}
 
 static void z_erofs_decompress_kickoff(struct z_erofs_decompressqueue *io,
 				       int bios)
@@ -1452,7 +1443,7 @@ static void z_erofs_decompress_kickoff(struct z_erofs_decompressqueue *io,
 
 	if (atomic_add_return(bios, &io->pending_bios))
 		return;
-	if (z_erofs_in_atomic()) {
+	if (bio_in_atomic()) {
 		/* See `sync_decompress` in sysfs-fs-erofs for more details */
 		if (sbi->sync_decompress == EROFS_SYNC_DECOMPRESS_AUTO)
 			sbi->sync_decompress = EROFS_SYNC_DECOMPRESS_FORCE_ON;
include/linux/bio.h +15 -0
index 8f33f717b14f..62a983758e09 100644
@@ -368,6 +368,21 @@ static inline struct bio *bio_alloc(struct block_device *bdev,
 
 void submit_bio(struct bio *bio);
 
+/**
+ * bio_in_atomic - check if the current context is unsafe for bio completion
+ *
+ * Return: %true in atomic contexts (e.g. hard/soft IRQ, preempt-disabled);
+ * %false when a bio can be safely completed in the current context.
+ */
+static inline bool bio_in_atomic(void)
+{
+	if (IS_ENABLED(CONFIG_PREEMPTION) && rcu_preempt_depth())
+		return true;
+	if (!IS_ENABLED(CONFIG_PREEMPT_COUNT))
+		return true;
+	return !preemptible();
+}
+
 extern void bio_endio(struct bio *);
 
 /**
Signature · 2 lines
-- 
2.39.5
Signature · 2 lines
Jan Kara <[email protected]>
SUSE Labs, CR
#4 lore raw

PATCH v7 2/5 block: add task-context bio completion infrastructure

In reply to Tal Zussman

Some bio completion handlers need to run from preemptible task context, but bio_endio() may be called from IRQ context (e.g., buffer_head writeback). Callers need a way to ensure their callback eventually runs from a sleepable context. Add infrastructure for that, in two forms:

1. BIO_COMPLETE_IN_TASK, a bio flag the submitter sets when it knows in advance that its callback needs task context (e.g., dropbehind writeback). bio_endio() sees the flag and offloads completion to a worker automatically.

2. bio_complete_in_task(), a helper that completion callbacks can invoke from within bi_end_io() when the deferral decision is dynamic (e.g., fserror reporting).

Both share a per-CPU list drained by a work item on a WQ_PERCPU workqueue. Producers push the bio onto the local CPU's list and schedule the work item, which then dispatches each bio's bi_end_io() from task context.

Both methods are gated on bio_in_atomic(), which returns true in any context where a sleeping bi_end_io() is unsafe, including non-preemptible task context.

Two CPU hotplug callbacks are used to drain remaining bios from the departing CPU's batch, while maintaining the per-CPU behavior. The CPUHP_AP_ONLINE_DYN callback disables the per-CPU work item while the CPU is still online, preventing it from running on an unbound worker later. CPUHP_BP_PREPARE_DYN then drains any bios added between disabling the work item and CPU offline.

---
 block/bio.c               | 132 +++++++++++++++++++++++++++++++++++++++++++++-
 include/linux/bio.h       |  24 +++++++++
 include/linux/blk_types.h |   1 +
 3 files changed, 156 insertions(+), 1 deletion(-)
block/bio.c +131 -1
index 6a2f6fc3413e..a9610950325b 100644
@@ -1741,6 +1741,61 @@ void bio_check_pages_dirty(struct bio *bio)
 	schedule_work(&bio_dirty_work);
 }
 
+/*
+ * Infrastructure for deferring bio completions to task-context via a per-CPU
+ * workqueue. Triggered either by the BIO_COMPLETE_IN_TASK bio flag (static
+ * decision at submit time) or by calling bio_complete_in_task() from
+ * bi_end_io() (dynamic decision at completion time).
+ */
+
+struct bio_complete_batch {
+	struct bio_list list;
+	struct work_struct work;
+	int cpu;
+};
+
+static DEFINE_PER_CPU(struct bio_complete_batch, bio_complete_batch);
+static struct workqueue_struct *bio_complete_wq;
+
+static void bio_complete_work_fn(struct work_struct *w)
+{
+	struct bio_complete_batch *batch =
+		container_of(w, struct bio_complete_batch, work);
+
+	while (1) {
+		struct bio_list list;
+		struct bio *bio;
+
+		local_irq_disable();
+		list = batch->list;
+		bio_list_init(&batch->list);
+		local_irq_enable();
+
+		if (bio_list_empty(&list))
+			break;
+
+		while ((bio = bio_list_pop(&list)))
+			bio->bi_end_io(bio);
+	}
+}
+
+void __bio_complete_in_task(struct bio *bio)
+{
+	struct bio_complete_batch *batch;
+	unsigned long flags;
+	bool was_empty;
+
+	local_irq_save(flags);
+	batch = this_cpu_ptr(&bio_complete_batch);
+	was_empty = bio_list_empty(&batch->list);
+	bio_list_add(&batch->list, bio);
+	local_irq_restore(flags);
+
+	if (was_empty)
+		queue_work_on(batch->cpu, bio_complete_wq, &batch->work);
+}
+EXPORT_SYMBOL_GPL(__bio_complete_in_task);
+
 static inline bool bio_remaining_done(struct bio *bio)
 {
 	/*
@@ -1815,7 +1870,9 @@ void bio_endio(struct bio *bio)
 	}
 #endif
 
-	if (bio->bi_end_io)
+	if (bio_flagged(bio, BIO_COMPLETE_IN_TASK) && bio_in_atomic())
+		__bio_complete_in_task(bio);
+	else if (bio->bi_end_io)
 		bio->bi_end_io(bio);
 }
 EXPORT_SYMBOL(bio_endio);
@@ -2001,6 +2058,55 @@ int bioset_init(struct bio_set *bs,
 }
 EXPORT_SYMBOL(bioset_init);
 
+static int bio_complete_batch_cpu_online(unsigned int cpu)
+{
+	struct bio_complete_batch *batch = &per_cpu(bio_complete_batch, cpu);
+
+	enable_work(&batch->work);
+	if (!bio_list_empty(&batch->list))
+		queue_work_on(cpu, bio_complete_wq, &batch->work);
+	return 0;
+}
+
+/*
+ * Disable this CPU's work item so that it cannot run on an unbound worker
+ * after the CPU is offlined.
+ */
+static int bio_complete_batch_cpu_down_prep(unsigned int cpu)
+{
+	disable_work_sync(&per_cpu(bio_complete_batch, cpu).work);
+	return 0;
+}
+
+/*
+ * Drain a dead CPU's deferred bio completions. The CPU is dead and the worker
+ * is canceled so no locking is needed.
+ */
+static int bio_complete_batch_cpu_dead(unsigned int cpu)
+{
+	struct bio_complete_batch *batch =
+		per_cpu_ptr(&bio_complete_batch, cpu);
+	struct bio *bio;
+
+	while ((bio = bio_list_pop(&batch->list)))
+		bio->bi_end_io(bio);
+
+	return 0;
+}
+
+static void __init bio_complete_batch_init(int cpu)
+{
+	struct bio_complete_batch *batch =
+		per_cpu_ptr(&bio_complete_batch, cpu);
+
+	bio_list_init(&batch->list);
+	INIT_WORK(&batch->work, bio_complete_work_fn);
+	batch->cpu = cpu;
+
+	if (!cpu_online(cpu))
+		disable_work_sync(&batch->work);
+}
+
 static int __init init_bio(void)
 {
 	int i;
@@ -2015,6 +2121,30 @@ static int __init init_bio(void)
 				SLAB_HWCACHE_ALIGN | SLAB_PANIC, NULL);
 	}
 
+	for_each_possible_cpu(i)
+		bio_complete_batch_init(i);
+
+	bio_complete_wq = alloc_workqueue("bio_complete",
+					   WQ_MEM_RECLAIM | WQ_PERCPU, 0);
+	if (!bio_complete_wq)
+		panic("bio: can't allocate bio_complete workqueue\n");
+
+	/*
+	 * bio task-context completion draining on hot-unplugged CPUs:
+	 *
+	 *   1. Stop the per-CPU work item while the CPU is still online, so
+	 *      that it cannot run on an unbound worker later.
+	 *   2. Drain leftover bios added between worker disabling and CPU
+	 *      offlining.
+	 */
+	cpuhp_setup_state_nocalls(CPUHP_AP_ONLINE_DYN,
+				  "block/bio:complete:online",
+				  bio_complete_batch_cpu_online,
+				  bio_complete_batch_cpu_down_prep);
+	cpuhp_setup_state_nocalls(CPUHP_BP_PREPARE_DYN,
+				  "block/bio:complete:dead",
+				  NULL, bio_complete_batch_cpu_dead);
+
 	cpuhp_setup_state_multi(CPUHP_BIO_DEAD, "block/bio:dead", NULL,
 					bio_cpu_dead);
 
include/linux/bio.h +24 -0
index 62a983758e09..a8091a3e9d87 100644
@@ -383,6 +383,30 @@ static inline bool bio_in_atomic(void)
 	return !preemptible();
 }
 
+void __bio_complete_in_task(struct bio *bio);
+
+/**
+ * bio_complete_in_task - ensure a bio is completed in preemptible task context
+ * @bio: bio to complete
+ *
+ * If called from non-task context, offload the bio completion to a worker
+ * thread and return %true. Else return %false and do nothing.
+ *
+ * Uses BIO_COMPLETE_IN_TASK as a sentinel: if set, the bio was already
+ * deferred and we are running in the worker — return %false so the
+ * callback proceeds instead of re-deferring.
+ */
+static inline bool bio_complete_in_task(struct bio *bio)
+{
+	if (bio_flagged(bio, BIO_COMPLETE_IN_TASK))
+		return false;
+	if (!bio_in_atomic())
+		return false;
+	bio_set_flag(bio, BIO_COMPLETE_IN_TASK);
+	__bio_complete_in_task(bio);
+	return true;
+}
+
 extern void bio_endio(struct bio *);
 
 /**
include/linux/blk_types.h +1 -0
index 8808ee76e73c..d49d97a050d0 100644
@@ -322,6 +322,7 @@ enum {
 	BIO_REMAPPED,
 	BIO_ZONE_WRITE_PLUGGING, /* bio handled through zone write plugging */
 	BIO_EMULATES_ZONE_APPEND, /* bio emulates a zone append operation */
+	BIO_COMPLETE_IN_TASK, /* complete bi_end_io() in task context */
 	BIO_FLAG_LAST
 };
 
Signature · 1 line
2.39.5
reviewer Reviewed-by on v7 2/5
#6 lore raw

Re: [PATCH v7 2/5] block: add task-context bio completion infrastructure

In reply to Tal Zussman
Quoted 30 lines · Tal Zussman

Some bio completion handlers need to run from preemptible task context, but bio_endio() may be called from IRQ context (e.g., buffer_head writeback). Callers need a way to ensure their callback eventually runs from a sleepable context. Add infrastructure for that, in two forms:

1. BIO_COMPLETE_IN_TASK, a bio flag the submitter sets when it knows in advance that its callback needs task context (e.g., dropbehind writeback). bio_endio() sees the flag and offloads completion to a worker automatically.

2. bio_complete_in_task(), a helper that completion callbacks can invoke from within bi_end_io() when the deferral decision is dynamic (e.g., fserror reporting).

Both share a per-CPU list drained by a work item on a WQ_PERCPU workqueue. Producers push the bio onto the local CPU's list and schedule the work item, which then dispatches each bio's bi_end_io() from task context.

Both methods are gated on bio_in_atomic(), which returns true in any context where a sleeping bi_end_io() is unsafe, including non-preemptible task context.

Two CPU hotplug callbacks are used to drain remaining bios from the departing CPU's batch, while maintaining the per-CPU behavior. The CPUHP_AP_ONLINE_DYN callback disables the per-CPU work item while the CPU is still online, preventing it from running on an unbound worker later. CPUHP_BP_PREPARE_DYN then drains any bios added between disabling the work item and CPU offline.

Quoted 4 lines

Looks good! Feel free to add:

Honza

Quoted 221 lines
---
 block/bio.c               | 132 +++++++++++++++++++++++++++++++++++++++++++++-
 include/linux/bio.h       |  24 +++++++++
 include/linux/blk_types.h |   1 +
 3 files changed, 156 insertions(+), 1 deletion(-)
block/bio.c +131 -1
index 6a2f6fc3413e..a9610950325b 100644
@@ -1741,6 +1741,61 @@ void bio_check_pages_dirty(struct bio *bio)
 	schedule_work(&bio_dirty_work);
 }
 
+/*
+ * Infrastructure for deferring bio completions to task-context via a per-CPU
+ * workqueue. Triggered either by the BIO_COMPLETE_IN_TASK bio flag (static
+ * decision at submit time) or by calling bio_complete_in_task() from
+ * bi_end_io() (dynamic decision at completion time).
+ */
+
+struct bio_complete_batch {
+	struct bio_list list;
+	struct work_struct work;
+	int cpu;
+};
+
+static DEFINE_PER_CPU(struct bio_complete_batch, bio_complete_batch);
+static struct workqueue_struct *bio_complete_wq;
+
+static void bio_complete_work_fn(struct work_struct *w)
+{
+	struct bio_complete_batch *batch =
+		container_of(w, struct bio_complete_batch, work);
+
+	while (1) {
+		struct bio_list list;
+		struct bio *bio;
+
+		local_irq_disable();
+		list = batch->list;
+		bio_list_init(&batch->list);
+		local_irq_enable();
+
+		if (bio_list_empty(&list))
+			break;
+
+		while ((bio = bio_list_pop(&list)))
+			bio->bi_end_io(bio);
+	}
+}
+
+void __bio_complete_in_task(struct bio *bio)
+{
+	struct bio_complete_batch *batch;
+	unsigned long flags;
+	bool was_empty;
+
+	local_irq_save(flags);
+	batch = this_cpu_ptr(&bio_complete_batch);
+	was_empty = bio_list_empty(&batch->list);
+	bio_list_add(&batch->list, bio);
+	local_irq_restore(flags);
+
+	if (was_empty)
+		queue_work_on(batch->cpu, bio_complete_wq, &batch->work);
+}
+EXPORT_SYMBOL_GPL(__bio_complete_in_task);
+
 static inline bool bio_remaining_done(struct bio *bio)
 {
 	/*
@@ -1815,7 +1870,9 @@ void bio_endio(struct bio *bio)
 	}
 #endif
 
-	if (bio->bi_end_io)
+	if (bio_flagged(bio, BIO_COMPLETE_IN_TASK) && bio_in_atomic())
+		__bio_complete_in_task(bio);
+	else if (bio->bi_end_io)
 		bio->bi_end_io(bio);
 }
 EXPORT_SYMBOL(bio_endio);
@@ -2001,6 +2058,55 @@ int bioset_init(struct bio_set *bs,
 }
 EXPORT_SYMBOL(bioset_init);
 
+static int bio_complete_batch_cpu_online(unsigned int cpu)
+{
+	struct bio_complete_batch *batch = &per_cpu(bio_complete_batch, cpu);
+
+	enable_work(&batch->work);
+	if (!bio_list_empty(&batch->list))
+		queue_work_on(cpu, bio_complete_wq, &batch->work);
+	return 0;
+}
+
+/*
+ * Disable this CPU's work item so that it cannot run on an unbound worker
+ * after the CPU is offlined.
+ */
+static int bio_complete_batch_cpu_down_prep(unsigned int cpu)
+{
+	disable_work_sync(&per_cpu(bio_complete_batch, cpu).work);
+	return 0;
+}
+
+/*
+ * Drain a dead CPU's deferred bio completions. The CPU is dead and the worker
+ * is canceled so no locking is needed.
+ */
+static int bio_complete_batch_cpu_dead(unsigned int cpu)
+{
+	struct bio_complete_batch *batch =
+		per_cpu_ptr(&bio_complete_batch, cpu);
+	struct bio *bio;
+
+	while ((bio = bio_list_pop(&batch->list)))
+		bio->bi_end_io(bio);
+
+	return 0;
+}
+
+static void __init bio_complete_batch_init(int cpu)
+{
+	struct bio_complete_batch *batch =
+		per_cpu_ptr(&bio_complete_batch, cpu);
+
+	bio_list_init(&batch->list);
+	INIT_WORK(&batch->work, bio_complete_work_fn);
+	batch->cpu = cpu;
+
+	if (!cpu_online(cpu))
+		disable_work_sync(&batch->work);
+}
+
 static int __init init_bio(void)
 {
 	int i;
@@ -2015,6 +2121,30 @@ static int __init init_bio(void)
 				SLAB_HWCACHE_ALIGN | SLAB_PANIC, NULL);
 	}
 
+	for_each_possible_cpu(i)
+		bio_complete_batch_init(i);
+
+	bio_complete_wq = alloc_workqueue("bio_complete",
+					   WQ_MEM_RECLAIM | WQ_PERCPU, 0);
+	if (!bio_complete_wq)
+		panic("bio: can't allocate bio_complete workqueue\n");
+
+	/*
+	 * bio task-context completion draining on hot-unplugged CPUs:
+	 *
+	 *   1. Stop the per-CPU work item while the CPU is still online, so
+	 *      that it cannot run on an unbound worker later.
+	 *   2. Drain leftover bios added between worker disabling and CPU
+	 *      offlining.
+	 */
+	cpuhp_setup_state_nocalls(CPUHP_AP_ONLINE_DYN,
+				  "block/bio:complete:online",
+				  bio_complete_batch_cpu_online,
+				  bio_complete_batch_cpu_down_prep);
+	cpuhp_setup_state_nocalls(CPUHP_BP_PREPARE_DYN,
+				  "block/bio:complete:dead",
+				  NULL, bio_complete_batch_cpu_dead);
+
 	cpuhp_setup_state_multi(CPUHP_BIO_DEAD, "block/bio:dead", NULL,
 					bio_cpu_dead);
 
include/linux/bio.h +24 -0
index 62a983758e09..a8091a3e9d87 100644
@@ -383,6 +383,30 @@ static inline bool bio_in_atomic(void)
 	return !preemptible();
 }
 
+void __bio_complete_in_task(struct bio *bio);
+
+/**
+ * bio_complete_in_task - ensure a bio is completed in preemptible task context
+ * @bio: bio to complete
+ *
+ * If called from non-task context, offload the bio completion to a worker
+ * thread and return %true. Else return %false and do nothing.
+ *
+ * Uses BIO_COMPLETE_IN_TASK as a sentinel: if set, the bio was already
+ * deferred and we are running in the worker — return %false so the
+ * callback proceeds instead of re-deferring.
+ */
+static inline bool bio_complete_in_task(struct bio *bio)
+{
+	if (bio_flagged(bio, BIO_COMPLETE_IN_TASK))
+		return false;
+	if (!bio_in_atomic())
+		return false;
+	bio_set_flag(bio, BIO_COMPLETE_IN_TASK);
+	__bio_complete_in_task(bio);
+	return true;
+}
+
 extern void bio_endio(struct bio *);
 
 /**
include/linux/blk_types.h +1 -0
index 8808ee76e73c..d49d97a050d0 100644
@@ -322,6 +322,7 @@ enum {
 	BIO_REMAPPED,
 	BIO_ZONE_WRITE_PLUGGING, /* bio handled through zone write plugging */
 	BIO_EMULATES_ZONE_APPEND, /* bio emulates a zone append operation */
+	BIO_COMPLETE_IN_TASK, /* complete bi_end_io() in task context */
 	BIO_FLAG_LAST
 };
 
Signature · 2 lines
-- 
2.39.5
Signature · 2 lines
Jan Kara <[email protected]>
SUSE Labs, CR
#7 lore raw

Re: [PATCH v7 2/5] block: add task-context bio completion infrastructure

In reply to Tal Zussman
Quoted 24 lines · Tal Zussman

Some bio completion handlers need to run from preemptible task context, but bio_endio() may be called from IRQ context (e.g., buffer_head writeback). Callers need a way to ensure their callback eventually runs from a sleepable context. Add infrastructure for that, in two forms:

1. BIO_COMPLETE_IN_TASK, a bio flag the submitter sets when it knows in advance that its callback needs task context (e.g., dropbehind writeback). bio_endio() sees the flag and offloads completion to a worker automatically.

2. bio_complete_in_task(), a helper that completion callbacks can invoke from within bi_end_io() when the deferral decision is dynamic (e.g., fserror reporting).

Both share a per-CPU list drained by a work item on a WQ_PERCPU workqueue. Producers push the bio onto the local CPU's list and schedule the work item, which then dispatches each bio's bi_end_io() from task context.

Both methods are gated on bio_in_atomic(), which returns true in any context where a sleeping bi_end_io() is unsafe, including non-preemptible task context.

Quoted 6 lines

Two CPU hotplug callbacks are used to drain remaining bios from the departing CPU's batch, while maintaining the per-CPU behavior. The CPUHP_AP_ONLINE_DYN callback disables the per-CPU work item while the CPU is still online, preventing it from running on an unbound worker later. CPUHP_BP_PREPARE_DYN then drains any bios added between disabling the work item and CPU offline.

Hi Tal,

Thanks very much for the patch. I’m just a bit confused that folio_end_dropbehind() only requires in_task(), and nothing more.

void folio_end_dropbehind(struct folio *folio)

{
        if (!folio_test_dropbehind(folio))
                return;

        /*
         * Hitting !in_task() should not happen off RWF_DONTCACHE writeback,
         * but can happen if normal writeback just happens to find dirty folios
         * that were created as part of uncached writeback, and that writeback
         * would otherwise not need non-IRQ handling. Just skip the
         * invalidation in that case.
         */
        if (in_task() && folio_trylock(folio)) {
                filemap_end_dropbehind(folio);
                folio_unlock(folio);
        }
}

But we’re applying a much stricter check with bio_in_atomic():

+static inline bool bio_in_atomic(void)
+{
+ if (IS_ENABLED(CONFIG_PREEMPTION) && rcu_preempt_depth())
+ return true;
+ if (!IS_ENABLED(CONFIG_PREEMPT_COUNT))
+ return true;
+ return !preemptible();
+}

I feel like something may be missing either in
folio_end_dropbehind() itself or somewhere in the
filesystem/block-device path.

Do you know why they don’t match each other?

Best Regards
Barry

#8 lore raw

Re: [PATCH v7 2/5] block: add task-context bio completion infrastructure

Quoted 34 lines · Barry Song

Hi Tal,

Thanks very much for the patch. I’m just a bit confused that folio_end_dropbehind() only requires in_task(), and nothing more.

void folio_end_dropbehind(struct folio *folio)

{
         if (!folio_test_dropbehind(folio))
                 return;

         /*
          * Hitting !in_task() should not happen off RWF_DONTCACHE writeback,
          * but can happen if normal writeback just happens to find dirty folios
          * that were created as part of uncached writeback, and that writeback
          * would otherwise not need non-IRQ handling. Just skip the
          * invalidation in that case.
          */
         if (in_task() && folio_trylock(folio)) {
                 filemap_end_dropbehind(folio);
                 folio_unlock(folio);
         }
}

But we’re applying a much stricter check with bio_in_atomic():

(quoted diff) +8 -0
+static inline bool bio_in_atomic(void)
+{
+     if (IS_ENABLED(CONFIG_PREEMPTION) && rcu_preempt_depth())
+           return true;
+     if (!IS_ENABLED(CONFIG_PREEMPT_COUNT))
+           return true;
+      return !preemptible();
+}
Quoted 9 lines

I feel like something may be missing either in
folio_end_dropbehind() itself or somewhere in the
filesystem/block-device path.

Do you know why they don’t match each other?

Best Regards
Barry

Hi Barry,

Apologies for the delay.

As far as I can tell, the in_task() in folio_end_dropbehind() wasn't meant to be a generic test for if it's safe to sleep, but rather filtered out the known unsafe case of buffer_head writeback completing from IRQ context. In contrast, the bio_in_atomic() check needs to be stricter since the bi_end_io() callbacks could sleep or the callers could be holding locks, RCU, etc.

I see there's some more discussion about this at [1], but I suspect there are other instances that are (incorrectly) using in_task() as a proxy for being able to sleep (like f2fs_read_end_io(), but I haven't looked too closely at it).

Additionally, it probably only makes sense to use bio_in_atomic() when there's some deferral path, since with CONFIG_PREEMPT_COUNT=n, it'll always return true.

Thanks,
Tal

[1]: https://lore.kernel.org/linux-fsdevel/[email protected]/T/

#9 lore raw

Re: [PATCH v7 2/5] block: add task-context bio completion infrastructure

Quoted 56 lines · Tal Zussman
Quoted 34 lines · Barry Song

Hi Tal,

Thanks very much for the patch. I’m just a bit confused that folio_end_dropbehind() only requires in_task(), and nothing more.

void folio_end_dropbehind(struct folio *folio)

{
         if (!folio_test_dropbehind(folio))
                 return;

         /*
          * Hitting !in_task() should not happen off RWF_DONTCACHE writeback,
          * but can happen if normal writeback just happens to find dirty folios
          * that were created as part of uncached writeback, and that writeback
          * would otherwise not need non-IRQ handling. Just skip the
          * invalidation in that case.
          */
         if (in_task() && folio_trylock(folio)) {
                 filemap_end_dropbehind(folio);
                 folio_unlock(folio);
         }
}

But we’re applying a much stricter check with bio_in_atomic():

(quoted diff) +8 -0
+static inline bool bio_in_atomic(void)
+{
+     if (IS_ENABLED(CONFIG_PREEMPTION) && rcu_preempt_depth())
+           return true;
+     if (!IS_ENABLED(CONFIG_PREEMPT_COUNT))
+           return true;
+      return !preemptible();
+}
Quoted 9 lines

I feel like something may be missing either in
folio_end_dropbehind() itself or somewhere in the
filesystem/block-device path.

Do you know why they don’t match each other?

Best Regards
Barry

Hi Barry,

Apologies for the delay.

As far as I can tell, the in_task() in folio_end_dropbehind() wasn't meant to be a generic test for if it's safe to sleep, but rather filtered out the known unsafe case of buffer_head writeback completing from IRQ context. In contrast, the bio_in_atomic() check needs to be stricter since the bi_end_io() callbacks could sleep or the callers could be holding locks, RCU, etc.

Quoted 6 lines

I see there's some more discussion about this at [1], but I suspect there are other instances that are (incorrectly) using in_task() as a proxy for being able to sleep (like f2fs_read_end_io(), but I haven't looked too closely at it).

Additionally, it probably only makes sense to use bio_in_atomic() when there's some deferral path, since with CONFIG_PREEMPT_COUNT=n, it'll always return true.

Tal, thanks very much for your reply and the kind clarification. It seems you're right, and I think we should also use bio_in_atomic() in folio_end_dropbehind().

On the other hand, as I mentioned in another thread[1], I really feel that bio_in_atomic() has nothing to do with bio at all; it's really a scheduler thing. We may want to move it to the scheduler layer and discuss it there.

[1] https://lore.kernel.org/linux-mm/CAGsJ_4xxDpeaPv9Uh_1qsuXsMLkx+J0X=2V6342idLFO5jkTBw@mail.gmail.com/

Best Regards
Barry

#10 lore raw

PATCH v7 3/5 iomap: use BIO_COMPLETE_IN_TASK for dropbehind writeback

In reply to Tal Zussman

Set BIO_COMPLETE_IN_TASK on iomap writeback bios when a dropbehind folio is added. This ensures that bi_end_io runs in task context, where folio_end_dropbehind() can safely invalidate folios.

With the bio layer now handling task-context deferral generically, IOMAP_IOEND_DONTCACHE is no longer needed, as XFS no longer needs to route DONTCACHE ioends through its completion workqueue. Remove the flag and its NOMERGE entry.

Without the NOMERGE, regular I/Os that get merged with a dropbehind folio will also have their completion deferred to task context.

---
 fs/iomap/ioend.c      | 5 +++--
 fs/xfs/xfs_aops.c     | 4 ----
 include/linux/iomap.h | 5 +----
 3 files changed, 4 insertions(+), 10 deletions(-)
fs/iomap/ioend.c +3 -2
index 30468d51b5ad..1ae8a8fb8503 100644
@@ -238,8 +238,6 @@ ssize_t iomap_add_to_ioend(struct iomap_writepage_ctx *wpc, struct folio *folio,
 
 	if (wpc->iomap.flags & IOMAP_F_SHARED)
 		ioend_flags |= IOMAP_IOEND_SHARED;
-	if (folio_test_dropbehind(folio))
-		ioend_flags |= IOMAP_IOEND_DONTCACHE;
 	if (pos == wpc->iomap.offset && (wpc->iomap.flags & IOMAP_F_BOUNDARY))
 		ioend_flags |= IOMAP_IOEND_BOUNDARY;
 
@@ -256,6 +254,9 @@ ssize_t iomap_add_to_ioend(struct iomap_writepage_ctx *wpc, struct folio *folio,
 	if (!bio_add_folio(&ioend->io_bio, folio, map_len, poff))
 		goto new_ioend;
 
+	if (folio_test_dropbehind(folio))
+		bio_set_flag(&ioend->io_bio, BIO_COMPLETE_IN_TASK);
+
 	/*
 	 * Clamp io_offset and io_size to the incore EOF so that ondisk
 	 * file size updates in the ioend completion are byte-accurate.
fs/xfs/xfs_aops.c +0 -4
index 51293b6f331f..cd8de8c82d78 100644
@@ -522,10 +522,6 @@ xfs_ioend_needs_wq_completion(
 	if (ioend->io_flags & (IOMAP_IOEND_UNWRITTEN | IOMAP_IOEND_SHARED))
 		return true;
 
-	/* Page cache invalidation cannot be done in irq context. */
-	if (ioend->io_flags & IOMAP_IOEND_DONTCACHE)
-		return true;
-
 	return false;
 }
 
include/linux/iomap.h +1 -4
index 56b43d594e6e..68af4935a107 100644
@@ -404,16 +404,13 @@ sector_t iomap_bmap(struct address_space *mapping, sector_t bno,
 #define IOMAP_IOEND_BOUNDARY		(1U << 2)
 /* is direct I/O */
 #define IOMAP_IOEND_DIRECT		(1U << 3)
-/* is DONTCACHE I/O */
-#define IOMAP_IOEND_DONTCACHE		(1U << 4)
 
 /*
  * Flags that if set on either ioend prevent the merge of two ioends.
  * (IOMAP_IOEND_BOUNDARY also prevents merges, but only one-way)
  */
 #define IOMAP_IOEND_NOMERGE_FLAGS \
-	(IOMAP_IOEND_SHARED | IOMAP_IOEND_UNWRITTEN | IOMAP_IOEND_DIRECT | \
-	 IOMAP_IOEND_DONTCACHE)
+	(IOMAP_IOEND_SHARED | IOMAP_IOEND_UNWRITTEN | IOMAP_IOEND_DIRECT)
 
 /*
  * Structure for writeback I/O completions.
Signature · 1 line
2.39.5
#11 lore raw

Re: [PATCH v7 3/5] iomap: use BIO_COMPLETE_IN_TASK for dropbehind writeback

Quoted 9 lines · Tal Zussman

Set BIO_COMPLETE_IN_TASK on iomap writeback bios when a dropbehind folio is added. This ensures that bi_end_io runs in task context, where folio_end_dropbehind() can safely invalidate folios.

With the bio layer now handling task-context deferral generically, IOMAP_IOEND_DONTCACHE is no longer needed, as XFS no longer needs to route DONTCACHE ioends through its completion workqueue. Remove the flag and its NOMERGE entry.

Quoted 5 lines

Without the NOMERGE, regular I/Os that get merged with a dropbehind folio will also have their completion deferred to task context.

This seems to be missing the bug fix you identified for the double deferral?

I although thought a bit how that could be done cleaner, see below:

fs/iomap/ioend.c ssize_t iomap_add_to_ioend(struct iomap_writepage_ctx *wpc, struct folio *folio, · line 242

Quoted 3 lines
fs/iomap/ioend.c +0 -2
 		ioend_flags |= IOMAP_IOEND_SHARED;
-	if (folio_test_dropbehind(folio))
-		ioend_flags |= IOMAP_IOEND_DONTCACHE;

I'd keep this flag.

fs/iomap/ioend.c ssize_t iomap_add_to_ioend(struct iomap_writepage_ctx *wpc, struct folio *folio, · line 258

Quoted 9 lines
fs/iomap/ioend.c +2 -0
 	if (pos == wpc->iomap.offset && (wpc->iomap.flags & IOMAP_F_BOUNDARY))
 		ioend_flags |= IOMAP_IOEND_BOUNDARY;
@@ -256,6 +254,9 @@ ssize_t iomap_add_to_ioend(struct iomap_writepage_ctx *wpc, struct folio *folio,
 	if (!bio_add_folio(&ioend->io_bio, folio, map_len, poff))
 		goto new_ioend;
 
+	if (folio_test_dropbehind(folio))
+		bio_set_flag(&ioend->io_bio, BIO_COMPLETE_IN_TASK);

And move setting the flag to iomap_ioend_writeback_submit, and only for the case where no io_bio.bi_end_io is already assigned.

#12 lore raw

Re: [PATCH v7 3/5] iomap: use BIO_COMPLETE_IN_TASK for dropbehind writeback

Quoted 10 lines · Christoph Hellwig
Quoted 9 lines · Tal Zussman

Set BIO_COMPLETE_IN_TASK on iomap writeback bios when a dropbehind folio is added. This ensures that bi_end_io runs in task context, where folio_end_dropbehind() can safely invalidate folios.

With the bio layer now handling task-context deferral generically, IOMAP_IOEND_DONTCACHE is no longer needed, as XFS no longer needs to route DONTCACHE ioends through its completion workqueue. Remove the flag and its NOMERGE entry.

Quoted 8 lines
Quoted 5 lines

Without the NOMERGE, regular I/Os that get merged with a dropbehind folio will also have their completion deferred to task context.

This seems to be missing the bug fix you identified for the double deferral?

Sorry, I misunderstood and assumed you'd take care of it in a follow-up. I'll adjust it as below and send v8.

Quoted 15 lines

I although thought a bit how that could be done cleaner, see below:

fs/iomap/ioend.c ssize_t iomap_add_to_ioend(struct iomap_writepage_ctx *wpc, struct folio *folio, · line 242

Quoted 3 lines
fs/iomap/ioend.c +0 -2
 		ioend_flags |= IOMAP_IOEND_SHARED;
-	if (folio_test_dropbehind(folio))
-		ioend_flags |= IOMAP_IOEND_DONTCACHE;

I'd keep this flag.

Quoted 7 lines
fs/iomap/ioend.c +0 -0
 	if (pos == wpc->iomap.offset && (wpc->iomap.flags & IOMAP_F_BOUNDARY))
 		ioend_flags |= IOMAP_IOEND_BOUNDARY;
@@ -256,6 +254,9 @@ ssize_t iomap_add_to_ioend(struct iomap_writepage_ctx *wpc, struct folio *folio,
 	if (!bio_add_folio(&ioend->io_bio, folio, map_len, poff))
 		goto new_ioend;
 
Quoted 6 lines

fs/iomap/ioend.c ssize_t iomap_add_to_ioend(struct iomap_writepage_ctx *wpc, struct folio *folio, · line 258

Quoted 2 lines
fs/iomap/ioend.c +2 -0
+	if (folio_test_dropbehind(folio))
+		bio_set_flag(&ioend->io_bio, BIO_COMPLETE_IN_TASK);

And move setting the flag to iomap_ioend_writeback_submit, and only for the case where no io_bio.bi_end_io is already assigned.

To confirm, this would only work because XFS provides its own task context and is the only consumer of iomap that sets its own io_bio.bi_end_io. If another filesystem set it and didn't provide task context, this would break, right? I can add a comment about this requirement, but I want to confirm the intent, as it feels a little fragile as-is.

#13 lore raw

Re: [PATCH v7 3/5] iomap: use BIO_COMPLETE_IN_TASK for dropbehind writeback

Quoted 2 lines · Tal Zussman

Sorry, I misunderstood and assumed you'd take care of it in a follow-up. I'll adjust it as below and send v8.

My plan was to move from the version you had to something better in a follow-up. Anyway, Jens merged it as-is so I'll look into a fixup. Travelling for two conferences this week might make this a bit slow, though.

Quoted 9 lines
Quoted 3 lines

And move setting the flag to iomap_ioend_writeback_submit, and only for the case where no io_bio.bi_end_io is already assigned.

To confirm, this would only work because XFS provides its own task context and is the only consumer of iomap that sets its own io_bio.bi_end_io. If another filesystem set it and didn't provide task context, this would break, right? I can add a comment about this requirement, but I want to confirm the intent, as it feels a little fragile as-is.

Yes. And I think this might be a good time to actually open code iomap_ioend_writeback_submit in XFS and remove the "accept an overriden bi_io_end" thing entirely. Back in the original code that was required, but now that the submission happens from a file system methods there is no good reason for keeping it.

#14 lore raw

PATCH v7 4/5 buffer: set BIO_COMPLETE_IN_TASK for dropbehind writeback

In reply to Tal Zussman

Set BIO_COMPLETE_IN_TASK in __bh_submit() for write bios when the folio has dropbehind set, so that buffer_head writeback completions get deferred to task context where folio_end_dropbehind() can safely invalidate folios.

Read completions are not deferred since dropbehind invalidation for reads is handled synchronously by the reader.

---
 fs/buffer.c | 3 +++
 1 file changed, 3 insertions(+)
fs/buffer.c +3 -0
index 9af5f061a1f8..6f099847240e 100644
@@ -1203,6 +1203,9 @@ static void __bh_submit(struct buffer_head *bh, blk_opf_t opf,
 
 	bio = bio_alloc(bh->b_bdev, 1, opf, GFP_NOIO);
 
+	if (folio_test_dropbehind(bh->b_folio) && op_is_write(opf))
+		bio_set_flag(bio, BIO_COMPLETE_IN_TASK);
+
 	if (IS_ENABLED(CONFIG_FS_ENCRYPTION))
 		buffer_set_crypto_ctx(bio, bh, GFP_NOIO);
 
Signature · 1 line
2.39.5
reviewer Reviewed-by on v7 4/5
#15 lore raw

Re: [PATCH v7 4/5] buffer: set BIO_COMPLETE_IN_TASK for dropbehind writeback

Quoted 10 lines · Tal Zussman

Set BIO_COMPLETE_IN_TASK in __bh_submit() for write bios when the folio has dropbehind set, so that buffer_head writeback completions get deferred to task context where folio_end_dropbehind() can safely invalidate folios.

Read completions are not deferred since dropbehind invalidation for reads is handled synchronously by the reader.

Looks good. Feel free to add:

Honza

Quoted 22 lines
---
 fs/buffer.c | 3 +++
 1 file changed, 3 insertions(+)
fs/buffer.c +3 -0
index 9af5f061a1f8..6f099847240e 100644
@@ -1203,6 +1203,9 @@ static void __bh_submit(struct buffer_head *bh, blk_opf_t opf,
 
 	bio = bio_alloc(bh->b_bdev, 1, opf, GFP_NOIO);
 
+	if (folio_test_dropbehind(bh->b_folio) && op_is_write(opf))
+		bio_set_flag(bio, BIO_COMPLETE_IN_TASK);
+
 	if (IS_ENABLED(CONFIG_FS_ENCRYPTION))
 		buffer_set_crypto_ctx(bio, bh, GFP_NOIO);
 
Signature · 2 lines
-- 
2.39.5
Signature · 2 lines
Jan Kara <[email protected]>
SUSE Labs, CR
#16 lore raw

PATCH v7 5/5 block: enable RWF_DONTCACHE for block devices

In reply to Tal Zussman

Block device buffered reads and writes already pass through filemap_read() and iomap_file_buffered_write() respectively, both of which handle IOCB_DONTCACHE. Enable RWF_DONTCACHE for block device files by setting FOP_DONTCACHE in def_blk_fops.

For CONFIG_BUFFER_HEAD=y, writeback goes through buffer_head's __bh_submit() which sets BIO_COMPLETE_IN_TASK on dropbehind folios. For CONFIG_BUFFER_HEAD=n, writeback goes through iomap which handles it via BIO_COMPLETE_IN_TASK on the ioend bio.

This support is useful for databases that operate on raw block devices, among other userspace applications.

---
 block/fops.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
block/fops.c +1 -1
index 15783a6180de..8b39c53b6393 100644
@@ -941,7 +941,7 @@ const struct file_operations def_blk_fops = {
 	.splice_write	= iter_file_splice_write,
 	.fallocate	= blkdev_fallocate,
 	.uring_cmd	= blkdev_uring_cmd,
-	.fop_flags	= FOP_BUFFER_RASYNC,
+	.fop_flags	= FOP_BUFFER_RASYNC | FOP_DONTCACHE,
 };
 
 static __init int blkdev_init(void)
Signature · 1 line
2.39.5
reviewer Reviewed-by on v7 5/5
#17 lore raw
Quoted 10 lines · Tal Zussman

Block device buffered reads and writes already pass through filemap_read() and iomap_file_buffered_write() respectively, both of which handle IOCB_DONTCACHE. Enable RWF_DONTCACHE for block device files by setting FOP_DONTCACHE in def_blk_fops.

For CONFIG_BUFFER_HEAD=y, writeback goes through buffer_head's __bh_submit() which sets BIO_COMPLETE_IN_TASK on dropbehind folios. For CONFIG_BUFFER_HEAD=n, writeback goes through iomap which handles it via BIO_COMPLETE_IN_TASK on the ioend bio.

Quoted 5 lines

This support is useful for databases that operate on raw block devices, among other userspace applications.

Looks good. Feel free to add:

Honza

Quoted 21 lines
---
 block/fops.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
block/fops.c +1 -1
index 15783a6180de..8b39c53b6393 100644
@@ -941,7 +941,7 @@ const struct file_operations def_blk_fops = {
 	.splice_write	= iter_file_splice_write,
 	.fallocate	= blkdev_fallocate,
 	.uring_cmd	= blkdev_uring_cmd,
-	.fop_flags	= FOP_BUFFER_RASYNC,
+	.fop_flags	= FOP_BUFFER_RASYNC | FOP_DONTCACHE,
 };
 
 static __init int blkdev_init(void)
Signature · 2 lines
-- 
2.39.5
Signature · 2 lines
Jan Kara <[email protected]>
SUSE Labs, CR
#18 lore raw
In reply to Tal Zussman
Quoted 25 lines · Tal Zussman

Block device buffered reads and writes already pass through filemap_read() and iomap_file_buffered_write() respectively, both of which handle IOCB_DONTCACHE. Enable RWF_DONTCACHE for block device files by setting FOP_DONTCACHE in def_blk_fops.

For CONFIG_BUFFER_HEAD=y, writeback goes through buffer_head's __bh_submit() which sets BIO_COMPLETE_IN_TASK on dropbehind folios. For CONFIG_BUFFER_HEAD=n, writeback goes through iomap which handles it via BIO_COMPLETE_IN_TASK on the ioend bio.

This support is useful for databases that operate on raw block devices, among other userspace applications.

---
 block/fops.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
block/fops.c +0 -0
index 15783a6180de..8b39c53b6393 100644

block/fops.c const struct file_operations def_blk_fops = { · line 947

Quoted 8 lines
block/fops.c +1 -1
@@ -941,7 +941,7 @@ const struct file_operations def_blk_fops = {
 	.splice_write	= iter_file_splice_write,
 	.fallocate	= blkdev_fallocate,
 	.uring_cmd	= blkdev_uring_cmd,
-	.fop_flags	= FOP_BUFFER_RASYNC,
+	.fop_flags	= FOP_BUFFER_RASYNC | FOP_DONTCACHE,
 };
 
 static __init int blkdev_init(void)

It looks like Sashiko found a couple pre-existing issues in this file [1]. I've reproduced the first and looking into the second. Will send patches once confirmed.

(FWIW the other issues Sashiko reported seem overblown.)

[1]: https://sashiko.dev/#/patchset/20260730-blk-dontcache-v7-0-3e8e6850068d%40columbia.edu?part=5

maintainer Applied
#19 lore raw
In reply to Tal Zussman
Quoted 8 lines · Tal Zussman

Add support for using RWF_DONTCACHE with block devices.

Dropbehind pruning needs to be done in non-IRQ context, but block devices complete writeback in IRQ context. To fix this, we defer dropbehind invalidation to task context. Add infrastructure that lets bi_end_io callbacks run from a worker, in two forms:

[...]

Applied, thanks!

[1/5] block: introduce bio_in_atomic()

      commit: 8ed0831b5263e1aa48d99063795a1f20e4e69b50
[2/5] block: add task-context bio completion infrastructure
      commit: 77e73fa61b3888c9765a549af02c928372522cc9
[3/5] iomap: use BIO_COMPLETE_IN_TASK for dropbehind writeback
      commit: efbde6f9f449da3306f5b5d32f08829954ffb44d
[4/5] buffer: set BIO_COMPLETE_IN_TASK for dropbehind writeback
      commit: a2c924c240e74dc2dd14ff763245dc86b93db714
[5/5] block: enable RWF_DONTCACHE for block devices
      commit: 8b5ffb43ae9d27eed9165e1962963d685d879219

Best regards,

Signature · 1 line
Jens Axboe
#20 lore raw

RWF_DONTCACHE regression fix for xfs

In reply to Tal Zussman

Below is Tal's suggested fixup for xfs with the whitespace damage fixeѕ. It should probably be attributed to Tal instead. Can you handled it from here and send it to Jens with your signoff/authorship to avoid the performance regression in XFS?

From: Christoph Hellwig <[email protected]>
Subject: xfs: avoid double deferrals for RWF_DONTCACHE writes

XFS already defers some writes to a workqueue when transactions are needed to process the I/O completion. Disable the block layer bio task completion in this case to avoid a major performance drop.

Fixes efbde6f9f449 ("iomap: use BIO_COMPLETE_IN_TASK for dropbehind writeback")
---
 fs/xfs/xfs_aops.c | 15 +++++++++++++--
 1 file changed, 13 insertions(+), 2 deletions(-)
fs/xfs/xfs_aops.c +13 -2
index f73e26d9c9f9..059846a8d65c 100644
@@ -372,10 +372,14 @@ xfs_writeback_submit(
 	}
 
 	/*
-	 * Send ioends that might require a transaction to the completion wq.
+	 * Send ioends that might require a transaction to the completion wq,
+	 * and disable the block layer task completion for them as there is no
+	 * need to defer twice.
 	 */
-	if (xfs_ioend_needs_wq_completion(ioend))
+	if (xfs_ioend_needs_wq_completion(ioend)) {
 		ioend->io_bio.bi_end_io = xfs_end_bio;
+		bio_clear_flag(&ioend->io_bio, BIO_COMPLETE_IN_TASK);
+	}
 
 	return iomap_ioend_writeback_submit(wpc, error);
 }
@@ -481,7 +485,14 @@ xfs_zoned_writeback_submit(
 {
 	struct iomap_ioend		*ioend = wpc->wb_ctx;
 
+	/*
+	 * Defer all completions to our workqueue as all zoned writes require a
+	 * transaction to be persisted.  This also means we never need the block
+	 * layer in-task completion for a task context.
+	 */
 	ioend->io_bio.bi_end_io = xfs_end_bio;
+	bio_clear_flag(&ioend->io_bio, BIO_COMPLETE_IN_TASK);
+
 	if (error) {
 		ioend->io_bio.bi_status = errno_to_blk_status(error);
 		bio_endio(&ioend->io_bio);
Signature · 1 line
2.53.0
#21 lore raw

Re: RWF_DONTCACHE regression fix for xfs

Quoted 5 lines · Christoph Hellwig

Below is Tal's suggested fixup for xfs with the whitespace damage fixeѕ. It should probably be attributed to Tal instead. Can you handled it from here and send it to Jens with your signoff/authorship to avoid the performance regression in XFS?

Yes, will do shortly. Thanks Christoph!

Quoted 51 lines

From: Christoph Hellwig <[email protected]>
Subject: xfs: avoid double deferrals for RWF_DONTCACHE writes

XFS already defers some writes to a workqueue when transactions are needed to process the I/O completion. Disable the block layer bio task completion in this case to avoid a major performance drop.

Fixes efbde6f9f449 ("iomap: use BIO_COMPLETE_IN_TASK for dropbehind writeback")
---
  fs/xfs/xfs_aops.c | 15 +++++++++++++--
  1 file changed, 13 insertions(+), 2 deletions(-)
fs/xfs/xfs_aops.c +13 -2
index f73e26d9c9f9..059846a8d65c 100644
@@ -372,10 +372,14 @@ xfs_writeback_submit(
  	}
 
  	/*
-	 * Send ioends that might require a transaction to the completion wq.
+	 * Send ioends that might require a transaction to the completion wq,
+	 * and disable the block layer task completion for them as there is no
+	 * need to defer twice.
  	 */
-	if (xfs_ioend_needs_wq_completion(ioend))
+	if (xfs_ioend_needs_wq_completion(ioend)) {
  		ioend->io_bio.bi_end_io = xfs_end_bio;
+		bio_clear_flag(&ioend->io_bio, BIO_COMPLETE_IN_TASK);
+	}
 
  	return iomap_ioend_writeback_submit(wpc, error);
  }
@@ -481,7 +485,14 @@ xfs_zoned_writeback_submit(
  {
  	struct iomap_ioend		*ioend = wpc->wb_ctx;
 
+	/*
+	 * Defer all completions to our workqueue as all zoned writes require a
+	 * transaction to be persisted.  This also means we never need the block
+	 * layer in-task completion for a task context.
+	 */
  	ioend->io_bio.bi_end_io = xfs_end_bio;
+	bio_clear_flag(&ioend->io_bio, BIO_COMPLETE_IN_TASK);
+
  	if (error) {
  		ioend->io_bio.bi_status = errno_to_blk_status(error);
  		bio_endio(&ioend->io_bio);
Signature · 1 line
2.53.0