block: charge passthrough requests to the submitter's cgroup

From: Tao Cui <[email protected]>

While looking at cgroup IO control coverage, I stumbled into a hole that is probably familiar to people who have worked on passthrough:
SG_IO, bsg, and the NVMe ioctl / uring command paths build requests directly and dispatch them through blk_execute_rq{,_nowait}(), never passing through submit_bio(). The mapped bio carries no blkcg association, so the bytes never show up in cgroup io.stat, and none of the rq_qos policies (iocost, iolatency, wbt) see these commands; blk-throttle, hooking submit_bio_noacct() directly, is equally blind. On a scsi_debug device with iocost enabled and vrate pinned to its 1% floor, a direct writer in a cgroup is throttled ~10x while the same cgroup issuing sg_dd writes runs at full device speed with io.stat staying at zero.

This RFC is a prototype asking for feedback on one possible direction, not a proposal I am attached to: is charging passthrough at dispatch time the right layer and the right semantics at all?

The patch associates the mapped bio with the submitter's blkcg in blk_execute_rq{,_nowait}(), runs the regular bio accounting (blk_cgroup_bio_start()) and the rq_qos throttle path with it, and gates by opcode (READ/WRITE/DRV_IN/DRV_OUT, mapping the latter two to READ/WRITE for io.stat classification) and to queues that already have a gendisk so probing stays exempt. On linux-next, with scsi_debug + sg_dd and qemu emulated nvme + an NVME_IOCTL_IO64_CMD loop, the transferred bytes are fully accounted in the issuing cgroup, and with vrate clamped the pacing bites: the same 128MB SG_IO run takes 0.025s unthrottled and 31.7s at the 1% floor, with the issuing task sleeping on the iocost waitqueue like any bio submitter.

Questions I am unsure about:

1. Is always-on the right default, or should this hide behind the queue/iostats_passthrough opt-in from the passthrough iostats series, keeping default behavior unchanged?
2. The throttle may sleep; all callers I found (ioctl / uring_cmd submit, target and error handling kthreads) are sleepable, and kthread-issued charges land on root and turn into no-ops - but I may have missed paths, and I'd rather ask than assume.
3. What should happen on nvme multipath failover, where the bio is re-routed to another queue but carries the original queue's blkg? Untested; guidance welcome.
4. wbt also gains passthrough throttling through the same rq_qos walk - probably benign since wbt targets buffered writeback, but it is a behavior change I did not set out to make.
5. blk-throttle does not benefit (it is not an rq_qos policy, and its queue-and-resubmit model does not fit bios already bound to a dispatched request); would a synchronous wait variant there be welcome as follow-up work, or is leaving it alone preferred?

Reproduction notes in case anyone tries: sg_dd coalesces into very large single commands by default (use bpt=1), and a short burst from a single cgroup rides the initial budget, so sustained transfers are what show the pacing.

Tao Cui (1):
block: charge passthrough requests to the submitter's cgroup

 block/blk-mq.c | 72 ++++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 72 insertions(+)

2.43.0

From: Tao Cui <[email protected]>

Passthrough requests (SG_IO, bsg, nvme passthrough ioctls and uring commands) are dispatched via blk_execute_rq{,_nowait}() without ever passing through submit_bio(), so the bio mapped by blk_rq_map_user() carries no blkcg association: the transferred bytes never show up in cgroup io.stat, and every rq_qos policy on the queue (iocost, iolatency, wbt) is bypassed, as is blk-throttle, which hooks submit_bio_noacct() directly rather than going through rq_qos.

A quick demonstration on a scsi_debug device with iocost enabled and vrate pinned to its 1% floor: a direct fio writer was throttled ~10x while the same cgroup issuing sg_dd writes ran at full device speed with zero io.stat accounting.

Associate the mapped bio with the submitter's blkcg at dispatch time and run the regular bio accounting (blk_cgroup_bio_start()) and rq_qos throttle paths with it. DRV_IN/DRV_OUT commands are mapped to READ/WRITE so io.stat classifies their bytes normally; request completion already pairs with the throttle through bio_endio() -> rq_qos_done_bio().

The charge is gated by opcode (READ/WRITE/DRV_IN/DRV_OUT) and to queues that already have a gendisk: commands issued during device probing (SCSI INQUIRY etc.) have no gendisk yet and stay exempt, following the same probe-exemption reasoning as the passthrough iostats support.

RFC notes:
- validated on linux-next with scsi_debug + sg_dd (SG_IO) and qemu emulated nvme + a NVME_IOCTL_IO64_CMD loop: in both cases the transferred bytes are fully accounted (wbytes/wios) in the issuing cgroup, and the issuing task is observed waiting on the iocost waitqueue with vrate clamped;
- iolatency gains the same coverage for free, and so does wbt:
passthrough writes now pass through wbt_wait() like bio-path writes, a behavior change worth calling out even though wbt targets buffered writeback and direct passthrough rarely hits it; blk-throttle does not: it hooks
submit_bio_noacct() directly and is not an rq_qos policy, so io.max stays unenforced for passthrough (measured). Its queue-and-resubmit throttling model would also need a synchronous variant for request-bound bios;
- nvme uring commands share the same blk_execute_rq_nowait() dispatch as the validated ioctl path;
- the charge runs in the submitter's context and may sleep in rq_qos throttling; all data-op callers found run in sleepable context, but this deserves reviewer attention.

---
 block/blk-mq.c | 72 ++++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 72 insertions(+)

index a26a11c73ee3..2dd59f6df321 100644
@@ -10,6 +10,7 @@
 #include <linux/backing-dev.h>
 #include <linux/bio.h>
 #include <linux/blkdev.h>
+#include "blk-cgroup.h"
 #include <linux/blk-integrity.h>
 #include <linux/kmemleak.h>
 #include <linux/mm.h>

@@ -1400,6 +1401,73 @@ static void blk_add_rq_to_plug(struct blk_plug *plug, struct request *rq)
 	plug->rq_count++;
 }
 
+/*
+ * Passthrough bios are mapped directly onto requests via
+ * blk_rq_map_user() and never pass through submit_bio(), so they carry
+ * no blkcg association and are invisible to cgroup io.stat and to every
+ * rq_qos policy (iocost, blk-throttle, iolatency).  Charge the ones that
+ * carry data to the submitter's blkcg at dispatch time and run the
+ * regular bio accounting and rq_qos throttle paths with the associated
+ * bio.
+ *
+ * Gated by opcode (READ/WRITE/DRV_IN/DRV_OUT, the latter two mapped
+ * to READ/WRITE for io.stat classification) and to queues that
+ * already have a gendisk: commands issued during device probing (SCSI
+ * INQUIRY and friends) have no gendisk yet and stay exempt.  The
+ * request bios may carry a stale ->bi_blkg from the mempool; the
+ * association helper drops the old reference and re-associates.
+ */
+static void blk_mq_pt_charge(struct request *rq)
+{
+	struct bio *bio = rq->bio;
+	enum req_op op = req_op(rq);
+
+	if (!bio || !rq->q->disk)
+		return;
+
+	switch (op) {
+	case REQ_OP_READ:
+	case REQ_OP_WRITE:
+	case REQ_OP_DRV_IN:
+	case REQ_OP_DRV_OUT:
+		break;
+	default:
+		return;
+	}
+	if (op == REQ_OP_DRV_IN)
+		op = REQ_OP_READ;
+	else if (op == REQ_OP_DRV_OUT)
+		op = REQ_OP_WRITE;
+
+	if (!bio->bi_bdev)
+		bio->bi_bdev = rq->q->disk->part0;
+	bio->bi_opf &= ~REQ_OP_MASK;
+	bio->bi_opf |= op;
+#ifdef CONFIG_BLK_CGROUP
+	/*
+	 * Issued from kthreads the css is root and the charge is a
+	 * no-op through the root exemptions; data-op issuers that
+	 * matter run in the submitter's task context.
+	 */
+	{
+		struct cgroup_subsys_state *css;
+
+		rcu_read_lock();
+		css = task_css(current, io_cgrp_id);
+		bio_associate_blkg_from_css(bio, css);
+		rcu_read_unlock();
+	}
+#endif
+	blk_cgroup_bio_start(bio);
+
+	/*
+	 * The rq_qos throttle path may sleep on the waitqueues like any
+	 * bio submitter; all callers found (ioctl / uring_cmd submit,
+	 * target and error handling kthreads) run in sleepable context.
+	 */
+	rq_qos_throttle(rq->q, bio);
+}
+
 /**
  * blk_execute_rq_nowait - insert a request to I/O scheduler for execution
  * @rq:		request to insert

@@ -1412,6 +1480,7 @@ static void blk_add_rq_to_plug(struct blk_plug *plug, struct request *rq)
  * Note:
  *    This function will invoke @done directly if the queue is dead.
  */
+
 void blk_execute_rq_nowait(struct request *rq, bool at_head)
 {
 	struct blk_mq_hw_ctx *hctx = rq->mq_hctx;

@@ -1419,6 +1488,7 @@ void blk_execute_rq_nowait(struct request *rq, bool at_head)
 	WARN_ON(irqs_disabled());
 	WARN_ON(!blk_rq_is_passthrough(rq));
 
+	blk_mq_pt_charge(rq);
 	blk_account_io_start(rq);
 
 	if (current->plug && !at_head) {

@@ -1484,6 +1554,8 @@ blk_status_t blk_execute_rq(struct request *rq, bool at_head)
 	WARN_ON(irqs_disabled());
 	WARN_ON(!blk_rq_is_passthrough(rq));
 
+	blk_mq_pt_charge(rq);
+
 	rq->end_io_data = &wait;
 	rq->end_io = blk_end_sync_rq;
 

2.43.0

Do you have an actual use case where this matters?

Thanks.

tejun

Hi Tejun,

I don't have a specific passthrough workload that triggered this in production. I ran into it while testing blk-iocost accounting coverage.

The relevant concern here is cgroup IO isolation. The expectation of blkcg IO control is that the IO consumed by a cgroup is reflected in its accounting and subject to blkcg IO policies. Today that is true for IO going through submit_bio(), but not for passthrough requests.

The concrete observation was that the same cgroup issuing equivalent writes through two paths gets different results: normal bio IO is accounted in io.stat and subject to blkcg IO policies, while SG_IO writes consume device bandwidth without accounting or blkcg IO control.

So the motivation is not a particular SG_IO application, but whether passthrough IO should be considered part of the IO usage controlled by blkcg.

I don't know yet whether the right answer is always-on behavior or an opt-in mechanism. The RFC was mainly to discuss whether passthrough paths should be covered by blkcg accounting/enforcement at all.

Thanks,
Tao

Hello,

SG_IO requires CAP_RAW_IO, so any kind of isolation isn't gonna work out that well. This is an imaginary use case. Please stop until it become an actual one.

Thanks.

tejun

Hi,

The cover letter summarized the motivation briefly, so let me add some background on how this came up and what the prototype is trying to address.

How this was found
------------------

This came from stress testing blk-iocost on linux-next with mixed IO workloads and cgroup churn. With the known bio-path accounting gaps (flush, zone append) already carrying posted fixes, the remaining missing cases had one property in common: they build requests directly and enter through blk_execute_rq(), bypassing submit_bio() entirely.

The invariant being violated is simple to state: any IO that consumes device bandwidth but bypasses submit_bio() currently bypasses blkcg IO accounting. Passthrough commands, including SG_IO, bsg, nvme ioctls and uring commands, currently fall into this category: they have no blkcg association, do not show up in io.stat, and do not participate in blkcg-aware rq_qos policies such as iocost and iolatency.

The observable difference
-------------------------

On the same scsi_debug device, with the same cgroup and iocost enabled with vrate pinned at its 1% floor:

bio writer (fio) throttled ~10x
SG_IO writer (sg_dd) full device speed,
io.stat remains zero

The NVMe ioctl path shows the same behavior: passthrough provides a path that bypasses the existing accounting boundary.

With the prototype patch
------------------------

The mapped bio is associated with the submitting blkcg in blk_execute_rq{,_nowait}(), and goes through the normal bio accounting and rq_qos paths.

With the same setup:

- a 128MB SG_IO write (bs=1M, bpt=1) is accounted to the issuing

    cgroup's io.stat, with wbytes/wios matching the transfer;

- the same workload takes 0.025s without throttling and 31.7s with vrate at the 1% floor, with the task sleeping in the iocost throttle path.

Two reproduction notes: sg_dd combines transfers into large commands by default, so bpt=1 is needed for scsi_debug. Also, short bursts can be covered by the initial budget; sustained transfers make the difference visible.

Relation to passthrough iostats
-------------------------------

This RFC builds on Keith's passthrough iostats work. That series made disk-level passthrough accounting possible and provided block-device association for passthrough bios where it is available.

This RFC is about the next layer: whether passthrough IO should also participate in blkcg accounting and enforcement.

The questions from the cover letter still stand. In particular, I am not sure whether attaching blkcg/accounting at this layer is the right abstraction, or whether this should remain an opt-in behavior like passthrough iostats.

Thanks,
Tao