xen/scsiback: unbind the IRQ before draining in-flight requests
Subsystems (4)
PATCH 1/1 xen/scsiback: unbind the IRQ before draining in-flight requests
scsiback_disconnect() drains the in-flight count before stopping the only thing that can raise it. scsiback_cmd_exec() (xen-scsiback.c:426) runs in the backend's threaded IRQ handler, so a request can be admitted just after wait_event() has seen zero, and its response is written later from target core context by scsiback_send_response() (:347) - after the ring has been unmapped, and on the scsiback_remove() path (:1268-1280) after kfree(info).
xen-blkback and xen-netback stop their request processing first:kthread_stop() before reading ring->inflight (xen-blkback/xenbus.c:290-304), and the kthreads and napi before unbinding and unmapping (xen-netback/interface.c:673-700).
Unbind and clear info->irq before the drain: nothing is admitted once the handler is gone, and the ring stays mapped until the drain finishes. The notify at the end of scsiback_send_response() is then a no-op, because notify_remote_via_irq() resolves irq 0 through evtchn_from_irq() to port 0, which VALID_EVTCHN() rejects (events_base.c:539-545, :411-421).
Reproduced in a nested-Xen lab with KASAN. Reaching the window needed two test-only usleep_range() delays, one after wait_event() returns and one at the top of scsiback_do_resp_with_sense() once the teardown has started. With both off neither arm produced anything in 4 rounds each; with both on the unpatched driver crashed in 6 of 6 valid cells, the patched in 0 of 7:
[ 168.546709] BUG: KASAN: vmalloc-out-of-bounds in scsiback_send_response+0x516/0x5d0 [ 168.551012] Write of size 4 at addr ffffc900000230a4 by task kworker/0:1/11 [ 168.554757] [ 168.555908] CPU: 0 UID: 0 PID: 11 Comm: kworker/0:1 Not tainted 7.3.0-kCTLblk2-15482-g21a7f44fd076-dirty #10 PREEMPT(full) [ 168.555914] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.16.3-debian-1.16.3-2 04/01/2014 [ 168.555918] Workqueue: target_completion target_complete_ok_work [ 168.555939] Call Trace: [ 168.555946] <TASK> [ 168.555949] dump_stack_lvl+0x64/0x80 [ 168.555958] print_report+0xd0/0x630 [ 168.555966] ? __pfx__raw_spin_lock_irqsave+0x10/0x10 [ 168.555978] ? scsiback_send_response+0x516/0x5d0 [ 168.555983] kasan_report+0xce/0x100 [ 168.555988] ? scsiback_send_response+0x516/0x5d0 [ 168.555992] scsiback_send_response+0x516/0x5d0 [ 168.555997] ? __pfx_scsiback_send_response+0x10/0x10 [ 168.556001] scsiback_do_resp_with_sense+0x87/0x230 [ 168.556007] scsiback_cmd_done+0xc0/0x290 [ 168.556011] ? update_cfs_rq_load_avg+0x5a/0x560 [ 168.556023] scsiback_queue_status+0xd3/0x100 [ 168.556028] target_complete_ok_work+0x467/0xc30 [ 168.556032] ? __pfx_target_complete_ok_work+0x10/0x10 [ 168.556036] ? put_prev_task_fair+0x44d/0xa30 [ 168.556039] ? finish_task_switch.isra.0+0x1d9/0xa70 [ 168.556047] process_one_work+0x697/0x1060 [ 168.556052] ? __pfx___schedule+0x10/0x10 [ 168.556058] ? __pfx_process_one_work+0x10/0x10 [ 168.556061] ? _raw_spin_lock_irq+0x86/0xe0 [ 168.556065] ? __pfx_target_complete_ok_work+0x10/0x10 [ 168.556069] ? assign_work+0x11d/0x370 [ 168.556074] worker_thread+0x45b/0xd10 [ 168.556078] ? __pfx_worker_thread+0x10/0x10 [ 168.556081] ? __pfx_worker_thread+0x10/0x10 [ 168.556083] kthread+0x2c8/0x3b0 [ 168.556090] ? recalc_sigpending+0x15c/0x1e0 [ 168.556094] ? __pfx_kthread+0x10/0x10 [ 168.556098] ret_from_fork+0x373/0x5b0 [ 168.556104] ? __pfx_ret_from_fork+0x10/0x10 [ 168.556107] ? __switch_to+0x3c4/0xb40 [ 168.556115] ? __pfx_kthread+0x10/0x10 [ 168.556120] ret_from_fork_asm+0x1a/0x30 [ 168.556127] </TASK> [ 168.556129] [ 168.639704] The buggy address belongs to a vmalloc virtual mapping [ 168.642324] Memory state around the buggy address: [ 168.644439] ffffc90000022f80: f8 f8 f8 f8 f8 f8 f8 f8 f8 f8 f8 f8 f8 f8 f8 f8 [ 168.647698] ffffc90000023000: f8 f8 f8 f8 f8 f8 f8 f8 f8 f8 f8 f8 f8 f8 f8 f8 [ 168.650951] >ffffc90000023080: f8 f8 f8 f8 f8 f8 f8 f8 f8 f8 f8 f8 f8 f8 f8 f8 [ 168.669199] ^ [ 168.671056] ffffc90000023100: f8 f8 f8 f8 f8 f8 f8 f8 f8 f8 f8 f8 f8 f8 f8 f8 [ 168.674050] ffffc90000023180: f8 f8 f8 f8 f8 f8 f8 f8 f8 f8 f8 f8 f8 f8 f8 f8 [ 168.676974] ==================================================================
[ 168.682117] BUG: unable to handle page fault for address: ffffc900000230a4 [ 168.684784] #PF: supervisor write access in kernel mode [ 168.686797] #PF: error_code(0x0002) - not-present page [ 168.688797] PGD 100000067 P4D 100000067 PUD 10019c067 PMD 1001a1067 PTE 0 [ 168.691319] Oops: Oops: 0002 [#1] SMP KASAN NOPTI [ 168.693246] CPU: 0 UID: 0 PID: 11 Comm: kworker/0:1 Tainted: G B 7.3.0-kCTLblk2-15482-g21a7f44fd076-dirty #10 PREEMPT(full) [ 168.699142] Tainted: [B]=BAD_PAGE [ 168.700896] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.16.3-debian-1.16.3-2 04/01/2014 [ 168.705056] Workqueue: target_completion target_complete_ok_work [ 168.707701] RIP: 0010:scsiback_send_response+0x1b6/0x5d0 [ 168.710147] Code: 48 89 f8 83 e0 07 83 c0 03 40 38 f0 7c 09 40 84 f6 0f 85 60 03 00 00 48 89 d8 48 c1 e0 06 48 29 d8 48 c1 e0 02 48 8d 7c 01 40 <89> 94 01 a4 00 00 00 48 b8 00 00 00 00 00 fc ff df 48 89 fa 48 c1 [ 168.718883] RSP: 0018:ffffc900000bfaa0 EFLAGS: 00010046 [ 168.721447] RAX: 0000000000000000 RBX: 0000000000000000 RCX: ffffc90000023000 [ 168.724785] RDX: 0000000000000000 RSI: 0000000000000008 RDI: ffffc90000023040 [ 168.727913] RBP: ffff88810a4a7238 R08: 0000000000000246 R09: fffffbfff0de2638 [ 168.730938] R10: ffffffff86f131c7 R11: ffff888100077320 R12: 1ffff92000017f59 [ 168.733902] R13: 0000000000000000 R14: 0000000000000000 R15: ffff88810a6de200 [ 168.736874] FS: 0000000000000000(0000) GS:ffff8881b034d000(0000) knlGS:0000000000000000 [ 168.740171] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 168.742490] CR2: ffffc900000230a4 CR3: 0000000103271005 CR4: 0000000000770ef0 [ 168.745350] PKRU: 55555554 [ 168.746577] Call Trace: [ 168.747772] <TASK> [ 168.748872] ? __pfx_scsiback_send_response+0x10/0x10 [ 168.751022] scsiback_do_resp_with_sense+0x87/0x230 [ 168.753109] scsiback_cmd_done+0xc0/0x290 [ 168.754847] ? update_cfs_rq_load_avg+0x5a/0x560 [ 168.756782] scsiback_queue_status+0xd3/0x100 [ 168.758630] target_complete_ok_work+0x467/0xc30 [ 168.760556] ? __pfx_target_complete_ok_work+0x10/0x10 [ 168.762644] ? put_prev_task_fair+0x44d/0xa30 [ 168.764512] ? finish_task_switch.isra.0+0x1d9/0xa70 [ 168.766584] process_one_work+0x697/0x1060 [ 168.768282] ? __pfx___schedule+0x10/0x10 [ 168.769973] ? __pfx_process_one_work+0x10/0x10 [ 168.771880] ? _raw_spin_lock_irq+0x86/0xe0 [ 168.773599] ? __pfx_target_complete_ok_work+0x10/0x10 [ 168.775721] ? assign_work+0x11d/0x370 [ 168.777324] worker_thread+0x45b/0xd10 [ 168.778899] ? __pfx_worker_thread+0x10/0x10 [ 168.780648] ? __pfx_worker_thread+0x10/0x10 [ 168.782415] kthread+0x2c8/0x3b0 [ 168.783877] ? recalc_sigpending+0x15c/0x1e0 [ 168.785697] ? __pfx_kthread+0x10/0x10 [ 168.787218] ret_from_fork+0x373/0x5b0 [ 168.788752] ? __pfx_ret_from_fork+0x10/0x10 [ 168.790500] ? __switch_to+0x3c4/0xb40 [ 168.792029] ? __pfx_kthread+0x10/0x10 [ 168.793555] ret_from_fork_asm+0x1a/0x30 [ 168.795180] </TASK> [ 168.796205] Modules linked in: [ 168.797596] CR2: ffffc900000230a4 [ 168.799084] ---[ end trace 0000000000000000 ]---
---
Reported to the Xen security team first; xen-scsiback is not security supported, so posting here as suggested.
drivers/xen/xen-scsiback.c | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-)
drivers/xen/xen-scsiback.c +4 -2
@@ -603,11 +603,13 @@ static int scsiback_gnttab_data_map(struct vscsiif_request *ring_req,static void scsiback_disconnect (struct vscsibk_info *info) { +/* Stop admitting requests before draining the in-flight ones. */ +unbind_from_irqhandler (info->irq, info); + info->irq =0 ; +wait_event (info->waiting_to_free,atomic_read (&info->nr_unreplied_reqs) ==0 ); -unbind_from_irqhandler (info->irq, info); - info->irq =0 ;xenbus_unmap_ring_vfree (info->dev, info->ring.sring); }
base-commit: 7475ea8b58616b33c04afb976cf0fbaad9dd63fb
Signature · 1 line
2.43.0
[email protected]
AI reviewThank you for your contribution! Sashiko AI review found 1 potential issue to consider: - [High] xen/scsiback: premature IRQ unbind in scsiback_disconnect() races with responses --
Thank you for your contribution! Sashiko AI review found 1 potential issue to consider:
- [High] xen/scsiback: premature IRQ unbind in scsiback_disconnect() races with responses
--
commit 3923ae443e53c8e678c6827dafcd0fcc2ee96487
Author: Yehyeong Lee <[email protected]>
xen/scsiback: unbind the IRQ before draining in-flight requests
This commit moves unbind_from_irqhandler() and clearing info->irq ahead of waiting for nr_unreplied_reqs to reach zero in scsiback_disconnect() so the threaded IRQ handler cannot admit new requests after the drain finishes.
Quoted 9 lines
drivers/xen/xen-scsiback.c +1
-0@@ -603,11 +603,13 @@ static int scsiback_gnttab_data_map(struct vscsiif_request *ring_req,static void scsiback_disconnect (struct vscsibk_info *info) { +/* Stop admitting requests before draining the in-flight ones. */
drivers/xen/xen-scsiback.c static int scsiback_gnttab_data_map(struct vscsiif_request *ring_req, · line 614
Quoted 10 lines
drivers/xen/xen-scsiback.c +3
-2+unbind_from_irqhandler (info->irq, info); + info->irq =0 ; +wait_event (info->waiting_to_free,atomic_read (&info->nr_unreplied_reqs) ==0 ); -unbind_from_irqhandler (info->irq, info); - info->irq =0 ;xenbus_unmap_ring_vfree (info->dev, info->ring.sring); }
[Severity: High]
Does unbinding and zeroing info->irq before wait_event() introduce a regression where scsiback_send_response() races with delayed_free_irq() or notifies IRQ 0 while in-flight requests are still draining?
In-flight commands complete on target_completion workqueues via scsiback_cmd_done(), which calls scsiback_do_resp_with_sense() -> scsiback_send_response() before scsiback_put(info) decrements info->nr_unreplied_reqs. In scsiback_send_response(), info->irq is read outside info->ring_lock and passed to notify_remote_via_irq() in preemptible context without RCU protection:
CPU0 CPU1 (kworker)scsiback_disconnect() scsiback_cmd_done()
...
scsiback_send_response()
irq = info->irq;
unbind_from_irqhandler(...)
...
xen_free_irq(irq)
queue_rcu_work(..., &rwork)
info->irq = 0;
wait_event(..., nr_unreplied_reqs == 0)
[RCU grace period elapses]
delayed_free_irq()
set_info_for_irq(irq, NULL)
kfree(irq_info)
xen_irq_free_desc(irq)
notify_remote_via_irq(irq)
evtchn_from_irq(irq)
irq_info->evtchn <-- use-after-free
scsiback_put(info)Once info->irq = 0 becomes visible during wait_event(), every completing request calls notify_remote_via_irq(0). IRQ 0 is a valid Xen IRQ number:xen_allocate_irq_dynamic() allocates descriptors via irq_alloc_desc_from(0, -1), so on platforms where nr_legacy_irqs() is 0 (such as ARM64 or x86 Xen PV domains with null_legacy_pic, where VIRQ_TIMER binds to IRQ 0 at boot), info_for_irq(0) is non-NULL with a valid event channel and notify_remote_via_irq(0) sends spurious notifications to that channel.
Unbinding before wait_event() also closes the event channel via xen_evtchn_close() before in-flight responses are pushed to the ring, dropping completion notifications during the drain. Calling disable_irq(info->irq) before wait_event() and keeping unbind_from_irqhandler(info->irq, info) after wait_event() would stop scsiback_irq_fn() synchronously while keeping info->irq valid until all in-flight requests have completed.
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=1
Quoted 131 lines · Yehyeong Lee
scsiback_disconnect()drains the in-flight count before stopping the only thing that can raise it.scsiback_cmd_exec()(xen-scsiback.c:426) runs in the backend's threaded IRQ handler, so a request can be admitted just afterwait_event()has seen zero, and its response is written later from target core context byscsiback_send_response()(:347) - after the ring has been unmapped, and on thescsiback_remove()path (:1268-1280) after kfree(info).xen-blkback and xen-netback stop their request processing first:
kthread_stop()before readingring->inflight(xen-blkback/xenbus.c:290-304), and the kthreads and napi before unbinding and unmapping (xen-netback/interface.c:673-700).Unbind and clear
info->irqbefore the drain: nothing is admitted once the handler is gone, and the ring stays mapped until the drain finishes. The notify at the end ofscsiback_send_response()is then a no-op, becausenotify_remote_via_irq()resolves irq 0 throughevtchn_from_irq()to port 0, whichVALID_EVTCHN()rejects (events_base.c:539-545, :411-421).Reproduced in a nested-Xen lab with KASAN. Reaching the window needed two test-only
usleep_range()delays, one afterwait_event()returns and one at the top ofscsiback_do_resp_with_sense()once the teardown has started. With both off neither arm produced anything in 4 rounds each; with both on the unpatched driver crashed in 6 of 6 valid cells, the patched in 0 of 7:[ 168.546709] BUG: KASAN: vmalloc-out-of-bounds in scsiback_send_response+0x516/0x5d0 [ 168.551012] Write of size 4 at addr ffffc900000230a4 by task kworker/0:1/11 [ 168.554757] [ 168.555908] CPU: 0 UID: 0 PID: 11 Comm: kworker/0:1 Not tainted 7.3.0-kCTLblk2-15482-g21a7f44fd076-dirty #10 PREEMPT(full) [ 168.555914] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.16.3-debian-1.16.3-2 04/01/2014 [ 168.555918] Workqueue: target_completion target_complete_ok_work [ 168.555939] Call Trace: [ 168.555946] <TASK> [ 168.555949] dump_stack_lvl+0x64/0x80 [ 168.555958] print_report+0xd0/0x630 [ 168.555966] ? __pfx__raw_spin_lock_irqsave+0x10/0x10 [ 168.555978] ? scsiback_send_response+0x516/0x5d0 [ 168.555983] kasan_report+0xce/0x100 [ 168.555988] ? scsiback_send_response+0x516/0x5d0 [ 168.555992] scsiback_send_response+0x516/0x5d0 [ 168.555997] ? __pfx_scsiback_send_response+0x10/0x10 [ 168.556001] scsiback_do_resp_with_sense+0x87/0x230 [ 168.556007] scsiback_cmd_done+0xc0/0x290 [ 168.556011] ? update_cfs_rq_load_avg+0x5a/0x560 [ 168.556023] scsiback_queue_status+0xd3/0x100 [ 168.556028] target_complete_ok_work+0x467/0xc30 [ 168.556032] ? __pfx_target_complete_ok_work+0x10/0x10 [ 168.556036] ? put_prev_task_fair+0x44d/0xa30 [ 168.556039] ? finish_task_switch.isra.0+0x1d9/0xa70 [ 168.556047] process_one_work+0x697/0x1060 [ 168.556052] ? __pfx___schedule+0x10/0x10 [ 168.556058] ? __pfx_process_one_work+0x10/0x10 [ 168.556061] ? _raw_spin_lock_irq+0x86/0xe0 [ 168.556065] ? __pfx_target_complete_ok_work+0x10/0x10 [ 168.556069] ? assign_work+0x11d/0x370 [ 168.556074] worker_thread+0x45b/0xd10 [ 168.556078] ? __pfx_worker_thread+0x10/0x10 [ 168.556081] ? __pfx_worker_thread+0x10/0x10 [ 168.556083] kthread+0x2c8/0x3b0 [ 168.556090] ? recalc_sigpending+0x15c/0x1e0 [ 168.556094] ? __pfx_kthread+0x10/0x10 [ 168.556098] ret_from_fork+0x373/0x5b0 [ 168.556104] ? __pfx_ret_from_fork+0x10/0x10 [ 168.556107] ? __switch_to+0x3c4/0xb40 [ 168.556115] ? __pfx_kthread+0x10/0x10 [ 168.556120] ret_from_fork_asm+0x1a/0x30 [ 168.556127] </TASK> [ 168.556129] [ 168.639704] The buggy address belongs to a vmalloc virtual mapping [ 168.642324] Memory state around the buggy address: [ 168.644439] ffffc90000022f80: f8 f8 f8 f8 f8 f8 f8 f8 f8 f8 f8 f8 f8 f8 f8 f8 [ 168.647698] ffffc90000023000: f8 f8 f8 f8 f8 f8 f8 f8 f8 f8 f8 f8 f8 f8 f8 f8 [ 168.650951] >ffffc90000023080: f8 f8 f8 f8 f8 f8 f8 f8 f8 f8 f8 f8 f8 f8 f8 f8 [ 168.669199] ^ [ 168.671056] ffffc90000023100: f8 f8 f8 f8 f8 f8 f8 f8 f8 f8 f8 f8 f8 f8 f8 f8 [ 168.674050] ffffc90000023180: f8 f8 f8 f8 f8 f8 f8 f8 f8 f8 f8 f8 f8 f8 f8 f8 [ 168.676974] ==================================================================[ 168.682117] BUG: unable to handle page fault for address: ffffc900000230a4 [ 168.684784] #PF: supervisor write access in kernel mode [ 168.686797] #PF: error_code(0x0002) - not-present page [ 168.688797] PGD 100000067 P4D 100000067 PUD 10019c067 PMD 1001a1067 PTE 0 [ 168.691319] Oops: Oops: 0002 [#1] SMP KASAN NOPTI [ 168.693246] CPU: 0 UID: 0 PID: 11 Comm: kworker/0:1 Tainted: G B 7.3.0-kCTLblk2-15482-g21a7f44fd076-dirty #10 PREEMPT(full) [ 168.699142] Tainted: [B]=BAD_PAGE [ 168.700896] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.16.3-debian-1.16.3-2 04/01/2014 [ 168.705056] Workqueue: target_completion target_complete_ok_work [ 168.707701] RIP: 0010:scsiback_send_response+0x1b6/0x5d0 [ 168.710147] Code: 48 89 f8 83 e0 07 83 c0 03 40 38 f0 7c 09 40 84 f6 0f 85 60 03 00 00 48 89 d8 48 c1 e0 06 48 29 d8 48 c1 e0 02 48 8d 7c 01 40 <89> 94 01 a4 00 00 00 48 b8 00 00 00 00 00 fc ff df 48 89 fa 48 c1 [ 168.718883] RSP: 0018:ffffc900000bfaa0 EFLAGS: 00010046 [ 168.721447] RAX: 0000000000000000 RBX: 0000000000000000 RCX: ffffc90000023000 [ 168.724785] RDX: 0000000000000000 RSI: 0000000000000008 RDI: ffffc90000023040 [ 168.727913] RBP: ffff88810a4a7238 R08: 0000000000000246 R09: fffffbfff0de2638 [ 168.730938] R10: ffffffff86f131c7 R11: ffff888100077320 R12: 1ffff92000017f59 [ 168.733902] R13: 0000000000000000 R14: 0000000000000000 R15: ffff88810a6de200 [ 168.736874] FS: 0000000000000000(0000) GS:ffff8881b034d000(0000) knlGS:0000000000000000 [ 168.740171] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 168.742490] CR2: ffffc900000230a4 CR3: 0000000103271005 CR4: 0000000000770ef0 [ 168.745350] PKRU: 55555554 [ 168.746577] Call Trace: [ 168.747772] <TASK> [ 168.748872] ? __pfx_scsiback_send_response+0x10/0x10 [ 168.751022] scsiback_do_resp_with_sense+0x87/0x230 [ 168.753109] scsiback_cmd_done+0xc0/0x290 [ 168.754847] ? update_cfs_rq_load_avg+0x5a/0x560 [ 168.756782] scsiback_queue_status+0xd3/0x100 [ 168.758630] target_complete_ok_work+0x467/0xc30 [ 168.760556] ? __pfx_target_complete_ok_work+0x10/0x10 [ 168.762644] ? put_prev_task_fair+0x44d/0xa30 [ 168.764512] ? finish_task_switch.isra.0+0x1d9/0xa70 [ 168.766584] process_one_work+0x697/0x1060 [ 168.768282] ? __pfx___schedule+0x10/0x10 [ 168.769973] ? __pfx_process_one_work+0x10/0x10 [ 168.771880] ? _raw_spin_lock_irq+0x86/0xe0 [ 168.773599] ? __pfx_target_complete_ok_work+0x10/0x10 [ 168.775721] ? assign_work+0x11d/0x370 [ 168.777324] worker_thread+0x45b/0xd10 [ 168.778899] ? __pfx_worker_thread+0x10/0x10 [ 168.780648] ? __pfx_worker_thread+0x10/0x10 [ 168.782415] kthread+0x2c8/0x3b0 [ 168.783877] ? recalc_sigpending+0x15c/0x1e0 [ 168.785697] ? __pfx_kthread+0x10/0x10 [ 168.787218] ret_from_fork+0x373/0x5b0 [ 168.788752] ? __pfx_ret_from_fork+0x10/0x10 [ 168.790500] ? __switch_to+0x3c4/0xb40 [ 168.792029] ? __pfx_kthread+0x10/0x10 [ 168.793555] ret_from_fork_asm+0x1a/0x30 [ 168.795180] </TASK> [ 168.796205] Modules linked in: [ 168.797596] CR2: ffffc900000230a4 [ 168.799084] ---[ end trace 0000000000000000 ]---
Quoted 3 lines
Fixes d9d660f6e562 ("xen-scsiback: Add Xen PV SCSI backend driver")Signed-off-by Yehyeong Lee <[email protected]>
Juergen