io_uring/zcrx: don't clear master_ctx from the import path
Subsystems (4)
PATCH 1/1 io_uring/zcrx: don't clear master_ctx from the import path
import_zcrx() attaches an existing ifq to another ring. It never calls zcrx_set_ring_ctx() and so never takes the ->master_ctx reference, but its error path still passes @ctx to zcrx_unregister(), which clears ->master_ctx and drops its percpu_ref whenever ifq->master_ctx == ctx.
That condition is reachable. A ring that registers an ifq with a non-zero event type_mask gets ->master_ctx pointed at itself, and nothing stops it from exporting that ifq with ZCRX_CTRL_EXPORT and importing the resulting fd back into the same ring. Failing the import after the refcount bumps -- an argument page mapped PROT_READ makes the copy_to_user() in import_zcrx() return -EFAULT -- then clears the ->master_ctx owned by the original registration, which is still live.
Refcounts stay balanced and nothing is freed early, so there is no splat. The ring silently stops receiving ZCRX_EVENT_ALLOC_FAIL andZCRX_EVENT_COPY: zcrx_send_notif() returns early on a NULL ->master_ctx, and ->master_ctx is only ever set on a freshly allocated ifq, so it cannot be restored without tearing the ring down.
Pass NULL instead, matching zcrx_box_release() and the zcrx_export() error path. io_register_zcrx() only gets away with passing @ctx because zcrx_set_ring_ctx() runs after its last goto err.
---
Found by code inspection, then confirmed on v7.2-rc5 in a QEMU guest by adding a pr_info() to zcrx_set_ring_ctx() and zcrx_unregister_user(). Writing A for the ring's io_ring_ctx:
Before, the failed self-import clears the binding:
set_ring_ctx: ctx=A -> master_ctx = A
unregister_user: ctx=A master=A -> CLEARING (failed self-import)
unregister_user: ctx=A master=0 (ring teardown, nothing left)
After, it survives and is released where it should be:
set_ring_ctx: ctx=A -> master_ctx = A
unregister_user: ctx=0 master=A (failed self-import)
unregister_user: ctx=A master=A -> CLEARING (ring teardown)
io_uring_register() returns -EFAULT either way.
The reproducer below needs no hardware: ZCRX_REG_NODEV leaves ifq->dev NULL, so no net device and no DMA mapping are involved. Plain syscalls, no liburing. Run as root (CAP_NET_ADMIN).
Reproducer:
#define _GNU_SOURCE
#include <errno.h>
#include <stdio.h>
#include <string.h>
#include <unistd.h>
#include <sys/mman.h>
#include <sys/syscall.h>
#include <linux/io_uring.h>
#include <linux/io_uring/zcrx.h>
#define AREA_PAGES 64
#define RING_BYTES (1024 * 1024)
int main(void)
{
struct io_uring_zcrx_ifq_reg reg, *ireg;
struct io_uring_zcrx_area_reg area_reg;
struct io_uring_region_desc region_reg;
struct zcrx_event_desc ev;
struct io_uring_params p;
struct zcrx_ctrl ctrl;
void *area, *ring, *ro;
long ps = sysconf(_SC_PAGESIZE);
int ringfd, boxfd, ret;
memset(&p, 0, sizeof(p));
p.flags = IORING_SETUP_DEFER_TASKRUN | IORING_SETUP_SINGLE_ISSUER |
IORING_SETUP_CQE32;
ringfd = syscall(__NR_io_uring_setup, 8, &p);
if (ringfd < 0)
return perror("io_uring_setup"), 1;
area = mmap(NULL, AREA_PAGES * ps, PROT_READ | PROT_WRITE,
MAP_ANONYMOUS | MAP_PRIVATE, -1, 0);
ring = mmap(NULL, RING_BYTES, PROT_READ | PROT_WRITE,
MAP_ANONYMOUS | MAP_PRIVATE, -1, 0);
/* non-zero type_mask is what makes zcrx_set_ring_ctx() run */
memset(&ev, 0, sizeof(ev));
ev.user_data = 0x5a5a5a5a;
ev.type_mask = 1U << ZCRX_EVENT_COPY;
memset(®ion_reg, 0, sizeof(region_reg));
region_reg.size = RING_BYTES;
region_reg.user_addr = (unsigned long)ring;
region_reg.flags = IORING_MEM_REGION_TYPE_USER;
memset(&area_reg, 0, sizeof(area_reg));
area_reg.addr = (unsigned long)area;
area_reg.len = AREA_PAGES * ps;
memset(®, 0, sizeof(reg));
reg.rq_entries = 64;
reg.flags = ZCRX_REG_NODEV;
reg.area_ptr = (unsigned long)&area_reg;
reg.region_ptr = (unsigned long)®ion_reg;
reg.event_desc = (unsigned long)&ev;
ret = syscall(__NR_io_uring_register, ringfd,
IORING_REGISTER_ZCRX_IFQ, ®, 1);
if (ret)
return perror("register ifq"), 1;
memset(&ctrl, 0, sizeof(ctrl));
ctrl.zcrx_id = reg.zcrx_id;
ctrl.op = ZCRX_CTRL_EXPORT;
ret = syscall(__NR_io_uring_register, ringfd,
IORING_REGISTER_ZCRX_CTRL, &ctrl, 0);
if (ret)
return perror("zcrx export"), 1;
boxfd = ctrl.zc_export.zcrx_fd;
/* import back into the same ring, with an unwritable argument page */
ro = mmap(NULL, ps, PROT_READ | PROT_WRITE,
MAP_ANONYMOUS | MAP_PRIVATE, -1, 0);
ireg = ro;
memset(ireg, 0, sizeof(*ireg));
ireg->flags = ZCRX_REG_IMPORT;
ireg->if_idx = boxfd;
mprotect(ro, ps, PROT_READ);
ret = syscall(__NR_io_uring_register, ringfd,
IORING_REGISTER_ZCRX_IFQ, ro, 1);
printf("self-import: ret=%d errno=%d (%s)\n", ret, errno,
strerror(errno)); /* -1 / EFAULT */
return 0;
}io_uring/zcrx.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)
io_uring/zcrx.c +2 -1
@@ -808,7 +808,8 @@ static int import_zcrx(struct io_ring_ctx *ctx, scoped_guard(mutex, &ctx->mmap_lock)xa_erase (&ctx->zcrx_ctxs, id); err: -zcrx_unregister (ifq, ctx); +/* the import path never took the ->master_ctx ref, don't drop it */ +zcrx_unregister (ifq,NULL );return ret; }
base-commit: 11028ab62899e4191e074ee364c712b77823a9c4
Signature · 1 line
2.45.4
Quoted 13 lines · Woraphat Khiaodaeng
import_zcrx()attaches an existing ifq to another ring. It never callszcrx_set_ring_ctx()and so never takes the ->master_ctx reference, but its error path still passes @ctx tozcrx_unregister(), which clears ->master_ctx and drops itspercpu_refwheneverifq->master_ctx== ctx.That condition is reachable. A ring that registers an ifq with a non-zero event
type_maskgets ->master_ctx pointed at itself, and nothing stops it from exporting that ifq withZCRX_CTRL_EXPORTand importing the resulting fd back into the same ring. Failing the import after the refcount bumps -- an argument page mappedPROT_READmakes thecopy_to_user()inimport_zcrx()return-EFAULT-- then clears the ->master_ctx owned by the original registration, which is still live.
Quoted 5 lines
Refcounts stay balanced and nothing is freed early, so there is no splat. The ring silently stops receiving
ZCRX_EVENT_ALLOC_FAILandZCRX_EVENT_COPY:zcrx_send_notif()returns early on a NULL ->master_ctx, and ->master_ctx is only ever set on a freshly allocated ifq, so it cannot be restored without tearing the ring down.
Yep, it effectively disables event CQEs.
...> diff --git a/io_uring/zcrx.c b/io_uring/zcrx.c
io_uring/zcrx.c static int import_zcrx(struct io_ring_ctx *ctx, · line 811
Quoted 9 lines
index
76b9b0d54af9e..fea2b27272ec2100644io_uring/zcrx.c +1
-1@@ -808,7 +808,8 @@ static int import_zcrx(struct io_ring_ctx *ctx, scoped_guard(mutex, &ctx->mmap_lock)xa_erase (&ctx->zcrx_ctxs, id); err: -zcrx_unregister (ifq, ctx); +/* the import path never took the ->master_ctx ref, don't drop it */
nit: I'd say "... it never set ->master_ctx ..." as it's about having a ctx and not references, but it's not worth of respinning.
Quoted 3 lines
io_uring/zcrx.c +1
-0+zcrx_unregister (ifq,NULL );return ret; }
Signature · 1 line
Pavel Begunkov
Quoted 13 lines · Woraphat Khiaodaeng
import_zcrx()attaches an existing ifq to another ring. It never callszcrx_set_ring_ctx()and so never takes the ->master_ctx reference, but its error path still passes @ctx tozcrx_unregister(), which clears ->master_ctx and drops itspercpu_refwheneverifq->master_ctx== ctx.That condition is reachable. A ring that registers an ifq with a non-zero event
type_maskgets ->master_ctx pointed at itself, and nothing stops it from exporting that ifq withZCRX_CTRL_EXPORTand importing the resulting fd back into the same ring. Failing the import after the refcount bumps -- an argument page mappedPROT_READmakes thecopy_to_user()inimport_zcrx()return-EFAULT-- then clears the ->master_ctx owned by the original registration, which is still live.
Quoted 1 line
[...]
Applied, thanks!
[1/1] io_uring/zcrx: don't clear master_ctx from the import path
commit: fa1ac3b9eb62918f039276d4d11cc02f682bf93c
Best regards,
Signature · 1 line
Jens Axboe