nvmet-tcp: fix use-after-free when TLS handshake completes before timeout is armed
Subsystems (2)
PATCH 1/1 nvmet-tcp: fix use-after-free when TLS handshake completes before timeout is armed
nvmet_tcp_tls_handshake() takes a queue reference for the handshake completion callback, submits the handshake with tls_server_hello_psk(), and only afterwards arms tls_handshake_tmo_work.
Once tls_server_hello_psk() has queued the request and notified the handshake agent, nvmet_tcp_tls_handshake_done() can run before the submitter reaches queue_delayed_work(). On failure the callback calls nvmet_tcp_schedule_release_queue() and then drops its own reference, which drops the last reference and queues release_work. nvmet_wq is not ordered, so nvmet_tcp_release_queue_work() can run concurrently:
its cancel_delayed_work_sync() finds no pending timeout work, and it frees the queue. The submitter then arms the delayed work inside the freed queue, and the timer fires on freed memory tls_handshake_timeout seconds later.
Reproduced on mainline during the v7.4 merge window on an NVMe/TCP port with addr_tsas=tls1.3, with clients whose handshakes tlshd rejects. It reproduces both with stock tlshd and with a minimal test agent that fails every request; a non-debug kernel panics in __run_timers. With KASAN, the unmodified kernel hit:
BUG: KASAN: slab-use-after-free in __run_timers+0x72a/0x8e0 Write of size 8 at addr ffff88800317a1d0 by task swapper/3/0 __run_timers run_timer_softirq Allocated by task 12: nvmet_tcp_accept_work Freed by task 132: kfree <- process_one_work (nvmet_tcp_release_queue_work) Last potentially related work creation:
queue_work_on <- handshake_complete <- handshake_nl_done_doit
The queue was freed 4ms after allocation and the timer fired 10s later. With an msleep() inserted between tls_server_hello_psk() and queue_delayed_work() the report is deterministic and points at the write in queue_delayed_work_on() from nvmet_tcp_accept_work().
Hold an extra queue reference across the submission and the arming of the timeout work. If the completion callback has already released the queue, the final put queues release_work, which cancels the timeout work that was just armed.
XXX-ADD-YOUR-OWN-Signed-off-by-LINE-HERE-AND-DELETE-THIS-LINE-XXX
---drivers/nvme/target/tcp.c | 9 +++++++++
1 file changed, 9 insertions(+)
drivers/nvme/target/tcp.c +0 -0
@@ -1910,6 +1910,14 @@ static int nvmet_tcp_tls_handshake(struct nvmet_tcp_queue *queue)
return -EINVAL;
}
+ /* Reference owned by the handshake completion callback */
+ kref_get(&queue->kref);
+ /*
+ * The completion callback may run, and drop its reference and the
+ * queue reference, before tls_server_hello_psk() returns. Hold an
+ * extra reference until the timeout work is armed so the queue
+ * cannot be released underneath us.
+ */kref_get(&queue->kref);pr_debug("queue %d: TLS ServerHello\n", queue->idx);memset(&args, 0, sizeof(args));
@@ -1927,6 +1935,7 @@ static int nvmet_tcp_tls_handshake(struct nvmet_tcp_queue *queue) queue_delayed_work(nvmet_wq, &queue->tls_handshake_tmo_work, tls_handshake_timeout * HZ);
}
+ kref_put(&queue->kref, nvmet_tcp_release_queue);
return ret;
}
#else
Keith Busch
maintainerQuoted 35 lines · Duc Anh Luu
nvmet_tcp_tls_handshake()takes a queue reference for the handshake completion callback, submits the handshake withtls_server_hello_psk(), and only afterwards armstls_handshake_tmo_work.Once
tls_server_hello_psk()has queued the request and notified the handshake agent,nvmet_tcp_tls_handshake_done()can run before the submitter reachesqueue_delayed_work(). On failure the callback callsnvmet_tcp_schedule_release_queue()and then drops its own reference, which drops the last reference and queuesrelease_work.nvmet_wqis not ordered, sonvmet_tcp_release_queue_work()can run concurrently:
itscancel_delayed_work_sync()finds no pending timeout work, and it frees the queue. The submitter then arms the delayed work inside the freed queue, and the timer fires on freed memorytls_handshake_timeoutseconds later.Reproduced on mainline during the v7.4 merge window on an NVMe/TCP port with addr_tsas=tls1.3, with clients whose handshakes tlshd rejects. It reproduces both with stock tlshd and with a minimal test agent that fails every request; a non-debug kernel panics in
__run_timers. With KASAN, the unmodified kernel hit:BUG: KASAN: slab-use-after-free in __run_timers+0x72a/0x8e0 Write of size 8 at addr ffff88800317a1d0 by task swapper/3/0 __run_timers run_timer_softirq Allocated by task 12: nvmet_tcp_accept_work Freed by task 132: kfree <- process_one_work (nvmet_tcp_release_queue_work) Last potentially related work creation:
queue_work_on<-handshake_complete<-handshake_nl_done_doitThe queue was freed 4ms after allocation and the timer fired 10s later. With an
msleep()inserted betweentls_server_hello_psk()andqueue_delayed_work()the report is deterministic and points at the write inqueue_delayed_work_on()fromnvmet_tcp_accept_work().
Quoted 4 lines
Hold an extra queue reference across the submission and the arming of the timeout work. If the completion callback has already released the queue, the final put queues
release_work, which cancels the timeout work that was just armed.
This is a rather long commit message for a simple problem. Just the first two paragraphs and the KASAN splat are sufficient for this changelog.
Quoted 3 lines
Fixes 675b453e0241 ("nvmet-tcp: enable TLS handshake upcall")XXX-ADD-YOUR-OWN-Signed-off-by-LINE-HERE-AND-DELETE-THIS-LINE-XXX
Umm...
Quoted 23 lines
(quoted diff) +9
-0@@ -1910,6 +1910,14 @@ static int nvmet_tcp_tls_handshake(struct nvmet_tcp_queue *queue) return -EINVAL; } + /* Reference owned by the handshake completion callback */ + kref_get(&queue->kref); + /* + * The completion callback may run, and drop its reference and the + * queue reference, before tls_server_hello_psk() returns. Hold an + * extra reference until the timeout work is armed so the queue + * cannot be released underneath us. + */ kref_get(&queue->kref); pr_debug("queue %d: TLS ServerHello\n", queue->idx); memset(&args, 0, sizeof(args));@@ -1927,6 +1935,7 @@ static int nvmet_tcp_tls_handshake(struct nvmet_tcp_queue *queue) queue_delayed_work(nvmet_wq, &queue->tls_handshake_tmo_work, tls_handshake_timeout * HZ); } + kref_put(&queue->kref, nvmet_tcp_release_queue); return ret; }
Quoted 1 line
#else
Your patch is white space damaged.