Skip to contentSkip to conversation

Fix-use-after-free-when-a-port-is-removed

PATCH Applied Israel Rukshin 2 participants 5 messages Oct 6, 2021 – Oct 12, 2021

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

cover Fix-use-after-free-when-a-port-is-removed

Hi all,
This series fixes an issue when a port is removed through configfs. This issue is a use-after-free bug for any command that dereferences req->port (like in nvmet_parse_io_cmd) after releasing the port.

The first patch fix the issue after the controller finished the connection flow and the second and the third patches fix it during connection creation time at nvmet-rdma and nvmet-tcp drivers.

Israel Rukshin (3):
nvmet: Fix use-after-free when a port is removed
nvmet_rdma: Fix use-after-free when a port is removed
nvmet_tcp: Fix use-after-free when a port is removed

 drivers/nvme/target/configfs.c |  2 ++
 drivers/nvme/target/rdma.c     | 24 ++++++++++++++++++++++++
 drivers/nvme/target/tcp.c      | 16 ++++++++++++++++
 3 files changed, 42 insertions(+)
Signature · 7 lines
2.16.3


_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme
#1 lore raw

PATCH 1/3 nvmet: Fix use-after-free when a port is removed

When a port is removed through configfs, any connected controllers are starting teardown flow asynchronously and can still send commands. This causes a use-after-free bug for any command that dereferences req->port (like in nvmet_parse_io_cmd).

To fix this, wait for all the teardown scheduled works to complete (like release_work at rdma/tcp drivers). This ensures there are no active controllers when the port is eventually removed.

---
 drivers/nvme/target/configfs.c | 2 ++
 1 file changed, 2 insertions(+)
drivers/nvme/target/configfs.c +2 -0
index d784f3c200b4..d6cfeb3b3eca 100644
@@ -1553,6 +1553,8 @@ static void nvmet_port_release(struct config_item *item)
 {
 	struct nvmet_port *port = to_nvmet_port(item);
 
+	/* Let inflight controllers teardown complete */
+	flush_scheduled_work();
 	list_del(&port->global_entry);
 
 	kfree(port->ana_state);
Signature · 7 lines
2.16.3


_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme
#2 lore raw

PATCH 2/3 nvmet_rdma: Fix use-after-free when a port is removed

In reply to Israel Rukshin

When removing a port, all its controllers are being removed, but there are queues on the port that doesn't belong to any controller (during connection time). This causes a use-after-free bug for any command that dereferences req->port (like in nvmet_alloc_ctrl). Those queues should be destroyed before freeing the port via configfs. Destroy the remaining queues after the RDMA-CM was destroyed guarantees that no new queue will be created.

---
 drivers/nvme/target/rdma.c | 24 ++++++++++++++++++++++++
 1 file changed, 24 insertions(+)
drivers/nvme/target/rdma.c +24 -0
index 891174ccd44b..f1eedbf493d5 100644
@@ -1818,12 +1818,36 @@ static void nvmet_rdma_delete_ctrl(struct nvmet_ctrl *ctrl)
 	mutex_unlock(&nvmet_rdma_queue_mutex);
 }
 
+static void nvmet_rdma_destroy_port_queues(struct nvmet_rdma_port *port)
+{
+	struct nvmet_rdma_queue *queue, *tmp;
+	struct nvmet_port *nport = port->nport;
+
+	mutex_lock(&nvmet_rdma_queue_mutex);
+	list_for_each_entry_safe(queue, tmp, &nvmet_rdma_queue_list,
+				 queue_list) {
+		if (queue->port != nport)
+			continue;
+
+		list_del_init(&queue->queue_list);
+		__nvmet_rdma_queue_disconnect(queue);
+	}
+	mutex_unlock(&nvmet_rdma_queue_mutex);
+}
+
 static void nvmet_rdma_disable_port(struct nvmet_rdma_port *port)
 {
 	struct rdma_cm_id *cm_id = xchg(&port->cm_id, NULL);
 
 	if (cm_id)
 		rdma_destroy_id(cm_id);
+
+	/*
+	 * Destroy the remaining queues, which are not belong to any
+	 * controller yet. Do it here after the RDMA-CM was destroyed
+	 * guarantees that no new queue will be created.
+	 */
+	nvmet_rdma_destroy_port_queues(port);
 }
 
 static int nvmet_rdma_enable_port(struct nvmet_rdma_port *port)
Signature · 7 lines
2.16.3


_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme
#3 lore raw

PATCH 3/3 nvmet_tcp: Fix use-after-free when a port is removed

In reply to Israel Rukshin

When removing a port, all its controllers are being removed, but there are queues on the port that doesn't belong to any controller (during connection time). This causes a use-after-free bug for any command that dereferences req->port (like in nvmet_alloc_ctrl). Those queues should be destroyed before freeing the port via configfs. Destroy the remaining queues after the accept_work was cancelled guarantees that no new queue will be created.

---
 drivers/nvme/target/tcp.c | 16 ++++++++++++++++
 1 file changed, 16 insertions(+)
drivers/nvme/target/tcp.c +16 -0
index 07ee347ea3f3..6eb0b3153477 100644
@@ -1737,6 +1737,17 @@ static int nvmet_tcp_add_port(struct nvmet_port *nport)
 	return ret;
 }
 
+static void nvmet_tcp_destroy_port_queues(struct nvmet_tcp_port *port)
+{
+	struct nvmet_tcp_queue *queue;
+
+	mutex_lock(&nvmet_tcp_queue_mutex);
+	list_for_each_entry(queue, &nvmet_tcp_queue_list, queue_list)
+		if (queue->port == port)
+			kernel_sock_shutdown(queue->sock, SHUT_RDWR);
+	mutex_unlock(&nvmet_tcp_queue_mutex);
+}
+
 static void nvmet_tcp_remove_port(struct nvmet_port *nport)
 {
 	struct nvmet_tcp_port *port = nport->priv;
@@ -1746,6 +1757,11 @@ static void nvmet_tcp_remove_port(struct nvmet_port *nport)
 	port->sock->sk->sk_user_data = NULL;
 	write_unlock_bh(&port->sock->sk->sk_callback_lock);
 	cancel_work_sync(&port->accept_work);
+	/*
+	 * Destroy the remaining queues, which are not belong to any
+	 * controller yet.
+	 */
+	nvmet_tcp_destroy_port_queues(port);
 
 	sock_release(port->sock);
 	kfree(port);
Signature · 7 lines
2.16.3


_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme