Skip to contentSkip to conversation

Fix use-after-free bug when ports are removed

PATCH Superseded Logan Gunthorpe 3 participants 9 messages Jul 3, 2019 – Jul 5, 2019

Subsystems (2)

v2 of 3 Superseded Latest is v3 →

lore
Share & tools
lore
#0 lore raw

v2 cover Fix use-after-free bug when ports are removed

Hey,

This is the second attempt at fixing this.

Per Sagi's feedback on the first attempt, I've found an approach that disconnects active controllers when the subsys is removed from the port (Patch 1). Patch 2 fixes a race that still exists in the loop transport which requires us to flush the nvme_delete_wq before freeing the port to prevent the use-after-free bug.

Logan

--

NVME target ports can be removed while there are still active controllers. Largely this is fine, except some admin commands can access the req->port (for example, id-ctrl uses the port's inline date size as part of it's response). This was found while testing with KASAN.

--

Logan Gunthorpe (2):
nvmet: Fix use-after-free bug when a port is removed
nvmet-loop: Flush nvme_delete_wq when removing the port

 drivers/nvme/target/configfs.c |  1 +
 drivers/nvme/target/core.c     | 12 ++++++++++++
 drivers/nvme/target/loop.c     |  8 ++++++++
 drivers/nvme/target/nvmet.h    |  3 +++
 4 files changed, 24 insertions(+)

--
2.20.1

#1 lore raw

PATCH v2 1/2 nvmet: Fix use-after-free bug when a port is removed

When a port is removed through configfs, any connected controllers are still active and can still send commands. This causes a use-after-free bug which is detected by KASAN for any admin command that dereferences req->port (like in nvmet_execute_identify_ctrl).

To fix this, disconnect all active controllers when a subsystem is removed from a port. This ensures there are no active controllers when the port is eventually removed.

Signed-off-by Logan Gunthorpe <logang at deltatee.com>
---
 drivers/nvme/target/configfs.c |  1 +
 drivers/nvme/target/core.c     | 12 ++++++++++++
 drivers/nvme/target/nvmet.h    |  3 +++
 3 files changed, 16 insertions(+)
drivers/nvme/target/configfs.c +1 -0
index 08dd5af357f7..3854363118cc 100644
@@ -673,6 +673,7 @@ static void nvmet_port_subsys_drop_link(struct config_item *parent,
 
 found:
 	list_del(&p->entry);
+	nvmet_port_del_ctrls(port, subsys);
 	nvmet_port_disc_changed(port, subsys);
 
 	if (list_empty(&port->subsystems))
drivers/nvme/target/core.c +12 -0
index 7734a6acff85..e4db9a441168 100644
@@ -277,6 +277,18 @@ void nvmet_unregister_transport(const struct nvmet_fabrics_ops *ops)
 }
 EXPORT_SYMBOL_GPL(nvmet_unregister_transport);
 
+void nvmet_port_del_ctrls(struct nvmet_port *port, struct nvmet_subsys *subsys)
+{
+	struct nvmet_ctrl *ctrl;
+
+	mutex_lock(&subsys->lock);
+	list_for_each_entry(ctrl, &subsys->ctrls, subsys_entry) {
+		if (ctrl->port == port)
+			ctrl->ops->delete_ctrl(ctrl);
+	}
+	mutex_unlock(&subsys->lock);
+}
+
 int nvmet_enable_port(struct nvmet_port *port)
 {
 	const struct nvmet_fabrics_ops *ops;
drivers/nvme/target/nvmet.h +3 -0
index c25d88fc9dec..b6b0d483e0c5 100644
@@ -415,6 +415,9 @@ void nvmet_port_send_ana_event(struct nvmet_port *port);
 int nvmet_register_transport(const struct nvmet_fabrics_ops *ops);
 void nvmet_unregister_transport(const struct nvmet_fabrics_ops *ops);
 
+void nvmet_port_del_ctrls(struct nvmet_port *port,
+			  struct nvmet_subsys *subsys);
+
 int nvmet_enable_port(struct nvmet_port *port);
 void nvmet_disable_port(struct nvmet_port *port);
 
Signature · 1 line
2.20.1
maintainer Reviewed-by on v2 1/2
#2 lore raw

Re: [PATCH v2 1/2] nvmet: Fix use-after-free bug when a port is removed

Reviewed-by Sagi Grimberg <sagi at grimberg.me>
#3 lore raw

Re: [PATCH v2 1/2] nvmet: Fix use-after-free bug when a port is removed

In reply to Logan Gunthorpe

Hi Logan,

Quoted 8 lines · Logan Gunthorpe

When a port is removed through configfs, any connected controllers are still active and can still send commands. This causes a use-after-free bug which is detected by KASAN for any admin command that dereferences req->port (like in nvmet_execute_identify_ctrl).

To fix this, disconnect all active controllers when a subsystem is removed from a port. This ensures there are no active controllers when the port is eventually removed.

so now we are enforcing controller existence with port configfs, right ? sounds reasonable.

Did you run your patches with other transport (RDMA/TCP/FC) ?

#4 lore raw

Re: [PATCH v2 1/2] nvmet: Fix use-after-free bug when a port is removed

Quoted 8 lines · Max Gurtovoy

Hi Logan,

Quoted 5 lines · Logan Gunthorpe

When a port is removed through configfs, any connected controllers are still active and can still send commands. This causes a use-after-free bug which is detected by KASAN for any admin command that dereferences req->port (like in nvmet_execute_identify_ctrl).

Quoted 6 lines
Quoted 3 lines

To fix this, disconnect all active controllers when a subsystem is removed from a port. This ensures there are no active controllers when the port is eventually removed.

so now we are enforcing controller existence with port configfs, right ? sounds reasonable.

Correct.

Quoted 1 line

Did you run your patches with other transport (RDMA/TCP/FC) ?

Just RDMA and loop. I suppose I could test with TCP but I don't have FC hardware.

Logan

Reviewed-by
#5 lore raw

Re: [PATCH v2 1/2] nvmet: Fix use-after-free bug when a port is removed

Quoted 13 lines · Logan Gunthorpe
Quoted 11 lines · Max Gurtovoy

Hi Logan,

Quoted 8 lines · Logan Gunthorpe

When a port is removed through configfs, any connected controllers are still active and can still send commands. This causes a use-after-free bug which is detected by KASAN for any admin command that dereferences req->port (like in nvmet_execute_identify_ctrl).

To fix this, disconnect all active controllers when a subsystem is removed from a port. This ensures there are no active controllers when the port is eventually removed.

Quoted 7 lines
Quoted 2 lines

so now we are enforcing controller existence with port configfs, right ? sounds reasonable.

Correct.

Quoted 1 line

Did you run your patches with other transport (RDMA/TCP/FC) ?

Just RDMA and loop. I suppose I could test with TCP but I don't have FC hardware.

Great.

the code looks good:

Reviewed-by Max Gurtovoy <maxg at mellanox.com>
Quoted 2 lines

Logan

#6 lore raw

PATCH v2 2/2 nvmet-loop: Flush nvme_delete_wq when removing the port

In reply to Logan Gunthorpe

After calling nvme_loop_delete_ctrl(), the controllers will not yet be deleted because nvme_delete_ctrl() only schedules work to do the delete.

This means a race can occur if a port is removed but there are still active controllers trying to access that memory.

To fix this, flush the nvme_delete_wq before returning from nvme_loop_remove_port() so that any controllers that might be in the process of being deleted won't access a freed port.

Signed-off-by Logan Gunthorpe <logang at deltatee.com>
---
 drivers/nvme/target/loop.c | 8 ++++++++
 1 file changed, 8 insertions(+)
drivers/nvme/target/loop.c +8 -0
index 9e211ad6bdd3..da9cd07461fb 100644
@@ -654,6 +654,14 @@ static void nvme_loop_remove_port(struct nvmet_port *port)
 	mutex_lock(&nvme_loop_ports_mutex);
 	list_del_init(&port->entry);
 	mutex_unlock(&nvme_loop_ports_mutex);
+
+	/*
+	 * Ensure any ctrls that are in the process of being
+	 * deleted are in fact deleted before we return
+	 * and free the port. This is to prevent active
+	 * ctrls from using a port after it's freed.
+	 */
+	flush_workqueue(nvme_delete_wq);
 }
 
 static const struct nvmet_fabrics_ops nvme_loop_ops = {
Signature · 1 line
2.20.1
maintainer Reviewed-by on v2 2/2
#7 lore raw

Re: [PATCH v2 2/2] nvmet-loop: Flush nvme_delete_wq when removing the port

Reviewed-by Sagi Grimberg <sagi at grimberg.me>
Reviewed-by on v2 2/2
#8 lore raw

Re: [PATCH v2 2/2] nvmet-loop: Flush nvme_delete_wq when removing the port

In reply to Logan Gunthorpe
Quoted 25 lines · Logan Gunthorpe

After calling nvme_loop_delete_ctrl(), the controllers will not yet be deleted because nvme_delete_ctrl() only schedules work to do the delete.

This means a race can occur if a port is removed but there are still active controllers trying to access that memory.

To fix this, flush the nvme_delete_wq before returning from nvme_loop_remove_port() so that any controllers that might be in the process of being deleted won't access a freed port.

Signed-off-by Logan Gunthorpe <logang at deltatee.com>
---
  drivers/nvme/target/loop.c | 8 ++++++++
  1 file changed, 8 insertions(+)
drivers/nvme/target/loop.c +1 -0
index 9e211ad6bdd3..da9cd07461fb 100644
@@ -654,6 +654,14 @@ static void nvme_loop_remove_port(struct nvmet_port *port)
  	mutex_lock(&nvme_loop_ports_mutex);
  	list_del_init(&port->entry);
  	mutex_unlock(&nvme_loop_ports_mutex);
+

drivers/nvme/target/loop.c static void nvme_loop_remove_port(struct nvmet_port *port) · line 667

Quoted 10 lines
drivers/nvme/target/loop.c +7 -0
+	/*
+	 * Ensure any ctrls that are in the process of being
+	 * deleted are in fact deleted before we return
+	 * and free the port. This is to prevent active
+	 * ctrls from using a port after it's freed.
+	 */
+	flush_workqueue(nvme_delete_wq);
  }
 
  static const struct nvmet_fabrics_ops nvme_loop_ops = {

Looks good:

Reviewed-by Max Gurtovoy <maxg at mellanox.com>