nvmet: fix subtle race in I/O processing and ns configuration
Subsystems (2)
cover nvmet: fix subtle race in I/O processing and ns configuration
Hi,
This series addresses two race conditions in the nvmet core code that handles I/O submission and namespace configuration.
The first patch addresses a race where target namespace attributes could be changed while I/O is in progress. The second patch addresses a subtle race that could allow I/O to be admitted even after the percpu reference count has been killed or marked dead.
As usual, feedback, comments, and suggestions are most welcome!
Thanks!
Nilay Shroff (2):
nvmet: defer setting ns->enabled to false in nvmet_ns_disable()
nvmet: don't allow I/O admission after percpu ns reference is killed
drivers/nvme/target/core.c | 23 +++++++++++++++-------- 1 file changed, 15 insertions(+), 8 deletions(-)
Signature · 1 line
2.53.0
PATCH 1/2 nvmet: defer setting ns->enabled to false in nvmet_ns_disable()
nvmet_ns_disable() currently clears ns->enabled before draining in-flight I/O references. This allows namespace configuration to be changed while existing I/O requests can still hold a reference to the namespace.
This can race with configuration of namespace attributes such as the device path, UUID, NGUID etc. These attributes can be accessed by I/O requests without holding subsys->lock and must not be modified while such requests are still using the namespace.
In nvmet_ns_disable(), keep ns->enabled set while existing namespace references are being drained, so namespace configuration remains blocked until all in-flight I/O has completed. Set ns->enabled to false only after the namespace references have been drained and the namespace device has been disabled.
Since setting ns->enabled to false is deferred in nvmet_ns_disable(), use the NVMET_NS_ENABLED XArray mark in nvmet_req_find_ns() to determine whether a namespace can accept new I/O.
Similarly, use the NVMET_NS_ENABLED XArray mark in nvmet_ns_disable() to prevent concurrent callers from starting namespace disable.
--- drivers/nvme/target/core.c | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-)
drivers/nvme/target/core.c +4 -3
@@ -448,7 +448,8 @@ u16 nvmet_req_find_ns(struct nvmet_req *req)struct nvmet_subsys *subsys =nvmet_req_subsys (req); req->ns =xa_load (&subsys->namespaces, nsid); -if (unlikely (!req->ns || !req->ns->enabled)) { +if (unlikely (!req->ns) || + !xa_get_mark (&subsys->namespaces, nsid,NVMET_NS_ENABLED )) { req->error_loc = offsetof(struct nvme_common_command , nsid);if (!req->ns)/* ns doesn't exist! */ return NVME_SC_INVALID_NS |NVME_STATUS_DNR ;
@@ -644,10 +645,9 @@ void nvmet_ns_disable(struct nvmet_ns *ns)struct nvmet_ctrl *ctrl;mutex_lock (&subsys->lock); -if (!ns->enabled) +if (!xa_get_mark (&subsys->namespaces, ns->nsid,NVMET_NS_ENABLED )) goto out_unlock; - ns->enabled = false;xa_clear_mark (&subsys->namespaces, ns->nsid,NVMET_NS_ENABLED );nvmet_debugfs_ns_free (ns);
@@ -675,6 +675,7 @@ void nvmet_ns_disable(struct nvmet_ns *ns)mutex_lock (&subsys->lock);nvmet_ns_changed (subsys, ns->nsid);nvmet_ns_dev_disable (ns); + ns->enabled = false; out_unlock:mutex_unlock (&subsys->lock); }
Signature · 1 line
2.53.0
Christoph Hellwig
maintainerRe: [PATCH 1/2] nvmet: defer setting ns->enabled to false in nvmet_ns_disable()
Quoted 6 lines · Nilay Shroff
In
nvmet_ns_disable(), keepns->enabledset while existing namespace references are being drained, so namespace configuration remains blocked until all in-flight I/O has completed. Setns->enabledto false only after the namespace references have been drained and the namespace device has been disabled.
Quoted 6 lines
Since setting
ns->enabledto false is deferred innvmet_ns_disable(), use theNVMET_NS_ENABLEDXArray mark innvmet_req_find_ns()to determine whether a namespace can accept new I/O.Similarly, use the
NVMET_NS_ENABLEDXArray mark innvmet_ns_disable()to prevent concurrent callers from starting namespace disable.
This does look sane, but have you looked at profiles if getting the xarray tag might touch a lot more cache lines vs the flag? It might make sense to turn enabled into an atomic bit mask and just have two flags in there.
Re: [PATCH 1/2] nvmet: defer setting ns->enabled to false in nvmet_ns_disable()
Quoted 11 lines · Christoph Hellwig
Quoted 10 lines · Nilay Shroff
In
nvmet_ns_disable(), keepns->enabledset while existing namespace references are being drained, so namespace configuration remains blocked until all in-flight I/O has completed. Setns->enabledto false only after the namespace references have been drained and the namespace device has been disabled.Since setting
ns->enabledto false is deferred innvmet_ns_disable(), use theNVMET_NS_ENABLEDXArray mark innvmet_req_find_ns()to determine whether a namespace can accept new I/O.
Quoted 7 lines
Quoted 2 lines
Similarly, use the
NVMET_NS_ENABLEDXArray mark innvmet_ns_disable()to prevent concurrent callers from starting namespace disable.This does look sane, but have you looked at profiles if getting the xarray tag might touch a lot more cache lines vs the flag? It might make sense to turn enabled into an atomic bit mask and just have two flags in there.
Yes, you were correct. Using the XArray mark in the I/O path touches significantly more cache lines.
I'll fix this as you suggested by using atomic bit flags and send out the next revision.
Thanks,
--Nilay
PATCH 2/2 nvmet: don't allow I/O admission after percpu ns reference is killed
In reply to Nilay Shroffnvmet_req_find_ns() uses percpu_ref_get() to obtain a reference to the namespace. However, percpu_ref_get() can acquire a reference even after the namespace reference has been killed (or marked DEAD).
This is undesirable during namespace disable because nvmet_ns_disable() kills the namespace reference and then waits for all outstanding references to drain. Acquiring a new reference after the reference is killed can therefore extend the namespace drain period.
Replace percpu_ref_get() in nvmet_req_find_ns() with percpu_ref_tryget_live_rcu(), which only acquires a reference while the namespace reference is still live. This handles the race where nvmet_req_find_ns() finds ns is enabled but before it acquires the reference to ns, its reference is killed in nvmet_ns_disable(). For instnace check this race:
CPU0 CPU1nvmet_req_find_ns(): nvmet_ns_disable():xa_load() -> ns
mark == set
xa_clear_mark() percpu_ref_kill() // DEAD
percpu_ref_get() synchronize_rcu()
| wait_for_completion()
+-- succeedReplacing percpu_ref_get() with percpu_ref_tryget_live_rcu() prevents the I/O request from acquiring a namespace reference once the reference has been marked DEAD.
Perform the namespace lookup and reference acquisition in nvmet_req_find_ns() within an RCU read-side critical section. nvmet_ns_disable() uses synchronize_rcu() before draining and exiting the namespace reference, ensuring that RCU readers which may be acquiring the namespace reference have completed before the reference is exited.
--- drivers/nvme/target/core.c | 18 ++++++++++++------ 1 file changed, 12 insertions(+), 6 deletions(-)
drivers/nvme/target/core.c +12 -6
@@ -446,21 +446,27 @@ u16 nvmet_req_find_ns(struct nvmet_req *req) {u32 nsid =le32_to_cpu (req->cmd->common.nsid);struct nvmet_subsys *subsys =nvmet_req_subsys (req); +u16 status =NVME_SC_SUCCESS ; +rcu_read_lock (); req->ns =xa_load (&subsys->namespaces, nsid);if (unlikely (!req->ns) || - !xa_get_mark (&subsys->namespaces, nsid,NVMET_NS_ENABLED )) { + !xa_get_mark (&subsys->namespaces, nsid,NVMET_NS_ENABLED ) || + !percpu_ref_tryget_live_rcu (&req->ns->ref)) { req->error_loc = offsetof(struct nvme_common_command , nsid); -if (!req->ns)/* ns doesn't exist! */ -return NVME_SC_INVALID_NS |NVME_STATUS_DNR ; +if (!req->ns) {/* ns doesn't exist! */ + status =NVME_SC_INVALID_NS |NVME_STATUS_DNR ; + goto unlock; + }/* ns exists but it's disabled */ req->ns =NULL ; -return NVME_SC_INTERNAL_PATH_ERROR ; + status =NVME_SC_INTERNAL_PATH_ERROR ; } +unlock: +rcu_read_unlock (); -percpu_ref_get (&req->ns->ref); -return NVME_SC_SUCCESS ; +return status; }static void nvmet_destroy_namespace (struct percpu_ref *ref)
Signature · 1 line
2.53.0