nvmet: fix max_qid race between configfs and controller allocation
Subsystems (2)
PATCH 1/1 nvmet: fix max_qid race between configfs and controller allocation
The function nvmet_subsys_attr_qid_max_store() can race against nvmet_alloc_ctrl() when a subsystem's max_qid limit is modified.
Suppose max_qid is currently 64. If nvmet_alloc_ctrl() executes:ctrl->sqs = kzalloc_objs(struct nvmet_sq *, subsys->max_qid + 1); and at this exact point, a userspace process changes max_qid to 128, nvmet_subsys_attr_qid_max_store() will set the new max_qid value. It attempts to delete active controllers to force a reconnect, but the new controller won't be deleted because it hasn't been added to the subsys->ctrls list yet.
nvmet_alloc_ctrl() then proceeds and adds the new controller to the subsys->ctrls list. Later, when nvmet_install_queue() is called, it will see max_qid set to 128, but the memory allocated for sqs is only sized for 64 entries. This results in a KASAN out-of-bounds warning and potential memory corruptions.
Fix this by protecting the queue allocations and list insertion in nvmet_alloc_ctrl() with down_read(&nvmet_config_sem). Because nvmet_subsys_attr_qid_max_store() acquires down_write(&nvmet_config_sem) to modify the attribute, this safely prevents the configfs writer from modifying max_qid during controller creation.
--- drivers/nvme/target/core.c | 39 ++++++++++++++++++++------------------ 1 file changed, 21 insertions(+), 18 deletions(-)
drivers/nvme/target/core.c +21 -18
@@ -1652,6 +1652,22 @@ struct nvmet_ctrl *nvmet_alloc_ctrl(struct nvmet_alloc_ctrl_args *args)if (!ctrl->changed_ns_list) goto out_free_ctrl; +/* +* Discovery controllers may use some arbitrary high value +* in order to cleanup stale discovery sessions +*/ +if (nvmet_is_disc_subsys (ctrl->subsys) && !kato) + kato =NVMET_DISC_KATO_MS ; + +/* keep-alive timeout in seconds */ + ctrl->kato =DIV_ROUND_UP (kato,1000 ); + + ctrl->err_counter =0 ; +spin_lock_init (&ctrl->error_lock); + +down_read (&nvmet_config_sem); +mutex_lock (&subsys->lock); + ctrl->sqs = kzalloc_objs(struct nvmet_sq *, subsys->max_qid + 1);if (!ctrl->sqs) goto out_free_changed_ns_list;
@@ -1669,22 +1685,6 @@ struct nvmet_ctrl *nvmet_alloc_ctrl(struct nvmet_alloc_ctrl_args *args) } ctrl->cntlid = ret; -/* -* Discovery controllers may use some arbitrary high value -* in order to cleanup stale discovery sessions -*/ -if (nvmet_is_disc_subsys (ctrl->subsys) && !kato) - kato =NVMET_DISC_KATO_MS ; - -/* keep-alive timeout in seconds */ - ctrl->kato =DIV_ROUND_UP (kato,1000 ); - - ctrl->err_counter =0 ; -spin_lock_init (&ctrl->error_lock); - -nvmet_start_keep_alive_timer (ctrl); - -mutex_lock (&subsys->lock); ret =nvmet_ctrl_init_pr (ctrl);if (ret) goto init_pr_fail;
@@ -1692,6 +1692,9 @@ struct nvmet_ctrl *nvmet_alloc_ctrl(struct nvmet_alloc_ctrl_args *args)nvmet_setup_p2p_ns_map (ctrl, args->p2p_client);nvmet_debugfs_ctrl_setup (ctrl);mutex_unlock (&subsys->lock); +up_read (&nvmet_config_sem); + +nvmet_start_keep_alive_timer (ctrl);if (args->hostid)uuid_copy (&ctrl->hostid, args->hostid);
@@ -1721,14 +1724,14 @@ struct nvmet_ctrl *nvmet_alloc_ctrl(struct nvmet_alloc_ctrl_args *args)return ctrl; init_pr_fail: -mutex_unlock (&subsys->lock); -nvmet_stop_keep_alive_timer (ctrl);ida_free (&cntlid_ida, ctrl->cntlid); out_free_cqs:kfree (ctrl->cqs); out_free_sqs:kfree (ctrl->sqs); out_free_changed_ns_list: +mutex_unlock (&subsys->lock); +up_read (&nvmet_config_sem);kfree (ctrl->changed_ns_list); out_free_ctrl:kfree (ctrl);
Signature · 1 line
2.55.0
Keith Busch
maintainerQuoted 5 lines · Maurizio Lombardi
Fix this by protecting the queue allocations and list insertion in
nvmet_alloc_ctrl()withdown_read(&nvmet_config_sem). Becausenvmet_subsys_attr_qid_max_store()acquiresdown_write(&nvmet_config_sem)to modify the attribute, this safely prevents the configfs writer from modifyingmax_qidduring controller creation.
I think there's still a race in nvmet_check_cqid() such that a command can access an out-of-bounds index if the target side changes subsys->max_qid to a larger value and the host sends a command with a larger cqid than the array was initially allocated. Should the max_qid be copied from the subystem to the controller that allocated the queue array?
Quoted 7 lines · Keith Busch
Quoted 5 lines · Maurizio Lombardi
Fix this by protecting the queue allocations and list insertion in
nvmet_alloc_ctrl()withdown_read(&nvmet_config_sem). Becausenvmet_subsys_attr_qid_max_store()acquiresdown_write(&nvmet_config_sem)to modify the attribute, this safely prevents the configfs writer from modifyingmax_qidduring controller creation.
Quoted 6 lines
I think there's still a race in
nvmet_check_cqid()such that a command can access an out-of-bounds index if the target side changessubsys->max_qidto a larger value and the host sends a command with a larger cqid than the array was initially allocated. Should themax_qidbe copied from the subystem to the controller that allocated the queue array?
Hmm I see, considering that changing max_qid triggers resets to all of the controllers, copying the value to the controllers' structure should indeed fix all these unsafe accesses.
I will submit a V2.
Maurizio