nvmet: force reconnect when number of queue changes
Subsystems (2)
PATCH v4 1/1 nvmet: force reconnect when number of queue changes
In order to test queue number changes we need to make sure that the host reconnects. Because only when the host disconnects from the target the number of queues are allowed to change according the spec.
The initial idea was to disable and re-enable the ports and have the host wait until the KATO timer expires, triggering error recovery. Though the host would see a DNR reply when trying to reconnect. Because of the DNR bit the connection is dropped completely. There is no point in trying to reconnect with the same parameters according the spec.
We can force to reconnect the host is by deleting all controllers. The host will observe any newly posted request to fail and thus starts the error recovery but this time without the DNR bit set.
---
v4:
- Rebased on -rc2
- invalid memory fix went in as separate fix 94f5a0688407 ("nvmet: fix invalid memory reference in nvmet_subsys_attr_qid_max_show")
v3:
- Updated Fixes tag
- Updated commit message with some additional information
from the v2 discussion
- https://lore.kernel.org/linux-nvme/[email protected]/
v2:
- instead preventing changes, force reconnect by delete ctrls
- renamed patch
- https://lore.kernel.org/linux-nvme/[email protected]/
v1:
- initial verison
- https://lore.kernel.org/linux-nvme/[email protected]/
drivers/nvme/target/configfs.c | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-)
drivers/nvme/target/configfs.c +8 -1
@@ -1290,6 +1290,8 @@ static ssize_t nvmet_subsys_attr_qid_max_show(struct config_item *item,static ssize_t nvmet_subsys_attr_qid_max_store (struct config_item *item,const char *page,size_t cnt) { +struct nvmet_subsys *subsys =to_subsys (item); +struct nvmet_ctrl *ctrl;u16 qid_max;if (sscanf (page,"%hu\n" , &qid_max) !=1 )
@@ -1299,8 +1301,13 @@ static ssize_t nvmet_subsys_attr_qid_max_store(struct config_item *item,return -EINVAL ;down_write (&nvmet_config_sem); -to_subsys (item)->max_qid = qid_max; + subsys->max_qid = qid_max; + +/* Force reconnect */ +list_for_each_entry (ctrl, &subsys->ctrls, subsys_entry) + ctrl->ops->delete_ctrl (ctrl);up_write (&nvmet_config_sem); +return cnt; }CONFIGFS_ATTR (nvmet_subsys_, attr_qid_max);
Signature · 1 line
2.38.0
Quoted 11 lines · Daniel Wagner
In order to test queue number changes we need to make sure that the host reconnects. Because only when the host disconnects from the target the number of queues are allowed to change according the spec.
The initial idea was to disable and re-enable the ports and have the host wait until the KATO timer expires, triggering error recovery. Though the host would see a DNR reply when trying to reconnect. Because of the DNR bit the connection is dropped completely. There is no point in trying to reconnect with the same parameters according the spec.
Quoted 4 lines
We can force to reconnect the host is by deleting all controllers. The host will observe any newly posted request to fail and thus starts the error recovery but this time without the DNR bit set.
Without looking into the spec, isn't some sort of AEN should be used for this ? please correct me if I'm wrong but deleting all the controllers and relaying on reconnect maybe overkill ? if it doesn't exists in the NVMe spec then perhaps we should think/work on it to update the spec ? Is it worth it ?
Quoted 4 lines
Reviewed-by Hannes Reinecke <[email protected]>Acked-by Sagi Grimberg <[email protected]>Signed-off-by Daniel Wagner <[email protected]>---
-ck
Quoted 17 lines · Chaitanya Kulkarni
Quoted 11 lines · Daniel Wagner
In order to test queue number changes we need to make sure that the host reconnects. Because only when the host disconnects from the target the number of queues are allowed to change according the spec.
The initial idea was to disable and re-enable the ports and have the host wait until the KATO timer expires, triggering error recovery. Though the host would see a DNR reply when trying to reconnect. Because of the DNR bit the connection is dropped completely. There is no point in trying to reconnect with the same parameters according the spec.
Quoted 4 lines
We can force to reconnect the host is by deleting all controllers. The host will observe any newly posted request to fail and thus starts the error recovery but this time without the DNR bit set.
Quoted 6 lines
Without looking into the spec, isn't some sort of AEN should be used for this ? please correct me if I'm wrong but deleting all the controllers and relaying on reconnect maybe overkill ? if it doesn't exists in the NVMe spec then perhaps we should think/work on it to update the spec ? Is it worth it ?
Problem is that changing the number of queues on the fly is really awkward, _and_ you have the problem that the number of queues is negotiated during the 'connect' call. So changing the number of queues from the controller side really requires us to reconnect to rectify the situation.
And no, I don't think it's overkill. It actually makes the situation easier as you start off from scratch with the correct number of queues.
Plus you have to quiesce all queues anyway when changing the number of queues; the tagset will be modified, requiring you to stop all I/O when doing so. So from the block layer side there isn't much of a difference here.
Cheers,
Hannes
Signature · 5 lines
Dr. Hannes Reinecke Kernel Storage Architect [email protected] +49 911 74053 688 SUSE Software Solutions GmbH, Maxfeldstr. 5, 90409 Nürnberg HRB 36809 (AG Nürnberg), Geschäftsführer: Ivo Totev, Andrew Myers, Andrew McDonald, Martje Boudien Moerman
Quoted 5 lines · Hannes Reinecke
Quoted 5 lines
Without looking into the spec, isn't some sort of AEN should be used for this ? please correct me if I'm wrong but deleting all the controllers and relaying on reconnect maybe overkill ? if it doesn't exists in the NVMe spec then perhaps we should think/work on it to update the spec ? Is it worth it ?
As it turns out there is no AEN which really matches this. We scenario had a short discussion on this topic with Fred during ALPSS and he was not against adding something if there is a good use case for it.
The issue is that no one really came up with a good general use case except for testing corner cases.
Quoted 7 lines
Problem is that changing the number of queues on the fly is really awkward, _and_ you have the problem that the number of queues is negotiated during the 'connect' call. So changing the number of queues from the controller side really requires us to reconnect to rectify the situation.
And no, I don't think it's overkill. It actually makes the situation easier as you start off from scratch with the correct number of queues.
There is also the argument, that this approach doesn't rely on the host to cooperate correctly. For example, if we had an AEN to tell the host 'hey, drop all connections and reconnect', the target still doesn't know when and if this has been done by the host.
Thanks,
applied to nvme-6.2.