nvme-fabrics: fix crash for no IO queues
PATCH 1/1 nvme-fabrics: fix crash for no IO queues
A crash happens when set feature(NVME_FEAT_NUM_QUEUES) timeout in nvme over rdma(roce) reconnection, the reason is use the queue which is not alloced.
If queue is not live, should not allow queue request.
--- drivers/nvme/host/fabrics.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
drivers/nvme/host/fabrics.h +1 -1
@@ -189,7 +189,7 @@ static inline bool nvmf_check_ready(struct nvme_ctrl *ctrl, struct request *rq, {if (likely (ctrl->state ==NVME_CTRL_LIVE || ctrl->state ==NVME_CTRL_DELETING )) -return true; +return queue_live;return __nvmf_check_ready (ctrl, rq, queue_live); }
Signature · 7 lines
2.16.4 _______________________________________________ Linux-nvme mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-nvme
Sagi Grimberg
maintainerQuoted 6 lines
A crash happens when set
feature(NVME_FEAT_NUM_QUEUES)timeout in nvme over rdma(roce) reconnection, the reason is use the queue which is not alloced.If queue is not live, should not allow queue request.
Can you describe exactly the scenario here? What is the state here? LIVE? or DELETING?
Quoted 11 lines
Signed-off-by Chao Leng <[email protected]>--- drivers/nvme/host/fabrics.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)drivers/nvme/host/fabrics.h +0
-0
drivers/nvme/host/fabrics.h static inline bool nvmf_check_ready(struct nvme_ctrl *ctrl, struct request *rq, · line 194
Quoted 7 lines
drivers/nvme/host/fabrics.h +1
-1@@ -189,7 +189,7 @@ static inline bool nvmf_check_ready(struct nvme_ctrl *ctrl, struct request *rq, {if (likely (ctrl->state ==NVME_CTRL_LIVE || ctrl->state ==NVME_CTRL_DELETING )) -return true; +return queue_live;return __nvmf_check_ready (ctrl, rq, queue_live); }
There were some issues in the past that made us allow submitting
requests in DELETING state and introducing DELETING_NOIO. See
patch ecca390e8056 ("nvme: fix deadlock in disconnect during scan_work and/or ana_work")
The driver should be able to accept I/O in DELETING because the core changes the state to DELETING_NOIO _before_ it calls ->delete_ctrl so I don't understand how you get to this if the queue is not allocated...
_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme
Quoted 9 lines · Sagi Grimberg
Quoted 6 lines
A crash happens when set
feature(NVME_FEAT_NUM_QUEUES)timeout in nvme over rdma(roce) reconnection, the reason is use the queue which is not alloced.If queue is not live, should not allow queue request.
Can you describe exactly the scenario here? What is the state here? LIVE? or DELETING?
If seting feature(NVME_FEAT_NUM_QUEUES) failed due to time out or the target return 0 io queues, nvme_set_queue_count will return 0, and then reconnection will continue and success. The state of controller is LIVE. The request will continue to deliver by call ->queue_rq(), and then crash happens.
Quoted 7 lines
Quoted 7 lines
Signed-off-by Chao Leng <[email protected]>---
drivers/nvme/host/fabrics.h| 2 +-1 file changed, 1 insertion(+), 1 deletion(-)
Quoted 16 lines
Quoted 6 lines
drivers/nvme/host/fabrics.h +0
-0@@ -189,7 +189,7 @@ static inline bool nvmf_check_ready(struct nvme_ctrl *ctrl, struct request *rq, {Quoted 6 lines
if (
likely(ctrl->state == NVME_CTRL_LIVE || ctrl->state == NVME_CTRL_DELETING)) - return true;
+ returnqueue_live;
return__nvmf_check_ready(ctrl, rq, queue_live); }There were some issues in the past that made us allow submitting
requests in DELETING state and introducingDELETING_NOIO. See
patchecca390e8056("nvme: fix deadlock in disconnect duringscan_workand/orana_work")
This doesn't make any difference. When in deleting state the queue is still live.
Quoted 4 lines
The driver should be able to accept I/O in DELETING because the core changes the state to
DELETING_NOIO_before_ it calls ->delete_ctrl so I don't understand how you get to this if the queue is not allocated...
The state of controller is live. The deletion process looks good.
Quoted 1 line
.
_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme
Sagi Grimberg
maintainerQuoted 7 lines
Quoted 7 lines
Quoted 6 lines
A crash happens when set
feature(NVME_FEAT_NUM_QUEUES)timeout in nvme over rdma(roce) reconnection, the reason is use the queue which is not alloced.If queue is not live, should not allow queue request.
Quoted 7 lines
Quoted 2 lines
Can you describe exactly the scenario here? What is the state here? LIVE? or DELETING?
If seting
feature(NVME_FEAT_NUM_QUEUES)failed due to time out or the target return 0 io queues,nvme_set_queue_countwill return 0, and then reconnection will continue and success. The state of controller is LIVE. The request will continue to deliver by call ->queue_rq(), and then crash happens.
Thinking about this again, we should absolutely fail the reconnection when we are unable to set any I/O queues, it is just wrong to keep this controller alive...
This should be fixed for both rdma and tcp.
_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme
Quoted 15 lines · Sagi Grimberg
Quoted 7 lines
Quoted 7 lines
Quoted 6 lines
A crash happens when set
feature(NVME_FEAT_NUM_QUEUES)timeout in nvme over rdma(roce) reconnection, the reason is use the queue which is not alloced.If queue is not live, should not allow queue request.
Quoted 7 lines
Quoted 2 lines
Can you describe exactly the scenario here? What is the state here? LIVE? or DELETING?
If seting
feature(NVME_FEAT_NUM_QUEUES)failed due to time out or the target return 0 io queues,nvme_set_queue_countwill return 0, and then reconnection will continue and success. The state of controller is LIVE. The request will continue to deliver by call ->queue_rq(), and then crash happens.
Quoted 3 lines
Thinking about this again, we should absolutely fail the reconnection when we are unable to set any I/O queues, it is just wrong to keep this controller alive...
Keith think keeping the controller alive for diagnose is better. This is the patch which failed the connection.
https://lore.kernel.org/linux-nvme/[email protected]/
Now we have 2 choice:
1.failed the connection when unable to set any I/O queues. 2.do not allow queue request when queue is not live.
From a service continuity perspective, I think it is better that failed the connection when unable to set any I/O queues.
Diagnose is less important, I prefer service continuity, because if failed the reconnection and then try new reconnection, it is possible to recover.
Quoted 3 lines
This should be fixed for both rdma and tcp.
.
_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme
Keith Busch
maintainerQuoted 21 lines · Chao Leng
Quoted 15 lines · Sagi Grimberg
Quoted 7 lines
Quoted 7 lines
Quoted 6 lines
A crash happens when set
feature(NVME_FEAT_NUM_QUEUES)timeout in nvme over rdma(roce) reconnection, the reason is use the queue which is not alloced.If queue is not live, should not allow queue request.
Quoted 7 lines
Quoted 2 lines
Can you describe exactly the scenario here? What is the state here? LIVE? or DELETING?
If seting
feature(NVME_FEAT_NUM_QUEUES)failed due to time out or the target return 0 io queues,nvme_set_queue_countwill return 0, and then reconnection will continue and success. The state of controller is LIVE. The request will continue to deliver by call ->queue_rq(), and then crash happens.Quoted 3 lines
Thinking about this again, we should absolutely fail the reconnection when we are unable to set any I/O queues, it is just wrong to keep this controller alive...
Quoted 7 lines
Keith think keeping the controller alive for diagnose is better. This is the patch which failed the connection.
https://lore.kernel.org/linux-nvme/[email protected]/Now we have 2 choice:
1.failed the connection when unable to set any I/O queues. 2.do not allow queue request when queue is not live.
Okay, so there are different views on how to handles this. I personally find in-band administration for a misbehaving device is a good thing to have, but I won't 'nak' if the consensus from the people using this is for the other way.
Quoted 4 lines
From a service continuity perspective, I think it is better that failed the connection when unable to set any I/O queues.
Diagnose is less important, I prefer service continuity, because if failed the reconnection and then try new reconnection, it is possible to recover.
_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme
Sagi Grimberg
maintainerQuoted 22 lines
Quoted 22 lines
Quoted 15 lines
Quoted 7 lines
Quoted 7 lines
Quoted 6 lines
A crash happens when set
feature(NVME_FEAT_NUM_QUEUES)timeout in nvme over rdma(roce) reconnection, the reason is use the queue which is not alloced.If queue is not live, should not allow queue request.
Quoted 7 lines
Quoted 2 lines
Can you describe exactly the scenario here? What is the state here? LIVE? or DELETING?
If seting
feature(NVME_FEAT_NUM_QUEUES)failed due to time out or the target return 0 io queues,nvme_set_queue_countwill return 0, and then reconnection will continue and success. The state of controller is LIVE. The request will continue to deliver by call ->queue_rq(), and then crash happens.Quoted 3 lines
Thinking about this again, we should absolutely fail the reconnection when we are unable to set any I/O queues, it is just wrong to keep this controller alive...
Keith think keeping the controller alive for diagnose is better. This is the patch which failed the connection.
https://lore.kernel.org/linux-nvme/[email protected]/
Quoted 7 lines
Quoted 3 lines
Now we have 2 choice:
1.failed the connection when unable to set any I/O queues. 2.do not allow queue request when queue is not live.Okay, so there are different views on how to handles this. I personally find in-band administration for a misbehaving device is a good thing to have, but I won't 'nak' if the consensus from the people using this is for the other way.
While I understand that this can be useful, I've seen it do more harm than good. It is really puzzling to people when the controller state reflected is live (and even optimized) and no I/O is making progress for unknown reason. And logs are rarely accessed in these cases.
I am also opting for failing it and rescheduling a reconnect.
_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme
Quoted 31 lines · Sagi Grimberg
Quoted 23 lines
Quoted 23 lines
Quoted 15 lines
Quoted 7 lines
Quoted 7 lines
Quoted 6 lines
A crash happens when set
feature(NVME_FEAT_NUM_QUEUES)timeout in nvme over rdma(roce) reconnection, the reason is use the queue which is not alloced.If queue is not live, should not allow queue request.
Quoted 7 lines
Quoted 2 lines
Can you describe exactly the scenario here? What is the state here? LIVE? or DELETING?
If seting
feature(NVME_FEAT_NUM_QUEUES)failed due to time out or the target return 0 io queues,nvme_set_queue_countwill return 0, and then reconnection will continue and success. The state of controller is LIVE. The request will continue to deliver by call ->queue_rq(), and then crash happens.Quoted 3 lines
Thinking about this again, we should absolutely fail the reconnection when we are unable to set any I/O queues, it is just wrong to keep this controller alive...
Keith think keeping the controller alive for diagnose is better. This is the patch which failed the connection.
https://lore.kernel.org/linux-nvme/[email protected]/Quoted 7 lines
Quoted 3 lines
Now we have 2 choice:
1.failed the connection when unable to set any I/O queues. 2.do not allow queue request when queue is not live.Okay, so there are different views on how to handles this. I personally find in-band administration for a misbehaving device is a good thing to have, but I won't 'nak' if the consensus from the people using this is for the other way.
Quoted 6 lines
While I understand that this can be useful, I've seen it do more harm than good. It is really puzzling to people when the controller state reflected is live (and even optimized) and no I/O is making progress for unknown reason. And logs are rarely accessed in these cases.
I am also opting for failing it and rescheduling a reconnect.
Agree with Sagi. We also hit this issue a long time ago and I made the same change (commit 834d3710a093a) that Sagi is suggesting: if the prior controller instance had io queues, but the new/reconnected controller fails to create io queues, then the controller create is failed and a reconnect is scheduled.
-- james
_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme
Keith Busch
maintainerQuoted 43 lines · James Smart
Quoted 42 lines · Sagi Grimberg
Quoted 30 lines
Quoted 21 lines
Quoted 18 lines
Quoted 9 lines
Quoted 9 lines
Quoted 8 lines
A crash happens when set
feature(NVME_FEAT_NUM_QUEUES)timeout in nvme over rdma(roce) reconnection, the reason is use the queue which is not
alloced.If queue is not live, should not allow queue request.
Quoted 8 lines
Quoted 2 lines
Can you describe exactly the scenario here? What is the state here? LIVE? or DELETING?
If seting
feature(NVME_FEAT_NUM_QUEUES)failed due to time out or the target return 0 io queues,nvme_set_queue_countwill return 0, and then reconnection will continue and success. The state of controller
is LIVE. The request will continue to deliver by call ->queue_rq(), and then crash happens.Quoted 3 lines
Thinking about this again, we should absolutely fail the reconnection when we are unable to set any I/O queues, it is just wrong to keep this controller alive...
Quoted 8 lines
Keith think keeping the controller alive for diagnose is better. This is the patch which failed the connection.
https://lore.kernel.org/linux-nvme/[email protected]/Now we have 2 choice:
1.failed the connection when unable to set any I/O queues. 2.do not allow queue request when queue is not live.Quoted 6 lines
Okay, so there are different views on how to handles this. I personally find
in-band administration for a misbehaving device is a good thing to have, but I
won't 'nak' if the consensus from the people using this is for the other way.While I understand that this can be useful, I've seen it do more harm than good. It is really puzzling to people when the controller state reflected is live (and even optimized) and no I/O is making progress for unknown reason. And logs are rarely accessed in these cases.
Quoted 7 lines
Quoted 1 line
I am also opting for failing it and rescheduling a reconnect.
Agree with Sagi. We also hit this issue a long time ago and I made the same change (commit
834d3710a093a) that Sagi is suggesting: if the prior controller instance had io queues, but the new/reconnected controller fails to create io queues, then the controller create is failed and a reconnect is scheduled.
Okay, fair enough.
One more question: if the controller is in such a bad way that it will never create IO queues without additional intervention, will this behavior have the driver schedule reconnect indefinitely?
_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme
Sagi Grimberg
maintainerQuoted 25 lines
Quoted 17 lines
Quoted 17 lines
Quoted 11 lines
Quoted 4 lines
Now we have 2 choice:
1.failed the connection when unable to set any I/O queues. 2.do not allow queue request when queue is not live.Okay, so there are different views on how to handles this. I personally find
in-band administration for a misbehaving device is a good thing to have, but I
won't 'nak' if the consensus from the people using this is for the other way.While I understand that this can be useful, I've seen it do more harm than good. It is really puzzling to people when the controller state reflected is live (and even optimized) and no I/O is making progress for unknown reason. And logs are rarely accessed in these cases.
Quoted 7 lines
Quoted 1 line
I am also opting for failing it and rescheduling a reconnect.
Agree with Sagi. We also hit this issue a long time ago and I made the same change (commit
834d3710a093a) that Sagi is suggesting: if the prior controller instance had io queues, but the new/reconnected controller fails to create io queues, then the controller create is failed and a reconnect is scheduled.
Quoted 5 lines
Okay, fair enough.
One more question: if the controller is in such a bad way that it will never create IO queues without additional intervention, will this behavior have the driver schedule reconnect indefinitely?
Until either ctrl_loss_tmo expires or the user is tired of this controller and manually disconnects.
_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme
Quoted 6 lines
One more question: if the controller is in such a bad way that it will never create IO queues without additional intervention, will this behavior have the driver schedule reconnect indefinitely?
No.. in fabrics, there is a max controller reconnect timeout that will expire (dflt 600s), stop the reconnect attempts and delete the controller.
-- james
_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme