Skip to contentSkip to conversation

nvme-fabrics: fix crash for no IO queues

PATCH In review Chao Leng 4 participants 11 messages Mar 4, 2021 – Mar 17, 2021 NVMe

lore
Share & tools
lore

Chao Leng

author
#0 lore raw

PATCH 1/1 nvme-fabrics: fix crash for no IO queues

A crash happens when set feature(NVME_FEAT_NUM_QUEUES) timeout in nvme over rdma(roce) reconnection, the reason is use the queue which is not alloced.

If queue is not live, should not allow queue request.

---
 drivers/nvme/host/fabrics.h | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
drivers/nvme/host/fabrics.h +1 -1
index 733010d2eafd..2479744fc349 100644
@@ -189,7 +189,7 @@ static inline bool nvmf_check_ready(struct nvme_ctrl *ctrl, struct request *rq,
 {
 	if (likely(ctrl->state == NVME_CTRL_LIVE ||
 		   ctrl->state == NVME_CTRL_DELETING))
-		return true;
+		return queue_live;
 	return __nvmf_check_ready(ctrl, rq, queue_live);
 }
 
Signature · 7 lines
2.16.4


_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme
maintainer
#1 lore raw
Quoted 6 lines

A crash happens when set feature(NVME_FEAT_NUM_QUEUES) timeout in nvme over rdma(roce) reconnection, the reason is use the queue which is not alloced.

If queue is not live, should not allow queue request.

Can you describe exactly the scenario here? What is the state here? LIVE? or DELETING?

Quoted 11 lines
---
  drivers/nvme/host/fabrics.h | 2 +-
  1 file changed, 1 insertion(+), 1 deletion(-)
drivers/nvme/host/fabrics.h +0 -0
index 733010d2eafd..2479744fc349 100644

drivers/nvme/host/fabrics.h static inline bool nvmf_check_ready(struct nvme_ctrl *ctrl, struct request *rq, · line 194

Quoted 7 lines
drivers/nvme/host/fabrics.h +1 -1
@@ -189,7 +189,7 @@ static inline bool nvmf_check_ready(struct nvme_ctrl *ctrl, struct request *rq,
  {
  	if (likely(ctrl->state == NVME_CTRL_LIVE ||
  		   ctrl->state == NVME_CTRL_DELETING))
-		return true;
+		return queue_live;
  	return __nvmf_check_ready(ctrl, rq, queue_live);
  }

There were some issues in the past that made us allow submitting
requests in DELETING state and introducing DELETING_NOIO. See
patch ecca390e8056 ("nvme: fix deadlock in disconnect during scan_work and/or ana_work")

The driver should be able to accept I/O in DELETING because the core changes the state to DELETING_NOIO _before_ it calls ->delete_ctrl so I don't understand how you get to this if the queue is not allocated...

_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme

author
#2 lore raw
Quoted 9 lines · Sagi Grimberg
Quoted 6 lines

A crash happens when set feature(NVME_FEAT_NUM_QUEUES) timeout in nvme over rdma(roce) reconnection, the reason is use the queue which is not alloced.

If queue is not live, should not allow queue request.

Can you describe exactly the scenario here? What is the state here? LIVE? or DELETING?

If seting feature(NVME_FEAT_NUM_QUEUES) failed due to time out or the target return 0 io queues, nvme_set_queue_count will return 0, and then reconnection will continue and success. The state of controller is LIVE. The request will continue to deliver by call ->queue_rq(), and then crash happens.

Quoted 7 lines
Quoted 7 lines

---
  drivers/nvme/host/fabrics.h | 2 +-

  1 file changed, 1 insertion(+), 1 deletion(-)
Quoted 16 lines
Quoted 6 lines
drivers/nvme/host/fabrics.h +0 -0
index 733010d2eafd..2479744fc349 100644
@@ -189,7 +189,7 @@ static inline bool nvmf_check_ready(struct nvme_ctrl *ctrl, struct request *rq,
   {
Quoted 6 lines

      if (likely(ctrl->state == NVME_CTRL_LIVE || ctrl->state == NVME_CTRL_DELETING)) -        return true;
+        return queue_live;
      return __nvmf_check_ready(ctrl, rq, queue_live); }

There were some issues in the past that made us allow submitting
requests in DELETING state and introducing DELETING_NOIO. See
patch ecca390e8056 ("nvme: fix deadlock in disconnect during scan_work and/or ana_work")

This doesn't make any difference. When in deleting state the queue is still live.

Quoted 4 lines

The driver should be able to accept I/O in DELETING because the core changes the state to DELETING_NOIO _before_ it calls ->delete_ctrl so I don't understand how you get to this if the queue is not allocated...

The state of controller is live. The deletion process looks good.

Quoted 1 line

.

_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme

maintainer
#3 lore raw
Quoted 7 lines
Quoted 7 lines
Quoted 6 lines

A crash happens when set feature(NVME_FEAT_NUM_QUEUES) timeout in nvme over rdma(roce) reconnection, the reason is use the queue which is not alloced.

If queue is not live, should not allow queue request.

Quoted 7 lines
Quoted 2 lines

Can you describe exactly the scenario here? What is the state here? LIVE? or DELETING?

If seting feature(NVME_FEAT_NUM_QUEUES) failed due to time out or the target return 0 io queues, nvme_set_queue_count will return 0, and then reconnection will continue and success. The state of controller is LIVE. The request will continue to deliver by call ->queue_rq(), and then crash happens.

Thinking about this again, we should absolutely fail the reconnection when we are unable to set any I/O queues, it is just wrong to keep this controller alive...

This should be fixed for both rdma and tcp.

_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme

author
#4 lore raw
Quoted 15 lines · Sagi Grimberg
Quoted 7 lines
Quoted 7 lines
Quoted 6 lines

A crash happens when set feature(NVME_FEAT_NUM_QUEUES) timeout in nvme over rdma(roce) reconnection, the reason is use the queue which is not alloced.

If queue is not live, should not allow queue request.

Quoted 7 lines
Quoted 2 lines

Can you describe exactly the scenario here? What is the state here? LIVE? or DELETING?

If seting feature(NVME_FEAT_NUM_QUEUES) failed due to time out or the target return 0 io queues, nvme_set_queue_count will return 0, and then reconnection will continue and success. The state of controller is LIVE. The request will continue to deliver by call ->queue_rq(), and then crash happens.

Quoted 3 lines

Thinking about this again, we should absolutely fail the reconnection when we are unable to set any I/O queues, it is just wrong to keep this controller alive...

Keith think keeping the controller alive for diagnose is better. This is the patch which failed the connection.
https://lore.kernel.org/linux-nvme/[email protected]/

Now we have 2 choice:
1.failed the connection when unable to set any I/O queues. 2.do not allow queue request when queue is not live.

From a service continuity perspective, I think it is better that failed the connection when unable to set any I/O queues.
Diagnose is less important, I prefer service continuity, because if failed the reconnection and then try new reconnection, it is possible to recover.

Quoted 3 lines

This should be fixed for both rdma and tcp.
.

_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme

maintainer
#5 lore raw
Quoted 21 lines · Chao Leng
Quoted 15 lines · Sagi Grimberg
Quoted 7 lines
Quoted 7 lines
Quoted 6 lines

A crash happens when set feature(NVME_FEAT_NUM_QUEUES) timeout in nvme over rdma(roce) reconnection, the reason is use the queue which is not alloced.

If queue is not live, should not allow queue request.

Quoted 7 lines
Quoted 2 lines

Can you describe exactly the scenario here? What is the state here? LIVE? or DELETING?

If seting feature(NVME_FEAT_NUM_QUEUES) failed due to time out or the target return 0 io queues, nvme_set_queue_count will return 0, and then reconnection will continue and success. The state of controller is LIVE. The request will continue to deliver by call ->queue_rq(), and then crash happens.

Quoted 3 lines

Thinking about this again, we should absolutely fail the reconnection when we are unable to set any I/O queues, it is just wrong to keep this controller alive...

Quoted 7 lines

Keith think keeping the controller alive for diagnose is better. This is the patch which failed the connection.
https://lore.kernel.org/linux-nvme/[email protected]/

Now we have 2 choice:
1.failed the connection when unable to set any I/O queues. 2.do not allow queue request when queue is not live.

Okay, so there are different views on how to handles this. I personally find in-band administration for a misbehaving device is a good thing to have, but I won't 'nak' if the consensus from the people using this is for the other way.

Quoted 4 lines

From a service continuity perspective, I think it is better that failed the connection when unable to set any I/O queues.
Diagnose is less important, I prefer service continuity, because if failed the reconnection and then try new reconnection, it is possible to recover.

_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme

maintainer
#6 lore raw
Quoted 22 lines
Quoted 22 lines
Quoted 15 lines
Quoted 7 lines
Quoted 7 lines
Quoted 6 lines

A crash happens when set feature(NVME_FEAT_NUM_QUEUES) timeout in nvme over rdma(roce) reconnection, the reason is use the queue which is not alloced.

If queue is not live, should not allow queue request.

Quoted 7 lines
Quoted 2 lines

Can you describe exactly the scenario here? What is the state here? LIVE? or DELETING?

If seting feature(NVME_FEAT_NUM_QUEUES) failed due to time out or the target return 0 io queues, nvme_set_queue_count will return 0, and then reconnection will continue and success. The state of controller is LIVE. The request will continue to deliver by call ->queue_rq(), and then crash happens.

Quoted 3 lines

Thinking about this again, we should absolutely fail the reconnection when we are unable to set any I/O queues, it is just wrong to keep this controller alive...

Keith think keeping the controller alive for diagnose is better. This is the patch which failed the connection.
https://lore.kernel.org/linux-nvme/[email protected]/

Quoted 7 lines
Quoted 3 lines

Now we have 2 choice:
1.failed the connection when unable to set any I/O queues. 2.do not allow queue request when queue is not live.

Okay, so there are different views on how to handles this. I personally find in-band administration for a misbehaving device is a good thing to have, but I won't 'nak' if the consensus from the people using this is for the other way.

While I understand that this can be useful, I've seen it do more harm than good. It is really puzzling to people when the controller state reflected is live (and even optimized) and no I/O is making progress for unknown reason. And logs are rarely accessed in these cases.

I am also opting for failing it and rescheduling a reconnect.

_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme

#7 lore raw
Quoted 31 lines · Sagi Grimberg
Quoted 23 lines
Quoted 23 lines
Quoted 15 lines
Quoted 7 lines
Quoted 7 lines
Quoted 6 lines

A crash happens when set feature(NVME_FEAT_NUM_QUEUES) timeout in nvme over rdma(roce) reconnection, the reason is use the queue which is not alloced.

If queue is not live, should not allow queue request.

Quoted 7 lines
Quoted 2 lines

Can you describe exactly the scenario here? What is the state here? LIVE? or DELETING?

If seting feature(NVME_FEAT_NUM_QUEUES) failed due to time out or the target return 0 io queues, nvme_set_queue_count will return 0, and then reconnection will continue and success. The state of controller is LIVE. The request will continue to deliver by call ->queue_rq(), and then crash happens.

Quoted 3 lines

Thinking about this again, we should absolutely fail the reconnection when we are unable to set any I/O queues, it is just wrong to keep this controller alive...

Keith think keeping the controller alive for diagnose is better. This is the patch which failed the connection.
https://lore.kernel.org/linux-nvme/[email protected]/

Quoted 7 lines
Quoted 3 lines

Now we have 2 choice:
1.failed the connection when unable to set any I/O queues. 2.do not allow queue request when queue is not live.

Okay, so there are different views on how to handles this. I personally find in-band administration for a misbehaving device is a good thing to have, but I won't 'nak' if the consensus from the people using this is for the other way.

Quoted 6 lines

While I understand that this can be useful, I've seen it do more harm than good. It is really puzzling to people when the controller state reflected is live (and even optimized) and no I/O is making progress for unknown reason. And logs are rarely accessed in these cases.

I am also opting for failing it and rescheduling a reconnect.

Agree with Sagi. We also hit this issue a long time ago and I made the same change (commit 834d3710a093a) that Sagi is suggesting: if the prior controller instance had io queues, but the new/reconnected controller fails to create io queues, then the controller create is failed and a reconnect is scheduled.

-- james

_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme

maintainer
#8 lore raw
Quoted 43 lines · James Smart
Quoted 42 lines · Sagi Grimberg
Quoted 30 lines
Quoted 21 lines
Quoted 18 lines
Quoted 9 lines
Quoted 9 lines
Quoted 8 lines

A crash happens when set feature(NVME_FEAT_NUM_QUEUES) timeout in nvme over rdma(roce) reconnection, the reason is use the queue which is not
alloced.

If queue is not live, should not allow queue request.

Quoted 8 lines
Quoted 2 lines

Can you describe exactly the scenario here? What is the state here? LIVE? or DELETING?

If seting feature(NVME_FEAT_NUM_QUEUES) failed due to time out or the target return 0 io queues, nvme_set_queue_count will return 0, and then reconnection will continue and success. The state of controller
is LIVE. The request will continue to deliver by call ->queue_rq(), and then crash happens.

Quoted 3 lines

Thinking about this again, we should absolutely fail the reconnection when we are unable to set any I/O queues, it is just wrong to keep this controller alive...

Quoted 8 lines

Keith think keeping the controller alive for diagnose is better. This is the patch which failed the connection.
https://lore.kernel.org/linux-nvme/[email protected]/

Now we have 2 choice:
1.failed the connection when unable to set any I/O queues. 2.do not allow queue request when queue is not live.

Quoted 6 lines

Okay, so there are different views on how to handles this. I personally find
in-band administration for a misbehaving device is a good thing to have, but I
won't 'nak' if the consensus from the people using this is for the other way.

While I understand that this can be useful, I've seen it do more harm than good. It is really puzzling to people when the controller state reflected is live (and even optimized) and no I/O is making progress for unknown reason. And logs are rarely accessed in these cases.

Quoted 7 lines
Quoted 1 line

I am also opting for failing it and rescheduling a reconnect.

Agree with Sagi. We also hit this issue a long time ago and I made the same change (commit 834d3710a093a) that Sagi is suggesting: if the prior controller instance had io queues, but the new/reconnected controller fails to create io queues, then the controller create is failed and a reconnect is scheduled.

Okay, fair enough.

One more question: if the controller is in such a bad way that it will never create IO queues without additional intervention, will this behavior have the driver schedule reconnect indefinitely?

_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme

maintainer
#9 lore raw
Quoted 25 lines
Quoted 17 lines
Quoted 17 lines
Quoted 11 lines
Quoted 4 lines

Now we have 2 choice:
1.failed the connection when unable to set any I/O queues. 2.do not allow queue request when queue is not live.

Okay, so there are different views on how to handles this. I personally find
in-band administration for a misbehaving device is a good thing to have, but I
won't 'nak' if the consensus from the people using this is for the other way.

While I understand that this can be useful, I've seen it do more harm than good. It is really puzzling to people when the controller state reflected is live (and even optimized) and no I/O is making progress for unknown reason. And logs are rarely accessed in these cases.

Quoted 7 lines
Quoted 1 line

I am also opting for failing it and rescheduling a reconnect.

Agree with Sagi. We also hit this issue a long time ago and I made the same change (commit 834d3710a093a) that Sagi is suggesting: if the prior controller instance had io queues, but the new/reconnected controller fails to create io queues, then the controller create is failed and a reconnect is scheduled.

Quoted 5 lines

Okay, fair enough.

One more question: if the controller is in such a bad way that it will never create IO queues without additional intervention, will this behavior have the driver schedule reconnect indefinitely?

Until either ctrl_loss_tmo expires or the user is tired of this controller and manually disconnects.

_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme

#10 lore raw
In reply to Keith Busch
Quoted 6 lines

One more question: if the controller is in such a bad way that it will never create IO queues without additional intervention, will this behavior have the driver schedule reconnect indefinitely?

No.. in fabrics, there is a max controller reconnect timeout that will expire (dflt 600s), stop the reconnect attempts and delete the controller.

-- james

_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme