Skip to contentSkip to conversation

nvme-rdma: fix crash for no IO queues

PATCH In review Chao Leng 4 participants 15 messages Feb 23, 2021 – Mar 3, 2021 NVMe

lore
Share & tools
lore

Chao Leng

author
#0 lore raw

PATCH 1/1 nvme-rdma: fix crash for no IO queues

A crash happens when set feature(NVME_FEAT_NUM_QUEUES) timeout in nvme over rdma(roce) reconnection, the reason is use the queue which is not alloced.

If it is not discovery and no io queues, the connection should fail.

---
 drivers/nvme/host/rdma.c | 7 +++++--
 1 file changed, 5 insertions(+), 2 deletions(-)
drivers/nvme/host/rdma.c +5 -2
index 53ac4d7442ba..7410972353ae 100644
@@ -736,8 +736,11 @@ static int nvme_rdma_alloc_io_queues(struct nvme_rdma_ctrl *ctrl)
 		return ret;
 
 	ctrl->ctrl.queue_count = nr_io_queues + 1;
-	if (ctrl->ctrl.queue_count < 2)
-		return 0;
+	if (ctrl->ctrl.queue_count < 2) {
+		if (opts->discovery_nqn)
+			return 0;
+		return -EAGAIN;
+	}
 
 	dev_info(ctrl->ctrl.device,
 		"creating %d I/O queues.\n", nr_io_queues);
Signature · 7 lines
2.16.4


_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme
#1 lore raw
Quoted 7 lines · Chao Leng

A crash happens when set feature(NVME_FEAT_NUM_QUEUES) timeout in nvme over rdma(roce) reconnection, the reason is use the queue which is not alloced.

If it is not discovery and no io queues, the connection should fail.

Can you please share more information about

"when set feature(NVME_FEAT_NUM_QUEUES) timeout" scenario ?

_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme

author
#2 lore raw
Quoted 5 lines · Chaitanya Kulkarni
Quoted 4 lines · Chao Leng

A crash happens when set feature(NVME_FEAT_NUM_QUEUES) timeout in nvme over rdma(roce) reconnection, the reason is use the queue which is not alloced.

Quoted 7 lines
Quoted 3 lines

If it is not discovery and no io queues, the connection should fail.

Can you please share more information about

"when set feature(NVME_FEAT_NUM_QUEUES) timeout" scenario ?

Inject a large number of bit errors intermittently. This will cause request time out, and then reconnection will trigger by error recovery. The requests of reconnect may also time out. If set feature (NVME_FEAT_NUM_QUEUES, which is called by nvme_set_queue_count) time out and the reconnection may success. Block will continue send request, and then crash due to use unallocated queue.

_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme

maintainer
#3 lore raw
In reply to Chao Leng
Quoted 5 lines · Chao Leng

A crash happens when set feature(NVME_FEAT_NUM_QUEUES) timeout in nvme over rdma(roce) reconnection, the reason is use the queue which is not alloced.

If it is not discovery and no io queues, the connection should fail.

If you're getting a timeout, we need to quit initialization. Hannes attempted making that status visible for fabrics here:

http://lists.infradead.org/pipermail/linux-nvme/2021-January/022353.html

There's still some corner cases that need handling, though.

_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme

author
#4 lore raw
Quoted 11 lines · Keith Busch
Quoted 5 lines · Chao Leng

A crash happens when set feature(NVME_FEAT_NUM_QUEUES) timeout in nvme over rdma(roce) reconnection, the reason is use the queue which is not alloced.

If it is not discovery and no io queues, the connection should fail.

If you're getting a timeout, we need to quit initialization. Hannes attempted making that status visible for fabrics here:

   http://lists.infradead.org/pipermail/linux-nvme/2021-January/022353.html

I know the patch. It can not solve the scenario: target may be an attacker or the target behavior is incorrect.
If target return 0 io queues or return other error code, the crash will still happen. We should not allow this to happen.

Quoted 4 lines

There's still some corner cases that need handling, though. .

_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme

#5 lore raw
Quoted 14 lines · Chao Leng
Quoted 11 lines · Keith Busch
Quoted 5 lines · Chao Leng

A crash happens when set feature(NVME_FEAT_NUM_QUEUES) timeout in nvme over rdma(roce) reconnection, the reason is use the queue which is not alloced.

If it is not discovery and no io queues, the connection should fail.

If you're getting a timeout, we need to quit initialization. Hannes attempted making that status visible for fabrics here:

Quoted 5 lines
Quoted 1 line

http://lists.infradead.org/pipermail/linux-nvme/2021-January/022353.html

I know the patch. It can not solve the scenario: target may be an attacker or the target behavior is incorrect.
If target return 0 io queues or return other error code, the crash will still happen. We should not allow this to happen.

I'm fully with you that we shouldn't crash, but at the same time a value of '0' for the number of I/O queues is considered valid. So we should fix the code to handle this scenario, and not disallowing zero I/O queues.

Cheers,

Hannes

Signature · 9 lines
Dr. Hannes Reinecke                Kernel Storage Architect
[email protected]                              +49 911 74053 688
SUSE Software Solutions GmbH, Maxfeldstr. 5, 90409 Nürnberg
HRB 36809 (AG Nürnberg), Geschäftsführer: Felix Imendörffer

_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme
author
#6 lore raw
Quoted 14 lines · Hannes Reinecke
Quoted 13 lines · Chao Leng
Quoted 10 lines · Keith Busch
Quoted 5 lines · Chao Leng

A crash happens when set feature(NVME_FEAT_NUM_QUEUES) timeout in nvme over rdma(roce) reconnection, the reason is use the queue which is not alloced.

If it is not discovery and no io queues, the connection should fail.

If you're getting a timeout, we need to quit initialization. Hannes attempted making that status visible for fabrics here:

Quoted 7 lines
Quoted 5 lines
Quoted 1 line

http://lists.infradead.org/pipermail/linux-nvme/2021-January/022353.html

I know the patch. It can not solve the scenario: target may be an attacker or the target behavior is incorrect.
If target return 0 io queues or return other error code, the crash will still happen. We should not allow this to happen.

I'm fully with you that we shouldn't crash, but at the same time a value of '0' for the number of I/O queues is considered valid. So we should fix the code to handle this scenario, and not disallowing zero I/O queues.

'0' I/O queues doesn't make any sense to nvme over fabrics, it is different with nvme over pci. If there is some bug with target, we can debug it in target instead of use admin queue in host. target may be an attacker or the target behavior is incorrect. So we should avoid crash. Another option: prohibit request delivery if io queue do not created.
I think failed connection with '0' I/O queues is a better choice.

Quoted 4 lines

Cheers,

Hannes

_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme

#7 lore raw
Quoted 27 lines · Chao Leng
Quoted 16 lines · Hannes Reinecke
Quoted 15 lines · Chao Leng
Quoted 12 lines · Keith Busch
Quoted 5 lines · Chao Leng

A crash happens when set feature(NVME_FEAT_NUM_QUEUES) timeout in nvme over rdma(roce) reconnection, the reason is use the queue which is not alloced.

If it is not discovery and no io queues, the connection should fail.

If you're getting a timeout, we need to quit initialization. Hannes attempted making that status visible for fabrics here:

http://lists.infradead.org/pipermail/linux-nvme/2021-January/022353.html

Quoted 8 lines
Quoted 4 lines

I know the patch. It can not solve the scenario: target may be an attacker or the target behavior is incorrect.
If target return 0 io queues or return other error code, the crash will still happen. We should not allow this to happen.

I'm fully with you that we shouldn't crash, but at the same time a value of '0' for the number of I/O queues is considered valid. So we should fix the code to handle this scenario, and not disallowing zero I/O queues.

Quoted 7 lines

'0' I/O queues doesn't make any sense to nvme over fabrics, it is different with nvme over pci. If there is some bug with target, we can debug it in target instead of use admin queue in host. target may be an attacker or the target behavior is incorrect. So we should avoid crash. Another option: prohibit  request delivery if io queue do not created.
I think failed connection with '0' I/O queues is a better choice.

Might be, but that's not for me to decide.
I tried that initially, but that patch got rejected as _technically_ the controller is reachable via its admin queue.
Sagi? Christoph?
Are controllers with 0 I/O queues valid or is this an error condition?

Cheers,

Hannes

Signature · 9 lines
Dr. Hannes Reinecke		           Kernel Storage Architect
[email protected]			                  +49 911 74053 688
SUSE Software Solutions Germany GmbH, Maxfeldstr. 5, 90409 Nürnberg
HRB 36809 (AG Nürnberg), GF: Felix Imendörffer

_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme
author
#8 lore raw
Quoted 36 lines · Hannes Reinecke
Quoted 27 lines · Chao Leng
Quoted 16 lines · Hannes Reinecke
Quoted 15 lines · Chao Leng
Quoted 12 lines · Keith Busch
Quoted 5 lines · Chao Leng

A crash happens when set feature(NVME_FEAT_NUM_QUEUES) timeout in nvme over rdma(roce) reconnection, the reason is use the queue which is not alloced.

If it is not discovery and no io queues, the connection should fail.

If you're getting a timeout, we need to quit initialization. Hannes attempted making that status visible for fabrics here:

http://lists.infradead.org/pipermail/linux-nvme/2021-January/022353.html

Quoted 8 lines
Quoted 4 lines

I know the patch. It can not solve the scenario: target may be an attacker or the target behavior is incorrect.
If target return 0 io queues or return other error code, the crash will still happen. We should not allow this to happen.

I'm fully with you that we shouldn't crash, but at the same time a value of '0' for the number of I/O queues is considered valid. So we should fix the code to handle this scenario, and not disallowing zero I/O queues.

Quoted 7 lines

'0' I/O queues doesn't make any sense to nvme over fabrics, it is different with nvme over pci. If there is some bug with target, we can debug it in target instead of use admin queue in host. target may be an attacker or the target behavior is incorrect. So we should avoid crash. Another option: prohibit  request delivery if io queue do not created.
I think failed connection with '0' I/O queues is a better choice.

Quoted 3 lines

Might be, but that's not for me to decide.
I tried that initially, but that patch got rejected as _technically_ the controller is reachable via its admin queue.

I know about your patch. That patch failed connection for all transports. It is not good for pcie transport, the controller can accept admin commands to get some diagnostics (perhaps an error log page), this is keith's thoughts.

Quoted 7 lines

Sagi? Christoph?
Are controllers with 0 I/O queues valid or is this an error condition?

Cheers,

Hannes

_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme

maintainer
#9 lore raw
Quoted 39 lines · Chao Leng
Quoted 36 lines · Hannes Reinecke
Quoted 34 lines · Chao Leng
Quoted 16 lines · Hannes Reinecke
Quoted 15 lines · Chao Leng
Quoted 12 lines · Keith Busch
Quoted 5 lines · Chao Leng

A crash happens when set feature(NVME_FEAT_NUM_QUEUES) timeout in nvme over rdma(roce) reconnection, the reason is use the queue which is not alloced.

If it is not discovery and no io queues, the connection should fail.

If you're getting a timeout, we need to quit initialization. Hannes attempted making that status visible for fabrics here:

http://lists.infradead.org/pipermail/linux-nvme/2021-January/022353.html

Quoted 8 lines
Quoted 4 lines

I know the patch. It can not solve the scenario: target may be an attacker or the target behavior is incorrect.
If target return 0 io queues or return other error code, the crash will still happen. We should not allow this to happen.

I'm fully with you that we shouldn't crash, but at the same time a value of '0' for the number of I/O queues is considered valid. So we should fix the code to handle this scenario, and not disallowing zero I/O queues.

'0' I/O queues doesn't make any sense to nvme over fabrics, it is different with nvme over pci. If there is some bug with target, we can debug it in target instead of use admin queue in host. target may be an attacker or the target behavior is incorrect. So we should avoid crash. Another option: prohibit  request delivery if io queue do not created.
I think failed connection with '0' I/O queues is a better choice.

Quoted 7 lines
Quoted 3 lines

Might be, but that's not for me to decide.
I tried that initially, but that patch got rejected as _technically_ the controller is reachable via its admin queue.

I know about your patch. That patch failed connection for all transports. It is not good for pcie transport, the controller can accept admin commands to get some diagnostics (perhaps an error log page), this is keith's thoughts.

We can continue to administrate a controller that didn't create IO queues, but the controller must provide a response to all commands. If it doesn't, the controller will either be reset or abandoned. This should be the same behavior for any transport, though; there's nothing special about PCIe for that.

_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme

author
#10 lore raw
Quoted 48 lines · Keith Busch
Quoted 39 lines · Chao Leng
Quoted 36 lines · Hannes Reinecke
Quoted 34 lines · Chao Leng
Quoted 16 lines · Hannes Reinecke
Quoted 15 lines · Chao Leng
Quoted 12 lines · Keith Busch
Quoted 5 lines · Chao Leng

A crash happens when set feature(NVME_FEAT_NUM_QUEUES) timeout in nvme over rdma(roce) reconnection, the reason is use the queue which is not alloced.

If it is not discovery and no io queues, the connection should fail.

If you're getting a timeout, we need to quit initialization. Hannes attempted making that status visible for fabrics here:

http://lists.infradead.org/pipermail/linux-nvme/2021-January/022353.html

Quoted 8 lines
Quoted 4 lines

I know the patch. It can not solve the scenario: target may be an attacker or the target behavior is incorrect.
If target return 0 io queues or return other error code, the crash will still happen. We should not allow this to happen.

I'm fully with you that we shouldn't crash, but at the same time a value of '0' for the number of I/O queues is considered valid. So we should fix the code to handle this scenario, and not disallowing zero I/O queues.

'0' I/O queues doesn't make any sense to nvme over fabrics, it is different with nvme over pci. If there is some bug with target, we can debug it in target instead of use admin queue in host. target may be an attacker or the target behavior is incorrect. So we should avoid crash. Another option: prohibit  request delivery if io queue do not created.
I think failed connection with '0' I/O queues is a better choice.

Quoted 7 lines
Quoted 3 lines

Might be, but that's not for me to decide.
I tried that initially, but that patch got rejected as _technically_ the controller is reachable via its admin queue.

I know about your patch. That patch failed connection for all transports. It is not good for pcie transport, the controller can accept admin commands to get some diagnostics (perhaps an error log page), this is keith's thoughts.

Quoted 5 lines

We can continue to administrate a controller that didn't create IO queues, but the controller must provide a response to all commands. If it doesn't, the controller will either be reset or abandoned. This should be the same behavior for any transport, though; there's nothing special about PCIe for that.

Though I don't see any useful scenarios for nvme over fabric now, Reserved for future possibilities may be a better choice.
I will send another patch. Prohibit request delivery if the queue is not live.

Quoted 2 lines

.

_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme

maintainer
#11 lore raw
Quoted 8 lines · Chao Leng
Quoted 5 lines · Keith Busch

We can continue to administrate a controller that didn't create IO queues, but the controller must provide a response to all commands. If it doesn't, the controller will either be reset or abandoned. This should be the same behavior for any transport, though; there's nothing special about PCIe for that.

Though I don't see any useful scenarios for nvme over fabric now, Reserved for future possibilities may be a better choice.

The admin queue may be the only way for a user to retrieve useful information on the malfunctioning controller. The telemetry log isn't unique to PCIe.

_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme

author
#12 lore raw
Quoted 7 lines · Keith Busch
Quoted 6 lines · Chao Leng
Quoted 5 lines · Keith Busch

We can continue to administrate a controller that didn't create IO queues, but the controller must provide a response to all commands. If it doesn't, the controller will either be reset or abandoned. This should be the same behavior for any transport, though; there's nothing special about PCIe for that.

Quoted 6 lines
Quoted 2 lines

Though I don't see any useful scenarios for nvme over fabric now, Reserved for future possibilities may be a better choice.

The admin queue may be the only way for a user to retrieve useful information on the malfunctioning controller. The telemetry log isn't unique to PCIe.

User can also directly visit target to get these information for nvme over fabrics.

Quoted 2 lines

.

_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme

#13 lore raw
Quoted 10 lines · Chao Leng
Quoted 7 lines · Keith Busch
Quoted 6 lines · Chao Leng
Quoted 5 lines · Keith Busch

We can continue to administrate a controller that didn't create IO queues, but the controller must provide a response to all commands. If it doesn't, the controller will either be reset or abandoned. This should be the same behavior for any transport, though; there's nothing special about PCIe for that.

Quoted 8 lines
Quoted 6 lines
Quoted 2 lines

Though I don't see any useful scenarios for nvme over fabric now, Reserved for future possibilities may be a better choice.

The admin queue may be the only way for a user to retrieve useful information on the malfunctioning controller. The telemetry log isn't unique to PCIe.

User can also directly visit target to get these information for nvme over fabrics.

Not necessarily. There is no requirement that the storage admin is identical to the host admin; in fact, in larger installations both roles are strictly separated.

Cheers,

Hannes

Signature · 9 lines
Dr. Hannes Reinecke		           Kernel Storage Architect
[email protected]			                  +49 911 74053 688
SUSE Software Solutions Germany GmbH, Maxfeldstr. 5, 90409 Nürnberg
HRB 36809 (AG Nürnberg), GF: Felix Imendörffer

_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme
maintainer
#14 lore raw
Quoted 17 lines · Hannes Reinecke
Quoted 16 lines · Chao Leng
Quoted 13 lines · Keith Busch
Quoted 8 lines · Chao Leng
Quoted 5 lines · Keith Busch

We can continue to administrate a controller that didn't create IO queues, but the controller must provide a response to all commands. If it doesn't, the controller will either be reset or abandoned. This should be the same behavior for any transport, though; there's nothing special about PCIe for that.

Though I don't see any useful scenarios for nvme over fabric now, Reserved for future possibilities may be a better choice.

The admin queue may be the only way for a user to retrieve useful information on the malfunctioning controller. The telemetry log isn't unique to PCIe.

Quoted 6 lines
Quoted 2 lines

User can also directly visit target to get these information for nvme over fabrics.

Not necessarily. There is no requirement that the storage admin is identical to the host admin; in fact, in larger installations both roles are strictly separated.

There's also no requirement that nvme targets providing anything but in-band management capabilities, so there is no justification for relying on a vendor specific method.

_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme