Skip to contentSkip to conversation

nvme: Add support for ACRE Command Interrupted status

PATCH In review Hannes Reinecke 5 participants 9 messages Nov 26, 2019 – Nov 27, 2019

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

PATCH 1/1 nvme: Add support for ACRE Command Interrupted status

From: John Meneghini <[email protected]>

This patch fixes a bug in nvme_complete_rq logic introduced by Enhanced Command Retry code. According to TP-4033 the controller only sets CDR when the Command Interrupted status is returned. The current code interprets Command Interrupted status as a BLK_STS_IOERR, which results in a controller reset if REQ_NVME_MPATH is set.

---
 drivers/nvme/host/core.c  | 2 ++
 include/linux/blk_types.h | 1 +
 include/linux/nvme.h      | 2 ++
 3 files changed, 5 insertions(+)
drivers/nvme/host/core.c +2 -0
index 108f60b46804..752a40daf2b3 100644
@@ -201,6 +201,8 @@ static blk_status_t nvme_error_status(u16 status)
 	switch (status & 0x7ff) {
 	case NVME_SC_SUCCESS:
 		return BLK_STS_OK;
+	case NVME_SC_COMMAND_INTERRUPTED:
+		return BLK_STS_RESOURCE;
 	case NVME_SC_CAP_EXCEEDED:
 		return BLK_STS_NOSPC;
 	case NVME_SC_LBA_RANGE:
include/linux/blk_types.h +1 -0
index d688b96d1d63..3a0d84528325 100644
@@ -84,6 +84,7 @@ static inline bool blk_path_error(blk_status_t error)
 	case BLK_STS_NEXUS:
 	case BLK_STS_MEDIUM:
 	case BLK_STS_PROTECTION:
+	case BLK_STS_RESOURCE:
 		return false;
 	}
 
include/linux/nvme.h +2 -0
index f61d6906e59d..6b21f3888cad 100644
@@ -1292,6 +1292,8 @@ enum {
 
 	NVME_SC_NS_WRITE_PROTECTED	= 0x20,
 
+	NVME_SC_COMMAND_INTERRUPTED	= 0x21,
+
 	NVME_SC_LBA_RANGE		= 0x80,
 	NVME_SC_CAP_EXCEEDED		= 0x81,
 	NVME_SC_NS_NOT_READY		= 0x82,
Signature · 7 lines
2.16.4


_______________________________________________
linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme
Reviewed-by Martin George on v1 1/1
#1 lore raw
Quoted 57 lines · Hannes Reinecke

From: John Meneghini <[email protected]>

This patch fixes a bug in nvme_complete_rq logic introduced by Enhanced Command Retry code. According to TP-4033 the controller only sets CDR when the Command Interrupted status is returned. The current code interprets Command Interrupted status as a BLK_STS_IOERR, which results in a controller reset if REQ_NVME_MPATH is set.

---
 drivers/nvme/host/core.c  | 2 ++
 include/linux/blk_types.h | 1 +
 include/linux/nvme.h      | 2 ++
 3 files changed, 5 insertions(+)
drivers/nvme/host/core.c +2 -0
index 108f60b46804..752a40daf2b3 100644
@@ -201,6 +201,8 @@ static blk_status_t nvme_error_status(u16 status)
        switch (status & 0x7ff) {
        case NVME_SC_SUCCESS:
                return BLK_STS_OK;
+       case NVME_SC_COMMAND_INTERRUPTED:
+               return BLK_STS_RESOURCE;
        case NVME_SC_CAP_EXCEEDED:
                return BLK_STS_NOSPC;
        case NVME_SC_LBA_RANGE:
include/linux/blk_types.h +1 -0
index d688b96d1d63..3a0d84528325 100644
@@ -84,6 +84,7 @@ static inline bool blk_path_error(blk_status_t
 error)
        case BLK_STS_NEXUS:
        case BLK_STS_MEDIUM:
        case BLK_STS_PROTECTION:
+       case BLK_STS_RESOURCE:
                return false;
        }
include/linux/nvme.h +2 -1
index f61d6906e59d..6b21f3888cad 100644
@@ -1292,6 +1292,8 @@ enum {
 
        NVME_SC_NS_WRITE_PROTECTED      = 0x20,
 
+       NVME_SC_COMMAND_INTERRUPTED     = 0x21,
+
        NVME_SC_LBA_RANGE               = 0x80,
        NVME_SC_CAP_EXCEEDED            = 0x81,
        NVME_SC_NS_NOT_READY            = 0x82,
--
Quoted 2 lines

2.16.4

Looks good.

_______________________________________________
linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme

maintainer
#2 lore raw
In reply to Hannes Reinecke

[cc'ing linux-block, Jens]

Quoted 6 lines · Hannes Reinecke

This patch fixes a bug in nvme_complete_rq logic introduced by Enhanced Command Retry code. According to TP-4033 the controller only sets CDR when the Command Interrupted status is returned. The current code interprets Command Interrupted status as a BLK_STS_IOERR, which results in a controller reset if REQ_NVME_MPATH is set.

I see that Command Interrupted status requires ACRE enabled, but I don't see the TP saying that the CQE CRD fields are used only with that status code. I'm pretty sure I've seen it used for Namespace Not Ready status as well. That would also fail MPATH for the same reason as this new status...

Quoted 18 lines
drivers/nvme/host/core.c +2 -0
index 108f60b46804..752a40daf2b3 100644
@@ -201,6 +201,8 @@ static blk_status_t nvme_error_status(u16 status)
 	switch (status & 0x7ff) {
 	case NVME_SC_SUCCESS:
 		return BLK_STS_OK;
+	case NVME_SC_COMMAND_INTERRUPTED:
+		return BLK_STS_RESOURCE;
 	case NVME_SC_CAP_EXCEEDED:
 		return BLK_STS_NOSPC;
 	case NVME_SC_LBA_RANGE:
include/linux/blk_types.h +0 -0
index d688b96d1d63..3a0d84528325 100644

include/linux/blk_types.h static inline bool blk_path_error(blk_status_t error) · line 89

Quoted 6 lines
include/linux/blk_types.h +1 -0
@@ -84,6 +84,7 @@ static inline bool blk_path_error(blk_status_t error)
 	case BLK_STS_NEXUS:
 	case BLK_STS_MEDIUM:
 	case BLK_STS_PROTECTION:
+	case BLK_STS_RESOURCE:
 		return false;
 	}

I agree we need to make this status a non-path error, but we at least need an Ack from Jens or have this patch go through linux-block if we're changing BLK_STS_RESOURCE to mean a non-path error.

include/linux/nvme.h enum { · line 1295

Quoted 9 lines
include/linux/nvme.h +1 -0
index f61d6906e59d..6b21f3888cad 100644
@@ -1292,6 +1292,8 @@ enum {
 
 	NVME_SC_NS_WRITE_PROTECTED	= 0x20,
 
+	NVME_SC_COMMAND_INTERRUPTED	= 0x21,

This command status was actually already defined in commit 48c9e85b23464, though with a slightly different name.

#3 lore raw
Quoted 4 lines · Keith Busch
Quoted 4 lines

+++ b/include/linux/blk_types.h

include/linux/blk_types.h +0 -0
@@ -84,6 +84,7 @@ static inline bool blk_path_error(blk_status_t error)
 	case BLK_STS_NEXUS:
 	case BLK_STS_MEDIUM:
Quoted 8 lines

include/linux/blk_types.h static inline bool blk_path_error(blk_status_t error) · line 89

Quoted 4 lines
include/linux/blk_types.h +1 -0
 	case BLK_STS_PROTECTION:
+	case BLK_STS_RESOURCE:
 		return false;
 	}

I agree we need to make this status a non-path error, but we at least need an Ack from Jens or have this patch go through linux-block if we're changing BLK_STS_RESOURCE to mean a non-path error.

most resource errors are per-path, so blindly changing this is wrong.

That's why I really just wanted to decode the nvme status codes inside nvme instead of going through a block layer mapping, as that is just bound to lose some information.

#4 lore raw

On 11/26/19, 11:24 AM, "Christoph Hellwig" <[email protected]> wrote:

On Wed, Nov 27, 2019 at 01:05:46AM +0900, Keith Busch wrote:
> > +++ b/include/linux/blk_types.h
> > @@ -84,6 +84,7 @@ static inline bool blk_path_error(blk_status_t error)
> > case BLK_STS_NEXUS:
> > case BLK_STS_MEDIUM:
> > case BLK_STS_PROTECTION:
> > + case BLK_STS_RESOURCE:
> > return false;
> > }
>
> I agree we need to make this status a non-path error, but we at least > need an Ack from Jens or have this patch go through linux-block if we're > changing BLK_STS_RESOURCE to mean a non-path error.

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

most resource errors are per-path, so blindly changing this is wrong.

That's why I really just wanted to decode the nvme status codes inside
nvme instead of going through a block layer mapping, as that is just
bound to lose some information.

It wasn't my intention to turn NVME_SC_CMD_INTERRUPTED into non-path related error. The goal was to make the command retry after CDR on the same controller. So why does this patch change the meaning of BLK_STS_RESOURCE?

/John

maintainer
#5 lore raw
Quoted 7 lines · John
Quoted 5 lines · Christoph Hellwig

I agree we need to make this status a non-path error, but we at least need an Ack from Jens or have this patch go through linux-block if we're changing BLK_STS_RESOURCE to mean a non-path error.

Quoted 1 line
Quoted 1 line

most resource errors are per-path, so blindly changing this is wrong.

Quoted 7 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

That's why I really just wanted to decode the nvme status codes inside
nvme instead of going through a block layer mapping, as that is just
bound to lose some information.

It wasn't my intention to turn NVME_SC_CMD_INTERRUPTED into non-path related error. The goal was to make the command retry after CDR on the same controller. So why does this patch change the meaning of BLK_STS_RESOURCE?

What I meant by a "non-path error" is that those types of errors for nvme go through the "normal" requeuing that checks CRD. The failover requeuing does not check CRD. But thinking again, I don't see why the failover side can't also be CRD aware.

#6 lore raw
Quoted 8 lines · Keith Busch
Quoted 3 lines

It wasn't my intention to turn NVME_SC_CMD_INTERRUPTED into non-path related error. The goal was to make the command retry after CDR on the same controller. So why does this patch change the meaning of BLK_STS_RESOURCE?

What I meant by a "non-path error" is that those types of errors for nvme go through the "normal" requeuing that checks CRD. The failover requeuing does not check CRD. But thinking again, I don't see why the failover side can't also be CRD aware.

Yes, the failover side can be CRD aware, but I don't see the point in failing over to a different path when CRD is set. As you said, this isn't a pathing error.

We can argue about whether the command is retried on the current controller or a different controller, but CRD must be observed before the command is retried.

Agreed?

/John

#7 lore raw
In reply to Keith Busch
Quoted 10 lines · Keith Busch

[cc'ing linux-block, Jens]

Quoted 6 lines · Hannes Reinecke

This patch fixes a bug in nvme_complete_rq logic introduced by Enhanced Command Retry code. According to TP-4033 the controller only sets CDR when the Command Interrupted status is returned. The current code interprets Command Interrupted status as a BLK_STS_IOERR, which results in a controller reset if REQ_NVME_MPATH is set.

Quoted 6 lines

I see that Command Interrupted status requires ACRE enabled, but I don't see the TP saying that the CQE CRD fields are used only with that status code. I'm pretty sure I've seen it used for Namespace Not Ready status as well. That would also fail MPATH for the same reason as this new status...

No, true, CRD is not directly related to 'command interrupted'.
According to the powers that be CRD evaluation is depending on the ACRE setting (and hence should be evaluated whenever a command needs to be retried), but 'command interrupted' will only be returned if ACRE is enabled. Consequently, whenever you get a 'command interrupted' you need to check the CRD setting to figure out the delay.

Quoted 20 lines
Quoted 20 lines
drivers/nvme/host/core.c +2 -0
index 108f60b46804..752a40daf2b3 100644
@@ -201,6 +201,8 @@ static blk_status_t nvme_error_status(u16 status)
  	switch (status & 0x7ff) {
  	case NVME_SC_SUCCESS:
  		return BLK_STS_OK;
+	case NVME_SC_COMMAND_INTERRUPTED:
+		return BLK_STS_RESOURCE;
  	case NVME_SC_CAP_EXCEEDED:
  		return BLK_STS_NOSPC;
  	case NVME_SC_LBA_RANGE:
include/linux/blk_types.h +0 -0
index d688b96d1d63..3a0d84528325 100644
@@ -84,6 +84,7 @@ static inline bool blk_path_error(blk_status_t error)
  	case BLK_STS_NEXUS:
  	case BLK_STS_MEDIUM:
Quoted 9 lines

include/linux/blk_types.h static inline bool blk_path_error(blk_status_t error) · line 89

Quoted 4 lines
include/linux/blk_types.h +1 -0
  	case BLK_STS_PROTECTION:
+	case BLK_STS_RESOURCE:
  		return false;
  	}

I agree we need to make this status a non-path error, but we at least need an Ack from Jens or have this patch go through linux-block if we're changing BLK_STS_RESOURCE to mean a non-path error.

Alternatively we can define a new return value, if we shouldn't re-use existing ones. Either way I'm fine with.

Quoted 6 lines
Quoted 6 lines
include/linux/nvme.h +0 -0
index f61d6906e59d..6b21f3888cad 100644
@@ -1292,6 +1292,8 @@ enum {
 
Quoted 7 lines

include/linux/nvme.h enum { · line 1295

Quoted 3 lines
include/linux/nvme.h +1 -0
  	NVME_SC_NS_WRITE_PROTECTED	= 0x20,
 
+	NVME_SC_COMMAND_INTERRUPTED	= 0x21,

This command status was actually already defined in commit 48c9e85b23464, though with a slightly different name.

Ah. Will be modifying the patch then.

Cheers,

Hannes

Signature · 5 lines
Dr. Hannes Reinecke            Teamlead Storage & Networking
[email protected]                              +49 911 74053 688
SUSE LINUX GmbH, Maxfeldstr. 5, 90409 Nürnberg
GF: Felix Imendörffer, Mary Higgins, Sri Rasiah
HRB 21284 (AG Nürnberg)
#8 lore raw

On 11/26/19, 11:32 AM, "Hannes Reinecke" <[email protected]> wrote:
On 11/26/19 5:05 PM, Keith Busch wrote:
> [cc'ing linux-block, Jens]
>
> On Tue, Nov 26, 2019 at 02:36:50PM +0100, Hannes Reinecke wrote:
>> This patch fixes a bug in nvme_complete_rq logic introduced by >> Enhanced Command Retry code. According to TP-4033 the controller >> only sets CDR when the Command Interrupted status is returned. >> The current code interprets Command Interrupted status as a >> BLK_STS_IOERR, which results in a controller reset if
>> REQ_NVME_MPATH is set.
>
> I see that Command Interrupted status requires ACRE enabled, but I don't > see the TP saying that the CQE CRD fields are used only with that status > code. I'm pretty sure I've seen it used for Namespace Not Ready status > as well. That would also fail MPATH for the same reason as this new > status...
>

> No, true, CRD is not directly related to 'command interrupted'. > According to the powers that be CRD evaluation is depending on the ACRE > setting (and hence should be evaluated whenever a command needs to be > retried), but 'command interrupted' will only be returned if ACRE is > enabled.

Yes, we had a long debate about this with Fred. CRD does not require the Command Interrupted status, but Command Interrupted requires CRD.

Quoted 2 lines

Consequently, whenever you get a 'command interrupted' you need to check the CRD setting to figure out the delay.

Whenever _any_ CQE sets CRD and clears DNR the host is required to implement the delay. The requirement is on the controller to always set CRD when Command Interrupted status is returned.

Also, the host is required to handle Command Interrupted if it enables ACRE.

But then that's the bug....

Quoted 29 lines
Quoted 18 lines
drivers/nvme/host/core.c +2 -0
index 108f60b46804..752a40daf2b3 100644
@@ -201,6 +201,8 @@ static blk_status_t nvme_error_status(u16 status)
     switch (status & 0x7ff) {
     case NVME_SC_SUCCESS:
             return BLK_STS_OK;
+    case NVME_SC_COMMAND_INTERRUPTED:
+            return BLK_STS_RESOURCE;
     case NVME_SC_CAP_EXCEEDED:
             return BLK_STS_NOSPC;
     case NVME_SC_LBA_RANGE:
include/linux/blk_types.h +0 -0
index d688b96d1d63..3a0d84528325 100644

include/linux/blk_types.h static inline bool blk_path_error(blk_status_t error) · line 89

Quoted 6 lines
include/linux/blk_types.h +1 -0
@@ -84,6 +84,7 @@ static inline bool blk_path_error(blk_status_t error)
     case BLK_STS_NEXUS:
     case BLK_STS_MEDIUM:
     case BLK_STS_PROTECTION:
+    case BLK_STS_RESOURCE:
             return false;
     }

I agree we need to make this status a non-path error, but we at least need an Ack from Jens or have this patch go through linux-block if we're changing BLK_STS_RESOURCE to mean a non-path error.

Quoted 2 lines

Alternatively we can define a new return value, if we shouldn't re-use existing ones. Either way I'm fine with.

It seems to me that the block layer is never going to see BLK_STS_RESOURCE unless nvme_req(req)->retries run out... and at that point it doesn't matter, does it?

What's the result if the controller returns Command Interrupted status for the same request repeatedly and we returning BLK_STS_RESOURCE after nvme_max_retries?

/John