Skip to contentSkip to conversation

nvme-core: mark passthru requests RQF_QUIET flag

PATCH In review Chaitanya Kulkarni 6 participants 23 messages Apr 6, 2022 – Apr 11, 2022

Subsystems (2)
lore
Share & tools
lore

Chaitanya Kulkarni

author maintainer
#0 lore raw

PATCH 1/1 nvme-core: mark passthru requests RQF_QUIET flag

From: Christoph Hellwig <[email protected]>

Since addition of the nvme_log_error() we are getting error message when running block tests framework due to internal passthru commands :-

[  612.754938] nvme nvme1: Removing ctrl: NQN "blktests-subsystem-1"
[  616.361730] run blktests nvme/012 at 2022-04-06 09:26:43
[  616.382902] loop0: detected capacity change from 0 to 2097152
[  616.392680] nvmet: adding nsid 1 to subsystem blktests-subsystem-1
[  616.400913] nvmet: creating nvm controller 1 for subsystem blktests-subsystem-1 for NQN testhostnqn.
[  616.401001] nvme1: Identify(0x6), Invalid Field in Command (sct 0x0 / sc 0x2) MORE DNR
[  627.427947] nvmet: adding nsid 1 to subsystem blktests-subsystem-1
[  627.437084] nvmet: creating nvm controller 1 for subsystem blktests-subsystem-1 for NQN testhostnqn.
[  627.437161] nvme1: Identify(0x6), Invalid Field in Command (sct 0x0 / sc 0x2) MORE DNR
[  627.438984] nvme nvme1: creating 48 I/O queues.
[  627.442620] nvme nvme1: new ctrl: "blktests-subsystem-1"
[  628.506885] XFS (nvme1n1): Mounting V5 Filesystem
[  628.516895] XFS (nvme1n1): Ending clean mount
[  628.519966] xfs filesystem being mounted at /mnt/blktests supports timestamps until 2038 (0x7fffffff)
[  704.852721] XFS (nvme1n1): Unmounting Filesystem
[  704.864724] nvme nvme1: Removing ctrl: NQN "blktests-subsystem-1"

This patch adds the passthru request with RQF_QUIET flag and ignores the error reporting for nvme_log_error() if RQF_QUIET flag is set when request is failed. With this patch we don't get the above error message.

[kch: make a formal patch & test with blktests]
---
 drivers/nvme/host/core.c | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)
drivers/nvme/host/core.c +3 -1
index f204c6f78b5b..b913a89c743e 100644
@@ -370,7 +370,8 @@ static inline void nvme_end_req(struct request *req)
 {
 	blk_status_t status = nvme_error_status(nvme_req(req)->status);
 
-	if (unlikely(nvme_req(req)->status != NVME_SC_SUCCESS))
+	if (unlikely(nvme_req(req)->status != NVME_SC_SUCCESS &&
+		    !(req->rq_flags & RQF_QUIET)))
 		nvme_log_error(req);
 	nvme_end_req_zoned(req);
 	nvme_trace_bio_complete(req);
@@ -651,6 +652,7 @@ void nvme_init_request(struct request *req, struct nvme_command *cmd)
 	cmd->common.flags &= ~NVME_CMD_SGL_ALL;
 
 	req->cmd_flags |= REQ_FAILFAST_DRIVER;
+	req->rq_flags |= RQF_QUIET;
 	if (req->mq_hctx->type == HCTX_TYPE_POLL)
 		req->cmd_flags |= REQ_POLLED;
 	nvme_clear_nvme_request(req);
Signature · 1 line
2.29.0
maintainer
#1 lore raw
Quoted 11 lines · Chaitanya Kulkarni
drivers/nvme/host/core.c +2 -1
@@ -370,7 +370,8 @@ static inline void nvme_end_req(struct request *req)
 {
 	blk_status_t status = nvme_error_status(nvme_req(req)->status);
 
-	if (unlikely(nvme_req(req)->status != NVME_SC_SUCCESS))
+	if (unlikely(nvme_req(req)->status != NVME_SC_SUCCESS &&
+		    !(req->rq_flags & RQF_QUIET)))
 		nvme_log_error(req);
 	nvme_end_req_zoned(req);
 	nvme_trace_bio_complete(req);

drivers/nvme/host/core.c void nvme_init_request(struct request *req, struct nvme_command *cmd) · line 655

Quoted 4 lines
drivers/nvme/host/core.c +1 -0
@@ -651,6 +652,7 @@ void nvme_init_request(struct request *req, struct nvme_command *cmd)
 	cmd->common.flags &= ~NVME_CMD_SGL_ALL;
 
 	req->cmd_flags |= REQ_FAILFAST_DRIVER;
+	req->rq_flags |= RQF_QUIET;

This defeats the admin error logging logic since every admin command comes through here. If you're sure we should do this, then I suppose you can remove that unreachable code.

#2 lore raw
Quoted 14 lines · Keith Busch
Quoted 13 lines · Chaitanya Kulkarni
drivers/nvme/host/core.c +2 -1
@@ -370,7 +370,8 @@ static inline void nvme_end_req(struct request *req)
  {
  	blk_status_t status = nvme_error_status(nvme_req(req)->status);
 
-	if (unlikely(nvme_req(req)->status != NVME_SC_SUCCESS))
+	if (unlikely(nvme_req(req)->status != NVME_SC_SUCCESS &&
+		    !(req->rq_flags & RQF_QUIET)))
  		nvme_log_error(req);
  	nvme_end_req_zoned(req);
  	nvme_trace_bio_complete(req);
@@ -651,6 +652,7 @@ void nvme_init_request(struct request *req, struct nvme_command *cmd)
  	cmd->common.flags &= ~NVME_CMD_SGL_ALL;
 
Quoted 6 lines

drivers/nvme/host/core.c void nvme_init_request(struct request *req, struct nvme_command *cmd) · line 655

Quoted 2 lines
drivers/nvme/host/core.c +1 -0
  	req->cmd_flags |= REQ_FAILFAST_DRIVER;
+	req->rq_flags |= RQF_QUIET;

This defeats the admin error logging logic since every admin command comes through here. If you're sure we should do this, then I suppose you can remove that unreachable code.

If you point out the unreachable code that will be great, I'll keep looking meanwhile...

-ck

maintainer
#3 lore raw
Quoted 22 lines · Chaitanya Kulkarni
Quoted 14 lines · Keith Busch
Quoted 13 lines · Chaitanya Kulkarni
drivers/nvme/host/core.c +2 -1
@@ -370,7 +370,8 @@ static inline void nvme_end_req(struct request *req)
  {
  	blk_status_t status = nvme_error_status(nvme_req(req)->status);
 
-	if (unlikely(nvme_req(req)->status != NVME_SC_SUCCESS))
+	if (unlikely(nvme_req(req)->status != NVME_SC_SUCCESS &&
+		    !(req->rq_flags & RQF_QUIET)))
  		nvme_log_error(req);
  	nvme_end_req_zoned(req);
  	nvme_trace_bio_complete(req);
@@ -651,6 +652,7 @@ void nvme_init_request(struct request *req, struct nvme_command *cmd)
  	cmd->common.flags &= ~NVME_CMD_SGL_ALL;
 
Quoted 6 lines

drivers/nvme/host/core.c void nvme_init_request(struct request *req, struct nvme_command *cmd) · line 655

Quoted 2 lines
drivers/nvme/host/core.c +1 -0
  	req->cmd_flags |= REQ_FAILFAST_DRIVER;
+	req->rq_flags |= RQF_QUIET;

This defeats the admin error logging logic since every admin command comes through here. If you're sure we should do this, then I suppose you can remove that unreachable code.

Quoted 2 lines

If you point out the unreachable code that will be great, I'll keep looking meanwhile...

The second half of nvme_log_error(), plus nvme_get_admin_opcode_str() and the array it defines are unreachable since all admin commands don't log errors with this change.

You could skip the RQF_QUIET setting and check blk_rq_is_passthrough() instead.

#4 lore raw
Quoted 27 lines · Keith Busch
Quoted 22 lines · Chaitanya Kulkarni
Quoted 20 lines · Keith Busch
Quoted 11 lines · Chaitanya Kulkarni
drivers/nvme/host/core.c +2 -1
@@ -370,7 +370,8 @@ static inline void nvme_end_req(struct request *req)
 {
 	blk_status_t status = nvme_error_status(nvme_req(req)->status);
 
-	if (unlikely(nvme_req(req)->status != NVME_SC_SUCCESS))
+	if (unlikely(nvme_req(req)->status != NVME_SC_SUCCESS &&
+		    !(req->rq_flags & RQF_QUIET)))
 		nvme_log_error(req);
 	nvme_end_req_zoned(req);
 	nvme_trace_bio_complete(req);

drivers/nvme/host/core.c void nvme_init_request(struct request *req, struct nvme_command *cmd) · line 655

Quoted 4 lines
drivers/nvme/host/core.c +1 -0
@@ -651,6 +652,7 @@ void nvme_init_request(struct request *req, struct nvme_command *cmd)
 	cmd->common.flags &= ~NVME_CMD_SGL_ALL;
 
 	req->cmd_flags |= REQ_FAILFAST_DRIVER;
+	req->rq_flags |= RQF_QUIET;

This defeats the admin error logging logic since every admin command comes through here. If you're sure we should do this, then I suppose you can remove that unreachable code.

Quoted 8 lines
Quoted 2 lines

If you point out the unreachable code that will be great, I'll keep looking meanwhile...

The second half of nvme_log_error(), plus nvme_get_admin_opcode_str() and the array it defines are unreachable since all admin commands don't log errors with this change.

You could skip the RQF_QUIET setting and check blk_rq_is_passthrough() instead.

Using RQF_QUIET or blk_rq_is_passrhrough() will mean no nvme admin-passthru command will log an error. I ran into this using the blktests I’m coding up for verbose errors. Is this the behavior we want?

Alan

#5 lore raw
Quoted 34 lines · Alan Adamson
Quoted 34 lines · Keith Busch
Quoted 22 lines · Chaitanya Kulkarni
Quoted 14 lines · Keith Busch
Quoted 13 lines · Chaitanya Kulkarni
(quoted diff) +2 -1
@@ -370,7 +370,8 @@ static inline void nvme_end_req(struct request *req)
  {
  	blk_status_t status = nvme_error_status(nvme_req(req)->status);
 
-	if (unlikely(nvme_req(req)->status != NVME_SC_SUCCESS))
+	if (unlikely(nvme_req(req)->status != NVME_SC_SUCCESS &&
+		    !(req->rq_flags & RQF_QUIET)))
  		nvme_log_error(req);
  	nvme_end_req_zoned(req);
  	nvme_trace_bio_complete(req);
@@ -651,6 +652,7 @@ void nvme_init_request(struct request *req, struct nvme_command *cmd)
  	cmd->common.flags &= ~NVME_CMD_SGL_ALL;
 
Quoted 6 lines

Review void nvme_init_request(struct request *req, struct nvme_command *cmd) · line 655

Quoted 2 lines
(quoted diff) +1 -0
  	req->cmd_flags |= REQ_FAILFAST_DRIVER;
+	req->rq_flags |= RQF_QUIET;

This defeats the admin error logging logic since every admin command comes through here. If you're sure we should do this, then I suppose you can remove that unreachable code.

Quoted 2 lines

If you point out the unreachable code that will be great, I'll keep looking meanwhile...

The second half of nvme_log_error(), plus nvme_get_admin_opcode_str() and the array it defines are unreachable since all admin commands don't log errors with this change.

Quoted 7 lines
Quoted 1 line

You could skip the RQF_QUIET setting and check blk_rq_is_passthrough() instead.

Using RQF_QUIET or blk_rq_is_passrhrough() will mean no nvme admin-passthru command will log an error. I ran into this using the blktests I’m coding up for verbose errors. Is this the behavior we want?

Alan

Sagi/Christoph/Hannes/Keith,

After debugging the issue following patch [1] makes errors disappear.

But I'm not sure if this behavior aligns with protocol or not.

I'll keep digging, meanwhile if anyone has an idea please provide a review on patch in [1] if make sense or not.

-ck

[1] mask invalid NVME_ID_CNS_CS_CTRL errors.

(quoted diff) +0 -0

b/drivers/nvme/target/admin-cmd.c
index 397daaf51f1b..e5eea2f0ac9c 100644

drivers/nvme/target/admin-cmd.c +0 -0
@@ -718,9 +718,16 @@ static void nvmet_execute_identify(struct nvmet_req 

*req)

                         switch (req->cmd->identify.csi) {
                         case NVME_CSI_ZNS:
                                 return 
nvmet_execute_identify_cns_cs_ctrl(req);
+                       case NVME_CSI_NVM:
+                               return nvmet_execute_identify_ctrl(req);
                         default:
                                 break;
                         }

+ } else {
+ switch (req->cmd->identify.csi) {
+ case NVME_CSI_NVM:
+ return nvmet_execute_identify_ctrl(req);
+ }

                 }
                 break;
         case NVME_ID_CNS_NS_ACTIVE_LIST:
(quoted diff) +0 -0

b/drivers/nvme/target/discovery.c
index c2162eef8ce1..34c7ed055674 100644

drivers/nvme/target/discovery.c +0 -0
@@ -254,7 +254,11 @@ static void nvmet_execute_disc_identify(struct 

nvmet_req *req)
if (!nvmet_check_transfer_len(req, NVME_IDENTIFY_DATA_SIZE))
return;

-       if (req->cmd->identify.cns != NVME_ID_CNS_CTRL) {
+       switch (req->cmd->identify.cns) {
+       case NVME_ID_CNS_CTRL:
+       case NVME_ID_CNS_CS_CTRL:
+               break;
+       default:
                 req->error_loc = offsetof(struct nvme_identify, cns);
                 status = NVME_SC_INVALID_FIELD | NVME_SC_DNR;
                 goto out;
#6 lore raw

I tried the patch with my tests. It resolves the error logging with nvme/012. My verbose errors tests (nvme/039) now passes, but we still see the error when the system boots.

root@localhost blktests]# ./check nvme/012
nvme/012 (run mkfs and data verification fio job on NVMeOF block device-backed ns) [passed]

    runtime  39.180s  ...  41.432s

[root@localhost blktests]#

[   77.946505] run blktests nvme/012 at 2022-04-06 19:22:23
[   78.013092] loop: module loaded
[   78.014344] loop0: detected capacity change from 0 to 2097152
[   78.024875] nvmet: adding nsid 1 to subsystem blktests-subsystem-1
[   78.052874] nvmet: creating nvm controller 1 for subsystem blktests-subsystem-1 for NQN nqn.2014-08.org.nvmexpress:uuid:00000000-0000-0000-0000-000000000000.
[   78.053107] nvme nvme1: creating 1 I/O queues.
[   78.053182] nvme nvme1: new ctrl: "blktests-subsystem-1"
[   79.125929] XFS (nvme1n1): Mounting V5 Filesystem
[   79.128321] XFS (nvme1n1): Ending clean mount
[   79.128480] xfs filesystem being mounted at /mnt/blktests supports timestamps until 2038 (0x7fffffff)
[  119.253246] XFS (nvme1n1): Unmounting Filesystem
[  119.264005] nvme nvme1: Removing ctrl: NQN "blktests-subsystem-1"

Verbose Errors blktests
=======================
[root@localhost blktests]# ./check nvme/039
nvme/039 => nvme0n1 (test error logging) [passed]

    runtime  0.060s  ...  0.058s

[root@localhost blktests]#

Boot
====

[    3.244056] nvme0: Identify(0x6), Invalid Field in Command (sct 0x0 / sc 0x2) DNR

Alan

~

Quoted 102 lines · Chaitanya Kulkarni
Quoted 34 lines · Alan Adamson
Quoted 34 lines · Keith Busch
Quoted 22 lines · Chaitanya Kulkarni
Quoted 14 lines · Keith Busch
Quoted 13 lines · Chaitanya Kulkarni
(quoted diff) +2 -1
@@ -370,7 +370,8 @@ static inline void nvme_end_req(struct request *req)
 {
 	blk_status_t status = nvme_error_status(nvme_req(req)->status);
 
-	if (unlikely(nvme_req(req)->status != NVME_SC_SUCCESS))
+	if (unlikely(nvme_req(req)->status != NVME_SC_SUCCESS &&
+		    !(req->rq_flags & RQF_QUIET)))
 		nvme_log_error(req);
 	nvme_end_req_zoned(req);
 	nvme_trace_bio_complete(req);
@@ -651,6 +652,7 @@ void nvme_init_request(struct request *req, struct nvme_command *cmd)
 	cmd->common.flags &= ~NVME_CMD_SGL_ALL;
 
Quoted 6 lines

Review void nvme_init_request(struct request *req, struct nvme_command *cmd) · line 655

Quoted 2 lines
(quoted diff) +1 -0
 	req->cmd_flags |= REQ_FAILFAST_DRIVER;
+	req->rq_flags |= RQF_QUIET;

This defeats the admin error logging logic since every admin command comes through here. If you're sure we should do this, then I suppose you can remove that unreachable code.

Quoted 2 lines

If you point out the unreachable code that will be great, I'll keep looking meanwhile...

The second half of nvme_log_error(), plus nvme_get_admin_opcode_str() and the array it defines are unreachable since all admin commands don't log errors with this change.

Quoted 7 lines
Quoted 1 line

You could skip the RQF_QUIET setting and check blk_rq_is_passthrough() instead.

Using RQF_QUIET or blk_rq_is_passrhrough() will mean no nvme admin-passthru command will log an error. I ran into this using the blktests I’m coding up for verbose errors. Is this the behavior we want?

Alan

Sagi/Christoph/Hannes/Keith,

After debugging the issue following patch [1] makes errors disappear.

But I'm not sure if this behavior aligns with protocol or not.

I'll keep digging, meanwhile if anyone has an idea please provide a review on patch in [1] if make sense or not.

-ck

[1] mask invalid NVME_ID_CNS_CS_CTRL errors.

drivers/nvme/target/admin-cmd.c +7 -0
index 397daaf51f1b..e5eea2f0ac9c 100644
@@ -718,9 +718,16 @@ static void nvmet_execute_identify(struct nvmet_req 
 *req)
                        switch (req->cmd->identify.csi) {
                        case NVME_CSI_ZNS:
                                return 
 nvmet_execute_identify_cns_cs_ctrl(req);
+                       case NVME_CSI_NVM:
+                               return nvmet_execute_identify_ctrl(req);
                        default:
                                break;
                        }
+               } else {
+                       switch (req->cmd->identify.csi) {
+                       case NVME_CSI_NVM:
+                               return nvmet_execute_identify_ctrl(req);
+                       }
                }
                break;
        case NVME_ID_CNS_NS_ACTIVE_LIST:
drivers/nvme/target/discovery.c +5 -1
index c2162eef8ce1..34c7ed055674 100644
@@ -254,7 +254,11 @@ static void nvmet_execute_disc_identify(struct 
 nvmet_req *req)
        if (!nvmet_check_transfer_len(req, NVME_IDENTIFY_DATA_SIZE))
                return;
 
-       if (req->cmd->identify.cns != NVME_ID_CNS_CTRL) {
+       switch (req->cmd->identify.cns) {
+       case NVME_ID_CNS_CTRL:
+       case NVME_ID_CNS_CS_CTRL:
+               break;
+       default:
                req->error_loc = offsetof(struct nvme_identify, cns);
                status = NVME_SC_INVALID_FIELD | NVME_SC_DNR;
                goto out;
#9 lore raw
In reply to Alan Adamson
Quoted 2 lines · Alan Adamson

Using RQF_QUIET or blk_rq_is_passrhrough() will mean no nvme admin-passthru command will log an error. I ran into this using the blktests I’m coding up for verbose errors. Is this the behavior we want?

Well, you submitted the logging so we're curious about your use case. SCSI skips logging errors for internally submitted commands, but not for userspace passthrough. So we could move the RQF_QUIET into __nvme_submit_sync_cmd / nvme_keep_alive_work / nvme_timeout / etc.

maintainer
#10 lore raw
Quoted 9 lines
Quoted 3 lines

Using RQF_QUIET or blk_rq_is_passrhrough() will mean no nvme admin-passthru command will log an error. I ran into this using the blktests I’m coding up for verbose errors. Is this the behavior we want?

Well, you submitted the logging so we're curious about your use case. SCSI skips logging errors for internally submitted commands, but not for userspace passthrough. So we could move the RQF_QUIET into __nvme_submit_sync_cmd / nvme_keep_alive_work / nvme_timeout / etc.

Makes sense to me

#11 lore raw
Quoted 6 lines · Sagi Grimberg
Quoted 6 lines
Quoted 5 lines

Using RQF_QUIET or blk_rq_is_passrhrough() will mean no nvme admin-passthru command will log an error.
I ran into this using the blktests I’m coding up for verbose errors. Is this the behavior we want?

Quoted 7 lines
Quoted 5 lines

Well, you submitted the logging so we're curious about your use case. SCSI skips logging errors for internally submitted commands, but not for userspace passthrough.  So we could move the RQF_QUIET into __nvme_submit_sync_cmd / nvme_keep_alive_work / nvme_timeout / etc.

Makes sense to me

This was my initial proposal to fix the call sites with internal passthru commands [1], so this should work.

-ck

[1]
http://lists.infradead.org/pipermail/linux-nvme/2022-April/031160.html

#12 lore raw
Quoted 19 lines · Chaitanya Kulkarni
Quoted 6 lines · Sagi Grimberg
Quoted 6 lines
Quoted 5 lines

Using RQF_QUIET or blk_rq_is_passrhrough() will mean no nvme admin-passthru command will log an error.
I ran into this using the blktests I’m coding up for verbose errors. Is this the behavior we want?

Quoted 7 lines
Quoted 5 lines

Well, you submitted the logging so we're curious about your use case. SCSI skips logging errors for internally submitted commands, but not for userspace passthrough. So we could move the RQF_QUIET into __nvme_submit_sync_cmd / nvme_keep_alive_work / nvme_timeout / etc.

Makes sense to me

Quoted 9 lines

This was my initial proposal to fix the call sites with internal passthru commands [1], so this should work.

-ck

[1]
http://lists.infradead.org/pipermail/linux-nvme/2022-April/031160.html

It works (suppresses logging) for driver initiated admin cmds which is good, but it also suppresses logging when running "nvme admin-passthru” which I don’t think we want.

Alan

#13 lore raw
Quoted 29 lines · Alan Adamson
Quoted 19 lines · Chaitanya Kulkarni
Quoted 6 lines · Sagi Grimberg
Quoted 6 lines
Quoted 5 lines

Using RQF_QUIET or blk_rq_is_passrhrough() will mean no nvme admin-passthru command will log an error.
I ran into this using the blktests I’m coding up for verbose errors. Is this the behavior we want?

Quoted 7 lines
Quoted 5 lines

Well, you submitted the logging so we're curious about your use case. SCSI skips logging errors for internally submitted commands, but not for userspace passthrough. So we could move the RQF_QUIET into __nvme_submit_sync_cmd / nvme_keep_alive_work / nvme_timeout / etc.

Makes sense to me

Quoted 9 lines

This was my initial proposal to fix the call sites with internal passthru commands [1], so this should work.

-ck

[1]
http://lists.infradead.org/pipermail/linux-nvme/2022-April/031160.html

Quoted 5 lines

It works (suppresses logging) for driver initiated admin cmds which is good, but it also suppresses logging when running "nvme admin-passthru” which I don’t think we want.

Alan

I think nvme admin-passthru and io-passthru uses different interface than internal passthru commnds i.e.

nvme admin-passthru :- NVME_IOCTL_ADMIN

nvme_submit_user_cmd()
req allocation and submission no internal passthru cmds.

nvme io-passthru:- NVME_IOCTL_IO_CMD

nvme_user_cmd()
nvme_submit_user_cmd()
req allocation and submission, no internal passthru cmds.

The internal passthru interface uses it's own request allocation and submission, so if we change that it should not affect the nvme admin-io passthru interface... unless we are using something else for internal passthru which is not going through __nvme_submit_sync_cmd() ultimately.

In short what I meant is following patch, this should help us remove the boot time messages you have reported and blktest warnings also, can you please confirm ?

if not then I'll keep digging..

nvme (nvme-5.18) # git diff

drivers/nvme/host/core.c +1 -1
index f204c6f78b5b..a1ea2f736d42 100644
@@ -370,7 +370,7 @@ static inline void nvme_end_req(struct request *req)
  {
         blk_status_t status = nvme_error_status(nvme_req(req)->status);
 
-       if (unlikely(nvme_req(req)->status != NVME_SC_SUCCESS))
+       if (unlikely(nvme_req(req)->status && !(req->rq_flags & RQF_QUIET)))
                 nvme_log_error(req);
         nvme_end_req_zoned(req);
         nvme_trace_bio_complete(req);
@@ -1086,9 +1086,11 @@ int __nvme_submit_sync_cmd(struct request_queue 

*q, struct nvme_command *cmd,
else
req = blk_mq_alloc_request_hctx(q, nvme_req_op(cmd), flags,
qid ? qid - 1 : 0);
-

         if (IS_ERR(req))
                 return PTR_ERR(req);
+
+       req->rq_flags |= RQF_QUIET;
+
         nvme_init_request(req, cmd);

         if (timeout)

if not then I'll keep digging..

-ck

#14 lore raw
Quoted 96 lines · Chaitanya Kulkarni
Quoted 29 lines · Alan Adamson
Quoted 19 lines · Chaitanya Kulkarni
Quoted 6 lines · Sagi Grimberg
Quoted 6 lines
Quoted 5 lines

Using RQF_QUIET or blk_rq_is_passrhrough() will mean no nvme admin-passthru command will log an error.
I ran into this using the blktests I’m coding up for verbose errors. Is this the behavior we want?

Quoted 7 lines
Quoted 5 lines

Well, you submitted the logging so we're curious about your use case. SCSI skips logging errors for internally submitted commands, but not for userspace passthrough. So we could move the RQF_QUIET into __nvme_submit_sync_cmd / nvme_keep_alive_work / nvme_timeout / etc.

Makes sense to me

Quoted 9 lines

This was my initial proposal to fix the call sites with internal passthru commands [1], so this should work.

-ck

[1]
http://lists.infradead.org/pipermail/linux-nvme/2022-April/031160.html

Quoted 5 lines

It works (suppresses logging) for driver initiated admin cmds which is good, but it also suppresses logging when running "nvme admin-passthru” which I don’t think we want.

Alan

I think nvme admin-passthru and io-passthru uses different interface than internal passthru commnds i.e.

nvme admin-passthru :- NVME_IOCTL_ADMIN

nvme_submit_user_cmd()
req allocation and submission no internal passthru cmds.

nvme io-passthru:- NVME_IOCTL_IO_CMD

nvme_user_cmd()
nvme_submit_user_cmd()
req allocation and submission, no internal passthru cmds.

The internal passthru interface uses it's own request allocation and submission, so if we change that it should not affect the nvme admin-io passthru interface... unless we are using something else for internal passthru which is not going through __nvme_submit_sync_cmd() ultimately.

In short what I meant is following patch, this should help us remove the boot time messages you have reported and blktest warnings also, can you please confirm ?

if not then I'll keep digging..

nvme (nvme-5.18) # git diff

drivers/nvme/host/core.c +4 -2
index f204c6f78b5b..a1ea2f736d42 100644
@@ -370,7 +370,7 @@ static inline void nvme_end_req(struct request *req)
 {
        blk_status_t status = nvme_error_status(nvme_req(req)->status);
 
-       if (unlikely(nvme_req(req)->status != NVME_SC_SUCCESS))
+       if (unlikely(nvme_req(req)->status && !(req->rq_flags & RQF_QUIET)))
                nvme_log_error(req);
        nvme_end_req_zoned(req);
        nvme_trace_bio_complete(req);
@@ -1086,9 +1086,11 @@ int __nvme_submit_sync_cmd(struct request_queue 
 *q, struct nvme_command *cmd,
        else
                req = blk_mq_alloc_request_hctx(q, nvme_req_op(cmd), flags,
                                                qid ? qid - 1 : 0);
-
        if (IS_ERR(req))
                return PTR_ERR(req);
+
+       req->rq_flags |= RQF_QUIET;
+
        nvme_init_request(req, cmd);
 

if (timeout)

Quoted 3 lines

if not then I'll keep digging..

-ck

I just tried the patch on my config, it properly suppresses the bootup message, but it also supresses messages from "nvme admin-passthru”.

Alan

#15 lore raw
Quoted 31 lines
Quoted 31 lines

nvme (nvme-5.18) # git diff

drivers/nvme/host/core.c +4 -2
index f204c6f78b5b..a1ea2f736d42 100644
@@ -370,7 +370,7 @@ static inline void nvme_end_req(struct request *req)
  {
         blk_status_t status = nvme_error_status(nvme_req(req)->status);
 
-       if (unlikely(nvme_req(req)->status != NVME_SC_SUCCESS))
+       if (unlikely(nvme_req(req)->status && !(req->rq_flags & RQF_QUIET)))
                 nvme_log_error(req);
         nvme_end_req_zoned(req);
         nvme_trace_bio_complete(req);
@@ -1086,9 +1086,11 @@ int __nvme_submit_sync_cmd(struct request_queue
 *q, struct nvme_command *cmd,
         else
                 req = blk_mq_alloc_request_hctx(q, nvme_req_op(cmd), flags,
                                                 qid ? qid - 1 : 0);
-
         if (IS_ERR(req))
                 return PTR_ERR(req);
+
+       req->rq_flags |= RQF_QUIET;
+
         nvme_init_request(req, cmd);
 

if (timeout)

Quoted 8 lines
Quoted 3 lines

if not then I'll keep digging..

-ck

I just tried the patch on my config, it properly suppresses the bootup message, but it also supresses messages from "nvme admin-passthru”.

Alan

Can you please share a command line for "nvme admin-passthru" where this patch supresses messages ?

-ck

#16 lore raw
Quoted 41 lines · Chaitanya Kulkarni
Quoted 38 lines
Quoted 32 lines

nvme (nvme-5.18) # git diff

drivers/nvme/host/core.c +4 -2
index f204c6f78b5b..a1ea2f736d42 100644
@@ -370,7 +370,7 @@ static inline void nvme_end_req(struct request *req)
 {
        blk_status_t status = nvme_error_status(nvme_req(req)->status);
 
-       if (unlikely(nvme_req(req)->status != NVME_SC_SUCCESS))
+       if (unlikely(nvme_req(req)->status && !(req->rq_flags & RQF_QUIET)))
                nvme_log_error(req);
        nvme_end_req_zoned(req);
        nvme_trace_bio_complete(req);
@@ -1086,9 +1086,11 @@ int __nvme_submit_sync_cmd(struct request_queue
 *q, struct nvme_command *cmd,
        else
                req = blk_mq_alloc_request_hctx(q, nvme_req_op(cmd), flags,
                                                qid ? qid - 1 : 0);
-
        if (IS_ERR(req))
                return PTR_ERR(req);
+
+       req->rq_flags |= RQF_QUIET;
+
        nvme_init_request(req, cmd);
 

if (timeout)

Quoted 3 lines

if not then I'll keep digging..

-ck

I just tried the patch on my config, it properly suppresses the bootup message, but it also supresses messages from "nvme admin-passthru”.

Quoted 5 lines
Quoted 2 lines

Alan

Can you please share a command line for "nvme admin-passthru" where this patch supresses messages ?

I have the NVME Fault Injector configured:

echo 0x286 > /sys/kernel/debug/${ctrl_dev}/fault_inject/status echo 1000 > /sys/kernel/debug/${ctrl_dev}/fault_inject/times echo 100 > /sys/kernel/debug/${ctrl_dev}/fault_inject/probability

nvme admin-passthru /dev/${ctrl_dev} --opcode=06 --data-len=4096 --cdw10=1 -r

echo 0 > /sys/kernel/debug/${ctrl_dev}/fault_inject/probability

Quoted 4 lines

-ck

#17 lore raw
Quoted 6 lines
Quoted 3 lines

Can you please share a command line for "nvme admin-passthru" where this patch supresses messages ?

I have the NVME Fault Injector configured:

Quoted 8 lines

echo 0x286 > /sys/kernel/debug/${ctrl_dev}/fault_inject/status echo 1000 > /sys/kernel/debug/${ctrl_dev}/fault_inject/times echo 100 > /sys/kernel/debug/${ctrl_dev}/fault_inject/probability

nvme admin-passthru /dev/${ctrl_dev} --opcode=06 --data-len=4096 --cdw10=1 -r

echo 0 > /sys/kernel/debug/${ctrl_dev}/fault_inject/probability

I was able to produce same admin-passthru error messages with my patch. See below detailed execution with the script and the log, can you please tell me what is missing ?

Quoted 4 lines
Quoted 3 lines

-ck

* Without this patch I get 3 Error message from fault injection enabled :-
1. [ 1743.353266] nvme1: Identify(0x6), Invalid Field in

    Command (sct 0x0 / sc 0x2) MORE DNR

2. [ 1744.370690] FAULT_INJECTION: forcing a failure.
name fault_inject, interval 1, probability 100, space 0,
times 1000

[ 1744.370698] CPU: 41 PID: 389 Comm: kworker/41:1 Tainted: G 
OE     5.17.0-rc2nvme+ #68
[ 1744.370702] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), 
BIOS rel-1.14.0-0-g155821a1990b-prebuilt.qemu.org 04/01/2014
[ 1744.370704] Workqueue: nvmet-wq nvme_loop_execute_work [nvme_loop]
[ 1744.370712] Call Trace:
[ 1744.370715]  <TASK>
[ 1744.370717]  dump_stack_lvl+0x48/0x5e
[ 1744.370724]  should_fail.cold+0x32/0x37
[ 1744.370729]  nvme_should_fail+0x38/0x90 [nvme_core]
[ 1744.370741]  nvme_loop_queue_response+0xc9/0x143 [nvme_loop]
[ 1744.370745]  nvmet_req_complete+0x11/0x50 [nvmet]
[ 1744.370754]  process_one_work+0x1af/0x380
[ 1744.370758]  worker_thread+0x50/0x3a0
[ 1744.370761]  ? rescuer_thread+0x370/0x370
[ 1744.370763]  kthread+0xe7/0x110
[ 1744.370767]  ? kthread_complete_and_exit+0x20/0x20
[ 1744.370770]  ret_from_fork+0x22/0x30
[ 1744.370776]  </TASK>
[ 1744.370783] nvme1: Identify(0x6), Access Denied
                (sct 0x2 / sc 0x86) DNR

3. [1744.375045] FAULT_INJECTION: forcing a failure.
name fault_inject, interval 1, probability 100, space 0,
times 999

[ 1744.375052] CPU: 42 PID: 391 Comm: kworker/42:1 Tainted: G 
OE     5.17.0-rc2nvme+ #68
[ 1744.375056] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), 
BIOS rel-1.14.0-0-g155821a1990b-prebuilt.qemu.org 04/01/2014
[ 1744.375058] Workqueue: nvmet-wq nvme_loop_execute_work [nvme_loop]
[ 1744.375066] Call Trace:
[ 1744.375070]  <TASK>
[ 1744.375072]  dump_stack_lvl+0x48/0x5e
[ 1744.375079]  should_fail.cold+0x32/0x37
[ 1744.375084]  nvme_should_fail+0x38/0x90 [nvme_core]
[ 1744.375096]  nvme_loop_queue_response+0xc9/0x143 [nvme_loop]
[ 1744.375100]  nvmet_req_complete+0x11/0x50 [nvmet]
[ 1744.375108]  process_one_work+0x1af/0x380
[ 1744.375113]  worker_thread+0x50/0x3a0
[ 1744.375115]  ? rescuer_thread+0x370/0x370
[ 1744.375118]  kthread+0xe7/0x110
[ 1744.375121]  ? kthread_complete_and_exit+0x20/0x20
[ 1744.375124]  ret_from_fork+0x22/0x30
[ 1744.375130]  </TASK>
[ 1744.375148] nvme1: Identify(0x6), Access Denied
                (sct 0x2 / sc 0x86) DNR

* With this patch I only get two from fault injection which are coming from nvme-admin-passthru and no from the internal passthru:-

1. [ 1765.175570] FAULT_INJECTION: forcing a failure.
name fault_inject, interval 1, probability 100, space 0,
times 1000

[ 1765.175579] CPU: 42 PID: 391 Comm: kworker/42:1 Tainted: G 
OE     5.17.0-rc2nvme+ #68
[ 1765.175583] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), 
BIOS rel-1.14.0-0-g155821a1990b-prebuilt.qemu.org 04/01/2014
[ 1765.175585] Workqueue: nvmet-wq nvme_loop_execute_work [nvme_loop]
[ 1765.175593] Call Trace:
[ 1765.175596]  <TASK>
[ 1765.175599]  dump_stack_lvl+0x48/0x5e
[ 1765.175605]  should_fail.cold+0x32/0x37
[ 1765.175610]  nvme_should_fail+0x38/0x90 [nvme_core]
[ 1765.175623]  nvme_loop_queue_response+0xc9/0x143 [nvme_loop]
[ 1765.175627]  nvmet_req_complete+0x11/0x50 [nvmet]
[ 1765.175636]  process_one_work+0x1af/0x380
[ 1765.175640]  worker_thread+0x50/0x3a0
[ 1765.175643]  ? rescuer_thread+0x370/0x370
[ 1765.175645]  kthread+0xe7/0x110
[ 1765.175648]  ? kthread_complete_and_exit+0x20/0x20
[ 1765.175652]  ret_from_fork+0x22/0x30
[ 1765.175658]  </TASK>
[ 1765.175664] nvme1: Identify(0x6), Access Denied
                (sct 0x2 / sc 0x86) DNR
2. [ 1765.179829] FAULT_INJECTION: forcing a failure.
                name fault_inject, interval 1, probability 100, space 0, 
times 999
[ 1765.179835] CPU: 44 PID: 9897 Comm: kworker/44:0 Tainted: G 
  OE     5.17.0-rc2nvme+ #68
[ 1765.179839] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), 
BIOS rel-1.14.0-0-g155821a1990b-prebuilt.qemu.org 04/01/2014
[ 1765.179841] Workqueue: nvmet-wq nvme_loop_execute_work [nvme_loop]
[ 1765.179850] Call Trace:
[ 1765.179853]  <TASK>
[ 1765.179855]  dump_stack_lvl+0x48/0x5e
[ 1765.179862]  should_fail.cold+0x32/0x37
[ 1765.179867]  nvme_should_fail+0x38/0x90 [nvme_core]
[ 1765.179879]  nvme_loop_queue_response+0xc9/0x143 [nvme_loop]
[ 1765.179884]  nvmet_req_complete+0x11/0x50 [nvmet]
[ 1765.179892]  process_one_work+0x1af/0x380
[ 1765.179897]  ? rescuer_thread+0x370/0x370
[ 1765.179899]  worker_thread+0x50/0x3a0
[ 1765.179902]  ? rescuer_thread+0x370/0x370
[ 1765.179903]  kthread+0xe7/0x110
[ 1765.179907]  ? kthread_complete_and_exit+0x20/0x20
[ 1765.179911]  ret_from_fork+0x22/0x30
[ 1765.179917]  </TASK>
[ 1765.179923] nvme1: Identify(0x6), Access Denied (sct 0x2 / sc 0x86) DNR

* Detailed test log with nvmeof nvme-loop controller configured and fault injection enabled with and without this patch to mask internal passthru. It shows only internal passthru error message is masked with this patch and internal passthru error message is present without this patch, nvme-admin-passthru error message present in both cases :-

nvme (nvme-5.18) # sh error_inject.sh
commit 7bec02cef3d11f3d3a80bfa8739f790377bac8d6 (HEAD -> nvme-5.18)
Merge: 226d991feef9 a4a6f3c8f61c
Author: Chaitanya Kulkarni <[email protected]>
Date: Fri Apr 8 16:57:45 2022 -0700

     Merge branch 'nvme-5.18' of git://git.infradead.org/nvme into nvme-5.18

+ umount /mnt/nvme0n1
+ clear_dmesg
./compile_nvme.sh: line 3: clear_dmesg: command not found
umount: /mnt/nvme0n1: no mount point specified.
+ ./delete.sh
+ NQN=testnqn
+ nvme disconnect -n testnqn
Failed to scan topoplogy: No such file or directory

real 0m0.002s
user 0m0.000s
sys 0m0.002s
+ rm -fr '/sys/kernel/config/nvmet/ports/1/subsystems/*'
+ rmdir /sys/kernel/config/nvmet/ports/1
rmdir: failed to remove '/sys/kernel/config/nvmet/ports/1': No such file or directory
+ for subsys in /sys/kernel/config/nvmet/subsystems/*
+ for ns in ${subsys}/namespaces/*
+ echo 0
./delete.sh: line 14:
/sys/kernel/config/nvmet/subsystems/*/namespaces/*/enable: No such file or directory
+ rmdir '/sys/kernel/config/nvmet/subsystems/*/namespaces/*'
rmdir: failed to remove '/sys/kernel/config/nvmet/subsystems/*/namespaces/*': No such file or directory
+ rmdir '/sys/kernel/config/nvmet/subsystems/*'
rmdir: failed to remove '/sys/kernel/config/nvmet/subsystems/*': No such file or directory
+ rmdir 'config/nullb/nullb*'
rmdir: failed to remove 'config/nullb/nullb*': No such file or directory
+ umount /mnt/nvme0n1
umount: /mnt/nvme0n1: no mount point specified.
+ umount /mnt/backend
umount: /mnt/backend: not mounted.
+ modprobe -r nvme_loop
+ modprobe -r nvme_fabrics
+ modprobe -r nvmet
+ modprobe -r nvme
+ modprobe -r null_blk
+ tree /sys/kernel/config
/sys/kernel/config

0 directories, 0 files
+ modprobe -r nvme-fabrics
+ modprobe -r nvme_loop
+ modprobe -r nvmet
+ modprobe -r nvme
+ sleep 1
+ modprobe -r nvme-core
+ lsmod
+ grep nvme
+ sleep 1
+ git diff
+ sleep 1
++ nproc
+ make -j 48 M=drivers/nvme/target/ clean
++ nproc
+ make -j 48 M=drivers/nvme/ modules

   CC [M]  drivers/nvme/target/core.o
   CC [M]  drivers/nvme/target/configfs.o
   CC [M]  drivers/nvme/target/admin-cmd.o
   CC [M]  drivers/nvme/target/fabrics-cmd.o
   CC [M]  drivers/nvme/target/discovery.o
   CC [M]  drivers/nvme/target/io-cmd-file.o
   CC [M]  drivers/nvme/target/io-cmd-bdev.o
   CC [M]  drivers/nvme/target/passthru.o
   CC [M]  drivers/nvme/target/zns.o
   CC [M]  drivers/nvme/target/trace.o
   CC [M]  drivers/nvme/target/loop.o
   CC [M]  drivers/nvme/target/rdma.o
   CC [M]  drivers/nvme/target/fc.o
   CC [M]  drivers/nvme/target/fcloop.o
   CC [M]  drivers/nvme/target/tcp.o
   CC [M]  drivers/nvme/host/core.o
   LD [M]  drivers/nvme/target/nvme-loop.o
   LD [M]  drivers/nvme/target/nvme-fcloop.o
   LD [M]  drivers/nvme/target/nvmet.o
   LD [M]  drivers/nvme/target/nvmet-tcp.o
   LD [M]  drivers/nvme/target/nvmet-fc.o
   LD [M]  drivers/nvme/target/nvmet-rdma.o
   LD [M]  drivers/nvme/host/nvme-core.o
   MODPOST drivers/nvme/Module.symvers
   LD [M]  drivers/nvme/host/nvme-core.ko
   CC [M]  drivers/nvme/target/nvme-fcloop.mod.o
   CC [M]  drivers/nvme/target/nvme-loop.mod.o
   CC [M]  drivers/nvme/target/nvmet-fc.mod.o
   CC [M]  drivers/nvme/target/nvmet-rdma.mod.o
   CC [M]  drivers/nvme/target/nvmet-tcp.mod.o
   CC [M]  drivers/nvme/target/nvmet.mod.o
   LD [M]  drivers/nvme/target/nvme-loop.ko
   LD [M]  drivers/nvme/target/nvme-fcloop.ko
   LD [M]  drivers/nvme/target/nvmet-rdma.ko
   LD [M]  drivers/nvme/target/nvmet-fc.ko
   LD [M]  drivers/nvme/target/nvmet-tcp.ko
   LD [M]  drivers/nvme/target/nvmet.ko

+ HOST=drivers/nvme/host
+ TARGET=drivers/nvme/target
++ uname -r
+ HOST_DEST=/lib/modules/5.17.0-rc2nvme+/kernel/drivers/nvme/host/ ++ uname -r
+ TARGET_DEST=/lib/modules/5.17.0-rc2nvme+/kernel/drivers/nvme/target/
+ cp drivers/nvme/host/nvme-core.ko drivers/nvme/host/nvme-fabrics.ko drivers/nvme/host/nvme-fc.ko drivers/nvme/host/nvme.ko drivers/nvme/host/nvme-rdma.ko drivers/nvme/host/nvme-tcp.ko /lib/modules/5.17.0-rc2nvme+/kernel/drivers/nvme/host//
+ cp drivers/nvme/target/nvme-fcloop.ko drivers/nvme/target/nvme-loop.ko drivers/nvme/target/nvmet-fc.ko drivers/nvme/target/nvmet.ko drivers/nvme/target/nvmet-rdma.ko drivers/nvme/target/nvmet-tcp.ko /lib/modules/5.17.0-rc2nvme+/kernel/drivers/nvme/target//
+ ls -lrth /lib/modules/5.17.0-rc2nvme+/kernel/drivers/nvme/host/ /lib/modules/5.17.0-rc2nvme+/kernel/drivers/nvme/target// /lib/modules/5.17.0-rc2nvme+/kernel/drivers/nvme/host/:
total 6.3M

-rw-r--r--. 1 root root 2.7M Apr  8 16:58 nvme-core.ko
-rw-r--r--. 1 root root 426K Apr  8 16:58 nvme-fabrics.ko
-rw-r--r--. 1 root root 925K Apr  8 16:58 nvme-fc.ko
-rw-r--r--. 1 root root 714K Apr  8 16:58 nvme.ko
-rw-r--r--. 1 root root 856K Apr  8 16:58 nvme-rdma.ko
-rw-r--r--. 1 root root 799K Apr  8 16:58 nvme-tcp.ko

/lib/modules/5.17.0-rc2nvme+/kernel/drivers/nvme/target//:
total 6.3M

-rw-r--r--. 1 root root 475K Apr  8 16:58 nvme-fcloop.ko
-rw-r--r--. 1 root root 419K Apr  8 16:58 nvme-loop.ko
-rw-r--r--. 1 root root 734K Apr  8 16:58 nvmet-fc.ko
-rw-r--r--. 1 root root 3.2M Apr  8 16:58 nvmet.ko
-rw-r--r--. 1 root root 822K Apr  8 16:58 nvmet-rdma.ko
-rw-r--r--. 1 root root 671K Apr  8 16:58 nvmet-tcp.ko

+ sync
+ sync
+ sync
+ modprobe nvme
+ echo 'Press enter to continue ...'
Press enter to continue ...
+ read next

++ NN=1
++ NQN=testnqn
++ let NR_DEVICES=NN+1
++ modprobe -r null_blk
++ modprobe -r nvme
++ modprobe null_blk nr_devices=0
++ modprobe nvme
++ modprobe nvme-fabrics
++ modprobe nvmet
++ modprobe nvme-loop
++ dmesg -c
++ sleep 2
++ tree /sys/kernel/config
/sys/kernel/config

├── nullb
│   └── features
└── nvmet
     ├── hosts
     ├── ports
     └── subsystems

5 directories, 1 file
++ sleep 1
++ mkdir /sys/kernel/config/nvmet/subsystems/testnqn
+++ shuf -i 1-1 -n 1
++ for i in shuf -i 1-$NN -n $NN
++ mkdir config/nullb/nullb1
++ echo 1
++ echo 4096
++ echo 2048
++ echo 1
+++ cat config/nullb/nullb1/index
++ IDX=0
++ mkdir /sys/kernel/config/nvmet/subsystems/testnqn/namespaces/1 ++ echo ' ####### /dev/nullb0'
####### /dev/nullb0
++ echo -n /dev/nullb0
++ cat /sys/kernel/config/nvmet/subsystems/testnqn/namespaces/1/device_path /dev/nullb0
++ echo 1
++ dmesg -c

[ 1740.356540] nvme nvme0: 48/0/0 default/read/poll queues
[ 1743.345785] nvmet: adding nsid 1 to subsystem testnqn

++ mkdir /sys/kernel/config/nvmet/ports/1/ ++ echo -n loop
++ echo -n 1
++ ln -s /sys/kernel/config/nvmet/subsystems/testnqn /sys/kernel/config/nvmet/ports/1/subsystems/ ++ echo transport=loop,nqn=testnqn
++ sleep 1
++ mount
++ column -t
++ grep nvme
++ dmesg -c

[ 1743.353177] nvmet: creating nvm controller 1 for subsystem testnqn 
for NQN 
nqn.2014-08.org.nvmexpress:uuid:510d0435-0ad7-49d4-ae4a-f1c1552b0f0c.
[ 1743.353266] nvme1: Identify(0x6), Invalid Field in Command (sct 0x0 / 
sc 0x2) MORE DNR
[ 1743.355380] nvme nvme1: creating 48 I/O queues.
[ 1743.359206] nvme nvme1: new ctrl: "testnqn"

Node SN Model Namespace Usage Format FW Rev
nvme1n1 8bdab4b79aca987d0eba Linux 1 2.15 GB / 2.15 GB 4 KiB + 0 B 5.17.0-r nvme0n1 foo QEMU NVMe Ctrl 1 1.07 GB / 1.07 GB 512 B + 0 B 1.0 NVMe status: Access Denied: Access to the namespace and/or LBA range is denied due to lack of access rights(0x4286)
Node SN Model Namespace Usage Format FW Rev
nvme0n1 foo QEMU NVMe Ctrl 1 1.07 GB / 1.07 GB 512 B + 0 B 1.0

[ 1744.370690] FAULT_INJECTION: forcing a failure.
                name fault_inject, interval 1, probability 100, space 0, 
times 1000
[ 1744.370698] CPU: 41 PID: 389 Comm: kworker/41:1 Tainted: G 
OE     5.17.0-rc2nvme+ #68
[ 1744.370702] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), 
BIOS rel-1.14.0-0-g155821a1990b-prebuilt.qemu.org 04/01/2014
[ 1744.370704] Workqueue: nvmet-wq nvme_loop_execute_work [nvme_loop]
[ 1744.370712] Call Trace:
[ 1744.370715]  <TASK>
[ 1744.370717]  dump_stack_lvl+0x48/0x5e
[ 1744.370724]  should_fail.cold+0x32/0x37
[ 1744.370729]  nvme_should_fail+0x38/0x90 [nvme_core]
[ 1744.370741]  nvme_loop_queue_response+0xc9/0x143 [nvme_loop]
[ 1744.370745]  nvmet_req_complete+0x11/0x50 [nvmet]
[ 1744.370754]  process_one_work+0x1af/0x380
[ 1744.370758]  worker_thread+0x50/0x3a0
[ 1744.370761]  ? rescuer_thread+0x370/0x370
[ 1744.370763]  kthread+0xe7/0x110
[ 1744.370767]  ? kthread_complete_and_exit+0x20/0x20
[ 1744.370770]  ret_from_fork+0x22/0x30
[ 1744.370776]  </TASK>
[ 1744.370783] nvme1: Identify(0x6), Access Denied (sct 0x2 / sc 0x86) DNR
[ 1744.375045] FAULT_INJECTION: forcing a failure.
                name fault_inject, interval 1, probability 100, space 0, 
times 999
[ 1744.375052] CPU: 42 PID: 391 Comm: kworker/42:1 Tainted: G 
OE     5.17.0-rc2nvme+ #68
[ 1744.375056] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), 
BIOS rel-1.14.0-0-g155821a1990b-prebuilt.qemu.org 04/01/2014
[ 1744.375058] Workqueue: nvmet-wq nvme_loop_execute_work [nvme_loop]
[ 1744.375066] Call Trace:
[ 1744.375070]  <TASK>
[ 1744.375072]  dump_stack_lvl+0x48/0x5e
[ 1744.375079]  should_fail.cold+0x32/0x37
[ 1744.375084]  nvme_should_fail+0x38/0x90 [nvme_core]
[ 1744.375096]  nvme_loop_queue_response+0xc9/0x143 [nvme_loop]
[ 1744.375100]  nvmet_req_complete+0x11/0x50 [nvmet]
[ 1744.375108]  process_one_work+0x1af/0x380
[ 1744.375113]  worker_thread+0x50/0x3a0
[ 1744.375115]  ? rescuer_thread+0x370/0x370
[ 1744.375118]  kthread+0xe7/0x110
[ 1744.375121]  ? kthread_complete_and_exit+0x20/0x20
[ 1744.375124]  ret_from_fork+0x22/0x30
[ 1744.375130]  </TASK>
[ 1744.375148] nvme1: Identify(0x6), Access Denied (sct 0x2 / sc 0x86) DNR

+ NQN=testnqn
+ nvme disconnect -n testnqn
NQN:testnqn disconnected 1 controller(s)

real 0m0.370s
user 0m0.001s
sys 0m0.004s
+ rm -fr /sys/kernel/config/nvmet/ports/1/subsystems/testnqn
+ rmdir /sys/kernel/config/nvmet/ports/1
+ for subsys in /sys/kernel/config/nvmet/subsystems/*
+ for ns in ${subsys}/namespaces/*
+ echo 0
+ rmdir /sys/kernel/config/nvmet/subsystems/testnqn/namespaces/1
+ rmdir /sys/kernel/config/nvmet/subsystems/testnqn
+ rmdir config/nullb/nullb1
+ umount /mnt/nvme0n1
umount: /mnt/nvme0n1: no mount point specified.
+ umount /mnt/backend
umount: /mnt/backend: not mounted.
+ modprobe -r nvme_loop
+ modprobe -r nvme_fabrics
+ modprobe -r nvmet
+ modprobe -r nvme
+ modprobe -r null_blk
+ tree /sys/kernel/config
/sys/kernel/config

0 directories, 0 files
From 2d552b2c756fce48f53d66c5c58ffb6b3e3cac6e Mon Sep 17 00:00:00 2001
From: Chaitanya Kulkarni <[email protected]>
Date: Fri, 8 Apr 2022 16:45:18 -0700
Subject: [PATCH] nvme-core: mark internal passthru req REQ_QUIET

---
  drivers/nvme/host/core.c | 6 ++++--
  1 file changed, 4 insertions(+), 2 deletions(-)
drivers/nvme/host/core.c +1 -1
index f204c6f78b5b..a1ea2f736d42 100644
@@ -370,7 +370,7 @@ static inline void nvme_end_req(struct request *req)
  {
  	blk_status_t status = nvme_error_status(nvme_req(req)->status);
 
-	if (unlikely(nvme_req(req)->status != NVME_SC_SUCCESS))
+	if (unlikely(nvme_req(req)->status && !(req->rq_flags & RQF_QUIET)))
  		nvme_log_error(req);
  	nvme_end_req_zoned(req);
  	nvme_trace_bio_complete(req);
@@ -1086,9 +1086,11 @@ int __nvme_submit_sync_cmd(struct request_queue 

*q, struct nvme_command *cmd,
else
req = blk_mq_alloc_request_hctx(q, nvme_req_op(cmd), flags,
qid ? qid - 1 : 0);
-
if (IS_ERR(req))
return PTR_ERR(req);
+
+ req->rq_flags |= RQF_QUIET;
+
nvme_init_request(req, cmd);

if (timeout)

Signature · 292 lines
2.29.0

Applying: nvme-core: mark internal passthru req REQ_QUIET

commit 3c7952e0f8fcc9affdfa0249ab8211c18a513338 (HEAD -> nvme-5.18)
Author: Chaitanya Kulkarni <[email protected]>
Date:   Fri Apr 8 16:45:18 2022 -0700

     nvme-core: mark internal passthru req REQ_QUIET

     Signed-off-by: Chaitanya Kulkarni <[email protected]>
+ umount /mnt/nvme0n1
+ clear_dmesg
./compile_nvme.sh: line 3: clear_dmesg: command not found
umount: /mnt/nvme0n1: no mount point specified.
+ ./delete.sh
+ NQN=testnqn
+ nvme disconnect -n testnqn
Failed to scan topoplogy: No such file or directory

real	0m0.002s
user	0m0.002s
sys	0m0.000s
+ rm -fr '/sys/kernel/config/nvmet/ports/1/subsystems/*'
+ rmdir /sys/kernel/config/nvmet/ports/1
rmdir: failed to remove '/sys/kernel/config/nvmet/ports/1': No such file 
or directory
+ for subsys in /sys/kernel/config/nvmet/subsystems/*
+ for ns in ${subsys}/namespaces/*
+ echo 0
./delete.sh: line 14: 
/sys/kernel/config/nvmet/subsystems/*/namespaces/*/enable: No such file 
or directory
+ rmdir '/sys/kernel/config/nvmet/subsystems/*/namespaces/*'
rmdir: failed to remove 
'/sys/kernel/config/nvmet/subsystems/*/namespaces/*': No such file or 
directory
+ rmdir '/sys/kernel/config/nvmet/subsystems/*'
rmdir: failed to remove '/sys/kernel/config/nvmet/subsystems/*': No such 
file or directory
+ rmdir 'config/nullb/nullb*'
rmdir: failed to remove 'config/nullb/nullb*': No such file or directory
+ umount /mnt/nvme0n1
umount: /mnt/nvme0n1: no mount point specified.
+ umount /mnt/backend
umount: /mnt/backend: not mounted.
+ modprobe -r nvme_loop
+ modprobe -r nvme_fabrics
+ modprobe -r nvmet
+ modprobe -r nvme
+ modprobe -r null_blk
+ tree /sys/kernel/config
/sys/kernel/config

0 directories, 0 files
+ modprobe -r nvme-fabrics
+ modprobe -r nvme_loop
+ modprobe -r nvmet
+ modprobe -r nvme
+ sleep 1
+ modprobe -r nvme-core
+ lsmod
+ grep nvme
+ sleep 1
+ git diff
+ sleep 1
++ nproc
+ make -j 48 M=drivers/nvme/target/ clean
++ nproc
+ make -j 48 M=drivers/nvme/ modules
   CC [M]  drivers/nvme/target/core.o
   CC [M]  drivers/nvme/target/configfs.o
   CC [M]  drivers/nvme/target/admin-cmd.o
   CC [M]  drivers/nvme/target/fabrics-cmd.o
   CC [M]  drivers/nvme/target/discovery.o
   CC [M]  drivers/nvme/target/io-cmd-file.o
   CC [M]  drivers/nvme/target/io-cmd-bdev.o
   CC [M]  drivers/nvme/target/passthru.o
   CC [M]  drivers/nvme/target/zns.o
   CC [M]  drivers/nvme/target/trace.o
   CC [M]  drivers/nvme/target/loop.o
   CC [M]  drivers/nvme/target/rdma.o
   CC [M]  drivers/nvme/target/fc.o
   CC [M]  drivers/nvme/target/fcloop.o
   CC [M]  drivers/nvme/target/tcp.o
   CC [M]  drivers/nvme/host/core.o
   LD [M]  drivers/nvme/target/nvme-loop.o
   LD [M]  drivers/nvme/target/nvme-fcloop.o
   LD [M]  drivers/nvme/target/nvmet.o
   LD [M]  drivers/nvme/target/nvmet-fc.o
   LD [M]  drivers/nvme/target/nvmet-tcp.o
   LD [M]  drivers/nvme/target/nvmet-rdma.o
   LD [M]  drivers/nvme/host/nvme-core.o
   MODPOST drivers/nvme/Module.symvers
   CC [M]  drivers/nvme/host/nvme-core.mod.o
   CC [M]  drivers/nvme/target/nvme-fcloop.mod.o
   CC [M]  drivers/nvme/target/nvme-loop.mod.o
   CC [M]  drivers/nvme/target/nvmet-fc.mod.o
   CC [M]  drivers/nvme/target/nvmet-rdma.mod.o
   CC [M]  drivers/nvme/target/nvmet-tcp.mod.o
   CC [M]  drivers/nvme/target/nvmet.mod.o
   LD [M]  drivers/nvme/host/nvme-core.ko
   LD [M]  drivers/nvme/target/nvmet-fc.ko
   LD [M]  drivers/nvme/target/nvmet-rdma.ko
   LD [M]  drivers/nvme/target/nvme-fcloop.ko
   LD [M]  drivers/nvme/target/nvmet.ko
   LD [M]  drivers/nvme/target/nvme-loop.ko
   LD [M]  drivers/nvme/target/nvmet-tcp.ko
+ HOST=drivers/nvme/host
+ TARGET=drivers/nvme/target
++ uname -r
+ HOST_DEST=/lib/modules/5.17.0-rc2nvme+/kernel/drivers/nvme/host/
++ uname -r
+ TARGET_DEST=/lib/modules/5.17.0-rc2nvme+/kernel/drivers/nvme/target/
+ cp drivers/nvme/host/nvme-core.ko drivers/nvme/host/nvme-fabrics.ko 
drivers/nvme/host/nvme-fc.ko drivers/nvme/host/nvme.ko 
drivers/nvme/host/nvme-rdma.ko drivers/nvme/host/nvme-tcp.ko 
/lib/modules/5.17.0-rc2nvme+/kernel/drivers/nvme/host//
+ cp drivers/nvme/target/nvme-fcloop.ko drivers/nvme/target/nvme-loop.ko 
drivers/nvme/target/nvmet-fc.ko drivers/nvme/target/nvmet.ko 
drivers/nvme/target/nvmet-rdma.ko drivers/nvme/target/nvmet-tcp.ko 
/lib/modules/5.17.0-rc2nvme+/kernel/drivers/nvme/target//
+ ls -lrth /lib/modules/5.17.0-rc2nvme+/kernel/drivers/nvme/host/ 
/lib/modules/5.17.0-rc2nvme+/kernel/drivers/nvme/target//
/lib/modules/5.17.0-rc2nvme+/kernel/drivers/nvme/host/:
total 6.3M
-rw-r--r--. 1 root root 2.7M Apr  8 16:59 nvme-core.ko
-rw-r--r--. 1 root root 426K Apr  8 16:59 nvme-fabrics.ko
-rw-r--r--. 1 root root 925K Apr  8 16:59 nvme-fc.ko
-rw-r--r--. 1 root root 714K Apr  8 16:59 nvme.ko
-rw-r--r--. 1 root root 856K Apr  8 16:59 nvme-rdma.ko
-rw-r--r--. 1 root root 799K Apr  8 16:59 nvme-tcp.ko

/lib/modules/5.17.0-rc2nvme+/kernel/drivers/nvme/target//:
total 6.3M
-rw-r--r--. 1 root root 475K Apr  8 16:59 nvme-fcloop.ko
-rw-r--r--. 1 root root 419K Apr  8 16:59 nvme-loop.ko
-rw-r--r--. 1 root root 734K Apr  8 16:59 nvmet-fc.ko
-rw-r--r--. 1 root root 3.2M Apr  8 16:59 nvmet.ko
-rw-r--r--. 1 root root 822K Apr  8 16:59 nvmet-rdma.ko
-rw-r--r--. 1 root root 671K Apr  8 16:59 nvmet-tcp.ko
+ sync
+ sync
+ sync
+ modprobe nvme
+ echo 'Press enter to continue ...'
Press enter to continue ...
+ read next

++ NN=1
++ NQN=testnqn
++ let NR_DEVICES=NN+1
++ modprobe -r null_blk
++ modprobe -r nvme
++ modprobe null_blk nr_devices=0
++ modprobe nvme
++ modprobe nvme-fabrics
++ modprobe nvmet
++ modprobe nvme-loop
++ dmesg -c
++ sleep 2
++ tree /sys/kernel/config
/sys/kernel/config
├── nullb
│   └── features
└── nvmet
     ├── hosts
     ├── ports
     └── subsystems

5 directories, 1 file
++ sleep 1
++ mkdir /sys/kernel/config/nvmet/subsystems/testnqn
+++ shuf -i 1-1 -n 1
++ for i in `shuf -i  1-$NN -n $NN`
++ mkdir config/nullb/nullb1
++ echo 1
++ echo 4096
++ echo 2048
++ echo 1
+++ cat config/nullb/nullb1/index
++ IDX=0
++ mkdir /sys/kernel/config/nvmet/subsystems/testnqn/namespaces/1
++ echo ' ####### /dev/nullb0'
  ####### /dev/nullb0
++ echo -n /dev/nullb0
++ cat /sys/kernel/config/nvmet/subsystems/testnqn/namespaces/1/device_path
/dev/nullb0
++ echo 1
++ dmesg -c
[ 1761.160765] nvme nvme0: 48/0/0 default/read/poll queues
[ 1764.151446] nvmet: adding nsid 1 to subsystem testnqn
++ mkdir /sys/kernel/config/nvmet/ports/1/
++ echo -n loop
++ echo -n 1
++ ln -s /sys/kernel/config/nvmet/subsystems/testnqn 
/sys/kernel/config/nvmet/ports/1/subsystems/
++ echo transport=loop,nqn=testnqn
++ sleep 1
++ mount
++ column -t
++ grep nvme
++ dmesg -c
[ 1764.158721] nvmet: creating nvm controller 1 for subsystem testnqn 
for NQN 
nqn.2014-08.org.nvmexpress:uuid:33bca6cd-e82c-4de0-bba2-f70070f69097.
[ 1764.158827] nvme nvme1: creating 48 I/O queues.
[ 1764.162745] nvme nvme1: new ctrl: "testnqn"
Node SN Model Namespace Usage Format FW Rev
nvme1n1 bc09a2ee2829a09471a3 Linux 1 2.15 GB / 2.15 GB 4 KiB + 0 B 5.17.0-r
nvme0n1 foo QEMU NVMe Ctrl 1 1.07 GB / 1.07 GB 512 B + 0 B 1.0
NVMe status: Access Denied: Access to the namespace and/or LBA range is 
denied due to lack of access rights(0x4286)
Node SN Model Namespace Usage Format FW Rev
nvme0n1 foo QEMU NVMe Ctrl 1 1.07 GB / 1.07 GB 512 B + 0 B 1.0
[ 1765.175570] FAULT_INJECTION: forcing a failure.
                name fault_inject, interval 1, probability 100, space 0, 
times 1000
[ 1765.175579] CPU: 42 PID: 391 Comm: kworker/42:1 Tainted: G 
OE     5.17.0-rc2nvme+ #68
[ 1765.175583] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), 
BIOS rel-1.14.0-0-g155821a1990b-prebuilt.qemu.org 04/01/2014
[ 1765.175585] Workqueue: nvmet-wq nvme_loop_execute_work [nvme_loop]
[ 1765.175593] Call Trace:
[ 1765.175596]  <TASK>
[ 1765.175599]  dump_stack_lvl+0x48/0x5e
[ 1765.175605]  should_fail.cold+0x32/0x37
[ 1765.175610]  nvme_should_fail+0x38/0x90 [nvme_core]
[ 1765.175623]  nvme_loop_queue_response+0xc9/0x143 [nvme_loop]
[ 1765.175627]  nvmet_req_complete+0x11/0x50 [nvmet]
[ 1765.175636]  process_one_work+0x1af/0x380
[ 1765.175640]  worker_thread+0x50/0x3a0
[ 1765.175643]  ? rescuer_thread+0x370/0x370
[ 1765.175645]  kthread+0xe7/0x110
[ 1765.175648]  ? kthread_complete_and_exit+0x20/0x20
[ 1765.175652]  ret_from_fork+0x22/0x30
[ 1765.175658]  </TASK>
[ 1765.175664] nvme1: Identify(0x6), Access Denied (sct 0x2 / sc 0x86) DNR
[ 1765.179829] FAULT_INJECTION: forcing a failure.
                name fault_inject, interval 1, probability 100, space 0, 
times 999
[ 1765.179835] CPU: 44 PID: 9897 Comm: kworker/44:0 Tainted: G 
  OE     5.17.0-rc2nvme+ #68
[ 1765.179839] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), 
BIOS rel-1.14.0-0-g155821a1990b-prebuilt.qemu.org 04/01/2014
[ 1765.179841] Workqueue: nvmet-wq nvme_loop_execute_work [nvme_loop]
[ 1765.179850] Call Trace:
[ 1765.179853]  <TASK>
[ 1765.179855]  dump_stack_lvl+0x48/0x5e
[ 1765.179862]  should_fail.cold+0x32/0x37
[ 1765.179867]  nvme_should_fail+0x38/0x90 [nvme_core]
[ 1765.179879]  nvme_loop_queue_response+0xc9/0x143 [nvme_loop]
[ 1765.179884]  nvmet_req_complete+0x11/0x50 [nvmet]
[ 1765.179892]  process_one_work+0x1af/0x380
[ 1765.179897]  ? rescuer_thread+0x370/0x370
[ 1765.179899]  worker_thread+0x50/0x3a0
[ 1765.179902]  ? rescuer_thread+0x370/0x370
[ 1765.179903]  kthread+0xe7/0x110
[ 1765.179907]  ? kthread_complete_and_exit+0x20/0x20
[ 1765.179911]  ret_from_fork+0x22/0x30
[ 1765.179917]  </TASK>
[ 1765.179923] nvme1: Identify(0x6), Access Denied (sct 0x2 / sc 0x86) DNR
+ NQN=testnqn
+ nvme disconnect -n testnqn
NQN:testnqn disconnected 1 controller(s)

real	0m0.347s
user	0m0.002s
sys	0m0.004s
+ rm -fr /sys/kernel/config/nvmet/ports/1/subsystems/testnqn
+ rmdir /sys/kernel/config/nvmet/ports/1
+ for subsys in /sys/kernel/config/nvmet/subsystems/*
+ for ns in ${subsys}/namespaces/*
+ echo 0
+ rmdir /sys/kernel/config/nvmet/subsystems/testnqn/namespaces/1
+ rmdir /sys/kernel/config/nvmet/subsystems/testnqn
+ rmdir config/nullb/nullb1
+ umount /mnt/nvme0n1
umount: /mnt/nvme0n1: no mount point specified.
+ umount /mnt/backend
umount: /mnt/backend: not mounted.
+ modprobe -r nvme_loop
+ modprobe -r nvme_fabrics
+ modprobe -r nvmet
+ modprobe -r nvme
+ modprobe -r null_blk
+ tree /sys/kernel/config
/sys/kernel/config

0 directories, 0 files
HEAD is now at 7bec02cef3d1 Merge branch 'nvme-5.18' of 
git://git.infradead.org/nvme into nvme-5.18
#18 lore raw
Quoted 14 lines · Chaitanya Kulkarni
Quoted 11 lines
Quoted 4 lines

Can you please share a command line for "nvme admin-passthru" where this patch supresses messages ?

I have the NVME Fault Injector configured:

echo 0x286 > /sys/kernel/debug/${ctrl_dev}/fault_inject/status echo 1000 > /sys/kernel/debug/${ctrl_dev}/fault_inject/times echo 100 > /sys/kernel/debug/${ctrl_dev}/fault_inject/probability

Quoted 8 lines
Quoted 4 lines

nvme admin-passthru /dev/${ctrl_dev} --opcode=06 --data-len=4096 --cdw10=1 -r

echo 0 > /sys/kernel/debug/${ctrl_dev}/fault_inject/probability

I was able to produce same admin-passthru error messages with my patch. See below detailed execution with the script and the log, can you please tell me what is missing ?

I reapplied your patch and reran my test and all looks good. I must of applied the incorrect patch last week.

Alan

Quoted 739 lines
Quoted 5 lines
Quoted 4 lines

-ck

* Without this patch I get 3 Error message from fault injection enabled :-
1. [ 1743.353266] nvme1: Identify(0x6), Invalid Field in Command (sct 0x0 / sc 0x2) MORE DNR

2. [ 1744.370690] FAULT_INJECTION: forcing a failure.
name fault_inject, interval 1, probability 100, space 0,
times 1000

[ 1744.370698] CPU: 41 PID: 389 Comm: kworker/41:1 Tainted: G 
OE     5.17.0-rc2nvme+ #68
[ 1744.370702] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), 
BIOS rel-1.14.0-0-g155821a1990b-prebuilt.qemu.org 04/01/2014
[ 1744.370704] Workqueue: nvmet-wq nvme_loop_execute_work [nvme_loop]
[ 1744.370712] Call Trace:
[ 1744.370715]  <TASK>
[ 1744.370717]  dump_stack_lvl+0x48/0x5e
[ 1744.370724]  should_fail.cold+0x32/0x37
[ 1744.370729]  nvme_should_fail+0x38/0x90 [nvme_core]
[ 1744.370741]  nvme_loop_queue_response+0xc9/0x143 [nvme_loop]
[ 1744.370745]  nvmet_req_complete+0x11/0x50 [nvmet]
[ 1744.370754]  process_one_work+0x1af/0x380
[ 1744.370758]  worker_thread+0x50/0x3a0
[ 1744.370761]  ? rescuer_thread+0x370/0x370
[ 1744.370763]  kthread+0xe7/0x110
[ 1744.370767]  ? kthread_complete_and_exit+0x20/0x20
[ 1744.370770]  ret_from_fork+0x22/0x30
[ 1744.370776]  </TASK>
[ 1744.370783] nvme1: Identify(0x6), Access Denied
               (sct 0x2 / sc 0x86) DNR

3. [1744.375045] FAULT_INJECTION: forcing a failure.
name fault_inject, interval 1, probability 100, space 0,
times 999

[ 1744.375052] CPU: 42 PID: 391 Comm: kworker/42:1 Tainted: G 
OE     5.17.0-rc2nvme+ #68
[ 1744.375056] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), 
BIOS rel-1.14.0-0-g155821a1990b-prebuilt.qemu.org 04/01/2014
[ 1744.375058] Workqueue: nvmet-wq nvme_loop_execute_work [nvme_loop]
[ 1744.375066] Call Trace:
[ 1744.375070]  <TASK>
[ 1744.375072]  dump_stack_lvl+0x48/0x5e
[ 1744.375079]  should_fail.cold+0x32/0x37
[ 1744.375084]  nvme_should_fail+0x38/0x90 [nvme_core]
[ 1744.375096]  nvme_loop_queue_response+0xc9/0x143 [nvme_loop]
[ 1744.375100]  nvmet_req_complete+0x11/0x50 [nvmet]
[ 1744.375108]  process_one_work+0x1af/0x380
[ 1744.375113]  worker_thread+0x50/0x3a0
[ 1744.375115]  ? rescuer_thread+0x370/0x370
[ 1744.375118]  kthread+0xe7/0x110
[ 1744.375121]  ? kthread_complete_and_exit+0x20/0x20
[ 1744.375124]  ret_from_fork+0x22/0x30
[ 1744.375130]  </TASK>
[ 1744.375148] nvme1: Identify(0x6), Access Denied
               (sct 0x2 / sc 0x86) DNR

* With this patch I only get two from fault injection which are coming from nvme-admin-passthru and no from the internal passthru:-

1. [ 1765.175570] FAULT_INJECTION: forcing a failure.
name fault_inject, interval 1, probability 100, space 0,
times 1000

[ 1765.175579] CPU: 42 PID: 391 Comm: kworker/42:1 Tainted: G 
OE     5.17.0-rc2nvme+ #68
[ 1765.175583] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), 
BIOS rel-1.14.0-0-g155821a1990b-prebuilt.qemu.org 04/01/2014
[ 1765.175585] Workqueue: nvmet-wq nvme_loop_execute_work [nvme_loop]
[ 1765.175593] Call Trace:
[ 1765.175596]  <TASK>
[ 1765.175599]  dump_stack_lvl+0x48/0x5e
[ 1765.175605]  should_fail.cold+0x32/0x37
[ 1765.175610]  nvme_should_fail+0x38/0x90 [nvme_core]
[ 1765.175623]  nvme_loop_queue_response+0xc9/0x143 [nvme_loop]
[ 1765.175627]  nvmet_req_complete+0x11/0x50 [nvmet]
[ 1765.175636]  process_one_work+0x1af/0x380
[ 1765.175640]  worker_thread+0x50/0x3a0
[ 1765.175643]  ? rescuer_thread+0x370/0x370
[ 1765.175645]  kthread+0xe7/0x110
[ 1765.175648]  ? kthread_complete_and_exit+0x20/0x20
[ 1765.175652]  ret_from_fork+0x22/0x30
[ 1765.175658]  </TASK>
[ 1765.175664] nvme1: Identify(0x6), Access Denied
               (sct 0x2 / sc 0x86) DNR
2. [ 1765.179829] FAULT_INJECTION: forcing a failure.
               name fault_inject, interval 1, probability 100, space 0, 
times 999
[ 1765.179835] CPU: 44 PID: 9897 Comm: kworker/44:0 Tainted: G 
 OE     5.17.0-rc2nvme+ #68
[ 1765.179839] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), 
BIOS rel-1.14.0-0-g155821a1990b-prebuilt.qemu.org 04/01/2014
[ 1765.179841] Workqueue: nvmet-wq nvme_loop_execute_work [nvme_loop]
[ 1765.179850] Call Trace:
[ 1765.179853]  <TASK>
[ 1765.179855]  dump_stack_lvl+0x48/0x5e
[ 1765.179862]  should_fail.cold+0x32/0x37
[ 1765.179867]  nvme_should_fail+0x38/0x90 [nvme_core]
[ 1765.179879]  nvme_loop_queue_response+0xc9/0x143 [nvme_loop]
[ 1765.179884]  nvmet_req_complete+0x11/0x50 [nvmet]
[ 1765.179892]  process_one_work+0x1af/0x380
[ 1765.179897]  ? rescuer_thread+0x370/0x370
[ 1765.179899]  worker_thread+0x50/0x3a0
[ 1765.179902]  ? rescuer_thread+0x370/0x370
[ 1765.179903]  kthread+0xe7/0x110
[ 1765.179907]  ? kthread_complete_and_exit+0x20/0x20
[ 1765.179911]  ret_from_fork+0x22/0x30
[ 1765.179917]  </TASK>
[ 1765.179923] nvme1: Identify(0x6), Access Denied (sct 0x2 / sc 0x86) DNR

* Detailed test log with nvmeof nvme-loop controller configured and fault injection enabled with and without this patch to mask internal passthru. It shows only internal passthru error message is masked with this patch and internal passthru error message is present without this patch, nvme-admin-passthru error message present in both cases :-

nvme (nvme-5.18) # sh error_inject.sh
commit 7bec02cef3d11f3d3a80bfa8739f790377bac8d6 (HEAD -> nvme-5.18)
Merge: 226d991feef9 a4a6f3c8f61c
Author: Chaitanya Kulkarni <[email protected]>
Date: Fri Apr 8 16:57:45 2022 -0700

    Merge branch 'nvme-5.18' of git://git.infradead.org/nvme into nvme-5.18

+ umount /mnt/nvme0n1
+ clear_dmesg
./compile_nvme.sh: line 3: clear_dmesg: command not found
umount: /mnt/nvme0n1: no mount point specified.
+ ./delete.sh
+ NQN=testnqn
+ nvme disconnect -n testnqn
Failed to scan topoplogy: No such file or directory

real 0m0.002s
user 0m0.000s
sys 0m0.002s
+ rm -fr '/sys/kernel/config/nvmet/ports/1/subsystems/*'
+ rmdir /sys/kernel/config/nvmet/ports/1
rmdir: failed to remove '/sys/kernel/config/nvmet/ports/1': No such file or directory
+ for subsys in /sys/kernel/config/nvmet/subsystems/*
+ for ns in ${subsys}/namespaces/*
+ echo 0
./delete.sh: line 14:
/sys/kernel/config/nvmet/subsystems/*/namespaces/*/enable: No such file or directory
+ rmdir '/sys/kernel/config/nvmet/subsystems/*/namespaces/*'
rmdir: failed to remove '/sys/kernel/config/nvmet/subsystems/*/namespaces/*': No such file or directory
+ rmdir '/sys/kernel/config/nvmet/subsystems/*'
rmdir: failed to remove '/sys/kernel/config/nvmet/subsystems/*': No such file or directory
+ rmdir 'config/nullb/nullb*'
rmdir: failed to remove 'config/nullb/nullb*': No such file or directory
+ umount /mnt/nvme0n1
umount: /mnt/nvme0n1: no mount point specified.
+ umount /mnt/backend
umount: /mnt/backend: not mounted.
+ modprobe -r nvme_loop
+ modprobe -r nvme_fabrics
+ modprobe -r nvmet
+ modprobe -r nvme
+ modprobe -r null_blk
+ tree /sys/kernel/config
/sys/kernel/config

0 directories, 0 files
+ modprobe -r nvme-fabrics
+ modprobe -r nvme_loop
+ modprobe -r nvmet
+ modprobe -r nvme
+ sleep 1
+ modprobe -r nvme-core
+ lsmod
+ grep nvme
+ sleep 1
+ git diff
+ sleep 1
++ nproc
+ make -j 48 M=drivers/nvme/target/ clean
++ nproc
+ make -j 48 M=drivers/nvme/ modules

  CC [M]  drivers/nvme/target/core.o
  CC [M]  drivers/nvme/target/configfs.o
  CC [M]  drivers/nvme/target/admin-cmd.o
  CC [M]  drivers/nvme/target/fabrics-cmd.o
  CC [M]  drivers/nvme/target/discovery.o
  CC [M]  drivers/nvme/target/io-cmd-file.o
  CC [M]  drivers/nvme/target/io-cmd-bdev.o
  CC [M]  drivers/nvme/target/passthru.o
  CC [M]  drivers/nvme/target/zns.o
  CC [M]  drivers/nvme/target/trace.o
  CC [M]  drivers/nvme/target/loop.o
  CC [M]  drivers/nvme/target/rdma.o
  CC [M]  drivers/nvme/target/fc.o
  CC [M]  drivers/nvme/target/fcloop.o
  CC [M]  drivers/nvme/target/tcp.o
  CC [M]  drivers/nvme/host/core.o
  LD [M]  drivers/nvme/target/nvme-loop.o
  LD [M]  drivers/nvme/target/nvme-fcloop.o
  LD [M]  drivers/nvme/target/nvmet.o
  LD [M]  drivers/nvme/target/nvmet-tcp.o
  LD [M]  drivers/nvme/target/nvmet-fc.o
  LD [M]  drivers/nvme/target/nvmet-rdma.o
  LD [M]  drivers/nvme/host/nvme-core.o
  MODPOST drivers/nvme/Module.symvers
  LD [M]  drivers/nvme/host/nvme-core.ko
  CC [M]  drivers/nvme/target/nvme-fcloop.mod.o
  CC [M]  drivers/nvme/target/nvme-loop.mod.o
  CC [M]  drivers/nvme/target/nvmet-fc.mod.o
  CC [M]  drivers/nvme/target/nvmet-rdma.mod.o
  CC [M]  drivers/nvme/target/nvmet-tcp.mod.o
  CC [M]  drivers/nvme/target/nvmet.mod.o
  LD [M]  drivers/nvme/target/nvme-loop.ko
  LD [M]  drivers/nvme/target/nvme-fcloop.ko
  LD [M]  drivers/nvme/target/nvmet-rdma.ko
  LD [M]  drivers/nvme/target/nvmet-fc.ko
  LD [M]  drivers/nvme/target/nvmet-tcp.ko
  LD [M]  drivers/nvme/target/nvmet.ko

+ HOST=drivers/nvme/host
+ TARGET=drivers/nvme/target
++ uname -r
+ HOST_DEST=/lib/modules/5.17.0-rc2nvme+/kernel/drivers/nvme/host/ ++ uname -r
+ TARGET_DEST=/lib/modules/5.17.0-rc2nvme+/kernel/drivers/nvme/target/
+ cp drivers/nvme/host/nvme-core.ko drivers/nvme/host/nvme-fabrics.ko drivers/nvme/host/nvme-fc.ko drivers/nvme/host/nvme.ko drivers/nvme/host/nvme-rdma.ko drivers/nvme/host/nvme-tcp.ko /lib/modules/5.17.0-rc2nvme+/kernel/drivers/nvme/host//
+ cp drivers/nvme/target/nvme-fcloop.ko drivers/nvme/target/nvme-loop.ko drivers/nvme/target/nvmet-fc.ko drivers/nvme/target/nvmet.ko drivers/nvme/target/nvmet-rdma.ko drivers/nvme/target/nvmet-tcp.ko /lib/modules/5.17.0-rc2nvme+/kernel/drivers/nvme/target//
+ ls -lrth /lib/modules/5.17.0-rc2nvme+/kernel/drivers/nvme/host/ /lib/modules/5.17.0-rc2nvme+/kernel/drivers/nvme/target// /lib/modules/5.17.0-rc2nvme+/kernel/drivers/nvme/host/:
total 6.3M

(quoted diff) +0 -6
-rw-r--r--. 1 root root 2.7M Apr  8 16:58 nvme-core.ko
-rw-r--r--. 1 root root 426K Apr  8 16:58 nvme-fabrics.ko
-rw-r--r--. 1 root root 925K Apr  8 16:58 nvme-fc.ko
-rw-r--r--. 1 root root 714K Apr  8 16:58 nvme.ko
-rw-r--r--. 1 root root 856K Apr  8 16:58 nvme-rdma.ko
-rw-r--r--. 1 root root 799K Apr  8 16:58 nvme-tcp.ko

/lib/modules/5.17.0-rc2nvme+/kernel/drivers/nvme/target//:
total 6.3M

(quoted diff) +5 -6
-rw-r--r--. 1 root root 475K Apr  8 16:58 nvme-fcloop.ko
-rw-r--r--. 1 root root 419K Apr  8 16:58 nvme-loop.ko
-rw-r--r--. 1 root root 734K Apr  8 16:58 nvmet-fc.ko
-rw-r--r--. 1 root root 3.2M Apr  8 16:58 nvmet.ko
-rw-r--r--. 1 root root 822K Apr  8 16:58 nvmet-rdma.ko
-rw-r--r--. 1 root root 671K Apr  8 16:58 nvmet-tcp.ko
+ sync
+ sync
+ sync
+ modprobe nvme
+ echo 'Press enter to continue ...'

Press enter to continue ...
+ read next

++ NN=1
++ NQN=testnqn
++ let NR_DEVICES=NN+1
++ modprobe -r null_blk
++ modprobe -r nvme
++ modprobe null_blk nr_devices=0
++ modprobe nvme
++ modprobe nvme-fabrics
++ modprobe nvmet
++ modprobe nvme-loop
++ dmesg -c
++ sleep 2
++ tree /sys/kernel/config
/sys/kernel/config

├── nullb
│   └── features
└── nvmet
    ├── hosts
    ├── ports
    └── subsystems

5 directories, 1 file
++ sleep 1
++ mkdir /sys/kernel/config/nvmet/subsystems/testnqn
+++ shuf -i 1-1 -n 1
++ for i in shuf -i 1-$NN -n $NN
++ mkdir config/nullb/nullb1
++ echo 1
++ echo 4096
++ echo 2048
++ echo 1
+++ cat config/nullb/nullb1/index
++ IDX=0
++ mkdir /sys/kernel/config/nvmet/subsystems/testnqn/namespaces/1 ++ echo ' ####### /dev/nullb0'
####### /dev/nullb0
++ echo -n /dev/nullb0
++ cat /sys/kernel/config/nvmet/subsystems/testnqn/namespaces/1/device_path /dev/nullb0
++ echo 1
++ dmesg -c

[ 1740.356540] nvme nvme0: 48/0/0 default/read/poll queues
[ 1743.345785] nvmet: adding nsid 1 to subsystem testnqn

++ mkdir /sys/kernel/config/nvmet/ports/1/ ++ echo -n loop
++ echo -n 1
++ ln -s /sys/kernel/config/nvmet/subsystems/testnqn /sys/kernel/config/nvmet/ports/1/subsystems/ ++ echo transport=loop,nqn=testnqn
++ sleep 1
++ mount
++ column -t
++ grep nvme
++ dmesg -c

[ 1743.353177] nvmet: creating nvm controller 1 for subsystem testnqn 
for NQN 
nqn.2014-08.org.nvmexpress:uuid:510d0435-0ad7-49d4-ae4a-f1c1552b0f0c.
[ 1743.353266] nvme1: Identify(0x6), Invalid Field in Command (sct 0x0 / 
sc 0x2) MORE DNR
[ 1743.355380] nvme nvme1: creating 48 I/O queues.
[ 1743.359206] nvme nvme1: new ctrl: "testnqn"

Node SN Model Namespace Usage Format FW Rev
nvme1n1 8bdab4b79aca987d0eba Linux 1 2.15 GB / 2.15 GB 4 KiB + 0 B 5.17.0-r nvme0n1 foo QEMU NVMe Ctrl 1 1.07 GB / 1.07 GB 512 B + 0 B 1.0 NVMe status: Access Denied: Access to the namespace and/or LBA range is denied due to lack of access rights(0x4286)
Node SN Model Namespace Usage Format FW Rev
nvme0n1 foo QEMU NVMe Ctrl 1 1.07 GB / 1.07 GB 512 B + 0 B 1.0

[ 1744.370690] FAULT_INJECTION: forcing a failure.
               name fault_inject, interval 1, probability 100, space 0, 
times 1000
[ 1744.370698] CPU: 41 PID: 389 Comm: kworker/41:1 Tainted: G 
OE     5.17.0-rc2nvme+ #68
[ 1744.370702] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), 
BIOS rel-1.14.0-0-g155821a1990b-prebuilt.qemu.org 04/01/2014
[ 1744.370704] Workqueue: nvmet-wq nvme_loop_execute_work [nvme_loop]
[ 1744.370712] Call Trace:
[ 1744.370715]  <TASK>
[ 1744.370717]  dump_stack_lvl+0x48/0x5e
[ 1744.370724]  should_fail.cold+0x32/0x37
[ 1744.370729]  nvme_should_fail+0x38/0x90 [nvme_core]
[ 1744.370741]  nvme_loop_queue_response+0xc9/0x143 [nvme_loop]
[ 1744.370745]  nvmet_req_complete+0x11/0x50 [nvmet]
[ 1744.370754]  process_one_work+0x1af/0x380
[ 1744.370758]  worker_thread+0x50/0x3a0
[ 1744.370761]  ? rescuer_thread+0x370/0x370
[ 1744.370763]  kthread+0xe7/0x110
[ 1744.370767]  ? kthread_complete_and_exit+0x20/0x20
[ 1744.370770]  ret_from_fork+0x22/0x30
[ 1744.370776]  </TASK>
[ 1744.370783] nvme1: Identify(0x6), Access Denied (sct 0x2 / sc 0x86) DNR
[ 1744.375045] FAULT_INJECTION: forcing a failure.
               name fault_inject, interval 1, probability 100, space 0, 
times 999
[ 1744.375052] CPU: 42 PID: 391 Comm: kworker/42:1 Tainted: G 
OE     5.17.0-rc2nvme+ #68
[ 1744.375056] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), 
BIOS rel-1.14.0-0-g155821a1990b-prebuilt.qemu.org 04/01/2014
[ 1744.375058] Workqueue: nvmet-wq nvme_loop_execute_work [nvme_loop]
[ 1744.375066] Call Trace:
[ 1744.375070]  <TASK>
[ 1744.375072]  dump_stack_lvl+0x48/0x5e
[ 1744.375079]  should_fail.cold+0x32/0x37
[ 1744.375084]  nvme_should_fail+0x38/0x90 [nvme_core]
[ 1744.375096]  nvme_loop_queue_response+0xc9/0x143 [nvme_loop]
[ 1744.375100]  nvmet_req_complete+0x11/0x50 [nvmet]
[ 1744.375108]  process_one_work+0x1af/0x380
[ 1744.375113]  worker_thread+0x50/0x3a0
[ 1744.375115]  ? rescuer_thread+0x370/0x370
[ 1744.375118]  kthread+0xe7/0x110
[ 1744.375121]  ? kthread_complete_and_exit+0x20/0x20
[ 1744.375124]  ret_from_fork+0x22/0x30
[ 1744.375130]  </TASK>
[ 1744.375148] nvme1: Identify(0x6), Access Denied (sct 0x2 / sc 0x86) DNR

+ NQN=testnqn
+ nvme disconnect -n testnqn
NQN:testnqn disconnected 1 controller(s)

real 0m0.370s
user 0m0.001s
sys 0m0.004s
+ rm -fr /sys/kernel/config/nvmet/ports/1/subsystems/testnqn
+ rmdir /sys/kernel/config/nvmet/ports/1
+ for subsys in /sys/kernel/config/nvmet/subsystems/*
+ for ns in ${subsys}/namespaces/*
+ echo 0
+ rmdir /sys/kernel/config/nvmet/subsystems/testnqn/namespaces/1
+ rmdir /sys/kernel/config/nvmet/subsystems/testnqn
+ rmdir config/nullb/nullb1
+ umount /mnt/nvme0n1
umount: /mnt/nvme0n1: no mount point specified.
+ umount /mnt/backend
umount: /mnt/backend: not mounted.
+ modprobe -r nvme_loop
+ modprobe -r nvme_fabrics
+ modprobe -r nvmet
+ modprobe -r nvme
+ modprobe -r null_blk
+ tree /sys/kernel/config
/sys/kernel/config

0 directories, 0 files
From 2d552b2c756fce48f53d66c5c58ffb6b3e3cac6e Mon Sep 17 00:00:00 2001
From: Chaitanya Kulkarni <[email protected]>
Date: Fri, 8 Apr 2022 16:45:18 -0700
Subject: [PATCH] nvme-core: mark internal passthru req REQ_QUIET

---
 drivers/nvme/host/core.c | 6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)
drivers/nvme/host/core.c +4 -2
index f204c6f78b5b..a1ea2f736d42 100644
@@ -370,7 +370,7 @@ static inline void nvme_end_req(struct request *req)
 {
 	blk_status_t status = nvme_error_status(nvme_req(req)->status);
 
-	if (unlikely(nvme_req(req)->status != NVME_SC_SUCCESS))
+	if (unlikely(nvme_req(req)->status && !(req->rq_flags & RQF_QUIET)))
 		nvme_log_error(req);
 	nvme_end_req_zoned(req);
 	nvme_trace_bio_complete(req);
@@ -1086,9 +1086,11 @@ int __nvme_submit_sync_cmd(struct request_queue 
 *q, struct nvme_command *cmd,
 	else
 		req = blk_mq_alloc_request_hctx(q, nvme_req_op(cmd), flags,
 						qid ? qid - 1 : 0);
-
 	if (IS_ERR(req))
 		return PTR_ERR(req);
+
+	req->rq_flags |= RQF_QUIET;
+
 	nvme_init_request(req, cmd);
 

if (timeout)

Signature · 126 lines
2.29.0

Applying: nvme-core: mark internal passthru req REQ_QUIET

commit 3c7952e0f8fcc9affdfa0249ab8211c18a513338 (HEAD -> nvme-5.18)
Author: Chaitanya Kulkarni <[email protected]>
Date:   Fri Apr 8 16:45:18 2022 -0700

    nvme-core: mark internal passthru req REQ_QUIET

    Signed-off-by: Chaitanya Kulkarni <[email protected]>
+ umount /mnt/nvme0n1
+ clear_dmesg
./compile_nvme.sh: line 3: clear_dmesg: command not found
umount: /mnt/nvme0n1: no mount point specified.
+ ./delete.sh
+ NQN=testnqn
+ nvme disconnect -n testnqn
Failed to scan topoplogy: No such file or directory

real	0m0.002s
user	0m0.002s
sys	0m0.000s
+ rm -fr '/sys/kernel/config/nvmet/ports/1/subsystems/*'
+ rmdir /sys/kernel/config/nvmet/ports/1
rmdir: failed to remove '/sys/kernel/config/nvmet/ports/1': No such file 
or directory
+ for subsys in /sys/kernel/config/nvmet/subsystems/*
+ for ns in ${subsys}/namespaces/*
+ echo 0
./delete.sh: line 14: 
/sys/kernel/config/nvmet/subsystems/*/namespaces/*/enable: No such file 
or directory
+ rmdir '/sys/kernel/config/nvmet/subsystems/*/namespaces/*'
rmdir: failed to remove 
'/sys/kernel/config/nvmet/subsystems/*/namespaces/*': No such file or 
directory
+ rmdir '/sys/kernel/config/nvmet/subsystems/*'
rmdir: failed to remove '/sys/kernel/config/nvmet/subsystems/*': No such 
file or directory
+ rmdir 'config/nullb/nullb*'
rmdir: failed to remove 'config/nullb/nullb*': No such file or directory
+ umount /mnt/nvme0n1
umount: /mnt/nvme0n1: no mount point specified.
+ umount /mnt/backend
umount: /mnt/backend: not mounted.
+ modprobe -r nvme_loop
+ modprobe -r nvme_fabrics
+ modprobe -r nvmet
+ modprobe -r nvme
+ modprobe -r null_blk
+ tree /sys/kernel/config
/sys/kernel/config

0 directories, 0 files
+ modprobe -r nvme-fabrics
+ modprobe -r nvme_loop
+ modprobe -r nvmet
+ modprobe -r nvme
+ sleep 1
+ modprobe -r nvme-core
+ lsmod
+ grep nvme
+ sleep 1
+ git diff
+ sleep 1
++ nproc
+ make -j 48 M=drivers/nvme/target/ clean
++ nproc
+ make -j 48 M=drivers/nvme/ modules
  CC [M]  drivers/nvme/target/core.o
  CC [M]  drivers/nvme/target/configfs.o
  CC [M]  drivers/nvme/target/admin-cmd.o
  CC [M]  drivers/nvme/target/fabrics-cmd.o
  CC [M]  drivers/nvme/target/discovery.o
  CC [M]  drivers/nvme/target/io-cmd-file.o
  CC [M]  drivers/nvme/target/io-cmd-bdev.o
  CC [M]  drivers/nvme/target/passthru.o
  CC [M]  drivers/nvme/target/zns.o
  CC [M]  drivers/nvme/target/trace.o
  CC [M]  drivers/nvme/target/loop.o
  CC [M]  drivers/nvme/target/rdma.o
  CC [M]  drivers/nvme/target/fc.o
  CC [M]  drivers/nvme/target/fcloop.o
  CC [M]  drivers/nvme/target/tcp.o
  CC [M]  drivers/nvme/host/core.o
  LD [M]  drivers/nvme/target/nvme-loop.o
  LD [M]  drivers/nvme/target/nvme-fcloop.o
  LD [M]  drivers/nvme/target/nvmet.o
  LD [M]  drivers/nvme/target/nvmet-fc.o
  LD [M]  drivers/nvme/target/nvmet-tcp.o
  LD [M]  drivers/nvme/target/nvmet-rdma.o
  LD [M]  drivers/nvme/host/nvme-core.o
  MODPOST drivers/nvme/Module.symvers
  CC [M]  drivers/nvme/host/nvme-core.mod.o
  CC [M]  drivers/nvme/target/nvme-fcloop.mod.o
  CC [M]  drivers/nvme/target/nvme-loop.mod.o
  CC [M]  drivers/nvme/target/nvmet-fc.mod.o
  CC [M]  drivers/nvme/target/nvmet-rdma.mod.o
  CC [M]  drivers/nvme/target/nvmet-tcp.mod.o
  CC [M]  drivers/nvme/target/nvmet.mod.o
  LD [M]  drivers/nvme/host/nvme-core.ko
  LD [M]  drivers/nvme/target/nvmet-fc.ko
  LD [M]  drivers/nvme/target/nvmet-rdma.ko
  LD [M]  drivers/nvme/target/nvme-fcloop.ko
  LD [M]  drivers/nvme/target/nvmet.ko
  LD [M]  drivers/nvme/target/nvme-loop.ko
  LD [M]  drivers/nvme/target/nvmet-tcp.ko
+ HOST=drivers/nvme/host
+ TARGET=drivers/nvme/target
++ uname -r
+ HOST_DEST=/lib/modules/5.17.0-rc2nvme+/kernel/drivers/nvme/host/
++ uname -r
+ TARGET_DEST=/lib/modules/5.17.0-rc2nvme+/kernel/drivers/nvme/target/
+ cp drivers/nvme/host/nvme-core.ko drivers/nvme/host/nvme-fabrics.ko 
drivers/nvme/host/nvme-fc.ko drivers/nvme/host/nvme.ko 
drivers/nvme/host/nvme-rdma.ko drivers/nvme/host/nvme-tcp.ko 
/lib/modules/5.17.0-rc2nvme+/kernel/drivers/nvme/host//
+ cp drivers/nvme/target/nvme-fcloop.ko drivers/nvme/target/nvme-loop.ko 
drivers/nvme/target/nvmet-fc.ko drivers/nvme/target/nvmet.ko 
drivers/nvme/target/nvmet-rdma.ko drivers/nvme/target/nvmet-tcp.ko 
/lib/modules/5.17.0-rc2nvme+/kernel/drivers/nvme/target//
+ ls -lrth /lib/modules/5.17.0-rc2nvme+/kernel/drivers/nvme/host/ 
/lib/modules/5.17.0-rc2nvme+/kernel/drivers/nvme/target//
/lib/modules/5.17.0-rc2nvme+/kernel/drivers/nvme/host/:
total 6.3M
drivers/nvme/host/core.c +0 -6
-rw-r--r--. 1 root root 2.7M Apr  8 16:59 nvme-core.ko
-rw-r--r--. 1 root root 426K Apr  8 16:59 nvme-fabrics.ko
-rw-r--r--. 1 root root 925K Apr  8 16:59 nvme-fc.ko
-rw-r--r--. 1 root root 714K Apr  8 16:59 nvme.ko
-rw-r--r--. 1 root root 856K Apr  8 16:59 nvme-rdma.ko
-rw-r--r--. 1 root root 799K Apr  8 16:59 nvme-tcp.ko

/lib/modules/5.17.0-rc2nvme+/kernel/drivers/nvme/target//:
total 6.3M

drivers/nvme/host/core.c +5 -6
-rw-r--r--. 1 root root 475K Apr  8 16:59 nvme-fcloop.ko
-rw-r--r--. 1 root root 419K Apr  8 16:59 nvme-loop.ko
-rw-r--r--. 1 root root 734K Apr  8 16:59 nvmet-fc.ko
-rw-r--r--. 1 root root 3.2M Apr  8 16:59 nvmet.ko
-rw-r--r--. 1 root root 822K Apr  8 16:59 nvmet-rdma.ko
-rw-r--r--. 1 root root 671K Apr  8 16:59 nvmet-tcp.ko
+ sync
+ sync
+ sync
+ modprobe nvme
+ echo 'Press enter to continue ...'

Press enter to continue ...
+ read next

++ NN=1
++ NQN=testnqn
++ let NR_DEVICES=NN+1
++ modprobe -r null_blk
++ modprobe -r nvme
++ modprobe null_blk nr_devices=0
++ modprobe nvme
++ modprobe nvme-fabrics
++ modprobe nvmet
++ modprobe nvme-loop
++ dmesg -c
++ sleep 2
++ tree /sys/kernel/config
/sys/kernel/config

├── nullb
│   └── features
└── nvmet
    ├── hosts
    ├── ports
    └── subsystems

5 directories, 1 file
++ sleep 1
++ mkdir /sys/kernel/config/nvmet/subsystems/testnqn
+++ shuf -i 1-1 -n 1
++ for i in shuf -i 1-$NN -n $NN
++ mkdir config/nullb/nullb1
++ echo 1
++ echo 4096
++ echo 2048
++ echo 1
+++ cat config/nullb/nullb1/index
++ IDX=0
++ mkdir /sys/kernel/config/nvmet/subsystems/testnqn/namespaces/1 ++ echo ' ####### /dev/nullb0'
####### /dev/nullb0
++ echo -n /dev/nullb0
++ cat /sys/kernel/config/nvmet/subsystems/testnqn/namespaces/1/device_path /dev/nullb0
++ echo 1
++ dmesg -c

[ 1761.160765] nvme nvme0: 48/0/0 default/read/poll queues
[ 1764.151446] nvmet: adding nsid 1 to subsystem testnqn

++ mkdir /sys/kernel/config/nvmet/ports/1/ ++ echo -n loop
++ echo -n 1
++ ln -s /sys/kernel/config/nvmet/subsystems/testnqn /sys/kernel/config/nvmet/ports/1/subsystems/ ++ echo transport=loop,nqn=testnqn
++ sleep 1
++ mount
++ column -t
++ grep nvme
++ dmesg -c

[ 1764.158721] nvmet: creating nvm controller 1 for subsystem testnqn 
for NQN 
nqn.2014-08.org.nvmexpress:uuid:33bca6cd-e82c-4de0-bba2-f70070f69097.
[ 1764.158827] nvme nvme1: creating 48 I/O queues.
[ 1764.162745] nvme nvme1: new ctrl: "testnqn"

Node SN Model Namespace Usage Format FW Rev
nvme1n1 bc09a2ee2829a09471a3 Linux 1 2.15 GB / 2.15 GB 4 KiB + 0 B 5.17.0-r nvme0n1 foo QEMU NVMe Ctrl 1 1.07 GB / 1.07 GB 512 B + 0 B 1.0 NVMe status: Access Denied: Access to the namespace and/or LBA range is denied due to lack of access rights(0x4286)
Node SN Model Namespace Usage Format FW Rev
nvme0n1 foo QEMU NVMe Ctrl 1 1.07 GB / 1.07 GB 512 B + 0 B 1.0

[ 1765.175570] FAULT_INJECTION: forcing a failure.
               name fault_inject, interval 1, probability 100, space 0, 
times 1000
[ 1765.175579] CPU: 42 PID: 391 Comm: kworker/42:1 Tainted: G 
OE     5.17.0-rc2nvme+ #68
[ 1765.175583] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), 
BIOS rel-1.14.0-0-g155821a1990b-prebuilt.qemu.org 04/01/2014
[ 1765.175585] Workqueue: nvmet-wq nvme_loop_execute_work [nvme_loop]
[ 1765.175593] Call Trace:
[ 1765.175596]  <TASK>
[ 1765.175599]  dump_stack_lvl+0x48/0x5e
[ 1765.175605]  should_fail.cold+0x32/0x37
[ 1765.175610]  nvme_should_fail+0x38/0x90 [nvme_core]
[ 1765.175623]  nvme_loop_queue_response+0xc9/0x143 [nvme_loop]
[ 1765.175627]  nvmet_req_complete+0x11/0x50 [nvmet]
[ 1765.175636]  process_one_work+0x1af/0x380
[ 1765.175640]  worker_thread+0x50/0x3a0
[ 1765.175643]  ? rescuer_thread+0x370/0x370
[ 1765.175645]  kthread+0xe7/0x110
[ 1765.175648]  ? kthread_complete_and_exit+0x20/0x20
[ 1765.175652]  ret_from_fork+0x22/0x30
[ 1765.175658]  </TASK>
[ 1765.175664] nvme1: Identify(0x6), Access Denied (sct 0x2 / sc 0x86) DNR
[ 1765.179829] FAULT_INJECTION: forcing a failure.
               name fault_inject, interval 1, probability 100, space 0, 
times 999
[ 1765.179835] CPU: 44 PID: 9897 Comm: kworker/44:0 Tainted: G 
 OE     5.17.0-rc2nvme+ #68
[ 1765.179839] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), 
BIOS rel-1.14.0-0-g155821a1990b-prebuilt.qemu.org 04/01/2014
[ 1765.179841] Workqueue: nvmet-wq nvme_loop_execute_work [nvme_loop]
[ 1765.179850] Call Trace:
[ 1765.179853]  <TASK>
[ 1765.179855]  dump_stack_lvl+0x48/0x5e
[ 1765.179862]  should_fail.cold+0x32/0x37
[ 1765.179867]  nvme_should_fail+0x38/0x90 [nvme_core]
[ 1765.179879]  nvme_loop_queue_response+0xc9/0x143 [nvme_loop]
[ 1765.179884]  nvmet_req_complete+0x11/0x50 [nvmet]
[ 1765.179892]  process_one_work+0x1af/0x380
[ 1765.179897]  ? rescuer_thread+0x370/0x370
[ 1765.179899]  worker_thread+0x50/0x3a0
[ 1765.179902]  ? rescuer_thread+0x370/0x370
[ 1765.179903]  kthread+0xe7/0x110
[ 1765.179907]  ? kthread_complete_and_exit+0x20/0x20
[ 1765.179911]  ret_from_fork+0x22/0x30
[ 1765.179917]  </TASK>
[ 1765.179923] nvme1: Identify(0x6), Access Denied (sct 0x2 / sc 0x86) DNR

+ NQN=testnqn
+ nvme disconnect -n testnqn
NQN:testnqn disconnected 1 controller(s)

real 0m0.347s
user 0m0.002s
sys 0m0.004s
+ rm -fr /sys/kernel/config/nvmet/ports/1/subsystems/testnqn
+ rmdir /sys/kernel/config/nvmet/ports/1
+ for subsys in /sys/kernel/config/nvmet/subsystems/*
+ for ns in ${subsys}/namespaces/*
+ echo 0
+ rmdir /sys/kernel/config/nvmet/subsystems/testnqn/namespaces/1
+ rmdir /sys/kernel/config/nvmet/subsystems/testnqn
+ rmdir config/nullb/nullb1
+ umount /mnt/nvme0n1
umount: /mnt/nvme0n1: no mount point specified.
+ umount /mnt/backend
umount: /mnt/backend: not mounted.
+ modprobe -r nvme_loop
+ modprobe -r nvme_fabrics
+ modprobe -r nvmet
+ modprobe -r nvme
+ modprobe -r null_blk
+ tree /sys/kernel/config
/sys/kernel/config

0 directories, 0 files
HEAD is now at 7bec02cef3d1 Merge branch 'nvme-5.18' of
git://git.infradead.org/nvme into nvme-5.18

#19 lore raw
Quoted 8 lines
Quoted 4 lines

I was able to produce same admin-passthru error messages with my patch. See below detailed execution with the script and the log, can you please tell me what is missing ?

I reapplied your patch and reran my test and all looks good. I must of applied the incorrect patch last week.

That's what I thought, it will be great if you can send tested by tag this so we can send out later this week :-

http://lists.infradead.org/pipermail/linux-nvme/2022-April/031349.html

Quoted 3 lines

Alan

Tested-by
#20 lore raw

- Verified the error log is no longer being seen at boot up.
- Verified errors can still be logged while using 'nvme admin-passthru’.

Quoted 21 lines · Chaitanya Kulkarni
Quoted 9 lines
Quoted 5 lines

I was able to produce same admin-passthru error messages with my patch. See below detailed execution with the script and the log, can you please tell me what is missing ?

I reapplied your patch and reran my test and all looks good. I must of applied the incorrect patch last week.

That's what I thought, it will be great if you can send tested by tag this so we can send out later this week :-

http://lists.infradead.org/pipermail/linux-nvme/2022-April/031349.html

Quoted 3 lines

Alan

#21 lore raw
In reply to [email protected]
Quoted 12 lines · On Apr 7, 2022, at 1:51 AM, [email protected] wrote:
Quoted 2 lines · Alan Adamson

Using RQF_QUIET or blk_rq_is_passrhrough() will mean no nvme admin-passthru command will log an error. I ran into this using the blktests I’m coding up for verbose errors. Is this the behavior we want?

Well, you submitted the logging so we're curious about your use case. SCSI skips logging errors for internally submitted commands, but not for userspace passthrough. So we could move the RQF_QUIET into __nvme_submit_sync_cmd / nvme_keep_alive_work / nvme_timeout / etc.

I think that's what we want, skip logging for nvme driver submitted commands, and allow logging for passthrough commands.

Alan

#22 lore raw
In reply to Keith Busch
Quoted 14 lines · Keith Busch
Quoted 13 lines · Chaitanya Kulkarni
drivers/nvme/host/core.c +2 -1
@@ -370,7 +370,8 @@ static inline void nvme_end_req(struct request *req)
  {
  	blk_status_t status = nvme_error_status(nvme_req(req)->status);
 
-	if (unlikely(nvme_req(req)->status != NVME_SC_SUCCESS))
+	if (unlikely(nvme_req(req)->status != NVME_SC_SUCCESS &&
+		    !(req->rq_flags & RQF_QUIET)))
  		nvme_log_error(req);
  	nvme_end_req_zoned(req);
  	nvme_trace_bio_complete(req);
@@ -651,6 +652,7 @@ void nvme_init_request(struct request *req, struct nvme_command *cmd)
  	cmd->common.flags &= ~NVME_CMD_SGL_ALL;
 
Quoted 6 lines

drivers/nvme/host/core.c void nvme_init_request(struct request *req, struct nvme_command *cmd) · line 655

Quoted 2 lines
drivers/nvme/host/core.c +1 -0
  	req->cmd_flags |= REQ_FAILFAST_DRIVER;
+	req->rq_flags |= RQF_QUIET;

This defeats the admin error logging logic since every admin command comes through here. If you're sure we should do this, then I suppose you can remove that unreachable code.

Perhaps we should do the req->rq_flags |= RQF_QUIET in the respective callers ? just thinking out loudly..

-ck