Skip to contentSkip to conversation

nvme: do not log errors for user commands

PATCH In review Daniel Wagner 5 participants 8 messages Oct 6, 2022 – Oct 17, 2022 NVMe

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 nvme: do not log errors for user commands

User space might issue commands which fail. Even though userland handles them correclty, these command get logged which irritates users.

E.g. libnvme's nvme_scan_topology function is using Namespace Identification Descriptor list (CNS Value 0x03) for discovering. It handles the case where it fails but still the kernel logs this as error:

nvme0: Admin Cmd(0x6), I/O Error (sct 0x0 / sc 0x2) DNR
nvme1: Admin Cmd(0x6), I/O Error (sct 0x0 / sc 0x2) DNR

This disables error logging via nvme_log_error for all passthrough commands. Instead reverting the complete patch, let's disable this bit until the interested parties in extended logging have figured out how to do it without introducing regressions. Removing the now non-functional part in nvme_log_error can be done later if needed.

Fixes bd83fe6f2cd2 ("nvme: add verbose error logging")
Reported-by [email protected]
Cc Alan Adamson <[email protected]>
Cc Chaitanya Kulkarni <[email protected]>
Cc Martin K. Petersen <[email protected]>

---

v1:
- SET RQF_QUIET flag instead of checking NVME_REQ_USERCMD in nvme_log_error.
v0:
- https://lore.kernel.org/linux-nvme/[email protected]/

 drivers/nvme/host/ioctl.c | 1 +
 1 file changed, 1 insertion(+)
drivers/nvme/host/ioctl.c +1 -0
index a48a79ed5c4c..b2c1b73f0a22 100644
@@ -151,6 +151,7 @@ static int nvme_submit_user_cmd(struct request_queue *q,
 	bio = req->bio;
 	ctrl = nvme_req(req)->ctrl;
 
+	req->rq_flags |= RQF_QUIET;
 	ret = nvme_execute_passthru_rq(req, &effects);
 
 	if (result)
Signature · 1 line
2.37.3
#1 lore raw
Quoted 17 lines · Daniel Wagner

User space might issue commands which fail. Even though userland handles them correclty, these command get logged which irritates users.

E.g. libnvme's nvme_scan_topology function is using Namespace Identification Descriptor list (CNS Value 0x03) for discovering. It handles the case where it fails but still the kernel logs this as error:

nvme0: Admin Cmd(0x6), I/O Error (sct 0x0 / sc 0x2) DNR
nvme1: Admin Cmd(0x6), I/O Error (sct 0x0 / sc 0x2) DNR

This disables error logging via nvme_log_error for all passthrough commands. Instead reverting the complete patch, let's disable this bit until the interested parties in extended logging have figured out how to do it without introducing regressions. Removing the now non-functional part in nvme_log_error can be done later if needed.

<snip>

drivers/nvme/host/ioctl.c static int nvme_submit_user_cmd(struct request_queue *q, · line 154

Quoted 5 lines
drivers/nvme/host/ioctl.c +1 -0
@@ -151,6 +151,7 @@ static int nvme_submit_user_cmd(struct request_queue *q,
 	bio = req->bio;
 	ctrl = nvme_req(req)->ctrl;
 
+	req->rq_flags |= RQF_QUIET;

With this, we are disabling logging for ADMIN and IO commands via the ioctl. IMO, the error logging for IO commands could be useful for debugging. But I do understand your regression argument here.

drivers/nvme/host/ioctl.c static int nvme_submit_user_cmd(struct request_queue *q, · line 157

Quoted 3 lines
drivers/nvme/host/ioctl.c +0 -0
 	ret = nvme_execute_passthru_rq(req, &effects);
 
 	if (result)

We can also send admin commands via the uring_cmd interface:
nvme_dev_uring_cmd(). Should we also enable RQF_QUIET flags for them?

P.S: I sent a patch today that touches the error logging in NVMe that sets the starting LBA as zero for IO passthrough requests.
https://lore.kernel.org/linux-nvme/[email protected]/

#2 lore raw
Quoted 5 lines · Pankaj Raghav

drivers/nvme/host/ioctl.c static int nvme_submit_user_cmd(struct request_queue *q, · line 154

Quoted 1 line
drivers/nvme/host/ioctl.c +1 -0
+	req->rq_flags |= RQF_QUIET;

With this, we are disabling logging for ADMIN and IO commands via the ioctl. IMO, the error logging for IO commands could be useful for debugging. But I do understand your regression argument here.

I do get the debugging argument, but this clearly regresses production code. Ideally, this code should be opt-in. The only way I see how to make this work is by introducing a new API which allows enable this feature.

Quoted 6 lines

drivers/nvme/host/ioctl.c static int nvme_submit_user_cmd(struct request_queue *q, · line 157

Quoted 3 lines
drivers/nvme/host/ioctl.c +0 -0
 	ret = nvme_execute_passthru_rq(req, &effects);
 
 	if (result)

We can also send admin commands via the uring_cmd interface:
nvme_dev_uring_cmd(). Should we also enable RQF_QUIET flags for them?

Good point, this path is likely to have the same problem.

Quoted 3 lines

P.S: I sent a patch today that touches the error logging in NVMe that sets the starting LBA as zero for IO passthrough requests.
https://lore.kernel.org/linux-nvme/[email protected]/

Yes, saw your patch, but with this patch it wont do match afterwards :)

#3 lore raw
Quoted 11 lines · Daniel Wagner
Quoted 5 lines · Pankaj Raghav

drivers/nvme/host/ioctl.c static int nvme_submit_user_cmd(struct request_queue *q, · line 154

Quoted 1 line
drivers/nvme/host/ioctl.c +1 -0
+	req->rq_flags |= RQF_QUIET;

With this, we are disabling logging for ADMIN and IO commands via the ioctl. IMO, the error logging for IO commands could be useful for debugging. But I do understand your regression argument here.

Quoted 4 lines

I do get the debugging argument, but this clearly regresses production code. Ideally, this code should be opt-in. The only way I see how to make this work is by introducing a new API which allows enable this feature.

Looks like a new API may need to be created. I’ll investigate that. With your change, there will need to be a blktests change.

nvme/039 => nvme0n1 (test error logging) [failed]

    runtime  0.120s  ...  0.118s
    --- tests/nvme/039.out	2022-09-21 17:14:12.760890663 -0400
    +++ /root/blktests/results/nvme0n1/nvme/039.out.bad	2022-10-06 19:11:45.846879996 -0400
    @@ -2,6 +2,4 @@
      Read(0x2) @ LBA 0, 1 blocks, Unrecovered Read Error (sct 0x2 / sc 0x81) DNR 
      Read(0x2) @ LBA 0, 1 blocks, Unknown (sct 0x3 / sc 0x75) DNR 
      Write(0x1) @ LBA 0, 1 blocks, Write Fault (sct 0x2 / sc 0x80) DNR 
    - Identify(0x6), Access Denied (sct 0x2 / sc 0x86) DNR 
    - Unknown(0x96), Invalid Command Opcode (sct 0x0 / sc 0x1) DNR 
     Test complete

[root@localhost blktests]#

Alan

Quoted 16 lines
Quoted 7 lines
Quoted 4 lines
	ret = nvme_execute_passthru_rq(req, &effects);

	if (result)

We can also send admin commands via the uring_cmd interface:
nvme_dev_uring_cmd(). Should we also enable RQF_QUIET flags for them?

Good point, this path is likely to have the same problem.

Quoted 3 lines

P.S: I sent a patch today that touches the error logging in NVMe that sets the starting LBA as zero for IO passthrough requests.
https://lore.kernel.org/linux-nvme/[email protected]/

Yes, saw your patch, but with this patch it wont do match afterwards :)

#4 lore raw

Hi Alan,

Quoted 7 lines · Alan Adamson
Quoted 4 lines

I do get the debugging argument, but this clearly regresses production code. Ideally, this code should be opt-in. The only way I see how to make this work is by introducing a new API which allows enable this feature.

Looks like a new API may need to be created. I’ll investigate that. With your change, there will need to be a blktests change.

Thanks for picking this up.

Quoted 11 lines

nvme/039 => nvme0n1 (test error logging) [failed]

    runtime  0.120s  ...  0.118s
    --- tests/nvme/039.out	2022-09-21 17:14:12.760890663 -0400
    +++ /root/blktests/results/nvme0n1/nvme/039.out.bad	2022-10-06 19:11:45.846879996 -0400
    @@ -2,6 +2,4 @@
      Read(0x2) @ LBA 0, 1 blocks, Unrecovered Read Error (sct 0x2 / sc 0x81) DNR 
      Read(0x2) @ LBA 0, 1 blocks, Unknown (sct 0x3 / sc 0x75) DNR 
      Write(0x1) @ LBA 0, 1 blocks, Write Fault (sct 0x2 / sc 0x80) DNR 
    - Identify(0x6), Access Denied (sct 0x2 / sc 0x86) DNR 
    - Unknown(0x96), Invalid Command Opcode (sct 0x0 / sc 0x1) DNR 
     Test complete

Hmm, that is not supposed to happen. Well, obviously this needs some more work :)

Thanks,
Daniel

#5 lore raw
In reply to Alan Adamson
Quoted 8 lines · Alan Adamson
Quoted 4 lines

I do get the debugging argument, but this clearly regresses production code. Ideally, this code should be opt-in. The only way I see how to make this work is by introducing a new API which allows enable this feature.

Looks like a new API may need to be created. I’ll investigate that. With your change, there
will need to be a blktests change.

Yes, I think we need to effectively revert this for user commands and then add an opt in - be that from the submitter, or through a sysfs file for all command.

maintainer
#6 lore raw
Quoted 6 lines · Christoph Hellwig
Quoted 5 lines · Alan Adamson
Quoted 4 lines

I do get the debugging argument, but this clearly regresses production code. Ideally, this code should be opt-in. The only way I see how to make this work is by introducing a new API which allows enable this feature.

Quoted 7 lines
Quoted 3 lines

Looks like a new API may need to be created. I’ll investigate that. With your change, there
will need to be a blktests change.

Yes, I think we need to effectively revert this for user commands and then add an opt in - be that from the submitter, or through a sysfs file for all command.

Yeah, I think we should revert logging errors for user admin commands (io command logs can probably remain). There are tools running at regular intervals that query all devices for optional pages, and spamming the kernel logs with these repeated errors is causing some problems.

#7 lore raw
Quoted 19 lines · Keith Busch
Quoted 6 lines · Christoph Hellwig
Quoted 5 lines · Alan Adamson
Quoted 4 lines

I do get the debugging argument, but this clearly regresses production code. Ideally, this code should be opt-in. The only way I see how to make this work is by introducing a new API which allows enable this feature.

Quoted 7 lines
Quoted 3 lines

Looks like a new API may need to be created. I’ll investigate that. With your change, there
will need to be a blktests change.

Yes, I think we need to effectively revert this for user commands and then add an opt in - be that from the submitter, or through a sysfs file for all command.

Quoted 5 lines

Yeah, I think we should revert logging errors for user admin commands (io command logs can probably remain). There are tools running at regular intervals that query all devices for optional pages, and spamming the kernel logs with these repeated errors is causing some problems.

I agree. I’ll look into a being able to opt in to logging of admin commands.

Alan