nvme: do not log errors for user commands
PATCH 1/1 nvme: do not log errors for user commands
User space might issue commands which fail. Even though userland handles them correclty, these command get logged which irritates users.
E.g. libnvme's nvme_scan_topology function is using Namespace Identification Descriptor list (CNS Value 0x03) for discovering. It handles the case where it fails but still the kernel logs this as error:
nvme0: Admin Cmd(0x6), I/O Error (sct 0x0 / sc 0x2) DNR
nvme1: Admin Cmd(0x6), I/O Error (sct 0x0 / sc 0x2) DNR
This disables error logging via nvme_log_error for all passthrough commands. Instead reverting the complete patch, let's disable this bit until the interested parties in extended logging have figured out how to do it without introducing regressions. Removing the now non-functional part in nvme_log_error can be done later if needed.
---
v1:
- SET RQF_QUIET flag instead of checking NVME_REQ_USERCMD in nvme_log_error.
v0:
- https://lore.kernel.org/linux-nvme/[email protected]/
drivers/nvme/host/ioctl.c | 1 + 1 file changed, 1 insertion(+)
drivers/nvme/host/ioctl.c +1 -0
@@ -151,6 +151,7 @@ static int nvme_submit_user_cmd(struct request_queue *q, bio = req->bio; ctrl =nvme_req (req)->ctrl; + req->rq_flags |=RQF_QUIET ; ret =nvme_execute_passthru_rq (req, &effects);if (result)
Signature · 1 line
2.37.3
Quoted 17 lines · Daniel Wagner
User space might issue commands which fail. Even though userland handles them correclty, these command get logged which irritates users.
E.g. libnvme's
nvme_scan_topologyfunction is using Namespace Identification Descriptor list (CNS Value0x03) for discovering. It handles the case where it fails but still the kernel logs this as error:nvme0: Admin Cmd(0x6), I/O Error (sct
0x0/ sc0x2) DNR
nvme1: Admin Cmd(0x6), I/O Error (sct0x0/ sc0x2) DNRThis disables error logging via
nvme_log_errorfor all passthrough commands. Instead reverting the complete patch, let's disable this bit until the interested parties in extended logging have figured out how to do it without introducing regressions. Removing the now non-functional part innvme_log_errorcan be done later if needed.
<snip>
drivers/nvme/host/ioctl.c static int nvme_submit_user_cmd(struct request_queue *q, · line 154
Quoted 5 lines
drivers/nvme/host/ioctl.c +1
-0@@ -151,6 +151,7 @@ static int nvme_submit_user_cmd(struct request_queue *q, bio = req->bio; ctrl =nvme_req (req)->ctrl; + req->rq_flags |=RQF_QUIET ;
With this, we are disabling logging for ADMIN and IO commands via the ioctl. IMO, the error logging for IO commands could be useful for debugging. But I do understand your regression argument here.
drivers/nvme/host/ioctl.c static int nvme_submit_user_cmd(struct request_queue *q, · line 157
Quoted 3 lines
drivers/nvme/host/ioctl.c +0
-0ret =nvme_execute_passthru_rq (req, &effects);if (result)
We can also send admin commands via the uring_cmd interface:nvme_dev_uring_cmd(). Should we also enable RQF_QUIET flags for them?
P.S: I sent a patch today that touches the error logging in NVMe that sets the starting LBA as zero for IO passthrough requests.
https://lore.kernel.org/linux-nvme/[email protected]/
Quoted 5 lines · Pankaj Raghav
drivers/nvme/host/ioctl.c static int nvme_submit_user_cmd(struct request_queue *q, · line 154
Quoted 1 line
drivers/nvme/host/ioctl.c +1
-0+ req->rq_flags |=RQF_QUIET ;With this, we are disabling logging for ADMIN and IO commands via the ioctl. IMO, the error logging for IO commands could be useful for debugging. But I do understand your regression argument here.
I do get the debugging argument, but this clearly regresses production code. Ideally, this code should be opt-in. The only way I see how to make this work is by introducing a new API which allows enable this feature.
Quoted 6 lines
drivers/nvme/host/ioctl.c static int nvme_submit_user_cmd(struct request_queue *q, · line 157
Quoted 3 lines
drivers/nvme/host/ioctl.c +0
-0ret =nvme_execute_passthru_rq (req, &effects);if (result)We can also send admin commands via the
uring_cmdinterface:nvme_dev_uring_cmd(). Should we also enableRQF_QUIETflags for them?
Good point, this path is likely to have the same problem.
Quoted 3 lines
P.S: I sent a patch today that touches the error logging in NVMe that sets the starting LBA as zero for IO passthrough requests.
https://lore.kernel.org/linux-nvme/[email protected]/
Yes, saw your patch, but with this patch it wont do match afterwards :)
Quoted 11 lines · Daniel Wagner
Quoted 5 lines · Pankaj Raghav
drivers/nvme/host/ioctl.c static int nvme_submit_user_cmd(struct request_queue *q, · line 154
Quoted 1 line
drivers/nvme/host/ioctl.c +1
-0+ req->rq_flags |=RQF_QUIET ;With this, we are disabling logging for ADMIN and IO commands via the ioctl. IMO, the error logging for IO commands could be useful for debugging. But I do understand your regression argument here.
Quoted 4 lines
I do get the debugging argument, but this clearly regresses production code. Ideally, this code should be opt-in. The only way I see how to make this work is by introducing a new API which allows enable this feature.
Looks like a new API may need to be created. I’ll investigate that. With your change, there will need to be a blktests change.
nvme/039 => nvme0n1 (test error logging) [failed]
runtime 0.120s ... 0.118s
--- tests/nvme/039.out 2022-09-21 17:14:12.760890663 -0400
+++ /root/blktests/results/nvme0n1/nvme/039.out.bad 2022-10-06 19:11:45.846879996 -0400
@@ -2,6 +2,4 @@
Read(0x2) @ LBA 0, 1 blocks, Unrecovered Read Error (sct 0x2 / sc 0x81) DNR
Read(0x2) @ LBA 0, 1 blocks, Unknown (sct 0x3 / sc 0x75) DNR
Write(0x1) @ LBA 0, 1 blocks, Write Fault (sct 0x2 / sc 0x80) DNR
- Identify(0x6), Access Denied (sct 0x2 / sc 0x86) DNR
- Unknown(0x96), Invalid Command Opcode (sct 0x0 / sc 0x1) DNR
Test complete[root@localhost blktests]#
Alan
Quoted 16 lines
Quoted 7 lines
Quoted 4 lines
ret = nvme_execute_passthru_rq(req, &effects); if (result)We can also send admin commands via the
uring_cmdinterface:nvme_dev_uring_cmd(). Should we also enableRQF_QUIETflags for them?Good point, this path is likely to have the same problem.
Quoted 3 lines
P.S: I sent a patch today that touches the error logging in NVMe that sets the starting LBA as zero for IO passthrough requests.
https://lore.kernel.org/linux-nvme/[email protected]/Yes, saw your patch, but with this patch it wont do match afterwards :)
Hi Alan,
Quoted 7 lines · Alan Adamson
Quoted 4 lines
I do get the debugging argument, but this clearly regresses production code. Ideally, this code should be opt-in. The only way I see how to make this work is by introducing a new API which allows enable this feature.
Looks like a new API may need to be created. I’ll investigate that. With your change, there will need to be a blktests change.
Thanks for picking this up.
Quoted 11 lines
nvme/039 => nvme0n1 (test error logging) [failed]
runtime 0.120s ... 0.118s --- tests/nvme/039.out 2022-09-21 17:14:12.760890663 -0400 +++ /root/blktests/results/nvme0n1/nvme/039.out.bad 2022-10-06 19:11:45.846879996 -0400 @@ -2,6 +2,4 @@ Read(0x2) @ LBA 0, 1 blocks, Unrecovered Read Error (sct 0x2 / sc 0x81) DNR Read(0x2) @ LBA 0, 1 blocks, Unknown (sct 0x3 / sc 0x75) DNR Write(0x1) @ LBA 0, 1 blocks, Write Fault (sct 0x2 / sc 0x80) DNR - Identify(0x6), Access Denied (sct 0x2 / sc 0x86) DNR - Unknown(0x96), Invalid Command Opcode (sct 0x0 / sc 0x1) DNR Test complete
Hmm, that is not supposed to happen. Well, obviously this needs some more work :)
Thanks,
Daniel
Quoted 8 lines · Alan Adamson
Quoted 4 lines
I do get the debugging argument, but this clearly regresses production code. Ideally, this code should be opt-in. The only way I see how to make this work is by introducing a new API which allows enable this feature.
Looks like a new API may need to be created. I’ll investigate that. With your change, there
will need to be a blktests change.
Yes, I think we need to effectively revert this for user commands and then add an opt in - be that from the submitter, or through a sysfs file for all command.
Keith Busch
maintainerQuoted 6 lines · Christoph Hellwig
Quoted 5 lines · Alan Adamson
Quoted 4 lines
I do get the debugging argument, but this clearly regresses production code. Ideally, this code should be opt-in. The only way I see how to make this work is by introducing a new API which allows enable this feature.
Quoted 7 lines
Quoted 3 lines
Looks like a new API may need to be created. I’ll investigate that. With your change, there
will need to be a blktests change.Yes, I think we need to effectively revert this for user commands and then add an opt in - be that from the submitter, or through a sysfs file for all command.
Yeah, I think we should revert logging errors for user admin commands (io command logs can probably remain). There are tools running at regular intervals that query all devices for optional pages, and spamming the kernel logs with these repeated errors is causing some problems.
Quoted 19 lines · Keith Busch
Quoted 6 lines · Christoph Hellwig
Quoted 5 lines · Alan Adamson
Quoted 4 lines
I do get the debugging argument, but this clearly regresses production code. Ideally, this code should be opt-in. The only way I see how to make this work is by introducing a new API which allows enable this feature.
Quoted 7 lines
Quoted 3 lines
Looks like a new API may need to be created. I’ll investigate that. With your change, there
will need to be a blktests change.Yes, I think we need to effectively revert this for user commands and then add an opt in - be that from the submitter, or through a sysfs file for all command.
Quoted 5 lines
Yeah, I think we should revert logging errors for user admin commands (io command logs can probably remain). There are tools running at regular intervals that query all devices for optional pages, and spamming the kernel logs with these repeated errors is causing some problems.
I agree. I’ll look into a being able to opt in to logging of admin commands.
Alan