nvme: Add verbose error logging
cover nvme: Add verbose error logging
This patch improves logging for NVMe errors. Currently, we only get a a vague idea as to why commands fail since only the block layer status is captured on error. This patch allows us to see why a command was failed by the controller. This is very useful when debugging problems in the field.
An example of an imnproved logged error:
[ 4652.812867] nvme0n1: Read @ LBA 1536, 1 blocks, Unrecovered Read Error (sct 0x2 / sc 0x81) DNR
Alan Adamson (1):
nvme: Add verbose error logging
drivers/nvme/host/Kconfig | 8 ++ drivers/nvme/host/Makefile | 1 + drivers/nvme/host/core.c | 4 + drivers/nvme/host/errors.c | 176 +++++++++++++++++++++++++++++++++++++ drivers/nvme/host/nvme.h | 6 ++ include/linux/nvme.h | 1 + 6 files changed, 196 insertions(+) create mode 100644 drivers/nvme/host/errors.c
Signature · 1 line
2.27.0
PATCH 1/1 nvme: Add verbose error logging
Enable more verbose reporting of NVMe errors. The kernel needs to be compiled with the NVME_VERBOSE_ERRORS kernel config option enabled.
Logged error example:
nvme0n1: Read @ LBA 2304, 1 blocks, Unrecovered Read Error (sct 0x2 / sc 0x81) DNR
--- drivers/nvme/host/Kconfig | 8 ++ drivers/nvme/host/Makefile | 1 + drivers/nvme/host/core.c | 4 + drivers/nvme/host/errors.c | 176 +++++++++++++++++++++++++++++++++++++ drivers/nvme/host/nvme.h | 6 ++ include/linux/nvme.h | 1 + 6 files changed, 196 insertions(+) create mode 100644 drivers/nvme/host/errors.c
drivers/nvme/host/Kconfig +8 -0
@@ -24,6 +24,14 @@ config NVME_MULTIPATH /dev/nvmeXnY device will show up for each NVMe namespace, even if it is accessible through multiple controllers. +config NVME_VERBOSE_ERRORS + bool "NVMe verbose error reporting" + depends on NVME_CORE + help + This option enables verbose reporting for NVMe errors. The + error translation table will grow the kernel image size by + about 4 KB. + config NVME_HWMON bool "NVMe hardware monitoring" depends on (NVME_CORE=y && HWMON=y) || (NVME_CORE=m && HWMON)
drivers/nvme/host/Makefile +1 -0
@@ -12,6 +12,7 @@ obj-$(CONFIG_NVME_TCP) += nvme-tcp.o nvme-core-y := core.o ioctl.o nvme-core-$(CONFIG_TRACING) += trace.o nvme-core-$(CONFIG_NVME_MULTIPATH) += multipath.o +nvme-core-$(CONFIG_NVME_VERBOSE_ERRORS) += errors.o nvme-core-$(CONFIG_BLK_DEV_ZONED) += zns.o nvme-core-$(CONFIG_FAULT_INJECTION_DEBUG_FS) += fault_inject.o nvme-core-$(CONFIG_NVME_HWMON) += hwmon.o
drivers/nvme/host/core.c +4 -0
@@ -346,6 +346,8 @@ static inline void nvme_end_req(struct request *req)void nvme_complete_rq (struct request *req) { +blk_status_t status =nvme_error_status (nvme_req (req)->status); +trace_nvme_complete_rq (req);nvme_cleanup_cmd (req);
@@ -354,6 +356,8 @@ void nvme_complete_rq(struct request *req)switch (nvme_decide_disposition (req)) {case COMPLETE : +if (unlikely (status !=BLK_STS_OK )) +nvme_error_log (req);nvme_end_req (req);return ;case RETRY :
drivers/nvme/host/errors.c new file +176 -0
176 lines not on this page · raw message
drivers/nvme/host/nvme.h +6 -0
@@ -922,4 +922,10 @@ static inline bool nvme_multi_css(struct nvme_ctrl *ctrl) return (ctrl->ctrl_config & NVME_CC_CSS_MASK) == NVME_CC_CSS_CSI; } +#ifdef CONFIG_NVME_VERBOSE_ERRORS +extern void nvme_error_log (struct request *req); +#else +static inline void nvme_error_log (struct request *req) {} +#endif /* CONFIG_NVME_VERBOSE_ERRORS */ +#endif /* _NVME_H */
include/linux/nvme.h +1 -0
@@ -1636,6 +1636,7 @@ enum {NVME_SC_HOST_ABORTED_CMD =0x371 ,NVME_SC_CRD =0x1800 , +NVME_SC_MORE =0x2000 , NVME_SC_DNR = 0x4000, };
Signature · 1 line
2.27.0
Alan,
Quoted 11 lines · Alan Adamson
Enable more verbose reporting of NVMe errors. The kernel needs to be compiled with the
NVME_VERBOSE_ERRORSkernel config option enabled.Logged error example:
nvme0n1: Read @ LBA 2304, 1 blocks, Unrecovered Read Error (sct
0x2/ sc0x81) DNRSigned-off-by Alan Adamson <[email protected]>Signed-off-by Martin K. Petersen <[email protected]>Reviewed-by Himanshu Madhani <[email protected]>---
Although I always prefer descriptive error messages, I've not seen even half of these errors in the field. This will :-
1. Increase the size of the driver code.
2. Require constant patching of the driver code when spec is updated, I'm afraid this will turn into churn.
Also, we can easily track down the error in the NVMe spec from the error code that driver is printing currently, is that not sufficient ? if not please provide a scenario, we can always improve bare minimum error logging.
-ck
Hi Chaitanya!
Quoted 6 lines
Although I always prefer descriptive error messages, I've not seen even half of these errors in the field. This will :-
1. Increase the size of the driver code.
2. Require constant patching of the driver code when spec is updated, I'm afraid this will turn into churn.
Updating the corresponding list in SCSI hasn't been a big problem.
Quoted 2 lines
Also, we can easily track down the error in the NVMe spec from the error code that driver is printing currently, is that not sufficient ?
Only the block layer status is captured.
In addition, customers are not really armed with the NVMe error decoder ring and are left with no explanation as to why an I/O failed. We have had good results in the field with the verbose SCSI error logging and that's why this patch was modeled to work the same way.
Signature · 1 line
Martin K. Petersen Oracle Linux Engineering
Keith Busch
maintainerQuoted 15 lines · Martin K. Petersen
Hi Chaitanya!
Quoted 6 lines
Although I always prefer descriptive error messages, I've not seen even half of these errors in the field. This will :-
1. Increase the size of the driver code.
2. Require constant patching of the driver code when spec is updated, I'm afraid this will turn into churn.Updating the corresponding list in SCSI hasn't been a big problem.
Quoted 2 lines
Also, we can easily track down the error in the NVMe spec from the error code that driver is printing currently, is that not sufficient ?
Quoted 6 lines
Only the block layer status is captured.
In addition, customers are not really armed with the NVMe error decoder ring and are left with no explanation as to why an I/O failed. We have had good results in the field with the verbose SCSI error logging and that's why this patch was modeled to work the same way.
If they are not familiar with the decoder, does seeing something like "Conflicting Attributes" really provide the user a better sense than "0x280"? Either way, it sounds like they'd just open a bug with someone who is familiar with the spec, in which case numeric codes should be sufficient.
Are the existing nvme trace events not providing enough information? The existing ones appear to have even greater detail than what this patch provides. Or is there some other reason why the kernel log is preferred over the event tracer for capturing these types of errors?
On the patch itself:
I'm not sure having it be a new Kconfig option is a good idea since it requires a recompile to enable/disable. Could this be toggled with a static_key instead so we may disable/enable verbose logging at runtime?
The new "struct nvme_string_table" is a bit inefficient. Unknown errors will scan the entire table, which will slow down our hot completion path on any error. The codes are small and mostly sequential, so I think you could create a sparse "static const char * const[]", then the "code" becomes the index that can be looked up in constant time. The total space of the array shouldn't be much different than the proposal.
A significant amount of the new strings are specific to admin commands, yet errors for admin commands skip the logging. Why define unreachable strings? Or perhaps a better question: why skip logging admin errors?
IO errors can happen in large batches, so perhaps pr_err() should be ratelimted.
Sagi Grimberg
maintainerQuoted 29 lines · Keith Busch
Quoted 15 lines · Martin K. Petersen
Hi Chaitanya!
Quoted 6 lines
Although I always prefer descriptive error messages, I've not seen even half of these errors in the field. This will :-
1. Increase the size of the driver code.
2. Require constant patching of the driver code when spec is updated, I'm afraid this will turn into churn.Updating the corresponding list in SCSI hasn't been a big problem.
Quoted 2 lines
Also, we can easily track down the error in the NVMe spec from the error code that driver is printing currently, is that not sufficient ?
Quoted 6 lines
Only the block layer status is captured.
In addition, customers are not really armed with the NVMe error decoder ring and are left with no explanation as to why an I/O failed. We have had good results in the field with the verbose SCSI error logging and that's why this patch was modeled to work the same way.
If they are not familiar with the decoder, does seeing something like "Conflicting Attributes" really provide the user a better sense than "
0x280"? Either way, it sounds like they'd just open a bug with someone who is familiar with the spec, in which case numeric codes should be sufficient.
Quoted 4 lines
Are the existing nvme trace events not providing enough information? The existing ones appear to have even greater detail than what this patch provides. Or is there some other reason why the kernel log is preferred over the event tracer for capturing these types of errors?
Maybe the string info be added to the trace events? If so, we may need a verbosity level for the trace to record "only errors".
Although the one advantage for the log would be the first time error status (or for generally rare errors).
Quoted 6 lines
On the patch itself:
I'm not sure having it be a new
Kconfigoption is a good idea since it requires a recompile to enable/disable. Could this be toggled with astatic_keyinstead so we may disable/enable verbose logging at runtime?
Quoted 6 lines
The new "struct
nvme_string_table" is a bit inefficient. Unknown errors will scan the entire table, which will slow down our hot completion path on any error. The codes are small and mostly sequential, so I think you could create a sparse "static const char * const[]", then the "code" becomes the index that can be looked up in constant time. The total space of the array shouldn't be much different than the proposal.
100% agree.
Quoted 6 lines
A significant amount of the new strings are specific to admin commands, yet errors for admin commands skip the logging. Why define unreachable strings? Or perhaps a better question: why skip logging admin errors?
IO errors can happen in large batches, so perhaps
pr_err()should be ratelimted.
Agree.
Quoted 50 lines · Sagi Grimberg
External email: Use caution opening links or attachments
Quoted 29 lines · Keith Busch
Quoted 15 lines · Martin K. Petersen
Hi Chaitanya!
Quoted 6 lines
Although I always prefer descriptive error messages, I've not seen even half of these errors in the field. This will :-
1. Increase the size of the driver code.
2. Require constant patching of the driver code when spec is updated, I'm afraid this will turn into churn.Updating the corresponding list in SCSI hasn't been a big problem.
Quoted 2 lines
Also, we can easily track down the error in the NVMe spec from the error code that driver is printing currently, is that not sufficient ?
Quoted 6 lines
Only the block layer status is captured.
In addition, customers are not really armed with the NVMe error decoder ring and are left with no explanation as to why an I/O failed. We have had good results in the field with the verbose SCSI error logging and that's why this patch was modeled to work the same way.
If they are not familiar with the decoder, does seeing something like "Conflicting Attributes" really provide the user a better sense than "
0x280"? Either way, it sounds like they'd just open a bug with someone who is familiar with the spec, in which case numeric codes should be sufficient.Quoted 4 lines
Are the existing nvme trace events not providing enough information? The existing ones appear to have even greater detail than what this patch provides. Or is there some other reason why the kernel log is preferred over the event tracer for capturing these types of errors?
Maybe the string info be added to the trace events? If so, we may need a verbosity level for the trace to record "only errors".
Although the one advantage for the log would be the first time error status (or for generally rare errors).
Quoted 6 lines
On the patch itself:
I'm not sure having it be a new
Kconfigoption is a good idea since it requires a recompile to enable/disable. Could this be toggled with astatic_keyinstead so we may disable/enable verbose logging at runtime?
Quoted 10 lines
Quoted 6 lines
The new "struct
nvme_string_table" is a bit inefficient. Unknown errors will scan the entire table, which will slow down our hot completion path on any error. The codes are small and mostly sequential, so I think you could create a sparse "static const char * const[]", then the "code" becomes the index that can be looked up in constant time. The total space of the array shouldn't be much different than the proposal.100% agree.
Ideally we should document the performance numbers if we are going to add anything in the fast path.
The main purpose of the patch is to get a bit more info into the logs on customer production systems.
What if we drop the string and just display the status, DNR and MORE bits:
[ 56.058003] nvme0n1: Read @ LBA 2048, 1 blocks, Unrecovered Read Error (sct 0x2 / sc 0x81) DNR MORE [ 56.058075] critical medium error, dev nvme0n1, sector 2048 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
This would be extremely useful when agoing through customer logs.
Alan
Quoted 63 lines · Sagi Grimberg
Quoted 22 lines · Keith Busch
Quoted 15 lines · Martin K. Petersen
Hi Chaitanya!
Quoted 6 lines
Although I always prefer descriptive error messages, I've not seen even half of these errors in the field. This will :-
1. Increase the size of the driver code.
2. Require constant patching of the driver code when spec is updated, I'm afraid this will turn into churn.Updating the corresponding list in SCSI hasn't been a big problem.
Quoted 2 lines
Also, we can easily track down the error in the NVMe spec from the error code that driver is printing currently, is that not sufficient ?
Quoted 6 lines
Only the block layer status is captured.
In addition, customers are not really armed with the NVMe error decoder ring and are left with no explanation as to why an I/O failed. We have had good results in the field with the verbose SCSI error logging and that's why this patch was modeled to work the same way.
Quoted 9 lines
If they are not familiar with the decoder, does seeing something like "Conflicting Attributes" really provide the user a better sense than "
0x280"? Either way, it sounds like they'd just open a bug with someone who is familiar with the spec, in which case numeric codes should be sufficient.
Are the existing nvme trace events not providing enough information? The existing ones appear to have even greater detail than what this patch provides. Or is there some other reason why the kernel log is preferred over the event tracer for capturing these types of errors?Maybe the string info be added to the trace events? If so, we may need a verbosity level for the trace to record "only errors".
Although the one advantage for the log would be the first time error status (or for generally rare errors).
Quoted 10 lines
On the patch itself:
I'm not sure having it be a newKconfigoption is a good idea since it requires a recompile to enable/disable. Could this be toggled with astatic_keyinstead so we may disable/enable verbose logging at runtime? The new "structnvme_string_table" is a bit inefficient. Unknown errors will scan the entire table, which will slow down our hot completion path on any error. The codes are small and mostly sequential, so I think you could create a sparse "static const char * const[]", then the "code" becomes the index that can be looked up in constant time. The total space of the array shouldn't be much different than the proposal.100% agree.
Quoted 5 lines
A significant amount of the new strings are specific to admin commands, yet errors for admin commands skip the logging. Why define unreachable strings? Or perhaps a better question: why skip logging admin errors? IO errors can happen in large batches, so perhaps
pr_err()should be ratelimted.Agree.
[email protected]
maintainerQuoted 3 lines · Keith Busch
I'm not sure having it be a new
Kconfigoption is a good idea since it requires a recompile to enable/disable. Could this be toggled with astatic_keyinstead so we may disable/enable verbose logging at runtime?
We need to think about module size. NVMe is not only used in big enterprise setups but also in deeply embedded scenarious these days.
I'm also not really convince of the usefulness (but also not strictly againt it), but if we support it it needs to be conditional.
Sagi Grimberg
maintainerQuoted 10 lines · Keith Busch
Quoted 3 lines
I'm not sure having it be a new
Kconfigoption is a good idea since it requires a recompile to enable/disable. Could this be toggled with astatic_keyinstead so we may disable/enable verbose logging at runtime?We need to think about module size. NVMe is not only used in big enterprise setups but also in deeply embedded scenarious these days.
I'm also not really convince of the usefulness (but also not strictly againt it), but if we support it it needs to be conditional.
Agree, the submitted patch made it conditional though.
Keith,
Quoted 5 lines
If they are not familiar with the decoder, does seeing something like "Conflicting Attributes" really provide the user a better sense than "
0x280"? Either way, it sounds like they'd just open a bug with someone who is familiar with the spec, in which case numeric codes should be sufficient.
An English error string is very valuable in a support situation. It is much easier to discuss the implications of "Unrecoverable Read Error" than some random hex number.
Quoted 4 lines
Are the existing nvme trace events not providing enough information? The existing ones appear to have even greater detail than what this patch provides. Or is there some other reason why the kernel log is preferred over the event tracer for capturing these types of errors?
Logging is imperative. We have to be able to root cause an issue after the fact.
Quoted 4 lines
I'm not sure having it be a new
Kconfigoption is a good idea since it requires a recompile to enable/disable. Could this be toggled with astatic_keyinstead so we may disable/enable verbose logging at runtime?
In SCSI it's a Kconfig option because the embedded folks do not want to waste 8K. We did it the same way here. Enabling verbose logging probably doesn't make much sense unless you're an enterprise distro.
Quoted 6 lines
The new "struct
nvme_string_table" is a bit inefficient. Unknown errors will scan the entire table, which will slow down our hot completion path on any error. The codes are small and mostly sequential, so I think you could create a sparse "static const char * const[]", then the "code" becomes the index that can be looked up in constant time. The total space of the array shouldn't be much different than the proposal.
We can look into that.
Quoted 4 lines
A significant amount of the new strings are specific to admin commands, yet errors for admin commands skip the logging. Why define unreachable strings? Or perhaps a better question: why skip logging admin errors?
Good point.
Quoted 2 lines
IO errors can happen in large batches, so perhaps
pr_err()should be ratelimted.
Sure.
Signature · 1 line
Martin K. Petersen Oracle Linux Engineering