nvme: allow passthru cmd error logging
v6 cover nvme: allow passthru cmd error logging
Hi,
In nvme_end_req() we only log errors which are for non-passthru commands. Add a helper function nvme_log_err_passthru() that allows us to log error for passthru commands by decoding cdw10-cdw15 values of nvme command.
Below is short testlog :-
* Admin Passsthru error log off, no errors are printed
* Admin Passsthru error log on, errors are printed
* IO Passsthru error log off, errors are printed
* IO Passsthru error log on, errors are printed
-ck
Chaitanya Kulkarni (1):
nvme: allow passthru cmd error logging
v6:
- Reabse, etest nvme-6.5 and add test log for admin and I/O passthru error log.
v5:
- Trim down code in the nvme_log_error_passthrough(). Use following to get the disk name as an arg to pr_err_ratelimited() :-
ns ? ns->disk->disk_name : dev_name(nr->ctrl->device),
Use following to get the admin vs I/O opcode string as an arg to
pr_err_ratelimited() :-
ns ? nvme_get_opcode_str(nr->cmd->common.opcode) :
nvme_get_admin_opcode_str(nr->cmd->common.opcode),- Rename nvme_log_error_passthrough() -> nvme_log_err_passthru().
- Remove else and return directly in nvme_passthru_err_log_show().
- Generate error on invalid values of the passthru_enable variable in nvme_passthru_log_store().
- Rename passthrough -> passthru.
- Rename sysfs attr from passthru_admin_err_logging -> passthru_log_err.
v4:
- Change sysfs attribute to passthru_admin_err_logging
- Only log passthrough admin commands. IO passthrough commands will always be logged.
v3:
- Log a passthrough specific message that dumps CDW* contents.
- Enable/disable vis sysfs rather than debugfs.
v2:
- Included Pankaj Raghav's patch 'nvme: ignore starting sector while error logging for passthrough requests'
with a couple changes.
- Moved error_logging flag to nvme_ctrl structure
- The entire nvme-debugfs.c does not need to be guarded by #ifdef CONFIG_FAULT_INJECTION_DEBUG_FS.
- Use IS_ENABLED((CONFIG_NVME_ERROR_LOGGING_DEBUG_FS)) to determine if error logging should be initialized.
- Various other nits.
drivers/nvme/host/core.c | 43 ++++++++++++++++++++++++++++++++++----- drivers/nvme/host/nvme.h | 1 + drivers/nvme/host/sysfs.c | 39 +++++++++++++++++++++++++++++++++++ 3 files changed, 78 insertions(+), 5 deletions(-)
* Admin Passsthru error log off, no errors are printed :-
nvme (nvme-6.5) #
nvme (nvme-6.5) # echo 0 > /sys/class/nvme/nvme0/passthru_err_log
nvme (nvme-6.5) # nvme telemetry-log -o /tmp/test /dev/nvme0
NVMe status: Invalid Field in Command: A reserved coded value or an unsupported value in a defined field(0x4002) Failed to acquire telemetry log 16386!
nvme (nvme-6.5) # cd -/sys/kernel/debug/tracing/events/nvme
nvme # cd -/mnt/data/nvme
nvme (nvme-6.5) # dmesg -c
* Admin Passsthru error log on, errors are printed :-
nvme (nvme-6.5) # echo 1 > /sys/class/nvme/nvme0/passthru_err_log
nvme (nvme-6.5) # nvme telemetry-log -o /tmp/test /dev/nvme0
NVMe status: Invalid Field in Command: A reserved coded value or an unsupported value in a defined field(0x4002) Failed to acquire telemetry log 16386!
nvme (nvme-6.5) # dmesg -c
[ 860.013105] nvme0: Get Log Page(0x2), Invalid Field in Command (sct 0x0 / sc 0x2) DNR cdw10=0x7f0107 cdw11=0x0 cdw12=0x0 cdw13=0x0 cdw14=0x0 cdw15=0x0
nvme (nvme-6.5) #
nvme (nvme-6.5) #
* IO Passsthru error log off, errors are printed :-
nvme (nvme-6.5) # echo 0 > /sys/class/nvme/nvme0/passthru_err_log
nvme (nvme-6.5) # nvme write-zeroes -n 1 -s 0x200000 -c 10 /dev/nvme0
NVMe status: LBA Out of Range: The command references an LBA that exceeds the size of the namespace(0x4080) nvme (nvme-6.5) # dmesg -c
[73675.769162] nvme nvme0: using deprecated NVME_IOCTL_IO_CMD ioctl on the char device! [73675.769233] nvme0n1: Write Zeroes(0x8), LBA Out of Range (sct 0x0 / sc 0x80) DNR cdw10=0x200000 cdw11=0x0 cdw12=0xa cdw13=0x0 cdw14=0x0 cdw15=0x0
* IO Passsthru error log on, errors are printed :-
nvme (nvme-6.5) # echo 1 > /sys/class/nvme/nvme0/passthru_err_log
nvme (nvme-6.5) # nvme write-zeroes -n 1 -s 0x200000 -c 10 /dev/nvme0
NVMe status: LBA Out of Range: The command references an LBA that exceeds the size of the namespace(0x4080) nvme (nvme-6.5) # dmesg -c
[73684.208241] nvme nvme0: using deprecated NVME_IOCTL_IO_CMD ioctl on the char device! [73684.208419] nvme0n1: Write Zeroes(0x8), LBA Out of Range (sct 0x0 / sc 0x80) DNR cdw10=0x200000 cdw11=0x0 cdw12=0xa cdw13=0x0 cdw14=0x0 cdw15=0x0
nvme (nvme-6.5) #
Signature · 1 line
2.40.0
PATCH v6 1/1 nvme: allow passthru cmd error logging
From: Alan Adamson <[email protected]>
Commit d7ac8dca938c ("nvme: quiet user passthrough command errors") disabled error logging for user passthrough commands. This commit adds the ability to opt-in to passthrough admin error logging. IO commands initiated as passthrough will always be logged.
The logging output for passthrough commands (Admin and IO) has been changed to include CDWXX fields.
nvme0n1: Read(0x2), LBA Out of Range (sct 0x0 / sc 0x80) DNR cdw10=0x0 cdw11=0x1
cdw12=0x70000 cdw13=0x0 cdw14=0x0 cdw15=0x0
Add a helper function nvme_log_err_passthru() which allows us to log error for passthru commands by decoding cdw10-cdw15 values of nvme command.
Add a new sysfs attr passthru_err_log that allows user to conditionally enable passthru command logging, by default it is disabled.
To enable passthrough admin error logging:
echo 1 > /sys/class/nvme/nvme0/passthru_err_log
To disable passthrough admin error logging:
echo 0 > /sys/class/nvme/nvme0/passthru_err_log
[kch] fix sevaral nits and trim down code, details in cover-letter.
--- drivers/nvme/host/core.c | 43 ++++++++++++++++++++++++++++++++++----- drivers/nvme/host/nvme.h | 1 + drivers/nvme/host/sysfs.c | 39 +++++++++++++++++++++++++++++++++++ 3 files changed, 78 insertions(+), 5 deletions(-)
drivers/nvme/host/core.c +38 -5
@@ -337,6 +337,30 @@ static void nvme_log_error(struct request *req) nr->status & NVME_SC_DNR ? "DNR " : ""); } +static void nvme_log_err_passthru (struct request *req) +{ +struct nvme_ns *ns = req->q->queuedata; +struct nvme_request *nr =nvme_req (req); + +pr_err_ratelimited ("%s: %s(0x%x), %s (sct 0x%x / sc 0x%x) %s%s" +"cdw10=0x%x cdw11=0x%x cdw12=0x%x cdw13=0x%x cdw14=0x%x cdw15=0x%x\n" , + ns ? ns->disk->disk_name :dev_name (nr->ctrl->device), + ns ?nvme_get_opcode_str (nr->cmd->common.opcode) : +nvme_get_admin_opcode_str (nr->cmd->common.opcode), + nr->cmd->common.opcode, +nvme_get_error_status_str (nr->status), + nr->status >>8 &7 ,/* Status Code Type */ + nr->status &0xff ,/* Status Code */ + nr->status &NVME_SC_MORE ?"MORE " :"" , + nr->status &NVME_SC_DNR ?"DNR " :"" , + nr->cmd->common.cdw10, + nr->cmd->common.cdw11, + nr->cmd->common.cdw12, + nr->cmd->common.cdw13, + nr->cmd->common.cdw14, + nr->cmd->common.cdw14); +} +enum nvme_disposition {COMPLETE ,RETRY ,
@@ -381,8 +405,12 @@ static inline void nvme_end_req(struct request *req) {blk_status_t status =nvme_error_status (nvme_req (req)->status); -if (unlikely (nvme_req (req)->status && !(req->rq_flags &RQF_QUIET ))) -nvme_log_error (req); +if (unlikely (nvme_req (req)->status && !(req->rq_flags &RQF_QUIET ))) { +if (blk_rq_is_passthrough (req)) +nvme_log_err_passthru (req); +else +nvme_log_error (req); + }nvme_end_req_zoned (req);nvme_trace_bio_complete (req);if (req->cmd_flags &REQ_NVME_MPATH )
@@ -666,10 +694,15 @@ static inline void nvme_clear_nvme_request(struct request *req)/* initialize a passthrough request */ void nvme_init_request (struct request *req,struct nvme_command *cmd) { +struct nvme_request *nr =nvme_req (req); +if (req->q->queuedata) req->timeout =NVME_IO_TIMEOUT ; -else /* no queuedata implies admin queue */ +else {/* no queuedata implies admin queue */ req->timeout =NVME_ADMIN_TIMEOUT ; +if (!nr->ctrl->passthru_log_err) + req->rq_flags |=RQF_QUIET ; + }/* passthru commands should let the driver set the SGL flags */ cmd->common.flags &= ~NVME_CMD_SGL_ALL ;
@@ -678,8 +711,7 @@ void nvme_init_request(struct request *req, struct nvme_command *cmd)if (req->mq_hctx->type ==HCTX_TYPE_POLL ) req->cmd_flags |=REQ_POLLED ;nvme_clear_nvme_request (req); - req->rq_flags |=RQF_QUIET ; -memcpy (nvme_req (req)->cmd, cmd,sizeof (*cmd)); +memcpy (nr->cmd, cmd,sizeof (*cmd)); }EXPORT_SYMBOL_GPL (nvme_init_request);
@@ -4483,6 +4515,7 @@ int nvme_init_ctrl(struct nvme_ctrl *ctrl, struct device *dev,int ret; ctrl->state =NVME_CTRL_NEW ; + ctrl->passthru_log_err = false;clear_bit (NVME_CTRL_FAILFAST_EXPIRED , &ctrl->flags);spin_lock_init (&ctrl->lock);mutex_init (&ctrl->scan_lock);
drivers/nvme/host/nvme.h +1 -0
@@ -249,6 +249,7 @@ enum nvme_ctrl_flags {struct nvme_ctrl {bool comp_seen;bool identified; +bool passthru_log_err;enum nvme_ctrl_state state;spinlock_t lock;struct mutex scan_lock;
drivers/nvme/host/sysfs.c +39 -0
@@ -35,6 +35,44 @@ static ssize_t nvme_sysfs_rescan(struct device *dev, } static DEVICE_ATTR(rescan_controller, S_IWUSR, NULL, nvme_sysfs_rescan); +static ssize_t nvme_passthru_err_log_show (struct device *dev, +struct device_attribute *attr,char *buf) +{ +struct nvme_ctrl *ctrl =dev_get_drvdata (dev); + +if (ctrl->passthru_log_err) +return sysfs_emit (buf,"on\n" ); + +return sysfs_emit (buf,"off\n" ); +} + +static ssize_t nvme_passthru_err_log_store (struct device *dev, +struct device_attribute *attr,const char *buf,size_t count) +{ +struct nvme_ctrl *ctrl =dev_get_drvdata (dev); +int passthru_enable, err; + + err =kstrtoint (buf,10 , &passthru_enable); +if (err) +return -EINVAL ; + +switch (passthru_enable) { +case true: +case false: + ctrl->passthru_log_err = passthru_enable; +break ; +default : +pr_err ("invlid value %d for admin error logging [on:1 off:0]\n" , + passthru_enable); +break ; + } +return count; +} + +static DEVICE_ATTR(passthru_err_log, S_IRUGO | S_IWUSR, + nvme_passthru_err_log_show, + nvme_passthru_err_log_store); +static inline struct nvme_ns_head *dev_to_ns_head (struct device *dev) {struct gendisk *disk =dev_to_disk (dev);
@@ -554,6 +592,7 @@ static struct attribute *nvme_dev_attrs[] = { &dev_attr_dhchap_secret.attr, &dev_attr_dhchap_ctrl_secret.attr, #endif + &dev_attr_passthru_err_log.attr,NULL };
Signature · 1 line
2.40.0
Christoph Hellwig
maintainerdrivers/nvme/host/sysfs.c static ssize_t nvme_sysfs_rescan(struct device *dev, · line 46
Quoted 5 lines
drivers/nvme/host/sysfs.c +5
-0+ +if (ctrl->passthru_log_err) +return sysfs_emit (buf,"on\n" ); + +return sysfs_emit (buf,"off\n" );
return sysfs_emit(buf, ctrl->passthru_log_err ? "on" : "off");
Quoted 13 lines
drivers/nvme/host/sysfs.c +13
-0+} + +static ssize_t nvme_passthru_err_log_store (struct device *dev, +struct device_attribute *attr,const char *buf,size_t count) +{ +struct nvme_ctrl *ctrl =dev_get_drvdata (dev); +int passthru_enable, err; + + err =kstrtoint (buf,10 , &passthru_enable); +if (err) +return -EINVAL ; + +switch (passthru_enable) {
drivers/nvme/host/sysfs.c static ssize_t nvme_sysfs_rescan(struct device *dev, · line 69
Quoted 10 lines
drivers/nvme/host/sysfs.c +10
-0+case true: +case false: + ctrl->passthru_log_err = passthru_enable; +break ; +default : +pr_err ("invlid value %d for admin error logging [on:1 off:0]\n" , + passthru_enable); +break ; + } +return count;
This should use kstrtobool.
Also I thik the flag feels a bit too sparse. Why not something like passthru_err_log_enabled that explains what it does?
Quoted 7 lines · Christoph Hellwig
drivers/nvme/host/sysfs.c static ssize_t nvme_sysfs_rescan(struct device *dev, · line 46
Quoted 5 lines
drivers/nvme/host/sysfs.c +5
-0+ +if (ctrl->passthru_log_err) +return sysfs_emit (buf,"on\n" ); + +return sysfs_emit (buf,"off\n" );return sysfs_emit(buf, ctrl->passthru_log_err ? "on" : "off");
okay..
Quoted 18 lines
Quoted 18 lines
drivers/nvme/host/sysfs.c +17
-0+} + +static ssize_t nvme_passthru_err_log_store (struct device *dev, +struct device_attribute *attr,const char *buf,size_t count) +{ +struct nvme_ctrl *ctrl =dev_get_drvdata (dev); +int passthru_enable, err; + + err =kstrtoint (buf,10 , &passthru_enable); +if (err) +return -EINVAL ; + +switch (passthru_enable) { +case true: +case false: + ctrl->passthru_log_err = passthru_enable; +break ;
Quoted 8 lines
drivers/nvme/host/sysfs.c static ssize_t nvme_sysfs_rescan(struct device *dev, · line 69
Quoted 6 lines
drivers/nvme/host/sysfs.c +6
-0+default : +pr_err ("invlid value %d for admin error logging [on:1 off:0]\n" , + passthru_enable); +break ; + } +return count;This should use kstrtobool.
indeed, forget to change it to bool will add to next version ..
Quoted 3 lines
Also I thik the flag feels a bit too sparse. Why not something like
passthru_err_log_enabledthat explains what it does?
sure, I'll send V7 with above changes ..
-ck
Quoted 7 lines · Chaitanya Kulkarni
Hi,
In
nvme_end_req()we only log errors which are for non-passthru commands. Add a helper functionnvme_log_err_passthru()that allows us to log error for passthru commands by decoding cdw10-cdw15 values of nvme command.
Quoted 8 lines
Below is short testlog :-
* Admin Passsthru error log off, no errors are printed
* Admin Passsthru error log on, errors are printed
* IO Passsthru error log off, errors are printed
* IO Passsthru error log on, errors are printed-ck
Sagi, Christoph, Keith,
Can you guys please have a look into this ?
-ck
Quoted 91 lines
Chaitanya Kulkarni (1):
nvme: allow passthru cmd error loggingv6:
- Reabse, etest nvme-6.5 and add test log for admin and I/O passthru error log.v5:
- Trim down code in thenvme_log_error_passthrough(). Use following to get the disk name as an arg topr_err_ratelimited() :- ns ? ns->disk->disk_name : dev_name(nr->ctrl->device), Use following to get the admin vs I/O opcode string as an arg to pr_err_ratelimited() :- ns ? nvme_get_opcode_str(nr->cmd->common.opcode) : nvme_get_admin_opcode_str(nr->cmd->common.opcode),- Rename
nvme_log_error_passthrough()->nvme_log_err_passthru().
- Remove else and return directly innvme_passthru_err_log_show().
- Generate error on invalid values of thepassthru_enablevariable innvme_passthru_log_store().
- Rename passthrough -> passthru.
- Rename sysfs attr frompassthru_admin_err_logging->passthru_log_err.v4:
- Change sysfs attribute topassthru_admin_err_logging
- Only log passthrough admin commands. IO passthrough commands will always be logged.v3:
- Log a passthrough specific message that dumps CDW* contents.
- Enable/disable vis sysfs rather than debugfs.v2:
- Included Pankaj Raghav's patch 'nvme: ignore starting sector while error logging for passthrough requests'
with a couple changes.
- Movederror_loggingflag tonvme_ctrlstructure
- The entirenvme-debugfs.cdoes not need to be guarded by #ifdefCONFIG_FAULT_INJECTION_DEBUG_FS.
- Use IS_ENABLED((CONFIG_NVME_ERROR_LOGGING_DEBUG_FS)) to determine if error logging should be initialized.
- Various other nits.drivers/nvme/host/core.c | 43 ++++++++++++++++++++++++++++++++++----- drivers/nvme/host/nvme.h | 1 + drivers/nvme/host/sysfs.c | 39 +++++++++++++++++++++++++++++++++++ 3 files changed, 78 insertions(+), 5 deletions(-)* Admin Passsthru error log off, no errors are printed :-
nvme (nvme-6.5) #
nvme (nvme-6.5) # echo 0 >/sys/class/nvme/nvme0/passthru_err_log
nvme (nvme-6.5) # nvme telemetry-log -o/tmp/test/dev/nvme0
NVMe status: Invalid Field in Command: A reserved coded value or an unsupported value in a defined field(0x4002) Failed to acquire telemetry log 16386!
nvme (nvme-6.5) # cd -/sys/kernel/debug/tracing/events/nvme
nvme # cd -/mnt/data/nvme
nvme (nvme-6.5) # dmesg -c* Admin Passsthru error log on, errors are printed :-
nvme (nvme-6.5) # echo 1 >
/sys/class/nvme/nvme0/passthru_err_log
nvme (nvme-6.5) # nvme telemetry-log -o/tmp/test/dev/nvme0
NVMe status: Invalid Field in Command: A reserved coded value or an unsupported value in a defined field(0x4002) Failed to acquire telemetry log 16386!
nvme (nvme-6.5) # dmesg -c[ 860.013105] nvme0: Get Log Page(0x2), Invalid Field in Command (sct 0x0 / sc 0x2) DNR cdw10=0x7f0107 cdw11=0x0 cdw12=0x0 cdw13=0x0 cdw14=0x0 cdw15=0x0nvme (nvme-6.5) #
nvme (nvme-6.5) #* IO Passsthru error log off, errors are printed :-
nvme (nvme-6.5) # echo 0 >
/sys/class/nvme/nvme0/passthru_err_log
nvme (nvme-6.5) # nvme write-zeroes -n 1 -s0x200000-c 10/dev/nvme0
NVMe status: LBA Out of Range: The command references an LBA that exceeds the size of the namespace(0x4080) nvme (nvme-6.5) # dmesg -c[73675.769162] nvme nvme0: using deprecated NVME_IOCTL_IO_CMD ioctl on the char device! [73675.769233] nvme0n1: Write Zeroes(0x8), LBA Out of Range (sct 0x0 / sc 0x80) DNR cdw10=0x200000 cdw11=0x0 cdw12=0xa cdw13=0x0 cdw14=0x0 cdw15=0x0* IO Passsthru error log on, errors are printed :-
nvme (nvme-6.5) # echo 1 >
/sys/class/nvme/nvme0/passthru_err_log
nvme (nvme-6.5) # nvme write-zeroes -n 1 -s0x200000-c 10/dev/nvme0
NVMe status: LBA Out of Range: The command references an LBA that exceeds the size of the namespace(0x4080) nvme (nvme-6.5) # dmesg -c[73684.208241] nvme nvme0: using deprecated NVME_IOCTL_IO_CMD ioctl on the char device! [73684.208419] nvme0n1: Write Zeroes(0x8), LBA Out of Range (sct 0x0 / sc 0x80) DNR cdw10=0x200000 cdw11=0x0 cdw12=0xa cdw13=0x0 cdw14=0x0 cdw15=0x0nvme (nvme-6.5) #