TP8028 Rapid Path Failure Recovery
Subsystems (3)
cover TP8028 Rapid Path Failure Recovery
This patchset adds support for TP8028 Rapid Path Failure Recovery for both nvme target and initiator. Rapid Path Failure Recovery brings Cross-Controller Reset (CCR) functionality to nvme. This allows nvme host to send an nvme command to source nvme controller to reset impacted nvme controller. Provided that both source and impacted controllers are in the same nvme subsystem.
The main use of CCR is when one path to nvme subsystem fails. Inflight IOs on impacted nvme controller need to be terminated first before they can be retried on another path. Otherwise data corruption may happen. CCR provides a quick way to terminate these IOs on the unreachable nvme controller allowing recovery to move quickly and avoiding unnecessary delays. In case of CCR is not possible, then inflight requests are held for duration defined by TP4129 KATO Corrections and Clarifications before they are allowed to be retried.
On the target side:
- New struct members have been added to support CCR. struct nvme_id_ctrl has been updated with CIU (Controller Instance Uniquifier), CIRN (Controller Instance Random Number), and CQT (Command Quiesce Time). The combination of CIU, CNTLID, and CIRN is used to identify impacted controller in CCR command.
- CCR nvme command implemented on the target causes impacted controller to fail and drop connections to host.
- CCR logpage contains the status of pending CCR requests. An entry is added to the logpage after CCR request is validated. Completed CCR requests are removed from the logpage when controller becomes ready or when requested in get logpage command.
- An AEN is sent when CCR completes to let the host know that it is safe to retry inflight requests.
On the host side:
- CIU, CIRN, and CQT have been added to struct nvme_ctrl. CIU and CIRN have been added to sysfs to make the values visible to user. CIU and CIRN can be used to construct and manually send admin-passthru CCR commands.
- New controller state NVME_CTRL_RECOVERING has been added to prevent cancelling timed out inflight requests while CCR is in progress. Controller flag NVME_CTRL_RECOVERED was also added to signal end of time-based recovery.
- Controller recovery in nvme_recover_ctrl() is invoked when LIVE controller hits an error or when a request times out. CCR is attempted to reset impacted controller.
- Updated nvme fabric transports nvme-tcp, nvme-rdma, and nvme-fc to use CCR recovery.
Ideally all inflight requests should be held during controller recovery and only retried after recovery is done. However, there are known situations that is not the case in this implementation. These gaps will be addressed in future patches:
- Manual controller reset from sysfs will result in controller going to RESETTING state and all inflight requests to be canceled immediately and maybe retried on another path.
- Manual controller delete from sysfs will also result in all inflight requests to be canceled immediately and maybe retried on another path.
- In nvme-fc nvme controller will be deleted if remote port disappears with no timeout specified. This results in immediate cancellation of requests that maybe retried on another path.
- In nvme-rdma if HCA is removed all nvme controllers will be deleted. This results in canceling inflight IOs and maybe they will be retred on another path.
- In nvme-fc if controller is LIVE and an IO ends with an error from LLDD, only this IO will be completed immediately. However, the rest of inflight IOs will be held correctly because the controller will have transitioned to RECOVERING state.
Mohamed Khalfella (14):
nvmet: Rapid Path Failure Recovery set controller identify fields nvmet/debugfs: Add ctrl uniquifier and random values
nvmet: Implement CCR nvme command
nvmet: Implement CCR logpage
nvmet: Send an AEN on CCR completion
nvme: Rapid Path Failure Recovery read controller identify fields
nvme: Add RECOVERING nvme controller state
nvme: Implement cross-controller reset recovery
nvme: Implement cross-controller reset completion
nvme-tcp: Use CCR to recover controller that hits an error
nvme-rdma: Use CCR to recover controller that hits an error
nvme-fc: Decouple error recovery from controller reset
nvme-fc: Use CCR to recover controller that hits an error
nvme-fc: Hold inflight requests while in RECOVERING state
drivers/nvme/host/constants.c | 1 + drivers/nvme/host/core.c | 197 +++++++++++++++++++++++++++++++- drivers/nvme/host/fc.c | 194 ++++++++++++++++++++----------- drivers/nvme/host/nvme.h | 24 ++++ drivers/nvme/host/rdma.c | 51 +++++++-- drivers/nvme/host/sysfs.c | 24 ++++ drivers/nvme/host/tcp.c | 52 +++++++-- drivers/nvme/target/admin-cmd.c | 127 ++++++++++++++++++++ drivers/nvme/target/core.c | 103 ++++++++++++++++- drivers/nvme/target/debugfs.c | 21 ++++ drivers/nvme/target/nvmet.h | 18 ++- include/linux/nvme.h | 57 ++++++++- 12 files changed, 778 insertions(+), 91 deletions(-)
base-commit: fd95357fd8c6778ac7dea6c57a19b8b182b6e91f
Signature · 1 line
2.51.2
RFC PATCH 1/14 nvmet: Rapid Path Failure Recovery set controller identify fields
TP8028 Rapid Path Failure Recovery defined new fields in controller identify response. The newly defined fields are:
- CIU (Controller Instance UNIQUIFIER): is an 8bit non-zero value that is assigned a random value when controller first created. The value is expected to be incremented when RDY bit in CSTS register is asserted
- CIRN (Controller Instance Random Number): is 64bit random value that gets generated when controller is crated. CIRN is regenerated everytime RDY bit is CSTS register is asserted.
- CCRL (Cross-Controller Reset Limit) is an 8bit value that defines the maximum number of in-progress controller reset operations. CCRL is hardcoded to 4 as recommended by TP8028.
TP4129 KATO Corrections and Clarifications defined CQT (Command Quiesce Time) which is used along with KATO (Keep Alive Timeout) to set an upper time limit for attempting Cross-Controller Recovery.
Make the new fields available for IO controllers only since TP8028 is not very useful for discovery controllers.
--- drivers/nvme/target/admin-cmd.c | 6 ++++++ drivers/nvme/target/core.c | 9 +++++++++ drivers/nvme/target/nvmet.h | 2 ++ include/linux/nvme.h | 15 ++++++++++++--- 4 files changed, 29 insertions(+), 3 deletions(-)
drivers/nvme/target/admin-cmd.c +6 -0
@@ -696,6 +696,12 @@ static void nvmet_execute_identify_ctrl(struct nvmet_req *req) id->cntlid =cpu_to_le16 (ctrl->cntlid); id->ver =cpu_to_le32 (ctrl->subsys->ver); +if (!nvmet_is_disc_subsys (ctrl->subsys)) { + id->cqt =NVMF_CQT_MS ; + id->ciu = ctrl->uniquifier; + id->cirn =cpu_to_le64 (ctrl->random); + id->ccrl =NVMF_CCR_LIMIT ; + }/* XXX: figure out what to do about RTD3R/RTD3 */ id->oaes =cpu_to_le32 (NVMET_AEN_CFG_OPTIONAL );
drivers/nvme/target/core.c +9 -0
@@ -1393,6 +1393,10 @@ static void nvmet_start_ctrl(struct nvmet_ctrl *ctrl)return ; } +if (!nvmet_is_disc_subsys (ctrl->subsys)) { + ctrl->uniquifier = ((u8)(ctrl->uniquifier +1 )) ? :1 ; + ctrl->random =get_random_u64 (); + } ctrl->csts =NVME_CSTS_RDY ;/*
@@ -1662,6 +1666,11 @@ struct nvmet_ctrl *nvmet_alloc_ctrl(struct nvmet_alloc_ctrl_args *args) } ctrl->cntlid = ret; +if (!nvmet_is_disc_subsys (ctrl->subsys)) { + ctrl->uniquifier =get_random_u8 () ? :1 ; + ctrl->random =get_random_u64 (); + } +/* * Discovery controllers may use some arbitrary high value * in order to cleanup stale discovery sessions
drivers/nvme/target/nvmet.h +2 -0
@@ -263,7 +263,9 @@ struct nvmet_ctrl {uuid_t hostid;u16 cntlid; +u8 uniquifier;u32 kato; +u64 random;struct nvmet_port *port;
include/linux/nvme.h +12 -3
@@ -21,6 +21,9 @@#define NVMF_TRADDR_SIZE 256#define NVMF_TSAS_SIZE 256 +#define NVMF_CQT_MS 0 +#define NVMF_CCR_LIMIT 4 +#define NVME_DISC_SUBSYS_NAME "nqn.2014-08.org.nvmexpress.discovery"#define NVME_NSID_ALL 0xffffffff
@@ -328,7 +331,10 @@ struct nvme_id_ctrl {__le16 crdt1;__le16 crdt2;__le16 crdt3; -__u8 rsvd134[122 ]; +__u8 rsvd134[1 ]; +__u8 ciu; +__le64 cirn; +__u8 rsvd144[112 ];__le16 oacs;__u8 acl;__u8 aerl;
@@ -362,7 +368,9 @@ struct nvme_id_ctrl {__u8 anacap;__le32 anagrpmax;__le32 nanagrpid; -__u8 rsvd352[160 ]; +__u8 rsvd352[34 ]; +__le16 cqt; +__u8 rsvd388[124 ];__u8 sqes;__u8 cqes;__le16 maxcmd;
@@ -389,7 +397,8 @@ struct nvme_id_ctrl {__u8 msdbd;__u8 rsvd1804[2 ];__u8 dctype; -__u8 rsvd1807[241 ]; +__u8 ccrl; +__u8 rsvd1808[240 ];struct nvme_id_power_state psd[32 ];__u8 vs[1024 ]; };
Signature · 1 line
2.51.2
Re: [RFC PATCH 01/14] nvmet: Rapid Path Failure Recovery set controller identify fields
Quoted 3 lines · Mohamed Khalfella
TP8028 Rapid Path Failure Recovery defined new fields in controller identify response.
RFC PATCH 2/14 nvmet/debugfs: Add ctrl uniquifier and random values
In reply to Mohamed KhalfellaExport ctrl->random and ctrl->uniquifier as debugfs files under controller debugfs directory.
--- drivers/nvme/target/debugfs.c | 21 +++++++++++++++++++++ 1 file changed, 21 insertions(+)
drivers/nvme/target/debugfs.c +21 -0
@@ -152,6 +152,23 @@ static int nvmet_ctrl_tls_concat_show(struct seq_file *m, void *p) }NVMET_DEBUGFS_ATTR (nvmet_ctrl_tls_concat);#endif +static int nvmet_ctrl_instance_uniquifier_show (struct seq_file *m,void *p) +{ +struct nvmet_ctrl *ctrl = m->private; + +seq_printf (m,"%02x\n" , ctrl->uniquifier); +return 0 ; +} +NVMET_DEBUGFS_ATTR (nvmet_ctrl_instance_uniquifier); + +static int nvmet_ctrl_instance_random_show (struct seq_file *m,void *p) +{ +struct nvmet_ctrl *ctrl = m->private; + +seq_printf (m,"%016llx\n" , ctrl->random); +return 0 ; +} +NVMET_DEBUGFS_ATTR (nvmet_ctrl_instance_random);int nvmet_debugfs_ctrl_setup (struct nvmet_ctrl *ctrl) {
@@ -184,6 +201,10 @@ int nvmet_debugfs_ctrl_setup(struct nvmet_ctrl *ctrl)debugfs_create_file ("tls_key" ,S_IRUSR , ctrl->debugfs_dir, ctrl, &nvmet_ctrl_tls_key_fops);#endif +debugfs_create_file ("uniquifier" ,S_IRUSR , ctrl->debugfs_dir, ctrl, + &nvmet_ctrl_instance_uniquifier_fops); +debugfs_create_file ("random" ,S_IRUSR , ctrl->debugfs_dir, ctrl, + &nvmet_ctrl_instance_random_fops);return 0 ; }
Signature · 1 line
2.51.2
Re: [RFC PATCH 02/14] nvmet/debugfs: Add ctrl uniquifier and random values
Quoted 5 lines · Mohamed Khalfella
Export
ctrl->randomandctrl->uniquifieras debugfs files under controller debugfs directory.Signed-off-by Mohamed Khalfella <[email protected]>
RFC PATCH 3/14 nvmet: Implement CCR nvme command
In reply to Mohamed KhalfellaDefined by TP8028 Rapid Path Failure Recovery, CCR (Cross-Controller Reset) command is an nvme command the is issued to source controller by initiator to reset impacted controller. Implement CCR command for linux nvme target.
--- drivers/nvme/target/admin-cmd.c | 79 +++++++++++++++++++++++++++++++++ drivers/nvme/target/core.c | 69 ++++++++++++++++++++++++++++ drivers/nvme/target/nvmet.h | 13 ++++++ include/linux/nvme.h | 23 ++++++++++ 4 files changed, 184 insertions(+)
drivers/nvme/target/admin-cmd.c +79 -0
@@ -376,7 +376,9 @@ static void nvmet_get_cmd_effects_admin(struct nvmet_ctrl *ctrl, log->acs[nvme_admin_get_features] = log->acs[nvme_admin_async_event] = log->acs[nvme_admin_keep_alive] = + log->acs[nvme_admin_cross_ctrl_reset] =cpu_to_le32 (NVME_CMD_EFFECTS_CSUPP ); + }static void nvmet_get_cmd_effects_nvm (struct nvme_effects_log *log)
@@ -1615,6 +1617,80 @@ void nvmet_execute_keep_alive(struct nvmet_req *req)nvmet_req_complete (req, status); } +void nvmet_execute_cross_ctrl_reset (struct nvmet_req *req) +{ +struct nvmet_ctrl *ictrl, *ctrl = req->sq->ctrl; +struct nvme_command *cmd = req->cmd; +struct nvmet_ccr *ccr, *new_ccr; +int ccr_active, ccr_total; +u16 cntlid, status =0 ; + + cntlid =le16_to_cpu (cmd->ccr.icid); +if (ctrl->cntlid == cntlid) { + req->error_loc = + offsetof(struct nvme_cross_ctrl_reset_cmd , icid); + status =NVME_SC_INVALID_FIELD |NVME_STATUS_DNR ; + goto out; + } + + ictrl =nvmet_ctrl_find_get_ccr (ctrl->subsys, ctrl->hostnqn, + cmd->ccr.ciu, cntlid, +le64_to_cpu (cmd->ccr.cirn)); +if (!ictrl) { +/* Immediate Reset Successful */ +nvmet_set_result (req,1 ); + status =NVME_SC_SUCCESS ; + goto out; + } + + new_ccr =kmalloc (sizeof (*ccr),GFP_KERNEL ); +if (!new_ccr) { + status =NVME_SC_INTERNAL ; + goto out_put_ctrl; + } + + ccr_total = ccr_active =0 ; +mutex_lock (&ctrl->lock); +list_for_each_entry (ccr, &ctrl->ccrs, entry) { +if (ccr->ctrl == ictrl) { + status =NVME_SC_CCR_IN_PROGRESS |NVME_STATUS_DNR ; + goto out_unlock; + } + + ccr_total++; +if (ccr->ctrl) + ccr_active++; + } + +if (ccr_active >=NVMF_CCR_LIMIT ) { + status =NVME_SC_CCR_LIMIT_EXCEEDED ; + goto out_unlock; + } +if (ccr_total >=NVMF_CCR_PER_PAGE ) { + status =NVME_SC_CCR_LOGPAGE_FULL ; + goto out_unlock; + } + + new_ccr->ciu = cmd->ccr.ciu; + new_ccr->icid = cntlid; + new_ccr->ctrl = ictrl; +list_add_tail (&new_ccr->entry, &ctrl->ccrs); +mutex_unlock (&ctrl->lock); + +nvmet_ctrl_fatal_error (ictrl); +nvmet_ctrl_put (ictrl); +nvmet_req_complete (req,0 ); +return ; + +out_unlock: +mutex_unlock (&ctrl->lock); +kfree (new_ccr); +out_put_ctrl: +nvmet_ctrl_put (ictrl); +out: +nvmet_req_complete (req, status); +} +u32 nvmet_admin_cmd_data_len (struct nvmet_req *req) {struct nvme_command *cmd = req->cmd;
@@ -1692,6 +1768,9 @@ u16 nvmet_parse_admin_cmd(struct nvmet_req *req)case nvme_admin_keep_alive: req->execute = nvmet_execute_keep_alive;return 0 ; +case nvme_admin_cross_ctrl_reset: + req->execute = nvmet_execute_cross_ctrl_reset; +return 0 ;default :return nvmet_report_invalid_opcode (req); }
drivers/nvme/target/core.c +69 -0
105 lines not on this page · raw message
drivers/nvme/target/nvmet.h +13 -0
@@ -267,6 +267,7 @@ struct nvmet_ctrl {u32 kato;u64 random; +struct list_head ccrs;struct nvmet_port *port;u32 aen_enabled;
@@ -314,6 +315,13 @@ struct nvmet_ctrl {struct nvmet_pr_log_mgr pr_log_mgr; }; +struct nvmet_ccr { +struct nvmet_ctrl *ctrl; +struct list_head entry; +u16 icid; +u8 ciu; +}; +struct nvmet_subsys {enum nvme_subsys_type type;
@@ -576,6 +584,7 @@ void nvmet_req_free_sgls(struct nvmet_req *req);void nvmet_execute_set_features (struct nvmet_req *req);void nvmet_execute_get_features (struct nvmet_req *req);void nvmet_execute_keep_alive (struct nvmet_req *req); +void nvmet_execute_cross_ctrl_reset (struct nvmet_req *req);u16 nvmet_check_cqid (struct nvmet_ctrl *ctrl,u16 cqid,bool create);u16 nvmet_check_io_cqid (struct nvmet_ctrl *ctrl,u16 cqid,bool create);
@@ -618,6 +627,10 @@ struct nvmet_ctrl *nvmet_alloc_ctrl(struct nvmet_alloc_ctrl_args *args);struct nvmet_ctrl *nvmet_ctrl_find_get (const char *subsysnqn,const char *hostnqn,u16 cntlid,struct nvmet_req *req); +struct nvmet_ctrl *nvmet_ctrl_find_get_ccr (struct nvmet_subsys *subsys, +const char *hostnqn,u8 ciu, +u16 cntlid,u64 cirn); +void nvmet_ctrl_cleanup_ccrs (struct nvmet_ctrl *ctrl,bool all);void nvmet_ctrl_put (struct nvmet_ctrl *ctrl);u16 nvmet_check_ctrl_status (struct nvmet_req *req); ssize_t nvmet_ctrl_host_traddr(struct nvmet_ctrl *ctrl,
include/linux/nvme.h +23 -0
@@ -23,6 +23,7 @@#define NVMF_CQT_MS 0#define NVMF_CCR_LIMIT 4 +#define NVMF_CCR_PER_PAGE 511#define NVME_DISC_SUBSYS_NAME "nqn.2014-08.org.nvmexpress.discovery"
@@ -1225,6 +1226,22 @@ struct nvme_zone_mgmt_recv_cmd {__le32 cdw14[2 ]; }; +struct nvme_cross_ctrl_reset_cmd { +__u8 opcode; +__u8 flags; +__u16 command_id; +__le32 nsid; +__le64 rsvd2[2 ]; +union nvme_data_ptr dptr; +__u8 rsvd10; +__u8 ciu; +__le16 icid; +__le32 cdw11; +__le64 cirn; +__le32 cdw14; +__le32 cdw15; +}; +struct nvme_io_mgmt_recv_cmd {__u8 opcode;__u8 flags;
@@ -1323,6 +1340,7 @@ enum nvme_admin_opcode { nvme_admin_virtual_mgmt =0x1c , nvme_admin_nvme_mi_send =0x1d , nvme_admin_nvme_mi_recv =0x1e , + nvme_admin_cross_ctrl_reset =0x38 , nvme_admin_dbbuf =0x7C , nvme_admin_format_nvm =0x80 , nvme_admin_security_send = 0x81,
@@ -1356,6 +1374,7 @@ enum nvme_admin_opcode {nvme_admin_opcode_name (nvme_admin_virtual_mgmt), \nvme_admin_opcode_name (nvme_admin_nvme_mi_send), \nvme_admin_opcode_name (nvme_admin_nvme_mi_recv), \ +nvme_admin_opcode_name (nvme_admin_cross_ctrl_reset), \nvme_admin_opcode_name (nvme_admin_dbbuf), \nvme_admin_opcode_name (nvme_admin_format_nvm), \ nvme_admin_opcode_name(nvme_admin_security_send), \
@@ -2009,6 +2028,7 @@ struct nvme_command {struct nvme_dbbuf dbbuf;struct nvme_directive_cmd directive;struct nvme_io_mgmt_recv_cmd imr; +struct nvme_cross_ctrl_reset_cmd ccr; }; };
@@ -2173,6 +2193,9 @@ enum {NVME_SC_PMR_SAN_PROHIBITED =0x123 ,NVME_SC_ANA_GROUP_ID_INVALID =0x124 ,NVME_SC_ANA_ATTACH_FAILED =0x125 , +NVME_SC_CCR_IN_PROGRESS =0x13f , +NVME_SC_CCR_LOGPAGE_FULL =0x140 , +NVME_SC_CCR_LIMIT_EXCEEDED =0x141 ,/* * I/O Command Set Specific - NVM commands:
Signature · 1 line
2.51.2
Re: [RFC PATCH 03/14] nvmet: Implement CCR nvme command
Quoted 5 lines · Mohamed Khalfella
Defined by TP8028 Rapid Path Failure Recovery, CCR (Cross-Controller Reset) command is an nvme command the is issued to source controller by initiator to reset impacted controller. Implement CCR command for linux nvme target.
Remove extraneous "the is" in second line.
Quoted 2 lines
Signed-off-by Mohamed Khalfella <[email protected]>
Re: [RFC PATCH 03/14] nvmet: Implement CCR nvme command
Quoted 8 lines · Randy Jennings
Quoted 5 lines · Mohamed Khalfella
Defined by TP8028 Rapid Path Failure Recovery, CCR (Cross-Controller Reset) command is an nvme command the is issued to source controller by initiator to reset impacted controller. Implement CCR command for linux nvme target.
Remove extraneous "the is" in second line.
Removed.
Quoted 5 lines
Quoted 3 lines
Signed-off-by Mohamed Khalfella <[email protected]>Reviewed-by Randy Jennings <[email protected]>
Sagi Grimberg
maintainerRe: [RFC PATCH 03/14] nvmet: Implement CCR nvme command
In reply to Mohamed KhalfellaQuoted 38 lines · Mohamed Khalfella
Defined by TP8028 Rapid Path Failure Recovery, CCR (Cross-Controller Reset) command is an nvme command the is issued to source controller by initiator to reset impacted controller. Implement CCR command for linux nvme target.
Signed-off-by Mohamed Khalfella <[email protected]>--- drivers/nvme/target/admin-cmd.c | 79 +++++++++++++++++++++++++++++++++ drivers/nvme/target/core.c | 69 ++++++++++++++++++++++++++++ drivers/nvme/target/nvmet.h | 13 ++++++ include/linux/nvme.h | 23 ++++++++++ 4 files changed, 184 insertions(+)drivers/nvme/target/admin-cmd.c +9
-0@@ -376,7 +376,9 @@ static void nvmet_get_cmd_effects_admin(struct nvmet_ctrl *ctrl, log->acs[nvme_admin_get_features] = log->acs[nvme_admin_async_event] = log->acs[nvme_admin_keep_alive] = + log->acs[nvme_admin_cross_ctrl_reset] =cpu_to_le32 (NVME_CMD_EFFECTS_CSUPP ); + }static void nvmet_get_cmd_effects_nvm (struct nvme_effects_log *log)@@ -1615,6 +1617,80 @@ void nvmet_execute_keep_alive(struct nvmet_req *req)nvmet_req_complete (req, status); } +void nvmet_execute_cross_ctrl_reset (struct nvmet_req *req) +{ +struct nvmet_ctrl *ictrl, *ctrl = req->sq->ctrl; +struct nvme_command *cmd = req->cmd; +struct nvmet_ccr *ccr, *new_ccr; +int ccr_active, ccr_total; +u16 cntlid, status =0 ;
drivers/nvme/target/admin-cmd.c void nvmet_execute_keep_alive(struct nvmet_req *req) · line 1636
Quoted 10 lines
drivers/nvme/target/admin-cmd.c +10
-0+ + cntlid =le16_to_cpu (cmd->ccr.icid); +if (ctrl->cntlid == cntlid) { + req->error_loc = + offsetof(struct nvme_cross_ctrl_reset_cmd , icid); + status =NVME_SC_INVALID_FIELD |NVME_STATUS_DNR ; + goto out; + } + + ictrl = nvmet_ctrl_find_get_ccr(ctrl->subsys, ctrl->hostnqn,
What does the 'i' stand for?
Quoted 4 lines
drivers/nvme/target/admin-cmd.c +4
-0+ cmd->ccr.ciu, cntlid, + le64_to_cpu(cmd->ccr.cirn)); +if (!ictrl) { +/* Immediate Reset Successful */
drivers/nvme/target/admin-cmd.c void nvmet_execute_keep_alive(struct nvmet_req *req) · line 1650
Quoted 10 lines
drivers/nvme/target/admin-cmd.c +10
-0+nvmet_set_result (req,1 ); + status =NVME_SC_SUCCESS ; + goto out; + } + + new_ccr =kmalloc (sizeof (*ccr),GFP_KERNEL ); +if (!new_ccr) { + status =NVME_SC_INTERNAL ; + goto out_put_ctrl; + }
Allocating this later when you actually use it would probably simplify error path.
Quoted 20 lines
drivers/nvme/target/admin-cmd.c +20
-0+ + ccr_total = ccr_active =0 ; +mutex_lock (&ctrl->lock); +list_for_each_entry (ccr, &ctrl->ccrs, entry) { +if (ccr->ctrl == ictrl) { + status =NVME_SC_CCR_IN_PROGRESS |NVME_STATUS_DNR ; + goto out_unlock; + } + + ccr_total++; +if (ccr->ctrl) + ccr_active++; + } + +if (ccr_active >=NVMF_CCR_LIMIT ) { + status =NVME_SC_CCR_LIMIT_EXCEEDED ; + goto out_unlock; + } +if (ccr_total >=NVMF_CCR_PER_PAGE ) { + status =NVME_SC_CCR_LOGPAGE_FULL ;
drivers/nvme/target/admin-cmd.c void nvmet_execute_keep_alive(struct nvmet_req *req) · line 1680
Quoted 10 lines
drivers/nvme/target/admin-cmd.c +10
-0+ goto out_unlock; + } + + new_ccr->ciu = cmd->ccr.ciu; + new_ccr->icid = cntlid; + new_ccr->ctrl = ictrl; +list_add_tail (&new_ccr->entry, &ctrl->ccrs); +mutex_unlock (&ctrl->lock); + +nvmet_ctrl_fatal_error (ictrl);
Don't you need to wait for it to complete?
e.g. flush_work(&ictrl->fatal_err_work);
Or is that done async? will need to look downstream...
Re: [RFC PATCH 03/14] nvmet: Implement CCR nvme command
Quoted 45 lines · Sagi Grimberg
Quoted 42 lines · Mohamed Khalfella
Defined by TP8028 Rapid Path Failure Recovery, CCR (Cross-Controller Reset) command is an nvme command the is issued to source controller by initiator to reset impacted controller. Implement CCR command for linux nvme target.
Signed-off-by Mohamed Khalfella <[email protected]>--- drivers/nvme/target/admin-cmd.c | 79 +++++++++++++++++++++++++++++++++ drivers/nvme/target/core.c | 69 ++++++++++++++++++++++++++++ drivers/nvme/target/nvmet.h | 13 ++++++ include/linux/nvme.h | 23 ++++++++++ 4 files changed, 184 insertions(+)drivers/nvme/target/admin-cmd.c +13
-0@@ -376,7 +376,9 @@ static void nvmet_get_cmd_effects_admin(struct nvmet_ctrl *ctrl, log->acs[nvme_admin_get_features] = log->acs[nvme_admin_async_event] = log->acs[nvme_admin_keep_alive] = + log->acs[nvme_admin_cross_ctrl_reset] =cpu_to_le32 (NVME_CMD_EFFECTS_CSUPP ); + }static void nvmet_get_cmd_effects_nvm (struct nvme_effects_log *log)@@ -1615,6 +1617,80 @@ void nvmet_execute_keep_alive(struct nvmet_req *req)nvmet_req_complete (req, status); } +void nvmet_execute_cross_ctrl_reset (struct nvmet_req *req) +{ +struct nvmet_ctrl *ictrl, *ctrl = req->sq->ctrl; +struct nvme_command *cmd = req->cmd; +struct nvmet_ccr *ccr, *new_ccr; +int ccr_active, ccr_total; +u16 cntlid, status =0 ; + + cntlid =le16_to_cpu (cmd->ccr.icid); +if (ctrl->cntlid == cntlid) { + req->error_loc =
Quoted 8 lines
drivers/nvme/target/admin-cmd.c void nvmet_execute_keep_alive(struct nvmet_req *req) · line 1636
Quoted 6 lines
drivers/nvme/target/admin-cmd.c +6
-0+ offsetof(struct nvme_cross_ctrl_reset_cmd , icid); + status =NVME_SC_INVALID_FIELD |NVME_STATUS_DNR ; + goto out; + } + + ictrl = nvmet_ctrl_find_get_ccr(ctrl->subsys, ctrl->hostnqn,What does the 'i' stand for?
'i' stands for impacted controller. Also, if you see sctrl the 's' stands for source controller. These terms are from TP8028.
Quoted 10 lines
Quoted 10 lines
drivers/nvme/target/admin-cmd.c +9
-0+ cmd->ccr.ciu, cntlid, + le64_to_cpu(cmd->ccr.cirn)); +if (!ictrl) { +/* Immediate Reset Successful */ +nvmet_set_result (req,1 ); + status =NVME_SC_SUCCESS ; + goto out; + } +
Quoted 8 lines
drivers/nvme/target/admin-cmd.c void nvmet_execute_keep_alive(struct nvmet_req *req) · line 1650
Quoted 5 lines
drivers/nvme/target/admin-cmd.c +5
-0+ new_ccr =kmalloc (sizeof (*ccr),GFP_KERNEL ); +if (!new_ccr) { + status =NVME_SC_INTERNAL ; + goto out_put_ctrl; + }Allocating this later when you actually use it would probably simplify error path.
Right, it will save us kfree(). Will do that.
Quoted 35 lines
Quoted 21 lines
drivers/nvme/target/admin-cmd.c +20
-0+ + ccr_total = ccr_active =0 ; +mutex_lock (&ctrl->lock); +list_for_each_entry (ccr, &ctrl->ccrs, entry) { +if (ccr->ctrl == ictrl) { + status =NVME_SC_CCR_IN_PROGRESS |NVME_STATUS_DNR ; + goto out_unlock; + } + + ccr_total++; +if (ccr->ctrl) + ccr_active++; + } + +if (ccr_active >=NVMF_CCR_LIMIT ) { + status =NVME_SC_CCR_LIMIT_EXCEEDED ; + goto out_unlock; + } +if (ccr_total >=NVMF_CCR_PER_PAGE ) { + status =NVME_SC_CCR_LOGPAGE_FULL ;drivers/nvme/target/admin-cmd.c void nvmet_execute_keep_alive(struct nvmet_req *req) · line 1680
Quoted 10 lines
drivers/nvme/target/admin-cmd.c +10
-0+ goto out_unlock; + } + + new_ccr->ciu = cmd->ccr.ciu; + new_ccr->icid = cntlid; + new_ccr->ctrl = ictrl; +list_add_tail (&new_ccr->entry, &ctrl->ccrs); +mutex_unlock (&ctrl->lock); + +nvmet_ctrl_fatal_error (ictrl);Don't you need to wait for it to complete?
e.g.flush_work(&ictrl->fatal_err_work);
Quoted 1 line
Or is that done async? will need to look downstream...
No, we do not need to wait for ictrl->fatal_err_work to complete. An AEN will be sent when ictrl exits. It is okay if AEN is sent before CCR request is completed. The initiator should expect this behavior and deal with it.
Sagi Grimberg
maintainerRe: [RFC PATCH 03/14] nvmet: Implement CCR nvme command
Quoted 19 lines
Quoted 11 lines
Quoted 11 lines
drivers/nvme/target/admin-cmd.c +10
-0+void nvmet_execute_cross_ctrl_reset (struct nvmet_req *req) +{ +struct nvmet_ctrl *ictrl, *ctrl = req->sq->ctrl; +struct nvme_command *cmd = req->cmd; +struct nvmet_ccr *ccr, *new_ccr; +int ccr_active, ccr_total; +u16 cntlid, status =0 ; + + cntlid =le16_to_cpu (cmd->ccr.icid); +if (ctrl->cntlid == cntlid) {Quoted 8 lines
drivers/nvme/target/admin-cmd.c void nvmet_execute_keep_alive(struct nvmet_req *req) · line 1636
Quoted 7 lines
drivers/nvme/target/admin-cmd.c +7
-0+ req->error_loc = + offsetof(struct nvme_cross_ctrl_reset_cmd , icid); + status =NVME_SC_INVALID_FIELD |NVME_STATUS_DNR ; + goto out; + } + + ictrl = nvmet_ctrl_find_get_ccr(ctrl->subsys, ctrl->hostnqn,What does the 'i' stand for?
Quoted 2 lines
'i' stands for impacted controller. Also, if you see sctrl the 's' stands for source controller. These terms are from TP8028.
Can you perhaps add a comment on this?
Quoted 10 lines
Quoted 10 lines
drivers/nvme/target/admin-cmd.c void nvmet_execute_keep_alive(struct nvmet_req *req) · line 1680
Quoted 7 lines
drivers/nvme/target/admin-cmd.c +7
-0+ new_ccr->ciu = cmd->ccr.ciu; + new_ccr->icid = cntlid; + new_ccr->ctrl = ictrl; +list_add_tail (&new_ccr->entry, &ctrl->ccrs); +mutex_unlock (&ctrl->lock); + +nvmet_ctrl_fatal_error (ictrl);Don't you need to wait for it to complete?
e.g.flush_work(&ictrl->fatal_err_work);
Quoted 5 lines
Quoted 1 line
Or is that done async? will need to look downstream...
No, we do not need to wait for
ictrl->fatal_err_workto complete. An AEN will be sent when ictrl exits. It is okay if AEN is sent before CCR request is completed. The initiator should expect this behavior and deal with it.
Yes, saw that in a later patch (didn't get to do a full review yet)
Re: [RFC PATCH 03/14] nvmet: Implement CCR nvme command
Quoted 19 lines · Sagi Grimberg
Quoted 19 lines
Quoted 19 lines
Quoted 8 lines
drivers/nvme/target/admin-cmd.c +7
-0+void nvmet_execute_cross_ctrl_reset (struct nvmet_req *req) +{ +struct nvmet_ctrl *ictrl, *ctrl = req->sq->ctrl; +struct nvme_command *cmd = req->cmd; +struct nvmet_ccr *ccr, *new_ccr; +int ccr_active, ccr_total; +u16 cntlid, status =0 ;drivers/nvme/target/admin-cmd.c void nvmet_execute_keep_alive(struct nvmet_req *req) · line 1636
Quoted 10 lines
drivers/nvme/target/admin-cmd.c +10
-0+ + cntlid =le16_to_cpu (cmd->ccr.icid); +if (ctrl->cntlid == cntlid) { + req->error_loc = + offsetof(struct nvme_cross_ctrl_reset_cmd , icid); + status =NVME_SC_INVALID_FIELD |NVME_STATUS_DNR ; + goto out; + } + + ictrl = nvmet_ctrl_find_get_ccr(ctrl->subsys, ctrl->hostnqn,What does the 'i' stand for?
Quoted 4 lines
Quoted 2 lines
'i' stands for impacted controller. Also, if you see sctrl the 's' stands for source controller. These terms are from TP8028.
Can you perhaps add a comment on this?
Okay, will do that.
Quoted 18 lines
Quoted 11 lines
Quoted 11 lines
drivers/nvme/target/admin-cmd.c void nvmet_execute_keep_alive(struct nvmet_req *req) · line 1680
Quoted 8 lines
drivers/nvme/target/admin-cmd.c +7
-0+ new_ccr->ciu = cmd->ccr.ciu; + new_ccr->icid = cntlid; + new_ccr->ctrl = ictrl; +list_add_tail (&new_ccr->entry, &ctrl->ccrs); +mutex_unlock (&ctrl->lock); + +nvmet_ctrl_fatal_error (ictrl);Don't you need to wait for it to complete?
e.g.flush_work(&ictrl->fatal_err_work);Quoted 5 lines
Quoted 1 line
Or is that done async? will need to look downstream...
No, we do not need to wait for
ictrl->fatal_err_workto complete. An AEN will be sent when ictrl exits. It is okay if AEN is sent before CCR request is completed. The initiator should expect this behavior and deal with it.Yes, saw that in a later patch (didn't get to do a full review yet)
RFC PATCH 4/14 nvmet: Implement CCR logpage
In reply to Mohamed KhalfellaDefined by TP8028 Rapid Path Failure Recovery, CCR (Cross-Controller Reset) log page contains an entry for each CCR request submitted to source controller. Implement CCR logpage for nvme linux target.
--- drivers/nvme/target/admin-cmd.c | 42 +++++++++++++++++++++++++++++++++ include/linux/nvme.h | 16 +++++++++++++ 2 files changed, 58 insertions(+)
drivers/nvme/target/admin-cmd.c +42 -0
@@ -220,6 +220,7 @@ static void nvmet_execute_get_supported_log_pages(struct nvmet_req *req) logs->lids[NVME_LOG_FEATURES ] =cpu_to_le32 (NVME_LIDS_LSUPP ); logs->lids[NVME_LOG_RMI ] =cpu_to_le32 (NVME_LIDS_LSUPP ); logs->lids[NVME_LOG_RESERVATION ] =cpu_to_le32 (NVME_LIDS_LSUPP ); + logs->lids[NVME_LOG_CCR ] =cpu_to_le32 (NVME_LIDS_LSUPP ); status =nvmet_copy_to_sgl (req,0 , logs,sizeof (*logs));kfree (logs);
@@ -608,6 +609,45 @@ static void nvmet_execute_get_log_page_features(struct nvmet_req *req)nvmet_req_complete (req, status); } +static void nvmet_execute_get_log_page_ccr (struct nvmet_req *req) +{ +struct nvmet_ctrl *ctrl = req->sq->ctrl; +struct nvmet_ccr *ccr; +struct nvme_ccr_log *log; +int index =0 ; +u16 status; + + log =kzalloc (sizeof (*log),GFP_KERNEL ); +if (!log) { + status =NVME_SC_INTERNAL ; + goto out; + } + +mutex_lock (&ctrl->lock); +list_for_each_entry (ccr, &ctrl->ccrs, entry) { + log->entries[index].icid =cpu_to_le16 (ccr->icid); + log->entries[index].ciu = ccr->ciu; + log->entries[index].acid =cpu_to_le16 (0xffff ); + +/* If ccr->ctrl is NULL then we know reset succeeded */ + log->entries[index].ccrs = ccr->ctrl ?0x00 :0x01 ; + log->entries[index].ccrf =0x03 ;/* Validated and Initiated */ + index++; + } + +/* Cleanup completed CCRs if requested */ +if (req->cmd->get_log_page.lsp &0x1 ) +nvmet_ctrl_cleanup_ccrs (ctrl, false); +mutex_unlock (&ctrl->lock); + + log->ne =cpu_to_le16 (index); +nvmet_clear_aen_bit (req,NVME_AEN_BIT_CCR_COMPLETE ); + status =nvmet_copy_to_sgl (req,0 , log,sizeof (*log)); +kfree (log); +out: +nvmet_req_complete (req, status); +} +static void nvmet_execute_get_log_page (struct nvmet_req *req) {if (!nvmet_check_transfer_len (req,nvmet_get_log_page_len (req->cmd)))
@@ -641,6 +681,8 @@ static void nvmet_execute_get_log_page(struct nvmet_req *req)return nvmet_execute_get_log_page_rmi (req);case NVME_LOG_RESERVATION :return nvmet_execute_get_log_page_resv (req); +case NVME_LOG_CCR : +return nvmet_execute_get_log_page_ccr (req); }pr_debug ("unhandled lid %d on qid %d\n" , req->cmd->get_log_page.lid, req->sq->qid);
include/linux/nvme.h +16 -0
@@ -1435,6 +1435,7 @@ enum {NVME_LOG_FDP_CONFIGS =0x20 ,NVME_LOG_DISC =0x70 ,NVME_LOG_RESERVATION =0x80 , +NVME_LOG_CCR =0x1E ,NVME_FWACT_REPL = (0 <<3 ),NVME_FWACT_REPL_ACTV = (1 <<3 ), NVME_FWACT_ACTV = (2 << 3),
@@ -1458,6 +1459,21 @@ enum { NVME_FIS_CSCPE = 1 << 21, }; +struct nvme_ccr_log_entry { +__le16 icid; +__u8 ciu; +__u8 rsvd3; +__le16 acid; +__u8 ccrs; +__u8 ccrf; +}; + +struct nvme_ccr_log { +__le16 ne; +__u8 rsvd2[6 ]; +struct nvme_ccr_log_entry entries[NVMF_CCR_PER_PAGE ]; +}; +/* NVMe Namespace Write Protect State */ enum {NVME_NS_NO_WRITE_PROTECT =0 ,
Signature · 1 line
2.51.2
Re: [RFC PATCH 04/14] nvmet: Implement CCR logpage
Quoted 6 lines · Mohamed Khalfella
Defined by TP8028 Rapid Path Failure Recovery, CCR (Cross-Controller Reset) log page contains an entry for each CCR request submitted to source controller. Implement CCR logpage for nvme linux target.
Signed-off-by Mohamed Khalfella <[email protected]>
RFC PATCH 5/14 nvmet: Send an AEN on CCR completion
In reply to Mohamed KhalfellaSend an AEN to initiator when impacted controller exists. The notification points to CCR log page that initiator can read to check which CCR operation completed.
--- drivers/nvme/target/core.c | 27 +++++++++++++++++++++++---- drivers/nvme/target/nvmet.h | 3 ++- include/linux/nvme.h | 3 +++ 3 files changed, 28 insertions(+), 5 deletions(-)
drivers/nvme/target/core.c +23 -4
@@ -202,7 +202,7 @@ static void nvmet_async_event_work(struct work_struct *work)nvmet_async_events_process (ctrl); } -void nvmet_add_async_event (struct nvmet_ctrl *ctrl,u8 event_type, +static void nvmet_add_async_event_locked (struct nvmet_ctrl *ctrl,u8 event_type,u8 event_info,u8 log_page) {struct nvmet_async_event *aen;
@@ -215,12 +215,17 @@ void nvmet_add_async_event(struct nvmet_ctrl *ctrl, u8 event_type, aen->event_info = event_info; aen->log_page = log_page; -mutex_lock (&ctrl->lock);list_add_tail (&aen->entry, &ctrl->async_events); -mutex_unlock (&ctrl->lock);queue_work (nvmet_wq, &ctrl->async_event_work); } +void nvmet_add_async_event (struct nvmet_ctrl *ctrl,u8 event_type, +u8 event_info,u8 log_page) +{ +mutex_lock (&ctrl->lock); +nvmet_add_async_event_locked (ctrl, event_type, event_info, log_page); +mutex_unlock (&ctrl->lock); +}static void nvmet_add_to_changed_ns_log (struct nvmet_ctrl *ctrl,__le32 nsid) {
@@ -1788,6 +1793,18 @@ struct nvmet_ctrl *nvmet_alloc_ctrl(struct nvmet_alloc_ctrl_args *args) }EXPORT_SYMBOL_GPL (nvmet_alloc_ctrl); +static void nvmet_ctrl_notify_ccr (struct nvmet_ctrl *ctrl) +{ +lockdep_assert_held (&ctrl->lock); + +if (nvmet_aen_bit_disabled (ctrl,NVME_AEN_BIT_CCR_COMPLETE )) +return ; + +nvmet_add_async_event_locked (ctrl,NVME_AER_NOTICE , +NVME_AER_NOTICE_CCR_COMPLETED , +NVME_LOG_CCR ); +} +static void nvmet_ctrl_complete_pending_ccr (struct nvmet_ctrl *ctrl) {struct nvmet_subsys *subsys = ctrl->subsys;
@@ -1801,8 +1818,10 @@ static void nvmet_ctrl_complete_pending_ccr(struct nvmet_ctrl *ctrl)list_for_each_entry (sctrl, &subsys->ctrls, subsys_entry) {mutex_lock (&sctrl->lock);list_for_each_entry (ccr, &sctrl->ccrs, entry) { -if (ccr->ctrl == ctrl) +if (ccr->ctrl == ctrl) { +nvmet_ctrl_notify_ccr (sctrl); ccr->ctrl =NULL ; + } }mutex_unlock (&sctrl->lock); }
drivers/nvme/target/nvmet.h +2 -1
@@ -44,7 +44,8 @@* Supported optional AENs: */ #define NVMET_AEN_CFG_OPTIONAL \ - (NVME_AEN_CFG_NS_ATTR | NVME_AEN_CFG_ANA_CHANGE) + (NVME_AEN_CFG_NS_ATTR | NVME_AEN_CFG_ANA_CHANGE | \ + NVME_AEN_CFG_CCR_COMPLETE)#define NVMET_DISC_AEN_CFG_OPTIONAL \ (NVME_AEN_CFG_DISC_CHANGE)
include/linux/nvme.h +3 -0
@@ -863,12 +863,14 @@ enum {NVME_AER_NOTICE_FW_ACT_STARTING =0x01 ,NVME_AER_NOTICE_ANA =0x03 ,NVME_AER_NOTICE_DISC_CHANGED =0xf0 , + NVME_AER_NOTICE_CCR_COMPLETED = 0xf4, };enum {NVME_AEN_BIT_NS_ATTR =8 ,NVME_AEN_BIT_FW_ACT =9 ,NVME_AEN_BIT_ANA_CHANGE =11 , +NVME_AEN_BIT_CCR_COMPLETE =20 ,NVME_AEN_BIT_DISC_CHANGE =31 , };
@@ -876,6 +878,7 @@ enum {NVME_AEN_CFG_NS_ATTR =1 <<NVME_AEN_BIT_NS_ATTR ,NVME_AEN_CFG_FW_ACT =1 <<NVME_AEN_BIT_FW_ACT ,NVME_AEN_CFG_ANA_CHANGE =1 <<NVME_AEN_BIT_ANA_CHANGE , +NVME_AEN_CFG_CCR_COMPLETE =1 <<NVME_AEN_BIT_CCR_COMPLETE , NVME_AEN_CFG_DISC_CHANGE = 1 << NVME_AEN_BIT_DISC_CHANGE, };
Signature · 1 line
2.51.2
Re: [RFC PATCH 05/14] nvmet: Send an AEN on CCR completion
Quoted 6 lines · Mohamed Khalfella
Send an AEN to initiator when impacted controller exists. The notification points to CCR log page that initiator can read to check which CCR operation completed.
Signed-off-by Mohamed Khalfella <[email protected]>
Sagi Grimberg
maintainerRe: [RFC PATCH 05/14] nvmet: Send an AEN on CCR completion
In reply to Mohamed KhalfellaQuoted 64 lines · Mohamed Khalfella
Send an AEN to initiator when impacted controller exists. The notification points to CCR log page that initiator can read to check which CCR operation completed.
Signed-off-by Mohamed Khalfella <[email protected]>--- drivers/nvme/target/core.c | 27 +++++++++++++++++++++++---- drivers/nvme/target/nvmet.h | 3 ++- include/linux/nvme.h | 3 +++ 3 files changed, 28 insertions(+), 5 deletions(-)drivers/nvme/target/core.c +20
-3@@ -202,7 +202,7 @@ static void nvmet_async_event_work(struct work_struct *work)nvmet_async_events_process (ctrl); } -void nvmet_add_async_event (struct nvmet_ctrl *ctrl,u8 event_type, +static void nvmet_add_async_event_locked (struct nvmet_ctrl *ctrl,u8 event_type,u8 event_info,u8 log_page) {struct nvmet_async_event *aen;@@ -215,12 +215,17 @@ void nvmet_add_async_event(struct nvmet_ctrl *ctrl, u8 event_type, aen->event_info = event_info; aen->log_page = log_page; -mutex_lock (&ctrl->lock);list_add_tail (&aen->entry, &ctrl->async_events); -mutex_unlock (&ctrl->lock);queue_work (nvmet_wq, &ctrl->async_event_work); } +void nvmet_add_async_event (struct nvmet_ctrl *ctrl,u8 event_type, +u8 event_info,u8 log_page) +{ +mutex_lock (&ctrl->lock); +nvmet_add_async_event_locked (ctrl, event_type, event_info, log_page); +mutex_unlock (&ctrl->lock); +}static void nvmet_add_to_changed_ns_log (struct nvmet_ctrl *ctrl,__le32 nsid) {@@ -1788,6 +1793,18 @@ struct nvmet_ctrl *nvmet_alloc_ctrl(struct nvmet_alloc_ctrl_args *args) }EXPORT_SYMBOL_GPL (nvmet_alloc_ctrl); +static void nvmet_ctrl_notify_ccr (struct nvmet_ctrl *ctrl) +{ +lockdep_assert_held (&ctrl->lock); + +if (nvmet_aen_bit_disabled (ctrl,NVME_AEN_BIT_CCR_COMPLETE )) +return ; + +nvmet_add_async_event_locked (ctrl,NVME_AER_NOTICE , +NVME_AER_NOTICE_CCR_COMPLETED , +NVME_LOG_CCR ); +} +static void nvmet_ctrl_complete_pending_ccr (struct nvmet_ctrl *ctrl) {struct nvmet_subsys *subsys = ctrl->subsys;
drivers/nvme/target/core.c static void nvmet_ctrl_complete_pending_ccr(struct nvmet_ctrl *ctrl) · line 1824
Quoted 8 lines
drivers/nvme/target/core.c +3
-1@@ -1801,8 +1818,10 @@ static void nvmet_ctrl_complete_pending_ccr(struct nvmet_ctrl *ctrl)list_for_each_entry (sctrl, &subsys->ctrls, subsys_entry) {mutex_lock (&sctrl->lock);list_for_each_entry (ccr, &sctrl->ccrs, entry) { -if (ccr->ctrl == ctrl) +if (ccr->ctrl == ctrl) { +nvmet_ctrl_notify_ccr (sctrl); ccr->ctrl =NULL ; + }
Is this double loop necessary? Would you have more than one controller cross resetting the same controller? Won't it be better to install a callback+opaque that the controller removal will call?
Re: [RFC PATCH 05/14] nvmet: Send an AEN on CCR completion
Quoted 70 lines · Sagi Grimberg
Quoted 67 lines · Mohamed Khalfella
Send an AEN to initiator when impacted controller exists. The notification points to CCR log page that initiator can read to check which CCR operation completed.
Signed-off-by Mohamed Khalfella <[email protected]>--- drivers/nvme/target/core.c | 27 +++++++++++++++++++++++---- drivers/nvme/target/nvmet.h | 3 ++- include/linux/nvme.h | 3 +++ 3 files changed, 28 insertions(+), 5 deletions(-)drivers/nvme/target/core.c +20
-3@@ -202,7 +202,7 @@ static void nvmet_async_event_work(struct work_struct *work)nvmet_async_events_process (ctrl); } -void nvmet_add_async_event (struct nvmet_ctrl *ctrl,u8 event_type, +static void nvmet_add_async_event_locked (struct nvmet_ctrl *ctrl,u8 event_type,u8 event_info,u8 log_page) {struct nvmet_async_event *aen;@@ -215,12 +215,17 @@ void nvmet_add_async_event(struct nvmet_ctrl *ctrl, u8 event_type, aen->event_info = event_info; aen->log_page = log_page; -mutex_lock (&ctrl->lock);list_add_tail (&aen->entry, &ctrl->async_events); -mutex_unlock (&ctrl->lock);queue_work (nvmet_wq, &ctrl->async_event_work); } +void nvmet_add_async_event (struct nvmet_ctrl *ctrl,u8 event_type, +u8 event_info,u8 log_page) +{ +mutex_lock (&ctrl->lock); +nvmet_add_async_event_locked (ctrl, event_type, event_info, log_page); +mutex_unlock (&ctrl->lock); +}static void nvmet_add_to_changed_ns_log (struct nvmet_ctrl *ctrl,__le32 nsid) {@@ -1788,6 +1793,18 @@ struct nvmet_ctrl *nvmet_alloc_ctrl(struct nvmet_alloc_ctrl_args *args) }EXPORT_SYMBOL_GPL (nvmet_alloc_ctrl); +static void nvmet_ctrl_notify_ccr (struct nvmet_ctrl *ctrl) +{ +lockdep_assert_held (&ctrl->lock); + +if (nvmet_aen_bit_disabled (ctrl,NVME_AEN_BIT_CCR_COMPLETE )) +return ; + +nvmet_add_async_event_locked (ctrl,NVME_AER_NOTICE , +NVME_AER_NOTICE_CCR_COMPLETED , +NVME_LOG_CCR ); +} +static void nvmet_ctrl_complete_pending_ccr (struct nvmet_ctrl *ctrl) {struct nvmet_subsys *subsys = ctrl->subsys;@@ -1801,8 +1818,10 @@ static void nvmet_ctrl_complete_pending_ccr(struct nvmet_ctrl *ctrl)list_for_each_entry (sctrl, &subsys->ctrls, subsys_entry) {mutex_lock (&sctrl->lock);list_for_each_entry (ccr, &sctrl->ccrs, entry) {
Quoted 8 lines
drivers/nvme/target/core.c static void nvmet_ctrl_complete_pending_ccr(struct nvmet_ctrl *ctrl) · line 1824
Quoted 5 lines
drivers/nvme/target/core.c +3
-1-if (ccr->ctrl == ctrl) +if (ccr->ctrl == ctrl) { +nvmet_ctrl_notify_ccr (sctrl); ccr->ctrl =NULL ; + }Is this double loop necessary? Would you have more than one controller cross resetting the same
As it is implemented now CCRs are linked to sctrl. This decision can be revisited if found suboptimal. At some point I had CCRs linked to ctrl->subsys but that led to lock ordering issues. Double loop is necessary to find all CCRs in all controllers and mark them done. Yes, it is possible to have more than one sctrl resetting the same ictrl.
Quoted 2 lines
controller? Won't it be better to install a callback+opaque that the controller removal will call?
Can you elaborate more on that? Better in what terms?
nvmet_ctrl_complete_pending_ccr() is called from nvmet_ctrl_free() when we know that ctrl->ref is zero and no new CCRs will be added to this controller because nvmet_ctrl_find_get_ccr() will not be able to get it.
Sagi Grimberg
maintainerRe: [RFC PATCH 05/14] nvmet: Send an AEN on CCR completion
Quoted 77 lines · Mohamed Khalfella
Quoted 68 lines · Sagi Grimberg
Quoted 66 lines · Mohamed Khalfella
Send an AEN to initiator when impacted controller exists. The notification points to CCR log page that initiator can read to check which CCR operation completed.
Signed-off-by Mohamed Khalfella <[email protected]>--- drivers/nvme/target/core.c | 27 +++++++++++++++++++++++---- drivers/nvme/target/nvmet.h | 3 ++- include/linux/nvme.h | 3 +++ 3 files changed, 28 insertions(+), 5 deletions(-)drivers/nvme/target/core.c +20
-3@@ -202,7 +202,7 @@ static void nvmet_async_event_work(struct work_struct *work)nvmet_async_events_process (ctrl); } -void nvmet_add_async_event (struct nvmet_ctrl *ctrl,u8 event_type, +static void nvmet_add_async_event_locked (struct nvmet_ctrl *ctrl,u8 event_type,u8 event_info,u8 log_page) {struct nvmet_async_event *aen;@@ -215,12 +215,17 @@ void nvmet_add_async_event(struct nvmet_ctrl *ctrl, u8 event_type, aen->event_info = event_info; aen->log_page = log_page; -mutex_lock (&ctrl->lock);list_add_tail (&aen->entry, &ctrl->async_events); -mutex_unlock (&ctrl->lock);queue_work (nvmet_wq, &ctrl->async_event_work); } +void nvmet_add_async_event (struct nvmet_ctrl *ctrl,u8 event_type, +u8 event_info,u8 log_page) +{ +mutex_lock (&ctrl->lock); +nvmet_add_async_event_locked (ctrl, event_type, event_info, log_page); +mutex_unlock (&ctrl->lock); +}static void nvmet_add_to_changed_ns_log (struct nvmet_ctrl *ctrl,__le32 nsid) {@@ -1788,6 +1793,18 @@ struct nvmet_ctrl *nvmet_alloc_ctrl(struct nvmet_alloc_ctrl_args *args) }EXPORT_SYMBOL_GPL (nvmet_alloc_ctrl); +static void nvmet_ctrl_notify_ccr (struct nvmet_ctrl *ctrl) +{ +lockdep_assert_held (&ctrl->lock); + +if (nvmet_aen_bit_disabled (ctrl,NVME_AEN_BIT_CCR_COMPLETE )) +return ; + +nvmet_add_async_event_locked (ctrl,NVME_AER_NOTICE , +NVME_AER_NOTICE_CCR_COMPLETED , +NVME_LOG_CCR ); +} +static void nvmet_ctrl_complete_pending_ccr (struct nvmet_ctrl *ctrl) {struct nvmet_subsys *subsys = ctrl->subsys;@@ -1801,8 +1818,10 @@ static void nvmet_ctrl_complete_pending_ccr(struct nvmet_ctrl *ctrl)list_for_each_entry (sctrl, &subsys->ctrls, subsys_entry) {mutex_lock (&sctrl->lock);Quoted 8 lines
drivers/nvme/target/core.c static void nvmet_ctrl_complete_pending_ccr(struct nvmet_ctrl *ctrl) · line 1824
Quoted 6 lines
drivers/nvme/target/core.c +3
-1list_for_each_entry (ccr, &sctrl->ccrs, entry) { -if (ccr->ctrl == ctrl) +if (ccr->ctrl == ctrl) { +nvmet_ctrl_notify_ccr (sctrl); ccr->ctrl =NULL ; + }Is this double loop necessary? Would you have more than one controller cross resetting the same
Quoted 6 lines
As it is implemented now CCRs are linked to sctrl. This decision can be revisited if found suboptimal. At some point I had CCRs linked to
ctrl->subsysbut that led to lock ordering issues. Double loop is necessary to find all CCRs in all controllers and mark them done. Yes, it is possible to have more than one sctrl resetting the same ictrl.
I'm more interested in simplifying.
Quoted 8 lines
Quoted 3 lines
controller? Won't it be better to install a callback+opaque that the controller removal will call?
Can you elaborate more on that? Better in what terms?
nvmet_ctrl_complete_pending_ccr()is called fromnvmet_ctrl_free()when we know thatctrl->refis zero and no new CCRs will be added to this controller becausenvmet_ctrl_find_get_ccr()will not be able to get it.
In nvmet, the controller is serving a single host. Hence I am not sure I understand how multiple source controllers will try to reset the impacted controller. So, if there is a 1-1 relationship between source and impacted controller, I'd perhaps suggest to simplify and install on the impacted controller callback+opaque (e.g. void *data) instead of having it iterate and then actually send the AEN from the impacted controller.
Re: [RFC PATCH 05/14] nvmet: Send an AEN on CCR completion
Quoted 98 lines · Sagi Grimberg
Quoted 77 lines · Mohamed Khalfella
Quoted 68 lines · Sagi Grimberg
Quoted 66 lines · Mohamed Khalfella
Send an AEN to initiator when impacted controller exists. The notification points to CCR log page that initiator can read to check which CCR operation completed.
Signed-off-by Mohamed Khalfella <[email protected]>--- drivers/nvme/target/core.c | 27 +++++++++++++++++++++++---- drivers/nvme/target/nvmet.h | 3 ++- include/linux/nvme.h | 3 +++ 3 files changed, 28 insertions(+), 5 deletions(-)drivers/nvme/target/core.c +20
-3@@ -202,7 +202,7 @@ static void nvmet_async_event_work(struct work_struct *work)nvmet_async_events_process (ctrl); } -void nvmet_add_async_event (struct nvmet_ctrl *ctrl,u8 event_type, +static void nvmet_add_async_event_locked (struct nvmet_ctrl *ctrl,u8 event_type,u8 event_info,u8 log_page) {struct nvmet_async_event *aen;@@ -215,12 +215,17 @@ void nvmet_add_async_event(struct nvmet_ctrl *ctrl, u8 event_type, aen->event_info = event_info; aen->log_page = log_page; -mutex_lock (&ctrl->lock);list_add_tail (&aen->entry, &ctrl->async_events); -mutex_unlock (&ctrl->lock);queue_work (nvmet_wq, &ctrl->async_event_work); } +void nvmet_add_async_event (struct nvmet_ctrl *ctrl,u8 event_type, +u8 event_info,u8 log_page) +{ +mutex_lock (&ctrl->lock); +nvmet_add_async_event_locked (ctrl, event_type, event_info, log_page); +mutex_unlock (&ctrl->lock); +}static void nvmet_add_to_changed_ns_log (struct nvmet_ctrl *ctrl,__le32 nsid) {@@ -1788,6 +1793,18 @@ struct nvmet_ctrl *nvmet_alloc_ctrl(struct nvmet_alloc_ctrl_args *args) }EXPORT_SYMBOL_GPL (nvmet_alloc_ctrl); +static void nvmet_ctrl_notify_ccr (struct nvmet_ctrl *ctrl) +{ +lockdep_assert_held (&ctrl->lock); + +if (nvmet_aen_bit_disabled (ctrl,NVME_AEN_BIT_CCR_COMPLETE )) +return ; + +nvmet_add_async_event_locked (ctrl,NVME_AER_NOTICE , +NVME_AER_NOTICE_CCR_COMPLETED , +NVME_LOG_CCR ); +} +static void nvmet_ctrl_complete_pending_ccr (struct nvmet_ctrl *ctrl) {struct nvmet_subsys *subsys = ctrl->subsys;@@ -1801,8 +1818,10 @@ static void nvmet_ctrl_complete_pending_ccr(struct nvmet_ctrl *ctrl)list_for_each_entry (sctrl, &subsys->ctrls, subsys_entry) {mutex_lock (&sctrl->lock);Quoted 8 lines
drivers/nvme/target/core.c static void nvmet_ctrl_complete_pending_ccr(struct nvmet_ctrl *ctrl) · line 1824
Quoted 6 lines
drivers/nvme/target/core.c +3
-1list_for_each_entry (ccr, &sctrl->ccrs, entry) { -if (ccr->ctrl == ctrl) +if (ccr->ctrl == ctrl) { +nvmet_ctrl_notify_ccr (sctrl); ccr->ctrl =NULL ; + }Is this double loop necessary? Would you have more than one controller cross resetting the same
Quoted 6 lines
As it is implemented now CCRs are linked to sctrl. This decision can be revisited if found suboptimal. At some point I had CCRs linked to
ctrl->subsysbut that led to lock ordering issues. Double loop is necessary to find all CCRs in all controllers and mark them done. Yes, it is possible to have more than one sctrl resetting the same ictrl.I'm more interested in simplifying.
Quoted 8 lines
Quoted 3 lines
controller? Won't it be better to install a callback+opaque that the controller removal will call?
Can you elaborate more on that? Better in what terms?
nvmet_ctrl_complete_pending_ccr()is called fromnvmet_ctrl_free()when we know thatctrl->refis zero and no new CCRs will be added to this controller becausenvmet_ctrl_find_get_ccr()will not be able to get it.
Quoted 8 lines
In nvmet, the controller is serving a single host. Hence I am not sure I understand how multiple source controllers will try to reset the impacted controller. So, if there is a 1-1 relationship between source and impacted controller, I'd perhaps suggest to simplify and install on the impacted controller
callback+opaque (e.g. void *data) instead of having it iterate and then actually send
the AEN from the impacted controller.
A controller is serving a single path for a given host. A host that is connected to nvme subsystem via multiple paths will have more than one controller. I can think of two reasons why we need to support resetting an impacted controller from multiple source controllers.
- It is possible for multiple paths to go down at the same time. The first source controller we use for CCR, even though we check to see if LIVE, might have lost connection to subsystem. It is a matter of time for it to see keepalive timeout and fail too. If CCR fails using this controller we should not give up. We need to try other paths.
- Some nvme subsystems might support resetting impacted controller from a subset of controllers connected to the host. An array that has multiple frontend engines might not support resetting controllers across engines. In fact, TP8028 allows for subsystem to suggest to host to use another source controller in Alternate Controller ID (ACID) fied on CCR logpage (not implemente in this patchset).
Sagi Grimberg
maintainerRe: [RFC PATCH 05/14] nvmet: Send an AEN on CCR completion
Quoted 106 lines · Mohamed Khalfella
Quoted 92 lines · Sagi Grimberg
Quoted 76 lines · Mohamed Khalfella
Quoted 67 lines · Sagi Grimberg
Quoted 66 lines · Mohamed Khalfella
Send an AEN to initiator when impacted controller exists. The notification points to CCR log page that initiator can read to check which CCR operation completed.
Signed-off-by Mohamed Khalfella<[email protected]>--- drivers/nvme/target/core.c | 27 +++++++++++++++++++++++---- drivers/nvme/target/nvmet.h | 3 ++- include/linux/nvme.h | 3 +++ 3 files changed, 28 insertions(+), 5 deletions(-)drivers/nvme/target/core.c +20
-3@@ -202,7 +202,7 @@ static void nvmet_async_event_work(struct work_struct *work)nvmet_async_events_process (ctrl); } -void nvmet_add_async_event (struct nvmet_ctrl *ctrl,u8 event_type, +static void nvmet_add_async_event_locked (struct nvmet_ctrl *ctrl,u8 event_type,u8 event_info,u8 log_page) {struct nvmet_async_event *aen;@@ -215,12 +215,17 @@ void nvmet_add_async_event(struct nvmet_ctrl *ctrl, u8 event_type, aen->event_info = event_info; aen->log_page = log_page; -mutex_lock (&ctrl->lock);list_add_tail (&aen->entry, &ctrl->async_events); -mutex_unlock (&ctrl->lock);queue_work (nvmet_wq, &ctrl->async_event_work); } +void nvmet_add_async_event (struct nvmet_ctrl *ctrl,u8 event_type, +u8 event_info,u8 log_page) +{ +mutex_lock (&ctrl->lock); +nvmet_add_async_event_locked (ctrl, event_type, event_info, log_page); +mutex_unlock (&ctrl->lock); +}static void nvmet_add_to_changed_ns_log (struct nvmet_ctrl *ctrl,__le32 nsid) {@@ -1788,6 +1793,18 @@ struct nvmet_ctrl *nvmet_alloc_ctrl(struct nvmet_alloc_ctrl_args *args) }EXPORT_SYMBOL_GPL (nvmet_alloc_ctrl); +static void nvmet_ctrl_notify_ccr (struct nvmet_ctrl *ctrl) +{ +lockdep_assert_held (&ctrl->lock); + +if (nvmet_aen_bit_disabled (ctrl,NVME_AEN_BIT_CCR_COMPLETE )) +return ; + +nvmet_add_async_event_locked (ctrl,NVME_AER_NOTICE , +NVME_AER_NOTICE_CCR_COMPLETED , +NVME_LOG_CCR ); +} +static void nvmet_ctrl_complete_pending_ccr (struct nvmet_ctrl *ctrl) {struct nvmet_subsys *subsys = ctrl->subsys;@@ -1801,8 +1818,10 @@ static void nvmet_ctrl_complete_pending_ccr(struct nvmet_ctrl *ctrl)list_for_each_entry (sctrl, &subsys->ctrls, subsys_entry) {mutex_lock (&sctrl->lock);Quoted 8 lines
drivers/nvme/target/core.c static void nvmet_ctrl_complete_pending_ccr(struct nvmet_ctrl *ctrl) · line 1824
Quoted 6 lines
drivers/nvme/target/core.c +3
-1list_for_each_entry (ccr, &sctrl->ccrs, entry) { -if (ccr->ctrl == ctrl) +if (ccr->ctrl == ctrl) { +nvmet_ctrl_notify_ccr (sctrl); ccr->ctrl =NULL ; + }Is this double loop necessary? Would you have more than one controller cross resetting the same
Quoted 6 lines
As it is implemented now CCRs are linked to sctrl. This decision can be revisited if found suboptimal. At some point I had CCRs linked to
ctrl->subsysbut that led to lock ordering issues. Double loop is necessary to find all CCRs in all controllers and mark them done. Yes, it is possible to have more than one sctrl resetting the same ictrl.I'm more interested in simplifying.
Quoted 7 lines
Quoted 2 lines
controller? Won't it be better to install a callback+opaque that the controller removal will call?
Can you elaborate more on that? Better in what terms?
nvmet_ctrl_complete_pending_ccr()is called fromnvmet_ctrl_free()when we know thatctrl->refis zero and no new CCRs will be added to this controller becausenvmet_ctrl_find_get_ccr()will not be able to get it.Quoted 8 lines
In nvmet, the controller is serving a single host. Hence I am not sure I understand how multiple source controllers will try to reset the impacted controller. So, if there is a 1-1 relationship between source and impacted controller, I'd perhaps suggest to simplify and install on the impacted controller
callback+opaque (e.g. void *data) instead of having it iterate and then actually send
the AEN from the impacted controller.A controller is serving a single path for a given host. A host that is connected to nvme subsystem via multiple paths will have more than one controller. I can think of two reasons why we need to support resetting an impacted controller from multiple source controllers.
Quoted 5 lines
- It is possible for multiple paths to go down at the same time. The first source controller we use for CCR, even though we check to see if LIVE, might have lost connection to subsystem. It is a matter of time for it to see keepalive timeout and fail too. If CCR fails using this controller we should not give up. We need to try other paths.
But the host is doing the cross-reset synchronously... it waits for kato for a completion of the reset, and then gives up, its not like it is sitting there waiting for the AEN...
Generally the fact that the spec states a capability/flexibility, it is still Linux's choice to choose weather to implement it. I'm trying to understand if we can simplify Linux host and controller in this non-trivial error recovery flow.
What is your expectation to happen in general? what are your expected kato/cqt values? how many attempts do we want the host to do?
Quoted 6 lines
- Some nvme subsystems might support resetting impacted controller from a subset of controllers connected to the host. An array that has multiple frontend engines might not support resetting controllers across engines. In fact, TP8028 allows for subsystem to suggest to host to use another source controller in Alternate Controller ID (ACID) fied on CCR logpage (not implemente in this patchset).
It is not a case though where the impacted controller will be reset from multiple source controller at the same time...
I'd also say that if indeed there are subsystems that require specific controllers to do cross recovery, they won't be able to use this at all... Are there any such arrays?
Re: [RFC PATCH 05/14] nvmet: Send an AEN on CCR completion
Quoted 113 lines · Sagi Grimberg
Quoted 106 lines · Mohamed Khalfella
Quoted 92 lines · Sagi Grimberg
Quoted 76 lines · Mohamed Khalfella
Quoted 67 lines · Sagi Grimberg
Quoted 66 lines · Mohamed Khalfella
Send an AEN to initiator when impacted controller exists. The notification points to CCR log page that initiator can read to check which CCR operation completed.
Signed-off-by Mohamed Khalfella<[email protected]>--- drivers/nvme/target/core.c | 27 +++++++++++++++++++++++---- drivers/nvme/target/nvmet.h | 3 ++- include/linux/nvme.h | 3 +++ 3 files changed, 28 insertions(+), 5 deletions(-)drivers/nvme/target/core.c +20
-3@@ -202,7 +202,7 @@ static void nvmet_async_event_work(struct work_struct *work)nvmet_async_events_process (ctrl); } -void nvmet_add_async_event (struct nvmet_ctrl *ctrl,u8 event_type, +static void nvmet_add_async_event_locked (struct nvmet_ctrl *ctrl,u8 event_type,u8 event_info,u8 log_page) {struct nvmet_async_event *aen;@@ -215,12 +215,17 @@ void nvmet_add_async_event(struct nvmet_ctrl *ctrl, u8 event_type, aen->event_info = event_info; aen->log_page = log_page; -mutex_lock (&ctrl->lock);list_add_tail (&aen->entry, &ctrl->async_events); -mutex_unlock (&ctrl->lock);queue_work (nvmet_wq, &ctrl->async_event_work); } +void nvmet_add_async_event (struct nvmet_ctrl *ctrl,u8 event_type, +u8 event_info,u8 log_page) +{ +mutex_lock (&ctrl->lock); +nvmet_add_async_event_locked (ctrl, event_type, event_info, log_page); +mutex_unlock (&ctrl->lock); +}static void nvmet_add_to_changed_ns_log (struct nvmet_ctrl *ctrl,__le32 nsid) {@@ -1788,6 +1793,18 @@ struct nvmet_ctrl *nvmet_alloc_ctrl(struct nvmet_alloc_ctrl_args *args) }EXPORT_SYMBOL_GPL (nvmet_alloc_ctrl); +static void nvmet_ctrl_notify_ccr (struct nvmet_ctrl *ctrl) +{ +lockdep_assert_held (&ctrl->lock); + +if (nvmet_aen_bit_disabled (ctrl,NVME_AEN_BIT_CCR_COMPLETE )) +return ; + +nvmet_add_async_event_locked (ctrl,NVME_AER_NOTICE , +NVME_AER_NOTICE_CCR_COMPLETED , +NVME_LOG_CCR ); +} +static void nvmet_ctrl_complete_pending_ccr (struct nvmet_ctrl *ctrl) {struct nvmet_subsys *subsys = ctrl->subsys;@@ -1801,8 +1818,10 @@ static void nvmet_ctrl_complete_pending_ccr(struct nvmet_ctrl *ctrl)list_for_each_entry (sctrl, &subsys->ctrls, subsys_entry) {mutex_lock (&sctrl->lock);Quoted 8 lines
drivers/nvme/target/core.c static void nvmet_ctrl_complete_pending_ccr(struct nvmet_ctrl *ctrl) · line 1824
Quoted 6 lines
drivers/nvme/target/core.c +3
-1list_for_each_entry (ccr, &sctrl->ccrs, entry) { -if (ccr->ctrl == ctrl) +if (ccr->ctrl == ctrl) { +nvmet_ctrl_notify_ccr (sctrl); ccr->ctrl =NULL ; + }Is this double loop necessary? Would you have more than one controller cross resetting the same
Quoted 6 lines
As it is implemented now CCRs are linked to sctrl. This decision can be revisited if found suboptimal. At some point I had CCRs linked to
ctrl->subsysbut that led to lock ordering issues. Double loop is necessary to find all CCRs in all controllers and mark them done. Yes, it is possible to have more than one sctrl resetting the same ictrl.I'm more interested in simplifying.
Quoted 7 lines
Quoted 2 lines
controller? Won't it be better to install a callback+opaque that the controller removal will call?
Can you elaborate more on that? Better in what terms?
nvmet_ctrl_complete_pending_ccr()is called fromnvmet_ctrl_free()when we know thatctrl->refis zero and no new CCRs will be added to this controller becausenvmet_ctrl_find_get_ccr()will not be able to get it.Quoted 8 lines
In nvmet, the controller is serving a single host. Hence I am not sure I understand how multiple source controllers will try to reset the impacted controller. So, if there is a 1-1 relationship between source and impacted controller, I'd perhaps suggest to simplify and install on the impacted controller
callback+opaque (e.g. void *data) instead of having it iterate and then actually send
the AEN from the impacted controller.A controller is serving a single path for a given host. A host that is connected to nvme subsystem via multiple paths will have more than one controller. I can think of two reasons why we need to support resetting an impacted controller from multiple source controllers.
Quoted 5 lines
- It is possible for multiple paths to go down at the same time. The first source controller we use for CCR, even though we check to see if LIVE, might have lost connection to subsystem. It is a matter of time for it to see keepalive timeout and fail too. If CCR fails using this controller we should not give up. We need to try other paths.
Quoted 3 lines
But the host is doing the cross-reset synchronously... it waits for kato for a completion of the reset, and then gives up, its not like it is sitting there waiting for the AEN...
The Linux host as Mohamed has implemented it is doing the CCR synchronously. I expect most hosts will also, until they time out one controller because the connection might be dying, and that timeout might come sooner than the target. Controllers publish how many outstanding CCRs they may be the source of, not how many controllers they may be impacted controllers for. However, it is my understanding (correct me if I am wrong) that the Linux nvmet implementation is primarily for testing the Linux nvme host implementation. Whatever limitations keep the nvmet code simple can make sense, even if they do not in a production system.
Quoted 9 lines
Generally the fact that the spec states a capability/flexibility, it is still Linux's
choice to choose weather to implement it. I'm trying to understand if we can simplify Linux host and controller in this non-trivial error recovery flow.What is your expectation to happen in general? what are your expected kato/cqt
values? how many attempts do we want the host to do?
For CQT, I see 30-60 seconds. For KATO, I see 5-60 seconds. I see KATO skewing lower than it should because of the cost to reliably avoid data corruption.
In general, I expect an HA-pair storage array to benefit most from implementing this feature. Often, a host would establish an association over 4 paths, 2 to each HA-pair. It is common for non-disruptive operation for a member of the pair to go down for software upgrades or hardware modifications while the other member stays up. In this configuration, 2 paths go down when the member goes down. The host has a 2 in 3 chance of picking a controller to which the host still has a connection. Given the tremendous reduction in failover time (even with a timeout on the CCR) and my perception of the complexity cost of a retry, I would much rather make the host have a 100% chance of using CCR successfully after a retry.
Quoted 7 lines
Quoted 6 lines
- Some nvme subsystems might support resetting impacted controller from a subset of controllers connected to the host. An array that has multiple frontend engines might not support resetting controllers across engines. In fact, TP8028 allows for subsystem to suggest to host to use another source controller in Alternate Controller ID (ACID) fied on CCR logpage (not implemente in this patchset).
Quoted 8 lines
It is not a case though where the impacted controller will be reset from multiple
source controller at the same time...I'd also say that if indeed there are subsystems that require specific controllers to do
cross recovery, they won't be able to use this at all... Are there any such arrays?
There are certainly systems out there (striping, Ceph-like, or stretched to support non-dispersed hosts) that have many more nodes than an HA-pair and may have communication barriers that prevent nvme controllers from being able to handle an CCR for another nvme controller. I still think they would benefit from retrying a CCR to find a path that can succeed.
Sincerely,
Randy Jennings
Re: [RFC PATCH 05/14] nvmet: Send an AEN on CCR completion
In reply to Sagi GrimbergQuoted 119 lines · Sagi Grimberg
Quoted 106 lines · Mohamed Khalfella
Quoted 92 lines · Sagi Grimberg
Quoted 76 lines · Mohamed Khalfella
Quoted 67 lines · Sagi Grimberg
Quoted 66 lines · Mohamed Khalfella
Send an AEN to initiator when impacted controller exists. The notification points to CCR log page that initiator can read to check which CCR operation completed.
Signed-off-by Mohamed Khalfella<[email protected]>--- drivers/nvme/target/core.c | 27 +++++++++++++++++++++++---- drivers/nvme/target/nvmet.h | 3 ++- include/linux/nvme.h | 3 +++ 3 files changed, 28 insertions(+), 5 deletions(-)drivers/nvme/target/core.c +20
-3@@ -202,7 +202,7 @@ static void nvmet_async_event_work(struct work_struct *work)nvmet_async_events_process (ctrl); } -void nvmet_add_async_event (struct nvmet_ctrl *ctrl,u8 event_type, +static void nvmet_add_async_event_locked (struct nvmet_ctrl *ctrl,u8 event_type,u8 event_info,u8 log_page) {struct nvmet_async_event *aen;@@ -215,12 +215,17 @@ void nvmet_add_async_event(struct nvmet_ctrl *ctrl, u8 event_type, aen->event_info = event_info; aen->log_page = log_page; -mutex_lock (&ctrl->lock);list_add_tail (&aen->entry, &ctrl->async_events); -mutex_unlock (&ctrl->lock);queue_work (nvmet_wq, &ctrl->async_event_work); } +void nvmet_add_async_event (struct nvmet_ctrl *ctrl,u8 event_type, +u8 event_info,u8 log_page) +{ +mutex_lock (&ctrl->lock); +nvmet_add_async_event_locked (ctrl, event_type, event_info, log_page); +mutex_unlock (&ctrl->lock); +}static void nvmet_add_to_changed_ns_log (struct nvmet_ctrl *ctrl,__le32 nsid) {@@ -1788,6 +1793,18 @@ struct nvmet_ctrl *nvmet_alloc_ctrl(struct nvmet_alloc_ctrl_args *args) }EXPORT_SYMBOL_GPL (nvmet_alloc_ctrl); +static void nvmet_ctrl_notify_ccr (struct nvmet_ctrl *ctrl) +{ +lockdep_assert_held (&ctrl->lock); + +if (nvmet_aen_bit_disabled (ctrl,NVME_AEN_BIT_CCR_COMPLETE )) +return ; + +nvmet_add_async_event_locked (ctrl,NVME_AER_NOTICE , +NVME_AER_NOTICE_CCR_COMPLETED , +NVME_LOG_CCR ); +} +static void nvmet_ctrl_complete_pending_ccr (struct nvmet_ctrl *ctrl) {struct nvmet_subsys *subsys = ctrl->subsys;@@ -1801,8 +1818,10 @@ static void nvmet_ctrl_complete_pending_ccr(struct nvmet_ctrl *ctrl)list_for_each_entry (sctrl, &subsys->ctrls, subsys_entry) {mutex_lock (&sctrl->lock);Quoted 8 lines
drivers/nvme/target/core.c static void nvmet_ctrl_complete_pending_ccr(struct nvmet_ctrl *ctrl) · line 1824
Quoted 6 lines
drivers/nvme/target/core.c +3
-1list_for_each_entry (ccr, &sctrl->ccrs, entry) { -if (ccr->ctrl == ctrl) +if (ccr->ctrl == ctrl) { +nvmet_ctrl_notify_ccr (sctrl); ccr->ctrl =NULL ; + }Is this double loop necessary? Would you have more than one controller cross resetting the same
Quoted 6 lines
As it is implemented now CCRs are linked to sctrl. This decision can be revisited if found suboptimal. At some point I had CCRs linked to
ctrl->subsysbut that led to lock ordering issues. Double loop is necessary to find all CCRs in all controllers and mark them done. Yes, it is possible to have more than one sctrl resetting the same ictrl.I'm more interested in simplifying.
Quoted 7 lines
Quoted 2 lines
controller? Won't it be better to install a callback+opaque that the controller removal will call?
Can you elaborate more on that? Better in what terms?
nvmet_ctrl_complete_pending_ccr()is called fromnvmet_ctrl_free()when we know thatctrl->refis zero and no new CCRs will be added to this controller becausenvmet_ctrl_find_get_ccr()will not be able to get it.Quoted 8 lines
In nvmet, the controller is serving a single host. Hence I am not sure I understand how multiple source controllers will try to reset the impacted controller. So, if there is a 1-1 relationship between source and impacted controller, I'd perhaps suggest to simplify and install on the impacted controller
callback+opaque (e.g. void *data) instead of having it iterate and then actually send
the AEN from the impacted controller.A controller is serving a single path for a given host. A host that is connected to nvme subsystem via multiple paths will have more than one controller. I can think of two reasons why we need to support resetting an impacted controller from multiple source controllers.
Quoted 5 lines
- It is possible for multiple paths to go down at the same time. The first source controller we use for CCR, even though we check to see if LIVE, might have lost connection to subsystem. It is a matter of time for it to see keepalive timeout and fail too. If CCR fails using this controller we should not give up. We need to try other paths.
But the host is doing the cross-reset synchronously... it waits for kato for a completion of the reset, and then gives up, its not like it is sitting there waiting for the AEN...
Quoted 8 lines
Generally the fact that the spec states a capability/flexibility, it is still Linux's
choice to choose weather to implement it. I'm trying to understand if we can simplify Linux host and controller in this non-trivial error recovery flow.What is your expectation to happen in general? what are your expected kato/cqt
values? how many attempts do we want the host to do?
For a target that support CCR it is expected to support CQT as well. If the initiator notices path failure then it either needs to get the impacted controller reset via CCR or wait for time-based recovery. Time based recovery adds long delay (n * kato + cqt), CCR is expected to take way less time. This is why initiator should try CCR for as long as there is time and paths to do so. The alternative is to sit and wait and do nothing.
kato defaults to 5 seconds. cqt depends on the implementation on the target. I tested these changes with 30s of cqt. With traffic based keepalive time-based recovery takes 3 * 5 + 30 = 45s. In case of partial path failure CCR finished in milliseconds. This is why it is better to try every possible path before giving up. There is no fixed number of attempts. As long as there is time to try CCR and there are paths to try the initiator should continue trying CCR. Again, we do not want to fallback to time-based recovery unless there are no options.
tmo = msecs_to_jiffies(max(ictrl->cqt, ictrl->kato * 1000));
if (!wait_for_completion_timeout(&ccr.complete, tmo)) {
ret = -ETIMEDOUT;
goto out;
}This is the code that waits for CCR to finish after CCR command was accepted by the target. The specs does not say how much time host should wait. max(cqt, kato) felt like reasonable value.
Quoted 8 lines
Quoted 7 lines
- Some nvme subsystems might support resetting impacted controller from a subset of controllers connected to the host. An array that has multiple frontend engines might not support resetting controllers across engines. In fact, TP8028 allows for subsystem to suggest to host to use another source controller in Alternate Controller ID (ACID) fied on CCR logpage (not implemente in this patchset).
Quoted 8 lines
It is not a case though where the impacted controller will be reset from multiple
source controller at the same time...I'd also say that if indeed there are subsystems that require specific controllers to do
cross recovery, they won't be able to use this at all... Are there any such arrays?
Not at all. CCR is still useful in the case of partial connectivity failure to the same engine. I can not name a product that does that today.
RFC PATCH 6/14 nvme: Rapid Path Failure Recovery read controller identify fields
In reply to Mohamed KhalfellaTP2028 Rapid path failure added new fileds to controller identify response. Read CIU (Controller Instance Uniquifier), CIRN (Controller Instance Random Number), and CCRL (Cross-Controller Reset Limit) from controller identify response. Expose CIU and CIRN as sysfs attributes so the values can be used directrly by user if needed.
TP4129 KATO Corrections and Clarifications defined CQT (Command Quiesce Time) which is used along with KATO (Keep Alive Timeout) to set an upper limite for attempting Cross-Controller Recovery.
--- drivers/nvme/host/core.c | 5 +++++ drivers/nvme/host/nvme.h | 11 +++++++++++ drivers/nvme/host/sysfs.c | 23 +++++++++++++++++++++++ 3 files changed, 39 insertions(+)
drivers/nvme/host/core.c +5 -0
@@ -3572,12 +3572,17 @@ static int nvme_init_identify(struct nvme_ctrl *ctrl) ctrl->crdt[1 ] =le16_to_cpu (id->crdt2); ctrl->crdt[2 ] =le16_to_cpu (id->crdt3); + ctrl->ciu = id->ciu; + ctrl->cirn =le64_to_cpu (id->cirn); +atomic_set (&ctrl->ccr_limit, id->ccrl); + ctrl->oacs =le16_to_cpu (id->oacs); ctrl->oncs =le16_to_cpu (id->oncs); ctrl->mtfa =le16_to_cpu (id->mtfa); ctrl->oaes =le32_to_cpu (id->oaes); ctrl->wctemp =le16_to_cpu (id->wctemp); ctrl->cctemp =le16_to_cpu (id->cctemp); + ctrl->cqt =le16_to_cpu (id->cqt);atomic_set (&ctrl->abort_limit, id->acl +1 ); ctrl->vwc = id->vwc;
drivers/nvme/host/nvme.h +11 -0
@@ -326,13 +326,17 @@ struct nvme_ctrl {u32 max_zone_append;#endif u16 crdt[3 ]; +u16 cqt;u16 oncs;u8 dmrl; +u8 ciu;u32 dmrsl; +u64 cirn;u16 oacs;u16 sqsize;u32 max_namespaces;atomic_t abort_limit; +atomic_t ccr_limit;u8 vwc;u32 vs;u32 sgls;
@@ -1218,4 +1222,11 @@ static inline bool nvme_multi_css(struct nvme_ctrl *ctrl) return (ctrl->ctrl_config & NVME_CC_CSS_MASK) == NVME_CC_CSS_CSI; } +static inline unsigned long nvme_recovery_timeout_ms (struct nvme_ctrl *ctrl) +{ +if (ctrl->ctratt &NVME_CTRL_ATTR_TBKAS ) +return 3 * ctrl->kato *1000 + ctrl->cqt; +return 2 * ctrl->kato *1000 + ctrl->cqt; +} +#endif /* _NVME_H */
drivers/nvme/host/sysfs.c +23 -0
@@ -388,6 +388,27 @@ nvme_show_int_function(queue_count);nvme_show_int_function (sqsize);nvme_show_int_function (kato); +static ssize_t nvme_sysfs_uniquifier_show (struct device *dev, +struct device_attribute *attr, +char *buf) +{ +struct nvme_ctrl *ctrl =dev_get_drvdata (dev); + +return sysfs_emit (buf,"%02x\n" , ctrl->ciu); +} +static DEVICE_ATTR(uniquifier, S_IRUGO, nvme_sysfs_uniquifier_show, NULL); + +static ssize_t nvme_sysfs_random_show (struct device *dev, +struct device_attribute *attr, +char *buf) +{ +struct nvme_ctrl *ctrl =dev_get_drvdata (dev); + +return sysfs_emit (buf,"%016llx\n" , ctrl->cirn); +} +static DEVICE_ATTR(random, S_IRUGO, nvme_sysfs_random_show, NULL); + +static ssize_t nvme_sysfs_delete (struct device *dev,struct device_attribute *attr,const char *buf,size_t count)
@@ -734,6 +755,8 @@ static struct attribute *nvme_dev_attrs[] = { &dev_attr_numa_node.attr, &dev_attr_queue_count.attr, &dev_attr_sqsize.attr, + &dev_attr_uniquifier.attr, + &dev_attr_random.attr, &dev_attr_hostnqn.attr, &dev_attr_hostid.attr, &dev_attr_ctrl_loss_tmo.attr,
Signature · 1 line
2.51.2
Re: [RFC PATCH 06/14] nvme: Rapid Path Failure Recovery read controller identify fields
Quoted 2 lines · Mohamed Khalfella
TP2028 Rapid path failure added new fileds to controller identify
TP8028
Quoted 8 lines
response. Read CIU (Controller Instance Uniquifier), CIRN (Controller Instance Random Number), and CCRL (Cross-Controller Reset Limit) from controller identify response. Expose CIU and CIRN as sysfs attributes so the values can be used directrly by user if needed.
TP4129 KATO Corrections and Clarifications defined CQT (Command Quiesce Time) which is used along with KATO (Keep Alive Timeout) to set an upper limite for attempting Cross-Controller Recovery.
"limite" -> "limit"
Quoted 13 lines
Signed-off-by Mohamed Khalfella <[email protected]>--- drivers/nvme/host/core.c | 5 +++++ drivers/nvme/host/nvme.h | 11 +++++++++++ drivers/nvme/host/sysfs.c | 23 +++++++++++++++++++++++ 3 files changed, 39 insertions(+)drivers/nvme/host/core.c +0
-0
drivers/nvme/host/core.c static int nvme_init_identify(struct nvme_ctrl *ctrl) · line 3577
Quoted 6 lines
drivers/nvme/host/core.c +3
-0@@ -3572,12 +3572,17 @@ static int nvme_init_identify(struct nvme_ctrl *ctrl) ctrl->crdt[1 ] =le16_to_cpu (id->crdt2); ctrl->crdt[2 ] =le16_to_cpu (id->crdt3); + ctrl->ciu = id->ciu; + ctrl->cirn =le64_to_cpu (id->cirn); +atomic_set (&ctrl->ccr_limit, id->ccrl);
Seems like it would be good for the target & init to use the same name for these fields. I have a preference for these over instance_uniquifier and random because they are more concise, but the preference is not strong.
drivers/nvme/host/core.c static int nvme_init_identify(struct nvme_ctrl *ctrl) · line 3588
Quoted 11 lines
drivers/nvme/host/core.c +2
-0+ ctrl->oacs =le16_to_cpu (id->oacs); ctrl->oncs =le16_to_cpu (id->oncs); ctrl->mtfa =le16_to_cpu (id->mtfa); ctrl->oaes =le32_to_cpu (id->oaes); ctrl->wctemp =le16_to_cpu (id->wctemp); ctrl->cctemp =le16_to_cpu (id->cctemp); + ctrl->cqt =le16_to_cpu (id->cqt);atomic_set (&ctrl->abort_limit, id->acl +1 ); ctrl->vwc = id->vwc;
I cannot discern an ordering to the attributes set here. Any particular reason, you placed cqt away from the others you added?
Quoted 42 lines
drivers/nvme/host/sysfs.c +23
-1@@ -388,6 +388,27 @@ nvme_show_int_function(queue_count);nvme_show_int_function (sqsize);nvme_show_int_function (kato); +static ssize_t nvme_sysfs_uniquifier_show (struct device *dev, +struct device_attribute *attr, +char *buf) +{ +struct nvme_ctrl *ctrl =dev_get_drvdata (dev); + +return sysfs_emit (buf,"%02x\n" , ctrl->ciu); +} +static DEVICE_ATTR(uniquifier, S_IRUGO, nvme_sysfs_uniquifier_show, NULL); + +static ssize_t nvme_sysfs_random_show (struct device *dev, +struct device_attribute *attr, +char *buf) +{ +struct nvme_ctrl *ctrl =dev_get_drvdata (dev); + +return sysfs_emit (buf,"%016llx\n" , ctrl->cirn); +} +static DEVICE_ATTR(random, S_IRUGO, nvme_sysfs_random_show, NULL); + +static ssize_t nvme_sysfs_delete (struct device *dev,struct device_attribute *attr,const char *buf,size_t count)@@ -734,6 +755,8 @@ static struct attribute *nvme_dev_attrs[] = { &dev_attr_numa_node.attr, &dev_attr_queue_count.attr, &dev_attr_sqsize.attr, + &dev_attr_uniquifier.attr, + &dev_attr_random.attr, &dev_attr_hostnqn.attr, &dev_attr_hostid.attr, &dev_attr_ctrl_loss_tmo.attr, --
Quoted 2 lines
2.51.2
These are the names used in the target code (uniquifer & random. I'd rather have them match (identify structure will have spec's abbreviations; ctrl & debug/sysfs for target & initiator either be ciu/cirn or uniquifer/random.
But this is small stuff.
Re: [RFC PATCH 06/14] nvme: Rapid Path Failure Recovery read controller identify fields
Quoted 5 lines · Randy Jennings
Quoted 2 lines · Mohamed Khalfella
TP2028 Rapid path failure added new fileds to controller identify
TP8028
Fixed.
Quoted 9 lines
Quoted 8 lines
response. Read CIU (Controller Instance Uniquifier), CIRN (Controller Instance Random Number), and CCRL (Cross-Controller Reset Limit) from controller identify response. Expose CIU and CIRN as sysfs attributes so the values can be used directrly by user if needed.
TP4129 KATO Corrections and Clarifications defined CQT (Command Quiesce Time) which is used along with KATO (Keep Alive Timeout) to set an upper limite for attempting Cross-Controller Recovery.
"limite" -> "limit"
Fixed.
Quoted 16 lines
Quoted 16 lines
Signed-off-by Mohamed Khalfella <[email protected]>--- drivers/nvme/host/core.c | 5 +++++ drivers/nvme/host/nvme.h | 11 +++++++++++ drivers/nvme/host/sysfs.c | 23 +++++++++++++++++++++++ 3 files changed, 39 insertions(+)drivers/nvme/host/core.c +0
-0@@ -3572,12 +3572,17 @@ static int nvme_init_identify(struct nvme_ctrl *ctrl) ctrl->crdt[1 ] =le16_to_cpu (id->crdt2); ctrl->crdt[2 ] =le16_to_cpu (id->crdt3);
Quoted 7 lines
drivers/nvme/host/core.c static int nvme_init_identify(struct nvme_ctrl *ctrl) · line 3577
Quoted 3 lines
drivers/nvme/host/core.c +3
-0+ ctrl->ciu = id->ciu; + ctrl->cirn =le64_to_cpu (id->cirn); +atomic_set (&ctrl->ccr_limit, id->ccrl);Seems like it would be good for the target & init to use the same name for these fields. I have a preference for these over
instance_uniquifierand random because they are more concise, but the preference is not strong.
The field names in the spec are concise, but they are also cryptic.
Quoted 10 lines
Quoted 10 lines
drivers/nvme/host/core.c +2
-0+ ctrl->oacs =le16_to_cpu (id->oacs); ctrl->oncs =le16_to_cpu (id->oncs); ctrl->mtfa =le16_to_cpu (id->mtfa); ctrl->oaes =le32_to_cpu (id->oaes); ctrl->wctemp =le16_to_cpu (id->wctemp); ctrl->cctemp =le16_to_cpu (id->cctemp); + ctrl->cqt =le16_to_cpu (id->cqt);
Quoted 4 lines
drivers/nvme/host/core.c static int nvme_init_identify(struct nvme_ctrl *ctrl) · line 3588
Quoted 2 lines
drivers/nvme/host/core.c +0
-0atomic_set (&ctrl->abort_limit, id->acl +1 ); ctrl->vwc = id->vwc;I cannot discern an ordering to the attributes set here. Any particular reason, you placed cqt away from the others you added?
No reason. Moved ctrl->cqt initialization up with other fields.
Quoted 42 lines
Quoted 42 lines
drivers/nvme/host/sysfs.c +23
-0@@ -388,6 +388,27 @@ nvme_show_int_function(queue_count);nvme_show_int_function (sqsize);nvme_show_int_function (kato); +static ssize_t nvme_sysfs_uniquifier_show (struct device *dev, +struct device_attribute *attr, +char *buf) +{ +struct nvme_ctrl *ctrl =dev_get_drvdata (dev); + +return sysfs_emit (buf,"%02x\n" , ctrl->ciu); +} +static DEVICE_ATTR(uniquifier, S_IRUGO, nvme_sysfs_uniquifier_show, NULL); + +static ssize_t nvme_sysfs_random_show (struct device *dev, +struct device_attribute *attr, +char *buf) +{ +struct nvme_ctrl *ctrl =dev_get_drvdata (dev); + +return sysfs_emit (buf,"%016llx\n" , ctrl->cirn); +} +static DEVICE_ATTR(random, S_IRUGO, nvme_sysfs_random_show, NULL); + +static ssize_t nvme_sysfs_delete (struct device *dev,struct device_attribute *attr,const char *buf,size_t count)@@ -734,6 +755,8 @@ static struct attribute *nvme_dev_attrs[] = { &dev_attr_numa_node.attr, &dev_attr_queue_count.attr, &dev_attr_sqsize.attr, + &dev_attr_uniquifier.attr, + &dev_attr_random.attr, &dev_attr_hostnqn.attr, &dev_attr_hostid.attr, &dev_attr_ctrl_loss_tmo.attr,
Quoted 8 lines
Quoted 3 lines
--
2.51.2These are the names used in the target code (uniquifer & random. I'd rather have them match (identify structure will have spec's abbreviations; ctrl & debug/sysfs for target & initiator either be ciu/cirn or uniquifer/random.
I think it matters for sysfs attributes. I do not know the right thing to do. Should we use spec names like "cirn" or call it "random"?
Quoted 4 lines
But this is small stuff.
Reviewed-by Randy Jennings <[email protected]>
Re: [RFC PATCH 06/14] nvme: Rapid Path Failure Recovery read controller identify fields
Quoted 115 lines · Mohamed Khalfella
Quoted 5 lines · Randy Jennings
Quoted 2 lines · Mohamed Khalfella
TP2028 Rapid path failure added new fileds to controller identify
TP8028
Fixed.
Quoted 9 lines
Quoted 8 lines
response. Read CIU (Controller Instance Uniquifier), CIRN (Controller Instance Random Number), and CCRL (Cross-Controller Reset Limit) from controller identify response. Expose CIU and CIRN as sysfs attributes so the values can be used directrly by user if needed.
TP4129 KATO Corrections and Clarifications defined CQT (Command Quiesce Time) which is used along with KATO (Keep Alive Timeout) to set an upper limite for attempting Cross-Controller Recovery.
"limite" -> "limit"
Fixed.
Quoted 16 lines
Quoted 16 lines
Signed-off-by Mohamed Khalfella <[email protected]>--- drivers/nvme/host/core.c | 5 +++++ drivers/nvme/host/nvme.h | 11 +++++++++++ drivers/nvme/host/sysfs.c | 23 +++++++++++++++++++++++ 3 files changed, 39 insertions(+)drivers/nvme/host/core.c +0
-0@@ -3572,12 +3572,17 @@ static int nvme_init_identify(struct nvme_ctrl *ctrl) ctrl->crdt[1 ] =le16_to_cpu (id->crdt2); ctrl->crdt[2 ] =le16_to_cpu (id->crdt3);Quoted 7 lines
drivers/nvme/host/core.c static int nvme_init_identify(struct nvme_ctrl *ctrl) · line 3577
Quoted 3 lines
drivers/nvme/host/core.c +3
-0+ ctrl->ciu = id->ciu; + ctrl->cirn =le64_to_cpu (id->cirn); +atomic_set (&ctrl->ccr_limit, id->ccrl);Seems like it would be good for the target & init to use the same name for these fields. I have a preference for these over
instance_uniquifierand random because they are more concise, but the preference is not strong.The field names in the spec are concise, but they are also cryptic.
Quoted 10 lines
Quoted 10 lines
drivers/nvme/host/core.c +2
-0+ ctrl->oacs =le16_to_cpu (id->oacs); ctrl->oncs =le16_to_cpu (id->oncs); ctrl->mtfa =le16_to_cpu (id->mtfa); ctrl->oaes =le32_to_cpu (id->oaes); ctrl->wctemp =le16_to_cpu (id->wctemp); ctrl->cctemp =le16_to_cpu (id->cctemp); + ctrl->cqt =le16_to_cpu (id->cqt);Quoted 4 lines
drivers/nvme/host/core.c static int nvme_init_identify(struct nvme_ctrl *ctrl) · line 3588
Quoted 2 lines
drivers/nvme/host/core.c +0
-0atomic_set (&ctrl->abort_limit, id->acl +1 ); ctrl->vwc = id->vwc;I cannot discern an ordering to the attributes set here. Any particular reason, you placed cqt away from the others you added?
No reason. Moved
ctrl->cqtinitialization up with other fields.Quoted 42 lines
Quoted 42 lines
drivers/nvme/host/sysfs.c +23
-0@@ -388,6 +388,27 @@ nvme_show_int_function(queue_count);nvme_show_int_function (sqsize);nvme_show_int_function (kato); +static ssize_t nvme_sysfs_uniquifier_show (struct device *dev, +struct device_attribute *attr, +char *buf) +{ +struct nvme_ctrl *ctrl =dev_get_drvdata (dev); + +return sysfs_emit (buf,"%02x\n" , ctrl->ciu); +} +static DEVICE_ATTR(uniquifier, S_IRUGO, nvme_sysfs_uniquifier_show, NULL); + +static ssize_t nvme_sysfs_random_show (struct device *dev, +struct device_attribute *attr, +char *buf) +{ +struct nvme_ctrl *ctrl =dev_get_drvdata (dev); + +return sysfs_emit (buf,"%016llx\n" , ctrl->cirn); +} +static DEVICE_ATTR(random, S_IRUGO, nvme_sysfs_random_show, NULL); + +static ssize_t nvme_sysfs_delete (struct device *dev,struct device_attribute *attr,const char *buf,size_t count)@@ -734,6 +755,8 @@ static struct attribute *nvme_dev_attrs[] = { &dev_attr_numa_node.attr, &dev_attr_queue_count.attr, &dev_attr_sqsize.attr, + &dev_attr_uniquifier.attr, + &dev_attr_random.attr, &dev_attr_hostnqn.attr, &dev_attr_hostid.attr, &dev_attr_ctrl_loss_tmo.attr,Quoted 8 lines
Quoted 3 lines
--
2.51.2These are the names used in the target code (uniquifer & random. I'd rather have them match (identify structure will have spec's abbreviations; ctrl & debug/sysfs for target & initiator either be ciu/cirn or uniquifer/random.
Quoted 2 lines
I think it matters for sysfs attributes. I do not know the right thing to do. Should we use spec names like "cirn" or call it "random"?
Now I am thinking about it, I think sticking to the spec's abbreviations makes more sense here. names like "random" and "uniquifier" in sysfs are not descriptive enough. I changed struct member names, sysfs, and debugfs file names to match the spec.
Quoted 5 lines
Quoted 5 lines
But this is small stuff.
Reviewed-by Randy Jennings <[email protected]>
RFC PATCH 7/14 nvme: Add RECOVERING nvme controller state
In reply to Mohamed KhalfellaAdd NVME_CTRL_RECOVERING as a new controller state to be used when impacted controller is being recovered. A LIVE controller enters RECOVERING state when an IO error is encountered. While recovering inflight IOs will not be canceled if they timeout. These IOs will be canceled after recovery finishes. Also, while recovering a controller can not be reset or deleted. This is intentional because reset or delete will result in canceling inflight IOs. When recovery finishes, the impacted controller transitions from RECOVERING state to RESETTING state. Reset codepath takes care of queues teardown and inflight requests cancellation.
Note, there is no transition from RECOVERING to RESETTING added to nvme_change_ctrl_state(). The reason is that user should not be allowed to reset or delete a controller that is being recovered.
Add NVME_CTRL_RECOVERED controller flag. This flag is set on a controller about to schedule delayed work for time based recovery.
--- drivers/nvme/host/core.c | 10 ++++++++++ drivers/nvme/host/nvme.h | 2 ++ drivers/nvme/host/sysfs.c | 1 + 3 files changed, 13 insertions(+)
drivers/nvme/host/core.c +10 -0
@@ -574,6 +574,15 @@ bool nvme_change_ctrl_state(struct nvme_ctrl *ctrl,break ; }break ; +case NVME_CTRL_RECOVERING : +switch (old_state) { +case NVME_CTRL_LIVE : + changed = true; + fallthrough; +default : +break ; + } +break ;case NVME_CTRL_RESETTING :switch (old_state) {case NVME_CTRL_NEW :
@@ -761,6 +770,7 @@ blk_status_t nvme_fail_nonready_command(struct nvme_ctrl *ctrl,if (state !=NVME_CTRL_DELETING_NOIO && state !=NVME_CTRL_DELETING && state !=NVME_CTRL_DEAD && + state !=NVME_CTRL_RECOVERING && !test_bit (NVME_CTRL_FAILFAST_EXPIRED , &ctrl->flags) && !blk_noretry_request (rq) && !(rq->cmd_flags &REQ_NVME_MPATH ))return BLK_STS_RESOURCE ;
drivers/nvme/host/nvme.h +2 -0
@@ -251,6 +251,7 @@ static inline u16 nvme_req_qid(struct request *req)enum nvme_ctrl_state {NVME_CTRL_NEW ,NVME_CTRL_LIVE , +NVME_CTRL_RECOVERING ,NVME_CTRL_RESETTING ,NVME_CTRL_CONNECTING ,NVME_CTRL_DELETING ,
@@ -275,6 +276,7 @@ enum nvme_ctrl_flags {NVME_CTRL_SKIP_ID_CNS_CS =4 ,NVME_CTRL_DIRTY_CAPABILITY =5 ,NVME_CTRL_FROZEN =6 , + NVME_CTRL_RECOVERED = 7, }; struct nvme_ctrl {
drivers/nvme/host/sysfs.c +1 -0
@@ -443,6 +443,7 @@ static ssize_t nvme_sysfs_show_state(struct device *dev,static const char *const state_name[] = { [NVME_CTRL_NEW ] ="new" , [NVME_CTRL_LIVE ] ="live" , + [NVME_CTRL_RECOVERING ] ="recovering" , [NVME_CTRL_RESETTING ] ="resetting" , [NVME_CTRL_CONNECTING ] ="connecting" , [NVME_CTRL_DELETING ] ="deleting" ,
Signature · 1 line
2.51.2
Re: [RFC PATCH 07/14] nvme: Add RECOVERING nvme controller state
Quoted 12 lines · Mohamed Khalfella
Add
NVME_CTRL_RECOVERINGas a new controller state to be used when impacted controller is being recovered. A LIVE controller enters RECOVERING state when an IO error is encountered. While recovering inflight IOs will not be canceled if they timeout. These IOs will be canceled after recovery finishes. Also, while recovering a controller can not be reset or deleted. This is intentional because reset or delete will result in canceling inflight IOs. When recovery finishes, the impacted controller transitions from RECOVERING state to RESETTING state. Reset codepath takes care of queues teardown and inflight requests cancellation.
Quoted 8 lines
Note, there is no transition from RECOVERING to RESETTING added to
nvme_change_ctrl_state(). The reason is that user should not be allowed to reset or delete a controller that is being recovered.Add
NVME_CTRL_RECOVEREDcontroller flag. This flag is set on a controller about to schedule delayed work for time based recovery.Signed-off-by Mohamed Khalfella <[email protected]>
Re: [RFC PATCH 07/14] nvme: Add RECOVERING nvme controller state
Quoted 6 lines · Randy Jennings
Quoted 3 lines · Mohamed Khalfella
Reset codepath takes care of queues teardown and inflight requests cancellation.
Note: Tearing down the connection (with the queues) after going through CCR-based recovery or time-based recovery is late. CCR will trigger a disconnect on the host side, and, because we stop traffic to the nvme controller, KATO should kick in if CCR does not, so I accept your argument that tearing down the connections first could be considered an optimization that can get implemented later.
Sincerely,
Randy Jennings
Sagi Grimberg
maintainerRe: [RFC PATCH 07/14] nvme: Add RECOVERING nvme controller state
In reply to Mohamed KhalfellaQuoted 10 lines · Mohamed Khalfella
Add
NVME_CTRL_RECOVERINGas a new controller state to be used when impacted controller is being recovered. A LIVE controller enters RECOVERING state when an IO error is encountered. While recovering inflight IOs will not be canceled if they timeout. These IOs will be canceled after recovery finishes. Also, while recovering a controller can not be reset or deleted. This is intentional because reset or delete will result in canceling inflight IOs. When recovery finishes, the impacted controller transitions from RECOVERING state to RESETTING state. Reset codepath takes care of queues teardown and inflight requests cancellation.
Is RECOVERING really capturing the nature of this state? Maybe RESETTLING? or QUIESCING?
Quoted 22 lines
Note, there is no transition from RECOVERING to RESETTING added to
nvme_change_ctrl_state(). The reason is that user should not be allowed to reset or delete a controller that is being recovered.Add
NVME_CTRL_RECOVEREDcontroller flag. This flag is set on a controller about to schedule delayed work for time based recovery.Signed-off-by Mohamed Khalfella <[email protected]>--- drivers/nvme/host/core.c | 10 ++++++++++ drivers/nvme/host/nvme.h | 2 ++ drivers/nvme/host/sysfs.c | 1 + 3 files changed, 13 insertions(+)drivers/nvme/host/core.c +0
-0@@ -574,6 +574,15 @@ bool nvme_change_ctrl_state(struct nvme_ctrl *ctrl,break ; }
drivers/nvme/host/core.c bool nvme_change_ctrl_state(struct nvme_ctrl *ctrl, · line 585
Quoted 10 lines
drivers/nvme/host/core.c +9
-0break ; +case NVME_CTRL_RECOVERING : +switch (old_state) { +case NVME_CTRL_LIVE : + changed = true; + fallthrough; +default : +break ; + } +break ;
That is a strange transition...
Re: [RFC PATCH 07/14] nvme: Add RECOVERING nvme controller state
Quoted 14 lines · Sagi Grimberg
Quoted 10 lines · Mohamed Khalfella
Add
NVME_CTRL_RECOVERINGas a new controller state to be used when impacted controller is being recovered. A LIVE controller enters RECOVERING state when an IO error is encountered. While recovering inflight IOs will not be canceled if they timeout. These IOs will be canceled after recovery finishes. Also, while recovering a controller can not be reset or deleted. This is intentional because reset or delete will result in canceling inflight IOs. When recovery finishes, the impacted controller transitions from RECOVERING state to RESETTING state. Reset codepath takes care of queues teardown and inflight requests cancellation.
Quoted 2 lines
Is RECOVERING really capturing the nature of this state? Maybe RESETTLING? or QUIESCING?
Naming is hard. QUIESCING sounds better, I will renaming it to QUIESCING.
Quoted 27 lines
Quoted 27 lines
Note, there is no transition from RECOVERING to RESETTING added to
nvme_change_ctrl_state(). The reason is that user should not be allowed to reset or delete a controller that is being recovered.Add
NVME_CTRL_RECOVEREDcontroller flag. This flag is set on a controller about to schedule delayed work for time based recovery.Signed-off-by Mohamed Khalfella <[email protected]>--- drivers/nvme/host/core.c | 10 ++++++++++ drivers/nvme/host/nvme.h | 2 ++ drivers/nvme/host/sysfs.c | 1 + 3 files changed, 13 insertions(+)drivers/nvme/host/core.c +3
-0@@ -574,6 +574,15 @@ bool nvme_change_ctrl_state(struct nvme_ctrl *ctrl,break ; }break ; +case NVME_CTRL_RECOVERING : +switch (old_state) { +case NVME_CTRL_LIVE :
Quoted 8 lines
drivers/nvme/host/core.c bool nvme_change_ctrl_state(struct nvme_ctrl *ctrl, · line 585
Quoted 6 lines
drivers/nvme/host/core.c +6
-0+ changed = true; + fallthrough; +default : +break ; + } +break ;That is a strange transition...
Why is it strange?
We transition to RECOVERING state only if controller is LIVE. This is when we expect to have inflight user IOs to be quiesced by CCR. We do not care about inflight requests in other states.
Sagi Grimberg
maintainerRe: [RFC PATCH 07/14] nvme: Add RECOVERING nvme controller state
Quoted 50 lines · Mohamed Khalfella
Quoted 12 lines · Sagi Grimberg
Quoted 10 lines · Mohamed Khalfella
Add
NVME_CTRL_RECOVERINGas a new controller state to be used when impacted controller is being recovered. A LIVE controller enters RECOVERING state when an IO error is encountered. While recovering inflight IOs will not be canceled if they timeout. These IOs will be canceled after recovery finishes. Also, while recovering a controller can not be reset or deleted. This is intentional because reset or delete will result in canceling inflight IOs. When recovery finishes, the impacted controller transitions from RECOVERING state to RESETTING state. Reset codepath takes care of queues teardown and inflight requests cancellation.Quoted 2 lines
Is RECOVERING really capturing the nature of this state? Maybe RESETTLING? or QUIESCING?
Naming is hard. QUIESCING sounds better, I will renaming it to QUIESCING.
Quoted 24 lines
Quoted 24 lines
Note, there is no transition from RECOVERING to RESETTING added to
nvme_change_ctrl_state(). The reason is that user should not be allowed to reset or delete a controller that is being recovered.Add
NVME_CTRL_RECOVEREDcontroller flag. This flag is set on a controller about to schedule delayed work for time based recovery.Signed-off-by Mohamed Khalfella <[email protected]>--- drivers/nvme/host/core.c | 10 ++++++++++ drivers/nvme/host/nvme.h | 2 ++ drivers/nvme/host/sysfs.c | 1 + 3 files changed, 13 insertions(+)drivers/nvme/host/core.c +2
-0@@ -574,6 +574,15 @@ bool nvme_change_ctrl_state(struct nvme_ctrl *ctrl,break ; }break ; +case NVME_CTRL_RECOVERING : +switch (old_state) {Quoted 8 lines
drivers/nvme/host/core.c bool nvme_change_ctrl_state(struct nvme_ctrl *ctrl, · line 585
Quoted 7 lines
drivers/nvme/host/core.c +7
-0+case NVME_CTRL_LIVE : + changed = true; + fallthrough; +default : +break ; + } +break ;That is a strange transition...
Quoted 5 lines
Why is it strange?
We transition to RECOVERING state only if controller is LIVE. This is when we expect to have inflight user IOs to be quiesced by CCR. We do not care about inflight requests in other states.
Sorry, got confused myself - I read it as the other way around... I am missing RECOVERING -> RESETTING transition in this patch.
Re: [RFC PATCH 07/14] nvme: Add RECOVERING nvme controller state
Quoted 53 lines · Sagi Grimberg
Quoted 50 lines · Mohamed Khalfella
Quoted 12 lines · Sagi Grimberg
Quoted 10 lines · Mohamed Khalfella
Add
NVME_CTRL_RECOVERINGas a new controller state to be used when impacted controller is being recovered. A LIVE controller enters RECOVERING state when an IO error is encountered. While recovering inflight IOs will not be canceled if they timeout. These IOs will be canceled after recovery finishes. Also, while recovering a controller can not be reset or deleted. This is intentional because reset or delete will result in canceling inflight IOs. When recovery finishes, the impacted controller transitions from RECOVERING state to RESETTING state. Reset codepath takes care of queues teardown and inflight requests cancellation.Quoted 2 lines
Is RECOVERING really capturing the nature of this state? Maybe RESETTLING? or QUIESCING?
Naming is hard. QUIESCING sounds better, I will renaming it to QUIESCING.
Quoted 32 lines
Quoted 21 lines
Note, there is no transition from RECOVERING to RESETTING added to
nvme_change_ctrl_state(). The reason is that user should not be allowed to reset or delete a controller that is being recovered.Add
NVME_CTRL_RECOVEREDcontroller flag. This flag is set on a controller about to schedule delayed work for time based recovery.Signed-off-by Mohamed Khalfella <[email protected]>--- drivers/nvme/host/core.c | 10 ++++++++++ drivers/nvme/host/nvme.h | 2 ++ drivers/nvme/host/sysfs.c | 1 + 3 files changed, 13 insertions(+)drivers/nvme/host/core.c +0
-0@@ -574,6 +574,15 @@ bool nvme_change_ctrl_state(struct nvme_ctrl *ctrl,break ; }drivers/nvme/host/core.c bool nvme_change_ctrl_state(struct nvme_ctrl *ctrl, · line 585
Quoted 10 lines
drivers/nvme/host/core.c +9
-0break ; +case NVME_CTRL_RECOVERING : +switch (old_state) { +case NVME_CTRL_LIVE : + changed = true; + fallthrough; +default : +break ; + } +break ;That is a strange transition...
Quoted 8 lines
Quoted 5 lines
Why is it strange?
We transition to RECOVERING state only if controller is LIVE. This is when we expect to have inflight user IOs to be quiesced by CCR. We do not care about inflight requests in other states.
Sorry, got confused myself - I read it as the other way around... I am missing RECOVERING -> RESETTING transition in this patch.
This is in patch 8 ("nvme: Implement cross-controller reset recovery"). It was not added to nvme_change_ctrl_state() because we do not want the controller to be reset while in RECOVERING state.
Sagi Grimberg
maintainerRe: [RFC PATCH 07/14] nvme: Add RECOVERING nvme controller state
In reply to Mohamed KhalfellaQuoted 13 lines · Mohamed Khalfella
Quoted 12 lines · Sagi Grimberg
Quoted 10 lines · Mohamed Khalfella
Add
NVME_CTRL_RECOVERINGas a new controller state to be used when impacted controller is being recovered. A LIVE controller enters RECOVERING state when an IO error is encountered. While recovering inflight IOs will not be canceled if they timeout. These IOs will be canceled after recovery finishes. Also, while recovering a controller can not be reset or deleted. This is intentional because reset or delete will result in canceling inflight IOs. When recovery finishes, the impacted controller transitions from RECOVERING state to RESETTING state. Reset codepath takes care of queues teardown and inflight requests cancellation.
Quoted 4 lines
Quoted 2 lines
Is RECOVERING really capturing the nature of this state? Maybe RESETTLING? or QUIESCING?
Naming is hard. QUIESCING sounds better, I will renaming it to QUIESCING.
I actually think that FENCING is probably best to describe what the state is used for...
Re: [RFC PATCH 07/14] nvme: Add RECOVERING nvme controller state
Quoted 16 lines · Sagi Grimberg
Quoted 13 lines · Mohamed Khalfella
Quoted 12 lines · Sagi Grimberg
Quoted 10 lines · Mohamed Khalfella
Add
NVME_CTRL_RECOVERINGas a new controller state to be used when impacted controller is being recovered. A LIVE controller enters RECOVERING state when an IO error is encountered. While recovering inflight IOs will not be canceled if they timeout. These IOs will be canceled after recovery finishes. Also, while recovering a controller can not be reset or deleted. This is intentional because reset or delete will result in canceling inflight IOs. When recovery finishes, the impacted controller transitions from RECOVERING state to RESETTING state. Reset codepath takes care of queues teardown and inflight requests cancellation.
Quoted 7 lines
Quoted 4 lines
Quoted 2 lines
Is RECOVERING really capturing the nature of this state? Maybe RESETTLING? or QUIESCING?
Naming is hard. QUIESCING sounds better, I will renaming it to QUIESCING.
I actually think that FENCING is probably best to describe what the state is used for...
FENCING is used in HA clusters with persistent reservations. I find it confusing to use it here. Let me know if you have strong preference.
Re: [RFC PATCH 07/14] nvme: Add RECOVERING nvme controller state
Quoted 19 lines · Mohamed Khalfella
Quoted 18 lines · Sagi Grimberg
Quoted 15 lines · Mohamed Khalfella
Quoted 14 lines · Sagi Grimberg
Quoted 10 lines · Mohamed Khalfella
Add
NVME_CTRL_RECOVERINGas a new controller state to be used when impacted controller is being recovered. A LIVE controller enters RECOVERING state when an IO error is encountered. While recovering inflight IOs will not be canceled if they timeout. These IOs will be canceled after recovery finishes. Also, while recovering a controller can not be reset or deleted. This is intentional because reset or delete will result in canceling inflight IOs. When recovery finishes, the impacted controller transitions from RECOVERING state to RESETTING state. Reset codepath takes care of queues teardown and inflight requests cancellation.Is RECOVERING really capturing the nature of this state? Maybe RESETTLING? or QUIESCING?
Quoted 8 lines
Quoted 5 lines
Quoted 2 lines
Naming is hard. QUIESCING sounds better, I will renaming it to QUIESCING.
I actually think that FENCING is probably best to describe what the state is used for...
FENCING is used in HA clusters with persistent reservations. I find it confusing to use it here. Let me know if you have strong preference.
Nevermind, I will rename it to FENCING.
RFC PATCH 8/14 nvme: Implement cross-controller reset recovery
In reply to Mohamed KhalfellaA host that has more than one path connecting to an nvme subsystem typically has an nvme controller associated with every path. This is mostly applicable to nvmeof. If one path goes down, inflight IOs on that path should not be retried immediately on another path because this could lead to data corruption as described in TP4129. TP8028 defines cross-controller reset mechanism that can be used by host to terminate IOs on the failed path using one of the remaining healthy paths. Only after IOs are terminated, or long enough time passes as defined by TP4129, inflight IOs should be retried on another path. Implement core cross-controller reset shared logic to be used by the transports.
--- drivers/nvme/host/constants.c | 1 + drivers/nvme/host/core.c | 133 ++++++++++++++++++++++++++++++++++ drivers/nvme/host/nvme.h | 10 +++ 3 files changed, 144 insertions(+)
drivers/nvme/host/constants.c +1 -0
@@ -46,6 +46,7 @@ static const char * const nvme_admin_ops[] = { [nvme_admin_virtual_mgmt] = "Virtual Management", [nvme_admin_nvme_mi_send] = "NVMe Send MI", [nvme_admin_nvme_mi_recv] = "NVMe Receive MI", + [nvme_admin_cross_ctrl_reset] = "Cross Controller Reset", [nvme_admin_dbbuf] = "Doorbell Buffer Config", [nvme_admin_format_nvm] = "Format NVM", [nvme_admin_security_send] = "Security Send",
drivers/nvme/host/core.c +133 -0
145 lines not on this page · raw message
drivers/nvme/host/nvme.h +10 -0
@@ -279,6 +279,13 @@ enum nvme_ctrl_flags { NVME_CTRL_RECOVERED = 7, }; +struct nvme_ccr_entry { +struct list_head list; +struct completion complete; +struct nvme_ctrl *ictrl; +u8 ccrs; +}; +struct nvme_ctrl {bool comp_seen;bool identified;
@@ -296,6 +303,7 @@ struct nvme_ctrl {struct blk_mq_tag_set *tagset;struct blk_mq_tag_set *admin_tagset;struct list_head namespaces; +struct list_head ccrs;struct mutex namespaces_lock;struct srcu_struct srcu;struct device ctrl_device;
@@ -805,6 +813,8 @@ blk_status_t nvme_host_path_error(struct request *req);bool nvme_cancel_request (struct request *req,void *data);void nvme_cancel_tagset (struct nvme_ctrl *ctrl);void nvme_cancel_admin_tagset (struct nvme_ctrl *ctrl); +unsigned long nvme_recover_ctrl (struct nvme_ctrl *ctrl); +void nvme_end_ctrl_recovery (struct nvme_ctrl *ctrl);bool nvme_change_ctrl_state (struct nvme_ctrl *ctrl,enum nvme_ctrl_state new_state);int nvme_disable_ctrl (struct nvme_ctrl *ctrl,bool shutdown);
Signature · 1 line
2.51.2
Re: [RFC PATCH 08/14] nvme: Implement cross-controller reset recovery
Quoted 15 lines · Mohamed Khalfella
A host that has more than one path connecting to an nvme subsystem typically has an nvme controller associated with every path. This is mostly applicable to nvmeof. If one path goes down, inflight IOs on that path should not be retried immediately on another path because this could lead to data corruption as described in TP4129. TP8028 defines cross-controller reset mechanism that can be used by host to terminate IOs on the failed path using one of the remaining healthy paths. Only after IOs are terminated, or long enough time passes as defined by TP4129, inflight IOs should be retried on another path. Implement core cross-controller reset shared logic to be used by the transports.
Signed-off-by Mohamed Khalfella <[email protected]>
drivers/nvme/host/core.c void nvme_cancel_admin_tagset(struct nvme_ctrl *ctrl) · line 671
Quoted 9 lines
drivers/nvme/host/core.c +6
-0+unsigned long nvme_recover_ctrl (struct nvme_ctrl *ictrl) + now = jiffies; + deadline = now +msecs_to_jiffies (timeout); +while (time_before (now, deadline)) { + now = jiffies; + }
I would use a for-loop to keep the advancing statement close to the condition.
Sagi Grimberg
maintainerRe: [RFC PATCH 08/14] nvme: Implement cross-controller reset recovery
In reply to Mohamed KhalfellaQuoted 36 lines · Mohamed Khalfella
A host that has more than one path connecting to an nvme subsystem typically has an nvme controller associated with every path. This is mostly applicable to nvmeof. If one path goes down, inflight IOs on that path should not be retried immediately on another path because this could lead to data corruption as described in TP4129. TP8028 defines cross-controller reset mechanism that can be used by host to terminate IOs on the failed path using one of the remaining healthy paths. Only after IOs are terminated, or long enough time passes as defined by TP4129, inflight IOs should be retried on another path. Implement core cross-controller reset shared logic to be used by the transports.
Signed-off-by Mohamed Khalfella <[email protected]>--- drivers/nvme/host/constants.c | 1 + drivers/nvme/host/core.c | 133 ++++++++++++++++++++++++++++++++++ drivers/nvme/host/nvme.h | 10 +++ 3 files changed, 144 insertions(+)drivers/nvme/host/constants.c +1
-0@@ -46,6 +46,7 @@ static const char * const nvme_admin_ops[] = { [nvme_admin_virtual_mgmt] = "Virtual Management", [nvme_admin_nvme_mi_send] = "NVMe Send MI", [nvme_admin_nvme_mi_recv] = "NVMe Receive MI", + [nvme_admin_cross_ctrl_reset] = "Cross Controller Reset", [nvme_admin_dbbuf] = "Doorbell Buffer Config", [nvme_admin_format_nvm] = "Format NVM", [nvme_admin_security_send] = "Security Send",drivers/nvme/host/core.c +0
-0@@ -554,6 +554,138 @@ void nvme_cancel_admin_tagset(struct nvme_ctrl *ctrl) }
drivers/nvme/host/core.c void nvme_cancel_admin_tagset(struct nvme_ctrl *ctrl) · line 564
Quoted 10 lines
drivers/nvme/host/core.c +8
-0EXPORT_SYMBOL_GPL (nvme_cancel_admin_tagset); +static struct nvme_ctrl *nvme_find_ccr_ctrl (struct nvme_ctrl *ictrl, +u32 min_cntlid) +{ +struct nvme_subsystem *subsys = ictrl->subsys; +struct nvme_ctrl *sctrl; +unsigned long flags; + +mutex_lock (&nvme_subsystems_lock);
This looks like the wrong lock to take here?
drivers/nvme/host/core.c void nvme_cancel_admin_tagset(struct nvme_ctrl *ctrl) · line 567
Quoted 3 lines
drivers/nvme/host/core.c +3
-0+list_for_each_entry (sctrl, &subsys->ctrls, subsys_entry) { +if (sctrl->cntlid < min_cntlid) +continue ;
The use of min_cntlid is not clear to me.
Quoted 50 lines
drivers/nvme/host/core.c +50
-0+ +if (atomic_dec_if_positive (&sctrl->ccr_limit) <0 ) +continue ; + +spin_lock_irqsave (&sctrl->lock, flags); +if (sctrl->state !=NVME_CTRL_LIVE ) { +spin_unlock_irqrestore (&sctrl->lock, flags); +atomic_inc (&sctrl->ccr_limit); +continue ; + } + +/* +* We got a good candidate source controller that is locked and +* LIVE. However, no guarantee sctrl will not be deleted after +* sctrl->lock is released. Get a ref of both sctrl and admin_q +* so they do not disappear until we are done with them. +*/ +WARN_ON_ONCE (!blk_get_queue (sctrl->admin_q)); +nvme_get_ctrl (sctrl); +spin_unlock_irqrestore (&sctrl->lock, flags); + goto found; + } + sctrl =NULL ; +found: +mutex_unlock (&nvme_subsystems_lock); +return sctrl; +} + +static int nvme_issue_wait_ccr (struct nvme_ctrl *sctrl,struct nvme_ctrl *ictrl) +{ +unsigned long flags, tmo, remain; +struct nvme_ccr_entry ccr = { }; +union nvme_result res = {0 }; +struct nvme_command c = { }; +u32 result; +int ret =0 ; + +init_completion (&ccr.complete); + ccr.ictrl = ictrl; + +spin_lock_irqsave (&sctrl->lock, flags); +list_add_tail (&ccr.list, &sctrl->ccrs); +spin_unlock_irqrestore (&sctrl->lock, flags); + + c.ccr.opcode = nvme_admin_cross_ctrl_reset; + c.ccr.ciu = ictrl->ciu; + c.ccr.icid =cpu_to_le16 (ictrl->cntlid); + c.ccr.cirn =cpu_to_le64 (ictrl->cirn); + ret =__nvme_submit_sync_cmd (sctrl->admin_q, &c, &res, +NULL ,0 ,NVME_QID_ANY ,0 );
drivers/nvme/host/core.c void nvme_cancel_admin_tagset(struct nvme_ctrl *ctrl) · line 627
Quoted 10 lines
drivers/nvme/host/core.c +10
-0+if (ret) + goto out; + + result =le32_to_cpu (res.u32); +if (result &0x01 )/* Immediate Reset */ + goto out; + + tmo =msecs_to_jiffies (max (ictrl->cqt, ictrl->kato *1000 )); + remain =wait_for_completion_timeout (&ccr.complete, tmo); +if (!remain)
I think remain is redundant here.
drivers/nvme/host/core.c void nvme_cancel_admin_tagset(struct nvme_ctrl *ctrl) · line 633
Quoted 6 lines
drivers/nvme/host/core.c +6
-0+ ret = -EAGAIN ; +out: +spin_lock_irqsave (&sctrl->lock, flags); +list_del (&ccr.list); +spin_unlock_irqrestore (&sctrl->lock, flags); +return ccr.ccrs ==1 ?0 : ret;
Why would you still return 0 and not EAGAIN? you expired on timeout but still return success if you have ccrs=1? btw you have ccrs in the ccr struct and in the controller as a list. Lets rename to distinguish the two.
drivers/nvme/host/core.c void nvme_cancel_admin_tagset(struct nvme_ctrl *ctrl) · line 637
Quoted 4 lines
drivers/nvme/host/core.c +4
-0+} + +unsigned long nvme_recover_ctrl (struct nvme_ctrl *ictrl) +{
I'd call it nvme_fence_controller()
Quoted 5 lines
drivers/nvme/host/core.c +5
-0+unsigned long deadline, now, timeout; +struct nvme_ctrl *sctrl; +u32 min_cntlid =0 ; +int ret; +
drivers/nvme/host/core.c void nvme_cancel_admin_tagset(struct nvme_ctrl *ctrl) · line 652
Quoted 10 lines
drivers/nvme/host/core.c +10
-0+ timeout =nvme_recovery_timeout_ms (ictrl); +dev_info (ictrl->device,"attempting CCR, timeout %lums\n" , timeout); + + now = jiffies; + deadline = now +msecs_to_jiffies (timeout); +while (time_before (now, deadline)) { + sctrl =nvme_find_ccr_ctrl (ictrl, min_cntlid); +if (!sctrl) { +/* CCR failed, switch to time-based recovery */ +return deadline - now;
It is not clear what is the return code semantics of this function. How about making it success/failure and have the caller choose what to do?
drivers/nvme/host/core.c void nvme_cancel_admin_tagset(struct nvme_ctrl *ctrl) · line 656
Quoted 4 lines
drivers/nvme/host/core.c +4
-0+ } + + ret =nvme_issue_wait_ccr (sctrl, ictrl); +atomic_inc (&sctrl->ccr_limit);
inc after you wait for the ccr? shouldn't this be before?
drivers/nvme/host/core.c void nvme_cancel_admin_tagset(struct nvme_ctrl *ctrl) · line 667
Quoted 11 lines
drivers/nvme/host/core.c +11
-0+ +if (!ret) { +dev_info (ictrl->device,"CCR succeeded using %s\n" , +dev_name (sctrl->device)); +blk_put_queue (sctrl->admin_q); +nvme_put_ctrl (sctrl); +return 0 ; + } + +/* Try another controller */ + min_cntlid = sctrl->cntlid +1 ;
OK, I see why min_cntlid is used. That is very non-intuitive.
I'm wandering if it will be simpler to take one-shot at ccr and if it fails fallback to crt. I mean, if the sctrl is alive, and it was unable to reset the ictrl in time, how would another ctrl do a better job here?
Quoted 6 lines
drivers/nvme/host/core.c +6
-0+blk_put_queue (sctrl->admin_q); +nvme_put_ctrl (sctrl); + now = jiffies; + } + +dev_info (ictrl->device,"CCR reached timeout, call it done\n" );
drivers/nvme/host/core.c void nvme_cancel_admin_tagset(struct nvme_ctrl *ctrl) · line 683
Quoted 10 lines
drivers/nvme/host/core.c +10
-0+return 0 ; +} +EXPORT_SYMBOL_GPL (nvme_recover_ctrl); + +void nvme_end_ctrl_recovery (struct nvme_ctrl *ctrl) +{ +unsigned long flags; + +spin_lock_irqsave (&ctrl->lock, flags); +WRITE_ONCE (ctrl->state,NVME_CTRL_RESETTING );
This needs to be a proper state transition.
Re: [RFC PATCH 08/14] nvme: Implement cross-controller reset recovery
Quoted 14 lines
drivers/nvme/host/core.c void nvme_cancel_admin_tagset(struct nvme_ctrl *ctrl) · line 667
Quoted 11 lines
drivers/nvme/host/core.c +11
-0+ +if (!ret) { +dev_info (ictrl->device,"CCR succeeded using %s\n" , +dev_name (sctrl->device)); +blk_put_queue (sctrl->admin_q); +nvme_put_ctrl (sctrl); +return 0 ; + } + +/* Try another controller */ + min_cntlid = sctrl->cntlid +1 ;OK, I see why
min_cntlidis used. That is very non-intuitive.
Quoted 4 lines
I'm wandering if it will be simpler to take one-shot at ccr and if it fails fallback to crt. I mean, if the sctrl is alive, and it was unable
to reset the ictrl in time, how would another ctrl do a better job here?
There are many different kinds of failures we are dealing with here that result in a dropped connection (association). It could be a problem with the specific link, or it could be that the node of an HA pair in the storage array went down. In the case of a specific link problem, maybe only one of the connections is down and any controller would work. In the case of the node of an HA pair, roughly half of the connections are going down, and there is a race between the controllers which are detected down first. There were some heuristics put into the spec about deciding which controller to use, but that is more code and a refinement that could come later (and they are still heuristics; they may not be helpful).
Because CCR offers a significant win of shortening the recovery time substantially, it is worth retrying on the other controllers. This time affects when we can start retrying IO. KATO is in seconds, and NVMEoF should have the capability of doing a significant amount of IOs in each of those seconds.
Besides, the alternative is just to wait. Might as well be actively trying to shorten that wait time. Besides a small increase in code complexity, is there a downside to doing so?
Sincerely,
Randy Jennings
the time.
Sagi Grimberg
maintainerRe: [RFC PATCH 08/14] nvme: Implement cross-controller reset recovery
Quoted 29 lines · Randy Jennings
Quoted 13 lines
drivers/nvme/host/core.c void nvme_cancel_admin_tagset(struct nvme_ctrl *ctrl) · line 667
Quoted 11 lines
drivers/nvme/host/core.c +11
-0+ +if (!ret) { +dev_info (ictrl->device,"CCR succeeded using %s\n" , +dev_name (sctrl->device)); +blk_put_queue (sctrl->admin_q); +nvme_put_ctrl (sctrl); +return 0 ; + } + +/* Try another controller */ + min_cntlid = sctrl->cntlid +1 ;OK, I see why
min_cntlidis used. That is very non-intuitive.Quoted 4 lines
I'm wandering if it will be simpler to take one-shot at ccr and if it fails fallback to crt. I mean, if the sctrl is alive, and it was unable
to reset the ictrl in time, how would another ctrl do a better job here?There are many different kinds of failures we are dealing with here that result in a dropped connection (association). It could be a problem with the specific link, or it could be that the node of an HA pair in the storage array went down. In the case of a specific link problem, maybe only one of the connections is down and any controller would work. In the case of the node of an HA pair, roughly half of the connections are going down, and there is a race between the controllers which are detected down first. There were some heuristics put into the spec about deciding which controller to use, but that is more code and a refinement that could come later (and they are still heuristics; they may not be helpful).
Quoted 5 lines
Because CCR offers a significant win of shortening the recovery time substantially, it is worth retrying on the other controllers. This time affects when we can start retrying IO. KATO is in seconds, and NVMEoF should have the capability of doing a significant amount of IOs in each of those seconds.
But it doesn't actually do I/O, it issues I/O and then wait for it to time out.
Quoted 4 lines
Besides, the alternative is just to wait. Might as well be actively trying to shorten that wait time. Besides a small increase in code complexity, is there a downside to doing so?
Simplicity is very important when it comes to non-trivial code paths like error recovery.
Re: [RFC PATCH 08/14] nvme: Implement cross-controller reset recovery
Quoted 30 lines · Sagi Grimberg
Quoted 29 lines · Randy Jennings
Quoted 13 lines
drivers/nvme/host/core.c void nvme_cancel_admin_tagset(struct nvme_ctrl *ctrl) · line 667
Quoted 11 lines
drivers/nvme/host/core.c +11
-0+ +if (!ret) { +dev_info (ictrl->device,"CCR succeeded using %s\n" , +dev_name (sctrl->device)); +blk_put_queue (sctrl->admin_q); +nvme_put_ctrl (sctrl); +return 0 ; + } + +/* Try another controller */ + min_cntlid = sctrl->cntlid +1 ;OK, I see why
min_cntlidis used. That is very non-intuitive.Quoted 4 lines
I'm wandering if it will be simpler to take one-shot at ccr and if it fails fallback to crt. I mean, if the sctrl is alive, and it was unable
to reset the ictrl in time, how would another ctrl do a better job here?There are many different kinds of failures we are dealing with here that result in a dropped connection (association). It could be a problem with the specific link, or it could be that the node of an HA pair in the storage array went down. In the case of a specific link problem, maybe only one of the connections is down and any controller would work. In the case of the node of an HA pair, roughly half of the connections are going down, and there is a race between the controllers which are detected down first. There were some heuristics put into the spec about deciding which controller to use, but that is more code and a refinement that could come later (and they are still heuristics; they may not be helpful).
Quoted 8 lines
Quoted 5 lines
Because CCR offers a significant win of shortening the recovery time substantially, it is worth retrying on the other controllers. This time affects when we can start retrying IO. KATO is in seconds, and NVMEoF should have the capability of doing a significant amount of IOs in each of those seconds.
But it doesn't actually do I/O, it issues I/O and then wait for it to time out.
Retrying CCR does not actually do I/O (trying to place your antecedent), but a successful CCR allows the host to get back to doing I/O. Every second saved can be a significant amount of I/O. If you were given a choice between a 1 second failover and a 60 second failover, of course, you would go for the 1 second failover. However, if I was given the option of a 10 second failover and a 60 second failover, I would still go for the 10 second failover. 50 seconds is still extremely valuable.
Quoted 8 lines
Quoted 5 lines
Besides, the alternative is just to wait. Might as well be actively trying to shorten that wait time. Besides a small increase in code complexity, is there a downside to doing so?
Simplicity is very important when it comes to non-trivial code paths like error recovery.
Okay, yes, unwarranted complexity, even with some benefit might not be worth it. I can see that my comment could be taken as flippant. But the extra complexity here yields an important and material benefit.
Sincerely,
Randy Jennings
Re: [RFC PATCH 08/14] nvme: Implement cross-controller reset recovery
In reply to Sagi GrimbergQuoted 41 lines · Sagi Grimberg
Quoted 38 lines · Mohamed Khalfella
A host that has more than one path connecting to an nvme subsystem typically has an nvme controller associated with every path. This is mostly applicable to nvmeof. If one path goes down, inflight IOs on that path should not be retried immediately on another path because this could lead to data corruption as described in TP4129. TP8028 defines cross-controller reset mechanism that can be used by host to terminate IOs on the failed path using one of the remaining healthy paths. Only after IOs are terminated, or long enough time passes as defined by TP4129, inflight IOs should be retried on another path. Implement core cross-controller reset shared logic to be used by the transports.
Signed-off-by Mohamed Khalfella <[email protected]>--- drivers/nvme/host/constants.c | 1 + drivers/nvme/host/core.c | 133 ++++++++++++++++++++++++++++++++++ drivers/nvme/host/nvme.h | 10 +++ 3 files changed, 144 insertions(+)drivers/nvme/host/constants.c +1
-0@@ -46,6 +46,7 @@ static const char * const nvme_admin_ops[] = { [nvme_admin_virtual_mgmt] = "Virtual Management", [nvme_admin_nvme_mi_send] = "NVMe Send MI", [nvme_admin_nvme_mi_recv] = "NVMe Receive MI", + [nvme_admin_cross_ctrl_reset] = "Cross Controller Reset", [nvme_admin_dbbuf] = "Doorbell Buffer Config", [nvme_admin_format_nvm] = "Format NVM", [nvme_admin_security_send] = "Security Send",drivers/nvme/host/core.c +0
-0@@ -554,6 +554,138 @@ void nvme_cancel_admin_tagset(struct nvme_ctrl *ctrl) }EXPORT_SYMBOL_GPL (nvme_cancel_admin_tagset);
Quoted 10 lines
drivers/nvme/host/core.c void nvme_cancel_admin_tagset(struct nvme_ctrl *ctrl) · line 564
Quoted 8 lines
drivers/nvme/host/core.c +8
-0+static struct nvme_ctrl *nvme_find_ccr_ctrl (struct nvme_ctrl *ictrl, +u32 min_cntlid) +{ +struct nvme_subsystem *subsys = ictrl->subsys; +struct nvme_ctrl *sctrl; +unsigned long flags; + +mutex_lock (&nvme_subsystems_lock);This looks like the wrong lock to take here?
This is similar to nvme_validate_cntlid()?
What is the correct lock to use?
Quoted 61 lines
drivers/nvme/host/core.c void nvme_cancel_admin_tagset(struct nvme_ctrl *ctrl) · line 567
Quoted 4 lines
drivers/nvme/host/core.c +3
-0+list_for_each_entry (sctrl, &subsys->ctrls, subsys_entry) { +if (sctrl->cntlid < min_cntlid) +continue ;The use of
min_cntlidis not clear to me.Quoted 54 lines
drivers/nvme/host/core.c +54
-0+ +if (atomic_dec_if_positive (&sctrl->ccr_limit) <0 ) +continue ; + +spin_lock_irqsave (&sctrl->lock, flags); +if (sctrl->state !=NVME_CTRL_LIVE ) { +spin_unlock_irqrestore (&sctrl->lock, flags); +atomic_inc (&sctrl->ccr_limit); +continue ; + } + +/* +* We got a good candidate source controller that is locked and +* LIVE. However, no guarantee sctrl will not be deleted after +* sctrl->lock is released. Get a ref of both sctrl and admin_q +* so they do not disappear until we are done with them. +*/ +WARN_ON_ONCE (!blk_get_queue (sctrl->admin_q)); +nvme_get_ctrl (sctrl); +spin_unlock_irqrestore (&sctrl->lock, flags); + goto found; + } + sctrl =NULL ; +found: +mutex_unlock (&nvme_subsystems_lock); +return sctrl; +} + +static int nvme_issue_wait_ccr (struct nvme_ctrl *sctrl,struct nvme_ctrl *ictrl) +{ +unsigned long flags, tmo, remain; +struct nvme_ccr_entry ccr = { }; +union nvme_result res = {0 }; +struct nvme_command c = { }; +u32 result; +int ret =0 ; + +init_completion (&ccr.complete); + ccr.ictrl = ictrl; + +spin_lock_irqsave (&sctrl->lock, flags); +list_add_tail (&ccr.list, &sctrl->ccrs); +spin_unlock_irqrestore (&sctrl->lock, flags); + + c.ccr.opcode = nvme_admin_cross_ctrl_reset; + c.ccr.ciu = ictrl->ciu; + c.ccr.icid =cpu_to_le16 (ictrl->cntlid); + c.ccr.cirn =cpu_to_le64 (ictrl->cirn); + ret =__nvme_submit_sync_cmd (sctrl->admin_q, &c, &res, +NULL ,0 ,NVME_QID_ANY ,0 ); +if (ret) + goto out; + + result =le32_to_cpu (res.u32);
Quoted 8 lines
drivers/nvme/host/core.c void nvme_cancel_admin_tagset(struct nvme_ctrl *ctrl) · line 627
Quoted 6 lines
drivers/nvme/host/core.c +6
-0+if (result &0x01 )/* Immediate Reset */ + goto out; + + tmo =msecs_to_jiffies (max (ictrl->cqt, ictrl->kato *1000 )); + remain =wait_for_completion_timeout (&ccr.complete, tmo); +if (!remain)I think remain is redundant here.
Deleted 'remain'.
Quoted 13 lines
drivers/nvme/host/core.c void nvme_cancel_admin_tagset(struct nvme_ctrl *ctrl) · line 633
Quoted 7 lines
drivers/nvme/host/core.c +6
-0+ ret = -EAGAIN ; +out: +spin_lock_irqsave (&sctrl->lock, flags); +list_del (&ccr.list); +spin_unlock_irqrestore (&sctrl->lock, flags); +return ccr.ccrs ==1 ?0 : ret;Why would you still return 0 and not EAGAIN? you expired on timeout but still
return success if you have ccrs=1? btw you have ccrs in the ccr struct and in the controller
as a list. Lets rename to distinguish the two.
True, we did expire timeout here. However, after we removed the ccr entry we found that it was marked as completed. We return success in this case even though we hit timeout.
Renamed ctrl->ccrs to ctrl->ccr_list.
Quoted 7 lines
drivers/nvme/host/core.c void nvme_cancel_admin_tagset(struct nvme_ctrl *ctrl) · line 637
Quoted 5 lines
drivers/nvme/host/core.c +4
-0+} + +unsigned long nvme_recover_ctrl (struct nvme_ctrl *ictrl) +{I'd call it
nvme_fence_controller()
Okay. I will do that. I will also rename the controller state FENCING.
Quoted 11 lines
Quoted 11 lines
drivers/nvme/host/core.c +10
-0+unsigned long deadline, now, timeout; +struct nvme_ctrl *sctrl; +u32 min_cntlid =0 ; +int ret; + + timeout =nvme_recovery_timeout_ms (ictrl); +dev_info (ictrl->device,"attempting CCR, timeout %lums\n" , timeout); + + now = jiffies; + deadline = now +msecs_to_jiffies (timeout);
Quoted 8 lines
drivers/nvme/host/core.c void nvme_cancel_admin_tagset(struct nvme_ctrl *ctrl) · line 652
Quoted 5 lines
drivers/nvme/host/core.c +5
-0+while (time_before (now, deadline)) { + sctrl =nvme_find_ccr_ctrl (ictrl, min_cntlid); +if (!sctrl) { +/* CCR failed, switch to time-based recovery */ +return deadline - now;It is not clear what is the return code semantics of this function. How about making it success/failure and have the caller choose what to do?
The function returns 0 on success. On failure it returns the time in jiffies to hold requests for before they are canceled. On failure the returned time is essentially the hold time defined in TP4129 minus the time it took to attempt CCR.
Quoted 7 lines
drivers/nvme/host/core.c void nvme_cancel_admin_tagset(struct nvme_ctrl *ctrl) · line 656
Quoted 5 lines
drivers/nvme/host/core.c +4
-0+ } + + ret =nvme_issue_wait_ccr (sctrl, ictrl); +atomic_inc (&sctrl->ccr_limit);inc after you wait for the ccr? shouldn't this be before?
I think it should be after we wait for CCR. sctrl->ccr_limit is the number of concurrent CCRs the controller supports. Only after we are done with CCR on this controller we increment it.
Quoted 15 lines
drivers/nvme/host/core.c void nvme_cancel_admin_tagset(struct nvme_ctrl *ctrl) · line 667
Quoted 12 lines
drivers/nvme/host/core.c +11
-0+ +if (!ret) { +dev_info (ictrl->device,"CCR succeeded using %s\n" , +dev_name (sctrl->device)); +blk_put_queue (sctrl->admin_q); +nvme_put_ctrl (sctrl); +return 0 ; + } + +/* Try another controller */ + min_cntlid = sctrl->cntlid +1 ;OK, I see why
min_cntlidis used. That is very non-intuitive.
Quoted 4 lines
I'm wandering if it will be simpler to take one-shot at ccr and if it fails fallback to crt. I mean, if the sctrl is alive, and it was unable
to reset the ictrl in time, how would another ctrl do a better job here?
We need to attempt CCR from multiple controllers for reason explained in another response. As you figured out min_cntlid is needed in order to not loop controller list forever. Do you have a better idea?
Quoted 11 lines
Quoted 11 lines
drivers/nvme/host/core.c +10
-0+blk_put_queue (sctrl->admin_q); +nvme_put_ctrl (sctrl); + now = jiffies; + } + +dev_info (ictrl->device,"CCR reached timeout, call it done\n" ); +return 0 ; +} +EXPORT_SYMBOL_GPL (nvme_recover_ctrl); +
Quoted 8 lines
drivers/nvme/host/core.c void nvme_cancel_admin_tagset(struct nvme_ctrl *ctrl) · line 683
Quoted 6 lines
drivers/nvme/host/core.c +6
-0+void nvme_end_ctrl_recovery (struct nvme_ctrl *ctrl) +{ +unsigned long flags; + +spin_lock_irqsave (&ctrl->lock, flags); +WRITE_ONCE (ctrl->state,NVME_CTRL_RESETTING );This needs to be a proper state transition.
We do not want to have proper transition from RECOVERING to RESETTING. The reason is that we do not want the controller to be reset while it is being recovered/fenced because requests should not be canceled. One way to keep the transitions in nvme_change_ctrl_state() is to use two states. Say FENCING and FENCED.
The allowed transitions are
- LIVE -> FENCING
- FENCING -> FENCED
- FENCED -> (RESETTING, DELETING)
This will also git rid of NVME_CTRL_RECOVERED
Does this sound good?
Sagi Grimberg
maintainerRe: [RFC PATCH 08/14] nvme: Implement cross-controller reset recovery
Quoted 50 lines · Mohamed Khalfella
Quoted 40 lines · Sagi Grimberg
Quoted 38 lines · Mohamed Khalfella
A host that has more than one path connecting to an nvme subsystem typically has an nvme controller associated with every path. This is mostly applicable to nvmeof. If one path goes down, inflight IOs on that path should not be retried immediately on another path because this could lead to data corruption as described in TP4129. TP8028 defines cross-controller reset mechanism that can be used by host to terminate IOs on the failed path using one of the remaining healthy paths. Only after IOs are terminated, or long enough time passes as defined by TP4129, inflight IOs should be retried on another path. Implement core cross-controller reset shared logic to be used by the transports.
Signed-off-by Mohamed Khalfella <[email protected]>--- drivers/nvme/host/constants.c | 1 + drivers/nvme/host/core.c | 133 ++++++++++++++++++++++++++++++++++ drivers/nvme/host/nvme.h | 10 +++ 3 files changed, 144 insertions(+)drivers/nvme/host/constants.c +1
-0@@ -46,6 +46,7 @@ static const char * const nvme_admin_ops[] = { [nvme_admin_virtual_mgmt] = "Virtual Management", [nvme_admin_nvme_mi_send] = "NVMe Send MI", [nvme_admin_nvme_mi_recv] = "NVMe Receive MI", + [nvme_admin_cross_ctrl_reset] = "Cross Controller Reset", [nvme_admin_dbbuf] = "Doorbell Buffer Config", [nvme_admin_format_nvm] = "Format NVM", [nvme_admin_security_send] = "Security Send",drivers/nvme/host/core.c +0
-0@@ -554,6 +554,138 @@ void nvme_cancel_admin_tagset(struct nvme_ctrl *ctrl) }EXPORT_SYMBOL_GPL (nvme_cancel_admin_tagset);Quoted 9 lines
drivers/nvme/host/core.c void nvme_cancel_admin_tagset(struct nvme_ctrl *ctrl) · line 564
Quoted 8 lines
drivers/nvme/host/core.c +8
-0+static struct nvme_ctrl *nvme_find_ccr_ctrl (struct nvme_ctrl *ictrl, +u32 min_cntlid) +{ +struct nvme_subsystem *subsys = ictrl->subsys; +struct nvme_ctrl *sctrl; +unsigned long flags; + +mutex_lock (&nvme_subsystems_lock);This looks like the wrong lock to take here?
Quoted 2 lines
This is similar to
nvme_validate_cntlid()?
What is the correct lock to use?
Not really, its only because it is called from nvme_init_subsystem which spans subsystems.
Quoted 80 lines
Quoted 59 lines
drivers/nvme/host/core.c void nvme_cancel_admin_tagset(struct nvme_ctrl *ctrl) · line 567
Quoted 4 lines
drivers/nvme/host/core.c +3
-0+list_for_each_entry (sctrl, &subsys->ctrls, subsys_entry) { +if (sctrl->cntlid < min_cntlid) +continue ;The use of
min_cntlidis not clear to me.Quoted 53 lines
drivers/nvme/host/core.c +53
-0+ +if (atomic_dec_if_positive (&sctrl->ccr_limit) <0 ) +continue ; + +spin_lock_irqsave (&sctrl->lock, flags); +if (sctrl->state !=NVME_CTRL_LIVE ) { +spin_unlock_irqrestore (&sctrl->lock, flags); +atomic_inc (&sctrl->ccr_limit); +continue ; + } + +/* +* We got a good candidate source controller that is locked and +* LIVE. However, no guarantee sctrl will not be deleted after +* sctrl->lock is released. Get a ref of both sctrl and admin_q +* so they do not disappear until we are done with them. +*/ +WARN_ON_ONCE (!blk_get_queue (sctrl->admin_q)); +nvme_get_ctrl (sctrl); +spin_unlock_irqrestore (&sctrl->lock, flags); + goto found; + } + sctrl =NULL ; +found: +mutex_unlock (&nvme_subsystems_lock); +return sctrl; +} + +static int nvme_issue_wait_ccr (struct nvme_ctrl *sctrl,struct nvme_ctrl *ictrl) +{ +unsigned long flags, tmo, remain; +struct nvme_ccr_entry ccr = { }; +union nvme_result res = {0 }; +struct nvme_command c = { }; +u32 result; +int ret =0 ; + +init_completion (&ccr.complete); + ccr.ictrl = ictrl; + +spin_lock_irqsave (&sctrl->lock, flags); +list_add_tail (&ccr.list, &sctrl->ccrs); +spin_unlock_irqrestore (&sctrl->lock, flags); + + c.ccr.opcode = nvme_admin_cross_ctrl_reset; + c.ccr.ciu = ictrl->ciu; + c.ccr.icid =cpu_to_le16 (ictrl->cntlid); + c.ccr.cirn =cpu_to_le64 (ictrl->cirn); + ret =__nvme_submit_sync_cmd (sctrl->admin_q, &c, &res, +NULL ,0 ,NVME_QID_ANY ,0 ); +if (ret) + goto out; +Quoted 8 lines
drivers/nvme/host/core.c void nvme_cancel_admin_tagset(struct nvme_ctrl *ctrl) · line 627
Quoted 7 lines
drivers/nvme/host/core.c +7
-0+ result =le32_to_cpu (res.u32); +if (result &0x01 )/* Immediate Reset */ + goto out; + + tmo =msecs_to_jiffies (max (ictrl->cqt, ictrl->kato *1000 )); + remain =wait_for_completion_timeout (&ccr.complete, tmo); +if (!remain)I think remain is redundant here.
Deleted 'remain'.
Quoted 11 lines
drivers/nvme/host/core.c void nvme_cancel_admin_tagset(struct nvme_ctrl *ctrl) · line 633
Quoted 6 lines
drivers/nvme/host/core.c +6
-0+ ret = -EAGAIN ; +out: +spin_lock_irqsave (&sctrl->lock, flags); +list_del (&ccr.list); +spin_unlock_irqrestore (&sctrl->lock, flags); +return ccr.ccrs ==1 ?0 : ret;Why would you still return 0 and not EAGAIN? you expired on timeout but still
return success if you have ccrs=1? btw you have ccrs in the ccr struct and in the controller
as a list. Lets rename to distinguish the two.
Quoted 3 lines
True, we did expire timeout here. However, after we removed the ccr entry we found that it was marked as completed. We return success in this case even though we hit timeout.
When does this happen? Why is it worth having the code non-intuitive for something that effectively never happens (unless I'm missing something?)
Quoted 27 lines
Renamed
ctrl->ccrstoctrl->ccr_list.Quoted 5 lines
drivers/nvme/host/core.c void nvme_cancel_admin_tagset(struct nvme_ctrl *ctrl) · line 637
Quoted 4 lines
drivers/nvme/host/core.c +4
-0+} + +unsigned long nvme_recover_ctrl (struct nvme_ctrl *ictrl) +{I'd call it
nvme_fence_controller()Okay. I will do that. I will also rename the controller state FENCING.
Quoted 9 lines
Quoted 9 lines
drivers/nvme/host/core.c +9
-0+unsigned long deadline, now, timeout; +struct nvme_ctrl *sctrl; +u32 min_cntlid =0 ; +int ret; + + timeout =nvme_recovery_timeout_ms (ictrl); +dev_info (ictrl->device,"attempting CCR, timeout %lums\n" , timeout); + + now = jiffies;Quoted 8 lines
drivers/nvme/host/core.c void nvme_cancel_admin_tagset(struct nvme_ctrl *ctrl) · line 652
Quoted 6 lines
drivers/nvme/host/core.c +6
-0+ deadline = now +msecs_to_jiffies (timeout); +while (time_before (now, deadline)) { + sctrl =nvme_find_ccr_ctrl (ictrl, min_cntlid); +if (!sctrl) { +/* CCR failed, switch to time-based recovery */ +return deadline - now;It is not clear what is the return code semantics of this function. How about making it success/failure and have the caller choose what to do?
Quoted 4 lines
The function returns 0 on success. On failure it returns the time in jiffies to hold requests for before they are canceled. On failure the returned time is essentially the hold time defined in TP4129 minus the time it took to attempt CCR.
I think it would be cleaner to simple have this function return status code and have the caller worry about time spent.
Quoted 9 lines
Quoted 6 lines
drivers/nvme/host/core.c void nvme_cancel_admin_tagset(struct nvme_ctrl *ctrl) · line 656
Quoted 5 lines
drivers/nvme/host/core.c +4
-0+ } + + ret =nvme_issue_wait_ccr (sctrl, ictrl); +atomic_inc (&sctrl->ccr_limit);inc after you wait for the ccr? shouldn't this be before?
I think it should be after we wait for CCR.
sctrl->ccr_limitis the number of concurrent CCRs the controller supports. Only after we are done with CCR on this controller we increment it.
Maybe it should be folded into nvme_issue_wait_ccr for symmetry?
Quoted 14 lines
Quoted 14 lines
drivers/nvme/host/core.c void nvme_cancel_admin_tagset(struct nvme_ctrl *ctrl) · line 667
Quoted 12 lines
drivers/nvme/host/core.c +11
-0+ +if (!ret) { +dev_info (ictrl->device,"CCR succeeded using %s\n" , +dev_name (sctrl->device)); +blk_put_queue (sctrl->admin_q); +nvme_put_ctrl (sctrl); +return 0 ; + } + +/* Try another controller */ + min_cntlid = sctrl->cntlid +1 ;OK, I see why
min_cntlidis used. That is very non-intuitive.
Quoted 7 lines
Quoted 4 lines
I'm wandering if it will be simpler to take one-shot at ccr and if it fails fallback to crt. I mean, if the sctrl is alive, and it was unable
to reset the ictrl in time, how would another ctrl do a better job here?We need to attempt CCR from multiple controllers for reason explained in another response. As you figured out
min_cntlidis needed in order to not loop controller list forever. Do you have a better idea?
No, just know that I don't like it very much :)
Quoted 26 lines
Quoted 10 lines
Quoted 10 lines
drivers/nvme/host/core.c +9
-0+blk_put_queue (sctrl->admin_q); +nvme_put_ctrl (sctrl); + now = jiffies; + } + +dev_info (ictrl->device,"CCR reached timeout, call it done\n" ); +return 0 ; +} +EXPORT_SYMBOL_GPL (nvme_recover_ctrl);Quoted 8 lines
drivers/nvme/host/core.c void nvme_cancel_admin_tagset(struct nvme_ctrl *ctrl) · line 683
Quoted 7 lines
drivers/nvme/host/core.c +7
-0+ +void nvme_end_ctrl_recovery (struct nvme_ctrl *ctrl) +{ +unsigned long flags; + +spin_lock_irqsave (&ctrl->lock, flags); +WRITE_ONCE (ctrl->state,NVME_CTRL_RESETTING );This needs to be a proper state transition.
We do not want to have proper transition from RECOVERING to RESETTING. The reason is that we do not want the controller to be reset while it is being recovered/fenced because requests should not be canceled. One way to keep the transitions in
nvme_change_ctrl_state()is to use two states. Say FENCING and FENCED.The allowed transitions are
Quoted 7 lines
- LIVE -> FENCING
- FENCING -> FENCED
- FENCED -> (RESETTING, DELETING)This will also git rid of
NVME_CTRL_RECOVEREDDoes this sound good?
We could do what failfast is doing, in case we get transition FENCING -> RESETTING/DELETING we flush the fence_work...
Re: [RFC PATCH 08/14] nvme: Implement cross-controller reset recovery
Quoted 53 lines · Sagi Grimberg
Quoted 50 lines · Mohamed Khalfella
Quoted 49 lines · Sagi Grimberg
Quoted 36 lines · Mohamed Khalfella
A host that has more than one path connecting to an nvme subsystem typically has an nvme controller associated with every path. This is mostly applicable to nvmeof. If one path goes down, inflight IOs on that path should not be retried immediately on another path because this could lead to data corruption as described in TP4129. TP8028 defines cross-controller reset mechanism that can be used by host to terminate IOs on the failed path using one of the remaining healthy paths. Only after IOs are terminated, or long enough time passes as defined by TP4129, inflight IOs should be retried on another path. Implement core cross-controller reset shared logic to be used by the transports.
Signed-off-by Mohamed Khalfella <[email protected]>--- drivers/nvme/host/constants.c | 1 + drivers/nvme/host/core.c | 133 ++++++++++++++++++++++++++++++++++ drivers/nvme/host/nvme.h | 10 +++ 3 files changed, 144 insertions(+)drivers/nvme/host/constants.c +1
-0@@ -46,6 +46,7 @@ static const char * const nvme_admin_ops[] = { [nvme_admin_virtual_mgmt] = "Virtual Management", [nvme_admin_nvme_mi_send] = "NVMe Send MI", [nvme_admin_nvme_mi_recv] = "NVMe Receive MI", + [nvme_admin_cross_ctrl_reset] = "Cross Controller Reset", [nvme_admin_dbbuf] = "Doorbell Buffer Config", [nvme_admin_format_nvm] = "Format NVM", [nvme_admin_security_send] = "Security Send",drivers/nvme/host/core.c +0
-0@@ -554,6 +554,138 @@ void nvme_cancel_admin_tagset(struct nvme_ctrl *ctrl) }drivers/nvme/host/core.c void nvme_cancel_admin_tagset(struct nvme_ctrl *ctrl) · line 564
Quoted 10 lines
drivers/nvme/host/core.c +8
-0EXPORT_SYMBOL_GPL (nvme_cancel_admin_tagset); +static struct nvme_ctrl *nvme_find_ccr_ctrl (struct nvme_ctrl *ictrl, +u32 min_cntlid) +{ +struct nvme_subsystem *subsys = ictrl->subsys; +struct nvme_ctrl *sctrl; +unsigned long flags; + +mutex_lock (&nvme_subsystems_lock);This looks like the wrong lock to take here?
Quoted 6 lines
Quoted 2 lines
This is similar to
nvme_validate_cntlid()?
What is the correct lock to use?Not really, its only because it is called from
nvme_init_subsystemwhich spans
subsystems.
Okay. I will use this lock for now. If this is not the right lock to use please point me to the right one.
Quoted 81 lines
Quoted 81 lines
Quoted 60 lines
drivers/nvme/host/core.c void nvme_cancel_admin_tagset(struct nvme_ctrl *ctrl) · line 567
Quoted 5 lines
drivers/nvme/host/core.c +3
-0+list_for_each_entry (sctrl, &subsys->ctrls, subsys_entry) { +if (sctrl->cntlid < min_cntlid) +continue ;The use of
min_cntlidis not clear to me.Quoted 53 lines
drivers/nvme/host/core.c +53
-0+ +if (atomic_dec_if_positive (&sctrl->ccr_limit) <0 ) +continue ; + +spin_lock_irqsave (&sctrl->lock, flags); +if (sctrl->state !=NVME_CTRL_LIVE ) { +spin_unlock_irqrestore (&sctrl->lock, flags); +atomic_inc (&sctrl->ccr_limit); +continue ; + } + +/* +* We got a good candidate source controller that is locked and +* LIVE. However, no guarantee sctrl will not be deleted after +* sctrl->lock is released. Get a ref of both sctrl and admin_q +* so they do not disappear until we are done with them. +*/ +WARN_ON_ONCE (!blk_get_queue (sctrl->admin_q)); +nvme_get_ctrl (sctrl); +spin_unlock_irqrestore (&sctrl->lock, flags); + goto found; + } + sctrl =NULL ; +found: +mutex_unlock (&nvme_subsystems_lock); +return sctrl; +} + +static int nvme_issue_wait_ccr (struct nvme_ctrl *sctrl,struct nvme_ctrl *ictrl) +{ +unsigned long flags, tmo, remain; +struct nvme_ccr_entry ccr = { }; +union nvme_result res = {0 }; +struct nvme_command c = { }; +u32 result; +int ret =0 ; + +init_completion (&ccr.complete); + ccr.ictrl = ictrl; + +spin_lock_irqsave (&sctrl->lock, flags); +list_add_tail (&ccr.list, &sctrl->ccrs); +spin_unlock_irqrestore (&sctrl->lock, flags); + + c.ccr.opcode = nvme_admin_cross_ctrl_reset; + c.ccr.ciu = ictrl->ciu; + c.ccr.icid =cpu_to_le16 (ictrl->cntlid); + c.ccr.cirn =cpu_to_le64 (ictrl->cirn); + ret =__nvme_submit_sync_cmd (sctrl->admin_q, &c, &res, +NULL ,0 ,NVME_QID_ANY ,0 ); +if (ret) + goto out; +Quoted 8 lines
drivers/nvme/host/core.c void nvme_cancel_admin_tagset(struct nvme_ctrl *ctrl) · line 627
Quoted 7 lines
drivers/nvme/host/core.c +7
-0+ result =le32_to_cpu (res.u32); +if (result &0x01 )/* Immediate Reset */ + goto out; + + tmo =msecs_to_jiffies (max (ictrl->cqt, ictrl->kato *1000 )); + remain =wait_for_completion_timeout (&ccr.complete, tmo); +if (!remain)I think remain is redundant here.
Deleted 'remain'.
Quoted 11 lines
drivers/nvme/host/core.c void nvme_cancel_admin_tagset(struct nvme_ctrl *ctrl) · line 633
Quoted 6 lines
drivers/nvme/host/core.c +6
-0+ ret = -EAGAIN ; +out: +spin_lock_irqsave (&sctrl->lock, flags); +list_del (&ccr.list); +spin_unlock_irqrestore (&sctrl->lock, flags); +return ccr.ccrs ==1 ?0 : ret;Why would you still return 0 and not EAGAIN? you expired on timeout but still
return success if you have ccrs=1? btw you have ccrs in the ccr struct and in the controller
as a list. Lets rename to distinguish the two.
Quoted 6 lines
Quoted 3 lines
True, we did expire timeout here. However, after we removed the ccr entry we found that it was marked as completed. We return success in this case even though we hit timeout.
When does this happen? Why is it worth having the code non-intuitive for something that effectively never happens (unless I'm missing something?)
Agree. It is a very low probability. I deleted the check for this condition.
Quoted 28 lines
Quoted 28 lines
Renamed
ctrl->ccrstoctrl->ccr_list.Quoted 5 lines
drivers/nvme/host/core.c void nvme_cancel_admin_tagset(struct nvme_ctrl *ctrl) · line 637
Quoted 4 lines
drivers/nvme/host/core.c +4
-0+} + +unsigned long nvme_recover_ctrl (struct nvme_ctrl *ictrl) +{I'd call it
nvme_fence_controller()Okay. I will do that. I will also rename the controller state FENCING.
Quoted 17 lines
Quoted 5 lines
drivers/nvme/host/core.c +5
-0+unsigned long deadline, now, timeout; +struct nvme_ctrl *sctrl; +u32 min_cntlid =0 ; +int ret; +drivers/nvme/host/core.c void nvme_cancel_admin_tagset(struct nvme_ctrl *ctrl) · line 652
Quoted 10 lines
drivers/nvme/host/core.c +10
-0+ timeout =nvme_recovery_timeout_ms (ictrl); +dev_info (ictrl->device,"attempting CCR, timeout %lums\n" , timeout); + + now = jiffies; + deadline = now +msecs_to_jiffies (timeout); +while (time_before (now, deadline)) { + sctrl =nvme_find_ccr_ctrl (ictrl, min_cntlid); +if (!sctrl) { +/* CCR failed, switch to time-based recovery */ +return deadline - now;It is not clear what is the return code semantics of this function. How about making it success/failure and have the caller choose what to do?
Quoted 8 lines
Quoted 4 lines
The function returns 0 on success. On failure it returns the time in jiffies to hold requests for before they are canceled. On failure the returned time is essentially the hold time defined in TP4129 minus the time it took to attempt CCR.
I think it would be cleaner to simple have this function return status code and
have the caller worry about time spent.
nvme_fence_ctrl() needs to track the time. It needs to be aware of how much time spent on attempting CCR in order to decide whether to continue trying CCR or give up.
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
drivers/nvme/host/core.c void nvme_cancel_admin_tagset(struct nvme_ctrl *ctrl) · line 656
Quoted 6 lines
drivers/nvme/host/core.c +4
-0+ } + + ret =nvme_issue_wait_ccr (sctrl, ictrl); +atomic_inc (&sctrl->ccr_limit);inc after you wait for the ccr? shouldn't this be before?
Quoted 5 lines
Quoted 3 lines
I think it should be after we wait for CCR.
sctrl->ccr_limitis the number of concurrent CCRs the controller supports. Only after we are done with CCR on this controller we increment it.Maybe it should be folded into nvme_issue_wait_ccr for symmetry?
Done.
Quoted 55 lines
Quoted 15 lines
Quoted 15 lines
drivers/nvme/host/core.c void nvme_cancel_admin_tagset(struct nvme_ctrl *ctrl) · line 667
Quoted 13 lines
drivers/nvme/host/core.c +11
-0+ +if (!ret) { +dev_info (ictrl->device,"CCR succeeded using %s\n" , +dev_name (sctrl->device)); +blk_put_queue (sctrl->admin_q); +nvme_put_ctrl (sctrl); +return 0 ; + } + +/* Try another controller */ + min_cntlid = sctrl->cntlid +1 ;OK, I see why
min_cntlidis used. That is very non-intuitive.Quoted 7 lines
Quoted 4 lines
I'm wandering if it will be simpler to take one-shot at ccr and if it fails fallback to crt. I mean, if the sctrl is alive, and it was unable
to reset the ictrl in time, how would another ctrl do a better job here?We need to attempt CCR from multiple controllers for reason explained in another response. As you figured out
min_cntlidis needed in order to not loop controller list forever. Do you have a better idea?No, just know that I don't like it very much :)
Quoted 30 lines
Quoted 10 lines
Quoted 10 lines
drivers/nvme/host/core.c +9
-0+blk_put_queue (sctrl->admin_q); +nvme_put_ctrl (sctrl); + now = jiffies; + } + +dev_info (ictrl->device,"CCR reached timeout, call it done\n" ); +return 0 ; +} +EXPORT_SYMBOL_GPL (nvme_recover_ctrl);Quoted 8 lines
drivers/nvme/host/core.c void nvme_cancel_admin_tagset(struct nvme_ctrl *ctrl) · line 683
Quoted 7 lines
drivers/nvme/host/core.c +7
-0+ +void nvme_end_ctrl_recovery (struct nvme_ctrl *ctrl) +{ +unsigned long flags; + +spin_lock_irqsave (&ctrl->lock, flags); +WRITE_ONCE (ctrl->state,NVME_CTRL_RESETTING );This needs to be a proper state transition.
We do not want to have proper transition from RECOVERING to RESETTING. The reason is that we do not want the controller to be reset while it is being recovered/fenced because requests should not be canceled. One way to keep the transitions in
nvme_change_ctrl_state()is to use two states. Say FENCING and FENCED.The allowed transitions are
- LIVE -> FENCING
- FENCING -> FENCED
- FENCED -> (RESETTING, DELETING)
Quoted 7 lines
Quoted 3 lines
This will also git rid of
NVME_CTRL_RECOVEREDDoes this sound good?
We could do what failfast is doing, in case we get transition FENCING -> RESETTING/DELETING we flush
thefence_work...
Yes. This is what v2 does.
RFC PATCH 9/14 nvme: Implement cross-controller reset completion
In reply to Mohamed KhalfellaAn nvme source controller that issues CCR command expects to receive an NVME_AER_NOTICE_CCR_COMPLETED when pending CCR succeeds or fails. Add sctrl->ccr_work to read NVME_LOG_CCR logpage and wakeup any thread waiting on CCR completion.
--- drivers/nvme/host/core.c | 49 +++++++++++++++++++++++++++++++++++++++- drivers/nvme/host/nvme.h | 1 + 2 files changed, 49 insertions(+), 1 deletion(-)
drivers/nvme/host/core.c +48 -1
@@ -1894,7 +1894,8 @@ EXPORT_SYMBOL_GPL(nvme_set_queue_count);#define NVME_AEN_SUPPORTED \ (NVME_AEN_CFG_NS_ATTR | NVME_AEN_CFG_FW_ACT | \ - NVME_AEN_CFG_ANA_CHANGE | NVME_AEN_CFG_DISC_CHANGE) + NVME_AEN_CFG_ANA_CHANGE | NVME_AEN_CFG_CCR_COMPLETE | \ + NVME_AEN_CFG_DISC_CHANGE)static void nvme_enable_aen (struct nvme_ctrl *ctrl) {
@@ -4860,6 +4861,47 @@ static void nvme_get_fw_slot_info(struct nvme_ctrl *ctrl)kfree (log); } +static void nvme_ccr_work (struct work_struct *work) +{ + struct nvme_ctrl *ctrl = container_of(work, struct nvme_ctrl, ccr_work); +struct nvme_ccr_entry *ccr; +struct nvme_ccr_log_entry *entry; +struct nvme_ccr_log *log; +unsigned long flags; +int ret, i; + + log =kmalloc (sizeof (*log),GFP_KERNEL ); +if (!log) +return ; + + ret =nvme_get_log (ctrl,0 ,NVME_LOG_CCR ,0x01 , +0x00 , log,sizeof (*log),0 ); +if (ret) + goto out; + +spin_lock_irqsave (&ctrl->lock, flags); +for (i =0 ; i <le16_to_cpu (log->ne); i++) { + entry = &log->entries[i]; +if (entry->ccrs ==0 )/* skip in progress entries */ +continue ; + +list_for_each_entry (ccr, &ctrl->ccrs, list) { +struct nvme_ctrl *ictrl = ccr->ictrl; + +if (ictrl->cntlid !=le16_to_cpu (entry->icid) || + ictrl->ciu != entry->ciu) +continue ; + +/* Complete matching entry */ + ccr->ccrs = entry->ccrs; +complete (&ccr->complete); + } + } +spin_unlock_irqrestore (&ctrl->lock, flags); +out: +kfree (log); +} +static void nvme_fw_act_work (struct work_struct *work) { struct nvme_ctrl *ctrl = container_of(work,
@@ -4936,6 +4978,9 @@ static bool nvme_handle_aen_notice(struct nvme_ctrl *ctrl, u32 result)case NVME_AER_NOTICE_DISC_CHANGED : ctrl->aen_result = result;break ; +case NVME_AER_NOTICE_CCR_COMPLETED : +queue_work (nvme_wq, &ctrl->ccr_work); +break ;default :dev_warn (ctrl->device,"async event result %08x\n" , result); }
@@ -5126,6 +5171,7 @@ void nvme_stop_ctrl(struct nvme_ctrl *ctrl)nvme_stop_failfast_work (ctrl);flush_work (&ctrl->async_event_work);cancel_work_sync (&ctrl->fw_act_work); +cancel_work_sync (&ctrl->ccr_work);if (ctrl->ops->stop_ctrl) ctrl->ops->stop_ctrl (ctrl); }
@@ -5247,6 +5293,7 @@ int nvme_init_ctrl(struct nvme_ctrl *ctrl, struct device *dev, ctrl->quirks = quirks; ctrl->numa_node =NUMA_NO_NODE ;INIT_WORK (&ctrl->scan_work, nvme_scan_work); +INIT_WORK (&ctrl->ccr_work, nvme_ccr_work);INIT_WORK (&ctrl->async_event_work, nvme_async_event_work);INIT_WORK (&ctrl->fw_act_work, nvme_fw_act_work);INIT_WORK (&ctrl->delete_work, nvme_delete_ctrl_work);
drivers/nvme/host/nvme.h +1 -0
@@ -366,6 +366,7 @@ struct nvme_ctrl {struct nvme_effects_log *effects;struct xarray cels;struct work_struct scan_work; +struct work_struct ccr_work;struct work_struct async_event_work;struct delayed_work ka_work;struct delayed_work failfast_work;
Signature · 1 line
2.51.2
Re: [RFC PATCH 09/14] nvme: Implement cross-controller reset completion
Quoted 7 lines · Mohamed Khalfella
An nvme source controller that issues CCR command expects to receive an
NVME_AER_NOTICE_CCR_COMPLETEDwhen pending CCR succeeds or fails. Addsctrl->ccr_workto readNVME_LOG_CCRlogpage and wakeup any thread waiting on CCR completion.Signed-off-by Mohamed Khalfella <[email protected]>
Sagi Grimberg
maintainerRe: [RFC PATCH 09/14] nvme: Implement cross-controller reset completion
In reply to Mohamed KhalfellaQuoted 5 lines
drivers/nvme/host/core.c +4
-0+ log =kmalloc (sizeof (*log),GFP_KERNEL ); +if (!log) +return ; +
drivers/nvme/host/core.c static void nvme_get_fw_slot_info(struct nvme_ctrl *ctrl) · line 4886
Quoted 10 lines
drivers/nvme/host/core.c +10
-0+ ret =nvme_get_log (ctrl,0 ,NVME_LOG_CCR ,0x01 , +0x00 , log,sizeof (*log),0 ); +if (ret) + goto out; + +spin_lock_irqsave (&ctrl->lock, flags); +for (i =0 ; i <le16_to_cpu (log->ne); i++) { + entry = &log->entries[i]; +if (entry->ccrs ==0 )/* skip in progress entries */ +continue ;
What does ccrs stand for?
Re: [RFC PATCH 09/14] nvme: Implement cross-controller reset completion
Quoted 17 lines · Sagi Grimberg
Quoted 5 lines
drivers/nvme/host/core.c +4
-0+ log =kmalloc (sizeof (*log),GFP_KERNEL ); +if (!log) +return ; +drivers/nvme/host/core.c static void nvme_get_fw_slot_info(struct nvme_ctrl *ctrl) · line 4886
Quoted 10 lines
drivers/nvme/host/core.c +10
-0+ ret =nvme_get_log (ctrl,0 ,NVME_LOG_CCR ,0x01 , +0x00 , log,sizeof (*log),0 ); +if (ret) + goto out; + +spin_lock_irqsave (&ctrl->lock, flags); +for (i =0 ; i <le16_to_cpu (log->ne); i++) { + entry = &log->entries[i]; +if (entry->ccrs ==0 )/* skip in progress entries */ +continue ;What does ccrs stand for?
Cross-Controller Reset Status
0x00 -> In Progress0x01 -> Success0x02 -> Failed0x03 - 0xff -> Reserved
Sagi Grimberg
maintainerRe: [RFC PATCH 09/14] nvme: Implement cross-controller reset completion
Quoted 16 lines · Mohamed Khalfella
Quoted 7 lines · Sagi Grimberg
Quoted 7 lines
drivers/nvme/host/core.c +7
-0+ log =kmalloc (sizeof (*log),GFP_KERNEL ); +if (!log) +return ; + + ret =nvme_get_log (ctrl,0 ,NVME_LOG_CCR ,0x01 , +0x00 , log,sizeof (*log),0 ); +if (ret)Quoted 8 lines
drivers/nvme/host/core.c static void nvme_get_fw_slot_info(struct nvme_ctrl *ctrl) · line 4886
Quoted 7 lines
drivers/nvme/host/core.c +7
-0+ goto out; + +spin_lock_irqsave (&ctrl->lock, flags); +for (i =0 ; i <le16_to_cpu (log->ne); i++) { + entry = &log->entries[i]; +if (entry->ccrs ==0 )/* skip in progress entries */ +continue ;What does ccrs stand for?
Quoted 6 lines
Cross-Controller Reset Status
0x00-> In Progress0x01-> Success0x02-> Failed0x03-0xff-> Reserved
Let's add it as a proper enumeration please.
Re: [RFC PATCH 09/14] nvme: Implement cross-controller reset completion
Quoted 19 lines · Sagi Grimberg
Quoted 16 lines · Mohamed Khalfella
Quoted 15 lines · Sagi Grimberg
Quoted 4 lines
drivers/nvme/host/core.c +4
-0+ log =kmalloc (sizeof (*log),GFP_KERNEL ); +if (!log) +return ; +drivers/nvme/host/core.c static void nvme_get_fw_slot_info(struct nvme_ctrl *ctrl) · line 4886
Quoted 10 lines
drivers/nvme/host/core.c +10
-0+ ret =nvme_get_log (ctrl,0 ,NVME_LOG_CCR ,0x01 , +0x00 , log,sizeof (*log),0 ); +if (ret) + goto out; + +spin_lock_irqsave (&ctrl->lock, flags); +for (i =0 ; i <le16_to_cpu (log->ne); i++) { + entry = &log->entries[i]; +if (entry->ccrs ==0 )/* skip in progress entries */ +continue ;What does ccrs stand for?
Quoted 9 lines
Quoted 6 lines
Cross-Controller Reset Status
0x00-> In Progress0x01-> Success0x02-> Failed0x03-0xff-> ReservedLet's add it as a proper enumeration please.
Done
RFC PATCH 10/14 nvme-tcp: Use CCR to recover controller that hits an error
In reply to Mohamed KhalfellaAn alive nvme controller that hits an error now will move to RECOVERING state instead of RESETTING state. In RECOVERING state ctrl->err_work will attempt to use cross-controller recovery to terminate inflight IOs on the controller. If CCR succeeds, then switch to RESETTING state and continue error recovery as usuall by tearing down controller and attempt reconnecting to target. If CCR fails, then the behavior of recovery depends on whether CQT is supported or not. If CQT is supported, switch to time-based recovery by holding inflight IOs until it is safe for them to be retried. If CQT is not supported proceed to retry requests immediately, as the code currently does.
To support implementing time-based recovery turn ctrl->err_work into delayed work. Update nvme_tcp_timeout() to not complete inflight IOs while controller in RECOVERING state.
--- drivers/nvme/host/tcp.c | 52 +++++++++++++++++++++++++++++++++++------ 1 file changed, 45 insertions(+), 7 deletions(-)
drivers/nvme/host/tcp.c +45 -7
@@ -193,7 +193,7 @@ struct nvme_tcp_ctrl {struct sockaddr_storage src_addr;struct nvme_ctrl ctrl; -struct work_struct err_work; +struct delayed_work err_work;struct delayed_work connect_work;struct nvme_tcp_request async_req;u32 io_queues[HCTX_MAX_TYPES ];
@@ -611,11 +611,12 @@ static void nvme_tcp_init_recv_ctx(struct nvme_tcp_queue *queue)static void nvme_tcp_error_recovery (struct nvme_ctrl *ctrl) { -if (!nvme_change_ctrl_state (ctrl,NVME_CTRL_RESETTING )) +if (!nvme_change_ctrl_state (ctrl,NVME_CTRL_RECOVERING ) && + !nvme_change_ctrl_state (ctrl,NVME_CTRL_RESETTING ))return ;dev_warn (ctrl->device,"starting error recovery\n" ); -queue_work (nvme_reset_wq, &to_tcp_ctrl (ctrl)->err_work); +queue_delayed_work (nvme_reset_wq, &to_tcp_ctrl (ctrl)->err_work,0 ); } static int nvme_tcp_process_nvme_cqe(struct nvme_tcp_queue *queue,
@@ -2470,12 +2471,48 @@ static void nvme_tcp_reconnect_ctrl_work(struct work_struct *work)nvme_tcp_reconnect_or_remove (ctrl, ret); } +static int nvme_tcp_recover_ctrl (struct nvme_ctrl *ctrl) +{ +unsigned long rem; + +if (test_and_clear_bit (NVME_CTRL_RECOVERED , &ctrl->flags)) { +dev_info (ctrl->device,"completed time-based recovery\n" ); + goto done; + } + + rem =nvme_recover_ctrl (ctrl); +if (!rem) + goto done; + +if (!ctrl->cqt) { +dev_info (ctrl->device, +"CCR failed, CQT not supported, skip time-based recovery\n" ); + goto done; + } + +dev_info (ctrl->device, +"CCR failed, switch to time-based recovery, timeout = %ums\n" , +jiffies_to_msecs (rem)); +set_bit (NVME_CTRL_RECOVERED , &ctrl->flags); +queue_delayed_work (nvme_reset_wq, &to_tcp_ctrl (ctrl)->err_work, rem); +return -EAGAIN ; + +done: +nvme_end_ctrl_recovery (ctrl); +return 0 ; +} +static void nvme_tcp_error_recovery_work (struct work_struct *work) { -struct nvme_tcp_ctrl *tcp_ctrl =container_of (work, +struct nvme_tcp_ctrl *tcp_ctrl =container_of (to_delayed_work (work), struct nvme_tcp_ctrl, err_work);struct nvme_ctrl *ctrl = &tcp_ctrl->ctrl; +if (nvme_ctrl_state (ctrl) ==NVME_CTRL_RECOVERING ) { +if (nvme_tcp_recover_ctrl (ctrl)) +return ; + } +if (nvme_tcp_key_revoke_needed (ctrl))nvme_auth_revoke_tls_key (ctrl);nvme_stop_keep_alive (ctrl);
@@ -2545,7 +2582,7 @@ static void nvme_reset_ctrl_work(struct work_struct *work)static void nvme_tcp_stop_ctrl (struct nvme_ctrl *ctrl) { -flush_work (&to_tcp_ctrl (ctrl)->err_work); +flush_delayed_work (&to_tcp_ctrl (ctrl)->err_work);cancel_delayed_work_sync (&to_tcp_ctrl (ctrl)->connect_work); }
@@ -2640,6 +2677,7 @@ static enum blk_eh_timer_return nvme_tcp_timeout(struct request *rq) {struct nvme_tcp_request *req =blk_mq_rq_to_pdu (rq);struct nvme_ctrl *ctrl = &req->queue->ctrl->ctrl; +enum nvme_ctrl_state state =nvme_ctrl_state (ctrl);struct nvme_tcp_cmd_pdu *pdu =nvme_tcp_req_cmd_pdu (req);struct nvme_command *cmd = &pdu->cmd;int qid =nvme_tcp_queue_id (req->queue);
@@ -2649,7 +2687,7 @@ static enum blk_eh_timer_return nvme_tcp_timeout(struct request *rq) rq->tag,nvme_cid (rq), pdu->hdr.type, cmd->common.opcode, nvme_fabrics_opcode_str(qid, cmd), qid); -if (nvme_ctrl_state (ctrl) !=NVME_CTRL_LIVE ) { +if (state !=NVME_CTRL_LIVE && state !=NVME_CTRL_RECOVERING ) {/* * If we are resetting, connecting or deleting we should * complete immediately because we may block controller
@@ -2903,7 +2941,7 @@ static struct nvme_tcp_ctrl *nvme_tcp_alloc_ctrl(struct device *dev,INIT_DELAYED_WORK (&ctrl->connect_work, nvme_tcp_reconnect_ctrl_work); -INIT_WORK (&ctrl->err_work, nvme_tcp_error_recovery_work); +INIT_DELAYED_WORK (&ctrl->err_work, nvme_tcp_error_recovery_work);INIT_WORK (&ctrl->ctrl.reset_work, nvme_reset_ctrl_work);if (!(opts->mask &NVMF_OPT_TRSVCID )) {
Signature · 1 line
2.51.2
Re: [RFC PATCH 10/14] nvme-tcp: Use CCR to recover controller that hits an error
Quoted 7 lines · Mohamed Khalfella
An alive nvme controller that hits an error now will move to RECOVERING state instead of RESETTING state. In RECOVERING state
ctrl->err_workwill attempt to use cross-controller recovery to terminate inflight IOs on the controller. If CCR succeeds, then switch to RESETTING state and continue error recovery as usuall by tearing down controller and attempt reconnecting to target. If CCR fails, then the behavior of recovery
"usuall" -> "usual"
"attempt reconnecting" -> "attempting to reconnect"
it would read better with "the" added:
"tearing down the controller"
"reconnect to the target"
Quoted 6 lines
depends on whether CQT is supported or not. If CQT is supported, switch to time-based recovery by holding inflight IOs until it is safe for them to be retried. If CQT is not supported proceed to retry requests immediately, as the code currently does.
drivers/nvme/host/tcp.c static void nvme_tcp_reconnect_ctrl_work(struct work_struct *work) · line 2498
Quoted 9 lines
drivers/nvme/host/tcp.c +7
-0+static int nvme_tcp_recover_ctrl(struct nvme_ctrl *ctrl) + dev_info(ctrl->device, +"CCR failed, switch to time-based recovery, timeout = %ums\n" , + jiffies_to_msecs(rem)); +set_bit (NVME_CTRL_RECOVERED , &ctrl->flags); +queue_delayed_work (nvme_reset_wq, &to_tcp_ctrl (ctrl)->err_work, rem); +return -EAGAIN ;
I see how setting this bit before the delayed work executes works to complete recovery, but it is kindof weird that the bit is called RECOVERED. I do not have a better name. TIME_BASED_RECOVERY? RECOVERY_WAIT?
Quoted 5 lines
static void
nvme_tcp_error_recovery_work(struct work_struct *work){drivers/nvme/host/tcp.c +1
-1-struct nvme_tcp_ctrl *tcp_ctrl =container_of (work, +struct nvme_tcp_ctrl *tcp_ctrl =container_of (to_delayed_work (work), struct nvme_tcp_ctrl, err_work);
drivers/nvme/host/tcp.c static void nvme_tcp_reconnect_ctrl_work(struct work_struct *work) · line 2518
Quoted 10 lines
drivers/nvme/host/tcp.c +5
-0struct nvme_ctrl *ctrl = &tcp_ctrl->ctrl; +if (nvme_ctrl_state (ctrl) ==NVME_CTRL_RECOVERING ) { +if (nvme_tcp_recover_ctrl (ctrl)) +return ; + } +if (nvme_tcp_key_revoke_needed (ctrl))nvme_auth_revoke_tls_key (ctrl);nvme_stop_keep_alive (ctrl);
The state of the controller should not be LIVE while waiting for recovery, so I do not think we will succeed in sending keep alives, but I think this should move to before (or inside of) nvme_tcp_recover_ctrl().
Sincerely,
Randy Jennings
Re: [RFC PATCH 10/14] nvme-tcp: Use CCR to recover controller that hits an error
Quoted 9 lines · Randy Jennings
Quoted 7 lines · Mohamed Khalfella
An alive nvme controller that hits an error now will move to RECOVERING state instead of RESETTING state. In RECOVERING state
ctrl->err_workwill attempt to use cross-controller recovery to terminate inflight IOs on the controller. If CCR succeeds, then switch to RESETTING state and continue error recovery as usuall by tearing down controller and attempt reconnecting to target. If CCR fails, then the behavior of recovery
Quoted 6 lines
"usuall" -> "usual"
"attempt reconnecting" -> "attempting to reconnect"it would read better with "the" added:
"tearing down the controller"
"reconnect to the target"
Updated as suggested.
Quoted 16 lines
Quoted 7 lines
depends on whether CQT is supported or not. If CQT is supported, switch to time-based recovery by holding inflight IOs until it is safe for them to be retried. If CQT is not supported proceed to retry requests immediately, as the code currently does.
drivers/nvme/host/tcp.c static void nvme_tcp_reconnect_ctrl_work(struct work_struct *work) · line 2498
Quoted 9 lines
drivers/nvme/host/tcp.c +7
-0+static int nvme_tcp_recover_ctrl(struct nvme_ctrl *ctrl) + dev_info(ctrl->device, +"CCR failed, switch to time-based recovery, timeout = %ums\n" , + jiffies_to_msecs(rem)); +set_bit (NVME_CTRL_RECOVERED , &ctrl->flags); +queue_delayed_work (nvme_reset_wq, &to_tcp_ctrl (ctrl)->err_work, rem); +return -EAGAIN ;
Quoted 4 lines
I see how setting this bit before the delayed work executes works to complete recovery, but it is kindof weird that the bit is called RECOVERED. I do not have a better name.
TIME_BASED_RECOVERY?RECOVERY_WAIT?
Agree. It does look weird. If we agree to add two states FENCING and FENCED then the flag might not be needed.
Quoted 16 lines
Quoted 6 lines
static void
nvme_tcp_error_recovery_work(struct work_struct *work){drivers/nvme/host/tcp.c +1
-1-struct nvme_tcp_ctrl *tcp_ctrl =container_of (work, +struct nvme_tcp_ctrl *tcp_ctrl =container_of (to_delayed_work (work), struct nvme_tcp_ctrl, err_work);drivers/nvme/host/tcp.c static void nvme_tcp_reconnect_ctrl_work(struct work_struct *work) · line 2518
Quoted 10 lines
drivers/nvme/host/tcp.c +5
-0struct nvme_ctrl *ctrl = &tcp_ctrl->ctrl; +if (nvme_ctrl_state (ctrl) ==NVME_CTRL_RECOVERING ) { +if (nvme_tcp_recover_ctrl (ctrl)) +return ; + } +if (nvme_tcp_key_revoke_needed (ctrl))nvme_auth_revoke_tls_key (ctrl);nvme_stop_keep_alive (ctrl);
Quoted 4 lines
The state of the controller should not be LIVE while waiting for recovery, so I do not think we will succeed in sending keep alives, but I think this should move to before (or inside of)
nvme_tcp_recover_ctrl().
This is correct, no keepalive traffic will be sent in RECOVERING state. If we split fencing work from existing error recovery work then this should removed. I think we are going in that direction.
Quoted 3 lines
Sincerely,
Randy Jennings
Sagi Grimberg
maintainerRe: [RFC PATCH 10/14] nvme-tcp: Use CCR to recover controller that hits an error
In reply to Mohamed KhalfellaQuoted 34 lines · Mohamed Khalfella
An alive nvme controller that hits an error now will move to RECOVERING state instead of RESETTING state. In RECOVERING state
ctrl->err_workwill attempt to use cross-controller recovery to terminate inflight IOs on the controller. If CCR succeeds, then switch to RESETTING state and continue error recovery as usuall by tearing down controller and attempt reconnecting to target. If CCR fails, then the behavior of recovery depends on whether CQT is supported or not. If CQT is supported, switch to time-based recovery by holding inflight IOs until it is safe for them to be retried. If CQT is not supported proceed to retry requests immediately, as the code currently does.To support implementing time-based recovery turn
ctrl->err_workinto delayed work. Updatenvme_tcp_timeout()to not complete inflight IOs while controller in RECOVERING state.Signed-off-by Mohamed Khalfella <[email protected]>--- drivers/nvme/host/tcp.c | 52 +++++++++++++++++++++++++++++++++++------ 1 file changed, 45 insertions(+), 7 deletions(-)drivers/nvme/host/tcp.c +1
-1@@ -193,7 +193,7 @@ struct nvme_tcp_ctrl {struct sockaddr_storage src_addr;struct nvme_ctrl ctrl; -struct work_struct err_work; +struct delayed_work err_work;struct delayed_work connect_work;struct nvme_tcp_request async_req;u32 io_queues[HCTX_MAX_TYPES ];
drivers/nvme/host/tcp.c static void nvme_tcp_init_recv_ctx(struct nvme_tcp_queue *queue) · line 615
Quoted 6 lines
drivers/nvme/host/tcp.c +2
-1@@ -611,11 +611,12 @@ static void nvme_tcp_init_recv_ctx(struct nvme_tcp_queue *queue)static void nvme_tcp_error_recovery (struct nvme_ctrl *ctrl) { -if (!nvme_change_ctrl_state (ctrl,NVME_CTRL_RESETTING )) +if (!nvme_change_ctrl_state (ctrl,NVME_CTRL_RECOVERING ) && + !nvme_change_ctrl_state (ctrl,NVME_CTRL_RESETTING ))
This warrants an explanation. It is not clear at all why we should allow two different transitions to allow error recovery to start...
Quoted 10 lines
drivers/nvme/host/tcp.c +1
-1return ;dev_warn (ctrl->device,"starting error recovery\n" ); -queue_work (nvme_reset_wq, &to_tcp_ctrl (ctrl)->err_work); +queue_delayed_work (nvme_reset_wq, &to_tcp_ctrl (ctrl)->err_work,0 ); } static int nvme_tcp_process_nvme_cqe(struct nvme_tcp_queue *queue,@@ -2470,12 +2471,48 @@ static void nvme_tcp_reconnect_ctrl_work(struct work_struct *work)nvme_tcp_reconnect_or_remove (ctrl, ret);
drivers/nvme/host/tcp.c static void nvme_tcp_reconnect_ctrl_work(struct work_struct *work) · line 2481
Quoted 10 lines
drivers/nvme/host/tcp.c +8
-0} +static int nvme_tcp_recover_ctrl (struct nvme_ctrl *ctrl) +{ +unsigned long rem; + +if (test_and_clear_bit (NVME_CTRL_RECOVERED , &ctrl->flags)) { +dev_info (ctrl->device,"completed time-based recovery\n" ); + goto done; + }
This is also not clear, why should we get here when NVME_CTRL_RECOVERED is set?
Quoted 7 lines
drivers/nvme/host/tcp.c +7
-0+ + rem =nvme_recover_ctrl (ctrl); +if (!rem) + goto done; + +if (!ctrl->cqt) { + dev_info(ctrl->device,
drivers/nvme/host/tcp.c static void nvme_tcp_reconnect_ctrl_work(struct work_struct *work) · line 2498
Quoted 10 lines
drivers/nvme/host/tcp.c +10
-0+ "CCR failed, CQT not supported, skip time-based recovery\n"); + goto done; + } + +dev_info (ctrl->device, +"CCR failed, switch to time-based recovery, timeout = %ums\n" , +jiffies_to_msecs (rem)); +set_bit (NVME_CTRL_RECOVERED , &ctrl->flags); +queue_delayed_work (nvme_reset_wq, &to_tcp_ctrl (ctrl)->err_work, rem); +return -EAGAIN ;
I don't think that reusing the same work to handle two completely different things is the right approach here.
How about splitting to fence_work and err_work? That should eliminate some of the ctrl state inspections and simplify error recovery.
Quoted 8 lines
drivers/nvme/host/tcp.c +6
-0+ +done: +nvme_end_ctrl_recovery (ctrl); +return 0 ; +} +static void nvme_tcp_error_recovery_work (struct work_struct *work) {
drivers/nvme/host/tcp.c static void nvme_tcp_reconnect_ctrl_work(struct work_struct *work) · line 2515
Quoted 10 lines
drivers/nvme/host/tcp.c +6
-1-struct nvme_tcp_ctrl *tcp_ctrl =container_of (work, +struct nvme_tcp_ctrl *tcp_ctrl =container_of (to_delayed_work (work), struct nvme_tcp_ctrl, err_work);struct nvme_ctrl *ctrl = &tcp_ctrl->ctrl; +if (nvme_ctrl_state (ctrl) ==NVME_CTRL_RECOVERING ) { +if (nvme_tcp_recover_ctrl (ctrl)) +return ; + } +
Yea, I think we want to rework the current design.
Re: [RFC PATCH 10/14] nvme-tcp: Use CCR to recover controller that hits an error
Quoted 1 line · Sagi Grimberg
On 26/11/2025 4:11, Mohamed Khalfella wrote:
...
Quoted 7 lines
drivers/nvme/host/tcp.c static void nvme_tcp_reconnect_ctrl_work(struct work_struct *work) · line 2498
Quoted 6 lines
drivers/nvme/host/tcp.c +6
-0+dev_info (ctrl->device, +"CCR failed, switch to time-based recovery, timeout = %ums\n" , +jiffies_to_msecs (rem)); +set_bit (NVME_CTRL_RECOVERED , &ctrl->flags); +queue_delayed_work (nvme_reset_wq, &to_tcp_ctrl (ctrl)->err_work, rem); +return -EAGAIN ;
Quoted 7 lines
I don't think that reusing the same work to handle two completely different things
is the right approach here.How about splitting to
fence_workanderr_work? That should eliminate some of the
ctrl state inspections and simplify error recovery.
If the work was independent and could happen separately (probably in parallel), I could understand having separate work structures. But they are not independent, and they have a definite relationship. Like Mohamed, I thought of them as different stages of the same work. Having an extra work item takes up more space (I would be concerned about scalability to thousands or 10s of thousands of associations and then go one order of magnitude higher for margin), and it also causes a connection object (referenced during IO) to take up more cache lines. Is it worth taking up that space, when the separate work items would be different, dependent stages in the same process?
Sincerely,
Randy Jennings
Sagi Grimberg
maintainerRe: [RFC PATCH 10/14] nvme-tcp: Use CCR to recover controller that hits an error
Quoted 13 lines · Randy Jennings
Quoted 1 line · Sagi Grimberg
On 26/11/2025 4:11, Mohamed Khalfella wrote:
...
Quoted 10 lines
drivers/nvme/host/tcp.c static void nvme_tcp_reconnect_ctrl_work(struct work_struct *work) · line 2498
Quoted 6 lines
drivers/nvme/host/tcp.c +6
-0+dev_info (ctrl->device, +"CCR failed, switch to time-based recovery, timeout = %ums\n" , +jiffies_to_msecs (rem)); +set_bit (NVME_CTRL_RECOVERED , &ctrl->flags); +queue_delayed_work (nvme_reset_wq, &to_tcp_ctrl (ctrl)->err_work, rem); +return -EAGAIN ;I don't think that reusing the same work to handle two completely different things
is the right approach here.
Quoted 6 lines
Quoted 3 lines
How about splitting to
fence_workanderr_work? That should eliminate some of the
ctrl state inspections and simplify error recovery.If the work was independent and could happen separately (probably in parallel), I could understand having separate work structures. But they are not independent, and they have a definite relationship.
The relationship that is defined here is that error recovery does not start before fencing is completed.
Quoted 8 lines
Like Mohamed,
I thought of them as different stages of the same work. Having an extra work item takes up more space (I would be concerned about scalability to thousands or 10s of thousands of associations and then go one order of magnitude higher for margin), and it also causes a connection object (referenced during IO) to take up more cache lines. Is it worth taking up that space, when the separate work items would be different, dependent stages in the same process?
Yes, IMO the added space of an additional work_struct is much better than adding more state around a single work handler that is queued up multiple times doing effectively different things.
Re: [RFC PATCH 10/14] nvme-tcp: Use CCR to recover controller that hits an error
In reply to Sagi GrimbergQuoted 38 lines · Sagi Grimberg
Quoted 35 lines · Mohamed Khalfella
An alive nvme controller that hits an error now will move to RECOVERING state instead of RESETTING state. In RECOVERING state
ctrl->err_workwill attempt to use cross-controller recovery to terminate inflight IOs on the controller. If CCR succeeds, then switch to RESETTING state and continue error recovery as usuall by tearing down controller and attempt reconnecting to target. If CCR fails, then the behavior of recovery depends on whether CQT is supported or not. If CQT is supported, switch to time-based recovery by holding inflight IOs until it is safe for them to be retried. If CQT is not supported proceed to retry requests immediately, as the code currently does.To support implementing time-based recovery turn
ctrl->err_workinto delayed work. Updatenvme_tcp_timeout()to not complete inflight IOs while controller in RECOVERING state.Signed-off-by Mohamed Khalfella <[email protected]>--- drivers/nvme/host/tcp.c | 52 +++++++++++++++++++++++++++++++++++------ 1 file changed, 45 insertions(+), 7 deletions(-)drivers/nvme/host/tcp.c +1
-1@@ -193,7 +193,7 @@ struct nvme_tcp_ctrl {struct sockaddr_storage src_addr;struct nvme_ctrl ctrl; -struct work_struct err_work; +struct delayed_work err_work;struct delayed_work connect_work;struct nvme_tcp_request async_req;u32 io_queues[HCTX_MAX_TYPES ];@@ -611,11 +611,12 @@ static void nvme_tcp_init_recv_ctx(struct nvme_tcp_queue *queue)
Quoted 9 lines
drivers/nvme/host/tcp.c static void nvme_tcp_init_recv_ctx(struct nvme_tcp_queue *queue) · line 615
Quoted 5 lines
drivers/nvme/host/tcp.c +2
-1static void nvme_tcp_error_recovery (struct nvme_ctrl *ctrl) { -if (!nvme_change_ctrl_state (ctrl,NVME_CTRL_RESETTING )) +if (!nvme_change_ctrl_state (ctrl,NVME_CTRL_RECOVERING ) && + !nvme_change_ctrl_state (ctrl,NVME_CTRL_RESETTING ))This warrants an explanation. It is not clear at all why we should allow two different
transitions to allow error recovery to start...
The behavior of the ctrl->err_work depends on the controller state. We go to RECOVERING only if the controller is LIVE. Otherwise, we attempt to got to RESETTING.
Quoted 13 lines
Quoted 13 lines
drivers/nvme/host/tcp.c +1
-1return ;dev_warn (ctrl->device,"starting error recovery\n" ); -queue_work (nvme_reset_wq, &to_tcp_ctrl (ctrl)->err_work); +queue_delayed_work (nvme_reset_wq, &to_tcp_ctrl (ctrl)->err_work,0 ); } static int nvme_tcp_process_nvme_cqe(struct nvme_tcp_queue *queue,@@ -2470,12 +2471,48 @@ static void nvme_tcp_reconnect_ctrl_work(struct work_struct *work)nvme_tcp_reconnect_or_remove (ctrl, ret); }
Quoted 11 lines
drivers/nvme/host/tcp.c static void nvme_tcp_reconnect_ctrl_work(struct work_struct *work) · line 2481
Quoted 8 lines
drivers/nvme/host/tcp.c +8
-0+static int nvme_tcp_recover_ctrl (struct nvme_ctrl *ctrl) +{ +unsigned long rem; + +if (test_and_clear_bit (NVME_CTRL_RECOVERED , &ctrl->flags)) { +dev_info (ctrl->device,"completed time-based recovery\n" ); + goto done; + }This is also not clear, why should we get here when
NVME_CTRL_RECOVEREDis set?
NVME_CTRL_RECOVERED flag is set before scheduling ctrl->err_work as delayed work. This is how how time-based recovery is implemented. We get here when ctrl->err_work runs for the second time, and at this point we know that it is safe to just reset the controller and cancel inflight requests.
Quoted 39 lines
Quoted 7 lines
drivers/nvme/host/tcp.c +7
-0+ + rem =nvme_recover_ctrl (ctrl); +if (!rem) + goto done; + +if (!ctrl->cqt) { + dev_info(ctrl->device,drivers/nvme/host/tcp.c static void nvme_tcp_reconnect_ctrl_work(struct work_struct *work) · line 2498
Quoted 10 lines
drivers/nvme/host/tcp.c +10
-0+ "CCR failed, CQT not supported, skip time-based recovery\n"); + goto done; + } + +dev_info (ctrl->device, +"CCR failed, switch to time-based recovery, timeout = %ums\n" , +jiffies_to_msecs (rem)); +set_bit (NVME_CTRL_RECOVERED , &ctrl->flags); +queue_delayed_work (nvme_reset_wq, &to_tcp_ctrl (ctrl)->err_work, rem); +return -EAGAIN ;I don't think that reusing the same work to handle two completely different things
is the right approach here.How about splitting to
fence_workanderr_work? That should eliminate some of the
ctrl state inspections and simplify error recovery.Quoted 13 lines
drivers/nvme/host/tcp.c +7
-1+ +done: +nvme_end_ctrl_recovery (ctrl); +return 0 ; +} +static void nvme_tcp_error_recovery_work (struct work_struct *work) { -struct nvme_tcp_ctrl *tcp_ctrl =container_of (work, +struct nvme_tcp_ctrl *tcp_ctrl =container_of (to_delayed_work (work), struct nvme_tcp_ctrl, err_work);struct nvme_ctrl *ctrl = &tcp_ctrl->ctrl;
Quoted 7 lines
drivers/nvme/host/tcp.c static void nvme_tcp_reconnect_ctrl_work(struct work_struct *work) · line 2515
Quoted 5 lines
drivers/nvme/host/tcp.c +5
-0+if (nvme_ctrl_state (ctrl) ==NVME_CTRL_RECOVERING ) { +if (nvme_tcp_recover_ctrl (ctrl)) +return ; + } +Yea, I think we want to rework the current design.
Good point. Splitting ctrl->fence_work simplifies things. The if condition above will be moved to fence_work. However, we will still need to reschedule ctrl->fence_work from within its self to implement time-based recovery. Is this good option?
If not, and we prefer to drop NVME_CTRL_RECOVERED flag above and not reschedule ctrl->fence_work from within its self, then we can add another ctr->fenced_work. How about that?
RFC PATCH 11/14 nvme-rdma: Use CCR to recover controller that hits an error
In reply to Mohamed KhalfellaAn alive nvme controller that hits an error will now move to RECOVERING state instead of RESETTING state. In RECOVERING state, ctrl->err_work will attempt to use cross-controller recovery to terminate inflight IOs on the controller. If CCR succeeds, then switch to RESETTING state and continue error recovery as usuall by tearing down the controller, and attempting reconnect to target. If CCR fails, the behavior of recovery depends on whether CQT is supported or not. If CQT is supported, switch to time-based recovery by holding inflight IOs until it is safe for them to be retried. If CQT is not supported proceed to retry requests immediately, as the code currently does.
To support implementing time-based recovery turn ctrl->err_work into delayed work. Update nvme_rdma_timeout() to not complete inflight IOs while controller in RECOVERING state.
--- drivers/nvme/host/rdma.c | 51 ++++++++++++++++++++++++++++++++++------ 1 file changed, 44 insertions(+), 7 deletions(-)
drivers/nvme/host/rdma.c +44 -7
@@ -106,7 +106,7 @@ struct nvme_rdma_ctrl {/* other member variables */ struct blk_mq_tag_set tag_set; -struct work_struct err_work; +struct delayed_work err_work;struct nvme_rdma_qe async_event_sqe;
@@ -961,7 +961,7 @@ static void nvme_rdma_stop_ctrl(struct nvme_ctrl *nctrl) {struct nvme_rdma_ctrl *ctrl =to_rdma_ctrl (nctrl); -flush_work (&ctrl->err_work); +flush_delayed_work (&ctrl->err_work);cancel_delayed_work_sync (&ctrl->reconnect_work); }
@@ -1120,11 +1120,46 @@ static void nvme_rdma_reconnect_ctrl_work(struct work_struct *work)nvme_rdma_reconnect_or_remove (ctrl, ret); } +static int nvme_rdma_recover_ctrl (struct nvme_ctrl *ctrl) +{ +unsigned long rem; + +if (test_and_clear_bit (NVME_CTRL_RECOVERED , &ctrl->flags)) { +dev_info (ctrl->device,"completed time-based recovery\n" ); + goto done; + } + + rem =nvme_recover_ctrl (ctrl); +if (!rem) + goto done; + +if (!ctrl->cqt) { +dev_info (ctrl->device, +"CCR failed, CQT not supported, skip time-based recovery\n" ); + goto done; + } + +dev_info (ctrl->device, +"CCR failed, switch to time-based recovery, timeout = %ums\n" , +jiffies_to_msecs (rem)); +set_bit (NVME_CTRL_RECOVERED , &ctrl->flags); +queue_delayed_work (nvme_reset_wq, &to_rdma_ctrl (ctrl)->err_work, rem); +return -EAGAIN ; + +done: +nvme_end_ctrl_recovery (ctrl); +return 0 ; +}static void nvme_rdma_error_recovery_work (struct work_struct *work) { -struct nvme_rdma_ctrl *ctrl =container_of (work, +struct nvme_rdma_ctrl *ctrl =container_of (to_delayed_work (work), struct nvme_rdma_ctrl, err_work); +if (nvme_ctrl_state (&ctrl->ctrl) ==NVME_CTRL_RECOVERING ) { +if (nvme_rdma_recover_ctrl (&ctrl->ctrl)) +return ; + } +nvme_stop_keep_alive (&ctrl->ctrl);flush_work (&ctrl->ctrl.async_event_work);nvme_rdma_teardown_io_queues (ctrl, false);
@@ -1147,11 +1182,12 @@ static void nvme_rdma_error_recovery_work(struct work_struct *work)static void nvme_rdma_error_recovery (struct nvme_rdma_ctrl *ctrl) { -if (!nvme_change_ctrl_state (&ctrl->ctrl,NVME_CTRL_RESETTING )) +if (!nvme_change_ctrl_state (&ctrl->ctrl,NVME_CTRL_RECOVERING ) && + !nvme_change_ctrl_state (&ctrl->ctrl,NVME_CTRL_RESETTING ))return ;dev_warn (ctrl->ctrl.device,"starting error recovery\n" ); -queue_work (nvme_reset_wq, &ctrl->err_work); +queue_delayed_work (nvme_reset_wq, &ctrl->err_work,0 ); }static void nvme_rdma_end_request (struct nvme_rdma_request *req)
@@ -1955,6 +1991,7 @@ static enum blk_eh_timer_return nvme_rdma_timeout(struct request *rq)struct nvme_rdma_request *req =blk_mq_rq_to_pdu (rq);struct nvme_rdma_queue *queue = req->queue;struct nvme_rdma_ctrl *ctrl = queue->ctrl; +enum nvme_ctrl_state state =nvme_ctrl_state (&ctrl->ctrl);struct nvme_command *cmd = req->req.cmd;int qid =nvme_rdma_queue_idx (queue);
@@ -1963,7 +2000,7 @@ static enum blk_eh_timer_return nvme_rdma_timeout(struct request *rq) rq->tag,nvme_cid (rq), cmd->common.opcode, nvme_fabrics_opcode_str(qid, cmd), qid); -if (nvme_ctrl_state (&ctrl->ctrl) !=NVME_CTRL_LIVE ) { +if (state !=NVME_CTRL_LIVE && state !=NVME_CTRL_RECOVERING ) {/* * If we are resetting, connecting or deleting we should * complete immediately because we may block controller
@@ -2280,7 +2317,7 @@ static struct nvme_rdma_ctrl *nvme_rdma_alloc_ctrl(struct device *dev,INIT_DELAYED_WORK (&ctrl->reconnect_work, nvme_rdma_reconnect_ctrl_work); -INIT_WORK (&ctrl->err_work, nvme_rdma_error_recovery_work); +INIT_DELAYED_WORK (&ctrl->err_work, nvme_rdma_error_recovery_work);INIT_WORK (&ctrl->ctrl.reset_work, nvme_rdma_reset_ctrl_work); ctrl->ctrl.queue_count = opts->nr_io_queues + opts->nr_write_queues +
Signature · 1 line
2.51.2
Re: [RFC PATCH 11/14] nvme-rdma: Use CCR to recover controller that hits an error
Quoted 7 lines · Mohamed Khalfella
An alive nvme controller that hits an error will now move to RECOVERING state instead of RESETTING state. In RECOVERING state,
ctrl->err_workwill attempt to use cross-controller recovery to terminate inflight IOs on the controller. If CCR succeeds, then switch to RESETTING state and continue error recovery as usuall by tearing down the controller, and attempting reconnect to target. If CCR fails, the behavior of recovery
"usuall" -> "usual"
"attempt reconnecting" -> "attempting to reconnect"
it would read better with "the" added:
"reconnect to the target"
Quoted 10 lines
depends on whether CQT is supported or not. If CQT is supported, switch to time-based recovery by holding inflight IOs until it is safe for them to be retried. If CQT is not supported proceed to retry requests immediately, as the code currently does.
drivers/nvme/host/rdma.c +0
-0
drivers/nvme/host/rdma.c static void nvme_rdma_reconnect_ctrl_work(struct work_struct *work)
Quoted 4 lines
drivers/nvme/host/rdma.c +2
-0+static int nvme_rdma_recover_ctrl(struct nvme_ctrl *ctrl) + queue_delayed_work(nvme_reset_wq, &to_rdma_ctrl(ctrl)->err_work, rem);
nvme_rdma_recover_ctrl is exactly the same as nvme_tcp_recover_ctrl. Seems like a core.c function nvme_recover_ctrl could take a delayed work queue, unifying the code.
drivers/nvme/host/rdma.c static void nvme_rdma_reconnect_ctrl_work(struct work_struct *work) · line 1163
Quoted 12 lines
static void
nvme_rdma_error_recovery_work(struct work_struct *work){drivers/nvme/host/rdma.c +6
-1-struct nvme_rdma_ctrl *ctrl =container_of (work, +struct nvme_rdma_ctrl *ctrl =container_of (to_delayed_work (work), struct nvme_rdma_ctrl, err_work); +if (nvme_ctrl_state (&ctrl->ctrl) ==NVME_CTRL_RECOVERING ) { +if (nvme_rdma_recover_ctrl (&ctrl->ctrl)) +return ; + } +nvme_stop_keep_alive (&ctrl->ctrl);
The state of the controller should not be LIVE while waiting for recovery, so I do not think we will succeed in sending keep alives, but I think this should move to before (or inside of) nvme_tcp_recover_ctrl().
Sincerely,
Randy Jennings
Sagi Grimberg
maintainerRe: [RFC PATCH 11/14] nvme-rdma: Use CCR to recover controller that hits an error
In reply to Mohamed KhalfellaSame comments from nvme-tcp...
RFC PATCH 12/14 nvme-fc: Decouple error recovery from controller reset
In reply to Mohamed Khalfellanvme_fc_error_recovery() called from nvme_fc_timeout() while controller in CONNECTING state results in deadlock reported in link below. Update nvme_fc_timeout() to schedule error recovery to avoid the deadlock.
Previous to this change, if controller was LIVE, error recovery resets the controller. This did not match nvme-tcp and nvme-rdma. Decouple error recovery from controller reset to match other fabric transports.
--- drivers/nvme/host/fc.c | 94 ++++++++++++++++++------------------------ 1 file changed, 41 insertions(+), 53 deletions(-)
drivers/nvme/host/fc.c +41 -53
@@ -227,6 +227,8 @@ static DEFINE_IDA(nvme_fc_ctrl_cnt);static struct device *fc_udev_device;static void nvme_fc_complete_rq (struct request *rq); +static void nvme_fc_start_ioerr_recovery (struct nvme_fc_ctrl *ctrl, +char *errmsg);/* *********************** FC-NVME Port Management ************************ */
@@ -786,7 +788,7 @@ nvme_fc_ctrl_connectivity_loss(struct nvme_fc_ctrl *ctrl) "Reconnect", ctrl->cnum);set_bit (ASSOC_FAILED , &ctrl->flags); -nvme_reset_ctrl (&ctrl->ctrl); +nvme_fc_start_ioerr_recovery (ctrl,"Connectivity Loss" ); }/**
@@ -983,7 +985,7 @@ fc_dma_unmap_sg(struct device *dev, struct scatterlist *sg, int nents,static void nvme_fc_ctrl_put (struct nvme_fc_ctrl *);static int nvme_fc_ctrl_get (struct nvme_fc_ctrl *); -static void nvme_fc_error_recovery (struct nvme_fc_ctrl *ctrl,char *errmsg); +static void nvme_fc_error_recovery (struct nvme_fc_ctrl *ctrl);static void __nvme_fc_finish_ls_req (struct nvmefc_ls_req_op *lsop)
@@ -1563,9 +1565,8 @@ nvme_fc_ls_disconnect_assoc(struct nvmefc_ls_rcv_op *lsop)* for the association have been ABTS'd by * nvme_fc_delete_association(). */ - -/* fail the association */ -nvme_fc_error_recovery (ctrl,"Disconnect Association LS received" ); +nvme_fc_start_ioerr_recovery (ctrl, +"Disconnect Association LS received" );/* release the reference taken by nvme_fc_match_disconn_ls() */ nvme_fc_ctrl_put (ctrl);
@@ -1867,7 +1868,7 @@ nvme_fc_ctrl_ioerr_work(struct work_struct *work)struct nvme_fc_ctrl *ctrl = container_of(work, struct nvme_fc_ctrl, ioerr_work); -nvme_fc_error_recovery (ctrl,"transport detected io error" ); +nvme_fc_error_recovery (ctrl); }/*
@@ -1888,6 +1889,17 @@ char *nvme_fc_io_getuuid(struct nvmefc_fcp_req *req) }EXPORT_SYMBOL_GPL (nvme_fc_io_getuuid); +static void nvme_fc_start_ioerr_recovery (struct nvme_fc_ctrl *ctrl, +char *errmsg) +{ +if (!nvme_change_ctrl_state (&ctrl->ctrl,NVME_CTRL_RESETTING )) +return ; + +dev_warn (ctrl->ctrl.device,"NVME-FC{%d}: starting error recovery %s\n" , + ctrl->cnum, errmsg); +queue_delayed_work (nvme_reset_wq, &ctrl->ioerr_work,0 ); +} +static void nvme_fc_fcpio_done (struct nvmefc_fcp_req *req) {
@@ -2045,9 +2057,8 @@ nvme_fc_fcpio_done(struct nvmefc_fcp_req *req)nvme_fc_complete_rq (rq); check_error: -if (terminate_assoc && -nvme_ctrl_state (&ctrl->ctrl) !=NVME_CTRL_RESETTING ) -queue_work (nvme_reset_wq, &ctrl->ioerr_work); +if (terminate_assoc) +nvme_fc_start_ioerr_recovery (ctrl,"io error" ); } static int
@@ -2497,39 +2508,6 @@ __nvme_fc_abort_outstanding_ios(struct nvme_fc_ctrl *ctrl, bool start_queues)nvme_unquiesce_admin_queue (&ctrl->ctrl); } -static void -nvme_fc_error_recovery (struct nvme_fc_ctrl *ctrl,char *errmsg) -{ -enum nvme_ctrl_state state =nvme_ctrl_state (&ctrl->ctrl); - -/* -* if an error (io timeout, etc) while (re)connecting, the remote -* port requested terminating of the association (disconnect_ls) -* or an error (timeout or abort) occurred on an io while creating -* the controller. Abort any ios on the association and let the -* create_association error path resolve things. -*/ -if (state ==NVME_CTRL_CONNECTING ) { -__nvme_fc_abort_outstanding_ios (ctrl, true); -dev_warn (ctrl->ctrl.device, -"NVME-FC{%d}: transport error during (re)connect\n" , - ctrl->cnum); -return ; - } - -/* Otherwise, only proceed if in LIVE state - e.g. on first error */ -if (state !=NVME_CTRL_LIVE ) -return ; - -dev_warn (ctrl->ctrl.device, -"NVME-FC{%d}: transport association event: %s\n" , - ctrl->cnum, errmsg); -dev_warn (ctrl->ctrl.device, -"NVME-FC{%d}: resetting controller\n" , ctrl->cnum); - -nvme_reset_ctrl (&ctrl->ctrl); -} -static enum blk_eh_timer_return nvme_fc_timeout (struct request *rq) {struct nvme_fc_fcp_op *op =blk_mq_rq_to_pdu (rq);
@@ -2538,24 +2516,14 @@ static enum blk_eh_timer_return nvme_fc_timeout(struct request *rq)struct nvme_fc_cmd_iu *cmdiu = &op->cmd_iu;struct nvme_command *sqe = &cmdiu->sqe; -/* -* Attempt to abort the offending command. Command completion -* will detect the aborted io and will fail the connection. -*/ dev_info (ctrl->ctrl.device,"NVME-FC{%d.%d}: io timeout: opcode %d fctype %d (%s) w10/11: " "x%08x/x%08x\n" , ctrl->cnum, qnum, sqe->common.opcode, sqe->fabrics.fctype,nvme_fabrics_opcode_str (qnum, sqe), sqe->common.cdw10, sqe->common.cdw11); -if (__nvme_fc_abort_op (ctrl, op)) -nvme_fc_error_recovery (ctrl,"io timeout abort failed" ); -/* -* the io abort has been initiated. Have the reset timer -* restarted and the abort completion will complete the io -* shortly. Avoids a synchronous wait while the abort finishes. -*/ +nvme_fc_start_ioerr_recovery (ctrl,"io timeout" );return BLK_EH_RESET_TIMER ; }
@@ -3347,6 +3315,26 @@ nvme_fc_reset_ctrl_work(struct work_struct *work) } } +static void +nvme_fc_error_recovery (struct nvme_fc_ctrl *ctrl) +{ +nvme_stop_keep_alive (&ctrl->ctrl); +nvme_stop_ctrl (&ctrl->ctrl); + +/* will block while waiting for io to terminate */ +nvme_fc_delete_association (ctrl); + +/* Do not reconnect if controller is being deleted */ +if (!nvme_change_ctrl_state (&ctrl->ctrl,NVME_CTRL_CONNECTING )) +return ; + +if (ctrl->rport->remoteport.port_state ==FC_OBJSTATE_ONLINE ) { +queue_delayed_work (nvme_wq, &ctrl->connect_work,0 ); +return ; + } + +nvme_fc_reconnect_or_delete (ctrl, -ENOTCONN ); +}static const struct nvme_ctrl_ops nvme_fc_ctrl_ops = { .name ="fc" ,
Signature · 1 line
2.51.2
Re: [RFC PATCH 12/14] nvme-fc: Decouple error recovery from controller reset
Quoted 9 lines · Mohamed Khalfella
drivers/nvme/host/fc.c +2
-0+static void nvme_fc_start_ioerr_recovery (struct nvme_fc_ctrl *ctrl, +char *errmsg);/* *********************** FC-NVME Port Management ************************ */
drivers/nvme/host/fc.c fc_dma_unmap_sg(struct device *dev, struct scatterlist *sg, int nents, · line 988
Quoted 5 lines
drivers/nvme/host/fc.c +1
-1@@ -983,7 +985,7 @@ fc_dma_unmap_sg(struct device *dev, struct scatterlist *sg, int nents,static void nvme_fc_ctrl_put (struct nvme_fc_ctrl *);static int nvme_fc_ctrl_get (struct nvme_fc_ctrl *); -static void nvme_fc_error_recovery (struct nvme_fc_ctrl *ctrl,char *errmsg); +static void nvme_fc_error_recovery (struct nvme_fc_ctrl *ctrl);
does it make sense for nvme_fc_error_recovery() to move to be with nvme_fc_start_ioerr_recovery()?
drivers/nvme/host/fc.c nvme_fc_reset_ctrl_work(struct work_struct *work) · line 3319
Quoted 13 lines
drivers/nvme/host/fc.c +2
-2@@ -2497,39 +2508,6 @@ __nvme_fc_abort_outstanding_ios(struct nvme_fc_ctrl *ctrl, bool start_queues)nvme_unquiesce_admin_queue (&ctrl->ctrl); } -static void -nvme_fc_error_recovery (struct nvme_fc_ctrl *ctrl,char *errmsg)@@ -3347,6 +3315,26 @@ nvme_fc_reset_ctrl_work(struct work_struct *work) } } +static void +nvme_fc_error_recovery (struct nvme_fc_ctrl *ctrl)
I'm curious about the motivation for moving the function.
drivers/nvme/host/fc.c nvme_fc_reset_ctrl_work(struct work_struct *work) · line 3321
Quoted 2 lines
drivers/nvme/host/fc.c +2
-0+{ +nvme_stop_keep_alive (&ctrl->ctrl);
This is a new addition. I think it is a good one; FC can use keep alives, and nvme_stop_keep_alive() is called in one other place, but I do not think the Linux nvme-fc driver uses keep alives.
RFC PATCH 13/14 nvme-fc: Use CCR to recover controller that hits an error
In reply to Mohamed KhalfellaAn alive nvme controller that hits an error will now move to RECOVERING state instead of RESETTING state. In RECOVERING state, ctrl->err_work will attempt to use cross-controller recovery to terminate inflight IOs on the controller. If CCR succeeds, then switch to RESETTING state and continue error recovery as usuall by tearing down the controller, and attempting reconnect to target. If CCR fails, the behavior of recovery depends on whether CQT is supported or not. If CQT is supported, switch to time-based recovery by holding inflight IOs until it is safe for them to be retried. If CQT is not supported proceed to retry requests immediately, as the code currently does.
Currently, inflight IOs can get completed during time-based recovery. This will be addressed in the next patch.
--- drivers/nvme/host/fc.c | 52 ++++++++++++++++++++++++++++++++++++------ 1 file changed, 45 insertions(+), 7 deletions(-)
drivers/nvme/host/fc.c +45 -7
@@ -166,7 +166,7 @@ struct nvme_fc_ctrl {struct blk_mq_tag_set admin_tag_set;struct blk_mq_tag_set tag_set; -struct work_struct ioerr_work; +struct delayed_work ioerr_work;struct delayed_work connect_work;struct kref ref;
@@ -1862,11 +1862,48 @@ __nvme_fc_fcpop_chk_teardowns(struct nvme_fc_ctrl *ctrl, } } +static int nvme_fc_recover_ctrl (struct nvme_ctrl *ctrl) +{ +unsigned long rem; + +if (test_and_clear_bit (NVME_CTRL_RECOVERED , &ctrl->flags)) { +dev_info (ctrl->device,"completed time-based recovery\n" ); + goto done; + } + + rem =nvme_recover_ctrl (ctrl); +if (!rem) + goto done; + +if (!ctrl->cqt) { +dev_info (ctrl->device, +"CCR failed, CQT not supported, skip time-based recovery\n" ); + goto done; + } + +dev_info (ctrl->device, +"CCR failed, switch to time-based recovery, timeout = %ums\n" , +jiffies_to_msecs (rem)); + +set_bit (NVME_CTRL_RECOVERED , &ctrl->flags); +queue_delayed_work (nvme_reset_wq, &to_fc_ctrl (ctrl)->ioerr_work, rem); +return -EAGAIN ; + +done: +nvme_end_ctrl_recovery (ctrl); +return 0 ; +} +static void nvme_fc_ctrl_ioerr_work (struct work_struct *work) { -struct nvme_fc_ctrl *ctrl = - container_of(work, struct nvme_fc_ctrl, ioerr_work); +struct nvme_fc_ctrl *ctrl =container_of (to_delayed_work (work), + struct nvme_fc_ctrl, ioerr_work); + +if (nvme_ctrl_state (&ctrl->ctrl) ==NVME_CTRL_RECOVERING ) { +if (nvme_fc_recover_ctrl (&ctrl->ctrl)) +return ; + }nvme_fc_error_recovery (ctrl); }
@@ -1892,7 +1929,8 @@ EXPORT_SYMBOL_GPL(nvme_fc_io_getuuid);static void nvme_fc_start_ioerr_recovery (struct nvme_fc_ctrl *ctrl,char *errmsg) { -if (!nvme_change_ctrl_state (&ctrl->ctrl,NVME_CTRL_RESETTING )) +if (!nvme_change_ctrl_state (&ctrl->ctrl,NVME_CTRL_RECOVERING ) && + !nvme_change_ctrl_state (&ctrl->ctrl,NVME_CTRL_RESETTING ))return ; dev_warn(ctrl->ctrl.device, "NVME-FC{%d}: starting error recovery %s\n",
@@ -3227,7 +3265,7 @@ nvme_fc_delete_ctrl(struct nvme_ctrl *nctrl) {struct nvme_fc_ctrl *ctrl =to_fc_ctrl (nctrl); -cancel_work_sync (&ctrl->ioerr_work); +cancel_delayed_work_sync (&ctrl->ioerr_work);cancel_delayed_work_sync (&ctrl->connect_work);/* * kill the association on the link side. this will block
@@ -3465,7 +3503,7 @@ nvme_fc_alloc_ctrl(struct device *dev, struct nvmf_ctrl_options *opts,INIT_WORK (&ctrl->ctrl.reset_work, nvme_fc_reset_ctrl_work);INIT_DELAYED_WORK (&ctrl->connect_work, nvme_fc_connect_ctrl_work); -INIT_WORK (&ctrl->ioerr_work, nvme_fc_ctrl_ioerr_work); +INIT_DELAYED_WORK (&ctrl->ioerr_work, nvme_fc_ctrl_ioerr_work);spin_lock_init (&ctrl->lock);/* io queue count */
@@ -3563,7 +3601,7 @@ nvme_fc_init_ctrl(struct device *dev, struct nvmf_ctrl_options *opts, fail_ctrl:nvme_change_ctrl_state (&ctrl->ctrl,NVME_CTRL_DELETING ); -cancel_work_sync (&ctrl->ioerr_work); +cancel_delayed_work_sync (&ctrl->ioerr_work);cancel_work_sync (&ctrl->ctrl.reset_work);cancel_delayed_work_sync (&ctrl->connect_work);
Signature · 1 line
2.51.2
Re: [RFC PATCH 13/14] nvme-fc: Use CCR to recover controller that hits an error
Quoted 7 lines · Mohamed Khalfella
An alive nvme controller that hits an error will now move to RECOVERING state instead of RESETTING state. In RECOVERING state,
ctrl->err_workwill attempt to use cross-controller recovery to terminate inflight IOs on the controller. If CCR succeeds, then switch to RESETTING state and continue error recovery as usuall by tearing down the controller, and attempting reconnect to target. If CCR fails, the behavior of recovery
"usuall" -> "usual"
"attempt reconnecting" -> "attempting to reconnect"
it would read better with "the" added:
"reconnect to the target"
drivers/nvme/host/fc.c __nvme_fc_fcpop_chk_teardowns(struct nvme_fc_ctrl *ctrl, · line 1865
Quoted 12 lines
depends on whether CQT is supported or not. If CQT is supported, switch to time-based recovery by holding inflight IOs until it is safe for them to be retried. If CQT is not supported proceed to retry requests immediately, as the code currently does.
drivers/nvme/host/fc.c +2
-0@@ -1862,11 +1862,48 @@ __nvme_fc_fcpop_chk_teardowns(struct nvme_fc_ctrl *ctrl, +static int nvme_fc_recover_ctrl(struct nvme_ctrl *ctrl) + queue_delayed_work(nvme_reset_wq, &to_fc_ctrl(ctrl)->ioerr_work, rem);
Just like nvme_rdma_recover_ctrl, nvme_fc_recover_ctrl is exactly the same as nvme_tcp_recover_ctrl. Seems like a core.c function nvme_recover_ctrl could take a delayed work queue, unifying the code.
drivers/nvme/host/fc.c __nvme_fc_fcpop_chk_teardowns(struct nvme_fc_ctrl *ctrl, · line 1908
Quoted 9 lines
drivers/nvme/host/fc.c +4
-0nvme_fc_ctrl_ioerr_work(struct work_struct *work) { +if (nvme_ctrl_state (&ctrl->ctrl) ==NVME_CTRL_RECOVERING ) { +if (nvme_fc_recover_ctrl (&ctrl->ctrl)) +return ; + }nvme_fc_error_recovery (ctrl);
Inside of nvme_fc_error_recovery(), we call nvme_stop_keep_alive(). The state of the controller should not be LIVE while waiting for recovery, so I do not think we will succeed in sending keep alives, but I think this should move to before (or inside of) nvme_fc_recover_ctrl(). You have replaced all the calls to nvme_fc_error_recovery() with nvme_fc_start_ioerr_recovery(), so that might be okay.
Sincerely,
Randy Jennings
RFC PATCH 14/14 nvme-fc: Hold inflight requests while in RECOVERING state
In reply to Mohamed Khalfellanvme_fc_delete_association() called from error recovery codepath waits for all requests to be completed. In RECOVERING state inflight IOs should be held until it is safe to for them to be retried. Update nvme_fc_fcpio_done() to not complete requests while in RECOVERING state. Update recovery codepath to cancel inflight requests similar to what nvme-tcp and nvme-rdma do today.
--- drivers/nvme/host/fc.c | 50 +++++++++++++++++++++++++++++++++--------- 1 file changed, 40 insertions(+), 10 deletions(-)
drivers/nvme/host/fc.c +40 -10
@@ -171,7 +171,7 @@ struct nvme_fc_ctrl {struct kref ref;unsigned long flags; -u32 iocnt; +atomic_t iocnt;wait_queue_head_t ioabort_wait;struct nvme_fc_fcp_op aen_ops[NVME_NR_AEN_COMMANDS ];
@@ -1816,7 +1816,7 @@ __nvme_fc_abort_op(struct nvme_fc_ctrl *ctrl, struct nvme_fc_fcp_op *op)atomic_set (&op->state, opstate); else if (test_bit(FCCTRL_TERMIO, &ctrl->flags)) { op->flags |=FCOP_FLAGS_TERMIO ; - ctrl->iocnt++; +atomic_inc (&ctrl->iocnt); }spin_unlock_irqrestore (&ctrl->lock, flags);
@@ -1846,20 +1846,29 @@ nvme_fc_abort_aen_ops(struct nvme_fc_ctrl *ctrl) }static inline void +__nvme_fc_fcpop_count_one_down (struct nvme_fc_ctrl *ctrl) +{ +if (atomic_dec_return (&ctrl->iocnt) ==0 ) +wake_up (&ctrl->ioabort_wait); +} + +static inline bool __nvme_fc_fcpop_chk_teardowns (struct nvme_fc_ctrl *ctrl,struct nvme_fc_fcp_op *op,int opstate) {unsigned long flags; +bool ret = false;if (opstate ==FCPOP_STATE_ABORTED ) {spin_lock_irqsave (&ctrl->lock, flags);if (test_bit (FCCTRL_TERMIO , &ctrl->flags) && op->flags &FCOP_FLAGS_TERMIO ) { -if (!--ctrl->iocnt) -wake_up (&ctrl->ioabort_wait); + ret = true; }spin_unlock_irqrestore (&ctrl->lock, flags); } + +return ret; }static int nvme_fc_recover_ctrl (struct nvme_ctrl *ctrl)
@@ -1950,7 +1959,7 @@ nvme_fc_fcpio_done(struct nvmefc_fcp_req *req)struct nvme_command *sqe = &op->cmd_iu.sqe;__le16 status =cpu_to_le16 (NVME_SC_SUCCESS <<1 );union nvme_result result; -bool terminate_assoc = true; +bool op_term, terminate_assoc = true;int opstate;/*
@@ -2083,17 +2092,34 @@ nvme_fc_fcpio_done(struct nvmefc_fcp_req *req) done:if (op->flags &FCOP_FLAGS_AEN ) {nvme_complete_async_event (&queue->ctrl->ctrl, status, &result); -__nvme_fc_fcpop_chk_teardowns (ctrl, op, opstate); +if (__nvme_fc_fcpop_chk_teardowns (ctrl, op, opstate)) +__nvme_fc_fcpop_count_one_down (ctrl);atomic_set (&op->state,FCPOP_STATE_IDLE ); op->flags =FCOP_FLAGS_AEN ;/* clear other flags */ nvme_fc_ctrl_put (ctrl); goto check_error; } -__nvme_fc_fcpop_chk_teardowns (ctrl, op, opstate); +/* +* We can not access op after the request is completed because it can +* be reused immediately. At the same time we want to wakeup the thread +* waiting for ongoing IOs _after_ requests are completed. This is +* necessary because that thread will start canceling inflight IOs +* and we want to avoid request completion racing with cancellation. +*/ + op_term =__nvme_fc_fcpop_chk_teardowns (ctrl, op, opstate); + +/* Error recovery completes inflight reqeusts when it is safe */ +if (nvme_ctrl_state (&ctrl->ctrl) ==NVME_CTRL_RECOVERING ) + goto check_op_term; +if (!nvme_try_complete_req (rq, status, result))nvme_fc_complete_rq (rq); +check_op_term: +if (op_term) +__nvme_fc_fcpop_count_one_down (ctrl); + check_error:if (terminate_assoc)nvme_fc_start_ioerr_recovery (ctrl,"io error" );
@@ -2737,7 +2763,8 @@ nvme_fc_start_fcp_op(struct nvme_fc_ctrl *ctrl, struct nvme_fc_queue *queue,* cmd with the csn was supposed to arrive. */ opstate =atomic_xchg (&op->state,FCPOP_STATE_COMPLETE ); -__nvme_fc_fcpop_chk_teardowns (ctrl, op, opstate); +if (__nvme_fc_fcpop_chk_teardowns (ctrl, op, opstate)) +__nvme_fc_fcpop_count_one_down (ctrl);if (!(op->flags &FCOP_FLAGS_AEN )) {nvme_fc_unmap_data (ctrl, op->rq, op);
@@ -3206,7 +3233,7 @@ nvme_fc_delete_association(struct nvme_fc_ctrl *ctrl)spin_lock_irqsave (&ctrl->lock, flags);set_bit (FCCTRL_TERMIO , &ctrl->flags); - ctrl->iocnt =0 ; +atomic_set (&ctrl->iocnt,0 );spin_unlock_irqrestore (&ctrl->lock, flags);__nvme_fc_abort_outstanding_ios (ctrl, false);
@@ -3215,8 +3242,8 @@ nvme_fc_delete_association(struct nvme_fc_ctrl *ctrl)nvme_fc_abort_aen_ops (ctrl);/* wait for all io that had to be aborted */ +wait_event (ctrl->ioabort_wait,atomic_read (&ctrl->iocnt) ==0 );spin_lock_irq (&ctrl->lock); -wait_event_lock_irq (ctrl->ioabort_wait, ctrl->iocnt ==0 , ctrl->lock);clear_bit (FCCTRL_TERMIO , &ctrl->flags);spin_unlock_irq (&ctrl->lock);
@@ -3362,6 +3389,9 @@ nvme_fc_error_recovery(struct nvme_fc_ctrl *ctrl)/* will block while waiting for io to terminate */ nvme_fc_delete_association (ctrl); +nvme_cancel_tagset (&ctrl->ctrl); +nvme_cancel_admin_tagset (&ctrl->ctrl); +/* Do not reconnect if controller is being deleted */ if (!nvme_change_ctrl_state (&ctrl->ctrl,NVME_CTRL_CONNECTING ))return ;
Signature · 1 line
2.51.2
Re: [RFC PATCH 14/14] nvme-fc: Hold inflight requests while in RECOVERING state
Quoted 6 lines · Mohamed Khalfella
Signed-off-by Mohamed Khalfella <[email protected]>drivers/nvme/host/fc.c +0
-0@@ -2083,17 +2092,34 @@ nvme_fc_fcpio_done(struct nvmefc_fcp_req *req)
drivers/nvme/host/fc.c nvme_fc_fcpio_done(struct nvmefc_fcp_req *req) · line 2102
Quoted 10 lines
drivers/nvme/host/fc.c +10
-0+/* +* We can not access op after the request is completed because it can +* be reused immediately. At the same time we want to wakeup the thread +* waiting for ongoing IOs _after_ requests are completed. This is +* necessary because that thread will start canceling inflight IOs +* and we want to avoid request completion racing with cancellation. +*/ + op_term =__nvme_fc_fcpop_chk_teardowns (ctrl, op, opstate); + +/* Error recovery completes inflight reqeusts when it is safe */
"reqeusts" -> "requests"
drivers/nvme/host/fc.c nvme_fc_fcpio_done(struct nvmefc_fcp_req *req) · line 2121
Quoted 9 lines
drivers/nvme/host/fc.c +6
-0+if (nvme_ctrl_state (&ctrl->ctrl) ==NVME_CTRL_RECOVERING ) + goto check_op_term; +if (!nvme_try_complete_req (rq, status, result))nvme_fc_complete_rq (rq); +check_op_term: +if (op_term) +__nvme_fc_fcpop_count_one_down (ctrl);
I think it is easier to grok:
+ op_term = __nvme_fc_fcpop_chk_teardowns(ctrl, op, opstate);
+
+ /* Error recovery completes inflight reqeusts when it is safe */
+ if (nvme_ctrl_state(&ctrl->ctrl) != NVME_CTRL_RECOVERING &&
+ !nvme_try_complete_req(rq, status, result))
nvme_fc_complete_rq(rq);
+
+ if (op_term)
+ __nvme_fc_fcpop_count_one_down(ctrl);