NVMe PCI endpoint target fixes
Subsystems (2)
cover NVMe PCI endpoint target fixes
Two bug fix patch and one cleanup patch in this series.
The first patch fixes an issue with completion queue entries initialization in the case of failed commands.
The second patch fixes
the initialization of the CC register to ensure that we can always detect that the host is attempting to enable the controller (with CC.EN).
The last patch cleans up/simplifies the management of the PCI link state.
Damien Le Moal (3):
nvmet: pci-epf: Always fully initialize completion entries
nvmet: pci-epf: Clear CC and CSTS when disabling the controller
nvmet: pci-epf: Cleanup link state management
drivers/nvme/target/pci-epf.c | 65 ++++++++++++++++++++++------------- 1 file changed, 42 insertions(+), 23 deletions(-)
Signature · 1 line
2.49.0
PATCH 3/3 nvmet: pci-epf: Cleanup link state management
Since the link_up boolean field of struct nvmet_pci_epf_ctrl is always set to true when nvmet_pci_epf_start_ctrl() is called, assign true to this field in nvmet_pci_epf_start_ctrl(). Conversely, since this field is set to false when nvmet_pci_epf_stop_ctrl() is called, set this field to false directly inside that function.
While at it, also add information messages to notify the user of the PCI link state changes to help troubleshoot any link stability issues without needing to enable debug messages.
--- drivers/nvme/target/pci-epf.c | 14 ++++++++------ 1 file changed, 8 insertions(+), 6 deletions(-)
drivers/nvme/target/pci-epf.c +8 -6
@@ -2086,11 +2086,18 @@ static int nvmet_pci_epf_create_ctrl(struct nvmet_pci_epf *nvme_epf,static void nvmet_pci_epf_start_ctrl (struct nvmet_pci_epf_ctrl *ctrl) { + +dev_info (ctrl->dev,"PCI link up\n" ); + ctrl->link_up = true; +schedule_delayed_work (&ctrl->poll_cc,NVMET_PCI_EPF_CC_POLL_INTERVAL ); }static void nvmet_pci_epf_stop_ctrl (struct nvmet_pci_epf_ctrl *ctrl) { +dev_info (ctrl->dev,"PCI link down\n" ); + ctrl->link_up = false; +cancel_delayed_work_sync (&ctrl->poll_cc);nvmet_pci_epf_disable_ctrl (ctrl, false);
@@ -2317,10 +2324,8 @@ static int nvmet_pci_epf_epc_init(struct pci_epf *epf)if (ret) goto out_clear_bar; -if (!epc_features->linkup_notifier) { - ctrl->link_up = true; +if (!epc_features->linkup_notifier)nvmet_pci_epf_start_ctrl (&nvme_epf->ctrl); - }return 0 ;
@@ -2336,7 +2341,6 @@ static void nvmet_pci_epf_epc_deinit(struct pci_epf *epf)struct nvmet_pci_epf *nvme_epf =epf_get_drvdata (epf);struct nvmet_pci_epf_ctrl *ctrl = &nvme_epf->ctrl; - ctrl->link_up = false;nvmet_pci_epf_destroy_ctrl (ctrl);nvmet_pci_epf_deinit_dma (nvme_epf);
@@ -2348,7 +2352,6 @@ static int nvmet_pci_epf_link_up(struct pci_epf *epf)struct nvmet_pci_epf *nvme_epf =epf_get_drvdata (epf);struct nvmet_pci_epf_ctrl *ctrl = &nvme_epf->ctrl; - ctrl->link_up = true;nvmet_pci_epf_start_ctrl (ctrl);return 0 ;
@@ -2359,7 +2362,6 @@ static int nvmet_pci_epf_link_down(struct pci_epf *epf)struct nvmet_pci_epf *nvme_epf =epf_get_drvdata (epf);struct nvmet_pci_epf_ctrl *ctrl = &nvme_epf->ctrl; - ctrl->link_up = false;nvmet_pci_epf_stop_ctrl (ctrl);return 0 ;
Signature · 1 line
2.49.0
PATCH 1/3 nvmet: pci-epf: Always fully initialize completion entries
In reply to Damien Le MoalFor a command that is normally processed through the command request execute() function, the completion entry for the command is initialized by __nvmet_req_complete() and nvmet_pci_epf_cq_work() only needs to set the status field and the phase of the completion entry before posting the entry to the completion queue.
However, for commands that are failed due to an internal error (e.g. the command data buffer allocation fails), the command request execute() function is not called and __nvmet_req_complete() is never executed for the command, leaving the command completion entry uninitialized. For such command failed before calling req->execute(), the host ends up seeing completion entries with an invalid submission queue ID and command ID.
Avoid such issue by always fully initilizing a command completion entry in nvmet_pci_epf_cq_work(), setting the entry submission queue ID and command ID.
--- drivers/nvme/target/pci-epf.c | 2 ++ 1 file changed, 2 insertions(+)
drivers/nvme/target/pci-epf.c +2 -0
@@ -1763,6 +1763,8 @@ static void nvmet_pci_epf_cq_work(struct work_struct *work)/* Post the IOD completion entry. */ cqe = &iod->cqe; + cqe->sq_id =cpu_to_le16 (iod->sq->qid); + cqe->command_id = iod->cmd.common.command_id; cqe->status =cpu_to_le16 ((iod->status <<1 ) | cq->phase); dev_dbg(ctrl->dev,
Signature · 1 line
2.49.0
PATCH 2/3 nvmet: pci-epf: Clear CC and CSTS when disabling the controller
In reply to Damien Le MoalWhen a host shuts down the controller when shutting down but does so without first disabling the controller, the enable bit remains set in the controller configuration register. When the host restarts and attempts to enable the controller again, the nvmet_pci_epf_poll_cc_work() function is unable to detect the change from 0 to 1 of the enable bit, and thus the controller is not enabled again, which result in a device scan timeout on the host. This problem also occurs if the host shuts down uncleanly or if the PCIe link goes
down: as the CC.EN value is not reset, the controller is not enabled again when the host restarts.
Fix this by introducing the function nvmet_pci_epf_clear_ctrl_config() to clear the CC and CSTS registers of the controller when the PCIe link is lost (nvmet_pci_epf_stop_ctrl() function), and when starting the controller fails (nvmet_pci_epf_enable_ctrl() fails). Also use this function in nvmet_pci_epf_init_bar() to simplify the initialization of the CC and CSTS registers.
Furthermore, modify the function nvmet_pci_epf_disable_ctrl() to clear the CC.EN bit and write this updated value to the BAR register when the controller is shutdown by the host, to ensure that upon restart, we can detect the host setting CC.EN.
--- drivers/nvme/target/pci-epf.c | 49 +++++++++++++++++++++++------------ 1 file changed, 32 insertions(+), 17 deletions(-)
drivers/nvme/target/pci-epf.c +32 -17
@@ -1802,6 +1802,21 @@ static void nvmet_pci_epf_cq_work(struct work_struct *work) NVMET_PCI_EPF_CQ_RETRY_INTERVAL); } +static void nvmet_pci_epf_clear_ctrl_config (struct nvmet_pci_epf_ctrl *ctrl) +{ +struct nvmet_ctrl *tctrl = ctrl->tctrl; + +/* Initialize controller status. */ + tctrl->csts =0 ; + ctrl->csts =0 ; +nvmet_pci_epf_bar_write32 (ctrl,NVME_REG_CSTS , ctrl->csts); + +/* Initialize controller configuration and start polling. */ + tctrl->cc =0 ; + ctrl->cc =0 ; +nvmet_pci_epf_bar_write32 (ctrl,NVME_REG_CC , ctrl->cc); +} +static int nvmet_pci_epf_enable_ctrl (struct nvmet_pci_epf_ctrl *ctrl) {u64 pci_addr, asq, acq;
@@ -1867,18 +1882,20 @@ static int nvmet_pci_epf_enable_ctrl(struct nvmet_pci_epf_ctrl *ctrl)return 0 ; err: - ctrl->csts =0 ; +nvmet_pci_epf_clear_ctrl_config (ctrl);return -EINVAL ; } -static void nvmet_pci_epf_disable_ctrl (struct nvmet_pci_epf_ctrl *ctrl) +static void nvmet_pci_epf_disable_ctrl (struct nvmet_pci_epf_ctrl *ctrl, +bool shutdown) {int qid;if (!ctrl->enabled)return ; -dev_info (ctrl->dev,"Disabling controller\n" ); +dev_info (ctrl->dev,"%s controller\n" , + shutdown ?"Shutting down" :"Disabling" ); ctrl->enabled = false;cancel_delayed_work_sync (&ctrl->poll_sqs);
@@ -1895,6 +1912,11 @@ static void nvmet_pci_epf_disable_ctrl(struct nvmet_pci_epf_ctrl *ctrl)nvmet_pci_epf_delete_cq (ctrl->tctrl,0 ); ctrl->csts &= ~NVME_CSTS_RDY ; +if (shutdown) { + ctrl->csts |=NVME_CSTS_SHST_CMPLT ; + ctrl->cc &= ~NVME_CC_ENABLE ; +nvmet_pci_epf_bar_write32 (ctrl,NVME_REG_CC , ctrl->cc); + } }static void nvmet_pci_epf_poll_cc_work (struct work_struct *work)
@@ -1921,12 +1943,10 @@ static void nvmet_pci_epf_poll_cc_work(struct work_struct *work) }if (!nvmet_cc_en (new_cc) &&nvmet_cc_en (old_cc)) -nvmet_pci_epf_disable_ctrl (ctrl); +nvmet_pci_epf_disable_ctrl (ctrl, false); -if (nvmet_cc_shn (new_cc) && !nvmet_cc_shn (old_cc)) { -nvmet_pci_epf_disable_ctrl (ctrl); - ctrl->csts |=NVME_CSTS_SHST_CMPLT ; - } +if (nvmet_cc_shn (new_cc) && !nvmet_cc_shn (old_cc)) +nvmet_pci_epf_disable_ctrl (ctrl, true);if (!nvmet_cc_shn (new_cc) &&nvmet_cc_shn (old_cc)) ctrl->csts &= ~NVME_CSTS_SHST_CMPLT ;
@@ -1965,16 +1985,10 @@ static void nvmet_pci_epf_init_bar(struct nvmet_pci_epf_ctrl *ctrl)/* Clear Controller Memory Buffer Supported (CMBS). */ ctrl->cap &= ~(0x1ULL <<57 ); -/* Controller configuration. */ - ctrl->cc = tctrl->cc & (~NVME_CC_ENABLE ); - -/* Controller status. */ - ctrl->csts = ctrl->tctrl->csts; -nvmet_pci_epf_bar_write64 (ctrl,NVME_REG_CAP , ctrl->cap);nvmet_pci_epf_bar_write32 (ctrl,NVME_REG_VS , tctrl->subsys->ver); -nvmet_pci_epf_bar_write32 (ctrl,NVME_REG_CSTS , ctrl->csts); -nvmet_pci_epf_bar_write32 (ctrl,NVME_REG_CC , ctrl->cc); + +nvmet_pci_epf_clear_ctrl_config (ctrl); } static int nvmet_pci_epf_create_ctrl(struct nvmet_pci_epf *nvme_epf,
@@ -2079,7 +2093,8 @@ static void nvmet_pci_epf_stop_ctrl(struct nvmet_pci_epf_ctrl *ctrl) {cancel_delayed_work_sync (&ctrl->poll_cc); -nvmet_pci_epf_disable_ctrl (ctrl); +nvmet_pci_epf_disable_ctrl (ctrl, false); +nvmet_pci_epf_clear_ctrl_config (ctrl); }static void nvmet_pci_epf_destroy_ctrl (struct nvmet_pci_epf_ctrl *ctrl)
Signature · 1 line
2.49.0
Quoted 5 lines · Damien Le Moal
Two bug fix patch and one cleanup patch in this series.
The first patch fixes an issue with completion queue entries initialization in the case of failed commands.
Quoted 7 lines
The second patch fixes
the initialization of the CC register to ensure that we can always detect that the host is attempting to enable the controller (with CC.EN).The last patch cleans up/simplifies the management of the PCI link state.
Ping ?
Quoted 9 lines
Damien Le Moal (3):
nvmet: pci-epf: Always fully initialize completion entries
nvmet: pci-epf: Clear CC and CSTS when disabling the controller
nvmet: pci-epf: Cleanup link state managementdrivers/nvme/target/pci-epf.c | 65 ++++++++++++++++++++++------------- 1 file changed, 42 insertions(+), 23 deletions(-)
Signature · 2 lines
Damien Le Moal Western Digital Research
Christoph Hellwig
maintainerQuoted 4 lines
Quoted 2 lines
The last patch cleans up/simplifies the management of the PCI link state.
Ping ?
Can I get someone else to look over them for an extra review? Niklas?
Re: [PATCH 1/3] nvmet: pci-epf: Always fully initialize completion entries
In reply to Damien Le MoalQuoted 14 lines · Damien Le Moal
For a command that is normally processed through the command request
execute()function, the completion entry for the command is initialized by__nvmet_req_complete()andnvmet_pci_epf_cq_work()only needs to set the status field and the phase of the completion entry before posting the entry to the completion queue.However, for commands that are failed due to an internal error (e.g. the command data buffer allocation fails), the command request
execute()function is not called and__nvmet_req_complete()is never executed for the command, leaving the command completion entry uninitialized. For such command failed before callingreq->execute(), the host ends up seeing completion entries with an invalid submission queue ID and command ID.
Quoted 3 lines
Avoid such issue by always fully initilizing a command completion entry in
nvmet_pci_epf_cq_work(), setting the entry submission queue ID and command ID.
Looks good.
Keith Busch
maintainerRe: [PATCH 2/3] nvmet: pci-epf: Clear CC and CSTS when disabling the controller
In reply to Damien Le Moaldrivers/nvme/target/pci-epf.c static void nvmet_pci_epf_disable_ctrl(struct nvmet_pci_epf_ctrl *ctrl) · line 1920
Quoted 10 lines · Damien Le Moal
drivers/nvme/target/pci-epf.c +5
-0@@ -1895,6 +1912,11 @@ static void nvmet_pci_epf_disable_ctrl(struct nvmet_pci_epf_ctrl *ctrl)nvmet_pci_epf_delete_cq (ctrl->tctrl,0 ); ctrl->csts &= ~NVME_CSTS_RDY ; +if (shutdown) { + ctrl->csts |=NVME_CSTS_SHST_CMPLT ; + ctrl->cc &= ~NVME_CC_ENABLE ; +nvmet_pci_epf_bar_write32 (ctrl,NVME_REG_CC , ctrl->cc); + } }
I think this is probably okay, but I don't know if it's necessary to be messing with CC.EN that the host didn't request. The qemu emulated nvme doesn't do this, at least. But it looks like that would all work out in the end anyway, so again, I think it's fine.
Re: [PATCH 3/3] nvmet: pci-epf: Cleanup link state management
In reply to Damien Le MoalQuoted 9 lines · Damien Le Moal
Since the
link_upboolean field of structnvmet_pci_epf_ctrlis always set to true whennvmet_pci_epf_start_ctrl()is called, assign true to this field innvmet_pci_epf_start_ctrl(). Conversely, since this field is set to false whennvmet_pci_epf_stop_ctrl()is called, set this field to false directly inside that function.While at it, also add information messages to notify the user of the PCI link state changes to help troubleshoot any link stability issues without needing to enable debug messages.
Looks good.
Christoph Hellwig
maintainerRe: [PATCH 1/3] nvmet: pci-epf: Always fully initialize completion entries
In reply to Damien Le Moaldrivers/nvme/target/pci-epf.c static void nvmet_pci_epf_cq_work(struct work_struct *work) · line 1767
Quoted 6 lines · Damien Le Moal
drivers/nvme/target/pci-epf.c +2
-0@@ -1763,6 +1763,8 @@ static void nvmet_pci_epf_cq_work(struct work_struct *work)/* Post the IOD completion entry. */ cqe = &iod->cqe; + cqe->sq_id =cpu_to_le16 (iod->sq->qid); + cqe->command_id = iod->cmd.common.command_id;
This could use a comment explaining why we are doing this seemingly duplicate work here.
Otherwise this looks good to me.
Re: [PATCH 1/3] nvmet: pci-epf: Always fully initialize completion entries
In reply to Damien Le MoalQuoted 30 lines · Damien Le Moal
For a command that is normally processed through the command request
execute()function, the completion entry for the command is initialized by__nvmet_req_complete()andnvmet_pci_epf_cq_work()only needs to set the status field and the phase of the completion entry before posting the entry to the completion queue.However, for commands that are failed due to an internal error (e.g. the command data buffer allocation fails), the command request
execute()function is not called and__nvmet_req_complete()is never executed for the command, leaving the command completion entry uninitialized. For such command failed before callingreq->execute(), the host ends up seeing completion entries with an invalid submission queue ID and command ID.Avoid such issue by always fully initilizing a command completion entry in
nvmet_pci_epf_cq_work(), setting the entry submission queue ID and command ID.Fixes 0faa0fe6f90e ("nvmet: New NVMe PCI endpoint function target driver")Signed-off-by Damien Le Moal <[email protected]>--- drivers/nvme/target/pci-epf.c | 2 ++ 1 file changed, 2 insertions(+)drivers/nvme/target/pci-epf.c +0
-0
drivers/nvme/target/pci-epf.c static void nvmet_pci_epf_cq_work(struct work_struct *work) · line 1768
Quoted 6 lines
drivers/nvme/target/pci-epf.c +2
-0@@ -1763,6 +1763,8 @@ static void nvmet_pci_epf_cq_work(struct work_struct *work)/* Post the IOD completion entry. */ cqe = &iod->cqe; + cqe->sq_id =cpu_to_le16 (iod->sq->qid); + cqe->command_id = iod->cmd.common.command_id; cqe->status =cpu_to_le16 ((iod->status <<1 ) | cq->phase);
Looking at e.g. __nvmet_req_complete() and __nvmet_fc_fcp_nvme_cmd_done() in addition to setting sq_id, command_id, and status, they also set:cqe->sq_head.
(__nvmet_req_complete() -> nvmet_update_sq_head() -> req->cqe->sq_head = ...)
Should we perhaps set sq_head too?
Kind regards,
Niklas
Re: [PATCH 2/3] nvmet: pci-epf: Clear CC and CSTS when disabling the controller
In reply to Damien Le MoalQuoted 11 lines · Damien Le Moal
When a host shuts down the controller when shutting down but does so without first disabling the controller, the enable bit remains set in the controller configuration register. When the host restarts and attempts to enable the controller again, the
nvmet_pci_epf_poll_cc_work()function is unable to detect the change from 0 to 1 of the enable bit, and thus the controller is not enabled again, which result in a device scan timeout on the host. This problem also occurs if the host shuts down uncleanly or if the PCIe link goes
down: as the CC.EN value is not reset, the controller is not enabled again when the host restarts.
Quoted 3 lines
Fix this by introducing the function
nvmet_pci_epf_clear_ctrl_config()to clear the CC and CSTS registers of the controller when the PCIe link is lost (nvmet_pci_epf_stop_ctrl()function), and when starting the
s/, and when/, or when/
Quoted 21 lines
controller fails (
nvmet_pci_epf_enable_ctrl()fails). Also use this function innvmet_pci_epf_init_bar()to simplify the initialization of the CC and CSTS registers.Furthermore, modify the function
nvmet_pci_epf_disable_ctrl()to clear the CC.EN bit and write this updated value to the BAR register when the controller is shutdown by the host, to ensure that upon restart, we can detect the host setting CC.EN.Fixes 0faa0fe6f90e ("nvmet: New NVMe PCI endpoint function target driver")Signed-off-by Damien Le Moal <[email protected]>--- drivers/nvme/target/pci-epf.c | 49 +++++++++++++++++++++++------------ 1 file changed, 32 insertions(+), 17 deletions(-)drivers/nvme/target/pci-epf.c +0
-0
drivers/nvme/target/pci-epf.c static void nvmet_pci_epf_cq_work(struct work_struct *work) · line 1811
Quoted 10 lines
drivers/nvme/target/pci-epf.c +7
-0@@ -1802,6 +1802,21 @@ static void nvmet_pci_epf_cq_work(struct work_struct *work) NVMET_PCI_EPF_CQ_RETRY_INTERVAL); } +static void nvmet_pci_epf_clear_ctrl_config (struct nvmet_pci_epf_ctrl *ctrl) +{ +struct nvmet_ctrl *tctrl = ctrl->tctrl; + +/* Initialize controller status. */ + tctrl->csts =0 ; + ctrl->csts =0 ;
This can be written on one line:tctrl->csts = ctrl->csts = 0;
drivers/nvme/target/pci-epf.c static void nvmet_pci_epf_cq_work(struct work_struct *work) · line 1816
Quoted 5 lines
drivers/nvme/target/pci-epf.c +5
-0+nvmet_pci_epf_bar_write32 (ctrl,NVME_REG_CSTS , ctrl->csts); + +/* Initialize controller configuration and start polling. */ + tctrl->cc =0 ; + ctrl->cc =0 ;
Same here.
Otherwise, this looks good to me:
Re: [PATCH 3/3] nvmet: pci-epf: Cleanup link state management
In reply to Damien Le MoalQuoted 6 lines · Damien Le Moal
Since the
link_upboolean field of structnvmet_pci_epf_ctrlis always set to true whennvmet_pci_epf_start_ctrl()is called, assign true to this field innvmet_pci_epf_start_ctrl(). Conversely, since this field is set to false whennvmet_pci_epf_stop_ctrl()is called, set this field to false directly inside that function.
Quoted 6 lines
While at it, also add information messages to notify the user of the PCI link state changes to help troubleshoot any link stability issues without needing to enable debug messages.
Signed-off-by Damien Le Moal <[email protected]>---
Re: [PATCH 2/3] nvmet: pci-epf: Clear CC and CSTS when disabling the controller
In reply to Keith BuschQuoted 4 lines · Keith Busch
Quoted 3 lines · Damien Le Moal
drivers/nvme/target/pci-epf.c +0
-0@@ -1895,6 +1912,11 @@ static void nvmet_pci_epf_disable_ctrl(struct nvmet_pci_epf_ctrl *ctrl)nvmet_pci_epf_delete_cq (ctrl->tctrl,0 );
Quoted 12 lines
drivers/nvme/target/pci-epf.c static void nvmet_pci_epf_disable_ctrl(struct nvmet_pci_epf_ctrl *ctrl) · line 1920
Quoted 7 lines
drivers/nvme/target/pci-epf.c +5
-0ctrl->csts &= ~NVME_CSTS_RDY ; +if (shutdown) { + ctrl->csts |=NVME_CSTS_SHST_CMPLT ; + ctrl->cc &= ~NVME_CC_ENABLE ; +nvmet_pci_epf_bar_write32 (ctrl,NVME_REG_CC , ctrl->cc); + } }I think this is probably okay, but I don't know if it's necessary to be messing with CC.EN that the host didn't request. The qemu emulated nvme doesn't do this, at least. But it looks like that would all work out in the end anyway, so again, I think it's fine.
Maybe qemu clears/resets all the registers of the BAR when it does a shutdown ? I did not check the code. But here, if we do not clear EN, *and* the host does not first disable the controller before trying to re-enable it (like the linux host pci driver does), after the shutdown, we fail to detect that the host set EN again since we can only detect that by polling and comparing to the previous value.
I also thought about using ctrl->enabled for determining what needs to be done, but then again, if we do not clear EN from the register, the next CC poll loop will see EN set and think that it is the host trying to re-enable the controller and we will re-enable right away.
I think that clearing EN is not necessary with qemu because there is no polling, since (I think) that qemu nvme driver will be looking at CC.EN only and only when the guest touches it (as that will trigger a guest exit into the hypervisor). So the qemu nvme driver always sees the newest value of CC, not the old one like we do with the endpoint.
Signature · 2 lines
Damien Le Moal Western Digital Research
Re: [PATCH 2/3] nvmet: pci-epf: Clear CC and CSTS when disabling the controller
In reply to Niklas CasselQuoted 10 lines · Niklas Cassel
drivers/nvme/target/pci-epf.c static void nvmet_pci_epf_cq_work(struct work_struct *work) · line 1811
Quoted 7 lines
drivers/nvme/target/pci-epf.c +7
-0+static void nvmet_pci_epf_clear_ctrl_config (struct nvmet_pci_epf_ctrl *ctrl) +{ +struct nvmet_ctrl *tctrl = ctrl->tctrl; + +/* Initialize controller status. */ + tctrl->csts =0 ; + ctrl->csts =0 ;This can be written on one line:
tctrl->csts=ctrl->csts= 0;
I am aware of this, but really dislike this style as I think it makes the code less readable. So unless the nvme maintainers insist on this change, I prefer leaving this as I wrote it.
Signature · 2 lines
Damien Le Moal Western Digital Research
Keith Busch
maintainerRe: [PATCH 2/3] nvmet: pci-epf: Clear CC and CSTS when disabling the controller
In reply to Damien Le MoalQuoted 18 lines · Damien Le Moal
Quoted 4 lines · Keith Busch
Quoted 3 lines · Damien Le Moal
drivers/nvme/target/pci-epf.c +0
-0@@ -1895,6 +1912,11 @@ static void nvmet_pci_epf_disable_ctrl(struct nvmet_pci_epf_ctrl *ctrl)nvmet_pci_epf_delete_cq (ctrl->tctrl,0 );Quoted 12 lines
drivers/nvme/target/pci-epf.c static void nvmet_pci_epf_disable_ctrl(struct nvmet_pci_epf_ctrl *ctrl) · line 1920
Quoted 7 lines
drivers/nvme/target/pci-epf.c +5
-0ctrl->csts &= ~NVME_CSTS_RDY ; +if (shutdown) { + ctrl->csts |=NVME_CSTS_SHST_CMPLT ; + ctrl->cc &= ~NVME_CC_ENABLE ; +nvmet_pci_epf_bar_write32 (ctrl,NVME_REG_CC , ctrl->cc); + } }I think this is probably okay, but I don't know if it's necessary to be messing with CC.EN that the host didn't request. The qemu emulated nvme doesn't do this, at least. But it looks like that would all work out in the end anyway, so again, I think it's fine.
Quoted 6 lines
Maybe qemu clears/resets all the registers of the BAR when it does a shutdown ? I did not check the code. But here, if we do not clear EN, *and* the host does not first disable the controller before trying to re-enable it (like the linux host pci driver does), after the shutdown, we fail to detect that the host set EN again since we can only detect that by polling and comparing to the previous value.
The spec has some text explaining the host needs to handle a shutdown controller with CC.EN set in section 3.6.1:
To start executing commands on the controller after that controller reports controller shutdown processing complete (...) utilizing the CC.EN bit:
o if the CC.EN bit is set to 1´, then a Controller Level Reset is required to clear the CC.EN bit to 0´ on that controller and the CC.EN bit is subsequently required to be set to `1´ as part of the initialization sequence
So I guess some hosts just aren't handling this? No biggie, linux will react accordingly either way.
I didn't find anything that requires a shutdown controller retain CC.EN, so clearing it on the condition is probably fine. I never really fully understood why a disabled vs shutdown-ready controller needed such distinctions anyway.
Re: [PATCH 1/3] nvmet: pci-epf: Always fully initialize completion entries
In reply to Damien Le MoalRe: [PATCH 3/3] nvmet: pci-epf: Cleanup link state management
In reply to Damien Le MoalRe: [PATCH 2/3] nvmet: pci-epf: Clear CC and CSTS when disabling the controller
In reply to Damien Le Moal