nvme-rdma: Stop queues when starting with error recovery
RFC 1/1 nvme-rdma: Stop queues when starting with error recovery
When we enter error recovery we should stop all queue activities and all armed timers.
For example, we could arming an ANATT timer right before we enter error recovery but do not successfully recover before the timer fires. The timer is supposed only be active when the controller is in LIVE state hence we should call nvme_stop_ctrl when starting with the recover activites.
---
The nvme_stop_ctrl() does cancel pending ANATT timers. But so far I don't got hold of logs when the two controllers get back live. So this might not work as expected.
My question is do we just want to cancel the timer or is nvme_stop_ctrl() the right function here. Obviously, the same problem exists for nvme-tcp.
[ 889.241541] nvme nvme0: creating 4 I/O queues. [ 892.341152] nvme nvme0: mapped 4/0/0 default/read/poll queues. [ 892.350942] nvme nvme0: new ctrl: NQN "XXX", addr 192.20.93.101:4420 [ 892.402493] nvme nvme1: creating 4 I/O queues. [ 895.392810] nvme nvme1: mapped 4/0/0 default/read/poll queues. [ 895.402029] nvme nvme1: new ctrl: NQN "XXX", addr 192.20.93.102:4420 [ 895.471730] nvme nvme2: creating 4 I/O queues. [ 898.509195] nvme nvme2: mapped 4/0/0 default/read/poll queues. [ 898.519015] nvme nvme2: new ctrl: NQN "XXX", addr 192.20.193.101:4420 [ 898.571169] nvme nvme3: creating 4 I/O queues. [ 901.592283] nvme nvme3: mapped 4/0/0 default/read/poll queues. [ 901.601832] nvme nvme3: new ctrl: NQN "XXX", addr 192.20.193.102:4420
[ 983.429977] nvme nvme3: I/O 0 QID 0 timeout [ 983.434472] nvme nvme3: starting error recovery [ 984.549958] nvme nvme0: I/O 0 QID 0 timeout [ 984.554452] nvme nvme0: starting error recovery [ 986.962375] nvme nvme3: failed nvme_keep_alive_end_io error=10 [ 986.986898] nvme nvme3: Reconnecting in 10 seconds...
[ 1226.486740] nvme nvme3: Reconnecting in 10 seconds... [ 1227.749980] nvme nvme0: rdma connection establishment failed (-110) [ 1227.761593] nvme nvme0: Failed reconnect attempt 18 [ 1227.766848] nvme nvme0: Reconnecting in 10 seconds...
[ 1235.685958] nvme nvme0: ANATT timeout, resetting controller. [ 1235.692107] nvme nvme3: ANATT timeout, resetting controller.
drivers/nvme/host/rdma.c | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-)
drivers/nvme/host/rdma.c +1 -2
@@ -1197,8 +1197,7 @@ static void nvme_rdma_error_recovery_work(struct work_struct *work)struct nvme_rdma_ctrl *ctrl =container_of (work, struct nvme_rdma_ctrl, err_work); -nvme_stop_keep_alive (&ctrl->ctrl); -flush_work (&ctrl->ctrl.async_event_work); +nvme_stop_ctrl (&ctrl->ctrl);nvme_rdma_teardown_io_queues (ctrl, false);nvme_start_queues (&ctrl->ctrl);nvme_rdma_teardown_admin_queue (ctrl, false);
Signature · 1 line
2.29.2
Sagi Grimberg
maintainerQuoted 8 lines · Daniel Wagner
When we enter error recovery we should stop all queue activities and all armed timers.
For example, we could arming an ANATT timer right before we enter error recovery but do not successfully recover before the timer fires. The timer is supposed only be active when the controller is in LIVE state hence we should call
nvme_stop_ctrlwhen starting with the recover activites.
This takes me back... IIRC the main reason why the error recovery handler in nvme-rdma (and nvme-tcp) does not start with nvme_stop_ctrl() is because back then, nvme_stop_ctrl() could block until an admin_timeout expired on an inflight admin command (I have a vague memory that if cancelled/flushed the scan_work), which delayed I/O failover, hence we just cherry-picked what needed to be stopped to make sure I/O failover will not be blocked by anything and progress quickly.
If I look at nvme_stop_queue it does not touch the scan_work, but it does flush ana_work which includes I/O, so it can still block on it to fail after admin_timeout expires... also flush failfast_work is taking the namespaces_rwsem (although read access). The fw_act_work is probably meaningless for fabrics...
Maybe we need to have something like nvme_stop_ctrl_fast, that will just stop the non-blocking stuff, so failover can work quickly, and then after we have failover, we can do the full nvme_stop_ctrl() or the reminder of nvme_stop_ctrl_fast...
thoughts?
Sagi Grimberg
maintainerQuoted 20 lines · Sagi Grimberg
Quoted 8 lines · Daniel Wagner
When we enter error recovery we should stop all queue activities and all armed timers.
For example, we could arming an ANATT timer right before we enter error recovery but do not successfully recover before the timer fires. The timer is supposed only be active when the controller is in LIVE state hence we should call
nvme_stop_ctrlwhen starting with the recover activites.This takes me back... IIRC the main reason why the error recovery handler in nvme-rdma (and nvme-tcp) does not start with
nvme_stop_ctrl()is because back then,nvme_stop_ctrl()could block until anadmin_timeoutexpired on an inflight admin command (I have a vague memory that if cancelled/flushed thescan_work), which delayed I/O failover, hence we just cherry-picked what needed to be stopped to make sure I/O failover will not be blocked by anything and progress quickly.
Quoted 1 line
If I look at
nvme_stop_queue
nvme_stop_ctrl obviously...
it does not touch the scan_work, but it
Quoted 11 lines
does flush
ana_workwhich includes I/O, so it can still block on it to fail afteradmin_timeoutexpires... also flushfailfast_workis taking thenamespaces_rwsem(although read access). Thefw_act_workis probably meaningless for fabrics...Maybe we need to have something like
nvme_stop_ctrl_fast, that will just stop the non-blocking stuff, so failover can work quickly, and then after we have failover, we can do the fullnvme_stop_ctrl()or the reminder ofnvme_stop_ctrl_fast...thoughts?
Quoted 23 lines · Sagi Grimberg
Quoted 20 lines · Sagi Grimberg
Quoted 8 lines · Daniel Wagner
When we enter error recovery we should stop all queue activities and all armed timers.
For example, we could arming an ANATT timer right before we enter error recovery but do not successfully recover before the timer fires. The timer is supposed only be active when the controller is in LIVE state hence we should call
nvme_stop_ctrlwhen starting with the recover activites.This takes me back... IIRC the main reason why the error recovery handler in nvme-rdma (and nvme-tcp) does not start with
nvme_stop_ctrl()is because back then,nvme_stop_ctrl()could block until anadmin_timeoutexpired on an inflight admin command (I have a vague memory that if cancelled/flushed thescan_work), which delayed I/O failover, hence we just cherry-picked what needed to be stopped to make sure I/O failover will not be blocked by anything and progress quickly.
Quoted 16 lines
Quoted 1 line
If I look at
nvme_stop_queue
nvme_stop_ctrlobviously...it does not touch the
scan_work, but itQuoted 11 lines
does flush
ana_workwhich includes I/O, so it can still block on it to fail afteradmin_timeoutexpires... also flushfailfast_workis taking thenamespaces_rwsem(although read access). Thefw_act_workis probably meaningless for fabrics...Maybe we need to have something like
nvme_stop_ctrl_fast, that will just stop the non-blocking stuff, so failover can work quickly, and then after we have failover, we can do the fullnvme_stop_ctrl()or the reminder ofnvme_stop_ctrl_fast...thoughts?
I suspected that nvme_stop_ctrl() might have unwanted side effects. Maybe factoring out the common bits from nvme_stop_ctrl() into nvme_stop_ctrl_fast() and call it from here sounds like a reasonable thing to do. The function name could be better, but naming is hard.
Sagi Grimberg
maintainerQuoted 33 lines
Quoted 33 lines
Quoted 18 lines
Quoted 9 lines
When we enter error recovery we should stop all queue activities and all armed timers.
For example, we could arming an ANATT timer right before we enter error recovery but do not successfully recover before the timer fires. The timer is supposed only be active when the controller is in LIVE state hence we should call
nvme_stop_ctrlwhen starting with the recover activites.This takes me back... IIRC the main reason why the error recovery handler in nvme-rdma (and nvme-tcp) does not start with
nvme_stop_ctrl()is because back then,nvme_stop_ctrl()could block until anadmin_timeoutexpired on an inflight admin command (I have a vague memory that if cancelled/flushed thescan_work), which delayed I/O failover, hence we just cherry-picked what needed to be stopped to make sure I/O failover will not be blocked by anything and progress quickly.Quoted 1 line
If I look at
nvme_stop_queue
nvme_stop_ctrlobviously...it does not touch the
scan_work, but itQuoted 10 lines
does flush
ana_workwhich includes I/O, so it can still block on it to fail afteradmin_timeoutexpires... also flushfailfast_workis taking thenamespaces_rwsem(although read access). Thefw_act_workis probably meaningless for fabrics...Maybe we need to have something like
nvme_stop_ctrl_fast, that will just stop the non-blocking stuff, so failover can work quickly, and then after we have failover, we can do the fullnvme_stop_ctrl()or the reminder ofnvme_stop_ctrl_fast...
Quoted 6 lines
Quoted 1 line
Quoted 1 line
thoughts?
I suspected that
nvme_stop_ctrl()might have unwanted side effects. Maybe factoring out the common bits fromnvme_stop_ctrl()intonvme_stop_ctrl_fast()and call it from here sounds like a reasonable thing to do. The function name could be better, but naming is hard.
The only name that comes to mind is nvme_stop_ctrl_noio, which is arguably worse... Or maybe just __nvme_stop_ctrl.
Maybe others have better names.
Quoted 7 lines · Sagi Grimberg
Quoted 4 lines
I suspected that
nvme_stop_ctrl()might have unwanted side effects. Maybe factoring out the common bits fromnvme_stop_ctrl()intonvme_stop_ctrl_fast()and call it from here sounds like a reasonable thing to do. The function name could be better, but naming is hard.The only name that comes to mind is
nvme_stop_ctrl_noio, which is arguably worse... Or maybe just__nvme_stop_ctrl.
I pondered for a while and no good idea. I'd prefer __nvme_stop_ctrl() clearly over nvme_stop_ctrl_foo() though, because I read __nvme_stop_ctrl() as to be called by nvme_stop_ctrl().