nvme: rdma/tcp: call nvme_mpath_stop() from reconnect workqueue
PATCH v2 1/1 nvme: rdma/tcp: call nvme_mpath_stop() from reconnect workqueue
From: Martin Wilck <[email protected]>
We have observed a few crashes run_timer_softirq(), where a broken timer_list struct belonging to an anatt_timer was encountered. The broken structures look like this, and we see actually multiple ones attached to the same timer base:
crash> struct timer_list 0xffff92471bcfdc90
struct timer_list {
entry = {
next = 0xdead000000000122, // LIST_POISON2
pprev = 0x0 },
expires = 4296022933,
function = 0xffffffffc06de5e0 <nvme_anatt_timeout>, flags = 20
}
If such a timer is encountered in run_timer_softirq(), the kernel crashes. The test scenario was an I/O load test with lots of NVMe controllers, some of which were removed and re-added on the storage side.
I think this may happen if the rdma recovery_work starts, in this call chain:
nvme_rdma_error_recovery_work()
/* this stops all sorts of activity for the controller, but not
the multipath-related work queue and timer */
nvme_rdma_reconnect_or_remove(ctrl)
=> kicks reconnect_work
work queue: reconnect_work
nvme_rdma_reconnect_ctrl_work()
nvme_rdma_setup_ctrl()
nvme_rdma_configure_admin_queue()
nvme_init_identify()
nvme_mpath_init()
# this sets some fields of the timer_list without taking a lock timer_setup()
nvme_read_ana_log()
mod_timer() or del_timer_sync()
Similar for TCP. The idea for the patch is based on the observation that nvme_rdma_reset_ctrl_work() calls nvme_stop_ctrl()->nvme_mpath_stop(), whereas nvme_rdma_error_recovery_work() stops only the keepalive timer, but not the anatt timer.
I admit that the root cause analysis isn't rock solid yet. In particular, I can't explain why we see LIST_POISON2 in the "next" pointer, which would indicate that the timer has been detached before; yet we find it linked to the timer base when the crash occurs.
OTOH, the anatt_timer is only touched in nvme_mpath_init() (see above) and nvme_mpath_stop(), so the hypothesis that modifying active timers may cause the issue isn't totally out of sight. I suspect that the LIST_POISON2 may come to pass in multiple steps.
If anyone has better ideas, please advise. The issue occurs very sporadically; verifying this by testing will be difficult.
----
Changes in v2: Moved call to nvme_mpath_stop() further down, directly before the call of nvme_rdma_reconnect_or_remove() (Chao Leng)
--- drivers/nvme/host/multipath.c | 1 + drivers/nvme/host/rdma.c | 1 + drivers/nvme/host/tcp.c | 1 + 3 files changed, 3 insertions(+)
drivers/nvme/host/multipath.c +1 -0
@@ -586,6 +586,7 @@ void nvme_mpath_stop(struct nvme_ctrl *ctrl)del_timer_sync (&ctrl->anatt_timer);cancel_work_sync (&ctrl->ana_work); } +EXPORT_SYMBOL_GPL (nvme_mpath_stop);#define SUBSYS_ATTR_RW (_name, _mode, _show, _store) \ struct device_attribute subsys_attr_##_name = \
drivers/nvme/host/rdma.c +1 -0
@@ -1202,6 +1202,7 @@ static void nvme_rdma_error_recovery_work(struct work_struct *work)return ; } +nvme_mpath_stop (&ctrl->ctrl);nvme_rdma_reconnect_or_remove (ctrl); }
drivers/nvme/host/tcp.c +1 -0
@@ -2068,6 +2068,7 @@ static void nvme_tcp_error_recovery_work(struct work_struct *work)return ; } +nvme_mpath_stop (ctrl);nvme_tcp_reconnect_or_remove (ctrl); }
Signature · 7 lines
2.31.1 _______________________________________________ Linux-nvme mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-nvme
A message that is not in the archive (sent off-list or not yet received): CAFL455k3aBLcZrZPq=Q-9aws4UesstA5gSOr_E7mEFrLT+KbKw@mail.gmail.com
Quoted 15 lines · Maurizio Lombardi
Hello,
pá 23. 4. 2021 v 15:49 odesílatel <[email protected]> napsal:
Quoted 12 lines
From: Martin Wilck <[email protected]>
drivers/nvme/host/tcp.c +0
-0@@ -2068,6 +2068,7 @@ static voidnvme_tcp_error_recovery_work(struct
work_struct*work)
return;
}
Quoted 10 lines
Quoted 4 lines
+ nvme_mpath_stop(ctrl);
nvme_tcp_reconnect_or_remove(ctrl);
}Do you think it would make sense to call
nvme_mpath_stop()inside nvme_tcp_reconnect_ctrl_work() too?
Right before it requeues itself by callingnvme_tcp_reconnect_or_remove(ctrl)?
Good point, probably yes. I'll have another look.
Thanks,
Martin
_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme
Quoted 27 lines · Martin Wilck
Quoted 15 lines · Maurizio Lombardi
Hello,
pá 23. 4. 2021 v 15:49 odesílatel <[email protected]> napsal:
Quoted 12 lines
From: Martin Wilck <[email protected]>
drivers/nvme/host/tcp.c +0
-0@@ -2068,6 +2068,7 @@ static voidnvme_tcp_error_recovery_work(struct
work_struct*work)
return;
}Quoted 10 lines
Quoted 4 lines
+ nvme_mpath_stop(ctrl);
nvme_tcp_reconnect_or_remove(ctrl);
}Do you think it would make sense to call
nvme_mpath_stop()inside nvme_tcp_reconnect_ctrl_work() too?
Right before it requeues itself by callingnvme_tcp_reconnect_or_remove(ctrl)?
Quoted 1 line
Good point, probably yes. I'll have another look.
I did now, and I don't think it's necessary. With my patch, nvme_mpath_stop() will be called the before the first invocation of nvme_tcp_reconnect_ctrl_work(). The anatt timer and mpath work queue will only be started if nvme_tcp_setup_ctrl() succeeds, so we don't need to stop them again when it fails.
Regards
Martin
_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme
drivers/nvme/host/rdma.c static void nvme_rdma_error_recovery_work(struct work_struct *work) · line 1206
Quoted 11 lines
drivers/nvme/host/rdma.c +1
-0@@ -1202,6 +1202,7 @@ static void nvme_rdma_error_recovery_work(struct work_struct *work)return ; } +nvme_mpath_stop (&ctrl->ctrl);nvme_rdma_reconnect_or_remove (ctrl);
Its pretty annoying to have to needlessly wait for the ana log page request to timeout... But this is also needed because we init ana_lock in nvme_mpath_init while it can be potentially taken in ana_work, which is a precipice for bad things...
So...
And this needs to go to stable, don't see a definitive offending commit here, so maybe just CC stable and have the title be more discriptive that it fixes a possible list-poison?
Something like:
nvme: rdma/tcp: fix a possible list corruption when reconnect work is racing with anatt timer
_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme
Christoph Hellwig
maintainerQuoted 15 lines · Sagi Grimberg
drivers/nvme/host/rdma.c static void nvme_rdma_error_recovery_work(struct work_struct *work) · line 1205
Quoted 10 lines
drivers/nvme/host/rdma.c +0
-0@@ -1202,6 +1202,7 @@ static void nvme_rdma_error_recovery_work(struct work_struct *work)return ; } +nvme_mpath_stop (&ctrl->ctrl);nvme_rdma_reconnect_or_remove (ctrl);Its pretty annoying to have to needlessly wait for the ana log page request to timeout... But this is also needed because we init
ana_lockinnvme_mpath_initwhile it can be potentially taken inana_work, which is a precipice for bad things...
I also really hate open coding this mpath detail in the transport drivers. Didn't you at some point have a series to factor out more common code from the whole reset and reconnect path?
_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme
Quoted 20 lines · Christoph Hellwig
Quoted 10 lines · Sagi Grimberg
Quoted 10 lines
drivers/nvme/host/rdma.c +0
-0@@ -1202,6 +1202,7 @@ static void nvme_rdma_error_recovery_work(struct work_struct *work) return; } + nvme_mpath_stop(&ctrl->ctrl);Quoted 8 lines
drivers/nvme/host/rdma.c static void nvme_rdma_error_recovery_work(struct work_struct *work) · line 1206
Quoted 1 line
drivers/nvme/host/rdma.c +0
-0nvme_rdma_reconnect_or_remove(ctrl);Its pretty annoying to have to needlessly wait for the ana log page request to timeout... But this is also needed because we init
ana_lock
innvme_mpath_initwhile it can be potentially taken inana_work, which
is a precipice for bad things...
Quoted 4 lines
I also really hate open coding this mpath detail in the transport drivers. Didn't you at some point have a series to factor out more common code from the whole reset and reconnect path?
So ... resubmit and cc stable as Sagi suggested, or work out something else?
Martin
_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme
Quoted 21 lines · Martin Wilck
Quoted 20 lines · Christoph Hellwig
Quoted 18 lines · Sagi Grimberg
drivers/nvme/host/rdma.c static void · line 1206
Quoted 11 lines
drivers/nvme/host/rdma.c +0
-0@@ -1202,6 +1202,7 @@ static void nvme_rdma_error_recovery_work(struct work_struct *work) return; } + nvme_mpath_stop(&ctrl->ctrl); nvme_rdma_reconnect_or_remove(ctrl);Its pretty annoying to have to needlessly wait for the ana log page request to timeout... But this is also needed because we init
ana_lock
innvme_mpath_initwhile it can be potentially taken inana_work, which
is a precipice for bad things...
Quoted 7 lines
Quoted 4 lines
I also really hate open coding this mpath detail in the transport drivers. Didn't you at some point have a series to factor out more common code from the whole reset and reconnect path?
So ... resubmit and cc stable as Sagi suggested, or work out something else?
Another option:call nvme_mpath_stop in nvme_mpath_init. In this way, transport drivers do not care mpath detail.
Quoted 6 lines
Martin
.
_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme
Quoted 33 lines · Chao Leng
Quoted 30 lines · Martin Wilck
Quoted 28 lines · Christoph Hellwig
Quoted 13 lines · Sagi Grimberg
drivers/nvme/host/rdma.c static void · line 1206
Quoted 12 lines
drivers/nvme/host/rdma.c +0
-0@@ -1202,6 +1202,7 @@ static void nvme_rdma_error_recovery_work(struct work_struct *work) return; } + nvme_mpath_stop(&ctrl->ctrl); nvme_rdma_reconnect_or_remove(ctrl);Quoted 8 lines
Its pretty annoying to have to needlessly wait for the ana log page
request to timeout... But this is also needed because we initana_lock
innvme_mpath_initwhile it can be potentially taken inana_work,
which
is a precipice for bad things...I also really hate open coding this mpath detail in the transport drivers. Didn't you at some point have a series to factor out more
common code from the whole reset and reconnect path?
Quoted 5 lines
Quoted 3 lines
So ... resubmit and cc stable as Sagi suggested, or work out something
else?Another option:call
nvme_mpath_stopinnvme_mpath_init. In this way, transport drivers do not care mpath detail.
That would require that we determine in nvme_mpath_init() whether we're called for the first time for the given controller, or whether the controller is re-initializing. It's not obvious to me how to do that reliably without introducing a new state flag.
Regards,
Martin
_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme
Quoted 5 lines · Martin Wilck
Quoted 4 lines · Chao Leng
Quoted 4 lines
So ... resubmit and cc stable as Sagi suggested, or work out something
else?
Quoted 9 lines
Quoted 2 lines
Another option:call
nvme_mpath_stopinnvme_mpath_init. In this way, transport drivers do not care mpath detail.That would require that we determine in
nvme_mpath_init()whether we're
called for the first time for the given controller, or whether the controller is re-initializing. It's not obvious to me how to do that reliably without introducing a new state flag.
I'll submit a v3 with changed subject as suggested by Sagi. That will fix the list corruption; we can figure out a better solution later.
Martin
_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme
Quoted 11 lines · Sagi Grimberg
Quoted 11 lines
drivers/nvme/host/rdma.c +0
-0@@ -1202,6 +1202,7 @@ static void nvme_rdma_error_recovery_work(struct work_struct *work) return; }
+ nvme_mpath_stop(&ctrl->ctrl);
Quoted 8 lines
Quoted 1 line
nvme_rdma_reconnect_or_remove(ctrl);
Its pretty annoying to have to needlessly wait for the ana log page request to timeout..
But this is also needed because we initana_lock
innvme_mpath_initwhile it can be potentially taken inana_work, which
is a precipice for bad things...
What if we move the ana-related fields into a separately kmalloc'd structure, rather than embedding it in nvme_ctrl? That way we could just detach it when the controller is re-initialized, without needing to wait. When the timeouts eventually occur, the worker func would realize that it's detached and just do nothing.
Regards
Martin
_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme
Looks good.
Quoted 102 lines · On 2021/4/23 21:38, [email protected] wrote:
From: Martin Wilck <[email protected]>
We have observed a few crashes
run_timer_softirq(), where a brokentimer_liststruct belonging to ananatt_timerwas encountered. The broken structures look like this, and we see actually multiple ones attached to the same timer base:crash> struct
timer_list0xffff92471bcfdc90struct timer_list { entry = { next = 0xdead000000000122, // LIST_POISON2 pprev = 0x0 }, expires = 4296022933, function = 0xffffffffc06de5e0 <nvme_anatt_timeout>, flags = 20 }If such a timer is encountered in
run_timer_softirq(), the kernel crashes. The test scenario was an I/O load test with lots of NVMe controllers, some of which were removed and re-added on the storage side.I think this may happen if the rdma
recovery_workstarts, in this call chain:
nvme_rdma_error_recovery_work()
/* this stops all sorts of activity for the controller, but not
the multipath-related work queue and timer */
nvme_rdma_reconnect_or_remove(ctrl)
=> kicksreconnect_workwork queue:
reconnect_work
nvme_rdma_reconnect_ctrl_work()
nvme_rdma_setup_ctrl()
nvme_rdma_configure_admin_queue()
nvme_init_identify()
nvme_mpath_init()
# this sets some fields of thetimer_listwithout taking a lock
timer_setup()
nvme_read_ana_log()
mod_timer()ordel_timer_sync()Similar for TCP. The idea for the patch is based on the observation that
nvme_rdma_reset_ctrl_work()calls nvme_stop_ctrl()->nvme_mpath_stop(), whereasnvme_rdma_error_recovery_work()stops only the keepalive timer, but not the anatt timer.I admit that the root cause analysis isn't rock solid yet. In particular, I can't explain why we see
LIST_POISON2in the "next" pointer, which would indicate that the timer has been detached before; yet we find it linked to the timer base when the crash occurs.OTOH, the
anatt_timeris only touched innvme_mpath_init()(see above) andnvme_mpath_stop(), so the hypothesis that modifying active timers may cause the issue isn't totally out of sight. I suspect that theLIST_POISON2may come to pass in multiple steps.If anyone has better ideas, please advise. The issue occurs very sporadically; verifying this by testing will be difficult.
Signed-off-by Martin Wilck <[email protected]>----
Changes in v2: Moved call tonvme_mpath_stop()further down, directly before the call ofnvme_rdma_reconnect_or_remove()(Chao Leng)--- drivers/nvme/host/multipath.c | 1 + drivers/nvme/host/rdma.c | 1 + drivers/nvme/host/tcp.c | 1 + 3 files changed, 3 insertions(+)drivers/nvme/host/multipath.c +1
-0@@ -586,6 +586,7 @@ void nvme_mpath_stop(struct nvme_ctrl *ctrl)del_timer_sync (&ctrl->anatt_timer);cancel_work_sync (&ctrl->ana_work); } +EXPORT_SYMBOL_GPL (nvme_mpath_stop);#define SUBSYS_ATTR_RW (_name, _mode, _show, _store) \ struct device_attribute subsys_attr_##_name = \drivers/nvme/host/rdma.c +1
-0@@ -1202,6 +1202,7 @@ static void nvme_rdma_error_recovery_work(struct work_struct *work)return ; } +nvme_mpath_stop (&ctrl->ctrl);nvme_rdma_reconnect_or_remove (ctrl); }drivers/nvme/host/tcp.c +0
-0
drivers/nvme/host/tcp.c static void nvme_tcp_error_recovery_work(struct work_struct *work) · line 2074
Quoted 7 lines
drivers/nvme/host/tcp.c +1
-0@@ -2068,6 +2068,7 @@ static void nvme_tcp_error_recovery_work(struct work_struct *work)return ; } +nvme_mpath_stop (ctrl);nvme_tcp_reconnect_or_remove (ctrl); }
_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme
Quoted 18 lines · On 4/23/21 3:38 PM, [email protected] wrote:
From: Martin Wilck <[email protected]>
We have observed a few crashes
run_timer_softirq(), where a brokentimer_liststruct belonging to ananatt_timerwas encountered. The broken structures look like this, and we see actually multiple ones attached to the same timer base:crash> struct
timer_list0xffff92471bcfdc90struct timer_list { entry = { next = 0xdead000000000122, // LIST_POISON2 pprev = 0x0 }, expires = 4296022933, function = 0xffffffffc06de5e0 <nvme_anatt_timeout>, flags = 20 }
Quoted 4 lines
If such a timer is encountered in
run_timer_softirq(), the kernel crashes. The test scenario was an I/O load test with lots of NVMe controllers, some of which were removed and re-added on the storage side.
...
But isn't this the result of detach_timer()? IE this suspiciously looks like perfectly normal operation; is you look at expire_timers() we're first calling 'detach_timer()' before calling the timer function, ie every crash in the timer function would have this signature. And, incidentally, so would any timer function which does not crash.
Sorry to kill your analysis ...
This doesn't mean that the patch isn't valid (in the sense that it resolve the issue), but we definitely will need to work on root cause analysis.
Cheera,
Hannes
Signature · 9 lines
Dr. Hannes Reinecke Kernel Storage Architect [email protected] +49 911 74053 688 SUSE Software Solutions GmbH, Maxfeldstr. 5, 90409 Nürnberg HRB 36809 (AG Nürnberg), Geschäftsführer: Felix Imendörffer _______________________________________________ Linux-nvme mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-nvme
Quoted 23 lines · Hannes Reinecke
Quoted 18 lines · On 4/23/21 3:38 PM, [email protected] wrote:
From: Martin Wilck <[email protected]>
We have observed a few crashes
run_timer_softirq(), where a brokentimer_liststruct belonging to ananatt_timerwas encountered. The broken structures look like this, and we see actually multiple ones attached to the same timer base:crash> struct
timer_list0xffff92471bcfdc90structtimer_list{
entry = {
next = 0xdead000000000122, //LIST_POISON2pprev =0x0
},
expires = 4296022933,
function =0xffffffffc06de5e0<nvme_anatt_timeout>, flags = 20
}Quoted 4 lines
If such a timer is encountered in
run_timer_softirq(), the kernel crashes. The test scenario was an I/O load test with lots of NVMe controllers, some of which were removed and re-added on the storage side.
Quoted 8 lines
...
But isn't this the result of
detach_timer()? IE this suspiciously looks like perfectly normal operation; is you look atexpire_timers()we're first calling 'detach_timer()' before calling the timer function, ie every crash in the timer function would have this signature. And, incidentally, so would any timer function which does not crash.Sorry to kill your analysis ...
This doesn't mean that the patch isn't valid (in the sense that it resolve the issue), but we definitely will need to work on root cause analysis.
The process maybe:1.ana_work add the timer;2.error recovery occurs, in reconnecting, reinitialize the timer and call nvme_read_ana_log, nvme_read_ana_log may add the timer again.
The same timer is added twice, crash will happens later.
Indeed ana_log_buf has the similar bug, it's been encountered in our testing. To fix this bug, I also make the same patch and tested for more than 2 weeks.
This patch can fix the both bugs.
Quoted 4 lines
Cheera,
Hannes
_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme
Quoted 21 lines · Chao Leng
Quoted 18 lines · Hannes Reinecke
Quoted 6 lines · On 4/23/21 3:38 PM, [email protected] wrote:
If such a timer is encountered in
run_timer_softirq(), the kernel crashes. The test scenario was an I/O load test with lots of NVMe controllers, some of which were removed and re-added on the storage side....
But isn't this the result of
detach_timer()? IE this suspiciously looks like perfectly normal operation; is you look atexpire_timers()we're first calling 'detach_timer()' before calling the timer function, ie every crash in the timer function would have this signature.
And, incidentally, so would any timer function which does not crash.Sorry to kill your analysis ...
Quoted 7 lines
Quoted 3 lines
This doesn't mean that the patch isn't valid (in the sense that it resolve the issue), but we definitely will need to work on root cause analysis.
The process maybe:1.ana_work add the timer;2.error recovery occurs, in reconnecting, reinitialize the timer and call
nvme_read_ana_log,nvme_read_ana_logmay add the timer again.
The same timer is added twice, crash will happens later.
Thanks a lot! I think you nailed it.
I had another look at my dump with this in mind, and indeed I was able to prove that the timer of the same NVMe controller had been started (at least) twice. This explains why it looks as if it had been cleanly detached, and is still running nonetheless.
In particular, I saw this:
[ 4389.939928] nvme nvme36: queue_size 128 > ctrl sqsize 32, clamping down [ 4389.962709] nvme nvme36: creating 4 I/O queues. [ 4389.978462] nvme nvme36: starting error recovery [ 4389.978509] nvme nvme36: failed to send request -32 [ 4389.978724] nvme nvme36: Failed to configure AEN (cfg 900) [ 4389.978732] nvme nvme36: Successfully reconnected (1 attempt) [ 4389.978739] nvme nvme36: failed to send request -32 [ 4389.991420] nvme nvme36: Reconnecting in 10 seconds... ... [ 4441.399435] nvme nvme36: queue_size 128 > ctrl sqsize 32, clamping down [ 4441.407647] nvme nvme36: creating 4 I/O queues. [ 4441.445655] nvme nvme36: Successfully reconnected (1 attempt) [ 4441.699665] nvme nvme36: starting error recovery [ 4441.719493] nvme nvme36: Identify Descriptors failed (881) [ 4441.735409] nvme nvme36: Reconnecting in 10 seconds... ... [ 4510.891400] nvme nvme36: ANATT timeout, resetting controller. ... [ 4517.035411] general protection fault: 0000 [#1] SMP PTI
The protection fault was caused by the anatt timer of another nvme controller. But I could see the anatt_timer of nvme36 still linked to an active timer_base, about to expire 44s in the future. Calculating back, that timer had been started around 4441s above, whereas the timer that expired a few seconds earlier must have been started at 4389.9.
Regards,
Martin
_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme
Quoted 28 lines · Hannes Reinecke
Quoted 25 lines · On 4/23/21 3:38 PM, [email protected] wrote:
From: Martin Wilck <[email protected]>
We have observed a few crashes
run_timer_softirq(), where a brokentimer_liststruct belonging to ananatt_timerwas encountered. The broken
structures look like this, and we see actually multiple ones attached to
the same timer base:crash> struct
timer_list0xffff92471bcfdc90structtimer_list{
entry = {
next = 0xdead000000000122, //LIST_POISON2pprev =0x0
},
expires = 4296022933,
function =0xffffffffc06de5e0<nvme_anatt_timeout>, flags = 20
}If such a timer is encountered in
run_timer_softirq(), the kernel crashes. The test scenario was an I/O load test with lots of NVMe controllers, some of which were removed and re-added on the storage side....
Quoted 7 lines
But isn't this the result of
detach_timer()? IE this suspiciously looks like perfectly normal operation; is you look atexpire_timers()we're first calling 'detach_timer()' before calling the timer function, ie every crash in the timer function would have this signature. And, incidentally, so would any timer function which does not crash.Sorry to kill your analysis ...
No problem, I realized this myself, and actually mentioned it in the commit description. OTOH, this timer is only modified in very few places, and all but nvme_mpath_init() use the proper APIs for modifying or deleting timers, so the initialization of a (possibly still) running timer is the only suspect, afaics.
My personal theory is that the corruption might happen in several steps, the first step being timer_setup() mofiying fields of a pending timer. But I couldn't figure it out completely, and found it too hand- waving to mention in the commit description.
Quoted 3 lines
This doesn't mean that the patch isn't valid (in the sense that it resolve the issue), but we definitely will need to work on root cause analysis.
I'd be grateful for any help figuring out the missing bits.
Thanks,
Martin
_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme
Quoted 48 lines · Martin Wilck
Quoted 28 lines · Hannes Reinecke
Quoted 25 lines · On 4/23/21 3:38 PM, [email protected] wrote:
From: Martin Wilck <[email protected]>
We have observed a few crashes
run_timer_softirq(), where a brokentimer_liststruct belonging to ananatt_timerwas encountered. The broken
structures look like this, and we see actually multiple ones attached to
the same timer base:crash> struct
timer_list0xffff92471bcfdc90structtimer_list{
entry = {
next = 0xdead000000000122, //LIST_POISON2pprev =0x0
},
expires = 4296022933,
function =0xffffffffc06de5e0<nvme_anatt_timeout>, flags = 20
}If such a timer is encountered in
run_timer_softirq(), the kernel crashes. The test scenario was an I/O load test with lots of NVMe controllers, some of which were removed and re-added on the storage side....
Quoted 7 lines
But isn't this the result of
detach_timer()? IE this suspiciously looks like perfectly normal operation; is you look atexpire_timers()we're first calling 'detach_timer()' before calling the timer function, ie every crash in the timer function would have this signature. And, incidentally, so would any timer function which does not crash.Sorry to kill your analysis ...
No problem, I realized this myself, and actually mentioned it in the commit description. OTOH, this timer is only modified in very few places, and all but
nvme_mpath_init()use the proper APIs for modifying or deleting timers, so the initialization of a (possibly still) running timer is the only suspect, afaics.My personal theory is that the corruption might happen in several steps, the first step being
timer_setup()mofiying fields of a pending timer. But I couldn't figure it out completely, and found it too hand- waving to mention in the commit description.
Quoted 6 lines
Quoted 3 lines
This doesn't mean that the patch isn't valid (in the sense that it resolve the issue), but we definitely will need to work on root cause analysis.
I'd be grateful for any help figuring out the missing bits.
No, the problem lies in the pecularities of NVMe error recovery.
The crucial point here is that during error recovery all I/O is aborted, but after that the queue is _NOT_ stopped.
So it's perfectly legal to send I/O even if the controller is not fully LIVE. And if _this_ I/O fails then we'll find ourselves in lots of interesting situations.
I guess the problem lies with nvme_init_identify().
We're calling nvme_mpath_init() (which will start the ANATT timer), but issue some more I/O afterwards.
If any of _that_ I/O fails, an error is registered with the caller (eg in nvme_tcp_configure_admin_queue(), _but_ the ANATT timer continues to run. Then we'll be calling nvme_tcp_reconnect_or_remove() and delete the controller. Completely oblivious to the running ANATT timer. Which will get quite unhappy once it triggers as the controller structure from which the timer was started is done for.
So with this analysis an alternative fix would be to move nvme_path_init() right at the end of nvme_init_identify(), such that we'll never get into the situation where nvme_init_identify() fails yet the ANATT timer is running.
Cheers,
Hannes
Signature · 9 lines
Dr. Hannes Reinecke Kernel Storage Architect [email protected] +49 911 74053 688 SUSE Software Solutions Germany GmbH, 90409 Nürnberg GF: F. Imendörffer, HRB 36809 (AG Nürnberg) _______________________________________________ Linux-nvme mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-nvme