Skip to contentSkip to conversation

nvme: rdma/tcp: call nvme_mpath_stop() from reconnect workqueue

PATCH In review [email protected] 5 participants 16 messages Apr 23, 2021 – Apr 27, 2021 1 message not in the archive NVMe

lore
Share & tools
lore
#0 lore raw

PATCH v2 1/1 nvme: rdma/tcp: call nvme_mpath_stop() from reconnect workqueue

From: Martin Wilck <[email protected]>

We have observed a few crashes run_timer_softirq(), where a broken timer_list struct belonging to an anatt_timer was encountered. The broken structures look like this, and we see actually multiple ones attached to the same timer base:

crash> struct timer_list 0xffff92471bcfdc90
struct timer_list {

  entry = {
    next = 0xdead000000000122,  // LIST_POISON2
    pprev = 0x0

},
expires = 4296022933,
function = 0xffffffffc06de5e0 <nvme_anatt_timeout>, flags = 20
}

If such a timer is encountered in run_timer_softirq(), the kernel crashes. The test scenario was an I/O load test with lots of NVMe controllers, some of which were removed and re-added on the storage side.

I think this may happen if the rdma recovery_work starts, in this call chain:

nvme_rdma_error_recovery_work()
/* this stops all sorts of activity for the controller, but not
the multipath-related work queue and timer */
nvme_rdma_reconnect_or_remove(ctrl)
=> kicks reconnect_work

work queue: reconnect_work

nvme_rdma_reconnect_ctrl_work()
nvme_rdma_setup_ctrl()
nvme_rdma_configure_admin_queue()
nvme_init_identify()
nvme_mpath_init()
# this sets some fields of the timer_list without taking a lock timer_setup()
nvme_read_ana_log()
mod_timer() or del_timer_sync()

Similar for TCP. The idea for the patch is based on the observation that nvme_rdma_reset_ctrl_work() calls nvme_stop_ctrl()->nvme_mpath_stop(), whereas nvme_rdma_error_recovery_work() stops only the keepalive timer, but not the anatt timer.

I admit that the root cause analysis isn't rock solid yet. In particular, I can't explain why we see LIST_POISON2 in the "next" pointer, which would indicate that the timer has been detached before; yet we find it linked to the timer base when the crash occurs.

OTOH, the anatt_timer is only touched in nvme_mpath_init() (see above) and nvme_mpath_stop(), so the hypothesis that modifying active timers may cause the issue isn't totally out of sight. I suspect that the LIST_POISON2 may come to pass in multiple steps.

If anyone has better ideas, please advise. The issue occurs very sporadically; verifying this by testing will be difficult.

----
Changes in v2: Moved call to nvme_mpath_stop() further down, directly before the call of nvme_rdma_reconnect_or_remove() (Chao Leng)

---
 drivers/nvme/host/multipath.c | 1 +
 drivers/nvme/host/rdma.c      | 1 +
 drivers/nvme/host/tcp.c       | 1 +
 3 files changed, 3 insertions(+)
drivers/nvme/host/multipath.c +1 -0
index a1d476e1ac02..c63dd5dfa7ff 100644
@@ -586,6 +586,7 @@ void nvme_mpath_stop(struct nvme_ctrl *ctrl)
 	del_timer_sync(&ctrl->anatt_timer);
 	cancel_work_sync(&ctrl->ana_work);
 }
+EXPORT_SYMBOL_GPL(nvme_mpath_stop);
 
 #define SUBSYS_ATTR_RW(_name, _mode, _show, _store)  \
 	struct device_attribute subsys_attr_##_name =	\
drivers/nvme/host/rdma.c +1 -0
index be905d4fdb47..fc07a7b0dc1d 100644
@@ -1202,6 +1202,7 @@ static void nvme_rdma_error_recovery_work(struct work_struct *work)
 		return;
 	}
 
+	nvme_mpath_stop(&ctrl->ctrl);
 	nvme_rdma_reconnect_or_remove(ctrl);
 }
 
drivers/nvme/host/tcp.c +1 -0
index a0f00cb8f9f3..46287b4f4d10 100644
@@ -2068,6 +2068,7 @@ static void nvme_tcp_error_recovery_work(struct work_struct *work)
 		return;
 	}
 
+	nvme_mpath_stop(ctrl);
 	nvme_tcp_reconnect_or_remove(ctrl);
 }
 
Signature · 7 lines
2.31.1


_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme

A message that is not in the archive (sent off-list or not yet received): CAFL455k3aBLcZrZPq=Q-9aws4UesstA5gSOr_E7mEFrLT+KbKw@mail.gmail.com

#2 lore raw
Quoted 15 lines · Maurizio Lombardi

Hello,

pá 23. 4. 2021 v 15:49 odesílatel <[email protected]> napsal:

Quoted 12 lines

From: Martin Wilck <[email protected]>

drivers/nvme/host/tcp.c +0 -0
index a0f00cb8f9f3..46287b4f4d10 100644
@@ -2068,6 +2068,7 @@ static void

nvme_tcp_error_recovery_work(struct
work_struct *work)
                return;
        }

Quoted 10 lines
Quoted 4 lines

+       nvme_mpath_stop(ctrl);
        nvme_tcp_reconnect_or_remove(ctrl);
 }

Do you think it would make sense to call nvme_mpath_stop() inside nvme_tcp_reconnect_ctrl_work() too?
Right before it requeues itself by calling nvme_tcp_reconnect_or_remove(ctrl)?

Good point, probably yes. I'll have another look.

Thanks,
Martin

_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme

#3 lore raw
Quoted 27 lines · Martin Wilck
Quoted 15 lines · Maurizio Lombardi

Hello,

pá 23. 4. 2021 v 15:49 odesílatel <[email protected]> napsal:

Quoted 12 lines

From: Martin Wilck <[email protected]>

drivers/nvme/host/tcp.c +0 -0
index a0f00cb8f9f3..46287b4f4d10 100644
@@ -2068,6 +2068,7 @@ static void

nvme_tcp_error_recovery_work(struct
work_struct *work)
                return;
        }

Quoted 10 lines
Quoted 4 lines

+       nvme_mpath_stop(ctrl);
        nvme_tcp_reconnect_or_remove(ctrl);
 }

Do you think it would make sense to call nvme_mpath_stop() inside nvme_tcp_reconnect_ctrl_work() too?
Right before it requeues itself by calling nvme_tcp_reconnect_or_remove(ctrl)?

Quoted 1 line

Good point, probably yes. I'll have another look.

I did now, and I don't think it's necessary. With my patch, nvme_mpath_stop() will be called the before the first invocation of nvme_tcp_reconnect_ctrl_work(). The anatt timer and mpath work queue will only be started if nvme_tcp_setup_ctrl() succeeds, so we don't need to stop them again when it fails.

Regards
Martin

_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme

maintainer Reviewed-by on v2 1/1
#4 lore raw
In reply to [email protected]

drivers/nvme/host/rdma.c static void nvme_rdma_error_recovery_work(struct work_struct *work) · line 1206

Quoted 11 lines
drivers/nvme/host/rdma.c +1 -0
index be905d4fdb47..fc07a7b0dc1d 100644
@@ -1202,6 +1202,7 @@ static void nvme_rdma_error_recovery_work(struct work_struct *work)
  		return;
  	}
 
+	nvme_mpath_stop(&ctrl->ctrl);
  	nvme_rdma_reconnect_or_remove(ctrl);

Its pretty annoying to have to needlessly wait for the ana log page request to timeout... But this is also needed because we init ana_lock in nvme_mpath_init while it can be potentially taken in ana_work, which is a precipice for bad things...

So...

And this needs to go to stable, don't see a definitive offending commit here, so maybe just CC stable and have the title be more discriptive that it fixes a possible list-poison?

Something like:
nvme: rdma/tcp: fix a possible list corruption when reconnect work is racing with anatt timer

_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme

#5 lore raw
Quoted 15 lines · Sagi Grimberg

drivers/nvme/host/rdma.c static void nvme_rdma_error_recovery_work(struct work_struct *work) · line 1205

Quoted 10 lines
drivers/nvme/host/rdma.c +0 -0
index be905d4fdb47..fc07a7b0dc1d 100644
@@ -1202,6 +1202,7 @@ static void nvme_rdma_error_recovery_work(struct work_struct *work)
  		return;
  	}
  +	nvme_mpath_stop(&ctrl->ctrl);
  	nvme_rdma_reconnect_or_remove(ctrl);

Its pretty annoying to have to needlessly wait for the ana log page request to timeout... But this is also needed because we init ana_lock in nvme_mpath_init while it can be potentially taken in ana_work, which is a precipice for bad things...

I also really hate open coding this mpath detail in the transport drivers. Didn't you at some point have a series to factor out more common code from the whole reset and reconnect path?

_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme

#6 lore raw
Quoted 20 lines · Christoph Hellwig
Quoted 10 lines · Sagi Grimberg
Quoted 10 lines
drivers/nvme/host/rdma.c +0 -0
index be905d4fdb47..fc07a7b0dc1d 100644
@@ -1202,6 +1202,7 @@ static void
 nvme_rdma_error_recovery_work(struct work_struct *work)
                 return;
         }
   +     nvme_mpath_stop(&ctrl->ctrl);
Quoted 8 lines

drivers/nvme/host/rdma.c static void nvme_rdma_error_recovery_work(struct work_struct *work) · line 1206

Quoted 1 line
drivers/nvme/host/rdma.c +0 -0
         nvme_rdma_reconnect_or_remove(ctrl);

Its pretty annoying to have to needlessly wait for the ana log page request to timeout... But this is also needed because we init ana_lock
in nvme_mpath_init while it can be potentially taken in ana_work, which
is a precipice for bad things...

Quoted 4 lines

I also really hate open coding this mpath detail in the transport drivers.  Didn't you at some point have a series to factor out more common code from the whole reset and reconnect path?

So ... resubmit and cc stable as Sagi suggested, or work out something else?

Martin

_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme

#7 lore raw
Quoted 21 lines · Martin Wilck
Quoted 20 lines · Christoph Hellwig
Quoted 18 lines · Sagi Grimberg

drivers/nvme/host/rdma.c static void · line 1206

Quoted 11 lines
drivers/nvme/host/rdma.c +0 -0
index be905d4fdb47..fc07a7b0dc1d 100644
@@ -1202,6 +1202,7 @@ static void
 nvme_rdma_error_recovery_work(struct work_struct *work)
                 return;
         }
   +     nvme_mpath_stop(&ctrl->ctrl);
         nvme_rdma_reconnect_or_remove(ctrl);

Its pretty annoying to have to needlessly wait for the ana log page request to timeout... But this is also needed because we init ana_lock
in nvme_mpath_init while it can be potentially taken in ana_work, which
is a precipice for bad things...

Quoted 7 lines
Quoted 4 lines

I also really hate open coding this mpath detail in the transport drivers.  Didn't you at some point have a series to factor out more common code from the whole reset and reconnect path?

So ... resubmit and cc stable as Sagi suggested, or work out something else?

Another option:call nvme_mpath_stop in nvme_mpath_init. In this way, transport drivers do not care mpath detail.

Quoted 6 lines

Martin

.

_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme

#8 lore raw
Quoted 33 lines · Chao Leng
Quoted 30 lines · Martin Wilck
Quoted 28 lines · Christoph Hellwig
Quoted 13 lines · Sagi Grimberg

drivers/nvme/host/rdma.c static void · line 1206

Quoted 12 lines
drivers/nvme/host/rdma.c +0 -0
index be905d4fdb47..fc07a7b0dc1d 100644
@@ -1202,6 +1202,7 @@ static void
 nvme_rdma_error_recovery_work(struct work_struct *work)
                  return;
          }
    +     nvme_mpath_stop(&ctrl->ctrl);
          nvme_rdma_reconnect_or_remove(ctrl);
Quoted 8 lines

Its pretty annoying to have to needlessly wait for the ana log page
request to timeout... But this is also needed because we init ana_lock
in nvme_mpath_init while it can be potentially taken in ana_work,
which
is a precipice for bad things...

I also really hate open coding this mpath detail in the transport drivers.  Didn't you at some point have a series to factor out more
common code from the whole reset and reconnect path?

Quoted 5 lines
Quoted 3 lines

So ... resubmit and cc stable as Sagi suggested, or work out something
else?

Another option:call nvme_mpath_stop in nvme_mpath_init. In this way, transport drivers do not care mpath detail.

That would require that we determine in nvme_mpath_init() whether we're called for the first time for the given controller, or whether the controller is re-initializing. It's not obvious to me how to do that reliably without introducing a new state flag.

Regards,
Martin

_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme

#9 lore raw
Quoted 5 lines · Martin Wilck
Quoted 4 lines · Chao Leng
Quoted 4 lines

So ... resubmit and cc stable as Sagi suggested, or work out something
else?

Quoted 9 lines
Quoted 2 lines

Another option:call nvme_mpath_stop in nvme_mpath_init. In this way, transport drivers do not care mpath detail.

That would require that we determine in nvme_mpath_init() whether we're
called for the first time for the given controller, or whether the controller is re-initializing. It's not obvious to me how to do that reliably without introducing a new state flag.

I'll submit a v3 with changed subject as suggested by Sagi. That will fix the list corruption; we can figure out a better solution later.

Martin

_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme

#10 lore raw
In reply to Sagi Grimberg
Quoted 11 lines · Sagi Grimberg
Quoted 11 lines
drivers/nvme/host/rdma.c +0 -0
index be905d4fdb47..fc07a7b0dc1d 100644
@@ -1202,6 +1202,7 @@ static void
 nvme_rdma_error_recovery_work(struct work_struct *work)
                 return;
         }


+       nvme_mpath_stop(&ctrl->ctrl);

Quoted 8 lines
Quoted 1 line

        nvme_rdma_reconnect_or_remove(ctrl);

Its pretty annoying to have to needlessly wait for the ana log page request to timeout..
But this is also needed because we init ana_lock
in nvme_mpath_init while it can be potentially taken in ana_work, which
is a precipice for bad things...

What if we move the ana-related fields into a separately kmalloc'd structure, rather than embedding it in nvme_ctrl? That way we could just detach it when the controller is re-initialized, without needing to wait. When the timeouts eventually occur, the worker func would realize that it's detached and just do nothing.

Regards
Martin

_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme

Reviewed-by on v2 1/1
#11 lore raw
In reply to [email protected]

Looks good.

Quoted 102 lines · On 2021/4/23 21:38, [email protected] wrote:

From: Martin Wilck <[email protected]>

We have observed a few crashes run_timer_softirq(), where a broken timer_list struct belonging to an anatt_timer was encountered. The broken structures look like this, and we see actually multiple ones attached to the same timer base:

crash> struct timer_list 0xffff92471bcfdc90

struct timer_list {
   entry = {
     next = 0xdead000000000122,  // LIST_POISON2
     pprev = 0x0
   },
   expires = 4296022933,
   function = 0xffffffffc06de5e0 <nvme_anatt_timeout>,
   flags = 20
}

If such a timer is encountered in run_timer_softirq(), the kernel crashes. The test scenario was an I/O load test with lots of NVMe controllers, some of which were removed and re-added on the storage side.

I think this may happen if the rdma recovery_work starts, in this call chain:

nvme_rdma_error_recovery_work()
/* this stops all sorts of activity for the controller, but not
the multipath-related work queue and timer */
nvme_rdma_reconnect_or_remove(ctrl)
=> kicks reconnect_work

work queue: reconnect_work

nvme_rdma_reconnect_ctrl_work()
nvme_rdma_setup_ctrl()
nvme_rdma_configure_admin_queue()
nvme_init_identify()
nvme_mpath_init()
# this sets some fields of the timer_list without taking a lock
timer_setup()
nvme_read_ana_log()
mod_timer() or del_timer_sync()

Similar for TCP. The idea for the patch is based on the observation that nvme_rdma_reset_ctrl_work() calls nvme_stop_ctrl()->nvme_mpath_stop(), whereas nvme_rdma_error_recovery_work() stops only the keepalive timer, but not the anatt timer.

I admit that the root cause analysis isn't rock solid yet. In particular, I can't explain why we see LIST_POISON2 in the "next" pointer, which would indicate that the timer has been detached before; yet we find it linked to the timer base when the crash occurs.

OTOH, the anatt_timer is only touched in nvme_mpath_init() (see above) and nvme_mpath_stop(), so the hypothesis that modifying active timers may cause the issue isn't totally out of sight. I suspect that the LIST_POISON2 may come to pass in multiple steps.

If anyone has better ideas, please advise. The issue occurs very sporadically; verifying this by testing will be difficult.

----
Changes in v2: Moved call to nvme_mpath_stop() further down, directly before the call of nvme_rdma_reconnect_or_remove() (Chao Leng)

---
  drivers/nvme/host/multipath.c | 1 +
  drivers/nvme/host/rdma.c      | 1 +
  drivers/nvme/host/tcp.c       | 1 +
  3 files changed, 3 insertions(+)
drivers/nvme/host/multipath.c +1 -0
index a1d476e1ac02..c63dd5dfa7ff 100644
@@ -586,6 +586,7 @@ void nvme_mpath_stop(struct nvme_ctrl *ctrl)
  	del_timer_sync(&ctrl->anatt_timer);
  	cancel_work_sync(&ctrl->ana_work);
  }
+EXPORT_SYMBOL_GPL(nvme_mpath_stop);
 
  #define SUBSYS_ATTR_RW(_name, _mode, _show, _store)  \
  	struct device_attribute subsys_attr_##_name =	\
drivers/nvme/host/rdma.c +1 -0
index be905d4fdb47..fc07a7b0dc1d 100644
@@ -1202,6 +1202,7 @@ static void nvme_rdma_error_recovery_work(struct work_struct *work)
  		return;
  	}
 
+	nvme_mpath_stop(&ctrl->ctrl);
  	nvme_rdma_reconnect_or_remove(ctrl);
  }
 
drivers/nvme/host/tcp.c +0 -0
index a0f00cb8f9f3..46287b4f4d10 100644

drivers/nvme/host/tcp.c static void nvme_tcp_error_recovery_work(struct work_struct *work) · line 2074

Quoted 7 lines
drivers/nvme/host/tcp.c +1 -0
@@ -2068,6 +2068,7 @@ static void nvme_tcp_error_recovery_work(struct work_struct *work)
  		return;
  	}
 
+	nvme_mpath_stop(ctrl);
  	nvme_tcp_reconnect_or_remove(ctrl);
  }
 

_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme

#12 lore raw
In reply to [email protected]
Quoted 18 lines · On 4/23/21 3:38 PM, [email protected] wrote:

From: Martin Wilck <[email protected]>

We have observed a few crashes run_timer_softirq(), where a broken timer_list struct belonging to an anatt_timer was encountered. The broken structures look like this, and we see actually multiple ones attached to the same timer base:

crash> struct timer_list 0xffff92471bcfdc90

struct timer_list {
   entry = {
     next = 0xdead000000000122,  // LIST_POISON2
     pprev = 0x0
   },
   expires = 4296022933,
   function = 0xffffffffc06de5e0 <nvme_anatt_timeout>,
   flags = 20
}
Quoted 4 lines

If such a timer is encountered in run_timer_softirq(), the kernel crashes. The test scenario was an I/O load test with lots of NVMe controllers, some of which were removed and re-added on the storage side.

...

But isn't this the result of detach_timer()? IE this suspiciously looks like perfectly normal operation; is you look at expire_timers() we're first calling 'detach_timer()' before calling the timer function, ie every crash in the timer function would have this signature. And, incidentally, so would any timer function which does not crash.

Sorry to kill your analysis ...

This doesn't mean that the patch isn't valid (in the sense that it resolve the issue), but we definitely will need to work on root cause analysis.

Cheera,

Hannes

Signature · 9 lines
Dr. Hannes Reinecke                Kernel Storage Architect
[email protected]                              +49 911 74053 688
SUSE Software Solutions GmbH, Maxfeldstr. 5, 90409 Nürnberg
HRB 36809 (AG Nürnberg), Geschäftsführer: Felix Imendörffer

_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme
#13 lore raw
Quoted 23 lines · Hannes Reinecke
Quoted 18 lines · On 4/23/21 3:38 PM, [email protected] wrote:

From: Martin Wilck <[email protected]>

We have observed a few crashes run_timer_softirq(), where a broken timer_list struct belonging to an anatt_timer was encountered. The broken structures look like this, and we see actually multiple ones attached to the same timer base:

crash> struct timer_list 0xffff92471bcfdc90 struct timer_list {
   entry = {
     next = 0xdead000000000122,  // LIST_POISON2 pprev = 0x0
   },
   expires = 4296022933,
   function = 0xffffffffc06de5e0 <nvme_anatt_timeout>, flags = 20
}

Quoted 4 lines

If such a timer is encountered in run_timer_softirq(), the kernel crashes. The test scenario was an I/O load test with lots of NVMe controllers, some of which were removed and re-added on the storage side.

Quoted 8 lines

...

But isn't this the result of detach_timer()? IE this suspiciously looks like perfectly normal operation; is you look at expire_timers() we're first calling 'detach_timer()' before calling the timer function, ie every crash in the timer function would have this signature. And, incidentally, so would any timer function which does not crash.

Sorry to kill your analysis ...

This doesn't mean that the patch isn't valid (in the sense that it resolve the issue), but we definitely will need to work on root cause analysis.

The process maybe:1.ana_work add the timer;2.error recovery occurs, in reconnecting, reinitialize the timer and call nvme_read_ana_log, nvme_read_ana_log may add the timer again.
The same timer is added twice, crash will happens later.

Indeed ana_log_buf has the similar bug, it's been encountered in our testing. To fix this bug, I also make the same patch and tested for more than 2 weeks.

This patch can fix the both bugs.

Quoted 4 lines

Cheera,

Hannes

_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme

#14 lore raw
Quoted 21 lines · Chao Leng
Quoted 18 lines · Hannes Reinecke
Quoted 6 lines · On 4/23/21 3:38 PM, [email protected] wrote:

If such a timer is encountered in run_timer_softirq(), the kernel crashes. The test scenario was an I/O load test with lots of NVMe controllers, some of which were removed and re-added on the storage side.

...

But isn't this the result of detach_timer()? IE this suspiciously looks like perfectly normal operation; is you look at expire_timers() we're first calling 'detach_timer()' before calling the timer function, ie every crash in the timer function would have this signature.
And, incidentally, so would any timer function which does not crash.

Sorry to kill your analysis ...

Quoted 7 lines
Quoted 3 lines

This doesn't mean that the patch isn't valid (in the sense that it resolve the issue), but we definitely will need to work on root cause analysis.

The process maybe:1.ana_work add the timer;2.error recovery occurs, in reconnecting, reinitialize the timer and call nvme_read_ana_log, nvme_read_ana_log may add the timer again.
The same timer is added twice, crash will happens later.

Thanks a lot! I think you nailed it.

I had another look at my dump with this in mind, and indeed I was able to prove that the timer of the same NVMe controller had been started (at least) twice. This explains why it looks as if it had been cleanly detached, and is still running nonetheless.

In particular, I saw this:

[ 4389.939928] nvme nvme36: queue_size 128 > ctrl sqsize 32, clamping down
[ 4389.962709] nvme nvme36: creating 4 I/O queues.
[ 4389.978462] nvme nvme36: starting error recovery
[ 4389.978509] nvme nvme36: failed to send request -32
[ 4389.978724] nvme nvme36: Failed to configure AEN (cfg 900)
[ 4389.978732] nvme nvme36: Successfully reconnected (1 attempt)
[ 4389.978739] nvme nvme36: failed to send request -32
[ 4389.991420] nvme nvme36: Reconnecting in 10 seconds...
...
[ 4441.399435] nvme nvme36: queue_size 128 > ctrl sqsize 32, clamping down
[ 4441.407647] nvme nvme36: creating 4 I/O queues.
[ 4441.445655] nvme nvme36: Successfully reconnected (1 attempt)
[ 4441.699665] nvme nvme36: starting error recovery
[ 4441.719493] nvme nvme36: Identify Descriptors failed (881)
[ 4441.735409] nvme nvme36: Reconnecting in 10 seconds...
...
[ 4510.891400] nvme nvme36: ANATT timeout, resetting controller.
...
[ 4517.035411] general protection fault: 0000 [#1] SMP PTI

The protection fault was caused by the anatt timer of another nvme controller. But I could see the anatt_timer of nvme36 still linked to an active timer_base, about to expire 44s in the future. Calculating back, that timer had been started around 4441s above, whereas the timer that expired a few seconds earlier must have been started at 4389.9.

Regards,
Martin

_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme

#15 lore raw
In reply to Hannes Reinecke
Quoted 28 lines · Hannes Reinecke
Quoted 25 lines · On 4/23/21 3:38 PM, [email protected] wrote:

From: Martin Wilck <[email protected]>

We have observed a few crashes run_timer_softirq(), where a broken timer_list struct belonging to an anatt_timer was encountered. The broken
structures look like this, and we see actually multiple ones attached to
the same timer base:

crash> struct timer_list 0xffff92471bcfdc90 struct timer_list {
   entry = {
     next = 0xdead000000000122,  // LIST_POISON2 pprev = 0x0
   },
   expires = 4296022933,
   function = 0xffffffffc06de5e0 <nvme_anatt_timeout>, flags = 20
}

If such a timer is encountered in run_timer_softirq(), the kernel crashes. The test scenario was an I/O load test with lots of NVMe controllers, some of which were removed and re-added on the storage side.

...

Quoted 7 lines

But isn't this the result of detach_timer()? IE this suspiciously looks like perfectly normal operation; is you look at expire_timers() we're first calling 'detach_timer()' before calling the timer function, ie every crash in the timer function would have this signature. And, incidentally, so would any timer function which does not crash.

Sorry to kill your analysis ...

No problem, I realized this myself, and actually mentioned it in the commit description. OTOH, this timer is only modified in very few places, and all but nvme_mpath_init() use the proper APIs for modifying or deleting timers, so the initialization of a (possibly still) running timer is the only suspect, afaics.

My personal theory is that the corruption might happen in several steps, the first step being timer_setup() mofiying fields of a pending timer. But I couldn't figure it out completely, and found it too hand- waving to mention in the commit description.

Quoted 3 lines

This doesn't mean that the patch isn't valid (in the sense that it resolve the issue), but we definitely will need to work on root cause analysis.

I'd be grateful for any help figuring out the missing bits.

Thanks,
Martin

_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme

#16 lore raw
Quoted 48 lines · Martin Wilck
Quoted 28 lines · Hannes Reinecke
Quoted 25 lines · On 4/23/21 3:38 PM, [email protected] wrote:

From: Martin Wilck <[email protected]>

We have observed a few crashes run_timer_softirq(), where a broken timer_list struct belonging to an anatt_timer was encountered. The broken
structures look like this, and we see actually multiple ones attached to
the same timer base:

crash> struct timer_list 0xffff92471bcfdc90 struct timer_list {
   entry = {
     next = 0xdead000000000122,  // LIST_POISON2 pprev = 0x0
   },
   expires = 4296022933,
   function = 0xffffffffc06de5e0 <nvme_anatt_timeout>, flags = 20
}

If such a timer is encountered in run_timer_softirq(), the kernel crashes. The test scenario was an I/O load test with lots of NVMe controllers, some of which were removed and re-added on the storage side.

...

Quoted 7 lines

But isn't this the result of detach_timer()? IE this suspiciously looks like perfectly normal operation; is you look at expire_timers() we're first calling 'detach_timer()' before calling the timer function, ie every crash in the timer function would have this signature. And, incidentally, so would any timer function which does not crash.

Sorry to kill your analysis ...

No problem, I realized this myself, and actually mentioned it in the commit description. OTOH, this timer is only modified in very few places, and all but nvme_mpath_init() use the proper APIs for modifying or deleting timers, so the initialization of a (possibly still) running timer is the only suspect, afaics.

My personal theory is that the corruption might happen in several steps, the first step being timer_setup() mofiying fields of a pending timer. But I couldn't figure it out completely, and found it too hand- waving to mention in the commit description.

Quoted 6 lines
Quoted 3 lines

This doesn't mean that the patch isn't valid (in the sense that it resolve the issue), but we definitely will need to work on root cause analysis.

I'd be grateful for any help figuring out the missing bits.

No, the problem lies in the pecularities of NVMe error recovery.

The crucial point here is that during error recovery all I/O is aborted, but after that the queue is _NOT_ stopped.
So it's perfectly legal to send I/O even if the controller is not fully LIVE. And if _this_ I/O fails then we'll find ourselves in lots of interesting situations.

I guess the problem lies with nvme_init_identify().

We're calling nvme_mpath_init() (which will start the ANATT timer), but issue some more I/O afterwards.
If any of _that_ I/O fails, an error is registered with the caller (eg in nvme_tcp_configure_admin_queue(), _but_ the ANATT timer continues to run. Then we'll be calling nvme_tcp_reconnect_or_remove() and delete the controller. Completely oblivious to the running ANATT timer. Which will get quite unhappy once it triggers as the controller structure from which the timer was started is done for.

So with this analysis an alternative fix would be to move nvme_path_init() right at the end of nvme_init_identify(), such that we'll never get into the situation where nvme_init_identify() fails yet the ANATT timer is running.

Cheers,

Hannes

Signature · 9 lines
Dr. Hannes Reinecke		        Kernel Storage Architect
[email protected]			               +49 911 74053 688
SUSE Software Solutions Germany GmbH, 90409 Nürnberg
GF: F. Imendörffer, HRB 36809 (AG Nürnberg)

_______________________________________________
Linux-nvme mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-nvme