Skip to contentSkip to conversation

cciss: Ignore stale commands after reboot

PATCH In review Hannes Reinecke 3 participants 9 messages Jul 2, 2009 – Jul 7, 2009 Block layer

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 cciss: Ignore stale commands after reboot

When doing an unexpected shutdown like kexec the cciss firmware might still have some commands in flight, which it is trying to complete.
The driver is doing it's best on resetting the HBA, but sadly there's a firmware issue causing the firmware _not_ to abort or drop old commands.
So the firmware will send us commands which we haven't accounted for, causing the driver to panic.

With this patch we're just ignoring these commands as there is nothing we could be doing with them anyway.

---
 drivers/block/cciss.c     |   14 ++++++++++++--
 drivers/block/cciss_cmd.h |    1 +
 2 files changed, 13 insertions(+), 2 deletions(-)
drivers/block/cciss.c +12 -2
index c7a527c..8dd4c0d 100644
@@ -226,7 +226,16 @@ static inline void addQ(struct hlist_head *list, CommandList_struct *c)
 
 static inline void removeQ(CommandList_struct *c)
 {
-	if (WARN_ON(hlist_unhashed(&c->list)))
+	/*
+	 * After kexec/dump some commands might still
+	 * be in flight, which the firmware will try
+	 * to complete. Resetting the firmware doesn't work
+	 * with old fw revisions, so we have to mark
+	 * them off as 'stale' to prevent the driver from
+	 * falling over.
+	 */
+	if (unlikely(hlist_unhashed(&c->list))) {
+		c->cmd_type = CMD_MSG_STALE;
 		return;
 
 	hlist_del_init(&c->list);
@@ -4246,7 +4255,8 @@ static void fail_all_cmds(unsigned long ctlr)
 	while (!hlist_empty(&h->cmpQ)) {
 		c = hlist_entry(h->cmpQ.first, CommandList_struct, list);
 		removeQ(c);
-		c->err_info->CommandStatus = CMD_HARDWARE_ERR;
+		if (c->cmd_type != CMD_MSG_STALE)
+			c->err_info->CommandStatus = CMD_HARDWARE_ERR;
 		if (c->cmd_type == CMD_RWREQ) {
 			complete_command(h, c, 0);
 		} else if (c->cmd_type == CMD_IOCTL_PEND)
drivers/block/cciss_cmd.h +1 -0
index cd665b0..dbaed1e 100644
@@ -274,6 +274,7 @@ typedef struct _ErrorInfo_struct {
 #define CMD_SCSI	0x03
 #define CMD_MSG_DONE	0x04
 #define CMD_MSG_TIMEOUT 0x05
+#define CMD_MSG_STALE	0xff
 
 /* This structure needs to be divisible by 8 for new
  * indexing method.
Signature · 1 line
1.5.3.2
#1 lore raw
Quoted 31 lines · Hannes Reinecke

When doing an unexpected shutdown like kexec the cciss firmware might still have some commands in flight, which it is trying to complete.
The driver is doing it's best on resetting the HBA, but sadly there's a firmware issue causing the firmware _not_ to abort or drop old commands.
So the firmware will send us commands which we haven't accounted for, causing the driver to panic.

With this patch we're just ignoring these commands as there is nothing we could be doing with them anyway.

---
 drivers/block/cciss.c     |   14 ++++++++++++--
 drivers/block/cciss_cmd.h |    1 +
 2 files changed, 13 insertions(+), 2 deletions(-)
drivers/block/cciss.c +3 -1
index c7a527c..8dd4c0d 100644
@@ -226,7 +226,16 @@ static inline void addQ(struct hlist_head *list, CommandList_struct *c)
 
 static inline void removeQ(CommandList_struct *c)
 {
-	if (WARN_ON(hlist_unhashed(&c->list)))
+	/*
+	 * After kexec/dump some commands might still
+	 * be in flight, which the firmware will try

drivers/block/cciss.c static inline void addQ(struct hlist_head *list, CommandList_struct *c) · line 241

Quoted 10 lines
drivers/block/cciss.c +7 -0
+	 * to complete. Resetting the firmware doesn't work
+	 * with old fw revisions, so we have to mark
+	 * them off as 'stale' to prevent the driver from
+	 * falling over.
+	 */
+	if (unlikely(hlist_unhashed(&c->list))) {
+		c->cmd_type = CMD_MSG_STALE;
 		return;
 
      hlist_del_init(&c->list);

Ehm, that looks rather dangerous. What's the level of testing this patch received?

Signature · 1 line
Jens Axboe
#2 lore raw
Quoted 40 lines · Jens Axboe
Quoted 39 lines · Hannes Reinecke

When doing an unexpected shutdown like kexec the cciss firmware might still have some commands in flight, which it is trying to complete.
The driver is doing it's best on resetting the HBA, but sadly there's a firmware issue causing the firmware _not_ to abort or drop old commands.
So the firmware will send us commands which we haven't accounted for, causing the driver to panic.

With this patch we're just ignoring these commands as there is nothing we could be doing with them anyway.

---
 drivers/block/cciss.c     |   14 ++++++++++++--
 drivers/block/cciss_cmd.h |    1 +
 2 files changed, 13 insertions(+), 2 deletions(-)
drivers/block/cciss.c +10 -1
index c7a527c..8dd4c0d 100644
@@ -226,7 +226,16 @@ static inline void addQ(struct hlist_head *list, CommandList_struct *c)
 
 static inline void removeQ(CommandList_struct *c)
 {
-	if (WARN_ON(hlist_unhashed(&c->list)))
+	/*
+	 * After kexec/dump some commands might still
+	 * be in flight, which the firmware will try
+	 * to complete. Resetting the firmware doesn't work
+	 * with old fw revisions, so we have to mark
+	 * them off as 'stale' to prevent the driver from
+	 * falling over.
+	 */
+	if (unlikely(hlist_unhashed(&c->list))) {
+		c->cmd_type = CMD_MSG_STALE;
 		return;
 
Quoted 5 lines

drivers/block/cciss.c static inline void addQ(struct hlist_head *list, CommandList_struct *c) · line 241

Quoted 1 line
drivers/block/cciss.c +0 -0
      hlist_del_init(&c->list);

Ehm, that looks rather dangerous. What's the level of testing this patch received?

Where is the danger here?

With the original code we would be issuing a warning and return.
But then we hit this codepath:

while (!hlist_empty(&h->cmpQ)) {
c = hlist_entry(h->cmpQ.first, CommandList_struct, list);

 		removeQ(c);
		c->err_info->CommandStatus = CMD_HARDWARE_ERR;

and the driver goes boom as c->err_info is not initialized.

This frequently happens if you're trying to do a kdump while the system is doing I/O.
If you object to the removed WARN() I can easily put this in, but without the fix there is a good chance that kdump fails on cciss machines.

And note we can't do anything with the stale commands anyway, as the context having sent the commands originally is long gone.

Cheers,

Hannes

Signature · 4 lines
Dr. Hannes Reinecke		      zSeries & Storage
[email protected]			      +49 911 74053 688
SUSE LINUX Products GmbH, Maxfeldstr. 5, 90409 Nürnberg
GF: Markus Rex, HRB 16746 (AG Nürnberg)
#3 lore raw
Quoted 41 lines · Hannes Reinecke
Quoted 40 lines · Jens Axboe
Quoted 39 lines · Hannes Reinecke

When doing an unexpected shutdown like kexec the cciss firmware might still have some commands in flight, which it is trying to complete.
The driver is doing it's best on resetting the HBA, but sadly there's a firmware issue causing the firmware _not_ to abort or drop old commands.
So the firmware will send us commands which we haven't accounted for, causing the driver to panic.

With this patch we're just ignoring these commands as there is nothing we could be doing with them anyway.

---
 drivers/block/cciss.c     |   14 ++++++++++++--
 drivers/block/cciss_cmd.h |    1 +
 2 files changed, 13 insertions(+), 2 deletions(-)
drivers/block/cciss.c +10 -1
index c7a527c..8dd4c0d 100644
@@ -226,7 +226,16 @@ static inline void addQ(struct hlist_head *list, CommandList_struct *c)
 
 static inline void removeQ(CommandList_struct *c)
 {
-	if (WARN_ON(hlist_unhashed(&c->list)))
+	/*
+	 * After kexec/dump some commands might still
+	 * be in flight, which the firmware will try
+	 * to complete. Resetting the firmware doesn't work
+	 * with old fw revisions, so we have to mark
+	 * them off as 'stale' to prevent the driver from
+	 * falling over.
+	 */
+	if (unlikely(hlist_unhashed(&c->list))) {
+		c->cmd_type = CMD_MSG_STALE;
 		return;
 
Quoted 6 lines
Quoted 5 lines

drivers/block/cciss.c static inline void addQ(struct hlist_head *list, CommandList_struct *c) · line 241

Quoted 1 line
drivers/block/cciss.c +0 -0
      hlist_del_init(&c->list);

Ehm, that looks rather dangerous. What's the level of testing this patch received?

Where is the danger here?

The danger is that the patch doesn't even compile :-)
At least it had the { at the end of the if, otherwise it would have been insta-hang.

Quoted 29 lines

With the original code we would be issuing a warning and return.
But then we hit this codepath:

drivers/block/cciss.c +0 -0
 	while (!hlist_empty(&h->cmpQ)) {
 		c = hlist_entry(h->cmpQ.first, CommandList_struct, list);
 		removeQ(c);
		c->err_info->CommandStatus = CMD_HARDWARE_ERR;

and the driver goes boom as c->err_info is not initialized.

This frequently happens if you're trying to do a kdump while the system is doing I/O.
If you object to the removed WARN() I can easily put this in, but without the fix there is a good chance that kdump fails on cciss machines.

And note we can't do anything with the stale commands anyway, as the context having sent the commands originally is long gone.

Cheers,

Hannes
--
Dr. Hannes Reinecke zSeries & Storage
[email protected] +49 911 74053 688
SUSE LINUX Products GmbH, Maxfeldstr. 5, 90409 Nürnberg
GF: Markus Rex, HRB 16746 (AG Nürnberg)

Signature · 1 line
Jens Axboe
#4 lore raw
Quoted 46 lines · Jens Axboe
Quoted 45 lines · Hannes Reinecke
Quoted 44 lines · Jens Axboe
Quoted 30 lines · Hannes Reinecke

When doing an unexpected shutdown like kexec the cciss firmware might still have some commands in flight, which it is trying to complete.
The driver is doing it's best on resetting the HBA, but sadly there's a firmware issue causing the firmware _not_ to abort or drop old commands.
So the firmware will send us commands which we haven't accounted for, causing the driver to panic.

With this patch we're just ignoring these commands as there is nothing we could be doing with them anyway.

---
 drivers/block/cciss.c     |   14 ++++++++++++--
 drivers/block/cciss_cmd.h |    1 +
 2 files changed, 13 insertions(+), 2 deletions(-)
drivers/block/cciss.c +3 -1
index c7a527c..8dd4c0d 100644
@@ -226,7 +226,16 @@ static inline void addQ(struct hlist_head *list, CommandList_struct *c)
 
 static inline void removeQ(CommandList_struct *c)
 {
-	if (WARN_ON(hlist_unhashed(&c->list)))
+	/*
+	 * After kexec/dump some commands might still
+	 * be in flight, which the firmware will try

drivers/block/cciss.c static inline void addQ(struct hlist_head *list, CommandList_struct *c) · line 241

Quoted 10 lines
drivers/block/cciss.c +7 -0
+	 * to complete. Resetting the firmware doesn't work
+	 * with old fw revisions, so we have to mark
+	 * them off as 'stale' to prevent the driver from
+	 * falling over.
+	 */
+	if (unlikely(hlist_unhashed(&c->list))) {
+		c->cmd_type = CMD_MSG_STALE;
 		return;
 
      hlist_del_init(&c->list);

Ehm, that looks rather dangerous. What's the level of testing this patch received?

Quoted 6 lines
Quoted 1 line

Where is the danger here?

The danger is that the patch doesn't even compile :-)
At least it had the { at the end of the if, otherwise it would have been insta-hang.

Bah. Should've said so.
That's what you get when you cut-n-paste from a different version.

OK, resending.

Cheers,

Hannes

Signature · 4 lines
Dr. Hannes Reinecke		      zSeries & Storage
[email protected]			      +49 911 74053 688
SUSE LINUX Products GmbH, Maxfeldstr. 5, 90409 Nürnberg
GF: Markus Rex, HRB 16746 (AG Nürnberg)
#5 lore raw
Quoted 47 lines · Hannes Reinecke
Quoted 46 lines · Jens Axboe
Quoted 45 lines · Hannes Reinecke
Quoted 44 lines · Jens Axboe
Quoted 30 lines · Hannes Reinecke

When doing an unexpected shutdown like kexec the cciss firmware might still have some commands in flight, which it is trying to complete.
The driver is doing it's best on resetting the HBA, but sadly there's a firmware issue causing the firmware _not_ to abort or drop old commands.
So the firmware will send us commands which we haven't accounted for, causing the driver to panic.

With this patch we're just ignoring these commands as there is nothing we could be doing with them anyway.

---
 drivers/block/cciss.c     |   14 ++++++++++++--
 drivers/block/cciss_cmd.h |    1 +
 2 files changed, 13 insertions(+), 2 deletions(-)
drivers/block/cciss.c +3 -1
index c7a527c..8dd4c0d 100644
@@ -226,7 +226,16 @@ static inline void addQ(struct hlist_head *list, CommandList_struct *c)
 
 static inline void removeQ(CommandList_struct *c)
 {
-	if (WARN_ON(hlist_unhashed(&c->list)))
+	/*
+	 * After kexec/dump some commands might still
+	 * be in flight, which the firmware will try

drivers/block/cciss.c static inline void addQ(struct hlist_head *list, CommandList_struct *c) · line 241

Quoted 10 lines
drivers/block/cciss.c +7 -0
+	 * to complete. Resetting the firmware doesn't work
+	 * with old fw revisions, so we have to mark
+	 * them off as 'stale' to prevent the driver from
+	 * falling over.
+	 */
+	if (unlikely(hlist_unhashed(&c->list))) {
+		c->cmd_type = CMD_MSG_STALE;
 		return;
 
      hlist_del_init(&c->list);

Ehm, that looks rather dangerous. What's the level of testing this patch received?

Quoted 7 lines
Quoted 6 lines
Quoted 1 line

Where is the danger here?

The danger is that the patch doesn't even compile :-)
At least it had the { at the end of the if, otherwise it would have been insta-hang.

Bah. Should've said so.

Sorry, just annoys me when people send out patches for inclusion that don't even compile. It usually means that some other form of the patch was tested and that this one hasn't even been run (obviously, since it doesn't compile).

Signature · 1 line
Jens Axboe
#6 lore raw

Jens Axboe wrote:
[ ... ]

Quoted 6 lines

Sorry, just annoys me when people send out patches for inclusion that don't even compile. It usually means that some other form of the patch was tested and that this one hasn't even been run (obviously, since it doesn't compile).

Accepted.

Apologies.

Cheers,

Hannes

Signature · 4 lines
Dr. Hannes Reinecke		      zSeries & Storage
[email protected]			      +49 911 74053 688
SUSE LINUX Products GmbH, Maxfeldstr. 5, 90409 Nürnberg
GF: Markus Rex, HRB 16746 (AG Nürnberg)
#7 lore raw
In reply to Hannes Reinecke
Quoted 9 lines · Hannes Reinecke

When doing an unexpected shutdown like kexec the cciss firmware might still have some commands in flight, which it is trying to complete.
The driver is doing it's best on resetting the HBA, but sadly there's a firmware issue causing the firmware _not_ to abort or drop old commands.
So the firmware will send us commands which we haven't accounted for, causing the driver to panic.

Quoted 4 lines

With this patch we're just ignoring these commands as there is nothing we could be doing with them anyway.

Pardon my ignorance here, but don't you have a bigger problem: if the reset is not dropping or aborting old commands, doesn't this also mean that these old commands can still be _executing_? In which case any (old) reads being executed could be scribbling over memory? (Memory that may be being used for other purposes?)

Alan D. Brunelle

#8 lore raw

Hi Alan,

Quoted 13 lines · Alan D. Brunelle
Quoted 12 lines · Hannes Reinecke

When doing an unexpected shutdown like kexec the cciss firmware might still have some commands in flight, which it is trying to complete.
The driver is doing it's best on resetting the HBA, but sadly there's a firmware issue causing the firmware _not_ to abort or drop old commands.
So the firmware will send us commands which we haven't accounted for, causing the driver to panic.

With this patch we're just ignoring these commands as there is nothing we could be doing with them anyway.

Quoted 8 lines
Quoted 1 line

Pardon my ignorance here, but don't you have a bigger problem: if the reset is not dropping or aborting old commands, doesn't this also mean that these old commands can still be _executing_? In which case any (old) reads being executed could be scribbling over memory? (Memory that may be being used for other purposes?)

Yes and no.

This scenario is being observed whilst doing a kexec/kdump reboot. IE a new kernel is started directly from the context of an already running kernel, so there is a fair chance that IO is still in flight.
In flight here means the kernel/driver has send the commands to the firmware but not yet received a reply/completion to them.

So the kdump kernel boots and initializes the driver. The driver itself tries to initializes the firmware, but due to the abovementioned bug this initialization does _not_ clear out old commands, so when the driver is up and running is receives command completions.
But these completions are not associated with any commands the driver has been sent, so we can as well drop them to the floor. Which is what this patch is all about.

So yes, there is some sort of overwrite in the sense the 'old' IO is being committed to disk by the time the new kernel starts. But no, it doesn't really matter to us as we're starting out with any operations only _after_ we have received these stale IO.

HTH.

Cheers,

Hannes

Signature · 4 lines
Dr. Hannes Reinecke		      zSeries & Storage
[email protected]			      +49 911 74053 688
SUSE LINUX Products GmbH, Maxfeldstr. 5, 90409 Nürnberg
GF: Markus Rex, HRB 16746 (AG Nürnberg)