Skip to contentSkip to conversation

kexec reboot broken with ioatdma?

DISCUSSION Roland Dreier 3 participants 13 messages Dec 16, 2009 – Dec 19, 2009

Subsystems (3)
lore
Share & tools
lore
#0 lore raw

I have a system with IOAT hardware, and rebooting with kexec fails with the latest 2.6.32-git kernel. I haven't really tried earlier kernels, but I suspect the issue comes from the ioatdma driver being autoloaded now.

The reboot gets stuck at:

ioatdma 0000:00:16.0: Self-test copy timed out, disabling ioatdma 0000:00:16.0: Freeing 2 in use descriptors! ioatdma 0000:00:16.0: Intel(R) I/OAT DMA Engine init failed

so presumably the IOAT hardware is left in a bad state that the ioatdma driver in the kexec'ed new kernel can't handle.

I notice that long ago, there was a commit 428ed602 ("I/OAT: fix I/OAT for kexec") that added a shutdown method to clean things up so kexec worked, and then more recently there was 4fac7fa5 ("ioat: do not perform removal actions at shutdown") that got rid of the shutdown hook.

I'm not sure what the correct fix is here: fix the shutdown order so everyone drops all references to IOAT stuff before IOAT is shutdown, or add some code to the ioatdma driver so it resets the hardware on startup so the new kernel can deal with an unspecified state.

This is on a system with the following hardware:

00:16.0 System peripheral [0880]: Intel Corporation 5520/5500/X58 Chipset QuickData Technology Devic
e [8086:3430] (rev 20)
00:16.1 System peripheral [0880]: Intel Corporation 5520/5500/X58 Chipset QuickData Technology Devic
e [8086:3431] (rev 20)
00:16.2 System peripheral [0880]: Intel Corporation 5520/5500/X58 Chipset QuickData Technology Devic
e [8086:3432] (rev 20)
00:16.3 System peripheral [0880]: Intel Corporation 5520/5500/X58 Chipset QuickData Technology Devic
e [8086:3433] (rev 20)
00:16.4 System peripheral [0880]: Intel Corporation 5520/5500/X58 Chipset QuickData Technology Devic
e [8086:3429] (rev 20)
00:16.5 System peripheral [0880]: Intel Corporation 5520/5500/X58 Chipset QuickData Technology Devic
e [8086:342a] (rev 20)
00:16.6 System peripheral [0880]: Intel Corporation 5520/5500/X58 Chipset QuickData Technology Device [8086:342b] (rev 20) 00:16.7 System peripheral [0880]: Intel Corporation 5520/5500/X58 Chipset QuickData Technology Device [8086:342c] (rev 20) 80:16.0 System peripheral [0880]: Intel Corporation 5520/5500/X58 Chipset QuickData Technology Device [8086:3430] (rev 20) 80:16.1 System peripheral [0880]: Intel Corporation 5520/5500/X58 Chipset QuickData Technology Device [8086:3431] (rev 20) 80:16.2 System peripheral [0880]: Intel Corporation 5520/5500/X58 Chipset QuickData Technology Device [8086:3432] (rev 20) 80:16.3 System peripheral [0880]: Intel Corporation 5520/5500/X58 Chipset QuickData Technology Device [8086:3433] (rev 20) 80:16.4 System peripheral [0880]: Intel Corporation 5520/5500/X58 Chipset QuickData Technology Device [8086:3429] (rev 20) 80:16.5 System peripheral [0880]: Intel Corporation 5520/5500/X58 Chipset QuickData Technology Device [8086:342a] (rev 20) 80:16.6 System peripheral [0880]: Intel Corporation 5520/5500/X58 Chipset QuickData Technology Device [8086:342b] (rev 20) 80:16.7 System peripheral [0880]: Intel Corporation 5520/5500/X58 Chipset QuickData Technology Device [8086:342c] (rev 20)

Thanks,
Roland

#1 lore raw
Quoted 18 lines · Roland Dreier

I have a system with IOAT hardware, and rebooting with kexec fails with the latest 2.6.32-git kernel. I haven't really tried earlier kernels, but I suspect the issue comes from the ioatdma driver being autoloaded now.

The reboot gets stuck at:

ioatdma 0000:00:16.0: Self-test copy timed out, disabling ioatdma 0000:00:16.0: Freeing 2 in use descriptors! ioatdma 0000:00:16.0: Intel(R) I/OAT DMA Engine init failed

so presumably the IOAT hardware is left in a bad state that the ioatdma driver in the kexec'ed new kernel can't handle.

I notice that long ago, there was a commit 428ed602 ("I/OAT: fix I/OAT for kexec") that added a shutdown method to clean things up so kexec worked, and then more recently there was 4fac7fa5 ("ioat: do not perform removal actions at shutdown") that got rid of the shutdown hook.

Quoted 4 lines

I'm not sure what the correct fix is here: fix the shutdown order so everyone drops all references to IOAT stuff before IOAT is shutdown, or add some code to the ioatdma driver so it resets the hardware on startup so the new kernel can deal with an unspecified state.

Hi Roland,

from a kexec point of view I believe that the preferred option is the former - shutdown the device so it can be initialised using standard paths in the second kernel.

_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec

#2 lore raw
Quoted 4 lines

from a kexec point of view I believe that the preferred option is the former - shutdown the device so it can be initialised using standard paths in the second kernel.

OK... however I'm not suggesting a separate kexec initialization path, simply adding a reset of the device in the standard initialization. This would be fairly normal for other types of device; for example, the BIOS may have left a NIC in an undefined state due to network boot. Of course BIOS is unlikely to use an IOAT DMA engine but the principle of limiting assumptions about platform state still stands I think.

From a quick look, it seems tricky to get a clean shutdown of IOAT stuff since there doesn't seem to be a clean ordering that makes sure the ioatdma stuff is shutdown after everything using it.

- R.

_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec

#3 lore raw
Quoted 10 lines · Roland Dreier
Quoted 3 lines

from a kexec point of view I believe that the preferred option is the former - shutdown the device so it can be initialised using standard paths in the second kernel.

OK... however I'm not suggesting a separate kexec initialization path, simply adding a reset of the device in the standard initialization. This would be fairly normal for other types of device; for example, the BIOS may have left a NIC in an undefined state due to network boot. Of course BIOS is unlikely to use an IOAT DMA engine but the principle of limiting assumptions about platform state still stands I think.

I agree that is more robust if the init path copes with hardware arriving in an unknown state. I'll look into adding a channel reset in the init path (something that should probably have been there since the beginning).

Quoted 3 lines

From a quick look, it seems tricky to get a clean shutdown of IOAT stuff since there doesn't seem to be a clean ordering that makes sure the ioatdma stuff is shutdown after everything using it.

The engines may be in use by multiple subsytems (net, raid) so coordinating shutdown ordering would indeed be a pain.

--
Dan

#4 lore raw
Quoted 5 lines

I agree that is more robust if the init path copes with hardware arriving in an unknown state. I'll look into adding a channel reset in the init path (something that should probably have been there since the beginning).

Great, let me know and I'll try it on my system with kexec.

Thanks,
Roland

#5 lore raw

Subject: ioat2,3: put channel hardware in known state at init

Put the ioat2 and ioat3 state machines in the halted state with all errors cleared.

The ioat1 init path is not disturbed for stability, there are no reported ioat1 initialization issues.

---

Quoted 7 lines · Roland Dreier
Quoted 4 lines

I agree that is more robust if the init path copes with hardware arriving in an unknown state. I'll look into adding a channel reset in the init path (something that should probably have been there since the beginning).

Great, let me know and I'll try it on my system with kexec.

I was able to recover stuck channels with this patch, let me know if it resolves the kexec issue.

Thanks,
Dan

 drivers/dma/ioat/dma.c       |    2 +
 drivers/dma/ioat/dma.h       |   18 +++++++++++
 drivers/dma/ioat/dma_v2.c    |   69 ++++++++++++++++++++++++++++++++----------
 drivers/dma/ioat/dma_v2.h    |    2 +
 drivers/dma/ioat/dma_v3.c    |   54 ++++++++++++++++++++++++---------
 drivers/dma/ioat/registers.h |    1 +
 6 files changed, 114 insertions(+), 32 deletions(-)
drivers/dma/ioat/dma.c +1 -1
index c524d36..dcc4ab7 100644
@@ -1032,7 +1032,7 @@ int __devinit ioat_probe(struct ioatdma_device *device)
 	dma->dev = &pdev->dev;
 
 	if (!dma->chancnt) {
-		dev_err(dev, "zero channels detected\n");
+		dev_err(dev, "channel enumeration error\n");
 		goto err_setup_interrupts;
 	}
 
drivers/dma/ioat/dma.h +18 -0
index 45edde9..bbc3e78 100644
@@ -60,6 +60,7 @@
  * @dca: direct cache access context
  * @intr_quirk: interrupt setup quirk (for ioat_v1 devices)
  * @enumerate_channels: hw version specific channel enumeration
+ * @reset_hw: hw version specific channel (re)initialization
  * @cleanup_tasklet: select between the v2 and v3 cleanup routines
  * @timer_fn: select between the v2 and v3 timer watchdog routines
  * @self_test: hardware version specific self test for each supported op type
@@ -78,6 +79,7 @@ struct ioatdma_device {
 	struct dca_provider *dca;
 	void (*intr_quirk)(struct ioatdma_device *device);
 	int (*enumerate_channels)(struct ioatdma_device *device);
+	int (*reset_hw)(struct ioat_chan_common *chan);
 	void (*cleanup_tasklet)(unsigned long data);
 	void (*timer_fn)(unsigned long data);
 	int (*self_test)(struct ioatdma_device *device);
@@ -264,6 +266,22 @@ static inline void ioat_suspend(struct ioat_chan_common *chan)
 	writeb(IOAT_CHANCMD_SUSPEND, chan->reg_base + IOAT_CHANCMD_OFFSET(ver));
 }
 
+static inline void ioat_reset(struct ioat_chan_common *chan)
+{
+	u8 ver = chan->device->version;
+
+	writeb(IOAT_CHANCMD_RESET, chan->reg_base + IOAT_CHANCMD_OFFSET(ver));
+}
+
+static inline bool ioat_reset_pending(struct ioat_chan_common *chan)
+{
+	u8 ver = chan->device->version;
+	u8 cmd;
+
+	cmd = readb(chan->reg_base + IOAT_CHANCMD_OFFSET(ver));
+	return (cmd & IOAT_CHANCMD_RESET) == IOAT_CHANCMD_RESET;
+}
+
 static inline void ioat_set_chainaddr(struct ioat_dma_chan *ioat, u64 addr)
 {
 	struct ioat_chan_common *chan = &ioat->base;
drivers/dma/ioat/dma_v2.c +52 -17

122 lines not on this page · raw message

drivers/dma/ioat/dma_v2.h +2 -0
index 1d849ef..3afad8d 100644
@@ -185,6 +185,8 @@ bool reshape_ring(struct ioat2_dma_chan *ioat, int order);
 void __ioat2_issue_pending(struct ioat2_dma_chan *ioat);
 void ioat2_cleanup_tasklet(unsigned long data);
 void ioat2_timer_event(unsigned long data);
+int ioat2_quiesce(struct ioat_chan_common *chan, unsigned long tmo);
+int ioat2_reset_sync(struct ioat_chan_common *chan, unsigned long tmo);
 extern struct kobj_type ioat2_ktype;
 extern struct kmem_cache *ioat2_cache;
 #endif /* IOATDMA_V2_H */
drivers/dma/ioat/dma_v3.c +40 -14

75 lines not on this page · raw message

drivers/dma/ioat/registers.h +1 -0
index f015ec1..e8ae63b 100644
@@ -27,6 +27,7 @@
 
 #define IOAT_PCI_DEVICE_ID_OFFSET		0x02
 #define IOAT_PCI_DMAUNCERRSTS_OFFSET		0x148
+#define IOAT_PCI_CHANERR_INT_OFFSET		0x180
 #define IOAT_PCI_CHANERRMASK_INT_OFFSET		0x184
 
 /* MMIO Device Registers */
author Tested-by
#6 lore raw
Quoted 5 lines

Subject: ioat2,3: put channel hardware in known state at init

Put the ioat2 and ioat3 state machines in the halted state with all errors cleared.

Yes, with this applied I was able to kexec reboot from a kernel with ioatdma loaded and have ioatdma work in the new kernel. So:

Thanks!

#7 lore raw
Quoted 9 lines · Roland Dreier
Quoted 4 lines

Subject: ioat2,3: put channel hardware in known state at init

Put the ioat2 and ioat3 state machines in the halted state with all errors cleared.

Yes, with this applied I was able to kexec reboot from a kernel with ioatdma loaded and have ioatdma work in the new kernel. So:

What was that, all of 8 minutes to turn that test around! :-) Nice.

Thanks,
Dan

#8 lore raw
Quoted 2 lines

What was that, all of 8 minutes to turn that test around! :-) Nice.

Well, a kexec reboot takes < 30 seconds, and this is a *very* fast box, so rebuilding the kernel is another 30 seconds. So I was really just slacking off for about 7 of those minutes :)

- R.

Acked-by
#9 lore raw
In reply to Roland Dreier
Quoted 10 lines · Roland Dreier
Quoted 5 lines

Subject: ioat2,3: put channel hardware in known state at init

Put the ioat2 and ioat3 state machines in the halted state with all errors cleared.

Yes, with this applied I was able to kexec reboot from a kernel with ioatdma loaded and have ioatdma work in the new kernel. So:

This seems fine to me from a kexec point of view

_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec

#10 lore raw
In reply to Roland Dreier
Quoted 11 lines · Roland Dreier
Quoted 4 lines

from a kexec point of view I believe that the preferred option is the former - shutdown the device so it can be initialised using standard paths in the second kernel.

OK... however I'm not suggesting a separate kexec initialization path, simply adding a reset of the device in the standard initialization. This would be fairly normal for other types of device; for example, the BIOS may have left a NIC in an undefined state due to network boot. Of course BIOS is unlikely to use an IOAT DMA engine but the principle of limiting assumptions about platform state still stands I think.

If its reset in a standard path, then yes that seems sensible to me. I'm mainly concerned about avoiding something along the lines of:

	if (is_booting_from_kexec)
		reset_hw();
Quoted 12 lines
Quoted 1 line

From a quick look, it seems tricky to get a clean shutdown of IOAT stuff

since there doesn't seem to be a clean ordering that makes sure the ioatdma stuff is shutdown after everything using it.

- R.

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

#11 lore raw
Quoted 6 lines

If its reset in a standard path, then yes that seems sensible to me. I'm mainly concerned about avoiding something along the lines of:

	if (is_booting_from_kexec)
		reset_hw();

I think we all agree... the idea is to change from

	init_hw()
	{
		/* assume hw is in pristine state */
		do_init();
	}

to

	init_hw()
	{
		/* hw is in unknown state */
		reset_hw();
		do_init();
	}

- R.

#12 lore raw
Quoted 17 lines · Roland Dreier
Quoted 6 lines

If its reset in a standard path, then yes that seems sensible to me. I'm mainly concerned about avoiding something along the lines of:

	if (is_booting_from_kexec)
		reset_hw();

I think we all agree... the idea is to change from

	init_hw()
	{
		/* assume hw is in pristine state */
		do_init();
	}

to

Quoted 8 lines
	init_hw()
	{
		/* hw is in unknown state */
		reset_hw();
		do_init();
	}

- R.

Yes, I think we all agree :-)

_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec