machveg=dig on hpzx1 platforms

Hello,

on most HP machines, the 'machveg=dig' parameter is needed for the kdump kernel to avoid problems with the HP ioc. In the original kdump patches, a function ioc_iova_disable() was used, which (as I understand) disabled the IOC before executing the new kernel. However, that function wasn't added to mainline [2].

However, with a 2.6.22-rc kernel, booting into the kdump kernel works surprisingly. I haven't found a fix in the git history that could be responsible. Was there a fix? Or is it just a random result on a machine I tested that it works?

Are there any plans to bring back ioc_iova_disable(), or would some automatic kernel code that uses machvec=dig in the kernel without forcing the user to pass an additional command line work?

Thanks,
Bernhard

[1] http://article.gmane.org/gmane.linux.ports.ia64/15192
[2] http://article.gmane.org/gmane.linux.ports.ia64/15147

ioc_iova_disable() was used during the initial development of kexec. In-flight DMA from improperly shutdown drivers would land on the booting kexec kernel and cause all sorts of hard to find problems. An MCA was actually preferred, to help identify errant drivers.

Some of kdump was based on this initial work and included ioc_iova_disable(). This function was later dropped - a kdump kernel boots in it's own memory space and shouldn't get incoming DMA dropped on it. The problem now is that during the boot of the kdump kernel, the ioc is re-initialized, so in-flight DMA causes an ioc IOTLB miss which leads to an MCA. With kdump, the idea is to get to the new kernel with as little code as we can, so shutting down drivers properly is not an option.

Two possible solutions are:
Fix the ioc to handle this situation, maybe by not clearing the IOTLB, and by reserving a few entries for the kdump kernel. I personally don't understand the IOC well enough to implement this.
Or
Use machvec=dig and avoid using the ioc altogether for the kdump kernel, (leaving the IOTLB intact). This is something of a hack, but it works.

Because this ioc is HP specific hardware, the machvec=dig hack is only needed for HP platforms.

In our testing, a lot of DMA was needed to cause this problem to show up. We would usually load the system with a lot of filesystem and network activity, then use a debug module to cause the kernel to oops within an interrupt handler. On a quiet system, kdump would usually work.

Here is a patch to do that. We use this internally, but I had forgotten to post it.

-T

---

@@ -42,6 +42,9 @@ #include <asm/dma.h>
  #include <asm/system.h>		/* wmb() */
 
  #include <asm/acpi-ext.h>
+#include <linux/crash_dump.h>
+
+extern int swiotlb_late_init_with_default_size (size_t size);
 
  #define PFX "IOC: "
 

@@ -2026,11 +2029,24 @@ sba_init(void)
  	if (!ia64_platform_is("hpzx1") && !ia64_platform_is("hpzx1_swiotlb"))
  		return 0;
 
+#ifdef CONFIG_CRASH_DUMP
+	/* If we are booting a kdump kernel, the sba_iommu will
+	 * cause devices that were not shutdown properly to MCA
+	 * as soon as they are turned back on.  Our only option for
+	 * a successful kdump kernel boot is to use the swiotlb.
+	 */
+	if (elfcorehdr_addr < ELFCORE_ADDR_MAX) {
+		if (swiotlb_late_init_with_default_size(64 * (1<<20)) != 0)
+			panic("Unable to initialize software I/O TLB:"
+				  " Try machvec=dig boot option");
+		machvec_init("dig");
+		return 0;
+	}
+#endif
+
  	acpi_bus_register_driver(&acpi_sba_ioc_driver);
  	if (!ioc_list) {
  #ifdef CONFIG_IA64_GENERIC
-		extern int swiotlb_late_init_with_default_size (size_t size);
-
  		/*
  		 * If we didn't find something sba_iommu can claim, we
  		 * need to setup the swiotlb and switch to the dig machvec.

Hi Terry,

Thanks for your explanation and the patch.

Can you try to bring it mainline? Since it affects only arch/ia64/hp/*, I don't see a reason why it should be rejected.

Thanks,
Bernhard

This is basically what the powerpc 64-bit port is doing, except we don't explicitly reserve any entries. See arch/powerpc/kernel/iommu.c near line 467. Basically, our generic iommu layer has a bitmap of pages for available for allocation per iommu table, and calls a platform-specific back-end to give the mapping to the iommu. If a method is supplied to read the existing mappings, then we will fetch them and mark any slot with an existing translation as busy in the kdump kernel.

For the normal (non-crash-dump) kernel, we will kill any preexisting translations the firmware may have left behind; if something was still using a translation we would get a machine check (or at least the slot would go into an error state, for the platforms that support Enhanced Error Handling (EEH)).

I am not familier with the IA64 code and don't know the the iommu allocates translation slots.

milton

_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec

Not that it matters a huge amount, but this has been in RHEL5 for sometime and does fix the "machvec=dig" nuisance.

(Sorry for not replying earlier Terry -- I had a huge backlog of things to get to before this ...)

P.

Terry,

Just got to applying some older patches ... perhaps this one has been sitting too long because I had some apply & build problems with it.

Building with arch/ia64/configs/zx1_defconfig and with this patch hand applied I get:

LD .tmp_vmlinux1
arch/ia64/hp/common/built-in.o(.init.text+0xcc2): In function `sba_init':
: undefined reference to swiotlb_late_init_with_default_size' arch/ia64/hp/common/built-in.o(.init.text+0xd02): In function sba_init':
: undefined reference to `machvec_init'
make: *** [.tmp_vmlinux1] Error 1

sba_init() uses these routines inside #ifdef CONFIG_I64_GENERIC, perhaps you need to do the same?

Other issues:
1) No "Signed-off-by"

2) +#include <linux/crash_dump.h>

We usually group all the <linux/*> includes together before the <asm/*> ones

3) +extern int swiotlb_late_init_with_default_size (size_t size);

extern declarations in ".c" files are frowned upon. This should be in a header file, which should be included (though I see that this guideline has already been ignored as you were just moving the declaration).

-Tony

Sorry - I'd meant to attend to this but got distracted. I'll repost it shortly and address the issues you mentioned.

-T