Skip to contentSkip to conversation

kdump/kexec on EFI-enabled x2apic platforms

DISCUSSION Jack Steiner 3 participants 5 messages Mar 29, 2010 – Apr 1, 2010

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

All -

I just started debugging kdump/kexec on our UV platform and have run into some problems. I suspect others have encountered these same or similar problems. Any help would be appreciated.

Our platform uses EFI boot. It is Nehalem based & has a large number of cpus. The BIOS enables x2apic mode and the kernel runs with interrupt remapping enabled. Note that some apicids have more than 8 bits - x2apic mode is required.

I am able to successfully kexec the dump kernel but run into several problems.

- because the initial kernel boots using EFI, BIOS does not build the legacy tables that are required to locate the RSDP using the legacy method in acpi_find_root_pointer(). (When booting with EFI, acpi_find_root_pointer() is not used. The ACPI tables are found from pointers in EFI tables.)

- it appears that kdump/kexec intentionally boots the kdump kernel in a mode does does enable efi mode. (Am I correct here???) This avoids the issues with EFI virtual mode. However, the result is that ACPI tables are not found. From the dump kernel:

ACPI Error: A valid RSDP was not found (20090903/tbxfroot-222)

- Because ACPI tables are not found, the dump kernel does not transition into x2apic mode. The hardware, however, is still in x2apic mode from the initial kernel. Later in the boot of the dump kernel, read_apic_id() tried to read memory-mapped apic registers instead of the MSRs that are used in x2apic mode. This is not allowed & the dump kernel panics with:

[ 0.000000] [<ffffffff81b52195>] early_idt_handler+0x55/0x68
[ 0.000000] [<ffffffff8101f393>] ? native_apic_mem_read+0x3/0x10
[ 0.000000] [<ffffffff8101a3c6>] ? read_apic_id+0x16/0x30
[ 0.000000] [<ffffffff81b5f857>] init_apic_mappings+0xe7/0x137
[ 0.000000] [<ffffffff81b559fd>] setup_arch+0x900/0xc33
[ 0.000000] [<ffffffff81b52bae>] start_kernel+0x6f/0x4a1

I checked an Intel Nehalem whitebox using the Intel BIOS. The dump kernel does not find the RDSP but the initial kernel does not enable x2apic mode either (possibly because of an old BIOS - not sure). As a result, the dump kernel does not hit the panic shown above. The kdump kernel successfully boots w/o having discovered ACPI tables.

How should I proceed?
- should I be running the dump kernel with EFI mode enabled?
- should I be fixing the issues with x2apic mode in a non-EFI dump kernel?
- or should BIOS be building the tables necessary to support both EFI & non-EFI boot.

--- jack

Jack Steiner ([email protected]) SGI - Silicon Graphics, Inc.

_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec

#1 lore raw
Quoted 13 lines · Jack Steiner

All -

I just started debugging kdump/kexec on our UV platform and have run into some problems. I suspect others have encountered these same or similar problems. Any help would be appreciated.

Our platform uses EFI boot. It is Nehalem based & has a large number of cpus. The BIOS enables x2apic mode and the kernel runs with interrupt remapping enabled. Note that some apicids have more than 8 bits - x2apic mode is required.

Quoted 6 lines

I am able to successfully kexec the dump kernel but run into several problems.

- because the initial kernel boots using EFI, BIOS does not build the legacy tables that are required to locate the RSDP using the legacy method in acpi_find_root_pointer(). (When booting with EFI, acpi_find_root_pointer() is not used. The ACPI tables are found from pointers in EFI tables.)

Ouch. EFI tables are a major pain to use because they are not 32/64 clean. Thus making them unreasonably difficult to work with.

I believe the boot loader should be passing in the location of acpi tables instead of expecting the kernel to wade through EFI tables.

Quoted 6 lines

- it appears that kdump/kexec intentionally boots the kdump kernel in a mode does does enable efi mode. (Am I correct here???) This avoids the issues with EFI virtual mode. However, the result is that ACPI tables are not found. From the dump kernel:

ACPI Error: A valid RSDP was not found (20090903/tbxfroot-222)

My personal opinion is that EFI virtual mode is a mistake. We should not have any interactions with the EFI bios of sufficient frequency that we need an efficient virtual mapping.

Quoted 3 lines

- Because ACPI tables are not found, the dump kernel does not transition into x2apic mode. The hardware, however, is still in x2apic mode from the initial kernel.

Hmm. That is a bug of many flavors. We should have transitions back into i8259 legacy mode, before calling kdump. Then regardless of what happen before we ran if the hardware is x2apic capable we should force the hardware into the mode we want it, not just assume x2apic mode is off by default.

Quoted 20 lines
	  Later in the boot of the dump kernel, read_apic_id()
	  tried to read memory-mapped apic registers instead of the MSRs that are
	  used in x2apic mode. This is not allowed & the dump kernel panics with:

		[    0.000000]  [<ffffffff81b52195>] early_idt_handler+0x55/0x68
		[    0.000000]  [<ffffffff8101f393>] ? native_apic_mem_read+0x3/0x10
		[    0.000000]  [<ffffffff8101a3c6>] ? read_apic_id+0x16/0x30
		[    0.000000]  [<ffffffff81b5f857>] init_apic_mappings+0xe7/0x137
		[    0.000000]  [<ffffffff81b559fd>] setup_arch+0x900/0xc33
		[    0.000000]  [<ffffffff81b52bae>] start_kernel+0x6f/0x4a1

I checked an Intel Nehalem whitebox using the Intel BIOS. The dump kernel does not find the RDSP but the initial kernel does not enable x2apic mode either (possibly because of an old BIOS - not sure). As a result, the dump kernel does not hit the panic shown above. The kdump kernel successfully boots w/o having discovered ACPI tables.

Quoted 2 lines

How should I proceed?
- should I be running the dump kernel with EFI mode enabled?

We can not interact with the BIOS/EFI in kdump in general so finding a way to make it work without interacting with the BIOS seems to be a proper path.

Quoted 1 line

- should I be fixing the issues with x2apic mode in a non-EFI dump kernel?

Definitely.

Quoted 1 line

- or should BIOS be building the tables necessary to support both EFI & non-EFI boot.

That too. Unless it is impossible it is good to be able to avoid EFI nonsense.

Eric

_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec

#2 lore raw
Quoted 2 lines · Eric W. Biederman

Jack Steiner <[email protected]> writes:

Thnaks for the quick reply. I am still digging thru the issues but am making good progress.

See comments below.

Quoted 20 lines
Quoted 13 lines

All -

I just started debugging kdump/kexec on our UV platform and have run into some problems. I suspect others have encountered these same or similar problems. Any help would be appreciated.

Our platform uses EFI boot. It is Nehalem based & has a large number of cpus. The BIOS enables x2apic mode and the kernel runs with interrupt remapping enabled. Note that some apicids have more than 8 bits - x2apic mode is required.

Quoted 6 lines

I am able to successfully kexec the dump kernel but run into several problems.

- because the initial kernel boots using EFI, BIOS does not build the legacy tables that are required to locate the RSDP using the legacy method in acpi_find_root_pointer(). (When booting with EFI, acpi_find_root_pointer() is not used. The ACPI tables are found from pointers in EFI tables.)

Quoted 5 lines

Ouch. EFI tables are a major pain to use because they are not 32/64 clean. Thus making them unreasonably difficult to work with.

I believe the boot loader should be passing in the location of acpi tables instead of expecting the kernel to wade through EFI tables.

I made a temporary fix to the BIOS to build the legacy table used by acpi_find_root_pointer() to locate the ACPI tables. I tested this on a system simulator & it seems to work. ACPI tables are discovered and parsed correctly - at least for the current BIOS & hardware configuration. I'll try this on real hardware later but don't expect any problems.

Quoted 11 lines
Quoted 7 lines

- it appears that kdump/kexec intentionally boots the kdump kernel in a mode does does enable efi mode. (Am I correct here???) This avoids the issues with EFI virtual mode. However, the result is that ACPI tables are not found. From the dump kernel:

ACPI Error: A valid RSDP was not found (20090903/tbxfroot-222)

My personal opinion is that EFI virtual mode is a mistake. We should not have any interactions with the EFI bios of sufficient frequency that we need an efficient virtual mapping.

Agree. The EFI virtual mode support was poorly architected as far as kdump/kexec is concerned. We had a lot of problem on IA64, too.

Quoted 10 lines
Quoted 5 lines

- Because ACPI tables are not found, the dump kernel does not transition into x2apic mode. The hardware, however, is still in x2apic mode from the initial kernel.

Hmm. That is a bug of many flavors. We should have transitions back into i8259 legacy mode, before calling kdump. Then regardless of what happen before we ran if the hardware is x2apic capable we should force the hardware into the mode we want it, not just assume x2apic mode is off by default.

It is not possible to transition back into non-x2apic mode. The initial transition INTO x2apic mode is done by the BIOS - not the OS. X2apic mode is required because apic ids are greater than 8 bits. Fortunately, now that we discover ACPI tables, the OS is handling x2apic mode correctly.

With the above changes, I can now successfully boot to single user mode (simulator) in the dump kernel. I may still have a few issues with lack-of EFI support but I think I can work thru them. I did not see any problems on the simulator but might hit them on hardware.

The only issue that still needs to be resolved (that I know of) is the memmap. The E820 table in the boot_param block only supports 128 entries. Additional entries are passed in the EFI memmap but this is missing w/o enabling EFI.

I suspect there are other problems waiting to be discovered but at least I'm making progress.

Thanks for the help.

--- jack

_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec

#3 lore raw
In reply to Jack Steiner
Quoted 11 lines · Jack Steiner

All -

I just started debugging kdump/kexec on our UV platform and have run into some problems. I suspect others have encountered these same or similar problems. Any help would be appreciated.

Our platform uses EFI boot. It is Nehalem based & has a large number of cpus. The BIOS enables x2apic mode and the kernel runs with interrupt remapping enabled. Note that some apicids have more than 8 bits - x2apic mode is required.

What are you using as a boot loader? I've tested kexec with elilo 3.8 and got good results. But, that was early last year, so I'll need to try it out with production hardware and the current firmware and sw stack. I'll try to get to that this week, and let you know the results.

Signature · 7 lines
David N. Lombard, Intel, Irvine, CA
I do not speak for Intel Corporation; all comments are strictly my own.

_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec
#4 lore raw
Quoted 9 lines · David N. Lombard
Quoted 8 lines · Jack Steiner

All -

I just started debugging kdump/kexec on our UV platform and have run into some problems. I suspect others have encountered these same or similar problems. Any help would be appreciated.

Quoted 8 lines
Quoted 3 lines

Our platform uses EFI boot. It is Nehalem based & has a large number of cpus. The BIOS enables x2apic mode and the kernel runs with interrupt remapping enabled. Note that some apicids have more than 8 bits - x2apic mode is required.

What are you using as a boot loader? I've tested kexec with elilo 3.8 and got good results. But, that was early last year, so I'll need to try it out with production hardware and the current firmware and sw stack. I'll try to get to that this week, and let you know the results.

We use:

	elilo-3.12-0.3.6

AFAICT, the following are the problem areas:

- nehalem platform using a BIOS that puts RSDP at a location that is not scanned by acpi_find_root_pointer() looking for ACPI tables

[ 0.000000] ACPI: RSDP 000000007b7fe014 00024 (v02 INTEL )
[ 0.000000] ACPI: XSDT 000000007b7fe120 0005C (v01 INTEL TIANO 00000000 01000013)
[ 0.000000] ACPI: FACP 000000007b7fc000 000F4 (v03 INTEL TIANO 00000000 MSFT 01000013)
...

Booting with EFI works because ACPI tables are pointed to by the EFI tables. However, the kdump kernel does not enable EFI and the OS uses an alternate method to locate the ACPI tables. The alternate method fails.

NOTE: with a hacked BIOS that puts the RSDP at 0x9ec00, ACPI tables are correctly discovered & booting the kdump kernel is successful (except that some memory is missing (see next item)).

- number of memory blocks exceeds the size supported by the E820 table. (I'm still learning - memory is missing from the kdump kernel memory maps. Does this affect the memory dumpped from the kernel that crashed???)

- x2apic enabled by the BIOS & interrupt remapping enabled by the OS (this may no longer be an issue but caused problems earlier)

Appreciate the help....

--- jack

_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec