Skip to contentSkip to conversation

Trying to test my gart/iommu vmcore problem on RH

DISCUSSION Vivek Goyal 3 participants 14 messages Aug 19, 2008 – Sep 23, 2008 5 messages not in the archive

Subsystems (2)
lore
Share & tools
lore

A message that is not in the archive (sent off-list or not yet received): [email protected]

A message that is not in the archive (sent off-list or not yet received): [email protected]

A message that is not in the archive (sent off-list or not yet received): [email protected]

A message that is not in the archive (sent off-list or not yet received): [email protected]

A message that is not in the archive (sent off-list or not yet received): [email protected]

#5 lore raw

Hi Bob,

I am CCing this thread to kexec mailing list. It is good to discuss the issue there to get the ideas.

I will summarize the discussion so far.

Bob is running into MCA in second kernel in kdump. Reason seems to be that second kernel is trying to access the memory area marked as GART aperture (by first kernel). Because GART aperture does not appear as "reserved" or something else in /proc/iomem (in case kernel has overridden the BIOS settings and has reserved a memory area), second kernel thinks it is a valid RAM area and tries to dump it and runs into issues.

Few options Bob is considering are.

- Update "e820" memory map to mark GART aperture as reserved, which will be reflected in /proc/iomem also. Kexec-tools will not pass reserved area to second kernel and it will not try to dump this area.

- Mark GART aperture as "GART aperture" in /proc/iomem and modify kexec-tools to filter out this memroy from memory map passed to second kernel.

- Disable cpu side GART access in first kernel so that even if second kernel tries to access it, it does not run into isseus.

Thanks
Vivek

Quoted 66 lines · Bob Montgomery
Quoted 5 lines · Vivek Goyal

I checked that aperture is allocated in mem_init(), which is little late in the game but bootmem allocator is still in effect and we have not released the pages to free list. May be it is possible to modify e820 memory map even now. Somebody will have to experiemnt..

I'll try to study this a bit this week.

Quoted 3 lines

If not, then I also like the idea of marking the region as "GART Aperture" in /proc/iomem and let kexec-tools filter it.

This of course requires two things to change to get a fix - the kernel and the kexec-tools.

Quoted 4 lines

Not very sure about the idea of disabling cpu side access. Will it run into issues like MCE if DMAs are still going on? It does MCA if one tries to disable GART when DMAs are going on.

I am disabling CPU side access in the *first kernel*, when the GART is initially set up. The kdump kernel just inherits that setting when it boots. So no DMA is going on when I do the disable. The reason it seems safe to me is that in the first kernel, CPU side access is effectively disabled by this (in arch/x86_64/pci-gart.c on our 2.6.18-based kernel):

/*
* Unmap the IOMMU part of the GART. The alias of the page is
* always mapped with cache enabled and there is no full cache
* coherency across the GART remapping. The unmapping avoids
* automatic prefetches from the CPU allocating cache lines in
* there. All CPU accesses are done via the direct mapping to
* the backing memory. The GART address is only used by PCI
* devices.
*/
clear_kernel_mapping((unsigned long)__va(iommu_bus_base),
iommu_size);

I notice some changes in the equivalent area in 2.6.26:
/*
* Unmap the IOMMU part of the GART. The alias of the page is
* always mapped with cache enabled and there is no full cache
* coherency across the GART remapping. The unmapping avoids
* automatic prefetches from the CPU allocating cache lines in
* there. All CPU accesses are done via the direct mapping to
* the backing memory. The GART address is only used by PCI
* devices.
*/
set_memory_np((unsigned long)__va(iommu_bus_base),
iommu_size >> PAGE_SHIFT);
/*
* Tricky. The GART table remaps the physical memory range,
* so the CPU wont notice potential aliases and if the memory
* is remapped to UC later on, we might surprise the PCI devices
* with a stray writeout of a cacheline. So play it sure and
* do an explicit, full-scale wbinvd() _after_ having marked all
* the pages as Not-Present:
*/
wbinvd();

Quoted 13 lines

set_memory_np does:

	change_page_attr_clear(addr, numpages, __pgprot(_PAGE_PRESENT));

wbinvd() does the wbinvd (write back and invalidate caches) instruction.

Bob Montgomery

Quoted 3 lines

Thanks
Vivek

_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec

#6 lore raw
Quoted 17 lines · Vivek Goyal

Hi Bob,

I am CCing this thread to kexec mailing list. It is good to discuss the issue there to get the ideas.

I will summarize the discussion so far.

Bob is running into MCA in second kernel in kdump. Reason seems to be that second kernel is trying to access the memory area marked as GART aperture (by first kernel). Because GART aperture does not appear as "reserved" or something else in /proc/iomem (in case kernel has overridden the BIOS settings and has reserved a memory area), second kernel thinks it is a valid RAM area and tries to dump it and runs into issues.

Few options Bob is considering are.

Quoted 8 lines

- Update "e820" memory map to mark GART aperture as reserved, which will be reflected in /proc/iomem also. Kexec-tools will not pass reserved area to second kernel and it will not try to dump this area.

- Mark GART aperture as "GART aperture" in /proc/iomem and modify kexec-tools to filter out this memroy from memory map passed to second kernel.

We should definitely reserve the resource, and it should definitely show up in /proc/iomem.

Quoted 2 lines

- Disable cpu side GART access in first kernel so that even if second kernel tries to access it, it does not run into isseus.

This is an interesting one. When I looked at this years ago I had the feeling that if we did this we could actually always use a 2G Aperture at a fixed address, and require going through the gart for all of lowmem.

But that is a little more than we are talking about.

Eric

_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec

#7 lore raw
Quoted 9 lines · Eric W. Biederman
Quoted 7 lines · Vivek Goyal

Few options Bob is considering are.

- Update "e820" memory map to mark GART aperture as reserved, which will be reflected in /proc/iomem also. Kexec-tools will not pass reserved area to second kernel and it will not try to dump this area.

Quoted 7 lines
Quoted 3 lines

- Mark GART aperture as "GART aperture" in /proc/iomem and modify kexec-tools to filter out this memroy from memory map passed to second kernel.

We should definitely reserve the resource, and it should definitely show up in /proc/iomem.

Reserving it as a child resource called "GART" in a "System RAM" resource is already in newer kernels than mine (at least in by 2.6.26). I haven't seen that kexec-tools does anything with that yet. kexec-tools looks for "Crash kernel" in /proc/iomem now and explicitly excludes that area.

Example:
000f0000-000fffff : System ROM
00100000-cfe4ffff : System RAM
00200000-0042635a : Kernel code
0042635b-00592037 : Kernel data
01000000-08ffffff : Crash kernel
0c000000-0fffffff : GART
cfe50000-cfe57fff : ACPI Tables
cfe58000-cfffffff : reserved

If it could be "reserved" earlier, so it isn't a child resource of a System Ram area, but a "reserved" area that divides two "System RAM" areas, then the current kexec-tools would exclude it (like it excludes all "reserved" areas from the /proc/vmcore map, and it would no longer be possible to trigger the MCE (or the mysterious hang) by reading from /proc/vmcore. But currently (in my older kernel) the original iomem_resource is constructed from the e820 map before I know where (and how big) the aperture will be created.

But either way we fix it in iomem to exclude it from /proc/vmcore, a read of /dev/oldmem in the aperture area would still trigger the MCE. At least it does on my system.

Quoted 3 lines
Quoted 3 lines

- Disable cpu side GART access in first kernel so that even if second kernel tries to access it, it does not run into isseus.

This has the advantage of "fixing" accesses through both /proc/vmcore
and /dev/oldmem. And for me, it's an easy patch to pci-gart.c in
init_k8_gatt that just sets bit 4 instead of clearing both 4 and 5:

-               ctl |= 1;
-               ctl &= ~((1<<4) | (1<<5));
+               ctl |= 1;       /* set GartEn */
+               ctl |= (1<<4);  /* set DisGartCpu */
+               ctl &= ~(1<<5); /* clear DisGartIO */
Quoted 4 lines

This is an interesting one. When I looked at this years ago I had the feeling that if we did this we could actually always use a 2G Aperture at a fixed address, and require going through the gart for all of lowmem.

During discussions here, a colleague suggested that with CPU-side access of the aperture disabled, we could allocate the crash kernel in the wasted memory "under" the aperture.

Quoted 2 lines

But that is a little more than we are talking about.

Yes, also.

Quoted 2 lines

Eric

Bob Montgomery

_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec

#8 lore raw

Hmm. Thinking about this we actually have 2 problems.
- Communication about what is going on.
- How to handle an iommu in the event of a crash dump scenario.

The current solution is to ignore the iommu, and use swiotlb. This solution does not look like it will work for future iommus.

The original plan (and it still sounds like a good one) was to reserve a section of the iommu (as we do for the physical memory). So we could have addresses that are only used for the crash dump kernel. Then have the crash dump kernel just use that section of the iommu.

Either we need to do that or we need to disable the iommu, before we use swiotlb.

The problem is we can not reliably kill on-going DMA transactions at the time of a kernel panic, and likely doing so would greatly decrease our kernel reliability.

Eric

_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec

#9 lore raw
Quoted 8 lines · Eric W. Biederman

Hmm. Thinking about this we actually have 2 problems.
- Communication about what is going on.
- How to handle an iommu in the event of a crash dump scenario.

The current solution is to ignore the iommu, and use swiotlb. This solution does not look like it will work for future iommus.

Does setting up of swiotlb require iommu to be disabled in second kernel? IOW, can swiotlb work reliably given the fact that iommu is active and there are some active mappings (as created by first kernel).

I am thinking is there a possibility that I set a DMA using swiotlb and the physical address can overlap with IO address setup in IOMMU and that DMA might go to a different buffer altogether.

Quoted 5 lines

The original plan (and it still sounds like a good one) was to reserve a section of the iommu (as we do for the physical memory). So we could have addresses that are only used for the crash dump kernel. Then have the crash dump kernel just use that section of the iommu.

This would also require that second kernel keeps using first kernel's iommu settings/tables and not try to initialize the iommu freshly.

One patch from Chandru is now mainline which seems to be solving the issue for calgary IOMMU. He seems to be re-using first kernel's iommu tables in second kernel hence avoiding re-initializing iommu and avoiding MCE.

git commit 95b68dec0d52c7b8fea3698b3938cf3ab936436b

This patch has the risk that second kernel might not find any free entries to setup DMA and that's why reserving a section of iommu will help.

Quoted 3 lines

Either we need to do that or we need to disable the iommu, before we use swiotlb.

I tought disabling iommu was not an option as it leads to MCE if there is a DMA going on.

Quoted 3 lines

The problem is we can not reliably kill on-going DMA transactions at the time of a kernel panic, and likely doing so would greatly decrease our kernel reliability.

May be re-using iommu tables in second kernel along with reserving some entries for kdump is the way to go..

Thanks
Vivek

_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec

#10 lore raw
Quoted 11 lines · Vivek Goyal
Quoted 8 lines · Eric W. Biederman

Hmm. Thinking about this we actually have 2 problems.
- Communication about what is going on.
- How to handle an iommu in the event of a crash dump scenario.

The current solution is to ignore the iommu, and use swiotlb. This solution does not look like it will work for future iommus.

Does setting up of swiotlb require iommu to be disabled in second kernel?

Not precisely. But in a full iommu all accesses go through the iommu, and the iommu start becoming per bus. So in practice either we need to disable full iommu or work with them.

Quoted 6 lines

IOW, can swiotlb work reliably given the fact that iommu is active and there are some active mappings (as created by first kernel).

I am thinking is there a possibility that I set a DMA using swiotlb and the physical address can overlap with IO address setup in IOMMU and that DMA might go to a different buffer altogether.

Yes. Which is why I would very much prefer to reserve some IOMMU entries. Instead of turning off an iommu altogether.

Quoted 8 lines
Quoted 5 lines

The original plan (and it still sounds like a good one) was to reserve a section of the iommu (as we do for the physical memory). So we could have addresses that are only used for the crash dump kernel. Then have the crash dump kernel just use that section of the iommu.

This would also require that second kernel keeps using first kernel's iommu settings/tables and not try to initialize the iommu freshly.

Not completely anyway.

Quoted 8 lines

One patch from Chandru is now mainline which seems to be solving the issue for calgary IOMMU. He seems to be re-using first kernel's iommu tables in second kernel hence avoiding re-initializing iommu and avoiding MCE.

git commit 95b68dec0d52c7b8fea3698b3938cf3ab936436b

This patch has the risk that second kernel might not find any free entries to setup DMA and that's why reserving a section of iommu will help.

Yes. That and we know there aren't any pending DMAs going to missetup entries.

Quoted 6 lines
Quoted 3 lines

Either we need to do that or we need to disable the iommu, before we use swiotlb.

I tought disabling iommu was not an option as it leads to MCE if there is a DMA going on.

Good point. Looks like I oversimplified.

Quoted 6 lines
Quoted 3 lines

The problem is we can not reliably kill on-going DMA transactions at the time of a kernel panic, and likely doing so would greatly decrease our kernel reliability.

May be re-using iommu tables in second kernel along with reserving some entries for kdump is the way to go..

That is the best plan we have been able to come up with. Making AMD's iommu look more like a full strength iommu should help reinforce that model.

Eric

_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec

#11 lore raw
Quoted 10 lines · Eric W. Biederman
Quoted 8 lines · Vivek Goyal
Quoted 7 lines · Eric W. Biederman

Hmm. Thinking about this we actually have 2 problems.
- Communication about what is going on.
- How to handle an iommu in the event of a crash dump scenario.

The current solution is to ignore the iommu, and use swiotlb. This solution does not look like it will work for future iommus.

Howdy all,

There are several aspects to this problem that make solutions come in and out of contention:

1. Kexec vs Kdump

Kexec: If we are kexec'ing normally, we assume that the shutdown has
successfully stopped DMAs prior to starting our new kernel, and if not, it's a bug in the previous kernel's driver shutdown. So no issue here, right?

Kdump: The driver shutdown has been skipped as we go down during a
crash, so assume that leftover DMA operations might be in progress as the kdump kernel comes up. BUT! They will be in progress to some area of memory other than the memory being used by the kdump kernel (it has its own crashkernel sandbox). And on my 2.6.18-based system, with an AMD64 NB GART-acting-as-IOMMU, the kdump kernel *does not* try to initialize or use an IOMMU when it comes up because its memory size is too small to need one (no one is setting crashkernel=4G@4G). So the kdump kernel can successfully ignore the old IOs using the old GART aperture IOMMU.

EXCEPT(!) for the fact the we've left CPU-side translations turned on in the GART NB hardware and the kdump kernel will currently read through that zone using /proc/vmcore or /dev/oldmem. That's why I like fixing my stone-age problem by turning off CPU-side access.

Note that real (future?) IOMMUs don't even have the concept of translating accesses from the CPU side. They only work on IO requests. So reading old memory areas from the crashed kernel shouldn't cause an IOMMU to "do" anything.

2. GART vs Calgary vs "new AMD IOMMU" vs "new Intel IOMMU"

The GART-as-IOMMU thing is not a "real" IOMMU. It doesn't offer much
of the interesting protection of a real IOMMU, just "valid", "coherent" and a translation address. An IO card is still free to screw up and hit other addresses outside the aperture if it wants to, or hit other pages in the aperture that really belong to some other driver, or to write to a page that it should only read, etc. Consequently, there isn't much desire to utilize the GART thing unless I really need 32-bit IO card access to 40-bit address space. Since I don't need that in the kdump kernel (currently), there's no reason to try to use the GART there, so it's safe to ignore it, as long as I don't provoke it :-)

BUT, if I had a real IOMMU that provided cool protection stuff and domain stuff, and not just address range expansion for old IO cards, then I might want to (or be forced to) use it all the time, independent of memory size, and then the kdump kernel might really need to deal with sharing it in some way with old leftover DMAs from the crashed kernel that we're dumping. And this, I think, is the only real issue looming.

But this should only be a kdump issue, and not a kexec issue (see #1 above), because the previous kernel should have shut all that down before it kexec'd, right?

3. IOMMU vs swiotlb

Isn't swiotlb just a way of hiding bounce buffer copies and management inside of the dma_map_single and dma_unmap_single calls? If so, it's just software(TM) and it just uses addresses in the kdump kernel sandbox, which (by definition) are not addresses that could have been used in the old kernel that crashed. There shouldn't be any conflict between kdump kernel and old crashed kernel if one or both are using swiotlb. Once again, in *my* current situation, there's no reason to use swiotlb in the kdump kernel because my memory range is restricted to my crashkernel sandbox and I don't need any IOMMU address translation capability.

If the original kernel had been using swiotlb, then there's really no issue, because any leftover DMAs are just writing to the old bounce buffers anyway, and there's no driver left waiting to call dma_unmap_single to copy the result into the real buffer.

What considerations have I missed?

Bob Montgomery
(vacation last week)

_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec

#12 lore raw
Quoted 16 lines · Bob Montgomery
Quoted 10 lines · Eric W. Biederman
Quoted 8 lines · Vivek Goyal
Quoted 7 lines · Eric W. Biederman

Hmm. Thinking about this we actually have 2 problems.
- Communication about what is going on.
- How to handle an iommu in the event of a crash dump scenario.

The current solution is to ignore the iommu, and use swiotlb. This solution does not look like it will work for future iommus.

Howdy all,

There are several aspects to this problem that make solutions come in and out of contention:

Quoted 6 lines

1. Kexec vs Kdump

Kexec: If we are kexec'ing normally, we assume that the shutdown has
successfully stopped DMAs prior to starting our new kernel, and if not, it's a bug in the previous kernel's driver shutdown. So no issue here, right?

Correct.

Quoted 16 lines

Kdump: The driver shutdown has been skipped as we go down during a
crash, so assume that leftover DMA operations might be in progress as the kdump kernel comes up. BUT! They will be in progress to some area of memory other than the memory being used by the kdump kernel (it has its own crashkernel sandbox). And on my 2.6.18-based system, with an AMD64 NB GART-acting-as-IOMMU, the kdump kernel *does not* try to initialize or use an IOMMU when it comes up because its memory size is too small to need one (no one is setting crashkernel=4G@4G). So the kdump kernel can successfully ignore the old IOs using the old GART aperture IOMMU.

EXCEPT(!) for the fact the we've left CPU-side translations turned on in the GART NB hardware and the kdump kernel will currently read through that zone using /proc/vmcore or /dev/oldmem. That's why I like fixing my stone-age problem by turning off CPU-side access.

Quoted 4 lines

Note that real (future?) IOMMUs don't even have the concept of translating accesses from the CPU side. They only work on IO requests. So reading old memory areas from the crashed kernel shouldn't cause an IOMMU to "do" anything.

Good point. I don't think linux uses the translations either.

The downside of this is that it increases the dependency of the kernel that crashed not to have done something bad. So at least long term it would be good to have code that can share do the right thing with iommus.

Quoted 13 lines

2. GART vs Calgary vs "new AMD IOMMU" vs "new Intel IOMMU"

The GART-as-IOMMU thing is not a "real" IOMMU. It doesn't offer much
of the interesting protection of a real IOMMU, just "valid", "coherent" and a translation address. An IO card is still free to screw up and hit other addresses outside the aperture if it wants to, or hit other pages in the aperture that really belong to some other driver, or to write to a page that it should only read, etc. Consequently, there isn't much desire to utilize the GART thing unless I really need 32-bit IO card access to 40-bit address space. Since I don't need that in the kdump kernel (currently), there's no reason to try to use the GART there, so it's safe to ignore it, as long as I don't provoke it :-)

Quoted 6 lines

BUT, if I had a real IOMMU that provided cool protection stuff and domain stuff, and not just address range expansion for old IO cards, then I might want to (or be forced to) use it all the time, independent of memory size, and then the kdump kernel might really need to deal with sharing it in some way with old leftover DMAs from the crashed kernel that we're dumping. And this, I think, is the only real issue looming.

Yes. How do we properly share an iommu is the looming issue.

Quoted 3 lines

But this should only be a kdump issue, and not a kexec issue (see #1 above), because the previous kernel should have shut all that down before it kexec'd, right?

Correct.

Quoted 19 lines

3. IOMMU vs swiotlb

Isn't swiotlb just a way of hiding bounce buffer copies and management inside of the dma_map_single and dma_unmap_single calls? If so, it's just software(TM) and it just uses addresses in the kdump kernel sandbox, which (by definition) are not addresses that could have been used in the old kernel that crashed. There shouldn't be any conflict between kdump kernel and old crashed kernel if one or both are using swiotlb. Once again, in *my* current situation, there's no reason to use swiotlb in the kdump kernel because my memory range is restricted to my crashkernel sandbox and I don't need any IOMMU address translation capability.

If the original kernel had been using swiotlb, then there's really no issue, because any leftover DMAs are just writing to the old bounce buffers anyway, and there's no driver left waiting to call dma_unmap_single to copy the result into the real buffer.

Quoted 5 lines

What considerations have I missed?

Bob Montgomery
(vacation last week)

If the BIOS provides us with an aperture in pci mmio space for the AMD GART there is no memory loss, and the issue that you are seeing can not occur.

This has a couple of interesting implications.
1) If you disable translation of cpu side accesses than you can continue to use the memory instead at the bus addresses used for the GART.
2) If you can continue to use the memory you can make the GART aperture it's maximum size 2G I think. This begins to provide protection from errant DMA addresses sent by devices.
3) If we enable access to the memory where the GART lives we have a situation where bus addresses and memory addresses are not always in the same domain. Giving us true iommu fun.

So specific recommendations.
1) Since cpu side accesses to the GART appear to just silly let's disable them.
2) We need to figure out how to communicate the disjoint address spaces that come with iommus in /sbin/kexec the user space code that sets this up.
3) If anyone has the oomph let's put the AMD K8 GART into large window mode and see if we can sort through all of the iommu issues on a platform that a lot of people have to work with.

Eric

_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec

#13 lore raw
In reply to Bob Montgomery
Quoted 83 lines · Bob Montgomery
Quoted 10 lines · Eric W. Biederman
Quoted 8 lines · Vivek Goyal
Quoted 7 lines · Eric W. Biederman

Hmm. Thinking about this we actually have 2 problems.
- Communication about what is going on.
- How to handle an iommu in the event of a crash dump scenario.

The current solution is to ignore the iommu, and use swiotlb. This solution does not look like it will work for future iommus.

Howdy all,

There are several aspects to this problem that make solutions come in and out of contention:

1. Kexec vs Kdump

Kexec: If we are kexec'ing normally, we assume that the shutdown has
successfully stopped DMAs prior to starting our new kernel, and if not, it's a bug in the previous kernel's driver shutdown. So no issue here, right?

Kdump: The driver shutdown has been skipped as we go down during a
crash, so assume that leftover DMA operations might be in progress as the kdump kernel comes up. BUT! They will be in progress to some area of memory other than the memory being used by the kdump kernel (it has its own crashkernel sandbox). And on my 2.6.18-based system, with an AMD64 NB GART-acting-as-IOMMU, the kdump kernel *does not* try to initialize or use an IOMMU when it comes up because its memory size is too small to need one (no one is setting crashkernel=4G@4G). So the kdump kernel can successfully ignore the old IOs using the old GART aperture IOMMU.

EXCEPT(!) for the fact the we've left CPU-side translations turned on in the GART NB hardware and the kdump kernel will currently read through that zone using /proc/vmcore or /dev/oldmem. That's why I like fixing my stone-age problem by turning off CPU-side access.

Note that real (future?) IOMMUs don't even have the concept of translating accesses from the CPU side. They only work on IO requests. So reading old memory areas from the crashed kernel shouldn't cause an IOMMU to "do" anything.

2. GART vs Calgary vs "new AMD IOMMU" vs "new Intel IOMMU"

The GART-as-IOMMU thing is not a "real" IOMMU. It doesn't offer much
of the interesting protection of a real IOMMU, just "valid", "coherent" and a translation address. An IO card is still free to screw up and hit other addresses outside the aperture if it wants to, or hit other pages in the aperture that really belong to some other driver, or to write to a page that it should only read, etc. Consequently, there isn't much desire to utilize the GART thing unless I really need 32-bit IO card access to 40-bit address space. Since I don't need that in the kdump kernel (currently), there's no reason to try to use the GART there, so it's safe to ignore it, as long as I don't provoke it :-)

BUT, if I had a real IOMMU that provided cool protection stuff and domain stuff, and not just address range expansion for old IO cards, then I might want to (or be forced to) use it all the time, independent of memory size, and then the kdump kernel might really need to deal with sharing it in some way with old leftover DMAs from the crashed kernel that we're dumping. And this, I think, is the only real issue looming.

But this should only be a kdump issue, and not a kexec issue (see #1 above), because the previous kernel should have shut all that down before it kexec'd, right?

3. IOMMU vs swiotlb

Isn't swiotlb just a way of hiding bounce buffer copies and management inside of the dma_map_single and dma_unmap_single calls? If so, it's just software(TM) and it just uses addresses in the kdump kernel sandbox, which (by definition) are not addresses that could have been used in the old kernel that crashed. There shouldn't be any conflict between kdump kernel and old crashed kernel if one or both are using swiotlb. Once again, in *my* current situation, there's no reason to use swiotlb in the kdump kernel because my memory range is restricted to my crashkernel sandbox and I don't need any IOMMU address translation capability.

Quoted 7 lines

If the original kernel had been using swiotlb, then there's really no issue, because any leftover DMAs are just writing to the old bounce buffers anyway, and there's no driver left waiting to call dma_unmap_single to copy the result into the real buffer.

What considerations have I missed?

Nice summary Bob. Few thoughts.

- So until and unless one is reserving memory for crashkernel above 4G, there is no need for initializing the IOMMU in second kernel (At this moment I am not too worried about need of isolation in second kernel). If that's the case, we shouldn't have initialized the calgary iommu in second kernel and just should have left it alone and things probably would have been fine?

The only issue is that how do you make sure that first kernel has not setup an IOMMU entry with same bus address which falls in crash kernel reserved area. I am not very familiar with the dma/iommu code and how bus addresses are selected. Because if there is bus address overlap in first kernel and second kernel, IOMMU will trap the second kernel's DMA attempts and redirect it somewhere else. If we don't run into this issue then it is fine otherwise we will be forced to use IOMMU in second kernel and try to find free bus addresses/entries so that we don't conflict with the first kernel's settings.

- Disabling cpu side access seems to makes sense. We can give it a try and hope we don't run into other hidden issues.

Thanks
Vivek

_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec

#14 lore raw
Quoted 8 lines · Vivek Goyal

Nice summary Bob. Few thoughts.

- So until and unless one is reserving memory for crashkernel above 4G, there is no need for initializing the IOMMU in second kernel (At this moment I am not too worried about need of isolation in second kernel). If that's the case, we shouldn't have initialized the calgary iommu in second kernel and just should have left it alone and things probably would have been fine?

Did you ever try booting the kdump kernel with iommu=off? That should have prevented the detection and initialization of the calgary iommu from the kdump kernel, which (I ass-u-me) is what you had problems with, if leftover IOs from the crashed kernel were still in progress when the kdump kernel initialized it?

As for 4G, the info in 2.6.26 Documentation/x86_64/boot-options.txt mentions 3G as the decision for using an iommu in several places. The test in our kernel for using the GART IOMMU is (end_pfn > MAX_DMA32_PFN) where MAX_DMA32_PFN is:
#define MAX_DMA32_PFN ((4UL*1024*1024*1024) >> PAGE_SHIFT)

Quoted 10 lines

The only issue is that how do you make sure that first kernel has not setup an IOMMU entry with same bus address which falls in crash kernel reserved area. I am not very familiar with the dma/iommu code and how bus addresses are selected. Because if there is bus address overlap in first kernel and second kernel, IOMMU will trap the second kernel's DMA attempts and redirect it somewhere else. If we don't run into this issue then it is fine otherwise we will be forced to use IOMMU in second kernel and try to find free bus addresses/entries so that we don't conflict with the first kernel's settings.

I finally thought about this long enough to figure out what you meant, I
think :-). If the existence of a real IOMMU in the first kernel
allows the use of completely virtualized addresses on the IO side, then there's no reason that they would have to avoid real RAM addresses when handing out addresses to IO cards with that IOMMU.

In particular, our little joke about allocating the crash kernel under the GART aperture (with CPU-side access turned off) would prevent the kdump kernel from doing non-iommu IO to the crash kernel address range, because the still-active GART from the old kernel would grab any of those addresses coming in from the IO side.

The kdump kernel wouldn't be in danger of being overwritten, it just might not be able to set up any IOs that work to its own address space if an IOMMU is out there waiting to grab them.

For the calgary case, we'd maybe have to add the Crash Kernel range to the list of things sent to iommu_range_reserve in calgary_reserve_regions, to prevent those addresses from ever being given out.

Bob M.

_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec

#15 lore raw
Quoted 10 lines · On Tue, 2008-09-09 at 21:12 +0000, I wrote:

The kdump kernel wouldn't be in danger of being overwritten, it just might not be able to set up any IOs that work to its own address space if an IOMMU is out there waiting to grab them.

For the calgary case, we'd maybe have to add the Crash Kernel range to the list of things sent to iommu_range_reserve in calgary_reserve_regions, to prevent those addresses from ever being given out.

Bob M.

While this, or something like it, is necessary, it isn't sufficient. I think what we would really need to do is to have the primary kernel set up an identity mapping for all pages in the Crash kernel range, or the subset of pages that could be IO targets when the kdump kernel is running. This would allow a still-running IOMMU from the primary kernel to translate kdump IOs to kdump addresses transparently.

And that leads to the Kdump IO Rule:

The primary kernel is responsible for setting up any necessary conditions to allow the kdump kernel to perform its required IO without detecting any iommu.

The kdump kernel must refrain from detecting and initializing any iommu.

This has a these effects:

A) Primary kernel: depending on what it is using for as an IOMMU, it may have to do some (or considerable) setup, to guarantee that the kdump kernel can have IO capability to its Crash kernel address range.

B) Primary kernel: the Crash kernel range must be set up in an address range whose physical addresses are accessible to IO cards without address remapping.

C) Kdump kernel: the kdump kernel must ignore any IOMMU hardware that might be "detectable".

The setup responsibilities for the primary kernel depend on what it is currently using for dma mapping:

1) no iommu (nommu_map_single): No setup is required for kdump. Leftover IOs will go to IO buffers allocated by the primary kernel outside of the Crash kernel area. Kdump IOs will go to IO buffers allocated by the kdump kernel in the Crash kernel area.

2) swiotlb (swiotlb_map_single_phys): No setup is required for kdump. Leftover IOs will go to the primary kernel bounce buffers outside of the Crash kernel area. Kdump IOs will go to IO buffers allocated by the kdump kernel in the Crash kernel area.

3) GART (gart_map_single): No setup is required for kdump. Leftover IOs will be mapped through the GART aperture to IO buffers allocated by the primary kernel outside of the Crash Kernel area. Kdump IOs will go to IO buffers allocated by the kdump kernel in the Crash kernel area.

4) Calgary IOMMU (calgary_map_single): The Crash kernel memory range must be pre-allocated for IO and identity-mapped, so any IO operation to an address in the Crash kernel range is allowed to complete to that same address. To preallocate for a 128MB Crash kernel area, 32K entries (256 Kbytes) are used from the Calgary table. For a 4GB system, the default size of the table is 1024K entries (8 Mbytes).

Leftover IOs will go to IO buffers allocated by the primary kernel and remapped by the Calgary IOMMU. Neither the IO-side address (iova) nor the physical address of a leftover IO will be in the Crash kernel area. Kdump IOs will go to IO buffers allocated by the kdump kernel, remapped by the Calgary IOMMU to those same addresses (iova equals physical address within the Crash kernel area).

5) Intel IOMMU (intel_map_single): The Crash kernel memory range must be pre-allocated and identity-mapped for each hw device that is needed by the kdump kernel, so any IO operation to an address in the Crash kernel range is allowed to complete to that same address.

Leftover IOs will go to IO buffers allocated by the primary kernel and remapped by the Intel IOMMU. Neither the IO-side address (iova) nor the physical address of a leftover IO will be in the Crash kernel area. Kdump IOs will go to IO buffers allocated by the kdump kernel, remapped by the Intel IOMMU to those same addresses (iova equals physical address within the Crash kernel area).

This all assumes no virtual machine stuff yet.

Possible? Comments? Corrections?

Bob Montgomery

_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec

#16 lore raw
Quoted 20 lines · Bob Montgomery
Quoted 10 lines · On Tue, 2008-09-09 at 21:12 +0000, I wrote:

The kdump kernel wouldn't be in danger of being overwritten, it just might not be able to set up any IOs that work to its own address space if an IOMMU is out there waiting to grab them.

For the calgary case, we'd maybe have to add the Crash Kernel range to the list of things sent to iommu_range_reserve in calgary_reserve_regions, to prevent those addresses from ever being given out.

Bob M.

While this, or something like it, is necessary, it isn't sufficient. I think what we would really need to do is to have the primary kernel set up an identity mapping for all pages in the Crash kernel range, or the subset of pages that could be IO targets when the kdump kernel is running. This would allow a still-running IOMMU from the primary kernel to translate kdump IOs to kdump addresses transparently.

Quoted 5 lines

And that leads to the Kdump IO Rule:

The primary kernel is responsible for setting up any necessary conditions to allow the kdump kernel to perform its required IO without detecting any iommu.

Reserving a range or addresses in the iommu I agree with. If that range of addresses allows for identity mapping I like it better.

I'm not certain about requiring it.

I don't like setting up the identity mapping before hand, it allows devices to trash the kdump kernel by accident.

Quoted 2 lines

The kdump kernel must refrain from detecting and initializing any iommu.

Why? I can fully understand avoiding addresses that are in flight. I can definitely see this being simpler in the kdump kernel. However this feels like it makes a less robust kdump kernel by not allowing it to touch the iommu.

Quoted 10 lines

This has a these effects:

A) Primary kernel: depending on what it is using for as an IOMMU, it may have to do some (or considerable) setup, to guarantee that the kdump kernel can have IO capability to its Crash kernel address range.

B) Primary kernel: the Crash kernel range must be set up in an address range whose physical addresses are accessible to IO cards without address remapping.

Below <= 16MB? That doesn't work in general.

Especially not if we are running on an SGI box and someone had unplugged node 0 (with all of the memory below 4G).

Quoted 55 lines

C) Kdump kernel: the kdump kernel must ignore any IOMMU hardware that might be "detectable".

The setup responsibilities for the primary kernel depend on what it is currently using for dma mapping:

1) no iommu (nommu_map_single): No setup is required for kdump. Leftover IOs will go to IO buffers allocated by the primary kernel outside of the Crash kernel area. Kdump IOs will go to IO buffers allocated by the kdump kernel in the Crash kernel area.

2) swiotlb (swiotlb_map_single_phys): No setup is required for kdump. Leftover IOs will go to the primary kernel bounce buffers outside of the Crash kernel area. Kdump IOs will go to IO buffers allocated by the kdump kernel in the Crash kernel area.

3) GART (gart_map_single): No setup is required for kdump. Leftover IOs will be mapped through the GART aperture to IO buffers allocated by the primary kernel outside of the Crash Kernel area. Kdump IOs will go to IO buffers allocated by the kdump kernel in the Crash kernel area.

4) Calgary IOMMU (calgary_map_single): The Crash kernel memory range must be pre-allocated for IO and identity-mapped, so any IO operation to an address in the Crash kernel range is allowed to complete to that same address. To preallocate for a 128MB Crash kernel area, 32K entries (256 Kbytes) are used from the Calgary table. For a 4GB system, the default size of the table is 1024K entries (8 Mbytes).

Leftover IOs will go to IO buffers allocated by the primary kernel and remapped by the Calgary IOMMU. Neither the IO-side address (iova) nor the physical address of a leftover IO will be in the Crash kernel area. Kdump IOs will go to IO buffers allocated by the kdump kernel, remapped by the Calgary IOMMU to those same addresses (iova equals physical address within the Crash kernel area).

5) Intel IOMMU (intel_map_single): The Crash kernel memory range must be pre-allocated and identity-mapped for each hw device that is needed by the kdump kernel, so any IO operation to an address in the Crash kernel range is allowed to complete to that same address.

Leftover IOs will go to IO buffers allocated by the primary kernel and remapped by the Intel IOMMU. Neither the IO-side address (iova) nor the physical address of a leftover IO will be in the Crash kernel area. Kdump IOs will go to IO buffers allocated by the kdump kernel, remapped by the Intel IOMMU to those same addresses (iova equals physical address within the Crash kernel area).

Quoted 3 lines

This all assumes no virtual machine stuff yet.

Possible? Comments? Corrections?

Possible.

I would very much like the option of doing the iommu setup, and possibly
fiddling in the kdump kernel. As long as we are not reusing the same
addresses in the iommu I don't see a problem.

I like the theoretical option of disabling ongoing DMA's, with the more complete IOMMUs. It isn't strictly necessary but I expect it would give a better result.

Eric

_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec

#17 lore raw
Quoted 7 lines · Eric W. Biederman
Quoted 5 lines · Bob Montgomery

And that leads to the Kdump IO Rule:

The primary kernel is responsible for setting up any necessary conditions to allow the kdump kernel to perform its required IO without detecting any iommu.

Quoted 8 lines

Reserving a range or addresses in the iommu I agree with. If that range of addresses allows for identity mapping I like it better.

I'm not certain about requiring it.

I don't like setting up the identity mapping before hand, it allows devices to trash the kdump kernel by accident.

The reason for having the primary kernel set up any mapping needed by a kdump kernel *in advance* is that for a HW IOMMU, this setup actually consists of modifying data structures (arrays, trees, lists) that are in the primary kernel's memory, as well as setting registers in the HW. When the kdump kernel comes up, none of those structures are in its memory range. They're just part of the artifacts left in /dev/oldmem. So yes, the kdump kernel could query any hardware that it found, verify that the hardware had previously been in use, read HW registers to get the root pointers, or list addresses or whatever, and then modify arrays, trees, or lists in that non-owned memory to map its DMA, but it's kind of an unprecedented step for the kdump kernel to take. (Blindly copying oldmem pages is one thing, manipulating live data structures in oldmem seems like quite another thing.)

Regarding the danger of trashing the kdump kernel prior to its launch:
Currently, any driver or errant kernel code can trash the kdump area. And any IO card on a non-IOMMU or swiotlb system can trash it. So it doesn't seem like much of an extension of a risk that already exists.

It does however negate one possibility to lower some of that risk.

Quoted 7 lines
Quoted 2 lines

The kdump kernel must refrain from detecting and initializing any iommu.

Why? I can fully understand avoiding addresses that are in flight. I can definitely see this being simpler in the kdump kernel. However this feels like it makes a less robust kdump kernel by not allowing it to touch the iommu.

As pointed out above, "touching the iommu" really includes touching its data structures created by the primary kernel in what is now the oldmem area. In addition, I'm not sure the kdump kernel can determine which addresses are in flight by querying either the HW or the oldmem
structures. It could probably determine which ones were unused at the
time of the crash.

Quoted 7 lines
Quoted 7 lines

This has a these effects:

A) Primary kernel: depending on what it is using for as an IOMMU, it may have to do some (or considerable) setup, to guarantee that the kdump kernel can have IO capability to its Crash kernel address range.

Quoted 5 lines
Quoted 3 lines

B) Primary kernel: the Crash kernel range must be set up in an address range whose physical addresses are accessible to IO cards without address remapping.

Below <= 16MB? That doesn't work in general.

I didn't think this was working now. Aren't most crash kernels allocated above 16 MB? And I assumed lots of systems don't have IOMMU capability. Do you have an example where this would be an issue for an IO card needed by the kdump kernel?

Quoted 2 lines

Especially not if we are running on an SGI box and someone had unplugged node 0 (with all of the memory below 4G).

How does an SGI box with an unplugged node 0 do kdump IO currently?

Quoted 7 lines
Quoted 1 line

Possible? Comments? Corrections?

Possible.

I would very much like the option of doing the iommu setup, and possibly
fiddling in the kdump kernel. As long as we are not reusing the same
addresses in the iommu I don't see a problem.

The problem I see is the oldmem area. Now we could come up with a plan to allow the primary kernel to do all of its iommu related allocations in the Crash kernel area, effectively creating an area of memory that is shared between the primary kernel and kdump. (This would be complicated in cases where the iommu state is in a dynamic tree vs. a fixed size array.) Then the kdump kernel would wake up and just take over maintenance of the iommu. But even the much smaller proposal to preallocate entries in the iommu data structures to allow the kdump kernel to do its IO is already violating one of the principals of kdump. It is making kdump operation dependent on the integrity of a primary kernel data structure. Actually taking over a shared iommu data structure from the primary kernel seems like an even bigger philosophical violation.

Quoted 3 lines

I like the theoretical option of disabling ongoing DMA's, with the more complete IOMMUs. It isn't strictly necessary but I expect it would give a better result.

It seems like this implies 1) stopping the DMA at the IOMMU, 2) surviving the resulting error condition when the IO card fails its next access (hopefully not MCE on a modern IOMMU), 3) verifying that the IO card won't try another access later after you've started using the IOMMU in the kdump kernel, and then 4) reinitializing and using the IOMMU. Is it doable?

Thanks for considering,
Bob Montgomery

_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec

#18 lore raw
In reply to Bob Montgomery
Quoted 34 lines · Bob Montgomery
Quoted 9 lines · Eric W. Biederman
Quoted 7 lines · Vivek Goyal

Few options Bob is considering are.

- Update "e820" memory map to mark GART aperture as reserved, which will be reflected in /proc/iomem also. Kexec-tools will not pass reserved area to second kernel and it will not try to dump this area.

Quoted 7 lines
Quoted 3 lines

- Mark GART aperture as "GART aperture" in /proc/iomem and modify kexec-tools to filter out this memroy from memory map passed to second kernel.

We should definitely reserve the resource, and it should definitely show up in /proc/iomem.

Reserving it as a child resource called "GART" in a "System RAM" resource is already in newer kernels than mine (at least in by 2.6.26). I haven't seen that kexec-tools does anything with that yet. kexec-tools looks for "Crash kernel" in /proc/iomem now and explicitly excludes that area.

Example:
000f0000-000fffff : System ROM
00100000-cfe4ffff : System RAM
00200000-0042635a : Kernel code
0042635b-00592037 : Kernel data
01000000-08ffffff : Crash kernel
0c000000-0fffffff : GART
cfe50000-cfe57fff : ACPI Tables
cfe58000-cfffffff : reserved

Quoted 9 lines

If it could be "reserved" earlier, so it isn't a child resource of a System Ram area, but a "reserved" area that divides two "System RAM" areas, then the current kexec-tools would exclude it (like it excludes all "reserved" areas from the /proc/vmcore map, and it would no longer be possible to trigger the MCE (or the mysterious hang) by reading from /proc/vmcore. But currently (in my older kernel) the original iomem_resource is constructed from the e820 map before I know where (and how big) the aperture will be created.

I think above /proc/iomem entries look good. It makes logical sense that "GART" is child of "System RAM".

It would be great if you could provide a patch for kexec-tools to explicitly exclude "GART" aperture from the memory map passed to second kernel.

Quoted 13 lines

But either way we fix it in iomem to exclude it from /proc/vmcore, a read of /dev/oldmem in the aperture area would still trigger the MCE. At least it does on my system.

Quoted 3 lines
Quoted 3 lines

- Disable cpu side GART access in first kernel so that even if second kernel tries to access it, it does not run into isseus.

This has the advantage of "fixing" accesses through both /proc/vmcore
and /dev/oldmem. And for me, it's an easy patch to pci-gart.c in
init_k8_gatt that just sets bit 4 instead of clearing both 4 and 5:

Review

Quoted 5 lines
(quoted diff) +3 -2
-               ctl |= 1;
-               ctl &= ~((1<<4) | (1<<5));
+               ctl |= 1;       /* set GartEn */
+               ctl |= (1<<4);  /* set DisGartCpu */
+               ctl &= ~(1<<5); /* clear DisGartIO */

This looks interesting from the point of view that it sloves the issue for /dev/oldmem also. But I am not sure if disabling cpu side access can have any side affects. Some GART/IOMMU expert needs to comment on this. You should post it as an independent patch on LKML and see if somebody can find an issue with above.

Even if disabling cpu patch is going in, I think we should still fix kexec-tools and older kernels to export "GART" info in /proc/iomem and modify kexec-tools to exclude that area.

Quoted 8 lines
Quoted 4 lines

This is an interesting one. When I looked at this years ago I had the feeling that if we did this we could actually always use a 2G Aperture at a fixed address, and require going through the gart for all of lowmem.

During discussions here, a colleague suggested that with CPU-side access of the aperture disabled, we could allocate the crash kernel in the wasted memory "under" the aperture.

Interesting. How big is the aperture? Generally it is 64MB and crash kdump kernel often requires more than that.

Thanks
Vivek

_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec