kdump: Fix for machine checkstop on DMA fault
PATCH 1/1 kdump: Fix for machine checkstop on DMA fault
Paul, If you are OK with this fix, please send it upstream.
Thanks
Haren
- Some machines checkstop on dma protection fault for ongoing DMA left in the first kernel. Since, we do not shutdown devices before the kdump boot, let them continue DMA to old kernel space.
arch/powerpc/kernel/iommu.c renamed from arch/powerpc/kernel/iommu.c.orig +2 -0
@@ -427,8 +427,10 @@ struct iommu_table *iommu_init_table(str tbl->it_largehint = tbl->it_halfpoint;spin_lock_init (&tbl->it_lock); +#ifndef CONFIG_CRASH_DUMP /* Clear the hardware table in case firmware left allocations in it */ ppc_md.tce_free (tbl, tbl->it_offset, tbl->it_size); +#endif if (!welcomed) { printk(KERN_INFO "IOMMU table initialized, virtual merging %s\n",
Quoted 8 lines · Haren Myneni
Paul, If you are OK with this fix, please send it upstream.
Thanks
Haren- Some machines checkstop on dma protection fault for ongoing DMA left in the first kernel. Since, we do not shutdown devices before the kdump boot, let them continue DMA to old kernel space.
How is this solved for regular kexec, doesn't the same problem exist there?
-Olof
Quoted 6 lines · Olof Johansson
Quoted 5 lines · Haren Myneni
Paul, If you are OK with this fix, please send it upstream.
Thanks
Haren
Quoted 6 lines
Quoted 3 lines
- Some machines checkstop on dma protection fault for ongoing DMA left in the first kernel. Since, we do not shutdown devices before the kdump boot, let them continue DMA to old kernel space.
How is this solved for regular kexec, doesn't the same problem exist there?
The idea for normal kexec is that the kernel should have shut everything down properly. It's a bug if there are still DMAs going on. Hopefully.
cheers
Signature · 8 lines
Michael Ellerman IBM OzLabs wwweb: http://michael.ellerman.id.au phone: +61 2 6212 1183 (tie line 70 21183) We do not inherit the earth from our ancestors, we borrow it from our children. - S.M.A.R.T Person
Quoted 11 lines · Michael Ellerman
Quoted 10 lines · Olof Johansson
Quoted 8 lines · Haren Myneni
Paul, If you are OK with this fix, please send it upstream.
Thanks
Haren- Some machines checkstop on dma protection fault for ongoing DMA left in the first kernel. Since, we do not shutdown devices before the kdump boot, let them continue DMA to old kernel space.
Quoted 5 lines
Quoted 2 lines
How is this solved for regular kexec, doesn't the same problem exist there?
The idea for normal kexec is that the kernel should have shut everything down properly. It's a bug if there are still DMAs going on. Hopefully.
Thanks Michael.
In that case, I have to NACK the original patch.
Out of luck, it'll probably work in most cases, but there's always the risk of a DMA still going on, the crash kernel remapping an entry, and getting memory scribbled over.
The crash kernel needs to be even more careful, and instead read out the entries that are mapped and reserve them. This would require a bit more plumbing since there's no way to read an entry right now, but it'd remove that hole.
-Olof
Quoted 4 lines · Olof Johansson
The crash kernel needs to be even more careful, and instead read out the entries that are mapped and reserve them. This would require a bit more plumbing since there's no way to read an entry right now, but it'd remove that hole.
Actually, what's probably easier is to allocate some entries when the purgatory is set up, and make the crash kernel only use those by modifying the device tree accordingly. Sort of how regular memory is handled right now. That'd be a cleaner solution with less changes needed.
The trick will be to get a decent size contiguous allocation, but the same applies for the memory reserve.
-Olof
Hi,
Quoted 4 lines · Haren Myneni
On JS21, immediately after the tce entries are initialized, the machine checkstops with an error "Internal CPU 1 Fault Error" on bladecenter MM. If we do not initialize tce entries for crash kernel, allows the ongoing DMA continue to the old kernel memory. I though that, ongoing DMA will be
The problem isn't when DMA is going to the old kernel memory. The problem is when that TCE entry gets reused by the crashdump kernel, and some other memory gets overwritten instead.
Quoted 3 lines
stopped when the device reset happens later by the drivers. I think, some hardening is already included in some drivers to take care of this behavior. I might be wrong. So far, I had e100 issue after testing on p5,
What assures that the crash kernel has drivers for all hardware in the system? If there's no driver, what will then be used to quiesce the device?
Quoted 3 lines
p4, js20 and js21. Probably, it could be lucky scenario.
So, will be keeping the same change (posted here) plus your suggestion. Right? Can we apply same approach even for power-4?
What you have now might be a 99%-of-the-time-it-works solution, but is that really good enough?
The last things you want from a crash kernel is:
1. Have it crash on it's own because of something getting overwritten (small chance, since most mappings are probably for writing out data for later analysis)
or:
2. Have it write corrupted data to the crash dump. This makes it more or less useless, since you can't trust what it wrote out: Did the machine go down because of the memory corruption you're spotting, or did that happen after the crash, while dumping it, etc?
Either way, a proper solution is needed, not a 99% one.
-Olof
[email protected] wrote on 03/23/2006 12:12:58 PM:
Quoted 5 lines · Olof Johansson
Quoted 4 lines
The crash kernel needs to be even more careful, and instead read out the entries that are mapped and reserve them. This would require a bit more plumbing since there's no way to read an entry right now, but
it'd
Quoted 4 lines
Quoted 1 line
remove that hole.
Actually, what's probably easier is to allocate some entries when the purgatory is set up, and make the crash kernel only use those by
modifying
Quoted 5 lines
the device tree accordingly. Sort of how regular memory is handled right now. That'd be a cleaner solution with less changes needed.
The trick will be to get a decent size contiguous allocation, but the same applies for the memory reserve.
Olof, Thanks for your comments/suggestions.
On JS21, immediately after the tce entries are initialized, the machine checkstops with an error "Internal CPU 1 Fault Error" on bladecenter MM. If we do not initialize tce entries for crash kernel, allows the ongoing DMA continue to the old kernel memory. I though that, ongoing DMA will be stopped when the device reset happens later by the drivers. I think, some hardening is already included in some drivers to take care of this behavior. I might be wrong. So far, I had e100 issue after testing on p5, p4, js20 and js21. Probably, it could be lucky scenario.
So, will be keeping the same change (posted here) plus your suggestion. Right? Can we apply same approach even for power-4?
Thanks
Haren
Quoted 7 lines
-Olof
_______________________________________________
Linuxppc-dev mailing list
[email protected]
https://ozlabs.org/mailman/listinfo/linuxppc-dev
Quoted 7 lines · Olof Johansson
Quoted 4 lines · Olof Johansson
The crash kernel needs to be even more careful, and instead read out the entries that are mapped and reserve them. This would require a bit more plumbing since there's no way to read an entry right now, but it'd remove that hole.
Quoted 7 lines
Actually, what's probably easier is to allocate some entries when the purgatory is set up, and make the crash kernel only use those by modifying the device tree accordingly. Sort of how regular memory is handled right now. That'd be a cleaner solution with less changes needed.
The trick will be to get a decent size contiguous allocation, but the same applies for the memory reserve.
I disagree. In most cases the kdump kernel will be loaded by the boot scripts, so reserving TCE space then is ~= reserving it for the life of the first kernel. Given that TCE space is a scarce commodity I don't think reserving it in the first kernel is a viable option.
What we should do is modify the second kernel so that instead of clearing the TCE tables it instead walks the tables and detects existing mappings, and then marks those as reserved so they're not overwritten.
This should give us 100% safety from the second kernel reusing a mapping and copping a rogue DMA, and doesn't inflict any penalty on the first kernel. It does fall down if there's no TCE space left for a device when the second kernel comes up, but I think that's the best trade off.
cheers
Signature · 8 lines
Michael Ellerman IBM OzLabs wwweb: http://michael.ellerman.id.au phone: +61 2 6212 1183 (tie line 70 21183) We do not inherit the earth from our ancestors, we borrow it from our children. - S.M.A.R.T Person
Quoted 4 lines · Michael Ellerman
I disagree. In most cases the kdump kernel will be loaded by the boot scripts, so reserving TCE space then is ~= reserving it for the life of the first kernel. Given that TCE space is a scarce commodity I don't think reserving it in the first kernel is a viable option.
Well, hopefully the kdump kernel doesn't need as much table space as a regular kernel, so the loss would be limited, but if you're willing to do the reserve instead; that'd be better.
Quoted 3 lines
What we should do is modify the second kernel so that instead of clearing the TCE tables it instead walks the tables and detects existing mappings, and then marks those as reserved so they're not overwritten.
Yep, that's exactly what my first proposal was.
Quoted 4 lines
This should give us 100% safety from the second kernel reusing a mapping and copping a rogue DMA, and doesn't inflict any penalty on the first kernel. It does fall down if there's no TCE space left for a device when the second kernel comes up, but I think that's the best trade off.
Correct. The time it could be a disadvantage is when there's been a driver bug that leaks mappings that causes the machine to go down (i.e. into the kdump kernel). Whatever device has been leaking might not be usable since the table will be 100% full.
-Olof