Dump driver module
Hi all.
I've already wrote a dump modules driver for linux-parisc. This module goal is to create a memory image on a swap area, and at reboot time to save it to disk with all kernel modules, in order to analyze it after by "support" people with tools like gdb/p4. The problem while dumping is that the dump modules cannot trust anymore the system, so dumping means: no interruption, no disk driver, no buffer, nothing. The solution is to use low level call. For parisc I use the IODC calls, for ia64 I think I'll use the EFI calls. My questions:
-do you think it is a "good" and realistic solution. -because I think "yes", does exist somewhere an example of reading/writing with EFI call on disk using BLOCK IO. I've already looked in elilo, but it seems that it use FS access.
Thanks.
Signature · 3 lines
Vidal Bruno, (770-4271)
SSD-HA Team, HP-UX & LINUX Support
[email protected]Salut Bruno,
Quoted 10 lines · Bruno Vidal
I've already wrote a dump modules driver for linux-parisc. This module goal is to create a memory image on a swap area, and at reboot time to save it to disk with all kernel modules, in order to analyze it after by "support" people with tools like gdb/p4. The problem while dumping is that the dump modules cannot trust anymore the system, so dumping means: no interruption, no disk driver, no buffer, nothing. The solution is to use low level call. For parisc I use the IODC calls, for ia64 I think I'll use the EFI calls. My questions:
-do you think it is a "good" and realistic solution.
I don't think it is because, I think you are missing an important feature of EFI. The last step taken by elilo before jumping to the first instruction of the kernel is to switch EFI into runtime mode (compared to boot time mode intially). There is no way back. In runtime mode, almost none of the EFI services you get at boot time remain available. The code/data memory areas are reclaimed by the kernel. The parts of EFI that remain are tagged 'runtime' and mostly consist of a subset of what you have at bootime: reset, gettimeofday, runtime drivers (fpswa). None of the I/O support is left.
the kernel, EFI is
Quoted 3 lines
-because I think "yes", does exist somewhere an example of reading/writing with EFI call on disk using BLOCK IO. I've already looked in elilo, but it seems that it use FS access.
Not quite. If you looked at the ext2 emulation, it uses the BlockIO protocol.
Signature · 1 line
-Stephane
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line · Bruno Vidal
Bruno> I've already wrote a dump modules driver for linux-parisc. Bruno> This module goal is to create a memory image on a swap area, and Bruno> at reboot time to save it to disk with all kernel modules, in Bruno> order to analyze it after by "support" people with tools like Bruno> gdb/p4. The problem while dumping is that the dump modules Bruno> cannot trust anymore the system, so dumping means: no interruption, Bruno> no disk driver, no buffer, nothing. The solution is to use low level Bruno> call. For parisc I use the IODC calls, for ia64 I think I'll Bruno> use the EFI calls. My questions:
Bruno> -do you think it is a "good" and realistic solution. Bruno> -because I think "yes", does exist somewhere an example of Bruno> reading/writing with EFI call on disk using BLOCK IO. I've already Bruno> looked in elilo, but it seems that it use FS access.
Once the kernel is booted, you can access only the EFI runtime services (see struct efi in include/linux/efi.h). I'm afraid there is no support for writing disk blocks. Given this limitation, I suspect you'd have to include your own (simple) disk driver(s).
--david
Quoted 17 lines · David Mosberger
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 1 line · Bruno Vidal
Bruno> I've already wrote a dump modules driver for linux-parisc. Bruno> This module goal is to create a memory image on a swap area, and Bruno> at reboot time to save it to disk with all kernel modules, in Bruno> order to analyze it after by "support" people with tools like Bruno> gdb/p4. The problem while dumping is that the dump modules Bruno> cannot trust anymore the system, so dumping means: no interruption, Bruno> no disk driver, no buffer, nothing. The solution is to use low level Bruno> call. For parisc I use the IODC calls, for ia64 I think I'll Bruno> use the EFI calls. My questions:
Bruno> -do you think it is a "good" and realistic solution. Bruno> -because I think "yes", does exist somewhere an example of Bruno> reading/writing with EFI call on disk using BLOCK IO. I've already Bruno> looked in elilo, but it seems that it use FS access.
Quoted 9 lines
Once the kernel is booted, you can access only the EFI runtime services (see struct efi in
include/linux/efi.h). I'm afraid there is no support for writing disk blocks. Given this limitation, I suspect you'd have to include your own (simple) disk driver(s).--david.
I've already seen this EFI runtime service, and there is nearly nothing in it (nothing usefull for IO), so what I can understand is I have to write (rewrite) a driver for each kind of scsi/ide interface....it is not really realistic :-( There is no other way to access EFI or firmware ? Or reset back EFI to have access to BLOCK IO interface ? For example in parisc, the first thing to do, is to a do a PDC_IO_RESET, it freeze the entire system and it is possible to have access to all firmware calls. Does exist something similar with EFI.
Thanks.
Signature · 3 lines
Vidal Bruno, (770-4271)
SSD-HA Team, HP-UX & LINUX Support
[email protected]Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line · Bruno Vidal
Bruno> There is no other way to access EFI or firmware ?
The EFI standard certainly won't help you here. Whether or not vendors have their own backdoors to a low-level disk driver, I don't know.
Bruno> Or reset back EFI to have access to BLOCK IO interface ?
You can warm-boot via EFI. But I'm not sure there is any guarantee that all kernel memory will be preserved.
--david
Have you looked at the LKCD project? It supports ia64 and does about what you are doing on the dump side, and has dump analysis tools.
If you've already considered this just delete this.
Thanks,
Dave Howell
These are my opinions and not official opinions of Intel Corp.
David Howell
Intel Corporation
Telco Server Development
Server Products Division
Voice: (803) 461-6112 Fax: (803) 461-6292
Intel Corporation
Columbia Design Center, CBA-2
250 Berryhill Road, Suite 100
Columbia, SC 29210
-----Original Message-----
From: Bruno Vidal [mailto:[email protected]]
Sent: Tuesday, May 13, 2003 3:03 AM
To: [email protected]
Subject: [Linux-ia64] Dump driver module
Hi all.
I've already wrote a dump modules driver for linux-parisc. This module goal is to create a memory image on a swap area, and at reboot time to save it to disk with all kernel modules, in order to analyze it after by "support" people with tools like gdb/p4. The problem while dumping is that the dump modules cannot trust anymore the system, so dumping means: no interruption, no disk driver, no buffer, nothing. The solution is to use low level call. For parisc I use the IODC calls, for ia64 I think I'll use the EFI calls. My questions:
-do you think it is a "good" and realistic solution. -because I think "yes", does exist somewhere an example of reading/writing with EFI call on disk using BLOCK IO. I've already looked in elilo, but it seems that it use FS access.
Thanks.
Signature · 9 lines
Vidal Bruno, (770-4271)
SSD-HA Team, HP-UX & LINUX Support
[email protected]
_______________________________________________
Linux-IA64 mailing list
[email protected]
http://lists.linuxia64.org/lists/listinfo/linux-ia64 Hi
Sorry, but LKCD is really not usefull because it use the buffer to write on the device. So it just hang in case of data page fault for example (because interuption are masked), LKCD is also not working in case of buffer corruption, disk driver pb, etc.... so LKCD is not usable in lot's of case (and it happen really often to have data page fault).
Cheers.
Quoted 48 lines · David P
Have you looked at the LKCD project? It supports ia64 and does about what you are doing on the dump side, and has dump analysis tools.
If you've already considered this just delete this.
Thanks,
Dave HowellThese are my opinions and not official opinions of Intel Corp.
David Howell
Intel Corporation
Telco Server Development
Server Products Division
Voice: (803) 461-6112 Fax: (803) 461-6292Intel Corporation
Columbia Design Center, CBA-2
250 Berryhill Road, Suite 100
Columbia, SC 29210-----Original Message-----
From: Bruno Vidal [mailto:[email protected]]
Sent: Tuesday, May 13, 2003 3:03 AM
To: [email protected]
Subject: [Linux-ia64] Dump driver moduleHi all.
I've already wrote a dump modules driver for linux-parisc. This module goal is to create a memory image on a swap area, and at reboot time to save it to disk with all kernel modules, in order to analyze it after by "support" people with tools like gdb/p4. The problem while dumping is that the dump modules cannot trust anymore the system, so dumping means: no interruption, no disk driver, no buffer, nothing. The solution is to use low level call. For parisc I use the IODC calls, for ia64 I think I'll use the EFI calls. My questions:(quoted diff) +0
-2-do you think it is a "good" and realistic solution. -because I think "yes", does exist somewhere an example ofreading/writing with EFI call on disk using BLOCK IO. I've already looked in elilo, but it seems that it use FS access.
Thanks.
Signature · 3 lines
Vidal Bruno, (770-4271)
SSD-HA Team, HP-UX & LINUX Support
[email protected]Quoted 5 lines · Bruno Vidal
Sorry, but LKCD is really not usefull because it use the buffer to write on the device. So it just hang in case of data page fault for example (because interuption are masked), LKCD is also not working in case of buffer corruption, disk driver pb, etc.... so LKCD is not usable in lot's of case (and it happen really often to have data page fault).
lkcd should not get data page faults. Are you sure that is what you are seeing and not some other problem that hangs lkcd?
Quoted 6 lines · Bruno Vidal
Hi
Sorry, but LKCD is really not usefull because it use the buffer to write on the device. So it just hang in case of data page fault for example (because interuption are masked), LKCD is also not working in case of buffer corruption, disk driver pb, etc.... so LKCD is not usable in lot's of case (and it happen really often to have data page fault).
There is also the netconsole/netdump system. This is supposed to rely on only a very minimal network driver. As you say, writing to disk when kernel memory may have been corrupted is a risky business, not only because you might hang but also because you might write over the wrong region. netdump doesn't do any disk IO for that reason. I seem to recall that Linus liked this property.
Of course it's only useful if the machine is on a network where there is another machine to catch the data, but I would expect that to be the case for most ia64 machines.
http://www.redhat.com/support/wpapers/redhat/netdump/
Signature · 1 line
Martin
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line · Martin Pool
Quoted 6 lines · Bruno Vidal
Quoted 6 lines
Hi
Sorry, but LKCD is really not usefull because it use the buffer to write on the device. So it just hang in case of data page fault for example (because interuption are masked), LKCD is also not working in case of buffer corruption, disk driver pb, etc.... so LKCD is not usable in lot's of case (and it happen really often to have data page fault).
Martin> There is also the netconsole/netdump system. This is supposed to rely Martin> on only a very minimal network driver. As you say, writing to disk Martin> when kernel memory may have been corrupted is a risky business, not Martin> only because you might hang but also because you might write over the Martin> wrong region. netdump doesn't do any disk IO for that reason. I seem Martin> to recall that Linus liked this property.
Martin> Of course it's only useful if the machine is on a network where there Martin> is another machine to catch the data, but I would expect that to be Martin> the case for most ia64 machines.
Martin> http://www.redhat.com/support/wpapers/redhat/netdump/
It strikes me that for a really reliable crash-dump, you'd want to read _all_ the bits needed to do the dumping from ROM. Assuming you have a (minimal) disk driver/network driver written in EFI byte code, all you'd need is the EFI byte-code interpreter which hopefully would fit in a fixed (and reasonable) amount of space. Hence you could reserve some memory for this purpose and on a crash, load the byte-code interpreter from ROM and get going that way.
--david
Quoted 2 lines · David Mosberger
It strikes me that for a really reliable crash-dump, you'd want to read _all_ the bits needed to do the dumping from ROM.
But you also need to know which disk to write to, and where. And presumably "which disk" actually means knowing the PCI path to the SCSI host adapter, and then the path to the disk. "Where" might be just the index of start block, if you reserved a contiguous partition, or it might be more complex. If any of this got screwed up, you might write your dump over /home instead -- and then rather than a possibly once-off kernel crash, you've lost lots of data.
So for a really reliable dump, you'd want to check and "lock down" these variables at startup into a region of memory not easily changed by the kernel, and accessible by EFI. NVRAM? I don't know if there's anything suitable on IPF.
It would be an interesting project, although since network dumps run a lower risk of clobbering filesystems they're probably a smarter design. It would be cool to do it in EFI, if that's possible.
Quoted 5 lines
Assuming you have a (minimal) disk driver/network driver written in EFI byte code, all you'd need is the EFI byte-code interpreter which hopefully would fit in a fixed (and reasonable) amount of space. Hence you could reserve some memory for this purpose and on a crash, load the byte-code interpreter from ROM and get going that way.
Signature · 1 line
Martin
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line · Martin Pool
Martin> So for a really reliable dump, you'd want to check and "lock Martin> down" these variables at startup into a region of memory not Martin> easily changed by the kernel, and accessible by EFI. NVRAM? Martin> I don't know if there's anything suitable on IPF.
EFI provides (persistent) environment variables for this purpose. Don't know how well they're protected, typically, but that's probably more of an implementation issue than anything else.
Martin> It would be an interesting project, although since network Martin> dumps run a lower risk of clobbering filesystems they're Martin> probably a smarter design. It would be cool to do it in Martin> EFI, if that's possible.
"Smart" is relative. I'm typing this on an Itanium box which doesn't have anything on the network that could serve as a dump device (it's own disk would be fine, though).
--david
Quoted 1 line · Martin Pool
But you also need to know which disk to write to, and where.
...
Quoted 3 lines
If any of this got screwed up, you might
write your dump over /home instead -- and then rather than a possibly once-off kernel crash, you've lost lots of data.
I'll take that risk since I've never seen it nor heard of it happening with HPUX after ~6 years of working HPUX IO subsystems. I'm sure HPUX has done things to help mitigate that risk. (like storing paths in NVRAM or some other hard to reach place.) All problem reports (including memory corruption problems) came with crash dumps.
Besides, fsck after the reboot is much more likely to trash data on disk than the original memory corruption problem.
While it's clearly feasible to dump over gigabit networks, I'm not sure how pervasive gigabit networks are. Probably standard now for core commercial/scientific centers. I fear dumping 4GB of memory over 100BT would be painful. (I pulled 4GB out of a hat; rx5670 supports 48GB of RAM).
grant
Hi
Many thanks, it seems that my question was interresting. I just want to answer to some point:
Quoted 8 lines
But you also need to know which disk to write to, and where. And presumably "which disk" actually means knowing the PCI path to the SCSI host adapter, and then the path to the disk. "Where" might be just the index of start block, if you reserved a contiguous partition, or it might be more complex. If any of this got screwed up, you might write your dump over /home instead -- and then rather than a possibly once-off kernel crash, you've lost lots of data.
HP-UX is already doing that. It keep in memory where to put the dump, and the best place is swap partition. And it is really easy to verify before writing that we are at the right place by checking any swap magic number in order to avoid any overwriting (with linux it is really easy to check that). And about the PCI path, it is not so hard to find it, and I saw that EFI provide a function to retrieve the media ID by using the hardware path. Now it is clear that in case of hardware failure their is still cases where it avoid to take dumps, but I never heard/seen HP-UX overwriting data while dumping, even in case of memory corruption.
Now there is some other solution in order to take a dump. It is another path I didn't try yet but if you can say what you think about it:
1. dump module reserve some amount of memory at the end of the memory (1Mb)
2. system call panic()
3. dump module start to compress memory, starting by the last address (without the pages reserved by the module), and put the result in the last reserved pages
4. free the compressed pages, and then compress new pages to put it in freed pages and so on until no more pages to compress
5. put a flag somewhere in NVRAM (or elsewhere)
6. do a warm boot in order to not reset the memory, and the compress memory dump pages must be marked as allocated be the new kernel.
7. now we have a brand new running kernel with all drivers and a bunch of memory pages that contain the dump, so we can push it directly on filesystem
The main problem with this kind of dump, is that it is not possible to takes early dumps. It means also that firmware is able to reboot without modifying any memory pages, it mean also that the kernel is able at boot time to flag a large amount of pages as allocated. But the good effects are: a quick dump (only memory access), a system up and running fairly fast (with less memory at boot time but for a short periode of time). It can be a good choice for critical application that require a low donwtime, and that have often a large amount of memory (if you already try to dump a superdome with 64Gb by using PDC call you understand what I mean).
Cheers.
Signature · 3 lines
Vidal Bruno, (770-4271)
SSD-HA Team, HP-UX & LINUX Support
[email protected]Quoted 7 lines · Grant Grundler
Quoted 1 line · Martin Pool
But you also need to know which disk to write to, and where.
...
Quoted 3 lines
If any of this got screwed up, you might
write your dump over /home instead -- and then rather than a possibly once-off kernel crash, you've lost lots of data.
Quoted 6 lines
I'll take that risk since I've never seen it nor heard of it happening with HPUX after ~6 years of working HPUX IO subsystems. I'm sure HPUX has done things to help mitigate that risk. (like storing paths in NVRAM or some other hard to reach place.) All problem reports (including memory corruption problems) came with crash dumps.
But those HPUX systems were on PA which has PDC which can write to the disc. The problem on ia64 is that EFI can't after the OS has started.
I seem to remember lamont eulogising the MPE way of doing crashdumps -- Transfer of Control really does transfer control to a completely different kernel that does the dump for you. That seems a much better way of doing things to me.
Signature · 3 lines
"It's not Hollywood. War is real, war is primarily not about defeat or victory, it is about death. I've seen thousands and thousands of dead bodies. Do you think I want to have an academic debate on this subject?" -- Robert Fisk
Quoted 2 lines · Matthew Wilcox
But those HPUX systems were on PA which has PDC which can write to the disc. The problem on ia64 is that EFI can't after the OS has started.
yup - I understood that. The "where to write" information getting corrupted is the same problem regardless of where the driver code to do the writes exists.
grant