KDB in the mainstream 2.4.x kernels?

One argument i have against it: KDB is incredibly ugly code. Before it could be even considered for merging it would need quite a lot of cleanup.

I actually started on porting the KDB backtracer recently to get reliable frame pointer based backtraces, but it turns out the code for that is so complicated and ugly that the chances of ever merging it would be very slim.

As for your home crash debugging I suspect you would be better of with LKCD or Mcore (crash dump, saving an crash image on some partition, with examining the crash dump after reboot)

KDB is usually not useful for debugging hangs on desktop boxes (and even many servers) because you have usually X running. When the machine crashes and goes in KDB you cannot see the text output and debug anything. I learned to type "go<return>" blind when I had still an KDB aware kernel, but it's not very useful overall.

On a development machine you can avoid that by connecting a serial cable, but that's usually not easily possible for a desktop box. Doing a post-mortem after reboot is more practical. That is what LKCD/mcore do.

Disadvantage is that the current crash dump mechanisms (lkcd, mcore crash, netdump) are all still not very reliable and have horrible error handling. And you can eat a lot of disk space for the dumps and they tend to overflow your file systems. But still it's the only realistic option for "desktop bugs"

BTW debugging on the X server works on linuxppc/mac with xmon because it has a fbcon based console and X server. The debugger can just work on the X background while the X server is stopped. Very nifty. But I don't see the x86 XFree86 switching to a similar fbcon model any time soon, so it's unlikely to help.

-Andi

Hi Andi,

I'm happy to get a response...

What in particular? I just looked at kdb/kdbmain.c and kdb/kdb_bt.c and it looks fine to me; fairly minimal even. I don't know about arch-specific code. Is there a particular file you're complaining about?

(very very off-topic: I love reading about the neat things that reiser v4 will do, but wow, every time I read reiserfs code, 'ugly' is indeed the word that flies to mind).

?

I have not (yet?) studied the code in detail, so point me at something ugly; I'm not sure what you are talking about. Now, stack traces are in general ugly because registers and args are splattered all over the place, and the struct pt_regs are even worse. So there's some inherent ugliness there ...

Since I live in KDB, I might have some spare time to cleanup/fix,etc. so nows a good time to talk ..

I'll look ... given that I own lots flaky IDE hardware, though, I'm catious. I get 'DriveReady SeekComplete Error' messages daily ... I learned the hard way that these aren't necessarily the fault of the hard drive, and I have suffered through corrupted fs's as a result ...

Generically, for servers, if you can just save the dump, reboot, and let the server go on, and analyze the dump at leisure, that is the prefered way to do things. Especially if you are doing customer support. (Linux is at the dawn of the era of having customers who have actually spent in excess of $100K or $1M on a server and who will be going apoplectic when it crashes. This will put a spotlight on dump tools).

I'm willing to put console on serial port. I've got enough machines & serial cables, this doesn't bother me.

This statement makes me nervous. One of the worst feelings one can get when debugging is not being able to trust the data you are looking at. Its too easy to loose a lot of time (and credibility) making incorrect hypothesis based on bad data.

Dedicating a partition that is unformated, and whose sole purpose in life is to record a dump -- that is a viable option, at least on servers, where high uptime is more important, and storage is cheap.

On my home machines, its sort of the other way around: I don't trust IDE, I don't have the disk space.

But you convinced me; I need more time on lkcd, etc.

--linas

Check the kdbsupport.c code too.

All the code together for the i386 backtracer is approaching 1000 LOC and it's quite ugly.

Typically you don't need a dedicated partition, you can dump on swap. netdump does also dump over the network. This may be the safer choice when you don't trust your block subsystem after crashes.

-Andi

On Fri, 18 Jul 2003 22:43:57 +0200,

Mainly because the kernel is full of special cases and i386 provides no unwind data to help decode those special cases, so all the special case code ends up in kdba_bt.c. Compare the complexity of i386 kdba_bt.c with ia64 kdba_bt.c, the latter is significantly simpler because ia64 mandates unwind data. Without unwind data, kdb has to use lots of awkward heuristics to even guess at an accurate backtrace. Don't blame kdb for the lack of i386 unwind data.

We could tell gcc to emit dwarf2 unwind tables on x86 for debugging kernel builds.

C code is not really an issue. Most of the unwind complexity is handling the special case asm code, interrupt handlers, out of line lock contention paths, anything in entry.S. Much of the IA64 asm code has explicit unwind directives in the asm code, i386 asm would need equivalent kernel changes.

Yes I agree. It is an ugly problem, which usually results in ugly solutions too.

-Andi

I believe there was a consensus reached that the ugliness was out of necessity. I hope my understanding is correct.

I personally think _BOTH_ belong in a kernel. KDB for locating problems in my code during development and a crash dump facility so I can look at the problems that I don't believe are part of my code at a later time. Additionally, when I am no longer debugging the code, I would rather turn KDB off and leave LKCD in there to capture the particular problem I am chasing. How many problems have been encountered on user machines which have resulted in a reboot and ignore method? This is no way to fix problems!

For me, and I believe I am not alone, the most infuriating problem is the one line change in your code which "tickles" someone else's bug and I spend 3 days trying to find what really went wrong.

I am sure that nearly anyone working in the bowels of the kernel has also had a user land program that terminated with just a bus error.

I believe that this could be addressed once KDB gets into the kernel. If this were clearly stated as a condition for getting KDB in, then I am sure someone can figure a method out.

I am not sure what you mean by LKCD is not reliable. I use LKCD to create crash dumps at work all the time. The only problems I have are when the NMI doesn't propagate correctly into LKCD and initiate the dump. I believe Keith Owens has improved LKCD's understanding of being called for KDB and now that seems to always work as well.

I think the error handling being referred to is the device driver itself. That should probably get fixed on a case-by-case basis instead of putting the blame on LKCD. I don't see driving force to fixing drivers for a crash dump facility without a clear direction as to which facility will be accepted.

As for space, I view that as an admin problem. If you are selling a machine with a lot of memory, you need to size for the dump you will typically get. I recently initiated an LKCD dump on a machine with 8GB of memory and had dumping of kernel pages including buffers. The uncompressed dump output was still only 161MB. I have also taken a dump from a machine with over 128GB of memory in 64K pages shortly after booting with init=/bin/sh. It took only 108MB. That doesn't seem too large in the days of 120GB drives. This can be reduced even further by using compression (Already supported by LKCD) and not dumping kernel buffers.

I believe that crash dumps and in core debugging are useful for a lot of others as well. If not, why have Red Hat's netdump and United Linux's LKCD implementations been done? It seems the only people saying "no" to dumps and in core debugging are the people who claim to benefit from it least. I view that as equally arrogant to the PPC people being able to say no to adding machine vectors required by ia64 on a "just because" basis without justifying based on technical merit.

I agree with _EVERYTHING_ that Linas from IBM pointed out. What needs to be done to get this in?

If you want to hear user testimonials, I have used KDB for the last year. It has been invaluable in locating race conditions with spin locks when you have large processor counts and you are stress testing a machine.

I have complained bitterly (ask Keith Owens) that LKCD needs to be working. One of my primary uses of LKCD since Keith got it working for our machine has been: hitting a problem; doing a cursory check to see if it is something I introduced and debug with KDB. When I am certain I am done using KDB, I take the dump and reanalyze using lcrash to ensure the answer is consistent and do a cursory check for other failures which may be hiding behind this bug. It also gives me a method of determining if my fix introduced a new bug or if a future failure was from a long running behavior which just now got bad enough. When I hit something new, I now have "perfect recall" of the previous failure.

I believe this should be in the 2.6 kernel as well. I believe it will save time during the bringup effort for countless people.

Robin Holt

Perhaps in the case where the console is on a vt, kdb could try to switch to the right vc before presenting its prompt? I realize calling into the vc code might be risky but it seems like there's not much to lose. (If you do have a bug in say the agp driver then you need a serial
console...) If it works, you'll be able to debug and continue.

It could even set the colors to white on blue. :-)

Martin

Only the X server can switch away, because only it knows how to talk to the graphic chipset. And running user space here is far too risky.

It's possible when the resolutions are controlled by the kernel in fbcon. That's the case on linux/ppc and you can indeed debug on top of an X server there. But it's unlikely to happen for linux/x86, the xfree86 people don't want to move parts of their drivers into the kernel.

-Andi

There was a proposal a long ways back to allow X to download instructions to the kernel on how to restore the video mode. The proposal was to code the instructions as a forth program that frobbed registers appropriately. The kernel would have a small forth interpretor to run it. Then switching resolutions could happen safely in the kernel.

greg

Did the proposal come with working code?

-Andi

Not that I recall. I'm going back, uh, probably 10-15 years. But it seems as relevant today as it was then.

greg

I've seen workable non forth versions of the proposal yes. It isnt actually that hard to do for most video cards

Hi!

We could make them use code for ACPI interpretter, that's already in and has advantage that graphics people might eventually ship it in card roms....
Pavel

When do you have a heart between your knees?
[Johanka's followup: and *two* hearts?]