Plea for help

I posted a couple of weeks ago about a kernel panic that shows up when I boot my Abit VP6 (dual-PIII) with ACPI turned on. It turns out that the kernel boots just fine with ACPI on when I build it with CONFIG_DEBUG_SLAB=y.

Alan Cox suggested that this is probably due to code that kmallocs memory and then relies on its content without clearing it, but I never received any response to my request for suggestions on how to identify the offending code.

I've just spent several hours adding printk statements to the ACPI code, and I've reached a point where the oops message is overwriting portions of my printk messages. I have the sinking feeling that this indicates that the process executing the ACPI code is not the one that's oops'ing.

I'm stuck. How can a relative newbie debug a problem like this, or at least gather enough information to enable someone with more expertise to do so?

TIA.

========================================================================
Ian Pilcher                                        [email protected]
========================================================================



-------------------------------------------------------
This SF.net email is sponsored by: IBM Linux Tutorials.
Become an expert in LINUX or just sharpen your skills.  Sign up for IBM's
Free Linux Tutorials.  Learn everything from the bash shell to sys admin.
Click now! http://ads.osdn.com/?ad_id=1278&alloc_id=3371&op=click

If you say that CONFIG_DEBUG_SLAB=y solves the problem, you could start by grepping in the kernel for CONFIG_DEBUG_SLAB and see what its doing. This sounds like one of two things.
Either CONFIG_DEBUG_SLAB=y clears some memory which isn't cleared properly somewhere else (there are quite a few kernel structures that are allocated as slabs iirc), or its a timing problem that CONFIG_DEBUG_SLAB solves by adding a printk somewhere that causes a short delay (the second one will probably be a lot harder to solve). I'll see if I have some time to look at it later.

-------------------------------------------------------
This SF.net email is sponsored by: IBM Linux Tutorials. Become an expert in LINUX or just sharpen your skills. Sign up for IBM's Free Linux Tutorials. Learn everything from the bash shell to sys admin. Click now! http://ads.osdn.com/?ad_id=1278&alloc_id=3371&op=click

I had a brief look at what CONFIG_DEBUG_SLAB=y option does. It looks that beside adding some debug output and sanity checks it also initializes the memory to a non null pattern (poison something). You could start with setting that pattern to 0x00 instead of what it is and see if that returns your oopses (don't know where it points, but thats probably won't do anything but could give you a null pointer access). Otherwise I would follow the changes (it changes the DEBUG flag and don't remember what else) and see if it changes anything (other then the poison thing) or just read tests the results and prints error messages. If that is the case than what you probably have is a race condition causing some kind of timing problem.

-------------------------------------------------------
This SF.net email is sponsored by: IBM Linux Tutorials. Become an expert in LINUX or just sharpen your skills. Sign up for IBM's Free Linux Tutorials. Learn everything from the bash shell to sys admin. Click now! http://ads.osdn.com/?ad_id=1278&alloc_id=3371&op=click

You can see the panic at

http://home.comcast.net/~i.pilcher/panics/1/panic.txt

The config used to produce it is at

http://home.comcast.net/~i.pilcher/panics/1/config

Unfortunately, if I turn CONFIG_FRAME_POINTER on I don't get the panic, just the hang at

http://home.comcast.net/~i.pilcher/panics/2/hang.txt

Config at

http://home.comcast.net/~i.pilcher/panics/2/config

And as I previously mentioned, turning CONFIG_DEBUG_SLAB on prevents the panic entirely.

========================================================================
Ian Pilcher                                        [email protected]
========================================================================



-------------------------------------------------------
This SF.net email is sponsored by: IBM Linux Tutorials.
Become an expert in LINUX or just sharpen your skills.  Sign up for IBM's
Free Linux Tutorials.  Learn everything from the bash shell to sys admin.
Click now! http://ads.osdn.com/?ad_id=1278&alloc_id=3371&op=click

On Fri, 19 Dec 2003 18:05:05 -0600 Ian Pilcher <[email protected]> wrote:

| Brian Perkins wrote:
| > First of all, what's the panic like?
| > Does it give a function?
| > You do have all of the kernel hacking options on, don't you?
| > 
| 
| You can see the panic at
| 
|    http://home.comcast.net/~i.pilcher/panics/1/panic.txt
| 
| The config used to produce it is at
| 
|    http://home.comcast.net/~i.pilcher/panics/1/config
| 
| Unfortunately, if I turn CONFIG_FRAME_POINTER on I don't get the panic,
| just the hang at
| 
|    http://home.comcast.net/~i.pilcher/panics/2/hang.txt
| 
| Config at
| 
|    http://home.comcast.net/~i.pilcher/panics/2/config
| 
| And as I previously mentioned, turning CONFIG_DEBUG_SLAB on prevents
| the panic entirely.

Can you also put the System.map file for that kernel on your website so that we can correlate kernel memory addresses?

--
~Randy
MOTD: Always include version info.

-------------------------------------------------------
This SF.net email is sponsored by: IBM Linux Tutorials. Become an expert in LINUX or just sharpen your skills. Sign up for IBM's Free Linux Tutorials. Learn everything from the bash shell to sys admin. Click now! http://ads.osdn.com/?ad_id=1278&alloc_id=3371&op=click

They're up now. The System.map for the panic (CONFIG_FRAME_POINTER off) is at

http://home.comcast.net/~i.pilcher/panics/1/System.map.txt

The map for the hang (CONFIG_FRAME_POINTER on) is at

http://home.comcast.net/~i.pilcher/panics/2/System.map.txt

========================================================================
Ian Pilcher                                        [email protected]
========================================================================



-------------------------------------------------------
This SF.net email is sponsored by: IBM Linux Tutorials.
Become an expert in LINUX or just sharpen your skills.  Sign up for IBM's
Free Linux Tutorials.  Learn everything from the bash shell to sys admin.
Click now! http://ads.osdn.com/?ad_id=1278&alloc_id=3371&op=click

On Fri, 19 Dec 2003 19:09:22 -0600 Ian Pilcher <[email protected]> wrote:

| Randy.Dunlap wrote:
| 
| > Can you also put the System.map file for that kernel on your website
| > so that we can correlate kernel memory addresses?
| 
| They're up now.  The System.map for the panic (CONFIG_FRAME_POINTER
| off) is at
| 
|    http://home.comcast.net/~i.pilcher/panics/1/System.map.txt
| 
| The map for the hang (CONFIG_FRAME_POINTER on) is at
| 
|    http://home.comcast.net/~i.pilcher/panics/2/System.map.txt

and exactly what kernel version is this?

--
~Randy

-------------------------------------------------------
This SF.net email is sponsored by: IBM Linux Tutorials. Become an expert in LINUX or just sharpen your skills. Sign up for IBM's Free Linux Tutorials. Learn everything from the bash shell to sys admin. Click now! http://ads.osdn.com/?ad_id=1278&alloc_id=3371&op=click

2.4.23 with the 20031203 ACPI patch. I get basically the same behavior with unpatched 2.4.23 or any of the Fedora Core kernels.

========================================================================
Ian Pilcher                                        [email protected]
========================================================================



-------------------------------------------------------
This SF.net email is sponsored by: IBM Linux Tutorials.
Become an expert in LINUX or just sharpen your skills.  Sign up for IBM's
Free Linux Tutorials.  Learn everything from the bash shell to sys admin.
Click now! http://ads.osdn.com/?ad_id=1278&alloc_id=3371&op=click

On Sat, 20 Dec 2003 13:47:12 -0600 Ian Pilcher <[email protected]> wrote:

| Randy.Dunlap wrote:
| > 
| > and exactly what kernel version is this?
| > 
| 
| 2.4.23 with the 20031203 ACPI patch.  I get basically the same behavior
| with unpatched 2.4.23 or any of the Fedora Core kernels.

Good. I only have unpatched 2.4.23 available currently (or 2.6.x).

Is there some reason that you build a kernel with both APM and ACPI support? Can you test it without APM support, to see if it changes any?

The dmesg files aren't complete boot logs. Can you provide complete boot logs for these 2 cases (panic & hang)?

--
~Randy

-------------------------------------------------------
This SF.net email is sponsored by: IBM Linux Tutorials. Become an expert in LINUX or just sharpen your skills. Sign up for IBM's Free Linux Tutorials. Learn everything from the bash shell to sys admin. Click now! http://ads.osdn.com/?ad_id=1278&alloc_id=3371&op=click

Sorry about the delay. I had to get all of these kernels, config files, etc. under control. (I now have a new appreciation for EXTRAVERSION.) As a result, I've got three completely new sets of data for your holiday enjoyment.

The starting point is the i686-smp kernel configuration from Fedora Core 1 (which has both APM and ACPI turned on). I've built kernel 2.4.23 with ACPI 20031203 with this configuration and called it 2.4.23fc_config. Then I disabled APM support and built 2.4.23no_apm. Finally, I turned on frame pointers and build 2.4.23frame_ptrs.

The kernel configs, System.map files, and boot logs for all three of these kernel are in http://home.comcast.net/~i.pilcher/panics.tar.gz.

I knew I wasn't going to get through this without buying a null-modem cable. The boot logs in the above tarball were all captured on my laptop; I hope that's what you're looking for. Unfortunately, the text of the panic message is garbled in several of the logs.

I also found that the behavior changes depending on whether I use a local console (in addition to the serial console) or not. In the case of the fc_config kernel, the system actually boots successfully if I use *only* a serial console. Check the kernel command line in each boot log to see what's what.

Am I crazy to think that this almost has to be a race condition?

I'll be out of town until the 29th without access to this system. I want to say thanks for taking the time to respond to my "plea".

Happy Holidays!

========================================================================
Ian Pilcher                                        [email protected]
========================================================================



-------------------------------------------------------
This SF.net email is sponsored by: IBM Linux Tutorials.
Become an expert in LINUX or just sharpen your skills.  Sign up for IBM's
Free Linux Tutorials.  Learn everything from the bash shell to sys admin.
Click now! http://ads.osdn.com/?ad_id=1278&alloc_id=3371&op=click