Skip to contentSkip to conversation

BISECTED: Re: source line numbers with x86_64 modules?

DISCUSSION Dave Anderson 4 participants 4 messages Jan 27, 2009 1 message not in the archive

lore
Share & tools
lore

A message that is not in the archive (sent off-list or not yet received): 952123357.823421233089955989.JavaMail.root@zmail02.collab.prod.int.phx2.redhat.com

#1 lore raw
Quoted 29 lines · ----- "Eric W. Biederman
Quoted 27 lines · Mike Snitzer

[I've trimmed the wide cc distribution that was inherited when I forked a different thread]

Quoted 5 lines · Eric W. Biederman
Quoted 3 lines · Mike Snitzer

Now if only I could fix line numbers when debugging crashes in x86_64 modules with the crash utility! :)

Quoted 7 lines

It's a userspace problem...

All of the little usability things are userspace problems.

I won't claim that it is trivial because it is a userspace problem, at the same time there is no reason to wait for any kernel features to merge etc. Someone just has to scratch an itch and go fix it.

Yes, the crash utility (userspace) is clearly having problems getting line number for symbols in x86_64 modules. But I finally took some time to bisect the point in the kernel where the crash utility first started to fail, it appears to be:

commit 7460ed2844ffad7141e30271c0c3da8336e66014

Author: john stultz <[email protected]>
Date:   Fri Feb 16 01:28:21 2007 -0800
Quoted 8 lines
Quoted 4 lines

I used version 4.0-7.6 of the crash utility to test if each commit was good or bad. I simply checked if ext3's module had correct line number info for the ext3_get_blocks_handle symbol with: sym ext3_get_blocks_handle

Weird. That patch doesn't appear to affect anything in that area. So my stab in the dark is that there is something in vmlinux that crash doesn't know how to cope with.

Actually it's not a problem with the vmlinux file, but rather with kernel module object files. The crash utility has an embedded gdb module which is invoked as "gdb vmlinux", and to get line numbers, the crash utility simply uses the relevant built-in gdb function to get them. And line numbers work fine with the base kernel code from the vmlinux file.

The debuginfo data of kernel modules can be subsequently added to the crash session by doing a gdb "add-symbol-file" command for any or all kernel modules. But getting correct line number information for kernel modules has been a crap-shoot in the past, depending upon architecture and/or kernel version. For example, they don't work with 2.6.9-based RHEL4 x86_64 kernel modules, but work fine with 2.6.18-based RHEL5 x86_64 kernels.

Looking at Mike's suspect kernel patch list, I don't see anything that would have any relationship to the issue. Perhaps there was a build tool change during the same timeframe?

Dave

#2 lore raw
Quoted 7 lines · On Tue, 27 Jan 2009 16:00:19 -0500 (EST) Dave Anderson <[email protected]>…

Actually it's not a problem with the vmlinux file, but rather with kernel module object files. The crash utility has an embedded gdb module which is invoked as "gdb vmlinux", and to get line numbers, the crash utility simply uses the relevant built-in gdb function to get them. And line numbers work fine with the base kernel code from the vmlinux file.

if the entire goal is to get a pinpoint of where the crash is, just using scripts/markup_oops.pl will do the right thing.

Signature · 3 lines
Arjan van de Ven 	Intel Open Source Technology Centre
For development, discussion and tips for power savings, 
visit http://www.lesswatts.org
#3 lore raw
In reply to Dave Anderson
Quoted 14 lines · Dave Anderson

Actually it's not a problem with the vmlinux file, but rather with kernel module object files. The crash utility has an embedded gdb module which is invoked as "gdb vmlinux", and to get line numbers, the crash utility simply uses the relevant built-in gdb function to get them. And line numbers work fine with the base kernel code from the vmlinux file.

The debuginfo data of kernel modules can be subsequently added to the crash session by doing a gdb "add-symbol-file" command for any or all kernel modules. But getting correct line number information for kernel modules has been a crap-shoot in the past, depending upon architecture and/or kernel version. For example, they don't work with 2.6.9-based RHEL4 x86_64 kernel modules, but work fine with 2.6.18-based RHEL5 x86_64 kernels.

Quoted 3 lines

Looking at Mike's suspect kernel patch list, I don't see anything that would have any relationship to the issue. Perhaps there was a build tool change during the same timeframe?

It look like Mike just built a series of kernels and had a problem, which should preclude a tool change.

That said. Does this feature of crash work in 2.6.29? If not is there enough interest to track this down, and fix it if it is a kernel bug?

If we are going to be using these tools we need them working on the latest and greatest kernels, not some weird enterprise branch, for fuddy duddies.

Eric

#4 lore raw
Quoted 16 lines · Eric W. Biederman
Quoted 14 lines · Dave Anderson

Actually it's not a problem with the vmlinux file, but rather with kernel module object files. The crash utility has an embedded gdb module which is invoked as "gdb vmlinux", and to get line numbers, the crash utility simply uses the relevant built-in gdb function to get them. And line numbers work fine with the base kernel code from the vmlinux file.

The debuginfo data of kernel modules can be subsequently added to the crash session by doing a gdb "add-symbol-file" command for any or all kernel modules. But getting correct line number information for kernel modules has been a crap-shoot in the past, depending upon architecture and/or kernel version. For example, they don't work with 2.6.9-based RHEL4 x86_64 kernel modules, but work fine with 2.6.18-based RHEL5 x86_64 kernels.

Quoted 6 lines
Quoted 3 lines

Looking at Mike's suspect kernel patch list, I don't see anything that would have any relationship to the issue. Perhaps there was a build tool change during the same timeframe?

It look like Mike just built a series of kernels and had a problem, which should preclude a tool change.

That is correct, I just built/booted/tested a series of kernels one after the other as part of the standard git bisect procedure. No tools were changed.

Quoted 7 lines

That said. Does this feature of crash work in 2.6.29? If not is there enough interest to track this down, and fix it if it is a kernel bug?

If we are going to be using these tools we need them working on the latest and greatest kernels, not some weird enterprise branch, for fuddy duddies.

This feature of crash does not work with 2.6.29, nor does it work with any kernel I've tried >= 2.6.21. Andi Kleen shared with me that he sees the same problem with a recent crash and 2.6.28.

AFAIK I found the regression point relative to linux (commit:
7460ed28). The verdict is clearly still out on where the actual bug lives (linux vs crash).

Mike