BISECTED: Re: source line numbers with x86_64 modules?
A message that is not in the archive (sent off-list or not yet received): 952123357.823421233089955989.JavaMail.root@zmail02.collab.prod.int.phx2.redhat.com
Quoted 29 lines · ----- "Eric W. Biederman
Quoted 27 lines · Mike Snitzer
[I've trimmed the wide cc distribution that was inherited when I forked a different thread]
Quoted 5 lines · Eric W. Biederman
Quoted 3 lines · Mike Snitzer
Now if only I could fix line numbers when debugging crashes in
x86_64modules with the crash utility! :)Quoted 7 lines
It's a userspace problem...
All of the little usability things are userspace problems.
I won't claim that it is trivial because it is a userspace problem, at the same time there is no reason to wait for any kernel features to merge etc. Someone just has to scratch an itch and go fix it.
Yes, the crash utility (userspace) is clearly having problems getting line number for symbols in
x86_64modules. But I finally took some time to bisect the point in the kernel where the crash utility first started to fail, it appears to be:commit
7460ed2844ffad7141e30271c0c3da8336e66014Author: john stultz <[email protected]> Date: Fri Feb 16 01:28:21 2007 -0800
Quoted 8 lines
Quoted 4 lines
I used version 4.0-7.6 of the crash utility to test if each commit was good or bad. I simply checked if ext3's module had correct line number info for the
ext3_get_blocks_handlesymbol with: symext3_get_blocks_handleWeird. That patch doesn't appear to affect anything in that area. So my stab in the dark is that there is something in vmlinux that crash doesn't know how to cope with.
Actually it's not a problem with the vmlinux file, but rather with kernel module object files. The crash utility has an embedded gdb module which is invoked as "gdb vmlinux", and to get line numbers, the crash utility simply uses the relevant built-in gdb function to get them. And line numbers work fine with the base kernel code from the vmlinux file.
The debuginfo data of kernel modules can be subsequently added to the crash session by doing a gdb "add-symbol-file" command for any or all kernel modules. But getting correct line number information for kernel modules has been a crap-shoot in the past, depending upon architecture and/or kernel version. For example, they don't work with 2.6.9-based RHEL4 x86_64 kernel modules, but work fine with 2.6.18-based RHEL5 x86_64 kernels.
Looking at Mike's suspect kernel patch list, I don't see anything that would have any relationship to the issue. Perhaps there was a build tool change during the same timeframe?
Dave
Quoted 7 lines · On Tue, 27 Jan 2009 16:00:19 -0500 (EST) Dave Anderson <[email protected]>…
Actually it's not a problem with the vmlinux file, but rather with kernel module object files. The crash utility has an embedded gdb module which is invoked as "gdb vmlinux", and to get line numbers, the crash utility simply uses the relevant built-in gdb function to get them. And line numbers work fine with the base kernel code from the vmlinux file.
if the entire goal is to get a pinpoint of where the crash is, just using scripts/markup_oops.pl will do the right thing.
Signature · 3 lines
Arjan van de Ven Intel Open Source Technology Centre For development, discussion and tips for power savings, visit http://www.lesswatts.org
Quoted 14 lines · Dave Anderson
Actually it's not a problem with the vmlinux file, but rather with kernel module object files. The crash utility has an embedded gdb module which is invoked as "gdb vmlinux", and to get line numbers, the crash utility simply uses the relevant built-in gdb function to get them. And line numbers work fine with the base kernel code from the vmlinux file.
The debuginfo data of kernel modules can be subsequently added to the crash session by doing a gdb "add-symbol-file" command for any or all kernel modules. But getting correct line number information for kernel modules has been a crap-shoot in the past, depending upon architecture and/or kernel version. For example, they don't work with 2.6.9-based RHEL4
x86_64kernel modules, but work fine with 2.6.18-based RHEL5x86_64kernels.
Quoted 3 lines
Looking at Mike's suspect kernel patch list, I don't see anything that would have any relationship to the issue. Perhaps there was a build tool change during the same timeframe?
It look like Mike just built a series of kernels and had a problem, which should preclude a tool change.
That said. Does this feature of crash work in 2.6.29? If not is there enough interest to track this down, and fix it if it is a kernel bug?
If we are going to be using these tools we need them working on the latest and greatest kernels, not some weird enterprise branch, for fuddy duddies.
Eric
Quoted 16 lines · Eric W. Biederman
Quoted 14 lines · Dave Anderson
Actually it's not a problem with the vmlinux file, but rather with kernel module object files. The crash utility has an embedded gdb module which is invoked as "gdb vmlinux", and to get line numbers, the crash utility simply uses the relevant built-in gdb function to get them. And line numbers work fine with the base kernel code from the vmlinux file.
The debuginfo data of kernel modules can be subsequently added to the crash session by doing a gdb "add-symbol-file" command for any or all kernel modules. But getting correct line number information for kernel modules has been a crap-shoot in the past, depending upon architecture and/or kernel version. For example, they don't work with 2.6.9-based RHEL4
x86_64kernel modules, but work fine with 2.6.18-based RHEL5x86_64kernels.
Quoted 6 lines
Quoted 3 lines
Looking at Mike's suspect kernel patch list, I don't see anything that would have any relationship to the issue. Perhaps there was a build tool change during the same timeframe?
It look like Mike just built a series of kernels and had a problem, which should preclude a tool change.
That is correct, I just built/booted/tested a series of kernels one after the other as part of the standard git bisect procedure. No tools were changed.
Quoted 7 lines
That said. Does this feature of crash work in 2.6.29? If not is there enough interest to track this down, and fix it if it is a kernel bug?
If we are going to be using these tools we need them working on the latest and greatest kernels, not some weird enterprise branch, for fuddy duddies.
This feature of crash does not work with 2.6.29, nor does it work with any kernel I've tried >= 2.6.21. Andi Kleen shared with me that he sees the same problem with a recent crash and 2.6.28.
AFAIK I found the regression point relative to linux (commit:7460ed28). The verdict is clearly still out on where the actual bug lives (linux vs crash).
Mike