frame unwinder patches
Hello,
up to 2.6.22 the -mm series had these nice frame unwinder patches, which gave beautiful stack traces. Unfortunately these frame unwinder patches seem to have been dropped :(
Anyone still maintaining updated patches?
Here is an example of a horrible stack trace, confusing my eyes
[ 855.504734] Call Trace: [ 855.504734] [<ffffffff80495af2>] ? scsi_request_fn+0x252/0x370 [ 855.504734] [<ffffffff8058b068>] io_schedule+0x28/0x40 [ 855.504734] [<ffffffff8038b876>] get_request_wait+0x96/0x110 [ 855.504734] [<ffffffff80249d40>] ? autoremove_wake_function+0x0/0x40 [ 855.504734] [<ffffffff8038bca8>] blk_get_request+0x38/0x80 [ 855.504734] [<ffffffff804950a3>] scsi_execute+0x43/0x140 [ 855.504734] [<ffffffffa00250dc>] :scsi_transport_spi:spi_execute+0x7c/0xd0 [ 855.504734] [<ffffffff8038accb>] ? blk_put_request+0x4b/0x60 [ 855.504734] [<ffffffff8049515f>] ? scsi_execute+0xff/0x140 [ 855.504734] [<ffffffffa0025b86>] :scsi_transport_spi:spi_dv_device_echo_buffer+0x1c6/0x290 [ 855.504734] [<ffffffffa00349f0>] ? :mptbase:mpt_timer_expired+0x0/0x70 [ 855.504734] [<ffffffffa0024dc3>] :scsi_transport_spi:spi_dv_retrain+0x63/0x240 [ 855.504734] [<ffffffffa00259c0>] ? :scsi_transport_spi:spi_dv_device_echo_buffer+0x0/0x290 [ 855.504734] [<ffffffffa00255f7>] :scsi_transport_spi:spi_dv_device+0x3b7/0x720 [ 855.504734] [<ffffffffa005017b>] :mptspi:mptspi_dv_device+0x6b/0x1a0 [ 855.504734] [<ffffffffa00502ce>] ? :mptspi:mptspi_dv_renegotiate_work+0x1e/0x100 [ 855.504734] [<ffffffffa005038f>] :mptspi:mptspi_dv_renegotiate_work+0xdf/0x100 [ 855.504734] [<ffffffffa00502b0>] ? :mptspi:mptspi_dv_renegotiate_work+0x0/0x100 [ 855.504734] [<ffffffffa00502b0>] ? :mptspi:mptspi_dv_renegotiate_work+0x0/0x100 [ 855.504734] [<ffffffff80245b36>] run_workqueue+0x176/0x220 [ 855.504734] [<ffffffff80246786>] worker_thread+0xa6/0x110 [ 855.504734] [<ffffffff80249d40>] ? autoremove_wake_function+0x0/0x40 [ 855.504734] [<ffffffff802466e0>] ? worker_thread+0x0/0x110 [ 855.504734] [<ffffffff8024990d>] kthread+0x4d/0x80 [ 855.504734] [<ffffffff8020c5e8>] child_rip+0xa/0x12 [ 855.504734] [<ffffffff8020bcff>] ? restore_args+0x0/0x30 [ 855.504734] [<ffffffff802498c0>] ? kthread+0x0/0x80 [ 855.504734] [<ffffffff8020c5de>] ? child_rip+0x0/0x12 [ 855.504734]
Probably all the "?" lines shouldn't be printed.
Thanks,
Bernd
Quoted 9 lines · On Thu, 04 Sep 2008 18:46:52 +0200 Bernd Schubert <[email protected]> wrote:
Hello,
up to 2.6.22 the -mm series had these nice frame unwinder patches, which gave beautiful stack traces. Unfortunately these frame unwinder patches seem to have been dropped :(
Anyone still maintaining updated patches?
Probably all the "?" lines shouldn't be printed.
printing the ? lines is the result of a compromise between getting the most reliable information out of a stack trace, and getting a useful data for the case where not everything is going as well as it should.
not printing the ? would be trivial (for some backtraces we do that), but so far hasn't been seen as desirable because you lose information in those hard-to-debug cases when you need anything you can get from the backtrace.
Signature · 3 lines
If you want to reach me at my work email, use [email protected] For development, discussion and tips for power savings, visit http://www.lesswatts.org
Quoted 13 lines · On Thursday 04 September 2008 19:29:36 Arjan van de Ven wrote:
On Thu, 04 Sep 2008 18:46:52 +0200
Quoted 9 lines · Bernd Schubert
Hello,
up to 2.6.22 the -mm series had these nice frame unwinder patches, which gave beautiful stack traces. Unfortunately these frame unwinder patches seem to have been dropped :(
Anyone still maintaining updated patches?
Probably all the "?" lines shouldn't be printed.
Quoted 8 lines
printing the ? lines is the result of a compromise between getting the most reliable information out of a stack trace, and getting a useful data for the case where not everything is going as well as it should.
not printing the ? would be trivial (for some backtraces we do that), but so far hasn't been seen as desirable because you lose information in those hard-to-debug cases when you need anything you can get from the backtrace.
I see your point, but in most cases it is rather annoying.
Bernd
Signature · 2 lines
Bernd Schubert Q-Leap Networks GmbH
Quoted 6 lines · On Fri, 5 Sep 2008 11:19:21 +0200 Bernd Schubert <[email protected]> wrote:
Quoted 4 lines
not printing the ? would be trivial (for some backtraces we do that), but so far hasn't been seen as desirable because you lose information in those hard-to-debug cases when you need anything you can get from the backtrace.
I see your point, but in most cases it is rather annoying.
... if you see backtraces regularly there's something wrong ;-)
(and if you really care it's 1 line of code to turn it off)
Signature · 3 lines
If you want to reach me at my work email, use [email protected] For development, discussion and tips for power savings, visit http://www.lesswatts.org
Quoted 11 lines · On Friday 05 September 2008 15:33:17 Arjan van de Ven wrote:
On Fri, 5 Sep 2008 11:19:21 +0200
Quoted 6 lines · Bernd Schubert
Quoted 4 lines
not printing the ? would be trivial (for some backtraces we do that), but so far hasn't been seen as desirable because you lose information in those hard-to-debug cases when you need anything you can get from the backtrace.
I see your point, but in most cases it is rather annoying.
... if you see backtraces regularly there's something wrong ;-)
It always depends on what you are doing. Presently I'm fixing the reset-handler of the mpt fusion driver and since I'm not $hardcore_kernel_hacker_with_super_scsi_knowlegde, but only know C and a little bit about the kernel API I insert dump_stack() and printks all over the place into the code to understand what is going on.
Well, this reset part is mostly done, now this problem:
http://marc.info/?l=linux-scsi&m=118039573814323&w=2
So yes, there is something wrong ;) Unfortunately so far nobody cared about these bugs we are hitting regularily.
Quoted 2 lines
(and if you really care it's 1 line of code to turn it off)
It is not only this, I think the dwarf2 stack unwinder patches provide by far better traces than the in-kernel unwinder. At least ever since I applied these patches to our kernels, I was able to read the stack dumps...
Cheers,
Bernd
Signature · 2 lines
Bernd Schubert Q-Leap Networks GmbH
Quoted 8 lines · On Fri, 5 Sep 2008 15:52:47 +0200 Bernd Schubert <[email protected]> wrote:
Quoted 2 lines
(and if you really care it's 1 line of code to turn it off)
It is not only this, I think the dwarf2 stack unwinder patches provide by far better traces than the in-kernel unwinder. At least ever since I applied these patches to our kernels, I was able to read the stack dumps...
they really wouldn't be different than the ones you get if you remove the "?" lines.
Signature · 3 lines
If you want to reach me at my work email, use [email protected] For development, discussion and tips for power savings, visit http://www.lesswatts.org
Quoted 5 lines · On Friday 05 September 2008 16:13:37 Arjan van de Ven wrote:
On Fri, 5 Sep 2008 15:52:47 +0200
Quoted 2 lines · Bernd Schubert
Quoted 1 line
(and if you really care it's 1 line of code to turn it off)
Quoted 7 lines
Quoted 4 lines
It is not only this, I think the dwarf2 stack unwinder patches provide by far better traces than the in-kernel unwinder. At least ever since I applied these patches to our kernels, I was able to read the stack dumps...
they really wouldn't be different than the ones you get if you remove the "?" lines.
Well may be, but then there is still the performace degrading, so I don't want to have it enabled on our production kernels. I admit I never measured what is the difference between of CONFIG_FRAME_POINTER=y and =n, but the fact the help text says there is a difference already makes me want to disable it (especially, since we have to provide benchmarks before we can sell a system).
Cheers,
Bernd
Signature · 2 lines
Bernd Schubert Q-Leap Networks GmbH
Quoted 14 lines · On Fri, 5 Sep 2008 16:48:52 +0200 Bernd Schubert <[email protected]> wrote:
Quoted 5 lines · On Friday 05 September 2008 16:13:37 Arjan van de Ven wrote:
On Fri, 5 Sep 2008 15:52:47 +0200
Quoted 2 lines · Bernd Schubert
Quoted 1 line
(and if you really care it's 1 line of code to turn it off)
Quoted 7 lines
Quoted 4 lines
It is not only this, I think the dwarf2 stack unwinder patches provide by far better traces than the in-kernel unwinder. At least ever since I applied these patches to our kernels, I was able to read the stack dumps...
they really wouldn't be different than the ones you get if you remove the "?" lines.
Quoted 6 lines
Well may be, but then there is still the performace degrading, so I don't want to have it enabled on our production kernels. I admit I never measured what is the difference between of
CONFIG_FRAME_POINTER=yand =n, but the fact the help text says there is a difference already makes me want to disable it (especially, since we have to provide benchmarks before we can sell a system).
to be honest, on 64 bit the overhead is quite small (the extra instructions it adds are optimized for by the modern cpus that you use in the systems you're selling); on 32 bit the overhead is.. well a little bigger but not THAT much. yes it loses a register for the compiler to use, but no it's not a general purpose register, and with the register renaming that today's cpus do, I'd be surprised if you could see anything significant.
Signature · 3 lines
If you want to reach me at my work email, use [email protected] For development, discussion and tips for power savings, visit http://www.lesswatts.org
Quoted 2 lines
to be honest, on 64 bit the overhead is quite small (the extra instructions it adds are optimized for by the modern cpus that you use
The pipeline dependency stalls yes, the icache/decode overhead no. Also CONFIG_FRAME_POINTER currently enables -fno-sibling-calls which generates significantly worse code for a lot of common kernel constructs.
Quoted 3 lines
in the systems you're selling); on 32 bit the overhead is.. well a little bigger but not THAT much. yes it loses a register for the compiler to use, but no it's not a general purpose register, and with
%rbp is a general purpose register. In fact it's even better than a general purpose register because it has often shorter encoding than the other registers, but is as versatile.
Quoted 2 lines
the register renaming that today's cpus do, I'd be surprised if you could see anything significant.
Even on modern CPUs you can measure it in macro benchmarks, at least that was the state last time that was investigated. On older CPUs without the magic hardware it was even more significant. Also there are even new CPUs like Atom which don't have the magic hardware.
-Andi
Quoted 7 lines · Bernd Schubert
Hello,
up to 2.6.22 the -mm series had these nice frame unwinder patches, which gave beautiful stack traces. Unfortunately these frame unwinder patches seem to have been dropped :(
Anyone still maintaining updated patches?
The SUSE kernel rpm should have an uptodate set
(ftp.suse.com/pub/projects/kernel/HEAD )
I agree that they were much better than the current unwinder, also they didn't require slowing down the kernel with frame pointers.
-Andi
Quoted 7 lines · On Thursday 04 September 2008 21:13:58 Andi Kleen wrote:
Quoted 6 lines · Bernd Schubert
Hello,
up to 2.6.22 the -mm series had these nice frame unwinder patches, which gave beautiful stack traces. Unfortunately these frame unwinder patches seem to have been dropped :(
Quoted 6 lines
Quoted 1 line
Anyone still maintaining updated patches?
The SUSE kernel rpm should have an uptodate set
(ftp.suse.com/pub/projects/kernel/HEAD )
Thanks Andi, found it! If someone else needs it, the correct path is ftp://ftp.suse.com/pub/projects/kernel/kotd/HEAD.
Quoted 3 lines
I agree that they were much better than the current unwinder, also they didn't require slowing down the kernel with frame pointers.
What it the reason this patch isn't mainline? It provides better traces, doesn't slow down the kernel (which is important in our HPC envirement), so what is the disadvantage?
Thanks again,
Bernd
Signature · 2 lines
Bernd Schubert Q-Leap Networks GmbH
Quoted 10 lines · Bernd Schubert
Quoted 9 lines · On Thursday 04 September 2008 21:13:58 Andi Kleen wrote:
Quoted 7 lines · Bernd Schubert
Hello,
up to 2.6.22 the -mm series had these nice frame unwinder patches, which gave beautiful stack traces. Unfortunately these frame unwinder patches seem to have been dropped :(
Anyone still maintaining updated patches?
Quoted 7 lines
Quoted 4 lines
The SUSE kernel rpm should have an uptodate set
(ftp.suse.com/pub/projects/kernel/HEAD )
Thanks Andi, found it! If someone else needs it, the correct path is ftp://ftp.suse.com/pub/projects/kernel/kotd/HEAD.
Thanks for the correction.
Quoted 7 lines
Quoted 3 lines
I agree that they were much better than the current unwinder, also they didn't require slowing down the kernel with frame pointers.
What it the reason this patch isn't mainline? It provides better traces, doesn't slow down the kernel (which is important in our HPC envirement), so what is the disadvantage?
The initial version that was merged had a few teething problems because it ran into tool chain bugs (which were quickly resolved). Unfortunately it lead to a few people developing irrational fears of dwarf2 in the process and it was reverted, annoying the original contributor. Right now someone would just need to retry the submission I guess.
-Andi
Signature · 1 line
[email protected]