Skip to contentSkip to conversation

frame unwinder patches

DISCUSSION Bernd Schubert 3 participants 12 messages Sep 4, 2008 – Sep 5, 2008

lore
Share & tools
lore
#0 lore raw

Hello,

up to 2.6.22 the -mm series had these nice frame unwinder patches, which gave beautiful stack traces. Unfortunately these frame unwinder patches seem to have been dropped :(

Anyone still maintaining updated patches?

Here is an example of a horrible stack trace, confusing my eyes

[  855.504734] Call Trace:
[  855.504734]  [<ffffffff80495af2>] ? scsi_request_fn+0x252/0x370
[  855.504734]  [<ffffffff8058b068>] io_schedule+0x28/0x40
[  855.504734]  [<ffffffff8038b876>] get_request_wait+0x96/0x110
[  855.504734]  [<ffffffff80249d40>] ? autoremove_wake_function+0x0/0x40
[  855.504734]  [<ffffffff8038bca8>] blk_get_request+0x38/0x80
[  855.504734]  [<ffffffff804950a3>] scsi_execute+0x43/0x140
[  855.504734]  [<ffffffffa00250dc>] :scsi_transport_spi:spi_execute+0x7c/0xd0
[  855.504734]  [<ffffffff8038accb>] ? blk_put_request+0x4b/0x60
[  855.504734]  [<ffffffff8049515f>] ? scsi_execute+0xff/0x140
[  855.504734]  [<ffffffffa0025b86>] :scsi_transport_spi:spi_dv_device_echo_buffer+0x1c6/0x290
[  855.504734]  [<ffffffffa00349f0>] ? :mptbase:mpt_timer_expired+0x0/0x70
[  855.504734]  [<ffffffffa0024dc3>] :scsi_transport_spi:spi_dv_retrain+0x63/0x240
[  855.504734]  [<ffffffffa00259c0>] ? :scsi_transport_spi:spi_dv_device_echo_buffer+0x0/0x290
[  855.504734]  [<ffffffffa00255f7>] :scsi_transport_spi:spi_dv_device+0x3b7/0x720
[  855.504734]  [<ffffffffa005017b>] :mptspi:mptspi_dv_device+0x6b/0x1a0
[  855.504734]  [<ffffffffa00502ce>] ? :mptspi:mptspi_dv_renegotiate_work+0x1e/0x100
[  855.504734]  [<ffffffffa005038f>] :mptspi:mptspi_dv_renegotiate_work+0xdf/0x100
[  855.504734]  [<ffffffffa00502b0>] ? :mptspi:mptspi_dv_renegotiate_work+0x0/0x100
[  855.504734]  [<ffffffffa00502b0>] ? :mptspi:mptspi_dv_renegotiate_work+0x0/0x100
[  855.504734]  [<ffffffff80245b36>] run_workqueue+0x176/0x220
[  855.504734]  [<ffffffff80246786>] worker_thread+0xa6/0x110
[  855.504734]  [<ffffffff80249d40>] ? autoremove_wake_function+0x0/0x40
[  855.504734]  [<ffffffff802466e0>] ? worker_thread+0x0/0x110
[  855.504734]  [<ffffffff8024990d>] kthread+0x4d/0x80
[  855.504734]  [<ffffffff8020c5e8>] child_rip+0xa/0x12
[  855.504734]  [<ffffffff8020bcff>] ? restore_args+0x0/0x30
[  855.504734]  [<ffffffff802498c0>] ? kthread+0x0/0x80
[  855.504734]  [<ffffffff8020c5de>] ? child_rip+0x0/0x12
[  855.504734]

Probably all the "?" lines shouldn't be printed.

Thanks,
Bernd

#1 lore raw
Quoted 9 lines · On Thu, 04 Sep 2008 18:46:52 +0200 Bernd Schubert <[email protected]> wrote:

Hello,

up to 2.6.22 the -mm series had these nice frame unwinder patches, which gave beautiful stack traces. Unfortunately these frame unwinder patches seem to have been dropped :(

Anyone still maintaining updated patches?

Probably all the "?" lines shouldn't be printed.

printing the ? lines is the result of a compromise between getting the most reliable information out of a stack trace, and getting a useful data for the case where not everything is going as well as it should.

not printing the ? would be trivial (for some backtraces we do that), but so far hasn't been seen as desirable because you lose information in those hard-to-debug cases when you need anything you can get from the backtrace.

Signature · 3 lines
If you want to reach me at my work email, use [email protected]
For development, discussion and tips for power savings, 
visit http://www.lesswatts.org
#9 lore raw
In reply to Bernd Schubert
Quoted 7 lines · Bernd Schubert

Hello,

up to 2.6.22 the -mm series had these nice frame unwinder patches, which gave beautiful stack traces. Unfortunately these frame unwinder patches seem to have been dropped :(

Anyone still maintaining updated patches?

The SUSE kernel rpm should have an uptodate set

(ftp.suse.com/pub/projects/kernel/HEAD )

I agree that they were much better than the current unwinder, also they didn't require slowing down the kernel with frame pointers.

-Andi

#2 lore raw
In reply to Arjan van de Ven
Quoted 13 lines · On Thursday 04 September 2008 19:29:36 Arjan van de Ven wrote:

On Thu, 04 Sep 2008 18:46:52 +0200

Quoted 9 lines · Bernd Schubert

Hello,

up to 2.6.22 the -mm series had these nice frame unwinder patches, which gave beautiful stack traces. Unfortunately these frame unwinder patches seem to have been dropped :(

Anyone still maintaining updated patches?

Probably all the "?" lines shouldn't be printed.

Quoted 8 lines

printing the ? lines is the result of a compromise between getting the most reliable information out of a stack trace, and getting a useful data for the case where not everything is going as well as it should.

not printing the ? would be trivial (for some backtraces we do that), but so far hasn't been seen as desirable because you lose information in those hard-to-debug cases when you need anything you can get from the backtrace.

I see your point, but in most cases it is rather annoying.

Bernd

Signature · 2 lines
Bernd Schubert
Q-Leap Networks GmbH
#10 lore raw
In reply to Andi Kleen
Quoted 7 lines · On Thursday 04 September 2008 21:13:58 Andi Kleen wrote:
Quoted 6 lines · Bernd Schubert

Hello,

up to 2.6.22 the -mm series had these nice frame unwinder patches, which gave beautiful stack traces. Unfortunately these frame unwinder patches seem to have been dropped :(

Quoted 6 lines
Quoted 1 line

Anyone still maintaining updated patches?

The SUSE kernel rpm should have an uptodate set

(ftp.suse.com/pub/projects/kernel/HEAD )

Thanks Andi, found it! If someone else needs it, the correct path is ftp://ftp.suse.com/pub/projects/kernel/kotd/HEAD.

Quoted 3 lines

I agree that they were much better than the current unwinder, also they didn't require slowing down the kernel with frame pointers.

What it the reason this patch isn't mainline? It provides better traces, doesn't slow down the kernel (which is important in our HPC envirement), so what is the disadvantage?

Thanks again,
Bernd

Signature · 2 lines
Bernd Schubert
Q-Leap Networks GmbH
#11 lore raw
Quoted 10 lines · Bernd Schubert
Quoted 9 lines · On Thursday 04 September 2008 21:13:58 Andi Kleen wrote:
Quoted 7 lines · Bernd Schubert

Hello,

up to 2.6.22 the -mm series had these nice frame unwinder patches, which gave beautiful stack traces. Unfortunately these frame unwinder patches seem to have been dropped :(

Anyone still maintaining updated patches?

Quoted 7 lines
Quoted 4 lines

The SUSE kernel rpm should have an uptodate set

(ftp.suse.com/pub/projects/kernel/HEAD )

Thanks Andi, found it! If someone else needs it, the correct path is ftp://ftp.suse.com/pub/projects/kernel/kotd/HEAD.

Thanks for the correction.

Quoted 7 lines
Quoted 3 lines

I agree that they were much better than the current unwinder, also they didn't require slowing down the kernel with frame pointers.

What it the reason this patch isn't mainline? It provides better traces, doesn't slow down the kernel (which is important in our HPC envirement), so what is the disadvantage?

The initial version that was merged had a few teething problems because it ran into tool chain bugs (which were quickly resolved). Unfortunately it lead to a few people developing irrational fears of dwarf2 in the process and it was reverted, annoying the original contributor. Right now someone would just need to retry the submission I guess.

-Andi

Signature · 1 line
[email protected]
#3 lore raw
In reply to Bernd Schubert
Quoted 6 lines · On Fri, 5 Sep 2008 11:19:21 +0200 Bernd Schubert <[email protected]> wrote:
Quoted 4 lines

not printing the ? would be trivial (for some backtraces we do that), but so far hasn't been seen as desirable because you lose information in those hard-to-debug cases when you need anything you can get from the backtrace.

I see your point, but in most cases it is rather annoying.

... if you see backtraces regularly there's something wrong ;-)

(and if you really care it's 1 line of code to turn it off)

Signature · 3 lines
If you want to reach me at my work email, use [email protected]
For development, discussion and tips for power savings, 
visit http://www.lesswatts.org
#4 lore raw
Quoted 11 lines · On Friday 05 September 2008 15:33:17 Arjan van de Ven wrote:

On Fri, 5 Sep 2008 11:19:21 +0200

Quoted 6 lines · Bernd Schubert
Quoted 4 lines

not printing the ? would be trivial (for some backtraces we do that), but so far hasn't been seen as desirable because you lose information in those hard-to-debug cases when you need anything you can get from the backtrace.

I see your point, but in most cases it is rather annoying.

... if you see backtraces regularly there's something wrong ;-)

It always depends on what you are doing. Presently I'm fixing the reset-handler of the mpt fusion driver and since I'm not $hardcore_kernel_hacker_with_super_scsi_knowlegde, but only know C and a little bit about the kernel API I insert dump_stack() and printks all over the place into the code to understand what is going on.

Well, this reset part is mostly done, now this problem:
http://marc.info/?l=linux-scsi&m=118039573814323&w=2

So yes, there is something wrong ;) Unfortunately so far nobody cared about these bugs we are hitting regularily.

Quoted 2 lines

(and if you really care it's 1 line of code to turn it off)

It is not only this, I think the dwarf2 stack unwinder patches provide by far better traces than the in-kernel unwinder. At least ever since I applied these patches to our kernels, I was able to read the stack dumps...

Cheers,
Bernd

Signature · 2 lines
Bernd Schubert
Q-Leap Networks GmbH
#5 lore raw
Quoted 8 lines · On Fri, 5 Sep 2008 15:52:47 +0200 Bernd Schubert <[email protected]> wrote:
Quoted 2 lines

(and if you really care it's 1 line of code to turn it off)

It is not only this, I think the dwarf2 stack unwinder patches provide by far better traces than the in-kernel unwinder. At least ever since I applied these patches to our kernels, I was able to read the stack dumps...

they really wouldn't be different than the ones you get if you remove the "?" lines.

Signature · 3 lines
If you want to reach me at my work email, use [email protected]
For development, discussion and tips for power savings, 
visit http://www.lesswatts.org
#6 lore raw
Quoted 5 lines · On Friday 05 September 2008 16:13:37 Arjan van de Ven wrote:

On Fri, 5 Sep 2008 15:52:47 +0200

Quoted 2 lines · Bernd Schubert
Quoted 1 line

(and if you really care it's 1 line of code to turn it off)

Quoted 7 lines
Quoted 4 lines

It is not only this, I think the dwarf2 stack unwinder patches provide by far better traces than the in-kernel unwinder. At least ever since I applied these patches to our kernels, I was able to read the stack dumps...

they really wouldn't be different than the ones you get if you remove the "?" lines.

Well may be, but then there is still the performace degrading, so I don't want to have it enabled on our production kernels. I admit I never measured what is the difference between of CONFIG_FRAME_POINTER=y and =n, but the fact the help text says there is a difference already makes me want to disable it (especially, since we have to provide benchmarks before we can sell a system).

Cheers,
Bernd

Signature · 2 lines
Bernd Schubert
Q-Leap Networks GmbH
#7 lore raw
Quoted 14 lines · On Fri, 5 Sep 2008 16:48:52 +0200 Bernd Schubert <[email protected]> wrote:
Quoted 5 lines · On Friday 05 September 2008 16:13:37 Arjan van de Ven wrote:

On Fri, 5 Sep 2008 15:52:47 +0200

Quoted 2 lines · Bernd Schubert
Quoted 1 line

(and if you really care it's 1 line of code to turn it off)

Quoted 7 lines
Quoted 4 lines

It is not only this, I think the dwarf2 stack unwinder patches provide by far better traces than the in-kernel unwinder. At least ever since I applied these patches to our kernels, I was able to read the stack dumps...

they really wouldn't be different than the ones you get if you remove the "?" lines.

Quoted 6 lines

Well may be, but then there is still the performace degrading, so I don't want to have it enabled on our production kernels. I admit I never measured what is the difference between of CONFIG_FRAME_POINTER=y and =n, but the fact the help text says there is a difference already makes me want to disable it (especially, since we have to provide benchmarks before we can sell a system).

to be honest, on 64 bit the overhead is quite small (the extra instructions it adds are optimized for by the modern cpus that you use in the systems you're selling); on 32 bit the overhead is.. well a little bigger but not THAT much. yes it loses a register for the compiler to use, but no it's not a general purpose register, and with the register renaming that today's cpus do, I'd be surprised if you could see anything significant.

Signature · 3 lines
If you want to reach me at my work email, use [email protected]
For development, discussion and tips for power savings, 
visit http://www.lesswatts.org
#8 lore raw
Quoted 2 lines

to be honest, on 64 bit the overhead is quite small (the extra instructions it adds are optimized for by the modern cpus that you use

The pipeline dependency stalls yes, the icache/decode overhead no. Also CONFIG_FRAME_POINTER currently enables -fno-sibling-calls which generates significantly worse code for a lot of common kernel constructs.

Quoted 3 lines

in the systems you're selling); on 32 bit the overhead is.. well a little bigger but not THAT much. yes it loses a register for the compiler to use, but no it's not a general purpose register, and with

%rbp is a general purpose register. In fact it's even better than a general purpose register because it has often shorter encoding than the other registers, but is as versatile.

Quoted 2 lines

the register renaming that today's cpus do, I'd be surprised if you could see anything significant.

Even on modern CPUs you can measure it in macro benchmarks, at least that was the state last time that was investigated. On older CPUs without the magic hardware it was even more significant. Also there are even new CPUs like Atom which don't have the magic hardware.

-Andi