Skip to contentSkip to conversation

latency-profiling

PATCH In review Roger Larsson 4 participants 4 messages Sep 20, 2001 – Sep 21, 2001

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 latency-profiling

Hi,

I ported my old latency-profiling patch to 2.4.10-pre10 with the reschedulable kernel patch. (I have not checked that it is preemption safe itself...)

This patch works a little different from Robert Loves. Since it samples the execution location at ticks. It is possible to instrument an ordinary kernel too... These are the results (>5 ms):

Sep 19 20:09:28 jeloin kernel: Latency  17ms PID   316 %% ifconfig
Sep 19 20:09:28 jeloin kernel: Trace: 
[mousedev:__insmod_mousedev_O/lib/modules/2.4.10-pre10/kernel/drivers+-93829/96] 
[__rdtsc_delay+20/28]
Sep 19 20:12:48 jeloin kernel: Latency  10ms PID   870 %% artsd
Sep 19 20:12:48 jeloin kernel: Trace: [console_softint+230/232] 
[__rdtsc_delay+20/28]
Sep 19 20:13:28 jeloin kernel: Latency   8ms PID   667 %% X
Sep 19 20:13:28 jeloin kernel: Trace: [__rdtsc_delay+22/28]
Sep 19 20:13:53 jeloin kernel: Latency  35ms PID   972 %% bash
Sep 19 20:13:53 jeloin kernel: Trace: [unmap_buffer+100/132] 
[kmem_cache_free+140/148] [block_flushpage+43/112]
Sep 19 20:14:19 jeloin kernel: Latency  40ms PID   979 %% bash
Sep 19 20:14:19 jeloin kernel: Trace: [__refile_buffer+95/108] 
[block_flushpage+43/112] [remove_inode_page+36/64] [__lru_cache_del+1/272]
Sep 19 20:15:55 jeloin kernel: Latency  21ms PID  1038 %% cp
Sep 19 20:15:55 jeloin kernel: Trace: [truncate_list_pages+200/456] 
[try_to_free_buffers+332/556]
Sep 19 20:32:18 jeloin kernel: Latency   6ms PID     7 %% kupdated
Sep 19 20:32:19 jeloin kernel: Latency   5ms PID  1214 %% mmap002
Sep 19 20:35:18 jeloin kernel: Latency  22ms PID  1214 %% mmap002
Sep 19 20:35:18 jeloin kernel: Trace: [zap_page_range+318/628] 
[zap_page_range+382/628]
Sep 19 21:49:22 jeloin kernel: Latency  10ms PID     0 %% swapper
Sep 19 21:49:22 jeloin kernel: Trace: [schedule+463/1032]
Sep 19 22:37:45 jeloin kernel: Latency   5ms PID  2439 %% y2bignfat
Sep 19 22:37:45 jeloin kernel: Latency   5ms PID  2446 %% y2bignfat
Sep 19 22:46:18 jeloin kernel: Latency   5ms PID  2552 %% y2bignfat
Sep 19 22:46:18 jeloin kernel: Trace: [zap_page_range+382/628]
Sep 19 22:46:18 jeloin kernel: Latency   5ms PID  2555 %% y2bignfat
Sep 19 22:46:18 jeloin kernel: Trace: [zap_page_range+318/628]
Sep 19 22:46:18 jeloin kernel: Latency   5ms PID  2559 %% y2bignfat
Sep 19 22:46:18 jeloin kernel: Latency   5ms PID  2571 %% y2bignfat
Sep 19 22:46:19 jeloin kernel: Latency   5ms PID  2573 %% y2bignfat
Sep 19 22:46:19 jeloin kernel: Latency   5ms PID  2574 %% y2bignfat
Sep 19 22:46:19 jeloin kernel: Latency   5ms PID  2578 %% y2bignfat
Sep 19 22:46:19 jeloin kernel: Latency   5ms PID  2579 %% y2bignfat
Sep 19 22:46:19 jeloin kernel: Trace: [zap_page_range+382/628]
Sep 19 22:46:19 jeloin kernel: Latency   5ms PID  2593 %% y2bignfat
Sep 19 22:46:19 jeloin kernel: Trace: [zap_page_range+318/628]
Sep 19 22:46:19 jeloin kernel: Latency   5ms PID  2594 %% y2bignfat
Sep 19 22:46:20 jeloin kernel: Latency   5ms PID  2596 %% y2bignfat
Sep 20 07:27:57 jeloin kernel: Latency  59ms PID   281 %% cardmgr
Sep 20 07:27:57 jeloin kernel: Trace: [__rdtsc_delay+20/28] 
[__rdtsc_delay+20/28] [__rdtsc_delay+22/28] [__rdtsc_delay+20/28] 
[__rdtsc_delay+20/28]
Sep 20 07:27:57 jeloin kernel: Trace: [__rdtsc_delay+20/28]
Sep 20 07:27:57 jeloin kernel: Latency  24ms PID   316 %% ifconfig
Sep 20 07:27:57 jeloin kernel: Trace: [__rdtsc_delay+20/28] 
[mousedev:__insmod_mousedev_O/lib/modules/2.4.10-pre10/kernel/drivers+-93829/96] 
[mousedev:__insmod_mousedev_O/lib/modules/2.4.10-pre10/kernel/drivers+-93562/96]
Sep 20 07:27:59 jeloin kernel: Latency   8ms PID   649 %% X
Sep 20 07:27:59 jeloin kernel: Trace: [aux_write_ack+54/60]
Signature · 3 lines
Roger Larsson
Skellefteå
Sweden
#1 lore raw
Quoted 10 lines · Roger Larsson

Hi,

I ported my old latency-profiling patch to 2.4.10-pre10 with the reschedulable kernel patch. (I have not checked that it is preemption safe itself...)

This patch works a little different from Robert Loves. Since it samples the execution location at ticks. It is possible to instrument an ordinary kernel too...

It gives experienced latencies rather than potential latencies, but more important from the developer/maintainers point of view, "Robert Loves" patch provides information on the bad guys, i.e. the reason for the long latency, which, hopefully, will allow them to be addressed by competent maintainers.

George

#2 lore raw
In reply to Roger Larsson
Quoted 11 lines · george anzinger
Quoted 9 lines · Roger Larsson

Hi,

I ported my old latency-profiling patch to 2.4.10-pre10 with the reschedulable kernel patch. (I have not checked that it is preemption safe itself...)

This patch works a little different from Robert Loves. Since it samples the execution location at ticks. It is possible to instrument an ordinary kernel too...

Quoted 6 lines

It gives experienced latencies rather than potential latencies, but more important from the developer/maintainers point of view, "Robert Loves" patch provides information on the bad guys, i.e. the reason for the long latency, which, hopefully, will allow them to be addressed by competent maintainers.

Yes, but my technique can be applied to any kernel. It does not require an preemptible kernel...

I.e. measures time from the moment a reschedulation was reqired to it is actually done - independent of how it is done...

And if you run something that needs periodic reshedules you will detect actual problems.

And thus it can be used as a bench, it is hard to describe skips in music, textual data is easier to send by mail...

But I agree that Roberts is better to find the critical ones!

/RogerL

Signature · 3 lines
Roger Larsson
Skellefteå
Sweden
#3 lore raw
In reply to Roger Larsson
Quoted 3 lines · Roger Larsson

This patch works a little different from Robert Loves. Since it samples the execution location at ticks. It is possible to instrument an ordinary kernel too...

You could also try the Linux Trace Toolkit which would provide you with much more accurate results and exact times instead of samples. LTT is here: http://www.opersys.com/LTT

Cheers,

Karim

===================================================
Karim Yaghmour
[email protected]
Embedded and Real-Time Linux Expert
===================================================