Looks like tick_sched_timer() is a major latency source
Hi
I was trying to get minimum possible in-kernel latencies on a freescale 8360 CPU based board (powerpc arch). There is a project where <30 us worst case latency is required, and question was - is it possible to use linux+preempt-rt (which is preferred for a number of reason) or not.
I used 2.6.29.5-rt21 kernel, which, for my best knowledge, was the last "release" of preempt-rt at the moment when I started.
I used 8360's Generic Timer Module to both to generate interrupts and to measure lanencies. GTM timers continue to count after firing an interrupt, so latency may be measured my reading current timer value and comparing that with the value at which interrupt was generated. I believe this method is both simple and accurate.
The result is:
- average latency up to IRQF_NODELAY interrupt header is 2-3 us, even on a system with high i/o and cpu load. To generate load, I used a flood ping and several 'while true; do true; done' cpu-eaters.
- but worst-case latency is BAD. I occasionally got >50 us even on idle system.
I tried hard to identify the latency source, and at some moment discovered tick_sched_timer() from kernel/time/tick-sched.c. This routine is called from timer interrupt with hardware interrupts disabled, and may execute for 50 us and more.
Could someone please comment on this?
Is it possible to move (part of) tick_sched_timer() call tree out of hardware-interrupts-disabled context without breaking things?
Nikita
Please CC: reply to my e-mail address. Thanks.
Quoted 6 lines · Nikita V. Youshchenko
I tried hard to identify the latency source, and at some moment discovered
tick_sched_timer()fromkernel/time/tick-sched.c. This routine is called from timer interrupt with hardware interrupts disabled, and may execute for 50 us and more.Could someone please comment on this?
Did you have CONFIG_NOHZ enabled ? If yes, please disable.
Quoted 2 lines
Is it possible to move (part of)
tick_sched_timer()call tree out of hardware-interrupts-disabled context without breaking things?
Yes, we can move things out and split the irq disabled regions. One particular thing which can be split out is the time update, which we already had in the timer softirq at some point, but we moved it back as it showed problems with NTP stability. It's one of the things which can be revisited. The whole call chain needs some investigation, but that's not in my main focus now.
Thanks,
tglx
Quoted 8 lines
Quoted 6 lines
I tried hard to identify the latency source, and at some moment discovered
tick_sched_timer()fromkernel/time/tick-sched.c. This routine is called from timer interrupt with hardware interrupts disabled, and may execute for 50 us and more.Could someone please comment on this?
Did you have
CONFIG_NOHZenabled ? If yes, please disable.
I have disabled CONFIG_NOHZ long ago, since it really influences latency.
Quoted 9 lines
Quoted 2 lines
Is it possible to move (part of)
tick_sched_timer()call tree out of hardware-interrupts-disabled context without breaking things?Yes, we can move things out and split the irq disabled regions. One particular thing which can be split out is the time update, which we already had in the timer softirq at some point, but we moved it back as it showed problems with NTP stability. It's one of the things which can be revisited. The whole call chain needs some investigation, but that's not in my main focus now.
Is that code with time update split out available anywhere?
Nikita