add_timer_on: in-kernel users _all_ buggy ?
Quoted 22 lines · * Mathieu Desnoyers ([email protected]) wrote:
Quoted 21 lines · * Thomas Gleixner ([email protected]) wrote:
Quoted 6 lines · Mathieu Desnoyers
Quoted 1 line
The function is called from an IPI. That's a LTTNG problem, not a RT one.
I use
del_timerin IPI to delete lttng per-cpu timers on all CPUs. I have to do this because timers created withadd_timer_onare documented to be incompatible withdel_timer_sync():Quoted 6 lines
* Synchronization rules: Callers must prevent restarting of the timer,
* otherwise this function is meaningless. It must not be called from
* interrupt contexts. The caller must not hold locks which would prevent
* completion of the timer's handler. The timer's handler must not call
*add_timer_on(). Upon exit the timer is not queued and the handler is
* not running on any CPU.Errm. The documentation says:
"The timer's handler must not call
add_timer_on()."It's not talking about a timer which was initialized with
add_timer_on().
Quoted 8 lines
Quoted 5 lines
And your per cpu timer handlers have no requirement to call
add_timer_on()simply because add/mod_timer() is requeueing the timer on the same cpu on which the handler runs.So the IPI is just a solution for a non existing problem.
Oh, right. Thanks for the explanation. I'll look into moving LTTng to a saner
del_timer_sync()scheme to delete the timers.
Double-checking this:
add_timer_on() needs to be paired with mod_timer_pinned(), otherwise NO_HZ SMP config can rebalance the timer to a different CPU. I am fixing this in lttng 0.194. These per-cpu timers, of course, should usually be deferrable (they are in lttng).
(looking at kernel 2.6.32.4 here)
Looking at the kernel/time/clocksource.c watchdog, I wonder how del_timer manages to synchronize the timer teardown. The handler, clocksource_watchdog(), uses add_timer_on(), which prohibits using del_timer_sync(). This seems rather odd. If we remove the watchdog and re-add it, it may still be in use while we initialize the timer structure.
Also, net/core/drop_monitor.c trace_drop_common usage of add_timer_on seems odd:
Executing (AFAIK) with preempt on, data points to a per-cpu timer:
if (!timer_pending(&data->send_timer)) {
data->send_timer.expires = jiffies + dm_delay * HZ;
add_timer_on(&data->send_timer, smp_processor_id());
}How is timer_pending synchronized with the target CPU timer wheel ?
Wait, there's more: arch/x86/kernel/cpu/mcheck/mce.c uses both add_timer_on in its handler and del_timer_sync (which is incorrect).
arch/x86/kernel/apic/x2apic_uv_x.c almost has it right, but maybe it should use del_timer_sync ?
arch/powerpc/platforms/chrp/setup.c should learn about mod_timer_pinned().
Which leads to the following question: is there _any_ add_timer_on() kernel user that's not currently buggy ? ;-) Maybe this calls for better documentation of this interface. From what I've learn from digging into cpufreq to debug its incorrect timer teardown last year, I fear there are lots and lots of buggy per-cpu _and_ standard timer interface users out there.
Maybe adding some debugging options, e.g. checking that a timer created with add_timer_on is always modified by mod_timer_pinned, and is always deferrable, and deleted by del_timer_sync could help discovering a couple of outlawyer.
Thanks,
Mathieu
Quoted 20 lines
Thanks,
Mathieu
Quoted 5 lines
Thanks,
tglx
Signature · 7 lines
Mathieu Desnoyers OpenPGP key fingerprint: 8CD5 52C3 8E3C 4140 715F BA06 3F25 A8FE 3BAE 9A68 _______________________________________________ ltt-dev mailing list [email protected] http://lists.casi.polymtl.ca/cgi-bin/mailman/listinfo/ltt-dev
Signature · 2 lines
Mathieu Desnoyers OpenPGP key fingerprint: 8CD5 52C3 8E3C 4140 715F BA06 3F25 A8FE 3BAE 9A68
Quoted 50 lines · Mathieu Desnoyers
Quoted 22 lines · * Mathieu Desnoyers ([email protected]) wrote:
Quoted 21 lines · * Thomas Gleixner ([email protected]) wrote:
Quoted 6 lines · Mathieu Desnoyers
Quoted 1 line
The function is called from an IPI. That's a LTTNG problem, not a RT one.
I use
del_timerin IPI to delete lttng per-cpu timers on all CPUs. I have to do this because timers created withadd_timer_onare documented to be incompatible withdel_timer_sync():Quoted 6 lines
* Synchronization rules: Callers must prevent restarting of the timer,
* otherwise this function is meaningless. It must not be called from
* interrupt contexts. The caller must not hold locks which would prevent
* completion of the timer's handler. The timer's handler must not call
*add_timer_on(). Upon exit the timer is not queued and the handler is
* not running on any CPU.Errm. The documentation says:
"The timer's handler must not call
add_timer_on()."It's not talking about a timer which was initialized with
add_timer_on().Quoted 8 lines
Quoted 5 lines
And your per cpu timer handlers have no requirement to call
add_timer_on()simply because add/mod_timer() is requeueing the timer on the same cpu on which the handler runs.So the IPI is just a solution for a non existing problem.
Oh, right. Thanks for the explanation. I'll look into moving LTTng to a saner
del_timer_sync()scheme to delete the timers.Double-checking this:
add_timer_on()needs to be paired withmod_timer_pinned(), otherwiseNO_HZSMP config can rebalance the timer to a different CPU. I am fixing this in lttng 0.194. These per-cpu timers, of course, should usually be deferrable (they are in lttng).(looking at kernel 2.6.32.4 here)
Looking at thekernel/time/clocksource.cwatchdog, I wonder howdel_timermanages to synchronize the timer teardown. The handler,clocksource_watchdog(), usesadd_timer_on(), which prohibits usingdel_timer_sync(). This seems rather odd. If we remove the watchdog and re-add it, it may still be in use while we initialize the timer structure.Also,
net/core/drop_monitor.ctrace_drop_commonusage ofadd_timer_onseems odd:
Quoted 9 lines
Executing (AFAIK) with preempt on, data points to a per-cpu timer:
if (!timer_pending(&data->send_timer)) { data->send_timer.expires = jiffies + dm_delay * HZ; add_timer_on(&data->send_timer, smp_processor_id()); }How is
timer_pendingsynchronized with the target CPU timer wheel ?
Hm, I think I see your point here. You're suggesting that a call to one of the tracepoint hooks in the drop monitor can race against a second call to the hook from an interrupt context that pre-empted the first, leading to double add of the timer? I agree, in fact I think its likely worse that that, the shared data on the skb that I modify there can get corrupted in that case as well. I expect a bit of refactoring paired with a local_irq_save/restore should fix that.
Thanks!
Neil
Quoted 51 lines
Wait, there's more:
arch/x86/kernel/cpu/mcheck/mce.cuses bothadd_timer_onin its handler anddel_timer_sync(which is incorrect).
arch/x86/kernel/apic/x2apic_uv_x.calmost has it right, but maybe it should usedel_timer_sync?
arch/powerpc/platforms/chrp/setup.cshould learn aboutmod_timer_pinned().Which leads to the following question: is there _any_
add_timer_on()kernel user that's not currently buggy ? ;-) Maybe this calls for better documentation of this interface. From what I've learn from digging into cpufreq to debug its incorrect timer teardown last year, I fear there are lots and lots of buggy per-cpu _and_ standard timer interface users out there.Maybe adding some debugging options, e.g. checking that a timer created with
add_timer_onis always modified bymod_timer_pinned, and is always deferrable, and deleted bydel_timer_synccould help discovering a couple of outlawyer.Thanks,
Mathieu
Quoted 20 lines
Thanks,
Mathieu
Quoted 5 lines
Thanks,
tglx
Signature · 7 lines
Mathieu Desnoyers OpenPGP key fingerprint: 8CD5 52C3 8E3C 4140 715F BA06 3F25 A8FE 3BAE 9A68 _______________________________________________ ltt-dev mailing list [email protected] http://lists.casi.polymtl.ca/cgi-bin/mailman/listinfo/ltt-devSignature · 2 lines
Mathieu Desnoyers OpenPGP key fingerprint: 8CD5 52C3 8E3C 4140 715F BA06 3F25 A8FE 3BAE 9A68
Quoted 58 lines · * Neil Horman ([email protected]) wrote:
Quoted 57 lines · Mathieu Desnoyers
Quoted 22 lines · * Mathieu Desnoyers ([email protected]) wrote:
Quoted 21 lines · * Thomas Gleixner ([email protected]) wrote:
Quoted 6 lines · Mathieu Desnoyers
Quoted 1 line
The function is called from an IPI. That's a LTTNG problem, not a RT one.
I use
del_timerin IPI to delete lttng per-cpu timers on all CPUs. I have to do this because timers created withadd_timer_onare documented to be incompatible withdel_timer_sync():Quoted 6 lines
* Synchronization rules: Callers must prevent restarting of the timer,
* otherwise this function is meaningless. It must not be called from
* interrupt contexts. The caller must not hold locks which would prevent
* completion of the timer's handler. The timer's handler must not call
*add_timer_on(). Upon exit the timer is not queued and the handler is
* not running on any CPU.Errm. The documentation says:
"The timer's handler must not call
add_timer_on()."It's not talking about a timer which was initialized with
add_timer_on().Quoted 8 lines
Quoted 5 lines
And your per cpu timer handlers have no requirement to call
add_timer_on()simply because add/mod_timer() is requeueing the timer on the same cpu on which the handler runs.So the IPI is just a solution for a non existing problem.
Oh, right. Thanks for the explanation. I'll look into moving LTTng to a saner
del_timer_sync()scheme to delete the timers.Double-checking this:
add_timer_on()needs to be paired withmod_timer_pinned(), otherwiseNO_HZSMP config can rebalance the timer to a different CPU. I am fixing this in lttng 0.194. These per-cpu timers, of course, should usually be deferrable (they are in lttng).(looking at kernel 2.6.32.4 here)
Looking at thekernel/time/clocksource.cwatchdog, I wonder howdel_timermanages to synchronize the timer teardown. The handler,clocksource_watchdog(), usesadd_timer_on(), which prohibits usingdel_timer_sync(). This seems rather odd. If we remove the watchdog and re-add it, it may still be in use while we initialize the timer structure.Also,
net/core/drop_monitor.ctrace_drop_commonusage ofadd_timer_onseems odd:Executing (AFAIK) with preempt on, data points to a per-cpu timer:
if (!timer_pending(&data->send_timer)) { data->send_timer.expires = jiffies + dm_delay * HZ; add_timer_on(&data->send_timer, smp_processor_id()); }
Quoted 6 lines
Quoted 2 lines
How is
timer_pendingsynchronized with the target CPU timer wheel ?Hm, I think I see your point here. You're suggesting that a call to one of the tracepoint hooks in the drop monitor can race against a second call to the hook from an interrupt context that pre-empted the first, leading to double add of the timer?
Yes.
Quoted 3 lines
I agree, in fact I think its likely worse that that, the shared data on the skb that I modify there can get corrupted in that case as well. I expect a bit of refactoring paired with a local_irq_save/restore should fix that.
For this current code, you don't seem to need to delete the timer. I think you'll have to find a way to do the timer_pending check and the add_timer_on operations atomically wrt timer wheel execution on the CPU (which can be a remote cpu because preemption is enabled). Your case is a bit weird.. the typical scenario here would be to use add_timer_on directly to re-arm the timer for a future expiration time (removing the current pending timer unconditionnally at the same time). But you leave the timer at its current expiration time if present, and only if not present add it. I wonder if it's really your intent ?
Thanks,
Mathieu
Quoted 55 lines
Thanks!
NeilQuoted 51 lines
Wait, there's more:
arch/x86/kernel/cpu/mcheck/mce.cuses bothadd_timer_onin its handler anddel_timer_sync(which is incorrect).
arch/x86/kernel/apic/x2apic_uv_x.calmost has it right, but maybe it should usedel_timer_sync?
arch/powerpc/platforms/chrp/setup.cshould learn aboutmod_timer_pinned().Which leads to the following question: is there _any_
add_timer_on()kernel user that's not currently buggy ? ;-) Maybe this calls for better documentation of this interface. From what I've learn from digging into cpufreq to debug its incorrect timer teardown last year, I fear there are lots and lots of buggy per-cpu _and_ standard timer interface users out there.Maybe adding some debugging options, e.g. checking that a timer created with
add_timer_onis always modified bymod_timer_pinned, and is always deferrable, and deleted bydel_timer_synccould help discovering a couple of outlawyer.Thanks,
Mathieu
Quoted 20 lines
Thanks,
Mathieu
Quoted 5 lines
Thanks,
tglx
Signature · 7 lines
Mathieu Desnoyers OpenPGP key fingerprint: 8CD5 52C3 8E3C 4140 715F BA06 3F25 A8FE 3BAE 9A68 _______________________________________________ ltt-dev mailing list [email protected] http://lists.casi.polymtl.ca/cgi-bin/mailman/listinfo/ltt-devSignature · 2 lines
Mathieu Desnoyers OpenPGP key fingerprint: 8CD5 52C3 8E3C 4140 715F BA06 3F25 A8FE 3BAE 9A68
Quoted 72 lines · Mathieu Desnoyers
Quoted 58 lines · * Neil Horman ([email protected]) wrote:
Quoted 57 lines · Mathieu Desnoyers
Quoted 22 lines · * Mathieu Desnoyers ([email protected]) wrote:
Quoted 21 lines · * Thomas Gleixner ([email protected]) wrote:
Quoted 6 lines · Mathieu Desnoyers
Quoted 1 line
The function is called from an IPI. That's a LTTNG problem, not a RT one.
I use
del_timerin IPI to delete lttng per-cpu timers on all CPUs. I have to do this because timers created withadd_timer_onare documented to be incompatible withdel_timer_sync():Quoted 6 lines
* Synchronization rules: Callers must prevent restarting of the timer,
* otherwise this function is meaningless. It must not be called from
* interrupt contexts. The caller must not hold locks which would prevent
* completion of the timer's handler. The timer's handler must not call
*add_timer_on(). Upon exit the timer is not queued and the handler is
* not running on any CPU.Errm. The documentation says:
"The timer's handler must not call
add_timer_on()."It's not talking about a timer which was initialized with
add_timer_on().Quoted 8 lines
Quoted 5 lines
And your per cpu timer handlers have no requirement to call
add_timer_on()simply because add/mod_timer() is requeueing the timer on the same cpu on which the handler runs.So the IPI is just a solution for a non existing problem.
Oh, right. Thanks for the explanation. I'll look into moving LTTng to a saner
del_timer_sync()scheme to delete the timers.Double-checking this:
add_timer_on()needs to be paired withmod_timer_pinned(), otherwiseNO_HZSMP config can rebalance the timer to a different CPU. I am fixing this in lttng 0.194. These per-cpu timers, of course, should usually be deferrable (they are in lttng).(looking at kernel 2.6.32.4 here)
Looking at thekernel/time/clocksource.cwatchdog, I wonder howdel_timermanages to synchronize the timer teardown. The handler,clocksource_watchdog(), usesadd_timer_on(), which prohibits usingdel_timer_sync(). This seems rather odd. If we remove the watchdog and re-add it, it may still be in use while we initialize the timer structure.Also,
net/core/drop_monitor.ctrace_drop_commonusage ofadd_timer_onseems odd:Executing (AFAIK) with preempt on, data points to a per-cpu timer:
if (!timer_pending(&data->send_timer)) { data->send_timer.expires = jiffies + dm_delay * HZ; add_timer_on(&data->send_timer, smp_processor_id()); }Quoted 6 lines
Quoted 2 lines
How is
timer_pendingsynchronized with the target CPU timer wheel ?Hm, I think I see your point here. You're suggesting that a call to one of the tracepoint hooks in the drop monitor can race against a second call to the hook from an interrupt context that pre-empted the first, leading to double add of the timer?
Yes.
Quoted 3 lines
I agree, in fact I think its likely worse that that, the shared data on the skb that I modify there can get corrupted in that case as well. I expect a bit of refactoring paired with a local_irq_save/restore should fix that.
Quoted 10 lines
For this current code, you don't seem to need to delete the timer. I think you'll have to find a way to do the
timer_pendingcheck and theadd_timer_onoperations atomically wrt timer wheel execution on the CPU (which can be a remote cpu because preemption is enabled). Your case is a bit weird.. the typical scenario here would be to useadd_timer_ondirectly to re-arm the timer for a future expiration time (removing the current pending timer unconditionnally at the same time). But you leave the timer at its current expiration time if present, and only if not present add it. I wonder if it's really your intent ?
Yes, thats my intent. The idea is that when I note a packet is lost, I record that in a data buffer that is part of an skb, and schedule a timer to send that skb a second or so later. Other drops in that time frame are recorded within the same skb. Since the drop monitor isn't built as a module currently, I don't need to worry about deleting the timer at all, I can just let it expire.
Sounds like I need to wrap up the data extract and time modifications in a prempt_disable/enable and local_irq_save/restore set
Neil
Quoted 5 lines
Thanks,
Mathieu
Quoted 90 lines · * Neil Horman ([email protected]) wrote:
Quoted 72 lines · Mathieu Desnoyers
Quoted 58 lines · * Neil Horman ([email protected]) wrote:
Quoted 57 lines · Mathieu Desnoyers
Quoted 22 lines · * Mathieu Desnoyers ([email protected]) wrote:
Quoted 21 lines · * Thomas Gleixner ([email protected]) wrote:
Quoted 6 lines · Mathieu Desnoyers
Quoted 1 line
The function is called from an IPI. That's a LTTNG problem, not a RT one.
I use
del_timerin IPI to delete lttng per-cpu timers on all CPUs. I have to do this because timers created withadd_timer_onare documented to be incompatible withdel_timer_sync():Quoted 6 lines
* Synchronization rules: Callers must prevent restarting of the timer,
* otherwise this function is meaningless. It must not be called from
* interrupt contexts. The caller must not hold locks which would prevent
* completion of the timer's handler. The timer's handler must not call
*add_timer_on(). Upon exit the timer is not queued and the handler is
* not running on any CPU.Errm. The documentation says:
"The timer's handler must not call
add_timer_on()."It's not talking about a timer which was initialized with
add_timer_on().Quoted 8 lines
Quoted 5 lines
And your per cpu timer handlers have no requirement to call
add_timer_on()simply because add/mod_timer() is requeueing the timer on the same cpu on which the handler runs.So the IPI is just a solution for a non existing problem.
Oh, right. Thanks for the explanation. I'll look into moving LTTng to a saner
del_timer_sync()scheme to delete the timers.Double-checking this:
add_timer_on()needs to be paired withmod_timer_pinned(), otherwiseNO_HZSMP config can rebalance the timer to a different CPU. I am fixing this in lttng 0.194. These per-cpu timers, of course, should usually be deferrable (they are in lttng).(looking at kernel 2.6.32.4 here)
Looking at thekernel/time/clocksource.cwatchdog, I wonder howdel_timermanages to synchronize the timer teardown. The handler,clocksource_watchdog(), usesadd_timer_on(), which prohibits usingdel_timer_sync(). This seems rather odd. If we remove the watchdog and re-add it, it may still be in use while we initialize the timer structure.Also,
net/core/drop_monitor.ctrace_drop_commonusage ofadd_timer_onseems odd:Executing (AFAIK) with preempt on, data points to a per-cpu timer:
if (!timer_pending(&data->send_timer)) { data->send_timer.expires = jiffies + dm_delay * HZ; add_timer_on(&data->send_timer, smp_processor_id()); }Quoted 6 lines
Quoted 2 lines
How is
timer_pendingsynchronized with the target CPU timer wheel ?Hm, I think I see your point here. You're suggesting that a call to one of the tracepoint hooks in the drop monitor can race against a second call to the hook from an interrupt context that pre-empted the first, leading to double add of the timer?
Yes.
Quoted 3 lines
I agree, in fact I think its likely worse that that, the shared data on the skb that I modify there can get corrupted in that case as well. I expect a bit of refactoring paired with a local_irq_save/restore should fix that.
Quoted 10 lines
For this current code, you don't seem to need to delete the timer. I think you'll have to find a way to do the
timer_pendingcheck and theadd_timer_onoperations atomically wrt timer wheel execution on the CPU (which can be a remote cpu because preemption is enabled). Your case is a bit weird.. the typical scenario here would be to useadd_timer_ondirectly to re-arm the timer for a future expiration time (removing the current pending timer unconditionnally at the same time). But you leave the timer at its current expiration time if present, and only if not present add it. I wonder if it's really your intent ?Yes, thats my intent. The idea is that when I note a packet is lost, I record that in a data buffer that is part of an skb, and schedule a timer to send that skb a second or so later. Other drops in that time frame are recorded within the same skb. Since the drop monitor isn't built as a module currently, I don't need to worry about deleting the timer at all, I can just let it expire.
Quoted 3 lines
Sounds like I need to wrap up the data extract and time modifications in a prempt_disable/enable and local_irq_save/restore set
Neil
I fear this will probably break the RT kernel, because the timer spinlocks become sleepable (I had the same issue within LTTng). Another approach you could try is to add a new member to the timer API, which hold the local timer base lock while:
if the timer is already pending
- waits for already executing timers to complete their execution.
- add_timer_on local cpu with new expiration.
else
- do nothing
Thanks,
Mathieu
Quoted 6 lines
Quoted 6 lines
Thanks,
Mathieu