Troubles with switchtest
Hi Gilles,
I'm currently facing a nasty effect with switchtest over latest git head (only tested this so far): running it inside my test VM (ie. with frequent excessive latencies) I get a stalled Linux timer IRQ quite quickly. System is otherwise still responsive, Xenomai timers are still being delivered, other Linux IRQs too. switchtest complained about
"Warning: Linux is compiled to use FPU in kernel-space."
when it was started. Kernels are 2.6.28.9/ipipe-x86-2.2-07 and 2.6.29.3/ipipe-x86-2.3-01 (LTTng patched in, but unused), both show the same effect.
Seen this before?
Jan
Signature · 2 lines
Siemens AG, Corporate Technology, CT SE 2 Corporate Competence Center Embedded Linux
Quoted 8 lines · Jan Kiszka
Hi Gilles,
I'm currently facing a nasty effect with switchtest over latest git head (only tested this so far): running it inside my test VM (ie. with frequent excessive latencies) I get a stalled Linux timer IRQ quite quickly. System is otherwise still responsive, Xenomai timers are still being delivered, other Linux IRQs too. switchtest complained about
Quoted 7 lines
"Warning: Linux is compiled to use FPU in kernel-space."
when it was started. Kernels are 2.6.28.9/ipipe-x86-2.2-07 and 2.6.29.3/ipipe-x86-2.3-01 (LTTng patched in, but unused), both show the same effect.
Seen this before?
The warning about Linux being compiled to use FPU in kernel-space means that you enabled soft RAID or compiled for K7, Geode, or any other configuration using 3DNow for such simple operations as memcpy. It is harmless, it simply means that switchtest can not use fpu in kernel-space.
The bug you have is probably the same as the one described here, which I am able to reproduce on my atom:
https://mail.gna.org/public/xenomai-help/2009-04/msg00200.html
Unfortunately, I for one am working on ARM issues and am not available to debug x86 issues. I think Philippe is busy too...
Signature · 1 line
Gilles.
Quoted 11 lines · Gilles Chanteperdrix
Quoted 10 lines · Jan Kiszka
Hi Gilles,
I'm currently facing a nasty effect with switchtest over latest git head (only tested this so far): running it inside my test VM (ie. with frequent excessive latencies) I get a stalled Linux timer IRQ quite quickly. System is otherwise still responsive, Xenomai timers are still being delivered, other Linux IRQs too. switchtest complained about
"Warning: Linux is compiled to use FPU in kernel-space."
Quoted 8 lines
Quoted 5 lines
when it was started. Kernels are 2.6.28.9/ipipe-x86-2.2-07 and 2.6.29.3/ipipe-x86-2.3-01 (LTTng patched in, but unused), both show the same effect.
Seen this before?
The warning about Linux being compiled to use FPU in kernel-space means that you enabled soft RAID or compiled for K7, Geode, or any other
RAID is on (ordinary server config).
Quoted 9 lines
configuration using 3DNow for such simple operations as memcpy. It is harmless, it simply means that switchtest can not use fpu in kernel-space.
The bug you have is probably the same as the one described here, which I am able to reproduce on my atom:
https://mail.gna.org/public/xenomai-help/2009-04/msg00200.htmlUnfortunately, I for one am working on ARM issues and am not available to debug x86 issues. I think Philippe is busy too...
OK, looks like I got the same flu here.
Philippe, did you find out any more details in the meantime? Then I'm afraid I have to pick this up.
Jan
Signature · 2 lines
Siemens AG, Corporate Technology, CT SE 2 Corporate Competence Center Embedded Linux
Quoted 30 lines · Jan Kiszka
Quoted 11 lines · Gilles Chanteperdrix
Quoted 10 lines · Jan Kiszka
Hi Gilles,
I'm currently facing a nasty effect with switchtest over latest git head (only tested this so far): running it inside my test VM (ie. with frequent excessive latencies) I get a stalled Linux timer IRQ quite quickly. System is otherwise still responsive, Xenomai timers are still being delivered, other Linux IRQs too. switchtest complained about
"Warning: Linux is compiled to use FPU in kernel-space."
Quoted 8 lines
Quoted 5 lines
when it was started. Kernels are 2.6.28.9/ipipe-x86-2.2-07 and 2.6.29.3/ipipe-x86-2.3-01 (LTTng patched in, but unused), both show the same effect.
Seen this before?
The warning about Linux being compiled to use FPU in kernel-space means that you enabled soft RAID or compiled for K7, Geode, or any other
RAID is on (ordinary server config).
Quoted 7 lines
configuration using 3DNow for such simple operations as memcpy. It is harmless, it simply means that switchtest can not use fpu in kernel-space.
The bug you have is probably the same as the one described here, which I am able to reproduce on my atom:
https://mail.gna.org/public/xenomai-help/2009-04/msg00200.html
Quoted 7 lines
Quoted 2 lines
Unfortunately, I for one am working on ARM issues and am not available to debug x86 issues. I think Philippe is busy too...
OK, looks like I got the same flu here.
Philippe, did you find out any more details in the meantime? Then I'm afraid I have to pick this up.
No, I did not resume this task yet. Working from the powerpc side of the universe here.
Quoted 3 lines
Jan
Signature · 1 line
Philippe.
PATCH 1/1 Fix host IRQ propagation (was: Troubles with switchtest)
Quoted 31 lines · Philippe Gerum
Quoted 30 lines · Jan Kiszka
Quoted 11 lines · Gilles Chanteperdrix
Quoted 10 lines · Jan Kiszka
Hi Gilles,
I'm currently facing a nasty effect with switchtest over latest git head (only tested this so far): running it inside my test VM (ie. with frequent excessive latencies) I get a stalled Linux timer IRQ quite quickly. System is otherwise still responsive, Xenomai timers are still being delivered, other Linux IRQs too. switchtest complained about
"Warning: Linux is compiled to use FPU in kernel-space."
Quoted 7 lines
Quoted 5 lines
when it was started. Kernels are 2.6.28.9/ipipe-x86-2.2-07 and 2.6.29.3/ipipe-x86-2.3-01 (LTTng patched in, but unused), both show the same effect.
Seen this before?
The warning about Linux being compiled to use FPU in kernel-space means that you enabled soft RAID or compiled for K7, Geode, or any other
RAID is on (ordinary server config).
Quoted 9 lines
configuration using 3DNow for such simple operations as memcpy. It is harmless, it simply means that switchtest can not use fpu in kernel-space.
The bug you have is probably the same as the one described here, which I am able to reproduce on my atom:
https://mail.gna.org/public/xenomai-help/2009-04/msg00200.htmlUnfortunately, I for one am working on ARM issues and am not available to debug x86 issues. I think Philippe is busy too...
Quoted 7 lines
Quoted 4 lines
OK, looks like I got the same flu here.
Philippe, did you find out any more details in the meantime? Then I'm afraid I have to pick this up.
No, I did not resume this task yet. Working from the powerpc side of the universe here.
Hoho, don't think this rain here over x86 would have never made it down to ARM or PPC land! ;)
Martin, could you check if this helps you, too?
Jan
(as usual, ready to be pulled from 'for-upstream')
--------->
Host IRQs may not only be triggered from non-root domains. But rthal_propagate_irq's implemenation in I-pipe assumes so, which broke host tick propagation under certain load scenarios. Besides that, rthal_schedule_irq_root is the more efficient service for this purpose anyway.
---
include/asm-generic/hal.h | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-)
include/asm-generic/hal.h +1 -1
@@ -437,7 +437,7 @@ int rthal_irq_host_release(unsigned irq,static inline void rthal_irq_host_pend (unsigned irq) { -rthal_propagate_irq (irq); +rthal_schedule_irq_root (irq); } int rthal_apc_alloc(const char *name,
Signature · 2 lines
Siemens AG, Corporate Technology, CT SE 2 Corporate Competence Center Embedded Linux
Re: [Xenomai-core] [PATCH] Fix host IRQ propagation (was: Troubles with switchtest)
Quoted 45 lines · Jan Kiszka
Quoted 31 lines · Philippe Gerum
Quoted 30 lines · Jan Kiszka
Quoted 11 lines · Gilles Chanteperdrix
Quoted 10 lines · Jan Kiszka
Hi Gilles,
I'm currently facing a nasty effect with switchtest over latest git head (only tested this so far): running it inside my test VM (ie. with frequent excessive latencies) I get a stalled Linux timer IRQ quite quickly. System is otherwise still responsive, Xenomai timers are still being delivered, other Linux IRQs too. switchtest complained about
"Warning: Linux is compiled to use FPU in kernel-space."
Quoted 7 lines
Quoted 5 lines
when it was started. Kernels are 2.6.28.9/ipipe-x86-2.2-07 and 2.6.29.3/ipipe-x86-2.3-01 (LTTng patched in, but unused), both show the same effect.
Seen this before?
The warning about Linux being compiled to use FPU in kernel-space means that you enabled soft RAID or compiled for K7, Geode, or any other
RAID is on (ordinary server config).
Quoted 9 lines
configuration using 3DNow for such simple operations as memcpy. It is harmless, it simply means that switchtest can not use fpu in kernel-space.
The bug you have is probably the same as the one described here, which I am able to reproduce on my atom:
https://mail.gna.org/public/xenomai-help/2009-04/msg00200.htmlUnfortunately, I for one am working on ARM issues and am not available to debug x86 issues. I think Philippe is busy too...
Quoted 7 lines
Quoted 4 lines
OK, looks like I got the same flu here.
Philippe, did you find out any more details in the meantime? Then I'm afraid I have to pick this up.
No, I did not resume this task yet. Working from the powerpc side of the universe here.
Hoho, don't think this rain here over x86 would have never made it down to ARM or PPC land! ;)
Martin, could you check if this helps you, too?
Quoted 7 lines
Jan
(as usual, ready to be pulled from 'for-upstream')
--------->
Host IRQs may not only be triggered from non-root domains.
Are you sure of this? I can't find any spot where this assumption would be wrong. host_pend() is basically there to relay RT timer ticks and device IRQs, and this only happens on behalf of the pipeline head. At least, this is how rthal_irq_host_pend() should be used in any case. If you did find a spot where this interface is being called from the lower stage, then this is the root bug to fix.
Quoted 5 lines
But
rthal_propagate_irq's implemenation in I-pipe assumes so, which broke host tick propagation under certain load scenarios. Besides that,rthal_schedule_irq_rootis the more efficient service for this purpose anyway.
Ack.
Quoted 21 lines
Signed-off-by Jan Kiszka <[email protected]>---
include/asm-generic/hal.h | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-)include/asm-generic/hal.h +1
-1@@ -437,7 +437,7 @@ int rthal_irq_host_release(unsigned irq,static inline void rthal_irq_host_pend (unsigned irq) { -rthal_propagate_irq (irq); +rthal_schedule_irq_root (irq); } int rthal_apc_alloc(const char *name,
Signature · 1 line
Philippe.
Re: [Xenomai-core] [PATCH] Fix host IRQ propagation
Quoted 50 lines · Philippe Gerum
Quoted 49 lines · Jan Kiszka
Quoted 29 lines · Philippe Gerum
Quoted 28 lines · Jan Kiszka
Quoted 11 lines · Gilles Chanteperdrix
Quoted 10 lines · Jan Kiszka
Hi Gilles,
I'm currently facing a nasty effect with switchtest over latest git head (only tested this so far): running it inside my test VM (ie. with frequent excessive latencies) I get a stalled Linux timer IRQ quite quickly. System is otherwise still responsive, Xenomai timers are still being delivered, other Linux IRQs too. switchtest complained about
"Warning: Linux is compiled to use FPU in kernel-space."
Quoted 7 lines
Quoted 5 lines
when it was started. Kernels are 2.6.28.9/ipipe-x86-2.2-07 and 2.6.29.3/ipipe-x86-2.3-01 (LTTng patched in, but unused), both show the same effect.
Seen this before?
The warning about Linux being compiled to use FPU in kernel-space means that you enabled soft RAID or compiled for K7, Geode, or any other
RAID is on (ordinary server config).
Quoted 7 lines
configuration using 3DNow for such simple operations as memcpy. It is harmless, it simply means that switchtest can not use fpu in kernel-space.
The bug you have is probably the same as the one described here, which I am able to reproduce on my atom:
https://mail.gna.org/public/xenomai-help/2009-04/msg00200.htmlQuoted 8 lines
Quoted 6 lines
Quoted 2 lines
Unfortunately, I for one am working on ARM issues and am not available to debug x86 issues. I think Philippe is busy too...
OK, looks like I got the same flu here.
Philippe, did you find out any more details in the meantime? Then I'm afraid I have to pick this up.
No, I did not resume this task yet. Working from the powerpc side of the universe here.
Hoho, don't think this rain here over x86 would have never made it down to ARM or PPC land! ;)
Martin, could you check if this helps you, too?
Jan
(as usual, ready to be pulled from 'for-upstream')
--------->
Quoted 8 lines
Quoted 1 line
Host IRQs may not only be triggered from non-root domains.
Are you sure of this? I can't find any spot where this assumption would be wrong.
host_pend()is basically there to relay RT timer ticks and device IRQs, and this only happens on behalf of the pipeline head. At least, this is howrthal_irq_host_pend()should be used in any case. If you did find a spot where this interface is being called from the lower stage, then this is the root bug to fix.
I haven't studied the I-pipe trace /wrt this in details yet, but I could imagine that some shadow task is interrupted in primary mode by the timer IRQ and then leaves the handler in secondary mode due to whatever events between schedule-out and in at the end of xnintr_clock_handler.
Jan
Signature · 2 lines
Siemens AG, Corporate Technology, CT SE 2 Corporate Competence Center Embedded Linux
Re: [Xenomai-core] [PATCH] Fix host IRQ propagation
Quoted 60 lines · Jan Kiszka
Quoted 50 lines · Philippe Gerum
Quoted 49 lines · Jan Kiszka
Quoted 29 lines · Philippe Gerum
Quoted 28 lines · Jan Kiszka
Quoted 11 lines · Gilles Chanteperdrix
Quoted 10 lines · Jan Kiszka
Hi Gilles,
I'm currently facing a nasty effect with switchtest over latest git head (only tested this so far): running it inside my test VM (ie. with frequent excessive latencies) I get a stalled Linux timer IRQ quite quickly. System is otherwise still responsive, Xenomai timers are still being delivered, other Linux IRQs too. switchtest complained about
"Warning: Linux is compiled to use FPU in kernel-space."
Quoted 7 lines
Quoted 5 lines
when it was started. Kernels are 2.6.28.9/ipipe-x86-2.2-07 and 2.6.29.3/ipipe-x86-2.3-01 (LTTng patched in, but unused), both show the same effect.
Seen this before?
The warning about Linux being compiled to use FPU in kernel-space means that you enabled soft RAID or compiled for K7, Geode, or any other
RAID is on (ordinary server config).
Quoted 7 lines
configuration using 3DNow for such simple operations as memcpy. It is harmless, it simply means that switchtest can not use fpu in kernel-space.
The bug you have is probably the same as the one described here, which I am able to reproduce on my atom:
https://mail.gna.org/public/xenomai-help/2009-04/msg00200.htmlQuoted 8 lines
Quoted 6 lines
Quoted 2 lines
Unfortunately, I for one am working on ARM issues and am not available to debug x86 issues. I think Philippe is busy too...
OK, looks like I got the same flu here.
Philippe, did you find out any more details in the meantime? Then I'm afraid I have to pick this up.
No, I did not resume this task yet. Working from the powerpc side of the universe here.
Hoho, don't think this rain here over x86 would have never made it down to ARM or PPC land! ;)
Martin, could you check if this helps you, too?
Jan
(as usual, ready to be pulled from 'for-upstream')
--------->
Quoted 8 lines
Quoted 1 line
Host IRQs may not only be triggered from non-root domains.
Are you sure of this? I can't find any spot where this assumption would be wrong.
host_pend()is basically there to relay RT timer ticks and device IRQs, and this only happens on behalf of the pipeline head. At least, this is howrthal_irq_host_pend()should be used in any case. If you did find a spot where this interface is being called from the lower stage, then this is the root bug to fix.
Quoted 5 lines
I haven't studied the I-pipe trace /wrt this in details yet, but I could imagine that some shadow task is interrupted in primary mode by the timer IRQ and then leaves the handler in secondary mode due to whatever events between schedule-out and in at the end of
xnintr_clock_handler.
You need a thread context to move to secondary, I just can't see how such scenario would be possible.
Quoted 2 lines
Jan
Signature · 1 line
Philippe.
Re: [Xenomai-core] [PATCH] Fix host IRQ propagation
Quoted 59 lines · Philippe Gerum
Quoted 58 lines · Jan Kiszka
Quoted 57 lines · Philippe Gerum
Quoted 43 lines · Jan Kiszka
Quoted 29 lines · Philippe Gerum
Quoted 28 lines · Jan Kiszka
Quoted 11 lines · Gilles Chanteperdrix
Quoted 10 lines · Jan Kiszka
Hi Gilles,
I'm currently facing a nasty effect with switchtest over latest git head (only tested this so far): running it inside my test VM (ie. with frequent excessive latencies) I get a stalled Linux timer IRQ quite quickly. System is otherwise still responsive, Xenomai timers are still being delivered, other Linux IRQs too. switchtest complained about
"Warning: Linux is compiled to use FPU in kernel-space."
Quoted 7 lines
Quoted 5 lines
when it was started. Kernels are 2.6.28.9/ipipe-x86-2.2-07 and 2.6.29.3/ipipe-x86-2.3-01 (LTTng patched in, but unused), both show the same effect.
Seen this before?
The warning about Linux being compiled to use FPU in kernel-space means that you enabled soft RAID or compiled for K7, Geode, or any other
RAID is on (ordinary server config).
Quoted 7 lines
configuration using 3DNow for such simple operations as memcpy. It is harmless, it simply means that switchtest can not use fpu in kernel-space.
The bug you have is probably the same as the one described here, which I am able to reproduce on my atom:
https://mail.gna.org/public/xenomai-help/2009-04/msg00200.htmlQuoted 8 lines
Quoted 6 lines
Quoted 2 lines
Unfortunately, I for one am working on ARM issues and am not available to debug x86 issues. I think Philippe is busy too...
OK, looks like I got the same flu here.
Philippe, did you find out any more details in the meantime? Then I'm afraid I have to pick this up.
No, I did not resume this task yet. Working from the powerpc side of the universe here.
Hoho, don't think this rain here over x86 would have never made it down to ARM or PPC land! ;)
Martin, could you check if this helps you, too?
Quoted 7 lines
Jan
(as usual, ready to be pulled from 'for-upstream')
--------->
Host IRQs may not only be triggered from non-root domains.
Are you sure of this? I can't find any spot where this assumption would be wrong.
host_pend()is basically there to relay RT timer ticks and device IRQs, and this only happens on behalf of the pipeline head. At least, this is howrthal_irq_host_pend()should be used in any case. If you did find a spot where this interface is being called from the lower stage, then this is the root bug to fix.
Quoted 8 lines
Quoted 5 lines
I haven't studied the I-pipe trace /wrt this in details yet, but I could imagine that some shadow task is interrupted in primary mode by the timer IRQ and then leaves the handler in secondary mode due to whatever events between schedule-out and in at the end of
xnintr_clock_handler.You need a thread context to move to secondary, I just can't see how such scenario would be possible.
Here is the trace of events:
=> Shadow task starts migration to secondary
=> in xnpod_suspend_thread, nklock is briefly released before
xnpod_schedule
=> timer IRQ intercepts
=> as the current CPU is marked for reschedule, we enter xnpod_schedule
before propagating the host tick
=> once the migrating thread comes in again, it will run the
xnintr_clock_handler tail, i.e. xnarch_relay_tick, already over the root domain
Jan
Signature · 2 lines
Siemens AG, Corporate Technology, CT SE 2 Corporate Competence Center Embedded Linux
Re: [Xenomai-core] [PATCH] Fix host IRQ propagation
Quoted 66 lines · Jan Kiszka
Quoted 65 lines · Philippe Gerum
Quoted 63 lines · Jan Kiszka
Quoted 50 lines · Philippe Gerum
Quoted 49 lines · Jan Kiszka
Quoted 29 lines · Philippe Gerum
Quoted 28 lines · Jan Kiszka
Quoted 11 lines · Gilles Chanteperdrix
Quoted 10 lines · Jan Kiszka
Hi Gilles,
I'm currently facing a nasty effect with switchtest over latest git head (only tested this so far): running it inside my test VM (ie. with frequent excessive latencies) I get a stalled Linux timer IRQ quite quickly. System is otherwise still responsive, Xenomai timers are still being delivered, other Linux IRQs too. switchtest complained about
"Warning: Linux is compiled to use FPU in kernel-space."
Quoted 7 lines
Quoted 5 lines
when it was started. Kernels are 2.6.28.9/ipipe-x86-2.2-07 and 2.6.29.3/ipipe-x86-2.3-01 (LTTng patched in, but unused), both show the same effect.
Seen this before?
The warning about Linux being compiled to use FPU in kernel-space means that you enabled soft RAID or compiled for K7, Geode, or any other
RAID is on (ordinary server config).
Quoted 7 lines
configuration using 3DNow for such simple operations as memcpy. It is harmless, it simply means that switchtest can not use fpu in kernel-space.
The bug you have is probably the same as the one described here, which I am able to reproduce on my atom:
https://mail.gna.org/public/xenomai-help/2009-04/msg00200.htmlQuoted 8 lines
Quoted 6 lines
Quoted 2 lines
Unfortunately, I for one am working on ARM issues and am not available to debug x86 issues. I think Philippe is busy too...
OK, looks like I got the same flu here.
Philippe, did you find out any more details in the meantime? Then I'm afraid I have to pick this up.
No, I did not resume this task yet. Working from the powerpc side of the universe here.
Hoho, don't think this rain here over x86 would have never made it down to ARM or PPC land! ;)
Martin, could you check if this helps you, too?
Jan
(as usual, ready to be pulled from 'for-upstream')
--------->
Quoted 7 lines
Quoted 1 line
Host IRQs may not only be triggered from non-root domains.
Are you sure of this? I can't find any spot where this assumption would be wrong.
host_pend()is basically there to relay RT timer ticks and device IRQs, and this only happens on behalf of the pipeline head. At least, this is howrthal_irq_host_pend()should be used in any case. If you did find a spot where this interface is being called from the lower stage, then this is the root bug to fix.I haven't studied the I-pipe trace /wrt this in details yet, but I could imagine that some shadow task is interrupted in primary mode by the timer IRQ and then leaves the handler in secondary mode due to whatever events between schedule-out and in at the end of
xnintr_clock_handler.
Quoted 8 lines
Quoted 2 lines
You need a thread context to move to secondary, I just can't see how such scenario would be possible.
Here is the trace of events:
=> Shadow task starts migration to secondary
=> inxnpod_suspend_thread, nklock is briefly released before
xnpod_schedule
Which is the root bug. Blame on me; this recent change in -head breaks a basic rule a lot of code is based on: a self-suspending thread may not be preempted while scheduling out, i.e. suspension and rescheduling must be atomically performed. xnshadow_relax() counts on this too.
Quoted 6 lines
=> timer IRQ intercepts
=> as the current CPU is marked for reschedule, we enterxnpod_schedule
before propagating the host tick
=> once the migrating thread comes in again, it will run the
xnintr_clock_handlertail, i.e.xnarch_relay_tick, already over the root domain
Ok, makes sense now. However, this can't happen with 2.4 which has no such lock release in xnpod_suspend_thread(). So the question is: was the "lost tick" bug observed also on 2.4, or not?
Quoted 3 lines
Jan
Signature · 1 line
Philippe.
Re: [Xenomai-core] [PATCH] Fix host IRQ propagation
Quoted 70 lines · Philippe Gerum
Quoted 69 lines · Jan Kiszka
Quoted 59 lines · Philippe Gerum
Quoted 58 lines · Jan Kiszka
Quoted 57 lines · Philippe Gerum
Quoted 43 lines · Jan Kiszka
Quoted 29 lines · Philippe Gerum
Quoted 28 lines · Jan Kiszka
Quoted 11 lines · Gilles Chanteperdrix
Quoted 10 lines · Jan Kiszka
Hi Gilles,
I'm currently facing a nasty effect with switchtest over latest git head (only tested this so far): running it inside my test VM (ie. with frequent excessive latencies) I get a stalled Linux timer IRQ quite quickly. System is otherwise still responsive, Xenomai timers are still being delivered, other Linux IRQs too. switchtest complained about
"Warning: Linux is compiled to use FPU in kernel-space."
Quoted 7 lines
Quoted 5 lines
when it was started. Kernels are 2.6.28.9/ipipe-x86-2.2-07 and 2.6.29.3/ipipe-x86-2.3-01 (LTTng patched in, but unused), both show the same effect.
Seen this before?
The warning about Linux being compiled to use FPU in kernel-space means that you enabled soft RAID or compiled for K7, Geode, or any other
RAID is on (ordinary server config).
Quoted 7 lines
configuration using 3DNow for such simple operations as memcpy. It is harmless, it simply means that switchtest can not use fpu in kernel-space.
The bug you have is probably the same as the one described here, which I am able to reproduce on my atom:
https://mail.gna.org/public/xenomai-help/2009-04/msg00200.htmlQuoted 8 lines
Quoted 6 lines
Quoted 2 lines
Unfortunately, I for one am working on ARM issues and am not available to debug x86 issues. I think Philippe is busy too...
OK, looks like I got the same flu here.
Philippe, did you find out any more details in the meantime? Then I'm afraid I have to pick this up.
No, I did not resume this task yet. Working from the powerpc side of the universe here.
Hoho, don't think this rain here over x86 would have never made it down to ARM or PPC land! ;)
Martin, could you check if this helps you, too?
Quoted 7 lines
Jan
(as usual, ready to be pulled from 'for-upstream')
--------->
Host IRQs may not only be triggered from non-root domains.
Are you sure of this? I can't find any spot where this assumption would be wrong.
host_pend()is basically there to relay RT timer ticks and device IRQs, and this only happens on behalf of the pipeline head. At least, this is howrthal_irq_host_pend()should be used in any case. If you did find a spot where this interface is being called from the lower stage, then this is the root bug to fix.Quoted 7 lines
Quoted 5 lines
I haven't studied the I-pipe trace /wrt this in details yet, but I could imagine that some shadow task is interrupted in primary mode by the timer IRQ and then leaves the handler in secondary mode due to whatever events between schedule-out and in at the end of
xnintr_clock_handler.You need a thread context to move to secondary, I just can't see how such scenario would be possible.
Here is the trace of events:
Quoted 8 lines
Quoted 3 lines
=> Shadow task starts migration to secondary
=> inxnpod_suspend_thread, nklock is briefly released before
xnpod_scheduleWhich is the root bug. Blame on me; this recent change in -head breaks a basic rule a lot of code is based on: a self-suspending thread may not be preempted while scheduling out, i.e. suspension and rescheduling must be atomically performed.
xnshadow_relax()counts on this too.
Oh, good that you insisted on this. Will you fix it soon? We are currently packaging a delivery of 2.5.git, and I would like to see this hole closed there already.
Quoted 11 lines
Quoted 7 lines
=> timer IRQ intercepts
=> as the current CPU is marked for reschedule, we enterxnpod_schedule
before propagating the host tick
=> once the migrating thread comes in again, it will run the
xnintr_clock_handlertail, i.e.xnarch_relay_tick, already over the root domainOk, makes sense now. However, this can't happen with 2.4 which has no such lock release in
xnpod_suspend_thread(). So the question is: was the "lost tick" bug observed also on 2.4, or not?
I haven't tested on 2.4, but Martin anyway reported that his problem is still unfixed for 2.5 even with my patch.
Jan
Signature · 2 lines
Siemens AG, Corporate Technology, CT SE 2 Corporate Competence Center Embedded Linux
Re: [Xenomai-core] [PATCH] Fix host IRQ propagation
Quoted 75 lines · Jan Kiszka
Quoted 74 lines · Philippe Gerum
Quoted 72 lines · Jan Kiszka
Quoted 59 lines · Philippe Gerum
Quoted 58 lines · Jan Kiszka
Quoted 57 lines · Philippe Gerum
Quoted 43 lines · Jan Kiszka
Quoted 29 lines · Philippe Gerum
Quoted 28 lines · Jan Kiszka
Quoted 11 lines · Gilles Chanteperdrix
Quoted 10 lines · Jan Kiszka
Hi Gilles,
I'm currently facing a nasty effect with switchtest over latest git head (only tested this so far): running it inside my test VM (ie. with frequent excessive latencies) I get a stalled Linux timer IRQ quite quickly. System is otherwise still responsive, Xenomai timers are still being delivered, other Linux IRQs too. switchtest complained about
"Warning: Linux is compiled to use FPU in kernel-space."
Quoted 7 lines
Quoted 5 lines
when it was started. Kernels are 2.6.28.9/ipipe-x86-2.2-07 and 2.6.29.3/ipipe-x86-2.3-01 (LTTng patched in, but unused), both show the same effect.
Seen this before?
The warning about Linux being compiled to use FPU in kernel-space means that you enabled soft RAID or compiled for K7, Geode, or any other
RAID is on (ordinary server config).
Quoted 7 lines
configuration using 3DNow for such simple operations as memcpy. It is harmless, it simply means that switchtest can not use fpu in kernel-space.
The bug you have is probably the same as the one described here, which I am able to reproduce on my atom:
https://mail.gna.org/public/xenomai-help/2009-04/msg00200.htmlQuoted 8 lines
Quoted 6 lines
Quoted 2 lines
Unfortunately, I for one am working on ARM issues and am not available to debug x86 issues. I think Philippe is busy too...
OK, looks like I got the same flu here.
Philippe, did you find out any more details in the meantime? Then I'm afraid I have to pick this up.
No, I did not resume this task yet. Working from the powerpc side of the universe here.
Hoho, don't think this rain here over x86 would have never made it down to ARM or PPC land! ;)
Martin, could you check if this helps you, too?
Quoted 7 lines
Jan
(as usual, ready to be pulled from 'for-upstream')
--------->
Host IRQs may not only be triggered from non-root domains.
Are you sure of this? I can't find any spot where this assumption would be wrong.
host_pend()is basically there to relay RT timer ticks and device IRQs, and this only happens on behalf of the pipeline head. At least, this is howrthal_irq_host_pend()should be used in any case. If you did find a spot where this interface is being called from the lower stage, then this is the root bug to fix.Quoted 7 lines
Quoted 5 lines
I haven't studied the I-pipe trace /wrt this in details yet, but I could imagine that some shadow task is interrupted in primary mode by the timer IRQ and then leaves the handler in secondary mode due to whatever events between schedule-out and in at the end of
xnintr_clock_handler.You need a thread context to move to secondary, I just can't see how such scenario would be possible.
Here is the trace of events:
=> Shadow task starts migration to secondary
=> inxnpod_suspend_thread, nklock is briefly released before
xnpod_schedule
Quoted 9 lines
Quoted 4 lines
Which is the root bug. Blame on me; this recent change in -head breaks a basic rule a lot of code is based on: a self-suspending thread may not be preempted while scheduling out, i.e. suspension and rescheduling must be atomically performed.
xnshadow_relax()counts on this too.Oh, good that you insisted on this. Will you fix it soon? We are currently packaging a delivery of 2.5.git, and I would like to see this hole closed there already.
I just pushed a fix for this.
Quoted 17 lines
Quoted 11 lines
Quoted 7 lines
=> timer IRQ intercepts
=> as the current CPU is marked for reschedule, we enterxnpod_schedule
before propagating the host tick
=> once the migrating thread comes in again, it will run the
xnintr_clock_handlertail, i.e.xnarch_relay_tick, already over the root domainOk, makes sense now. However, this can't happen with 2.4 which has no such lock release in
xnpod_suspend_thread(). So the question is: was the "lost tick" bug observed also on 2.4, or not?I haven't tested on 2.4, but Martin anyway reported that his problem is still unfixed for 2.5 even with my patch.
Jan
Signature · 1 line
Philippe.
Re: [Xenomai-core] [PATCH] Fix host IRQ propagation
In reply to Philippe GerumQuoted 70 lines · Philippe Gerum
Quoted 69 lines · Jan Kiszka
Quoted 59 lines · Philippe Gerum
Quoted 58 lines · Jan Kiszka
Quoted 57 lines · Philippe Gerum
Quoted 43 lines · Jan Kiszka
Quoted 29 lines · Philippe Gerum
Quoted 28 lines · Jan Kiszka
Quoted 11 lines · Gilles Chanteperdrix
Quoted 10 lines · Jan Kiszka
Hi Gilles,
I'm currently facing a nasty effect with switchtest over latest git head (only tested this so far): running it inside my test VM (ie. with frequent excessive latencies) I get a stalled Linux timer IRQ quite quickly. System is otherwise still responsive, Xenomai timers are still being delivered, other Linux IRQs too. switchtest complained about
"Warning: Linux is compiled to use FPU in kernel-space."
Quoted 7 lines
Quoted 5 lines
when it was started. Kernels are 2.6.28.9/ipipe-x86-2.2-07 and 2.6.29.3/ipipe-x86-2.3-01 (LTTng patched in, but unused), both show the same effect.
Seen this before?
The warning about Linux being compiled to use FPU in kernel-space means that you enabled soft RAID or compiled for K7, Geode, or any other
RAID is on (ordinary server config).
Quoted 7 lines
configuration using 3DNow for such simple operations as memcpy. It is harmless, it simply means that switchtest can not use fpu in kernel-space.
The bug you have is probably the same as the one described here, which I am able to reproduce on my atom:
https://mail.gna.org/public/xenomai-help/2009-04/msg00200.htmlQuoted 8 lines
Quoted 6 lines
Quoted 2 lines
Unfortunately, I for one am working on ARM issues and am not available to debug x86 issues. I think Philippe is busy too...
OK, looks like I got the same flu here.
Philippe, did you find out any more details in the meantime? Then I'm afraid I have to pick this up.
No, I did not resume this task yet. Working from the powerpc side of the universe here.
Hoho, don't think this rain here over x86 would have never made it down to ARM or PPC land! ;)
Martin, could you check if this helps you, too?
Quoted 7 lines
Jan
(as usual, ready to be pulled from 'for-upstream')
--------->
Host IRQs may not only be triggered from non-root domains.
Are you sure of this? I can't find any spot where this assumption would be wrong.
host_pend()is basically there to relay RT timer ticks and device IRQs, and this only happens on behalf of the pipeline head. At least, this is howrthal_irq_host_pend()should be used in any case. If you did find a spot where this interface is being called from the lower stage, then this is the root bug to fix.Quoted 7 lines
Quoted 5 lines
I haven't studied the I-pipe trace /wrt this in details yet, but I could imagine that some shadow task is interrupted in primary mode by the timer IRQ and then leaves the handler in secondary mode due to whatever events between schedule-out and in at the end of
xnintr_clock_handler.You need a thread context to move to secondary, I just can't see how such scenario would be possible.
Here is the trace of events:
Quoted 8 lines
Quoted 3 lines
=> Shadow task starts migration to secondary
=> inxnpod_suspend_thread, nklock is briefly released before
xnpod_scheduleWhich is the root bug. Blame on me; this recent change in -head breaks a basic rule a lot of code is based on: a self-suspending thread may not be preempted while scheduling out, i.e. suspension and rescheduling must be atomically performed.
xnshadow_relax()counts on this too.
Actually, I think the idea was mine in the first place... Maybe we can specify a special flag to xnpod_suspend_thread to ask fo the atomic suspension (maybe reuse XNATOMIC ?).
Signature · 1 line
Gilles.
Re: [Xenomai-core] [PATCH] Fix host IRQ propagation
Quoted 75 lines · Gilles Chanteperdrix
Quoted 74 lines · Philippe Gerum
Quoted 72 lines · Jan Kiszka
Quoted 59 lines · Philippe Gerum
Quoted 58 lines · Jan Kiszka
Quoted 57 lines · Philippe Gerum
Quoted 43 lines · Jan Kiszka
Quoted 29 lines · Philippe Gerum
Quoted 28 lines · Jan Kiszka
Quoted 11 lines · Gilles Chanteperdrix
Quoted 10 lines · Jan Kiszka
Hi Gilles,
I'm currently facing a nasty effect with switchtest over latest git head (only tested this so far): running it inside my test VM (ie. with frequent excessive latencies) I get a stalled Linux timer IRQ quite quickly. System is otherwise still responsive, Xenomai timers are still being delivered, other Linux IRQs too. switchtest complained about
"Warning: Linux is compiled to use FPU in kernel-space."
Quoted 7 lines
Quoted 5 lines
when it was started. Kernels are 2.6.28.9/ipipe-x86-2.2-07 and 2.6.29.3/ipipe-x86-2.3-01 (LTTng patched in, but unused), both show the same effect.
Seen this before?
The warning about Linux being compiled to use FPU in kernel-space means that you enabled soft RAID or compiled for K7, Geode, or any other
RAID is on (ordinary server config).
Quoted 7 lines
configuration using 3DNow for such simple operations as memcpy. It is harmless, it simply means that switchtest can not use fpu in kernel-space.
The bug you have is probably the same as the one described here, which I am able to reproduce on my atom:
https://mail.gna.org/public/xenomai-help/2009-04/msg00200.htmlQuoted 8 lines
Quoted 6 lines
Quoted 2 lines
Unfortunately, I for one am working on ARM issues and am not available to debug x86 issues. I think Philippe is busy too...
OK, looks like I got the same flu here.
Philippe, did you find out any more details in the meantime? Then I'm afraid I have to pick this up.
No, I did not resume this task yet. Working from the powerpc side of the universe here.
Hoho, don't think this rain here over x86 would have never made it down to ARM or PPC land! ;)
Martin, could you check if this helps you, too?
Quoted 7 lines
Jan
(as usual, ready to be pulled from 'for-upstream')
--------->
Host IRQs may not only be triggered from non-root domains.
Are you sure of this? I can't find any spot where this assumption would be wrong.
host_pend()is basically there to relay RT timer ticks and device IRQs, and this only happens on behalf of the pipeline head. At least, this is howrthal_irq_host_pend()should be used in any case. If you did find a spot where this interface is being called from the lower stage, then this is the root bug to fix.Quoted 7 lines
Quoted 5 lines
I haven't studied the I-pipe trace /wrt this in details yet, but I could imagine that some shadow task is interrupted in primary mode by the timer IRQ and then leaves the handler in secondary mode due to whatever events between schedule-out and in at the end of
xnintr_clock_handler.You need a thread context to move to secondary, I just can't see how such scenario would be possible.
Here is the trace of events:
=> Shadow task starts migration to secondary
=> inxnpod_suspend_thread, nklock is briefly released before
xnpod_schedule
Quoted 9 lines
Quoted 4 lines
Which is the root bug. Blame on me; this recent change in -head breaks a basic rule a lot of code is based on: a self-suspending thread may not be preempted while scheduling out, i.e. suspension and rescheduling must be atomically performed.
xnshadow_relax()counts on this too.Actually, I think the idea was mine in the first place... Maybe we can specify a special flag to
xnpod_suspend_threadto ask fo the atomic suspension (maybe reuse XNATOMIC ?).
I don't think so. We really need the basic assumption to hold in any case, because this is expected by most of the callers, and this micro-optimization is not worth the risk of introducing a race if misused.
Signature · 1 line
Philippe.
Re: [Xenomai-core] [PATCH] Fix host IRQ propagation
Quoted 84 lines · Philippe Gerum
Quoted 74 lines · Gilles Chanteperdrix
Quoted 73 lines · Philippe Gerum
Quoted 72 lines · Jan Kiszka
Quoted 59 lines · Philippe Gerum
Quoted 58 lines · Jan Kiszka
Quoted 57 lines · Philippe Gerum
Quoted 43 lines · Jan Kiszka
Quoted 29 lines · Philippe Gerum
Quoted 28 lines · Jan Kiszka
Quoted 11 lines · Gilles Chanteperdrix
Quoted 10 lines · Jan Kiszka
Hi Gilles,
I'm currently facing a nasty effect with switchtest over latest git head (only tested this so far): running it inside my test VM (ie. with frequent excessive latencies) I get a stalled Linux timer IRQ quite quickly. System is otherwise still responsive, Xenomai timers are still being delivered, other Linux IRQs too. switchtest complained about
"Warning: Linux is compiled to use FPU in kernel-space."
Quoted 7 lines
Quoted 5 lines
when it was started. Kernels are 2.6.28.9/ipipe-x86-2.2-07 and 2.6.29.3/ipipe-x86-2.3-01 (LTTng patched in, but unused), both show the same effect.
Seen this before?
The warning about Linux being compiled to use FPU in kernel-space means that you enabled soft RAID or compiled for K7, Geode, or any other
RAID is on (ordinary server config).
Quoted 7 lines
configuration using 3DNow for such simple operations as memcpy. It is harmless, it simply means that switchtest can not use fpu in kernel-space.
The bug you have is probably the same as the one described here, which I am able to reproduce on my atom:
https://mail.gna.org/public/xenomai-help/2009-04/msg00200.htmlQuoted 8 lines
Quoted 6 lines
Quoted 2 lines
Unfortunately, I for one am working on ARM issues and am not available to debug x86 issues. I think Philippe is busy too...
OK, looks like I got the same flu here.
Philippe, did you find out any more details in the meantime? Then I'm afraid I have to pick this up.
No, I did not resume this task yet. Working from the powerpc side of the universe here.
Hoho, don't think this rain here over x86 would have never made it down to ARM or PPC land! ;)
Martin, could you check if this helps you, too?
Quoted 7 lines
Jan
(as usual, ready to be pulled from 'for-upstream')
--------->
Host IRQs may not only be triggered from non-root domains.
Are you sure of this? I can't find any spot where this assumption would be wrong.
host_pend()is basically there to relay RT timer ticks and device IRQs, and this only happens on behalf of the pipeline head. At least, this is howrthal_irq_host_pend()should be used in any case. If you did find a spot where this interface is being called from the lower stage, then this is the root bug to fix.Quoted 7 lines
Quoted 5 lines
I haven't studied the I-pipe trace /wrt this in details yet, but I could imagine that some shadow task is interrupted in primary mode by the timer IRQ and then leaves the handler in secondary mode due to whatever events between schedule-out and in at the end of
xnintr_clock_handler.You need a thread context to move to secondary, I just can't see how such scenario would be possible.
Here is the trace of events:
=> Shadow task starts migration to secondary
=> inxnpod_suspend_thread, nklock is briefly released before
xnpod_scheduleQuoted 8 lines
Quoted 4 lines
Which is the root bug. Blame on me; this recent change in -head breaks a basic rule a lot of code is based on: a self-suspending thread may not be preempted while scheduling out, i.e. suspension and rescheduling must be atomically performed.
xnshadow_relax()counts on this too.Actually, I think the idea was mine in the first place... Maybe we can specify a special flag to
xnpod_suspend_threadto ask fo the atomic suspension (maybe reuse XNATOMIC ?).
Quoted 4 lines
I don't think so. We really need the basic assumption to hold in any case, because this is expected by most of the callers, and this micro-optimization is not worth the risk of introducing a race if misused.
Well, I tend to disagree. The assumption that the thread is suspended from the point of view of the scheduler still holds even when the nklock is released, and it is what callers like rt_cond_wait are expecting. The assumptions of xnshadow_relax do not seem to me like a common assumption.
To go further, maybe forwarding the tick in xnintr_clock_handler epilogue is wrong in the first place, precisely because the current thread where xnintr_clock_handler happens may be scheduled out for a long time by the clock handler itself.
Signature · 1 line
Gilles.
Re: [Xenomai-core] [PATCH] Fix host IRQ propagation
Quoted 90 lines · Gilles Chanteperdrix
Quoted 84 lines · Philippe Gerum
Quoted 74 lines · Gilles Chanteperdrix
Quoted 73 lines · Philippe Gerum
Quoted 72 lines · Jan Kiszka
Quoted 59 lines · Philippe Gerum
Quoted 58 lines · Jan Kiszka
Quoted 57 lines · Philippe Gerum
Quoted 43 lines · Jan Kiszka
Quoted 29 lines · Philippe Gerum
Quoted 28 lines · Jan Kiszka
Quoted 11 lines · Gilles Chanteperdrix
Quoted 10 lines · Jan Kiszka
Hi Gilles,
I'm currently facing a nasty effect with switchtest over latest git head
(only tested this so far): running it inside my test VM (ie. with
frequent excessive latencies) I get a stalled Linux timer IRQ quite
quickly. System is otherwise still responsive, Xenomai timers are still
being delivered, other Linux IRQs too. switchtest complained about"Warning: Linux is compiled to use FPU in kernel-space."
Quoted 7 lines
Quoted 5 lines
when it was started. Kernels are 2.6.28.9/ipipe-x86-2.2-07 and
2.6.29.3/ipipe-x86-2.3-01 (LTTng patched in, but unused), both show the
same effect.Seen this before?
The warning about Linux being compiled to use FPU in kernel-space means that you enabled soft RAID or compiled for K7, Geode, or any other
RAID is on (ordinary server config).
Quoted 7 lines
configuration using 3DNow for such simple operations as memcpy. It is harmless, it simply means that switchtest can not use fpu in kernel-space.
The bug you have is probably the same as the one described here, which I am able to reproduce on my atom:
https://mail.gna.org/public/xenomai-help/2009-04/msg00200.htmlQuoted 8 lines
Quoted 6 lines
Quoted 2 lines
Unfortunately, I for one am working on ARM issues and am not available to debug x86 issues. I think Philippe is busy too...
OK, looks like I got the same flu here.
Philippe, did you find out any more details in the meantime? Then I'm afraid I have to pick this up.
No, I did not resume this task yet. Working from the powerpc side of the universe here.
Hoho, don't think this rain here over x86 would have never made it down to ARM or PPC land! ;)
Martin, could you check if this helps you, too?
Quoted 7 lines
Jan
(as usual, ready to be pulled from 'for-upstream')
--------->
Host IRQs may not only be triggered from non-root domains.
Are you sure of this? I can't find any spot where this assumption would be wrong.
host_pend()is basically there to relay RT timer ticks and device IRQs, and this only happens on behalf of the pipeline head. At least, this is howrthal_irq_host_pend()should be used in any case. If you did find a spot where this interface is being called from the lower stage, then this is the root bug to fix.Quoted 7 lines
Quoted 5 lines
I haven't studied the I-pipe trace /wrt this in details yet, but I could imagine that some shadow task is interrupted in primary mode by the timer IRQ and then leaves the handler in secondary mode due to whatever events between schedule-out and in at the end of
xnintr_clock_handler.You need a thread context to move to secondary, I just can't see how such scenario would be possible.
Here is the trace of events:
=> Shadow task starts migration to secondary
=> inxnpod_suspend_thread, nklock is briefly released before
xnpod_scheduleQuoted 8 lines
Quoted 4 lines
Which is the root bug. Blame on me; this recent change in -head breaks a basic rule a lot of code is based on: a self-suspending thread may not be preempted while scheduling out, i.e. suspension and rescheduling must be atomically performed.
xnshadow_relax()counts on this too.Actually, I think the idea was mine in the first place... Maybe we can specify a special flag to
xnpod_suspend_threadto ask fo the atomic suspension (maybe reuse XNATOMIC ?).Quoted 4 lines
I don't think so. We really need the basic assumption to hold in any case, because this is expected by most of the callers, and this micro-optimization is not worth the risk of introducing a race if misused.
Quoted 5 lines
Well, I tend to disagree. The assumption that the thread is suspended from the point of view of the scheduler still holds even when the nklock is released, and it is what callers like
rt_cond_waitare expecting. The assumptions ofxnshadow_relaxdo not seem to me like a common assumption.
The assumption is that the thread has been suspended _and_ scheduled out atomically, not only put in a suspended thread, which is quite different when considered from an interrupt context. I'm worried by the fact that re-enabling interrupts in the middle of this critical transition breaks the unspoken rule that sched->curr may not be seen as bearing any block bit in its status word from anywhere in the code executed from the local CPU but xnpod_suspend_thread().
Another issue may arise in the SMP case, where xnpod_suspend_thread() would block a thread running on a remote CPU; in theory, re-enabling interrupts before the IPIs are sent from xnpod_schedule() - to kick the remote resched procedure - may introduce an undefined delay due to local interrupts being processed, albeit some runnable thread may be waiting for the CPU to be released on the remote CPU. We would then end up with a local scheduling artefact leaking remotely. I don't quite like this, in fact.
Admittedly, we could plug each and every subtle hole we find like this one, introducing XNATOMIC to fix the migration case as well, but my point is that it may not be worth opening pandora's box just for that micro-optimization.
Quoted 5 lines
To go further, maybe forwarding the tick in
xnintr_clock_handlerepilogue is wrong in the first place, precisely because the current thread wherexnintr_clock_handlerhappens may be scheduled out for a long time by the clock handler itself.
It was mostly ok in the early days before RPI came in and with legacy kernels, since the root thread would always be the last to be scheduled; it is now wrong because:
- RPI may boost the root thread, so we could pend the timer IRQ long after the root domain has regained control.
- Linux tickless timing requires that no tick is lost due to the current IRQ context not returning from xnpod_schedule(), such as when calling xnpod_delete_thread() on behalf of a kernel-based RT thread, previously preempted by a timer IRQ. The same goes if xnpod_suspend_thread() is called upon this thread as well.
So in short, yes, you are definitely right. We would be better off moving this code close to the deferred tick handling instead, since we know for sure that the root thread is about to be scheduled in at that point. I just committed a fix for -head; this is a good candidate for a backport to 2.4.x.
Signature · 1 line
Philippe.
Re: [Xenomai-core] [PATCH] Fix host IRQ propagation
In reply to Jan KiszkaQuoted 68 lines · Jan Kiszka
Quoted 29 lines · Philippe Gerum
Quoted 28 lines · Jan Kiszka
Quoted 11 lines · Gilles Chanteperdrix
Quoted 10 lines · Jan Kiszka
Hi Gilles,
I'm currently facing a nasty effect with switchtest over latest git head (only tested this so far): running it inside my test VM (ie. with frequent excessive latencies) I get a stalled Linux timer IRQ quite quickly. System is otherwise still responsive, Xenomai timers are still being delivered, other Linux IRQs too. switchtest complained about
"Warning: Linux is compiled to use FPU in kernel-space."
Quoted 7 lines
Quoted 5 lines
when it was started. Kernels are 2.6.28.9/ipipe-x86-2.2-07 and 2.6.29.3/ipipe-x86-2.3-01 (LTTng patched in, but unused), both show the same effect.
Seen this before?
The warning about Linux being compiled to use FPU in kernel-space means that you enabled soft RAID or compiled for K7, Geode, or any other
RAID is on (ordinary server config).
Quoted 7 lines
configuration using 3DNow for such simple operations as memcpy. It is harmless, it simply means that switchtest can not use fpu in kernel-space.
The bug you have is probably the same as the one described here, which I am able to reproduce on my atom:
https://mail.gna.org/public/xenomai-help/2009-04/msg00200.htmlQuoted 8 lines
Quoted 6 lines
Quoted 2 lines
Unfortunately, I for one am working on ARM issues and am not available to debug x86 issues. I think Philippe is busy too...
OK, looks like I got the same flu here.
Philippe, did you find out any more details in the meantime? Then I'm afraid I have to pick this up.
No, I did not resume this task yet. Working from the powerpc side of the universe here.
Hoho, don't think this rain here over x86 would have never made it down to ARM or PPC land! ;)
Martin, could you check if this helps you, too?
Jan
(as usual, ready to be pulled from 'for-upstream')
--------->
Host IRQs may not only be triggered from non-root domains. But rthal_propagate_irq's implemenation in I-pipe assumes so, which broke host tick propagation under certain load scenarios. Besides that,
rthal_schedule_irq_rootis the more efficient service for this purpose anyway.Signed-off-by Jan Kiszka <[email protected]>---
include/asm-generic/hal.h | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-)include/asm-generic/hal.h +0
-0
include/asm-generic/hal.h int rthal_irq_host_release(unsigned irq, · line 443
Quoted 8 lines
include/asm-generic/hal.h +1
-1@@ -437,7 +437,7 @@ int rthal_irq_host_release(unsigned irq,static inline void rthal_irq_host_pend (unsigned irq) { -rthal_propagate_irq (irq); +rthal_schedule_irq_root (irq); } int rthal_apc_alloc(const char *name,
Well, I would assume this could fix head, with the newly delayed root tick irq propagation. But the user reports this bug with 2.4 too...
Signature · 1 line
Gilles.
Re: [Xenomai-core] [PATCH] Fix host IRQ propagation
Quoted 74 lines · Gilles Chanteperdrix
Quoted 73 lines · Jan Kiszka
Quoted 29 lines · Philippe Gerum
Quoted 28 lines · Jan Kiszka
Quoted 11 lines · Gilles Chanteperdrix
Quoted 10 lines · Jan Kiszka
Hi Gilles,
I'm currently facing a nasty effect with switchtest over latest git head (only tested this so far): running it inside my test VM (ie. with frequent excessive latencies) I get a stalled Linux timer IRQ quite quickly. System is otherwise still responsive, Xenomai timers are still being delivered, other Linux IRQs too. switchtest complained about
"Warning: Linux is compiled to use FPU in kernel-space."
Quoted 7 lines
Quoted 5 lines
when it was started. Kernels are 2.6.28.9/ipipe-x86-2.2-07 and 2.6.29.3/ipipe-x86-2.3-01 (LTTng patched in, but unused), both show the same effect.
Seen this before?
The warning about Linux being compiled to use FPU in kernel-space means that you enabled soft RAID or compiled for K7, Geode, or any other
RAID is on (ordinary server config).
Quoted 7 lines
configuration using 3DNow for such simple operations as memcpy. It is harmless, it simply means that switchtest can not use fpu in kernel-space.
The bug you have is probably the same as the one described here, which I am able to reproduce on my atom:
https://mail.gna.org/public/xenomai-help/2009-04/msg00200.htmlQuoted 8 lines
Quoted 6 lines
Quoted 2 lines
Unfortunately, I for one am working on ARM issues and am not available to debug x86 issues. I think Philippe is busy too...
OK, looks like I got the same flu here.
Philippe, did you find out any more details in the meantime? Then I'm afraid I have to pick this up.
No, I did not resume this task yet. Working from the powerpc side of the universe here.
Hoho, don't think this rain here over x86 would have never made it down to ARM or PPC land! ;)
Martin, could you check if this helps you, too?
Jan
(as usual, ready to be pulled from 'for-upstream')
--------->
Host IRQs may not only be triggered from non-root domains. But rthal_propagate_irq's implemenation in I-pipe assumes so, which broke host tick propagation under certain load scenarios. Besides that,
rthal_schedule_irq_rootis the more efficient service for this purpose anyway.Signed-off-by Jan Kiszka <[email protected]>---
include/asm-generic/hal.h | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-)include/asm-generic/hal.h +1
-1@@ -437,7 +437,7 @@ int rthal_irq_host_release(unsigned irq,static inline void rthal_irq_host_pend (unsigned irq) { -rthal_propagate_irq (irq); +rthal_schedule_irq_root (irq); }
Quoted 5 lines
include/asm-generic/hal.h int rthal_irq_host_release(unsigned irq, · line 443
Quoted 2 lines
include/asm-generic/hal.h +0
-0int rthal_apc_alloc(const char *name,Well, I would assume this could fix head, with the newly delayed root tick irq propagation. But the user reports this bug with 2.4 too...
According to my instrumentation, a delayed tick was not involved. The problem was 'propagate' vs. non-root trigger domain. So 2.4 should suffer from the same issue.
Jan
Signature · 2 lines
Siemens AG, Corporate Technology, CT SE 2 Corporate Competence Center Embedded Linux
Re: [Xenomai-core] [PATCH] Fix host IRQ propagation (was: Troubles with switchtest)
In reply to Jan KiszkaQuoted 2 lines · Jan Kiszka
...
Martin, could you check if this helps you, too?
It doesn't appear to help. To check, first I turned on the HPET and PM timer options, and recompiled the kernel without your patch, to verify that this reproduced the problem. I then manually applied your patch to include/asm-generic/xenomai/hal.h in the kernel source tree, recompiled the kernel, installed etc, and rebooted. However even with the patch in place, whenever I ran:
dd if=/dev/zero of=/dev/null count=20000000
then initially top showed that dd was getting very little CPU time (<10%), then after 30 seconds or so, the system became completely unresponsive until the dd ended. This is how it acts without the patch as well. So it doesn't appear that the patch has made any difference to this problem.
Note that I applied the patch to Xenomai 2.5-rc1 in linux kernel 2.6.29.1. If the patch somehow relies on head, tell me, and I'll endeavor to set up a new kernel using that.
Martin
Re: [Xenomai-core] [PATCH] Fix host IRQ propagation
Quoted 11 lines · Martin Shepherd
Quoted 2 lines · Jan Kiszka
...
Martin, could you check if this helps you, too?It doesn't appear to help. To check, first I turned on the HPET and PM timer options, and recompiled the kernel without your patch, to verify that this reproduced the problem. I then manually applied your patch to
include/asm-generic/xenomai/hal.hin the kernel source tree, recompiled the kernel, installed etc, and rebooted. However even with the patch in place, whenever I ran:
Quoted 7 lines
dd if=/dev/zero of=/dev/null count=20000000
then initially top showed that dd was getting very little CPU time (<10%), then after 30 seconds or so, the system became completely unresponsive until the dd ended. This is how it acts without the patch as well. So it doesn't appear that the patch has made any difference to this problem.
Hmm, so there is no Xenomai task running during this test, just Linux load? Tried to reproduce with my kernel here (both HPET and PM enabled, as usual), but no success. Strange.
Quoted 3 lines
Note that I applied the patch to Xenomai 2.5-rc1 in linux kernel 2.6.29.1. If the patch somehow relies on head, tell me, and I'll endeavor to set up a new kernel using that.
The patch must cleanly apply as is on -rc1, any later changes did not affect its context.
Jan
Signature · 2 lines
Siemens AG, Corporate Technology, CT SE 2 Corporate Competence Center Embedded Linux
Re: [Xenomai-core] [PATCH] Fix host IRQ propagation
Quoted 19 lines · Jan Kiszka
Quoted 10 lines · Martin Shepherd
Quoted 2 lines · Jan Kiszka
...
Martin, could you check if this helps you, too?It doesn't appear to help. To check, first I turned on the HPET and PM timer options, and recompiled the kernel without your patch, to verify that this reproduced the problem. I then manually applied your patch to
include/asm-generic/xenomai/hal.hin the kernel source tree, recompiled the kernel, installed etc, and rebooted. However even with the patch in place, whenever I ran:Quoted 7 lines
dd if=/dev/zero of=/dev/null count=20000000
then initially top showed that dd was getting very little CPU time (<10%), then after 30 seconds or so, the system became completely unresponsive until the dd ended. This is how it acts without the patch as well. So it doesn't appear that the patch has made any difference to this problem.
Quoted 3 lines
Hmm, so there is no Xenomai task running during this test, just Linux load? Tried to reproduce with my kernel here (both HPET and PM enabled, as usual), but no success. Strange.
Just having HPET and PM_TIMER enabled is not enough: your CPU needs to actually have an HPET timer or PM timer.
Signature · 1 line
Gilles.
Re: [Xenomai-core] [PATCH] Fix host IRQ propagation
Quoted 19 lines · Gilles Chanteperdrix
Quoted 18 lines · Jan Kiszka
Quoted 17 lines · Martin Shepherd
Quoted 2 lines · Jan Kiszka
...
Martin, could you check if this helps you, too?It doesn't appear to help. To check, first I turned on the HPET and PM timer options, and recompiled the kernel without your patch, to verify that this reproduced the problem. I then manually applied your patch to
include/asm-generic/xenomai/hal.hin the kernel source tree, recompiled the kernel, installed etc, and rebooted. However even with the patch in place, whenever I ran:dd if=/dev/zero of=/dev/null count=20000000
then initially top showed that dd was getting very little CPU time (<10%), then after 30 seconds or so, the system became completely unresponsive until the dd ended. This is how it acts without the patch as well. So it doesn't appear that the patch has made any difference to this problem.
Quoted 6 lines
Quoted 3 lines
Hmm, so there is no Xenomai task running during this test, just Linux load? Tried to reproduce with my kernel here (both HPET and PM enabled, as usual), but no success. Strange.
Just having HPET and
PM_TIMERenabled is not enough: your CPU needs to actually have an HPET timer or PM timer.
That's the case for practically any modern system. So I bet the "trick" is to make Linux _use_ one of them - in favor of the normally better suited LAPIC. Need to check again what the reasons for this switch could be. One that comes to my mind is problems of LAPIC vs. power management.
Jan
Signature · 2 lines
Siemens AG, Corporate Technology, CT SE 2 Corporate Competence Center Embedded Linux
Quoted 16 lines · Jan Kiszka
Quoted 15 lines · Gilles Chanteperdrix
Quoted 14 lines · Jan Kiszka
Hi Gilles,
I'm currently facing a nasty effect with switchtest over latest git head (only tested this so far): running it inside my test VM (ie. with frequent excessive latencies) I get a stalled Linux timer IRQ quite quickly. System is otherwise still responsive, Xenomai timers are still being delivered, other Linux IRQs too. switchtest complained about
"Warning: Linux is compiled to use FPU in kernel-space."
when it was started. Kernels are 2.6.28.9/ipipe-x86-2.2-07 and 2.6.29.3/ipipe-x86-2.3-01 (LTTng patched in, but unused), both show the same effect.
Quoted 5 lines
Quoted 3 lines
Quoted 1 line
Seen this before?
The warning about Linux being compiled to use FPU in kernel-space means that you enabled soft RAID or compiled for K7, Geode, or any other
RAID is on (ordinary server config).
By the way, I wonder how MMX accelerated software raid works on K7, since the way I understand the code, calls to kernel_fpu_begin() can not be nested.
If you think they can be nested, then we can make switchtest test fpu in Linux kernel-space when these options are enabled.
Signature · 1 line
Gilles.
Quoted 20 lines · Gilles Chanteperdrix
Quoted 19 lines · Jan Kiszka
Quoted 18 lines · Gilles Chanteperdrix
Quoted 8 lines · Jan Kiszka
Hi Gilles,
I'm currently facing a nasty effect with switchtest over latest git head (only tested this so far): running it inside my test VM (ie. with frequent excessive latencies) I get a stalled Linux timer IRQ quite quickly. System is otherwise still responsive, Xenomai timers are still being delivered, other Linux IRQs too. switchtest complained about
Quoted 7 lines
"Warning: Linux is compiled to use FPU in kernel-space."
when it was started. Kernels are 2.6.28.9/ipipe-x86-2.2-07 and 2.6.29.3/ipipe-x86-2.3-01 (LTTng patched in, but unused), both show the same effect.
Seen this before?
The warning about Linux being compiled to use FPU in kernel-space means that you enabled soft RAID or compiled for K7, Geode, or any other
Quoted 8 lines
Quoted 1 line
RAID is on (ordinary server config).
By the way, I wonder how MMX accelerated software raid works on K7, since the way I understand the code, calls to
kernel_fpu_begin()can not be nested.If you think they can be nested, then we can make switchtest test fpu in Linux kernel-space when these options are enabled.
Sorry, I haven't looked that close into the in-kernel FPU handling so far. What users are nested in the standard kernel? Or is it RAID in itself?
Jan
Signature · 2 lines
Siemens AG, Corporate Technology, CT SE 2 Corporate Competence Center Embedded Linux
Quoted 26 lines · Jan Kiszka
Quoted 25 lines · Gilles Chanteperdrix
Quoted 12 lines · Jan Kiszka
Quoted 11 lines · Gilles Chanteperdrix
Quoted 10 lines · Jan Kiszka
Hi Gilles,
I'm currently facing a nasty effect with switchtest over latest git head (only tested this so far): running it inside my test VM (ie. with frequent excessive latencies) I get a stalled Linux timer IRQ quite quickly. System is otherwise still responsive, Xenomai timers are still being delivered, other Linux IRQs too. switchtest complained about
"Warning: Linux is compiled to use FPU in kernel-space."
Quoted 8 lines
Quoted 7 lines
Quoted 5 lines
when it was started. Kernels are 2.6.28.9/ipipe-x86-2.2-07 and 2.6.29.3/ipipe-x86-2.3-01 (LTTng patched in, but unused), both show the same effect.
Seen this before?
The warning about Linux being compiled to use FPU in kernel-space means that you enabled soft RAID or compiled for K7, Geode, or any other
RAID is on (ordinary server config).
By the way, I wonder how MMX accelerated software raid works on K7, since the way I understand the code, calls to
kernel_fpu_begin()can not be nested.
Quoted 5 lines
Quoted 2 lines
If you think they can be nested, then we can make switchtest test fpu in Linux kernel-space when these options are enabled.
Sorry, I haven't looked that close into the in-kernel FPU handling so far. What users are nested in the standard kernel? Or is it RAID in itself?
RAID uses FPU. And on K7, basic things like clearing or copying user pages, or even large memcpy do use FPU too. So, I would expect that when enabling both, they could happen to be nested.
Signature · 1 line
Gilles.
Quoted 27 lines · Gilles Chanteperdrix
Quoted 26 lines · Jan Kiszka
Quoted 25 lines · Gilles Chanteperdrix
Quoted 12 lines · Jan Kiszka
Quoted 11 lines · Gilles Chanteperdrix
Quoted 10 lines · Jan Kiszka
Hi Gilles,
I'm currently facing a nasty effect with switchtest over latest git head (only tested this so far): running it inside my test VM (ie. with frequent excessive latencies) I get a stalled Linux timer IRQ quite quickly. System is otherwise still responsive, Xenomai timers are still being delivered, other Linux IRQs too. switchtest complained about
"Warning: Linux is compiled to use FPU in kernel-space."
Quoted 8 lines
Quoted 7 lines
Quoted 5 lines
when it was started. Kernels are 2.6.28.9/ipipe-x86-2.2-07 and 2.6.29.3/ipipe-x86-2.3-01 (LTTng patched in, but unused), both show the same effect.
Seen this before?
The warning about Linux being compiled to use FPU in kernel-space means that you enabled soft RAID or compiled for K7, Geode, or any other
RAID is on (ordinary server config).
By the way, I wonder how MMX accelerated software raid works on K7, since the way I understand the code, calls to
kernel_fpu_begin()can not be nested.
Quoted 9 lines
Quoted 4 lines
Quoted 2 lines
If you think they can be nested, then we can make switchtest test fpu in Linux kernel-space when these options are enabled.
Sorry, I haven't looked that close into the in-kernel FPU handling so far. What users are nested in the standard kernel? Or is it RAID in itself?
RAID uses FPU. And on K7, basic things like clearing or copying user pages, or even large memcpy do use FPU too. So, I would expect that when enabling both, they could happen to be nested.
I think the trick against /this/ is preempt_disable/enable in kernel_fpu/begin/end. But that won't work for Xenomai, of course.
Jan
Signature · 2 lines
Siemens AG, Corporate Technology, CT SE 2 Corporate Competence Center Embedded Linux
Quoted 2 lines · Jan Kiszka
I think the trick against /this/ is preempt_disable/enable in kernel_fpu/begin/end. But that won't work for Xenomai, of course.
Well, that does not prevent an IRQ or a page fault from computing a RAID cheksum...
Signature · 1 line
Gilles.
Quoted 6 lines · Gilles Chanteperdrix
Quoted 2 lines · Jan Kiszka
I think the trick against /this/ is preempt_disable/enable in kernel_fpu/begin/end. But that won't work for Xenomai, of course.
Well, that does not prevent an IRQ or a page fault from computing a RAID cheksum...
If this is allowed, then... My impression was that FPU access is not legal from IRQ context.
Jan
Signature · 2 lines
Siemens AG, Corporate Technology, CT SE 2 Corporate Competence Center Embedded Linux
Quoted 9 lines · Jan Kiszka
Quoted 5 lines · Gilles Chanteperdrix
Quoted 2 lines · Jan Kiszka
I think the trick against /this/ is preempt_disable/enable in kernel_fpu/begin/end. But that won't work for Xenomai, of course.
Well, that does not prevent an IRQ or a page fault from computing a RAID cheksum...
If this is allowed, then... My impression was that FPU access is not legal from IRQ context.
If you have a mapped file on a raid disk and a page fault triggers a write, I think the RAID routines will be used. But you are right, maybe there is a test to use the plain integer RAID routines when called from interrupt context.
Signature · 1 line
Gilles.