Skip to contentSkip to conversation

Troubles with switchtest

Xenomai-core In review Jan Kiszka 4 participants 29 messages May 13, 2009 – May 17, 2009 Generic include/asm header files

lore
Share & tools
lore

Jan Kiszka

author
#0 lore raw

Hi Gilles,

I'm currently facing a nasty effect with switchtest over latest git head (only tested this so far): running it inside my test VM (ie. with frequent excessive latencies) I get a stalled Linux timer IRQ quite quickly. System is otherwise still responsive, Xenomai timers are still being delivered, other Linux IRQs too. switchtest complained about

"Warning: Linux is compiled to use FPU in kernel-space."

when it was started. Kernels are 2.6.28.9/ipipe-x86-2.2-07 and 2.6.29.3/ipipe-x86-2.3-01 (LTTng patched in, but unused), both show the same effect.

Seen this before?

Jan

Signature · 2 lines
Siemens AG, Corporate Technology, CT SE 2
Corporate Competence Center Embedded Linux
#1 lore raw
Quoted 8 lines · Jan Kiszka

Hi Gilles,

I'm currently facing a nasty effect with switchtest over latest git head (only tested this so far): running it inside my test VM (ie. with frequent excessive latencies) I get a stalled Linux timer IRQ quite quickly. System is otherwise still responsive, Xenomai timers are still being delivered, other Linux IRQs too. switchtest complained about

Quoted 7 lines

"Warning: Linux is compiled to use FPU in kernel-space."

when it was started. Kernels are 2.6.28.9/ipipe-x86-2.2-07 and 2.6.29.3/ipipe-x86-2.3-01 (LTTng patched in, but unused), both show the same effect.

Seen this before?

The warning about Linux being compiled to use FPU in kernel-space means that you enabled soft RAID or compiled for K7, Geode, or any other configuration using 3DNow for such simple operations as memcpy. It is harmless, it simply means that switchtest can not use fpu in kernel-space.

The bug you have is probably the same as the one described here, which I am able to reproduce on my atom:
https://mail.gna.org/public/xenomai-help/2009-04/msg00200.html

Unfortunately, I for one am working on ARM issues and am not available to debug x86 issues. I think Philippe is busy too...

Signature · 1 line
                                                 Gilles.
author
#2 lore raw
Quoted 11 lines · Gilles Chanteperdrix
Quoted 10 lines · Jan Kiszka

Hi Gilles,

I'm currently facing a nasty effect with switchtest over latest git head (only tested this so far): running it inside my test VM (ie. with frequent excessive latencies) I get a stalled Linux timer IRQ quite quickly. System is otherwise still responsive, Xenomai timers are still being delivered, other Linux IRQs too. switchtest complained about

"Warning: Linux is compiled to use FPU in kernel-space."

Quoted 8 lines
Quoted 5 lines

when it was started. Kernels are 2.6.28.9/ipipe-x86-2.2-07 and 2.6.29.3/ipipe-x86-2.3-01 (LTTng patched in, but unused), both show the same effect.

Seen this before?

The warning about Linux being compiled to use FPU in kernel-space means that you enabled soft RAID or compiled for K7, Geode, or any other

RAID is on (ordinary server config).

Quoted 9 lines

configuration using 3DNow for such simple operations as memcpy. It is harmless, it simply means that switchtest can not use fpu in kernel-space.

The bug you have is probably the same as the one described here, which I am able to reproduce on my atom:
https://mail.gna.org/public/xenomai-help/2009-04/msg00200.html

Unfortunately, I for one am working on ARM issues and am not available to debug x86 issues. I think Philippe is busy too...

OK, looks like I got the same flu here.

Philippe, did you find out any more details in the meantime? Then I'm afraid I have to pick this up.

Jan

Signature · 2 lines
Siemens AG, Corporate Technology, CT SE 2
Corporate Competence Center Embedded Linux
#3 lore raw
Quoted 30 lines · Jan Kiszka
Quoted 11 lines · Gilles Chanteperdrix
Quoted 10 lines · Jan Kiszka

Hi Gilles,

I'm currently facing a nasty effect with switchtest over latest git head (only tested this so far): running it inside my test VM (ie. with frequent excessive latencies) I get a stalled Linux timer IRQ quite quickly. System is otherwise still responsive, Xenomai timers are still being delivered, other Linux IRQs too. switchtest complained about

"Warning: Linux is compiled to use FPU in kernel-space."

Quoted 8 lines
Quoted 5 lines

when it was started. Kernels are 2.6.28.9/ipipe-x86-2.2-07 and 2.6.29.3/ipipe-x86-2.3-01 (LTTng patched in, but unused), both show the same effect.

Seen this before?

The warning about Linux being compiled to use FPU in kernel-space means that you enabled soft RAID or compiled for K7, Geode, or any other

RAID is on (ordinary server config).

Quoted 7 lines

configuration using 3DNow for such simple operations as memcpy. It is harmless, it simply means that switchtest can not use fpu in kernel-space.

The bug you have is probably the same as the one described here, which I am able to reproduce on my atom:
https://mail.gna.org/public/xenomai-help/2009-04/msg00200.html

Quoted 7 lines
Quoted 2 lines

Unfortunately, I for one am working on ARM issues and am not available to debug x86 issues. I think Philippe is busy too...

OK, looks like I got the same flu here.

Philippe, did you find out any more details in the meantime? Then I'm afraid I have to pick this up.

No, I did not resume this task yet. Working from the powerpc side of the universe here.

Quoted 3 lines

Jan

Signature · 1 line
Philippe.
author
#4 lore raw

PATCH 1/1 Fix host IRQ propagation (was: Troubles with switchtest)

Quoted 31 lines · Philippe Gerum
Quoted 30 lines · Jan Kiszka
Quoted 11 lines · Gilles Chanteperdrix
Quoted 10 lines · Jan Kiszka

Hi Gilles,

I'm currently facing a nasty effect with switchtest over latest git head (only tested this so far): running it inside my test VM (ie. with frequent excessive latencies) I get a stalled Linux timer IRQ quite quickly. System is otherwise still responsive, Xenomai timers are still being delivered, other Linux IRQs too. switchtest complained about

"Warning: Linux is compiled to use FPU in kernel-space."

Quoted 7 lines
Quoted 5 lines

when it was started. Kernels are 2.6.28.9/ipipe-x86-2.2-07 and 2.6.29.3/ipipe-x86-2.3-01 (LTTng patched in, but unused), both show the same effect.

Seen this before?

The warning about Linux being compiled to use FPU in kernel-space means that you enabled soft RAID or compiled for K7, Geode, or any other

RAID is on (ordinary server config).

Quoted 9 lines

configuration using 3DNow for such simple operations as memcpy. It is harmless, it simply means that switchtest can not use fpu in kernel-space.

The bug you have is probably the same as the one described here, which I am able to reproduce on my atom:
https://mail.gna.org/public/xenomai-help/2009-04/msg00200.html

Unfortunately, I for one am working on ARM issues and am not available to debug x86 issues. I think Philippe is busy too...

Quoted 7 lines
Quoted 4 lines

OK, looks like I got the same flu here.

Philippe, did you find out any more details in the meantime? Then I'm afraid I have to pick this up.

No, I did not resume this task yet. Working from the powerpc side of the universe here.

Hoho, don't think this rain here over x86 would have never made it down to ARM or PPC land! ;)

Martin, could you check if this helps you, too?

Jan

(as usual, ready to be pulled from 'for-upstream')

--------->

Host IRQs may not only be triggered from non-root domains. But rthal_propagate_irq's implemenation in I-pipe assumes so, which broke host tick propagation under certain load scenarios. Besides that, rthal_schedule_irq_root is the more efficient service for this purpose anyway.

---

 include/asm-generic/hal.h |    2 +-
 1 files changed, 1 insertions(+), 1 deletions(-)
include/asm-generic/hal.h +1 -1
index b37e476..8137856 100644
@@ -437,7 +437,7 @@ int rthal_irq_host_release(unsigned irq,
 
 static inline void rthal_irq_host_pend(unsigned irq)
 {
-	rthal_propagate_irq(irq);
+	rthal_schedule_irq_root(irq);
 }
 
 int rthal_apc_alloc(const char *name,
Signature · 2 lines
Siemens AG, Corporate Technology, CT SE 2
Corporate Competence Center Embedded Linux
#5 lore raw

Re: [Xenomai-core] [PATCH] Fix host IRQ propagation (was: Troubles with switchtest)

Quoted 45 lines · Jan Kiszka
Quoted 31 lines · Philippe Gerum
Quoted 30 lines · Jan Kiszka
Quoted 11 lines · Gilles Chanteperdrix
Quoted 10 lines · Jan Kiszka

Hi Gilles,

I'm currently facing a nasty effect with switchtest over latest git head (only tested this so far): running it inside my test VM (ie. with frequent excessive latencies) I get a stalled Linux timer IRQ quite quickly. System is otherwise still responsive, Xenomai timers are still being delivered, other Linux IRQs too. switchtest complained about

"Warning: Linux is compiled to use FPU in kernel-space."

Quoted 7 lines
Quoted 5 lines

when it was started. Kernels are 2.6.28.9/ipipe-x86-2.2-07 and 2.6.29.3/ipipe-x86-2.3-01 (LTTng patched in, but unused), both show the same effect.

Seen this before?

The warning about Linux being compiled to use FPU in kernel-space means that you enabled soft RAID or compiled for K7, Geode, or any other

RAID is on (ordinary server config).

Quoted 9 lines

configuration using 3DNow for such simple operations as memcpy. It is harmless, it simply means that switchtest can not use fpu in kernel-space.

The bug you have is probably the same as the one described here, which I am able to reproduce on my atom:
https://mail.gna.org/public/xenomai-help/2009-04/msg00200.html

Unfortunately, I for one am working on ARM issues and am not available to debug x86 issues. I think Philippe is busy too...

Quoted 7 lines
Quoted 4 lines

OK, looks like I got the same flu here.

Philippe, did you find out any more details in the meantime? Then I'm afraid I have to pick this up.

No, I did not resume this task yet. Working from the powerpc side of the universe here.

Hoho, don't think this rain here over x86 would have never made it down to ARM or PPC land! ;)

Martin, could you check if this helps you, too?

Quoted 7 lines

Jan

(as usual, ready to be pulled from 'for-upstream')

--------->

Host IRQs may not only be triggered from non-root domains.

Are you sure of this? I can't find any spot where this assumption would be wrong. host_pend() is basically there to relay RT timer ticks and device IRQs, and this only happens on behalf of the pipeline head. At least, this is how rthal_irq_host_pend() should be used in any case. If you did find a spot where this interface is being called from the lower stage, then this is the root bug to fix.

Quoted 5 lines

But
rthal_propagate_irq's implemenation in I-pipe assumes so, which broke host tick propagation under certain load scenarios. Besides that, rthal_schedule_irq_root is the more efficient service for this purpose anyway.

Ack.

Quoted 21 lines

---

 include/asm-generic/hal.h |    2 +-
 1 files changed, 1 insertions(+), 1 deletions(-)
include/asm-generic/hal.h +1 -1
index b37e476..8137856 100644
@@ -437,7 +437,7 @@ int rthal_irq_host_release(unsigned irq,
 
 static inline void rthal_irq_host_pend(unsigned irq)
 {
-	rthal_propagate_irq(irq);
+	rthal_schedule_irq_root(irq);
 }
 
 int rthal_apc_alloc(const char *name,
Signature · 1 line
Philippe.
author
#6 lore raw

Re: [Xenomai-core] [PATCH] Fix host IRQ propagation

Quoted 50 lines · Philippe Gerum
Quoted 49 lines · Jan Kiszka
Quoted 29 lines · Philippe Gerum
Quoted 28 lines · Jan Kiszka
Quoted 11 lines · Gilles Chanteperdrix
Quoted 10 lines · Jan Kiszka

Hi Gilles,

I'm currently facing a nasty effect with switchtest over latest git head (only tested this so far): running it inside my test VM (ie. with frequent excessive latencies) I get a stalled Linux timer IRQ quite quickly. System is otherwise still responsive, Xenomai timers are still being delivered, other Linux IRQs too. switchtest complained about

"Warning: Linux is compiled to use FPU in kernel-space."

Quoted 7 lines
Quoted 5 lines

when it was started. Kernels are 2.6.28.9/ipipe-x86-2.2-07 and 2.6.29.3/ipipe-x86-2.3-01 (LTTng patched in, but unused), both show the same effect.

Seen this before?

The warning about Linux being compiled to use FPU in kernel-space means that you enabled soft RAID or compiled for K7, Geode, or any other

RAID is on (ordinary server config).

Quoted 7 lines

configuration using 3DNow for such simple operations as memcpy. It is harmless, it simply means that switchtest can not use fpu in kernel-space.

The bug you have is probably the same as the one described here, which I am able to reproduce on my atom:
https://mail.gna.org/public/xenomai-help/2009-04/msg00200.html

Quoted 8 lines
Quoted 6 lines
Quoted 2 lines

Unfortunately, I for one am working on ARM issues and am not available to debug x86 issues. I think Philippe is busy too...

OK, looks like I got the same flu here.

Philippe, did you find out any more details in the meantime? Then I'm afraid I have to pick this up.

No, I did not resume this task yet. Working from the powerpc side of the universe here.

Hoho, don't think this rain here over x86 would have never made it down to ARM or PPC land! ;)

Martin, could you check if this helps you, too?

Jan

(as usual, ready to be pulled from 'for-upstream')

--------->

Quoted 8 lines
Quoted 1 line

Host IRQs may not only be triggered from non-root domains.

Are you sure of this? I can't find any spot where this assumption would be wrong. host_pend() is basically there to relay RT timer ticks and device IRQs, and this only happens on behalf of the pipeline head. At least, this is how rthal_irq_host_pend() should be used in any case. If you did find a spot where this interface is being called from the lower stage, then this is the root bug to fix.

I haven't studied the I-pipe trace /wrt this in details yet, but I could imagine that some shadow task is interrupted in primary mode by the timer IRQ and then leaves the handler in secondary mode due to whatever events between schedule-out and in at the end of xnintr_clock_handler.

Jan

Signature · 2 lines
Siemens AG, Corporate Technology, CT SE 2
Corporate Competence Center Embedded Linux
#7 lore raw

Re: [Xenomai-core] [PATCH] Fix host IRQ propagation

Quoted 60 lines · Jan Kiszka
Quoted 50 lines · Philippe Gerum
Quoted 49 lines · Jan Kiszka
Quoted 29 lines · Philippe Gerum
Quoted 28 lines · Jan Kiszka
Quoted 11 lines · Gilles Chanteperdrix
Quoted 10 lines · Jan Kiszka

Hi Gilles,

I'm currently facing a nasty effect with switchtest over latest git head (only tested this so far): running it inside my test VM (ie. with frequent excessive latencies) I get a stalled Linux timer IRQ quite quickly. System is otherwise still responsive, Xenomai timers are still being delivered, other Linux IRQs too. switchtest complained about

"Warning: Linux is compiled to use FPU in kernel-space."

Quoted 7 lines
Quoted 5 lines

when it was started. Kernels are 2.6.28.9/ipipe-x86-2.2-07 and 2.6.29.3/ipipe-x86-2.3-01 (LTTng patched in, but unused), both show the same effect.

Seen this before?

The warning about Linux being compiled to use FPU in kernel-space means that you enabled soft RAID or compiled for K7, Geode, or any other

RAID is on (ordinary server config).

Quoted 7 lines

configuration using 3DNow for such simple operations as memcpy. It is harmless, it simply means that switchtest can not use fpu in kernel-space.

The bug you have is probably the same as the one described here, which I am able to reproduce on my atom:
https://mail.gna.org/public/xenomai-help/2009-04/msg00200.html

Quoted 8 lines
Quoted 6 lines
Quoted 2 lines

Unfortunately, I for one am working on ARM issues and am not available to debug x86 issues. I think Philippe is busy too...

OK, looks like I got the same flu here.

Philippe, did you find out any more details in the meantime? Then I'm afraid I have to pick this up.

No, I did not resume this task yet. Working from the powerpc side of the universe here.

Hoho, don't think this rain here over x86 would have never made it down to ARM or PPC land! ;)

Martin, could you check if this helps you, too?

Jan

(as usual, ready to be pulled from 'for-upstream')

--------->

Quoted 8 lines
Quoted 1 line

Host IRQs may not only be triggered from non-root domains.

Are you sure of this? I can't find any spot where this assumption would be wrong. host_pend() is basically there to relay RT timer ticks and device IRQs, and this only happens on behalf of the pipeline head. At least, this is how rthal_irq_host_pend() should be used in any case. If you did find a spot where this interface is being called from the lower stage, then this is the root bug to fix.

Quoted 5 lines

I haven't studied the I-pipe trace /wrt this in details yet, but I could imagine that some shadow task is interrupted in primary mode by the timer IRQ and then leaves the handler in secondary mode due to whatever events between schedule-out and in at the end of xnintr_clock_handler.

You need a thread context to move to secondary, I just can't see how such scenario would be possible.

Quoted 2 lines

Jan

Signature · 1 line
Philippe.
author
#8 lore raw

Re: [Xenomai-core] [PATCH] Fix host IRQ propagation

Quoted 59 lines · Philippe Gerum
Quoted 58 lines · Jan Kiszka
Quoted 57 lines · Philippe Gerum
Quoted 43 lines · Jan Kiszka
Quoted 29 lines · Philippe Gerum
Quoted 28 lines · Jan Kiszka
Quoted 11 lines · Gilles Chanteperdrix
Quoted 10 lines · Jan Kiszka

Hi Gilles,

I'm currently facing a nasty effect with switchtest over latest git head (only tested this so far): running it inside my test VM (ie. with frequent excessive latencies) I get a stalled Linux timer IRQ quite quickly. System is otherwise still responsive, Xenomai timers are still being delivered, other Linux IRQs too. switchtest complained about

"Warning: Linux is compiled to use FPU in kernel-space."

Quoted 7 lines
Quoted 5 lines

when it was started. Kernels are 2.6.28.9/ipipe-x86-2.2-07 and 2.6.29.3/ipipe-x86-2.3-01 (LTTng patched in, but unused), both show the same effect.

Seen this before?

The warning about Linux being compiled to use FPU in kernel-space means that you enabled soft RAID or compiled for K7, Geode, or any other

RAID is on (ordinary server config).

Quoted 7 lines

configuration using 3DNow for such simple operations as memcpy. It is harmless, it simply means that switchtest can not use fpu in kernel-space.

The bug you have is probably the same as the one described here, which I am able to reproduce on my atom:
https://mail.gna.org/public/xenomai-help/2009-04/msg00200.html

Quoted 8 lines
Quoted 6 lines
Quoted 2 lines

Unfortunately, I for one am working on ARM issues and am not available to debug x86 issues. I think Philippe is busy too...

OK, looks like I got the same flu here.

Philippe, did you find out any more details in the meantime? Then I'm afraid I have to pick this up.

No, I did not resume this task yet. Working from the powerpc side of the universe here.

Hoho, don't think this rain here over x86 would have never made it down to ARM or PPC land! ;)

Martin, could you check if this helps you, too?

Quoted 7 lines

Jan

(as usual, ready to be pulled from 'for-upstream')

--------->

Host IRQs may not only be triggered from non-root domains.

Are you sure of this? I can't find any spot where this assumption would be wrong. host_pend() is basically there to relay RT timer ticks and device IRQs, and this only happens on behalf of the pipeline head. At least, this is how rthal_irq_host_pend() should be used in any case. If you did find a spot where this interface is being called from the lower stage, then this is the root bug to fix.

Quoted 8 lines
Quoted 5 lines

I haven't studied the I-pipe trace /wrt this in details yet, but I could imagine that some shadow task is interrupted in primary mode by the timer IRQ and then leaves the handler in secondary mode due to whatever events between schedule-out and in at the end of xnintr_clock_handler.

You need a thread context to move to secondary, I just can't see how such scenario would be possible.

Here is the trace of events:

=> Shadow task starts migration to secondary
=> in xnpod_suspend_thread, nklock is briefly released before
xnpod_schedule
=> timer IRQ intercepts
=> as the current CPU is marked for reschedule, we enter xnpod_schedule
before propagating the host tick
=> once the migrating thread comes in again, it will run the
xnintr_clock_handler tail, i.e. xnarch_relay_tick, already over the root domain

Jan

Signature · 2 lines
Siemens AG, Corporate Technology, CT SE 2
Corporate Competence Center Embedded Linux
#9 lore raw

Re: [Xenomai-core] [PATCH] Fix host IRQ propagation

Quoted 66 lines · Jan Kiszka
Quoted 65 lines · Philippe Gerum
Quoted 63 lines · Jan Kiszka
Quoted 50 lines · Philippe Gerum
Quoted 49 lines · Jan Kiszka
Quoted 29 lines · Philippe Gerum
Quoted 28 lines · Jan Kiszka
Quoted 11 lines · Gilles Chanteperdrix
Quoted 10 lines · Jan Kiszka

Hi Gilles,

I'm currently facing a nasty effect with switchtest over latest git head (only tested this so far): running it inside my test VM (ie. with frequent excessive latencies) I get a stalled Linux timer IRQ quite quickly. System is otherwise still responsive, Xenomai timers are still being delivered, other Linux IRQs too. switchtest complained about

"Warning: Linux is compiled to use FPU in kernel-space."

Quoted 7 lines
Quoted 5 lines

when it was started. Kernels are 2.6.28.9/ipipe-x86-2.2-07 and 2.6.29.3/ipipe-x86-2.3-01 (LTTng patched in, but unused), both show the same effect.

Seen this before?

The warning about Linux being compiled to use FPU in kernel-space means that you enabled soft RAID or compiled for K7, Geode, or any other

RAID is on (ordinary server config).

Quoted 7 lines

configuration using 3DNow for such simple operations as memcpy. It is harmless, it simply means that switchtest can not use fpu in kernel-space.

The bug you have is probably the same as the one described here, which I am able to reproduce on my atom:
https://mail.gna.org/public/xenomai-help/2009-04/msg00200.html

Quoted 8 lines
Quoted 6 lines
Quoted 2 lines

Unfortunately, I for one am working on ARM issues and am not available to debug x86 issues. I think Philippe is busy too...

OK, looks like I got the same flu here.

Philippe, did you find out any more details in the meantime? Then I'm afraid I have to pick this up.

No, I did not resume this task yet. Working from the powerpc side of the universe here.

Hoho, don't think this rain here over x86 would have never made it down to ARM or PPC land! ;)

Martin, could you check if this helps you, too?

Jan

(as usual, ready to be pulled from 'for-upstream')

--------->

Quoted 7 lines
Quoted 1 line

Host IRQs may not only be triggered from non-root domains.

Are you sure of this? I can't find any spot where this assumption would be wrong. host_pend() is basically there to relay RT timer ticks and device IRQs, and this only happens on behalf of the pipeline head. At least, this is how rthal_irq_host_pend() should be used in any case. If you did find a spot where this interface is being called from the lower stage, then this is the root bug to fix.

I haven't studied the I-pipe trace /wrt this in details yet, but I could imagine that some shadow task is interrupted in primary mode by the timer IRQ and then leaves the handler in secondary mode due to whatever events between schedule-out and in at the end of xnintr_clock_handler.

Quoted 8 lines
Quoted 2 lines

You need a thread context to move to secondary, I just can't see how such scenario would be possible.

Here is the trace of events:

=> Shadow task starts migration to secondary
=> in xnpod_suspend_thread, nklock is briefly released before
xnpod_schedule

Which is the root bug. Blame on me; this recent change in -head breaks a basic rule a lot of code is based on: a self-suspending thread may not be preempted while scheduling out, i.e. suspension and rescheduling must be atomically performed. xnshadow_relax() counts on this too.

Quoted 6 lines

=> timer IRQ intercepts
=> as the current CPU is marked for reschedule, we enter xnpod_schedule
before propagating the host tick
=> once the migrating thread comes in again, it will run the
xnintr_clock_handler tail, i.e. xnarch_relay_tick, already over the root domain

Ok, makes sense now. However, this can't happen with 2.4 which has no such lock release in xnpod_suspend_thread(). So the question is: was the "lost tick" bug observed also on 2.4, or not?

Quoted 3 lines

Jan

Signature · 1 line
Philippe.
author
#10 lore raw

Re: [Xenomai-core] [PATCH] Fix host IRQ propagation

Quoted 70 lines · Philippe Gerum
Quoted 69 lines · Jan Kiszka
Quoted 59 lines · Philippe Gerum
Quoted 58 lines · Jan Kiszka
Quoted 57 lines · Philippe Gerum
Quoted 43 lines · Jan Kiszka
Quoted 29 lines · Philippe Gerum
Quoted 28 lines · Jan Kiszka
Quoted 11 lines · Gilles Chanteperdrix
Quoted 10 lines · Jan Kiszka

Hi Gilles,

I'm currently facing a nasty effect with switchtest over latest git head (only tested this so far): running it inside my test VM (ie. with frequent excessive latencies) I get a stalled Linux timer IRQ quite quickly. System is otherwise still responsive, Xenomai timers are still being delivered, other Linux IRQs too. switchtest complained about

"Warning: Linux is compiled to use FPU in kernel-space."

Quoted 7 lines
Quoted 5 lines

when it was started. Kernels are 2.6.28.9/ipipe-x86-2.2-07 and 2.6.29.3/ipipe-x86-2.3-01 (LTTng patched in, but unused), both show the same effect.

Seen this before?

The warning about Linux being compiled to use FPU in kernel-space means that you enabled soft RAID or compiled for K7, Geode, or any other

RAID is on (ordinary server config).

Quoted 7 lines

configuration using 3DNow for such simple operations as memcpy. It is harmless, it simply means that switchtest can not use fpu in kernel-space.

The bug you have is probably the same as the one described here, which I am able to reproduce on my atom:
https://mail.gna.org/public/xenomai-help/2009-04/msg00200.html

Quoted 8 lines
Quoted 6 lines
Quoted 2 lines

Unfortunately, I for one am working on ARM issues and am not available to debug x86 issues. I think Philippe is busy too...

OK, looks like I got the same flu here.

Philippe, did you find out any more details in the meantime? Then I'm afraid I have to pick this up.

No, I did not resume this task yet. Working from the powerpc side of the universe here.

Hoho, don't think this rain here over x86 would have never made it down to ARM or PPC land! ;)

Martin, could you check if this helps you, too?

Quoted 7 lines

Jan

(as usual, ready to be pulled from 'for-upstream')

--------->

Host IRQs may not only be triggered from non-root domains.

Are you sure of this? I can't find any spot where this assumption would be wrong. host_pend() is basically there to relay RT timer ticks and device IRQs, and this only happens on behalf of the pipeline head. At least, this is how rthal_irq_host_pend() should be used in any case. If you did find a spot where this interface is being called from the lower stage, then this is the root bug to fix.

Quoted 7 lines
Quoted 5 lines

I haven't studied the I-pipe trace /wrt this in details yet, but I could imagine that some shadow task is interrupted in primary mode by the timer IRQ and then leaves the handler in secondary mode due to whatever events between schedule-out and in at the end of xnintr_clock_handler.

You need a thread context to move to secondary, I just can't see how such scenario would be possible.

Here is the trace of events:

Quoted 8 lines
Quoted 3 lines

=> Shadow task starts migration to secondary
=> in xnpod_suspend_thread, nklock is briefly released before
xnpod_schedule

Which is the root bug. Blame on me; this recent change in -head breaks a basic rule a lot of code is based on: a self-suspending thread may not be preempted while scheduling out, i.e. suspension and rescheduling must be atomically performed. xnshadow_relax() counts on this too.

Oh, good that you insisted on this. Will you fix it soon? We are currently packaging a delivery of 2.5.git, and I would like to see this hole closed there already.

Quoted 11 lines
Quoted 7 lines

=> timer IRQ intercepts
=> as the current CPU is marked for reschedule, we enter xnpod_schedule
before propagating the host tick
=> once the migrating thread comes in again, it will run the
xnintr_clock_handler tail, i.e. xnarch_relay_tick, already over the root domain

Ok, makes sense now. However, this can't happen with 2.4 which has no such lock release in xnpod_suspend_thread(). So the question is: was the "lost tick" bug observed also on 2.4, or not?

I haven't tested on 2.4, but Martin anyway reported that his problem is still unfixed for 2.5 even with my patch.

Jan

Signature · 2 lines
Siemens AG, Corporate Technology, CT SE 2
Corporate Competence Center Embedded Linux
#11 lore raw

Re: [Xenomai-core] [PATCH] Fix host IRQ propagation

Quoted 75 lines · Jan Kiszka
Quoted 74 lines · Philippe Gerum
Quoted 72 lines · Jan Kiszka
Quoted 59 lines · Philippe Gerum
Quoted 58 lines · Jan Kiszka
Quoted 57 lines · Philippe Gerum
Quoted 43 lines · Jan Kiszka
Quoted 29 lines · Philippe Gerum
Quoted 28 lines · Jan Kiszka
Quoted 11 lines · Gilles Chanteperdrix
Quoted 10 lines · Jan Kiszka

Hi Gilles,

I'm currently facing a nasty effect with switchtest over latest git head (only tested this so far): running it inside my test VM (ie. with frequent excessive latencies) I get a stalled Linux timer IRQ quite quickly. System is otherwise still responsive, Xenomai timers are still being delivered, other Linux IRQs too. switchtest complained about

"Warning: Linux is compiled to use FPU in kernel-space."

Quoted 7 lines
Quoted 5 lines

when it was started. Kernels are 2.6.28.9/ipipe-x86-2.2-07 and 2.6.29.3/ipipe-x86-2.3-01 (LTTng patched in, but unused), both show the same effect.

Seen this before?

The warning about Linux being compiled to use FPU in kernel-space means that you enabled soft RAID or compiled for K7, Geode, or any other

RAID is on (ordinary server config).

Quoted 7 lines

configuration using 3DNow for such simple operations as memcpy. It is harmless, it simply means that switchtest can not use fpu in kernel-space.

The bug you have is probably the same as the one described here, which I am able to reproduce on my atom:
https://mail.gna.org/public/xenomai-help/2009-04/msg00200.html

Quoted 8 lines
Quoted 6 lines
Quoted 2 lines

Unfortunately, I for one am working on ARM issues and am not available to debug x86 issues. I think Philippe is busy too...

OK, looks like I got the same flu here.

Philippe, did you find out any more details in the meantime? Then I'm afraid I have to pick this up.

No, I did not resume this task yet. Working from the powerpc side of the universe here.

Hoho, don't think this rain here over x86 would have never made it down to ARM or PPC land! ;)

Martin, could you check if this helps you, too?

Quoted 7 lines

Jan

(as usual, ready to be pulled from 'for-upstream')

--------->

Host IRQs may not only be triggered from non-root domains.

Are you sure of this? I can't find any spot where this assumption would be wrong. host_pend() is basically there to relay RT timer ticks and device IRQs, and this only happens on behalf of the pipeline head. At least, this is how rthal_irq_host_pend() should be used in any case. If you did find a spot where this interface is being called from the lower stage, then this is the root bug to fix.

Quoted 7 lines
Quoted 5 lines

I haven't studied the I-pipe trace /wrt this in details yet, but I could imagine that some shadow task is interrupted in primary mode by the timer IRQ and then leaves the handler in secondary mode due to whatever events between schedule-out and in at the end of xnintr_clock_handler.

You need a thread context to move to secondary, I just can't see how such scenario would be possible.

Here is the trace of events:

=> Shadow task starts migration to secondary
=> in xnpod_suspend_thread, nklock is briefly released before
xnpod_schedule

Quoted 9 lines
Quoted 4 lines

Which is the root bug. Blame on me; this recent change in -head breaks a basic rule a lot of code is based on: a self-suspending thread may not be preempted while scheduling out, i.e. suspension and rescheduling must be atomically performed. xnshadow_relax() counts on this too.

Oh, good that you insisted on this. Will you fix it soon? We are currently packaging a delivery of 2.5.git, and I would like to see this hole closed there already.

I just pushed a fix for this.

Quoted 17 lines
Quoted 11 lines
Quoted 7 lines

=> timer IRQ intercepts
=> as the current CPU is marked for reschedule, we enter xnpod_schedule
before propagating the host tick
=> once the migrating thread comes in again, it will run the
xnintr_clock_handler tail, i.e. xnarch_relay_tick, already over the root domain

Ok, makes sense now. However, this can't happen with 2.4 which has no such lock release in xnpod_suspend_thread(). So the question is: was the "lost tick" bug observed also on 2.4, or not?

I haven't tested on 2.4, but Martin anyway reported that his problem is still unfixed for 2.5 even with my patch.

Jan

Signature · 1 line
Philippe.
#12 lore raw

Re: [Xenomai-core] [PATCH] Fix host IRQ propagation

In reply to Philippe Gerum
Quoted 70 lines · Philippe Gerum
Quoted 69 lines · Jan Kiszka
Quoted 59 lines · Philippe Gerum
Quoted 58 lines · Jan Kiszka
Quoted 57 lines · Philippe Gerum
Quoted 43 lines · Jan Kiszka
Quoted 29 lines · Philippe Gerum
Quoted 28 lines · Jan Kiszka
Quoted 11 lines · Gilles Chanteperdrix
Quoted 10 lines · Jan Kiszka

Hi Gilles,

I'm currently facing a nasty effect with switchtest over latest git head (only tested this so far): running it inside my test VM (ie. with frequent excessive latencies) I get a stalled Linux timer IRQ quite quickly. System is otherwise still responsive, Xenomai timers are still being delivered, other Linux IRQs too. switchtest complained about

"Warning: Linux is compiled to use FPU in kernel-space."

Quoted 7 lines
Quoted 5 lines

when it was started. Kernels are 2.6.28.9/ipipe-x86-2.2-07 and 2.6.29.3/ipipe-x86-2.3-01 (LTTng patched in, but unused), both show the same effect.

Seen this before?

The warning about Linux being compiled to use FPU in kernel-space means that you enabled soft RAID or compiled for K7, Geode, or any other

RAID is on (ordinary server config).

Quoted 7 lines

configuration using 3DNow for such simple operations as memcpy. It is harmless, it simply means that switchtest can not use fpu in kernel-space.

The bug you have is probably the same as the one described here, which I am able to reproduce on my atom:
https://mail.gna.org/public/xenomai-help/2009-04/msg00200.html

Quoted 8 lines
Quoted 6 lines
Quoted 2 lines

Unfortunately, I for one am working on ARM issues and am not available to debug x86 issues. I think Philippe is busy too...

OK, looks like I got the same flu here.

Philippe, did you find out any more details in the meantime? Then I'm afraid I have to pick this up.

No, I did not resume this task yet. Working from the powerpc side of the universe here.

Hoho, don't think this rain here over x86 would have never made it down to ARM or PPC land! ;)

Martin, could you check if this helps you, too?

Quoted 7 lines

Jan

(as usual, ready to be pulled from 'for-upstream')

--------->

Host IRQs may not only be triggered from non-root domains.

Are you sure of this? I can't find any spot where this assumption would be wrong. host_pend() is basically there to relay RT timer ticks and device IRQs, and this only happens on behalf of the pipeline head. At least, this is how rthal_irq_host_pend() should be used in any case. If you did find a spot where this interface is being called from the lower stage, then this is the root bug to fix.

Quoted 7 lines
Quoted 5 lines

I haven't studied the I-pipe trace /wrt this in details yet, but I could imagine that some shadow task is interrupted in primary mode by the timer IRQ and then leaves the handler in secondary mode due to whatever events between schedule-out and in at the end of xnintr_clock_handler.

You need a thread context to move to secondary, I just can't see how such scenario would be possible.

Here is the trace of events:

Quoted 8 lines
Quoted 3 lines

=> Shadow task starts migration to secondary
=> in xnpod_suspend_thread, nklock is briefly released before
xnpod_schedule

Which is the root bug. Blame on me; this recent change in -head breaks a basic rule a lot of code is based on: a self-suspending thread may not be preempted while scheduling out, i.e. suspension and rescheduling must be atomically performed. xnshadow_relax() counts on this too.

Actually, I think the idea was mine in the first place... Maybe we can specify a special flag to xnpod_suspend_thread to ask fo the atomic suspension (maybe reuse XNATOMIC ?).

Signature · 1 line
                                                 Gilles.
#13 lore raw

Re: [Xenomai-core] [PATCH] Fix host IRQ propagation

Quoted 75 lines · Gilles Chanteperdrix
Quoted 74 lines · Philippe Gerum
Quoted 72 lines · Jan Kiszka
Quoted 59 lines · Philippe Gerum
Quoted 58 lines · Jan Kiszka
Quoted 57 lines · Philippe Gerum
Quoted 43 lines · Jan Kiszka
Quoted 29 lines · Philippe Gerum
Quoted 28 lines · Jan Kiszka
Quoted 11 lines · Gilles Chanteperdrix
Quoted 10 lines · Jan Kiszka

Hi Gilles,

I'm currently facing a nasty effect with switchtest over latest git head (only tested this so far): running it inside my test VM (ie. with frequent excessive latencies) I get a stalled Linux timer IRQ quite quickly. System is otherwise still responsive, Xenomai timers are still being delivered, other Linux IRQs too. switchtest complained about

"Warning: Linux is compiled to use FPU in kernel-space."

Quoted 7 lines
Quoted 5 lines

when it was started. Kernels are 2.6.28.9/ipipe-x86-2.2-07 and 2.6.29.3/ipipe-x86-2.3-01 (LTTng patched in, but unused), both show the same effect.

Seen this before?

The warning about Linux being compiled to use FPU in kernel-space means that you enabled soft RAID or compiled for K7, Geode, or any other

RAID is on (ordinary server config).

Quoted 7 lines

configuration using 3DNow for such simple operations as memcpy. It is harmless, it simply means that switchtest can not use fpu in kernel-space.

The bug you have is probably the same as the one described here, which I am able to reproduce on my atom:
https://mail.gna.org/public/xenomai-help/2009-04/msg00200.html

Quoted 8 lines
Quoted 6 lines
Quoted 2 lines

Unfortunately, I for one am working on ARM issues and am not available to debug x86 issues. I think Philippe is busy too...

OK, looks like I got the same flu here.

Philippe, did you find out any more details in the meantime? Then I'm afraid I have to pick this up.

No, I did not resume this task yet. Working from the powerpc side of the universe here.

Hoho, don't think this rain here over x86 would have never made it down to ARM or PPC land! ;)

Martin, could you check if this helps you, too?

Quoted 7 lines

Jan

(as usual, ready to be pulled from 'for-upstream')

--------->

Host IRQs may not only be triggered from non-root domains.

Are you sure of this? I can't find any spot where this assumption would be wrong. host_pend() is basically there to relay RT timer ticks and device IRQs, and this only happens on behalf of the pipeline head. At least, this is how rthal_irq_host_pend() should be used in any case. If you did find a spot where this interface is being called from the lower stage, then this is the root bug to fix.

Quoted 7 lines
Quoted 5 lines

I haven't studied the I-pipe trace /wrt this in details yet, but I could imagine that some shadow task is interrupted in primary mode by the timer IRQ and then leaves the handler in secondary mode due to whatever events between schedule-out and in at the end of xnintr_clock_handler.

You need a thread context to move to secondary, I just can't see how such scenario would be possible.

Here is the trace of events:

=> Shadow task starts migration to secondary
=> in xnpod_suspend_thread, nklock is briefly released before
xnpod_schedule

Quoted 9 lines
Quoted 4 lines

Which is the root bug. Blame on me; this recent change in -head breaks a basic rule a lot of code is based on: a self-suspending thread may not be preempted while scheduling out, i.e. suspension and rescheduling must be atomically performed. xnshadow_relax() counts on this too.

Actually, I think the idea was mine in the first place... Maybe we can specify a special flag to xnpod_suspend_thread to ask fo the atomic suspension (maybe reuse XNATOMIC ?).

I don't think so. We really need the basic assumption to hold in any case, because this is expected by most of the callers, and this micro-optimization is not worth the risk of introducing a race if misused.

Signature · 1 line
Philippe.
#14 lore raw

Re: [Xenomai-core] [PATCH] Fix host IRQ propagation

Quoted 84 lines · Philippe Gerum
Quoted 74 lines · Gilles Chanteperdrix
Quoted 73 lines · Philippe Gerum
Quoted 72 lines · Jan Kiszka
Quoted 59 lines · Philippe Gerum
Quoted 58 lines · Jan Kiszka
Quoted 57 lines · Philippe Gerum
Quoted 43 lines · Jan Kiszka
Quoted 29 lines · Philippe Gerum
Quoted 28 lines · Jan Kiszka
Quoted 11 lines · Gilles Chanteperdrix
Quoted 10 lines · Jan Kiszka

Hi Gilles,

I'm currently facing a nasty effect with switchtest over latest git head (only tested this so far): running it inside my test VM (ie. with frequent excessive latencies) I get a stalled Linux timer IRQ quite quickly. System is otherwise still responsive, Xenomai timers are still being delivered, other Linux IRQs too. switchtest complained about

"Warning: Linux is compiled to use FPU in kernel-space."

Quoted 7 lines
Quoted 5 lines

when it was started. Kernels are 2.6.28.9/ipipe-x86-2.2-07 and 2.6.29.3/ipipe-x86-2.3-01 (LTTng patched in, but unused), both show the same effect.

Seen this before?

The warning about Linux being compiled to use FPU in kernel-space means that you enabled soft RAID or compiled for K7, Geode, or any other

RAID is on (ordinary server config).

Quoted 7 lines

configuration using 3DNow for such simple operations as memcpy. It is harmless, it simply means that switchtest can not use fpu in kernel-space.

The bug you have is probably the same as the one described here, which I am able to reproduce on my atom:
https://mail.gna.org/public/xenomai-help/2009-04/msg00200.html

Quoted 8 lines
Quoted 6 lines
Quoted 2 lines

Unfortunately, I for one am working on ARM issues and am not available to debug x86 issues. I think Philippe is busy too...

OK, looks like I got the same flu here.

Philippe, did you find out any more details in the meantime? Then I'm afraid I have to pick this up.

No, I did not resume this task yet. Working from the powerpc side of the universe here.

Hoho, don't think this rain here over x86 would have never made it down to ARM or PPC land! ;)

Martin, could you check if this helps you, too?

Quoted 7 lines

Jan

(as usual, ready to be pulled from 'for-upstream')

--------->

Host IRQs may not only be triggered from non-root domains.

Are you sure of this? I can't find any spot where this assumption would be wrong. host_pend() is basically there to relay RT timer ticks and device IRQs, and this only happens on behalf of the pipeline head. At least, this is how rthal_irq_host_pend() should be used in any case. If you did find a spot where this interface is being called from the lower stage, then this is the root bug to fix.

Quoted 7 lines
Quoted 5 lines

I haven't studied the I-pipe trace /wrt this in details yet, but I could imagine that some shadow task is interrupted in primary mode by the timer IRQ and then leaves the handler in secondary mode due to whatever events between schedule-out and in at the end of xnintr_clock_handler.

You need a thread context to move to secondary, I just can't see how such scenario would be possible.

Here is the trace of events:

=> Shadow task starts migration to secondary
=> in xnpod_suspend_thread, nklock is briefly released before
xnpod_schedule

Quoted 8 lines
Quoted 4 lines

Which is the root bug. Blame on me; this recent change in -head breaks a basic rule a lot of code is based on: a self-suspending thread may not be preempted while scheduling out, i.e. suspension and rescheduling must be atomically performed. xnshadow_relax() counts on this too.

Actually, I think the idea was mine in the first place... Maybe we can specify a special flag to xnpod_suspend_thread to ask fo the atomic suspension (maybe reuse XNATOMIC ?).

Quoted 4 lines

I don't think so. We really need the basic assumption to hold in any case, because this is expected by most of the callers, and this micro-optimization is not worth the risk of introducing a race if misused.

Well, I tend to disagree. The assumption that the thread is suspended from the point of view of the scheduler still holds even when the nklock is released, and it is what callers like rt_cond_wait are expecting. The assumptions of xnshadow_relax do not seem to me like a common assumption.

To go further, maybe forwarding the tick in xnintr_clock_handler epilogue is wrong in the first place, precisely because the current thread where xnintr_clock_handler happens may be scheduled out for a long time by the clock handler itself.

Signature · 1 line
                                                 Gilles.
#15 lore raw

Re: [Xenomai-core] [PATCH] Fix host IRQ propagation

Quoted 90 lines · Gilles Chanteperdrix
Quoted 84 lines · Philippe Gerum
Quoted 74 lines · Gilles Chanteperdrix
Quoted 73 lines · Philippe Gerum
Quoted 72 lines · Jan Kiszka
Quoted 59 lines · Philippe Gerum
Quoted 58 lines · Jan Kiszka
Quoted 57 lines · Philippe Gerum
Quoted 43 lines · Jan Kiszka
Quoted 29 lines · Philippe Gerum
Quoted 28 lines · Jan Kiszka
Quoted 11 lines · Gilles Chanteperdrix
Quoted 10 lines · Jan Kiszka

Hi Gilles,

I'm currently facing a nasty effect with switchtest over latest git head
(only tested this so far): running it inside my test VM (ie. with
frequent excessive latencies) I get a stalled Linux timer IRQ quite
quickly. System is otherwise still responsive, Xenomai timers are still
being delivered, other Linux IRQs too. switchtest complained about

"Warning: Linux is compiled to use FPU in kernel-space."

Quoted 7 lines
Quoted 5 lines

when it was started. Kernels are 2.6.28.9/ipipe-x86-2.2-07 and
2.6.29.3/ipipe-x86-2.3-01 (LTTng patched in, but unused), both show the
same effect.

Seen this before?

The warning about Linux being compiled to use FPU in kernel-space means that you enabled soft RAID or compiled for K7, Geode, or any other

RAID is on (ordinary server config).

Quoted 7 lines

configuration using 3DNow for such simple operations as memcpy. It is harmless, it simply means that switchtest can not use fpu in kernel-space.

The bug you have is probably the same as the one described here, which I am able to reproduce on my atom:
https://mail.gna.org/public/xenomai-help/2009-04/msg00200.html

Quoted 8 lines
Quoted 6 lines
Quoted 2 lines

Unfortunately, I for one am working on ARM issues and am not available to debug x86 issues. I think Philippe is busy too...

OK, looks like I got the same flu here.

Philippe, did you find out any more details in the meantime? Then I'm afraid I have to pick this up.

No, I did not resume this task yet. Working from the powerpc side of the universe here.

Hoho, don't think this rain here over x86 would have never made it down to ARM or PPC land! ;)

Martin, could you check if this helps you, too?

Quoted 7 lines

Jan

(as usual, ready to be pulled from 'for-upstream')

--------->

Host IRQs may not only be triggered from non-root domains.

Are you sure of this? I can't find any spot where this assumption would be wrong. host_pend() is basically there to relay RT timer ticks and device IRQs, and this only happens on behalf of the pipeline head. At least, this is how rthal_irq_host_pend() should be used in any case. If you did find a spot where this interface is being called from the lower stage, then this is the root bug to fix.

Quoted 7 lines
Quoted 5 lines

I haven't studied the I-pipe trace /wrt this in details yet, but I could imagine that some shadow task is interrupted in primary mode by the timer IRQ and then leaves the handler in secondary mode due to whatever events between schedule-out and in at the end of xnintr_clock_handler.

You need a thread context to move to secondary, I just can't see how such scenario would be possible.

Here is the trace of events:

=> Shadow task starts migration to secondary
=> in xnpod_suspend_thread, nklock is briefly released before
xnpod_schedule

Quoted 8 lines
Quoted 4 lines

Which is the root bug. Blame on me; this recent change in -head breaks a basic rule a lot of code is based on: a self-suspending thread may not be preempted while scheduling out, i.e. suspension and rescheduling must be atomically performed. xnshadow_relax() counts on this too.

Actually, I think the idea was mine in the first place... Maybe we can specify a special flag to xnpod_suspend_thread to ask fo the atomic suspension (maybe reuse XNATOMIC ?).

Quoted 4 lines

I don't think so. We really need the basic assumption to hold in any case, because this is expected by most of the callers, and this micro-optimization is not worth the risk of introducing a race if misused.

Quoted 5 lines

Well, I tend to disagree. The assumption that the thread is suspended from the point of view of the scheduler still holds even when the nklock is released, and it is what callers like rt_cond_wait are expecting. The assumptions of xnshadow_relax do not seem to me like a common assumption.

The assumption is that the thread has been suspended _and_ scheduled out atomically, not only put in a suspended thread, which is quite different when considered from an interrupt context. I'm worried by the fact that re-enabling interrupts in the middle of this critical transition breaks the unspoken rule that sched->curr may not be seen as bearing any block bit in its status word from anywhere in the code executed from the local CPU but xnpod_suspend_thread().

Another issue may arise in the SMP case, where xnpod_suspend_thread() would block a thread running on a remote CPU; in theory, re-enabling interrupts before the IPIs are sent from xnpod_schedule() - to kick the remote resched procedure - may introduce an undefined delay due to local interrupts being processed, albeit some runnable thread may be waiting for the CPU to be released on the remote CPU. We would then end up with a local scheduling artefact leaking remotely. I don't quite like this, in fact.

Admittedly, we could plug each and every subtle hole we find like this one, introducing XNATOMIC to fix the migration case as well, but my point is that it may not be worth opening pandora's box just for that micro-optimization.

Quoted 5 lines

To go further, maybe forwarding the tick in xnintr_clock_handler epilogue is wrong in the first place, precisely because the current thread where xnintr_clock_handler happens may be scheduled out for a long time by the clock handler itself.

It was mostly ok in the early days before RPI came in and with legacy kernels, since the root thread would always be the last to be scheduled; it is now wrong because:
- RPI may boost the root thread, so we could pend the timer IRQ long after the root domain has regained control.
- Linux tickless timing requires that no tick is lost due to the current IRQ context not returning from xnpod_schedule(), such as when calling xnpod_delete_thread() on behalf of a kernel-based RT thread, previously preempted by a timer IRQ. The same goes if xnpod_suspend_thread() is called upon this thread as well.

So in short, yes, you are definitely right. We would be better off moving this code close to the deferred tick handling instead, since we know for sure that the root thread is about to be scheduled in at that point. I just committed a fix for -head; this is a good candidate for a backport to 2.4.x.

Signature · 1 line
Philippe.
#16 lore raw

Re: [Xenomai-core] [PATCH] Fix host IRQ propagation

In reply to Jan Kiszka
Quoted 68 lines · Jan Kiszka
Quoted 29 lines · Philippe Gerum
Quoted 28 lines · Jan Kiszka
Quoted 11 lines · Gilles Chanteperdrix
Quoted 10 lines · Jan Kiszka

Hi Gilles,

I'm currently facing a nasty effect with switchtest over latest git head (only tested this so far): running it inside my test VM (ie. with frequent excessive latencies) I get a stalled Linux timer IRQ quite quickly. System is otherwise still responsive, Xenomai timers are still being delivered, other Linux IRQs too. switchtest complained about

"Warning: Linux is compiled to use FPU in kernel-space."

Quoted 7 lines
Quoted 5 lines

when it was started. Kernels are 2.6.28.9/ipipe-x86-2.2-07 and 2.6.29.3/ipipe-x86-2.3-01 (LTTng patched in, but unused), both show the same effect.

Seen this before?

The warning about Linux being compiled to use FPU in kernel-space means that you enabled soft RAID or compiled for K7, Geode, or any other

RAID is on (ordinary server config).

Quoted 7 lines

configuration using 3DNow for such simple operations as memcpy. It is harmless, it simply means that switchtest can not use fpu in kernel-space.

The bug you have is probably the same as the one described here, which I am able to reproduce on my atom:
https://mail.gna.org/public/xenomai-help/2009-04/msg00200.html

Quoted 8 lines
Quoted 6 lines
Quoted 2 lines

Unfortunately, I for one am working on ARM issues and am not available to debug x86 issues. I think Philippe is busy too...

OK, looks like I got the same flu here.

Philippe, did you find out any more details in the meantime? Then I'm afraid I have to pick this up.

No, I did not resume this task yet. Working from the powerpc side of the universe here.

Hoho, don't think this rain here over x86 would have never made it down to ARM or PPC land! ;)

Martin, could you check if this helps you, too?

Jan

(as usual, ready to be pulled from 'for-upstream')

--------->

Host IRQs may not only be triggered from non-root domains. But rthal_propagate_irq's implemenation in I-pipe assumes so, which broke host tick propagation under certain load scenarios. Besides that, rthal_schedule_irq_root is the more efficient service for this purpose anyway.

---

 include/asm-generic/hal.h |    2 +-
 1 files changed, 1 insertions(+), 1 deletions(-)
include/asm-generic/hal.h +0 -0
index b37e476..8137856 100644

include/asm-generic/hal.h int rthal_irq_host_release(unsigned irq, · line 443

Quoted 8 lines
include/asm-generic/hal.h +1 -1
@@ -437,7 +437,7 @@ int rthal_irq_host_release(unsigned irq,
 
 static inline void rthal_irq_host_pend(unsigned irq)
 {
-	rthal_propagate_irq(irq);
+	rthal_schedule_irq_root(irq);
 }
 
 int rthal_apc_alloc(const char *name,

Well, I would assume this could fix head, with the newly delayed root tick irq propagation. But the user reports this bug with 2.4 too...

Signature · 1 line
					    Gilles.
author
#17 lore raw

Re: [Xenomai-core] [PATCH] Fix host IRQ propagation

Quoted 74 lines · Gilles Chanteperdrix
Quoted 73 lines · Jan Kiszka
Quoted 29 lines · Philippe Gerum
Quoted 28 lines · Jan Kiszka
Quoted 11 lines · Gilles Chanteperdrix
Quoted 10 lines · Jan Kiszka

Hi Gilles,

I'm currently facing a nasty effect with switchtest over latest git head (only tested this so far): running it inside my test VM (ie. with frequent excessive latencies) I get a stalled Linux timer IRQ quite quickly. System is otherwise still responsive, Xenomai timers are still being delivered, other Linux IRQs too. switchtest complained about

"Warning: Linux is compiled to use FPU in kernel-space."

Quoted 7 lines
Quoted 5 lines

when it was started. Kernels are 2.6.28.9/ipipe-x86-2.2-07 and 2.6.29.3/ipipe-x86-2.3-01 (LTTng patched in, but unused), both show the same effect.

Seen this before?

The warning about Linux being compiled to use FPU in kernel-space means that you enabled soft RAID or compiled for K7, Geode, or any other

RAID is on (ordinary server config).

Quoted 7 lines

configuration using 3DNow for such simple operations as memcpy. It is harmless, it simply means that switchtest can not use fpu in kernel-space.

The bug you have is probably the same as the one described here, which I am able to reproduce on my atom:
https://mail.gna.org/public/xenomai-help/2009-04/msg00200.html

Quoted 8 lines
Quoted 6 lines
Quoted 2 lines

Unfortunately, I for one am working on ARM issues and am not available to debug x86 issues. I think Philippe is busy too...

OK, looks like I got the same flu here.

Philippe, did you find out any more details in the meantime? Then I'm afraid I have to pick this up.

No, I did not resume this task yet. Working from the powerpc side of the universe here.

Hoho, don't think this rain here over x86 would have never made it down to ARM or PPC land! ;)

Martin, could you check if this helps you, too?

Jan

(as usual, ready to be pulled from 'for-upstream')

--------->

Host IRQs may not only be triggered from non-root domains. But rthal_propagate_irq's implemenation in I-pipe assumes so, which broke host tick propagation under certain load scenarios. Besides that, rthal_schedule_irq_root is the more efficient service for this purpose anyway.

---

 include/asm-generic/hal.h |    2 +-
 1 files changed, 1 insertions(+), 1 deletions(-)
include/asm-generic/hal.h +1 -1
index b37e476..8137856 100644
@@ -437,7 +437,7 @@ int rthal_irq_host_release(unsigned irq,
 
 static inline void rthal_irq_host_pend(unsigned irq)
 {
-	rthal_propagate_irq(irq);
+	rthal_schedule_irq_root(irq);
 }
 
Quoted 5 lines

include/asm-generic/hal.h int rthal_irq_host_release(unsigned irq, · line 443

Quoted 2 lines
include/asm-generic/hal.h +0 -0
 int rthal_apc_alloc(const char *name,

Well, I would assume this could fix head, with the newly delayed root tick irq propagation. But the user reports this bug with 2.4 too...

According to my instrumentation, a delayed tick was not involved. The problem was 'propagate' vs. non-root trigger domain. So 2.4 should suffer from the same issue.

Jan

Signature · 2 lines
Siemens AG, Corporate Technology, CT SE 2
Corporate Competence Center Embedded Linux
#18 lore raw

Re: [Xenomai-core] [PATCH] Fix host IRQ propagation (was: Troubles with switchtest)

In reply to Jan Kiszka
Quoted 2 lines · Jan Kiszka

...
Martin, could you check if this helps you, too?

It doesn't appear to help. To check, first I turned on the HPET and PM timer options, and recompiled the kernel without your patch, to verify that this reproduced the problem. I then manually applied your patch to include/asm-generic/xenomai/hal.h in the kernel source tree, recompiled the kernel, installed etc, and rebooted. However even with the patch in place, whenever I ran:

dd if=/dev/zero of=/dev/null count=20000000

then initially top showed that dd was getting very little CPU time (<10%), then after 30 seconds or so, the system became completely unresponsive until the dd ended. This is how it acts without the patch as well. So it doesn't appear that the patch has made any difference to this problem.

Note that I applied the patch to Xenomai 2.5-rc1 in linux kernel 2.6.29.1. If the patch somehow relies on head, tell me, and I'll endeavor to set up a new kernel using that.

Martin

author
#19 lore raw

Re: [Xenomai-core] [PATCH] Fix host IRQ propagation

Quoted 11 lines · Martin Shepherd
Quoted 2 lines · Jan Kiszka

...
Martin, could you check if this helps you, too?

It doesn't appear to help. To check, first I turned on the HPET and PM timer options, and recompiled the kernel without your patch, to verify that this reproduced the problem. I then manually applied your patch to include/asm-generic/xenomai/hal.h in the kernel source tree, recompiled the kernel, installed etc, and rebooted. However even with the patch in place, whenever I ran:

Quoted 7 lines

dd if=/dev/zero of=/dev/null count=20000000

then initially top showed that dd was getting very little CPU time (<10%), then after 30 seconds or so, the system became completely unresponsive until the dd ended. This is how it acts without the patch as well. So it doesn't appear that the patch has made any difference to this problem.

Hmm, so there is no Xenomai task running during this test, just Linux load? Tried to reproduce with my kernel here (both HPET and PM enabled, as usual), but no success. Strange.

Quoted 3 lines

Note that I applied the patch to Xenomai 2.5-rc1 in linux kernel 2.6.29.1. If the patch somehow relies on head, tell me, and I'll endeavor to set up a new kernel using that.

The patch must cleanly apply as is on -rc1, any later changes did not affect its context.

Jan

Signature · 2 lines
Siemens AG, Corporate Technology, CT SE 2
Corporate Competence Center Embedded Linux
#20 lore raw

Re: [Xenomai-core] [PATCH] Fix host IRQ propagation

Quoted 19 lines · Jan Kiszka
Quoted 10 lines · Martin Shepherd
Quoted 2 lines · Jan Kiszka

...
Martin, could you check if this helps you, too?

It doesn't appear to help. To check, first I turned on the HPET and PM timer options, and recompiled the kernel without your patch, to verify that this reproduced the problem. I then manually applied your patch to include/asm-generic/xenomai/hal.h in the kernel source tree, recompiled the kernel, installed etc, and rebooted. However even with the patch in place, whenever I ran:

Quoted 7 lines

dd if=/dev/zero of=/dev/null count=20000000

then initially top showed that dd was getting very little CPU time (<10%), then after 30 seconds or so, the system became completely unresponsive until the dd ended. This is how it acts without the patch as well. So it doesn't appear that the patch has made any difference to this problem.

Quoted 3 lines

Hmm, so there is no Xenomai task running during this test, just Linux load? Tried to reproduce with my kernel here (both HPET and PM enabled, as usual), but no success. Strange.

Just having HPET and PM_TIMER enabled is not enough: your CPU needs to actually have an HPET timer or PM timer.

Signature · 1 line
                                                 Gilles.
author
#21 lore raw

Re: [Xenomai-core] [PATCH] Fix host IRQ propagation

Quoted 19 lines · Gilles Chanteperdrix
Quoted 18 lines · Jan Kiszka
Quoted 17 lines · Martin Shepherd
Quoted 2 lines · Jan Kiszka

...
Martin, could you check if this helps you, too?

It doesn't appear to help. To check, first I turned on the HPET and PM timer options, and recompiled the kernel without your patch, to verify that this reproduced the problem. I then manually applied your patch to include/asm-generic/xenomai/hal.h in the kernel source tree, recompiled the kernel, installed etc, and rebooted. However even with the patch in place, whenever I ran:

dd if=/dev/zero of=/dev/null count=20000000

then initially top showed that dd was getting very little CPU time (<10%), then after 30 seconds or so, the system became completely unresponsive until the dd ended. This is how it acts without the patch as well. So it doesn't appear that the patch has made any difference to this problem.

Quoted 6 lines
Quoted 3 lines

Hmm, so there is no Xenomai task running during this test, just Linux load? Tried to reproduce with my kernel here (both HPET and PM enabled, as usual), but no success. Strange.

Just having HPET and PM_TIMER enabled is not enough: your CPU needs to actually have an HPET timer or PM timer.

That's the case for practically any modern system. So I bet the "trick" is to make Linux _use_ one of them - in favor of the normally better suited LAPIC. Need to check again what the reasons for this switch could be. One that comes to my mind is problems of LAPIC vs. power management.

Jan

Signature · 2 lines
Siemens AG, Corporate Technology, CT SE 2
Corporate Competence Center Embedded Linux
#22 lore raw
In reply to Jan Kiszka
Quoted 16 lines · Jan Kiszka
Quoted 15 lines · Gilles Chanteperdrix
Quoted 14 lines · Jan Kiszka

Hi Gilles,

I'm currently facing a nasty effect with switchtest over latest git head (only tested this so far): running it inside my test VM (ie. with frequent excessive latencies) I get a stalled Linux timer IRQ quite quickly. System is otherwise still responsive, Xenomai timers are still being delivered, other Linux IRQs too. switchtest complained about

"Warning: Linux is compiled to use FPU in kernel-space."

when it was started. Kernels are 2.6.28.9/ipipe-x86-2.2-07 and 2.6.29.3/ipipe-x86-2.3-01 (LTTng patched in, but unused), both show the same effect.

Quoted 5 lines
Quoted 3 lines
Quoted 1 line

Seen this before?

The warning about Linux being compiled to use FPU in kernel-space means that you enabled soft RAID or compiled for K7, Geode, or any other

RAID is on (ordinary server config).

By the way, I wonder how MMX accelerated software raid works on K7, since the way I understand the code, calls to kernel_fpu_begin() can not be nested.

If you think they can be nested, then we can make switchtest test fpu in Linux kernel-space when these options are enabled.

Signature · 1 line
                                                 Gilles.
author
#23 lore raw
Quoted 20 lines · Gilles Chanteperdrix
Quoted 19 lines · Jan Kiszka
Quoted 18 lines · Gilles Chanteperdrix
Quoted 8 lines · Jan Kiszka

Hi Gilles,

I'm currently facing a nasty effect with switchtest over latest git head (only tested this so far): running it inside my test VM (ie. with frequent excessive latencies) I get a stalled Linux timer IRQ quite quickly. System is otherwise still responsive, Xenomai timers are still being delivered, other Linux IRQs too. switchtest complained about

Quoted 7 lines

"Warning: Linux is compiled to use FPU in kernel-space."

when it was started. Kernels are 2.6.28.9/ipipe-x86-2.2-07 and 2.6.29.3/ipipe-x86-2.3-01 (LTTng patched in, but unused), both show the same effect.

Seen this before?

The warning about Linux being compiled to use FPU in kernel-space means that you enabled soft RAID or compiled for K7, Geode, or any other

Quoted 8 lines
Quoted 1 line

RAID is on (ordinary server config).

By the way, I wonder how MMX accelerated software raid works on K7, since the way I understand the code, calls to kernel_fpu_begin() can not be nested.

If you think they can be nested, then we can make switchtest test fpu in Linux kernel-space when these options are enabled.

Sorry, I haven't looked that close into the in-kernel FPU handling so far. What users are nested in the standard kernel? Or is it RAID in itself?

Jan

Signature · 2 lines
Siemens AG, Corporate Technology, CT SE 2
Corporate Competence Center Embedded Linux
#24 lore raw
Quoted 26 lines · Jan Kiszka
Quoted 25 lines · Gilles Chanteperdrix
Quoted 12 lines · Jan Kiszka
Quoted 11 lines · Gilles Chanteperdrix
Quoted 10 lines · Jan Kiszka

Hi Gilles,

I'm currently facing a nasty effect with switchtest over latest git head (only tested this so far): running it inside my test VM (ie. with frequent excessive latencies) I get a stalled Linux timer IRQ quite quickly. System is otherwise still responsive, Xenomai timers are still being delivered, other Linux IRQs too. switchtest complained about

"Warning: Linux is compiled to use FPU in kernel-space."

Quoted 8 lines
Quoted 7 lines
Quoted 5 lines

when it was started. Kernels are 2.6.28.9/ipipe-x86-2.2-07 and 2.6.29.3/ipipe-x86-2.3-01 (LTTng patched in, but unused), both show the same effect.

Seen this before?

The warning about Linux being compiled to use FPU in kernel-space means that you enabled soft RAID or compiled for K7, Geode, or any other

RAID is on (ordinary server config).

By the way, I wonder how MMX accelerated software raid works on K7, since the way I understand the code, calls to kernel_fpu_begin() can not be nested.

Quoted 5 lines
Quoted 2 lines

If you think they can be nested, then we can make switchtest test fpu in Linux kernel-space when these options are enabled.

Sorry, I haven't looked that close into the in-kernel FPU handling so far. What users are nested in the standard kernel? Or is it RAID in itself?

RAID uses FPU. And on K7, basic things like clearing or copying user pages, or even large memcpy do use FPU too. So, I would expect that when enabling both, they could happen to be nested.

Signature · 1 line
					    Gilles.
author
#25 lore raw
Quoted 27 lines · Gilles Chanteperdrix
Quoted 26 lines · Jan Kiszka
Quoted 25 lines · Gilles Chanteperdrix
Quoted 12 lines · Jan Kiszka
Quoted 11 lines · Gilles Chanteperdrix
Quoted 10 lines · Jan Kiszka

Hi Gilles,

I'm currently facing a nasty effect with switchtest over latest git head (only tested this so far): running it inside my test VM (ie. with frequent excessive latencies) I get a stalled Linux timer IRQ quite quickly. System is otherwise still responsive, Xenomai timers are still being delivered, other Linux IRQs too. switchtest complained about

"Warning: Linux is compiled to use FPU in kernel-space."

Quoted 8 lines
Quoted 7 lines
Quoted 5 lines

when it was started. Kernels are 2.6.28.9/ipipe-x86-2.2-07 and 2.6.29.3/ipipe-x86-2.3-01 (LTTng patched in, but unused), both show the same effect.

Seen this before?

The warning about Linux being compiled to use FPU in kernel-space means that you enabled soft RAID or compiled for K7, Geode, or any other

RAID is on (ordinary server config).

By the way, I wonder how MMX accelerated software raid works on K7, since the way I understand the code, calls to kernel_fpu_begin() can not be nested.

Quoted 9 lines
Quoted 4 lines
Quoted 2 lines

If you think they can be nested, then we can make switchtest test fpu in Linux kernel-space when these options are enabled.

Sorry, I haven't looked that close into the in-kernel FPU handling so far. What users are nested in the standard kernel? Or is it RAID in itself?

RAID uses FPU. And on K7, basic things like clearing or copying user pages, or even large memcpy do use FPU too. So, I would expect that when enabling both, they could happen to be nested.

I think the trick against /this/ is preempt_disable/enable in kernel_fpu/begin/end. But that won't work for Xenomai, of course.

Jan

Signature · 2 lines
Siemens AG, Corporate Technology, CT SE 2
Corporate Competence Center Embedded Linux
#26 lore raw
Quoted 2 lines · Jan Kiszka

I think the trick against /this/ is preempt_disable/enable in kernel_fpu/begin/end. But that won't work for Xenomai, of course.

Well, that does not prevent an IRQ or a page fault from computing a RAID cheksum...

Signature · 1 line
                                                 Gilles.
author
#27 lore raw
Quoted 6 lines · Gilles Chanteperdrix
Quoted 2 lines · Jan Kiszka

I think the trick against /this/ is preempt_disable/enable in kernel_fpu/begin/end. But that won't work for Xenomai, of course.

Well, that does not prevent an IRQ or a page fault from computing a RAID cheksum...

If this is allowed, then... My impression was that FPU access is not legal from IRQ context.

Jan

Signature · 2 lines
Siemens AG, Corporate Technology, CT SE 2
Corporate Competence Center Embedded Linux
#28 lore raw
Quoted 9 lines · Jan Kiszka
Quoted 5 lines · Gilles Chanteperdrix
Quoted 2 lines · Jan Kiszka

I think the trick against /this/ is preempt_disable/enable in kernel_fpu/begin/end. But that won't work for Xenomai, of course.

Well, that does not prevent an IRQ or a page fault from computing a RAID cheksum...

If this is allowed, then... My impression was that FPU access is not legal from IRQ context.

If you have a mapped file on a raid disk and a page fault triggers a write, I think the RAID routines will be used. But you are right, maybe there is a test to use the plain integer RAID routines when called from interrupt context.

Signature · 1 line
                                                 Gilles.