2.6.33.7-rt29 PREEMPT_RT worse latency than PREEMPT_DESKTOP on AT91?

Hello,

I am fine tuning the configuration for an ARM system derived from AT91SAM9263-EK.

My goal is to minimize latency, and I am using "cyclictest" from rt-tools v0.78 for measuring it.

I get consistently better latency with PREEMPT_DESKTOP over what I get with PREEMPT_RT. This is an example for a very simple test run, which reflects the overall results I am getting:

### PREEMPT-RT, HRT, no NO_HZ, no RTC, no USE_SLOW_CLOCK

   root@at91sam9263cpc:~# time cyclictest -i 700000 -r -p 80 -l 33
   Clock resolution: 0.000000001 (s.ns)
   policy: fifo: loadavg: 0.11 0.10 0.04 1/49 884

T: 0 ( 861) P:80 I:700000 C: 33 Min: 370 Act: 627 Avg: 437 Max: 627
real 0m 23.51s
user 0m 0.38s
sys 0m 2.20s

### PREEMPT-DESKTOP (no RT), HRT, no NO_HZ, no USE_SLOW_CLOCK

   root@at91sam9263cpc:~# time cyclictest -i 700000 -r -p 80 -l 33
   Clock resolution: 0.000000001 (s.ns)
   policy: fifo: loadavg: 0.24 0.08 0.03 1/46 600

T: 0 ( 574) P:80 I:700000 C: 33 Min: 173 Act: 196 Avg: 222 Max: 378
real 0m 23.66s
user 0m 0.32s
sys 0m 1.29s

That is, I get much better min, avg, and max latency with "PREEMPT_DESKTOP". How can this be?

Anyway, the results seem worse than the <150us worst case) obtained with 2.6.24.7-rt27 by Remy Bohmer <https://rt.wiki.kernel.org/index.php/CONFIG_PREEMPT_RT_Patch#Platforms_Tested_and_in_Use_with_CONFIG_PREEMPT_RT>, though he was apparently/ not using 'Tickless and HRT' /because of CPU usage issues... But without those I can't get cyclictest to run properly!

My guess is that the RT patch adds code (checks, locks) to may critical sections that happens to be specially slow on ARM9...

Clues & comments welcome!
--Agustín.

[CG logo]

Agustín Ferrín Pozuelo

Embedded Systems Engineer

CG Power Systems Ireland Limited

Automation Systems Division

Herbert House, Harmony Row, Dublin 2, Ireland.

Phone: +353 1 4153700 Web: www.cgglobal.com <http://www.cgglobal.com>

Save the environment. Please print only if essential.



CG DISCLAIMER: This email contains confidential information. It is intended exclusively for the addressees. If you are not an addressee, you must not store, transmit or disclose its contents. Instead please notify the sender immediately; and permanently delete this e-mail from your computer systems. We have taken reasonable precautions to ensure that no viruses are present. However, you must check this email and the attachments, for viruses. We accept no liability whatsoever, for any detriment caused by any transmitted virus.

--
To unsubscribe from this list: send the line "unsubscribe linux-rt-users" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html

33 loops are not really giving you any useful information. Also run both tests with some background load and not on a fully idle system.

Thanks,

tglx

Sorry, my test setup is not very comprehensive at the moment. I have some overnight/overweekend results at hand with millions of loops:

    PREEMPT-DESKTOP:
    root@at91sam9263cpc:~# time cyclictest -i 70000 -p 80 -h 700 -r
    Clock resolution: 0.000000001 (s.ns)
    policy: fifo: loadavg: 0.07 0.09 0.02 1/38 8770
    T: 0 (  980) P:80 I:70000 C:1328241 Min:    135 Act:  218 Avg:  204 

Max: 720

    PREEMPT-RT:
    root@at91sam9263cpc:~# time cyclictest -i 70000 -p 80 -h 700 -r
    Clock resolution: 0.000000001 (s.ns)
    policy: fifo: loadavg: 0.97 1.12 1.18 4/51 31290
    T: 0 (  675) P:80 I:70000 C:3688312 Min:  244    Act:  358 Avg:  

395 Max: 732

Min and avg latencies much worse in PREEMPT-RT on otherwise same setup. Worst case similar though.

System is not loaded but many interrupts happen because it is running it interactively through ssh+ethernet. (Had to force ethernet to 10Mbit to avoid frequent buffer underruns due to sam9263 bug).

I am assuming context switching is more expensive on PREEMPT-RT under ARM9, where it seems already a bit expensive.

--Agustín.

[CG logo]

Agustín Ferrín Pozuelo

Embedded Systems Engineer

CG Power Systems Ireland Limited

Automation Systems Division

Herbert House, Harmony Row, Dublin 2, Ireland.

Phone: +353 1 4153700 Web: www.cgglobal.com <http://www.cgglobal.com>

Save the environment. Please print only if essential.



CG DISCLAIMER: This email contains confidential information. It is intended exclusively for the addressees. If you are not an addressee, you must not store, transmit or disclose its contents. Instead please notify the sender immediately; and permanently delete this e-mail from your computer systems. We have taken reasonable precautions to ensure that no viruses are present. However, you must check this email and the attachments, for viruses. We accept no liability whatsoever, for any detriment caused by any transmitted virus.

--
To unsubscribe from this list: send the line "unsubscribe linux-rt-users" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Oh, you are running cyclictest with a signal based timer and relative time. Any reason for this ?

What happens if you change the command line to

cyclictest -i 70000 -p 80 -n

No, the context switch is equally expensive, but signal delivery might be a bit more overhead on -RT

Thanks,

tglx

Not sure about the "signal based timer", is there an alternative to it? (clock_nanosleep?)

The relative time setting "-r" I put there to prevent issues with some specific configurations, specially when the interval was smaller than max latency, in which case cyclictest would freeze or give unusable results. That was happening often before using the workaround for the 9263 ethernet bug which induced long latencies (non RT at all).

Interesting, "-n" setting (use clock_nanosleep) brings down the min and avg latencies for 35~40 us. (~70 when using "-r") on the PREEMPT-DESKTOP configuration:

    root@at91sam9263cpc:~# cyclictest -i 70000 -p 80 -l 1000
    Clock resolution: 0.000000001 (s.ns)
    policy: fifo: loadavg: 0.14 0.13 0.06 1/36 954
    T: 0 (  876) P:80 I:70000 C:   1000 Min:    107 Act:  135 Avg:  164 
Max:     262

    root@at91sam9263cpc:~# cyclictest -i 70000 -p 80 -l 1000 -n
    Clock resolution: 0.000000001 (s.ns)
    policy: fifo: loadavg: 0.14 0.12 0.05 1/37 874
    T: 0 (  795) P:80 I:70000 C:   1000 Min:     72 Act:  127 Avg:  125 
Max:     275

    root@at91sam9263cpc:~# cyclictest -i 70000 -p 80 -l 1000 -r
    Clock resolution: 0.000000001 (s.ns)
    policy: fifo: loadavg: 0.04 0.10 0.06 1/36 1035
    T: 0 (  956) P:80 I:70000 C:   1000 Min:    145 Act:  205 Avg:  203 
Max:     348

    root@at91sam9263cpc:~# cyclictest -i 70000 -p 80 -l 1000 -r -n
    Clock resolution: 0.000000001 (s.ns)
    policy: fifo: loadavg: 0.01 0.07 0.05 2/37 1115
    T: 0 ( 1037) P:80 I:70000 C:   1000 Min:     87 Act:  135 Avg:  150 
Max:     439

(Longer runs giving similar results)
Note that"-r" seems to lower CPU usage, is loadavg 0.14 == 14% or 0.0014%?
Right now I don't have the PREEMPT-RT version available for this test, should I expect big improvement there?

Thanks for the clarification and tips!
Regards,
--Agustín.

[CG logo]

Agustín Ferrín Pozuelo

Embedded Systems Engineer

CG Power Systems Ireland Limited

Automation Systems Division

Herbert House, Harmony Row, Dublin 2, Ireland.

Phone: +353 1 4153700 Web: www.cgglobal.com <http://www.cgglobal.com>

Save the environment. Please print only if essential.



CG DISCLAIMER: This email contains confidential information. It is intended exclusively for the addressees. If you are not an addressee, you must not store, transmit or disclose its contents. Instead please notify the sender immediately; and permanently delete this e-mail from your computer systems. We have taken reasonable precautions to ensure that no viruses are present. However, you must check this email and the attachments, for viruses. We accept no liability whatsoever, for any detriment caused by any transmitted virus.

--
To unsubscribe from this list: send the line "unsubscribe linux-rt-users" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html