how to measure scheduler latency on powerpc? realfeel doesn't work due to /dev/rtc issues

I'm trying to test the scheduler latency on a powerpc platform. It appears that a realfeel type of program won't work since you can't program /dev/rtc to generated interrupts on powerpc. Is there anything similar which could be done?

Thanks,

Chris

Chris Friesen                    | MailStop: 043/33/F10
Nortel Networks                  | work: (613) 765-0557
3500 Carling Avenue              | fax:  (613) 765-2986
Nepean, ON K2H 8E9 Canada        | email: [email protected]

Why would you want to use an interrupt? Just count jiffies in sched.c

-- wli

I don't know what he does mean for scheduler latency, but if it is the ctx switch one something like get_cycles() will be better instead of jiffies.

- Davide

True, if you're looking for performance tweaks and not pathologies (which I was) you'll need something that accurate.

-- wli

Does it have a printer port like the Intel machines? If so, set it up to generate interrupts on the 'event' pin (paper-out, etc.) and have the ISR parrot the status bits out the printer-port bits.

Start it up and put function generator on the event bit. measure the delay beteen that bin and the data data bit(s) with a 'scope. This tells you the whole story, the total time necessary for an ISR to actually do something.

Cheers,
Dick Johnson
Penguin : Linux version 2.4.20 on an i686 machine (797.90 BogoMips). Why is the government concerned about the lunatic fringe? Think about it.

I'm trying to get a feel for the maximum time from an interrupt coming in until the userspace handler gets notified. On intel you can program the hardware to generate interupts through /dev/rtc. The powerpc doesn't seem to support this.

Jiffies are not accurate enough, I am expecting max latencies in the 1-2 ms range.

Unfortunately no. USB/Firewire/Ethernet on the desktop, ethernet/serial for compactPCI.

I want to find an additional programmable interrupt source. It bites that cheap PCs have this, and the powerpc doesn't.

Chris

Chris Friesen                    | MailStop: 043/33/F10
Nortel Networks                  | work: (613) 765-0557
3500 Carling Avenue              | fax:  (613) 765-2986
Nepean, ON K2H 8E9 Canada        | email: [email protected]

Chris> I want to find an additional programmable interrupt source. Chris> It bites that cheap PCs have this, and the powerpc doesn't.

I don't know which PowerPC CPU you are using, but for example the IBM 4xx series (and all Book E processors) have a "fixed interval timer" interrupt that is currently not used at all by Linux. (The "programmable interval timer" is used to increment jiffies)

On something like the Motorola 74xx, you might be able to use the something like the performance monitor to generate an exception.

Best,
Roland

That's the chip. I'll look into that further--thanks for the suggestion.

Chris

Chris Friesen                    | MailStop: 043/33/F10
Nortel Networks                  | work: (613) 765-0557
3500 Carling Avenue              | fax:  (613) 765-2986
Nepean, ON K2H 8E9 Canada        | email: [email protected]

Try the timebase instead.

-- wli

The timestamp is not hard to get. The problem is getting a medium-frequency (2KHz or so) hardware interrupt to drive the test.

On intel, you can do this by programming /dev/rtc. This does not work in powerpc.

Chris

Chris Friesen                    | MailStop: 043/33/F10
Nortel Networks                  | work: (613) 765-0557
3500 Carling Avenue              | fax:  (613) 765-2986
Nepean, ON K2H 8E9 Canada        | email: [email protected]

I don't understand why you're obsessed with interrupts. Just run your load and spray the scheduler latency stats out /proc/

-- wli

I'm obsessed with interrupts because it gives me a higher sampling rate.

I could set up and itimer for a recurring 10ms timeout and see how much extra I waited, but then I can only get 100 samples/sec.

With /dev/rtc (on intel) you can get 20x more samples in the same amount of time.

Chris

Chris Friesen                    | MailStop: 043/33/F10
Nortel Networks                  | work: (613) 765-0557
3500 Carling Avenue              | fax:  (613) 765-2986
Nepean, ON K2H 8E9 Canada        | email: [email protected]

Why not just keep track of it in the scheduler? The statistic is well- defined in terms of things measurable at context switch and wakeup. Just stamping processes with when they became runnable and when they were first run with the timebase and/or TSC and/or whatever would seem to provide the answer you want for T(x). W(x) is slightly more involved but can be measured properly in the same way.

I think your stats will be more accurate and serve your own (not mine; I have _zero_ cause to fish this stuff out myself besides curiostiy) purposes better if measured in the way I'm suggesting. That said, only your own purposes really matter for this in the end so if I misunderstand you by all means generate all the interrupts you want.

-- wli

This would measure context switch latency. Or something.

By definition, scheduling latency is the time from an interrupt which wakes the task up until the task is actually running.

Historically, it has been measured by things like realfeel or amlat or whatever which generate interrupts and wake a waiting task up. You then measure the latency between the interrupt and when the task actually runs in user-space.

So Chris can then go run this test under varying loads and see how bad the latency gets. I understand his question, but (sorry Chris) I have no idea of the solution on PPC.

Robert Love

Not at all. Just stamp at wakeup and difference when it runs.

-- wli

Well, since time was short I didn't want to hack the kernel. What I ended up doing was setting up an itimer for 10ms intervals and then measuring how much longer than 10ms I was asleep for. While this only collects samples at 100Hz, it will work with the default kernel.

Thanks for the suggestions,

Chris

Chris Friesen                    | MailStop: 043/33/F10
Nortel Networks                  | work: (613) 765-0557
3500 Carling Avenue              | fax:  (613) 765-2986
Nepean, ON K2H 8E9 Canada        | email: [email protected]

Reload the decrementer with a smaller value, and read the decrementer value at the beginning of every timer_interrupt. Timekeeping only depends on the timebase on PPC (I wrote most of the code), taking additional decrementer interrupts does not harm. Actually the simplest and most efficient way to perform a self IPI on PPC is to write 0 to the decrementer (2 machine instructions).

Actually, a way to measure the longest time during which interrupts are masked would be:

- when disabling interrupts, set decrementer to zero if they were previously enabled.
- read the decrementer on every timer/external interrupt, if the value is negative and too large print the value and the point at which the interrupts were reenabled.

That's not perfect (it will generate not too many false positives but quite some overhead, but it should work).

Gabriel

On all Moto PPCs I've checked, the time-base runs at 1/4 the bus clock rather than at the core clock like we're used to on x86.

I believe many PPCs allow you to program one of the performance counters to count core clocks. Be advised though that many Moto PPCs in the 750-7410 range have an errata that prevents using performance counter interrupts, so you'd have to run the counter in non-interrupting mode.

This isn't available to user-space (yet), so you'd also have to hack the kernel to use this facility.

If you are using a legacy Mac, you can use the serial port. If you have a sound chip, you can use that one. All archs have a programmable timer, even if /dev/rtc doesn't suport it. Try to ask in the linuxppc-dev mailing list.

Bye.