Skip to contentSkip to conversation

about interrupt latency

DISCUSSION Francesco Oppedisano 4 participants 7 messages Mar 8, 2005 – Mar 14, 2005

lore
Share & tools
lore
#0 lore raw

Hi,
i'm trying to estimate the interrupt latency (time between hardware interrrupt and the start of the ISR) of a linux kernel 2.4.29 and i used a simple tecnique: inside the do_timer_interrupt i read the 8259 counter to obtain the elapsed time.
By this mean i found a latency of about 6/7 microseconds that is very similar to the time measured in some articles but with CPU much slower while i expected the latency was shorter on faster CPUs. So, my questions are:
1)what's the depency between the interrupt latency and the CPU speed? 2)what are the factors at the origin of th interrupt latency?

Than u very much

Francesco Oppedisano

#1 lore raw
Quoted 12 lines · Francesco Oppedisano

Hi,
i'm trying to estimate the interrupt latency (time between hardware interrrupt and the start of the ISR) of a linux kernel 2.4.29 and i used a simple tecnique: inside the do_timer_interrupt i read the 8259 counter to obtain the elapsed time.
By this mean i found a latency of about 6/7 microseconds that is very similar to the time measured in some articles but with CPU much slower while i expected the latency was shorter on faster CPUs. So, my questions are:
1)what's the depency between the interrupt latency and the CPU speed? 2)what are the factors at the origin of th interrupt latency?

Quoted 3 lines

Than u very much

Francesco Oppedisano

You can't measure interrupt latency that way even though you may get the "correct" answer!

Make a simple module that uses IRQ7, the printer-port interrupt. Inside your ISR, you toggle one of the printer-port bits. Program the printer port to generate the interrupt when its control bit is triggered.

Now, connect a function generator to toggle the printer control bit. Also use this transition to trigger an oscilloscope while looking at its trace on one channel. Connect the other channel to the bit that's being toggled in your ISR.

Observe the time between the trigger-trace and the toggle-trace. That, minus the few nanoseconds necessary to execute your ISR code, is the interrupt latency when using that specific interrupt source. PCI/Bus devices have lower latencies.

The CPU speed seems to have little to do with interrupt latency now that we have fast CPUs. The limiting action is the memory speed (front-side bus). You can seldom count on having your ISR code inside the cache, so it needs to be fetched. It also takes more cache-flushes to switch from user-mode to a kernel stack, set up new segments, etc. That's the reason why you must MEASURE the latency if it is important. Guessing that an interrupt occurred when a timer went to zero, then measuring the residual in that same ISR will give you the wrong answers, altough in this case, it's probably close.

Cheers,
Dick Johnson
Penguin : Linux version 2.6.10 on an i686 machine (5537.79 BogoMips).
Notice : All mail here is now cached for review by Dictator Bush.
98.36% of all statistics are fiction.

#3 lore raw
In reply to Francesco Oppedisano

Hi Francesco,

Quoted 9 lines · Francesco Oppedisano

i'm trying to estimate the interrupt latency (time between hardware interrrupt and the start of the ISR) of a linux kernel 2.4.29 and i used a simple tecnique: inside the do_timer_interrupt i read the 8259 counter to obtain the elapsed time. By this mean i found a latency of about 6/7 microseconds that is very similar to the time measured in some articles but with CPU much slower while i expected the latency was shorter on faster CPUs. So, my questions are: 1)what's the depency between the interrupt latency and the CPU speed? 2)what are the factors at the origin of th interrupt latency?

At some cpu frequency point on i386 the main cause of your interrupt service latency will be in the interrupt controller and how long from irq assertion to the signal being recognised, resultant vector being dispatched to the processor and the necessary interrupt controller acknowledge steps required. This is also helped by the fact that the Linux/i386 interrupt vector stubs are very small and fast, so there isn't all that much code to execute to reach the ISR from the vector table. I'm not sure if you've tested this, but you may notice that timer interrupt via Local APIC will have lower dispatch latency than timer interrupt via i8259 only. But that's all at the lower end of the latency graph, you will most likely run into other sources on a busy system.

Zwane

#2 lore raw
In reply to linux-os

On Tue, 8 Mar 2005 14:03:21 -0500 (EST), linux-os

Quoted 2 lines

You can't measure interrupt latency that way even though you may get the "correct" answer!

Why do you think the technique is not valid?

Quoted 12 lines

Make a simple module that uses IRQ7, the printer-port interrupt. Inside your ISR, you toggle one of the printer-port bits. Program the printer port to generate the interrupt when its control bit is triggered.

Now, connect a function generator to toggle the printer control bit. Also use this transition to trigger an oscilloscope while looking at its trace on one channel. Connect the other channel to the bit that's being toggled in your ISR.

Quoted 5 lines

Observe the time between the trigger-trace and the toggle-trace. That, minus the few nanoseconds necessary to execute your ISR code, is the interrupt latency when using that specific interrupt source. PCI/Bus devices have lower latencies.

I do not have such an instrumentation :-(

Quoted 14 lines

The CPU speed seems to have little to do with interrupt latency now that we have fast CPUs. The limiting action is the memory speed (front-side bus). You can seldom count on having your ISR code inside the cache, so it needs to be fetched. It also takes more cache-flushes to switch from user-mode to a kernel stack, set up new segments, etc. That's the reason why you must MEASURE the latency if it is important. Guessing that an interrupt occurred when a timer went to zero, then measuring the residual in that same ISR will give you the wrong answers, altough in this case, it's probably close.

Sorry but i cannot understand this:
i'm trying to estimate interrupt latency (i need only the order of magnitude) by measuring the interrupt latency of the timer interrupt. So when the 8254 counter reaches zero it issues an interrupt so i'm sure that the time when the interrupt has been issued was when the counter was to zero. From that time the counter returns to a value of 19832 (or something else) and at that time the countdown restarts. When i reach the ISR i read the counter and obtain the latency...what's wrong in this simple strategy?
Probably i'm guessing that the latency experienced by a timer interrupt is the same as e.g the latency experienced by a NIC interrupt, but remember that i need only a coarse measure.

Nevertheless, the information about the ISR in cache is very interesting, probably it's what i was looking for....thank u.

Francesco Oppedisano

#4 lore raw
In reply to Zwane Mwaikambo
Quoted 12 lines · Zwane Mwaikambo

At some cpu frequency point on i386 the main cause of your interrupt service latency will be in the interrupt controller and how long from irq assertion to the signal being recognised, resultant vector being dispatched to the processor and the necessary interrupt controller acknowledge steps required. This is also helped by the fact that the Linux/i386 interrupt vector stubs are very small and fast, so there isn't all that much code to execute to reach the ISR from the vector table. I'm not sure if you've tested this, but you may notice that timer interrupt via Local APIC will have lower dispatch latency than timer interrupt via i8259 only. But that's all at the lower end of the latency graph, you will most likely run into other sources on a busy system.

Quoted 2 lines

Zwane

Very interesting zwane....i haven't tested the local APIC....do you think this dispatch time can vary with the system I/O load (many pending interrupts in the PIC)?

I think the interrupt latency is influenced even by the code inside the kernel: if a lot of code is running with interrupts disabled then the interrupt latency will grow. Am i right?

So probably we can state that the factors influencing the interrupt latency are:
1)Dispatching time in the PIC
2)Waiting time on the PIC (if there are pending interrupt of lower vector) 3)fetching ISR from main memory
4)wait time when CPU is running with disabled interrupt

Do U agree?

Thank u very much
Francesco Oppedisano

#5 lore raw
Quoted 15 lines · Francesco Oppedisano
Quoted 12 lines · Zwane Mwaikambo

At some cpu frequency point on i386 the main cause of your interrupt service latency will be in the interrupt controller and how long from irq assertion to the signal being recognised, resultant vector being dispatched to the processor and the necessary interrupt controller acknowledge steps required. This is also helped by the fact that the Linux/i386 interrupt vector stubs are very small and fast, so there isn't all that much code to execute to reach the ISR from the vector table. I'm not sure if you've tested this, but you may notice that timer interrupt via Local APIC will have lower dispatch latency than timer interrupt via i8259 only. But that's all at the lower end of the latency graph, you will most likely run into other sources on a busy system.

Quoted 5 lines
Quoted 2 lines

Zwane

Very interesting zwane....i haven't tested the local APIC....do you think this dispatch time can vary with the system I/O load (many pending interrupts in the PIC)?

The PIC/i386 setup cannot really pend interrupts so i would say no i don't think the dispatch time would be affected.

Quoted 3 lines

I think the interrupt latency is influenced even by the code inside the kernel: if a lot of code is running with interrupts disabled then the interrupt latency will grow. Am i right?

Yes that's correct, in fact this will be your major/main cause of interrupt latency.

Quoted 3 lines

So probably we can state that the factors influencing the interrupt latency are:
1)Dispatching time in the PIC
2)Waiting time on the PIC (if there are pending interrupt of lower vector)

With APIC/i386 this is more possible, if you're really interested you should try and calculate the dispatch time using the same system (must have an IOAPIC) and testing with the following combinations;

PIC only (noapic, nolapic)
PIC + LAPIC (noapic)
IOAPIC + LAPIC

You will probably find that the IOAPIC + LAPIC has the lowest dispatch time. Also worth noting that the LAPIC can queue upto 2 vectors (on P4), this allows for more interrupts to be ready for dispatch.

Quoted 4 lines

3)fetching ISR from main memory
4)wait time when CPU is running with disabled interrupt

Do U agree?

A bit more on (4)

4a) wait time when CPU is running firmware (e.g. SMI/SMM, VGA BIOSes etc)

Otherwise, yes i agree, good luck with your research.

Zwane

#6 lore raw
Quoted 58 lines · Zwane Mwaikambo
Quoted 15 lines · Francesco Oppedisano
Quoted 12 lines · Zwane Mwaikambo

At some cpu frequency point on i386 the main cause of your interrupt service latency will be in the interrupt controller and how long from irq assertion to the signal being recognised, resultant vector being dispatched to the processor and the necessary interrupt controller acknowledge steps required. This is also helped by the fact that the Linux/i386 interrupt vector stubs are very small and fast, so there isn't all that much code to execute to reach the ISR from the vector table. I'm not sure if you've tested this, but you may notice that timer interrupt via Local APIC will have lower dispatch latency than timer interrupt via i8259 only. But that's all at the lower end of the latency graph, you will most likely run into other sources on a busy system.

Quoted 5 lines
Quoted 2 lines

Zwane

Very interesting zwane....i haven't tested the local APIC....do you think this dispatch time can vary with the system I/O load (many pending interrupts in the PIC)?

The PIC/i386 setup cannot really pend interrupts so i would say no i don't think the dispatch time would be affected.

Quoted 3 lines

I think the interrupt latency is influenced even by the code inside the kernel: if a lot of code is running with interrupts disabled then the interrupt latency will grow. Am i right?

Yes that's correct, in fact this will be your major/main cause of interrupt latency.

Quoted 3 lines

So probably we can state that the factors influencing the interrupt latency are:
1)Dispatching time in the PIC
2)Waiting time on the PIC (if there are pending interrupt of lower vector)

With APIC/i386 this is more possible, if you're really interested you should try and calculate the dispatch time using the same system (must have an IOAPIC) and testing with the following combinations;

PIC only (noapic, nolapic)
PIC + LAPIC (noapic)
IOAPIC + LAPIC

You will probably find that the IOAPIC + LAPIC has the lowest dispatch time. Also worth noting that the LAPIC can queue upto 2 vectors (on P4), this allows for more interrupts to be ready for dispatch.

Quoted 4 lines

3)fetching ISR from main memory
4)wait time when CPU is running with disabled interrupt

Do U agree?

A bit more on (4)

4a) wait time when CPU is running firmware (e.g. SMI/SMM, VGA BIOSes etc)

Quoted 9 lines

Otherwise, yes i agree, good luck with your research.

Zwane
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

I am just a user, not sure if it is related.
But with CONFIG_X86_UP_APIC and/or CONFIG_X86_UP_IOAPIC set my mouse behaviour is totally different, when playing games everything feels more "laggy".
And it gives me what gamers call "negative acceleration".