Serial port latency
Is the computer otherwise idle?
I've seen one unexplainable report with atm problems that disappeared (!) if a kernel compile was running.
Could you try to run a simple cpu hog (with nice 20)?
<<main()
{
for(;;) getpid();
}<<
I'm aware of one bug that could cause a delay of up to 20 ms (cpu_idle() doesn't check for pending softirq's before sleeping), but that doesn't explain your 500 ms delay.
--
Manfred