Large thread wakeup (scheduling) delay spikes
I posted something about this back in October, but received little response. Maybe others have run into problems with this since then.
I've noticed much less deterministic and more widely varying thread wakeup (scheduling) delays on recent kernels. Even with isolated processors, the maximum delay to wakeup has gotten much longer (configured with or without CONFIG_PREEMPT).
The maximum delay to wakeup is now more than 10x longer than it was in 2.6.13.4 and previous kernels, and that's on isolated processors (as much as 300 usec on a 1GHz cpu), although nominal values remain largely unchanged. The latest version I've tested is 2.6.15-rc5.
Delving into this further I discovered that this is due to the execution time of file_free_rcu(), running from rcu_process_callbacks() in ksoftirqd. It appears that the modification that caused this was:
http://www.kernel.org/git/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commit;h=ab2af1f5005069321c5d130f09cce577b03f43ef
By simply making the following change things return to more consistent thread wakeup delays on isolated cpus, similiar to what we had on kernels previous to the above mentioned mod (I know this change is incorrect, it is just for test purposes):
fs/file_table.c
@@ -62,7 +62,7 @@
static inline void file_free(struct file *f)
{
- call_rcu(&f->f_rcuhead, file_free_rcu);
+ kmem_cache_free(filp_cachep, f);
}
I am wondering if there is some way we can return to consistently fast and predictable scheduling of threads to be woken? If not on the system in general, maybe at least on certain specified processors?
Dimitri
Quoted 7 lines · Dimitri Sivanich
I posted something about this back in October, but received little response. Maybe others have run into problems with this since then.
I've noticed much less deterministic and more widely varying thread wakeup (scheduling) delays on recent kernels. Even with isolated processors, the maximum delay to wakeup has gotten much longer (configured with or without
CONFIG_PREEMPT).
Interesting -- what workload are you running, and what mechanism are you using to check scheduling delays?
What happens when you run this workload on a -rt kernel?
Thanx, Paul
Quoted 31 lines
The maximum delay to wakeup is now more than 10x longer than it was in 2.6.13.4 and previous kernels, and that's on isolated processors (as much as 300 usec on a 1GHz cpu), although nominal values remain largely unchanged. The latest version I've tested is 2.6.15-rc5.
Delving into this further I discovered that this is due to the execution time of
file_free_rcu(), running fromrcu_process_callbacks()in ksoftirqd. It appears that the modification that caused this was:http://www.kernel.org/git/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commit;h=ab2af1f5005069321c5d130f09cce577b03f43efBy simply making the following change things return to more consistent thread wakeup delays on isolated cpus, similiar to what we had on kernels previous to the above mentioned mod (I know this change is incorrect, it is just for test purposes):
fs/file_table.c(quoted diff) +1
-1@@ -62,7 +62,7 @@ static inline void file_free(struct file *f) { - call_rcu(&f->f_rcuhead, file_free_rcu); + kmem_cache_free(filp_cachep, f); }I am wondering if there is some way we can return to consistently fast and predictable scheduling of threads to be woken? If not on the system in general, maybe at least on certain specified processors?
Dimitri
Paul,
Quoted 11 lines · Paul E. McKenney
Quoted 7 lines · Dimitri Sivanich
I posted something about this back in October, but received little response. Maybe others have run into problems with this since then.
I've noticed much less deterministic and more widely varying thread wakeup (scheduling) delays on recent kernels. Even with isolated processors, the maximum delay to wakeup has gotten much longer (configured with or without
CONFIG_PREEMPT).Interesting -- what workload are you running, and what mechanism are you using to check scheduling delays?
For this issue, it takes just a simple mechanism and little workload. Aside from the 2 processors involved in the test, the rest of the system is fairly idle. Basically, it looks like the following:
Thread pinned to processor A waits in the kernel on a mutex.
Thread pinned to processor B sets up some timer hardware to interrupt processor A at a specified instant in time.
Processor A gets the interrupt and records an RTC time stamp, then trips the mutex to wake the sleeping thread on this same cpu.
Thread on A wakes up and records another RTC time stamp.
Thread on A opens a results file. Should the difference between the timestamps be > some threshold, data is written out to the file. The file is then closed.
Thread on A waits again on the mutex.
If I look at the stack for processor A at the threshold time, it's always somewhere in file_free_rcu. Nominal values for this test are well below threshold (even now).
And again, setting either CONFIG_PREEMPT or the equivalent of CONFIG_PREEMPT_VOLUNTARY (setting might_resched()=cond_resched()) makes no difference.
Quoted 2 lines
What happens when you run this workload on a -rt kernel?
I haven't tried it.
Thanks.
Dimitri
Quoted 34 lines
Thanx, Paul
Quoted 31 lines
The maximum delay to wakeup is now more than 10x longer than it was in 2.6.13.4 and previous kernels, and that's on isolated processors (as much as 300 usec on a 1GHz cpu), although nominal values remain largely unchanged. The latest version I've tested is 2.6.15-rc5.
Delving into this further I discovered that this is due to the execution time of
file_free_rcu(), running fromrcu_process_callbacks()in ksoftirqd. It appears that the modification that caused this was:http://www.kernel.org/git/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commit;h=ab2af1f5005069321c5d130f09cce577b03f43efBy simply making the following change things return to more consistent thread wakeup delays on isolated cpus, similiar to what we had on kernels previous to the above mentioned mod (I know this change is incorrect, it is just for test purposes):
fs/file_table.c(quoted diff) +1
-1@@ -62,7 +62,7 @@ static inline void file_free(struct file *f) { - call_rcu(&f->f_rcuhead, file_free_rcu); + kmem_cache_free(filp_cachep, f); }I am wondering if there is some way we can return to consistently fast and predictable scheduling of threads to be woken? If not on the system in general, maybe at least on certain specified processors?
Dimitri