sched_yield() makes OpenLDAP slow
Hello,
I've been investigating a performance problem on a server using OpenLDAP 2.2.26 for nss resolution and running kernel 2.6.12.
When a CPU bound process such as GCC is running in the background (even at nice 10), many trivial commands such as "su" or "groups" become extremely slow and take a few seconds to complete.
strace revealed that data exchange over the slapd socket was where most of the time was spent. Looking at the slapd side, I see several calls to sched_yield() like this:
[pid 8780] 0.000033 stat64("gidNumber.dbb", 0xb7b3ebcc) = -1 EACCES (Permission denied)
[pid 8780] 0.000059 pread(20, "\0\0\0\0\1\0\0\0\1\0\0\0\0\0\0\0\0\0\0\0\2\0\344\17\2\3"..., 4096, 4096) = 4096
[pid 8780] 0.000083 pread(20, "\0\0\0\0\1\0\0\0\4\0\0\0\3\0\0\0\0\0\0\0\222\0<\7\1\5\370"..., 4096, 16384) = 4096
[pid 8780] 0.000078 time(NULL) = 1124322520
[pid 8780] 0.000066 pread(11, "\0\0\0\0\1\0\0\0\250\0\0\0\231\0\0\0\235\0\0\0\16\0000"..., 4096, 688128) = 4096
[pid 8780] 0.000241 write(19, "0e\2\1\3d`\4$cn=bernie,ou=group,dc=d"..., 103) = 103
[pid 8780] 0.000137 sched_yield( <unfinished ...>
[pid 8781] 0.050020 <... sched_yield resumed> ) = 0
[pid 8780] 0.000025 <... sched_yield resumed> ) = 0
[pid 8781] 0.000060 futex(0x925ab20, FUTEX_WAIT, 33, NULL <unfinished ...>
[pid 8780] 0.000026 write(19, "0\f\2\1\3e\7\n\1\0\4\0\4\0", 14) = 14
[pid 8774] 0.000774 <... select resumed> ) = 1 (in [19])The relative timestamp reveals that slapd is spending 50ms after yielding. Meanwhile, GCC is probably being scheduled for a whole quantum.
Reading the man-page of sched_yield() it seems this isn't the correct behavior:
Note: If the current process is the only process in the highest priority list at that time, this process will continue to run after a call to sched_yield.
I also think OpenLDAP is wrong. First, it should be calling pthread_yield() because slapd is a multithreading process and it just wants to run the other threads. See:
intldap_pvt_thread_yield( void )
{
#if HAVE_THR_YIELD
return thr_yield();#elif HAVE_PTHREADS == 10
return sched_yield();
#elif defined(_POSIX_THREAD_IS_GNU_PTH)
sched_yield();
return 0;#elif HAVE_PTHREADS == 6
pthread_yield(NULL);
return 0;
#else
pthread_yield();
return 0;#endif
}
Signature · 2 lines
// Bernardo Innocenti - Develer S.r.l., R&D dept. \X/ http://www.develer.com/
Quoted 41 lines · Bernardo Innocenti
Hello,
I've been investigating a performance problem on a server using OpenLDAP 2.2.26 for nss resolution and running kernel 2.6.12.
When a CPU bound process such as GCC is running in the background (even at nice 10), many trivial commands such as "su" or "groups" become extremely slow and take a few seconds to complete.
strace revealed that data exchange over the slapd socket was where most of the time was spent. Looking at the slapd side, I see several calls to
sched_yield()like this:[pid 8780] 0.000033
stat64("gidNumber.dbb", 0xb7b3ebcc)= -1 EACCES
(Permission denied) [pid 8780] 0.000059 pread(20,
"\0\0\0\0\1\0\0\0\1\0\0\0\0\0\0\0\0\0\0\0\2\0\344\17\2\3"..., 4096, 4096) =
4096 [pid 8780] 0.000083 pread(20,
"\0\0\0\0\1\0\0\0\4\0\0\0\3\0\0\0\0\0\0\0\222\0<\7\1\5\370"..., 4096,
16384) = 4096 [pid 8780] 0.000078 time(NULL) = 1124322520
[pid 8780] 0.000066 pread(11,
"\0\0\0\0\1\0\0\0\250\0\0\0\231\0\0\0\235\0\0\0\16\0000"..., 4096, 688128)
= 4096 [pid 8780] 0.000241 write(19,
"0e\2\1\3d`\4$cn=bernie,ou=group,dc=d"..., 103) = 103 [pid 8780] 0.000137 sched_yield( <unfinished ...>[pid 8781] 0.050020 <... sched_yield resumed> ) = 0 [pid 8780] 0.000025 <... sched_yield resumed> ) = 0 [pid 8781] 0.000060 futex(0x925ab20, FUTEX_WAIT, 33, NULL <unfinished ...> [pid 8780] 0.000026 write(19, "0\f\2\1\3e\7\n\1\0\4\0\4\0", 14) = 14 [pid 8774] 0.000774 <... select resumed> ) = 1 (in [19])The relative timestamp reveals that slapd is spending 50ms after yielding. Meanwhile, GCC is probably being scheduled for a whole quantum.
Reading the man-page of
sched_yield()it seems this isn't the correct behavior:
Quoted 7 lines
Note: If the current process is the only process in the highest priority list at that time, this process will continue to run after a call to
sched_yield.I also think OpenLDAP is wrong. First, it should be calling
pthread_yield()because slapd is a multithreading process and it just wants to run the other threads. See:
sched_yield behaviour changed in 2.5 series more than 3 years ago and applications that use this as a locking primitive should be updated.
Cheers,
Con
Quoted 10 lines · Bernardo Innocenti
The relative timestamp reveals that slapd is spending 50ms after yielding. Meanwhile, GCC is probably being scheduled for a whole quantum.
Reading the man-page of
sched_yield()it seems this isn't the correct behavior:Note: If the current process is the only process in the highest priority list at that time, this process will continue to run after a call to
sched_yield.
The behavior of sched_yield changed for 2.6. I suppose the man
page didn't get updated.
Quoted 1 line
From
linux/Documentation/post-halloween.txt:
| - The behavior of sched_yield() changed a lot. A task that uses | this system call should now expect to sleep for possibly a very | long time. Tasks that do not really desire to give up the | processor for a while should probably not make heavy use of this | function. Unfortunately, some GUI programs (like Open Office) | do make excessive use of this call and under load their | performance is poor. It seems this new 2.6 behavior is optimal | but some user-space applications may need fixing.
This is pretty much all I know about it; I just thought I'd point
it out.
Quoted 3 lines
I also think OpenLDAP is wrong. First, it should be calling
pthread_yield()because slapd is a multithreading process and it just wants to run the other threads. See:
Is it possible that this problem has been noticed and fixed
already?
--
Joseph Fannin
[email protected]
/* So there I am, in the middle of my `netfilter-is-wonderful'
talk in Sydney, and someone asks What happens if you try to enlarge a 64k packet here?'. I think I said something eloquent like fuck'. - RR */
Quoted 2 lines · Joseph Fannin
The behavior of
sched_yieldchanged for 2.6. I suppose the man
page didn't get updated.
Now I remember reading about that on LWN or maybe KernelTraffic. Thanks!
Quoted 6 lines
Quoted 3 lines
I also think OpenLDAP is wrong. First, it should be calling
pthread_yield()because slapd is a multithreading process and it just wants to run the other threads. See:Is it possible that this problem has been noticed and fixed
already?
The OpenLDAP 2.3.5 source still looks like this.
I've filed a report in OpenLDAP's issue tracker:
http://www.openldap.org/its/index.cgi/Incoming?id=3950;page=2
Signature · 2 lines
// Bernardo Innocenti - Develer S.r.l., R&D dept. \X/ http://www.develer.com/
Quoted 10 lines · Joseph Fannin
Quoted 7 lines · Bernardo Innocenti
The relative timestamp reveals that slapd is spending 50ms after yielding. Meanwhile, GCC is probably being scheduled for a whole quantum.
Reading the man-page of
sched_yield()it seems this isn't the correct behavior:
Quoted 9 lines
Quoted 4 lines
Note: If the current process is the only process in the highest priority list at that time, this process will continue to run after a call to
sched_yield.The behavior of
sched_yieldchanged for 2.6. I suppose the man
page didn't get updated.
We class the SCHED_OTHER policy as having a single priority, which I believe is allowed (and even makes good sense, because dynamic and even nice priorities aren't really well defined).
That also makes our sched_yield() behaviour correct.
AFAIKS, sched_yield should only really be used by realtime applications that know exactly what they're doing.
Send instant messages to your online friends http://au.messenger.yahoo.com
Hello Con,
Quoted 2 lines · you
sched_yieldbehaviour changed in 2.5 series more than 3 years ago and applications that use this as a locking primitive should be updated.
I remember open office had a problem with excessive use of sched_yield() during 2.5. I guess they changed it but I have not checked. Does anyone know ?
Back then oo was having serious latency problems on 2.5
Regards,
Maciej
Quoted 8 lines · Nick Piggin
We class the
SCHED_OTHERpolicy as having a single priority, which I believe is allowed (and even makes good sense, because dynamic and even nice priorities aren't really well defined).That also makes our
sched_yield()behaviour correct.AFAIKS,
sched_yieldshould only really be used by realtime applications that know exactly what they're doing.
I'm pretty sure this has already been discussed in the past, but I fail to see why this new behavior of sched_yield() would be more correct.
In the OpenLDAP bug discussion, one of the developers considers this a Linux quirk needing a workaround, not a real bug in OpenLDAP.
As I understand it, the old behavior was to push the yielding process to the end of the queue for processes with the same niceness. This is somewhat closer to the (vague) definition in the POSIX man pages:
The sched_yield() function shall force the running thread to relinquish the processor until it again becomes the head of its thread list. It takes no arguments.
Pushing the process far behind in the queue, even after niced CPU crunchers, appears a bit extreme. It seems most programs expect sched_yield() to only reschedule the calling thread wrt its sibling threads, to be used to implement do-it-yourself spinlocks and the like.
Signature · 2 lines
// Bernardo Innocenti - Develer S.r.l., R&D dept. \X/ http://www.develer.com/