Skip to contentSkip to conversation

sched_yield() makes OpenLDAP slow

DISCUSSION Andi Kleen 3 participants 7 messages Aug 20, 2005 – Aug 22, 2005 3 messages not in the archive

lore
Share & tools
lore

Andi Kleen

author
#3 lore raw
In reply to a message not in the archive
Quoted 3 lines · Howard Chu

In this specific example, we use whatever
BerkeleyDB provides and we're certainly not about to write our own transactional embedded database engine just for this.

BerkeleyDB is free software after all that comes with source code. Surely it can be fixed without rewriting it from scratch.

Has anybody contacted the Sleepycat people with a description of the problem yet?

-Andi

#4 lore raw

* Andi Kleen:

Quoted 2 lines

Has anybody contacted the Sleepycat people with a description of the problem yet?

Berkeley DB does not call sched_yield, but OpenLDAP does in some wrapper code around the Berkeley DB backend.

#7 lore raw
In reply to Andi Kleen
Quoted 11 lines · Andi Kleen
Quoted 5 lines

processes (PTHREAD_SCOPE_SYSTEM). The previous comment about slapd only needing to yield within a single process is inaccurate; since we allow slapcat to run concurrently with slapd (to allow hot backups) we need BerkeleyDB's locking/yield functions to work in System scope.

That's broken by design - it means you can be arbitarily starved by other processes running in parallel. You are basically assuming your application is the only thing running on the system which is wrong. Also there are enough synchronization primitives that can synchronize multiple processes without making such broken assumptions.

Again, I think you overstate the problem. "Arbitrarily starved by other processes" implies that the process scheduler will do a poor job and will allow the slapd process to be starved. We do not assume we're the only app on the system, we just assume that eventually we will get the CPU back. If that's not a valid assumption, then there is something wrong with the underlying system environment.

Something you ought to keep in mind - correctness and compliance are well and good, but worthless if the end result isn't useful. Windows NT has a POSIX-compliant subsystem but it is utterly useless. That's what you wind up with when all you do is conform to the letter of the spec.

Signature · 4 lines
 -- Howard Chu
 Chief Architect, Symas Corp.  http://www.symas.com
 Director, Highland Sun        http://highlandsun.com/hyc
 OpenLDAP Core Team            http://www.openldap.org/project/
author
#6 lore raw
In reply to Howard Chu
Quoted 4 lines

processes (PTHREAD_SCOPE_SYSTEM). The previous comment about slapd only needing to yield within a single process is inaccurate; since we allow slapcat to run concurrently with slapd (to allow hot backups) we need BerkeleyDB's locking/yield functions to work in System scope.

That's broken by design - it means you can be arbitarily starved by other processes running in parallel. You are basically assuming your application is the only thing running on the system which is wrong. Also there are enough synchronization primitives that can synchronize multiple processes without making such broken assumptions.

-Andi

#8 lore raw
In reply to Howard Chu

* Howard Chu:

Quoted 6 lines
Quoted 5 lines
Quoted 2 lines

Has anybody contacted the Sleepycat people with a description of the problem yet?

Berkeley DB does not call sched_yield, but OpenLDAP does in some wrapper code around the Berkeley DB backend.

Quoted 6 lines

That's not the complete story. BerkeleyDB provides a db_env_set_func_yield() hook to tell it what yield function it should use when its internal locking routines need such a function. If you don't set a specific hook, it just uses sleep(). The OpenLDAP backend will invoke this hook during some (not necessarily all) init sequences, to tell it to use the thread yield function that we selected in autoconf.

And this helps to increase performance substantially?

Quoted 2 lines

Note that (on systems that support inter-process mutexes) a BerkeleyDB database environment may be used by multiple processes concurrently.

Yes, I know this, and I haven't experienced that much trouble with deadlocks. Maybe the way you structure and access the database environment can be optimized for deadlock avoidance?

Quoted 3 lines

As such, the yield function that is provided must work both for threads within a single process (PTHREAD_SCOPE_PROCESS) as well as between processes (PTHREAD_SCOPE_SYSTEM).

If I understand you correctly, what you really need is a syscall along the lines "don't run me again until all threads T that share property X have run, where the Ts aren't necessarily in the same process". The kernel is psychic, it can't really know which processes to schedule to satisfy such a requirement. I don't even think "has joined the Berkeley DB environment" is the desired property, but something like "is part of this cycle in the wait-for graph" or something similar.

I would have to check the Berkeley DB internals in order to tell what is feasible to implement. This code shouldn't be on the fast path, so some kernel-based synchronization is probably sufficient.

#5 lore raw
In reply to Florian Weimer
Quoted 8 lines · Florian Weimer

* Andi Kleen:

Quoted 2 lines

Has anybody contacted the Sleepycat people with a description of the problem yet?

Berkeley DB does not call sched_yield, but OpenLDAP does in some wrapper code around the Berkeley DB backend.

That's not the complete story. BerkeleyDB provides a db_env_set_func_yield() hook to tell it what yield function it should use when its internal locking routines need such a function. If you don't set a specific hook, it just uses sleep(). The OpenLDAP backend will invoke this hook during some (not necessarily all) init sequences, to tell it to use the thread yield function that we selected in autoconf.

Note that (on systems that support inter-process mutexes) a BerkeleyDB database environment may be used by multiple processes concurrently. As such, the yield function that is provided must work both for threads within a single process (PTHREAD_SCOPE_PROCESS) as well as between processes (PTHREAD_SCOPE_SYSTEM). The previous comment about slapd only needing to yield within a single process is inaccurate; since we allow slapcat to run concurrently with slapd (to allow hot backups) we need BerkeleyDB's locking/yield functions to work in System scope.

Signature · 4 lines
 -- Howard Chu
 Chief Architect, Symas Corp.  http://www.symas.com
 Director, Highland Sun        http://highlandsun.com/hyc
 OpenLDAP Core Team            http://www.openldap.org/project/
#9 lore raw
In reply to Florian Weimer
Quoted 10 lines · Florian Weimer

* Howard Chu:

Quoted 7 lines

That's not the complete story. BerkeleyDB provides a db_env_set_func_yield() hook to tell it what yield function it should use when its internal locking routines need such a function. If you don't set a specific hook, it just uses sleep(). The OpenLDAP backend will invoke this hook during some (not necessarily all) init sequences, to tell it to use the thread yield function that we selected in autoconf.

And this helps to increase performance substantially?

When the caller is a threaded program, yes, there is a substantial (measurable and noticable) difference. Given that sleep() blocks the entire process, the difference is obvious.

Quoted 7 lines
Quoted 3 lines

Note that (on systems that support inter-process mutexes) a BerkeleyDB database environment may be used by multiple processes concurrently.

Yes, I know this, and I haven't experienced that much trouble with deadlocks. Maybe the way you structure and access the database environment can be optimized for deadlock avoidance?

Maybe we already did this deadlock analysis and optimization, years ago when we first started developing this backend? Do you think everyone else in the world is a total fool?

Quoted 4 lines
Quoted 3 lines

As such, the yield function that is provided must work both for threads within a single process (PTHREAD_SCOPE_PROCESS) as well as between processes (PTHREAD_SCOPE_SYSTEM).

Quoted 8 lines

If I understand you correctly, what you really need is a syscall along the lines "don't run me again until all threads T that share property X have run, where the Ts aren't necessarily in the same process". The kernel is psychic, it can't really know which processes to schedule to satisfy such a requirement. I don't even think "has joined the Berkeley DB environment" is the desired property, but something like "is part of this cycle in the wait-for graph" or something similar.

You seem to believe we're looking for special treatment for the processes we're concerned with, and that's not true. If the system is busy with other processes, so be it, the system is busy. If you want better performance, you build a dedicated server and don't let anything else make the system busy. This is the way mission-critical services are delivered, regardless of the service. If you're not running on a dedicated system, then your deployment must not be mission critical, and so you shouldn't be surprised if a large gcc run slows down some other activities in the meantime. If you have a large nice'd job running before your normal priority jobs get their timeslice, then you should certainly wonder wtf the scheduler is doing, and why your system even claims to support nice() when clearly it doesn't mean anything on that system.

Quoted 3 lines

I would have to check the Berkeley DB internals in order to tell what is feasible to implement. This code shouldn't be on the fast path, so some kernel-based synchronization is probably sufficient.

pthread_cond_wait() probably would be just fine here, but BerkeleyDB doesn't work that way.

Signature · 4 lines
 -- Howard Chu
 Chief Architect, Symas Corp.  http://www.symas.com
 Director, Highland Sun        http://highlandsun.com/hyc
 OpenLDAP Core Team            http://www.openldap.org/project/