Strange thread behaviour on 8-way x86 machine
Hi,
I have observed a rather strange behaviour doing a multi-threaded CPU benchmark on an 8-way machine running 2.4.2 SMP kernel. Even when the priority is reniced to the highest possible value, I am still unable to reach more than 50% CPU utilization. My benchmark just creates a bunch of threads with pthread_create(), and then runs a simple integer computation in each thread. On a dual with 2.4.3 kernel, and a 4-way with 2.4.2 kernel, I am able to reach full CPU utilization.
At first glance, it appears to be like some overzealous fairness problem in the scheduler that is exhibited only when you have more than 4 CPUs. Before I start scrutinizing the source trying to understand the inner workings of the scheduler, I would like to get some feedback from people that know something about the subject. Any ideas/suggestions would be appreciated.
Signature · 7 lines
MySQL Development Team For technical support contracts, visit https://order.mysql.com/ __ ___ ___ ____ __ / |/ /_ __/ __/ __ \/ / Sasha Pachev <[email protected]> / /|_/ / // /\ \/ /_/ / /__ MySQL AB, http://www.mysql.com/ /_/ /_/\_, /___/\___\_\___/ Provo, Utah, USA <___/
Quoted 9 lines · Sasha Pachev
Hi,
I have observed a rather strange behaviour doing a multi-threaded CPU benchmark on an 8-way machine running 2.4.2 SMP kernel. Even when the priority is reniced to the highest possible value, I am still unable to reach more than 50% CPU utilization. My benchmark just creates a bunch of threads with
pthread_create(), and then runs a simple integer computation in each thread. On a dual with 2.4.3 kernel, and a 4-way with 2.4.2 kernel, I am able to reach full CPU utilization.
I haven't had any problem fully utilizing 8 CPUs on 2.4.* kernels. This may seem obvious, but do you have more than 4 CPUs worth of work for the system to do? What is the runqueue length during this benchmark?
Signature · 2 lines
Mike Kravetz [email protected] IBM Linux Technology Center
Quoted 6 lines · Mike Kravetz
Quoted 5 lines · Sasha Pachev
Hi,
I have observed a rather strange behaviour doing a multi-threaded CPU benchmark on an 8-way machine running 2.4.2 SMP kernel. Even when the priority is reniced to the highest possible value, I am still unable to
reach
Quoted 1 line
Quoted 1 line
more than 50% CPU utilization. My benchmark just creates a bunch of
threads
Quoted 2 lines
Quoted 2 lines
with
pthread_create(), and then runs a simple integer computation in each thread. On a dual with 2.4.3 kernel, and a 4-way with 2.4.2 kernel, I am
able
Quoted 5 lines
Quoted 1 line
to reach full CPU utilization.
I haven't had any problem fully utilizing 8 CPUs on 2.4.* kernels. This may seem obvious, but do you have more than 4 CPUs worth of work for the system to do? What is the runqueue length during this benchmark?
Upon further investigation and testing, it turned out that the kernel was not at fault - the problem was high mutex contention, which caused frequent context switches, and the idle CPU was apparently from the scheduler waiting for the original CPU to become available too often.
On a side note, it would be nice if a process could communicate to the kernel that it would rather run on the first available CPU than wait for the perfect one to become available.
Signature · 7 lines
MySQL Development Team For technical support contracts, visit https://order.mysql.com/ __ ___ ___ ____ __ / |/ /_ __/ __/ __ \/ / Sasha Pachev <[email protected]> / /|_/ / // /\ \/ /_/ / /__ MySQL AB, http://www.mysql.com/ /_/ /_/\_, /___/\___\_\___/ Provo, Utah, USA <___/
Quoted 8 lines · Sasha Pachev
Upon further investigation and testing, it turned out that the kernel was not at fault - the problem was high mutex contention, which caused frequent context switches, and the idle CPU was apparently from the scheduler waiting for the original CPU to become available too often.
On a side note, it would be nice if a process could communicate to the kernel that it would rather run on the first available CPU than wait for the perfect one to become available.
The kernel already does this.
Rik
--
Executive summary of a recent Microsoft press release:
"we are concerned about the GNU General Public License (GPL)"
http://www.surriel.com/
Quoted 3 lines · Rik van Riel
Quoted 1 line · Sasha Pachev
Upon further investigation and testing, it turned out that the kernel was
not
Quoted 2 lines
Quoted 2 lines
at fault - the problem was high mutex contention, which caused frequent context switches, and the idle CPU was apparently from the scheduler
waiting
Quoted 7 lines
Quoted 5 lines
for the original CPU to become available too often.
On a side note, it would be nice if a process could communicate to the kernel that it would rather run on the first available CPU than wait for the perfect one to become available.
The kernel already does this.
Thanks for the info. Would you mind proving a one line pointer on how to tell this to the kernel?
Signature · 7 lines
MySQL Development Team For technical support contracts, visit https://order.mysql.com/ __ ___ ___ ____ __ / |/ /_ __/ __/ __ \/ / Sasha Pachev <[email protected]> / /|_/ / // /\ \/ /_/ / /__ MySQL AB, http://www.mysql.com/ /_/ /_/\_, /___/\___\_\___/ Provo, Utah, USA <___/
Quoted 10 lines · Sasha Pachev
Quoted 3 lines · Rik van Riel
Quoted 1 line · Sasha Pachev
Upon further investigation and testing, it turned out that the kernel was
not
Quoted 2 lines
Quoted 2 lines
at fault - the problem was high mutex contention, which caused frequent context switches, and the idle CPU was apparently from the scheduler
waiting
Quoted 2 lines
Quoted 2 lines
for the original CPU to become available too often.
Quoted 8 lines
Quoted 5 lines
Quoted 3 lines
On a side note, it would be nice if a process could communicate to the kernel that it would rather run on the first available CPU than wait for the perfect one to become available.
The kernel already does this.
Thanks for the info. Would you mind proving a one line pointer on how to tell this to the kernel?
It always does this, by default. AFAIK you cannot turn it off.
Rik
--
Executive summary of a recent Microsoft press release:
"we are concerned about the GNU General Public License (GPL)"
http://www.surriel.com/