scheduler: should_we_balance() kills filesystem performance
Hi folks,
I just updated my performance test VM to the current 3.12-git tree after the XFS dev branch was merged. The first test I ran which was a 16-way concurrent fsmark test to create lots of files gave me a number about 30% lower than I expected - ~180k files/s when I was expecting somewhere around 250k files/s.
I did a bisect, and the bisect landed on this commit:
commit 23f0d2093c789e612185180c468fa09063834e87
Author: Joonsoo Kim <[email protected]> Date: Tue Aug 6 17:36:42 2013 +0900
sched: Factor out code to should_we_balance()
Now checking whether this cpu is appropriate to balance or not is embedded into update_sg_lb_stats() and this checking has no direct relationship to this function. There is not enough reason to place this checking at update_sg_lb_stats(), except saving one iteration for sched_group_cpus.
....
Now, i couldn't revert that patch by itself, but I reverted the series of about 10 scheduler patches in that series total from a current TOT and the regression went away. Hence I'm pretty confident that the this is the patch causing the issue as i've verified it in more than one way and the difference between "good" and "bad" was signficantlt greater than the variance of the test (1.5-2 stddev difference).
In more detail:
v4 filesystem v5 filesystem
3.11+xfsdev: 220k files/s 225k files/s
3.12-git 180k files/s 185k files/s
3.12-git-revert 245k files/s 247k files/s
The test vm is a 16p/16GB RAM VM, with a sparse 100TB filesystem image sitting on a 4-way RAID0 SSD array formatted with XFS and the image file is accessed by virtio+direct IO. The fsmark command line is:
time ./fs_mark -D 10000 -S0 -n 100000 -s 0 -L 32 \
-d /mnt/scratch/0 -d /mnt/scratch/1 \
-d /mnt/scratch/2 -d /mnt/scratch/3 \
-d /mnt/scratch/4 -d /mnt/scratch/5 \
-d /mnt/scratch/6 -d /mnt/scratch/7 \
-d /mnt/scratch/8 -d /mnt/scratch/9 \
-d /mnt/scratch/10 -d /mnt/scratch/11 \
-d /mnt/scratch/12 -d /mnt/scratch/13 \
-d /mnt/scratch/14 -d /mnt/scratch/15 \
| tee >(stats --trim-outliers | tail -1 1>&2)The workload on XFS runs to almost being CPU bound - the effect of the above patch was that there was a lot of idle time left in the system. The workload consumed the same amount of user and system CPU, just instantaneous CPU usage was reduced by 20-30% and the elaspsed time was increased by 20-30%.
Cheers,
Dave.
Signature · 2 lines
Dave Chinner [email protected]
Quoted 54 lines · Dave Chinner
Hi folks,
I just updated my performance test VM to the current 3.12-git tree after the XFS dev branch was merged. The first test I ran which was a 16-way concurrent fsmark test to create lots of files gave me a number about 30% lower than I expected - ~180k files/s when I was expecting somewhere around 250k files/s.
I did a bisect, and the bisect landed on this commit:
commit
23f0d2093c789e612185180c468fa09063834e87Author: Joonsoo Kim <[email protected]> Date: Tue Aug 6 17:36:42 2013 +0900sched: Factor out code to
should_we_balance()Now checking whether this cpu is appropriate to balance or not is embedded into
update_sg_lb_stats()and this checking has no direct relationship to this function. There is not enough reason to place this checking atupdate_sg_lb_stats(), except saving one iteration forsched_group_cpus.
....Now, i couldn't revert that patch by itself, but I reverted the series of about 10 scheduler patches in that series total from a current TOT and the regression went away. Hence I'm pretty confident that the this is the patch causing the issue as i've verified it in more than one way and the difference between "good" and "bad" was signficantlt greater than the variance of the test (1.5-2 stddev difference).
In more detail:
v4 filesystem v5 filesystem
3.11+xfsdev: 220k files/s 225k files/s
3.12-git 180k files/s 185k files/s
3.12-git-revert 245k files/s 247k files/sThe test vm is a 16p/16GB RAM VM, with a sparse 100TB filesystem image sitting on a 4-way RAID0 SSD array formatted with XFS and the image file is accessed by virtio+direct IO. The fsmark command line is:
time ./fs_mark -D 10000 -S0 -n 100000 -s 0 -L 32 \ -d /mnt/scratch/0 -d /mnt/scratch/1 \ -d /mnt/scratch/2 -d /mnt/scratch/3 \ -d /mnt/scratch/4 -d /mnt/scratch/5 \ -d /mnt/scratch/6 -d /mnt/scratch/7 \ -d /mnt/scratch/8 -d /mnt/scratch/9 \ -d /mnt/scratch/10 -d /mnt/scratch/11 \ -d /mnt/scratch/12 -d /mnt/scratch/13 \ -d /mnt/scratch/14 -d /mnt/scratch/15 \ | tee >(stats --trim-outliers | tail -1 1>&2)
Quoted 5 lines
The workload on XFS runs to almost being CPU bound - the effect of the above patch was that there was a lot of idle time left in the system. The workload consumed the same amount of user and system CPU, just instantaneous CPU usage was reduced by 20-30% and the elaspsed time was increased by 20-30%.
Hello, Dave.
Now, I look again this patch and find one mistake.
If we find that we are appropriate cpu for balancing, should_we_balance() should return 1. But current code doesn't do so. This correspond with your observation that a lot of idle time left.
Could you re-test your benchmark with below?
Thanks.
------------------->8-------------------------
kernel/sched/fair.c +1 -1
@@ -5151,7 +5151,7 @@ static int should_we_balance(struct lb_env *env)* First idle cpu or the first cpu(busiest) in this sched group * is eligible for doing load balancing at this and above domains. */ -return balance_cpu != env->dst_cpu; +return balance_cpu == env->dst_cpu; }/*
Dave Chinner
Tested-byQuoted 16 lines · Joonsoo Kim
Quoted 15 lines · Dave Chinner
Hi folks,
I just updated my performance test VM to the current 3.12-git tree after the XFS dev branch was merged. The first test I ran which was a 16-way concurrent fsmark test to create lots of files gave me a number about 30% lower than I expected - ~180k files/s when I was expecting somewhere around 250k files/s.
I did a bisect, and the bisect landed on this commit:
commit
23f0d2093c789e612185180c468fa09063834e87Author: Joonsoo Kim <[email protected]> Date: Tue Aug 6 17:36:42 2013 +0900sched: Factor out code to
should_we_balance()
.....
Quoted 28 lines
Quoted 22 lines
v4 filesystem v5 filesystem
3.11+xfsdev: 220k files/s 225k files/s
3.12-git 180k files/s 185k files/s
3.12-git-revert 245k files/s 247k files/sThe test vm is a 16p/16GB RAM VM, with a sparse 100TB filesystem image sitting on a 4-way RAID0 SSD array formatted with XFS and the image file is accessed by virtio+direct IO. The fsmark command line is:
time ./fs_mark -D 10000 -S0 -n 100000 -s 0 -L 32 \ -d /mnt/scratch/0 -d /mnt/scratch/1 \ -d /mnt/scratch/2 -d /mnt/scratch/3 \ -d /mnt/scratch/4 -d /mnt/scratch/5 \ -d /mnt/scratch/6 -d /mnt/scratch/7 \ -d /mnt/scratch/8 -d /mnt/scratch/9 \ -d /mnt/scratch/10 -d /mnt/scratch/11 \ -d /mnt/scratch/12 -d /mnt/scratch/13 \ -d /mnt/scratch/14 -d /mnt/scratch/15 \ | tee >(stats --trim-outliers | tail -1 1>&2)Quoted 5 lines
The workload on XFS runs to almost being CPU bound - the effect of the above patch was that there was a lot of idle time left in the system. The workload consumed the same amount of user and system CPU, just instantaneous CPU usage was reduced by 20-30% and the elaspsed time was increased by 20-30%.
Quoted 8 lines
Hello, Dave.
Now, I look again this patch and find one mistake.
If we find that we are appropriate cpu for balancing,should_we_balance()should return 1. But current code doesn't do so. This correspond with your observation that a lot of idle time left.Could you re-test your benchmark with below?
Sure. It looks like your patch fixes the problem:
v4 filesystem v5 filesystem
3.11+xfsdev: 220k files/s 225k files/s
3.12-git 180k files/s 185k files/s
3.12-git-revert 245k files/s 247k files/s
3.12-git-fix 249k files/s 248k files/s
Thanks for the quick turnaround :)
Cheers,
Dave.
Signature · 2 lines
Dave Chinner [email protected]
Joonsoo Kim
Tested-by Dave ChinnerQuoted 63 lines · Dave Chinner
Quoted 16 lines · Joonsoo Kim
Quoted 15 lines · Dave Chinner
Hi folks,
I just updated my performance test VM to the current 3.12-git tree after the XFS dev branch was merged. The first test I ran which was a 16-way concurrent fsmark test to create lots of files gave me a number about 30% lower than I expected - ~180k files/s when I was expecting somewhere around 250k files/s.
I did a bisect, and the bisect landed on this commit:
commit
23f0d2093c789e612185180c468fa09063834e87Author: Joonsoo Kim <[email protected]> Date: Tue Aug 6 17:36:42 2013 +0900sched: Factor out code to
should_we_balance().....
Quoted 28 lines
Quoted 22 lines
v4 filesystem v5 filesystem
3.11+xfsdev: 220k files/s 225k files/s
3.12-git 180k files/s 185k files/s
3.12-git-revert 245k files/s 247k files/sThe test vm is a 16p/16GB RAM VM, with a sparse 100TB filesystem image sitting on a 4-way RAID0 SSD array formatted with XFS and the image file is accessed by virtio+direct IO. The fsmark command line is:
time ./fs_mark -D 10000 -S0 -n 100000 -s 0 -L 32 \ -d /mnt/scratch/0 -d /mnt/scratch/1 \ -d /mnt/scratch/2 -d /mnt/scratch/3 \ -d /mnt/scratch/4 -d /mnt/scratch/5 \ -d /mnt/scratch/6 -d /mnt/scratch/7 \ -d /mnt/scratch/8 -d /mnt/scratch/9 \ -d /mnt/scratch/10 -d /mnt/scratch/11 \ -d /mnt/scratch/12 -d /mnt/scratch/13 \ -d /mnt/scratch/14 -d /mnt/scratch/15 \ | tee >(stats --trim-outliers | tail -1 1>&2)Quoted 5 lines
The workload on XFS runs to almost being CPU bound - the effect of the above patch was that there was a lot of idle time left in the system. The workload consumed the same amount of user and system CPU, just instantaneous CPU usage was reduced by 20-30% and the elaspsed time was increased by 20-30%.
Quoted 8 lines
Hello, Dave.
Now, I look again this patch and find one mistake.
If we find that we are appropriate cpu for balancing,should_we_balance()should return 1. But current code doesn't do so. This correspond with your observation that a lot of idle time left.Could you re-test your benchmark with below?
Sure. It looks like your patch fixes the problem:
v4 filesystem v5 filesystem
3.11+xfsdev: 220k files/s 225k files/s
3.12-git 180k files/s 185k files/s
3.12-git-revert 245k files/s 247k files/s
3.12-git-fix 249k files/s 248k files/s
Quoted 4 lines
Thanks for the quick turnaround :)
Tested-by Dave Chinner <[email protected]>
Thanks for the quick turnaround, too. :)
Hello, Ingo.
I attach the formatted patch with proper SOBs and commit message. Please merge this to fix above problem.
Thanks.
--------------------->8-------------------------
Quoted 1 line
From
cf8ca492c2206e72d91ca0a1f6c59c5436132c61Mon Sep 17 00:00:00 2001
From: Joonsoo Kim <[email protected]>
Date: Tue, 10 Sep 2013 15:28:10 +0900
Subject: [PATCH] sched: return 1 if this cpu is proper for balancing in
should_we_balance()
Commit 23f0d20('sched: Factor out code to should_we_balance()') introduces should_we_balance() function. This function should return 1 if this cpu is appropriate for balancing. But current code doesn't do so. When this happens, it returns 0, instead of 1.
This introduces performance regression which Dave Chinner reports.
v4 filesystem v5 filesystem 3.11+xfsdev: 220k files/s 225k files/s 3.12-git 180k files/s 185k files/s 3.12-git-revert 245k files/s 247k files/s
You can find more detailed information in below link.
https://lkml.org/lkml/2013/9/10/1
This patch corrects the return value of should_we_balance() function as orignally intended.
With this patch, Dave Chinner said that regression are gone.
v4 filesystem v5 filesystem 3.11+xfsdev: 220k files/s 225k files/s 3.12-git 180k files/s 185k files/s 3.12-git-revert 245k files/s 247k files/s 3.12-git-fix 249k files/s 248k files/s
kernel/sched/fair.c +1 -1
@@ -5151,7 +5151,7 @@ static int should_we_balance(struct lb_env *env)* First idle cpu or the first cpu(busiest) in this sched group * is eligible for doing load balancing at this and above domains. */ -return balance_cpu != env->dst_cpu; +return balance_cpu == env->dst_cpu; }/*
Signature · 1 line
1.7.9.5
tip-bot for Joonsoo Kim
merge bot[tip:sched/urgent] sched: Fix load balancing performance regression in should_we_balance()
sched: Fix load balancing performance regression in should_we_balance()
Commit-ID: b0cff9d88ce2f3030f73138078c5b1019f17e1cc Gitweb: http://git.kernel.org/tip/b0cff9d88ce2f3030f73138078c5b1019f17e1cc Author: Joonsoo Kim <[email protected]> AuthorDate: Tue, 10 Sep 2013 15:54:49 +0900 Committer: Ingo Molnar <[email protected]> CommitDate: Tue, 10 Sep 2013 09:20:42 +0200
sched: Fix load balancing performance regression in should_we_balance()
Commit 23f0d20 ("sched: Factor out code to should_we_balance()") introduces the should_we_balance() function. This function should return 1 if this cpu is appropriate for balancing. But the newly introduced code doesn't do so, it returns 0 instead of 1.
This introduces performance regression, reported by Dave Chinner:
v4 filesystem v5 filesystem 3.11+xfsdev: 220k files/s 225k files/s 3.12-git 180k files/s 185k files/s 3.12-git-revert 245k files/s 247k files/s
You can find more detailed information at:
https://lkml.org/lkml/2013/9/10/1
This patch corrects the return value of should_we_balance() function as orignally intended.
With this patch, Dave Chinner reports that the regression is gone:
v4 filesystem v5 filesystem 3.11+xfsdev: 220k files/s 225k files/s 3.12-git 180k files/s 185k files/s 3.12-git-revert 245k files/s 247k files/s 3.12-git-fix 249k files/s 248k files/s
--- kernel/sched/fair.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
kernel/sched/fair.c +1 -1
@@ -5151,7 +5151,7 @@ static int should_we_balance(struct lb_env *env)* First idle cpu or the first cpu(busiest) in this sched group * is eligible for doing load balancing at this and above domains. */ -return balance_cpu != env->dst_cpu; +return balance_cpu == env->dst_cpu; }/*
Peter Zijlstra
maintainerkernel/sched/fair.c static int should_we_balance(struct lb_env *env) · line 5155
Quoted 11 lines · Joonsoo Kim
kernel/sched/fair.c +1
-1@@ -5151,7 +5151,7 @@ static int should_we_balance(struct lb_env *env)* First idle cpu or the first cpu(busiest) in this sched group * is eligible for doing load balancing at this and above domains. */ -return balance_cpu != env->dst_cpu; +return balance_cpu == env->dst_cpu; }
Duh.. /me kicks himself for not seeing that.
Thanks!