vdso_gettimeofday hogs all my CPU
Hi Peter,
I'm guessing that x86 vdso problems are in your area of expertise, if not can you point me at the right person to bug?
I just spun up my XFS performance tests on 3.15-rc1, the first of which runs a concurrent fsmark workload. fsmark uses gettimeofday to do per-operation timing and performance is horrendous. 3.14.0 on this workload does around 300,000 file creates per second. 3.15-rc1 does:
# ./fs_mark -D 10000 -S0 -n 100000 -s 0 -L 32 -d /mnt/scratch/0 -d /mnt/scratch/1 -d /mnt/scratch/2 -d /mnt/scratch/3 -d /mnt/scratch/4 -d /mnt/scratch/5 -d /mnt/scratch/6 -d /mnt/scratch/7 -d /mnt/scratch/8 -d /mnt/scratch/9 -d /mnt/scratch/10 -d /mnt/scratch/11 -d /mnt/scratch/12 -d /mnt/scratch/13 -d /mnt/scratch/14 -d /mnt/scratch/15
# Version 3.3, 16 thread(s) starting at Tue Apr 15 17:37:20 2014 # Sync method: NO SYNC: Test does not issue sync() or fsync() calls. # Directories: Time based hash between directories across 10000 subdirectories with 180 seconds per subdirectory. # File names: 40 bytes long, (16 initial bytes of time stamp with 24 random bytes at end of name) # Files info: size 0 bytes, written with an IO size of 16384 bytes per write # App overhead is time in microseconds spent in the test not doing file writing related system calls.
FSUse% Count Size Files/sec App Overhead
0 1600000 0 70950.5 155219877
0 3200000 0 71832.4 150138649
0 4800000 0 71195.5 150935842
0 6400000 0 71215.4 150906768
0 8000000 0 63707.7 166569461.....
about 70,000 files/sec, so it's way, way down on 3.14. The load has pegged all 16 CPUs in the VM, and perf top tells me:
- 63.11% [vdso] [.] __vdso_gettimeofday
- __vdso_gettimeofday
44.24% stop
28.18% do_run
13.98% write_file
13.60% setup_file_namethat it's the gettimeofday calls that are causing the excessive CPU load.
I haven't changed anything in userspace for the past couple of months, the only thing that has changed is the kernel. I haven't looked at this any further and have no idea what to do to debug it further, so let me know if you need more information....
Cheers,
Dave.
Signature · 2 lines
Dave Chinner [email protected]
Quoted 4 lines · Dave Chinner
Hi Peter,
I'm guessing that x86 vdso problems are in your area of expertise, if not can you point me at the right person to bug?
And you can ignore it. The VM wasn't running the kernel I thought it was - it was an old kernel from half way through last week. A 3.15-rc1 kernel, which I'm now testing, doesn't show the regression:
FSUse% Count Size Files/sec App Overhead
0 1600000 0 281779.0 9964735
0 3200000 0 280343.0 10174512
0 4800000 0 268615.2 11330160
0 6400000 0 264782.4 11072193.....
It shows all sorts of interesting new inode cache reclaim imbalances, but other than that the peak performance is back to normal...
Sorry for the noise.
Cheers,
Dave.
Signature · 2 lines
Dave Chinner [email protected]