Skip to contentSkip to conversation

Clear performance regression on reaim7 in 2.6.15-git6

DISCUSSION Martin J. Bligh 4 participants 9 messages Apr 15, 2006

lore
Share & tools
lore
#0 lore raw

OK, looking back through the perf results, these two graphs clearly show
a perf regression in reaim7 from 2.6.15 to 2.6.16-rc1. We're loosing over 50% of the performance.

http://test.kernel.org/abat/perf/reaim.moe.png
http://test.kernel.org/abat/perf/reaim.elm3b67.png

Drilling down (there's not enough detail on the graphs for releases that far back), I see it's actually between -git5 and -git6

These are both ia-32 NUMA machines, one is an x440, the other is NUMA-Q.

#1 lore raw
Quoted 7 lines · Martin J. Bligh

OK, looking back through the perf results, these two graphs clearly show a perf regression in reaim7 from 2.6.15 to 2.6.16-rc1. We're loosing over 50% of the performance.

http://test.kernel.org/abat/perf/reaim.moe.png
http://test.kernel.org/abat/perf/reaim.elm3b67.png

Something very bad has allegedly been happening in -mm. It would have been nice to have heard about this problem in that multi-month wwindow there. Does no human look at these numbers?

Quoted 4 lines

Drilling down (there's not enough detail on the graphs for releases that far back), I see it's actually between -git5 and -git6

These are both ia-32 NUMA machines, one is an x440, the other is NUMA-Q.

The 2.6.15-git5 -> 2.6.15-git6 diff is enormous. I'd assume some bad thing got transferred into mainline then.

How does one reproduce this? Which type of filesystem, which command line/config file?

Thanks.

#2 lore raw
Quoted 10 lines · Andrew Morton
Quoted 5 lines · Martin J. Bligh

OK, looking back through the perf results, these two graphs clearly show
a perf regression in reaim7 from 2.6.15 to 2.6.16-rc1. We're loosing over 50% of the performance.

http://test.kernel.org/abat/perf/reaim.moe.png
http://test.kernel.org/abat/perf/reaim.elm3b67.png

Something very bad has allegedly been happening in -mm. It would have been
nice to have heard about this problem in that multi-month wwindow there. Does no human look at these numbers?

Obviously not ;-(
I do look every so often, but not frequently. I'm working on reformatting the site right now, which is why I was poking around. reaim I often don't look at, as it's not as stable as I'd like, but
that is a *huge* drop.

You can see I collated stuff down into a collection of tests per machine-release combo, so the dbench and LTP results are folded into the main matrix now. Next stage is to add performance regression comparions in there, but it's not easy unless we run multiple times so that I get std dev as well as average. All of this should make it easier to (a) see and (b) send out proactive notifications.

Quoted 11 lines
Quoted 4 lines

Drilling down (there's not enough detail on the graphs for releases that far back), I see it's actually between -git5 and -git6

These are both ia-32 NUMA machines, one is an x440, the other is NUMA-Q.

The 2.6.15-git5 -> 2.6.15-git6 diff is enormous. I'd assume some bad thing got transferred into mainline then.

How does one reproduce this? Which type of filesystem, which command line/config file?

Sigh ... this is exactly why I want the new autotest harness, I could just hand you the control file and you could run the same test (at your box or OSDL or whatever). Not finished yet though ;-(

drilling down into the results directories can get you the command line, looks like "reaim -f workfile.short -s 1 -e 10 -i 2" to me. Buggered if I can recall what that did though.

(http://test.kernel.org/abat/20229/004.reaim.test/results/cmdline)

I *think* it's only ia32 NUMA boxes, at least as far as I can see from
a quick poke around. Which would make me guess at scheduler code. Gitweb
would be nice to use, but it doesn't tag the -git snapshots, AFAICS, which is a real shame. Nor does the git snapshot include the git tag, as far as I know. Grrrr. I guess I'll download the snapshots and diff
them, and try to pull out the sched changes by hand. Much suckage.

M.

#3 lore raw
Quoted 5 lines · Martin J. Bligh

drilling down into the results directories can get you the command line,
looks like "reaim -f workfile.short -s 1 -e 10 -i 2" to me. Buggered if I can recall what that did though.

Quoted 8 lines

(http://test.kernel.org/abat/20229/004.reaim.test/results/cmdline)

I *think* it's only ia32 NUMA boxes, at least as far as I can see from a quick poke around. Which would make me guess at scheduler code. Gitweb would be nice to use, but it doesn't tag the -git snapshots, AFAICS, which is a real shame. Nor does the git snapshot include the git tag, as far as I know. Grrrr. I guess I'll download the snapshots and diff them, and try to pull out the sched changes by hand. Much suckage.

The diffstat for 2.6.15-git5 -> 2.6.15-git6 is at
http://www.zip.com.au/~akpm/linux/patches/stuff/2 - only a single line changed in sched.c:

kernel/sched.c +1 -0
diff -uNr 2.6.15-git5/kernel/sched.c 2.6.15-git6/kernel/sched.c
@@ -4386,6 +4386,7 @@
 	} while_each_thread(g, p);
 
 	read_unlock(&tasklist_lock);
+	mutex_debug_show_all_locks();
 }
 
 /**

The below patches went from -mm into mainline on that day. It'll be pretty simple to bisection search these once we have a testcase.

"tiny-make-id16-support-optional" fixes
nvidiafb: Fixes for new G5
printk levels for i386 oops code.
ipmi: fix compile errors with PROC_FS=n
Remove set_fs() in stop_machine()
spufs: fix for recent "shrink dentry_struct" patch
printk levels for spinlock debug
kdump: i386 save ss esp bug fix
kdump: dynamic per cpu allocation of memory for saving cpu registers
kdump: export per cpu crash notes pointer through sysfs (fix) dump_thread() cleanup
Kdump: i386 compiler warning fix
Add list_for_each_entry_safe_reverse()
fix /sys/class/net/<if>/wireless without dev->get_wireless_stats
kexec: increase max segment limit
Kdump documentation update
kdump: x86_64 kexec on panic
kdump: export per cpu crash notes pointer through sysfs
remove ext3 xattr permission checks
Kdump: powerpc and s390 build failure fix
kdump: save registers early (inline functions)
Disable rio on 64-bit platforms
kdump: x86_64: add memmmap command line option
remove ext2 xattr permission checks
kdump: x86_64: add elfcorehdr command line option
remove reiserfs xattr permission checks
kprobes: cleanup include/asm/kprobes.h
remove xfs xattr permission checks
move xattr permission checks into the VFS
remove jfs xattr permission checks
i386: GPIO driver for AMD CS5535/CS5536
per-mountpoint noatime/nodiratime
switch autofs4 to touch_atime()
->compat_ioctl for 390 tape_char
kdump: read previous kernel's memory
remove update_atime
hrtimer: coding style and white space cleanup
remove asm/serial.h from synclink_gt.
media-radio: Maestro types change
kdump: x86_64 save cpu registers upon crash
kprobes: arch_remove_kprobe
hrtimer: clean up mktime and make arguments const kprobes-changed-from-using-spinlock-to-mutex fix
hrtimer: remove unused clock constants
hrtimer: move div_long_long_rem out of jiffies.h
hrtimer: hrtimer documentation
9p: remove superflous MS_NODIRATIME assignment
hrtimer: hrtimer core code
nvidiafb: i2c bus name beautification
__deprecated_for_modules the lookup_hash() prototype
switch fs3270 to ->compat_ioctl
media-radio: Pci probing for maestro radio
kprobes: enable funcions only for required arch
fbcon: Sanitize fbcon
kexec: change CONFIG_PHYSICAL_START dependency
fbdev: rivafb: Driver cleanups
hrtimer: introduce ktime_t time format
hrtimer: switch itimers to hrtimer
hrtimer: coding style and white space cleanup 2
hrtimer: make clockid_t arguments const
Generic ioctl.h
remove TIOCGSERIAL/TIOCSSERIAL compat_ioctl entries for 390
hrtimer: validate timespec of do_sys_settimeofday
sanitize building of fs/compat_ioctl.c
matroxfb: Remove fbcon.h from the main header file
ntfs: remove superflous MS_NOATIME/MS_NODIRATIME assignments don't include ioctl32.h in drivers
replace inode_update_time with file_update_time
hrtimer: coding style clean up of clock constants
hrtimer: switch sys_nanosleep to hrtimer
fbcon: Store struct display when setting all vcs
kprobes: changed from using spinlock to mutex
hrtimer: export deinlined mktime
hrtimer: create and use timespec_valid macro
vesafb: Drop blank hook
add vfs_* helpers for xattr operations
d_instantiate_unique / NFS inode leakage
Kprobes: conversion from kcalloc to kzalloc
media-radio: Maestro avoid accessing private structures directly
nfs: sleep_on() removal
Switch getnstimestamp() calls to ktime_get_ts()
savagefb: One more I2C-enabled device in savagefb
common compat_sys_timer_create
media-radio: Maestro radio delete owner line from video device Remove getnstimestamp()
kprobes: fix build breakage
Export ktime_get_ts()
move rtc compat ioctl handling to fs/compat_ioctl.c
add ->compat_ioctl to dasd
hrtimer: convert posix timers completely
hrtimer: switch clock_nanosleep to hrtimer nanosleep API
fbdev: i810fb: Driver cleanups
hrtimer: deinline mktime and set_normalized_timespec
aty: remove unnecessary CONFIG_PCI
media-radio: Maestro radio Lindent
hrtimer: remove duplicate div_long_long_rem implementation
Add sysfs entry to disable framebuffer access
hrtimer: create hrtimer nanosleep API
hrtimer: introduce nsec_t type and conversion functions
fbdev: savagefb: Driver cleanup
fbdev: neofb: Driver cleanups
fbdev: tdfxfb: Driver cleanups
fbdev: hgafb: Convert to platform device
fbdev: asiliantfb: Driver cleanups
atyfb: Fix spelling
Fix vesafb display panning regression
nvidiafb: Add support for some pci-e chipsets
DocBook: fix kernel-doc comments
Docs update: remove obsolete patch from locks.txt
turn "const static" into "static const"
Fix console blanking
fbdev: Fix return code of fb_read and fb_write
fbdev: imsttfb: Driver cleanups
fbdev: nvidiafb: Driver cleanup
Docs update: small spelling, formating etc fixes for filesystems/ext3.txt char/isicom: Other little changes
vga16fb: Trim vga16fb_pan_display
rivafb: Trim rivafb_pan_display
Docs update: small fixes to stable_kernel_rules.txt
Serial: disable jsm in ppc64 defconfig
fbdev: sstfb: Driver cleanups
vesafb: Trim vesafb_pan_display
skeletonfb: Documentation update
fs/binfmt_elf: Remove unneeded kmalloc() return value casts
fbdev: pm2fb: Driver cleanups
Add git tree for DocBook
CodingStyle correction
char/isicom: Type conversion and variables deletion include/video/newport.h: "extern inline" -> "static inline"
DocBook: warn for missing macro parameters
fs/ext3/: small cleanups
fbcon: Code cleanups
fbdev: fbdev: Cleanup
nvidiafb: Add boot option 'bpp'
savagefb: Trim savagefb_pan_display
drivers/video/: possible cleanups
DocBook: add .gitignore file
non-linear frame buffer read/write access
fs/ext2/bitmap.c: ext2_count_free() is only required #ifdef EXT2FS_DEBUG
fbdev: Reduce stack usage
s3c2410fb: cleanup and fix
fbdev: atyfb: Remove BIOS-less booting
drivers/char: Use ARRAY_SIZE macro
fbcon: disable ywrap if not supported by fbcon scrolling code
atyfb: Reduce verbosity
Docs update: typos, corrections and additions to applying-patches.txt
i810fb: Fix suspend and resume hooks
atyfb: LT/LG cleanup
atyfb: Rage XL/XC cleanup
include/asm-sh64/: "extern inline" -> "static inline" tty-layer-buffering-revamp: jsm is broken
atyfb: VT/GT cleanup
atyfb: Don't stretch with CRT
atyfb: Set ECP divider
fbdev: kyrofb: Driver cleanups
fbdev: Replace kmalloc with kzalloc
fs/hfsplus/: remove the hfsplus_inode_check() debug function Decrease number of pointer derefs in exit.c
atyfb: Fix CRTC_FIFO_LWM mask
video/matrox/matroxfb_misc.c: remove dead code
atyfb: Fix interlaced modes
nvidiafb: Reduce stack usage
vr41xx: ARRAY_SIZE cleanup
char/isicom: Firmware loading
include/linux/sched.h: no need to guard the normalize_rt_tasks() prototype char/isicom: Whitespace cleanup
char/isicom: More whitespaces and coding style
char/isicom: Pci probing added
Decrease number of pointer derefs in multipath.c drivers/net/irda/irport.c: cleanups
kernel/resource.c: __check_region(): remove pointless __deprecated
fbdev: Typos in Kconfig
let MAGIC_SYSRQ no longer depend on DEBUG_KERNEL
lib/zlib*: cleanups
atyfb: Improve blanking
selinux: Remove unneeded k[cm]alloc() return value casts n_hdlc.c: remove unused declaration
clean up computone remaining cli use
TTY layer buffering revamp

#4 lore raw
Quoted 23 lines · Andrew Morton
Quoted 11 lines · Martin J. Bligh

drilling down into the results directories can get you the command line, looks like "reaim -f workfile.short -s 1 -e 10 -i 2" to me. Buggered if I can recall what that did though.

(http://test.kernel.org/abat/20229/004.reaim.test/results/cmdline)

I *think* it's only ia32 NUMA boxes, at least as far as I can see from
a quick poke around. Which would make me guess at scheduler code. Gitweb
would be nice to use, but it doesn't tag the -git snapshots, AFAICS, which is a real shame. Nor does the git snapshot include the git tag, as far as I know. Grrrr. I guess I'll download the snapshots and diff
them, and try to pull out the sched changes by hand. Much suckage.

The diffstat for 2.6.15-git5 -> 2.6.15-git6 is at
http://www.zip.com.au/~akpm/linux/patches/stuff/2 - only a single line changed in sched.c:

diff -uNr 2.6.15-git5/kernel/sched.c 2.6.15-git6/kernel/sched.c

kernel/sched.c line 4390

Quoted 5 lines
kernel/sched.c +1 -0
@@ -4386,6 +4386,7 @@
 	} while_each_thread(g, p);
 
 	read_unlock(&tasklist_lock);
+	mutex_debug_show_all_locks();
 }

Hmm. whilst it's probably not that call, it does look like mutex debugging. Look at the profiles from reaim between -git5 and -git6:

before:
http://test.kernel.org/abat/20115/004.reaim.test/profiling/profile.text

after:
http://test.kernel.org/abat/20229/004.reaim.test/profiling/profile.text

1262 kfree 3.5056
820 mutex_lock_interruptible 164.0000

   752 __mutex_lock_slowpath                      0.8754
    43 schedule                                   0.0284
    35 _spin_lock                                 2.3333
    25 debug_mutex_set_owner                      0.1613
    23 debug_mutex_add_waiter                     0.1586

/me hugs his huge stacks of data.

--------------------------------

config DEBUG_MUTEXES
bool "Mutex debugging, deadlock detection"
default y
depends on DEBUG_KERNEL
help
This allows mutex semantics violations and mutex related deadlocks (lockups) to be detected and reported automatically.

--------------------------------

Hmmm. Who the hell thought defaulting that to 'y' was a good plan???? It's still broken in 17-rc1 ... will send you a patch in a sec.

M.

PS. Jeff Garzik pointed out there *are* git tags for each -git snapshot
http://www.kernel.org/pub/linux/kernel/v2.6/snapshots/patch-2.6.17-rc1-git11.id
which is really helpful.

#5 lore raw
Quoted 41 lines · Martin J. Bligh
Quoted 24 lines · Andrew Morton
Quoted 4 lines · Martin J. Bligh

drilling down into the results directories can get you the command line, looks like "reaim -f workfile.short -s 1 -e 10 -i 2" to me. Buggered if I can recall what that did though.

Quoted 8 lines

(http://test.kernel.org/abat/20229/004.reaim.test/results/cmdline)

I *think* it's only ia32 NUMA boxes, at least as far as I can see from a quick poke around. Which would make me guess at scheduler code. Gitweb would be nice to use, but it doesn't tag the -git snapshots, AFAICS, which is a real shame. Nor does the git snapshot include the git tag, as far as I know. Grrrr. I guess I'll download the snapshots and diff them, and try to pull out the sched changes by hand. Much suckage.

The diffstat for 2.6.15-git5 -> 2.6.15-git6 is at
http://www.zip.com.au/~akpm/linux/patches/stuff/2 - only a single line changed in sched.c:

diff -uNr 2.6.15-git5/kernel/sched.c 2.6.15-git6/kernel/sched.c

kernel/sched.c line 4390

Quoted 5 lines
kernel/sched.c +1 -0
@@ -4386,6 +4386,7 @@
 	} while_each_thread(g, p);
 
 	read_unlock(&tasklist_lock);
+	mutex_debug_show_all_locks();
 }

Hmm. whilst it's probably not that call, it does look like mutex debugging. Look at the profiles from reaim between -git5 and -git6:

before:
http://test.kernel.org/abat/20115/004.reaim.test/profiling/profile.text

after:
http://test.kernel.org/abat/20229/004.reaim.test/profiling/profile.text

Quoted 7 lines

1262 kfree 3.5056
820 mutex_lock_interruptible 164.0000
752 __mutex_lock_slowpath 0.8754
43 schedule 0.0284
35 _spin_lock 2.3333
25 debug_mutex_set_owner 0.1613
23 debug_mutex_add_waiter 0.1586

hm.

Quoted 12 lines

/me hugs his huge stacks of data.

--------------------------------

config DEBUG_MUTEXES
bool "Mutex debugging, deadlock detection"
default y
depends on DEBUG_KERNEL
help
This allows mutex semantics violations and mutex related deadlocks (lockups) to be detected and reported automatically.

Quoted 3 lines

--------------------------------

Hmmm. Who the hell thought defaulting that to 'y' was a good plan????

Ingo ;)

Quoted 1 line

It's still broken in 17-rc1 ... will send you a patch in a sec.

I already have kconfigdebug-set-debug_mutex-to-off-by-default.patch queued up.

#6 lore raw
Quoted 4 lines
Quoted 1 line

It's still broken in 17-rc1 ... will send you a patch in a sec.

I already have kconfigdebug-set-debug_mutex-to-off-by-default.patch queued up.

OK. That explains why -mm2 bounced back at the end of this graph:
http://test.kernel.org/perf/reaim.moe.png

But ... it's still way below what 2.6.15 was. There's another thunk down between 2.6.16 and 2.6.16-git18, AFAICS.

M.

#7 lore raw
Quoted 4 lines · Martin J. Bligh
Quoted 4 lines
Quoted 1 line

It's still broken in 17-rc1 ... will send you a patch in a sec.

Quoted 8 lines
Quoted 1 line

I already have kconfigdebug-set-debug_mutex-to-off-by-default.patch queued up.

OK. That explains why -mm2 bounced back at the end of this graph:
http://test.kernel.org/perf/reaim.moe.png

But ... it's still way below what 2.6.15 was. There's another thunk down between 2.6.16 and 2.6.16-git18, AFAICS.

OK, I futzed with the graphs a bit. -git2 to -git3 definitely drops off. I think -git6 to git7 does as well, though that's much more difficult to discern amongst the noise. I'll take a look more.

Andy, if there's any way you could rerun all the -git ones on moe with the CONFIG_DEBUG_MUTEXES stuff disabled, might be easier to see ...

M.

#8 lore raw
In reply to Andrew Morton
Quoted 31 lines · Andrew Morton
Quoted 5 lines · Martin J. Bligh

drilling down into the results directories can get you the command line,
looks like "reaim -f workfile.short -s 1 -e 10 -i 2" to me. Buggered if I can recall what that did though.

Quoted 8 lines

(http://test.kernel.org/abat/20229/004.reaim.test/results/cmdline)

I *think* it's only ia32 NUMA boxes, at least as far as I can see from a quick poke around. Which would make me guess at scheduler code. Gitweb would be nice to use, but it doesn't tag the -git snapshots, AFAICS, which is a real shame. Nor does the git snapshot include the git tag, as far as I know. Grrrr. I guess I'll download the snapshots and diff them, and try to pull out the sched changes by hand. Much suckage.

The diffstat for 2.6.15-git5 -> 2.6.15-git6 is at
http://www.zip.com.au/~akpm/linux/patches/stuff/2 - only a single line changed in sched.c:

diff -uNr 2.6.15-git5/kernel/sched.c 2.6.15-git6/kernel/sched.c

kernel/sched.c +1 -0
@@ -4386,6 +4386,7 @@
 	} while_each_thread(g, p);
 
 	read_unlock(&tasklist_lock);
+	mutex_debug_show_all_locks();
 }
 
 /**
Quoted 2 lines

The below patches went from -mm into mainline on that day. It'll be pretty simple to bisection search these once we have a testcase.

config DEBUG_MUTEXES
bool "Mutex debugging, deadlock detection"
default y
depends on DEBUG_KERNEL
help
This allows mutex semantics violations and mutex related deadlocks (lockups) to be detected and reported automatically.

?

By default, it's y, so it seems likely.

Signature · 1 line
OGAWA Hirofumi <[email protected]>