MMTests 0.05
MMTests 0.05 is a configurable test suite that runs a number of common workloads of interest to MM developers. The biggest addition this time around is separate extraction, compare and reporting scripts. This may help people run their own analysis in another tool if they wish.
Changelog since V0.04
o Move driver and config scripts into their own directory
o Add bin/extract-mmtests.pl and bin/compare-mmtests.pl
o Remove references to Irish kernel.org mirror
o Small tidy up
At LSF/MM at some point a request was made that a series of tests be identified that were of interest to MM developers and that could be used for testing the Linux memory management subsystem. There is renewed interest in some sort of general testing framework during discussions for Kernel Summit 2012 so here is what I use.
http://www.csn.ul.ie/~mel/projects/mmtests/
http://www.csn.ul.ie/~mel/projects/mmtests/mmtests-0.05-mmtests-0.01.tar.gz
There are a number of stock configurations stored in configs/. For example config-global-dhp__pagealloc-performance runs a number of tests that may be able to identify performance regressions or gains in the page allocator. Similarly there network and scheduler configs. There are also more complex options. config-global-dhp__parallelio-memcachetest will run memcachetest in the foreground while doing IO of different sizes in the background to measure how much unrelated IO affects the throughput of an in-memory database.
This release is also a little rough and the extraction scripts could have been tidier but they were mostly written in an airport and for the most part they work as advertised. I'll fix bugs as according as they are brought to my attention.
The stats reporting still needs work because while some tests know how to make a better estimate of mean by filtering outliers it is not being handled consistently and the methodology needs work. I know filtering statistics like this is a major flaw in the methodology but the decision was made in this case in the interest of the benchmarks with unstable results completing in a reasonable time.
Out of the box it should now do something useful so here is a demo of the page fault microbenchmark. At the most recent memcg meeting I used this benchmark to demonstrate how memory control groups have between 6% and 15% overhead even when not in use. If someone is interested in reproducing that I'll send on a patch that configures profiling so you can reproduce it.
# Download and "install"
mel@machina:~ > wget -q http://www.csn.ul.ie/~mel/projects/mmtests/mmtests-0.05-mmtests-0.01.tar.gz
mel@machina:~ > tar -xf mmtests-0.05-mmtests-0.01.tar.gz
mel@machina:~ > cd mmtests-0.05-mmtests-0.01/
# Run with the default "config" file. It runs a a page fault microbenchmark. # There are some warnings displayed about root, some tests require root but
# this is not one of them specifically. It also wars about libnuma.h not # being available but on this machine it doesn't matter mel@machina:~/mmtests-0.05-mmtests-0.01 > ./run-mmtests.sh test-run-1 Tuning the system for run: test-run-1 monitor: yes Using default swap configuration Swap configuration Filename Type Size Used Priority Configuring ftrace mount: only root can do that Skipping warmup run
/home/mel/mmtests-0.05-mmtests-0.01/shellpacks/common.sh: line 144: /sys/kernel/mm/transparent_hugepage/enabled: Permission denied Starting monitors
Started monitor proc-vmstat gzip pid 8595,8597
Started monitor top gzip pid 8653,8655
Started monitor slabinfo gzip pid 8706,8708
Started monitor vmstat latency pid 6146 6149 8759 8762,8760
Started monitor iostat latency pid 8801,8802
Starting test pft
cat: /proc/sys/kernel/stack_tracer_enabled: No such file or directory /home/mel/mmtests-0.05-mmtests-0.01/shellpacks/common.sh: line 137:/sys/kernel/mm/transparent_hugepage/enabled: Permission denied
pft-install: Fetching from mirror http://mcp/~gormanm/pft/pft-0.12x.tar.gz/pft-0.12x.tar.gz
pft-install: Fetching from internet http://free.linux.hp.com/~lts/Tools/pft-0.12x.tar.gz
~/mmtests-0.05-mmtests-0.01/work/testdisk/sources/pft-0.12x-installed ~/mmtests-0.05-mmtests-0.01/work/testdisk/sources ~/mmtests-0.05-mmtests-0.01/work/log/pft
WARNING: PFT REQUIRES NUMA.H AND IT IS NOT AVAILABLE
WORKING AROUND, BUT MAY NOT BEHAVE AS EXPECTED
patching file Makefile
patching file numa_stubs.h
patching file pft.c
cc -std=gnu99 -pthread -O3 -D_GNU_SOURCE -DUSE_RUSAGE_THREAD -UUSE_NOCLEAR -c -o pft.o pft.c
cc -o pft -std=gnu99 pft.o -lpthread -lrt
pft installed successfully
1 1 1 0.07s 0.59s 0.65s 622936.989 620037.781 1 1 1 0.08s 0.50s 0.58s 699060.613 695857.289 1 1 1 0.06s 0.52s 0.58s 701475.702 697679.908 1 1 1 0.07s 0.51s 0.58s 702689.514 699572.271 1 1 1 0.04s 0.51s 0.55s 737121.449 733980.458
[ ..... test run continues ......]
# In a normal situation you would probably install a different kernel as
# part of some comparison and run the test again. Here we'll just run # it a second time. mel@machina:~/mmtests-0.05-mmtests-0.01 > ./run-mmtests.sh test-run-2 [ ..... test runs a second time .....]
# Extract the raw results from the test
mel@machina:~/mmtests-0.05-mmtests-0.01 > ./bin/extract-mmtests.pl -d work/log -b pft -n test-run-1 --print-header
Clients User System Elapsed Faults/cpu Faults/sec
1 0.07 0.59 0.65 622936.989 620037.781 1 0.08 0.50 0.58 699060.613 695857.289 1 0.06 0.52 0.58 701475.702 697679.908 1 0.07 0.51 0.58 702689.514 699572.271 1 0.04 0.51 0.55 737121.449 733980.458
[ .... raw results for each sample taken is displayed .... ]
# Print a summary of the results. For this test, a summary shows the
# mean of each sample taken. Other tests summarise differently mel@machina:~/mmtests-0.05-mmtests-0.01 > ./bin/extract-mmtests.pl -d work/log -b pft -n test-run-1 --print-header --print-summary Clients User System Elapsed Faults/cpu Faults/sec 1 0.06 0.52 0.58 705602.805 702256.189 2 0.08 0.70 0.40 524681.533 1028256.231 3 0.09 0.91 0.36 404728.859 1137961.182 4 0.10 0.97 0.30 382681.532 1371008.438 5 0.10 1.01 0.25 367927.082 1649637.507 6 0.10 1.10 0.22 337179.610 1832504.839 7 0.11 1.25 0.21 299583.769 1966789.329 8 0.11 1.37 0.21 273358.740 1956777.078
# Compare test-run-1 and test-run-2. The results are unstable because this
# is running on my laptop which was also doing other work at the time.
mel@machina:~/mmtests-0.05-mmtests-0.01 > ./bin/compare-mmtests.pl -d work/log/ -b pft -n test-run-1,test-run-2
test test
run-1 run-2
User 1 0.0610 ( 0.00%) 0.0600 ( 1.64%)
User 2 0.0785 ( 0.00%) 0.0645 ( 17.83%)
User 3 0.0920 ( 0.00%) 0.0830 ( 9.78%)
User 4 0.0960 ( 0.00%) 0.0930 ( 3.12%)
User 5 0.0975 ( 0.00%) 0.0955 ( 2.05%)
User 6 0.0995 ( 0.00%) 0.1040 ( -4.52%)
User 7 0.1050 ( 0.00%) 0.1095 ( -4.29%)
User 8 0.1100 ( 0.00%) 0.1130 ( -2.73%)
System 1 0.5160 ( 0.00%) 0.4780 ( 7.36%)
System 2 0.7000 ( 0.00%) 0.5985 ( 14.50%)
System 3 0.9125 ( 0.00%) 0.7345 ( 19.51%)
System 4 0.9660 ( 0.00%) 0.8525 ( 11.75%)
System 5 1.0085 ( 0.00%) 0.9680 ( 4.02%)
System 6 1.1030 ( 0.00%) 1.1125 ( -0.86%)
System 7 1.2500 ( 0.00%) 1.2460 ( 0.32%)
System 8 1.3745 ( 0.00%) 1.3645 ( 0.73%)
Elapsed 1 0.5815 ( 0.00%) 0.5395 ( 7.22%)
Elapsed 2 0.3990 ( 0.00%) 0.3355 ( 15.91%)
Elapsed 3 0.3585 ( 0.00%) 0.2740 ( 23.57%)
Elapsed 4 0.2975 ( 0.00%) 0.2470 ( 16.97%)
Elapsed 5 0.2455 ( 0.00%) 0.2410 ( 1.83%)
Elapsed 6 0.2215 ( 0.00%) 0.2225 ( -0.45%)
Elapsed 7 0.2065 ( 0.00%) 0.2060 ( 0.24%)
Elapsed 8 0.2075 ( 0.00%) 0.2105 ( -1.45%)
Faults/cpu 1 705602.8052 ( 0.00%) 756530.8582 ( 7.22%)
Faults/cpu 2 524681.5331 ( 0.00%) 612777.6817 ( 16.79%)
Faults/cpu 3 404728.8590 ( 0.00%) 495572.8665 ( 22.45%)
Faults/cpu 4 382681.5322 ( 0.00%) 429694.5717 ( 12.29%)
Faults/cpu 5 367927.0821 ( 0.00%) 381027.2533 ( 3.56%)
Faults/cpu 6 337179.6097 ( 0.00%) 333885.0524 ( -0.98%)
Faults/cpu 7 299583.7693 ( 0.00%) 299253.1277 ( -0.11%)
Faults/cpu 8 273358.7403 ( 0.00%) 274967.2918 ( 0.59%)
Faults/sec 1 702256.1889 ( 0.00%) 752642.3956 ( 7.17%)
Faults/sec 21028256.2315 ( 0.00%)1213980.6263 ( 18.06%)
Faults/sec 31137961.1816 ( 0.00%)1471511.1024 ( 29.31%)
Faults/sec 41371008.4380 ( 0.00%)1651793.2132 ( 20.48%)
Faults/sec 51649637.5069 ( 0.00%)1685023.2429 ( 2.15%)
Faults/sec 61832504.8389 ( 0.00%)1828275.9043 ( -0.23%)
Faults/sec 71966789.3289 ( 0.00%)1970410.1478 ( 0.18%)
Faults/sec 81956777.0777 ( 0.00%)1930608.9724 ( -1.34%)# Compare the running times
mel@machina:~/mmtests-0.05-mmtests-0.01 > ./bin/compare-mmtests.pl -d work/log/ -b pft -n test-run-1,test-run-2 --print-monitor duration
test test
run-1 run-2
User 15.52 14.83
System 196.24 184.81
Elapsed 69.88 61.51
# Compare vmstat information
mel@machina:~/mmtests-0.05-mmtests-0.01 > ./bin/compare-mmtests.pl -d work/log/ -b pft -n test-run-1,test-run-2 --print-monitor vmstat
test test
run-1 run-2
Page Ins 4440 0
Page Outs 34876 4440
Swap Ins 0 0 Swap Outs 0 0 Direct pages scanned 0 0 Kswapd pages scanned 199986 0 Kswapd pages reclaimed 199981 0 Direct pages reclaimed 0 0
Kswapd efficiency 99% 100%
Kswapd velocity 2861.849 0.000
Direct efficiency 100% 100%
Direct velocity 0.000 0.000
Percentage direct scans 0% 0%
Page writes by reclaim 0 0 Page writes file 0 0 Page writes anon 0 0 Page reclaim immediate 0 0 Page rescued immediate 0 0 Slabs scanned 145408 0 Direct inode steals 0 0 Kswapd inode steals 33392 0 Kswapd skipped wait 0 0 THP fault alloc 0 0 THP collapse alloc 0 0 THP splits 0 0 THP fault fallback 0 0 THP collapse fail 0 0 Compaction stalls 0 0 Compaction success 0 0 Compaction failures 0 0 Compaction pages moved 0 0 Compaction move failure 0 0
Feedback welcome.
Signature · 8 lines
Mel Gorman SUSE Labs -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Quoted 1 line · Mel Gorman
./run-mmtests.sh test-run-1
Mel, would you share with us the command line and config tweaks you had in place to run the memcg tests you presented in the memcg summit?
Thanks!
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Quoted 6 lines · Glauber Costa
Quoted 1 line · Mel Gorman
./run-mmtests.sh test-run-1
Mel, would you share with us the command line and config tweaks you had in place to run the memcg tests you presented in the memcg summit?
Apply the following patch to mmtests 0.05 and then from within the mmtests directory do
./run-mmtests.sh testrun
At the very least you should have oprofile installed. Optionally install libnuma-devel but the test will cope if it's not available. Automatic package installation will be in 0.06 for opensuse at least but other distros can be easily supported if I know the names of the equivalent packages.
The above command will run both with and without profiling. The profiles will be in work/log/pft-testrun/fine-profile-timer/base/ and an annotated profile will be included in the file. If you have "recode" installed the annotated profile will be compressed and can be extracted with something like
grep -A 9999999 "=== annotate ===" oprofile-compressed.report | grep -v annotate | recode /b64..char | gunzip -c
Each of the memcg functions will be small but when all the functions that are in mm/memcontrol.c are added together it becomes a big problem. What I actually showed at the meeting was based on piping the oprofile report through another quick and dirty script to match functions to filenames.
The bulk of this patch is renaming profile-disabled-hooks-a.sh to profile-hooks-a.sh. Let me know if you run into problems.
---8<--
mmtests: Configure for PFT profile
config +3 -3
@@ -9,9 +9,9 @@ export SKIP_WARMUP=yes # Profiling parameters export SKIP_NOPROFILE=no -export SKIP_FINEPROFILE=yes +export SKIP_FINEPROFILE=no export SKIP_COARSEPROFILE=yes -export OPROFILE_REPORT_ANNOTATE=no +export OPROFILE_REPORT_ANNOTATE=yes # Fixups if [ "`which check-confidence.pl 2> /dev/null`" = "" ]; then
@@ -57,7 +57,7 @@ export SWAP_NBD_PORT=100`ifconfig eth0 | sed -n 2p | cut -d ":" -f2 | cut -d " " #export TESTDISK_NBD_PORT=100`ifconfig eth0 | sed -n 2p | cut -d ":" -f2 | cut -d " " -f1 | cut -d "." -f4` # List of monitors -export RUN_MONITOR=yes +export RUN_MONITOR=no export MONITORS_ALWAYS= export MONITORS_PLAIN= export MONITORS_GZIP="proc-vmstat top slabinfo"
profile-disabled-hooks-a.sh deleted +0 -48
48 lines not on this page · raw message
profile-hooks-a.sh new file +48 -0
48 lines not on this page · raw message
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Quoted 25 lines · Mel Gorman
Quoted 6 lines · Glauber Costa
Quoted 1 line · Mel Gorman
./run-mmtests.sh test-run-1
Mel, would you share with us the command line and config tweaks you had in place to run the memcg tests you presented in the memcg summit?
Apply the following patch to mmtests 0.05 and then from within the mmtests directory do
./run-mmtests.sh testrun
At the very least you should have oprofile installed. Optionally install libnuma-devel but the test will cope if it's not available. Automatic package installation will be in 0.06 for opensuse at least but other distros can be easily supported if I know the names of the equivalent packages.
The above command will run both with and without profiling. The profiles will be in work/log/pft-testrun/fine-profile-timer/base/ and an annotated profile will be included in the file. If you have "recode" installed the annotated profile will be compressed and can be extracted with something like
grep -A 9999999 "=== annotate ===" oprofile-compressed.report | grep -v annotate | recode /b64..char | gunzip -c
Quoted 7 lines
Each of the memcg functions will be small but when all the functions that are in
mm/memcontrol.care added together it becomes a big problem. What I actually showed at the meeting was based on piping the oprofile report through another quick and dirty script to match functions to filenames.The bulk of this patch is renaming
profile-disabled-hooks-a.shtoprofile-hooks-a.sh. Let me know if you run into problems.
FYI: I get this:
Can't locate TLBC/Report.pm in @INC (@INC contains:/home/glauber/mmtests-0.05-mmtests-0.01/vmr/bin /usr/local/lib64/perl5 /usr/local/share/perl5 /usr/lib64/perl5/vendor_perl /usr/share/perl5/vendor_perl /usr/lib64/perl5 /usr/share/perl5 .) at /home/glauber/mmtests-0.05-mmtests-0.01/vmr/bin/oprofile_map_events.pl line 11.
Investigating, it seems that hugetlbfs packages in fedora doesn't install any perl scripts, unlike SuSE.
I downloaded the library manually, and pointed perl path to it, and it seems to work.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Quoted 43 lines · Glauber Costa
Quoted 25 lines · Mel Gorman
Quoted 6 lines · Glauber Costa
Quoted 1 line · Mel Gorman
./run-mmtests.sh test-run-1
Mel, would you share with us the command line and config tweaks you had in place to run the memcg tests you presented in the memcg summit?
Apply the following patch to mmtests 0.05 and then from within the mmtests directory do
./run-mmtests.sh testrun
At the very least you should have oprofile installed. Optionally install libnuma-devel but the test will cope if it's not available. Automatic package installation will be in 0.06 for opensuse at least but other distros can be easily supported if I know the names of the equivalent packages.
The above command will run both with and without profiling. The profiles will be in work/log/pft-testrun/fine-profile-timer/base/ and an annotated profile will be included in the file. If you have "recode" installed the annotated profile will be compressed and can be extracted with something like
grep -A 9999999 "=== annotate ===" oprofile-compressed.report | grep -v annotate | recode /b64..char | gunzip -c
Quoted 7 lines
Each of the memcg functions will be small but when all the functions that are in
mm/memcontrol.care added together it becomes a big problem. What I actually showed at the meeting was based on piping the oprofile report through another quick and dirty script to match functions to filenames.The bulk of this patch is renaming
profile-disabled-hooks-a.shtoprofile-hooks-a.sh. Let me know if you run into problems.FYI: I get this:
Can't locate TLBC/Report.pm in @INC (@INC contains:
/home/glauber/mmtests-0.05-mmtests-0.01/vmr/bin/usr/local/lib64/perl5/usr/local/share/perl5/usr/lib64/perl5/vendor_perl/usr/share/perl5/vendor_perl/usr/lib64/perl5/usr/share/perl5.) at/home/glauber/mmtests-0.05-mmtests-0.01/vmr/bin/oprofile_map_events.plline 11.
Quoted 3 lines
Investigating, it seems that hugetlbfs packages in fedora doesn't install any perl scripts, unlike SuSE.
That is unexpected but thanks for pointing it out. I'll pull in the necessary support files into mmtests itself to avoid the problem in the future.
Quoted 3 lines
I downloaded the library manually, and pointed perl path to it, and it seems to work.
Good news, thanks.
Signature · 8 lines
Mel Gorman SUSE Labs -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>