backports: add Coccinelle SmPL profiling support to gentree.py
PATCH 1/2 backports: add Coccinelle SmPL profiling support to gentree.py
This adds support to let you pass --profile when running Coccinelle. This will skip all patches, rewriting Makefiles, Kconfigs, etc, and it will also keep a copy of the original src directory prior to applying the spatch. You pass the spatch file as an argument when using this.
Screenshot of relevant output on 11-dev-pm-ops.cocci:
$ ./gentree.py --clean --verbose --profile-cocci \ patches/collateral-evolutions/network/11-dev-pm-ops.cocci \ /home/mcgrof/linux-next/ \ /home/mcgrof/build/backports-20131206
On big iron backports server:
---------------------
profiling result
---------------------
Main total : 275.327 sec 1 count Main.outfiles computation : 274.761 sec 1 count full_engine : 272.034 sec 291 count C parsing : 158.143 sec 346 count TOTAL : 158.141 sec 346 count HACK : 69.680 sec 696 count C parsing.tokens : 54.000 sec 693 count Parsing: 1st pass : 43.870 sec 22728 count YACC : 42.921 sec 22615 count C parsing.fix_cpp : 36.855 sec 349 count MACRO mgmt prep 2 : 32.787 sec 346 count TAC.annotate_program : 32.779 sec 318 count flow : 31.251 sec 21004 count LEXING : 26.415 sec 346 count bigloop : 15.298 sec 291 count process_a_ctl_a_env_a_toplevel : 14.840 sec 46661 count C parsing.lookahead : 13.254 sec 1965389 count mysat : 12.777 sec 46661 count show_xxx : 11.933 sec 49645 count C parsing.fix_define : 6.957 sec 693 count Type_c.type_of_s : 6.729 sec 135896 count module_pci : 6.653 sec 291 count rule starting on line 28 : 6.630 sec 291 count fix_flow : 6.498 sec 20038 count C parsing.lex_ident : 6.388 sec 1695857 count C consistencycheck : 6.213 sec 346 count Pattern3.match_re_node : 6.110 sec 969235 count Common.full_charpos_to_pos_large : 5.684 sec 693 count C parsing.mk_info_item : 3.556 sec 22728 count worth_trying : 2.683 sec 1902 count Parsing: multi pass : 2.574 sec 206 count simple_dev_pm : 2.011 sec 291 count TAC.unwrap_unfold_env : 2.007 sec 171026 count TAC.typedef_fix : 1.948 sec 273295 count TAC.lookup_env : 1.583 sec 236568 count TAC.add_binding : 0.896 sec 57620 count MACRO managment : 0.439 sec 118 count Main.result analysis : 0.418 sec 1 count Common.=~ : 0.305 sec 80558 count C unparsing : 0.168 sec 41 count MACRO mgmt prep 1 : 0.148 sec 346 count parse cocci : 0.115 sec 1 count pre_engine : 0.115 sec 1 count Common.info_from_charpos : 0.102 sec 54 count Main.infiles computation : 0.033 sec 1 count ctl : 0.019 sec 94 count Transformation3.transform : 0.006 sec 27 count TAC.lookup_typedef : 0.004 sec 332 count check_duplicate : 0.003 sec 1 count Common.group_assoc_bykey_eff : 0.003 sec 1 count merge_env : 0.003 sec 649 count post_engine : 0.000 sec 1 count get_glimpse_constants : 0.000 sec 1 count Common.full_charpos_to_pos : 0.000 sec 2 count asttoctl2 : 0.000 sec 1 count
On a Chromebook Pixel:
---------------------
profiling result
---------------------
Main total : 379.349 sec 1 count Main.outfiles computation : 379.139 sec 1 count full_engine : 372.905 sec 1902 count HACK : 96.134 sec 2785 count C parsing.tokens : 76.053 sec 2769 count Parsing: 1st pass : 57.708 sec 82287 count YACC : 56.033 sec 81918 count C parsing.fix_cpp : 52.167 sec 1400 count TAC.annotate_program : 46.560 sec 1356 count MACRO mgmt prep 2 : 43.976 sec 1384 count flow : 41.027 sec 80563 count LEXING : 38.111 sec 1384 count bigloop : 17.631 sec 1329 count process_a_ctl_a_env_a_toplevel : 17.174 sec 161123 count C parsing.lookahead : 15.138 sec 6737589 count mysat : 14.537 sec 161123 count Type_c.type_of_s : 11.815 sec 675989 count Common.full_charpos_to_pos_large : 9.507 sec 2769 count fix_flow : 8.685 sec 77269 count module_pci : 8.459 sec 1329 count rule starting on line 28 : 8.400 sec 1329 count C consistencycheck : 8.137 sec 1384 count C parsing.lex_ident : 7.810 sec 5665983 count C parsing.fix_define : 7.343 sec 2769 count Pattern3.match_re_node : 6.827 sec 3046889 count C parsing.mk_info_item : 5.787 sec 82287 count show_xxx : 4.825 sec 174022 count Parsing: multi pass : 2.802 sec 679 count TAC.lookup_env : 2.556 sec 858456 count TAC.typedef_fix : 2.540 sec 976842 count TAC.unwrap_unfold_env : 2.278 sec 587709 count TAC.add_binding : 1.157 sec 204618 count simple_dev_pm : 0.762 sec 1329 count MACRO managment : 0.610 sec 394 count Common.=~ : 0.377 sec 253422 count MACRO mgmt prep 1 : 0.299 sec 1384 count Main.result analysis : 0.200 sec 1 count Common.info_from_charpos : 0.135 sec 248 count C unparsing : 0.132 sec 41 count pre_engine : 0.050 sec 1 count parse cocci : 0.050 sec 1 count C parsing : 0.016 sec 1384 count check_duplicate : 0.012 sec 1 count Common.group_assoc_bykey_eff : 0.012 sec 1 count TAC.lookup_typedef : 0.011 sec 1314 count Main.infiles computation : 0.010 sec 1 count ctl : 0.009 sec 94 count merge_env : 0.005 sec 2725 count TOTAL : 0.004 sec 1384 count Transformation3.transform : 0.003 sec 27 count Common.full_charpos_to_pos : 0.002 sec 2 count C unparsing.new_tabbing : 0.000 sec 149 count get_glimpse_constants : 0.000 sec 1 count asttoctl2 : 0.000 sec 1 count post_engine : 0.000 sec 1 count
--- gentree.py | 36 ++++++++++++++++++++++++++++++++---- 1 file changed, 32 insertions(+), 4 deletions(-)
gentree.py +32 -4
@@ -472,6 +472,8 @@ def _main(): 'however run `kup ls` on the target paths so ' + 'at the very least we test your kup configuration. ' + 'If this is your first time uploading use this first!') + parser.add_argument('--profile-cocci', metavar='<sp_file>', type=str, default=None, + help='Only use the cocci file passed and pass --profile to Coccinelle.') args = parser.parse_args() def logwrite(msg):
@@ -486,19 +488,22 @@ def _main(): extra_driver=args.extra_driver, kup=args.kup, kup_test=args.kup_test, + profile_cocci=args.profile_cocci, logwrite=logwrite) def process(kerneldir, outdir, copy_list_file, git_revision=None, clean=False, refresh=False, base_name="Linux", gitdebug=False, verbose=False, extra_driver=[], kup=False, kup_test=False, + profile_cocci=None, logwrite=lambda x:None, git_tracked_version=False): class Args(object): def __init__(self, kerneldir, outdir, copy_list_file, git_revision, clean, refresh, base_name, gitdebug, verbose, extra_driver, kup, - kup_test): + kup_test, + profile_cocci): self.kerneldir = kerneldir self.outdir = outdir self.copy_list = copy_list_file
@@ -511,6 +516,7 @@ def process(kerneldir, outdir, copy_list_file, git_revision=None, self.extra_driver = extra_driver self.kup = kup self.kup_test = kup_test + self.profile_cocci = profile_cocci def git_paranoia(tree=None, logwrite=lambda x:None): data = git.paranoia(tree) if (data['r'] != 0):
@@ -522,7 +528,8 @@ def process(kerneldir, outdir, copy_list_file, git_revision=None, args = Args(kerneldir, outdir, copy_list_file, git_revision, clean, refresh, base_name, - gitdebug, verbose, extra_driver, kup, kup_test) + gitdebug, verbose, extra_driver, kup, kup_test, + profile_cocci) rel_prep = None # start processing ...
@@ -592,14 +599,27 @@ def process(kerneldir, outdir, copy_list_file, git_revision=None, bpcfg.disable_symbols(disable_list) git_debug_snapshot(args, 'Add automatic backports') + # Extend with other tests for Coccinelle + test_cocci = args.profile_cocci + if test_cocci: + test_cocci = test_cocci.split('/')[-1] + orig_dir = args.outdir + '.orig' + shutil.rmtree(orig_dir, ignore_errors=True) + copytree(args.outdir, orig_dir) + logwrite('Apply patches ...') patches = [] sempatches = [] for root, dirs, files in os.walk(os.path.join(source_dir, 'patches')): for f in files: - if f.endswith('.patch'): + if not test_cocci and f.endswith('.patch'): patches.append(os.path.join(root, f)) if f.endswith('.cocci'): + if test_cocci: + if f != test_cocci: + continue + if args.profile_cocci: + logwrite("Profiling Coccinelle SmPL patch: %s" % test_cocci) sempatches.append(os.path.join(root, f)) patches.sort() prefix_len = len(os.path.join(source_dir, 'patches')) + 1
@@ -681,12 +701,16 @@ def process(kerneldir, outdir, copy_list_file, git_revision=None, sempatches.sort() prefix_len = len(os.path.join(source_dir, 'patches')) + 1 for cocci_file in sempatches: + profile = '' + if args.profile_cocci: + profile = '--profile' print_name = cocci_file[prefix_len:] if args.verbose: logwrite("Applying patch %s" % print_name) process = subprocess.Popen(['spatch', '--sp-file', cocci_file, '--in-place', - '--backup-suffix', '.cocci_backup', '--dir', '.'], + '--backup-suffix', '.cocci_backup', profile, + '--dir', '.'], stdout=subprocess.PIPE, stderr=subprocess.STDOUT, close_fds=True, universal_newlines=True, cwd=args.outdir)
@@ -711,6 +735,10 @@ def process(kerneldir, outdir, copy_list_file, git_revision=None, os.unlink(os.path.join(root, f)) git_debug_snapshot(args, "apply backport patch %s" % print_name) + if test_cocci: + logwrite('Done!') + return 0 + # some post-processing is required configtree = kconfig.ConfigTree(os.path.join(args.outdir, 'Kconfig')) logwrite('Modify Kconfig tree ...')
Signature · 1 line
1.8.4.3
PATCH 2/2 backports: add support for testing only a single Coccinelle SmPL patch
You can just pass now one cocci file with the --test-cocci to test only that single Coccinelle SmPL patch.
On my Chromebook Pixel:
$ time ./gentree.py --clean --verbose --test-cocci \ patches/collateral-evolutions/network/11-dev-pm-ops.cocci \ /home/mcgrof/linux-next/ \ /home/mcgrof/build/backports-20131206
real 5m39.361s user 5m24.148s sys 0m7.772s
Note run time with --profile-cocci on the same system:
real 6m25.778s user 6m4.912s sys 0m8.060s
The an original copy of the souce directory will be kept under the target directory, renamed with .orig.
--- gentree.py | 16 ++++++++++++---- 1 file changed, 12 insertions(+), 4 deletions(-)
gentree.py +12 -4
@@ -472,6 +472,8 @@ def _main(): 'however run `kup ls` on the target paths so ' + 'at the very least we test your kup configuration. ' + 'If this is your first time uploading use this first!') + parser.add_argument('--test-cocci', metavar='<sp_file>', type=str, default=None, + help='Only use the cocci file passed for Coccinelle, don\'t do anything else') parser.add_argument('--profile-cocci', metavar='<sp_file>', type=str, default=None, help='Only use the cocci file passed and pass --profile to Coccinelle.') args = parser.parse_args()
@@ -488,6 +490,7 @@ def _main(): extra_driver=args.extra_driver, kup=args.kup, kup_test=args.kup_test, + test_cocci=args.test_cocci, profile_cocci=args.profile_cocci, logwrite=logwrite)
@@ -495,6 +498,7 @@ def process(kerneldir, outdir, copy_list_file, git_revision=None, clean=False, refresh=False, base_name="Linux", gitdebug=False, verbose=False, extra_driver=[], kup=False, kup_test=False, + test_cocci=None, profile_cocci=None, logwrite=lambda x:None, git_tracked_version=False):
@@ -503,6 +507,7 @@ def process(kerneldir, outdir, copy_list_file, git_revision=None, git_revision, clean, refresh, base_name, gitdebug, verbose, extra_driver, kup, kup_test, + test_cocci, profile_cocci): self.kerneldir = kerneldir self.outdir = outdir
@@ -516,6 +521,7 @@ def process(kerneldir, outdir, copy_list_file, git_revision=None, self.extra_driver = extra_driver self.kup = kup self.kup_test = kup_test + self.test_cocci = test_cocci self.profile_cocci = profile_cocci def git_paranoia(tree=None, logwrite=lambda x:None): data = git.paranoia(tree)
@@ -529,7 +535,7 @@ def process(kerneldir, outdir, copy_list_file, git_revision=None, args = Args(kerneldir, outdir, copy_list_file, git_revision, clean, refresh, base_name, gitdebug, verbose, extra_driver, kup, kup_test, - profile_cocci) + test_cocci, profile_cocci) rel_prep = None # start processing ...
@@ -599,8 +605,8 @@ def process(kerneldir, outdir, copy_list_file, git_revision=None, bpcfg.disable_symbols(disable_list) git_debug_snapshot(args, 'Add automatic backports') - # Extend with other tests for Coccinelle - test_cocci = args.profile_cocci + test_cocci = args.profile_cocci or \ + args.test_cocci if test_cocci: test_cocci = test_cocci.split('/')[-1] orig_dir = args.outdir + '.orig'
@@ -618,7 +624,9 @@ def process(kerneldir, outdir, copy_list_file, git_revision=None, if test_cocci: if f != test_cocci: continue - if args.profile_cocci: + if args.test_cocci: + logwrite("Testing Coccinelle SmPL patch: %s" % test_cocci) + elif args.profile_cocci: logwrite("Profiling Coccinelle SmPL patch: %s" % test_cocci) sempatches.append(os.path.join(root, f)) patches.sort()
Signature · 1 line
1.8.4.3
Re: [PATCH 2/2] backports: add support for testing only a single Coccinelle SmPL patch
Quoted 20 lines · Luis R. Rodriguez
You can just pass now one cocci file with the --test-cocci to test only that single Coccinelle SmPL patch.
On my Chromebook Pixel:
$ time ./gentree.py --clean --verbose --test-cocci \ patches/collateral-evolutions/network/11-dev-pm-ops.cocci \ /home/mcgrof/linux-next/ \ /home/mcgrof/build/backports-20131206real 5m39.361s user 5m24.148s sys 0m7.772sNote run time with --profile-cocci on the same system:
real 6m25.778s user 6m4.912s sys 0m8.060s
Quoted 8 lines
The an original copy of the souce directory will be kept under the target directory, renamed with .orig.
Cc Peter Senna <[email protected]>Cc Julia Lawall <[email protected]>Cc Gilles Muller <[email protected]>Signed-off-by Luis R. Rodriguez <[email protected]>---
Hi Luis,
these patches are conflicting with a patch from Johannes [0] and I haven't applied it.
[0]: http://www.spinics.net/lists/backports/msg02188.html
Hauke
Re: [PATCH 2/2] backports: add support for testing only a single Coccinelle SmPL patch
Quoted 4 lines · Hauke Mehrtens
Quoted 4 lines
$ time ./gentree.py --clean --verbose --test-cocci \ patches/collateral-evolutions/network/11-dev-pm-ops.cocci \ /home/mcgrof/linux-next/ \ /home/mcgrof/build/backports-20131206
I'm not really convinced that adding profile options for spatch is a good thing in gentree anyway. It's complicated enough as is, and running the spatch manually isn't all that difficult. If you use the --gitdebug option to gentree, you could even roll back to just before the patch was applied before running it manually.
Now, I don't want to control what goes into backports, but this way I might end up having to rewrite it again ;-)
johannes
Re: [PATCH 2/2] backports: add support for testing only a single Coccinelle SmPL patch
Quoted 7 lines · Johannes Berg
Quoted 4 lines · Hauke Mehrtens
Quoted 4 lines
$ time ./gentree.py --clean --verbose --test-cocci \ patches/collateral-evolutions/network/11-dev-pm-ops.cocci \ /home/mcgrof/linux-next/ \ /home/mcgrof/build/backports-20131206
Quoted 8 lines
I'm not really convinced that adding profile options for spatch is a good thing in gentree anyway. It's complicated enough as is, and running the spatch manually isn't all that difficult. If you use the --gitdebug option to gentree, you could even roll back to just before the patch was applied before running it manually.
Now, I don't want to control what goes into backports, but this way I might end up having to rewrite it again ;-)
I'm not sure to understand the issue. Spatch has a --profile option, but I don't see it in the above code. From my point of view, it could be nice if people are somehow encouraged to use that option from time to time, because it increases the chance that they will report back on performance problems. But there is no need for it to bein the default execution path.
julia
Re: [PATCH 2/2] backports: add support for testing only a single Coccinelle SmPL patch
Quoted 11 lines · Julia Lawall
Quoted 10 lines
Quoted 4 lines
Quoted 4 lines
$ time ./gentree.py --clean --verbose --test-cocci \ patches/collateral-evolutions/network/11-dev-pm-ops.cocci \ /home/mcgrof/linux-next/ \ /home/mcgrof/build/backports-20131206I'm not really convinced that adding profile options for spatch is a good thing in gentree anyway. It's complicated enough as is, and running the spatch manually isn't all that difficult. If you use the --gitdebug option to gentree, you could even roll back to just before the patch was applied before running it manually.
Quoted 5 lines
I'm not sure to understand the issue. Spatch has a --profile option, but I don't see it in the above code. From my point of view, it could be nice if people are somehow encouraged to use that option from time to time, because it increases the chance that they will report back on performance problems. But there is no need for it to bein the default execution path.
I'm just saying that making 'gentree' the kitchen sink of backports generation, spatch profiling, patch refreshing, etc. will just make for really messy code (it's not particularly clean as is) and that I don't see the need for adding everything into it since running spatch --profile or other such manually isn't terribly difficult (so there's little reason to add it to gentree.)
johannes