Skip to contentSkip to conversation

coccinelle: extend testing with check-pycocci

PATCH New Luis R. Rodriguez 3 participants 9 messages May 5, 2016 – May 9, 2016 Kbuild

lore
Share & tools
lore
#0 lore raw

cover coccinelle: extend testing with check-pycocci

Coccinelle has its own test framework built-in to spatch, this is used to test against small tests cases written in tests/*.c with their respective tests/*.cocci and tests/*.res file. This is however a bit limited as it only handles single C file. In real world projects where more files are used we need to support testing against larger series of file structures.

pycocci already had SmPL <=> Patch equivalence support, we can leverage this to extend testing in a more generic fashion. This does that work.

For now, we disable pycocci-tests given we touch your git tree for these tests.

Luis R. Rodriguez (5):
pycocci: use os.path.abspath() for gitname
pycocci: add git branch support to git library
pycocci: skip proof if git dir is dirty
pycocci: add clean version of SmPL <=> Patch equivalence proof support
pycocci: enable SmPL <=> Patch equivalence tests

 Makefile                                           |  3 +
 docs/pycocci.1                                     | 24 +++++-
 scripts/pycocci-check.sh                           |  8 ++
 tests/pycocci/code/0001-proto3/proto3.c            |  7 ++
 tests/pycocci/code/0001-proto3/proto3.h            |  3 +
 tests/pycocci/patches/0001-proto3.cocci            | 11 +++
 .../0001-proto3/0001-change-prototypes.patch       | 18 +++++
 tools/pycocci                                      | 89 ++++++++++++++++------
 8 files changed, 140 insertions(+), 23 deletions(-)
 create mode 100755 scripts/pycocci-check.sh
 create mode 100644 tests/pycocci/code/0001-proto3/proto3.c
 create mode 100644 tests/pycocci/code/0001-proto3/proto3.h
 create mode 100644 tests/pycocci/patches/0001-proto3.cocci
 create mode 100644 tests/pycocci/patches/0001-proto3/0001-change-prototypes.patch
Signature · 1 line
2.7.2
#1 lore raw

PATCH 1/5 pycocci: use os.path.abspath() for gitname

gitname() is currently used for SmPL <=> Patch equivalence proof support. It uses git to test for proofs and also show any differences found. If the target directory passed where the user wishes to do work on has is not part of a git tree, it creates a git tree for you there.

When using relative paths the check for if a target directory is part of a git tree currently always fails, the consequences of which are that if you try to show proof on an existing git tree you'll end up with a sub-git project, and as such a new .git under some directory under your target path.

Correct this by using absolute paths.

---
 tools/pycocci | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
tools/pycocci +2 -1
index 43bb9a4dd412..a7b0f69b25f0 100755
@@ -246,9 +246,10 @@ def git_rev_parse(tree=None, extra_args=None):
     return stdout.split('\n', 1)[0]
 
 def gitname(path=None):
+    path = os.path.abspath(path)
     work_dir = path
     if not os.path.isdir(path):
-        work_dir = os.path.dirname(path)
+        work_dir = os.path.abspath(os.path.dirname(path))
     process = subprocess.Popen(['git', 'rev-parse', '--show-toplevel', path],
                                stdout=subprocess.PIPE, stderr=subprocess.STDOUT,
                                close_fds=True, universal_newlines=True, cwd=work_dir)
Signature · 1 line
2.7.2
#2 lore raw

PATCH 2/5 pycocci: add git branch support to git library

In reply to Luis R. Rodriguez

This will be used later on an new command for pycocci.

---
 tools/pycocci | 9 +++++++++
 1 file changed, 9 insertions(+)
tools/pycocci +9 -0
index a7b0f69b25f0..b53bcb778be2 100755
@@ -276,6 +276,15 @@ def git_checkout(tree=None, extra_args=None):
     process.wait()
     _check(process)
 
+def git_branch(tree=None, extra_args=None):
+    cmd = ['git', 'branch' ] + extra_args
+    process = subprocess.Popen(cmd,
+                               stdout=subprocess.PIPE, stderr=subprocess.STDOUT,
+                               close_fds=True, universal_newlines=True, cwd=tree)
+    stdout = process.communicate()[0]
+    process.wait()
+    _check(process)
+
 def git_commit_all(message, tree=None):
     git_add('.', tree=tree)
     process = subprocess.Popen(['git', 'commit', '--allow-empty', '-a', '-m', message],
Signature · 1 line
2.7.2
#3 lore raw

PATCH 3/5 pycocci: skip proof if git dir is dirty

In reply to Luis R. Rodriguez

SmPL <=> Patch equivalence proof support requires your git tree to be clean, if it is dirty it will fail. To avoid confusing users complain if their tree is dirty.

You will either need commit any local pending changes or discard them before proceeding with SmPL <=> Patch equivalence proof support.

Extend documentation for this.

---
 docs/pycocci.1 | 3 +++
 tools/pycocci  | 9 +++++++++
 2 files changed, 12 insertions(+)
docs/pycocci.1 +3 -0
index 63df731fdf46..e7f6b072589c 100644
@@ -201,6 +201,9 @@ will set the tree back to the master branch. Each run of \fBpycocci --show-proof
 will create two new git branches. 8 random characters are postixed to each new
 git branch created to avoid conflicts with previous runs.
 
+You must run this option with a clean git tree, if you have any pending changes
+you must commit them or discard them.
+
 .SH AUTHOR
 \fBpycocci\fP and this man page was written by Luis R. Rodriguez <[email protected]>
 
tools/pycocci +9 -0
index b53bcb778be2..4bcfb46af24b 100755
@@ -575,6 +575,15 @@ def _main():
             logwrite("Path (%s) not part of a git tree, creating one for you...\n" % (args.target_dir))
             git_init(tree=args.target_dir)
             git_commit_all(tree=args.target_dir, message="Initial commit")
+        else:
+            diff_stat = git_diff(tree=args.target_dir, extra_args = [ '--stat' ] )
+            if len(diff_stat) != 0:
+                logwrite("SmPL <--> Patch proof equivalence requires a "
+                         "clean tree.\nYou have a dirty tree with "
+                         "uncommitted changes:\n\n")
+                logwrite(diff_stat)
+                logwrite("\nEither commit these changes, or stash them away somewhere\n")
+                return -2
         cmd = [ '--abbrev-ref', 'HEAD' ]
         current_branch = git_rev_parse(tree=args.target_dir, extra_args = cmd)
         logwrite("\n")
Signature · 1 line
2.7.2
#4 lore raw

PATCH 4/5 pycocci: add clean version of SmPL <=> Patch equivalence proof support

In reply to Luis R. Rodriguez

When working with automated tests against proofs you expect mosts tests to pass. Currently SmPL <=> Patch equivalence proof support is too chatty for automated tests, and it also leaves in place the two git branches it uses to test for proof. Add a new option for pycocci which will be silent unless errors occur, and will also remove the temporary git branches used for the proof unless an error is found.

Extend documentation for this. Note that these tests use git branches on your tree, pycocc-* branches are only left in your tree if an error has occured. Even if successful, even if branches are deleted there will be a period of time during which you will still see these branches on your git reflog, by default this is about 30 days. If you want to expedite clearing your reflog you can use this after testing:

	git reflog expire --all --expire=now
---
 docs/pycocci.1 | 21 +++++++++++++++++-
 tools/pycocci  | 68 ++++++++++++++++++++++++++++++++++++++++------------------
 2 files changed, 67 insertions(+), 22 deletions(-)
docs/pycocci.1 +20 -1
index e7f6b072589c..2410022d641d 100644
@@ -204,8 +204,27 @@ git branch created to avoid conflicts with previous runs.
 You must run this option with a clean git tree, if you have any pending changes
 you must commit them or discard them.
 
+.B\-c | \-\-clean\-proof
+This does what -\-show\-proof does but this is completely silent unless an
+error occurs. It will also remove the git branches used to test for the
+equivalence proof, unless an error is found. If an error is found you can
+inspect the two branches used to test for proof, refer to the documentation
+on \-\-show\-proof for details about these branches. This option is useful
+if you want to automate tests with proofs.
+
+Note that using this method will have created and subsequently if successful
+deleted two git branches on your git tree. As a consequence of using git
+branches your git reflog will show these branches, if you push your tree
+out these branches will not be pushed as they were deleted, your reflog
+however will keep these references locally until git expires them, by default
+this is 30 days. If this is too chatty for you, you can run:
+
+	git reflog expire \-\-all \-\-expire=now
+
+This will immediately clear old entries from your reflog.
+
 .SH AUTHOR
-\fBpycocci\fP and this man page was written by Luis R. Rodriguez <[email protected]>
+\fBpycocci\fP and this man page was written by Luis R. Rodriguez <[email protected]>
 
 .SH REPORTING BUGS
 Send an mail to <[email protected]>
tools/pycocci +47 -21

146 lines not on this page · raw message

Signature · 1 line
2.7.2
#5 lore raw

PATCH 5/5 pycocci: enable SmPL <=> Patch equivalence tests

In reply to Luis R. Rodriguez

Coccinelle has integrated tests support into it, these tests are performed really fast, the current test framework enables testing with respective single C file and a respective cocci file, ie: foo.c, foo.cocci, foo.res. If one wants to add extensive tests, to reflect more real-life sized projects the current test scheme is a bit limited.

We can extend Coccinelle's existing test framework to support multiple files, header files, but a different can also consist of making use of SmPL <=> Patch equivalence support. Instead of relying on .res files, this relies on patches that provide a outline of differences using standard diff. This enables extending test coverage to more complex cases. This is not meant to replace Coccinelle's internal test functionality but rather complement it.

For now, since I can't figure out how to only remove from the reflog the temporary branch creation stuff without expiring your entire relog immediately, just keep the pycocci-check separate from the standard tests. If we're happy to have a chatty reflog when testing we can later enable this by default, but this just doesn't seem appropriate as a default setting.

---
 Makefile                                               |  3 +++
 scripts/pycocci-check.sh                               |  8 ++++++++
 tests/pycocci/code/0001-proto3/proto3.c                |  7 +++++++
 tests/pycocci/code/0001-proto3/proto3.h                |  3 +++
 tests/pycocci/patches/0001-proto3.cocci                | 11 +++++++++++
 .../patches/0001-proto3/0001-change-prototypes.patch   | 18 ++++++++++++++++++
 6 files changed, 50 insertions(+)
 create mode 100755 scripts/pycocci-check.sh
 create mode 100644 tests/pycocci/code/0001-proto3/proto3.c
 create mode 100644 tests/pycocci/code/0001-proto3/proto3.h
 create mode 100644 tests/pycocci/patches/0001-proto3.cocci
 create mode 100644 tests/pycocci/patches/0001-proto3/0001-change-prototypes.patch
Makefile +3 -0
index a98eefd2d83b..364e1cc68154 100644
@@ -489,6 +489,9 @@ check: scripts/spatch
 	@$(ECHO) running the test suite
 	COCCINELLE_HOME="$$(pwd)" ./scripts/spatch --testall --no-update-score-file
 
+pycocci-check:
+	COCCINELLE_HOME="$$(pwd)" ./scripts/pycocci-check.sh
+
 # -inline 0  to see all the functions in the profile.
 # Can also use the profile framework in commons/ and run your program
 # with -profile.
scripts/pycocci-check.sh new file +8 -0
new file mode 100755
index 000000000000..ca81ca9cb156
@@ -0,0 +1,8 @@
+#!/bin/bash
+set -e
+
+for i in tests/pycocci/patches/*.cocci; do
+	TEST=${i%%.cocci}
+	CODE_TEST=tests/pycocci/code/$(basename $TEST)
+	./tools/pycocci --clean-proof $i $CODE_TEST
+done
tests/pycocci/code/0001-proto3/proto3.c new file +7 -0
new file mode 100644
index 000000000000..db3818afd5da
@@ -0,0 +1,7 @@
+#include "proto3.h"
+
+static void bch_sched_event(struct BCState *bcs, int event)
+{
+	bcs->event |= 1 << event;
+	schedule_work(&bcs->work);
+}
tests/pycocci/code/0001-proto3/proto3.h new file +3 -0
new file mode 100644
index 000000000000..d4365e8c890e
@@ -0,0 +1,3 @@
+void bch_l2l1(struct PStack *st, int pr, void *arg);
+void bch_sched_event(struct BCState *bcs, int event);
+void bch_empty_fifo(struct BCState *bcs, int count);
tests/pycocci/patches/0001-proto3.cocci new file +11 -0
new file mode 100644
index 000000000000..d81678ef6d6e
@@ -0,0 +1,11 @@
+@@
+identifier fn2;
+identifier bcs, ev;
+@@
+
+- fn2(struct BCState *bcs, int ev) {
++ fn2(int ev) {
+   ...
+   bcs->event |= 1 << ev;
+   schedule_work(&bcs->work);
+ }
tests/pycocci/patches/0001-proto3/0001-change-prototypes.patch new file +18 -0
new file mode 100644
index 000000000000..e5b504e0acbb
@@ -0,0 +1,18 @@
+--- a/proto3.c
++++ b/proto3.c
+@@ -1,7 +1,6 @@
+ #include "proto3.h"
+ 
+-static void bch_sched_event(struct BCState *bcs, int event)
+-{
++static void bch_sched_event(int event) {
+ 	bcs->event |= 1 << event;
+ 	schedule_work(&bcs->work);
+ }
+--- a/proto3.h
++++ b/proto3.h
+@@ -1,3 +1,3 @@
+ void bch_l2l1(struct PStack *st, int pr, void *arg);
+-void bch_sched_event(struct BCState *bcs, int event);
++void bch_sched_event(int event);
+ void bch_empty_fifo(struct BCState *bcs, int count);
Signature · 1 line
2.7.2
#6 lore raw

PATCH 5/5 pycocci: enable SmPL <=> Patch equivalence tests

Quoted 5 lines

We can extend Coccinelle's existing test framework to support multiple files, header files, but a different can also consist of making use of SmPL <=> Patch equivalence support. Instead of relying on .res files, this relies on patches that provide a outline of differences using standard diff.

Does this wording need a bit more fine-tuning to make the understanding easier for the suggested software development?

Regards,
Markus

#7 lore raw

cover coccinelle: extend testing with check-pycocci

In reply to Luis R. Rodriguez

for all of the patches in this series.

julia

Quoted 44 lines · Luis R. Rodriguez

Coccinelle has its own test framework built-in to spatch, this is used to test against small tests cases written in tests/*.c with their respective tests/*.cocci and tests/*.res file. This is however a bit limited as it only handles single C file. In real world projects where more files are used we need to support testing against larger series of file structures.

pycocci already had SmPL <=> Patch equivalence support, we can leverage this to extend testing in a more generic fashion. This does that work.

For now, we disable pycocci-tests given we touch your git tree for these tests.

Luis R. Rodriguez (5):
pycocci: use os.path.abspath() for gitname
pycocci: add git branch support to git library
pycocci: skip proof if git dir is dirty
pycocci: add clean version of SmPL <=> Patch equivalence proof support
pycocci: enable SmPL <=> Patch equivalence tests

 Makefile                                           |  3 +
 docs/pycocci.1                                     | 24 +++++-
 scripts/pycocci-check.sh                           |  8 ++
 tests/pycocci/code/0001-proto3/proto3.c            |  7 ++
 tests/pycocci/code/0001-proto3/proto3.h            |  3 +
 tests/pycocci/patches/0001-proto3.cocci            | 11 +++
 .../0001-proto3/0001-change-prototypes.patch       | 18 +++++
 tools/pycocci                                      | 89 ++++++++++++++++------
 8 files changed, 140 insertions(+), 23 deletions(-)
 create mode 100755 scripts/pycocci-check.sh
 create mode 100644 tests/pycocci/code/0001-proto3/proto3.c
 create mode 100644 tests/pycocci/code/0001-proto3/proto3.h
 create mode 100644 tests/pycocci/patches/0001-proto3.cocci
 create mode 100644 tests/pycocci/patches/0001-proto3/0001-change-prototypes.patch
Signature · 6 lines
2.7.2

_______________________________________________
Cocci mailing list
Cocci at systeme.lip6.fr
https://systeme.lip6.fr/mailman/listinfo/cocci