Skip to contentSkip to conversation

coccinelle: extend test coverage

RFC New Luis R. Rodriguez 3 participants 5 messages May 5, 2016 Kbuild

lore
Share & tools
lore
#0 lore raw

cover coccinelle: extend test coverage

I wanted to extend tests coverage for coccinelle but noticed that tests are limited to single c file, these are attached to respective .res files, but this also means that you have a direct relationship and requirement between tests from a single C file and a respective cocci file.

To go wild we want to be able to support really complex types of examples. Fortunatley the backports project has been using Coccinelle extensively daily, when a backports developers needed to replace a series of patches with a Coccinelle patch pycocci was extended to provide SmPL <--> Patch equivalence support. We can easily take advantage of this to extend Coccinelle test cases way beyond a single C file, but to more complex real life style projects.

I don't think we should convert all test cases to this new mechanism but, I'd like to see what are folks' thoughts over using pycocci SmPL <--> Patch equivalence proof support to help expand the tests cases for Coccinelle.

Luis R. Rodriguez (2):
pycocci: add missing new line
tests: convert sp.cocci test to pycocci proof

 Makefile                            | 1 +
 scripts/test.sh                     | 8 ++++++++
 tests/{sp.cocci => 0001-sp.cocci}   | 0
 tests/0001-sp/replace-typedef.patch | 9 +++++++++
 tests/{ => code/0001-sp}/sp.c       | 0
 tests/sp.res                        | 4 ----
 tools/pycocci                       | 2 +-
 7 files changed, 19 insertions(+), 5 deletions(-)
 create mode 100755 scripts/test.sh
 rename tests/{sp.cocci => 0001-sp.cocci} (100%)
 create mode 100644 tests/0001-sp/replace-typedef.patch
 rename tests/{ => code/0001-sp}/sp.c (100%)
 delete mode 100644 tests/sp.res
Signature · 1 line
2.7.2
#1 lore raw

RFC 1/2 pycocci: add missing new line

---
 tools/pycocci | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
tools/pycocci +1 -1
index 266957bd90cd..43bb9a4dd412 100755
@@ -608,7 +608,7 @@ def _main():
         cmd = [ '--stat', patch_branch_name + ".." + smpl_branch_name ]
         diff_stat = git_diff(tree=args.target_dir, extra_args = cmd)
         if len(diff_stat) == 0:
-            logwrite('\nSmPL patch fully replaces patch series!')
+            logwrite('\nSmPL patch fully replaces patch series!\n')
         else:
             logwrite('\nDifferences found:\n\n')
             logwrite('Change directory to %s and run:\n\n\tgit diff %s..%s\n\n' % (args.target_dir, patch_branch_name, smpl_branch_name))
Signature · 1 line
2.7.2
#2 lore raw

RFC 1/2 pycocci: add missing new line

Quoted 1 line · Luis R. Rodriguez
Quoted 25 lines
---
 tools/pycocci | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
tools/pycocci +1 -1
index 266957bd90cd..43bb9a4dd412 100755
@@ -608,7 +608,7 @@ def _main():
         cmd = [ '--stat', patch_branch_name + ".." + smpl_branch_name ]
         diff_stat = git_diff(tree=args.target_dir, extra_args = cmd)
         if len(diff_stat) == 0:
-            logwrite('\nSmPL patch fully replaces patch series!')
+            logwrite('\nSmPL patch fully replaces patch series!\n')
         else:
             logwrite('\nDifferences found:\n\n')
             logwrite('Change directory to %s and run:\n\n\tgit diff %s..%s\n\n' % (args.target_dir, patch_branch_name, smpl_branch_name))
Signature · 6 lines
2.7.2

_______________________________________________
Cocci mailing list
Cocci at systeme.lip6.fr
https://systeme.lip6.fr/mailman/listinfo/cocci
#3 lore raw

RFC 2/2 tests: convert sp.cocci test to pycocci proof

In reply to Luis R. Rodriguez
---
 Makefile                            | 1 +
 scripts/test.sh                     | 8 ++++++++
 tests/{sp.cocci => 0001-sp.cocci}   | 0
 tests/0001-sp/replace-typedef.patch | 9 +++++++++
 tests/{ => code/0001-sp}/sp.c       | 0
 tests/sp.res                        | 4 ----
 6 files changed, 18 insertions(+), 4 deletions(-)
 create mode 100755 scripts/test.sh
 rename tests/{sp.cocci => 0001-sp.cocci} (100%)
 create mode 100644 tests/0001-sp/replace-typedef.patch
 rename tests/{ => code/0001-sp}/sp.c (100%)
 delete mode 100644 tests/sp.res
Makefile +1 -0
index a98eefd2d83b..312f287fd61d 100644
@@ -488,6 +488,7 @@ check: scripts/spatch
 		COCCINELLE_HOME="$$(pwd)" ./scripts/spatch --sp-file demos/hello/hello-python.cocci demos/hello/helloworld.c --very-quiet | grep -q 'Hello at: 2'; fi
 	@$(ECHO) running the test suite
 	COCCINELLE_HOME="$$(pwd)" ./scripts/spatch --testall --no-update-score-file
+	COCCINELLE_HOME="$$(pwd)" ./scripts/test.sh
 
 # -inline 0  to see all the functions in the profile.
 # Can also use the profile framework in commons/ and run your program
scripts/test.sh new file +8 -0
new file mode 100755
index 000000000000..efaaa88e67be
@@ -0,0 +1,8 @@
+#!/bin/bash
+set -e
+
+for i in tests/0001-sp.cocci; do
+	TEST=${i%%.cocci}
+	CODE_TEST=tests/code/$(basename $TEST)
+	./tools/pycocci -v -s $i $CODE_TEST
+done
tests/0001-sp.cocci renamed from tests/sp.cocci +0 -0
similarity index 100%
rename from tests/sp.cocci
rename to tests/0001-sp.cocci
tests/0001-sp/replace-typedef.patch new file +9 -0
new file mode 100644
index 000000000000..5097b94cbfca
@@ -0,0 +1,9 @@
+--- a/sp.c
++++ b/sp.c
+@@ -1,4 +1,4 @@
+-typedef struct {
++struct name {
+ 	unsigned long gcr;
+ 	struct pci_dev *pci;
+-} snd_card_als4000_t;
++};
tests/code/0001-sp/sp.c renamed from tests/sp.c +0 -0
similarity index 100%
rename from tests/sp.c
rename to tests/code/0001-sp/sp.c
tests/sp.res deleted +0 -4
deleted file mode 100644
index a97fecceb3f0..000000000000
@@ -1,4 +0,0 @@
-struct name {
-	unsigned long gcr;
-	struct pci_dev *pci;
-};
Signature · 1 line
2.7.2
#4 lore raw

cover coccinelle: extend test coverage

In reply to Luis R. Rodriguez
Quoted 5 lines

I wanted to extend tests coverage for coccinelle but noticed that tests are limited to single c file, these are attached to respective .res files, but this also means that you have a direct relationship and requirement between tests from a single C file and a respective cocci file.

Do you imagine to measure test coverage with any advanced techniques?

Quoted 2 lines

To go wild we want to be able to support really complex types of examples.

Is there any kind of target conflict between nice software demonstrations and the desire for better test procedures?

Quoted 3 lines

We can easily take advantage of this to extend Coccinelle test cases way beyond a single C file, but to more complex real life style projects.

How much would you like to improve the automatic checking of generated patches?

Regards,
Markus