scripts: provide a script for checking glib symbol usage
PATCH 1/1 scripts: provide a script for checking glib symbol usage
There are frequently patches submitted for merge which use symbols only defined by newer glib versions that the max version currently permitted by QEMU (ie glib==2.22). The glib 2.32 release introduced GLIB_VERSION_MIN_REQUIRED and GLIB_VERSION_MAX_ALLOWED macros to allow apps to declare what they want. Unfortunately QEMU has a glib-compat.h file which backports certain functions from newer glib which confuses the glib version checking macros. This is effectively preventing their use in QEMU. The version check macros also don't cover all versions back to 2.22 so any checks are incomplete.
This patch takes a different approach, by providing a script that can validate that source files are only using glib functions & macros present in the minimum declared glib version, by doing static analysis of the source and comparing to a previously defined list of permitted symbols.
The list of functions/macros was extracted from the glib 2.22 HTML docs symbol index using
$ grep '<dt>' /usr/share/gtk-doc/html/glib/ix01.html | \
awk {'print $1}' | \
sed -e 's/<dt>//' -e 's/,//' | \
grep -E -i '^g_' | \
sort > scripts/glib-syms.txtThe script works by simply searching through the .c and .h files looking for anything that looks like a function or macro call, starting with a 'G_' or 'g_' prefix. This is not a 100% exact science, but with a few override lists it provides a practical/usable level of detection of mistakes.
This patch creates a 'scripts/Makefile' file which is intended to accumulate various other style checks, similar to those done by 'checkpatch.pl', but operating against the entire source tree rather than just new patches.
The check can be invoked using 'make check-style' in the top level directory.
Example output when the check fails would look like:
$ make check-style tests/test-io-task.c:151: 'g_thread_ref' symbol not provided in min glib version tests/test-io-task.c:172: 'g_thread_ref' symbol not provided in min glib version tests/test-io-task.c:216: 'g_thread_unref' symbol not provided in min glib version tests/test-io-task.c:217: 'g_thread_unref' symbol not provided in min glib version tests/test-io-task.c:260: 'g_thread_unref' symbol not provided in min glib version tests/test-io-task.c:261: 'g_thread_unref' symbol not provided in min glib version
--- Makefile | 1 + scripts/Makefile | 24 + scripts/glib-syms.pl | 131 +++++ scripts/glib-syms.txt | 1513 +++++++++++++++++++++++++++++++++++++++++++++++++ 4 files changed, 1669 insertions(+) create mode 100644 scripts/Makefile create mode 100644 scripts/glib-syms.pl create mode 100644 scripts/glib-syms.txt
Makefile +1 -0
@@ -165,6 +165,7 @@ dummy := $(call unnest-vars,, \ ifneq ($(wildcard config-host.mak),) include $(SRC_PATH)/tests/Makefile +include $(SRC_PATH)/scripts/Makefile endif all: $(DOCS) $(TOOLS) $(HELPERS-y) recurse-all modules
scripts/Makefile new file +24 -0
@@ -0,0 +1,24 @@ +# +# This makefile runs various style checks across the entire +# source tree. +# +# This is similar in concept of checkpatch.pl, but enforces +# rules across the entire codebase, not just new patches +# + +STYLE_CHECKS = \ + cs-glib-syms + +ALL_FILES = \ + $(shell git ls-tree -r HEAD . | awk '{print $$4}') + +C_CODE_FILES = $(filter %.c %.h, $(ALL_FILES)) + +check-style: $(STYLE_CHECKS) + +# Check that we only use glib symbols present in our +# minimum declared glib version +GLIB_SYMS_LIST = scripts/glib-syms.txt + +cs-glib-syms: + @perl scripts/glib-syms.pl $(GLIB_SYMS_LIST) $(C_CODE_FILES)
scripts/glib-syms.pl new file +131 -0
131 lines not on this page · raw message
scripts/glib-syms.txt new file +1513 -0
1,513 lines not on this page · raw message
Signature · 1 line
2.5.0
Quoted 20 lines · Daniel P. Berrange
scripts/Makefile new file +14
-0@@ -0,0 +1,24 @@ +# +# This makefile runs various style checks across the entire +# source tree. +# +# This is similar in concept of checkpatch.pl, but enforces +# rules across the entire codebase, not just new patches +# + +STYLE_CHECKS = \ + cs-glib-syms + +ALL_FILES = \ + $(shell git ls-tree -r HEAD . | awk '{print $$4}') +
scripts/Makefile line 24
Quoted 10 lines
scripts/Makefile new file +10
-0+C_CODE_FILES = $(filter %.c %.h, $(ALL_FILES)) + +check-style: $(STYLE_CHECKS) + +# Check that we only use glib symbols present in our +# minimum declared glib version +GLIB_SYMS_LIST = scripts/glib-syms.txt + +cs-glib-syms: + @perl scripts/glib-syms.pl $(GLIB_SYMS_LIST) $(C_CODE_FILES)
Does this need to be included, or could it be a separate Makefile invoked with e.g. make -f scripts/Makefile.style?
Quoted 52 lines
scripts/Makefile new file +52
-0+# Symbols not present in the release that we depend +# on, but which have wrappers in include/glib-compat.h +my @compatsyms = qw( + g_get_monotonic_time + + g_assert_true + g_assert_false + g_assert_null + g_assert_nonnull + g_assert_cmpmem + + g_hash_table_add + + g_cond_clear + g_cond_init + g_cond_wait_until + + g_mutex_init + g_mutex_clear + + g_thread_new + + g_private_replace + G_PRIVATE_INIT + + G_TIME_SPAN_SECOND +); + + +# Functions defined inside QEMU which are using the +# the same "g_" function name prefix as glib, so +# get mis-detected as glib symbols +my @blacklist = qw( + g_to_float64 + g_assert_no_errno + g_cclosure_new_swap + g_free_rcu + g_test_trap_subprocess + g_poll_fixed + g_list_insert_sorted_merged + G_BYTE +); + +# GObject stuff used by gtk frontend +my @gobjectsums = qw( + g_object_ref + g_object_unref + g_object_set_data + g_signal_connect + G_CALLBACK +); +
scripts/Makefile
Quoted 10 lines
scripts/Makefile new file +10
-0+# Functions defined by glib which are strangely +# missing from their docs header index +my @missingindex = qw( + g_assertion_message + g_assertion_message_expr + g_assertion_message_cmpstr + g_assertion_message_cmpnum + g_assertion_message_error +); +
Can we "parse" #.* as comments, and put these in a glib-syms-extra.txt file? Then we can hypothetically do the same tests in checkpatch.pl too.
I'm not opposing the scripts/Makefile concept, and I'm not asking you to patch checkpatch.pl; I just would like to keep the door open for that.
Paolo
Quoted 29 lines · Paolo Bonzini
Quoted 26 lines · Daniel P. Berrange
scripts/Makefile new file +20
-0@@ -0,0 +1,24 @@ +# +# This makefile runs various style checks across the entire +# source tree. +# +# This is similar in concept of checkpatch.pl, but enforces +# rules across the entire codebase, not just new patches +# + +STYLE_CHECKS = \ + cs-glib-syms + +ALL_FILES = \ + $(shell git ls-tree -r HEAD . | awk '{print $$4}') + +C_CODE_FILES = $(filter %.c %.h, $(ALL_FILES)) + +check-style: $(STYLE_CHECKS) + +# Check that we only use glib symbols present in our +# minimum declared glib version
Quoted 8 lines
scripts/Makefile line 24
Quoted 4 lines
scripts/Makefile new file +4
-0+GLIB_SYMS_LIST = scripts/glib-syms.txt + +cs-glib-syms: + @perl scripts/glib-syms.pl $(GLIB_SYMS_LIST) $(C_CODE_FILES)Does this need to be included, or could it be a separate
Makefileinvoked with e.g. make -fscripts/Makefile.style?
Any particular reason to favour that over include ? I did it this way because QEMU in general seems to be biased towards includes and not recursive make
Quoted 57 lines
Quoted 57 lines
scripts/Makefile new file +57
-0+# Symbols not present in the release that we depend +# on, but which have wrappers in include/glib-compat.h +my @compatsyms = qw( + g_get_monotonic_time + + g_assert_true + g_assert_false + g_assert_null + g_assert_nonnull + g_assert_cmpmem + + g_hash_table_add + + g_cond_clear + g_cond_init + g_cond_wait_until + + g_mutex_init + g_mutex_clear + + g_thread_new + + g_private_replace + G_PRIVATE_INIT + + G_TIME_SPAN_SECOND +); + + +# Functions defined inside QEMU which are using the +# the same "g_" function name prefix as glib, so +# get mis-detected as glib symbols +my @blacklist = qw( + g_to_float64 + g_assert_no_errno + g_cclosure_new_swap + g_free_rcu + g_test_trap_subprocess + g_poll_fixed + g_list_insert_sorted_merged + G_BYTE +); + +# GObject stuff used by gtk frontend +my @gobjectsums = qw( + g_object_ref + g_object_unref + g_object_set_data + g_signal_connect + G_CALLBACK +); + +# Functions defined by glib which are strangely +# missing from their docs header index +my @missingindex = qw( + g_assertion_message + g_assertion_message_expr
Quoted 8 lines
scripts/Makefile
Quoted 5 lines
scripts/Makefile new file +5
-0+ g_assertion_message_cmpstr + g_assertion_message_cmpnum + g_assertion_message_error +); +Can we "parse" #.* as comments, and put these in a glib-syms-extra.txt file? Then we can hypothetically do the same tests in checkpatch.pl too.
Yep, that works.
Regards,
Daniel
Signature · 4 lines
|: http://berrange.com -o- http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :|
Quoted 11 lines · Daniel P. Berrange
Quoted 7 lines
scripts/Makefile line 24
Quoted 3 lines
scripts/Makefile new file +3
-0+ +cs-glib-syms: + @perl scripts/glib-syms.pl $(GLIB_SYMS_LIST) $(C_CODE_FILES)Does this need to be included, or could it be a separate
Makefileinvoked with e.g. make -fscripts/Makefile.style?Any particular reason to favour that over include ? I did it this way because QEMU in general seems to be biased towards includes and not recursive make
That would not be recursive make, but rather a completely separate Makefile to be manually invoked with -f.
Paolo
Quoted 11 lines · * Paolo Bonzini ([email protected]) wrote:
Quoted 8 lines · Daniel P. Berrange
Quoted 7 lines
scripts/Makefile line 24
Quoted 3 lines
scripts/Makefile new file +3
-0+ +cs-glib-syms: + @perl scripts/glib-syms.pl $(GLIB_SYMS_LIST) $(C_CODE_FILES)Does this need to be included, or could it be a separate
Makefileinvoked with e.g. make -fscripts/Makefile.style?
Quoted 6 lines
Quoted 3 lines
Any particular reason to favour that over include ? I did it this way because QEMU in general seems to be biased towards includes and not recursive make
That would not be recursive make, but rather a completely separate
Makefileto be manually invoked with -f.
Hmm but wouldn't this Makefile also be a good place for small-fast style check scripts that could be included in make check ?
Dave
Quoted 2 lines
Paolo
--
Dr. David Alan Gilbert / [email protected] / Manchester, UK
Quoted 16 lines · Dr. David Alan Gilbert
Quoted 15 lines · * Paolo Bonzini ([email protected]) wrote:
Quoted 11 lines · Daniel P. Berrange
Quoted 7 lines
scripts/Makefile line 24
Quoted 3 lines
scripts/Makefile new file +3
-0+ +cs-glib-syms: + @perl scripts/glib-syms.pl $(GLIB_SYMS_LIST) $(C_CODE_FILES)Does this need to be included, or could it be a separate
Makefileinvoked with e.g. make -fscripts/Makefile.style?Any particular reason to favour that over include ? I did it this way because QEMU in general seems to be biased towards includes and not recursive make
Quoted 5 lines
Quoted 2 lines
That would not be recursive make, but rather a completely separate
Makefileto be manually invoked with -f.Hmm but wouldn't this
Makefilealso be a good place for small-fast style check scripts that could be included in make check ?
Nothing about "make check" is fast, so we could probably just wire it all into make check by default, as qtest & the block tests take some considerable time to run. IOW people who want speed will be running "make check-unit" or individual tests already.
Regards,
Daniel
Signature · 4 lines
|: http://berrange.com -o- http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :|
Quoted 20 lines · * Daniel P. Berrange ([email protected]) wrote:
Quoted 19 lines · Dr. David Alan Gilbert
Quoted 17 lines · * Paolo Bonzini ([email protected]) wrote:
Quoted 11 lines · Daniel P. Berrange
Quoted 7 lines
scripts/Makefile line 24
Quoted 3 lines
scripts/Makefile new file +3
-0+ +cs-glib-syms: + @perl scripts/glib-syms.pl $(GLIB_SYMS_LIST) $(C_CODE_FILES)Does this need to be included, or could it be a separate
Makefileinvoked with e.g. make -fscripts/Makefile.style?Any particular reason to favour that over include ? I did it this way because QEMU in general seems to be biased towards includes and not recursive make
That would not be recursive make, but rather a completely separate
Makefileto be manually invoked with -f.
Quoted 7 lines
Quoted 2 lines
Hmm but wouldn't this
Makefilealso be a good place for small-fast style check scripts that could be included in make check ?Nothing about "make check" is fast, so we could probably just wire it all into make check by default, as qtest & the block tests take some considerable time to run. IOW people who want speed will be running "make check-unit" or individual tests already.
"make check" takes under 2m on my laptop, so I take that to be reasonably fast, i.e. I'd be happy to add other small (few second) tests to it. e.g. with your Makefile included like you had done, the scripts to check ordering of some files could go in it.
Dave
Quoted 8 lines
Regards,
DanielSignature · 4 lines
|: http://berrange.com -o- http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :|
--
Dr. David Alan Gilbert / [email protected] / Manchester, UK
Quoted 2 lines · Dr. David Alan Gilbert
"make check" takes under 2m on my laptop, so I take that to be reasonably fast, i.e. I'd be happy to add other small (few second) tests to it.
Is that a 'make check' for a complete all-targets configure? Those are not fast, especially on slower hosts...
thanks
-- PMM
Quoted 7 lines · * Peter Maydell ([email protected]) wrote:
Quoted 2 lines · Dr. David Alan Gilbert
"make check" takes under 2m on my laptop, so I take that to be reasonably fast, i.e. I'd be happy to add other small (few second) tests to it.
Is that a 'make check' for a complete all-targets configure? Those are not fast, especially on slower hosts...
Ah no, that is just for x86_64-softmmu.
Dave
Quoted 3 lines
thanks
-- PMM
--
Dr. David Alan Gilbert / [email protected] / Manchester, UK
Quoted 5 lines · Dr. David Alan Gilbert
Quoted 2 lines
That would not be recursive make, but rather a completely separate
Makefileto be manually invoked with -f.Hmm but wouldn't this
Makefilealso be a good place for small-fast style check scripts that could be included in make check ?
Yes, but there is no benefit from inclusion, since these tests do not depend on anything having been built already.
Paolo
Quoted 11 lines · Paolo Bonzini
Quoted 5 lines · Dr. David Alan Gilbert
Quoted 2 lines
That would not be recursive make, but rather a completely separate
Makefileto be manually invoked with -f.Hmm but wouldn't this
Makefilealso be a good place for small-fast style check scripts that could be included in make check ?Yes, but there is no benefit from inclusion, since these tests do not depend on anything having been built already.
In particular, if it doesn't require a preceding configure or build I can just run it once as part of my merge testing rather than farming it out to be run in parallel on five machines and ten different configs :-)
thanks
-- PMM
Quoted 10 lines · Peter Maydell
Quoted 9 lines · Paolo Bonzini
Quoted 5 lines · Dr. David Alan Gilbert
Quoted 2 lines
That would not be recursive make, but rather a completely separate
Makefileto be manually invoked with -f.Hmm but wouldn't this
Makefilealso be a good place for small-fast style check scripts that could be included in make check ?
Quoted 7 lines
Quoted 2 lines
Yes, but there is no benefit from inclusion, since these tests do not depend on anything having been built already.
In particular, if it doesn't require a preceding configure or build I can just run it once as part of my merge testing rather than farming it out to be run in parallel on five machines and ten different configs :-)
Ok, yeah, that makes total sense - it certainly shouldn't depend on any configure / build step.
Regards,
Daniel
Signature · 4 lines
|: http://berrange.com -o- http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :|