support/scripts/cve.py: use fast ijson backend if available on old ijson versions

ijson < 2.5 (as available in Debian 10) use the slow python backend by default instead of the most efficient one available like modern ijson versions, significantly slowing down cve checking. E.G.:

time ./support/scripts/pkg-stats --nvd-path ~/.nvd -p avahi --html foobar.html

Goes from
174,44s user 2,11s system 99% cpu 2:58,04 total

To
93,53s user 2,00s system 98% cpu 1:36,65 total

E.G. almost 2x as fast.

As a workaround, detect when the python backend is used and try to use a more efficient one instead. Use the yajl2_cffi backend as recommended by upstream, as it is most likely to work, and print a warning (and continue) if we fail to load it.

The detection is slightly complicated by the fact that ijson.backends used to be a reference to a backend module, but is nowadays a string (without the ijson.backends prefix).

Signed-off-by Peter Korsgaard <[email protected]>

---
 support/scripts/cve.py | 6 ++++++
 1 file changed, 6 insertions(+)

support/scripts/cve.py +6 -0

index 6e97ea193f..965fc2a466 100755
@@ -28,6 +28,12 @@ import operator
 
 try:
     import ijson
+    # backend is a module in < 2.5, a string in >= 2.5
+    if 'python' in getattr(ijson.backend, '__name__', ijson.backend):
+        try:
+            import ijson.backends.yajl2_cffi as ijson
+        except ImportError:
+            sys.stderr.write('Warning: Using slow ijson python backend\n')
 except ImportError:
     sys.stderr.write("You need ijson to parse NVD for CVE check\n")
     exit(1)

Signature · 6 lines

2.20.1

_______________________________________________
buildroot mailing list
buildroot at busybox.net
http://lists.busybox.net/mailman/listinfo/buildroot

Quoted 1 line

Quoted 1 line

Quoted 1 line

Quoted 1 line

"Peter" == Peter Korsgaard <[email protected]> writes:

ijson < 2.5 (as available in Debian 10) use the slow python backend by default instead of the most efficient one available like modern ijson versions, significantly slowing down cve checking. E.G.:

time ./support/scripts/pkg-stats --nvd-path ~/.nvd -p avahi --html foobar.html

Goes from
174,44s user 2,11s system 99% cpu 2:58,04 total

To
93,53s user 2,00s system 98% cpu 1:36,65 total

E.G. almost 2x as fast.

As a workaround, detect when the python backend is used and try to use a more efficient one instead. Use the yajl2_cffi backend as recommended by upstream, as it is most likely to work, and print a warning (and continue) if we fail to load it.

The detection is slightly complicated by the fact that ijson.backends used to be a reference to a backend module, but is nowadays a string (without the ijson.backends prefix).

Signed-off-by Peter Korsgaard <[email protected]>