xen: Build fixes related to Python3
v2 cover xen: Build fixes related to Python3
Patch 3 is a fix for a problem reported on IRC. It is a very-nice-to-have considering our attempt to make Xen 4.13 Py3-clean.
While testing patch 3, it became apparent that XSM/Flask isn't Py3-clean, and this is a blocker. It is addressed in patch 1.
Patch 2 addresses a bug spotted by Gitlab while testing v1 of this series. It isn't strictly a Py3 bug, but is a build system robustness fix.
Andrew Cooper (3):
xen/flask: Drop the gen-policy.py script xen/banner: Drop the fig-to-oct.py script xen/build: Automatically locate a suitable python interpreter
xen/Makefile | 10 +++++++++- xen/tools/fig-to-oct.py | 18 ------------------ xen/xsm/flask/Makefile | 6 ++---- xen/xsm/flask/flask-policy.S | 20 ++++++++++++++++++++ xen/xsm/flask/gen-policy.py | 23 ----------------------- 5 files changed, 31 insertions(+), 46 deletions(-) delete mode 100644 xen/tools/fig-to-oct.py create mode 100644 xen/xsm/flask/flask-policy.S delete mode 100644 xen/xsm/flask/gen-policy.py
Signature · 7 lines
2.11.0 _______________________________________________ Xen-devel mailing list [email protected] https://lists.xenproject.org/mailman/listinfo/xen-devel
PATCH v2 1/3 xen/flask: Drop the gen-policy.py script
The script is Python 2 specific, and fails with string/binary issues with Python 3:
Traceback (most recent call last):
File "gen-policy.py", line 14, in <module>
for char in sys.stdin.read():
File "/usr/lib/python3.5/codecs.py", line 321, in decode
(result, consumed) = self._buffer_decode(data, self.errors, final)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x8c in position 0: invalid start byte
Fixing the script to be compatible isn't hard, but using python here is wasteful. Drop the script entirely, and write a short flask-policy.S instead.
---
CC: Daniel De Graaf <[email protected]>
CC: Juergen Gross <[email protected]>
v2:
* Fix tabs vs spaces issues
For 4.13. This is a blocker to our intent to by Py3-clean in this release.
Discovered entirely accidently when testing the final patch.
--- xen/xsm/flask/Makefile | 6 ++---- xen/xsm/flask/flask-policy.S | 20 ++++++++++++++++++++ xen/xsm/flask/gen-policy.py | 23 ----------------------- 3 files changed, 22 insertions(+), 27 deletions(-) create mode 100644 xen/xsm/flask/flask-policy.S delete mode 100644 xen/xsm/flask/gen-policy.py
xen/xsm/flask/Makefile +2 -4
@@ -27,7 +27,8 @@ $(FLASK_H_FILES): $(FLASK_H_DEPEND) $(AV_H_FILES): $(AV_H_DEPEND) $(CONFIG_SHELL) policy/mkaccess_vector.sh $(AWK) $(AV_H_DEPEND) -obj-$(CONFIG_XSM_FLASK_POLICY) += policy.o +obj-bin-$(CONFIG_XSM_FLASK_POLICY) += flask-policy.o +flask-policy.o: policy.bin FLASK_BUILD_DIR := $(CURDIR) POLICY_SRC := $(FLASK_BUILD_DIR)/xenpolicy-$(XEN_FULLVERSION)
@@ -36,9 +37,6 @@ policy.bin: FORCE $(MAKE) -f $(XEN_ROOT)/tools/flask/policy/Makefile.common -C $(XEN_ROOT)/tools/flask/policy FLASK_BUILD_DIR=$(FLASK_BUILD_DIR) cmp -s $(POLICY_SRC) $@ || cp $(POLICY_SRC) $@ -policy.c: policy.bin gen-policy.py - $(PYTHON) gen-policy.py < $< > $@ - .PHONY: clean clean:: rm -f $(ALL_H_FILES) *.o $(DEPS_RM) policy.* $(POLICY_SRC)
xen/xsm/flask/flask-policy.S new file +20 -0
@@ -0,0 +1,20 @@ + .section .init.rodata, "a", @progbits + +/* const unsigned char xsm_flask_init_policy[] __initconst */ + .align 4 + .global xsm_flask_init_policy +xsm_flask_init_policy: + .incbin "policy.bin" +.Lend: + + .type xsm_flask_init_policy, @object + .size xsm_flask_init_policy, . - xsm_flask_init_policy + +/* const unsigned int __initconst xsm_flask_init_policy_size */ + .align 4 + .global xsm_flask_init_policy_size +xsm_flask_init_policy_size: + .long .Lend - xsm_flask_init_policy + + .type xsm_flask_init_policy_size, @object + .size xsm_flask_init_policy_size, . - xsm_flask_init_policy_size
xen/xsm/flask/gen-policy.py deleted +0 -23
@@ -1,23 +0,0 @@ -#!/usr/bin/env python -import sys - -policy_size = 0 - -sys.stdout.write(""" -/* This file is autogenerated by gen_policy.py */ -#include <xen/init.h> -#include <xsm/xsm.h> - -const unsigned char xsm_flask_init_policy[] __initconst = { -""") - -for char in sys.stdin.read(): - sys.stdout.write(" 0x%02x," % ord(char)) - policy_size = policy_size + 1 - if policy_size % 13 == 0: - sys.stdout.write("\n") - -sys.stdout.write(""" -}; -const unsigned int __initconst xsm_flask_init_policy_size = %d; -""" % policy_size)
Signature · 7 lines
2.11.0 _______________________________________________ Xen-devel mailing list [email protected] https://lists.xenproject.org/mailman/listinfo/xen-devel
PATCH v2 2/3 xen/banner: Drop the fig-to-oct.py script
In reply to Andrew CooperThe script is 664 rather than 775, so the banner conversion doesn't actually work if $(PYTHON) is empty:
/bin/sh: tools/fig-to-oct.py: Permission denied
make[3]: *** [include/xen/compile.h] Error 126
make[3]: Leaving directory `/builds/xen-project/people/andyhhp/xen/xen'
Fixing this is easy, but using python here is wasteful. compile.h doesn't need XEN_BANNER rendering in octal, and text is much more simple to handle. Replace fig-to-oct.py with a sed oneliner.
While changing this logic, take the opportunity to optimise the banner space (and time on the serial port) by dropping trailing whitespace, which is 84 characters for current staging.
---
CC: George Dunlap <[email protected]>
CC: Ian Jackson <[email protected]>
CC: Jan Beulich <[email protected]>
CC: Konrad Rzeszutek Wilk <[email protected]>
CC: Stefano Stabellini <[email protected]>
CC: Wei Liu <[email protected]>
CC: Julien Grall <[email protected]>
CC: Juergen Gross <[email protected]>
v2:
* New
Spotted by Gitlab CI, caused by which not being present in some of the CentOS containers. While this is more of a container bug than anything else, it does highlight that the build ought to cope.
--- xen/Makefile | 5 ++++- xen/tools/fig-to-oct.py | 18 ------------------ 2 files changed, 4 insertions(+), 19 deletions(-) delete mode 100644 xen/tools/fig-to-oct.py
xen/Makefile +4 -1
@@ -163,6 +163,9 @@ delete-unfresh-files: @mv -f [email protected] $@ # compile.h contains dynamic build info. Rebuilt on every 'make' invocation. +# +# For .banner sed-ary, strip trailing whitespace, escape backslashes, and wrap +# each line in '"...\n" \' to become a valid C string include/xen/compile.h: include/xen/compile.h.in .banner @sed -e 's/@@date@@/$(XEN_BUILD_DATE)/g' \ -e 's/@@time@@/$(XEN_BUILD_TIME)/g' \
@@ -176,7 +179,7 @@ include/xen/compile.h: include/xen/compile.h.in .banner -e 's!@@changeset@@!$(shell tools/scmversion $(XEN_ROOT) || echo "unavailable")!g' \ < include/xen/compile.h.in > [email protected] @cat .banner - @$(PYTHON) tools/fig-to-oct.py < .banner >> [email protected] + @sed -e 's_[ ]*$$__' -e 's_\\_\\\\_g' -e 's_\(.*\)_"\1\\n" \\_' < .banner >> [email protected] @mv -f [email protected] $@ include/asm-$(TARGET_ARCH)/asm-offsets.h: arch/$(TARGET_ARCH)/asm-offsets.s
xen/tools/fig-to-oct.py deleted +0 -18
@@ -1,18 +0,0 @@ -#!/usr/bin/env python -import sys - -chars_per_line = 18 -chars_so_far = 0 - -sys.stdout.write('"') - -for char in sys.stdin.read(): - - sys.stdout.write("\\%03o" % ord(char)) - chars_so_far = chars_so_far + 1 - - if chars_so_far == chars_per_line: - chars_so_far = 0 - sys.stdout.write('" \\\n"') - -sys.stdout.write('"\n')
Signature · 7 lines
2.11.0 _______________________________________________ Xen-devel mailing list [email protected] https://lists.xenproject.org/mailman/listinfo/xen-devel
PATCH v2 3/3 xen/build: Automatically locate a suitable python interpreter
In reply to Andrew CooperNeeding to pass PYTHON=python3 into hypervisor builds is irritating and unnecessary. Locate a suitable interpreter automatically, defaulting to Py3 if it is available.
---
CC: George Dunlap <[email protected]>
CC: Ian Jackson <[email protected]>
CC: Jan Beulich <[email protected]>
CC: Konrad Rzeszutek Wilk <[email protected]>
CC: Stefano Stabellini <[email protected]>
CC: Wei Liu <[email protected]>
CC: Julien Grall <[email protected]>
CC: Steven Haigh <[email protected]>
CC: Juergen Gross <[email protected]>
v2:
* Cope with which not being present in the system.
* Only evaulate the shell command one, rather than once per $(PTHON) usage
For 4.13. This is a very-nice-to-have WRT our Py3-clean intention.
--- xen/Makefile | 5 +++++ 1 file changed, 5 insertions(+)
xen/Makefile +5 -0
@@ -13,6 +13,11 @@ export XEN_BUILD_TIME ?= $(shell LC_ALL=C date +%T) export XEN_BUILD_HOST ?= $(shell hostname) export XEN_CONFIG_EXPERT ?= n +# Best effort attempt to find a python interpreter, defaulting to Python 3 if +# available. Fall back to just `python` if `which` is nowhere to be found. +PYTHON_INTERPRETER := $(word 1,$(shell which python3 python python2 2>/dev/null) python) +export PYTHON ?= $(PYTHON_INTERPRETER) + export BASEDIR := $(CURDIR) export XEN_ROOT := $(BASEDIR)/..
Signature · 7 lines
2.11.0 _______________________________________________ Xen-devel mailing list [email protected] https://lists.xenproject.org/mailman/listinfo/xen-devel