Skip to contentSkip to conversation

tests: Force Python I/O encoding for check-qapi-schema

PATCH In review Eduardo Habkost 5 participants 8 messages May 6, 2019 – May 9, 2019

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 tests: Force Python I/O encoding for check-qapi-schema

test-qapi.py doesn't force a specific encoding for stderr or stdout, but the reference files used by check-qapi-schema are in UTF-8. This breaks check-qapi-schema under certain circumstances (e.g. if using the C locale and Python < 3.7).

We need to make sure test-qapi.py always generate UTF-8 output somehow. On Python 3.7+ we can do it using sys.stdout.reconfigure(...), but we need a solution that works with older Python versions.

Instead of trying a hack like reopening sys.stdout and sys.stderr, we can just tell Python to use UTF-8 for I/O encoding when running test-qapi.py. Do it by setting PYTHONIOENCODING.

---
 tests/Makefile.include | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
tests/Makefile.include +1 -1
index 7c8b9c84b2..af88ab6f8b 100644
@@ -1103,7 +1103,7 @@ check-tests/qemu-iotests-quick.sh: tests/qemu-iotests-quick.sh qemu-img$(EXESUF)
 .PHONY: $(patsubst %, check-%, $(check-qapi-schema-y))
 $(patsubst %, check-%, $(check-qapi-schema-y)): check-%.json: $(SRC_PATH)/%.json
 	$(call quiet-command, PYTHONPATH=$(SRC_PATH)/scripts \
-		$(PYTHON) $(SRC_PATH)/tests/qapi-schema/test-qapi.py \
+		PYTHONIOENCODING=utf-8 $(PYTHON) $(SRC_PATH)/tests/qapi-schema/test-qapi.py \
 		$^ >$*.test.out 2>$*.test.err; \
 		echo $$? >$*.test.exit, \
 		"TEST","$*.out")
Signature · 1 line
2.18.0.rc1.1.g3f1ff2140
Reviewed-by on v1 1/1
#1 lore raw
Quoted 10 lines · Eduardo Habkost

test-qapi.py doesn't force a specific encoding for stderr or stdout, but the reference files used by check-qapi-schema are in UTF-8. This breaks check-qapi-schema under certain circumstances (e.g. if using the C locale and Python < 3.7).

We need to make sure test-qapi.py always generate UTF-8 output somehow. On Python 3.7+ we can do it using sys.stdout.reconfigure(...), but we need a solution that works with older Python versions.

Quoted 7 lines

Instead of trying a hack like reopening sys.stdout and sys.stderr, we can just tell Python to use UTF-8 for I/O encoding when running test-qapi.py. Do it by setting PYTHONIOENCODING.

Quoted 18 lines
---
 tests/Makefile.include | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
tests/Makefile.include +1 -1
index 7c8b9c84b2..af88ab6f8b 100644
@@ -1103,7 +1103,7 @@ check-tests/qemu-iotests-quick.sh: tests/qemu-iotests-quick.sh qemu-img$(EXESUF)
 .PHONY: $(patsubst %, check-%, $(check-qapi-schema-y))
 $(patsubst %, check-%, $(check-qapi-schema-y)): check-%.json: $(SRC_PATH)/%.json
 	$(call quiet-command, PYTHONPATH=$(SRC_PATH)/scripts \
-		$(PYTHON) $(SRC_PATH)/tests/qapi-schema/test-qapi.py \
+		PYTHONIOENCODING=utf-8 $(PYTHON) $(SRC_PATH)/tests/qapi-schema/test-qapi.py \
 		$^ >$*.test.out 2>$*.test.err; \
 		echo $$? >$*.test.exit, \
 		"TEST","$*.out")
#2 lore raw
In reply to Eduardo Habkost
Quoted 26 lines · Eduardo Habkost

test-qapi.py doesn't force a specific encoding for stderr or stdout, but the reference files used by check-qapi-schema are in UTF-8. This breaks check-qapi-schema under certain circumstances (e.g. if using the C locale and Python < 3.7).

We need to make sure test-qapi.py always generate UTF-8 output somehow. On Python 3.7+ we can do it using sys.stdout.reconfigure(...), but we need a solution that works with older Python versions.

Instead of trying a hack like reopening sys.stdout and sys.stderr, we can just tell Python to use UTF-8 for I/O encoding when running test-qapi.py. Do it by setting PYTHONIOENCODING.

---
 tests/Makefile.include | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
tests/Makefile.include +0 -0
index 7c8b9c84b2..af88ab6f8b 100644

tests/Makefile.include check-tests/qemu-iotests-quick.sh: tests/qemu-iotests-quick.sh qemu-img$(EXESUF) · line 1106

Quoted 5 lines
tests/Makefile.include +1 -1
@@ -1103,7 +1103,7 @@ check-tests/qemu-iotests-quick.sh: tests/qemu-iotests-quick.sh qemu-img$(EXESUF)
 .PHONY: $(patsubst %, check-%, $(check-qapi-schema-y))
 $(patsubst %, check-%, $(check-qapi-schema-y)): check-%.json: $(SRC_PATH)/%.json
 	$(call quiet-command, PYTHONPATH=$(SRC_PATH)/scripts \
-		$(PYTHON) $(SRC_PATH)/tests/qapi-schema/test-qapi.py \
+		PYTHONIOENCODING=utf-8 $(PYTHON) $(SRC_PATH)/tests/qapi-schema/test-qapi.py \

I see PYTHONIOENCODING exists since 2.6 which is nice.

How about we actually change $(PYTHON) so that it always includes PYTHONIOENCODING=utf-8 ?

That way we avoid continuing to play whack-a-mole with more utf-8 bugs in future.

It would also let us revert this:

commit de685ae5e9a4b523513033bd6cadc8187a227170
Author: Markus Armbruster <[email protected]>
Date: Mon Jun 18 19:59:57 2018 +0200

qapi: Open files with encoding='utf-8'

which had to provide separate logic for py2 vs py3 :-(

Regards,
Daniel

Signature · 3 lines
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|
#3 lore raw
Quoted 43 lines · Daniel P. Berrangé
Quoted 26 lines · Eduardo Habkost

test-qapi.py doesn't force a specific encoding for stderr or stdout, but the reference files used by check-qapi-schema are in UTF-8. This breaks check-qapi-schema under certain circumstances (e.g. if using the C locale and Python < 3.7).

We need to make sure test-qapi.py always generate UTF-8 output somehow. On Python 3.7+ we can do it using sys.stdout.reconfigure(...), but we need a solution that works with older Python versions.

Instead of trying a hack like reopening sys.stdout and sys.stderr, we can just tell Python to use UTF-8 for I/O encoding when running test-qapi.py. Do it by setting PYTHONIOENCODING.

---
 tests/Makefile.include | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
tests/Makefile.include +0 -0
index 7c8b9c84b2..af88ab6f8b 100644

tests/Makefile.include check-tests/qemu-iotests-quick.sh: tests/qemu-iotests-quick.sh qemu-img$(EXESUF) · line 1106

Quoted 5 lines
tests/Makefile.include +1 -1
@@ -1103,7 +1103,7 @@ check-tests/qemu-iotests-quick.sh: tests/qemu-iotests-quick.sh qemu-img$(EXESUF)
 .PHONY: $(patsubst %, check-%, $(check-qapi-schema-y))
 $(patsubst %, check-%, $(check-qapi-schema-y)): check-%.json: $(SRC_PATH)/%.json
 	$(call quiet-command, PYTHONPATH=$(SRC_PATH)/scripts \
-		$(PYTHON) $(SRC_PATH)/tests/qapi-schema/test-qapi.py \
+		PYTHONIOENCODING=utf-8 $(PYTHON) $(SRC_PATH)/tests/qapi-schema/test-qapi.py \

I see PYTHONIOENCODING exists since 2.6 which is nice.

How about we actually change $(PYTHON) so that it always includes PYTHONIOENCODING=utf-8 ?

That way we avoid continuing to play whack-a-mole with more utf-8 bugs in future.

It would also let us revert this:

Quoted 7 lines

commit de685ae5e9a4b523513033bd6cadc8187a227170
Author: Markus Armbruster <[email protected]>
Date: Mon Jun 18 19:59:57 2018 +0200

qapi: Open files with encoding='utf-8'

which had to provide separate logic for py2 vs py3 :-(

Not every Python script in the QEMU tree is run by our makefiles and scripts using $(PYTHON). We need to ensure our scripts and modules won't break when run directly from the command line, too. Setting PYTHONIOENCODING everywhere would just hide these bugs from us.

Signature · 1 line
Eduardo
#4 lore raw
Quoted 44 lines · Eduardo Habkost
Quoted 43 lines · Daniel P. Berrangé
Quoted 26 lines · Eduardo Habkost

test-qapi.py doesn't force a specific encoding for stderr or stdout, but the reference files used by check-qapi-schema are in UTF-8. This breaks check-qapi-schema under certain circumstances (e.g. if using the C locale and Python < 3.7).

We need to make sure test-qapi.py always generate UTF-8 output somehow. On Python 3.7+ we can do it using sys.stdout.reconfigure(...), but we need a solution that works with older Python versions.

Instead of trying a hack like reopening sys.stdout and sys.stderr, we can just tell Python to use UTF-8 for I/O encoding when running test-qapi.py. Do it by setting PYTHONIOENCODING.

---
 tests/Makefile.include | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
tests/Makefile.include +0 -0
index 7c8b9c84b2..af88ab6f8b 100644

tests/Makefile.include check-tests/qemu-iotests-quick.sh: tests/qemu-iotests-quick.sh qemu-img$(EXESUF) · line 1106

Quoted 5 lines
tests/Makefile.include +1 -1
@@ -1103,7 +1103,7 @@ check-tests/qemu-iotests-quick.sh: tests/qemu-iotests-quick.sh qemu-img$(EXESUF)
 .PHONY: $(patsubst %, check-%, $(check-qapi-schema-y))
 $(patsubst %, check-%, $(check-qapi-schema-y)): check-%.json: $(SRC_PATH)/%.json
 	$(call quiet-command, PYTHONPATH=$(SRC_PATH)/scripts \
-		$(PYTHON) $(SRC_PATH)/tests/qapi-schema/test-qapi.py \
+		PYTHONIOENCODING=utf-8 $(PYTHON) $(SRC_PATH)/tests/qapi-schema/test-qapi.py \

I see PYTHONIOENCODING exists since 2.6 which is nice.

How about we actually change $(PYTHON) so that it always includes PYTHONIOENCODING=utf-8 ?

That way we avoid continuing to play whack-a-mole with more utf-8 bugs in future.

It would also let us revert this:

Quoted 7 lines
Quoted 7 lines

commit de685ae5e9a4b523513033bd6cadc8187a227170
Author: Markus Armbruster <[email protected]>
Date: Mon Jun 18 19:59:57 2018 +0200

qapi: Open files with encoding='utf-8'

which had to provide separate logic for py2 vs py3 :-(

The separate logic will soon be history. I'd welcome getting rid of the remainder anyway.

Quoted 5 lines

Not every Python script in the QEMU tree is run by our makefiles and scripts using $(PYTHON). We need to ensure our scripts and modules won't break when run directly from the command line, too. Setting PYTHONIOENCODING everywhere would just hide these bugs from us.

I agree for Python scripts that are meant to be run that way (assuming such scripts exist). For all the others (including all the QAPI-related scripts), I'd be quite fine with

1. Our build system runs all Python scripts with the PYTHONIOENCODING=utf-8

2. If you run a Python script yourself, you get to specify the PYTHONIOENCODING=utf-8, or use a suitable locale. Enabling UTF-8 mode with PYTHONUTF8=1 or -X utf8 could also work.

#5 lore raw
Quoted 50 lines · Markus Armbruster
Quoted 48 lines · Eduardo Habkost
Quoted 47 lines · Daniel P. Berrangé
Quoted 26 lines · Eduardo Habkost

test-qapi.py doesn't force a specific encoding for stderr or stdout, but the reference files used by check-qapi-schema are in UTF-8. This breaks check-qapi-schema under certain circumstances (e.g. if using the C locale and Python < 3.7).

We need to make sure test-qapi.py always generate UTF-8 output somehow. On Python 3.7+ we can do it using sys.stdout.reconfigure(...), but we need a solution that works with older Python versions.

Instead of trying a hack like reopening sys.stdout and sys.stderr, we can just tell Python to use UTF-8 for I/O encoding when running test-qapi.py. Do it by setting PYTHONIOENCODING.

---
 tests/Makefile.include | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
tests/Makefile.include +0 -0
index 7c8b9c84b2..af88ab6f8b 100644

tests/Makefile.include check-tests/qemu-iotests-quick.sh: tests/qemu-iotests-quick.sh qemu-img$(EXESUF) · line 1106

Quoted 5 lines
tests/Makefile.include +1 -1
@@ -1103,7 +1103,7 @@ check-tests/qemu-iotests-quick.sh: tests/qemu-iotests-quick.sh qemu-img$(EXESUF)
 .PHONY: $(patsubst %, check-%, $(check-qapi-schema-y))
 $(patsubst %, check-%, $(check-qapi-schema-y)): check-%.json: $(SRC_PATH)/%.json
 	$(call quiet-command, PYTHONPATH=$(SRC_PATH)/scripts \
-		$(PYTHON) $(SRC_PATH)/tests/qapi-schema/test-qapi.py \
+		PYTHONIOENCODING=utf-8 $(PYTHON) $(SRC_PATH)/tests/qapi-schema/test-qapi.py \

I see PYTHONIOENCODING exists since 2.6 which is nice.

How about we actually change $(PYTHON) so that it always includes PYTHONIOENCODING=utf-8 ?

That way we avoid continuing to play whack-a-mole with more utf-8 bugs in future.

It would also let us revert this:

commit de685ae5e9a4b523513033bd6cadc8187a227170
Author: Markus Armbruster <[email protected]>
Date: Mon Jun 18 19:59:57 2018 +0200

Quoted 6 lines
Quoted 3 lines
Quoted 3 lines

qapi: Open files with encoding='utf-8'

which had to provide separate logic for py2 vs py3 :-(

The separate logic will soon be history. I'd welcome getting rid of the remainder anyway.

Which remainder? Do you mean the encoding='utf-8' arguments to open()?

Quoted 9 lines
Quoted 6 lines

Not every Python script in the QEMU tree is run by our makefiles and scripts using $(PYTHON). We need to ensure our scripts and modules won't break when run directly from the command line, too. Setting PYTHONIOENCODING everywhere would just hide these bugs from us.

I agree for Python scripts that are meant to be run that way (assuming such scripts exist). [...]

All scripts inside ./scripts are meant to be run directly from the command line, aren't they?

Quoted 9 lines

[...] For all the others (including all the QAPI-related
scripts), I'd be quite fine with

1. Our build system runs all Python scripts with the PYTHONIOENCODING=utf-8

2. If you run a Python script yourself, you get to specify the PYTHONIOENCODING=utf-8, or use a suitable locale. Enabling UTF-8 mode with PYTHONUTF8=1 or -X utf8 could also work.

I'm OK if we don't actively try to fix those bugs and just expect people to set PYTHONIOENCODING. But I don't think we should reject patches that make the Python code work with non-utf8 locales if it's an easy fix.

Signature · 1 line
Eduardo
#6 lore raw
Quoted 53 lines · Eduardo Habkost
Quoted 52 lines · Markus Armbruster
Quoted 50 lines · Eduardo Habkost
Quoted 49 lines · Daniel P. Berrangé
Quoted 26 lines · Eduardo Habkost

test-qapi.py doesn't force a specific encoding for stderr or stdout, but the reference files used by check-qapi-schema are in UTF-8. This breaks check-qapi-schema under certain circumstances (e.g. if using the C locale and Python < 3.7).

We need to make sure test-qapi.py always generate UTF-8 output somehow. On Python 3.7+ we can do it using sys.stdout.reconfigure(...), but we need a solution that works with older Python versions.

Instead of trying a hack like reopening sys.stdout and sys.stderr, we can just tell Python to use UTF-8 for I/O encoding when running test-qapi.py. Do it by setting PYTHONIOENCODING.

---
 tests/Makefile.include | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
tests/Makefile.include +0 -0
index 7c8b9c84b2..af88ab6f8b 100644

tests/Makefile.include check-tests/qemu-iotests-quick.sh: tests/qemu-iotests-quick.sh qemu-img$(EXESUF) · line 1106

Quoted 5 lines
tests/Makefile.include +1 -1
@@ -1103,7 +1103,7 @@ check-tests/qemu-iotests-quick.sh: tests/qemu-iotests-quick.sh qemu-img$(EXESUF)
 .PHONY: $(patsubst %, check-%, $(check-qapi-schema-y))
 $(patsubst %, check-%, $(check-qapi-schema-y)): check-%.json: $(SRC_PATH)/%.json
 	$(call quiet-command, PYTHONPATH=$(SRC_PATH)/scripts \
-		$(PYTHON) $(SRC_PATH)/tests/qapi-schema/test-qapi.py \
+		PYTHONIOENCODING=utf-8 $(PYTHON) $(SRC_PATH)/tests/qapi-schema/test-qapi.py \

I see PYTHONIOENCODING exists since 2.6 which is nice.

How about we actually change $(PYTHON) so that it always includes PYTHONIOENCODING=utf-8 ?

That way we avoid continuing to play whack-a-mole with more utf-8 bugs in future.

It would also let us revert this:

commit de685ae5e9a4b523513033bd6cadc8187a227170
Author: Markus Armbruster <[email protected]>
Date: Mon Jun 18 19:59:57 2018 +0200

qapi: Open files with encoding='utf-8'

Quoted 7 lines
Quoted 4 lines
Quoted 1 line
Quoted 1 line

which had to provide separate logic for py2 vs py3 :-(

The separate logic will soon be history. I'd welcome getting rid of the remainder anyway.

Which remainder? Do you mean the encoding='utf-8' arguments to open()?

I'd welcome a revert the whole commit.

Quoted 11 lines
Quoted 8 lines
Quoted 5 lines

Not every Python script in the QEMU tree is run by our makefiles and scripts using $(PYTHON). We need to ensure our scripts and modules won't break when run directly from the command line, too. Setting PYTHONIOENCODING everywhere would just hide these bugs from us.

I agree for Python scripts that are meant to be run that way (assuming such scripts exist). [...]

All scripts inside ./scripts are meant to be run directly from the command line, aren't they?

Counter-example: you're welcome to run scripts/qapi-gen.py by hand for whatever purpose, but if you mess up your build that way, you're on your own.

Quoted 6 lines
Quoted 6 lines

[...] For all the others (including all the QAPI-related
scripts), I'd be quite fine with

1. Our build system runs all Python scripts with the PYTHONIOENCODING=utf-8

Quoted 8 lines
Quoted 3 lines

2. If you run a Python script yourself, you get to specify the PYTHONIOENCODING=utf-8, or use a suitable locale. Enabling UTF-8 mode with PYTHONUTF8=1 or -X utf8 could also work.

I'm OK if we don't actively try to fix those bugs and just expect people to set PYTHONIOENCODING. But I don't think we should reject patches that make the Python code work with non-utf8 locales if it's an easy fix.

I'm not going to interfere with easy fixes to code I don't maintain :)

Reviewed-by Tested-by on v1 1/1
#7 lore raw
In reply to Eduardo Habkost
Quoted 10 lines · Eduardo Habkost

test-qapi.py doesn't force a specific encoding for stderr or stdout, but the reference files used by check-qapi-schema are in UTF-8. This breaks check-qapi-schema under certain circumstances (e.g. if using the C locale and Python < 3.7).

We need to make sure test-qapi.py always generate UTF-8 output somehow. On Python 3.7+ we can do it using sys.stdout.reconfigure(...), but we need a solution that works with older Python versions.

Quoted 7 lines

Instead of trying a hack like reopening sys.stdout and sys.stderr, we can just tell Python to use UTF-8 for I/O encoding when running test-qapi.py. Do it by setting PYTHONIOENCODING.

Quoted 9 lines
---
 tests/Makefile.include | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
tests/Makefile.include +0 -0
index 7c8b9c84b2..af88ab6f8b 100644

tests/Makefile.include check-tests/qemu-iotests-quick.sh: tests/qemu-iotests-quick.sh qemu-img$(EXESUF) · line 1109

Quoted 8 lines
tests/Makefile.include +1 -1
@@ -1103,7 +1103,7 @@ check-tests/qemu-iotests-quick.sh: tests/qemu-iotests-quick.sh qemu-img$(EXESUF)
 .PHONY: $(patsubst %, check-%, $(check-qapi-schema-y))
 $(patsubst %, check-%, $(check-qapi-schema-y)): check-%.json: $(SRC_PATH)/%.json
 	$(call quiet-command, PYTHONPATH=$(SRC_PATH)/scripts \
-		$(PYTHON) $(SRC_PATH)/tests/qapi-schema/test-qapi.py \
+		PYTHONIOENCODING=utf-8 $(PYTHON) $(SRC_PATH)/tests/qapi-schema/test-qapi.py \
 		$^ >$*.test.out 2>$*.test.err; \
 		echo $$? >$*.test.exit, \
 		"TEST","$*.out")

--
Alex Bennée