Skip to contentSkip to conversation

tools: Python 3 compatibility

RFC PATCH Superseded Wei Liu 5 participants 27 messages Mar 5, 2019 – Mar 6, 2019

v1 of 4 Superseded Latest is v4 →

lore
Share & tools
lore

Wei Liu

author
#0 lore raw

cover tools: Python 3 compatibility

This series makes tools build with Python 3.

Compile test only with 2.7 and 3.5 thus far, hence the RFC. This should be able to give people some idea what sort of work is involved.

You will also need Andrew's "tools/xen-foreign: Update python scripts to be Py3 compatible".

Wei.

Wei Liu (4):
build/m4: make python_devel.m4 work with both python 2 and 3
libxl: make python scripts work with python 2 and 3
pygrub: convert python files with 2to3
pygrub: make it build with python 3

 m4/python_devel.m4                 | 27 ++++++-----
 tools/configure                    | 34 ++++++++------
 tools/libxl/gentest.py             |  4 +-
 tools/libxl/gentypes.py            | 12 ++---
 tools/libxl/idl.py                 | 21 ++++-----
 tools/pygrub/src/ExtLinuxConf.py   | 16 +++----
 tools/pygrub/src/GrubConf.py       | 36 +++++++--------
 tools/pygrub/src/LiloConf.py       | 16 +++----
 tools/pygrub/src/fsimage/fsimage.c | 92 ++++++++++++++++++++++++++++++++++----
 9 files changed, 175 insertions(+), 83 deletions(-)
Signature · 7 lines
2.11.0


_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
author
#1 lore raw

RFC PATCH 1/4 build/m4: make python_devel.m4 work with both python 2 and 3

Do the following:

1. Change the form of "print".
2. Check for ABI flags -- this is complicated because it is only introduced in 3.2.
3. Fix library name in AC_CHECK_LIB.
4. Remove other-libs in AC_CHECK_LIB.

---
I doubt the non python-pkg branch works, because the paths generated seem rather off. It definitely doesn't work on my machine, but I don't know how other systems could possibly be configured before the existence of python-config.

---
 m4/python_devel.m4 | 27 ++++++++++++++++-----------
 tools/configure    | 34 ++++++++++++++++++++--------------
 2 files changed, 36 insertions(+), 25 deletions(-)
m4/python_devel.m4 +16 -11
index 05ea4ef7e2..1e2f41b6aa 100644
@@ -2,37 +2,42 @@ AC_DEFUN([AX_CHECK_PYTHON_DEVEL], [
 ac_previous_cppflags=$CPPFLAGS
 ac_previous_ldflags=$LDFLAGS
 ac_python_version=`$PYTHON -c 'import distutils.sysconfig; \
-    print distutils.sysconfig.get_config_var("VERSION")'`
+    print(distutils.sysconfig.get_config_var("VERSION"))'`
+ac_python_abiflags=
 AC_PATH_PROG([pyconfig], [$PYTHON-config], [no])
 AS_IF([test x"$pyconfig" = x"no"], [
     dnl For those that don't have python-config
     CPPFLAGS="$CFLAGS `$PYTHON -c 'import distutils.sysconfig; \
         print "-I" + distutils.sysconfig.get_config_var("INCLUDEPY")'`"
     CPPFLAGS="$CPPFLAGS `$PYTHON -c 'import distutils.sysconfig; \
-        print distutils.sysconfig.get_config_var("CFLAGS")'`"
+        print(distutils.sysconfig.get_config_var("CFLAGS"))'`"
     PYTHON_LIBS="$LDFLAGS `$PYTHON -c 'import distutils.sysconfig; \
-        print distutils.sysconfig.get_config_var("LIBS")'`"
+        print(distutils.sysconfig.get_config_var("LIBS"))'`"
     PYTHON_LIBS="$LDFLAGS `$PYTHON -c 'import distutils.sysconfig; \
-        print distutils.sysconfig.get_config_var("SYSLIBS")'`"
+        print(distutils.sysconfig.get_config_var("SYSLIBS"))'`"
     LDFLAGS="$LDFLAGS `$PYTHON -c 'import distutils.sysconfig; \
-        print "-L" + distutils.sysconfig.get_python_lib(plat_specific=1,\
-        standard_lib=1) + "/config"'`"
+        print("-L" + distutils.sysconfig.get_python_lib(plat_specific=1,\
+        standard_lib=1) + "/config")'`"
     LDFLAGS="$LDFLAGS `$PYTHON -c 'import distutils.sysconfig; \
-        print distutils.sysconfig.get_config_var("LINKFORSHARED")'`"
+        print(distutils.sysconfig.get_config_var("LINKFORSHARED"))'`"
     LDFLAGS="$LDFLAGS `$PYTHON -c 'import distutils.sysconfig; \
-        print distutils.sysconfig.get_config_var("LDFLAGS")'`"
+        print(distutils.sysconfig.get_config_var("LDFLAGS"))'`"
 ], [
     dnl If python-config is found use it
     CPPFLAGS="$CFLAGS `$PYTHON-config --cflags`"
     LDFLAGS="$LDFLAGS `$PYTHON-config --ldflags`"
     PYTHON_LIBS="$LIBS `$PYTHON-config --libs`"
+    abiflags="`$PYTHON-config --abiflags`"
+    if test "$?" == "0"
+    then
+        ac_python_abiflags="$abiflags"
+    fi
 ])
 
 AC_CHECK_HEADER([Python.h], [],
     [AC_MSG_ERROR([Unable to find Python development headers])],)
-AC_CHECK_LIB(python$ac_python_version, PyArg_ParseTuple, [],
-    [AC_MSG_ERROR([Unable to find a suitable python development library])],
-    [$PYTHON_LIBS])
+AC_CHECK_LIB(python$ac_python_version$ac_python_abiflags, PyArg_ParseTuple, [],
+    [AC_MSG_ERROR([Unable to find a suitable python development library])])
 CPPFLAGS=$ac_previous_cppflags
 LDFLAGS=$ac_previous_ldflags
 ])
tools/configure +20 -14
index acc857510e..8cc216b669 100755
@@ -7419,7 +7419,8 @@ if test "$cross_compiling" != yes; then :
 ac_previous_cppflags=$CPPFLAGS
 ac_previous_ldflags=$LDFLAGS
 ac_python_version=`$PYTHON -c 'import distutils.sysconfig; \
-    print distutils.sysconfig.get_config_var("VERSION")'`
+    print(distutils.sysconfig.get_config_var("VERSION"))'`
+ac_python_abiflags=
 # Extract the first word of "$PYTHON-config", so it can be a program name with args.
 set dummy $PYTHON-config; ac_word=$2
 { $as_echo "$as_me:${as_lineno-$LINENO}: checking for $ac_word" >&5
@@ -7466,24 +7467,29 @@ if test x"$pyconfig" = x"no"; then :
         CPPFLAGS="$CFLAGS `$PYTHON -c 'import distutils.sysconfig; \
         print "-I" + distutils.sysconfig.get_config_var("INCLUDEPY")'`"
     CPPFLAGS="$CPPFLAGS `$PYTHON -c 'import distutils.sysconfig; \
-        print distutils.sysconfig.get_config_var("CFLAGS")'`"
+        print(distutils.sysconfig.get_config_var("CFLAGS"))'`"
     PYTHON_LIBS="$LDFLAGS `$PYTHON -c 'import distutils.sysconfig; \
-        print distutils.sysconfig.get_config_var("LIBS")'`"
+        print(distutils.sysconfig.get_config_var("LIBS"))'`"
     PYTHON_LIBS="$LDFLAGS `$PYTHON -c 'import distutils.sysconfig; \
-        print distutils.sysconfig.get_config_var("SYSLIBS")'`"
+        print(distutils.sysconfig.get_config_var("SYSLIBS"))'`"
     LDFLAGS="$LDFLAGS `$PYTHON -c 'import distutils.sysconfig; \
-        print "-L" + distutils.sysconfig.get_python_lib(plat_specific=1,\
-        standard_lib=1) + "/config"'`"
+        print("-L" + distutils.sysconfig.get_python_lib(plat_specific=1,\
+        standard_lib=1) + "/config")'`"
     LDFLAGS="$LDFLAGS `$PYTHON -c 'import distutils.sysconfig; \
-        print distutils.sysconfig.get_config_var("LINKFORSHARED")'`"
+        print(distutils.sysconfig.get_config_var("LINKFORSHARED"))'`"
     LDFLAGS="$LDFLAGS `$PYTHON -c 'import distutils.sysconfig; \
-        print distutils.sysconfig.get_config_var("LDFLAGS")'`"
+        print(distutils.sysconfig.get_config_var("LDFLAGS"))'`"
 
 else
 
         CPPFLAGS="$CFLAGS `$PYTHON-config --cflags`"
     LDFLAGS="$LDFLAGS `$PYTHON-config --ldflags`"
     PYTHON_LIBS="$LIBS `$PYTHON-config --libs`"
+    abiflags="`$PYTHON-config --abiflags`"
+    if test "$?" == "0"
+    then
+        ac_python_abiflags="$abiflags"
+    fi
 
 fi
 
@@ -7495,14 +7501,14 @@ else
 fi
 
 
-as_ac_Lib=`$as_echo "ac_cv_lib_python$ac_python_version''_PyArg_ParseTuple" | $as_tr_sh`
-{ $as_echo "$as_me:${as_lineno-$LINENO}: checking for PyArg_ParseTuple in -lpython$ac_python_version" >&5
-$as_echo_n "checking for PyArg_ParseTuple in -lpython$ac_python_version... " >&6; }
+as_ac_Lib=`$as_echo "ac_cv_lib_python$ac_python_version$ac_python_abiflags''_PyArg_ParseTuple" | $as_tr_sh`
+{ $as_echo "$as_me:${as_lineno-$LINENO}: checking for PyArg_ParseTuple in -lpython$ac_python_version$ac_python_abiflags" >&5
+$as_echo_n "checking for PyArg_ParseTuple in -lpython$ac_python_version$ac_python_abiflags... " >&6; }
 if eval \${$as_ac_Lib+:} false; then :
   $as_echo_n "(cached) " >&6
 else
   ac_check_lib_save_LIBS=$LIBS
-LIBS="-lpython$ac_python_version $PYTHON_LIBS $LIBS"
+LIBS="-lpython$ac_python_version$ac_python_abiflags  $LIBS"
 cat confdefs.h - <<_ACEOF >conftest.$ac_ext
 /* end confdefs.h.  */
 
@@ -7535,10 +7541,10 @@ eval ac_res=\$$as_ac_Lib
 $as_echo "$ac_res" >&6; }
 if eval test \"x\$"$as_ac_Lib"\" = x"yes"; then :
   cat >>confdefs.h <<_ACEOF
-#define `$as_echo "HAVE_LIBpython$ac_python_version" | $as_tr_cpp` 1
+#define `$as_echo "HAVE_LIBpython$ac_python_version$ac_python_abiflags" | $as_tr_cpp` 1
 _ACEOF
 
-  LIBS="-lpython$ac_python_version $LIBS"
+  LIBS="-lpython$ac_python_version$ac_python_abiflags $LIBS"
 
 else
   as_fn_error $? "Unable to find a suitable python development library" "$LINENO" 5
Signature · 7 lines
2.11.0


_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
#2 lore raw

Re: [PATCH for-next RFC 1/4] build/m4: make python_devel.m4 work with both python 2 and 3

Quoted 5 lines · Wei Liu

Do the following:

1. Change the form of "print".
2. Check for ABI flags -- this is complicated because it is only introduced in 3.2.

Is this a recommanded way of doing this? I may have a better way of fixing this macro, see below.

Quoted 2 lines

3. Fix library name in AC_CHECK_LIB.
4. Remove other-libs in AC_CHECK_LIB.

Why did you remove the other libs? Also, with this change, PYTHON_LIBS isn't used anywhere anymore, and can be removed.

Quoted 57 lines

---
I doubt the non python-pkg branch works, because the paths generated seem rather off. It definitely doesn't work on my machine, but I don't know how other systems could possibly be configured before the existence of python-config.

---
 m4/python_devel.m4 | 27 ++++++++++++++++-----------
 tools/configure    | 34 ++++++++++++++++++++--------------
 2 files changed, 36 insertions(+), 25 deletions(-)
m4/python_devel.m4 +13 -8
index 05ea4ef7e2..1e2f41b6aa 100644
@@ -2,37 +2,42 @@ AC_DEFUN([AX_CHECK_PYTHON_DEVEL], [
 ac_previous_cppflags=$CPPFLAGS
 ac_previous_ldflags=$LDFLAGS
 ac_python_version=`$PYTHON -c 'import distutils.sysconfig; \
-    print distutils.sysconfig.get_config_var("VERSION")'`
+    print(distutils.sysconfig.get_config_var("VERSION"))'`
+ac_python_abiflags=
 AC_PATH_PROG([pyconfig], [$PYTHON-config], [no])
 AS_IF([test x"$pyconfig" = x"no"], [
     dnl For those that don't have python-config
     CPPFLAGS="$CFLAGS `$PYTHON -c 'import distutils.sysconfig; \
         print "-I" + distutils.sysconfig.get_config_var("INCLUDEPY")'`"
     CPPFLAGS="$CPPFLAGS `$PYTHON -c 'import distutils.sysconfig; \
-        print distutils.sysconfig.get_config_var("CFLAGS")'`"
+        print(distutils.sysconfig.get_config_var("CFLAGS"))'`"
     PYTHON_LIBS="$LDFLAGS `$PYTHON -c 'import distutils.sysconfig; \
-        print distutils.sysconfig.get_config_var("LIBS")'`"
+        print(distutils.sysconfig.get_config_var("LIBS"))'`"
     PYTHON_LIBS="$LDFLAGS `$PYTHON -c 'import distutils.sysconfig; \
-        print distutils.sysconfig.get_config_var("SYSLIBS")'`"
+        print(distutils.sysconfig.get_config_var("SYSLIBS"))'`"
     LDFLAGS="$LDFLAGS `$PYTHON -c 'import distutils.sysconfig; \
-        print "-L" + distutils.sysconfig.get_python_lib(plat_specific=1,\
-        standard_lib=1) + "/config"'`"
+        print("-L" + distutils.sysconfig.get_python_lib(plat_specific=1,\
+        standard_lib=1) + "/config")'`"
     LDFLAGS="$LDFLAGS `$PYTHON -c 'import distutils.sysconfig; \
-        print distutils.sysconfig.get_config_var("LINKFORSHARED")'`"
+        print(distutils.sysconfig.get_config_var("LINKFORSHARED"))'`"
     LDFLAGS="$LDFLAGS `$PYTHON -c 'import distutils.sysconfig; \
-        print distutils.sysconfig.get_config_var("LDFLAGS")'`"
+        print(distutils.sysconfig.get_config_var("LDFLAGS"))'`"
 ], [
     dnl If python-config is found use it
     CPPFLAGS="$CFLAGS `$PYTHON-config --cflags`"
     LDFLAGS="$LDFLAGS `$PYTHON-config --ldflags`"
     PYTHON_LIBS="$LIBS `$PYTHON-config --libs`"
+    abiflags="`$PYTHON-config --abiflags`"
+    if test "$?" == "0"
+    then
+        ac_python_abiflags="$abiflags"

m4/python_devel.m4 AC_DEFUN([AX_CHECK_PYTHON_DEVEL], [ · line 40

Quoted 10 lines
m4/python_devel.m4 +3 -3
+    fi
 ])
 
 AC_CHECK_HEADER([Python.h], [],
     [AC_MSG_ERROR([Unable to find Python development headers])],)
-AC_CHECK_LIB(python$ac_python_version, PyArg_ParseTuple, [],
-    [AC_MSG_ERROR([Unable to find a suitable python development library])],
-    [$PYTHON_LIBS])
+AC_CHECK_LIB(python$ac_python_version$ac_python_abiflags, PyArg_ParseTuple, [],
+    [AC_MSG_ERROR([Unable to find a suitable python development library])])

So, AC_CHECK_LIB seems to only be used to check if PyArg_ParseTuple exist, and requires as argument the name of the lib which is now complicated.

But, AC_CHECK_LIB do test compilation using the LDFLAGS which already contain the python lib we want, so instead, we could only do the part of the jobs that we need:

AC_LINK_IFELSE([AC_LANG_CALL([], [PyArg_ParseTuple])], [],
[AC_MSG_ERROR([Unable to find a suitable python development library])])

That generate a main.c with PyArg_ParseTuple() call like AC_CHECK_LIB do, and do build/link. If that fails, throw an error.

That avoid to use the --abiflags, which we don't need.

What do you thing?

Some progress message can be added, similair to AC_CHECK_LIB:
AC_MSG_CHECKING([for PyArg_ParseTuple])
and [AC_MSG_RESULT([yes])] on success.

(I think AC_CHECK_LIB would also update $LIBS, but I don't think our build system is using that.)

Quoted 3 lines
m4/python_devel.m4 +0 -0
 CPPFLAGS=$ac_previous_cppflags
 LDFLAGS=$ac_previous_ldflags
 ])
Signature · 6 lines
Anthony PERARD

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
author
#3 lore raw

Re: [PATCH for-next RFC 1/4] build/m4: make python_devel.m4 work with both python 2 and 3

Quoted 9 lines · Anthony PERARD
Quoted 5 lines · Wei Liu

Do the following:

1. Change the form of "print".
2. Check for ABI flags -- this is complicated because it is only introduced in 3.2.

Is this a recommanded way of doing this? I may have a better way of fixing this macro, see below.

Nope. I figured this out myself. There doesn't seem to be a recommended way to do it.

Quoted 7 lines
Quoted 3 lines

3. Fix library name in AC_CHECK_LIB.
4. Remove other-libs in AC_CHECK_LIB.

Why did you remove the other libs? Also, with this change, PYTHON_LIBS isn't used anywhere anymore, and can be removed.

Because

"The other-libraries argument should be limited to cases where it is desirable to test for one library in the presence of another that is not already in LIBS."

Its original usage is wrong. At least in the python-config case, PYTHON_LIBS is "-lpython2.7 -lpthread -ldl -lutil -lm". ./configure already knows how to use LDFLAGS as far as I can tell from the log.

Yes. PYTHON_LIBS can be removed. I think.

Quoted 79 lines
Quoted 57 lines

---
I doubt the non python-pkg branch works, because the paths generated seem rather off. It definitely doesn't work on my machine, but I don't know how other systems could possibly be configured before the existence of python-config.

---
 m4/python_devel.m4 | 27 ++++++++++++++++-----------
 tools/configure    | 34 ++++++++++++++++++++--------------
 2 files changed, 36 insertions(+), 25 deletions(-)
m4/python_devel.m4 +13 -8
index 05ea4ef7e2..1e2f41b6aa 100644
@@ -2,37 +2,42 @@ AC_DEFUN([AX_CHECK_PYTHON_DEVEL], [
 ac_previous_cppflags=$CPPFLAGS
 ac_previous_ldflags=$LDFLAGS
 ac_python_version=`$PYTHON -c 'import distutils.sysconfig; \
-    print distutils.sysconfig.get_config_var("VERSION")'`
+    print(distutils.sysconfig.get_config_var("VERSION"))'`
+ac_python_abiflags=
 AC_PATH_PROG([pyconfig], [$PYTHON-config], [no])
 AS_IF([test x"$pyconfig" = x"no"], [
     dnl For those that don't have python-config
     CPPFLAGS="$CFLAGS `$PYTHON -c 'import distutils.sysconfig; \
         print "-I" + distutils.sysconfig.get_config_var("INCLUDEPY")'`"
     CPPFLAGS="$CPPFLAGS `$PYTHON -c 'import distutils.sysconfig; \
-        print distutils.sysconfig.get_config_var("CFLAGS")'`"
+        print(distutils.sysconfig.get_config_var("CFLAGS"))'`"
     PYTHON_LIBS="$LDFLAGS `$PYTHON -c 'import distutils.sysconfig; \
-        print distutils.sysconfig.get_config_var("LIBS")'`"
+        print(distutils.sysconfig.get_config_var("LIBS"))'`"
     PYTHON_LIBS="$LDFLAGS `$PYTHON -c 'import distutils.sysconfig; \
-        print distutils.sysconfig.get_config_var("SYSLIBS")'`"
+        print(distutils.sysconfig.get_config_var("SYSLIBS"))'`"
     LDFLAGS="$LDFLAGS `$PYTHON -c 'import distutils.sysconfig; \
-        print "-L" + distutils.sysconfig.get_python_lib(plat_specific=1,\
-        standard_lib=1) + "/config"'`"
+        print("-L" + distutils.sysconfig.get_python_lib(plat_specific=1,\
+        standard_lib=1) + "/config")'`"
     LDFLAGS="$LDFLAGS `$PYTHON -c 'import distutils.sysconfig; \
-        print distutils.sysconfig.get_config_var("LINKFORSHARED")'`"
+        print(distutils.sysconfig.get_config_var("LINKFORSHARED"))'`"
     LDFLAGS="$LDFLAGS `$PYTHON -c 'import distutils.sysconfig; \
-        print distutils.sysconfig.get_config_var("LDFLAGS")'`"
+        print(distutils.sysconfig.get_config_var("LDFLAGS"))'`"
 ], [
     dnl If python-config is found use it
     CPPFLAGS="$CFLAGS `$PYTHON-config --cflags`"
     LDFLAGS="$LDFLAGS `$PYTHON-config --ldflags`"
     PYTHON_LIBS="$LIBS `$PYTHON-config --libs`"
+    abiflags="`$PYTHON-config --abiflags`"
+    if test "$?" == "0"
+    then
+        ac_python_abiflags="$abiflags"

m4/python_devel.m4 AC_DEFUN([AX_CHECK_PYTHON_DEVEL], [ · line 40

Quoted 10 lines
m4/python_devel.m4 +3 -3
+    fi
 ])
 
 AC_CHECK_HEADER([Python.h], [],
     [AC_MSG_ERROR([Unable to find Python development headers])],)
-AC_CHECK_LIB(python$ac_python_version, PyArg_ParseTuple, [],
-    [AC_MSG_ERROR([Unable to find a suitable python development library])],
-    [$PYTHON_LIBS])
+AC_CHECK_LIB(python$ac_python_version$ac_python_abiflags, PyArg_ParseTuple, [],
+    [AC_MSG_ERROR([Unable to find a suitable python development library])])

So, AC_CHECK_LIB seems to only be used to check if PyArg_ParseTuple exist, and requires as argument the name of the lib which is now complicated.

But, AC_CHECK_LIB do test compilation using the LDFLAGS which already contain the python lib we want, so instead, we could only do the part of the jobs that we need:

AC_LINK_IFELSE([AC_LANG_CALL([], [PyArg_ParseTuple])], [],
[AC_MSG_ERROR([Unable to find a suitable python development library])])

Quoted 6 lines

That generate a main.c with PyArg_ParseTuple() call like AC_CHECK_LIB do, and do build/link. If that fails, throw an error.

That avoid to use the --abiflags, which we don't need.

What do you thing?

Definitely looks better. Let me try this.

Quoted 8 lines

Some progress message can be added, similair to AC_CHECK_LIB:
AC_MSG_CHECKING([for PyArg_ParseTuple])
and [AC_MSG_RESULT([yes])] on success.

(I think AC_CHECK_LIB would also update $LIBS, but I don't think our build system is using that.)

Yes it updates LIBS and yes xen build system doesn't need it.

Wei.

Quoted 6 lines
Quoted 3 lines
m4/python_devel.m4 +0 -0
 CPPFLAGS=$ac_previous_cppflags
 LDFLAGS=$ac_previous_ldflags
 ])
Signature · 1 line
Anthony PERARD

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

#4 lore raw

Re: [PATCH for-next RFC 1/4] build/m4: make python_devel.m4 work with both python 2 and 3

Quoted 18 lines · Wei Liu
Quoted 17 lines · Anthony PERARD

m4/python_devel.m4 AC_DEFUN([AX_CHECK_PYTHON_DEVEL], [ · line 40

Quoted 5 lines
m4/python_devel.m4 +2 -3
-AC_CHECK_LIB(python$ac_python_version, PyArg_ParseTuple, [],
-    [AC_MSG_ERROR([Unable to find a suitable python development library])],
-    [$PYTHON_LIBS])
+AC_CHECK_LIB(python$ac_python_version$ac_python_abiflags, PyArg_ParseTuple, [],
+    [AC_MSG_ERROR([Unable to find a suitable python development library])])

So, AC_CHECK_LIB seems to only be used to check if PyArg_ParseTuple exist, and requires as argument the name of the lib which is now complicated.

But, AC_CHECK_LIB do test compilation using the LDFLAGS which already contain the python lib we want, so instead, we could only do the part of the jobs that we need:

AC_LINK_IFELSE([AC_LANG_CALL([], [PyArg_ParseTuple])], [],
[AC_MSG_ERROR([Unable to find a suitable python development library])])

Quoted 8 lines
Quoted 6 lines

That generate a main.c with PyArg_ParseTuple() call like AC_CHECK_LIB do, and do build/link. If that fails, throw an error.

That avoid to use the --abiflags, which we don't need.

What do you thing?

Definitely looks better. Let me try this.

Actually, I think I just found better, same things but using a macro that already do what we want:

AC_CHECK_FUNC([PyArg_ParseTuple], [],
[AC_MSG_ERROR([Unable to find a suitable python development library])])

Signature · 6 lines
Anthony PERARD

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
author
#5 lore raw

Re: [PATCH for-next RFC 1/4] build/m4: make python_devel.m4 work with both python 2 and 3

Quoted 26 lines · Anthony PERARD
Quoted 25 lines · Wei Liu
Quoted 23 lines · Anthony PERARD

m4/python_devel.m4 AC_DEFUN([AX_CHECK_PYTHON_DEVEL], [ · line 40

Quoted 5 lines
m4/python_devel.m4 +2 -3
-AC_CHECK_LIB(python$ac_python_version, PyArg_ParseTuple, [],
-    [AC_MSG_ERROR([Unable to find a suitable python development library])],
-    [$PYTHON_LIBS])
+AC_CHECK_LIB(python$ac_python_version$ac_python_abiflags, PyArg_ParseTuple, [],
+    [AC_MSG_ERROR([Unable to find a suitable python development library])])

So, AC_CHECK_LIB seems to only be used to check if PyArg_ParseTuple exist, and requires as argument the name of the lib which is now complicated.

But, AC_CHECK_LIB do test compilation using the LDFLAGS which already contain the python lib we want, so instead, we could only do the part of the jobs that we need:

AC_LINK_IFELSE([AC_LANG_CALL([], [PyArg_ParseTuple])], [],
[AC_MSG_ERROR([Unable to find a suitable python development library])])

That generate a main.c with PyArg_ParseTuple() call like AC_CHECK_LIB do, and do build/link. If that fails, throw an error.

That avoid to use the --abiflags, which we don't need.

What do you thing?

Quoted 8 lines
Quoted 1 line

Definitely looks better. Let me try this.

Actually, I think I just found better, same things but using a macro that already do what we want:

AC_CHECK_FUNC([PyArg_ParseTuple], [],
[AC_MSG_ERROR([Unable to find a suitable python development library])])

This works. Thank you.

Wei.

Quoted 2 lines
Signature · 1 line
Anthony PERARD

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

author
#6 lore raw

RFC PATCH 2/4 libxl: make python scripts work with python 2 and 3

In reply to Wei Liu

All scripts are transformed by 2to3.

The only addition is "from __future__ import print_function" so that print("BLAH", file=sys.stderr) can work.

https://python-future.org/compatible_idioms.html

Tested with 2.7 and 3.5.

---
I don't have environment to test 2.4 -- it is almost 15 years old. We may want to consider bumping the minimum requirement to 2.7?

---
 tools/libxl/gentest.py  |  4 +++-
 tools/libxl/gentypes.py | 12 +++++++-----
 tools/libxl/idl.py      | 21 +++++++++++----------
 3 files changed, 21 insertions(+), 16 deletions(-)
tools/libxl/gentest.py +3 -1
index 989959fc68..1cc7eebc82 100644
@@ -1,5 +1,7 @@
 #!/usr/bin/python
 
+from __future__ import print_function
+
 import os
 import sys
 import re
@@ -86,7 +88,7 @@ def gen_rand_init(ty, v, indent = "    ", parent = None):
 
 if __name__ == '__main__':
     if len(sys.argv) < 3:
-        print >>sys.stderr, "Usage: gentest.py <idl> <implementation>"
+        print("Usage: gentest.py <idl> <implementation>", file=sys.stderr)
         sys.exit(1)
 
     random.seed(os.getenv('LIBXL_TESTIDL_SEED'))
tools/libxl/gentypes.py +7 -5
index 88e5c5f30e..6417c9dd8c 100644
@@ -1,5 +1,7 @@
 #!/usr/bin/python
 
+from __future__ import print_function
+
 import sys
 import re
 
@@ -576,14 +578,14 @@ def libxl_C_enum_from_string(ty, str, e, indent = "    "):
 
 if __name__ == '__main__':
     if len(sys.argv) != 6:
-        print >>sys.stderr, "Usage: gentypes.py <idl> <header> <header-private> <header-json> <implementation>"
+        print("Usage: gentypes.py <idl> <header> <header-private> <header-json> <implementation>", file=sys.stderr)
         sys.exit(1)
 
     (_, idlname, header, header_private, header_json, impl) = sys.argv
 
     (builtins,types) = idl.parse(idlname)
 
-    print "outputting libxl type definitions to %s" % header
+    print("outputting libxl type definitions to %s" % header)
 
     f = open(header, "w")
 
@@ -633,7 +635,7 @@ if __name__ == '__main__':
     f.write("""#endif /* %s */\n""" % (header_define))
     f.close()
 
-    print "outputting libxl JSON definitions to %s" % header_json
+    print("outputting libxl JSON definitions to %s" % header_json)
 
     f = open(header_json, "w")
 
@@ -657,7 +659,7 @@ if __name__ == '__main__':
     f.write("""#endif /* %s */\n""" % header_json_define)
     f.close()
 
-    print "outputting libxl type internal definitions to %s" % header_private
+    print("outputting libxl type internal definitions to %s" % header_private)
 
     f = open(header_private, "w")
 
@@ -683,7 +685,7 @@ if __name__ == '__main__':
     f.write("""#endif /* %s */\n""" % header_json_define)
     f.close()
 
-    print "outputting libxl type implementations to %s" % impl
+    print("outputting libxl type implementations to %s" % impl)
 
     f = open(impl, "w")
     f.write("""
tools/libxl/idl.py +11 -10
index 2a7f3c44fe..27bd1053a6 100644
@@ -1,3 +1,5 @@
+from __future__ import print_function
+
 import sys
 
 PASS_BY_VALUE = 1
@@ -11,7 +13,7 @@ DIR_BOTH = 3
 _default_namespace = ""
 def namespace(s):
     if type(s) != str:
-        raise TypeError, "Require a string for the default namespace."
+        raise TypeError("Require a string for the default namespace.")
     global _default_namespace
     _default_namespace = s
 
@@ -269,7 +271,7 @@ class KeyedUnion(Aggregate):
         if not isinstance(keyvar_type, Enumeration):
             raise ValueError
 
-        kv_kwargs = dict([(x.lstrip('keyvar_'),y) for (x,y) in kwargs.items() if x.startswith('keyvar_')])
+        kv_kwargs = dict([(x.lstrip('keyvar_'),y) for (x,y) in list(kwargs.items()) if x.startswith('keyvar_')])
         
         self.keyvar = Field(keyvar_type, keyvar_name, **kv_kwargs)
 
@@ -315,7 +317,7 @@ class Array(Type):
         kwargs.setdefault('json_parse_type', 'JSON_ARRAY')
         Type.__init__(self, namespace=elem_type.namespace, typename=elem_type.rawname + " *", **kwargs)
 
-        lv_kwargs = dict([(x.lstrip('lenvar_'),y) for (x,y) in kwargs.items() if x.startswith('lenvar_')])
+        lv_kwargs = dict([(x.lstrip('lenvar_'),y) for (x,y) in list(kwargs.items()) if x.startswith('lenvar_')])
 
         self.lenvar = Field(integer, lenvar_name, **lv_kwargs)
         self.elem_type = elem_type
@@ -346,12 +348,12 @@ class OrderedDict(dict):
         return [(x,self[x]) for x in self.__ordered]
 
 def parse(f):
-    print >>sys.stderr, "Parsing %s" % f
+    print("Parsing %s" % f, file=sys.stderr)
 
     globs = {}
     locs = OrderedDict()
 
-    for n,t in globals().items():
+    for n,t in list(globals().items()):
         if isinstance(t, Type):
             globs[n] = t
         elif isinstance(t,type(object)) and issubclass(t, Type):
@@ -362,11 +364,10 @@ def parse(f):
             globs[n] = t
 
     try:
-        execfile(f, globs, locs)
-    except SyntaxError,e:
-        raise SyntaxError, \
-              "Errors were found at line %d while processing %s:\n\t%s"\
-              %(e.lineno,f,e.text)
+        exec(compile(open(f).read(), f, 'exec'), globs, locs)
+    except SyntaxError as e:
+        raise SyntaxError("Errors were found at line %d while processing %s:\n\t%s"\
+              %(e.lineno,f,e.text))
 
     types = [t for t in locs.ordered_values() if isinstance(t,Type)]
 
Signature · 7 lines
2.11.0


_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
#7 lore raw

Re: [PATCH for-next RFC 2/4] libxl: make python scripts work with python 2 and 3

Quoted 5 lines · Wei Liu

All scripts are transformed by 2to3.

The only addition is "from __future__ import print_function" so that print("BLAH", file=sys.stderr) can work.

Quoted 8 lines

https://python-future.org/compatible_idioms.html

Tested with 2.7 and 3.5.

---
I don't have environment to test 2.4 -- it is almost 15 years old. We may want to consider bumping the minimum requirement to 2.7?

The compatible way to do this is  sys.stderr.write(msg + "\n") and using print() without the future import.

tools/libxl/idl.py class KeyedUnion(Aggregate): · line 274

Quoted 6 lines
tools/libxl/idl.py +1 -1
@@ -269,7 +271,7 @@ class KeyedUnion(Aggregate):
         if not isinstance(keyvar_type, Enumeration):
             raise ValueError
 
-        kv_kwargs = dict([(x.lstrip('keyvar_'),y) for (x,y) in kwargs.items() if x.startswith('keyvar_')])
+        kv_kwargs = dict([(x.lstrip('keyvar_'),y) for (x,y) in list(kwargs.items()) if x.startswith('keyvar_')])

This shouldn't need changing.  List comprehensions are one of the few uses of .items() which is compatible with older versions of python IIRC.

tools/libxl/idl.py def parse(f): · line 368

Quoted 11 lines
tools/libxl/idl.py +2 -5
@@ -362,11 +364,10 @@ def parse(f):
             globs[n] = t
 
     try:
-        execfile(f, globs, locs)
-    except SyntaxError,e:
-        raise SyntaxError, \
-              "Errors were found at line %d while processing %s:\n\t%s"\
-              %(e.lineno,f,e.text)
+        exec(compile(open(f).read(), f, 'exec'), globs, locs)
+    except SyntaxError as e:

This is the only really awkward bit, and isn't Py 2.4 compatible.

The only option here to retain pre 2.6 compatibility is:

try:
    ...
except SyntaxError:
    _, e = sys.exc_info()[:2]
    ...

~Andrew

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

author
#8 lore raw

Re: [PATCH for-next RFC 2/4] libxl: make python scripts work with python 2 and 3

Quoted 42 lines · Andrew Cooper
Quoted 5 lines · Wei Liu

All scripts are transformed by 2to3.

The only addition is "from __future__ import print_function" so that print("BLAH", file=sys.stderr) can work.

Quoted 8 lines

https://python-future.org/compatible_idioms.html

Tested with 2.7 and 3.5.

---
I don't have environment to test 2.4 -- it is almost 15 years old. We may want to consider bumping the minimum requirement to 2.7?

The compatible way to do this is  sys.stderr.write(msg + "\n") and using print() without the future import.

tools/libxl/idl.py class KeyedUnion(Aggregate): · line 274

Quoted 6 lines
tools/libxl/idl.py +1 -1
@@ -269,7 +271,7 @@ class KeyedUnion(Aggregate):
         if not isinstance(keyvar_type, Enumeration):
             raise ValueError
 
-        kv_kwargs = dict([(x.lstrip('keyvar_'),y) for (x,y) in kwargs.items() if x.startswith('keyvar_')])
+        kv_kwargs = dict([(x.lstrip('keyvar_'),y) for (x,y) in list(kwargs.items()) if x.startswith('keyvar_')])

This shouldn't need changing.  List comprehensions are one of the few uses of .items() which is compatible with older versions of python IIRC.

tools/libxl/idl.py def parse(f): · line 368

Quoted 11 lines
tools/libxl/idl.py +2 -5
@@ -362,11 +364,10 @@ def parse(f):
             globs[n] = t
 
     try:
-        execfile(f, globs, locs)
-    except SyntaxError,e:
-        raise SyntaxError, \
-              "Errors were found at line %d while processing %s:\n\t%s"\
-              %(e.lineno,f,e.text)
+        exec(compile(open(f).read(), f, 'exec'), globs, locs)
+    except SyntaxError as e:

This is the only really awkward bit, and isn't Py 2.4 compatible.

Quoted 7 lines

The only option here to retain pre 2.6 compatibility is:

try:
    ...
except SyntaxError:
    _, e = sys.exc_info()[:2]
    ...

Since we will bump python requirement to 2.6, I think the transformation made by 2to3 should be fine.

Wei.

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

author
#9 lore raw

RFC PATCH 3/4 pygrub: convert python files with 2to3

In reply to Wei Liu

---
Not sure this works with python 2.4, but it should work with 2.7 since the changes look more or less in the same vein as the changes in libxl.

The conversion of the import is interesting. This definitely needs some testing.

---
 tools/pygrub/src/ExtLinuxConf.py | 16 ++++++++--------
 tools/pygrub/src/GrubConf.py     | 36 ++++++++++++++++++------------------
 tools/pygrub/src/LiloConf.py     | 16 ++++++++--------
 3 files changed, 34 insertions(+), 34 deletions(-)
tools/pygrub/src/ExtLinuxConf.py +8 -8
index d1789bf020..60da960c4b 100644
@@ -12,7 +12,7 @@
 
 import sys, re, os
 import logging
-import GrubConf
+from . import GrubConf
 
 class ExtLinuxImage(object):
     def __init__(self, lines, path):
@@ -32,7 +32,7 @@ class ExtLinuxImage(object):
         self.lines = []
         self.path = path
         self.root = ""
-        map(self.set_from_line, lines)
+        list(map(self.set_from_line, lines))
 
     def set_from_line(self, line, replace = None):
         (com, arg) = GrubConf.grub_exact_split(line, 2)
@@ -67,7 +67,7 @@ class ExtLinuxImage(object):
                         setattr(self, "initrd", a.replace("initrd=", ""))
                         arg = arg.replace(a, "")
 
-        if com is not None and self.commands.has_key(com):
+        if com is not None and com in self.commands:
             if self.commands[com] is not None:
                 setattr(self, self.commands[com], re.sub('^"(.+)"$', r"\1", arg.strip()))
             else:
@@ -136,7 +136,7 @@ class ExtLinuxConfigFile(object):
     def parse(self, buf = None):
         if buf is None:
             if self.filename is None:
-                raise ValueError, "No config file defined to parse!"
+                raise ValueError("No config file defined to parse!")
 
             f = open(self.filename, 'r')
             lines = f.readlines()
@@ -167,7 +167,7 @@ class ExtLinuxConfigFile(object):
 
             (com, arg) = GrubConf.grub_exact_split(l, 2)
             com = com.lower()
-            if self.commands.has_key(com):
+            if com in self.commands:
                 if self.commands[com] is not None:
                     setattr(self, self.commands[com], arg.strip())
                 else:
@@ -207,8 +207,8 @@ class ExtLinuxConfigFile(object):
         
 if __name__ == "__main__":
     if len(sys.argv) < 2:
-        raise RuntimeError, "Need a configuration file to read"
+        raise RuntimeError("Need a configuration file to read")
     g = ExtLinuxConfigFile(sys.argv[1])
     for i in g.images:
-        print i
-    print g.default
+        print(i)
+    print(g.default)
tools/pygrub/src/GrubConf.py +18 -18
index dc810d55cb..17f6e686cc 100644
@@ -44,7 +44,7 @@ def get_path(s):
         return (None, s)
     idx = s.find(')')
     if idx == -1:
-        raise ValueError, "Unable to find matching ')'"
+        raise ValueError("Unable to find matching ')'")
     d = s[:idx]
     return (GrubDiskPart(d), s[idx + 1:])
 
@@ -100,7 +100,7 @@ class _GrubImage(object):
                 "  initrd: %s\n" %(self.title, self.root, self.kernel,
                                    self.args, self.initrd))
     def _parse(self, lines):
-        map(self.set_from_line, lines)
+        list(map(self.set_from_line, lines))
 
     def reset(self, lines):
         self._root = self._initrd = self._kernel = self._args = None
@@ -141,7 +141,7 @@ class GrubImage(_GrubImage):
     def set_from_line(self, line, replace = None):
         (com, arg) = grub_exact_split(line, 2)
 
-        if self.commands.has_key(com):
+        if com in self.commands:
             if self.commands[com] is not None:
                 setattr(self, self.commands[com], arg.strip())
             else:
@@ -177,7 +177,7 @@ class _GrubConfigFile(object):
             self.parse()
 
     def parse(self, buf = None):
-        raise RuntimeError, "unimplemented parse function"   
+        raise RuntimeError("unimplemented parse function")   
 
     def hasPasswordAccess(self):
         return self.passwordAccess
@@ -201,7 +201,7 @@ class _GrubConfigFile(object):
                 import crypt
                 if crypt.crypt(password, pwd[1]) == pwd[1]:
                     return True
-            except Exception, e:
+            except Exception as e:
                 self.passExc = "Can't verify password: %s" % str(e)
                 return False
 
@@ -213,7 +213,7 @@ class _GrubConfigFile(object):
 
     def set(self, line):
         (com, arg) = grub_exact_split(line, 2)
-        if self.commands.has_key(com):
+        if com in self.commands:
             if self.commands[com] is not None:
                 setattr(self, self.commands[com], arg.strip())
             else:
@@ -233,7 +233,7 @@ class _GrubConfigFile(object):
             self._default = val
 
         if self._default < 0:
-            raise ValueError, "default must be positive number"
+            raise ValueError("default must be positive number")
     default = property(_get_default, _set_default)
 
     def set_splash(self, val):
@@ -265,7 +265,7 @@ class GrubConfigFile(_GrubConfigFile):
     def parse(self, buf = None):
         if buf is None:
             if self.filename is None:
-                raise ValueError, "No config file defined to parse!"
+                raise ValueError("No config file defined to parse!")
 
             f = open(self.filename, 'r')
             lines = f.readlines()
@@ -296,7 +296,7 @@ class GrubConfigFile(_GrubConfigFile):
                 continue
 
             (com, arg) = grub_exact_split(l, 2)
-            if self.commands.has_key(com):
+            if com in self.commands:
                 if self.commands[com] is not None:
                     setattr(self, self.commands[com], arg.strip())
                 else:
@@ -328,7 +328,7 @@ class Grub2Image(_GrubImage):
         if com == "set":
             (com,arg) = grub2_handle_set(arg)
             
-        if self.commands.has_key(com):
+        if com in self.commands:
             if self.commands[com] is not None:
                 setattr(self, self.commands[com], arg.strip())
             else:
@@ -364,7 +364,7 @@ class Grub2ConfigFile(_GrubConfigFile):
     def parse(self, buf = None):
         if buf is None:
             if self.filename is None:
-                raise ValueError, "No config file defined to parse!"
+                raise ValueError("No config file defined to parse!")
 
             f = open(self.filename, 'r')
             lines = f.readlines()
@@ -398,7 +398,7 @@ class Grub2ConfigFile(_GrubConfigFile):
             title_match = re.match('^menuentry ["\'](.*?)["\'] (.*){', l)
             if title_match:
                 if img is not None:
-                    raise RuntimeError, "syntax error: cannot nest menuentry (%d %s)" % (len(img),img)
+                    raise RuntimeError("syntax error: cannot nest menuentry (%d %s)" % (len(img),img))
                 img = []
                 title = title_match.group(1)
                 continue
@@ -413,7 +413,7 @@ class Grub2ConfigFile(_GrubConfigFile):
                         menu_level -= 1
                         continue
                     else:
-                        raise RuntimeError, "syntax error: closing brace without menuentry"
+                        raise RuntimeError("syntax error: closing brace without menuentry")
 
                 self.add_image(Grub2Image(title, img))
                 img = None
@@ -428,7 +428,7 @@ class Grub2ConfigFile(_GrubConfigFile):
             if com == "set":
                 (com,arg) = grub2_handle_set(arg)
                 
-            if self.commands.has_key(com):
+            if com in self.commands:
                 if self.commands[com] is not None:
                     arg_strip = arg.strip()
                     if arg_strip == "${saved_entry}" or arg_strip == "${next_entry}":
@@ -443,7 +443,7 @@ class Grub2ConfigFile(_GrubConfigFile):
                 logging.warning("Unknown directive %s" %(com,))
             
         if img is not None:
-            raise RuntimeError, "syntax error: end of file with open menuentry(%d %s)" % (len(img),img)
+            raise RuntimeError("syntax error: end of file with open menuentry(%d %s)" % (len(img),img))
 
         if self.hasPassword():
             self.setPasswordAccess(False)
@@ -462,12 +462,12 @@ class Grub2ConfigFile(_GrubConfigFile):
         
 if __name__ == "__main__":
     if len(sys.argv) < 3:
-        raise RuntimeError, "Need a grub version (\"grub\" or \"grub2\") and a grub.conf or grub.cfg to read"
+        raise RuntimeError("Need a grub version (\"grub\" or \"grub2\") and a grub.conf or grub.cfg to read")
     if sys.argv[1] == "grub":
         g = GrubConfigFile(sys.argv[2])
     elif sys.argv[1] == "grub2":
         g = Grub2ConfigFile(sys.argv[2])
     else:
-        raise RuntimeError, "Unknown config type %s" % sys.argv[1]
+        raise RuntimeError("Unknown config type %s" % sys.argv[1])
     for i in g.images:
-        print i #, i.title, i.root, i.kernel, i.args, i.initrd
+        print(i) #, i.title, i.root, i.kernel, i.args, i.initrd
tools/pygrub/src/LiloConf.py +8 -8
index 2cb649f115..8c080e697c 100644
@@ -4,7 +4,7 @@
 
 import sys, re, os
 import logging
-import GrubConf
+from . import GrubConf
 
 class LiloImage(object):
     def __init__(self, lines, path):
@@ -24,12 +24,12 @@ class LiloImage(object):
         self.lines = []
         self.path = path
         self.root = ""
-        map(self.set_from_line, lines)
+        list(map(self.set_from_line, lines))
 
     def set_from_line(self, line, replace = None):
         (com, arg) = GrubConf.grub_exact_split(line, 2)
 
-        if self.commands.has_key(com):
+        if com in self.commands:
             if self.commands[com] is not None:
                 setattr(self, self.commands[com], re.sub('^"(.+)"$', r"\1", arg.strip()))
             else:
@@ -97,7 +97,7 @@ class LiloConfigFile(object):
     def parse(self, buf = None):
         if buf is None:
             if self.filename is None:
-                raise ValueError, "No config file defined to parse!"
+                raise ValueError("No config file defined to parse!")
 
             f = open(self.filename, 'r')
             lines = f.readlines()
@@ -127,7 +127,7 @@ class LiloConfigFile(object):
                 continue
 
             (com, arg) = GrubConf.grub_exact_split(l, 2)
-            if self.commands.has_key(com):
+            if com in self.commands:
                 if self.commands[com] is not None:
                     setattr(self, self.commands[com], arg.strip())
                 else:
@@ -170,8 +170,8 @@ class LiloConfigFile(object):
 
 if __name__ == "__main__":
     if len(sys.argv) < 2:
-        raise RuntimeError, "Need a lilo.conf to read"
+        raise RuntimeError("Need a lilo.conf to read")
     g = LiloConfigFile(sys.argv[1])
     for i in g.images:
-        print i #, i.title, i.root, i.kernel, i.args, i.initrd
-    print g.default
+        print(i) #, i.title, i.root, i.kernel, i.args, i.initrd
+    print(g.default)
Signature · 7 lines
2.11.0


_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
#10 lore raw

Re: [PATCH for-next RFC 3/4] pygrub: convert python files with 2to3

Quoted 19 lines · Wei Liu

---
Not sure this works with python 2.4, but it should work with 2.7 since the changes look more or less in the same vein as the changes in libxl.

The conversion of the import is interesting. This definitely needs some testing.

---
 tools/pygrub/src/ExtLinuxConf.py | 16 ++++++++--------
 tools/pygrub/src/GrubConf.py     | 36 ++++++++++++++++++------------------
 tools/pygrub/src/LiloConf.py     | 16 ++++++++--------
 3 files changed, 34 insertions(+), 34 deletions(-)
tools/pygrub/src/ExtLinuxConf.py +0 -0
index d1789bf020..60da960c4b 100644

tools/pygrub/src/ExtLinuxConf.py line 15

Quoted 5 lines
tools/pygrub/src/ExtLinuxConf.py +1 -1
@@ -12,7 +12,7 @@
 
 import sys, re, os
 import logging
-import GrubConf
+from . import GrubConf

Relative imports definitely don't exist in Py 2.4

tools/pygrub/src/ExtLinuxConf.py class ExtLinuxImage(object): · line 35

Quoted 9 lines
tools/pygrub/src/ExtLinuxConf.py +1 -1
 class ExtLinuxImage(object):
     def __init__(self, lines, path):
@@ -32,7 +32,7 @@ class ExtLinuxImage(object):
         self.lines = []
         self.path = path
         self.root = ""
-        map(self.set_from_line, lines)
+        list(map(self.set_from_line, lines))

This an abuse of map() in the first place, but the automatic transformation makes the result even more confusing.

Instead, I'd rewrite the code as

for line in lines:
    self.set_from_line(line)

which makes it clear what is actually going on.

~Andrew

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

author
#11 lore raw

Re: [PATCH for-next RFC 3/4] pygrub: convert python files with 2to3

Quoted 29 lines · Andrew Cooper
Quoted 19 lines · Wei Liu

---
Not sure this works with python 2.4, but it should work with 2.7 since the changes look more or less in the same vein as the changes in libxl.

The conversion of the import is interesting. This definitely needs some testing.

---
 tools/pygrub/src/ExtLinuxConf.py | 16 ++++++++--------
 tools/pygrub/src/GrubConf.py     | 36 ++++++++++++++++++------------------
 tools/pygrub/src/LiloConf.py     | 16 ++++++++--------
 3 files changed, 34 insertions(+), 34 deletions(-)
tools/pygrub/src/ExtLinuxConf.py +0 -0
index d1789bf020..60da960c4b 100644

tools/pygrub/src/ExtLinuxConf.py line 15

Quoted 5 lines
tools/pygrub/src/ExtLinuxConf.py +1 -1
@@ -12,7 +12,7 @@
 
 import sys, re, os
 import logging
-import GrubConf
+from . import GrubConf

Relative imports definitely don't exist in Py 2.4

Quoted 1 line
Quoted 11 lines

tools/pygrub/src/ExtLinuxConf.py class ExtLinuxImage(object): · line 35

Quoted 8 lines

class ExtLinuxImage(object):

     def __init__(self, lines, path):
tools/pygrub/src/ExtLinuxConf.py +1 -1
@@ -32,7 +32,7 @@ class ExtLinuxImage(object):
         self.lines = []
         self.path = path
         self.root = ""
-        map(self.set_from_line, lines)
+        list(map(self.set_from_line, lines))

This an abuse of map() in the first place, but the automatic transformation makes the result even more confusing.

Right. I tried to find the justification for this transformation but the document doesn't provide that.

I will drop this and the relative import -- I _think_ the original code should still work with python 3.

Wei.

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

#12 lore raw

Re: [PATCH for-next RFC 3/4] pygrub: convert python files with 2to3

Quoted 39 lines · Wei Liu
Quoted 28 lines · Andrew Cooper
Quoted 19 lines · Wei Liu

---
Not sure this works with python 2.4, but it should work with 2.7 since the changes look more or less in the same vein as the changes in libxl.

The conversion of the import is interesting. This definitely needs some testing.

---
 tools/pygrub/src/ExtLinuxConf.py | 16 ++++++++--------
 tools/pygrub/src/GrubConf.py     | 36 ++++++++++++++++++------------------
 tools/pygrub/src/LiloConf.py     | 16 ++++++++--------
 3 files changed, 34 insertions(+), 34 deletions(-)
tools/pygrub/src/ExtLinuxConf.py +0 -0
index d1789bf020..60da960c4b 100644

tools/pygrub/src/ExtLinuxConf.py line 15

Quoted 5 lines
tools/pygrub/src/ExtLinuxConf.py +1 -1
@@ -12,7 +12,7 @@
 
 import sys, re, os
 import logging
-import GrubConf
+from . import GrubConf

Relative imports definitely don't exist in Py 2.4

Quoted 1 line
Quoted 10 lines

tools/pygrub/src/ExtLinuxConf.py class ExtLinuxImage(object): · line 35

Quoted 8 lines

class ExtLinuxImage(object):

     def __init__(self, lines, path):
tools/pygrub/src/ExtLinuxConf.py +1 -1
@@ -32,7 +32,7 @@ class ExtLinuxImage(object):
         self.lines = []
         self.path = path
         self.root = ""
-        map(self.set_from_line, lines)
+        list(map(self.set_from_line, lines))

This an abuse of map() in the first place, but the automatic transformation makes the result even more confusing.

Quoted 2 lines

Right. I tried to find the justification for this transformation but the document doesn't provide that.

The expected use of map is in the form:

x = map(fn, y)

which would leave x as a list in Py2, and a generator in Py3.

In most code, wrapping map with list() is the correct transformation to make, because a) a lot of code written for Py2 expects it to be a list and b) you cant programmatically evaluate whether leaving it in its generator form is safe in context.

For this piece of code (and the other similar examples), map() is not the correct construct to use in the first place, and probably wants fixing for clarity alone, irrespective of the Py3 transformation.

~Andrew

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

author
#13 lore raw

Re: [PATCH for-next RFC 3/4] pygrub: convert python files with 2to3

Quoted 40 lines · Andrew Cooper
Quoted 39 lines · Wei Liu
Quoted 38 lines · Andrew Cooper
Quoted 19 lines · Wei Liu

---
Not sure this works with python 2.4, but it should work with 2.7 since the changes look more or less in the same vein as the changes in libxl.

The conversion of the import is interesting. This definitely needs some testing.

---
 tools/pygrub/src/ExtLinuxConf.py | 16 ++++++++--------
 tools/pygrub/src/GrubConf.py     | 36 ++++++++++++++++++------------------
 tools/pygrub/src/LiloConf.py     | 16 ++++++++--------
 3 files changed, 34 insertions(+), 34 deletions(-)
tools/pygrub/src/ExtLinuxConf.py +0 -0
index d1789bf020..60da960c4b 100644

tools/pygrub/src/ExtLinuxConf.py line 15

Quoted 5 lines
tools/pygrub/src/ExtLinuxConf.py +1 -1
@@ -12,7 +12,7 @@
 
 import sys, re, os
 import logging
-import GrubConf
+from . import GrubConf

Relative imports definitely don't exist in Py 2.4

tools/pygrub/src/ExtLinuxConf.py class ExtLinuxImage(object): · line 35

Quoted 9 lines
tools/pygrub/src/ExtLinuxConf.py +1 -1
 class ExtLinuxImage(object):
     def __init__(self, lines, path):
@@ -32,7 +32,7 @@ class ExtLinuxImage(object):
         self.lines = []
         self.path = path
         self.root = ""
-        map(self.set_from_line, lines)
+        list(map(self.set_from_line, lines))

This an abuse of map() in the first place, but the automatic transformation makes the result even more confusing.

Quoted 8 lines
Quoted 2 lines

Right. I tried to find the justification for this transformation but the document doesn't provide that.

The expected use of map is in the form:

x = map(fn, y)

which would leave x as a list in Py2, and a generator in Py3.

Oh, so we should indeed force it to evaluate.

Quoted 10 lines

In most code, wrapping map with list() is the correct transformation to make, because a) a lot of code written for Py2 expects it to be a list and b) you cant programmatically evaluate whether leaving it in its generator form is safe in context.

For this piece of code (and the other similar examples), map() is not the correct construct to use in the first place, and probably wants fixing for clarity alone, irrespective of the Py3 transformation.

Sure.

Wei.

Quoted 1 line

~Andrew

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

author
#14 lore raw

RFC PATCH 4/4 pygrub: make it build with python 3

In reply to Wei Liu

With the help of two porting guides and cpython source code:

1. Use PyUnicode to replace PyString counterparts.
2. Use PyVarObject_HEAD_INIT and provide compatibility for 2.5 and earlier.
3. Remove usage of Py_FindMethod.
4. Use new module initialisation routine.

For #3, Py_FindMethod was removed, yet an alternative wasn't documented. The code is the result of reverse-engineering cpython commit 6116d4a1d1

https://docs.python.org/3/howto/cporting.html
http://python3porting.com/cextensions.html

---
XXX Build test only!

---
 tools/pygrub/src/fsimage/fsimage.c | 92 ++++++++++++++++++++++++++++++++++----
 1 file changed, 84 insertions(+), 8 deletions(-)
tools/pygrub/src/fsimage/fsimage.c +84 -8
index 743a3fb7b8..c914253741 100644
@@ -26,11 +26,15 @@
 #include <xenfsimage.h>
 #include <stdlib.h>
 
+#if PY_MAJOR_VERSION < 3
 #if (PYTHON_API_VERSION >= 1011)
 #define PY_PAD 0L,0L,0L,0L,0L,0L,0L,0L,0L,0L,0L,0L,0L,0L,0L,0L,0L,0L,0L,0L,0L,0L,0L,0L
 #else
 #define PY_PAD 0L,0L,0L,0L
 #endif
+#else
+#define PY_PAD 0L,0L,0L,0L,0L,0L,0L,0L,0L,0L,0L,0L,0L,0L,0L,0L,0L
+#endif
 
 typedef struct fsimage_fs {
 	PyObject_HEAD
@@ -66,12 +70,24 @@ fsimage_file_read(fsimage_file_t *file, PyObject *args, PyObject *kwargs)
 
 	bufsize = size ? size : 4096;
 
-	if ((buffer = PyString_FromStringAndSize(NULL, bufsize)) == NULL)
+	buffer =
+#if PY_MAJOR_VERSION < 3
+		PyString_FromStringAndSize(NULL, bufsize);
+#else
+		PyUnicode_FromStringAndSize(NULL, bufsize);
+#endif
+
+	if (buffer == NULL)
 		return (NULL);
  
 	while (1) {
 		int err;
-		void *buf = PyString_AS_STRING(buffer) + bytesread;
+		void *buf =
+#if PY_MAJOR_VERSION < 3
+			PyString_AS_STRING(buffer) + bytesread;
+#else
+			PyUnicode_AS_UNICODE(buffer) + bytesread;
+#endif
 
 		err = fsi_pread_file(file->file, buf, bufsize,
 		    bytesread + offset);
@@ -91,12 +107,20 @@ fsimage_file_read(fsimage_file_t *file, PyObject *args, PyObject *kwargs)
 			if (bufsize == 0)
 				break;
 		} else {
+#if PY_MAJOR_VERSION < 3
 			if (_PyString_Resize(&buffer, bytesread + bufsize) < 0)
+#else
+			if (PyUnicode_Resize(&buffer, bytesread + bufsize) < 0)
+#endif
 				return (NULL);
 		}
 	}
 
+#if PY_MAJOR_VERSION < 3
 	_PyString_Resize(&buffer, bytesread);
+#else
+	PyUnicode_Resize(&buffer, bytesread);
+#endif
 	return (buffer);
 }
 
@@ -113,11 +137,13 @@ static struct PyMethodDef fsimage_file_methods[] = {
 	{ NULL, NULL, 0, NULL }	
 };
 
+#if PY_MAJOR_VERSION < 3
 static PyObject *
 fsimage_file_getattr(fsimage_file_t *file, char *name)
 {
 	return (Py_FindMethod(fsimage_file_methods, (PyObject *)file, name));
 }
+#endif
 
 static void
 fsimage_file_dealloc(fsimage_file_t *file)
@@ -128,16 +154,25 @@ fsimage_file_dealloc(fsimage_file_t *file)
 	PyObject_DEL(file);
 }
 
+/* Compatibility for 2.5 and earlier */
+#ifndef PyVarObject_HEAD_INIT
+#define PyVarObject_HEAD_INIT(type, size) \
+	PyObject_HEAD_INIT(type) size,
+#endif
+
 static char fsimage_file_type__doc__[] = "Filesystem image file";
 PyTypeObject fsimage_file_type = {
-	PyObject_HEAD_INIT(&PyType_Type)
-	0,					/* ob_size */
+	PyVarObject_HEAD_INIT(&PyType_Type, 0)
 	"xenfsimage.file",			/* tp_name */
 	sizeof(fsimage_file_t),			/* tp_size */
 	0,					/* tp_itemsize */
 	(destructor) fsimage_file_dealloc, 	/* tp_dealloc */
 	0,					/* tp_print */
+#if PY_MAJOR_VERSION < 3
 	(getattrfunc) fsimage_file_getattr, 	/* tp_getattr */
+#else
+	0,					/* tp_getattr */
+#endif
 	0,					/* tp_setattr */
 	0,					/* tp_compare */
 	0,					/* tp_repr */
@@ -151,7 +186,16 @@ PyTypeObject fsimage_file_type = {
 	0,					/* tp_setattro */
 	0,					/* tp_as_buffer */
 	Py_TPFLAGS_DEFAULT,			/* tp_flags */
-	fsimage_file_type__doc__,
+	fsimage_file_type__doc__,		/* tp_doc */
+#if PY_MAJOR_VERSION >= 3
+	0,					/* tp_traverse */
+	0,					/* tp_clear */
+	0,					/* tp_richcompare */
+	0,					/* tp_weaklistoffset */
+	0,					/* tp_iter */
+	0,					/* tp_iternext */
+	fsimage_file_methods,			/* tp_methods */
+#endif
 	PY_PAD
 };
 
@@ -215,11 +259,13 @@ static struct PyMethodDef fsimage_fs_methods[] = {
 	{ NULL, NULL, 0, NULL }	
 };
 
+#if PY_MAJOR_VERSION < 3
 static PyObject *
 fsimage_fs_getattr(fsimage_fs_t *fs, char *name)
 {
 	return (Py_FindMethod(fsimage_fs_methods, (PyObject *)fs, name));
 }
+#endif
 
 static void
 fsimage_fs_dealloc (fsimage_fs_t *fs)
@@ -232,14 +278,17 @@ fsimage_fs_dealloc (fsimage_fs_t *fs)
 PyDoc_STRVAR(fsimage_fs_type__doc__, "Filesystem image");
 
 PyTypeObject fsimage_fs_type = {
-	PyObject_HEAD_INIT(&PyType_Type)
-	0,					/* ob_size */
+	PyVarObject_HEAD_INIT(&PyType_Type, 0)
 	"xenfsimage.fs",			/* tp_name */
 	sizeof(fsimage_fs_t),			/* tp_size */
 	0,					/* tp_itemsize */
 	(destructor) fsimage_fs_dealloc, 	/* tp_dealloc */
 	0,					/* tp_print */
+#if PY_MAJOR_VERSION < 3
 	(getattrfunc) fsimage_fs_getattr, 	/* tp_getattr */
+#else
+	0,					/* tp_getattr */
+#endif
 	0,					/* tp_setattr */
 	0,					/* tp_compare */
 	0,					/* tp_repr */
@@ -253,7 +302,16 @@ PyTypeObject fsimage_fs_type = {
 	0,					/* tp_setattro */
 	0,					/* tp_as_buffer */
 	Py_TPFLAGS_DEFAULT,			/* tp_flags */
-	fsimage_fs_type__doc__,
+	fsimage_fs_type__doc__,			/* tp_doc */
+#if PY_MAJOR_VERSION >= 3
+	0,					/* tp_traverse */
+	0,					/* tp_clear */
+	0,					/* tp_richcompare */
+	0,					/* tp_weaklistoffset */
+	0,					/* tp_iter */
+	0,					/* tp_iternext */
+	fsimage_fs_methods,			/* tp_methods */
+#endif
 	PY_PAD
 };
 
@@ -316,8 +374,26 @@ static struct PyMethodDef fsimage_module_methods[] = {
 	{ NULL, NULL, 0, NULL }
 };
 
+#if PY_MAJOR_VERSION >= 3
+static struct PyModuleDef fsimage_module_def = {
+	PyModuleDef_HEAD_INIT,
+	"xenfsimage",		/* m_name */
+	"",			/* m_doc */
+	-1,			/* m_size */
+	fsimage_module_methods,	/* m_methods */
+	NULL,			/* m_reload */
+	NULL,			/* m_traverse */
+	NULL,			/* m_clear */
+	NULL			/* m_free */
+};
+#endif
+
 PyMODINIT_FUNC
 initxenfsimage(void)
 {
+#if PY_MAJOR_VERSION < 3
 	Py_InitModule("xenfsimage", fsimage_module_methods);
+#else
+	return PyModule_Create(&fsimage_module_def);
+#endif
 }
Signature · 7 lines
2.11.0


_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
#15 lore raw

Re: [PATCH for-next RFC 4/4] pygrub: make it build with python 3

Quoted 8 lines · Wei Liu

With the help of two porting guides and cpython source code:

1. Use PyUnicode to replace PyString counterparts.
2. Use PyVarObject_HEAD_INIT and provide compatibility for 2.5 and earlier.
3. Remove usage of Py_FindMethod.
4. Use new module initialisation routine.

Quoted 8 lines

For #3, Py_FindMethod was removed, yet an alternative wasn't documented. The code is the result of reverse-engineering cpython commit 6116d4a1d1

https://docs.python.org/3/howto/cporting.html
http://python3porting.com/cextensions.html

Marek already made the tools/python/* libraries compatible with Py2 and Py3

The following commits are the relevant ones:

* be6b316 - python: handle long type in scripts (2 years ago) <Marek Marczykowski-Górecki>
* e16c705 - python: adjust module initalization for Python3 (2 years ago) <Marek Marczykowski-Górecki>
* dd986cd - python: use PyLong_* for constructing 'int' type in Python3 (2 years ago) <Marek Marczykowski-Górecki>
* 121d9d4 - python: use PyBytes/PyUnicode instead of PyString (2 years ago) <Marek Marczykowski-Górecki>
* 0c8981f - python: initialize specific fields of PyTypeObject (2 years ago) <Marek Marczykowski-Górecki>
* 7b1e5f7 - python: use Py_TYPE instead of looking directly into PyObject_HEAD (2 years ago) <Marek Marczykowski-Górecki>
* 96d1ee6 - python: drop tp_getattr implementation (2 years ago) <Marek Marczykowski-Górecki>
* 6b28df3 - python: check return value of PyErr_NewException (2 years ago) <Marek Marczykowski-Górecki>

Which in particular handle strings differently in the Py2 case.

~Andrew

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

author
#16 lore raw

Re: [PATCH for-next RFC 4/4] pygrub: make it build with python 3

Quoted 22 lines · Andrew Cooper
Quoted 8 lines · Wei Liu

With the help of two porting guides and cpython source code:

1. Use PyUnicode to replace PyString counterparts.
2. Use PyVarObject_HEAD_INIT and provide compatibility for 2.5 and earlier.
3. Remove usage of Py_FindMethod.
4. Use new module initialisation routine.

Quoted 8 lines

For #3, Py_FindMethod was removed, yet an alternative wasn't documented. The code is the result of reverse-engineering cpython commit 6116d4a1d1

https://docs.python.org/3/howto/cporting.html
http://python3porting.com/cextensions.html

Marek already made the tools/python/* libraries compatible with Py2 and Py3

The following commits are the relevant ones:

Quoted 8 lines

* be6b316 - python: handle long type in scripts (2 years ago) <Marek Marczykowski-Górecki>
* e16c705 - python: adjust module initalization for Python3 (2 years ago) <Marek Marczykowski-Górecki>
* dd986cd - python: use PyLong_* for constructing 'int' type in Python3 (2 years ago) <Marek Marczykowski-Górecki>
* 121d9d4 - python: use PyBytes/PyUnicode instead of PyString (2 years ago) <Marek Marczykowski-Górecki>
* 0c8981f - python: initialize specific fields of PyTypeObject (2 years ago) <Marek Marczykowski-Górecki>
* 7b1e5f7 - python: use Py_TYPE instead of looking directly into PyObject_HEAD (2 years ago) <Marek Marczykowski-Górecki>
* 96d1ee6 - python: drop tp_getattr implementation (2 years ago) <Marek Marczykowski-Górecki>
* 6b28df3 - python: check return value of PyErr_NewException (2 years ago) <Marek Marczykowski-Górecki>

I knew.

Quoted 2 lines

Which in particular handle strings differently in the Py2 case.

I am not sure his changes for the string APIs are correct -- they seem to deviate from the official porting guide. But hey, I don't use these bindings myself, so he probably knows better.

CC Marek.

Wei.

Quoted 2 lines

~Andrew

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

#17 lore raw

Re: [PATCH for-next RFC 4/4] pygrub: make it build with python 3

Quoted 35 lines · Wei Liu
Quoted 22 lines · Andrew Cooper
Quoted 8 lines · Wei Liu

With the help of two porting guides and cpython source code:

1. Use PyUnicode to replace PyString counterparts.
2. Use PyVarObject_HEAD_INIT and provide compatibility for 2.5 and earlier.
3. Remove usage of Py_FindMethod.
4. Use new module initialisation routine.

Quoted 8 lines

For #3, Py_FindMethod was removed, yet an alternative wasn't documented. The code is the result of reverse-engineering cpython commit 6116d4a1d1

https://docs.python.org/3/howto/cporting.html
http://python3porting.com/cextensions.html

Marek already made the tools/python/* libraries compatible with Py2 and Py3

The following commits are the relevant ones:

Quoted 8 lines

* be6b316 - python: handle long type in scripts (2 years ago) <Marek Marczykowski-Górecki>
* e16c705 - python: adjust module initalization for Python3 (2 years ago) <Marek Marczykowski-Górecki>
* dd986cd - python: use PyLong_* for constructing 'int' type in Python3 (2 years ago) <Marek Marczykowski-Górecki>
* 121d9d4 - python: use PyBytes/PyUnicode instead of PyString (2 years ago) <Marek Marczykowski-Górecki>
* 0c8981f - python: initialize specific fields of PyTypeObject (2 years ago) <Marek Marczykowski-Górecki>
* 7b1e5f7 - python: use Py_TYPE instead of looking directly into PyObject_HEAD (2 years ago) <Marek Marczykowski-Górecki>
* 96d1ee6 - python: drop tp_getattr implementation (2 years ago) <Marek Marczykowski-Górecki>
* 6b28df3 - python: check return value of PyErr_NewException (2 years ago) <Marek Marczykowski-Górecki>

I knew.

Quoted 1 line
Quoted 6 lines
Quoted 1 line

Which in particular handle strings differently in the Py2 case.

I am not sure his changes for the string APIs are correct -- they seem to deviate from the official porting guide. But hey, I don't use these bindings myself, so he probably knows better.

That was intentional, because in py2 str type is the same as bytes types, and in fact some of those str should really be bytes. It's in the commit message:

python: use PyBytes/PyUnicode instead of PyString

In Python2 PyBytes is the same as PyString, but in Python3 PyString is gone and 'str' is really PyUnicode in C-API.
When handling arbitrary data, use PyBytes - which is the right thing to do in Python3, and pose no API change in Python2. When handling xenstore paths and transaction ids, which have well defined format, use PyUnicode - to ease API usage - no need to prefix all xenstore paths with 'b' when migrating scripts to Python3.

I'm not sure if the same reasoning applies to pygrub, but I guess it may. For example fsimage_file_read sounds like handling binary data, not really UTF-8 strings. Using PyUnicode for arbitrary binary data may lead to various UnicodeDecodeErrors.

Signature · 5 lines
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

author
#18 lore raw

Re: [PATCH for-next RFC 4/4] pygrub: make it build with python 3

Quoted 57 lines · Marek Marczykowski-Górecki
Quoted 35 lines · Wei Liu
Quoted 22 lines · Andrew Cooper
Quoted 8 lines · Wei Liu

With the help of two porting guides and cpython source code:

1. Use PyUnicode to replace PyString counterparts.
2. Use PyVarObject_HEAD_INIT and provide compatibility for 2.5 and earlier.
3. Remove usage of Py_FindMethod.
4. Use new module initialisation routine.

Quoted 8 lines

For #3, Py_FindMethod was removed, yet an alternative wasn't documented. The code is the result of reverse-engineering cpython commit 6116d4a1d1

https://docs.python.org/3/howto/cporting.html
http://python3porting.com/cextensions.html

Marek already made the tools/python/* libraries compatible with Py2 and Py3

The following commits are the relevant ones:

Quoted 8 lines

* be6b316 - python: handle long type in scripts (2 years ago) <Marek Marczykowski-Górecki>
* e16c705 - python: adjust module initalization for Python3 (2 years ago) <Marek Marczykowski-Górecki>
* dd986cd - python: use PyLong_* for constructing 'int' type in Python3 (2 years ago) <Marek Marczykowski-Górecki>
* 121d9d4 - python: use PyBytes/PyUnicode instead of PyString (2 years ago) <Marek Marczykowski-Górecki>
* 0c8981f - python: initialize specific fields of PyTypeObject (2 years ago) <Marek Marczykowski-Górecki>
* 7b1e5f7 - python: use Py_TYPE instead of looking directly into PyObject_HEAD (2 years ago) <Marek Marczykowski-Górecki>
* 96d1ee6 - python: drop tp_getattr implementation (2 years ago) <Marek Marczykowski-Górecki>
* 6b28df3 - python: check return value of PyErr_NewException (2 years ago) <Marek Marczykowski-Górecki>

I knew.

Quoted 1 line
Quoted 6 lines
Quoted 1 line

Which in particular handle strings differently in the Py2 case.

I am not sure his changes for the string APIs are correct -- they seem to deviate from the official porting guide. But hey, I don't use these bindings myself, so he probably knows better.

That was intentional, because in py2 str type is the same as bytes types, and in fact some of those str should really be bytes. It's in the commit message:

python: use PyBytes/PyUnicode instead of PyString

In Python2 PyBytes is the same as PyString, but in Python3 PyString is gone and 'str' is really PyUnicode in C-API.
When handling arbitrary data, use PyBytes - which is the right thing to do in Python3, and pose no API change in Python2. When handling xenstore paths and transaction ids, which have well defined format, use PyUnicode - to ease API usage - no need to prefix all xenstore paths with 'b' when migrating scripts to Python3.

Quoted 4 lines

I'm not sure if the same reasoning applies to pygrub, but I guess it may. For example fsimage_file_read sounds like handling binary data, not really UTF-8 strings. Using PyUnicode for arbitrary binary data may lead to various UnicodeDecodeErrors.

Good point. Let me dig into this a bit more. Frankly I trust you more than I trust myself on this matter. :-)

Wei.

Quoted 7 lines
Signature · 6 lines
-- 
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

author
#19 lore raw

Re: [PATCH for-next RFC 4/4] pygrub: make it build with python 3

In reply to Wei Liu
Quoted 3 lines · Wei Liu

+
PyMODINIT_FUNC
initxenfsimage(void)

So Python 3 requires the initialisation function to be called PyInit_xenfsimage, otherwise it can't find the entry point of this module.

I have fixed this in my next version.

Wei.

Quoted 10 lines

{

tools/pygrub/src/fsimage/fsimage.c +4 -0
+#if PY_MAJOR_VERSION < 3
 	Py_InitModule("xenfsimage", fsimage_module_methods);
+#else
+	return PyModule_Create(&fsimage_module_def);
+#endif
 }
Signature · 1 line
2.11.0

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

#20 lore raw
In reply to Wei Liu
Quoted 7 lines · Wei Liu

This series makes tools build with Python 3.

Compile test only with 2.7 and 3.5 thus far, hence the RFC. This should be able to give people some idea what sort of work is involved.

You will also need Andrew's "tools/xen-foreign: Update python scripts to be Py3 compatible".

Tossing this out there: Given that python2 is (theoretically) EOL in less than a year, is it worth maintaining compatibility with python2, or would it be better to just go whole-hog into python3?

-George

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

#21 lore raw
Quoted 6 lines · George Dunlap
Quoted 5 lines · Wei Liu

This series makes tools build with Python 3.

Compile test only with 2.7 and 3.5 thus far, hence the RFC. This should be able to give people some idea what sort of work is involved.

Quoted 6 lines
Quoted 2 lines

You will also need Andrew's "tools/xen-foreign: Update python scripts to be Py3 compatible".

Tossing this out there: Given that python2 is (theoretically) EOL in less than a year, is it worth maintaining compatibility with python2, or would it be better to just go whole-hog into python3?

I mean -- looking at some of the discussions about how differently certain kinds of things are interpreted between python2 and python3, it seems a bit mad to try to write code that works in both: we're just asking for trouble.

-George

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

author
#22 lore raw
Quoted 10 lines · George Dunlap
Quoted 9 lines · George Dunlap
Quoted 7 lines · Wei Liu

This series makes tools build with Python 3.

Compile test only with 2.7 and 3.5 thus far, hence the RFC. This should be able to give people some idea what sort of work is involved.

You will also need Andrew's "tools/xen-foreign: Update python scripts to be Py3 compatible".

Quoted 8 lines
Quoted 3 lines

Tossing this out there: Given that python2 is (theoretically) EOL in less than a year, is it worth maintaining compatibility with python2, or would it be better to just go whole-hog into python3?

I mean -- looking at some of the discussions about how differently certain kinds of things are interpreted between python2 and python3, it seems a bit mad to try to write code that works in both: we're just asking for trouble.

I think Python is not very consistent even within a major version. New syntax and constructs get added to point releases. Some of the discussions re code isn't about the differences between 2 and 3. There is at least one about 2.4 vs 2.7.

I am all for moving away from 2.4.

Wei.

Quoted 2 lines

-George

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

#23 lore raw
Quoted 20 lines · Wei Liu
Quoted 10 lines · George Dunlap
Quoted 9 lines · George Dunlap
Quoted 7 lines · Wei Liu

This series makes tools build with Python 3.

Compile test only with 2.7 and 3.5 thus far, hence the RFC. This should be able to give people some idea what sort of work is involved.

You will also need Andrew's "tools/xen-foreign: Update python scripts to be Py3 compatible".

Quoted 8 lines
Quoted 3 lines

Tossing this out there: Given that python2 is (theoretically) EOL in less than a year, is it worth maintaining compatibility with python2, or would it be better to just go whole-hog into python3?

I mean -- looking at some of the discussions about how differently certain kinds of things are interpreted between python2 and python3, it seems a bit mad to try to write code that works in both: we're just asking for trouble.

Quoted 4 lines

I think Python is not very consistent even within a major version. New syntax and constructs get added to point releases. Some of the discussions re code isn't about the differences between 2 and 3. There is at least one about 2.4 vs 2.7.

I think it's because they backport syntax changes from 3.X to 2.6 and 2.7. Also all the _future_X imports. All of that makes it easier to have scripts compatible with both 2.7 and 3.X.

Quoted 1 line

I am all for moving away from 2.4.

Moving to a min of 2.6 would be nice. CentOS 6 comes it.

Signature · 6 lines
Anthony PERARD

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel
author
#24 lore raw
Quoted 26 lines · Anthony PERARD
Quoted 20 lines · Wei Liu
Quoted 10 lines · George Dunlap
Quoted 9 lines · George Dunlap
Quoted 7 lines · Wei Liu

This series makes tools build with Python 3.

Compile test only with 2.7 and 3.5 thus far, hence the RFC. This should be able to give people some idea what sort of work is involved.

You will also need Andrew's "tools/xen-foreign: Update python scripts to be Py3 compatible".

Quoted 8 lines
Quoted 3 lines

Tossing this out there: Given that python2 is (theoretically) EOL in less than a year, is it worth maintaining compatibility with python2, or would it be better to just go whole-hog into python3?

I mean -- looking at some of the discussions about how differently certain kinds of things are interpreted between python2 and python3, it seems a bit mad to try to write code that works in both: we're just asking for trouble.

Quoted 4 lines

I think Python is not very consistent even within a major version. New syntax and constructs get added to point releases. Some of the discussions re code isn't about the differences between 2 and 3. There is at least one about 2.4 vs 2.7.

Quoted 7 lines

I think it's because they backport syntax changes from 3.X to 2.6 and 2.7. Also all the _future_X imports. All of that makes it easier to have scripts compatible with both 2.7 and 3.X.

Quoted 1 line

I am all for moving away from 2.4.

Moving to a min of 2.6 would be nice. CentOS 6 comes it.

My next version will have a patch to do that. Any objection?

Wei.

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

author
#25 lore raw
In reply to George Dunlap
Quoted 6 lines · George Dunlap
Quoted 5 lines · Wei Liu

This series makes tools build with Python 3.

Compile test only with 2.7 and 3.5 thus far, hence the RFC. This should be able to give people some idea what sort of work is involved.

Quoted 6 lines
Quoted 2 lines

You will also need Andrew's "tools/xen-foreign: Update python scripts to be Py3 compatible".

Tossing this out there: Given that python2 is (theoretically) EOL in less than a year, is it worth maintaining compatibility with python2, or would it be better to just go whole-hog into python3?

Some enterprise-y distros still ship ancient python AFAICT.

Given that the work involved seems to be manageable so far I would say let's keep the compatibility for now.

Wei.

Quoted 2 lines

-George

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel

#26 lore raw
In reply to George Dunlap
Quoted 11 lines · George Dunlap
Quoted 7 lines · Wei Liu

This series makes tools build with Python 3.

Compile test only with 2.7 and 3.5 thus far, hence the RFC. This should be able to give people some idea what sort of work is involved.

You will also need Andrew's "tools/xen-foreign: Update python scripts to be Py3 compatible".

Tossing this out there: Given that python2 is (theoretically) EOL in less than a year, is it worth maintaining compatibility with python2, or would it be better to just go whole-hog into python3?

Not when we've got supported distros which only ship Py2 by default. (CentOS 6 - I'm looking at you in particular...)

~Andrew

_______________________________________________
Xen-devel mailing list
[email protected]
https://lists.xenproject.org/mailman/listinfo/xen-devel