Skip to contentSkip to conversation

pynfs python 3 flag day

DISCUSSION J. Bruce Fields 2 participants 3 messages Feb 14, 2020 – Mar 10, 2020

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

I'm hearing more noise about deprecating Python 2, so decided I can't keep ignoring Python 3.

Getting pynfs working on Python 3 is a bigger project than I expected. Keeping it working under Python 2 looks like another project. So, I'm planning a flag day after which pynfs will require Python 3.

That isn't the way I'd prefer to do it, but there's only so much time I want to spend on this.

I've mostly got the 4.0 server tests working under python 3. I hope a few more days will be enough to get the 4.1 tests working as well.

When I switch over, I'm afraid a few things will be left broken: any tests that I don't personally run may still have minor python 3 bugs, and I haven't touched the python server code that's used for client testing.

If you stumble across something broken, and you can give me a simple reproducer, feel free to share it with me and I'll take a look.

But for anything complicated, I'll probably need patches.

Again, I apologize for any extra work that creates for anyone, but for now this seems like the best compromise to keep things mostly working without it becoming a bigger time sink for me.

Work so far is in the "python3" branch at

git://linux-nfs.org/~bfields/pynfs.git

The history will probably be cleaned up an rewritten before it's done. I'm hoping that'll be in the next week.

It's mostly just a matter of separating out unicode strings and byte arrays. Protocol data is all the latter (even if the protocol prefers some field to be UTF8, pynfs still needs to be able to handle non-UTF8). But some things have to be unicode strings.

--b.

#1 lore raw

I've pushed the python 3 changes to the master branch, so pynfs now *only* works with python 3:

git://linux-nfs.org/~bfields/pynfs.git

This works for me, but, as I've said, some of the functionality that I don't use is probably broken. Apologies--let me know if you run across anything.

The last hold-up was the gssapi code. But it turns out pynfs gssapi has been broken for a while. So, I'm still working on it, but I couldn't see any reason to hold back the python 3 changes for it.

--b.

J. Bruce Fields (21):
st_delegations: don't reimplement join()
Fix module imports for python 3
python 3 has no long type
python3: exception scope
python3: open results file in binary mode
python3: / no longer does integer division
showresults: remove unnecessary import code
python3: StandardError no longer defined
python3: the socket structure has changed
python3: machinename should be bytes, not string python 3 map returns iterator not list
python3: sort works differently
python3: make "path" commandline argument a byte array
python3: RPCClient.ipaddress should be bytes
python3: tag should be bytes NFS4Client.create_obj() should expect bytes
python3: need binary array instead of t.code string
python3: make a lot of strings byte arrays python3 has no xrange()
python3: loop over dict while removing entries
use_obj: expect bytes, not string

 nfs4.0/TODO                                 |   2 +-
 nfs4.0/lib/rpc/rpc.py                       |  35 ++--
 nfs4.0/lib/rpc/rpcsec/base.py               |   2 +-
 nfs4.0/lib/rpc/rpcsec/sec_auth_sys.py       |   7 +-
 nfs4.0/nfs4lib.py                           |  70 ++++----
 nfs4.0/servertests/environment.py           | 121 ++++++-------
 nfs4.0/servertests/st_acl.py                |   8 +-
 nfs4.0/servertests/st_close.py              |  74 ++++----
 nfs4.0/servertests/st_commit.py             |   6 +-
 nfs4.0/servertests/st_compound.py           |   2 +-
 nfs4.0/servertests/st_create.py             |  38 ++--
 nfs4.0/servertests/st_delegation.py         | 159 ++++++++---------
 nfs4.0/servertests/st_fslocations.py        |   2 +-
 nfs4.0/servertests/st_link.py               |  40 ++---
 nfs4.0/servertests/st_lock.py               | 262 ++++++++++++++--------------
 nfs4.0/servertests/st_lockt.py              |  30 ++--
 nfs4.0/servertests/st_locku.py              |  78 ++++-----
 nfs4.0/servertests/st_lookup.py             |  68 ++++----
 nfs4.0/servertests/st_lookupp.py            |   4 +-
 nfs4.0/servertests/st_nverify.py            |   2 +-
 nfs4.0/servertests/st_open.py               | 184 +++++++++----------
 nfs4.0/servertests/st_openconfirm.py        |  16 +-
 nfs4.0/servertests/st_opendowngrade.py      |  44 ++---
 nfs4.0/servertests/st_putfh.py              |   8 +-
 nfs4.0/servertests/st_read.py               |  36 ++--
 nfs4.0/servertests/st_readdir.py            |  48 ++---
 nfs4.0/servertests/st_reboot.py             |  92 +++++-----
 nfs4.0/servertests/st_releaselockowner.py   |   6 +-
 nfs4.0/servertests/st_remove.py             |  78 ++++-----
 nfs4.0/servertests/st_rename.py             | 209 +++++++++++-----------
 nfs4.0/servertests/st_renew.py              |   4 +-
 nfs4.0/servertests/st_replay.py             |  66 +++----
 nfs4.0/servertests/st_secinfo.py            |  16 +-
 nfs4.0/servertests/st_setattr.py            | 122 ++++++-------
 nfs4.0/servertests/st_setclientid.py        |  74 ++++----
 nfs4.0/servertests/st_setclientidconfirm.py |  12 +-
 nfs4.0/servertests/st_spoof.py              |   8 +-
 nfs4.0/servertests/st_verify.py             |   2 +-
 nfs4.0/servertests/st_write.py              |  94 +++++-----
 nfs4.0/testserver.py                        |  22 ++-
 nfs4.1/client41tests/environment.py         |  10 +-
 nfs4.1/nfs4client.py                        |  16 +-
 nfs4.1/nfs4commoncode.py                    |   4 +-
 nfs4.1/nfs4lib.py                           |  44 ++---
 nfs4.1/nfs4server.py                        |   4 +-
 nfs4.1/server41tests/environment.py         | 103 +++++------
 nfs4.1/server41tests/st_compound.py         |   4 +-
 nfs4.1/server41tests/st_create_session.py   |  24 +--
 nfs4.1/server41tests/st_current_stateid.py  |  12 +-
 nfs4.1/server41tests/st_debug.py            |   2 +-
 nfs4.1/server41tests/st_delegation.py       |  24 +--
 nfs4.1/server41tests/st_destroy_clientid.py |   6 +-
 nfs4.1/server41tests/st_destroy_session.py  |   2 +-
 nfs4.1/server41tests/st_exchange_id.py      |  16 +-
 nfs4.1/server41tests/st_open.py             |  24 +--
 nfs4.1/server41tests/st_putfh.py            |   2 +-
 nfs4.1/server41tests/st_reboot.py           |  24 +--
 nfs4.1/server41tests/st_reclaim_complete.py |   6 +-
 nfs4.1/server41tests/st_rename.py           |  92 +++++-----
 nfs4.1/server41tests/st_secinfo.py          |   6 +-
 nfs4.1/server41tests/st_secinfo_no_name.py  |   2 +-
 nfs4.1/server41tests/st_sequence.py         |  32 ++--
 nfs4.1/server41tests/st_sparse.py           |   2 +-
 nfs4.1/testmod.py                           |  14 +-
 nfs4.1/testserver.py                        |  13 +-
 rpc/rpc.py                                  |  30 ++--
 rpc/rpclib.py                               |  12 +-
 rpc/security.py                             |  10 +-
 showresults.py                              |  19 +-
 69 files changed, 1357 insertions(+), 1353 deletions(-)
Quoted 40 lines · bfields

I'm hearing more noise about deprecating Python 2, so decided I can't keep ignoring Python 3.

Getting pynfs working on Python 3 is a bigger project than I expected. Keeping it working under Python 2 looks like another project. So, I'm planning a flag day after which pynfs will require Python 3.

That isn't the way I'd prefer to do it, but there's only so much time I want to spend on this.

I've mostly got the 4.0 server tests working under python 3. I hope a few more days will be enough to get the 4.1 tests working as well.

When I switch over, I'm afraid a few things will be left broken: any tests that I don't personally run may still have minor python 3 bugs, and I haven't touched the python server code that's used for client testing.

If you stumble across something broken, and you can give me a simple reproducer, feel free to share it with me and I'll take a look.

But for anything complicated, I'll probably need patches.

Again, I apologize for any extra work that creates for anyone, but for now this seems like the best compromise to keep things mostly working without it becoming a bigger time sink for me.

Work so far is in the "python3" branch at

git://linux-nfs.org/~bfields/pynfs.git

The history will probably be cleaned up an rewritten before it's done. I'm hoping that'll be in the next week.

It's mostly just a matter of separating out unicode strings and byte arrays. Protocol data is all the latter (even if the protocol prefers some field to be UTF8, pynfs still needs to be able to handle non-UTF8). But some things have to be unicode strings.

--b.

#2 lore raw

Hi Bruce,

works for me! Thanks for the effort.

Tigran.

----- Original Message -----

Quoted 156 lines

From: "J. Bruce Fields" <[email protected]>
To: "linux-nfs" <[email protected]>
Sent: Thursday, February 27, 2020 2:59:25 PM
Subject: Re: pynfs python 3 flag day

I've pushed the python 3 changes to the master branch, so pynfs now *only* works with python 3:

git://linux-nfs.org/~bfields/pynfs.git

This works for me, but, as I've said, some of the functionality that I don't use is probably broken. Apologies--let me know if you run across anything.

The last hold-up was the gssapi code. But it turns out pynfs gssapi has been broken for a while. So, I'm still working on it, but I couldn't see any reason to hold back the python 3 changes for it.

--b.

J. Bruce Fields (21):
st_delegations: don't reimplement join()
Fix module imports for python 3
python 3 has no long type
python3: exception scope
python3: open results file in binary mode
python3: / no longer does integer division
showresults: remove unnecessary import code
python3: StandardError no longer defined
python3: the socket structure has changed
python3: machinename should be bytes, not string python 3 map returns iterator not list
python3: sort works differently
python3: make "path" commandline argument a byte array
python3: RPCClient.ipaddress should be bytes
python3: tag should be bytes NFS4Client.create_obj() should expect bytes
python3: need binary array instead of t.code string
python3: make a lot of strings byte arrays python3 has no xrange()
python3: loop over dict while removing entries
use_obj: expect bytes, not string

nfs4.0/TODO                                 |   2 +-
nfs4.0/lib/rpc/rpc.py                       |  35 ++--
nfs4.0/lib/rpc/rpcsec/base.py               |   2 +-
nfs4.0/lib/rpc/rpcsec/sec_auth_sys.py       |   7 +-
nfs4.0/nfs4lib.py                           |  70 ++++----
nfs4.0/servertests/environment.py           | 121 ++++++-------
nfs4.0/servertests/st_acl.py                |   8 +-
nfs4.0/servertests/st_close.py              |  74 ++++----
nfs4.0/servertests/st_commit.py             |   6 +-
nfs4.0/servertests/st_compound.py           |   2 +-
nfs4.0/servertests/st_create.py             |  38 ++--
nfs4.0/servertests/st_delegation.py         | 159 ++++++++---------
nfs4.0/servertests/st_fslocations.py        |   2 +-
nfs4.0/servertests/st_link.py               |  40 ++---
nfs4.0/servertests/st_lock.py               | 262 ++++++++++++++--------------
nfs4.0/servertests/st_lockt.py              |  30 ++--
nfs4.0/servertests/st_locku.py              |  78 ++++-----
nfs4.0/servertests/st_lookup.py             |  68 ++++----
nfs4.0/servertests/st_lookupp.py            |   4 +-
nfs4.0/servertests/st_nverify.py            |   2 +-
nfs4.0/servertests/st_open.py               | 184 +++++++++----------
nfs4.0/servertests/st_openconfirm.py        |  16 +-
nfs4.0/servertests/st_opendowngrade.py      |  44 ++---
nfs4.0/servertests/st_putfh.py              |   8 +-
nfs4.0/servertests/st_read.py               |  36 ++--
nfs4.0/servertests/st_readdir.py            |  48 ++---
nfs4.0/servertests/st_reboot.py             |  92 +++++-----
nfs4.0/servertests/st_releaselockowner.py   |   6 +-
nfs4.0/servertests/st_remove.py             |  78 ++++-----
nfs4.0/servertests/st_rename.py             | 209 +++++++++++-----------
nfs4.0/servertests/st_renew.py              |   4 +-
nfs4.0/servertests/st_replay.py             |  66 +++----
nfs4.0/servertests/st_secinfo.py            |  16 +-
nfs4.0/servertests/st_setattr.py            | 122 ++++++-------
nfs4.0/servertests/st_setclientid.py        |  74 ++++----
nfs4.0/servertests/st_setclientidconfirm.py |  12 +-
nfs4.0/servertests/st_spoof.py              |   8 +-
nfs4.0/servertests/st_verify.py             |   2 +-
nfs4.0/servertests/st_write.py              |  94 +++++-----
nfs4.0/testserver.py                        |  22 ++-
nfs4.1/client41tests/environment.py         |  10 +-
nfs4.1/nfs4client.py                        |  16 +-
nfs4.1/nfs4commoncode.py                    |   4 +-
nfs4.1/nfs4lib.py                           |  44 ++---
nfs4.1/nfs4server.py                        |   4 +-
nfs4.1/server41tests/environment.py         | 103 +++++------
nfs4.1/server41tests/st_compound.py         |   4 +-
nfs4.1/server41tests/st_create_session.py   |  24 +--
nfs4.1/server41tests/st_current_stateid.py  |  12 +-
nfs4.1/server41tests/st_debug.py            |   2 +-
nfs4.1/server41tests/st_delegation.py       |  24 +--
nfs4.1/server41tests/st_destroy_clientid.py |   6 +-
nfs4.1/server41tests/st_destroy_session.py  |   2 +-
nfs4.1/server41tests/st_exchange_id.py      |  16 +-
nfs4.1/server41tests/st_open.py             |  24 +--
nfs4.1/server41tests/st_putfh.py            |   2 +-
nfs4.1/server41tests/st_reboot.py           |  24 +--
nfs4.1/server41tests/st_reclaim_complete.py |   6 +-
nfs4.1/server41tests/st_rename.py           |  92 +++++-----
nfs4.1/server41tests/st_secinfo.py          |   6 +-
nfs4.1/server41tests/st_secinfo_no_name.py  |   2 +-
nfs4.1/server41tests/st_sequence.py         |  32 ++--
nfs4.1/server41tests/st_sparse.py           |   2 +-
nfs4.1/testmod.py                           |  14 +-
nfs4.1/testserver.py                        |  13 +-
rpc/rpc.py                                  |  30 ++--
rpc/rpclib.py                               |  12 +-
rpc/security.py                             |  10 +-
showresults.py                              |  19 +-
69 files changed, 1357 insertions(+), 1353 deletions(-)
Quoted 40 lines · bfields

I'm hearing more noise about deprecating Python 2, so decided I can't keep ignoring Python 3.

Getting pynfs working on Python 3 is a bigger project than I expected. Keeping it working under Python 2 looks like another project. So, I'm planning a flag day after which pynfs will require Python 3.

That isn't the way I'd prefer to do it, but there's only so much time I want to spend on this.

I've mostly got the 4.0 server tests working under python 3. I hope a few more days will be enough to get the 4.1 tests working as well.

When I switch over, I'm afraid a few things will be left broken: any tests that I don't personally run may still have minor python 3 bugs, and I haven't touched the python server code that's used for client testing.

If you stumble across something broken, and you can give me a simple reproducer, feel free to share it with me and I'll take a look.

But for anything complicated, I'll probably need patches.

Again, I apologize for any extra work that creates for anyone, but for now this seems like the best compromise to keep things mostly working without it becoming a bigger time sink for me.

Work so far is in the "python3" branch at

git://linux-nfs.org/~bfields/pynfs.git

The history will probably be cleaned up an rewritten before it's done. I'm hoping that'll be in the next week.

It's mostly just a matter of separating out unicode strings and byte arrays. Protocol data is all the latter (even if the protocol prefers some field to be UTF8, pynfs still needs to be able to handle non-UTF8). But some things have to be unicode strings.

--b.