utf-8 issues on b4 master
Passing message id [email protected] to b4 gives this backtrace:
Traceback (most recent call last):
File "/scm/b4/b4/command.py", line 263, in <module>
cmd()
File "/scm/b4/b4/command.py", line 246, in cmd
cmdargs.func(cmdargs)
File "/scm/b4/b4/command.py", line 41, in cmd_mbox
b4.mbox.main(cmdargs)
File "/scm/b4/b4/mbox.py", line 581, in main
msgid, msgs = get_msgs(cmdargs)
File "/scm/b4/b4/mbox.py", line 523, in get_msgs
msgid = b4.get_msgid(cmdargs)
File "/scm/b4/b4/__init__.py", line 2080, in get_msgid
msgid = get_msgid_from_stdin()
File "/scm/b4/b4/__init__.py", line 2072, in get_msgid_from_stdin
message = email.message_from_string(sys.stdin.read())
File "/usr/lib64/python3.9/codecs.py", line 322, in decode
(result, consumed) = self._buffer_decode(data, self.errors, final)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xd4 in position 5886: invalid continuation byte
mutt does not seem to have trouble decoding this ... weird.
Signature · 1 line
MST
Quoted 23 lines · Michael S. Tsirkin
Passing message id [email protected] to b4 gives this backtrace:
Traceback (most recent call last):
File "/scm/b4/b4/command.py", line 263, in <module>cmd()File "/scm/b4/b4/command.py", line 246, in cmd
cmdargs.func(cmdargs)
File "/scm/b4/b4/command.py", line 41, incmd_mbox
b4.mbox.main(cmdargs)
File "/scm/b4/b4/mbox.py", line 581, in main
msgid, msgs =get_msgs(cmdargs)
File "/scm/b4/b4/mbox.py", line 523, inget_msgs
msgid =b4.get_msgid(cmdargs)
File "/scm/b4/b4/__init__.py", line 2080, inget_msgid
msgid =get_msgid_from_stdin()
File "/scm/b4/b4/__init__.py", line 2072, inget_msgid_from_stdin
message = email.message_from_string(sys.stdin.read())
File "/usr/lib64/python3.9/codecs.py", line 322, in decode
(result, consumed) =self._buffer_decode(data, self.errors, final)
UnicodeDecodeError: 'utf-8' codec can't decode byte0xd4in position 5886: invalid continuation byte
Quoted 1 line
mutt does not seem to have trouble decoding this ... weird.
I'm confused by that backtrace. I think get_msgid_from_stdin() should be called only when a message is fed on stdin. You say you're passing a message ID. That's as a positional argument, right?
Fwiw I wasn't able to trigger the issue on my end.
$ b4 am [email protected] Looking up https://lore.kernel.org/r/bbe52a89-c7ea-c155-6226-0397f223cd80%40linux.alibaba.com Analyzing 5 messages in the thread --- [PATCH] virtio-balloon: Use virtio_find_vqs() helper + Reviewed-by: David Hildenbrand <[email protected]> --- Total patches: 1 --- Link: https://lore.kernel.org/r/[email protected] Base: not specified git am ./20210713_xianting_tian_virtio_balloon_use_virtio_find_vqs_helper.mbx
$ b4 mbox [email protected] Looking up https://lore.kernel.org/r/bbe52a89-c7ea-c155-6226-0397f223cd80%40linux.alibaba.com 5 messages in the thread Saved ./[email protected]
That's with
Python 3.7.3
b4 v0.7.0-32-g45ef591
patatt v0.4.6
Quoted 25 lines · Kyle Meyer
Quoted 23 lines · Michael S. Tsirkin
Passing message id [email protected] to b4 gives this backtrace:
Traceback (most recent call last):
File "/scm/b4/b4/command.py", line 263, in <module>cmd()File "/scm/b4/b4/command.py", line 246, in cmd
cmdargs.func(cmdargs)
File "/scm/b4/b4/command.py", line 41, incmd_mbox
b4.mbox.main(cmdargs)
File "/scm/b4/b4/mbox.py", line 581, in main
msgid, msgs =get_msgs(cmdargs)
File "/scm/b4/b4/mbox.py", line 523, inget_msgs
msgid =b4.get_msgid(cmdargs)
File "/scm/b4/b4/__init__.py", line 2080, inget_msgid
msgid =get_msgid_from_stdin()
File "/scm/b4/b4/__init__.py", line 2072, inget_msgid_from_stdin
message = email.message_from_string(sys.stdin.read())
File "/usr/lib64/python3.9/codecs.py", line 322, in decode
(result, consumed) =self._buffer_decode(data, self.errors, final)
UnicodeDecodeError: 'utf-8' codec can't decode byte0xd4in position 5886: invalid continuation byte
Quoted 5 lines
Quoted 1 line
mutt does not seem to have trouble decoding this ... weird.
I'm confused by that backtrace. I think
get_msgid_from_stdin()should be called only when a message is fed on stdin. You say you're passing a message ID. That's as a positional argument, right?
Sorry. I passed the message on the stdin. I supplied the message ID so you can get the original from the list archives.
To reproduce:
wget -O - https://lore.kernel.org/lkml/[email protected]/raw | b4 mbox
Quoted 20 lines
Fwiw I wasn't able to trigger the issue on my end.
$ b4 am [email protected] Looking up https://lore.kernel.org/r/bbe52a89-c7ea-c155-6226-0397f223cd80%40linux.alibaba.com Analyzing 5 messages in the thread --- [PATCH] virtio-balloon: Use virtio_find_vqs() helper + Reviewed-by: David Hildenbrand <[email protected]> --- Total patches: 1 --- Link: https://lore.kernel.org/r/[email protected] Base: not specified git am ./20210713_xianting_tian_virtio_balloon_use_virtio_find_vqs_helper.mbx$ b4 mbox [email protected] Looking up https://lore.kernel.org/r/bbe52a89-c7ea-c155-6226-0397f223cd80%40linux.alibaba.com 5 messages in the thread Saved ./[email protected]
Quoted 5 lines
That's with
Python 3.7.3
b4 v0.7.0-32-g45ef591
patatt v0.4.6
b4 v0.7.0-32-g45ef591
python3-3.9.5-2.fc33.x86_64
I don't know about patatt.
Signature · 1 line
MST
cover Avoid decoding errors when extracting message ID from stdin
Quoted 21 lines · Michael S. Tsirkin
Quoted 14 lines · Kyle Meyer
Quoted 12 lines · Michael S. Tsirkin
Passing message id [email protected] to b4 gives this backtrace:
Traceback (most recent call last):
[....]
File "/scm/b4/b4/__init__.py", line 2072, inget_msgid_from_stdin
message = email.message_from_string(sys.stdin.read())
File "/usr/lib64/python3.9/codecs.py", line 322, in decode
(result, consumed) =self._buffer_decode(data, self.errors, final)
UnicodeDecodeError: 'utf-8' codec can't decode byte0xd4in position 5886: invalid continuation byteQuoted 5 lines
Quoted 1 line
mutt does not seem to have trouble decoding this ... weird.
I'm confused by that backtrace. I think
get_msgid_from_stdin()should be called only when a message is fed on stdin. You say you're passing a message ID. That's as a positional argument, right?
Quoted 6 lines
Sorry. I passed the message on the stdin. I supplied the message ID so you can get the original from the list archives.
To reproduce:
wget -O - https://lore.kernel.org/lkml/[email protected]/raw | b4 mbox
Thanks. I can trigger that on my end too. Here's a possible fix. The first patch is the actual fix. The second patch makes this code path do a little less work but isn't necessary.
[1/2] Avoid decoding errors when extracting message ID from stdin [2/2] Parse just headers when extracting message ID from stdin mbox
b4/__init__.py | 4 +++- b4/pr.py | 5 ++++- 2 files changed, 7 insertions(+), 2 deletions(-)
base-commit: 06cc7c8820aea85d1329911b785d7bf4ecaacb1f
Signature · 1 line
2.32.0
PATCH 2/2 Parse just headers when extracting message ID from stdin mbox
When the pr, mbox, and am subcommands grab a message ID from the mbox on stdin, they call message_from_bytes(), which in turn calls BytesParser().parsebytes(s).
parsebytes() has a headersonly parameter that can be used to tell it to stop parsing after reading the headers. The headers are all that's needed here, so use BytesParser directly and set headersonly.
---
Note: I've tested only b4 am/mbox' with the reproducer message mentioned in upthread; I haven't tested b4 pr'.
b4/__init__.py | 4 +++- b4/pr.py | 5 ++++- 2 files changed, 7 insertions(+), 2 deletions(-)
b4/__init__.py +3 -1
@@ -1948,7 +1948,9 @@ def get_requests_session(): def get_msgid_from_stdin(): if not sys.stdin.isatty(): - message = email.message_from_bytes(sys.stdin.buffer.read()) + from email.parser import BytesParser + message = BytesParser().parsebytes( + sys.stdin.buffer.read(), headersonly=True) return message.get('Message-ID', None) return None
b4/pr.py +4 -1
@@ -433,7 +433,10 @@ def main(cmdargs): if not sys.stdin.isatty(): logger.debug('Getting PR message from stdin') - msg = email.message_from_bytes(sys.stdin.buffer.read()) + from email.parser import BytesHeaderParser + from email.parser import BytesHeaderParser + msg = BytesParser().parsebytes( + sys.stdin.buffer.read(), headersonly=True) msgid = b4.LoreMessage.get_clean_msgid(msg) lmsg = parse_pr_data(msg) else:
Signature · 1 line
2.32.0
PATCH 1/2 Avoid decoding errors when extracting message ID from stdin
In reply to Kyle MeyerThe mbox, am, and pr subcommands accept an mbox on stdin and extract the message ID. When stdin.read() is called, Python assumes the encoding is locale.getpreferredencoding(False). This may not match the content encoding, leading to a decoding error.
Instead feed the stdin bytes to message_from_bytes(), which leads to a decode('ASCII', errors='surrogateescape') underneath. That's sufficient to get the message ID from the ASCII headers.
---
Note: I've tested only b4 am/mbox' with the reproducer message mentioned in upthread; I haven't tested b4 pr'.
b4/__init__.py | 2 +- b4/pr.py | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-)
b4/__init__.py +1 -1
@@ -1948,7 +1948,7 @@ def get_requests_session(): def get_msgid_from_stdin(): if not sys.stdin.isatty(): - message = email.message_from_string(sys.stdin.read()) + message = email.message_from_bytes(sys.stdin.buffer.read()) return message.get('Message-ID', None) return None
b4/pr.py +1 -1
@@ -433,7 +433,7 @@ def main(cmdargs): if not sys.stdin.isatty(): logger.debug('Getting PR message from stdin') - msg = email.message_from_string(sys.stdin.read()) + msg = email.message_from_bytes(sys.stdin.buffer.read()) msgid = b4.LoreMessage.get_clean_msgid(msg) lmsg = parse_pr_data(msg) else:
Signature · 1 line
2.32.0
Re: [PATCH b4 2/2] Parse just headers when extracting message ID from stdin mbox
In reply to Kyle Meyerb4/pr.py def main(cmdargs): · line 437
Quoted 11 lines · Kyle Meyer
b4/pr.py +2
-1@@ -433,7 +433,10 @@ def main(cmdargs): if not sys.stdin.isatty(): logger.debug('Getting PR message from stdin') - msg = email.message_from_bytes(sys.stdin.buffer.read()) + from email.parser import BytesHeaderParser + from email.parser import BytesHeaderParser
Doh, there's a repeated import here :/
Re: [PATCH b4 0/2] Avoid decoding errors when extracting message ID from stdin
In reply to Kyle MeyerQuoted 6 lines · Kyle Meyer
Thanks. I can trigger that on my end too. Here's a possible fix. The first patch is the actual fix. The second patch makes this code path do a little less work but isn't necessary.
[1/2] Avoid decoding errors when extracting message ID from stdin [2/2] Parse just headers when extracting message ID from stdin mbox
I've applied 1/2 and partial 2/2. In the case of b4/pr.py, we actually want to parse the entire message, not just headers.
Thanks!
-K